Event Driven Cycling

Get a copy of this example

$ cylc get-resources examples/event-driven-cycling

Cylc is good at orchestrating tasks to a schedule, e.g:

• PT1H - every hour

• P1D - every day

• P1M - every month

• PT1H ! (T00, T12) - every hour, except midnight and midday.

But sometimes the things you want to run don’t have a schedule.

This example uses cylc ext-trigger to establish a pattern where Cylc waits for an external signal and starts a new cycle every time a signal is received.

The signal can carry data using the ext-trigger ID, this example sets the ID as a file path containing some data that we want to make available to the tasks that run in the cycle it triggers.

To use this example, first start the workflow as normal:

cylc vip event-driven-cycling

Then, when you’re ready, kick off a new cycle, specifying any environment variables you want to configure this cycle with:

./bin/trigger <workflow-id> WORLD=earth

Replacing <workflow-id> with the ID you installed this workflow as.

Example - CI/CD

This pattern is good for CI/CD type workflows where you’re waiting on external events. This pattern is especially powerful when used with sub-workflows where it provides a solution to two-dimensional cycling problems.

Example - Polar satellite data processing

Polar satellites pass overhead at irregular intervals. This makes it tricky to schedule data processing because you don’t know when the satellite will pass over the receiver station. With the event driven cycling approach you could start a new cycle every time data arrives.

Note

• The number of parallel cycles can be adjusted by changing the [scheduling]runahead limit.

• To avoid hitting the runahead limit, ensure that failures are handled in the graph.

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    runahead limit = P5  # max number of cycles which can run in parallel
    [[special tasks]]
        # register the external trigger, it must be given a name,
        # here, 'trigger' is used as a placeholder, the bash script will
        # need to be updated if this is changed
        external-trigger = configure("trigger")
    [[graph]]
        P1 = """
            # use a "?" to prevent failures causing runahead stalls
            configure? => run
        """

[runtime]
    [[configure]]
        # this task reads in the broadcast file the trigger wrote
        # and broadcasts any variables set to all tasks in this cycle
        script = """
            echo "received new ext-trigger ID=$CYLC_EXT_TRIGGER_ID"
            TRIGGER_FILE="${CYLC_WORKFLOW_RUN_DIR}/triggers/${CYLC_EXT_TRIGGER_ID}.cylc"
            cylc broadcast "${CYLC_WORKFLOW_ID}" \
                -p "${CYLC_TASK_CYCLE_POINT}" \
                -F "${TRIGGER_FILE}"
        """

    [[run]]
        # this task could be a sub-workflow
        script = """
            echo "Hello $WORLD!"
        """