Message Triggers

Message triggers allow us to trigger dependent tasks before the upstream task has completed.

Explanation

We have seen before that tasks can have qualifiers for different task states. Message triggers are essentially custom qualifiers. We can produce a bespoke output while our task is still running. This output could be, for example, a report or perhaps another task.

Usage

Message triggers are particularly useful if we have a long running task and we want to produce multiple tailored outputs whilst this task is running, rather than having to wait for the task to complete.

We could also set up message triggers to, for example, send an email to inform us that a submission has failed, making use of Cylc’s task event handling system. More information is available on these in the Cylc User Guide.

Message triggers provide a superior solution to the problem of file system polling. We could, for example, design our workflow such that we check if our task is finished by polling at intervals. It is inefficient to ‘spam’ task hosts with polling commands, it is preferable to set up a message trigger.

How to create a message trigger

In order to get our workflow to trigger messages, we need to:

specify our custom message in a section called [[outputs]] within the
[runtime] section of our workflow,
add cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" "YOUR CHOSEN TRIGGER MESSAGE"
to the script section of [runtime], your chosen trigger message should be unique and should exactly match the message defined in [[outputs]].
Refer to these messages in the [dependencies] section of our workflow.

Note

The message will be recorded in the workflow’s scheduler log. See Cylc Message for details of how messages appear.

These outputs are then triggered during the running of the task. We can use these to manage tasks dependent on partially completed tasks.

So, a basic example, where we have a task foo, that when partially completed triggers another task bar and when fully completed triggers another task, baz.

[scheduling]
    [[graph]]
        R1 = """
            foo:out1 => bar
            foo => baz
        """
[runtime]
    [[foo]]
        script = """
            sleep 5
            cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" "file 1 done"
            sleep 10
        """
        [[[outputs]]]
            out1 = "file 1 done"

    [[bar, baz]]
        script = sleep 10

Practical

In this practical example, we will create a workflow to demonstrate message triggers. We will use message triggers to both produce a report and trigger a new task from a partially completed task.

Create a new directory.

Within your ~/cylc-src directory create a new directory called

message-triggers and move into it:
```
mkdir ~/cylc-src/message-triggers
cd ~/cylc-src/message-triggers
```
Install the script needed for our workflow

The workflow we will be designing requires a bash script, random.sh, to produce our report. It will simply create a text file report.txt with some random numbers in it. This will be executed when the associated task is run.

Scripts should be kept in the bin sub-directory within the run directory. If a /bin exists in the run directory, it will be prepended $PATH at run time.

Create a /bin directory.
```
mkdir ~/cylc-src/message-triggers/bin
```
Create a bash script in the bin directory:
```
touch bin/random.sh
```
We will need to make this script executable.
```
chmod +x bin/random.sh
```
Open the file and paste the following basic bash script into it:
```
#!/usr/bin/env bash
set -eu  # Prevent bash script failing quietly.

counter=1

while [ $counter -le 10 ]; do
    newrand=$(( (( RANDOM % 40) + 1 ) ));
    echo $newrand >> report.txt;
    counter=$((counter + 1));
done
```

Create a new workflow.

Create a flow.cylc file and paste the following basic workflow into it:

[meta]
    title = "test workflow to demo message triggers"

[scheduler]
    UTC mode = True

[scheduling]
    initial cycle point = 2019-06-27T00Z
    final cycle point = 2019-10-27T00Z
    [[graph]]
        P2M = """
            long_forecasting_task =>  another_weather_task
            long_forecasting_task => different_weather_task
            long_forecasting_task[-P2M] => long_forecasting_task
        """

This is a basic workflow, currently it does not have any message triggers attached to any task.

Define our tasks in the runtime section.

Next we want to create our runtime section of our workflow. First we define what the tasks do. In this example long_forecasting_task will sleep, create a file containing some random numbers and produce a message. (Note that the random number generator bash script has already been preloaded into your bin directory.) another_weather_task and different_weather_task simply sleep.

Add the following code to the flow.cylc file.
```
[runtime]

    [[long_forecasting_task]]
        script = """
            sleep 2
            random.sh

            sleep 2
            random.sh

            sleep 2
            random.sh
        """

    [[another_weather_task, different_weather_task]]
        script = sleep 1
```

Create message triggers.

We now have a workflow with a task, long_forecasting_task which, after it has fully completed, triggers two more tasks, another_weather_task and different_weather_task.

Suppose we want another_weather_task and different_weather_task to start before long_forecasting_task has fully completed, perhaps after some data has become available.

In this case, we shall trigger another_weather_task after one set of random numbers has been created and different_weather_task after a second set of random numbers has been created.

There are three aspects of creating message triggers. The first is to create the messages. Within runtime, TASK in our workflow, we need to create a sub-section called outputs. Here we create our custom outputs.

+        [[[outputs]]]
+            update1 = "Task partially complete, report ready to view"
+            update2 = "Task partially complete, report updated"

The second thing we need to do is to create a cylc message in our script. This should be placed where you want the message to be called. In our case, this is after each of the first two set of random numbers are generated.

Tip

Remember that the cylc message should exactly match the outputs stated in our [[[outputs]]] section.

Modify the [[long_forecasting_task]] script in the flow.cylc file as follows:

[runtime]

    [[long_forecasting_task]]
        script = """
            sleep 2
            random.sh
+           cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
+                "Task partially complete, report ready to view"
            sleep 2
            random.sh
+           cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
+               "Task partially complete, report updated"
            sleep 2
            random.sh
        """

Lastly, we need to make reference to the messages in the graph section. This will ensure your tasks trigger off of the messages correctly.

Adapt the [[dependencies]] section in the flow.cylc file to read as follows:

         [[[P2M]]]
             graph = """
-               long_forecasting_task =>  another_weather_task
-               long_forecasting_task => different_weather_task
+               long_forecasting_task:update1 =>  another_weather_task
+               long_forecasting_task:update2 => different_weather_task
                long_forecasting_task[-P2M] => long_forecasting_task
            """

This completes our flow.cylc file.

Our final workflow should look like this:

Solution

[meta]
    title = "test workflow to demo message triggers"

[scheduler]
    UTC mode = True

[scheduling]
    initial cycle point = 2019-06-27T00Z
    final cycle point = 2019-10-27T00Z

    [[graph]]
        P2M = """
            long_forecasting_task:update1 =>  another_weather_task
            long_forecasting_task:update2 => different_weather_task
            long_forecasting_task[-P2M] => long_forecasting_task
        """

[runtime]
    [[long_forecasting_task]]
        script = """
            sleep 2
            random.sh
            cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
                "Task partially complete, report ready to view"
            sleep 2
            random.sh
            cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
                "Task partially complete, report updated"
            sleep 2
            random.sh
        """
        [[[outputs]]]
            update1 = "Task partially complete, report ready to view"
            update2 = "Task partially complete, report updated"

    [[another_weather_task, different_weather_task]]
        script = sleep 1

Validate the workflow.

It is a good idea to check that our flow.cylc file does not have any configuration issues.

Run cylc validate to check for any errors:
```
cylc validate .
```
Install and Play the workflow.

Now we are ready to run our workflow. Validate, install, then open the GUI or TUI and play the workflow.
```
cylc validate .
cylc install
cylc play message-triggers
```
Your workflow should now run, the tasks should succeed.
Inspect the work directory.

You can now check for your report outputs. These should appear in the work directory of the workflow. All being well, our first cycle point should produce a test file with some random numbers, and each subsequent cycle point file should have more random numbers added.

Extension.

Suppose now we would like to send an email alerting us to the reports being ready to view.

We will need to add to our flow.cylc file.

In the runtime section, add a sub-section called [[[events]]]. Within this section we will make use of the built-in setting mail events. Here, we specify a list of events for which notifications should be sent.

The events we are interested in are, in this case, our outputs.

Add the following code to your [[[events]]] section.

[[[events]]]
    mail events = update1, update2
Our updated workflow should look like this:

Solution

[scheduler]
    UTC mode = True
[meta]
    title = "test workflow to demo message triggers"
[scheduling]
    initial cycle point = 2019-06-27T00Z
    final cycle point = 2019-10-27T00Z
    [[graph]]
        P2M = """
            long_forecasting_task:update1 =>  another_weather_task
            long_forecasting_task:update2 => different_weather_task
            long_forecasting_task[-P2M] => long_forecasting_task
        """
[runtime]
    [[long_forecasting_task]]
        script = """
            sleep 2
            random.sh
            cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
                "Task partially complete, report ready to view"
            sleep 2
            random.sh
            cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" \
                "Task partially complete, report updated"
            sleep 2
            random.sh
        """

        [[[outputs]]]
            update1 = "Task partially complete, report ready to view"
            update2 = "Task partially complete, report updated"

        [[[events]]]
            mail events = update1, update2

    [[another_weather_task, different_weather_task]]
        script = sleep 1

Save your changes and run your workflow. Check your emails and you should have, one email for the first update and, a second email alerting you to the subsequent updated reports being ready.

Note that the second email automatically bundles the messages to prevent your inbox from being flooded.