Interventions

Sometimes things don’t go to plan!

So Cylc allows you to take manual control of your workflow while it’s running allowing you to do things like edit a task’s configuration, re-run a section of your graph or override task outputs.

This section of the documentation covers some of the common interventions you might want to perform in various scenarios.

You can perform these interventions in multiple ways:

GUI

The Cylc web app (GUI) can be launched with the cylc gui command.

Some sites provide a central deployment for you where you log in via the web browser rather than launching it yourself.

With the Tui

Cylc Tui is an interactive in-terminal application, like a minimal version of the GUI. Start it with the cylc tui command.

Launch it with the cylc tui command.

On the CLI

The Cylc command line interface (CLI) can do everything the GUI can.

To see the full list of available commands run cylc help all.

Re-Run a Task

Example:

A task failed and I want to give it another try.

Solution:

Trigger the failed task.

GUI

Tui

CLI

$ cylc trigger <workflow>//<task>

Re-Run Multiple Tasks

Example:

I need to make changes to a task, then re-run it and everything downstream of it.

Solution:

Edit the tasks configuration, then trigger it in a new flow.

GUI

CLI

$ cylc trigger --flow=new <workflow>//<cycle>/<task>

Re-Run All Failed Tasks

Example:

Multiple tasks failed due to factors external to the workflow. I’ve fixed the problem, now I want them to run again.

Solution:

Trigger all failed tasks.

Note

You can use this intervention with other states too, e.g. “submit-failed”.

GUI

CLI

$ cylc trigger <workflow>:failed

Set Task Outputs

Example:

My task failed, I’ve gone and fixed the problem, now I want to tell Cylc that the task succeeded.

Solution:

Set the “succeeded” output on the failed task.

GUI

By default this sets the “succeeded” output, press the pencil icon next to the trigger command to specify a different output.

Tui

By default, this sets the “succeeded” output. Use cylc set --output to specify a different output.

CLI

$ cylc set <workflow>//<task>

By default, this sets the “succeeded” output. Use the --output option to specify a different output.

You can also set task prerequisites. Should I set task outputs or prerequisites?

• If you set a task’s outputs, then tasks downstream of it may start to run.

• If you set a task’s prerequisites, the task itself may start to run.

Set Task Prerequisites

Example:

A task is not running yet, because one or more of its prerequisites are not satisfied (e.g. upstream tasks have not run yet or have failed). I want it to ignore one or more of these prerequisites.

Solution:

Set the task’s prerequisites as satisfied.

Note

If you want the task to run right away, then trigger it using the same intervention as Re-Run a Task.

GUI

CLI

$ cylc set \
>   --pre <prereq-cycle>/<prereq-task>:<prereq-output> \
>   <workflow>//<cycle><task>

You can also set task outputs. Should I set task outputs or prerequisites?

• If you set a task’s outputs, then tasks downstream of it may start to run.

• If you set a task’s prerequisites, the task itself may start to run.

Set a Switch Task

Example:

I have a branched workflow, I want to direct it to take a particular path ahead of time:

start => switch

# by default, the workflow follows the "normal" path
switch:normal? => normal

# but it can follow an alternate route if desired
switch:alternate? => alternate

normal | alternate => end

Solution:

Set “succeeded” as well as the desired output(s) on the task and use the -wait option. Cylc will follow the desired path when it gets there.

Note

We set the “succeeded” output to prevent the task from being re-run when the flow approaches it.

GUI

CLI

$ cylc set --wait --output=succeeded,alternate <workflow>

Edit a Task’s Configuration and Re-Run It

Example:

I’m developing a task and need to be able to quickly make changes and re-run the task.

Solution:

Edit the task’s definition and trigger it.

GUI

Note

Any changes you make apply only to this one instance of the task, not to any future instances.

To change future instances, either use “broadcast” or see Edit The Workflow Configuration.

CLI

$ cylc broadcast <workflow> -p <cycle> -n <task> -s 'script=true'
$ cylc trigger <workflow>//<cycle>/<task>

Edit The Workflow Configuration

Example:

I want to change the configuration of multiple tasks or add/remove tasks without stopping the workflow or starting it from the beginning again.

Solution:

Edit the workflow configuration in the source directory, then reinstall and reload the workflow.

Tui

CLI

$ vim ~/cylc-src/myworkflow  # edit the workflow configuration
$ cylc vr myworkflow         # reinstall and reload the workflow

Note

For information on how graph changes (e.g. the adding or removing of tasks) are applied, see Restarting or Reloading after Graph Changes.

Orphan a “Stuck” Job Submission

Example:

I have a job which cannot be killed (e.g. the platform has gone down), I want Cylc to forget about it.

Solution:

Set the “failed” output on the task.

Use the same intervention as Set Task Outputs, but you will probably want to specify the “failed” output rather than “succeeded”.

Terminate a Chain of Automatic Retries

Example:

I have a failed task which has been configured to automatically retry, I want to cancel these retries because I know it can’t succeed.

Solution:

Set the “failed” output on the task.

Use the same intervention as Set Task Outputs, but specify the “failed” output rather than “succeeded”.

Hold The Workflow And Trigger Tasks One By One

Example:

I want to hold back the workflow while I manually run one or more tasks to fix a problem or test a task.

Solution:

• Hold all tasks after a specified cycle.

• Trigger the task(s) you want to run.

• When you’re done triggering, release the held tasks to allow the workflow to continue.

GUI

CLI

$ # hold all tasks after the cycle "2000"
$ cylc hold --after=2000 <workflow>

$ # trigger the task(s) you want to run
$ cylc trigger <workflow>//<cycle>/<task>

$ # release the "hold point" to allow the workflow to continue
$ cylc release --all <workflow>

Note

The difference between the workflow “paused” state and the task “held” state:

Workflow Pause

When a workflow is paused, no new jobs will be submitted. This gives you an opportunity to make changes to the workflow, however, if you trigger tasks, they will not run until the workflow is resumed.

Task Hold

When a task is held, then it will not submit (if ready to submit) until released.

I want to Skip a cycle of tasks and allow the workflow to continue

Example:

I want to skip a cycle (or group) of tasks and continue as if they had run and succeeded.

Solution:

Set the run mode of the tasks to skip and Cylc will pretend that they have run (very quickly).

GUI

CLI

cylc broadcast -p '<cycle>' -n root -s 'run mode = skip'

Note

-n root matches all tasks in a cycle. Similarly, it is possible to broadcast this setting to the root namespace of any family, or to multiple named tasks.