Workflow Configuration

The flow.cylc File

Cylc workflows are defined in flow.cylc files that specify the tasks to be managed by the Cylc scheduler, the dependencies between them, and the schedules to run them to.

The .cylc file syntax is described in the File Format Reference.

The configurations you can use are documented in Workflow Configuration.

Cylc also supports the Jinja2 template processor for use in the flow.cylc file.

Workflow Configuration Directories

A Cylc source directory contains:

flow.cylc

The file which configures the workflow.

bin/ (optional)

A directory for scripts and executables used by workflow tasks. It is added to $PATH in job environments.

Jobs can also run scripting defined in the flow.cylc file, executables installed to user-defined locations of the workflow run directory, and external executables.

lib/python/ (optional)

A directory for Python modules. It is added to $PYTHONPATH in the scheduler and job execution environments. It can be used by:

Other files and folders

Such as documentation, configuration files, etc.

When the workflow is installed these will be copied over to the run directory.

Note

If your workflow needs to create or install scripts or executables at runtime and you don’t want Cylc to delete them on re-installation, you can use equivalent directories in the Workflow Share Directories.

Understanding Variables and Code in Workflow Configurations

The flow.cylc file is a static configuration file that configures an instance of the Cylc scheduler at startup, to manage your workflow.

The .cylc file format itself does not support variables and loop constructs etc., but Jinja2 templating code can be embedded anywhere in the file, and string-valued script and [environment] items, which are written to task job scripts, contain Bash code that will be evaluated when jobs run.

Cylc uses the Jinja2 templating engine as a preprocessor, to allow programmatic generation of workflow configurations. The result after Jinja2 preprocessing must be a valid plain workflow configuration. Jinja2 preprocessing is done first by all file-parsing commands such as cylc validate, and at scheduler start-up on the run host.

To see the result of Jinja2 template preprocessing:

# print the workflow configuration, processed but not parsed:
$ cylc view --process <workflow-id>

# print the workflow configuration, processed and parsed:
$ cylc config <workflow-id>

Here’s an example of a simple Jinja2 template:

#!Jinja2
{% set tasks = ["cat", "dog", "fish"] %}
[scheduling]
    [[graph]]
        R1 = """
{% for task in tasks %}
           start => {{task}}
{% endfor %}
        """
[runtime]
    [[start]]
       script = "echo 'hello'"
{% for task in tasks %}
    [[{{task}}]]
        script = "echo 'I am a {{task}}'"
{% endfor %}

And the resulting workflow configuration:

[scheduling]
    [[graph]]
        R1 = """
            start => cat
            start => dog
            start => fish
        """
[runtime]
    [[start]]
        script = "echo 'hello'"
    [[cat]]
        script = "echo 'I am a cat'"
    [[dog]]
        script = "echo 'I am a dog'"
    [[fish]]
        script = "echo 'I am a fish'"

These uses of a Jinja2 variable result in a valid configuration:

{% set TASK = "model" %}
[runtime]
    # This generates a valid task definition heading "[[model]]":
    [[{{TASK}}]]
        # This generates: script = 'echo "I am a model"'
        script = 'echo "I am a {{TASK}}"'

This example shows valid use of shell variables:

[runtime]
    [[model]]
        # The "script" item gets written verbatim to job scripts so
        # Bash will evaluate $OWNER (and $USER) when the job runs:
        script = 'echo "I am $OWNER"'
        [[[environment]]]
            OWNER = "${USER}-the-wizard"

And this shows bad use of a shell variable:

[runtime]
    # ERROR: task definition headings are not evaluated by the shell
    # and the literal "[[${USER}-the-wizard]]" is not valid as such.
    [[${USER}-the-wizard]]

Jinja2 code can read the local environment (or the local filesystem, via custom Python functions) during template processing - i.e., when and where the file is parsed. Jinja2 has its own syntax for reading environment variables:

{# This evaluates $USER when the file is parsed. #}
{% set OWNER = environ["USER"] ~ "-the-wizard" %}
[runtime]
    # For Bob, this generates task definition "[[bob-the-wizard]]":
    [[{{OWNER}}]]

Warning

Jinja2 code gets evaluated when the file is parsed, so embedding it in task scripting does not mean it will execute on the task job host at task run time. Rather, the job will run the scripting that was generated by template preprocessing at scheduler start-up on the scheduler run host.

Here’s an example of valid use of Jinja2 to generate a string that references an environment variable, for use in a task job script:

{# This defines a string literal "${USER}-the-wizard": #}
{% set OWNER = "${USER}-the-wizard" %}
[runtime]
    [[model]]
        # This generates:
        #   script = 'echo "my owner is ${USER}-the-wizard"'
        # which Bash will interpret when the job runs:
        script = 'echo "my owner is {{OWNER}}"'

And here’s an example of bad use of a Jinja2-generated string that references a task environment variable:

{# This does not evaluate $USER, it's just a string literal #}
{% set OWNER = "${USER}-the-wizard" %}
[runtime]
    # ERROR: literal "${USER}-the-wizard" is not a valid task name:
    [[{{OWNER}}]]

Finally, whether or not it makes sense to pass shell variables (or rather, strings containing them) to a Jinja2 macro depends on what the macro does with its arguments. As illustrated above, valid task names can’t be generated that way (because shell variables won’t be evaluated in that context) but task scripting can be. If in doubt, view the processed result with cylc view -p.

See also

Jinja2

Syntax Highlighting For Workflow Configuration

Cylc provides syntax plugins for the following editors:

Atom & Pulsar

Install the language-cylc extension.

Emacs

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc-mode.el . installation instructions are at the top of the file.

Gedit

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.lang . installation instructions are at the top of the file.

Helix

Built-in support.

Kate

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.xml . installation instructions are at the top of the file.

Nano

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.nanorc ~/.config/nano installation instructions are at the top of the file.

NeoVim

Install the cylc.vim plugin.

PyCharm

Install the Cylc.tmbundle.

Vim

Install the cylc.vim plugin.

Visual Studio Code

Install the vscode-cylc extension.

Sublime Text 3

Install the Cylc.tmbundle.

TextMate

Install the Cylc.tmbundle.

WebStorm

Install the Cylc.tmbundle.

Zed

Install the zed-cylc extension.

Gross File Structure

Cylc flow.cylc files consist of configuration items grouped under several top level section headings:

[meta]

Information about the workflow e.g. title and description.

[scheduler]

Non task-specific workflow configuration.

[task parameters]

Parameters for use when defining graphs and tasks. See Task Parameters.

[scheduling]

Determines when tasks are ready to run.

  • special scheduling constraints e.g. external triggers.

  • the dependency graph, which defines the relationships between tasks

[runtime]

Determines how, where, and what to execute when tasks are ready

  • script, environment, job submission, remote hosting, etc.

  • workflow-wide defaults in the root family

  • a nested family hierarchy with common properties inherited by related tasks

Validation

The cylc validate command evaluates the flow.cylc file against a specification that defines all legal entries, values and options. It also performs some integrity checks designed to catch certain configuration issues and impossible scheduling constraints.

These checks are also performed by cylc play before starting a workflow.

All legal entries are documented in the flow.cylc reference.

If a flow.cylc file uses include-files cylc view will show an inlined copy of the workflow with correct line numbers.

Linting

The cylc lint command checks code style, deprecated syntax and other issues in Cylc configuration files.

See also

How to configure Cylc lint at project level using a pyproject.toml.

cylc lint

Cylc configuration linter.

Checks code style, deprecated syntax and other issues.

Individual errors can be ignored using the # noqa line comment. It is good practice to specify specific errors you wish to ignore using # noqa: S002 S007 U999

Style

S001

Use multiple spaces, not tabs

S002

Item not indented.

S003

Top level sections should not be indented.

S004

Second level sections should be indented exactly 4 spaces.

S005

Third level sections should be indented exactly 8 spaces.

S006

trailing whitespace.

S007

Family name contains lowercase characters.

S008

jinja2 found: no shebang (#!jinja2)

S009

Host Selection Script may be redundant with platform

S010

Using backticks to invoke subshell is deprecated

S011

Cylc will process commented Jinja2!

S012

line > <max_line_len> characters. Max line length set in pyproject.toml (default 130)

S013

Items should be indented in 4 space blocks.

S014

Use flow.cylc[runtime][<namespace>]execution time limit rather than directly specifying a timeout directive, otherwise Cylc has no way of knowing when the job should have finished. Cylc automatically translates the execution time limit to the correct timeout directive for the particular job runner.

See Task Execution Time Limits.

S015

=> implies line continuation without \.

S016

Task outputs {outputs}: {description}.

S017

Run mode is not live: This task will only appear to run.

7 to 8 upgrades

U001

[scheduling][dependencies][<recurrence>]graph = -> [scheduling][graph]<recurrence> = ([dependencies] detected)

U002

[scheduling][dependencies][<recurrence>]graph = -> [scheduling][graph]<recurrence> = (graph = detected)

U003

settings in [runtime][<namespace>][remote] have been moved to [runtime][<namespace>] and global.cylc[platforms][<platforms name>]

U004

settings in [runtime][<namespace>][job] have been moved to [runtime][<namespace>] and global.cylc[platforms][<platforms name>]

U005

flow.cylc[runtime][<namespace>][job]batch system -> global.cylc[platforms][<platform name>]job runner

U006

Using backticks to invoke subshell will fail at Cylc 8.

U007

Use built in platform selection instead of rose host-select.

U009

This line contains an obsolete Cylc CLI command.

U010

rose suite-hook is deprecated at Rose 2,

U011

Leading zeros are no longer valid for Jinja2 integers.

U012

The following environment variables are deprecated:

Deprecated Variable

New Variable

CYLC_SUITE_HOST

CYLC_WORKFLOW_HOST

CYLC_SUITE_OWNER

CYLC_WORKFLOW_OWNER

CYLC_SUITE_SHARE_DIR

CYLC_WORKFLOW_SHARE_DIR

CYLC_SUITE_SHARE_PATH

CYLC_WORKFLOW_SHARE_PATH

CYLC_SUITE_NAME

CYLC_WORKFLOW_ID

CYLC_SUITE_LOG_DIR

CYLC_WORKFLOW_LOG_DIR

CYLC_SUITE_INITIAL_CYCLE_POINT

CYLC_WORKFLOW_INITIAL_CYCLE_POINT

CYLC_SUITE_INITIAL_CYCLE_TIME

CYLC_WORKFLOW_INITIAL_CYCLE_TIME

CYLC_SUITE_FINAL_CYCLE_POINT

CYLC_WORKFLOW_FINAL_CYCLE_POINT

CYLC_SUITE_FINAL_CYCLE_TIME

CYLC_WORKFLOW_FINAL_CYCLE_TIME

CYLC_SUITE_WORK_DIR

CYLC_WORKFLOW_WORK_DIR

CYLC_SUITE_UUID

CYLC_WORKFLOW_UUID

CYLC_SUITE_RUN_DIR

CYLC_WORKFLOW_RUN_DIR
U013

The following environment variables are obsolete:

  • CYLC_SUITE_DEF_PATH

  • CYLC_SUITE_DEF_PATH_ON_SUITE_HOST

U014

For datetime operations in task scripts:

  • Use isodatetime instead of rose date

  • Use isodatetime ref instead of rose date -c for the current cycle point

U015

The following deprecated template variables, mostly used in event handlers, should be replaced:

  • suite ⇒ workflow

  • suite_uuid ⇒ uuid

  • batch_sys_name ⇒ job_runner_name

  • batch_sys_job_id ⇒ job_id

  • user@host ⇒ platform_name

  • task_url ⇒ URL (if set in [meta]URL)

  • workflow_url ⇒ workflow_URL (if set in [runtime][<namespace>][meta]URL)

U016

It is no longer necessary to configure the environment variables CYLC_VERSION, ROSE_VERSION or FCM_VERSION.

U017

& and | imply line continuation without \

U018

clock-trigger is deprecated, please use @wall_clock

Auto Generated 7 to 8 upgrades

U998 and U999 represent automatically generated sets of deprecations and upgrades.

  • [runtime][__MANY__][workflow state polling]run-dir is now [runtime][__MANY__][workflow state polling]alt-cylc-run-dir

  • [cylc][authentication] is not available at Cylc 8

  • [cylc]include at start-up is not available at Cylc 8

  • [cylc]exclude at start-up is not available at Cylc 8

  • [cylc]log resolved dependencies is not available at Cylc 8

  • [cylc]required run mode is not available at Cylc 8

  • [cylc]health check interval is not available at Cylc 8

  • [runtime][__MANY__][events]mail retry delays is not available at Cylc 8

  • [runtime][__MANY__]extra log files is not available at Cylc 8

  • [runtime][__MANY__][job]shell is not available at Cylc 8

  • [runtime][__MANY__][remote]suite definition directory is not available at Cylc 8

  • [cylc]abort if any task fails is not available at Cylc 8

  • [cylc]disable automatic shutdown is not available at Cylc 8

  • [cylc][environment] is not available at Cylc 8

  • [cylc][reference test] is not available at Cylc 8

  • [cylc][simulation]disable suite event handlers is not available at Cylc 8

  • [cylc][simulation] is not available at Cylc 8

  • [visualization] is not available at Cylc 8

  • [scheduling]spawn to max active cycle points is not available at Cylc 8

  • [cylc]task event mail interval is now [cylc][mail]task event batch interval

  • [runtime][__MANY__][suite state polling] is now [runtime][__MANY__][workflow state polling]

  • [runtime][__MANY__][workflow state polling]host is not available at Cylc 8

  • [runtime][__MANY__][workflow state polling]user is not available at Cylc 8

  • [cylc][parameters] is now [task parameters]

  • [cylc][parameter templates] is now [task parameters][templates]

  • [cylc][events]mail to is now [cylc][mail]to

  • [cylc][events]mail from is now [cylc][mail]from

  • [cylc][events]mail footer is now [cylc][mail]footer

  • [runtime][__MANY__][events]mail to is now [runtime][__MANY__][mail]to

  • [runtime][__MANY__][events]mail from is now [runtime][__MANY__][mail]from

  • [cylc][events]mail smtp is not available at Cylc 8

  • [runtime][__MANY__][events]mail smtp is not available at Cylc 8

  • [scheduling]max active cycle points is now [scheduling]runahead limit

  • [scheduling]hold after point is now [scheduling]hold after cycle point

  • [cylc][events]timeout is now [cylc][events]stall timeout

  • [cylc][events]abort on timeout is now [cylc][events]abort on stall timeout

  • [cylc][events]inactivity is now [cylc][events]inactivity timeout

  • [cylc][events]abort on inactivity is now [cylc][events]abort on inactivity timeout

  • [cylc][events]startup handler is now [cylc][events]startup handlers

  • [cylc][events]shutdown handler is now [cylc][events]shutdown handlers

  • [cylc][events]timeout handler is now [cylc][events]stall timeout handlers

  • [cylc][events]stalled handler is now [cylc][events]stall handlers

  • [cylc][events]aborted handler is now [cylc][events]abort handlers

  • [cylc][events]inactivity handler is now [cylc][events]inactivity timeout handlers

  • [runtime][__MANY__][events]expired handler is now [runtime][__MANY__][events]expired handlers

  • [runtime][__MANY__][events]late handler is now [runtime][__MANY__][events]late handlers

  • [runtime][__MANY__][events]submitted handler is now [runtime][__MANY__][events]submitted handlers

  • [runtime][__MANY__][events]started handler is now [runtime][__MANY__][events]started handlers

  • [runtime][__MANY__][events]succeeded handler is now [runtime][__MANY__][events]succeeded handlers

  • [runtime][__MANY__][events]failed handler is now [runtime][__MANY__][events]failed handlers

  • [runtime][__MANY__][events]submission failed handler is now [runtime][__MANY__][events]submission failed handlers

  • [runtime][__MANY__][events]warning handler is now [runtime][__MANY__][events]warning handlers

  • [runtime][__MANY__][events]critical handler is now [runtime][__MANY__][events]critical handlers

  • [runtime][__MANY__][events]retry handler is now [runtime][__MANY__][events]retry handlers

  • [runtime][__MANY__][events]submission retry handler is now [runtime][__MANY__][events]submission retry handlers

  • [runtime][__MANY__][events]execution timeout handler is now [runtime][__MANY__][events]execution timeout handlers

  • [runtime][__MANY__][events]submission timeout handler is now [runtime][__MANY__][events]submission timeout handlers

  • [runtime][__MANY__][events]custom handler is now [runtime][__MANY__][events]custom handlers

  • [cylc][events]abort on stalled is not available at Cylc 8

  • [cylc][events]abort if startup handler fails is not available at Cylc 8

  • [cylc][events]abort if shutdown handler fails is not available at Cylc 8

  • [cylc][events]abort if timeout handler fails is not available at Cylc 8

  • [cylc][events]abort if inactivity handler fails is not available at Cylc 8

  • [cylc][events]abort if stalled handler fails is not available at Cylc 8

  • [cylc] is now [scheduler]

Configure cylc lint at project level

You can configure cylc lint for each workflow using a pyproject.toml file.

To define settings for cylc lint use a [tool.cylc.lint] section. Within the [tool.cylc.lint] section you may define the following:

rulesets

A list of rulesets to use.

Allowed values: '728', 'style', 'all'.

(You can override this on the command line.)

ignore

Individual rules to ignore: A list of rule codes, such as S007.

exclude

A list of files or glob patterns for files which will not be checked.

max-line-length

Set longest line length to permit in Cylc configs for this project.

Default: 130.

Note

Changed in version 8.8.0: Support for the legacy [cylc-lint] section has been removed. Use [tool.cylc.lint] instead.

An example pyproject.toml might look like this:

[tool.cylc.lint]
# Enforce a line limit of 99 chars
max-line-length = 99

# Ignore style [S] rule 007 (It's good practice comment with a reason)
ignore = ['S007']   # Family names start with lowercase in this workflow

# Don't check files matching these globs
exclude = ['history/*.old.cylc', 'someother.cylc']

# By default check for style
rulesets = ['style']

[some-other-section]
# Cylc lint won't pay any attention to this.