Workflow Configuration

The flow.cylc File

Cylc workflows are defined in flow.cylc files that specify the tasks to be managed by the Cylc scheduler, the dependencies between them, and the schedules to run time on.

General file syntax is described in the File Format Reference.

Legal configuration settings are documented in Workflow Configuration.

Cylc also supports two template processors for use in the flow.cylc file:

Workflow Configuration Directories

A Cylc source directory contains:

flow.cylc

The file which configures the workflow.

bin/ (optional)

A directory for scripts and executables used by workflow tasks. It is added to $PATH in job environments.

Jobs can also run scripting defined in the flow.cylc file, executables installed to user-defined locations of the workflow run directory, and external executables.

lib/python/ (optional)

A directory for Python modules. It is added to $PYTHONPATH in the scheduler and job execution environments. It can be used by:

Other files and folders may be placed in the source directory too: documentation, configuration files, etc. When the workflow is installed these will be copied over to the run directory.

Syntax Highlighting For Workflow Configuration

Cylc provides syntax plugins for the following editors:

Atom

install the language-cylc extension.

Emacs

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc-mode.el . installation instructions are at the top of the file.

Gedit

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.lang . installation instructions are at the top of the file.

Kate

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.xml . installation instructions are at the top of the file.

PyCharm

Install the Cylc.tmbundle.

Vim

The syntax file can be obtained from the Cylc library by running the following command cylc get-resources syntax/cylc.vim . installation instructions are at the top of the file.

Visual Studio Code

Install the vscode-cylc extension.

Sublime Text 3

Install the Cylc.tmbundle.

TextMate

Install the Cylc.tmbundle.

WebStorm

Install the Cylc.tmbundle.

Gross File Structure

Cylc flow.cylc files consist of configuration items grouped under several top level section headings:

[meta]

Information about the workflow e.g. title and description.

[scheduler]

Non task-specific workflow configuration.

[scheduling]

Determines when tasks are ready to run.

  • special scheduling constraints e.g. external triggers.

  • the dependency graph, which defines the relationships between tasks

[runtime]

Determines how, where, and what to execute when tasks are ready

  • script, environment, job submission, remote hosting, etc.

  • workflow-wide defaults in the root family

  • a nested family hierarchy with common properties inherited by related tasks

Validation

The cylc validate command evaluates the flow.cylc file against a specification that defines all legal entries, values and options. It also performs some integrity checks designed to catch certain configuration issues and impossible scheduling constraints.

These checks are also performed by cylc play before starting a workflow.

All legal entries are documented in the flow.cylc reference.

If a flow.cylc file uses include-files cylc view will show an inlined copy of the workflow with correct line numbers.

cylc lint

Cylc configuration linter.

Checks code style, deprecated syntax and other issues.

7 to 8 upgrades

U001

[runtime][__MANY__][suite state polling]template is not available at Cylc 8

U002

[runtime][__MANY__][events]reset timer is not available at Cylc 8

U003

[cylc][events]reset inactivity timer is not available at Cylc 8

U004

[cylc]force run mode is not available at Cylc 8

U005

[cylc][authentication] is not available at Cylc 8

U006

[cylc]include at start-up is not available at Cylc 8

U007

[cylc]exclude at start-up is not available at Cylc 8

U008

[cylc]log resolved dependencies is not available at Cylc 8

U009

[cylc]required run mode is not available at Cylc 8

U010

[cylc]health check interval is not available at Cylc 8

U011

[runtime][__MANY__][events]mail retry delays is not available at Cylc 8

U012

[runtime][__MANY__]extra log files is not available at Cylc 8

U013

[runtime][__MANY__][job]shell is not available at Cylc 8

U014

[runtime][__MANY__][remote]suite definition directory is not available at Cylc 8

U015

[cylc]abort if any task fails is not available at Cylc 8

U016

[cylc]disable automatic shutdown is not available at Cylc 8

U017

[cylc][environment] is not available at Cylc 8

U018

[cylc][reference test] is not available at Cylc 8

U019

[cylc][simulation]disable suite event handlers is not available at Cylc 8

U020

[cylc][simulation] is not available at Cylc 8

U021

[visualization] is not available at Cylc 8

U022

[scheduling]spawn to max active cycle points is not available at Cylc 8

U023

[cylc]task event mail interval is now [cylc][mail]task event batch interval

U024

[cylc][parameters] is now [task parameters]

U025

[cylc][parameter templates] is now [task parameters][templates]

U026

[runtime][__MANY__][events]mail to is now [runtime][__MANY__][mail]to

U027

[runtime][__MANY__][events]mail from is now [runtime][__MANY__][mail]from

U028

[cylc][events]mail footer is now [cylc][mail]footer

U029

[runtime][__MANY__][events]mail smtp is not available at Cylc 8

U030

[scheduling]max active cycle points is now [scheduling]runahead limit

U031

[scheduling]hold after point is now [scheduling]hold after cycle point

U032

[runtime][__MANY__][suite state polling] is now [runtime][__MANY__][workflow state polling]

U033

[cylc][events]timeout is now [cylc][events]stall timeout

U034

[cylc][events]abort on timeout is now [cylc][events]abort on stall timeout

U035

[cylc][events]inactivity is now [cylc][events]inactivity timeout

U036

[cylc][events]abort on inactivity is now [cylc][events]abort on inactivity timeout

U037

[cylc][events]startup handler is now [cylc][events]startup handlers

U038

[cylc][events]shutdown handler is now [cylc][events]shutdown handlers

U039

[cylc][events]timeout handler is now [cylc][events]stall timeout handlers

U040

[cylc][events]stalled handler is now [cylc][events]stall handlers

U041

[cylc][events]aborted handler is now [cylc][events]abort handlers

U042

[cylc][events]inactivity handler is now [cylc][events]inactivity timeout handlers

U043

[runtime][__MANY__][events]expired handler is now [runtime][__MANY__][events]expired handlers

U044

[runtime][__MANY__][events]late handler is now [runtime][__MANY__][events]late handlers

U045

[runtime][__MANY__][events]submitted handler is now [runtime][__MANY__][events]submitted handlers

U046

[runtime][__MANY__][events]started handler is now [runtime][__MANY__][events]started handlers

U047

[runtime][__MANY__][events]succeeded handler is now [runtime][__MANY__][events]succeeded handlers

U048

[runtime][__MANY__][events]failed handler is now [runtime][__MANY__][events]failed handlers

U049

[runtime][__MANY__][events]submission failed handler is now [runtime][__MANY__][events]submission failed handlers

U050

[runtime][__MANY__][events]warning handler is now [runtime][__MANY__][events]warning handlers

U051

[runtime][__MANY__][events]critical handler is now [runtime][__MANY__][events]critical handlers

U052

[runtime][__MANY__][events]retry handler is now [runtime][__MANY__][events]retry handlers

U053

[runtime][__MANY__][events]submission retry handler is now [runtime][__MANY__][events]submission retry handlers

U054

[runtime][__MANY__][events]execution timeout handler is now [runtime][__MANY__][events]execution timeout handlers

U055

[runtime][__MANY__][events]submission timeout handler is now [runtime][__MANY__][events]submission timeout handlers

U056

[runtime][__MANY__][events]custom handler is now [runtime][__MANY__][events]custom handlers

U057

[cylc][events]abort on stalled is not available at Cylc 8

U058

[cylc][events]abort if startup handler fails is not available at Cylc 8

U059

[cylc][events]abort if shutdown handler fails is not available at Cylc 8

U060

[cylc][events]abort if timeout handler fails is not available at Cylc 8

U061

[cylc][events]abort if inactivity handler fails is not available at Cylc 8

U062

[cylc][events]abort if stalled handler fails is not available at Cylc 8

U063

[cylc] is now [scheduler]

U064

[scheduling][dependencies][<recurrence>]graph = -> [scheduling][graph]<recurrence> =

U065

[scheduling][dependencies][<recurrence>]graph = -> [scheduling][graph]<recurrence> =

U066

settings in [runtime][<namespace>][remote] have been moved to [runtime][<namespace>] and global.cylc[platforms][<platforms name>]

U067

settings in [runtime][<namespace>][job] have been moved to [runtime][<namespace>] and global.cylc[platforms][<platforms name>]

U068

flow.cylc[runtime][<namespace>][job]batch system -> global.cylc[platforms][<platform name>]job runner

U069

Using backticks to invoke subshell will fail at Cylc 8.

Style

S001

Use multiple spaces, not tabs

S002

Item not indented.

S003

Top level sections should not be indented.

S004

Second level sections should be indented exactly 4 spaces.

S005

Third level sections should be indented exactly 8 spaces.

S006

trailing whitespace.

S007

Family name contains lowercase characters.

S008

jinja2 found: no shebang (#!jinja2)

S009

Host Selection Script may be redundant with platform

S010

Using backticks to invoke subshell is deprecated

S011

Cylc will process commented Jinja2!

S008

line > <max_line_len> characters. Max line length set in pyproject.toml (default 130)

Configure cylc lint at project level

You can configure cylc lint for each workflow using a pyproject.toml file.

To define settings for cylc lint use a [cylc-lint] section. Within the [cylc-lint] section you may define the following:

rulesets

A list of rulesets to use. If you run cylc lint without setting rulesets on the command line this value will override the default (['728', 'style']).

Allowed Values: ‘728’, ‘style’

ignore

Individual rules to ignore: A list of rule codes, such as S007.

exclude

A list of files or glob patterns for files which will not be checked.

max-line-length

Set longest line length to permit in Cylc Configs for this project. If unset, line length is not checked.

An example pyproject.toml might look like this:

[cylc-lint]
# Enforce a line limit of 99 chars
max-line-length = 99

# Ignore style [S] rule 007 (It's good practice comment with a reason)
ignore = ['S007']   # Family names start with lowercase in this workflow

# Don't check files matching these globs
exclude = ['history/*.old.cylc', 'someother.cylc']

# By default check for style
rulesets = ['style']

[some-other-section]
# Cylc lint won't pay any attention to this.