The Cylc Suite Engine
User Guide
7.4.0
GNU GPL v3.0 Software License
Copyright (C) 2008-2017 NIWA

Hilary Oliver

May 16, 2017
PIC
PIC

Contents

1 Introduction: How Cylc Works
2 Cylc Screenshots
3 Installation
4 Cylc Terminology
5 Workflows For Cycling Systems
6 Global (Site, User) Configuration Files
7 Tutorial
8 Suite Name Registration
9 Suite Definition
10 Task Implementation
11 Task Job Submission and Management
12 Running Suites
13 Suite Storage, Discovery, Revision Control, and Deployment
A Suite.rc Reference
B Global (Site, User) Config File Reference
C Gcylc Config File Reference
D Cylc Gscan Config File Reference
E Command Reference
F The gcylc Graph View
G Cylc README File
H Cylc INSTALL File
I Cylc Development History - Major Changes
J Communication Method
K Cylc 6 Migration Reference
L Known Issues
M GNU GENERAL PUBLIC LICENSE v3.0

1 Introduction: How Cylc Works

 1.1 Scheduling Forecast Suites
 1.2 EcoConnect
 1.3 Dependence Between Tasks
 1.4 The Cylc Scheduling Algorithm

1.1 Scheduling Forecast Suites

Environmental forecasting suites generate forecast products from a potentially large group of interdependent scientific models and associated data processing tasks. They are constrained by availability of external driving data: typically one or more tasks will wait on real time observations and/or model data from an external system, and these will drive other downstream tasks, and so on. The dependency diagram for a single forecast cycle point in such a system is a Directed Acyclic Graph as shown in Figure 1 (in our terminology, a forecast cycle point is comprised of all tasks with a common cycle point, which is the nominal analysis time or start time of the forecast models in the group). In real time operation processing will consist of a series of distinct forecast cycle points that are each initiated, after a gap, by arrival of the new cycle point’s external driving data.

From a job scheduling perspective task execution order in such a system must be carefully controlled in order to avoid dependency violations. Ideally, each task should be queued for execution at the instant its last prerequisite is satisfied; this is the best that can be done even if queued tasks are not able to execute immediately because of resource contention.

1.2 EcoConnect

Cylc was developed for the EcoConnect Forecasting System at NIWA (National Institute of Water and Atmospheric Research, New Zealand). EcoConnect takes real time atmospheric and stream flow observations, and operational global weather forecasts from the Met Office (UK), and uses these to drive global sea state and regional data assimilating weather models, which in turn drive regional sea state, storm surge, and catchment river models, plus tide prediction, and a large number of associated data collection, quality control, preprocessing, post-processing, product generation, and archiving tasks.1 The global sea state forecast runs once daily. The regional weather forecast runs four times daily but it supplies surface winds and pressure to several downstream models that run only twice daily, and precipitation accumulations to catchment river models that run on an hourly cycle assimilating real time stream flow observations and using the most recently available regional weather forecast. EcoConnect runs on heterogeneous distributed hardware, including a massively parallel supercomputer and several Linux servers.

1.3 Dependence Between Tasks

1.3.1 Intra-cycle Dependence

Most dependence between tasks applies within a single forecast cycle point. Figure 1 shows the dependency diagram for a single forecast cycle point of a simple example suite of three forecast models (a, b, and c) and three post processing or product generation tasks (d, e and f). A scheduler capable of handling this must manage, within a single forecast cycle point, multiple parallel streams of execution that branch when one task generates output for several downstream tasks, and merge when one task takes input from several upstream tasks.


PIC


Figure 1: The dependency graph for a single forecast cycle point of a simple example suite. Tasks a, b, and c represent forecast models, d, e and f are post processing or product generation tasks, and x represents external data that the upstream forecast model depends on.



PIC


Figure 2: The optimal job schedule for two consecutive cycle points of our example suite during real time operation, assuming that all tasks trigger off upstream tasks finishing completely. The horizontal extent of a task bar represents its execution time, and the vertical blue lines show when the external driving data becomes available.


Figure 2 shows the optimal job schedule for two consecutive cycle points of the example suite in real time operation, given execution times represented by the horizontal extent of the task bars. There is a time gap between cycle points as the suite waits on new external driving data. Each task in the example suite happens to trigger off upstream tasks finishing, rather than off any intermediate output or event; this is merely a simplification that makes for clearer diagrams.


PIC


Figure 3: If the external driving data is available in advance, can we start running the next cycle point early?



PIC


Figure 4: A naive attempt to overlap two consecutive cycle points using the single-cycle-point dependency graph. The red shaded tasks will fail because of dependency violations (or will not be able to run because of upstream dependency violations).



PIC


Figure 5: The best that can be done in general when inter-cycle dependence is ignored.


Now the question arises, what happens if the external driving data for upcoming cycle points is available in advance, as it would be after a significant delay in operations, or when running a historical case study? While the forecast model a appears to depend only on the external data x at this stage of the discussion, in fact it would typically also depend on its own previous instance for the model background state used in initializing the new forecast. Thus, as alluded to in Figure 3, task a could in principle start as soon as its predecessor has finished. Figure 4 shows, however, that starting a whole new cycle point at this point is dangerous - it results in dependency violations in half of the tasks in the example suite. In fact the situation could be even worse than this - imagine that task b in the first cycle point is delayed for some reason after the second cycle point has been launched. Clearly we must consider handling inter-cycle dependence explicitly or else agree not to start the next cycle point early, as is illustrated in Figure 5.

1.3.2 Inter-Cycle Dependence

Forecast models typically depend on their own most recent previous forecast for background state or restart files of some kind (this is called warm cycling) but there can also be inter-cycle dependence between different tasks. In an atmospheric forecast analysis suite, for instance, the weather model may generate background states for observation processing and data-assimilation tasks in the next cycle point as well as for the next forecast model run. In real time operation inter-cycle dependence can be ignored because it is automatically satisfied when one cycle point finishes before the next begins. If it is not ignored it drastically complicates the dependency graph by blurring the clean boundary between cycle points. Figure 6 illustrates the problem for our simple example suite assuming minimal inter-cycle dependence: the warm cycled models (a, b, and c) each depend on their own previous instances.

For this reason, and because we tend to see forecasting suites in terms of their real time characteristics, other metaschedulers have ignored inter-cycle dependence and are thus restricted to running entire cycle points in sequence at all times. This does not affect normal real time operation but it can be a serious impediment when advance availability of external driving data makes it possible, in principle, to run some tasks from upcoming cycle points before the current cycle point is finished - as was suggested at the end of the previous section. This can occur, for instance, after operational delays (late arrival of external data, system maintenance, etc.) and to an even greater extent in historical case studies and parallel test suites started behind a real time operation. It can be a serious problem for suites that have little downtime between forecast cycle points and therefore take many cycle points to catch up after a delay. Without taking account of inter-cycle dependence, the best that can be done, in general, is to reduce the gap between cycle points to zero as shown in Figure 5. A limited crude overlap of the single cycle point job schedule may be possible for specific task sets but the allowable overlap may change if new tasks are added, and it is still dangerous: it amounts to running different parts of a dependent system as if they were not dependent and as such it cannot be guaranteed that some unforeseen delay in one cycle point, after the next cycle point has begun, (e.g. due to resource contention or task failures) won’t result in dependency violations.


PIC


Figure 6: The complete dependency graph for the example suite, assuming the least possible inter-cycle dependence: the forecast models (a, b, and c) depend on their own previous instances. The dashed arrows show connections to previous and subsequent forecast cycle points.



PIC


Figure 7: The optimal two cycle job schedule when the next cycle’s driving data is available in advance, possible in principle when inter-cycle dependence is handled explicitly.


Figure 7 shows, in contrast to Figure 4, the optimal two cycle point job schedule obtained by respecting all inter-cycle dependence. This assumes no delays due to resource contention or otherwise - i.e. every task runs as soon as it is ready to run. The scheduler running this suite must be able to adapt dynamically to external conditions that impact on multi-cycle-point scheduling in the presence of inter-cycle dependence or else, again, risk bringing the system down with dependency violations.


PIC


Figure 8: Job schedules for the example suite after a delay of almost one whole forecast cycle point, when inter-cycle dependence is taken into account (above the time axis), and when it is not (below the time axis). The colored lines indicate the time that each cycle point is delayed, and normal “caught up” cycle points are shaded gray.



PIC


Figure 9: Job schedules for the example suite in case study mode, or after a long delay, when the external driving data are available many cycle points in advance. Above the time axis is the optimal schedule obtained when the suite is constrained only by its true dependencies, as in Figure 3, and underneath is the best that can be done, in general, when inter-cycle dependence is ignored.


To further illustrate the potential benefits of proper inter-cycle dependency handling, Figure 8 shows an operational delay of almost one whole cycle point in a suite with little downtime between cycle points. Above the time axis is the optimal schedule that is possible in principle when inter-cycle dependence is taken into account, and below it is the only safe schedule possible in general when it is ignored. In the former case, even the cycle point immediately after the delay is hardly affected, and subsequent cycle points are all on time, whilst in the latter case it takes five full cycle points to catch up to normal real time operation.

Similarly, Figure 9 shows example suite job schedules for an historical case study, or when catching up after a very long delay; i.e. when the external driving data are available many cycle points in advance. Task a, which as the most upstream forecast model is likely to be a resource intensive atmosphere or ocean model, has no upstream dependence on co-temporal tasks and can therefore run continuously, regardless of how much downstream processing is yet to be completed in its own, or any previous, forecast cycle point (actually, task a does depend on co-temporal task x which waits on the external driving data, but that returns immediately when the data is available in advance, so the result stands). The other forecast models can also cycle continuously or with a short gap between, and some post processing tasks, which have no previous-instance dependence, can run continuously or even overlap (e.g. e in this case). Thus, even for this very simple example suite, tasks from three or four different cycle points can in principle run simultaneously at any given time.

In fact, if our tasks are able to trigger off internal outputs of upstream tasks (message triggers) rather than waiting on full completion, then successive instances of the forecast models could overlap as well (because model restart outputs are generally completed early in the forecast) for an even more efficient job schedule.

1.4 The Cylc Scheduling Algorithm


PIC


Figure 10: How cylc sees a suite, in contrast to the multi-cycle-point dependency graph of Figure 6. Task colors represent different cycle points, and the small squares and circles represent different prerequisites and outputs. A task can run when its prerequisites are satisfied by the outputs of other tasks in the pool.


Cylc manages a pool of proxy objects that represent the real tasks in a suite. Task proxies know how to run the real tasks that they represent, and they receive progress messages from the tasks as they run (usually reports of completed outputs). There is no global cycling mechanism to advance the suite; instead individual task proxies have their own private cycle point and spawn their own successors when the time is right. Task proxies are self-contained - they know their own prerequisites and outputs but are not aware of the wider suite. Inter-cycle dependence is not treated as special, and the task pool can be populated with tasks with many different cycle points. The task pool is illustrated in Figure 10. Whenever any task changes state due to completion of an output, every task checks to see if its own prerequisites have been satisfied. In effect, cylc gets a pool of tasks to self-organize by negotiating their own dependencies so that optimal scheduling, as described in the previous section, emerges naturally at run time.

2 Cylc Screenshots


PIC


Figure 11: gcylc graph and dot views.



PIC


Figure 12: gcylc text view.



PIC


Figure 13: gscan multi-suite state summary GUI.



PIC


Figure 14: A large-ish suite graphed by cylc.


3 Installation

 3.1 External Software Packages
 3.2 Software Bundled With Cylc
 3.3 Installing Cylc Itself
 3.4 Automated Tests

Cylc runs on Unix variants, usually Linux, and including Apple OS X.

3.1 External Software Packages

Python >= 2.6 is required (but not yet Python 3). Python should already be installed in your Linux system. https://python.org.

For Cylc’s HTTPS communications layer:

The following packages are highly recommended, but are technically optional as you can construct and run suites without dependency graph visualisation or the Cylc GUIs:

The User Guide is generated from LATEXsource files by running make in the top level Cylc directory. The specific packages required may vary by distribution, e.g.:

To generate the HTML User Guide ImageMagick is also needed.

In most modern Linux distributions all of the software above can be installed via the system package manager. Otherwise download packages manually and follow their native installation instructions. To check that all (non LATEXpackages) are installed properly:

$ cylc check-software 
Checking for Python >= 2.6 ... found 2.7.6 ... ok 
Checking for non-Python packages: 
 + Graphviz ... ok 
Checking for Python packages: 
 + pygraphviz ... ok 
 + pygtk ... ok

If errors are reported then the packages concerned are either not installed or not in your Python search path. (Note that cylc check-software has become quite trivial as we’ve removed or bundled some former dependencies, but in future we intend to make it print a comprehensive list of library versions etc. to include in with bug reports.)

3.2 Software Bundled With Cylc

Cylc bundles several third party packages which do not need to be installed separately.

3.3 Installing Cylc Itself

Cylc releases can be downloaded from from https://cylc.github.io/cylc.

The wrapper script admin/cylc-wrapper should be installed as cylc in the system executable search path (e.g. /usr/local/bin/) and modified slightly to point to a location such as /opt where successive Cylc releases will be unpacked side by side.

To install Cylc for the first time simply unpack the release tarball in that location, e.g. /opt/cylc-7.4.0, type make inside the unpacked release directory, and set site defaults - if necessary - in a site global config file (below).

In the installed location, make a symbolic link from cylc to the latest installed version: ln -s /opt/cylc-7.4.0 cylc. This is the version of Cylc that will be invoked by the central wrapper if a specific version is not requested e.g. by CYLC_VERSION=7.4.0.

Installing subsequent releases is just a matter of unpacking the new tarballs next to the previous releases, running make in them, and copying in (possibly with modifications) the previous site global config file.

3.3.1 Local User Installation

It is easy to install Cylc under your own user account if you don’t have root or sudo access to the system: just put the central Cylc wrapper in $HOME/bin/ (making sure that is in your $PATH) and modify it to point to a directory such as $HOME/cylc/ where you will unpack and install release tarballs. Local installation of third party dependencies like Graphviz is also possible, but that depends on the particular installation methods used and is outside of the scope of this document.

3.3.2 Create A Site Config File

Site and user global config files define some important parameters that affect all suites, some of which may need to be customized for your site. See 6 for how to generate an initial site file and where to install it. All legal site and user global config items are defined in B.

3.3.3 Configure Site Environment on Job Hosts

If your users submit task jobs to hosts other than the hosts they use to run their suites, you should ensure that the job hosts have the correct environment for running cylc. A cylc suite generates task job scripts that normally invoke bash. The job will attempt to source the first of these files it finds to set up its environment:

The ${CYLC_DIR}/conf/job-init-env-default.sh file is provided in the cylc distribution, and will attempt to source /etc/profile and ${HOME}/.profile. If this behaviour is not desirable, you should override it by adding a ${CYLC_DIR}/conf/job-init-env.sh file and populate it with the appropriate contents.

3.4 Automated Tests

The cylc test battery is primarily intended for developers to check that changes to the source code don’t break existing functionality. Note that some test failures can be expected to result from suites timing out, even if nothing is wrong, if you run too many tests in parallel. See cylc test-battery --help.

4 Cylc Terminology

 4.1 Jobs and Tasks
 4.2 Cycle Points

4.1 Jobs and Tasks

A job is a program or script that runs on a computer, and a task is a workflow abstraction - a node in the suite dependency graph - that represents a job.

4.2 Cycle Points

A cycle point is a particular date-time (or integer) point in a sequence of date-time (or integer) points. Each cylc task has a private cycle point and can advance independently to subsequent cycle points. It may sometimes be convenient, however, to refer to the “current cycle point” of a suite (or the previous or next one, etc.) with reference to a particular task, or in the sense of all tasks instances that “belong to” a particular cycle point. But keep in mind that different tasks may pass through the “current cycle point” (etc.) at different times as the suite evolves.

5 Workflows For Cycling Systems

 5.1 Cycling Workflows
 5.2 Parameterized Tasks as a Proxy for Cycling
 5.3 Mixed Cycling Workflows

A model run and associated processing may need to be cycled for the following reasons:

Cylc provides two ways of constructing workflows for cycling systems: cycling workflows and parameterized tasks.

5.1 Cycling Workflows

This is cylc’s classic cycling mode as described in the Introduction. Each instance of a cycling job is represented by a new instance of the same task, with a new cycle point. The suite configuration defines patterns for extending the workflow on the fly, so it can keep running indefinitely if necessary. For example, to cycle model.exe on a monthly sequence we could define a single task model, an initial cycle point, and a monthly sequence. Cylc then generates the date-time sequence and creates a new task instance for each cycle point as it comes up. Workflow dependencies are defined generically with respect to the “current cycle point” of the tasks involved.

This is the only sensible way to run very large suites or operational suites that need to continue cycling indefinitely. The cycling is configured with standards-based ISO 8601 date-time recurrence expressions. Multiple cycling sequences can be used at once in the same suite. See Section 9.3.

5.2 Parameterized Tasks as a Proxy for Cycling

It is also possible to run cycling jobs with a pre-defined static workflow in which each instance of a cycling job is represented by a different task: as far as the abstract workflow is concerned there is no cycling. The sequence of tasks can be constructed efficiently, however, using cylc’s built-in suite parameters (9.6.7) or explicit Jinja2 loops (9.7).

For example, to run model.exe 12 times on a monthly cycle we could loop over an integer parameter R = 0, 1, 2, ..., 11 to define tasks model-R0, model-R1, model-R2, ...model-R11, and the parameter values could be multiplied by the interval P1M (one month) to get the start point point for the corresponding model run.

This method is only good for smaller workflows of finite duration because every single task has to be mapped out in advance, and cylc has to be aware of all of them throughout the entire run. Additionally Cylc’s cycling workflow capabilities (above) are more powerful, more flexible, and generally easier to use (Cylc will do the date-time arithmetic for you, for instance), so that is the recommended way to drive most cycling systems.

The primary use for parameterized tasks in cylc is to generate ensembles and other groups of related tasks at the same cycle point, not as a proxy for cycling.

5.3 Mixed Cycling Workflows

For completeness we note that parameterized cycling can be used within a cycling workflow. For example, in a daily cycling workflow long (daily) model runs could be split into four shorter runs by parameterized cycling. A simpler six-hourly cycling workflow should be considered first, however.

6 Global (Site, User) Configuration Files

Cylc site and user global configuration files contain settings that affect all suites. Some of these, such as the range of network ports used by cylc, should be set at site level,

# cylc site global config file 
/path/to/cylc/conf/global.rc 
# Deprecated path to cylc site global config file 
/path/to/cylc/conf/siterc/site.rc

Others, such as the preferred text editor for suite definitions, can be overridden by users,

# cylc user global config file 
~/.cylc/global.rc 
# Deprecated cylc user global config file 
~/.cylc/user.rc

The cylc get-site-config command retrieves current global settings consisting of cylc defaults overridden by site settings, if any, overridden by user settings, if any. To generate an initial site or user global config file:

$ cylc get-site-config > $HOME/.cylc/global.rc

Settings that do not need to be changed should be deleted or commented out of user global config files so that they don’t override future changes to the site file.

Legal items, values, and system defaults are documented in (B).

7 Tutorial

 7.1 User Config File
 7.2 User Interfaces
 7.3 Suite Definitions
 7.4 Suite Registration
 7.5 Suite Passphrases
 7.6 Import The Example Suites
 7.7 Rename The Imported Tutorial Suites
 7.8 Suite Validation
 7.9 Hello World in Cylc
 7.10 Editing Suites
 7.11 Running Suites
 7.12 Discovering Running Suites
 7.13 Task Identifiers
 7.14 Job Submission: How Tasks Are Executed
 7.15 Locating Suite And Task Output
 7.16 Remote Tasks
 7.17 Task Triggering
 7.18 Runtime Inheritance
 7.19 Triggering Families
 7.20 Triggering Off Of Families
 7.21 Suite Visualization
 7.22 External Task Scripts
 7.23 Cycling Tasks
 7.24 Jinja2
 7.25 Task Retry On Failure
 7.26 Other Users’ Suites
 7.27 Other Things To Try

This section provides a hands-on tutorial introduction to basic cylc functionality.

7.1 User Config File

Some settings affecting cylc’s behaviour can be defined in site and user global config files. For example, to choose the text editor invoked by cylc on suite definitions:

# $HOME/.cylc/global.rc 
[editors] 
    terminal = vim 
    gui = gvim -f

7.1.1 Configure Environment on Job Hosts

If you submit task jobs to hosts other than the hosts you use to run your suites, you may need to customise the environment for running cylc. A cylc suite generates task job scripts that normally invoke bash. The job will attempt to source the first of these files it finds to set up its environment:

The ${CYLC_DIR}/conf/job-init-env-default.sh file is provided in the cylc distribution, and will attempt to source /etc/profile and ${HOME}/.profile. If this behaviour is not desirable, your site administrator should have overridden it by adding a ${CYLC_DIR}/conf/job-init-env.sh file and populate it with the appropriate contents. If customisation is still required, you can add your own ${HOME}/.cylc/job-init-env.sh file and populate it with the appropriate contents.

7.2 User Interfaces

You should have access to the cylc command line (CLI) and graphical (GUI) user interfaces once cylc has been installed as described in Section 3.3.

7.2.1 Command Line Interface (CLI)

The command line interface is unified under a single top level cylc command that provides access to many sub-commands and their help documentation.

$ cylc help       # Top level command help. 
$ cylc run --help # Example command-specific help.

Command help transcripts are printed in E and are available from the GUI Help menu.

Cylc is scriptable - the error status returned by commands can be relied on.

7.2.2 Graphical User Interface (GUI)

The cylc GUI covers the same functionality as the CLI, but it has more sophisticated suite monitoring capability. It can start and stop suites, or connect to suites that are already running; in either case, shutting down the GUI does not affect the suite itself.

$ gcylc & # or: 
$ cylc gui & # Single suite control GUI. 
$ cylc gscan & # Multi-suite monitor GUI.

Clicking on a suite in gscan, shown in Figure 13, opens a gcylc instance for it.

7.3 Suite Definitions

Cylc suites are defined by extended-INI format suite.rc files (the main file format extension is section nesting). These reside in suite definition directories that may also contain a bin directory and any other suite-related files.

7.4 Suite Registration

Suite registration creates a run directory (under ~/cylc-run/ by default) and populates it with authentication files and a symbolic link to a suite definition directory. Cylc commands that parse suite definitions can take the file path or the suite name as input. Commands that interact with running suites have to target the suite by name.

# Target a suite by file path: 
$ cylc validate /path/to/my/suite/suite.rc 
$ cylc graph /path/to/my/suite/suite.rc 
 
# Register a suite: 
$ cylc register my.suite /path/to/my/suite/ 
 
# Target a suite by name: 
$ cylc graph my.suite 
$ cylc validate my.suite 
$ cylc run my.suite 
$ cylc stop my.suite 
# etc.

7.5 Suite Passphrases

Registration (above) also generates a suite-specific passphrase file under .service/ in the suite run directory. It is loaded by the suite daemon at start-up and used to authenticate connections from client programs.

Possession of a suite’s passphrase file gives full control over it. Without it, the information avaiable to a client is determined by the suite’s public access privilege level.

For more on connection authentication, suite passphrases, and public access, see 12.6.

7.6 Import The Example Suites

Run the following command to copy cylc’s example suites and register them for your own use:

$ cylc import-examples /tmp

7.7 Rename The Imported Tutorial Suites

Suites can be renames by simply renaming (i.e. moving) their run directories. Make the tutorial suite names shorter, and print their locations with cylc print:

$ mv ~/cylc-run/$(cylc --version)/examples/tutorial ~/cylc-run/tut 
$ cylc print -ya tut 
tut/oneoff/jinja2  | /tmp/cylc-examples/7.0.0/tutorial/oneoff/jinja2 
tut/cycling/two    | /tmp/cylc-examples/7.0.0/tutorial/cycling/two 
tut/cycling/three  | /tmp/cylc-examples/7.0.0/tutorial/cycling/three 
# ...

See cylc print --help for other display options.

7.8 Suite Validation

Suite definitions can be validated to detect syntax (and other) errors:

# pass: 
$ cylc validate tut/oneoff/basic 
Valid for cylc-6.0.0 
$ echo $? 
0 
# fail: 
$ cylc validate my/bad/suite 
Illegal item: [scheduling]special tusks 
$ echo $? 
1

7.9 Hello World in Cylc

suite: tut/oneoff/basic

Here’s the traditional Hello World program rendered as a cylc suite:

 
title = "The cylc Hello World! suite" 
[scheduling] 
    [[dependencies]] 
        graph = "hello" 
[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!"

Cylc suites feature a clean separation of scheduling configuration, which determines when tasks are ready to run; and runtime configuration, which determines what to run (and where and how to run it) when a task is ready. In this example the [scheduling] section defines a single task called hello that triggers immediately when the suite starts up. When the task finishes the suite shuts down. That this is a dependency graph will be more obvious when more tasks are added. Under the [runtime] section the script item defines a simple inlined implementation for hello: it sleeps for ten seconds, then prints Hello World!, and exits. This ends up in a job script generated by cylc to encapsulate the task (below) and, thanks to some defaults designed to allow quick prototyping of new suites, it is submitted to run as a background job on the suite host. In fact cylc even provides a default task implementation that makes the entire [runtime] section technically optional:

 
title = "The minimal complete runnable cylc suite" 
[scheduling] 
    [[dependencies]] 
        graph = "foo" 
# (actually, 'title' is optional too ... and so is this comment)

(the resulting dummy task just prints out some identifying information and exits).

7.10 Editing Suites

The text editor invoked by cylc on suite definitions is determined by cylc site and user global config files, as shown above in 7.2. Check that you have renamed the tutorial examples suites as described just above and open the Hello World suite definition in your text editor:

$ cylc edit tut/oneoff/basic # in-terminal 
$ cylc edit -g tut/oneoff/basic & # or GUI

Alternatively, start gcylc on the suite:

$ gcylc tut/oneoff/basic &

and choose Suite Edit from the menu.

The editor will be invoked from within the suite definition directory for easy access to other suite files (in this case there are none). There are syntax highlighting control files for several text editors under /path/to/cylc/conf/; see in-file comments for installation instructions.

7.11 Running Suites

7.11.1 CLI

Run tut/oneoff/basic using the cylc run command. As a suite runs detailed timestamped information is written to a suite log and progress can be followed with cylc’s suite monitoring tools (below). By default a running suite daemonizes after printing a short message so that you can exit the terminal or even log out without killing the suite:
$ cylc run tut/oneoff/basic 
            ._. 
            | |                 The Cylc Suite Engine [7.0.0] 
._____._. ._| |_____.           Copyright (C) 2008-2017 NIWA 
| .___| | | | | .___|  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
| !___| !_! | | !___.  This program comes with ABSOLUTELY NO WARRANTY; 
!_____!___. |_!_____!  see cylc warranty‘.  It is free software, you 
      .___! |           are welcome to redistribute it under certain 
      !_____!                conditions; see cylc conditions‘. 
 
⋆⋆⋆ listening on nwp-1:43027 ⋆⋆⋆ 
 
To view suite daemon contact information: 
 $ cylc get-suite-contact tut/oneoff/basic 
 
Other ways to see if the suite is still running: 
 $ cylc scan -n '\btut/oneoff/basic\b' nwp-1 
 $ cylc ping -v --host=nwp-1 tut/oneoff/basic 
 $ ssh nwp-1 "pgrep -a -P 1 -fu $USER 'cylc-r.⋆ \btut/oneoff/basic\b'"

If you’re quick enough (this example only takes 10-15 seconds to run) the cylc scan command will detect the running suite:

$ cylc scan 
tut/oneoff/basic oliverh@nwp-1:43027

Note you can use the --no-detach and --debug options to cylc-run to prevent the suite from daemonizing (i.e. to make it stay attached to your terminal until it exits).

When a task is ready cylc generates a job script to run it, by default as a background jobs on the suite host. The job process ID is captured, and job output is directed to log files in standard locations under the suite run directory.

Log file locations relative to the suite run directory look like job/1/hello/01/ where the first digit is the cycle point of the task hello (for non-cycling tasks this is just ‘1’); and the final 01 is the submit number (tasks can be made to retry on failure or manually retriggered - each time a new log directory is used to avoid overwriting previous output).

The suite shuts down automatically once all tasks have succeeded.

7.11.2 GUI

The cylc GUI can start and stop suites, or (re)connect to suites that are already running:

$ cylc gui tut/oneoff/basic &

Use the tool bar Play button, or the Control Run menu item, to run the suite again. You may want to alter the suite definition slightly to make the task take longer to run. Try right-clicking on the hello task to view its output logs. The relative merits of the three suite views - dot, text, and graph - will be more apparent later when we have more tasks. Closing the GUI does not affect the suite itself.

7.12 Discovering Running Suites

Suites that are currently running can be detected with command line or GUI tools:

# list currently running suites and their port numbers: 
$ cylc scan 
tut/oneoff/basic oliverh@nwp-1:43001 
 
# GUI summary view of running suites: 
$ cylc gscan &

The scan GUI is shown in Figure 13; clicking on a suite in it opens gcylc.

7.13 Task Identifiers

At run time, task instances are identified by name, which is determined entirely by the suite definition, and a cycle point which is usually a date-time or an integer:

foo.20100808T00Z   # a task with a date-time cycle point 
bar.1              # a task with an integer cycle point (could be non-cycling)

Non-cycling tasks usually just have the cycle point 1, but this still has to be used to target the task instance with cylc commands.

7.14 Job Submission: How Tasks Are Executed

suite: tut/oneoff/jobsub

Task job scripts are generated by cylc to wrap the task implementation specified in the suite definition (environment, script, etc.) in error trapping code, messaging calls to report task progress back to the suite daemon, and so forth. Job scripts are written to the suite job log directory where they can be viewed alongside the job output logs. They can be accessed at run time by right-clicking on the task in the cylc GUI, or printed to the terminal:

$ cylc cat-log tut/oneoff/basic hello.1

This command can also print the suite log (and stdout and stderr for suites in daemon mode) and task stdout and stderr logs (see cylc cat-log --help). A new job script can also be generated on the fly for inspection:

$ cylc jobscript tut/oneoff/basic hello.1

Take a look at the job script generated for hello.1 during the suite run above. The custom scripting should be clearly visible toward the bottom of the file.

The hello task in the first tutorial suite defaults to running as a background job on the suite host. To submit it to the Unix at scheduler instead, configure its job submission settings as in tut/oneoff/jobsub:

[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
        [[[job]]] 
            batch system = at

Run the suite again after checking that atd is running on your system.

Cylc supports a number of different batch systems. Tasks submitted to external batch queuing systems like at, PBS, SLURM, Moab, or LoadLeveler, are displayed as submitted in the cylc GUI until they start executing.

7.15 Locating Suite And Task Output

If the --no-detach option is not used, suite stdout and stderr will be directed to the suite run directory along with the time-stamped suite log file, and task job scripts and job logs (task stdout and stderr). The default suite run directory location is $HOME/cylc-run:

$ tree $HOME/cylc-run/tut/oneoff/basic/ 
|-- .service              # location of run time service files 
|    |-- contact          # detail on how to contact the running suite 
|    |-- db               # private suite run database 
|    |-- passphrase       # passphrase for client authentication 
|    |-- source           # symbolic link to source directory 
|    |-- ssl.cert         # SSL certificate for the suite server 
|    ‘-- ssl.pem          # SSL private key 
|-- cylc-suite.db         # back compat symlink to public suite run database 
|-- share                 # suite share directory (not used in this example) 
|-- work                  # task work space (sub-dirs are deleted if not used) 
|    ‘-- 1                   # task cycle point directory (or 1) 
|        ‘-- hello              # task work directory (deleted if not used) 
|-- log                   # suite log directory 
|   |-- db                   # public suite run database 
|   |-- job                  # task job log directory 
|   |   ‘-- 1                   # task cycle point directory (or 1) 
|   |       ‘-- hello              # task name 
|   |           |-- 01                # task submission number 
|   |           |   |-- job              # task job script 
|   |           |   ‘-- job-activity.log # task job activity log 
|   |           |   |-- job.err          # task stderr log 
|   |           |   |-- job.out          # task stdout log 
|   |           |   ‘-- job.status       # task status file 
|   |           ‘-- NN -> 01          # symlink to latest submission number 
|   ‘-- suite                # suite daemon log directory 
|       |-- err                 # suite daemon stderr log (daemon mode only) 
|       |-- out                 # suite daemon stdout log (damon mode only) 
|       ‘-- log                 # suite daemon event log (timestamped info)

The suite run database files, suite environment file, and task status files are used internally by cylc. Tasks execute in private work/ directories that are deleted automatically if empty when the task finishes. The suite share/ directory is made available to all tasks (by $CYLC_SUITE_SHARE_DIR) as a common share space. The task submission number increments from 1 if a task retries on failure; this is used a sub-directory of the log tree to avoid overwriting log files from earlier job submissions.

The top level run directory location can be changed in site and user config files if necessary, and the suite share and work locations can be configured separately because of the potentially larger disk space requirement.

Task job logs can be viewed by right-clicking on tasks in the gcylc GUI (so long as the task proxy is live in the suite), manually accessed from the log directory (of course), or printed to the terminal with the cylc cat-log command:

# suite logs: 
$ cylc cat-log    tut/oneoff/basic           # suite event log 
$ cylc cat-log -o tut/oneoff/basic           # suite stdout log 
$ cylc cat-log -e tut/oneoff/basic           # suite stderr log 
# task logs: 
$ cylc cat-log    tut/oneoff/basic hello.1   # task job script 
$ cylc cat-log -o tut/oneoff/basic hello.1   # task stdout log 
$ cylc cat-log -e tut/oneoff/basic hello.1   # task stderr log

7.16 Remote Tasks

suite: tut/oneoff/remote

The hello task in the first two tutorial suites defaults to running on the suite host. To make it run on a remote host instead change its runtime configuration as in tut/oneoff/remote:

[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
        [[[remote]]] 
            host = server1.niwa.co.nz

For remote task hosting to work several requirements must be satisfied:

If your username is different on the task host the [[[remote]]] section also supports an owner=username item, or your $HOME/.ssh/config file can be configured for username translation.

If you configure a task host according to the requirements cylc will create remote log directories, source login scripts on the remote host to ensure cylc is visible there, send the task job script over, and submit it to run there by the configured batch system:

Remote task job logs are saved to the suite run directory on the task host, not on the suite host. They can be retrieved by right-clicking on the task in the GUI, or to have cylc pull them back to the suite host automatically do this:

[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
        [[[remote]]] 
            host = server1.niwa.co.nz 
            retrieve job logs = True

This suite will attempt to rsync job logs from the remote host each time a task job completes.

Some batch systems have considerable delays between the time when the job completes and when it writes the job logs in its normal location. If this is the case, you can configure an initial delay and retry delays for job log retrieval by setting some delays. E.g.:

[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
        [[[remote]]] 
            host = server1.niwa.co.nz 
            retrieve job logs = True 
            # Retry after 10 seconds, 1 minute and 3 minutes 
            retrieve job logs retry delays = PT10S, PT1M, PT3M

Finally, if the disk space of the suite host is limited, you may want to set [[[remote]]]retrieve job logs max size=SIZE. The value of SIZE can be anything that is accepted by the --max-size=SIZE option of the rsync command. E.g.:

[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
        [[[remote]]] 
            host = server1.niwa.co.nz 
            retrieve job logs = True 
            # Don't get anything bigger than 10MB 
            retrieve job logs max size = 10M

It is worth noting that cylc uses the existence of a job’s job.out or job.err in the local file system to indicate a successful job log retrieval. If retrieve job logs max sizeSIZE= is set and both job.out and job.err are bigger than SIZE then cylc will consider the retrieval as failed. If retry delays are specified, this will trigger some useless (but harmless) retries. If this occurs regularly, you should try the following:

7.17 Task Triggering

suite: tut/oneoff/goodbye

To make a second task called goodbye trigger after hello finishes successfully, return to the original example, tut/oneoff/basic, and change the suite graph as in tut/oneoff/goodbye:

[scheduling] 
    [[dependencies]] 
        graph = "hello => goodbye"

or to trigger it at the same time as hello,

[scheduling] 
    [[dependencies]] 
        graph = "hello & goodbye"

and configure the new task’s behaviour under [runtime]:

[runtime] 
    [[goodbye]] 
        script = "sleep 10; echo Goodbye World!"

Run tut/oneoff/goodbye and check the output from the new task:

$ cat ~/cylc-run/tut/oneoff/goodbye/log/job/1/goodbye/01/job.out 
  # or 
$ cylc cat-log -o tut/oneoff/goodbye goodbye.1 
JOB SCRIPT STARTING 
cylc (scheduler - 2014-08-14T15:09:30+12): goodbye.1 started at 2014-08-14T15:09:30+12 
cylc Suite and Task Identity: 
  Suite Name  : tut/oneoff/goodbye 
  Suite Host  : oliverh-34403dl.niwa.local 
  Suite Port  : 43001 
  Suite Owner : oliverh 
  Task ID     : goodbye.1 
  Task Host   : nwp-1 
  Task Owner  : oliverh 
  Task Try No.: 1 
 
Goodbye World! 
cylc (scheduler - 2014-08-14T15:09:40+12): goodbye.1 succeeded at 2014-08-14T15:09:40+12 
JOB SCRIPT EXITING (TASK SUCCEEDED)

7.17.1 Task Failure And Suicide Triggering

suite: tut/oneoff/suicide

Task names in the graph string can be qualified with a state indicator to trigger off task states other than success:

    graph = """ 
 a => b        # trigger b if a succeeds 
 c:submit => d # trigger d if c submits 
 e:finish => f # trigger f if e succeeds or fails 
 g:start  => h # trigger h if g starts executing 
 i:fail   => j # trigger j if i fails 
            """

A common use of this is to automate recovery from known modes of failure:

    graph = "goodbye:fail => really_goodbye"

i.e. if task goodbye fails, trigger another task that (presumably) really says goodbye.

Failure triggering generally requires use of suicide triggers as well, to remove the recovery task if it isn’t required (otherwise it would hang about indefinitely in the waiting state):

[scheduling] 
    [[dependencies]] 
        graph = """hello => goodbye 
            goodbye:fail => really_goodbye 
         goodbye => !really_goodbye # suicide"""

This means if goodbye fails, trigger really_goodbye; and otherwise, if goodbye succeeds, remove really_goodbye from the suite.

Try running tut/oneoff/suicide, which also configures the hello task’s runtime to make it fail, to see how this works.

7.18 Runtime Inheritance

suite: tut/oneoff/inherit

The [runtime] section is actually a multiple inheritance hierarchy. Each subsection is a namespace that represents a task, or if it is inherited by other namespaces, a family. This allows common configuration to be factored out of related tasks very efficiently.

 
title = "Simple runtime inheritance example" 
[scheduling] 
    [[dependencies]] 
        graph = "hello => goodbye" 
[runtime] 
    [[root]] 
        script = "sleep 10; echo $GREETING World!" 
    [[hello]] 
        [[[environment]]] 
            GREETING = Hello 
    [[goodbye]] 
        [[[environment]]] 
            GREETING = Goodbye

The [root] namespace provides defaults for all tasks in the suite. Here both tasks inherit script from root, which they customize with different values of the environment variable $GREETING. Note that inheritance from root is implicit; from other parents an explicit inherit = PARENT is required, as shown below.

7.19 Triggering Families

suite: tut/oneoff/ftrigger1

Task families defined by runtime inheritance can also be used as shorthand in graph trigger expressions. To see this, consider two “greeter” tasks that trigger off another task foo:

[scheduling] 
    [[dependencies]] 
        graph = "foo => greeter_1 & greeter_2"

If we put the common greeting functionality of greeter_1 and greeter_2 into a special GREETERS family, the graph can be expressed more efficiently like this:

[scheduling] 
    [[dependencies]] 
        graph = "foo => GREETERS"

i.e. if foo succeeds, trigger all members of GREETERS at once. Here’s the full suite with runtime hierarchy shown:

 
title = "Triggering a family of tasks" 
[scheduling] 
    [[dependencies]] 
        graph = "foo => GREETERS" 
[runtime] 
    [[root]] 
        pre-script = "sleep 10" 
    [[foo]] 
        # empty (creates a dummy task) 
    [[GREETERS]] 
        script = "echo $GREETING World!" 
    [[greeter_1]] 
        inherit = GREETERS 
        [[[environment]]] 
            GREETING = Hello 
    [[greeter_2]] 
        inherit = GREETERS 
        [[[environment]]] 
            GREETING = Goodbye

(Note that we recommend given ALL-CAPS names to task families to help distinguish them from task names. However, this is just a convention).

Experiment with the tut/oneoff/ftrigger1 suite to see how this works.

7.20 Triggering Off Of Families

suite: tut/oneoff/ftrigger2

Tasks (or families) can also trigger off other families, but in this case we need to specify what the trigger means in terms of the upstream family members. Here’s how to trigger another task bar if all members of GREETERS succeed:

[scheduling] 
    [[dependencies]] 
        graph = """foo => GREETERS 
            GREETERS:succeed-all => bar"""

Verbose validation in this case reports:

$ cylc val -v tut/oneoff/ftrigger2 
... 
Graph line substitutions occurred: 
  IN: GREETERS:succeed-all => bar 
  OUT: greeter_1:succeed & greeter_2:succeed => bar 
...

Cylc ignores family member qualifiers like succeed-all on the right side of a trigger arrow, where they don’t make sense, to allow the two graph lines above to be combined in simple cases:

[scheduling] 
    [[dependencies]] 
        graph = "foo => GREETERS:succeed-all => bar"

Any task triggering status qualified by -all or -any, for the members, can be used with a family trigger. For example, here’s how to trigger bar if all members of GREETERS finish (succeed or fail) and any of them them succeed:

[scheduling] 
    [[dependencies]] 
        graph = """foo => GREETERS 
    GREETERS:finish-all & GREETERS:succeed-any => bar"""

(use of GREETERS:succeed-any by itself here would trigger bar as soon as any one member of GREETERS completed successfully). Verbose validation now begins to show how family triggers can simplify complex graphs, even for this tiny two-member family:

$ cylc val -v tut/oneoff/ftrigger2 
... 
Graph line substitutions occurred: 
  IN: GREETERS:finish-all & GREETERS:succeed-any => bar 
  OUT: ( greeter_1:succeed | greeter_1:fail ) & \ 
       ( greeter_2:succeed | greeter_2:fail ) & \ 
       ( greeter_1:succeed | greeter_2:succeed ) => bar 
...

Experiment with tut/oneoff/ftrigger2 to see how this works.

7.21 Suite Visualization

You can style dependency graphs with an optional [visualization] section, as shown in tut/oneoff/ftrigger2:

[visualization] 
    default node attributes = "style=filled" 
    [[node attributes]] 
        foo = "fillcolor=#6789ab", "color=magenta" 
        GREETERS = "fillcolor=#ba9876" 
        bar = "fillcolor=#89ab67"

To display the graph in an interactive viewer:

$ cylc graph tut/oneoff/ftrigger2 &    # dependency graph 
$ cylc graph -n tut/oneoff/ftrigger2 & # runtime inheritance graph

It should look like Figure 15 (with the GREETERS family node expanded on the right).


PIC       PIC       PIC


Figure 15: The tut/oneoff/ftrigger2 dependency and runtime inheritance graphs


Graph styling can be applied to entire families at once, and custom “node groups” can also be defined for non-family groups.

7.22 External Task Scripts

suite: tut/oneoff/external

The tasks in our examples so far have all had inlined implementation, in the suite definition, but real tasks often need to call external commands, scripts, or executables. To try this, let’s return to the basic Hello World suite and cut the implementation of the task hello out to a file hello.sh in the suite bin directory:

 
#!/bin/sh 
 
set -e 
 
GREETING=${GREETING:-Goodbye} 
echo "$GREETING World! from $0"

Make the task script executable, and change the hello task runtime section to invoke it:

 
title = "Hello World! from an external task script" 
[scheduling] 
    [[dependencies]] 
        graph = "hello" 
[runtime] 
    [[hello]] 
        pre-script = sleep 10 
        script = hello.sh 
        [[[environment]]] 
            GREETING = Hello

If you run the suite now the new greeting from the external task script should appear in the hello task stdout log. This works because cylc automatically adds the suite bin directory to $PATH in the environment passed to tasks via their job scripts. To execute scripts (etc.) located elsewhere you can refer to the file by its full file path, or set $PATH appropriately yourself (this could be done via $HOME/.profile, which is sourced at the top of the task job script, or in the suite definition itself).

Note the use of set -e above to make the script abort on error. This allows the error trapping code in the task job script to automatically detect unforeseen errors.

7.23 Cycling Tasks

suite: tut/cycling/one

So far we’ve considered non-cycling tasks, which finish without spawning a successor.

Cycling is based around iterating through date-time or integer sequences. A cycling task may run at each cycle point in a given sequence (cycle). For example, a sequence might be a set of date-times every 6 hours starting from a particular date-time. A cycling task may run for each date-time item (cycle point) in that sequence.

There may be multiple instances of this type of task running in parallel, if the opportunity arises and their dependencies allow it. Alternatively, a sequence can be defined with only one valid cycle point - in that case, a task belonging to that sequence may only run once.

Open the tut/cycling/one suite:

 
title = "Two cycling tasks, no inter-cycle dependence" 
[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130812T00 
    [[dependencies]] 
        [[[T00,T12]]] # 00 and 12 hours UTC every day 
            graph = "foo => bar" 
[visualization] 
    initial cycle point = 20130808T00 
    final cycle point = 20130809T00 
    [[node attributes]] 
        foo = "color=red" 
        bar = "color=blue"

The difference between cycling and non-cycling suites is all in the [scheduling] section, so we will leave the [runtime] section alone for now (this will result in cycling dummy tasks). Note that the graph is now defined under a new section heading that makes each task under it have a succession of cycle points ending in 00 or 12 hours, between specified initial and final cycle points (or indefinitely if no final cycle point is given), as shown in Figure 16.


PIC


Figure 16: The tut/cycling/one suite


If you run this suite instances of foo will spawn in parallel out to the runahead limit, and each bar will trigger off the corresponding instance of foo at the same cycle point. The runahead limit, which defaults to a few cycles but is configurable, prevents uncontrolled spawning of cycling tasks in suites that are not constrained by clock triggers in real time operation.

Experiment with tut/cycling/one to see how cycling tasks work.

7.23.1 ISO 8601 Date-Time Syntax

The suite above is a very simple example of a cycling date-time workflow. More generally, cylc comprehensively supports the ISO 8601 standard for date-time instants, intervals, and sequences. Cycling graph sections can be specified using full ISO 8601 recurrence expressions, but these may be simplified by assuming context information from the suite - namely initial and final cycle points. One form of the recurrence syntax looks like Rn/start-date-time/period (Rn means run n times). In the example above, if the initial cycle point is always at 00 or 12 hours then [[[T00,T12]]] could be written as [[[PT12H]]], which is short for [[[R/initial-cycle-point/PT12H/]]] - i.e. run every 12 hours indefinitely starting at the initial cycle point. It is possible to add constraints to the suite to only allow initial cycle points at 00 or 12 hours e.g.

[scheduling] 
    initial cycle point = 20130808T00 
    initial cycle point constraints = T00, T12

7.23.2 Inter-Cycle Triggers

suite: tut/cycling/two

The tut/cycling/two suite adds inter-cycle dependence to the previous example:

[scheduling] 
    [[dependencies]] 
        # Repeat with cycle points of 00 and 12 hours every day: 
        [[[T00,T12]]] 
            graph = "foo[-PT12H] => foo => bar"

For any given cycle point in the sequence defined by the cycling graph section heading, bar triggers off foo as before, but now foo triggers off its own previous instance foo[-PT12H]. Date-time offsets in inter-cycle triggers are expressed as ISO 8601 intervals (12 hours in this case). Figure 17 shows how this connects the cycling graph sections together.


PIC


Figure 17: The tut/cycling/two suite


Experiment with this suite to see how inter-cycle triggers work. Note that the first instance of foo, at suite start-up, will trigger immediately in spite of its inter-cycle trigger, because cylc ignores dependence on points earlier than the initial cycle point. However, the presence of an inter-cycle trigger usually implies something special has to happen at start-up. If a model depends on its own previous instance for restart files, for example, then some special process has to generate the initial set of restart files when there is no previous cycle point to do it. The following section shows one way to handle this in cylc suites.

7.23.3 Initial Non-Repeating (R1) Tasks

suite: tut/cycling/three

Sometimes we want to be able to run a task at the initial cycle point, but refrain from running it in subsequent cycles. We can do this by writing an extra set of dependencies that are only valid at a single date-time cycle point. If we choose this to be the initial cycle point, these will only apply at the very start of the suite.

The cylc syntax for writing this single date-time cycle point occurrence is R1, which stands for R1/no-specified-date-time/no-specified-period. This is an adaptation of part of the ISO 8601 date-time standard’s recurrence syntax (Rn/date-time/period) with some special context information supplied by cylc for the no-specified-⋆ data.

The 1 in the R1 means run once. As we’ve specified no date-time, Cylc will use the initial cycle point date-time by default, which is what we want. We’ve also missed out specifying the period - this is set by cylc to a zero amount of time in this case (as it never repeats, this is not significant).

For example, in tut/cycling/three:

[cylc] 
    cycle point time zone = +13 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130812T00 
    [[dependencies]] 
        [[[R1]]] 
            graph = "prep => foo" 
        [[[T00,T12]]] 
            graph = "foo[-PT12H] => foo => bar"

This is shown in Figure 18.

Note that the time zone has been set to +1300 in this case, instead of UTC (Z) as before. If no time zone or UTC mode was set, the local time zone of your machine will be used in the cycle points.

At the initial cycle point, foo will depend on foo[-PT12H] and also on prep:

prep.20130808T0000+13 & foo.20130807T1200+13 => foo.20130808T0000+13

Thereafter, it will just look like e.g.:

foo.20130808T0000+13 => foo.20130808T1200+13

However, in our initial cycle point example, the dependence on foo.20130807T1200+13 will be ignored, because that task’s cycle point is earlier than the suite’s initial cycle point and so it cannot run. This means that the initial cycle point dependencies for foo actually look like:

prep.20130808T0000+13 => foo.20130808T0000+13


PIC


Figure 18: The tut/cycling/three suite


7.23.4 Integer Cycling

suite: tut/cycling/integer

Cylc can do also do integer cycling for repeating workflows that are not date-time based.

Open the tut/cycling/integer suite, which is plotted in Figure 19.

 
 
[scheduling] 
    cycling mode = integer 
    initial cycle point = 1 
    final cycle point = 3 
    [[dependencies]] 
        [[[R1]]] # = R1/1/? 
            graph = start => foo 
        [[[P1]]] # = R/1/P1 
            graph = foo[-P1] => foo => bar 
        [[[R2/P1]]] # = R2/P1/3 
            graph = bar => stop 
 
[visualization] 
    [[node attributes]] 
        start = "style=filled", "fillcolor=skyblue" 
        foo = "style=filled", "fillcolor=slategray" 
        bar = "style=filled", "fillcolor=seagreen3" 
        stop = "style=filled", "fillcolor=orangered"


PIC


Figure 19: The tut/cycling/integer suite


The integer cycling notation is intended to look similar to the ISO 8601 date-time notation, but it is simpler for obvious reasons. The example suite illustrates two recurrence forms, Rn/start-point/period and Rn/period/stop-point, simplified somewhat using suite context information (namely the initial and final cycle points). The first form is used to run one special task called start at start-up, and for the main cycling body of the suite; and the second form to run another special task called stop in the final two cycles. The P character denotes period (interval) just like in the date-time notation. R/1/P2 would generate the sequence of points 1,3,5,....

7.24 Jinja2

suite: tut/oneoff/jinja2

Cylc has built in support for the Jinja2 template processor, which allows us to embed code in suite definitions to generate the final result seen by cylc.

The tut/oneoff/jinja2 suite illustrates two common uses of Jinja2: changing suite content or structure based on the value of a logical switch; and iteratively generating dependencies and runtime configuration for groups of related tasks:

 
#!jinja2 
 
{% set MULTI = True %} 
{% set N_GOODBYES = 3 %} 
 
title = "A Jinja2 Hello World! suite" 
[scheduling] 
    [[dependencies]] 
{% if MULTI %} 
        graph = "hello => BYE" 
{% else %} 
        graph = "hello" 
{% endif %} 
 
[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
{% if MULTI %} 
    [[BYE]] 
        script = "sleep 10; echo Goodbye World!" 
    {% for I in range(0,N_GOODBYES) %} 
    [[ goodbye_{{I}} ]] 
        inherit = BYE 
    {% endfor %} 
{% endif %}

To view the result of Jinja2 processing with the Jinja2 flag MULTI set to False:

$ cylc view --jinja2 --stdout tut/oneoff/jinja2
title = "A Jinja2 Hello World! suite" 
[scheduling] 
    [[dependencies]] 
        graph = "hello" 
[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!"

And with MULTI set to True:

$ cylc view --jinja2 --stdout tut/oneoff/jinja2
title = "A Jinja2 Hello World! suite" 
[scheduling] 
    [[dependencies]] 
        graph = "hello => BYE" 
[runtime] 
    [[hello]] 
        script = "sleep 10; echo Hello World!" 
    [[BYE]] 
        script = "sleep 10; echo Goodbye World!" 
    [[ goodbye_0 ]] 
        inherit = BYE 
    [[ goodbye_1 ]] 
        inherit = BYE 
    [[ goodbye_2 ]] 
        inherit = BYE

7.25 Task Retry On Failure

suite: tut/oneoff/retry

Tasks can be configured to retry a number of times if they fail. An environment variable $CYLC_TASK_TRY_NUMBER increments from 1 on each successive try, and is passed to the task to allow different behaviour on the retry:

 
title = "A task with automatic retry on failure" 
[scheduling] 
    [[dependencies]] 
        graph = "hello" 
[runtime] 
    [[hello]] 
        script = """ 
sleep 10 
if [[ $CYLC_TASK_TRY_NUMBER < 3 ]]; then 
    echo "Hello ... aborting!" 
    exit 1 
else 
    echo "Hello World!" 
fi""" 
        [[[job]]] 
            execution retry delays = 2⋆PT6S # retry twice after 6-second delays

When a task with configured retries fails, its cylc task proxy goes into the retrying state until the next retry delay is up, then it resubmits. It only enters the failed state on a final definitive failure.

Experiment with tut/oneoff/retry to see how this works.

7.26 Other Users’ Suites

If you have read access to another user’s account (even on another host) it is possible to use cylc monitor to look at their suite’s progress without full shell access to their account. To do this, you will need to copy their suite passphrase to

    $HOME/.cylc/SUITE_OWNER@SUITE_HOST/SUITE_NAME/passphrase

(use of the host and owner names is optional here - see 12.6.1) and also retrieve the port number of the running suite from:

    ~SUITE_OWNER/cylc-run/SUITE_NAME/.service/contact

Once you have this information, you can run

$ cylc monitor --user=SUITE_OWNER --port=SUITE_PORT SUITE_NAME

to view the progress of their suite.

Other suite-connecting commands work in the same way; see 12.10.

7.27 Other Things To Try

Almost every feature of cylc can be tested quickly and easily with a simple dummy suite. You can write your own, or start from one of the example suites in /path/to/cylc/examples (see use of cylc import-examples above) - they all run “out the box” and can be copied and modified at will.

8 Suite Name Registration

Cylc commands target suites via their names, which are relative path names under the suite run directory (~/cylc-run/ by default). Suites can be grouped together under sub-directories. E.g.:

$ cylc print -t nwp 
nwp 
 |-oper 
 | |-region1  Local Model Region1       /home/oliverh/cylc-run/nwp/oper/region1 
 | ‘-region2  Local Model Region2       /home/oliverh/cylc-run/nwp/oper/region2 
 ‘-test 
   ‘-region1  Local Model TEST Region1  /home/oliverh/cylc-run/nwp/test/region1

Suites can be pre-registered with a name using the cylc register command. The creates the essential directory structure for the suite, and generates some service files underneath it. Otherwise, cylc run will create these files on suite start up.

9 Suite Definition

 9.1 Suite Definition Directories
 9.2 Suite.rc File Overview
 9.3 Scheduling - Dependency Graphs
 9.4 Runtime - Task Configuration
 9.5 Visualization
 9.6 Parameterized Tasks
 9.7 Jinja2
 9.8 Omitting Tasks At Runtime
 9.9 Naked Dummy Tasks And Strict Validation

Cylc suites are defined in structured, validated, suite.rc files that concisely specify the properties of, and the relationships between, the various tasks managed by the suite. This section of the User Guide deals with the format and content of the suite.rc file, including task definition. Task implementation - what’s required of the real commands, scripts, or programs that do the processing that the tasks represent - is covered in 10; and task job submission - how tasks are submitted to run - is in 11.

9.1 Suite Definition Directories

A cylc suite definition directory contains:

A typical example:

/path/to/my/suite   # suite definition directory 
    suite.rc           # THE SUITE DEFINITION FILE 
    bin/               # scripts and executables used by tasks 
        foo.sh 
        bar.sh 
        ... 
    # (OPTIONAL) any other suite-related files, for example: 
    inc/               # suite.rc include-files 
        nwp-tasks.rc 
        globals.rc 
        ... 
    doc/               # documentation 
    control/           # control files 
    ancil/             # ancillary files 
    ...

9.2 Suite.rc File Overview

Suite.rc files are an extended-INI format with section nesting.

Embedded template processor expressions may also be used in the file, to programatically generate the final suite definition seen by cylc. Currently the Jinja2 template processor is supported (http://jinja.pocoo.org/docs); see 9.7 for examples. In the future cylc may provide a plug-in interface to allow use of other template engines too.

9.2.1 Syntax

The following defines legal suite.rc syntax:

Suites that embed Jinja2 code (see 9.7) must process to raw suite.rc syntax.

9.2.2 Include-Files

Cylc has native support for suite.rc include-files, which may help to organize large suites. Inclusion boundaries are completely arbitrary - you can think of include-files as chunks of the suite.rc file simply cut-and-pasted into another file. Include-files may be included multiple times in the same file, and even nested. Include-file paths can be specified portably relative to the suite definition directory, e.g.:

# include the file $CYLC_SUITE_DEF_PATH/inc/foo.rc: 
%include inc/foo.rc

Editing Temporarily Inlined Suites Cylc’s native file inclusion mechanism supports optional inlined editing:

$ cylc edit --inline SUITE

The suite will be split back into its constituent include-files when you exit the edit session. While editing, the inlined file becomes the official suite definition so that changes take effect whenever you save the file. See cylc prep edit --help for more information.

Include-Files via Jinja2 Jinja2 (9.7) also has template inclusion functionality.

9.2.3 Syntax Highlighting For Suite Definitions

Cylc comes with syntax files for a number of text editors:

$CYLC_DIR/conf/cylc.vim     # vim 
$CYLC_DIR/conf/cylc-mode.el # emacs 
$CYLC_DIR/conf/cylc.lang    # gedit (and other gtksourceview programs) 
$CYLC_DIR/conf/cylc.xml     # kate

Refer to comments at the top of each file to see how to use them.

9.2.4 Gross File Structure

Cylc suite.rc files consist of a suite title and description followed by configuration items grouped under several top level section headings:

9.2.5 Validation

Cylc suite.rc files are automatically validated against a specification that defines all legal entries, values, options, and defaults. This detects formatting errors, typographic errors, illegal items and illegal values prior to run time. Some values are complex strings that require further parsing by cylc to determine their correctness (this is also done during validation). All legal entries are documented in the Suite.rc Reference (A).

The validator reports the line numbers of detected errors. Here’s an example showing a section heading with a missing right bracket:

$ cylc validate my.suite 
    [[special tasks] 
'Section bracket mismatch, line 19'

If the suite.rc file uses include-files cylc view will show an inlined copy of the suite with correct line numbers (you can also edit suites in a temporarily inlined state with cylc edit --inline).

Validation does not check the validity of chosen batch systems.

9.3 Scheduling - Dependency Graphs

The [scheduling] section of a suite.rc file defines the relationships between tasks in a suite - the information that allows cylc to determine when tasks are ready to run. The most important component of this is the suite dependency graph. Cylc graph notation makes clear textual graph representations that are very concise because sections of the graph that repeat at different hours of the day, say, only have to be defined once. Here’s an example with dependencies that vary depending on the particular cycle point:

[scheduling] 
    initial cycle point = 20200401 
    final cycle point = 20200405 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] # validity (hours) 
            graph = """ 
A => B & C   # B and C trigger off A 
A[-PT6H] => A  # Model A restart trigger 
                    """ 
        [[[T06,T18]]] # hours 
            graph = "C => X"

Figure 20 shows the complete suite.rc listing alongside the suite graph. This is a complete, valid, runnable suite (it will use default task runtime properties such as script).


title = "Dependency Example 1" 
[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20200401 
    final cycle point = 20200405 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] # validity (hours) 
            graph = """ 
A => B & C   # B and C trigger off A 
A[-PT6H] => A  # Model A restart trigger 
                    """ 
        [[[T06,T18]]] # hours 
            graph = "C => X" 
[visualization] 
    initial cycle point = 20200401 
    final cycle point = 20200401T06 
    [[node attributes]] 
        X = "color=red"

PIC


Figure 20: Example Suite


9.3.1 Graph String Syntax

Multiline graph strings may contain:

9.3.2 Interpreting Graph Strings

Suite dependency graphs can be broken down into pairs in which the left side (which may be a single task or family, or several that are conditionally related) defines a trigger for the task or family on the right. For instance the “word graph” C triggers off B which triggers off A can be deconstructed into pairs C triggers off B and B triggers off A. In this section we use only the default trigger type, which is to trigger off the upstream task succeeding; see 9.3.5 for other available triggers.

In the case of cycling tasks, the triggers defined by a graph string are valid for cycle points matching the list of hours specified for the graph section. For example this graph:

[scheduling] 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "A => B"

implies that B triggers off A for cycle points in which the hour matches 00 or 12.

To define inter-cycle dependencies, attach an offset indicator to the left side of a pair:

[scheduling] 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "A[-PT12H] => B"

This means B[time] triggers off A[time-PT12H] (12 hours before) for cycle points with hours matching 00 or 12. time is implicit because this keeps graphs clean and concise, given that the majority of tasks will typically depend only on others with the same cycle point. Cycle point offsets can only appear on the left of a pair, because a pairs define triggers for the right task at cycle point time. However, A => B[-PT6H], which is illegal, can be reformulated as a future trigger A[+PT6H] => B (see 9.3.5.11). It is also possible to combine multiple offsets within a cycle point offset e.g.

[scheduling] 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "A[-P1D-PT12H] => B"

This means that B[Time] triggers off A[time-P1D-PT12H] (1 day and 12 hours before).

Triggers can be chained together. This graph:

    graph = """A => B  # B triggers off A 
               B => C  # C triggers off B"""

is equivalent to this:

    graph = "A => B => C"

Each trigger in the graph must be unique but the same task can appear in multiple pairs or chains. Separately defined triggers for the same task have an AND relationship. So this:

    graph = """A => X  # X triggers off A 
               B => X  # X also triggers off B"""

is equivalent to this:

    graph = "A & B => X"  # X triggers off A AND B

In summary, the branching tree structure of a dependency graph can be partitioned into lines (in the suite.rc graph string) of pairs or chains, in any way you like, with liberal use of internal white space and comments to make the graph structure as clear as possible.

# B triggers if A succeeds, then C and D trigger if B succeeds: 
    graph = "A => B => C & D" 
# which is equivalent to this: 
    graph = """A => B => C 
               B => D""" 
# and to this: 
    graph = """A => B => D 
               B => C""" 
# and to this: 
    graph = """A => B 
               B => C 
               B => D""" 
# and it can even be written like this: 
    graph = """A => B # blank line follows: 
 
               B => C # comment ... 
               B => D"""

Splitting Up Long Graph Lines It is not necessary to use the general line continuation marker \ to split long graph lines. Just break at dependency arrows, or split long chains into smaller ones. This graph:

    graph = "A => B => C"

is equivalent to this:

    graph = """A => B => 
                 C"""

and also to this:

    graph = """A => B 
               B => C"""

9.3.3 Graph Types

A suite definition can contain multiple graph strings that are combined to generate the final graph.

One-off (Non-Cycling) Figure 21 shows a small suite of one-off non-cycling tasks; these all share a single cycle point (1) and don’t spawn successors (once they’re all finished the suite just exits). The integer 1 attached to each graph node is just an arbitrary label here.


title = some one-off tasks 
[scheduling] 
    [[dependencies]] 
        graph = "foo => bar & baz => qux"

PIC


Figure 21: One-off (Non-Cycling) Tasks.


Cycling Graphs For cycling tasks the graph section heading defines a sequence of cycle points for which the subsequent graph section is valid. Figure 22 shows a small suite of cycling tasks.


title = some cycling tasks 
# (no dependence between cycle points) 
[scheduling] 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "foo => bar & baz => qux"

PIC


Figure 22: Cycling Tasks.


9.3.4 Graph Section Headings

Graph section headings define recurrence expressions, the graph within a graph section heading defines a workflow at each point of the recurrence. For example in the following scenario:

[scheduling] 
    [[dependencies]] 
        [[[ T06 ]]]  # A graph section heading 
            graph = foo => bar

T06 means ”Run every day starting at 06:00 after the initial cycle point”. Cylc allows you to start (or end) at any particular time, repeat at whatever frequency you like, and even optionally limit the number of repetitions.

Graph section heading can also be used with integer cycling see 9.3.4.7.

Syntax Rules Date-time cycling information is made up of a starting date-time, an interval, and an optional limit.

The time is assumed to be in the local time zone unless you set [cylc]cycle point time zone or [cylc]UTC mode. The calendar is assumed to be the proleptic Gregorian calendar unless you set [scheduling]cycling mode.

The syntax for representations is based on the ISO 8601 date-time standard. This includes the representation of date-time, interval. What we define for cylc’s cycling syntax is our own optionally-heavily-condensed form of ISO 8601 recurrence syntax. The most common full form is: R[limit?]/[date-time]/[interval]. However, we allow omitting information that can be guessed from the context (rules below). This means that it can be written as:

R[limit?]/[date-time] 
R[limit?]//[interval] 
[date-time]/[interval] 
R[limit?] # Special limit of 1 case 
[date-time] 
[interval]

with example graph headings for each form being:

[[[ R5/T00 ]]]           # Run 5 times at 00:00 every day 
[[[ R//PT1H ]]]          # Run every hour (Note the R// is redundant) 
[[[ 20000101T00Z/P1D ]]] # Run every day starting at 00:00 1st Jan 2000 
[[[ R1 ]]]               # Run once at the initial cycle point 
[[[ 20000101T00Z ]]]     # Run once at 00:00 1st Jan 2000 
[[[ P1Y ]]]              # Run every year

Note that T00 is an example of [date-time], with an inferred 1 day period and no limit.

Where some or all date-time information is omitted, it is inferred to be relative to the initial date-time cycle point. For example, T00 by itself would mean the next occurrence of midnight that follows, or is, the initial cycle point. Entering +PT6H would mean 6 hours after the initial cycle point. Entering -P1D would mean 1 day before the initial cycle point. Entering no information for the date-time implies the initial cycle point date-time itself.

Where the interval is omitted and some (but not all) date-time information is omitted, it is inferred to be a single unit above the largest given specific date-time unit. For example, the largest given specific unit in T00 is hours, so the inferred interval is 1 day (daily), P1D.

Where the limit is omitted, unlimited cycling is assumed. This will be bounded by the final cycle point’s date-time if given.

Another supported form of ISO 8601 recurrence is: R[limit?]/[interval]/[date-time]. This form uses the date-time as the end of the cycling sequence rather than the start. For example, R3/P5D/20140430T06 means:

20140420T06 
20140425T06 
20140430T06

This kind of form can be used for specifying special behaviour near the end of the suite, at the final cycle point’s date-time. We can also represent this in cylc with a collapsed form:

R[limit?]/[interval] 
R[limit?]//[date-time] 
[interval]/[date-time]

So, for example, you can write:

[[[ R1//+P0D ]]]  # Run once at the final cycle point 
[[[ R5/P1D ]]]    # Run 5 times, every 1 day, ending at the final 
                  # cycle point 
[[[ P2W/T00 ]]]   # Run every 2 weeks ending at 00:00 following 
                  # the final cycle point 
[[[ R//T00 ]]]    # Run every 1 day ending at 00:00 following the 
                  # final cycle point

Referencing The Initial And Final Cycle Points For convenience the caret and dollar symbols may be used as shorthand for the initial and final cycle points. Using this shorthand you can write:

[[[ R1/^+PT12H ]]]  # Repeat once 12 hours after the initial cycle point 
                    # R[limit]/[date-time] 
                    # Equivalent to [[[ R1/+PT12H ]]] 
[[[ R1/$ ]]]        # Repeat once at the final cycle point 
                    # R[limit]/[date-time] 
                    # Equivalent to [[[ R1//+P0D ]]] 
[[[ $-P2D/PT3H ]]]  # Repeat 3 hourly starting two days before the 
                    # [date-time]/[interval] 
                    # final cycle point

Note that there can be multiple ways to write the same headings, for instance the following all run once at the final cycle point:

[[[ R1/P0Y ]]]      # R[limit]/[interval] 
[[[ R1/P0Y/$ ]]]    # R[limit]/[interval]/[date-time] 
[[[ R1/$ ]]]        # R[limit]/[date-time]

Excluding Dates Datetimes can be excluded from a recurrence by an exclamation mark for example [[[PT1D!20000101 ]]] means run daily except on the first of January 2000.

This syntax can be used to exclude one or multiple datetimes from a recurrence. Multiple datetimes are excluded using the syntax [[[PT1D!(20000101,20000102,...) ]]]. All datetimes listed within the parentheses after the exclamation mark will be excluded. Note that the ^ and $ symbols (shorthand for the initial and final cycle points) are both datetimes so [[[T12!$-PT1D ]]] is valid.

If using a run limit in combination with an exclusion, the heading might not run the number of times specified in the limit. For example in the following suite foo will only run once as its second run has been excluded.

[scheduling] 
    initial cycle point = 20000101T00Z 
    final cycle point = 20000105T00Z 
    [[dependencies]] 
        [[[ R2/P1D!20000102 ]]] 
            graph = foo

How Multiple Graph Strings Combine For a cycling graph with multiple validity sections for different hours of the day, the different sections add to generate the complete graph. Different graph sections can overlap (i.e. the same hours may appear in multiple section headings) and the same tasks may appear in multiple sections, but individual dependencies should be unique across the entire graph. For example, the following graph defines a duplicate prerequisite for task C:

[scheduling] 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] 
            graph = "A => B => C" 
        [[[T06,T18]]] 
            graph = "B => C => X" 
            # duplicate prerequisite: B => C already defined at T06, T18

This does not affect scheduling, but for the sake of clarity and brevity the graph should be written like this:

[scheduling] 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] 
            graph = "A => B => C" 
        [[[T06,T18]]] 
            # X triggers off C only at 6 and 18 hours 
            graph = "C => X"

Advanced Examples The following examples show the various ways of writing graph headings in cylc.

[[[ R1 ]]]         # Run once at the initial cycle point 
[[[ P1D ]]]        # Run every day starting at the initial cycle point 
[[[ PT5M ]]]       # Run every 5 minutes starting at the initial cycle 
                   # point 
[[[ T00/P2W ]]]    # Run every 2 weeks starting at 00:00 after the 
                   # initial cycle point 
[[[ +P5D/P1M ]]]   # Run every month, starting 5 days after the initial 
                   # cycle point 
[[[ R1/T06 ]]]     # Run once at 06:00 after the initial cycle point 
[[[ R1/P0Y ]]]     # Run once at the final cycle point 
[[[ R1/$ ]]]       # Run once at the final cycle point (alternative 
                   # form) 
[[[ R1/$-P3D ]]]   # Run once three days before the final cycle point 
[[[ R3/T0830 ]]]   # Run 3 times, every day at 08:30 after the initial 
                   # cycle point 
[[[ R3/01T00 ]]]   # Run 3 times, every month at 00:00 on the first 
                   # of the month after the initial cycle point 
[[[ R5/W-1/P1M ]]] # Run 5 times, every month starting on Monday 
                   # following the initial cycle point 
[[[ T00!^ ]]]      # Run at the first occurrence of T00 that isn't the 
                   # initial cycle point 
[[[ PT1D!20000101 ]]]  # Run every day days excluding 1st Jan 2000 
[[[ 20140201T06/P1D ]]]    # Run every day starting at 20140201T06 
[[[ R1/min(T00,T06,T12,T18) ]]]  # Run once at the first instance 
                                 # of either T00, T06, T12 or T18 
                                 # starting at the initial cycle 
                                 # point

Advanced Starting Up Dependencies that are only valid at the initial cycle point can be written using the R1 notation (e.g. as in 7.23.3. For example:

[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130812T00 
    [[dependencies]] 
        [[[R1]]] 
            graph = "prep => foo" 
        [[[T00]]] 
            graph = "foo[-P1D] => foo => bar"

In the example above, R1 implies R1/20130808T00, so prep only runs once at that cycle point (the initial cycle point). At that cycle point, foo will have a dependence on prep - but not at subsequent cycle points.

However, it is possible to have a suite that has multiple effective initial cycles - for example, one starting at T00 and another starting at T12. What if they need to share an initial task?

Let’s suppose that we add the following section to the suite example above:

[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130812T00 
    [[dependencies]] 
        [[[R1]]] 
            graph = "prep => foo" 
        [[[T00]]] 
            graph = "foo[-P1D] => foo => bar" 
        [[[T12]]] 
            graph = "baz[-P1D] => baz => qux"

We’ll also say that there should be a starting dependence between prep and our new task baz - but we still want to have a single prep task, at a single cycle.

We can write this using a special case of the task[-interval] syntax - if the interval is null, this implies the task at the initial cycle point.

For example, we can write our suite like 23.


[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130812T00 
    [[dependencies]] 
        [[[R1]]] 
            graph = "prep" 
        [[[R1/T00]]] 
# ^ implies the initial cycle point: 
     graph = "prep[^] => foo" 
        [[[R1/T12]]] 
# ^ is initial cycle point, as above: 
     graph = "prep[^] => baz" 
        [[[T00]]] 
     graph = "foo[-P1D] => foo => bar" 
        [[[T12]]] 
     graph = "baz[-P1D] => baz => qux" 
[visualization] 
    initial cycle point = 20130808T00 
    final cycle point = 20130810T00 
    [[node attributes]] 
        foo = "color=red" 
        bar = "color=orange" 
        baz = "color=green" 
        qux = "color=blue"

PIC


Figure 23: Staggered Start Suite


This neatly expresses what we want - a task running at the initial cycle point that has one-off dependencies with other task sets at different cycles.


[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20130808T00 
    final cycle point = 20130808T18 
    [[dependencies]] 
        [[[R1]]] 
            graph = "setup_foo => foo" 
        [[[+PT6H/PT6H]]] 
            graph = """ 
                foo[-PT6H] => foo 
                foo => bar 
            """ 
[visualization] 
    initial cycle point = 20130808T00 
    final cycle point = 20130808T18 
    [[node attributes]] 
        foo = "color=red" 
        bar = "color=orange"

PIC


Figure 24: Restricted First Cycle Point Suite


A different kind of requirement is displayed in Figure 24. Usually, we want to specify additional tasks and dependencies at the initial cycle point. What if we want our first cycle point to be entirely special, with some tasks missing compared to subsequent cycle points?

In Figure 24, bar will not be run at the initial cycle point, but will still run at subsequent cycle points. [[[+PT6H/PT6H]]] means start at +PT6H (6 hours after the initial cycle point) and then repeat every PT6H (6 hours).

Some suites may have staggered start-up sequences where different tasks need running once but only at specific cycle points, potentially due to differing data sources at different cycle points with different possible initial cycle points. To allow this cylc provides a min( ) function that can be used as follows:

[cylc] 
    UTC mode = True 
[scheduling] 
    initial cycle point = 20100101T03 
    [[dependencies]] 
        [[[R1/min(T00,T12)]]] 
            graph = "prep1 => foo" 
        [[[R1/min(T06,T18)]]] 
            graph = "prep2 => foo" 
        [[[T00,T06,T12,T18]]] 
            graph = "foo => bar"

In this example the initial cycle point is 20100101T03, so the prep1 task will run once at 20100101T12 and the prep2 task will run once at 20100101T06 as these are the first cycle points after the initial cycle point in the respective min( ) entries.

Integer Cycling In addition to non-repeating and date-time cycling workflows, cylc can do integer cycling for repeating workflows that are not date-time based.

To construct an integer cycling suite, set [scheduling]cycling mode = integer, and specify integer values for the initial and (optional) final cycle points. The notation for intervals, offsets, and recurrences (sequences) is similar to the date-time cycling notation, except for the simple integer values.

The full integer recurrence expressions supported are:

But, as for date-time cycling, sequence start and end points can be omitted where suite initial and final cycle points can be assumed. Some examples:

[[[ R1 ]]]        # Run once at the initial cycle point 
                  # (short for R1/initial-point/?) 
[[[ P1 ]]]        # Repeat with step 1 from the initial cycle point 
                  # (short for R/initial-point/P1) 
[[[ P5 ]]]        # Repeat with step 5 from the initial cycle point 
                  # (short for R/initial-point/P5) 
[[[ R2//P2 ]]]    # Run twice with step 3 from the initial cycle point 
                  # (short for R2/initial-point/P2) 
[[[ R/+P1/P2 ]]]  # Repeat with step 2, from 1 after the initial cycle point 
[[[ R2/P2 ]]]     # Run twice with step 2, to the final cycle point 
                  # (short for R2/P2/final-point) 
[[[ R1/P0 ]]]     # Run once at the final cycle point 
                  # (short for R1/P0/final-point)

Example The tutorial illustrates integer cycling in 7.23.4, and $CYLC_DIR/examples/satellite/ is a self-contained example of a realistic use for integer cycling. It simulates the processing of incoming satellite data: each new dataset arrives after a random (as far as the suite is concerned) interval, and is labeled by an arbitrary (as far as the suite is concerned) ID in the filename. A task called get_data at the top of the repeating workflow waits on the next dataset and, when it finds one, moves it to a cycle-point-specific shared workspace for processing by the downstream tasks. When get_data.1 finishes, get_data.2 triggers and begins waiting for the next dataset at the same time as the downstream tasks in cycle point 1 are processing the first one, and so on. In this way multiple datasets can be processed at once if they happen to come in quickly. A single shutdown task runs at the end of the final cycle to collate results. The suite graph is shown in Figure 25.


PIC


Figure 25: The examples/satellite integer suite


Advanced Integer Cycling Syntax The same syntax used to reference the initial and final cycle points introduced in 9.3.4.2) for use with datetime cycling can also be used for integer cycling. For example you can write:

[[[ R1/^ ]]]     # Run once at the initial cycle point 
[[[ R1/$ ]]]     # Run once at the final cycle point 
[[[ R3/^/P2 ]]]  # Run three times with step two starting at the 
                 # initial cycle point

Likewise the syntax introduced in 9.3.4.3 for excluding a particular point from a recurrence also works for integer cycling. For example:

[[[ R/P4!8 ]]]       # Run with step 4, to the final cycle point 
                     # but not at point 8 
[[[ R3/3/P2!5 ]]]    # Run with step 2 from point 3 but not at 
                     # point 5 
[[[ R/+P1/P6!14 ]]]  # Run with step 6 from 1 step after the 
                     # initial cycle point but not at point 14

9.3.5 Trigger Types

Trigger type, indicated by :type after the upstream task (or family) name, determines what kind of event results in the downstream task (or family) triggering.

Success Triggers The default, with no trigger type specified, is to trigger off the upstream task succeeding:

# B triggers if A SUCCEEDS: 
    graph = "A => B"

For consistency and completeness, however, the success trigger can be explicit:

# B triggers if A SUCCEEDS: 
    graph = "A => B" 
# or: 
    graph = "A:succeed => B"

Failure Triggers To trigger off the upstream task reporting failure:

# B triggers if A FAILS: 
    graph = "A:fail => B"

Suicide triggers can be used to remove task B here if A does not fail, see 9.3.5.8.

Start Triggers To trigger off the upstream task starting to execute:

# B triggers if A STARTS EXECUTING: 
    graph = "A:start => B"

This can be used to trigger tasks that monitor other tasks once they (the target tasks) start executing. Consider a long-running forecast model, for instance, that generates a sequence of output files as it runs. A postprocessing task could be launched with a start trigger on the model (model:start => post) to process the model output as it becomes available. Note, however, that there are several alternative ways of handling this scenario: both tasks could be triggered at the same time (foo => model & post), but depending on external queue delays this could result in the monitoring task starting to execute first; or a different postprocessing task could be triggered off a message output for each data file (model:out1 => post1 etc.; see 9.3.5.5), but this may not be practical if the number of output files is large or if it is difficult to add cylc messaging calls to the model.

Finish Triggers To trigger off the upstream task succeeding or failing, i.e. finishing one way or the other:

# B triggers if A either SUCCEEDS or FAILS: 
    graph = "A | A:fail => B" 
# or 
    graph = "A:finish => B"

Message Triggers Tasks can also trigger off custom output messages. These must be registered in the [runtime] section of the emitting task, and reported using the cylc message command in task scripting. The graph trigger notation refers to the item name of the registered output message. The example suite $CYLC_DIR/examples/message-triggers illustrates message triggering.

 
title = "test suite for cylc-6 message triggers" 
 
[scheduling] 
    initial cycle point = 20140801T00 
    final cycle point = 20141201T00 
    [[dependencies]] 
        [[[P2M]]] 
           graph = """foo:out1 => bar 
                      foo[-P2M]:out2 => baz""" 
[runtime] 
    [[foo]] 
        script = """ 
sleep 5 
cylc message "file 1 done" 
sleep 10 
cylc message "file 2 done" 
sleep 10""" 
        [[[outputs]]] 
            out1 = "file 1 done" 
            out2 = "file 2 done" 
    [[bar, baz]] 
        script = sleep 10

Job Submission Triggers It is also possible to trigger off a task submitting, or failing to submit:

# B triggers if A submits successfully: 
    graph = "A:submit => B" 
# D triggers if C fails to submit successfully: 
    graph = "C:submit-fail => D"

A possible use case for submit-fail triggers: if a task goes into the submit-failed state, possibly after several job submission retries, another task that inherits the same runtime but sets a different job submission method and/or host could be triggered to, in effect, run the same job on a different platform.

Conditional Triggers AND operators (&) can appear on both sides of an arrow. They provide a concise alternative to defining multiple triggers separately:

# 1/ this: 
    graph = "A & B => C" 
# is equivalent to: 
    graph = """A => C 
               B => C""" 
# 2/ this: 
    graph = "A => B & C" 
# is equivalent to: 
    graph = """A => B 
               A => C""" 
# 3/ and this: 
    graph = "A & B => C & D" 
# is equivalent to this: 
    graph = """A => C 
               B => C 
               A => D 
               B => D"""

OR operators (|) which result in true conditional triggers, can only appear on the left,2

# C triggers when either A or B finishes: 
    graph = "A | B => C"

Forecasting suites typically have simple conditional triggering requirements, but any valid conditional expression can be used, as shown in Figure 26 (conditional triggers are plotted with open arrow heads).


        graph = """ 
# D triggers if A or (B and C) succeed 
A | B & C => D 
# just to align the two graph sections 
D => W 
# Z triggers if (W or X) and Y succeed 
(W|X) & Y => Z 
                """

PIC


Figure 26: Conditional triggers are plotted with open arrow heads.


Suicide Triggers Suicide triggers take tasks out of the suite. This can be used for automated failure recovery. The suite.rc listing and accompanying graph in Figure 27 show how to define a chain of failure recovery tasks that trigger if they’re needed but otherwise remove themselves from the suite (you can run the AutoRecover.async example suite to see how this works). The dashed graph edges ending in solid dots indicate suicide triggers, and the open arrowheads indicate conditional triggers as usual. Suicide triggers are ignored by default in the graph view, unless you toggle them on with View - Options - Ignore Suicide Triggers.


title = automated failure recovery 
description = """ 
Model task failure triggers diagnosis 
and recovery tasks, which take themselves 
out of the suite if model succeeds. Model 
post processing triggers off model OR 
recovery tasks. 
              """ 
[scheduling] 
    [[dependencies]] 
        graph = """ 
pre => model 
model:fail => diagnose => recover 
model => !diagnose & !recover 
model | recover => post 
                """ 
[runtime] 
    [[model]] 
        # UNCOMMENT TO TEST FAILURE: 
        # script = /bin/false

PIC


Figure 27: Automated failure recovery via suicide triggers.


Note that multiple suicide triggers combine in the same way as other triggers, so this:

foo => !baz 
bar => !baz

is equivalent to this:

foo & bar => !baz

i.e. both foo and bar must succeed for baz to be taken out of the suite. If you really want a task to be taken out if any one of several events occurs then be careful to write it that way:

foo | bar => !baz

A word of warning on the meaning of “bare suicide triggers”. Consider the following suite:

[scheduling] 
    [[dependencies]] 
        graph = "foo => !bar"

Task bar has a suicide trigger but no normal prerequisites (a suicide trigger is not a task triggering prerequisite, it is a task removal prerequisite) so this is entirely equivalent to:

[scheduling] 
    [[dependencies]] 
        graph = """ 
            foo & bar 
           foo => !bar 
                """

In other words both tasks will trigger immediately, at the same time, and then bar will be removed if foo succeeds.

If an active task proxy (currently in the submitted or running states) is removed from the suite by a suicide trigger, a warning will be logged.

Family Triggers Families defined by the namespace inheritance hierarchy ( 9.4) can be used in the graph trigger whole groups of tasks at the same time (e.g. forecast model ensembles and groups of tasks for processing different observation types at the same time) and for triggering downstream tasks off families as a whole. Higher level families, i.e. families of families, can also be used, and are reduced to the lowest level member tasks. Note that tasks can also trigger off individual family members if necessary.

To trigger an entire task family at once:

[scheduling] 
    [[dependencies]] 
        graph = "foo => FAM" 
[runtime] 
    [[FAM]]    # a family (because others inherit from it) 
    [[m1,m2]]  # family members (inherit from namespace FAM) 
        inherit = FAM

This is equivalent to:

[scheduling] 
    [[dependencies]] 
        graph = "foo => m1 & m2" 
[runtime] 
    [[FAM]] 
    [[m1,m2]] 
        inherit = FAM

To trigger other tasks off families we have to specify whether to triggering off all members starting, succeeding, failing, or finishing, or off any members (doing the same). Legal family triggers are thus:

[scheduling] 
    [[dependencies]] 
        graph = """ 
      # all-member triggers: 
    FAM:start-all => one 
    FAM:succeed-all => one 
    FAM:fail-all => one 
    FAM:finish-all => one 
      # any-member triggers: 
    FAM:start-any => one 
    FAM:succeed-any => one 
    FAM:fail-any => one 
    FAM:finish-any => one 
                """

Here’s how to trigger downstream processing after if one or more family members succeed, but only after all members have finished (succeeded or failed):

[scheduling] 
    [[dependencies]] 
        graph = """ 
    FAM:finish-all & FAM:succeed-any => foo 
                """

Writing Efficient Inter-Family Triggering While cylc allows writing dependencies between two families it is important to consider the number of dependencies this will generate. In the following example, each member of FAM2 has dependencies pointing at all the members of FAM1.

[scheduling] 
    [[dependencies]] 
        graph = """ 
    FAM1:succeed-any => FAM2 
                """

Expanding this out, you generate N  M dependencies, where N is the number of members of FAM1 and M is the number of members of FAM2. This can result in high memory use as the number of members of these families grows, potentially rendering the suite impractical for running on some systems.

You can greatly reduce the number of dependencies generated in these situations by putting dummy tasks in the graphing to represent the state of the family you want to trigger off. For example, if FAM2 should trigger off any member of FAM1 succeeding you can create a dummy task FAM1_succeed_any_marker and place a dependency on it as follows:

[scheduling] 
    [[dependencies]] 
        graph = """ 
    FAM1:succeed-any => FAM1_succeed_any_marker => FAM2 
                """ 
[runtime] 
... 
    [[FAM1_succeed_any_marker]] 
        script = true 
...

This graph generates only N + M dependencies, which takes significantly less memory and CPU to store and evaluate.

Inter-Cycle Triggers Typically most tasks in a suite will trigger off others in the same cycle point, but some may depend on others with other cycle points. This notably applies to warm-cycled forecast models, which depend on their own previous instances (see below); but other kinds of inter-cycle dependence are possible too.3 Here’s how to express this kind of relationship in cylc:

[dependencies] 
    [[PT6H]] 
        # B triggers off A in the previous cycle point 
        graph = "A[-PT6H] => B"

inter-cycle and trigger type (or message trigger) notation can be combined:

    # B triggers if A in the previous cycle point fails: 
    graph = "A[-PT6H]:fail => B"

At suite start-up inter-cycle triggers refer to a previous cycle point that does not exist. This does not cause the dependent task to wait indefinitely, however, because cylc ignores triggers that reach back beyond the initial cycle point. That said, the presence of an inter-cycle trigger does normally imply that something special has to happen at start-up. If a model depends on its own previous instance for restart files, for instance, then an initial set of restart files has to be generated somehow or the first model task will presumably fail with missing input files. There are several ways to handle this in cylc using different kinds of one-off (non-cycling) tasks that run at suite start-up. They are illustrated in the Tutorial (7.23.2); to summarize here briefly:

R1, or R1/date-time tasks are the recommended way to specify unusual start up conditions. They allow you to specify a clean distinction between the dependencies of initial cycles and the dependencies of the subsequent cycles.

Initial tasks can be used for real model cold-start processes, whereby a warm-cycled model at any given cycle point can in principle have its inputs satisfied by a previous instance of itself, or by an initial task with (nominally) the same cycle point.

In effect, the R1 task masquerades as the previous-cycle-point trigger of its associated cycling task. At suite start-up initial tasks will trigger the first cycling tasks, and thereafter the inter-cycle trigger will take effect.

If a task has a dependency on another task in a different cycle point, the dependency can be written using the [offset] syntax such as [-PT12H] in foo[-PT12H] => foo. This means that foo at the current cycle point depends on a previous instance of foo at 12 hours before the current cycle point. Unlike the cycling section headings (e.g. [[[T00,T12]]]), dependencies assume that relative times are relative to the current cycle point, not the initial cycle point.

However, it can be useful to have specific dependencies on tasks at or near the initial cycle point. You can switch the context of the offset to be the initial cycle point by using the caret symbol: ^.

For example, you can write foo[^] to mean foo at the initial cycle point, and foo[^+PT6H] to mean foo 6 hours after the initial cycle point. Usually, this kind of dependency will only apply in a limited number of cycle points near the start of the suite, so you may want to write it in R1-based cycling sections. Here’s the example inter-cycle R1 suite from above again.

[scheduling] 
    [[dependencies]] 
        [[[R1]]] 
            graph = "prep" 
        [[[R1/T00,R1/T12]]] 
            graph = "prep[^] => foo" 
        [[[T00,T12]]] 
            graph = "foo[-PT12H] => foo => bar"

You can see there is a dependence on the initial R1 task prep for foo at the first T00 cycle point, and at the first T12 cycle point. Thereafter, foo just depends on its previous (12 hours ago) instance.

Finally, it is also possible to have a dependency on a task at a specific cycle point.

[scheduling] 
    [[dependencies]] 
        [[[R1/20200202]]] 
            graph = "baz[20200101] => qux"

However, in a long running suite, a repeating cycle should avoid having a dependency on a task with a specific cycle point (including the initial cycle point) - as it can currently cause performance issue. In the following example, all instances of qux will depend on baz.20200101, which will never be removed from the task pool.:

[scheduling] 
    initial cycle point = 2010 
    [[dependencies]] 
        # Can cause performance issue! 
        [[[P1D]]] 
            graph = "baz[20200101] => qux"

Special Sequential Tasks If a cycling task does not generate files required by its own successor, then successive instances can run in parallel if the opportunity arises. However, if such a task would interfere with its own siblings for internal reasons (e.g. use of a hardwired non cycle dependent temporary file or similar) then it can be forced to run sequentially. This can be done with explicit inter-cycle triggers in the graph:

[scheduling] 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "foo[-PT12H] => foo => bar"

or by declaring the task to be sequential:

[scheduling] 
    [[special tasks]] 
        sequential = foo 
    [[dependencies]] 
        [[[T00,T12]]] 
            graph = "foo => bar"

The sequential declaration also results in each instance of foo triggering off its own predecessor, exactly as in the explicit version. The only difference is that implicit triggers will not appear in graph visualizations. The implicit version can also be considerably simpler when the task appears in multiple graph sections or in a non-uniform cycling sequence: this suite:

[scheduling] 
    [[special tasks]] 
        sequential = foo 
    [[dependencies]] 
        [[[T00,T03,T11]]] 
            graph = "foo => bar"

is equivalent to this one:

[scheduling] 
    [[dependencies]] 
        [[[T00,T03,T11]]] 
            graph = "foo => bar" 
        [[[T00]]] 
            graph = "foo[-PT13H] => foo" 
        [[[T03]]] 
            graph = "foo[-PT3H] => foo" 
        [[[T11]]] 
            graph = "foo[-PT8H] => foo"

Future Triggers Cylc also supports inter-cycle triggering off tasks “in the future” (with respect to cycle point):

[[dependencies]] 
    [[[T00,T06,T12,T18]]] 
        graph = """ 
    # A runs in this cycle: 
            A 
    # B in this cycle triggers off A in the next cycle. 
            A[PT6H] => B 
        """

(Recall that A[t+PT6H] can run before B[t] because tasks in cylc have private cycle points). Future triggers present a problem at the suite shutdown rather than at start-up. Here, B at the final cycle point wants to trigger off an instance of A that will never exist because it is beyond the suite stop point. Consequently cylc prevents tasks from spawning successors that depend on other tasks beyond the stop point.

Clock Triggers In addition to depending on other tasks (and on external events - see 9.3.5.16) tasks can depend on the wall clock: specifically, they can trigger off a wall clock time expressed as an offset from their own cycle point:

[scheduling] 
    [[special tasks]] 
        clock-trigger = foo(PT2H) 
    [[dependencies]] 
        [[[T00]]] 
            graph = foo

Here, foo[2015-08-23T00] would trigger (other dependencies allowing) when the wall clock time reaches 2015-08-23T02. Clock-trigger offsets are normally positive, to trigger some time after the wall-clock time is equal to task cycle point.

Clock-triggers have no effect on scheduling if the suite is running sufficiently far behind the clock (e.g. after a delay, or because it is processing archived historical data) that the trigger times, which are relative to task cycle point, have already passed.

Clock-Expire Triggers Tasks can be configured to expire - i.e. to skip job submission and enter the expired state - if they are too far behind the wall clock when they become ready to run, and other tasks can trigger off this. As a possible use case, consider a cycling task that copies the latest of a set of files to overwrite the previous set: if the task is delayed by more than one cycle there may be no point in running it because the freshly copied files will just be overwritten immediately by the next task instance as the suite catches back up to real time operation. Clock-expire tasks are configured like clock-trigger tasks, with a date-time offset relative to cycle point (A.3.11.2). The offset should be positive to make the task expire if the wall-clock time has gone beyond the cycle point. Triggering off an expired task typically requires suicide triggers to remove the workflow that runs if the task has not expired. Here a task called copy expires, and its downstream workflow is skipped, if it is more than one day behind the wall-clock (see also examples/clock-expire):

[cylc] 
   cycle point format = %Y-%m-%dT%H 
[scheduling] 
    initial cycle point = 2015-08-15T00 
    [[special tasks]] 
        clock-expire = copy(-P1D) 
    [[dependencies]] 
        [[[P1D]]] 
            graph = """ 
        model[-P1D] => model => copy => proc 
              copy:expired => !proc"""

External Triggers In addition to depending on other tasks (and on the wall clock - see 9.3.5.14) tasks can trigger off events reported by an external system. For example, an external process could detect incoming data on an ftp server, and then notify a suite containing a task to retrieve the new data for processing. This is an alternative to long-running tasks that poll for external events.

Note that cylc does not currently support triggering off “filesystem events” (e.g. inotify on Linux). However, external watcher processes can use filesystem events to detect triggering conditions, if that is appropriate, before notifying a suite with our general external event system.

The external triggering process must call cylc ext-trigger with the name of the target suite, the message that identifies this type of event to the suite, and an ID that distinguishes this particular event instance from others (the name of the target task or its current cycle point is not required). The event ID is just an arbitary string to cylc, but it typically identifies the filename(s) of the latest dataset in some way. When the suite daemon receives the external event notification it will trigger the next instance of any task waiting on that trigger (whatever its cycle point) and then broadcast (see 12.19) the event ID to the cycle point of the triggered task as $CYLC_EXT_TRIGGER_ID. Downstream tasks with the same cycle point therefore know the new event ID too and can use it, if they need to, to identify the same new dataset. In this way a whole workflow can be associated with each new dataset, and multiple datasets can be processed in parallel if they happen to arrive in quick succession.

An externally-triggered task must register the event it waits on in the suite scheduling section:

# suite "sat-proc" 
[scheduling] 
    cycling mode = integer 
    initial cycle point = 1 
    [[special tasks]] 
        external-trigger = get-data("new sat X data avail") 
    [[dependencies]] 
        [[[P1]]] 
            graph = get-data => conv-data => products

Then, each time a new dataset arrives the external detection system should notify the suite like this:

$ cylc ext-trigger sat-proc "new sat X data avail" passX12334a

where “sat-proc” is the suite name and “passX12334a” is the ID string for the new event. The suite passphrase must be installed on triggering account.

Note that only one task in a suite can trigger off a particular external message. Other tasks can trigger off the externally triggered task as required, of course.

$CYLC_DIR/examples/satellite/ext-triggers/suite.rc is a working example of a simulated satellite processing suite.

External triggers are not normally needed in date-time cycling suites driven by real time data that comes in at regular intervals. In these cases a data retrieval task can be clock-triggered (and have appropriate retry intervals supplied) to submit at the expected data arrival time, so little time if any is wasted in polling. However, if the arrival time of the cycle-point-specific data is highly variable, external triggering may be used with the cycle point embedded in the message:

# suite "data-proc" 
[scheduling] 
    initial cycle point = 20150125T00 
    final cycle point   = 20150126T00 
    [[special tasks]] 
        external-trigger = get-data("data arrived for $CYLC_TASK_CYCLE_POINT") 
    [[dependencies]] 
        [[[T00]]] 
            graph = init-process => get-data => post-process

Once the variable-length waiting is finished, an external detection system should notify the suite like this:

$ cylc ext-trigger data-proc "data arrived for 20150126T00" passX12334a

where “data-proc” is the suite name, the cycle point has replaced the variable in the trigger string, and “passX12334a” is the ID string for the new event. The suite passphrase must be installed on the triggering account. In this case, the event will trigger for the second cycle point but not the first because of the cycle-point matching.

9.3.6 Model Restart Dependencies

Warm-cycled forecast models generate restart files, e.g. model background fields, to initialize the next forecast. This kind of dependence requires an inter-cycle trigger:

[scheduling] 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] 
            graph = "A[-PT6H] => A"

If your model is configured to write out additional restart files to allow one or more cycle points to be skipped in an emergency do not represent these potential dependencies in the suite graph as they should not be used under normal circumstances. For example, the following graph would result in task A erroneously triggering off A[T-24] as a matter of course, instead of off A[T-6], because A[T-24] will always be finished first:

[scheduling] 
    [[dependencies]] 
        [[[T00,T06,T12,T18]]] 
            # DO NOT DO THIS (SEE ACCOMPANYING TEXT): 
            graph = "A[-PT24H] | A[-PT18H] | A[-PT12H] | A[-PT6H] => A"

9.3.7 How The Graph Determines Task Instantiation

A graph trigger pair like foo => bar determines the existence and prerequisites (dependencies) of the downstream task bar, for the cycle points defined by the associated graph section heading. In general it does not say anything about the dependencies or existence of the upstream task foo. However if the trigger has no cycle point offset Cylc will infer that bar must exist at the same cycle points as foo. This is a convenience to allow this:

graph = "foo => bar"

to be written as shorthand for this:

graph = """foo 
           foo => bar"""

(where foo by itself means <nothing> => foo, i.e. the task exists at these cycle points but has no prerequisites - although other prerequisites may be defined for it in other parts of the graph).

Cylc does not infer the existence of the upstream task in offset triggers like foo[-P1D]=> bar because, as explained in Section K.5, a typo in the offset interval should generate an error rather than silently creating tasks on an erroneous cycling sequence.

As a result you need to be careful not to define inter-cycle dependencies that cannot be satisfied at run time. Suite validation catches this kind of error if the existence of the cycle offset task is not defined anywhere at all:

[scheduling] 
    initial cycle point = 2020 
    [[dependencies]] 
        [[[P1Y]]] 
            # ERROR 
            graph = "foo[-P1Y] => bar"
$ cylc validate SUITE 
'ERROR: No cycling sequences defined for foo'

To fix this, use another line in the graph to tell Cylc to define foo at each cycle point:

[scheduling] 
    initial cycle point = 2020 
    [[dependencies]] 
        [[[P1Y]]] 
            graph = """ 
                foo 
                foo[-P1Y] => bar"""

But validation does not catch this kind of error if the offset task is defined only on a different cycling sequence:

[scheduling] 
    initial cycle point = 2020 
    [[dependencies]] 
        [[[P2Y]]] 
            graph = """foo 
                # ERROR 
                foo[-P1Y] => bar"""

This suite will validate OK, but it will stall at runtime with bar waiting on foo[-P1Y] at the intermediate years where it does not exist. The offset [-P1Y] is presumably an error (it should be [-P2Y]), or else another graph line is needed to generate foo instances on the yearly sequence:

[scheduling] 
    initial cycle point = 2020 
    [[dependencies]] 
        [[[P1Y]]] 
            graph = "foo" 
        [[[P2Y]]] 
            graph = "foo[-P1Y] => bar"

Similarly the following suite will validate OK, but it will stall at runtime with bar waiting on foo[-P1Y] in every cycle point, when only a single instance of it exists, at the initial cycle point:

[scheduling] 
    initial cycle point = 2020 
    [[dependencies]] 
        [[[R1]]] 
            graph = foo 
        [[[P1Y]]] 
            # ERROR 
            graph = foo[-P1Y] => bar

9.4 Runtime - Task Configuration

The [runtime] section of a suite definition configures what to execute (and where and how to execute it) when each task is ready to run, in a multiple inheritance hierarchy of namespaces culminating in individual tasks. This allows all common configuration detail to be factored out and defined in one place.

Any namespace can configure any or all of the items defined in the Suite.rc Reference (A).

Namespaces that do not explicitly inherit from others automatically inherit from the root namespace (below).

Nested namespaces define task families that can be used in the graph as convenient shorthand for triggering all member tasks at once, or for triggering other tasks off all members at once - see 9.3.5.9. Nested namespaces can be progressively expanded and collapsed in the dependency graph viewer, and in the gcylc graph and text views. Only the first parent of each namespace (as for single-inheritance) is used for suite visualization purposes.

9.4.1 Namespace Names

Namespace names may contain letters, digits, underscores, and hyphens.

Note that task names need not be hardwired into task implementations because task and suite identity can be extracted portably from the task execution environment supplied by the suite daemon (9.4.7) - then to rename a task you can just change its name in the suite definition.

9.4.2 Root - Runtime Defaults

The root namespace, at the base of the inheritance hierarchy, provides default configuration for all tasks in the suite. Most root items are unset by default, but some have default values sufficient to allow test suites to be defined by dependency graph alone. The script item, for example, defaults to code that prints a message then sleeps for between 1 and 15 seconds and exits. Default values are documented with each item in A. You can override the defaults or provide your own defaults by explicitly configuring the root namespace.

9.4.3 Defining Multiple Namespaces At Once

If a namespace section heading is a comma-separated list of names then the subsequent configuration applies to each list member. Particular tasks can be singled out at run time using the $CYLC_TASK_NAME variable.

As an example, consider a suite containing an ensemble of closely related tasks that each invokes the same script but with a unique argument that identifies the calling task name:

[runtime] 
    [[ENSEMBLE]] 
        script = "run-model.sh $CYLC_TASK_NAME" 
    [[m1, m2, m3]] 
        inherit = ENSEMBLE

For large ensembles Jinja2 template processing can be used to automatically generate the member names and associated dependencies (see 9.7).

9.4.4 Runtime Inheritance - Single

The following listing of the inherit.single.one example suite illustrates basic runtime inheritance with single parents.

 
# SUITE.RC 
title = "User Guide [runtime] example." 
[cylc] 
    required run mode = simulation # (no task implementations) 
[scheduling] 
    initial cycle point = 20110101T06 
    final cycle point = 20110102T00 
    [[dependencies]] 
        [[[T00]]] 
            graph = """foo => OBS 
                 OBS:succeed-all => bar""" 
[runtime] 
    [[root]] # base namespace for all tasks (defines suite-wide defaults) 
        [[[job]]] 
            batch system = at 
        [[[environment]]] 
            COLOR = red 
    [[OBS]]  # family (inherited by land, ship); implicitly inherits root 
        script = run-${CYLC_TASK_NAME}.sh 
        [[[environment]]] 
            RUNNING_DIR = $HOME/running/$CYLC_TASK_NAME 
    [[land]] # a task (a leaf on the inheritance tree) in the OBS family 
        inherit = OBS 
        description = land obs processing 
    [[ship]] # a task (a leaf on the inheritance tree) in the OBS family 
        inherit = OBS 
        description = ship obs processing 
        [[[job]]] 
            batch system = loadleveler 
        [[[environment]]] 
            RUNNING_DIR = $HOME/running/ship  # override OBS environment 
            OUTPUT_DIR = $HOME/output/ship    # add to OBS environment 
    [[foo]] 
        # (just inherits from root) 
 
    # The task [[bar]] is implicitly defined by its presence in the 
    # graph; it is also a dummy task that just inherits from root.

9.4.5 Runtime Inheritance - Multiple

If a namespace inherits from multiple parents the linear order of precedence (which namespace overrides which) is determined by the so-called C3 algorithm used to find the linear method resolution order for class hierarchies in Python and several other object oriented programming languages. The result of this should be fairly obvious for typical use of multiple inheritance in cylc suites, but for detailed documentation of how the algorithm works refer to the official Python documentation here: http://www.python.org/download/releases/2.3/mro/.

The inherit.multi.one example suite, listed here, makes use of multiple inheritance:

 
 
title = "multiple inheritance example" 
 
description = """To see how multiple inheritance works: 
 
 % cylc list -tb[m] SUITE # list namespaces 
 % cylc graph -n SUITE # graph namespaces 
 % cylc graph SUITE # dependencies, collapse on first-parent namespaces 
 
  % cylc get-config --sparse --item [runtime]ops_s1 SUITE 
  % cylc get-config --sparse --item [runtime]var_p2 foo""" 
 
[scheduling] 
    [[dependencies]] 
        graph = "OPS:finish-all => VAR" 
 
[runtime] 
    [[root]] 
    [[OPS]] 
        script = echo "RUN: run-ops.sh" 
    [[VAR]] 
        script = echo "RUN: run-var.sh" 
    [[SERIAL]] 
        [[[directives]]] 
            job_type = serial 
    [[PARALLEL]] 
        [[[directives]]] 
            job_type = parallel 
    [[ops_s1, ops_s2]] 
        inherit = OPS, SERIAL 
 
    [[ops_p1, ops_p2]] 
        inherit = OPS, PARALLEL 
 
    [[var_s1, var_s2]] 
        inherit = VAR, SERIAL 
 
    [[var_p1, var_p2]] 
        inherit = VAR, PARALLEL 
 
[visualization] 
    # NOTE ON VISUALIZATION AND MULTIPLE INHERITANCE: overlapping 
    # family groups can have overlapping attributes, so long as 
    # non-conflictling attributes are used to style each group. Below, 
    # for example, OPS tasks are filled green and SERIAL tasks are 
    # outlined blue, so that ops_s1 and ops_s2 are green with a blue 
    # outline. But if the SERIAL tasks are explicitly styled as "not 
    # filled" (by setting "style=") this will override the fill setting 
    # in the (previously defined and therefore lower precedence) OPS 
    # group, making ops_s1 and ops_s2 unfilled with a blue outline. 
    # Alternatively you can just create a manual node group for ops_s1 
    # and ops_s2 and style them separately. 
    [[node groups]] 
        #(see comment above:) 
        #serial_ops = ops_s1, ops_s2 
    [[node attributes]] 
        OPS = "style=filled", "fillcolor=green" 
        SERIAL = "color=blue" #(see comment above:), "style=" 
        #(see comment above:) 
        #serial_ops = "color=blue", "style=filled", "fillcolor=green"

cylc get-suite-config provides an easy way to check the result of inheritance in a suite. You can extract specific items, e.g.:

$ cylc get-suite-config --item '[runtime][var_p2]script' \ 
    inherit.multi.one 
echo ‘‘RUN: run-var.sh''

or use the --sparse option to print entire namespaces without obscuring the result with the dense runtime structure obtained from the root namespace:

$ cylc get-suite-config --sparse --item '[runtime]ops_s1' inherit.multi.one 
script = echo ‘‘RUN: run-ops.sh'' 
inherit = ['OPS', 'SERIAL'] 
[directives] 
   job_type = serial

Suite Visualization And Multiple Inheritance The first parent inherited by a namespace is also used as the collapsible family group when visualizing the suite. If this is not what you want, you can demote the first parent for visualization purposes, without affecting the order of inheritance of runtime properties:

[runtime] 
    [[BAR]] 
        # ... 
    [[foo]] 
        # inherit properties from BAR, but stay under root for visualization: 
        inherit = None, BAR

9.4.6 How Runtime Inheritance Works

The linear precedence order of ancestors is computed for each namespace using the C3 algorithm. Then any runtime items that are explicitly configured in the suite definition are “inherited” up the linearized hierarchy for each task, starting at the root namespace: if a particular item is defined at multiple levels in the hierarchy, the level nearest the final task namespace takes precedence. Finally, root namespace defaults are applied for every item that has not been configured in the inheritance process (this is more efficient than carrying the full dense namespace structure through from root from the beginning).

9.4.7 Task Execution Environment

The task execution environment contains suite and task identity variables provided by the suite daemon, and user-defined environment variables. The environment is explicitly exported (by the task job script) prior to executing the task script (see 11).

Suite and task identity are exported first, so that user-defined variables can refer to them. Order of definition is preserved throughout so that variable assignment expressions can safely refer to previously defined variables.

Additionally, access to cylc itself is configured prior to the user-defined environment, so that variable assignment expressions can make use of cylc utility commands:

[runtime] 
    [[foo]] 
        [[[environment]]] 
            REFERENCE_TIME = $( cylc util cycletime --offset-hours=6 )

User Environment Variables A task’s user-defined environment results from its inherited [[[environment]]] sections:

[runtime] 
    [[root]] 
        [[[environment]]] 
            COLOR = red 
            SHAPE = circle 
    [[foo]] 
        [[[environment]]] 
            COLOR = blue  # root override 
            TEXTURE = rough # new variable

This results in a task foo with SHAPE=circle, COLOR=blue, and TEXTURE=rough in its environment.

Overriding Environment Variables When you override inherited namespace items the original parent item definition is replaced by the new definition. This applies to all items including those in the environment sub-sections which, strictly speaking, are not “environment variables” until they are written, post inheritance processing, to the task job script that executes the associated task. Consequently, if you override an environment variable you cannot also access the original parent value:

[runtime] 
    [[FOO]] 
        [[[environment]]] 
            COLOR = red 
    [[bar]] 
        inherit = FOO 
        [[[environment]]] 
            tmp = $COLOR        # !! ERROR: $COLOR is undefined here 
            COLOR = dark-$tmp   # !! as this overrides COLOR in FOO.

The compressed variant of this, COLOR = dark-$COLOR, is also in error for the same reason. To achieve the desired result you must use a different name for the parent variable:

[runtime] 
    [[FOO]] 
        [[[environment]]] 
            FOO_COLOR = red 
    [[bar]] 
        inherit = FOO 
        [[[environment]]] 
            COLOR = dark-$FOO_COLOR  # OK

Task Job Script Variables These are variables that can be referenced (but should not be modified) in a task job script.

The task job script may export the following environment variables:

CYLC_DEBUG                      # Debug mode, true or not defined 
CYLC_DIR                        # Location of cylc installation used 
CYLC_VERSION                    # Version of cylc installation used 
 
CYLC_CYCLING_MODE               # Cycling mode, e.g. gregorian 
CYLC_SUITE_FINAL_CYCLE_POINT    # Final cycle point 
CYLC_SUITE_INITIAL_CYCLE_POINT  # Initial cycle point 
CYLC_SUITE_NAME                 # Suite name 
CYLC_UTC                        # UTC mode, True or False 
CYLC_VERBOSE                    # Verbose mode, True or False 
TZ                              # Set to "UTC" in UTC mode or not defined 
 
CYLC_SUITE_RUN_DIR              # Location of the suite run directory in 
                                # job host, e.g. ~/cylc-run/foo 
CYLC_SUITE_DEF_PATH             # Location of the suite definition directory in 
                                # job host, e.g. ~/cylc-run/foo 
CYLC_SUITE_HOST                 # Host running the suite process 
CYLC_SUITE_OWNER                # User ID running the suite process 
CYLC_SUITE_DEF_PATH_ON_SUITE_HOST 
                                # Location of the suite definition directory in