Cylc (“silk”) is a workflow engine for cycling systems

Use cases for cycling include:

  • Running successive cycles of an environmental forecasting system (where in real time operation new forecasts are initiated at regular intervals; but in catch-up or historical mode dependencies may allow concurrent cycles).

  • Splitting long model runs into a sequence of smaller runs, with associated processing tasks for each chunk.

  • Running successive steps in some multi-task iterative process, such as for optimizing model parameters.

  • Processing a series of datasets (potentially concurrently, to the extent possible) as they are generated or received.

Cylc was originally developed for operational environmental forecasting at NIWA by Dr Hilary Oliver. It is now an Open Source collaboration involving NIWA, Met Office, and others. It is available under the GPL v3 license.

Workflows are defined in a human-readable config file format - so you can use software development power tools for suite development.

Configure scheduling with an efficient graph description notation, and task runtime properties in an efficient inheritance hierarchy (to factor out all commonality).

Cylc respects inter-cycle dependence and dynamically generates new workflow without being constrained by a global cycle loop, so that cycles can interleave naturally for much improved scheduling during catch-up from delays.

Cylc has low admin overhead and a small security footprint, because - as a distributed system - there is no central server process to manage workflows for all users.

Plus many other features to support both clock-triggered real time and free-flow metascheduling in research and operational environments.

Please let us know if your organization should be included in the list of Cylc users.

Here’s the DOI to use when citing Cylc: DOI

See also publications and citations