Cylc (“silk”) is a workflow engine for cycling systems - it orchestrates
distributed suites of interdependent cycling tasks that may continue to
There are several reasons why tasks might need to be cycled:
To run successive cycles of an environmental forecasting system (where in
real time operation new forecasts are initiated at regular intervals; but in
catch-up or historical mode dependencies may allow concurrent cycles).
To split long model runs into a sequence of smaller runs, with associated
processing tasks for each chunk.
To run successive steps in some multi-task iterative process, such as for
optimizing model parameters.
To process a series of datasets (potentially concurrently, to the extent
possible) as they are generated or received.
Suites are defined in a human-readable config
file format - so you can use software development power tools for suite
Configure scheduling with an efficient graph
description notation, and task runtime properties in an efficient inheritance
hierarchy (to factor out all commonality).
Cylc dynamically generates new workflow
without being constrained by a global cycle loop. Cycles interleave
naturally, suites flow around failed or delayed tasks, and they adapt to
insertion and removal of tasks.
Cylc has low admin overhead and a small
security footprint, because - as a distributed system - there is no central
server process to manage workflows for all users.
Plus many other
features to support both clock-triggered real time and free-flow
metascheduling in research and operational environments.