Cylc (“silk”) is a workflow engine for cycling systems - it orchestrates distributed suites of interdependent cycling tasks that may continue to run indefinitely.
There are several reasons why tasks might need to be cycled:
To run successive cycles of an environmental forecasting system (where in real time operation new forecasts are initiated at regular intervals; but in catch-up or historical mode dependencies may allow concurrent cycles).
To split long model runs into a sequence of smaller runs, with associated processing tasks for each chunk.
To run successive steps in some multi-task iterative process, such as for optimizing model parameters.
To process a series of datasets (potentially concurrently, to the extent possible) as they are generated or received.
Cylc was originally developed for operational environmental forecasting at NIWA by Dr Hilary Oliver. It is now an Open Source collaboration involving NIWA, Met Office, and others. It is available under the GPL v3 license.
See also publications and citations