Cylc is a general purpose workflow engine that orchestrates cycling workflows very efficiently. It was designed for production weather, climate, and environmental forecasting systems, but it is not specialized to those domains and is completely agnostic to the applications it manages.

H. Oliver et al., “Workflow Automation for Cycling Systems: The Cylc Workflow Engine”, Computing in Science & Engineering Vol 21, Issue 4, July/Aug 2019. DOI: 10.1109/MCSE.2019.2906593

Next Release:  ()

0% 100%

Cylc Does Cycling Properly

Cylc does not merely repeat-run a workflow (immediately, or on a real time schedule). It "unwinds the loop" to create a single potentially infinite non-cycling workflow composed of repeating tasks. Consequently:

  • Cylc can interleave cycles for fast catch-up from delays, and sustained high-throughput off the clock
  • It seamlessly transitions between fast catch-up and clock-limited real-time scheduling
  • It supports multiple cycle intervals within a single workflow

Why Another Workflow Engine?

Production Ready: Cylc is not just a research tool. It has been used 24/7 since 2010 for production weather forecasting - which is notorious for the size and complexity of its application workflows.


Efficient Cycling: On deployment to NIWA production systems Cylc's automatic cycle interleaving cut ~24 hours of catch-up from delays down to ~30 minutes. (This is the original reason for Cylc. It also predates some current non-cycling systems).


Open Source: Cylc is available on the GPLv3 license and is actively and openly developed on GitHub.


Distributed Architecture: There are no central workflow or database servers to manage, so Cylc has low admin overhead and a small security footprint. Each workflow gets its own ad hoc server program, which runs as the user, and only relies on the filesystem.


Cross-workflow Triggering: Via the upstream workflow database, so the upstream workflow doesn't even have to be running at the time.


Distributed Workflows: Cylc easily manages task jobs on multiple remote hosts.


Workflow Configuration: A human-readable config file, so modest workflows are easy to write, even for non-programmers. This is not "just a static config file" however: efficient programmatic generation of large complex workflows is supported by task inheritance and parameterization, and Python-like general template processing. (A Python API is also planned for Cylc 9).


Workflow Scaling: Single workflows scale to thousands of tasks per cycle (however, Cylc is a distributed system so several smaller workflows may be preferred).


System Scaling: Cylc is built to transparently use a pool of host VMs, with load balancing at start-up, and it scales arbitrarily well with the number of host VMs. Running workflows can self-migrate to another host at maintenance time.


Powerful UIs: GUIs for workflow visualization, and runtime monitoring and control (GTK for Cylc 7; a web UI for Cylc 8, coming next year). The CLI can flexibly operate on many tasks at once, e.g. to re-trigger all failed tasks matching a certain name pattern in a particular cycle point.


Flexibility and Generality: Cylc can be used equally for every kind of workflow in climate, weather, and weather-driven forecasting (for example), in research, test, and production. Portable workflows are feasible that run "out of the box" in different environments; and even at different sites. This can lead to huge savings: transitioning complex systems from research to operations, for instance, is notoriously difficult if the workflows are different; and much worse again if conversion to a different workflow engine is required.


HPC: Cylc has great support for common HPC batch systems (workload managers) like PBS and Slurm. It can also manage plain background jobs on any Linux host.

Remote Job Hosts: If you need to use ssh to reach a remote job host, Cylc supports that as well. This includes remote job query and kill, filesystem configuration and file installation, and job log retrieval. (Some of this is via Rose but soon to be built in to Cylc).


General External Triggering: Trigger tasks off of anything in the external world - just supply a Python function that checks your external condition.


Resilience: Cylc has many features to help deal with system problems, e.g.:


Other Capabilities in Brief:

If you use Cylc please cite it in your publications and let us know if your organization should be listed on this website.