Cylc is a general purpose workflow engine with a particular gift for cycling.

What is Cylc?

Cylc is a decentralised, distributed, DAG/DCG scheduler.

Scheduler

Cylc can schedule inter-dependent collections of tasks (aka workflows) to run on platforms of your choice.

Decentralised

Cylc employs a decentralised architecture where each workflow execution is fully independent. This means that workflows are isolated for robustness and Cylc deployments scale horizontally.

Distributed

Cylc workflows can be distributed across a pool of servers to facilitate large multi-user setups. Workflows can distribute their workload across multiple platforms.

DAG / DCG

Cylc can schedule Directed Acyclic Graphs of tasks (i.e. workflows which do not cycle) similar to other tools.

Cylc's unique cycling model allows it to run cyclic workflows too. This makes Cylc especially powerful for cases where later repetitions of tasks depend on the output of earlier instances. Tasks can repeat on different intervals within the same workflow and complex inter-cycle dependencies can be defined between them.

Features

Scalable

Thanks to Cylc's horizontal architecture, deployments can scale to any number of workflows. Cylc's novel scheduling algorithm allows it to schedule complex and even infinite workflows.

Flexible

Start with a simple linear pipeline, grow it into a complex branched cycling workflow.

Production Ready

Used in production since 2010, trusted by institutions worldwide.

Cycling

Cylc's cycling can do much more than just run a DAG to a schedule.

Different parts of your workflow can cycle to different schedules, tasks can have inter-cycle dependencies and multiple cycles can run simultaneously.

Advanced Scheduling

Cylc supports branched graphs where runtime events direct the direction the workflow takes.

Cycling workflows can run in real-time or simulated time with multiple calendars supported. And much more

Powerful User Interfaces

The Cylc dashboard (the GUI) provides a multi-workflow multi-pane interface for monitoring and controlling workflows.

View job logs and outputs, re-run subgraphs, modify the workflow configuration on the fly and more.

Alongside the GUI is a feature complete command line with an interactive in-terminal user interface (the Tui).

Jupyter Integration

Deploy the Cylc dashboard via Jupyter Hub to unlock institution-scale multi-user deployments with fine-grained authorisation allowing users to be granted permissions to view and control workflows running under other user accounts.

Deploy the Cylc dashboard alongside Jupyter Lab for an all-in-one virtual workstation in the browser.

Publications

If you use Cylc in your work, please consider citing us.

Workflow Automation for Cycling Systems: The Cylc Workflow Engine
H. Oliver et al
Computing in Science & Engineering Vol 21, Issue 4, July/Aug 2019. DOI: 10.1109/MCSE.2019.2906593

Cylc: A Workflow Engine for Cycling Systems
Oliver et al., (2018).
Journal of Open Source Software, 3(27), 737. DOI: 10.21105/joss.00737

Citable DOI for Cylc code releases.

Trusted By Sites Worldwide

Support

Community support is available on the Discourse forum .

Commercial support for Cylc is available from Altair, a leading provider of workload management and scheduling resources for the weather industry. Learn more about the Altair Weather Solution .

Testimony

Cylc has been used to orchestrate all climate, weather, and weather-driven forecasting workflows at NIWA, for research and production, since 2010.
On deployment to production at NIWA, Cylc's automatic cycle-interleaving cut catch-up time from serious delays down from ~24 hours to ~30 minutes.
- Hillary Oliver, NIWA

Cylc has been used at the Met Office since 2012 and operationally since 2014.
“Every week we have 300+ users who rely on Cylc. At any one time we typically have 700+ active workflows. Of the 11.7 million jobs run on our HPC systems in February 2020, 99.6% of them were submitted by Cylc.”
- David Matthews, Met Office