Slidev - Slidev

An introduction to the Cylc workflow engine
Oliver Sanders (Met Office)
IS-ENES3 has received funding from the European Union's
Horizon 2020 research and innovation programme under
grant agreement No 824084.

Cylc is a
decentralised
distributed
DAG/DCG
scheduler

Cylc is a scheduler

Cylc is a system for workflow definition and execution.

Cylc is a DAG scheduler

Cylc can schedule directed acyclic graphs (DAGs).

Cylc is a DCG scheduler

Cylc can schedule directed cyclic graphs (DCGs).

Cylc is decentralised

Cylc is distributed

Cylc is open source

Cylc runs on unix systems:

$ conda install -c conda-forge cylc-flow
$ pip install cylc-flow

Hello World!

This is a DAG

[scheduling]
    [[graph]]
        R1 = """
            a => b => d => f
            b => c
            e => f
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is a DAG

[scheduling]
    [[graph]]
        R1 = """
            a => b => d => f
            b => c
            e => f
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is a DAG

[scheduling]
    [[graph]]
        R1 = """
            a => b => d => f
            b => c
            e => f
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is a DAG

[scheduling]
    [[graph]]
        R1 = """
            a => b => d => f
            b => c
            e => f
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is a DCG

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    [[graph]]
        P1 = """
            a => b => d => f
            b => c
            e => f

            b[-P1] => b
            f[-P1] => e
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is a DCG

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    [[graph]]
        P1 = """
            a => b => d => f
            b => c
            e => f

            b[-P1] => b
            f[-P1] => e
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is date-cycling DCG

(simulated time)

[scheduling]
    cycling mode = gregorian
    initial cycle point = now
    [[graph]]
        P1D = """
            a => b => d => f
            b => c
            e => f

            b[-P1D] => b
            f[-P1D] => e
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

This is date-cycling DCG

(real time)

[scheduling]
    cycling mode = gregorian
    initial cycle point = now
    [[graph]]
        P1D = """
            @wall_clock => a

            a => b => d => f
            b => c
            e => f

            b[-P1D] => b
            f[-P1D] => e
        """

[runtime]
    [[a, b, c, d, e, f]]
        script = echo "Hello $CYLC_TASK_NAME"

Scheduler Features

Cylc constructs a moving window over a cycling workflow.

∞ Workflows can be infinite.
Xtriggers can respond to external events.
Inter-workflow triggers can trigger off of tasks in other workflows.
Internal queueing mechanism for controlling parallelism and submission load.

Branched graphs with stall detection.

# branch the graph depending on the outcome of "a"
a:x? => x1 => x2
a:y? => y1 => y2
a:z? => z1 => z2

# join the graph back together
x2 | y2 | z2 => b

Schedule tasks with ISO8601 recurrences (with extensions):

P1D = foo        # every day
P1D ! P3D = bar  # every day, except every third day
01T-30 = baz     # every hour, at half past on the first day of the month

Alternate calendar support:

360-day
365-day
366-day

Utilities for date-time arthritic:

$ isodatetime 20000101T00 --offset P1Y --offset P1DT12H
20010102T12

$ isodatetime R/2020/P1Y --max=3
2020-01-01T00:00:00Z
2021-01-01T00:00:00Z
2022-01-01T00:00:00Z

Submit jobs to industry strength batch systems out of the box:

[platforms]
  [[hpc2]]
    hosts = hpc1.login.1
    job runner = pbs
  [[hpc2]]
    hosts = hpc2.login.1
    job runner = sge
  [[cluster]]
    job runner = slurm

Generic interface for defining integration with other job runners.

class SLURMHandler():
    DIRECTIVE_PREFIX = "#SBATCH "
    KILL_CMD_TMPL = "scancel '%(job_id)s'"
    POLL_CMD = "squeue -h"
    REC_ID_FROM_POLL_OUT = re.compile(r"^ *(?P<id>\d+)")
    SUBMIT_CMD_TMPL = "sbatch '%(job)s'"