Irregular Cycling

We typically schedule tasks on regular intervals, e.g. P1D (every day) or PT1H (every hour), however, sometimes our intervals are irregular.

Exclusions can be used to “subtract” dates or entire recurrences e.g:

PT1H!PT6H

Every hour, except every six hours.

PT1H!(T00, T12)

Every hour, except at 00:00 and 12:00.

However, sometimes we want to schedule tasks on completely irregular intervals or at arbitrary dates. E.g, when working on case studies, you might have a list or range of arbitrary dates to work with.

Simple Example

Get a copy of this example

$ cylc get-resources examples/cycle-over-irregular-dates/simple

This example uses Jinja2 to define the list of dates and write out a scheduling section for each.

#!Jinja2

{%
    set DATES = [
        '2000-01-01T00:00Z',
        '2000-01-01T06:00Z',
        '2000-03-05T12:00Z',
        '2000-05-28T13:36Z',
        '2001-01-05T23:24',
        '2002-04-30T04:20',
    ]
%}

[meta]
    title = Irregular cycling example
    description = """
        A workflow that runs a group of tasks on arbitrary dates.
    """

[scheduling]
    # start cycling at the first date
    initial cycle point = {{ DATES[0] }}
    [[graph]]
        # define the tasks you want to run on startup
        R1 = install

{# loop over the list of dates #}
{% for date in DATES %}
        # schedule the tasks to run at each date
        R1/{{ date }} = """
            # NOTE: install[^] references the task "install" in the first cycle
            install[^] => prep => run_model => plot
        """
{% endfor %}

[runtime]
    [[install]]
    [[prep]]
    [[run_model]]
    [[plot]]

Tip

You can see the result of this Jinja2 code by running the cylc view -p command.

Example with inter-cycle dependencies

Get a copy of this example

$ cylc get-resources examples/cycle-over-irregular-dates/inter-dependent

If you have dependencies between the cycles, you can make this work by using the Jinja2 loop variable.

For example, the previous iteration of the {% for date in DATES %} loop is available as loop.previtem and the next as loop.nextitem.

If you need to make the tasks which cycle on irregular intervals dependent on tasks which cycle on regular intervals, then you might find the strftime function helpful as a way of determining the nearest matching cycle.

#!Jinja2

{%
    set DATES = [
        '2000-01-01T00:00Z',
        '2000-01-01T06:00Z',
        '2000-03-05T12:00Z',
        '2000-05-28T13:36Z',
        '2001-01-05T23:24',
        '2002-04-30T04:20',
    ]
%}

[meta]
    title = Irregular cycling example
    description = """
        A workflow that runs a group of tasks on arbitrary dates
        with inter-cycle dependencies between those dates.
    """

[scheduling]
    initial cycle point = 2000
    [[graph]]
        # define the tasks you want to run on startup
        R1 = install

        # run this graph every year
        P1Y = """
            install[^] => prep
        """

{# loop over the list of dates #}
{% for date in DATES %}
        # schedule the tasks to run at each date
        R1/{{ date }} = """
            # make "run_model" depend on the "prep" task from the same year
            prep[{{ date | strftime('%Y') }}] => run_model => plot

    {# include this for all but the first date #}
    {% if not loop.first %}
            # make the run_model task depend on its previous instance
            run_model[ {{ loop.previtem }} ] => run_model
    {% endif %}
        """
{% endfor %}

[runtime]
    [[install]]
    [[prep]]
    [[run_model]]
    [[plot]]

You can see how the cycles are linked together using the cylc graph command: