You may want to repeat the same workflow multiple times. In Cylc this is called cycling, and each repetition is called a cycle.
Each cycle is given a unique label, called a cycle point. For now these will be integers, but they can also be datetimes as we will see in the next section.
To make a workflow repeat we must tell Cylc three things:
- The recurrence.
How often to repeat the workflow (or part of it).
- The initial cycle point.
The cycle point to start from.
- The final cycle point (optional).
We can also tell Cylc where to stop the workflow.
Let’s take the bakery example from the previous section. Bread is baked in batches so the bakery will repeat this workflow for each batch. We can make the workflow repeat by adding three lines:
[scheduling] + cycling mode = integer + initial cycle point = 1 [[graph]] - R1 = """ + P1 = """ buy_ingredients => make_dough pre_heat_oven & make_dough => bake_bread bake_bread => sell_bread & clean_oven """
cycling mode = integertells Cylc to give our cycle points integer labels.
initial cycle point = 1tells Cylc to start counting cycle points from 1.
P1is the recurrence; a
P1graph string repeats at every integer cycle point.
The first three cycles look like this, with the entire workflow repeated at each cycle point:
The number under each task shows which cycle point it belongs to.
We’ve just seen how to write a workflow that repeats every cycle.
Cylc runs tasks as soon as their dependencies are met, regardless of cycle point, so cycles will not necessarily run in order. This can be efficient, but it could also cause problems. For instance we could find ourselves pre-heating the oven in one cycle while still cleaning it in another.
To resolve this we can add dependencies between
cycles, to the graph. To ensure that
clean_oven completes before
pre_heat_oven starts in the next cycle, we can write this:
clean_oven[-P1] => pre_heat_oven
P1 recurrence, the suffix
[-P1] means the previous cycle point,
[-P2] refers back two cycles. The new dependency can be added
to the workflow graph like this:
[scheduling] cycling mode = integer initial cycle point = 1 [[graph]] P1 = """ buy_ingredients => make_dough pre_heat_oven & make_dough => bake_bread bake_bread => sell_bread & clean_oven + clean_oven[-P1] => pre_heat_oven """
And the resulting workflow looks like this:
The intercycle dependency forces the connected tasks, in different cycle points, to run in order.
Note that the
buy_ingredients task has no arrows pointing at it.
This means it has no parent tasks to wait on, upstream in the graph.
buy_ingredients tasks (out to a configurable
runahead limit) want to run straight away.
This could cause our bakery to run into cash-flow problems by purchasing
ingredients too far in advance of using them.
To solve this problem without running out of ingredients, the bakery wants to purchase ingredients two batches ahead. This can be achieved by adding the following dependency:
[scheduling] cycling mode = integer initial cycle point = 1 [[graph]] P1 = """ buy_ingredients => make_dough pre_heat_oven & make_dough => bake_bread bake_bread => sell_bread & clean_oven clean_oven[-P1] => pre_heat_oven + sell_bread[-P2] => buy_ingredients """
This means that
buy_ingredients will run after the
two cycles earlier.
[-P2] suffix references a task two cycles back. For the first two
cycles this doesn’t make sense, so those dependencies (and indeed any before
the initial cycle point) will be ignored.
In the previous examples we used a
P1recurrence to make the workflow repeat at successive integer
cycle points. Similarly
P2 means repeat every other cycle, and so
on. We can use multiple recurrences to build more complex workflows:
[scheduling] cycling mode = integer initial cycle point = 1 [[graph]] # Repeat every cycle. P1 = foo # Repeat every second cycle. P2 = bar # Repeat every third cycle. P3 = baz
We can also tell Cylc where to start a recurrence sequence.
- From the initial cycle point:
By default, recurrences start at the: initial cycle point.
- From an arbitrary cycle point:
We can give a different start point like this:
5/P3means repeat every third cycle, starting from cycle number 5. To run a graph at every other cycle point, use
- Offset from the initial cycle point:
The start point of a recurrence can also be defined as an offset from the initial cycle point For example,
+P5/P3means repeat every third cycle from 5 cycles after the initial cycle point.
In this practical we will turn the workflow of the previous section into a cycling workflow.
If you have not completed the previous practical use the following code for
[scheduler] allow implicit tasks = True [scheduling] [[graph]] R1 = """ a & c => b => d & f d => e """
Create a new workflow.
Create a new source directory
~/cylc-src/, and move into it:
mkdir -p ~/cylc-src/integer-cycling cd ~/cylc-src/integer-cycling
Copy the above code into a
flow.cylcfile in that directory.
Make the workflow cycle.
Add the following lines to your
[scheduling] + cycling mode = integer + initial cycle point = 1 [[graph]] - R1 = """ + P1 = """ a & c => b => d & f d => e """
Visualise the workflow.
Try visualising your workflow using
cylc graph .
You can use the
--cycles) option to draw a box around each cycle:
cylc graph -c .
cylc graphdisplays the first three cycles of the graph, but you can specify the range of cycles on the command line. Here’s how to display cycles
cylc graph . 1 5
Add another recurrence.
Suppose we wanted the
etask to run every other cycle as opposed to every cycle. We can do this by adding another recurrence.
Make the following changes to your
[scheduling] cycling mode = integer initial cycle point = 1 [[graph]] P1 = """ a & c => b => d & f - d => e """ + P2 = """ + d => e + """
cylc graphto see the effect this has on the workflow.
Now we will add three intercycle dependencies:
ffrom the previous cycle and
dfrom the previous cycle and
aevery odd cycle (e.g. 2/d => 3/a).
efrom the previous cycle and
aevery even cycle (e.g. 1/e => 2/a).
Try adding these to your
flow.cylcfile to make your workflow match the diagram below.
P2means every other cycle, from the initial cycle point.
2/P2means every other cycle, from cycle point 2.
[scheduler] allow implicit tasks = True [scheduling] cycling mode = integer initial cycle point = 1 [[graph]] P1 = """ a & c => b => d & f f[-P1] => c # (1) """ P2 = """ d => e d[-P1] => a # (2) """ 2/P2 = """ e[-P1] => a # (3) """