Queues are used to put a limit on the number of tasks that will be active at any one time, even if their dependencies are satisfied. This avoids swamping systems with too many tasks at once.
In this example, our workflow manages a particularly understaffed restaurant.
Create a new workflow called
rose tutorial queues-tutorial cd ~/cylc-run/queues-tutorial
You will now have a
flow.cylc file that looks like this:
[scheduling] [[graph]] R1 = """ open_restaurant => steak1 & steak2 & pasta1 & pasta2 & pasta3 & \ pizza1 & pizza2 & pizza3 & pizza4 steak1 => ice_cream1 steak2 => cheesecake1 pasta1 => ice_cream2 pasta2 => sticky_toffee1 pasta3 => cheesecake2 pizza1 => ice_cream3 pizza2 => ice_cream4 pizza3 => sticky_toffee2 pizza4 => ice_cream5 """ [runtime] [[open_restaurant]] [[MAINS]] [[DESSERT]] [[steak1,steak2,pasta1,pasta2,pasta3,pizza1,pizza2,pizza3,pizza4]] inherit = MAINS [[ice_cream1,ice_cream2,ice_cream3,ice_cream4,ice_cream5]] inherit = DESSERT [[cheesecake1,cheesecake2,sticky_toffee1,sticky_toffee2]] inherit = DESSERT
In graph sections backslash (
\) is a line continuation character i.e. the
following two examples are equivalent:
foo => bar & \ baz
foo => bar & baz
cylc gui then run the workflow:
cylc gui queues-tutorial & cylc play queues-tutorial
You will see that all the
pizza tasks are run
at once, swiftly followed by all the
sticky_toffee tasks as the customers order from the dessert menu.
This will overwhelm our restaurant staff! The chef responsible for
can only handle 3 tasks at any given time, and the
DESSERT chef can only
We need to add some queues. Add a
[queues] section to the
section like so:
[scheduling] [[queues]] [[[mains_chef_queue]]] limit = 3 # Only 3 mains dishes at one time. members = MAINS [[[dessert_chef_queue]]] limit = 2 # Only 2 dessert dishes at one time. members = DESSERT
cylc gui if you have closed it and re-run the workflow.
You should see that there are now never more than 3 active
running and never more than 2 active
DESSERT tasks running.
The customers will obviously have to wait!