.. _tutorial-cylc-parameterization: Parameterized Tasks =================== Parameterized tasks (see :term:`parameterization`) provide a way of implicitly looping over tasks without the need for Jinja2. Cylc Parameters --------------- .. ifnotslides:: Parameters are defined in their own section, e.g: .. code-block:: cylc [task parameters] world = Mercury, Venus, Earth .. ifnotslides:: They can then be referenced by writing the name of the parameter in angle brackets, e.g: .. code-block:: cylc [scheduling] [[graph]] R1 = start => hello => end [runtime] [[hello]] script = echo 'Hello World!' .. nextslide:: .. ifnotslides:: When the :cylc:conf:`flow.cylc` file is read by Cylc, the parameters will be expanded. For example the code above is equivalent to: .. code-block:: cylc [scheduling] [[graph]] R1 = """ start => hello_Mercury => end start => hello_Venus => end start => hello_Earth => end """ [runtime] [[hello_Mercury]] script = echo 'Hello World!' [[hello_Venus]] script = echo 'Hello World!' [[hello_Earth]] script = echo 'Hello World!' .. nextslide:: .. ifnotslides:: We can refer to a specific parameter by writing it after an ``=`` sign: .. code-block:: cylc [runtime] [[hello]] script = echo 'Greetings Earth!' Environment Variables --------------------- .. ifnotslides:: The name of the parameter is provided to the job as an environment variable called ``CYLC_TASK_PARAM_`` where ```` is the name of the parameter (in the present case ``world``): .. code-block:: cylc [runtime] [[hello]] script = echo "Hello ${CYLC_TASK_PARAM_world}!" Parameter Types --------------- Parameters can be either strings or integers: .. code-block:: cylc [task parameters] foo = 1..5 bar = 1..5..2 baz = pub, qux, bol .. nextslide:: .. hint:: Remember that by default Cylc automatically inserts an underscore between the task and the parameter, e.g. the following lines are equivalent: .. code-block:: cylc-graph task task_pub .. nextslide:: .. hint:: .. ifnotslides:: When using integer parameters, to prevent confusion, Cylc prefixes the parameter value with the parameter name. For example: .. ifslides:: Cylc prefixes integer parameters with the parameter name: .. code-block:: cylc [scheduling] [[graph]] R1 = """ # task would result in: task_bar1 task_bar3 task_bar5 # task would result in: task_pub task_qux task_bol """ .. nextslide:: .. ifnotslides:: Using parameters the ``get_observations`` configuration could be written like so: .. code-block:: cylc [scheduling] [[graph]] T00/PT3H = """ get_observations => consolidate_observations """ [runtime] [[get_observations]] script = get-observations [[[environment]]] API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx [[get_observations]] [[[environment]]] SITE_ID = 3917 [[get_observations]] [[[environment]]] SITE_ID = 3808 [[get_observations]] [[[environment]]] SITE_ID = 3772 [[get_observations]] [[[environment]]] SITE_ID = 3005 .. nextslide:: .. ifnotslides:: For more information see the `Cylc User Guide`_. .. ifslides:: .. rubric:: This practical continues on from the :ref:`Jinja2 practical `. Next section: :ref:`Which approach to use ` .. _cylc-tutorial-parameters-practical: .. practical:: .. rubric:: This practical continues on from the :ref:`Jinja2 practical `. 4. **Use Parameterization To Consolidate The** ``get_observations`` **Tasks**. Next we will parameterize the ``get_observations`` tasks. Add a parameter called ``station``: .. code-block:: diff +[task parameters] + station = aldergrove, camborne, heathrow, shetland [scheduler] UTC mode = True Remove the four ``get_observations`` tasks and insert the following code in their place: .. code-block:: cylc [[get_observations]] script = get-observations [[[environment]]] API_KEY = {{ API_KEY }} Using ``cylc config`` you should see that Cylc replaces the ```` with each of the stations in turn, creating a new task for each: .. code-block:: bash cylc config . -i "[runtime]" The ``get_observations`` tasks are now missing the ``SITE_ID`` environment variable. Add a new section for each station with a ``SITE_ID``: .. code-block:: cylc [[get_observations]] [[[environment]]] SITE_ID = 3772 .. hint:: The relevant IDs are: * Aldergrove - ``3917`` * Camborne - ``3808`` * Heathrow - ``3772`` * Shetland - ``3005`` .. spoiler:: Solution warning .. code-block:: cylc [[get_observations]] [[[environment]]] SITE_ID = 3917 [[get_observations]] [[[environment]]] SITE_ID = 3808 [[get_observations]] [[[environment]]] SITE_ID = 3772 [[get_observations]] [[[environment]]] SITE_ID = 3005 Using ``cylc config`` you should now see four ``get_observations`` tasks, each with a ``script``, an ``API_KEY`` and a ``SITE_ID``: .. code-block:: bash cylc config . -i "[runtime]" Finally we can use this parameterization to simplify the workflow's graphing. Replace the ``get_observations`` lines in the graph with ``get_observations``: .. code-block:: diff # Repeat every three hours starting at the initial cycle point. PT3H = """ - get_observations_aldergrove => consolidate_observations - get_observations_camborne => consolidate_observations - get_observations_heathrow => consolidate_observations - get_observations_shetland => consolidate_observations + get_observations => consolidate_observations """ .. hint:: The ``cylc config`` command does not expand parameters or families in the graph so you must use ``cylc graph`` to inspect changes to the graphing. #. **Use Parameterization To Consolidate The** ``post_process`` **Tasks**. At the moment we only have one ``post_process`` task (``post_process_exeter``), but suppose we wanted to add a second task for Edinburgh. Create a new parameter called ``site`` and set it to contain ``exeter`` and ``edinburgh``. Parameterize the ``post_process`` task using this parameter. .. hint:: The first argument to the ``post-process`` task is the name of the site. We can use the ``CYLC_TASK_PARAM_site`` environment variable to avoid having to write out this section twice. .. TODO - use parameter environment templates instead of CYLC_TASK_PARAM. .. spoiler:: Solution warning First we must create the ``site`` parameter: .. code-block:: diff [scheduler] UTC mode = True [task parameters] station = aldergrove, camborne, heathrow, shetland + site = exeter, edinburgh Next we parameterize the task in the graph: .. code-block:: diff -get_rainfall => forecast => post_process_exeter +get_rainfall => forecast => post_process And also the runtime: .. code-block:: diff -[[post_process_exeter]] +[[post_process]] # Generate a forecast for Exeter 60 minutes in the future. - script = post-process exeter 60 + script = post-process $CYLC_TASK_PARAM_site 60