Jinja2

Jinja2 is a templating language often used in web design, with some similarities to Python. It can be used to make a workflow definition more dynamic.

The Jinja2 Language

In Jinja2 statements are wrapped with {% characters, i.e:

{% ... %}

Variables are initialised with the set statement, e.g:

{% set foo = 3 %}

Expressions wrapped with {{ characters will be replaced with the evaluated expression, e.g:

There are {{ foo }} methods for consolidating the flow.cylc file

Would result in:

There are 3 methods for consolidating the flow.cylc file

Loops are written with for statements, e.g:

{% for x in range(foo) %}
   {{ x }}
{% endfor %}

Would result in:

0
1
2

To enable Jinja2 in the flow.cylc file, add the following shebang to the top of the file:

#!Jinja2

For more information see the Jinja2 documentation.

Example

To consolidate the configuration for the get_observations tasks we could define a dictionary of station and ID pairs:

{% set stations = {'aldergrove': 3917,
                   'camborne': 3808,
                   'heathrow': 3772,
                   'shetland': 3005} %}

We could then loop over the stations like so:

{% for station in stations %}
    {{ station }}
{% endfor %}

After processing, this would result in:

aldergrove
camborne
heathrow
shetland

We could also loop over both the stations and corresponding IDs like so:

{% for station, id in stations.items() %}
    {{ station }} - {{ id }}
{% endfor %}

This would result in:

aldergrove - 3917
camborne - 3808
heathrow - 3772
shetland - 3005

Putting this all together, the get_observations configuration could be written as follows:

#!Jinja2

{% set stations = {'aldergrove': 3917,
                   'camborne': 3808,
                   'heathrow': 3772,
                   'shetland': 3005} %}

[scheduler]
    allow implicit tasks = True

[scheduling]
    [[graph]]
        T00/PT3H = """
{% for station in stations %}
            get_observations_{{station}} => consolidate_observations
{% endfor %}
        """

[runtime]
{% for station, id in stations.items() %}
    [[get_observations_{{station}}]]
        script = get-observations
        [[[environment]]]
            SITE_ID = {{ id }}
            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

{% endfor %}

Practical

This practical continues on from the families practical.

3. Use Jinja2 To Avoid Duplication.

   The API_KEY environment variable is used by both the get_observations and get_rainfall tasks. Rather than writing it out multiple times we will use Jinja2 to centralise this configuration.

   At the top of the flow.cylc file add the Jinja2 shebang line. Then copy the value of the API_KEY environment variable and use it to define an API_KEY Jinja2 variable:

   #!Jinja2
   
   {% set API_KEY = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' %}
   

   Next replace the key, where it appears in the workflow, with {{ API_KEY }}:

    [runtime]
        [[get_observations_heathrow]]
            script = get-observations
            [[[environment]]]
                SITE_ID = 3772
   -            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   +            API_KEY = {{ API_KEY }}
        [[get_observations_camborne]]
            script = get-observations
            [[[environment]]]
                SITE_ID = 3808
   -            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   +            API_KEY = {{ API_KEY }}
        [[get_observations_shetland]]
            script = get-observations
            [[[environment]]]
               SITE_ID = 3005
   -            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   +            API_KEY = {{ API_KEY }}
        [[get_observations_aldergrove]]
            script = get-observations
            [[[environment]]]
                SITE_ID = 3917
   -            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   +            API_KEY = {{ API_KEY }}
       [[get_rainfall]]
           script = get-rainfall
           [[[environment]]]
               # The key required to get weather data from the DataPoint service.
               # To use archived data comment this line out.
   -            API_KEY = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   +            API_KEY = {{ API_KEY }}
   

   Check the result with cylc config. The Jinja2 will be processed so you should not see any difference after making these changes.