Jinja2

Related Tutorial

Cylc supports use of the Jinja2 template processor in workflow configurations. Jinja2 variables, expressions, loop control structures, conditional logic, etc., are processed to generate the final configuration. To the template processor, Jinja2 code is embedded in arbitrary text, but the result after processing must be valid Cylc syntax.

To use Jinja2, put a hash-bang comment in the first line of flow.cylc:

#!jinja2

Template processing is the first thing done on parsing a workflow configuration so Jinja2 can appear anywhere in the file.

Embedded Jinja2 code should be reasonably easy to understand for those with coding experience; but if not, Jinja2 is well documented here.

Uses of Jinja2 in Cylc include:

Inclusion or exclusion of config sections by logical switch, e.g. to make portable workflows

Computation of config values from in input data

Inclusion of files and sub-templates

Loop over parameters to generate groups of similar tasks and associated dependencies - but see Parameterized Tasks for a simpler alternative to this use case

../../_images/jinja2-ensemble-graph.png — The Jinja2 ensemble example workflow graph.

The graph above shows an ensemble of similar tasks generated with a Jinja2 loop:

#!jinja2
{% set N_MEMBERS = 5 %}
[scheduling]
    [[graph]]
        R1 = """
{# generate ensemble dependencies #}
{% for I in range( 0, N_MEMBERS ) %}
            foo => mem_{{ I }} => post_{{ I }} => bar
{% endfor %}
        """

Note that Jinja2 code is encapsulated in curly braces to distinguish it from the surrounding text.

Jinja2 Syntax

Description

{# comment #}

Comment

{% if true %}

Expression

{{ var }}

Print statement

Here is the workflow configuration after Jinja2 processing:

#!jinja2
[scheduling]
    [[graph]]
        R1 = """
            foo => mem_0 => post_0 => bar
            foo => mem_1 => post_1 => bar
            foo => mem_2 => post_2 => bar
            foo => mem_3 => post_3 => bar
            foo => mem_4 => post_4 => bar
        """

This example illustrates Jinja2 loops nicely, but note it is now easier to generate task names automatically with built-in task parameters:

[task parameters]
    m = 0..4
[scheduling]
    [[graph]]
        R1 = "foo => mem<m> => post<m> => bar"

The next example, which generates weather forecasts over a number of cities, is more complex. To add a new city and associated tasks and dependencies just add the new city name to list at the top of the file. It makes use of Jinja2 variables, loops, math, and logical flags to include or exclude tasks.

Tip

This example could also be simplified with built in task parameters

#!Jinja2
[meta]
    title = "Jinja2 city workflow example."
    description = """
        Illustrates use of variables and math expressions, and programmatic
        generation of groups of related dependencies and runtime properties.
    """

[scheduler]
    allow implicit tasks = True

{% set HOST = "SuperComputer" %}
{% set CITIES = 'NewYork', 'Philadelphia', 'Newark', 'Houston', 'SantaFe', 'Chicago' %}
{% set CITYJOBS = 'one', 'two', 'three', 'four' %}
{% set LIMIT_MINS = 20 %}

{% set CLEANUP = True %}

[scheduling]
    initial cycle point = 2011-08-08T12
    [[graph]]
{% if CLEANUP %}
        T23 = "clean"
{% endif %}
        T00,T12 = """
            setup => get_lbc & get_ic # foo
    {% for CITY in CITIES %} {# comment #}
            get_lbc => {{ CITY }}_one
            get_ic => {{ CITY }}_two
            {{ CITY }}_one & {{ CITY }}_two => {{ CITY }}_three & {{ CITY }}_four
        {% if CLEANUP %}
            {{ CITY }}_three & {{ CITY }}_four => cleanup
    {% endif %}
{% endfor %}
        """

[runtime]
    [[on_{{ HOST }} ]]
        [[[remote]]]
            host = {{ HOST }}
            # (remote cylc directory is set in site/user config for this host)
        [[[directives]]]
            wall_clock_limit = "00:{{ LIMIT_MINS|int() + 2 }}:00,00:{{ LIMIT_MINS }}:00"

{% for CITY in CITIES %}
    [[ {{ CITY }} ]]
        inherit = on_{{ HOST }}
    {% for JOB in CITYJOBS %}
    [[ {{ CITY }}_{{ JOB }} ]]
        inherit = {{ CITY }}
    {% endfor %}
{% endfor %}

../../_images/jinja2-workflow-graph.png — Jinja2 cities example workflow graph, with the New York City task family expanded.

Accessing Environment Variables

Cylc automatically imports the environment to the template’s global namespace (see Custom Jinja2 Filters, Tests and Globals) in a dictionary called environ:

#!Jinja2
#...
[runtime]
    [[root]]
        [[[environment]]]
            WORKFLOW_OWNER_HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}}

In addition, the following variables are exported to this environment (hence are available in the environ dict) to provide workflow context:

CYLC_VERBOSE                    # Verbose mode, true or false
CYLC_DEBUG                      # Debug mode (even more verbose), true or false

CYLC_WORKFLOW_ID                # Workflow ID
CYLC_WORKFLOW_NAME              # Workflow name
                                # (the ID with the run name removed)

CYLC_WORKFLOW_LOG_DIR           # Workflow log directory.
CYLC_WORKFLOW_RUN_DIR           # Location of the run directory in
                                # workflow host, e.g. ~/cylc-run/foo
CYLC_WORKFLOW_SHARE_DIR         # Workflow (or task post parsing!)
                                # shared directory.
CYLC_WORKFLOW_WORK_DIR          # Workflow work directory.

Warning

The environment is read on the workflow host when the configuration is parsed. It is not read at run time by jobs on the job platform.

The following Jinja2 variables are also available (i.e. standalone, not in the environ dict):

CYLC_VERSION: Version of Cylc used.
CYLC_TEMPLATE_VARS: All variables set by the -s, --set-file or --set-list options, or by a plugin.

Custom Jinja2 Filters, Tests and Globals

Jinja2 has three namespaces that separate “globals”, “filters” and “tests”. Globals are template-wide variables and functions. Cylc extends this namespace with the environ dictionary above, and raise and assert functions for raising exceptions to abort Cylc config parsing.

Filters can be used to modify variable values and are applied using pipe notation. For example, the built-in trim filter strips leading and trailing white space from a string:

{% set MyString = "   dog   " %}
{{ MyString | trim() }}  # "dog"

Variable values can be tested using the is keyword followed by the name of the test, e.g. {% if VARIABLE is defined %}. See Jinja2 documentation for available built-in globals, filters and tests.

Cylc also supports custom Jinja2 globals, filters and tests. A custom global, filter or test is a single Python function in a source file with the same name as the function (plus .py extension). These must be located in a subdirectory of the run directory called Jinja2Filters, Jinja2Globals or Jinja2Tests respectively.

In the argument list of a filter or test function, the first argument is the variable value to be filtered or tested, and subsequent arguments can be whatever is needed. Currently three custom filters are supplied:

`cylc.flow.jinja.filters.pad.pad`	Pads a string to some length with a fill character
`cylc.flow.jinja.filters.strftime.strftime`	Format an ISO8601 datetime string using an strftime string.
`cylc.flow.jinja.filters.duration_as.duration_as`	Format an ISO8601 duration string as the specified units.

cylc.flow.jinja.filters.pad.pad(value, length, fillchar=' ')

Pads a string to some length with a fill character

Useful for generating task names and related values in ensemble workflows.

Args:

value (str):: The string to pad.
length (int/str):: The length for the returned string.
fillchar (str - optional):: The character to fill in surplus space (space by default).

Returns:

str: value padded to the left with fillchar to length length.

Python Examples:

>>> pad('13', 3, '0')
'013'
>>> pad('foo', 6)
'   foo'
>>> pad('foo', 2)
'foo'

Jinja2 Examples:

{% for i in range(0,100) %}  # 0, 1, ..., 99
    {% set j = i | pad(2,'0') %}
    [[A_{{j}}]]         # [[A_00]], [[A_01]], ..., [[A_99]]
{% endfor %}

cylc.flow.jinja.filters.strftime.strftime(iso8601_datetime, strftime_str, strptime_str=None)

Format an ISO8601 datetime string using an strftime string.

{{ '10661004T08+01' | strftime('%H') }}  # 00

It is also possible to parse non-standard date-time strings by passing a strptime string as the second argument.

Args:

iso8601_datetime (str):: Any valid ISO8601 datetime as a string.
strftime_str (str):: A valid strftime string to format the output datetime.
strptime_str (str - optional):: A valid strptime string defining the format of the provided iso8601_datetime.

Return:

The result of applying the strftime to the iso8601_datetime as parsed by the strptime string if provided.

Raises:

ISO8601SyntaxError: In the event of an invalid datetime string. StrftimeSyntaxError: In the event of an invalid strftime string.

Python Examples:

>>> # Basic usage.
>>> strftime('2000-01-01T00Z', '%H')
'00'
>>> strftime('2000', '%H')
'00'
>>> strftime('2000', '%Y/%m/%d %H:%M:%S')
'2000/01/01 00:00:00'
>>> strftime('10661014T08+01', '%z')  # Timezone offset.
'+0100'
>>> strftime('10661014T08+01', '%j')  # Day of the year
'287'

>>> # Strptime.
>>> strftime('12,30,2000', '%m', '%m,%d,%Y')
'12'
>>> strftime('1066/10/14 08:00:00', '%Y%m%dT%H', '%Y/%m/%d %H:%M:%S')
'10661014T08'

>>> # Exceptions.
>>> strftime('invalid', '%H')  
Traceback (most recent call last):
<class 'metomi.isodatetime.exceptions.ISO8601SyntaxError'>
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601         date representation: invalid
>>> strftime('2000', '%invalid')  
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i
>>> strftime('2000', '%Y', '%invalid')
... 
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i

Jinja2 Examples:

{% set START_CYCLE = '10661004T08+01' %}

{{START_CYCLE | strftime('%Y')}}  # 1066
{{START_CYCLE | strftime('%m')}}  # 10
{{START_CYCLE | strftime('%d')}}  # 14
{{START_CYCLE | strftime('%H:%M:%S %z')}}  # 08:00:00 +01
{{'12,30,2000' | strftime('%m', '%m,%d,%Y')}}  # 12

cylc.flow.jinja.filters.duration_as.duration_as(iso8601_duration, units)

Format an ISO8601 duration string as the specified units.

Units for the conversion can be specified in a case-insensitive short or long form:

Seconds - “s” or “seconds”
Minutes - “m” or “minutes”
Hours - “h” or “hours”
Days - “d” or “days”
Weeks - “w” or “weeks”

While the filtered value is a floating-point number, it is often required to supply an integer to workflow entities (e.g. environment variables) that require it. This is accomplished by chaining filters:

{{CYCLE_INTERVAL | duration_as('h') | int}} - 24
{{CYCLE_SUBINTERVAL | duration_as('h') | int}} - 0
{{CYCLE_INTERVAL | duration_as('s') | int}} - 86400
{{CYCLE_SUBINTERVAL | duration_as('s') | int}} - 1800

Args:

iso8601_duration (str): Any valid ISO8601 duration as a string. units (str): Destination unit for the duration conversion

Return:

The total number of the specified unit contained in the specified duration as a floating-point number.

Raises:

ISO8601SyntaxError: In the event of an invalid datetime string.

Python Examples:

>>> # Basic usage.
>>> duration_as('PT1M', 's')
60.0
>>> duration_as('PT1H', 'seconds')
3600.0

>>> # Exceptions.
>>> duration_as('invalid value', 's')  
Traceback (most recent call last):
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601        duration representation: invalid value
>>> duration_as('invalid unit', '#')  
Traceback (most recent call last):
ValueError: No matching units found for #

Jinja2 Examples:

{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('h') }}  # 24.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('hours') }}  # 0.5
{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('s') }}  # 86400.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('seconds') }}  # 1800.0

Associative Arrays In Jinja2

Associative arrays (or dictionaries) are very useful. For example:

#!Jinja2
{% set obs_types = ['airs', 'iasi'] %}
{% set resource = { 'airs':'ncpus=9', 'iasi':'ncpus=20' } %}

[scheduling]
    [[graph]]
        R1 = OBS
[runtime]
    [[OBS]]
        platform = platform_using_pbs
    {% for i in obs_types %}
    [[ {{i}} ]]
        inherit = OBS
        [[[directives]]]
             -I = {{ resource[i] }}
     {% endfor %}

Here’s the result:

$ cylc config -i [runtime][airs]directives <workflow-id>
-I = ncpus=9

Default Values and Template Variables

You can provide template variables to Cylc in 4 ways:

Using the --set-file (-S) option.
Using the --set (-s) option.
Using the --set-list (-z) option.
Using a plugin, such as Cylc Rose.

Note

If the same variable is set by more than one method, the last source in the above list is used.

The `-s`, `-z` and `--set-file` Options

$ # set the Jinja2 variable "answer" to 42
$ cylc play <workflow-id> -s answer=42

A Python string-list is a valid value, but a lot to type, so --set-list (-z) is provided as a convenience:

# The set syntax
$ cylc play <workflow-id> -s "answers=['mice', 'dolphins']"
# ... can  be shortened to:
$ cylc play <workflow-id> -z answers=mice,dolphins

If you need to define a lot of variables, you can so in a file using the --set-file option:

$ # create a set file
$ cat > my-set-file <<__SET_FILE__
question='the meaning of life, the universe and everything'
answer=42
host='deep-thought'
__SET_FILE__

$ # run using the options in the set file
$ cylc play <workflow-id> --set-file my-set-file

Values must be Python literals e.g:

"string"   # string
123        # integer
12.34      # float
True       # boolean
None       # None type
[1, 2, 3]  # list
(1, 2, 3)  # tuple
{1, 2, 3}  # set
{"a": 1, "b": 2, "c": 3}  # dictionary

Note

On the command line you may need to wrap strings with an extra pair of quotes as the shell you are using (e.g. Bash) will strip the outer pair of quotes.

$ # wrap the key=value pair in single quotes stop the shell from
$ # stripping the inner quotes around the string:
$ cylc play <workflow-id> -s 'my_string="a b c"'

Here’s an example:

#!Jinja2
[meta]
    title = "Jinja2 example: use of defaults and external input"

    description = """
        The template variable N_MEMBERS can be set on the command line with
        --set or --set-file=FILE; but if not a default values is supplied.
    """

[scheduling]
    initial cycle point = 20100808T00
    final cycle point   = 20100816T00
    [[graph]]
        T00 = """
            foo => ENS
            ENS:succeed-all => bar
        """

[runtime]
    [[foo, bar]]
    [[ENS]]
{% for I in range( 0, N_MEMBERS | default( 3 )) %}
    [[ mem_{{ I }} ]]
        inherit = ENS
{% endfor %}

Here’s the result:

$ cylc list <workflow-id>
Jinja2 Template Error
'FIRST_TASK' is undefined
cylc-list <workflow-id>  failed:  1

$ # Note: quoting "bob" so that it is evaluated as a string
$ cylc list --set 'FIRST_TASK="bob"' <workflow-id>
bob
baz
mem_2
mem_1
mem_0

$ cylc list --set 'FIRST_TASK="bob"' --set 'LAST_TASK="alice"' <workflow-id>
bob
alice
mem_2
mem_1
mem_0

$ # Note: no quotes required for N_MEMBERS since it is an integer
$ cylc list --set 'FIRST_TASK="bob"' --set N_MEMBERS=10 <workflow-id>
mem_9
mem_8
mem_7
mem_6
mem_5
mem_4
mem_3
mem_2
mem_1
mem_0
baz
bob

Note also that cylc view --set FIRST_TASK=bob --jinja2 <workflow-id> will show the workflow with the Jinja2 variables as set.

Note

Workflows started with template variables set on the command line will restart with the same settings. You can set them again on the cylc play command line if they need to be overridden.

Using a plugin

Template plugins such as Cylc Rose should provide a set of template variables which can be provided to Cylc. For example, using Cylc Rose you add a rose-suite.conf file containing a [template variables] section which the plugin makes available to Cylc:

rose-suite.conf

[template variables]
ICP=1068

flow.cylc

#!jinja2
[scheduler]
   allow implicit tasks = True
[scheduling]
   initial cycle point = {{ICP}}
   [[dependencies]]
      P1Y = Task1

$ cylc config . -i "[scheduling]initial cycle point"
1068

Jinja2 Variable Scope

Jinja2 variable scoping rules may be surprising. For instance, variables set inside a for loop can’t be accessed outside of the block, so the following will not print # FOO is True:

{% set FOO = False %}
{% for item in items %}
    {% if item.check_something() %}
        {% set FOO = True %}
    {% endif %}
{% endfor %}
# FOO is {{FOO}}

Jinja2 documentation suggests using alternative constructs like the loop else block or the special loop variable. More complex use cases can be handled using namespace objects that allow propagating of changes across scopes:

{% set ns = namespace(foo=false) %}
{% for item in items %}
    {% if item.check_something() %}
        {% set ns.foo = true %}
    {% endif %}
{% endfor %}
# FOO is {{ns.foo}}

For detail, see Jinja2 Template Designer Documentation - Assignments

Raising Exceptions

Cylc provides two functions for raising exceptions in Jinja2 code. These exceptions are raised when the flow.cylc file is loaded and will prevent a workflow from running.

Note

These functions must be contained within {{ Jinja2 print statements, not {% code blocks.

Raise

The raise function will result in an error containing the provided text.

{% if not VARIABLE is defined %}
    {{ raise('VARIABLE must be defined for this workflow.') }}
{% endif %}

Assert

The assert function will raise an exception containing the text provided in the second argument providing that the first argument evaluates as False. The following example is equivalent to the “raise” example above.

{{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }}

Importing Python modules

Jinja2 allows to gather variable and macro definitions in a separate template that can be imported into (and thus shared among) other templates.

{% import "flow-utils.cylc" as utils %}
{% from "flow-utils.cylc" import VARIABLE as ALIAS %}
{{ utils.VARIABLE is equalto(ALIAS)) }}

Cylc extends this functionality to allow import of arbitrary Python modules.

{% from "itertools" import product %}
[runtime]
{% for group, member in product(['a', 'b'], [0, 1, 2]) %}
    [[{{group}}_{{member}}]]
{% endfor %}

For better clarity and disambiguation Python modules can be prefixed with __python__:

{% from "__python__.itertools" import product %}

Logging

It is possible to output messages to the Cylc log from within Jinja2, these messages will appear on the console when validating or starting a workflow. This can be useful for development or debugging.

Example flow.cylc:

#!Jinja2
{% from "cylc.flow" import LOG %}
{% do LOG.debug("Hello World!") %}

Example output:

$ cylc validate . --debug
DEBUG - Loading site/user config files
DEBUG - Reading file <file>
DEBUG - Processing with Jinja2
DEBUG - Hello World!
...
Valid for cylc-<version>

Log messages will appear whenever the workflow configuration is loaded so it is advisable to use the DEBUG logging level which is suppressed unless the --debug option is provided.

Debugging

It is possible to run Python debuggers from within Jinja2 via the import mechanism.

For example to use a PDB breakpoint you could do the following:

#!Jinja2

{% set ANSWER = 42 %}

{% from "pdb" import set_trace %}
{% do set_trace() %}

The debugger will open within the Jinja2 code, local variables can be accessed via the _Context__self variable e.g:

$ cylc validate <id>
(Pdb) _Context__self['ANSWER']
42

Jinja2

Accessing Environment Variables

Custom Jinja2 Filters, Tests and Globals

Associative Arrays In Jinja2

Default Values and Template Variables

The -s, -z and --set-file Options

Using a plugin

Jinja2 Variable Scope

Raising Exceptions

Raise

Assert

Importing Python modules

Logging

Debugging

The `-s`, `-z` and `--set-file` Options