Jinja2

Access to Workflow Files

Your Jinja2 code can see the workflow directory by using Python code that simply reads from the current working directory.

This will be the source directory if parsing a source workflow, or the run directory if parsing an installed workflow.

Workflow Context variables

Jinja2 CYLC variables available when parsing any workflow (source or installed):

CYLC_VERSION	Version of Cylc parsing the configuration
CYLC_WORKFLOW_NAME	Workflow name (source, or run ID minus run name)
CYLC_WORKFLOW_SRC_DIR	Path of the source directory.
CYLC_TEMPLATE_VARS	Variables set by ‘–set’ CLI options or plugins

Additional Jinja2 CYLC variables available when parsing an installed workflow:

CYLC_WORKFLOW_ID	Workflow ID
CYLC_WORKFLOW_RUN_DIR	Workflow run directory

Additional Jinja2 CYLC variables available when the scheduler is parsing an installed workflow at run time:

CYLC_WORKFLOW_LOG_DIR	Workflow scheduler’s log directory
CYLC_WORKFLOW_SHARE_DIR	Workflow share sub-directory
CYLC_WORKFLOW_WORK_DIR	Workflow work sub-directory

Note

Set default values for CYLC variables that are only defined for installed or running workflows, to allow successful parsing in other contexts as well. For example:

{{ CYLC_WORKFLOW_RUN_DIR | default("not-defined") }}

Jinja2 Filters, Tests and Globals

Jinja2 provides “globals”, “filters” and “tests” which can be helpful in workflow writing.

Globals

Regular Python functions.

Jinja2 builtins:: Jinja2 Built-in Globals
Cylc builtins:: Cylc Built-in Globals
Custom directory:: Jinja2Globals

Filters

Special functions which “chain” using the pipe character (|).

Jinja2 builtins:: Jinja2 Built-in Filters
Cylc builtins:: Cylc Built-in Filters
Custom directory:: Jinja2Filters

Tests

Special functions which work with the is operator.

Jinja2 builtins:: Jinja2 Built-in Tests
Custom directory:: Jinja2Tests

For example, this flow.cylc file uses the pad filter to help write out task definitions:

[runtime]
{% for x in range(3) %}
    [[task_{{ x | pad(3) }}]]
        script = sleep {{ x }}
{% endfor %}

The Jinja2 would be expanded like so:

[runtime]
    [[x_001]]
        script = sleep 1
    [[x_002]]
        script = sleep 2
    [[x_003]]
        script = sleep 3

In addition to the built-ins that Jinja2 and Cylc provide, you can also define your own custom filters (see Custom Jinja2 Extensions).

Cylc Built-in Globals

`environ`	Access environment variables.
`raise()`	Raise an error.
`assert()`	Raise an error if a condition is not met.

environ

Provides access to environment variables.

Note, these are the “parse-time” environment variables - i.e, the environment that is set when the workflow’s flow.cylc file is processed. This happens when a workflow is validated or started, not when jobs are submitted. Jinja2 does not have access to dynamic environment variables available to jobs.

Jinja2 Examples:

[runtime]
    [[root]]
        [[[environment]]]
            HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}}

raise(error_message)

The raise function will result in an error containing the provided text.

Calling this will cause cylc validate to fail with the provided error message and will prevent the workflow from being started. It’s useful for validating input template variables.

Jinja2 Examples:

{% if VARIABLE is not defined %}
    {{ raise('VARIABLE must be defined for this workflow.') }}
{% endif %}

assert(condition, error_message)

The assert function will raise an exception containing the text provided in the second argument providing that the first argument evaluates as False. The following example is equivalent to the “raise” example above.

Assertion errors will cylc validate to fail with the provided error message and will prevent the workflow from being started. It’s useful for validating input template variables.

Jinja2 Examples:

{{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }}

Cylc Built-in Filters

`cylc.flow.jinja.filters.pad.pad`	Pads a string to some length with a fill character
`cylc.flow.jinja.filters.strftime.strftime`	Format an ISO8601 datetime string using an strftime string.
`cylc.flow.jinja.filters.duration_as.duration_as`	Format an ISO8601 duration string as the specified units.

cylc.flow.jinja.filters.pad.pad(value, length, fillchar=' ')[source]

Pads a string to some length with a fill character

Useful for generating task names and related values in ensemble workflows.

Parameters:

value (str) – The string to pad.
length (int | str) – The length for the returned string.
fillchar (str) – The character to fill in surplus space (space by default).

Returns:

value padded to the left with fillchar to length length.

Return type:

str

Python Examples:

>>> pad('13', 3, '0')
'013'
>>> pad('foo', 6)
'   foo'
>>> pad('foo', 2)
'foo'

Jinja2 Examples:

{% for i in range(0,100) %}  # 0, 1, ..., 99
    {% set j = i | pad(2,'0') %}
    [[A_{{j}}]]         # [[A_00]], [[A_01]], ..., [[A_99]]
{% endfor %}

cylc.flow.jinja.filters.strftime.strftime(iso8601_datetime, strftime_str, strptime_str=None)[source]

Format an ISO8601 datetime string using an strftime string.

{{ '10661004T08+01' | strftime('%H') }}  # 00

It is also possible to parse non-standard date-time strings by passing a strptime string as the second argument.

Parameters:

iso8601_datetime (str) – Any valid ISO8601 datetime as a string.
strftime_str (str) – A valid strftime string to format the output datetime.
strptime_str (str | None) – A valid strptime string defining the format of the provided iso8601_datetime.

Returns:

The result of applying the strftime to the iso8601_datetime as parsed by the strptime string if provided.

Raises:

ISO8601SyntaxError – In the event of an invalid datetime string.
StrftimeSyntaxError – In the event of an invalid strftime string.

Python Examples:

>>> # Basic usage.
>>> strftime('2000-01-01T00Z', '%H')
'00'
>>> strftime('2000', '%H')
'00'
>>> strftime('2000', '%Y/%m/%d %H:%M:%S')
'2000/01/01 00:00:00'
>>> strftime('10661014T08+01', '%z')  # Timezone offset.
'+0100'
>>> strftime('10661014T08+01', '%j')  # Day of the year
'287'

>>> # Strptime.
>>> strftime('12,30,2000', '%m', '%m,%d,%Y')
'12'
>>> strftime('1066/10/14 08:00:00', '%Y%m%dT%H', '%Y/%m/%d %H:%M:%S')
'10661014T08'

>>> # Exceptions.
>>> strftime('invalid', '%H')
Traceback (most recent call last):
<class 'metomi.isodatetime.exceptions.ISO8601SyntaxError'>
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601         date representation: invalid
>>> strftime('2000', '%invalid')
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i
>>> strftime('2000', '%Y', '%invalid')
...
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i

Jinja2 Examples:

{% set START_CYCLE = '10661004T08+01' %}

{{START_CYCLE | strftime('%Y')}}  # 1066
{{START_CYCLE | strftime('%m')}}  # 10
{{START_CYCLE | strftime('%d')}}  # 14
{{START_CYCLE | strftime('%H:%M:%S %z')}}  # 08:00:00 +01
{{'12,30,2000' | strftime('%m', '%m,%d,%Y')}}  # 12

cylc.flow.jinja.filters.duration_as.duration_as(iso8601_duration, units)[source]

Format an ISO8601 duration string as the specified units.

Units for the conversion can be specified in a case-insensitive short or long form:

Seconds - “s” or “seconds”
Minutes - “m” or “minutes”
Hours - “h” or “hours”
Days - “d” or “days”
Weeks - “w” or “weeks”

While the filtered value is a floating-point number, it is often required to supply an integer to workflow entities (e.g. environment variables) that require it. This is accomplished by chaining filters:

{{CYCLE_INTERVAL | duration_as('h') | int}} - 24
{{CYCLE_SUBINTERVAL | duration_as('h') | int}} - 0
{{CYCLE_INTERVAL | duration_as('s') | int}} - 86400
{{CYCLE_SUBINTERVAL | duration_as('s') | int}} - 1800

Parameters:

iso8601_duration (str) – Any valid ISO8601 duration as a string.
units (str) – Destination unit for the duration conversion

Returns:

The total number of the specified unit contained in the specified duration as a floating-point number.

Raises:

ISO8601SyntaxError – In the event of an invalid datetime string.

Return type:

float

Python Examples:

>>> # Basic usage.
>>> duration_as('PT1M', 's')
60.0
>>> duration_as('PT1H', 'seconds')
3600.0

>>> # Exceptions.
>>> duration_as('invalid value', 's')
Traceback (most recent call last):
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601        duration representation: invalid value
>>> duration_as('invalid unit', '#')
Traceback (most recent call last):
ValueError: No matching units found for #

Jinja2 Examples:

{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('h') }}  # 24.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('hours') }}  # 0.5
{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('s') }}  # 86400.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('seconds') }}  # 1800.0

Custom Jinja2 Extensions

You can define your own custom Jinja2 globals, filters and tests within workflows.

These extensions must be Python modules containing a function with the same name as the module (e.g, a module called foo.py should contain a function called foo).

Jinja2 globals go in the workflow source directory in a subdirectory called Jinja2Globals, filters in Jinja2Filters and tests in Jinja2Tests.

This example defines one of each and demonstrates how to use them:

flow.cylc

#!Jinja2

# "globals" are regular Python functions
{{ square(5) }}

# filters are special functions which chain using the pipe character
{{ ('run', 1) | display_name }}

# tests are special functions which work with the "is" operator
{{ 42 is even }}

Jinja2Filters/display_name.py

def display_name(argument):
    name, number = argument
    return f'{name}_x{number:03d}'

Jinja2Globals/square.py

def square(number):
    return number ** 2

Jinja2Tests/even.py

def even(number):
    return number % 2 == 0

Associative Arrays In Jinja2

Associative arrays (or dictionaries) are very useful. For example:

#!Jinja2
{% set obs_types = ['airs', 'iasi'] %}
{% set resource = { 'airs':'ncpus=9', 'iasi':'ncpus=20' } %}

[scheduling]
    [[graph]]
        R1 = OBS
[runtime]
    [[OBS]]
        platform = platform_using_pbs
    {% for i in obs_types %}
    [[ {{i}} ]]
        inherit = OBS
        [[[directives]]]
             -I = {{ resource[i] }}
     {% endfor %}

Here’s the result:

$ cylc config -i [runtime][airs]directives <workflow-id>
-I = ncpus=9

Default Values and Template Variables

You can provide template variables to Cylc in 4 ways:

Using the --set-file (-S) option.
Using the --set (-s) option.
Using the --set-list (-z) option.
Using a plugin, such as Cylc Rose.

Note

If the same variable is set by more than one method, the last source in the above list is used.

The `-s`, `-z` and `--set-file` Options

$ # set the Jinja2 variable "answer" to 42
$ cylc play <workflow-id> -s answer=42

A Python string-list is a valid value, but a lot to type, so --set-list (-z) is provided as a convenience:

# The set syntax
$ cylc play <workflow-id> -s "answers=['mice', 'dolphins']"
# ... can  be shortened to:
$ cylc play <workflow-id> -z answers=mice,dolphins

If you need to define a lot of variables, you can so in a file using the --set-file option:

$ # create a set file
$ cat > my-set-file <<__SET_FILE__
question='the meaning of life, the universe and everything'
answer=42
host='deep-thought'
__SET_FILE__

$ # run using the options in the set file
$ cylc play <workflow-id> --set-file my-set-file

Values must be Python literals e.g:

"string"   # string
123        # integer
12.34      # float
True       # boolean
None       # None type
[1, 2, 3]  # list
(1, 2, 3)  # tuple
{1, 2, 3}  # set
{"a": 1, "b": 2, "c": 3}  # dictionary

Note

On the command line you may need to wrap strings with an extra pair of quotes as the shell you are using (e.g. Bash) will strip the outer pair of quotes.

$ # wrap the key=value pair in single quotes stop the shell from
$ # stripping the inner quotes around the string:
$ cylc play <workflow-id> -s 'my_string="a b c"'

Here’s an example:

#!Jinja2
[meta]
    title = "Jinja2 example: use of defaults and external input"

    description = """
        The template variable N_MEMBERS can be set on the command line with
        --set or --set-file=FILE; but if not a default values is supplied.
    """

[scheduling]
    initial cycle point = 20100808T00
    final cycle point   = 20100816T00
    [[graph]]
        T00 = """
            foo => ENS
            ENS:succeed-all => bar
        """

[runtime]
    [[foo, bar]]
    [[ENS]]
{% for I in range( 0, N_MEMBERS | default( 3 )) %}
    [[ mem_{{ I }} ]]
        inherit = ENS
{% endfor %}

Here’s the result:

$ cylc list <workflow-id>
Jinja2 Template Error
'FIRST_TASK' is undefined
cylc-list <workflow-id>  failed:  1

$ # Note: quoting "bob" so that it is evaluated as a string
$ cylc list --set 'FIRST_TASK="bob"' <workflow-id>
bob
baz
mem_2
mem_1
mem_0

$ cylc list --set 'FIRST_TASK="bob"' --set 'LAST_TASK="alice"' <workflow-id>
bob
alice
mem_2
mem_1
mem_0

$ # Note: no quotes required for N_MEMBERS since it is an integer
$ cylc list --set 'FIRST_TASK="bob"' --set N_MEMBERS=10 <workflow-id>
mem_9
mem_8
mem_7
mem_6
mem_5
mem_4
mem_3
mem_2
mem_1
mem_0
baz
bob

Note also that cylc view --set FIRST_TASK=bob --jinja2 <workflow-id> will show the workflow with the Jinja2 variables as set.

Note

Workflows started with template variables set on the command line will restart with the same settings. You can set them again on the cylc play command line if they need to be overridden.

Using a plugin

Template plugins such as Cylc Rose should provide a set of template variables which can be provided to Cylc. For example, using Cylc Rose you add a rose-suite.conf file containing a [template variables] section which the plugin makes available to Cylc:

rose-suite.conf

[template variables]
ICP=1068

flow.cylc

#!jinja2
[scheduler]
   allow implicit tasks = True
[scheduling]
   initial cycle point = {{ICP}}
   [[dependencies]]
      P1Y = Task1

$ cylc config . -i "[scheduling]initial cycle point"
1068

Jinja2 Variable Scope

Jinja2 variable scoping rules may be surprising. For instance, variables set inside a for loop can’t be accessed outside of the block, so the following will not print # FOO is True:

{% set FOO = False %}
{% for item in items %}
    {% if item.check_something() %}
        {% set FOO = True %}
    {% endif %}
{% endfor %}
# FOO is {{FOO}}

Jinja2 documentation suggests using alternative constructs like the loop else block or the special loop variable. More complex use cases can be handled using namespace objects that allow propagating of changes across scopes:

{% set ns = namespace(foo=false) %}
{% for item in items %}
    {% if item.check_something() %}
        {% set ns.foo = true %}
    {% endif %}
{% endfor %}
# FOO is {{ns.foo}}

For detail, see Jinja2 Template Designer Documentation - Assignments

Importing Python modules

Jinja2 allows to gather variable and macro definitions in a separate template that can be imported into (and thus shared among) other templates. For example, if we have a file in the source directory called utils.cylc, we can use it in flow.cylc in a couple of ways:

{% import "utils.cylc" as utils %}
{% from "utils.cylc" import VARIABLE as ALIAS %}
{{ utils.VARIABLE is equalto(ALIAS)) }}

Cylc extends this functionality to allow import of arbitrary Python modules.

{% from "itertools" import product %}
[runtime]
{% for group, member in product(['a', 'b'], [0, 1, 2]) %}
    [[{{group}}_{{member}}]]
{% endfor %}

For better clarity and disambiguation Python modules can be prefixed with __python__:

{% from "__python__.itertools" import product %}

Important

As before, you can only import modules that are available in the environment used to run Cylc, or the lib/python directory.

Macros

Jinja2 macros can be used to automatically construct parts of your workflow based on input parameters (macro arguments).

Here’s an example macro that adds a task to print a given word (default “hello”), after a given task in your graph:

{% macro say_it(
    after_task,
    word = "hello"
) %}
[scheduling]
    [[graph]]
        R1 = {{ after_task }} => say_{{ word }}
[runtime]
    [[say_{{ word }}]]
        script = "echo {{ word }}"
{% endmacro %}

If we have written this to a file in the source directory called macros.cylc, it can be used in flow.cylc like so:

#!Jinja2
{% from "macros.cylc" import say_it %}

{{ say_it(after_task="b") }}
{{ say_it(after_task="c", word="goodbye") }}
[scheduling]
    [[graph]]
        R1 = a => b => c
[runtime]
    [[a, b, c]]

Here’s the result after template processing and config parsing:

[scheduling]
    [[graph]]
        R1 = """
            b => say_hello
            c => say_goodbye
            a => b => c
        """
[runtime]
    [[say_hello]]
        script = echo hello
    [[say_goodbye]]
        script = echo goodbye
    [[a, b, c]]

Logging

It is possible to output messages to the Cylc log from within Jinja2, these messages will appear on the console when validating or starting a workflow. This can be useful for development or debugging.

Example flow.cylc:

#!Jinja2
{% from "cylc.flow" import LOG %}
{% do LOG.debug("Hello World!") %}

Example output:

$ cylc validate . --debug
DEBUG - Loading site/user config files
DEBUG - Reading file <file>
DEBUG - Processing with Jinja2
DEBUG - Hello World!
...
Valid for cylc-<version>

Log messages will appear whenever the workflow configuration is loaded so it is advisable to use the DEBUG logging level which is suppressed unless the --debug option is provided.

Debugging

It is possible to run Python debuggers from within Jinja2 via the import mechanism.

For example to use a PDB breakpoint you could do the following:

#!Jinja2

{% set ANSWER = 42 %}

{% from "pdb" import set_trace %}
{% do set_trace() %}

The debugger will open within the Jinja2 code, local variables can be accessed via the _Context__self variable e.g:

$ cylc validate <id>
(Pdb) _Context__self['ANSWER']
42

Jinja2 Syntax	Description
`{# comment #}`	Comment
`{% if true %}`	Expression
`{{ var }}`	Print statement

Jinja2

Access to Workflow Files

Workflow Context variables

Jinja2 Filters, Tests and Globals

Cylc Built-in Globals

Cylc Built-in Filters

Custom Jinja2 Extensions

Associative Arrays In Jinja2

Default Values and Template Variables

The -s, -z and --set-file Options

Using a plugin

Jinja2 Variable Scope

Importing Python modules

Macros

Logging

Debugging

The `-s`, `-z` and `--set-file` Options