Jinja2

Access to Workflow Files

Your Jinja2 code can see the workflow directory by using Python code that simply reads from the current working directory.

This will be the source directory if parsing a source workflow, or the run directory if parsing an installed workflow.

Workflow Context variables

Jinja2 CYLC variables available when parsing any workflow (source or installed):

CYLC_VERSION	Version of Cylc parsing the configuration
CYLC_WORKFLOW_NAME	Workflow name (source, or run ID minus run name)
CYLC_TEMPLATE_VARS	Variables set by ‘–set’ CLI options or plugins

Additional Jinja2 CYLC variables available when parsing an installed workflow:

CYLC_WORKFLOW_ID	Workflow ID
CYLC_WORKFLOW_RUN_DIR	Workflow run directory

Additional Jinja2 CYLC variables available when the scheduler is parsing an installed workflow at run time:

CYLC_WORKFLOW_LOG_DIR	Workflow log sub-directory
CYLC_WORKFLOW_SHARE_DIR	Workflow share sub-directory
CYLC_WORKFLOW_WORK_DIR	Workflow work sub-directory

Note

Set default values for CYLC variables that are only defined for installed or running workflows, to allow successful parsing in other contexts as well: {{CYLC_WORKFLOW_RUN_DIR | default("not-defined")}}.

Environment Variables

Cylc automatically imports the parse-time environment to the template processor’s global namespace (see Custom Jinja2 Filters, Tests and Globals), in a dictionary called environ:

#!Jinja2
#...
[runtime]
    [[root]]
        [[[environment]]]
            HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}}

Warning

The environment is read during configuration parsing. It is not the run time job environment.

Custom Jinja2 Filters, Tests and Globals

Jinja2 has three namespaces that separate “globals”, “filters” and “tests”. Globals are template-wide variables and functions. Cylc extends this namespace with the environ dictionary above, and raise and assert functions for raising exceptions to abort Cylc config parsing.

Filters can be used to modify variable values and are applied using pipe notation. For example, the built-in trim filter strips leading and trailing white space from a string:

{% set MyString = "   dog   " %}
{{ MyString | trim() }}  # "dog"

Variable values can be tested using the is keyword followed by the name of the test, e.g. {% if VARIABLE is defined %}. See Jinja2 documentation for available built-in globals, filters and tests.

Cylc also supports custom Jinja2 globals, filters and tests. A custom global, filter or test is a single Python function in a source file with the same name as the function (plus .py extension). These must be located in a subdirectory of the run directory called Jinja2Filters, Jinja2Globals or Jinja2Tests respectively.

In the argument list of a filter or test function, the first argument is the variable value to be filtered or tested, and subsequent arguments can be whatever is needed. Currently three custom filters are supplied:

`cylc.flow.jinja.filters.pad.pad`	Pads a string to some length with a fill character
`cylc.flow.jinja.filters.strftime.strftime`	Format an ISO8601 datetime string using an strftime string.
`cylc.flow.jinja.filters.duration_as.duration_as`	Format an ISO8601 duration string as the specified units.

cylc.flow.jinja.filters.pad.pad(value, length, fillchar=' ')[source]

Pads a string to some length with a fill character

Useful for generating task names and related values in ensemble workflows.

Parameters:

value (str) – The string to pad.
length (int | str) – The length for the returned string.
fillchar (str) – The character to fill in surplus space (space by default).

Returns:

value padded to the left with fillchar to length length.

Return type:

str

Python Examples:

>>> pad('13', 3, '0')
'013'
>>> pad('foo', 6)
'   foo'
>>> pad('foo', 2)
'foo'

Jinja2 Examples:

{% for i in range(0,100) %}  # 0, 1, ..., 99
    {% set j = i | pad(2,'0') %}
    [[A_{{j}}]]         # [[A_00]], [[A_01]], ..., [[A_99]]
{% endfor %}

cylc.flow.jinja.filters.strftime.strftime(iso8601_datetime, strftime_str, strptime_str=None)[source]

Format an ISO8601 datetime string using an strftime string.

{{ '10661004T08+01' | strftime('%H') }}  # 00

It is also possible to parse non-standard date-time strings by passing a strptime string as the second argument.

Parameters:

iso8601_datetime (str) – Any valid ISO8601 datetime as a string.
strftime_str (str) – A valid strftime string to format the output datetime.
strptime_str (str | None) – A valid strptime string defining the format of the provided iso8601_datetime.

Returns:

The result of applying the strftime to the iso8601_datetime as parsed by the strptime string if provided.

Raises:

ISO8601SyntaxError – In the event of an invalid datetime string.
StrftimeSyntaxError – In the event of an invalid strftime string.

Python Examples:

>>> # Basic usage.
>>> strftime('2000-01-01T00Z', '%H')
'00'
>>> strftime('2000', '%H')
'00'
>>> strftime('2000', '%Y/%m/%d %H:%M:%S')
'2000/01/01 00:00:00'
>>> strftime('10661014T08+01', '%z')  # Timezone offset.
'+0100'
>>> strftime('10661014T08+01', '%j')  # Day of the year
'287'

>>> # Strptime.
>>> strftime('12,30,2000', '%m', '%m,%d,%Y')
'12'
>>> strftime('1066/10/14 08:00:00', '%Y%m%dT%H', '%Y/%m/%d %H:%M:%S')
'10661014T08'

>>> # Exceptions.
>>> strftime('invalid', '%H')  
Traceback (most recent call last):
<class 'metomi.isodatetime.exceptions.ISO8601SyntaxError'>
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601         date representation: invalid
>>> strftime('2000', '%invalid')  
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i
>>> strftime('2000', '%Y', '%invalid')
... 
Traceback (most recent call last):
metomi.isodatetime.exceptions.StrftimeSyntaxError: Invalid         strftime/strptime representation: %i

Jinja2 Examples:

{% set START_CYCLE = '10661004T08+01' %}

{{START_CYCLE | strftime('%Y')}}  # 1066
{{START_CYCLE | strftime('%m')}}  # 10
{{START_CYCLE | strftime('%d')}}  # 14
{{START_CYCLE | strftime('%H:%M:%S %z')}}  # 08:00:00 +01
{{'12,30,2000' | strftime('%m', '%m,%d,%Y')}}  # 12

cylc.flow.jinja.filters.duration_as.duration_as(iso8601_duration, units)[source]

Format an ISO8601 duration string as the specified units.

Units for the conversion can be specified in a case-insensitive short or long form:

Seconds - “s” or “seconds”
Minutes - “m” or “minutes”
Hours - “h” or “hours”
Days - “d” or “days”
Weeks - “w” or “weeks”

While the filtered value is a floating-point number, it is often required to supply an integer to workflow entities (e.g. environment variables) that require it. This is accomplished by chaining filters:

{{CYCLE_INTERVAL | duration_as('h') | int}} - 24
{{CYCLE_SUBINTERVAL | duration_as('h') | int}} - 0
{{CYCLE_INTERVAL | duration_as('s') | int}} - 86400
{{CYCLE_SUBINTERVAL | duration_as('s') | int}} - 1800

Parameters:

iso8601_duration (str) – Any valid ISO8601 duration as a string.
units (str) – Destination unit for the duration conversion

Returns:

The total number of the specified unit contained in the specified duration as a floating-point number.

Raises:

ISO8601SyntaxError – In the event of an invalid datetime string.

Return type:

float

Python Examples:

>>> # Basic usage.
>>> duration_as('PT1M', 's')
60.0
>>> duration_as('PT1H', 'seconds')
3600.0

>>> # Exceptions.
>>> duration_as('invalid value', 's')  
Traceback (most recent call last):
metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601        duration representation: invalid value
>>> duration_as('invalid unit', '#')  
Traceback (most recent call last):
ValueError: No matching units found for #

Jinja2 Examples:

{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('h') }}  # 24.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('hours') }}  # 0.5
{% set CYCLE_INTERVAL = 'PT1D' %}
{{ CYCLE_INTERVAL | duration_as('s') }}  # 86400.0
{% set CYCLE_SUBINTERVAL = 'PT30M' %}
{{ CYCLE_SUBINTERVAL | duration_as('seconds') }}  # 1800.0

Associative Arrays In Jinja2

Associative arrays (or dictionaries) are very useful. For example:

#!Jinja2
{% set obs_types = ['airs', 'iasi'] %}
{% set resource = { 'airs':'ncpus=9', 'iasi':'ncpus=20' } %}

[scheduling]
    [[graph]]
        R1 = OBS
[runtime]
    [[OBS]]
        platform = platform_using_pbs
    {% for i in obs_types %}
    [[ {{i}} ]]
        inherit = OBS
        [[[directives]]]
             -I = {{ resource[i] }}
     {% endfor %}

Here’s the result:

$ cylc config -i [runtime][airs]directives <workflow-id>
-I = ncpus=9

Default Values and Template Variables

You can provide template variables to Cylc in 4 ways:

Using the --set-file (-S) option.
Using the --set (-s) option.
Using the --set-list (-z) option.
Using a plugin, such as Cylc Rose.

Note

If the same variable is set by more than one method, the last source in the above list is used.

The `-s`, `-z` and `--set-file` Options

$ # set the Jinja2 variable "answer" to 42
$ cylc play <workflow-id> -s answer=42

A Python string-list is a valid value, but a lot to type, so --set-list (-z) is provided as a convenience:

# The set syntax
$ cylc play <workflow-id> -s "answers=['mice', 'dolphins']"
# ... can  be shortened to:
$ cylc play <workflow-id> -z answers=mice,dolphins

If you need to define a lot of variables, you can so in a file using the --set-file option:

$ # create a set file
$ cat > my-set-file <<__SET_FILE__
question='the meaning of life, the universe and everything'
answer=42
host='deep-thought'
__SET_FILE__

$ # run using the options in the set file
$ cylc play <workflow-id> --set-file my-set-file

Values must be Python literals e.g:

"string"   # string
123        # integer
12.34      # float
True       # boolean
None       # None type
[1, 2, 3]  # list
(1, 2, 3)  # tuple
{1, 2, 3}  # set
{"a": 1, "b": 2, "c": 3}  # dictionary

Note

On the command line you may need to wrap strings with an extra pair of quotes as the shell you are using (e.g. Bash) will strip the outer pair of quotes.

$ # wrap the key=value pair in single quotes stop the shell from
$ # stripping the inner quotes around the string:
$ cylc play <workflow-id> -s 'my_string="a b c"'

Here’s an example:

#!Jinja2
[meta]
    title = "Jinja2 example: use of defaults and external input"

    description = """
        The template variable N_MEMBERS can be set on the command line with
        --set or --set-file=FILE; but if not a default values is supplied.
    """

[scheduling]
    initial cycle point = 20100808T00
    final cycle point   = 20100816T00
    [[graph]]
        T00 = """
            foo => ENS
            ENS:succeed-all => bar
        """

[runtime]
    [[foo, bar]]
    [[ENS]]
{% for I in range( 0, N_MEMBERS | default( 3 )) %}
    [[ mem_{{ I }} ]]
        inherit = ENS
{% endfor %}

Here’s the result:

$ cylc list <workflow-id>
Jinja2 Template Error
'FIRST_TASK' is undefined
cylc-list <workflow-id>  failed:  1

$ # Note: quoting "bob" so that it is evaluated as a string
$ cylc list --set 'FIRST_TASK="bob"' <workflow-id>
bob
baz
mem_2
mem_1
mem_0

$ cylc list --set 'FIRST_TASK="bob"' --set 'LAST_TASK="alice"' <workflow-id>
bob
alice
mem_2
mem_1
mem_0

$ # Note: no quotes required for N_MEMBERS since it is an integer
$ cylc list --set 'FIRST_TASK="bob"' --set N_MEMBERS=10 <workflow-id>
mem_9
mem_8
mem_7
mem_6
mem_5
mem_4
mem_3
mem_2
mem_1
mem_0
baz
bob

Note also that cylc view --set FIRST_TASK=bob --jinja2 <workflow-id> will show the workflow with the Jinja2 variables as set.

Note

Workflows started with template variables set on the command line will restart with the same settings. You can set them again on the cylc play command line if they need to be overridden.

Using a plugin

Template plugins such as Cylc Rose should provide a set of template variables which can be provided to Cylc. For example, using Cylc Rose you add a rose-suite.conf file containing a [template variables] section which the plugin makes available to Cylc:

rose-suite.conf

[template variables]
ICP=1068

flow.cylc

#!jinja2
[scheduler]
   allow implicit tasks = True
[scheduling]
   initial cycle point = {{ICP}}
   [[dependencies]]
      P1Y = Task1

$ cylc config . -i "[scheduling]initial cycle point"
1068

Jinja2 Variable Scope

Jinja2 variable scoping rules may be surprising. For instance, variables set inside a for loop can’t be accessed outside of the block, so the following will not print # FOO is True:

{% set FOO = False %}
{% for item in items %}
    {% if item.check_something() %}
        {% set FOO = True %}
    {% endif %}
{% endfor %}
# FOO is {{FOO}}

Jinja2 documentation suggests using alternative constructs like the loop else block or the special loop variable. More complex use cases can be handled using namespace objects that allow propagating of changes across scopes:

{% set ns = namespace(foo=false) %}
{% for item in items %}
    {% if item.check_something() %}
        {% set ns.foo = true %}
    {% endif %}
{% endfor %}
# FOO is {{ns.foo}}

For detail, see Jinja2 Template Designer Documentation - Assignments

Raising Exceptions

Cylc provides two functions for raising exceptions in Jinja2 code. These exceptions are raised when the flow.cylc file is loaded and will prevent a workflow from running.

Note

These functions must be contained within {{ Jinja2 print statements, not {% code blocks.

Raise

The raise function will result in an error containing the provided text.

{% if not VARIABLE is defined %}
    {{ raise('VARIABLE must be defined for this workflow.') }}
{% endif %}

Assert

The assert function will raise an exception containing the text provided in the second argument providing that the first argument evaluates as False. The following example is equivalent to the “raise” example above.

{{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }}

Importing Python modules

Jinja2 allows to gather variable and macro definitions in a separate template that can be imported into (and thus shared among) other templates.

{% import "flow-utils.cylc" as utils %}
{% from "flow-utils.cylc" import VARIABLE as ALIAS %}
{{ utils.VARIABLE is equalto(ALIAS)) }}

Cylc extends this functionality to allow import of arbitrary Python modules.

{% from "itertools" import product %}
[runtime]
{% for group, member in product(['a', 'b'], [0, 1, 2]) %}
    [[{{group}}_{{member}}]]
{% endfor %}

For better clarity and disambiguation Python modules can be prefixed with __python__:

{% from "__python__.itertools" import product %}

Logging

It is possible to output messages to the Cylc log from within Jinja2, these messages will appear on the console when validating or starting a workflow. This can be useful for development or debugging.

Example flow.cylc:

#!Jinja2
{% from "cylc.flow" import LOG %}
{% do LOG.debug("Hello World!") %}

Example output:

$ cylc validate . --debug
DEBUG - Loading site/user config files
DEBUG - Reading file <file>
DEBUG - Processing with Jinja2
DEBUG - Hello World!
...
Valid for cylc-<version>

Log messages will appear whenever the workflow configuration is loaded so it is advisable to use the DEBUG logging level which is suppressed unless the --debug option is provided.

Debugging

It is possible to run Python debuggers from within Jinja2 via the import mechanism.

For example to use a PDB breakpoint you could do the following:

#!Jinja2

{% set ANSWER = 42 %}

{% from "pdb" import set_trace %}
{% do set_trace() %}

The debugger will open within the Jinja2 code, local variables can be accessed via the _Context__self variable e.g:

$ cylc validate <id>
(Pdb) _Context__self['ANSWER']
42

Jinja2 Syntax	Description
`{# comment #}`	Comment
`{% if true %}`	Expression
`{{ var }}`	Print statement

Jinja2

Access to Workflow Files

Workflow Context variables

Environment Variables

Custom Jinja2 Filters, Tests and Globals

Associative Arrays In Jinja2

Default Values and Template Variables

The -s, -z and --set-file Options

Using a plugin

Jinja2 Variable Scope

Raising Exceptions

Raise

Assert

Importing Python modules

Logging

Debugging

The `-s`, `-z` and `--set-file` Options