Cylc 7 Compatibility Mode

Does This Change Affect Me?

This will affect you if you want to run Cylc 7 (suite.rc) workflows using Cylc 8.

Overview

Cylc 8 can run most Cylc 7 workflows “as is”. The suite.rc filename triggers a backward compatibility mode in which:

• implicit tasks are allowed by default

  • (unless a rose-suite.conf file is found in the run directory for consistency with rose suite-run behaviour)

  • (Cylc 8 does not allow implicit tasks by default)

• cycle point time zone defaults to the local time zone

  • (Cylc 8 defaults to UTC)

• waiting tasks are pre-spawned to mimic the Cylc 7 scheduling algorithm and stall behaviour, and these require suicide triggers for alternate graph branching

  • (Cylc 8 spawns tasks on demand, and suicide triggers are not needed for branching)

• only succeeded task outputs are *required*, meaning the scheduler will retain tasks that do not succeed as incomplete

  • (in Cylc 8, all outputs are required unless marked as *optional* by the new ? syntax)

Required Changes

Providing your Cylc 7 workflow does not use syntax that was deprecated at Cylc 7, you may be able to run it using Cylc 8 without any modifications while in compatibility mode.

First, run cylc validate with Cylc 7 on your suite.rc workflow to check for deprecation warnings and fix those before validating with Cylc 8. See below for an example.

Warning

cylc validate operates on the processed suite.rc, which means it will not detect any deprecated syntax that is inside a currently-unused Jinja2/EmPy if...else branch.

Some workflows may require modifications to either upgrade to Cylc 8 or make interoperable with Cylc 8 backward compatibility mode. Read on for more details.

Cylc commands in task scripts

Check for any use of Cylc commands in task scripting. Some Cylc 7 commands have been removed and some others now behave differently. However, cylc message and cylc broadcast have not changed. See the full list of command line interface changes and see below for an example.

Python 2 to 3

Whereas Cylc 7 runs using Python 2, Cylc 8 runs using Python 3. This affects: - modules imported in Jinja2 - Jinja2 filters, tests and globals - custom xtrigger functions

Note that task scripts are not affected - they run in an independent environment.

See Python 2 => 3 for more information and examples of how to implement interoperability if your workflows extend Cylc or Jinja2 with custom Python scripts.

Other caveats

• Cylc 8 cannot restart a partially completed Cylc 7 workflow in-place. If possible, complete the run with Cylc 7. Otherwise, see Continuing a Cylc 7 Workflow with Cylc 8.

• Cylc 8 only transfers certain files and directories by default during remote installation. See Remote Installation for more information.

• Cylc 8 does not support excluding/including tasks at start-up. If your workflow used this old functionality, it may have been used in combination with the cylc insert command (which has been removed from Cylc 8) and cylc remove (which still exists but is much less needed).

• Cylc 8 does not support specifying remote usernames using flow.cylc[runtime][<namespace>][remote]owner.

Examples

Validating with Cylc 7

Consider this configuration:

suite.rc

[scheduling]
    initial cycle point = 11000101T00
    [[dependencies]]
        [[[R1]]]
            graph = task

[runtime]
    [[task]]
        pre-command scripting = echo "Hello World"

Running cylc validate at Cylc 7 we see that the workflow is valid, but we are warned that pre-command scripting was replaced by pre-script at 6.4.0:

Cylc 7 validation

$ cylc validate .
WARNING - deprecated items were automatically upgraded in 'suite definition':
WARNING -  * (6.4.0) [runtime][task][pre-command scripting] -> [runtime][task][pre-script] - value unchanged
Valid for cylc-7.8.7

Note

Cylc 7 has handled this deprecation for us, but at Cylc 8 this workflow will fail validation.

Cylc 8 validation

$ cylc validate .
IllegalItemError: [runtime][task]pre-command scripting

You must change the configuration yourself. In this case:

-     pre-command scripting = echo "Hello World"
+     pre-script = echo "Hello World"

Validation will now succeed.

Cylc commands in task scripts

You might have a task script that calls a Cylc command like so:

[runtime]
    [[foo]]
        script = cylc hold "$CYLC_SUITE_NAME"

The cylc hold command has changed in Cylc 8. It is now used for holding tasks only; use cylc pause for entire workflows. (Additionally, $CYLC_SUITE_NAME is deprecated in favour of $CYLC_WORKFLOW_ID, though still supported.)

In order to make this interoperable, so that you can run it with both Cylc 7 and Cylc 8 backward compatibility mode, you could do something like this in the bash script:

[runtime]
    [[foo]]
        script = """
            if [[ "${CYLC_VERSION:0:1}" == 7 ]]; then
                cylc hold "$CYLC_SUITE_NAME"
            else
                cylc pause "$CYLC_WORKFLOW_ID"
            fi
        """

Note this logic (and the $CYLC_VERSION environment variable) is executed at runtime on the job host.

Alternatively, you could use Jinja2 like so:

[runtime]
    [[foo]]
        {% if CYLC_VERSION is defined and CYLC_VERSION[0] == '8' %}
            script = cylc pause "$CYLC_WORKFLOW_ID"
        {% else %}
            script = cylc hold "$CYLC_SUITE_NAME"
        {% endif %}

Note this logic (and the CYLC_VERSION Jinja2 variable) is executed locally prior to Cylc parsing the workflow configuration.

Renaming to flow.cylc

When your workflow runs successfully in backward compatibility mode, it is ready for renaming suite.rc to flow.cylc. Doing this will turn off backward compatibility mode, and validation in Cylc 8 will show deprecation warnings.

See also

Configuration Changes at Cylc 8

Important

More complex workflows (e.g. those with suicide triggers) may fail validation once backward compatibility is off - see Graph branching, optional outputs and suicide triggers