Start, Restart, Reload

The cylc play command starts a new instance of the Cylc scheduler program to manage a workflow, for the duration of the run. It also resumes paused workflows.

There are several ways to start a workflow that is not running:

  • cylc play from the initial cycle point

    • This is the default for a workflow that has not run yet

    • The initial cycle point is taken from flow.cylc or the command line

    • Dependence on earlier tasks is ignored, for convenience

    • (In Cylc 7, this was called a cold start)

  • cylc play --start-cycle-point=POINT from a later start cycle point

    • This is an option for a workflow that has not run yet

    • The start point must be given on the command line

    • Dependence on earlier tasks is ignored, for convenience

    • The initial cycle point value defined in flow.cylc is preserved

    • (In Cylc 7, this was called a warm start)

  • cylc play --start-task=TASK-ID from any task(s) in the graph

    • This is an option for a workflow that has not run yet

    • The start task(s) must be given on the command line

    • The flow will follow the graph onward from the start task(s)

    • (In Cylc 7, this was not possible)

    • (This makes Cylc 7 restart-from-checkpoint obsolete)

  • cylc play to restart from previous state

    • This is automatic for a workflow that previously ran

    • The scheduler will carry on from exactly where it got to before

    • At start-up, it will automatically find out what happened to any active tasks that were orphaned during shutdown

    • Failed tasks are not automatically resubmitted at restart, in case the underlying problem has not been addressed

    • (In Cylc 7, this used the now-obsolete restart command)

Note

If a workflow gets stopped or killed it can be restarted, but to avoid corrupting an existing run directory it cannot be started again from scratch (unless you delete certain files from the run directory). To start from scratch again, install a new copy of the workflow to a new run directory.

See also

Start and Stop vs Initial and Final Cycle Points

All workflows have an initial cycle point and some may have a final cycle point. These define extent of the graph of tasks that Cylc can schedule to run.

Start and stop cycle points, if used, define a sub-section of the graph that the scheduler actually does run. For example:

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 5
    [[graph]]
        # every cycle: 1, 2, 3, 4, 5
        P1 = foo
        # every other cycle: 1, 3, 5
        P2 = bar

With a start cycle point of 2 and a stop cycle point of 4, the task foo would run at cycles 2, 3 & 4 and the task bar would only run at cycle 3.

../../_images/initial-start-stop-final-cp.svg

Restarts and the Initial, Final, Start and Stop Cycle Points

When a workflow is restarted or reloaded, any changes to [scheduling]final cycle point or [scheduling]stop after cycle point are normally picked up by Cylc.

However, this is not the case if you used the --final-cycle-point or --stop-cycle-point options in a prior run. If you use either of these options, the values you specify are stored in the workflow database, so the original values will be used if you restart the workflow later on. You can override these original values using the options again, and the new value will persist over restarts. Or, you can pick up the workflow config values by passing reload to the options (e.g. --final-cycle-point=reload); this will remove the stored value from the database so future restarts will go back to picking up any changes to the config.

Note

If the workflow reached the final cycle point and shut down, it is finished and cannot be restarted; the --final-cycle-point option will have no effect.

The --initial-cycle-point and --start-cycle-point options cannot be used in a restart; workflows always start from the cycle point where they previously stopped.

Reloading the Workflow Configuration at Runtime

The cylc reload command tells the target scheduler to reload its workflow configuration at run time. This is an alternative to shutting a workflow down and restarting it after making changes.

Note

If writing your workflows in a source directory, then be sure to reinstall your changes to the run directory before doing a reload.

If you make an error in the flow.cylc file before a reload, the workflow log will report an error and the reload will have no effect.

Note, Remote Initialization will be redone for each job platform, when the first job is submitted there after a reload.

Restarting or Reloading after Graph Changes

If dependencies have changed, tasks that were already active will spawn children according to their original outputs. Subsequent instances will have the new settings.

If tasks were removed from the graph, any active instances will be left to finish, but they will not spawn children. They can be removed manually if necessary, with cylc remove.

If new tasks were added to the graph, instances will be spawned automatically as upstream tasks complete the outputs that they depend on. If they have no parents to do that, you can trigger the first ones manually with cylc trigger.

Remote Initialization

For workflows that run on remote platforms, i.e. using a host other than localhost, Cylc performs an initialization process. This involves transferring files and directories required to run jobs, including authentication keys (see Authentication Files for more information).

The default directories included in the remote install are:

ana/

For workflow-level rose_ana analysis modules.

app/

For Application Configuration.

bin/

For scripts and executables to be run by tasks.

etc/

For miscellaneous resources.

lib/

For shared libraries e.g. Custom Jinja2 Filters, Tests and Globals.

These will be transferred from the workflow run directory on the scheduler host to the remote host. In addition, custom file and directories configured in flow.cylc[scheduler]install will be included in the transfer.

This remote initialization process also creates symlinks on the remote platform, if these are configured using global.cylc[install][symlink dirs]. This provides a way to manage disk space.

A log file is created on the scheduler to report information relating to the remote file installation process. There will be a separate log created per install target. These can be found in $HOME/cylc-run/<workflow-id>/log/remote-install/.

Installing Custom Files At Start-up

Cylc supports adding custom directories and files to the file installation.

If, for example, you wished to install directories dir1, dir2, and files file1, file2, add the following configuration to your flow.cylc, under the section [scheduler]install. To mark an item as a directory, add a trailing slash.

[scheduler]
    install = dir1/, dir2/, file1, file2

Note

Ensure files and directories to be installed are located in the top level of your workflow.

Install tasks are preferred for time-consuming installations because they don’t slow the workflow start-up process, they can be monitored, they can run directly on target platforms, and you can rerun them later without restarting the workflow.

Note

Files configured for installation to remote job platforms can be reinstalled by doing a reload. The reinstallation is done when the first job submits to a platform after the reload.

Troubleshooting

There are certain scenarios where remote initialization may fail. Cylc will return a REMOTE INIT FAILED message.

Timeout

Remote initialization has a timeout set at 10 minutes, after which remote initialization will fail. If you have particularly large files files to transfer, which you expect to exceed the 10 minute timeout, consider using an install task in your workflow.

Misconfiguration

Platforms must be correctly configured to ensure authentication keys, which are responsible for secure communication between the scheduler and the platform, are correctly in place. Sites can configure these platforms, insuring they match up with the correct install target. Cylc uses install targets as a way of recognising which platforms share the same file system. For more information, see What Are Install Targets?.

Files created at workflow start

Configuration Logs

A folder log/config is created where the workflow configuration is recorded, with all templating expanded:

  • flow-processed.cylc - A record of the current workflow configuration with templating expanded, but without being fully parsed: Duplicate sections will not be merged.

  • <datetime-stamp>-<start/restart/reload> - A record of the config at the time a workflow was started, restarted or reloaded, parsed by Cylc: Duplicate sections will be merged.

Note

These are particularly useful files to look at if the workflow configuration contains many template variables, to see how they are filled in.

The Workflow Contact File

The scheduler writes a contact file at $HOME/cylc-run/<workflow-name>/.service/contact that records workflow host, user, port number, process ID, Cylc version, and other information. Client commands read this file to find the scheduler.

The contact file gets removed automatically at shutdown (assuming the scheduler shuts down cleanly).

Authentication Files

See Authentication Files.