Workflow Logs

Cylc produces log files which record information about how the workflow was installed, configured when started or changed, and about tasks that have run and their outcomes.

Log Files

Cylc log files are located in the workflow run directory.

log/scheduler/

Contains the scheduler log files, these detail everything that a workflow did whilst it ran and are key to debugging issues.

Every time you start or restart a workflow a new log file is created in this location:

log/scheduler/<start_number>-<type>-<file_number>.log

For more information on this file, see The Scheduler Log File.

log/config/

Contains a record of the workflow configuration files:

log/config/<start_number>-<type>-<file_number>.cylc

The parsed contents of the flow.cylc (or suite.rc) file from each start, restart or reload of the workflow.

log/config/flow-processed.cylc

The unparsed contents of the flow.cylc (or suite.rc) file that the workflow was most recently started or restarted from. This file has not been parsed by Cylc so is still in the definition order, however, Jinja2 process has been performed making it useful for debugging Jinja2 issues.

log/config/<time>-rose-suite.conf

The parsed contents of the rose-suite.conf file if present in the workflow.

log/install/

Contains the workflow installation log (i.e. the output of the cylc install command.

log/install/<start_number>-install.log
log/remote-install/

Contains a record of any remote platforms that the workflow has been installed onto.

log/remote-install/<start_number>-<type>-<platform>.log
log/version/

If the workflow source directory is version controlled with Git or SVN, this directory will contain information extracted from the version control system at the time the workflow was installed:P

log/version/uncommitted.diff

Any uncommitted diff in unified diff format.

log/version/vcs.json

Information including the commit hash / revision number.

opt/rose-suite-cylc-install.conf

If Rose is used, this file will contain any Rose options that were used with the cylc install command (i.e. any overrides which were applied when the workflow was installed).

Note

Where the <variables> in the above paths are:

start_number

Is 1 when you first start the workflow. This number increments with each restart.

type

Is either start, restart or reload.

file_number

Is initially 1, if the log file exceeds the maximum size in bytes, it will “roll over” into a new log file and this number will increment. If the number of “roll over” log files exceeds the rolling archive length, then Cylc will remove an old log file before creating a new one.

time

Is a timestamp in ISO8601 format.

The Scheduler Log File

Each workflow maintains its own log of time-stamped events in the workflow log directory ($HOME/cylc-run/<workflow-id>/log/scheduler/).

The information logged here includes:

  • Event timestamps, at the start of each line

  • Workflow server host, port and process ID

  • Workflow initial and final cycle points

  • Workflow (re)start number (1 for the first play, 2 or more for restarts)

  • Task events (task started, succeeded, failed, etc.)

  • Workflow stalled warnings.

  • Client commands (e.g. cylc hold)

  • Job IDs.

  • Information relating to the remote file installation is contained in a separate log file, which can be found in $HOME/cylc-run/<workflow-id>/log/remote-install/.

Note

Workflow log files are primarily intended for human eyes. If you need to have an external system to monitor workflow events automatically, use:

Rather than parse the log files.

Cylc Message

The Scheduler Log also records messages sent by cylc message allowing you to add custom messages to this log.

For example, if your task contained the following code:

cylc message -- "ERROR:some_function failed."

Your log should produce output similar to:

ERROR - [21491012T0410Z/mytask running job:01 flows:1] (received)some_function failed ERROR at 2023-04-14T11:36:35+01:00

Severity levels are the same as those used by Python’s logger:

  • CRITICAL

  • ERROR

  • WARNING

  • INFO

  • DEBUG

Messages logged at “DEBUG” will only appear in the scheduler log if the workflow is played with cylc play --debug.