Installing Workflows

Cylc commands identify workflows via their names, which are relative path names under the cylc-run directory, ~/cylc-run/ by default.

Workflows can be grouped together under sub-directories. E.g. there are three workflows in the example below: nwp/oper/region1, nwp/oper/region2 and nwp/test/region1.

$ cylc scan --state=all --name nwp --format=tree
nwp
├─oper
│ ├─ region1
│ └─ region2
└─test
   └─ region1

This chapter will demonstrate how to install a workflow from an arbitrary location, called a source directory. cylc install will create a new run directory in the cylc-run directory for each installation of a workflow.

The Cylc Install Command

Workflows can be installed with the cylc install command, which creates the run directory structure and some service files underneath it.

Note

It is possible to run a workflow without installation by writing it directly in the run directory. However, we recommend that you write your workflow in a source directory and use cylc install to create a fresh run directory.

Using Cylc Install

cylc install accepts as its only argument either:

  • a workflow source name (see Configurable Source Directories below) e.g. foo/bar

  • a path to the source directory e.g. ./foo/bar or /path/to/foo/bar (note that relative paths must begin with ./ to distinguish them from workflow source names)

If no argument is supplied, the current working directory is used as the source directory.

Note

To avoid confusion, cylc install does not permit any of the following reserved directory names: share, suite.rc, log, .service, work, _cylc-install, runN, flow.cylc

Options

The table below illustrates several command line options that control naming of run directories for installed workflows (the current working directory in these examples is ~/cylc-src/my-flow):

Command

Results in Installation in Directory

cylc install

~/cylc-run/my-flow/run1

cylc install --no-run-name

~/cylc-run/my-flow

cylc install --run-name=new-run

~/cylc-run/my-flow/new-run

cylc install --workflow-name=new-name

~/cylc-run/new-name/run1

Configurable Source Directories

You can configure a list of directories that cylc install SOURCE_NAME and the GUI will search for source directories inside, using global.cylc[install]source dirs. For example, if you have

# global.cylc
[install]
    source dirs = ~/cylc-src, ~/roses

then cylc install dogs/fido will search for a workflow source directory ~/cylc-src/dogs/fido, or, failing that, ~/roses/dogs/fido, and install the first match (into ~/cylc-run/dogs/fido/run1).

Numbered Runs

By default, cylc install creates numbered run directories, i.e. ~/cylc-run/<workflow-name>/run<number>, provided the options --run-name or --no-run-name are not used. The run number automatically increments each time cylc install is run, and a symlink runN is created/updated to point to the run.

Example: A typical run directory structure, after three executions of cylc install will look as follows.

├── _cylc-install
│   └── source -> /home/cylc-src/test-flow
├── run1
│   ├── flow.cylc
│   └── log
│       └── install
│           └── <time-stamp>-install.log
├── run2
│   ├── flow.cylc
│   └── log
│       └── install
│           └── <time-stamp>-install.log
├── run3
│   ├── flow.cylc
│   └── log
│       └── install
│           └── <time-stamp>-install.log
└── runN -> /home/cylc-run/test-flow/run3

The numbered runs option may be overridden, using either the --no-run-name or the --run-name options.

Named Runs

As an alternative to numbered runs, it is possible to name the runs, using the --run-name option. In this case, the runN symlink will not be created. This option cannot be used if numbered runs are already present. Likewise, numbered runs cannot be used if named runs are already present.

The Cylc Install Process

There are two main parts of the cylc install process.

  1. File Installation

  2. Symlinking of Directories

1. File Installation

Installation will involve copying the files found in the source directory into a new run directory. If you wish to install files into an existing run directory, use cylc reinstall, see Reinstalling a Workflow.

Excluding Items From Installation

By default, cylc install will exclude .git, .svn directories. To configure excluded files and directories from the file installation, create a .cylcignore file in your source directory, this supports pattern matching.

The following example will detail how to install a workflow, including configuring files to be excluded from the installation.

Example Installation

For example: We will look at running the cylc install command inside the directory ~/cylc-src/test-flow with the following directory structure:

$ pwd
/home/cylc-src/test-flow
$ tree -all
├── .cylcignore
├── dir1
│   ├── another-file
│   └── file
├── dir2
│   ├── another-file
│   └── file
├── file1
├── file2
├── file3
├── flow.cylc
├── textfile1.txt
└── textfile2.txt

We wish to omit any files matching the pattern *.txt, the file file1, the contents of dir1 and the contents of dir2 including the directory itself.

$ cat .cylcignore
*.txt
file1
dir1/*
dir2

Now we are ready to install our workflow.

$ cylc install
INSTALLED test-flow from home/cylc-src/test-flow -> home/cylc-run/test-flow/run1

Looking at the directory structure that has been created

$ tree -all home/cylc-run/test-flow/run1
├── dir1
├── file2
├── file3
├── flow.cylc
├── log
│   └── install
│       └── <time-stamp>-install.log
└── .service

2. Symlinking of Directories

If symlink directories are configured in the [[[localhost]]] section in global.cylc[install][symlink dirs], cylc install will create these symlinks and their target locations (symlinks for remote install targets do not get created until starting the workflow).

Automatically Generated Directories and Files

Running cylc install will generate some extra files in your workflow run directory.

  • The service directory will be created in preparation for running the workflow. This is needed to store essential files used by Cylc.

  • A _cylc-install directory containing a source symlink to the source directory. This is needed to enable Cylc to determine the original workflow source for cylc reinstall.

  • A new install directory in the workflow’s log directory, with a time-stamped install log file containing information about the installation.

Cylc plugins (such as Cylc Rose) may generate additional files.

Reinstalling a Workflow

To apply changes from your source directory to the installed run directory, use cylc reinstall. Changes made will be recorded in the workflow log directory.

For example:

$ cylc reinstall myflow/run1

Cylc will determine the source directory and update your workflow.

Returning to the example from above (see Example Installation).

The source directory, ~/cylc-src/test-flow has been altered as follows:

$ tree -all ~/cylc-src/test-flow
├── .cylcignore
├── dir1
│   ├── another-file
│   └── file
├── dir2
│   ├── another-file
│   └── file
├── dir3
│   ├── another-file
│   └── file
├── file1
├── file2
├── file3
├── flow.cylc
├── textfile1.txt
└── textfile2.txt
$ cat .cylcignore
*.txt
file1
dir2

We wish to update our ~/cylc-run/test-flow/run1 with the directories dir1 and dir3:

$ cylc reinstall test-flow/run1

The run directory now looks as follows:

$ tree -all home/cylc-run/test-flow/run1
├── dir1
│   ├── another-file
│   └── file
├── dir3
│   ├── another-file
│   └── file
├── file2
├── file3
├── flow.cylc
├── log
│   └── install
│       └── <time-stamp>-install.log
│       └── <time-stamp>-reinstall.log
└── .service

Note

If your workflow needs to create or install scripts or executables at runtime and you don’t want Cylc to delete them on re-installation, you can use bin and lib/python directories in the workflow share directory.

Expected Errors

There are some occasions when installation is expected to fail:

  • log, share, work or _cylc-install directories exist in the source directory

  • Neither flow.cylc nor the deprecated suite.rc are found in the source directory

  • Both flow.cylc and the deprecated suite.rc are found in the source directory. Only one should be present.

  • The workflow name is an absolute path or invalid

    Workflow names are validated by cylc.flow.unicode_rules.WorkflowNameValidator.

    class cylc.flow.unicode_rules.WorkflowNameValidator

    The rules for valid workflow names:

    • must be between 1 and 254 characters long

    • cannot start with: ., -, numbers

    • can only contain: alphanumeric, /, _, +, -, ., @

  • The workflow name contains a directory name that is any of these reserved filenames: share, suite.rc, log, .service, work, _cylc-install, runN, flow.cylc

  • The install would create nested install directories. Neither a new installation in a subdirectory of an existing one, nor a directory containing an existing installation are permitted. For example, having installed a workflow in bar you would be unable to install one in foo or foo/bar/baz.

    foo
    └── bar
        ├── _cylc-install
        ├── baz
        ├── run1
        └── runN
    

    This means you cannot install using --no-run-name for a workflow that has installed numbered/named runs, nor can you install numbered/named runs for a workflow where --no-run-name was used.

  • Trying to install a workflow into an already existing run directory. cylc reinstall should be used for this, see Reinstalling a Workflow.

  • The source directory path does not match the source directory path of a previous installation. i.e. running cylc install in ~/cylc-src/my-flow, followed by running cylc install from ~/different/my-flow.