Installing Workflows
Cylc commands identify workflows via their names, which are relative path names
under the cylc-run directory, ~/cylc-run/
by default.
Workflows can be grouped together under sub-directories. E.g. there are three
workflows in the example below: nwp/oper/region1
, nwp/oper/region2
and
nwp/test/region1
.
$ cylc scan --state=all --name nwp --format=tree
nwp
├─oper
│ ├─ region1
│ └─ region2
└─test
└─ region1
This chapter will demonstrate how to install a workflow from an arbitrary
location, called a source directory.
cylc install
will create a new run directory in the cylc-run directory
for each installation of a workflow.
The Cylc Install Command
Workflows can be installed with the cylc install
command, which creates
the run directory structure and some service files underneath it.
Note
It is possible to run a workflow without installation by writing it
directly in the run directory.
However, we recommend that you write your workflow in a source
directory and use cylc install
to create a fresh run directory.
Using Cylc Install
cylc install
accepts as its only argument either:
a workflow source name (see Configurable Source Directories below) e.g.
foo/bar
a path to the source directory e.g.
./foo/bar
or/path/to/foo/bar
(note that relative paths must begin with./
to distinguish them from workflow source names)
If no argument is supplied, the current working directory is used as the source directory.
Note
To avoid confusion, cylc install
does not permit any of the following
reserved directory names: log
, .service
, _cylc-install
, work
, runN
, suite.rc
, share
, flow.cylc
Options
The table below illustrates several command line options that control naming
of run directories for installed workflows (the current working directory in
these examples is ~/cylc-src/my-flow
):
Command |
Results in Installation in Directory |
---|---|
|
|
|
|
|
|
|
|
Configurable Source Directories
You can configure a list of directories that cylc install SOURCE_NAME
and the GUI will search for source directories inside, using
global.cylc[install]source dirs
. For example, if you have
# global.cylc
[install]
source dirs = ~/cylc-src, ~/roses
then cylc install dogs/fido
will search for a workflow source directory
~/cylc-src/dogs/fido
, or, failing that, ~/roses/dogs/fido
, and install
the first match (into ~/cylc-run/dogs/fido/run1
).
Numbered Runs
By default, cylc install
creates numbered run directories, i.e.
~/cylc-run/<workflow-name>/run<number>
, provided the options
--run-name
or --no-run-name
are not used. The run number automatically
increments each time cylc install
is run, and a symlink runN
is
created/updated to point to the run.
Example: A typical run directory structure, after three executions of
cylc install
will look as follows.
├── _cylc-install
│ └── source -> /home/cylc-src/test-flow
├── run1
│ ├── flow.cylc
│ └── log
│ └── install
│ └── <time-stamp>-install.log
├── run2
│ ├── flow.cylc
│ └── log
│ └── install
│ └── <time-stamp>-install.log
├── run3
│ ├── flow.cylc
│ └── log
│ └── install
│ └── <time-stamp>-install.log
└── runN -> /home/cylc-run/test-flow/run3
The numbered runs option may be overridden, using either the --no-run-name
or the --run-name
options.
Named Runs
As an alternative to numbered runs, it is possible to name the runs, using the
--run-name
option.
In this case, the runN
symlink will not be created.
This option cannot be used if numbered runs are already present. Likewise,
numbered runs cannot be used if named runs are already present.
Symlink Directories
You can configure workflow run directories and certain
sub-directories as symlinks to other locations. This is a useful way of
offloading data onto other drives to limit the disk space taken up by
~/cylc-run
.
The symlink targets are configured per install target in
global.cylc[install][symlink dirs]
. For more information see
Symlinking Directories
The Cylc Install Process
There are two main parts of the cylc install
process.
File Installation
Symlinking of Directories
1. File Installation
Installation will involve copying the files found in the source directory into
a new run directory. If you wish to install files into an existing run
directory, use cylc reinstall
, see Reinstalling a Workflow.
Excluding Items From Installation
By default, cylc install will exclude .git
, .svn
directories.
To configure excluded files and directories from the file installation,
create a .cylcignore
file in your source directory, this supports
pattern matching.
The following example will detail how to install a workflow, including configuring files to be excluded from the installation.
Example Installation
For example:
We will look at running the cylc install command inside the directory
~/cylc-src/test-flow
with the following directory structure:
$ pwd
/home/cylc-src/test-flow
$ tree -all
├── .cylcignore
├── dir1
│ ├── another-file
│ └── file
├── dir2
│ ├── another-file
│ └── file
├── file1
├── file2
├── file3
├── flow.cylc
├── textfile1.txt
└── textfile2.txt
We wish to omit any files matching the pattern *.txt
, the file
file1
, the contents of dir1
and the contents of dir2
including the
directory itself.
$ cat .cylcignore
*.txt
file1
dir1/*
dir2
Now we are ready to install our workflow.
$ cylc install
INSTALLED test-flow from home/cylc-src/test-flow -> home/cylc-run/test-flow/run1
Looking at the directory structure that has been created
$ tree -all home/cylc-run/test-flow/run1
├── dir1
├── file2
├── file3
├── flow.cylc
├── log
│ └── install
│ └── <time-stamp>-install.log
└── .service
2. Symlinking of Directories
If symlink directories are configured in the [[[localhost]]]
section in
global.cylc[install][symlink dirs]
,
cylc install
will create these symlinks and their target locations
(symlinks for remote install targets do not get created until
starting the workflow).
Override default symlink locations
You can override the default locations set in
global.cylc[install][symlink dirs]
by using the --symlink-dirs
option with cylc install
.
For example, using the command line option
--symlink-dirs="log=$DIR, run=/path/to/dir,..."
will symlink:
$DIR -> ~/cylc-run/workflow/log
/path/to/dir -> ~/cylc-run/workflow/run
Note
If configuring symlink dirs on the command line, the global configured symlink dirs will not be used to source directories not included in the command line list.
To skip making localhost symlinks
Use --symlink-dirs=""
with the cylc install
command.
Automatically Generated Directories and Files
Running cylc install
will generate some extra files in your workflow run
directory.
The service directory will be created in preparation for running the workflow. This is needed to store essential files used by Cylc.
A
_cylc-install
directory containing asource
symlink to the source directory. This is needed to enable Cylc to determine the original workflow source forcylc reinstall
.A new
install
directory in the workflow’s log directory, with a time-stamped install log file containing information about the installation.
Cylc plugins (such as Cylc Rose) may generate additional files.
Reinstalling a Workflow
To apply changes from your source directory to the installed run directory,
use cylc reinstall
. Changes made will be recorded in the workflow log
directory.
For example:
$ cylc reinstall myflow/run1
Cylc will determine the source directory and update your workflow.
Returning to the example from above (see Example Installation).
The source directory, ~/cylc-src/test-flow
has been altered as follows:
$ tree -all ~/cylc-src/test-flow
├── .cylcignore
├── dir1
│ ├── another-file
│ └── file
├── dir2
│ ├── another-file
│ └── file
├── dir3
│ ├── another-file
│ └── file
├── file1
├── file2
├── file3
├── flow.cylc
├── textfile1.txt
└── textfile2.txt
$ cat .cylcignore
*.txt
file1
dir2
We wish to update our ~/cylc-run/test-flow/run1
with the directories dir1
and dir3
:
$ cylc reinstall test-flow/run1
The run directory now looks as follows:
$ tree -all home/cylc-run/test-flow/run1
├── dir1
│ ├── another-file
│ └── file
├── dir3
│ ├── another-file
│ └── file
├── file2
├── file3
├── flow.cylc
├── log
│ └── install
│ └── <time-stamp>-install.log
│ └── <time-stamp>-reinstall.log
└── .service
Note
If your workflow needs to create or install scripts or executables at runtime
and you don’t want Cylc to delete them on re-installation, you can use
bin
and lib/python
directories in the
workflow share directory.
Expected Errors
There are some occasions when installation is expected to fail:
log
,share
,work
or_cylc-install
directories exist in the source directoryNeither
flow.cylc
nor the deprecatedsuite.rc
are found in the source directoryBoth
flow.cylc
and the deprecatedsuite.rc
are found in the source directory. Only one should be present.The workflow name is an absolute path or invalid
Workflow names are validated by
cylc.flow.unicode_rules.WorkflowNameValidator
.The workflow name contains a directory name that is any of these reserved filenames:
log
,.service
,_cylc-install
,work
,runN
,suite.rc
,share
,flow.cylc
The install would create nested install directories. Neither a new installation in a subdirectory of an existing one, nor a directory containing an existing installation are permitted. For example, having installed a workflow in
bar
you would be unable to install one infoo
orfoo/bar/baz
.foo └── bar ├── _cylc-install ├── baz ├── run1 └── runN
This means you cannot install using
--no-run-name
for a workflow that has installed numbered/named runs, nor can you install numbered/named runs for a workflow where--no-run-name
was used.Trying to install a workflow into an already existing run directory.
cylc reinstall
should be used for this, see Reinstalling a Workflow.The source directory path does not match the source directory path of a previous installation. i.e. running
cylc install
in~/cylc-src/my-flow
, followed by runningcylc install
from~/different/my-flow
.
Compound Commands
Cylc provides compound commands which carry out multiple operations.
For example Cylc provides a command to validate, install and play a workflow.
Compound commands make common ways of working easier.
Tip
Use cylc command --help
to get help for each compound command,
including a full list of available options.
cylc vip
(Validate, Install and Play)
cylc vip /home/me/cylc-src/my-workflow
is the same as running:
$ cylc validate /home/me/cylc-src/my-workflow
$ cylc install /home/me/cylc-src/my-workflow
INSTALLED my-workflow/run1 from /home/me/cylc-src/my-workflow
$ cylc play my-workflow
cylc vr
(Validate, Reinstall and Reload or Play)
cylc vr my-workflow
is the same as running:
# Check that the changes you want to
# make will be valid after installation:
$ cylc validate my-workflow --against-source
$ cylc reinstall my-workflow
# If workflow is running:
$ cylc reload my-workflow
# If workflow is stopped:
$ cylc play my-workflow