Proposal - rose suite-run functionality migration to cylc

The work described in this document aims to:

This will complete cylc-flow #1885.

Work Plan

  1. Implement basic platforms support. See platforms proposal.

  2. Add new platforms features required to support rose suite-run functionality. See platforms proposal.

  3. Define new cylc command to replace rose suite-run (and adapt rose suite-run to provide backwards compatibility). See below.

The remaining points can happen any time after 1) is complete although note that 2) and 3) are higher priority.

  1. Support generation of cylc config reference documentation from the code and ensure all settings are documented. See cylc-doc #11. Alternatively, review and update the existing documentation.

  2. Implement further enhancements to the platforms support. See platforms proposal. Note that these are not essential for Cylc 8.0.

  3. Review all config settings and make any agreed changes. See cylc-flow #3422. Note that these are not essential for Cylc 8.0 although we should try to address any non-controversial changes.

Replacing rose suite-run

Key Functionality

  1. Installation of suites on platforms and support for moving the share, work & log directories to different locations. This will be included in the platforms support.

  2. Support installation of a suite from the source location (working copy) to the run location (cylc-run).
    • This functionality assumes the source location (working copy) and the run location (cylc-run) are both accessible on the host where the command is run.
    • The installation needs to create the run time directory structure which has been configured for the localhost platform.
    • Need to be able to update from the source location as part of reload.
  3. On start up, archive any previous logs in a tar.gz.
    • Propose to stop supporting this since, in most cases, we will use a separate run directory each time a suite is installed.
    • Continue to support a timestamped log dir for the special occasions where we re-use an existing run directory (see the discussion in the CLI changes below).
  4. On start up and reload, validate suite (using --strict).
    • Hopefully this can become a standard part of suite start-up rather than a separate step. This was difficult previously because cylc loads the suite after it has been detached?
    • Consider making all validation strict? See cylc-flow #3866.
  5. Handle rose-suite.conf functionality including optional configurations. See cylc-flow #3819 and rose #2412.

  6. Utility to remove installed locations (on all platforms) on demand (equivalent to rose suite-clean).

  7. Use a different run directory for each installation of a suite:
    • This is big change in working practice but Cylc 8 is a good time to introduce it.
    • Use a symbolic link to identify the latest run which is the default name when performing operations on a suite?
  8. Consider rose prune functionality.
    • Need to make sure it will continue to work correctly.
    • Longer term we should build housekeeping into Cylc, see cylc-flow #1159.
  9. Record version control information when you install a suite, see cylc-flow #3849.
    • rose suite-run current supports this for Subversion only (svn info + svn status + svn diff).
    • Extend this to support equivalent information for git (and preferably make it easy to plugin support for other systems as well).

CLI changes

cylc install - new command to install workflows.

cylc play - new command to run an installed workflow (replacing cylc run|start).

cylc reinstall - new command to re-install workflows.

cylc reload - no change from existing command.

cylc validate - as per existing command with the following changes:

Several combinations of the above commands will be commonly used. Propose to support these as separate commands:

cylc pause|release - new commands with aliases pause = hold and release = unpause = unhold.

cylc register - remove: functionality no longer required.

cylc clean - new command to replace rose suite-clean.

For all commands which have been replaced (run, restart, register,etc) we will replace them with (hidden, not listed in CLI help) commands which simply report what commands should be used in their place.

Once the main CLI changes are in place we should do a complete review of the entire cylc command set to see whether there are further commands which should be retired or altered.

Rose commands:

Related issues / notes:

Log file changes

Proposed contents of the log directory in the run directory.