cylc-admin

Proposal - rose suite-run functionality migration to cylc

The work described in this document aims to:

This will complete cylc-flow #1885.

Work Plan

  1. Implement basic platforms support. See platforms proposal.

  2. Add new platforms features required to support rose suite-run functionality. See platforms proposal.

  3. Define new cylc command to replace rose suite-run (and adapt rose suite-run to provide backwards compatibility). See below.

The remaining points can happen any time after 1) is complete although note that 2) and 3) are higher priority.

  1. Support generation of cylc config reference documentation from the code and ensure all settings are documented. See cylc-doc #11. Alternatively, review and update the existing documentation.

  2. Implement further enhancements to the platforms support. See platforms proposal. Note that these are not essential for Cylc 8.0.

  3. Review all config settings and make any agreed changes. See cylc-flow #3422. Note that these are not essential for Cylc 8.0 although we should try to address any non-controversial changes.

Replacing rose suite-run

Key Functionality

  1. Installation of suites on platforms and support for moving the share, work & log directories to different locations. Currently this is done by rose suite-run but we will include this in the platforms support.

  2. On start up and reload, install the suite from the source location (working copy) to the run location (cylc-run).
    • Currently this functionality assumes a shared filesystem but we should support workflow server hosts with separate filesystems (future enhancement?).
    • Needs to create the run time directory structure which has been configured for the localhost platform.
  3. On start up, housekeep / archive logs and other items generated by the suite.

  4. On start up and reload, validate suite (using --strict).
    • Hopefully this can become a standard part of suite start-up rather than a separate step. This was difficult previously because cylc loads the suite after it has been detached?
    • Consider making all validation strict? If we want really want to allow naked dummy tasks in some circumstances then we would be better to add a suite configuration for this.
  5. Handle rose-suite.conf functionality
    • Pull items from other locations / sources on installation (e.g. svn).
    • Configure Jinja2 / EmPy inputs.
    • Set environment variables exported to the suite daemon?
    • Continue to handle rose-suite.conf using a plugin preprocessor.
    • Decide which features to support natively in Cylc and how.
  6. Utility to remove installed locations (on all platforms) on demand (equivalent to rose suite-clean).

  7. On starting up a new run of the suite assign a new run name:
    mi-aa001/
      run1
      run2
    
    • This is new functionality but Cylc 8 is a good time to introduce it.
    • Use a symbolic link to identify the latest run which is the default name when performing operations on a suite?
  8. Consider the behaviour created by the CYLC_VERSION and ROSE_VERSION variables in rose-suite.conf. These allow rose suite-run to:
    • Change the version of Cylc and check the version of Rose used to run the suite.
    • Prevent suites accidentally upgrading or downgrading versions on restart.
  9. Consider rose prune functionality.
    • Need to make sure it will continue to work correctly.
    • Need Cylc equivalent but this should be done as part of cylc-flow #1159.

CLI

We need a new Cylc command to replace rose suite-run. See cylc-flow #1030.