Workflow Design Guide

Cylc/Rose Workflow Design Best Practice Guide.

This document provides guidance on making complex Cylc + Rose workflows that are clear, maintainable, and portable. Note that best practice advice may evolve over time with the capabilities of Rose and Cylc.

Content is drawn from the Rose and Cylc user guides, earlier Met Office workflow design and operational workflow review documents, experience with real workflows across the Unified Model Consortium, and discussion among members of the UM TISD (Technical Infrastructure Workflow Design) working group.

We start with the most general topics (coding style, general principles), move on to more advanced topics (efficiency and maintainability, portable workflows), and end with some pointers to future developments.

Note

A good working knowledge of Cylc and Rose is assumed. For further details, please consult the:

• Cylc User Guide

• Rose Documentation

Note

For non-Rose users: this document comes out of the Unified Model Consortium wherein Cylc is used within the Rose workflow management framework. However, the bulk of the information in this guide is about Cylc workflow design; which parts are Rose-specific should be clear from context.

• Style Guidelines
  • Tab Characters
  • Trailing Whitespace
  • Indentation
  • Comments
  • Titles, Descriptions, And URLs
  • Line Length And Continuation
  • Task Naming Conventions
  • Rose Config Files
• Basic Principles
  • Cycle point time zone
  • Fine Or Coarse-Grained Workflows
  • Monolithic Or Interdependent Workflows
  • Self-Contained Workflows
  • Task Host Selection
  • Task Scripting
  • Rose Apps
  • Rose Metadata Compliance
  • Task Independence
  • Clock-Triggered Tasks
  • Rose App File Polling
  • Task Execution Time Limits
  • Restricting Workflow Activity
  • Workflow Housekeeping
  • Complex Jinja2 Code
  • Shared Configuration
  • Automating Failure Recovery
  • Include Files
• Efficiency And Maintainability
  • The Task Family Hierarchy
  • Generating Tasks Automatically
  • Optional App Config Files
• Portable Workflows
  • The Jinja2 SITE Variable
  • Site Include-Files
  • Site-Specific Graphs
  • Inlined Site-Switching
  • Site-Specific Workflow Variables
  • Site-Specific Optional Workflow Configs
  • Site-Agnostic File Paths in App Configs
  • Site-Specific Optional App Configs
  • An Example
  • Collaborative Development Model
  • Research-To-Operations Transition