Cylc documentation¶
The Cylc Suite Engine
Current release: 7.9.3
Released Under the GNU GPL v3.0 Software License
Copyright (C) NIWA & British Crown (Met Office) & Contributors.
Cylc (“silk”) is a workflow engine for cycling systems - it orchestrates distributed suites of interdependent cycling tasks that may continue to run indefinitely.
Table of Contents:
- 1. Introduction: How Cylc Works
- 2. Cylc Screenshots
- 3. Installation
- 4. Cylc Terminology
- 5. Workflows For Cycling Systems
- 6. Global (Site, User) Configuration Files
- 7. Tutorial
- 7.1. User Config File
- 7.2. User Interfaces
- 7.3. Suite Configuration
- 7.4. Suite Registration
- 7.5. Suite Passphrases
- 7.6. Import The Example Suites
- 7.7. Rename The Imported Tutorial Suites
- 7.8. Suite Validation
- 7.9. Hello World in Cylc
- 7.10. Editing Suites
- 7.11. Running Suites
- 7.12. Remote Suites
- 7.13. Discovering Running Suites
- 7.14. Task Identifiers
- 7.15. Job Submission: How Tasks Are Executed
- 7.16. Locating Suite And Task Output
- 7.17. Viewing Suite Logs in a Web Browser: Cylc Review
- 7.18. Remote Tasks
- 7.19. Task Triggering
- 7.20. Runtime Inheritance
- 7.21. Triggering Families
- 7.22. Triggering Off Of Families
- 7.23. Suite Visualization
- 7.24. External Task Scripts
- 7.25. Cycling Tasks
- 7.26. Jinja2
- 7.27. Task Retry On Failure
- 7.28. Other Users’ Suites
- 7.29. Other Things To Try
- 8. Suite Name Registration
- 9. Suite Configuration
- 9.1. Suite Configuration Directories
- 9.2. Suite.rc File Overview
- 9.3. Scheduling - Dependency Graphs
- 9.4. Runtime - Task Configuration
- 9.4.1. Task and Namespace Names
- 9.4.2. Root - Runtime Defaults
- 9.4.3. Defining Multiple Namespaces At Once
- 9.4.4. Runtime Inheritance - Single
- 9.4.5. Runtime Inheritance - Multiple
- 9.4.6. How Runtime Inheritance Works
- 9.4.7. Task Execution Environment
- 9.4.8. How Tasks Get Access To The Suite Directory
- 9.4.9. Remote Task Hosting
- 9.5. Visualization
- 9.6. Parameterized Tasks
- 9.7. Jinja2
- 9.8. EmPy
- 9.9. Omitting Tasks At Runtime
- 9.10. Naked Dummy Tasks And Strict Validation
- 10. Task Implementation
- 11. Task Job Submission and Management
- 12. External Triggers
- 13. Running Suites
- 13.1. Suite Start-Up
- 13.2. Reloading The Suite Configuration At Runtime
- 13.3. Task Job Access To Cylc
- 13.4. The Suite Contact File
- 13.5. Task Job Polling
- 13.6. Tracking Task State
- 13.7. The Suite Service Directory
- 13.8. File-Reading Commands
- 13.9. Client-Server Interaction
- 13.10. GUI-to-Suite Interaction
- 13.11. Remote Control
- 13.12. Scan And Gscan
- 13.13. Task States Explained
- 13.14. What The Suite Control GUI Shows
- 13.15. Network Connection Timeouts
- 13.16. Runahead Limiting
- 13.17. Limiting Activity With Internal Queues
- 13.18. Automatic Task Retry On Failure
- 13.19. Task Event Handling
- 13.20. Managing External Command Execution
- 13.21. Handling Job Preemption
- 13.22. Manual Task Triggering and Edit-Run
- 13.23. Cylc Broadcast
- 13.24. The Meaning And Use Of Initial Cycle Point
- 13.25. Simulating Suite Behaviour
- 13.26. Automated Reference Test Suites
- 13.27. Triggering Off Of Tasks In Other Suites
- 13.28. Suite Server Logs
- 13.29. Suite Run Databases
- 13.30. Disaster Recovery
- 13.31. Auto Stop-Restart
- 13.32. Alternate Suite Run Directories
- 13.33. Sub-Suites
- 14. Suite Storage, Discovery, Revision Control, and Deployment
- 15. Appendices
- 15.1. Suite.rc Reference
- 15.2. Global (Site, User) Config File Reference
- 15.2.1. Top Level Items
- 15.2.2. [task messaging]
- 15.2.3. [suite logging]
- 15.2.4. [documentation]
- 15.2.5. [document viewers]
- 15.2.6. [editors]
- 15.2.7. [communication]
- 15.2.8. [monitor]
- 15.2.9. [hosts]
- 15.2.10. [suite servers]
- 15.2.11. [suite host self-identification]
- 15.2.12. [task events]
- 15.2.13. [test battery]
- 15.2.14. [cylc]
- 15.2.15. [authentication]
- 15.3. Gcylc GUI (cylc gui) Config File Reference
- 15.4. Gscan GUI (cylc gscan) Config File Reference
- 15.5. Remote Job Management
- 15.6. Command Reference
- 15.7. The gcylc Graph View
- 15.8. Cylc README File
- 15.9. Cylc INSTALL File
- 15.10. Cylc Development History - Major Changes
- 15.11. Cylc 6 Migration Reference
- 15.12. Known Issues
- 15.13. GNU GENERAL PUBLIC LICENSE v3.0
- 16. Suite Design Guide
- 16.1. Introduction
- 16.2. Style Guidelines
- 16.3. Basic Principles
- 16.3.1. UTC Mode
- 16.3.2. Fine Or Coarse-Grained Suites
- 16.3.3. Monolithic Or Interdependent Suites
- 16.3.4. Self-Contained Suites
- 16.3.5. Task Host Selection
- 16.3.6. Task Scripting
- 16.3.7. Rose Apps
- 16.3.8. Rose Metadata Compliance
- 16.3.9. Task Independence
- 16.3.10. Clock-Triggered Tasks
- 16.3.11. Rose App File Polling
- 16.3.12. Task Execution Time Limits
- 16.3.13. Restricting Suite Activity
- 16.3.14. Suite Housekeeping
- 16.3.15. Complex Jinja2 Code
- 16.3.16. Shared Configuration
- 16.3.17. Automating Failure Recovery
- 16.3.18. Include Files
- 16.4. Efficiency And Maintainability
- 16.5. Portable Suites
- 16.5.1. The Jinja2 SITE Variable
- 16.5.2. Site Include-Files
- 16.5.3. Site-Specific Graphs
- 16.5.4. Inlined Site-Switching
- 16.5.5. Site-Specific Suite Variables
- 16.5.6. Site-Specific Optional Suite Configs
- 16.5.7. Site-Agnostic File Paths in App Configs
- 16.5.8. Site-Specific Optional App Configs
- 16.5.9. An Example
- 16.5.10. Collaborative Development Model
- 16.5.11. Research-To-Operations Transition
- 16.6. Roadmap