Cylc Workshop 2020 Logos

CylcCon 2020 Report

Cylc Development Workshop 10-14 Feb, NIWA, Wellington, New Zealand

Hilary Oliver, NIWA

Participants attended from NIWA, Met Office, BOM, and NRL.

Acknowledgement: Thanks to NIWA for hosting, and the UM Partnership and IS-ENES3 for funding travel for some participants.

Table of Contents

Goal and topics

The primary goal of the workshop was to review Cylc 8 progress and chart the roadmap to completion by early 2021.

For specific discussion topics covered during the workshop and links to relevant background material see the Workshop Agenda. For additional detail see workshop notes by HO and JR.

Executive summary

Python 2 end of life, obsoletion of the PyGTK GUI toolkit, and a near-universal modern preference for web UIs over traditional desktop GUIs, has led to an urgent need to re-engineer Cylc to future proof our investment in workflow orchestration.

The new Cylc 8 architecture was agreed during the December 2018 Cylc workshop at BOM. Much has been achieved since then despite some unexpected resourcing issues (staff turnover etc.). At this point the back-end workflow engine and CLI have been migrated to Python 3, and the major components Cylc 8 and the connections between them are all in place and working. It is already possible to run workflows of moderate size with Cylc 8 and view and control them in the new web UI. For details, see Progress to date below.

However, there is still much to do before Cylc 8 is production-ready. Every component needs detailing, extensive testing, and performance optimisation; and some important sub-systems are still missing. For example, the primary tree workflow status view does not yet perform well enough for large workflows; control functionality is not integrated with the view components; we don’t have efficient incremental data updates flowing through to the UI; some aspects of security have not been addressed; and we haven’t implemented fine-grained authorization yet. For details of what remains to be done, see Prioritization of remaining tasks.

We are aiming to have a major preview release of Cylc 8 ready for widespread testing in September of this year, and to officially release cylc-8.0 in the first quarter of 2021.

Progress to date

(First read the Executive summary.)

A recent screenshot of the new web UI showing live workflow data: Cylc 8 web UI as of February 2020

As of February 2020:

Prioritization of remaining tasks

(First read the Executive summary.)

The following items were thoroughly discussed, and problems worked out, in the workshop. Now they just need to be implemented or completed.

Essential for Cylc 8.0

Nice to have for Cylc 8.0

These can be bumped to 8.1 if necessary, but we’ll do as much as we can to get them done before then:

Cylc 8.1

Tentative development timeline

The following are endpoints to aim for, for the major tasks. (Smaller tasks above are not listed here as they can be fitted in around the bigger ones as time allows). Many of these items are already in progress or have at least been planned in advance. Initials have been given for those leading the development, but others may also be involved, and we have to be flexible depending on availability.

Cylc Workshop 2020 Logos