Hilary Oliver, January 2019
In December 2018 the Australian Bureau of Meteorology (BOM) hosted the first-ever Cylc development workshop, with participants from NIWA (3), Met Office (3), ESiWACE (1), BOM (1), and Altair Engineering (1). Travel funding was generously provided by the Unified Model Partnership, ESiWACE, and Altair.
The primary goal of the workshop was to plan how to port Cylc from Python 2 to Python 3, with the complication that the current Cylc (and Rose) GUIs need to be completely replaced because they are based on a now-obsolete framework (PyGTK) that will never be ported to Python 3. This has to be done with some urgency due to the impending demise of Python 2, which will officially be retired at the end of 2019. The week in Melbourne also provided an ideal opportunity to introduce new team members, and to meet the BOM and Altair staff building NWP production infrastructure around PBS Pro, Cylc, and Kafka, and to learn about that project.
The workshop was a great success. We settled (with caveats) on communication and working practices for the widely distributed team, and worked through a porting plan and a radically new Cylc architecture to support a modern web UI, with site identity management integration and a single point of access for users.
See below for more information on background, the workshop agenda, and outcomes including a link to Cylc-8 architecture documentation.
The official Python 2 end-of-life date is 31 December 2019. After that the language will no longer be developed or maintained, not even for security patches. Even worse, the current Cylc GUIs, which embody several developer-years of effort, are built on the obsolete PyGTK framework which will never be ported to Python 3. Major Linux distributions have already announced they will soon be shipping without Python 2 and PyGTK. Cylc is still Python 2-based as we have not previously had the resources to address these major issues.
So there is an urgent need to migrate Cylc to Python 3 and entirely replace the current GUIs with something more modern, which by general agreement should be a web UI. This is a challenging task that requires new system components and a complete rework of Cylc’s local client-server architecture, which is not web-compatible (current clients make use of the local system in ways that browsers do not allow, for instance). And the first major release of the new system, Cylc-8, must be performant “out of the box” for existing critical production suites.
The primary workshop goal was (therefore) to:
Toward this end, sub-goals were to:
The WORKSHOP AGENDA shows the topics covered each day during the week, to support these goals.
Technical working practices centered on git and GitHub were discussed, but already well-established in the project. With new team members and people located all over the world, however, we agreed on the need for regular video conferences (monthly); instant communication via a chat platform of some kind; possibly a replacement for our old Google Groups email forum (mainly used for release announcements and infrequent questions from remote users); and something like Trello boards to help with project management as well. Unfortunately it is not so easy to solve these problems because our respective institutions all use mutually incompatible video conferencing technologies and have wildly different restrictions on what can be used on site. Popular platforms such as Slack also place significant restrictions on free accounts (such as loss of history beyond some point) or charge a lot for full functionality.
We decided to:
(Since the workshop one of our major sites has decided to block Riot.im for security reasons, forcing some of us to use personal devices to stay in touch … so effective communication remains challenging).
The new architecture is written up in some detail, and illustrated, in a separate report: CYLC-8 ARCHITECTURE.
The implementation plan can be summarized as follows.
Then begin Cylc-8 development:
pip install cylcetc.)
Concurrent with Python 3 porting:
Thanks to BOM for hosting the workshop; and the UM Partnership, ESiWACE, and Altair for providing travel funding.