cylc-admin
Cylc Development Workshop
Bureau of Meteorology, Melbourne, Australia, 3-7 December 2018
Table of Contents
Context
Over the next 12 months we need to migrate Cylc to Python 3, and replace the aging PyGTK GUIs and simple client/server architecture with a more complex (but more powerful) architecture and a web GUI. And the new system has to be performant “out of the box” for existing critical production use. This is a challenging task!!!
Format
Mainly discussion sessions, with some demoing and coding (to the extent possible at this early stage of development, and with our varying levels of experience)- not formal presentations (except perhaps Wednesday afternoon).
Preparation
Read and understand (as far as possible at this stage):
- Cylc-7 Architecture
- GUI Replacement Options
- Cylc-8 Architecture and Web GUI Roadmap
- those of us who have researched particular aspects in depth should be prepared to talk about it
- A few Cylc Issues that are relevant:
- Jupyter Hub Architecture - we think we need something very much like this (with notebook = suite daemon)
- Consider what you might like to work on, and how much effort you can contribute in the next year or so.
And for the new people:
- Understand how to define, run, and interact with basic Cylc workflows, non-cycling and cycling
- Work through the basic tutorial in the Cylc User Guide and the better Cylc tutorial in the newly-rewritten Rose User Guide
What to bring
- Your development laptop, with a Cylc repository clone on board and tested
- Working demos or other material to present, if you have any
Goals
By the end of the week we need to understand:
- How we will work together to develop the new system
- in principle we have a bigger team than ever before, and we should be able to make great progress
- Who will do what?
- Where technological or architectural choices have to be made,
- which way will we go?
- or (at the least) how to decide very quickly which way to go
- How to develop and test the different components separately, given their interdependence
- When can intermediate milestones be achieved along the way?
- (Is the team big enough now that we can start on some important “future Cylc” work concurrently with GUI development? - see Possible Cylc Futures and Vision for Cylc beyond 2018/2019 Priorities)
AGENDA
Items listed below are to guide the discussion, but we can be flexible if needed. Several of us are relatively expert on selected topics already and can start by walking the rest of us through it. Tentative decisions have already been made in a few areas (e.g. to use Vue.js for the GUI). We can consider coding and demoing at any point if time allows.
Main discussion points are noted below. For the technical topics see Cylc 8 Roadmap - and the other references above - for more detail.
Monday
- Morning
- (Combined session with the BoM/Altair Control Panel Stream)
- Introductions, with interpretive dance
- Overview and showcase of work and plans on the BoM/Altair/Cylc project (Control Panel, Apache Kafka, reporting DB, authentication etc.)
- Then: begin afternoon topics, if possible
- Afternoon:
- (development working practices - we have a bunch of new team members!)
- git, GitHub, GitHub Flow, testing, Travis CI, Codacy
- 2FA about to be enforced on cylc repository
- Riot.im? (UK team now has access); or Slack?; and/or Discourse?
- Regular video conferences, e.g. monthly?
- project governance
- how (and by whom) decisions will be made
- how much time can newcomers donate, and what determines what they work on?
- Python code style guidelines (add to CONTRIBUTING.md)?
- Contributing guidelines: non-trivial features (and implementation) should be discussed and agreed in advance!
- Development roadmap: what’s been discussed and decided so far?
- Then: begin Architecture discussion, if possible
Tuesday
- All day:
- (Architecture) - see Cylc 8 Roadmap
- components: GUI, “hub”, proxy, sub-services for suite discovery and suite start-up etc., GUI server, “state server”, …?
- How closely can we follow/steal from Jupyter Hub? see
- What runs privileged, and as the user?
- Server-side framework and API: flask, tornado, other?
- Inter-component communication
- WebSocket and GraphQL API, rather than HTTPS and REST API?
- Demo: GraphQL with cylc-7 (David)
- Simplified architecture for individual use?
- or just run all components as the user?
Wednesday
- Morning
- (Combined session with the BoM/Altair Control Panel Stream)
- Software testing, packaging and distribution
- Scheduler prediction tool
- Cylc packaging via
setup.py - (Begin afternoon topics, if possible)
- Afternoon
- (BoM Cylc user feedback and discussion - if needed)
- Deficiencies, feature requests, problem solving
- A presentation or two, if called for
- e.g. the Brussels Workflow Workshop Cylc and Rose keynote presentation?
- Demo: Cylc with Docker (Bruno)
Thursday
- Morning
- (Authentication and authorization)
- Follow JupyterHub?
- Site integration (plugins?)
- Session management (and for CLI commands?)
- Automatic job authentication: one-time tokens?
- Authentication by the Hub or Proxy, then how do suite daemons trust the Hub?
- SSL client certificate? (if not using HTTPS?)
- Authorization - how to do it?
- Afternoon
- Begin the Web GUI discussion (see Friday morning)
Friday
- Morning
- (Web GUI)
- Framework: Vue, or React, or Angular?
- Provide a small number of built-in themes? or give users control over the look?
- Accessibility
- UI Design
- Do we need (/can we get?) professional UI design advice?
- How to display very large suites effectively and efficiently
- Interaction with other components, incremental update, WebSocket (no polling), GraphQL
- Who serves the actual HTML pages (as opposed to just data)?
- suite daemon, or GUI server? (covered in “Architecture” above)
- Automated UI testing
- Afternoon
- (Tie it up and nail it down, with reference to the Workshop Goals)
- Retro
- Review of risks
- Delivery timeline
- Who can work on what?
Participants
- Hilary Oliver - NIWA, (Wellington, New Zealand) - hilary.oliver@niwa.co.nz
- Bruno Kinoshita - NIWA, (Auckland, New Zealand - bruno.kinoshita@niwa.co.nz
- David Sutherland - NIWA, (Wellington, New Zealand) - david.sutherland@niwa.co.nz
- Dave Matthews - ESiWACE, Met Office, (Exeter, UK) - david.matthews@metoffice.gov.uk
- Sadie Bartholomew - Met Office, (Exeter, UK) - sadie.bartholomew@metoffice.gov.uk
- Matt Shin - Met Office, (Exeter, UK) - matthew.shin@metoffice.gov.uk
- Oliver Sanders - Met Office, (Exeter, UK) - oliver.sanders@metoffice.gov.uk
- Martin Ryan - BoM, (Melbourne, Australia) - martin.ryan@bom.gov.au
- Sujata Patnaik - Altair, (Bangalore, India) - sujata.patnaik@altair.com
Acknowledgements
Thanks to BoM, Altair, the UM Partnership, and ESiWACE for sponsoring this workshop, by providing a venue, and/or travel funding, and/or development resourcing!