Cylc Development Workshop 2020
10-14 Feb, NIWA, Wellington, New Zealand
Primarily: to review Cylc 8 progress and chart the roadmap to completion. Cylc
8 has a more complex architecture than Cylc 7 (and earlier versions) and it makes
use of a much wider variety of technologies and protocols. This is an
opportunity for us all to get a better understanding of the complete system as
well as plan next steps.
For each of the main topics below we will cover:
- What we are aiming for (and why)
- What we have achieved so far
- What is still to be done
- and any problems still to be solved
Participants should bring a notebook computer capable of installing and running
Cylc 8 if possible (Linux with admin access to install software as needed). If
not possible, your locked-down work laptop will do for viewing code and
documentation, and showing slides and documents as needed.
Those who have worked on aspects of the project already should be prepared to
explain and demo their parts of the system to others. And we should all be
prepared to ask questions until we understand the entire system.
(NOTE: we have been using several provisional names for the back-end
workflow manager before settling on “Cylc Scheduler” and the following docs
have not been updated yet … “suite daemon”, “suite server program”,
“workflow server”, “workflow service” are all the same thing.)
To squeeze in somewhere if needed and/or if time allows:
- BOM and NRL perspectives and priorities?
- Beyond Cylc 8
Monday: Intro, Architecture and Data Provision
- 9-10am Welcome and Introductions etc.
- Project status
- How we run the project
- Update on funding etc.
- Cylc-8 Architecture recap
- Components and code repositories
- Quick Cylc 8 DEMO
- Architectural issues that still need some thought:
- Remote spawning
- Scaling and load balancing
- CLI via the UI Server (e.g for task communication)
- Access to other-user workflows
- Single User usage
- Hub Sub-services: workflow start-up, Cylc review, tasks beyond the active window, other?
- (Cloud, containers?)
- Data provision to the UI (David S):
- Data stores and incremental update via ZMQ, Protobuf, GraphQL
- Practical debugging session:
- UI (browser and Vue tools), WFS (incl. subprocesses), UIS (incl. async
routines), traffic between components
Tuesday: UI plus Spawn-on-Demand and Friends
- Web UI overview:
- Design (latest mock-up at top of page)
- Task/job separation and implications
- Vue.js, common data store, subscriptions, GraphQL, websocket, etc.
- Mutations (commands) and API-on-the-fly
- Performance considerations
- Treeview status and plans
- Other views TBD: dot, graph, etc.
- how does the ideal graph view differ from our current POC?
- Whiteboard/inkscape session on remaining design issues:
- Gantt view
- selecting sharing (drag n drop?)
- representing xtriggers
- representing dependencies in non-graph views?
- CLI: entry points, new
cylc monitor, CLI simplification? (post
Tuesday or Wednesday?
- The future is sooner than you think
- Spawn-on-Demand: comparison with Spawn-on-Submit, implications
- Towards a fully event-driven scheduler (main loop asyncio…)
Wednesday: Configuring and Running Workflows
- finish morning discussions
- visit NIWA operations?
- visit NIWA HPC facility?
- (DM, HO, and JR side meeting?)
Thursday: Authentication, Authorization, Security
- Hub: authentication plugins; sessions; managing identity; which config options?
- Back-end connection authentication:
- Hub-UIS, UIS-WFS, CLI-WFS, CLI-UIS
- WFS and ZMQ auth files
- Single user issues, e.g. standalone (sans Hub) UIS? (like Jupyter notebook)
- BOM security concerns:
- review cylc-7 pen testing report
- review threat modeling notes
- All handled by the UIS (not WFS)?
- Can we settle on authorization levels and corresponding privileges?
- Configuration and implementation
Friday: Deployment and Documentation
- Hilary Oliver - NIWA, (Wellington, New Zealand) - firstname.lastname@example.org
- Bruno Kinoshita - NIWA, (Auckland, New Zealand - email@example.com
- David Sutherland - NIWA, (Wellington, New Zealand) - firstname.lastname@example.org
- Dave Matthews - IS-ENES3, Met Office, (Exeter, UK) - email@example.com
- Tim Pillinger - Met Office, (Exeter, UK) - firstname.lastname@example.org
- Mel Hall - Met Office, (Exeter, UK) - email@example.com
- Oliver Sanders - Met Office, (Exeter, UK) - firstname.lastname@example.org
- Jacinta Richardson - BoM, (Melbourne, Australia) - email@example.com
- Tim Whitcomb - NRL, (Monterey, USA) - firstname.lastname@example.org
Thanks to NIWA for hosting the workshop, and the UM Partnership and IS-ENES3
for sponsoring travel for relevant participants.
- WFS = Workflow Scheduler
- UIS = UI Server
- WUI = Web UI