cylc-admin

Cylc-8 Architecture

Last Updated: November 2019.

Author: Hilary Oliver.

Contributors: Dave Matthews, Matt Shin, Oliver Sanders, Sadie Bartholomew, Martin Ryan, Bruno Kinoshita, David Sutherland, Sujata Patnaik.

This document is a primary output of the 3-7 December 2018 Cylc Development Workshop at the Bureau of Meteorology, Melbourne, Australia.

Table of Contents

TOP

Appendices:

Background

TOP

(A Cylc workflow is a single (possibly cycling) suite of inter-dependent tasks).

(A Cylc workflow service is workflow manager program for a single workflow, formerly known as a suite server program or a suite daemon. (Cylc has no central server - each workflow gets its own ad-hoc service that runs as the user)).

Cylc-7 is written in Python 2, with PyGTK desktop GUIs and relatively simple local client/server architecture in which everything runs as the user, all clients are treated equally (user GUI and CLI, and job CLI), clients get some server information via the filesystem and port scanning, and automatic owner-only authentication via a suite-specific passphrase file. Unfortunately:

We have decided not to port the existing Cylc GUIs to PyGObject (the successor to PyGTK) because there is strong demand for a new architecture that supports an in-browser web GUI and integration with site identity management systems. The full web architecture is (necessarily) more complicated, but it is more powerful. It will enable us to:

  1. Provide a single point of access to many Cylc workflows on a pool of servers.
  2. Run and interact with workflows via a web browser, without requiring:
    • a Cylc installation on the front-end (browser) platform
    • a shared filesystem or SSH access between the front-end and workflow platforms
  3. Drop the requirement for port scanning by users.
  4. Retire the suite-specific passphrase files and self-signed SSL certificates.
  5. Integrate with site identity management.
  6. Support fine-grained authorized access to individual Workflow Services.

Cylc-8 Architecture Diagram

TOP

Cylc-8 Architecture

Figure 1 Cylc-8 Architecture: The “user A” box represents processes owned by one user, but potentially spread over multiple workflow hosts (on a shared filesystem) and multiple job hosts. The term “HPC Platform” is used rather loosely - potentially only the jobs reside on actual HPC nodes. Yellow boxes show the technologies and protocols that will be used to implement each component.

Cylc Hub

TOP

The Hub and Proxy architecture described below is inspired by JupyterHub. JupyterHub is a proven technology that solves a very similar problem of managing back-end services spawned into user accounts. And it is commonly used in scientific modeling and HPC contexts, See below for details: Similarity with Jupyter Hub).

We hope to use JupyterHub “out of the box” for the Hub and Proxy components of the new archtitecture. Our back-end components are very different from Jupyter Notebook, but some of the technologies involved remain relevant.

Overview:

Detail:

For more detail on component interaction, including session management, see JupyterHub Technical Overview.

Cylc UI Server

TOP

Overview:

Detail:

Cylc UI

TOP

Cylc Workflow Services

TOP

Largely unchanged from Cylc-7 “suite server programs”, except:

Command Line Interface

TOP

Authorization

TOP

Appendix

TOP

Deployment

Similarity with JupyterHub

TOP

Jupyter Notebooks are a proven technology commonly used in scientific and educational institutions worldwide for interactive programming and sharing web documents that contain embedded code. Architecturally, the user’s browser talks to a Python Tornado-based Notebook Server that communicates via the ZeroMQ network library with a back-end Notebook Kernel that executes the code. Both the notebook server and kernel run as the user.

JupyterHub is used to deploy and manage single-user notebooks for large numbers of users within an institution. It consists of a privileged multi-user Hub (a Python Tornado process) that:

JupyterHub is extremely well documented, including (for example) a security overview of the architecture.

This architecture is (almost) exactly what we need for Cylc-8, with:

For an in-browser GUI to spawn and access a distributed set of Cylc workflow services running under various user accounts, we need a central “hub” that can spawn user processes and proxy requests to them, and a “UI Server” component to construct the UI (HTML/Javascript) around workflow status data obtained from the workflow services (and also static data about stopped workflows).

JupyterHub is Open Source, with the 3-Clause BSD License)

Differences from Jupyter Notebook

The Jupyter Notebook back-end is highly specific to the Notebook Document format. For Cylc, we have Workflow Services instead of Kernels, and we need a Cylc-specific UI Server (with a bunch of Cylc-specific sub-services for suite discovery and start-up etc.) instead of the Notebook Server. Further, Jupyter Notebook Servers fronts a single Kernel, whereas a Cylc UI Server may need to access (and possibly collate) data from multiple workflow services.

However, we do share the two-level “web server / kernel” back-end model, and we have therefore decided to use ZeroMQ for back-end communications. It is a well-used, efficient network library, and we may be able to learn by studying the Jupyter model - including the automatic trust (server authentication) mechanism.

Differences from JupyterHub

There is only one difference that we’re aware of:

However, multi-user access is presumably as simple as allowing access to /user/{other-name}/ URLs (in the Jupyter case the show-stopper is that the Notebooks themselves, at the back end, don’t support shared use yet).

We therefore hope we can use JupyterHub “out of the box” for the Hub and Proxy components of the Cylc-8 architecture. The only question is, can the (minor?) multi-user access change be done (e.g. by plugin) without modifying the core of JupyterHub so that we can treat it as a third-party software requirement, or can we contribute a change back to JupyterHub to enable that, or do we need to fork the project and maintain our own “Cylc Hub” in the future?

Technology Glossary

TOP

(Hyperlinks in the text below point here for further information).