Data Flow

This section looks at how Cylc synchronises data between its different components, the transports, security and formats used to do so.

Protocols, Transport & Security

Cylc uses the following schemes for data and control:

ZMQ (Over TCP)

Cylc uses ZMQ over TCP for most server side interaction.

All ZMQ connections are secured by CurveZMQ which uses security keys which are stored on the filesystem with appropriate permissions.

Where remote communication is involved Cylc synchronises the relevant keys to the appropriate platforms.

For more information on the key files see Authentication Files.

SSH

SSH connections are used for some purposes such as installing the key files required for ZMQ (Over TCP) transport.

HTTPS (Hypertext Transfer Protocol Secure)

HTTPS connections are used in the browser-based Cylc UI. The certificate used must be configured with the Cylc UI Server / Jupyter Server.

WSS (WebSocket Secure)

Websocket connections are used in the browser-based Cylc UI. These connections are upgraded from HTTPS connections and share the same security.

Formats

Cylc uses the following transports to communicate over these protocols.

Protobuf

Protobuf is both a storage utility and transport.

Protobuf is used for internal data stores and the synchronisation of those data stores.

GraphQL

GraphQL is a query language which is intended to be the user-facing data and control interface for Cylc workflows.

It contains both data and mutations (actions) defined by a self-documenting schema.

The GraphQL servers get their data from Protobuf data stores.

Component Interactions

Here are some additional details on how the components interact.

Job -> Scheduler

Default Protocol:

ZMQ (Over TCP)

Format:

GraphQL

Jobs can communicate status updates back to the Scheduler using different methods. The method used is configured on a per-platform basis by the global.cylc[platforms][<platform name>]communication method.

Client -> Scheduler

Protocol:

ZMQ (Over TCP)

Format:

Protobuf

The subcommands in the Cylc command line interface map onto GraphQL mutations.

Mutations are issued through ZMQ connections.

Scheduler -> UI Server

Protocol:

ZMQ (Over TCP)

Formats:

Protobuf (data) and GraphQL (control)

The Scheduler maintains an in-memory Protobuf data store which is backed up by a SQLite3 database.

The database provides crash resilience and restart capability.

The UI Server also maintains an in-memory Protobuf data store containing relevant data for the workflows it is actively monitoring.

The synchronisation of the Scheduler and UI Server data stores is done using one ZMQ (Over TCP) connection per Scheduler.

The Schedulers that the UI Server connects to are determined by the active subscriptions registered. If there are no active subscriptions the UI Server will have no active Scheduler connections.

UI Server -> UI

Protocols:

WSS (WebSocket Secure), HTTPS (Hypertext Transfer Protocol Secure)

Format:

GraphQL

Most UI functionality involves subscribing to “delta” updates. For these subscriptions the UI Server sends only the added/removed/updated data (a delta) to the UI enabling it to update its internal data store.

The UI maintains a flat “lookup” which contains all objects in the store indexed by their ID. It also maintains a “tree” which contains references to the data in the “lookup” (but does not duplicate it) which it holds in a hierarchical structure more suitable for presentation purposes.

The Cylc Web UI uses Apollo Client to handle GraphQL requests. It will have one WebSocket per user session.

Every message received by the server is added to a queue, and processed by the server as soon as possible.

It uses the Cylc UI Server schema and resolvers to validate the query and to fetch data from the data store for the query response.

The query result is then serialized as JSON and sent back to the client. The work of the Apollo Client ends after it pushes the data to the Vuex store.

The communication between client and server follows a protocol called graphql-ws protocol.

After a channel between client and server is open, the messages follow that protocol, starting by the connection init message, that simply expects an ack message back from the server, where the ack is an acknowledgement to the client - note that the protocol does not define an ack as a MUST, but rather as a MAY, so a client may interpret not receiving an error as an acknowledgement to proceed as well.

The next message will be a start, which will contain the GraphQL query subscription. If there were no errors, the client and server subscription is established, and the client will start receiving the GraphQL responses.

The protocol also supports other messages, such as stop, to tell the server it doesn’t need to send any more data as that subscription is now cancelled.