This section looks at how Cylc synchronises data between its different components, the transports, security and formats used to do so.
Protocols, Transport & Security
Cylc uses the following schemes for data and control:
ZMQ (Over TCP)
Cylc uses ZMQ over TCP for most server side interaction.
All ZMQ connections are secured by CurveZMQ which uses security keys which are stored on the filesystem with appropriate permissions.
Where remote communication is involved Cylc synchronises the relevant keys to the appropriate platforms.
For more information on the key files see Authentication Files.
SSH connections are used for some purposes such as installing the key files required for ZMQ (Over TCP) transport.
HTTPS (Hypertext Transfer Protocol Secure)
HTTPS connections are used in the browser-based Cylc UI. The certificate used must be configured with the Cylc UI Server / Jupyter Server.
WSS (WebSocket Secure)
Websocket connections are used in the browser-based Cylc UI. These connections are upgraded from HTTPS connections and share the same security.
Cylc uses the following transports to communicate over these protocols.
Protobuf is both a storage utility and transport.
Protobuf is used for internal data stores and the synchronisation of those data stores.
GraphQL is a query language which is intended to be the user-facing data and control interface for Cylc workflows.
It contains both data and mutations (actions) defined by a self-documenting schema.
The GraphQL servers get their data from Protobuf data stores.
Here are some additional details on how the components interact.
Job -> Scheduler
- Default Protocol:
Jobs can communicate status updates back to the Scheduler using different
methods. The method used is configured on a per-platform basis by the
global.cylc[platforms][<platform name>]communication method.
Client -> Scheduler
The subcommands in the Cylc command line interface map onto GraphQL mutations.
Mutations are issued through ZMQ connections.
Scheduler -> UI Server
The Scheduler maintains an in-memory Protobuf data store which is backed up by a SQLite3 database.
The database provides crash resilience and restart capability.
The UI Server also maintains an in-memory Protobuf data store containing relevant data for the workflows it is actively monitoring.
The synchronisation of the Scheduler and UI Server data stores is done using one ZMQ (Over TCP) connection per Scheduler.
The Schedulers that the UI Server connects to are determined by the active subscriptions registered. If there are no active subscriptions the UI Server will have no active Scheduler connections.
UI Server -> UI
WSS (WebSocket Secure), HTTPS (Hypertext Transfer Protocol Secure)
Most UI functionality involves subscribing to “delta” updates. For these subscriptions the UI Server sends only the added/removed/updated data (a delta) to the UI enabling it to update its internal data store.
The UI maintains a flat “lookup” which contains all objects in the store indexed by their ID. It also maintains a “tree” which contains references to the data in the “lookup” (but does not duplicate it) which it holds in a hierarchical structure more suitable for presentation purposes.
The Cylc Web UI uses Apollo Client to handle GraphQL requests. It will have one WebSocket per user session.
Every message received by the server is added to a queue, and processed by the server as soon as possible.
It uses the Cylc UI Server schema and resolvers to validate the query and to fetch data from the data store for the query response.
The query result is then serialized as JSON and sent back to the client. The work of the Apollo Client ends after it pushes the data to the Vuex store.
The communication between client and server follows a protocol called graphql-ws protocol.
After a channel between client and server is open, the messages
follow that protocol, starting by the
connection init message,
that simply expects an
ack message back from the server,
ack is an acknowledgement to the client - note
that the protocol does not define an
ack as a MUST, but
rather as a MAY, so a client may interpret not receiving an
error as an acknowledgement to proceed as well.
The next message will be a
start, which will contain the
GraphQL query subscription. If there were no errors, the client and
server subscription is established, and the client will start
receiving the GraphQL responses.
The protocol also supports other messages, such as
tell the server it doesn’t need to send any more data as that
subscription is now cancelled.