Intro

David Sutherland, NIWA

What's Important?

• Workflow state.
• Job config, state, and messages.
• Hierarchy data (tree view):
• Child/parent Node relationships.
• Graph data:
• Nodes & Edges.

What data?

1. In memory
   • Parsed workflow config/definition (cylc-flow.conf).
   • Task handlers/proxies/instances (Task Pool).
   • Job config/file creation & message handling.


2. File System
   • Workflow sqlite DB.
   • Contact file (Cylc/API verison, ports ...etc).
   • Job logs.
   • Contact file.
   • Security keys.

Architecture

Overview & considerations

Workflow Scheduler (WFS) Interaction paths

• Directly with WFS (CLI, Jobs ..etc)
• UI Server (UIS) CLI.
• Web UI via hub proxy and UIS.

Desirables & Consequences

• One WUI per user (instead of per workflow with Cylc7).
• Unified CLI across alternate paths.
• Data efficiency of WSF <=> UIS.
• Isolate load on WFS by Web UI.
• Scalability; Number & size of WFS.

GraphQL

Why?

(It's better than REST!)

• Allows the client to determine the data's structure and content.
• Agnostic to transport protocol.
• Strongly typed schema
• Subscriptions via existing schema object types.
• Introspection (api-on-the-fly)
• Many supported libraries & active community

Consequences

• Fine-grained authorisation.
• Unified CLI with identical schema at both WFS & UIS.
• ZeroMQ has no GraphQL packages, so we need to create these tools.

Data-store

Why?

• GraphQL requires backing data.
• In memory mapping (via Id) provides fast retreival for query resolving.

Replication/Sync of data-store at UIS

• With unified CLI & the same GraphQL schema, input data needs to be identical.
• Isolates WUI load from WFS.
• Facilitates efficient cross/multi workflow queries.
• Size is no barrier to scalability (~1Mb for large workflows).

Protobuf

• Efficient transport format.
• GraphQL can resolve directly off Protobuf messages.
• Type checking.
• GraphQL input != GraphQL output.

Progress Report

Done:

• GraphQL at UIS and WFS.
• Data-store sync.
• Queries.
• Mutations (api-on-the-fly! thanks Oliver).
• N-distance Nodes-Edges query (follows edges).
• Tornado web-sockets GraphQL subscriptions (thanks Bruno!).

Underway:

• UIS delta subscriptions:
• https://github.com/cylc/cylc-flow/pull/3500#
• https://github.com/cylc/cylc-uiserver/pull/118#

To Do:

• CLI migration to GraphQL.
• ZeroMQ GraphQL subscription module.
• Authorisation middleware.

Future

To Discuss...