David Sutherland, NIWA

What's Important?

  • Workflow state.
  • Job config, state, and messages.
  • Hierarchy data (tree view):
    • Child/parent Node relationships.
  • Graph data:
    • Nodes & Edges.

What data?

  1. In memory
    • Parsed workflow config/definition (cylc-flow.conf).
    • Task handlers/proxies/instances (Task Pool).
    • Job config/file creation & message handling.

  2. File System
    • Workflow sqlite DB.
    • Contact file (Cylc/API verison, ports ...etc).
    • Job logs.
    • Contact file.
    • Security keys.


Overview & considerations

Workflow Scheduler (WFS) Interaction paths

  • Directly with WFS (CLI, Jobs ..etc)
  • UI Server (UIS) CLI.
  • Web UI via hub proxy and UIS.

Desirables & Consequences

  • One WUI per user (instead of per workflow with Cylc7).
  • Unified CLI across alternate paths.
  • Data efficiency of WSF <=> UIS.
  • Isolate load on WFS by Web UI.
  • Scalability; Number & size of WFS.



(It's better than REST!)

  • Allows the client to determine the data's structure and content.
  • Agnostic to transport protocol.
  • Strongly typed schema
  • Subscriptions via existing schema object types.
  • Introspection (api-on-the-fly)
  • Many supported libraries & active community


  • Fine-grained authorisation.
  • Unified CLI with identical schema at both WFS & UIS.
  • ZeroMQ has no GraphQL packages, so we need to create these tools.



  • GraphQL requires backing data.
  • In memory mapping (via Id) provides fast retreival for query resolving.

Replication/Sync of data-store at UIS

  • With unified CLI & the same GraphQL schema, input data needs to be identical.
  • Isolates WUI load from WFS.
  • Facilitates efficient cross/multi workflow queries.
  • Size is no barrier to scalability (~1Mb for large workflows).


  • Efficient transport format.
  • GraphQL can resolve directly off Protobuf messages.
  • Type checking.
  • GraphQL input != GraphQL output.

Progress Report


  • GraphQL at UIS and WFS.
  • Data-store sync.
  • Queries.
  • Mutations (api-on-the-fly! thanks Oliver).
  • N-distance Nodes-Edges query (follows edges).
  • Tornado web-sockets GraphQL subscriptions (thanks Bruno!).


  • UIS delta subscriptions:

To Do:

  • CLI migration to GraphQL.
  • ZeroMQ GraphQL subscription module.
  • Authorisation middleware.


To Discuss...