Main Loop Plugins

Plugins for running Python code inside of the Cylc scheduler.

Built In Plugins

Cylc Flow provides the following plugins:

`cylc.flow.main_loop.auto_restart`	Automatically restart workflows if they are running on bad servers.
`cylc.flow.main_loop.health_check`	Checks the integrity of the workflow run directory.
`cylc.flow.main_loop.log_data_store`	Log the number and size of each type of object in the data store.
`cylc.flow.main_loop.log_main_loop`	Main loop plugin for monitoring main loop plugins.
`cylc.flow.main_loop.log_memory`	Log the memory usage of a running scheduler over time.
`cylc.flow.main_loop.reset_bad_hosts`	Resets the list of bad hosts.

Configuring

Main loop plugins can be activated either by:

Using the --main-loop option with cylc play e.g:

$ # run a workflow using the "health check" and "auto restart" plugins:
$ cylc play my-workflow --main-loop 'health check' --main-loop 'auto restart'

Adding them to the default list of plugins in global.cylc[scheduler][main loop]plugins e.g:

[scheduler]
    [[main loop]]
        plugins = health check, auto restart

Main loop plugins can be individually configured in their global.cylc[scheduler][main loop][<plugin name>] section e.g:

[scheduler]
    [[main loop]]
        [[[health check]]]
            interval = PT5M  # perform check every 5 minutes

Developing Main Loop Plugins

Main loop plugins are Python modules containing asynchronous function(s) (sometimes referred to as coroutines) which Cylc Flow executes within the scheduler.

Hello World

Here is the “hello world” of main loop plugins:

my_plugin.py

from cylc.flow import LOG
from cylc.flow.main_loop import startup

@startup
async def my_startup_coroutine(schd, state):
   # write Hello <workflow name> to the Cylc log.
   LOG.info(f'Hello {schd.workflow}')

Plugins are registered by registering them with the cylc.main_loop entry point:

setup.py

# plugins must be properly installed, in-place PYTHONPATH meddling will
# not work.

from setuptools import setup

setup(
    name='my-plugin',
    version='1.0',
    py_modules=['my_plugin'],
    entry_points={
       # register this plugin with Cylc
       'cylc.main_loop': [
         # name = python.namespace.of.module
         'my_plugin=my_plugin.my_plugin'
       ]
    }
)

Examples

For examples see the built-in plugins in the cylc.flow.main_loop module which are registered in the Cylc Flow setup.cfg file.

Coroutines

Plugins provide asynchronous functions (coroutines) which Cylc will then run inside the scheduler.

Coroutines should be fast running (read as gentle on the scheduler) and perform IO asynchronously.

Coroutines shouldn’t meddle with the state of the scheduler and should be parallel-safe with other plugins.

Event Types

Coroutines must be decorated using one of the main loop decorators. The choice of decorator effects when the coroutine is called and what arguments are provided to it.

The available event types are:

cylc.flow.main_loop.startup(fcn)[source]

Decorates a coroutine which is run at workflow startup.

The decorated coroutine should have the signature:

async coroutine(scheduler, plugin_state) -> None

Exceptions:

Regular Exceptions are caught and logged.

Exceptions which subclass CylcError are re-raised as MainLoopPluginException

cylc.flow.main_loop.shutdown(fcn)[source]

Decorates a coroutine which is run at workflow shutdown.

Note shutdown refers to “clean” shutdown as opposed to workflow abort.

The decorated coroutine should have the signature:

async coroutine(scheduler, plugin_state) -> None

Exceptions:

Regular Exceptions are caught and logged.

Exceptions which subclass CylcError are re-raised as MainLoopPluginException

cylc.flow.main_loop.periodic(fcn)[source]

Decorates a coroutine which is run at a set interval.

The decorated coroutine should have the signature:

async coroutine(scheduler, plugin_state) -> None

Exceptions:

Regular Exceptions are caught and logged.

Exceptions which subclass CylcError are re-raised as MainLoopPluginException

Configuration:

The interval of execution can be altered using global.cylc[scheduler][main loop][<plugin name>]interval