Creating a release for Cylc

Before you start (First time)

Create pre-release Pull requests

Test the build

   cd my/repository/

   # This may seem slow as this will build the wheel as well as the source
   # distributions
   python3 bdist_wheel sdist

   # check that you've successfully created wheel and source files
   ls dist/

   # Tidy up
   rm -r dist

Test the docs

For any projects which are auto-documented by cylc-doc, currently:

Ensure the docs build against master by manually triggering the test workflow in cylc-doc.

This will catch syntax errors, broken urls etc which need to be fixed prior to releasing the project.

Tag the version

    # Examine previous tags using
    git tag -ln

    # Create a new tag  "tag_name" is usually a version number.
    git tag -a -s <tag_name>

    # Push tags:
    git push --tags upstream <tag_name>

Create a release on Github

Upload build to PyPI

    # Build your distribution inside the repository folder this time
    cd my/repository

    # Create your build
    python3 bdist_wheel sdist

    # Check that the dist folder contains the right artifacts:
    ls dist/

    # Upload your build to PyPI. n.b. This will not work if your build has the
    # same version number as one already on PyPI.
    twine upload dist/*

Check PyPI for your upload.

Upload to Conda

NOTE: the GitHub and PYPI releases are prerequisites for releasing to Conda Forge!

The Conda Forge release process is mostly automated by bots. These bots are set up during the initial project bootstrap, which has been done for all the projects that are part of Cylc 8.

As there are inter-dependencies amongst the Cylc 8 parts, you should know the dependency tree, and start by the leaf nodes (i.e. a module A may have multiple dependants, but no dependency to other modules).

For example, Cylc UI Server depends on Cylc Flow. So unless you are releasing only Cylc UI Server, you should release Cylc Flow first.

Prepare the Conda Releases

On GitHub, navigate to the project repository on GitHub, e.g.

On each of these repositories, the release process should be pretty much the same. Your work will be mainly (if not exclusive) on the recipe/meta.yaml file.

This is a possible order of action:

From past experience, I’ve seen Conda packages available in Conda Forge in under 30 minutes, but have also had to wait hours (<8 hours) for it to show in Conda Forge. For example, if Azure or GitHub infrastructure have issues (as I experienced during the first releases) it may take days. So keep that in mind before announcing the Conda packages.

Testing the Packages

To test packages locally, first you should make sure that Conda is configured to avoid automatically upload the package. Open your ~/.condarc and check that you have something similar to:

     - defaults
     - conda-forge
   ssl_verify: true
   anaconda_upload: false

Now create a Conda environment for your tests, e.g.: conda create -n cylc1, and then activate it conda activate cylc1. Then to build and install locally:

   # Where $CONDA_FORGE_REPOSITORY could be, for example,
   # cylc-uiserver-feedstock.
   # Your package should not be listed!
   conda list
   # This will take some minutes and print useful information.
   conda build recipe/
   # The following command will install the locally created package. Before
   # installing it will ask you to confirm. Scroll up and search the
   # package name. The right-side column must show a location like
   # .../anaconda3/conda-bld/linux-64::cylc-uiserver-0.1-py37_1.
   # This confirms you are installing the local build. Here $PACKAGE_NAME
   # could be something like cylc-uiserver.
   conda install --use-local $PACKAGE_NAME

At this point you should be good to go. Test the package with commands such as which $PACKAGE_NAME, or $PACKAGE_NAME --version, etc. Of if you are testing the metapackage, try running the complete system with a workflow and the UI or tui, and check if there is nothing wrong.

Troubleshooting Conda Forge issues

Issues building or installing the package locally with conda-build and `conda install

If you found problems while testing locally, try troubleshooting locally, and either mark the pull request as draft, and close it. Merging the pull request will create a release.

Marking a release as invalid or broken

To undo a release, you will have to liaise with the maintainers via Gitter or Github. Or, alternatively, bump up the build number +1, and release it again.

conda-install is picking the wrong version or build of a package

If conda install cylc returns the incorrect build number, it could be an issue that is very difficult to debug (a posteriori). In fact it may not even be a bug:

Question to conda Gitter channel: (@hjoliver Jul 21 17:23):

I have a conda metapackage just published to conda-forge, which (on conda install) does not install the latest build numbers of several of our component packages. The conda docs cite bug fixes as a primary use case for incrementing build numbers, in which case I would have thought the environment solver should never select older builds? If I install the component packages directly, I do get the latest builds, but not via the metapackage. Anyone know what’s going on here?


@hjoliver this article describes how the conda solver works: It specifically says that build numbers are only maximized for explicit specs

@jjhelmus: @hjoliver changes to the build number can also occur when a package is built against a different version of a library which is incompatible with earlier version. For example, one build of python 3.7.7 might be compiled against libffi 3.2 while the next build number against libffi 3.3. Depending on what other packages are installed in the environment it may not be possible to install the package with the largest built number. In the example if cffi is only available as a package built against libffi 3.2 then the older build of python 3.7.7 would be required.

Interpretation: build number is treated much like version number, if not pinned down exactly conda can choose an older build if that works best for solving all the dependencies in the system. Conda would presumably(?) choose the newest build if it does not affect dependencies; we may just need to check what happens if dependencies get changed by a new build.

If you are sure that you have your new version or build available from, and that conda install --dry-run $package should work, then you could have hit a bug in either conda client or in Conda Forge repository data.

You can try the following:

Conda documentation has details about their package math specification. Basically, there are three parts, where the last one is optional.

So you will have to specify the exact version, and use a glob expression for the build number. For example, from our alpha2 release:

  - python
  - cylc-flow 8.0a2 *_3
  - cylc-uiserver 0.2 *_2

In the example above, we are specifying the packages and versions we want. Plus, we also give it a pattern to the build string, so conda will match cylc-flow-8.0a2-py37_2, or ``cylc-flow-8.0a2-py38_2, or cylc-flow-8.0a2-py37h2387c_2` (all valid build strings).

Announce on Element and Discourse?

Bask in the glory of having created a release.