cylc-admin

Creating a release for Cylc

Note: The release process has been substantially automated. For reference, the old release process is documented at create-a-release-old.md.

Warning: The release process is still in flux, these instructions will drift from ground truth.

Release Process

Create a new release issue on this repository, follow the instructions there.

Before you start (First time)

Test the docs

For any projects which are auto-documented by cylc-doc, currently:

Ensure the docs build against master by manually triggering the test workflow in cylc-doc.

This will catch syntax errors, broken URLs etc which need to be fixed prior to releasing the project.

Stage 1: trigger the GitHub Actions workflow

Stage 2: merge the release PR

Notes

It is also possible to manually create a release PR with the conditions:

If anything goes wrong, you can also do the whole process manually by following the old instructions.

Stage 3: release on Conda

Note: The GitHub and PYPI releases are prerequisites for releasing to Conda Forge!

Note: The Conda Forge release process is mostly automated by bots. These bots are set up during the initial project bootstrap, which has been done for all the projects that are part of Cylc 8.

After a Python project has been pushed to PyPi a new PR should be automatically created on the conda-forge feedstock. If not you can DIY, you will need to update the checksum to the value on PyPi.

As there are inter-dependencies amongst the Cylc 8 parts, you should know the dependency tree, and start by the leaf nodes (i.e. a module A may have multiple dependants, but no dependency to other modules).

For example, Cylc UI Server depends on Cylc Flow. So unless you are releasing only Cylc UI Server, you should release Cylc Flow first.

Prepare the Conda Releases

On GitHub, navigate to the project repository on GitHub, e.g.

On each of these repositories, the release process should be pretty much the same. Your work will be mainly (if not exclusive) on the recipe/meta.yaml file.

This is a possible order of action:

From past experience, I’ve seen Conda packages available in Conda Forge in under 30 minutes, but have also had to wait hours (<8 hours) for it to show in Conda Forge. For example, if Azure or GitHub infrastructure have issues (as I experienced during the first releases) it may take days. So keep that in mind before announcing the Conda packages.

Testing the Packages

To test packages locally, first you should make sure that Conda is configured to avoid automatically upload the package. Open your ~/.condarc and check that you have something similar to:

channels:
  - conda-forge
ssl_verify: true
anaconda_upload: false

Now create a Conda environment for your tests, e.g.: conda create -n cylc1, and then activate it conda activate cylc1. Then to build and install locally:

# Where $CONDA_FORGE_REPOSITORY could be, for example,
# cylc-uiserver-feedstock.
cd $CONDA_FORGE_REPOSITORY
# Your package should not be listed!
conda list
# This will take some minutes and print useful information.
conda build recipe/
# The following command will install the locally created package. Before
# installing it will ask you to confirm. Scroll up and search the
# package name. The right-side column must show a location like
# .../anaconda3/conda-bld/linux-64::cylc-uiserver-0.1-py37_1.
# This confirms you are installing the local build. Here $PACKAGE_NAME
# could be something like cylc-uiserver.
conda install --use-local $PACKAGE_NAME
# If the last command doesn't use the right build, try:
conda install -c ${CONDA_PREFIX}/conda-bld/

At this point you should be good to go. Test the package with commands such as which $PACKAGE_NAME, or $PACKAGE_NAME --version, etc. Of if you are testing the metapackage, try running the complete system with a workflow and the UI or tui, and check if there is nothing wrong.

Troubleshooting Conda Forge issues

Issues building or installing the package locally with conda-build and conda install

If you found problems while testing locally, try troubleshooting locally, and either mark the pull request as draft, and close it. Merging the pull request will create a release.

Marking a release as invalid or broken

To undo a release, you will have to liaise with the maintainers via Gitter or Github. Or, alternatively, bump up the build number +1, and release it again.

conda-install is picking the wrong version or build of a package

If conda install cylc returns the incorrect build number, it could be an issue that is very difficult to debug (a posteriori). In fact it may not even be a bug:

Question to conda Gitter channel: (@hjoliver Jul 21 17:23):

I have a conda metapackage just published to conda-forge, which (on conda install) does not install the latest build numbers of several of our component packages. The conda docs cite bug fixes as a primary use case for incrementing build numbers, in which case I would have thought the environment solver should never select older builds? If I install the component packages directly, I do get the latest builds, but not via the metapackage. Anyone know what’s going on here?

Replies:

@wolfv: @hjoliver this article describes how the conda solver works: https://www.anaconda.com/blog/understanding-and-improving-condas-performance. It specifically says that build numbers are only maximized for explicit specs

@jjhelmus: @hjoliver changes to the build number can also occur when a package is built against a different version of a library which is incompatible with earlier version. For example, one build of python 3.7.7 might be compiled against libffi 3.2 while the next build number against libffi 3.3. Depending on what other packages are installed in the environment it may not be possible to install the package with the largest built number. In the example if cffi is only available as a package built against libffi 3.2 then the older build of python 3.7.7 would be required.

Interpretation: build number is treated much like version number, if not pinned down exactly conda can choose an older build if that works best for solving all the dependencies in the system. Conda would presumably(?) choose the newest build if it does not affect dependencies; we may just need to check what happens if dependencies get changed by a new build.

If you are sure that you have your new version or build available from Anaconda.org, and that conda install --dry-run $package should work, then you could have hit a bug in either conda client or in Conda Forge repository data.

You can try the following:

Conda documentation has details about their package math specification. Basically, there are three parts, where the last one is optional.

So you will have to specify the exact version, and use a glob expression for the build number. For example, from our alpha2 release:

run:
  - python
  - cylc-flow 8.0a2 *_3
  - cylc-uiserver 0.2 *_2

In the example above, we are specifying the packages and versions we want. Plus, we also give it a pattern to the build string, so conda will match cylc-flow-8.0a2-py37_2, or cylc-flow-8.0a2-py38_2, or cylc-flow-8.0a2-py37h2387c_2 (all valid build strings).

Announce on Element and Discourse?

Bask in the glory of having created a release.

This site is open source. Improve this page.