Baserock release process

These are the steps to go through to make a Baserock release. The release process is a work in progress, so expect it to require a lot of improvement over time. Also, further tool support is needed.

Releases tend to involve uploading lots of data. As a rule of thumb, you should aim to be uploading the release images and artifacts the night before the date that the release is due.

Prerequisites for making a release

Note: Typically, only Baserock maintainers have enough access to complete this process.

You will need:

Push access to the git.baserock.org Trove.
Write access to the backing store for download.baserock.org.
A fresh x86_64 (or x86_32 at a push) VM to coordinate the release. This VM should have enough space to coordinate the release artifacts. In the past, 30G or more was required. Please refer to other guides for setting up such a VM.

You will also need access to:

A local Trove synced with git.baserock.org.
An x86_64 distributed build infrastructure.
An x86_32 distributed build infrastructure.
An armv7lhf distributed build infrastructure.
Configuration for the above suitable for use with your coordination VM.

Configuration

Ensure that morph.conf on your coordination VM is configured to use your internal Trove by setting trove-id and trove-host as appropriate.

Deliverables

See release.morph in definitions.git for the systems we release. In addition to the system images and other files produced by release.morph, the release requires manually producing or updating the following:

Release notes, on wiki.baserock.org.
Updated symlinks to "current" images and tarballs

For a GENIVI release, the following artifacts are produced:

GENIVI baseline manifest and license manifest
GENIVI release notes

Compatibility considerations

Before making the release, make sure that the previous release of Baserock can build the new release. The 'master' branch of definitions is continuously built by http://mason-x86-64.baserock.org and http://mason-x86-32.baserock.org/ so, as long as these are running the previous release, you can be sure that there are no compatibility issues.

If there are planned incompatible changes to definitions, make sure that the version of Morph in the release is capable of understanding the new versions, so that the changes can be committed to 'master' of definitions.git after you tag the release.

Release steps

You should do the release in a clean instance of the previous Baserock devel system release, preferably on x86 (for speed). This makes it possible for others to reproduce the release images you created.

Create a new branch (baserock/release/baserock-X.Y) in definitions.git. This is the "release branch", and any changes made during the release will happen there, avoiding the need to freeze master while the release is happening.
Get the branch mirrored to your local Trove (push to git.baserock.org, promote on your local Trove to get it lorried quickly).
- log into your local Trove, run this: curl -X POST -d path=baserock/baserock/definitions http://localhost:12765/1.0/move-to-top
- then wait and check the Trove's status page until the definitions repository has been lorried
Run scripts/release-build to produce release artifacts. This script builds all the artifacts needed for the release (using distbuild, so they end up on your local Trove), and creates the release artifacts (all the images and tar archives etc).
- See the scripts/release-build.test.conf sample configuration file and use that as a basis for your own.
Compress the release artifacts with gzip. (FIXME: Maybe release-upload should do this.)
Run scripts/release-upload release.morph to publish release artifacts. This copies build artifacts from your local Trove to git.baserock.org, and other files to download.baserock.org. After this script has finished successfully, Baserock users can start using the new release.
- See the scripts/release-upload.test.conf sample configuration file and use that as a basis for your own.
Manually create the "current" symlinks pointing at the new release. (FIXME: The upload script should do this automatically, but that code hasn't been written yet.)
Tag the HEAD of the release branch with the new release (baserock-X.Y). Push the release branch, including tags (git push --tags).
Prepare or update release notes, wikis, and websites.
- Drafting these can start early on, while building and uploading are running.

The upload script assumes that you have appropriate credentials to upload files to the servers using rsync. If a password is required, rsync will prompt for your password at that stage of the script. If you rely on SSH agent forwarding to provide access, and are leaving the release script running overnight, remember that the machine running your SSH agent must also remain running overnight!

GENIVI Baseline release steps

For the GENIVI Baseline systems there are several extra steps. Beware: these scripts are slow and can take several hours!

Generate a manifest for each baseline system, using the morph generate-manifest command:

 morph generate-manifest /src/cache/artifacts/*.genivi-baseline-system-armv7lhf-jetson-rootfs > \
     /src/release/genivi-baseline-system-armv7lhf-jetson-manifest.txt
 morph generate-manifest /src/cache/artifacts/*.genivi-baseline-system-x86_64-generic-rootfs > \
     /src/release/genivi-baseline-system-x86_64-generic-manifest.txt

NOTE: You will need the rootfs artifacts because while generate-manifest claims to operate from disk images, the code is not in-sync with our current raw disk generator

Generate a license manifest for each baseline system, using the license-check script in definitions.git:

 scripts/licensecheck.py systems/genivi-baseline-system-armv7lhf-jetson.morph > \
     /src/release/genivi-baseline-system-armv7lhf-jetson-license-manifest.txt
 scripts/licensecheck.py systems/genivi-baseline-system-x86_64-generic.morph > \
     /src/release/genivi-baseline-system-x86_64-generic-license-manifest.txt

Extract the kernel zImage file from the ARMv7lhf system, by loopback mounting it and copying the file out. (FIXME: there is an unmerged patch in morph.git branch baserock/richardipsum/write_kernel to make morph deploy handle this automatically).
Write GENIVI release notes. These should summarise the changes relevant to developers of systems based on the Baserock GENIVI Baseline system, and link to the Baserock release notes. See http://projects.genivi.org/GENIVI_Baselines/genivi-baserock/blog for examples.

You'll need to upload these extra files to the download.baserock.org server manually; the script doesn't take care of this yet (see 'future plans', below).

Announcing the release

Once you've successfully completed all the steps to produce and upload all the files for the release:

Update all pages on wiki.baserock.org that refer to the previous release number to the new release number.
Send release notes to baserock-announce@baserock.org and baserock-dev@baserock.org.

Announcing the GENIVI release

Send announcement email to genivi-dev@mail.genivi.org and genivi-baserock@lists.genivi.org
Post release notes into blog at http://projects.genivi.org/GENIVI_Baselines/genivi-baserock. Currently Jonathan Maw (jonathanmaw), Pedro Alvarez (pedroalvarez), Sam Thursfield (ssam2) and Rob Taylor (robtaylor) have admin access to do this. WARNING: the 'preview' renderer will show you something quite different to the actual rendering after you've pressed 'post'.
Review wiki.baserock.org's GENIVI related how-tos.

Post-release

Any incompatible changes to definitions that have been planned, but were waiting for a Baserock release with a version of Morph that could understand them, can now be merged to 'master' of definitions.git.

If you are able, upgrade http://mason-x86-64.baserock.org/ and http://mason-x86-32.baserock.org/, so that new features of Morph from the new release can be used. If you are not an op, remind a member of the operations team to do this.

Future goals

Define and deploy a virtual machine configured specifically for doing releases, which can be used by anyone reasonably trusted that wants to make a release. This should replace the first couple of sections of this documentation, which quite lengthy and manual. Our cloud infrastructure has very fast upload bandwidth which makes releases much less time-consuming.

Checksum everything we release, to guard against corruption at all stages of the release process.

Consider the bit-for-bit reproducibility of the release images, and how we can make it easy for those who want to reproduce the Baserock release images that are produced as part of this process.

Allow greater flexibility in what we release. Some of us think that Baserock's release process should, in future, be as follows: the build-system image is released every week (this image is capable of building any other Baserock image), other systems can optionally be released at the same time according either to schedule or need. For example, we can continue to release the GENIVI Baseline demo system every 6 weeks, following the GENIVI Baseline release process. We can release other example systems when they have received significant changes to merit a new release.

Automate GENIVI releases. Ideally, this should be done in a generic way, treating the GENIVI Baseline system as if it was one of many example Baserock systems which each had their own release requirements.

Ensure that this script is useful for projects derived from Baserock, as well as for the Baserock project itself.