Upgrading Baserock systems

Baserock aims to provide tooling that allows you to build a custom operating system. There's no 'universal upgrade mechanism' that would manage to satisfy every possible use case, and if you are building a product with Baserock you will need to think about your specific needs when deciding how to deliver system updates to your users.

The developer tooling does provide a mechanism for upgrading the reference systems that are used during development, which is described below.

This guide is aimed both at people who are interested in how Baserock upgrades work, and people who are responsible for maintaining Baserock development infrastructure who need to know how to keep their deployed systems up to date.

Upgrading Baserock devel systems

Notes:
1. This method can not be used to upgrade a Baserock system running in a chroot.
2. This method works only for upgrading Baserock 14 releases or newer.
3. This method works only for upgrading Baserock systems using btrfs.

To perform an upgrade, first get the branch containing the system to which you want to upgrade:

git clone git://git.baserock.org/baserock/baserock/definitions
cd definitions
git checkout branch-containing-system-you-want

It's best to create a new branch for this (ie: replace 'your-branch' with a branch name that doesn't exist yet). This will avoid some errors.

Go to the definitions directory in your-branch, then run:

morph build systems/system-you-want

Now use clusters/upgrade-devel.morph to deploy the result of that build as an upgrade to your system. Choose a version label that makes sense to you: 'devel' is fine, but some people use the current date as the version label, or other things. The label you choose must be valid as a filename.

NOTE: If your system image is not one of the prebuilt ones from baserock.org, then when upgrading you should use the cluster .morph file that you used to deploy it, instead of clusters/upgrade-devel.morph. That way, any special configuration from the original deployment will be preserved in the upgraded version.

NOTE: Ensure that your root user has passwordless SSH access to 127.0.0.1 with ssh root@127.0.0.1 whoami. If not, run ssh-copy-id root@127.0.0.1.

morph upgrade clusters/upgrade-devel.morph self.HOSTNAME=$(hostname) self.VERSION_LABEL=devel

If you get a "No space left on device" error, you may need to remove some old versions from btrfs. The command system-version-manager list will show all the current bootable systems in btrfs. You can delete any of them other than 'factory' and the one marked '(running)'. system-version-manager remove <old-version> will remove it. system-version-manager --help will give more information on usage.

Your configuration in /etc should be propagated to the new system, but there may be merge conflicts. Check /etc for files named '.rej' and '.orig' in the new system, which will indicate that there are changes from the old system that must be merged manually. You can get a nice diff from the old /etc as follows:

mount /dev/sda /mnt
git diff --no-index /mnt/systems/factory/run/etc /mnt/systems/$VERSION_LABEL/run/etc

To use the upgraded system, simply reboot. The most recently deployed system will be the default.

Upgrading a devel chroot

You can add a new chroot using the manage-baserock add command, passing it a label for the chroot and a URL or path for the tarball to download.

You can enter the new version with enter-baserock LABEL (where LABEL is the label you gave it). You can also make the new version the default with manage-baserock set-default LABEL. You can remove older versions using manage-baserock rm.

Only the /src directory is shared between chroots at present, because the tooling is very simple. Other configuration will need to be recreated in the new chroot.

Upgrading a Trove

See the 'upgrade Trove' guide.

Upgrading a distbuild network deployed with NFS on a Calxeda Highbank ARM server

See the distbuild on Calxeda guide.

A simple developer workflow for x86 Baserock VM or NVIDIA Jetson ARM board

The 'cycle.sh' script in definitions.git gives you a workflow for building and testing changes within a deployed Baserock 'build' or 'devel' system, by deploying the result of the build as a temporary upgrade.

See the 'simple build-deploy workflow' guide for instructions on this workflow.

Best practices for maintaining Baserock infrastructure

A team working with Baserock tooling may need a bunch of shared infrastructure systems set up. For example, you might have a Trove for storing source code and a distbuild network running on a build server or cluster of devboards. You might deploy web tools like issue trackers and patch trackers using Baserock as well.

We recommend that all your infrastructure is deployed from a single infrastructure.git repo. You can start one by cloning the Baserock reference definitions.git repo and pushing it somewhere of your own. As you deploy systems, commit all the files needed to deploy them to this repo. If the repo is public then you can't commit secret files like SSH private keys, you'll have to store them somewhere else, but there should only be a few such files. The goal is to make it so 'master' of your infrastructure.git repo reflects exactly what you have running, which makes it easier to debug when things go wrong, and is useful to know when you are thinking about backups.

When you upgrade your infrastructure (for example, when there has been a new release of the reference definitions.git repo tagged) then follow this process. We'll assume the new Baserock release is version 20.00 and your infrastructure repo is called $infrastructure.git.

First, clone your infrastructure definitions.

git clone $infrastructure.git
cd infrastructure

Merge in the latest tagged release of the Baserock reference systems from the reference systems Git repository.

cd you/upgrade-to-20.00/$infrastructure.git
git remote add upstream git://git.baserock.org/baserock/baserock/definitions.git
git remote update upstream
git merge --no-ff baserock-20.00

Build and deploy the new version of each system you want to upgrade. See below for specific guides. If you are building with [distbuild], you need to push your branch first, so the distbuild network can see it.

If anything goes wrong, you can always roll back to the previous version, by using system-version-manager set-default to set the previous version as the default, and then rebooting the failed system.

If you're confident that the upgraded systems are working, merge your branch to 'master' and clean up. Now 'master' reflects the state of your actual infrastructure again.

git checkout master
git merge you/upgrade-to-20.00
git push origin master
git push --delete origin you/upgrade-to-20.00
git branch -d you/upgrade-to-20.00

You might want to create a tag as well.

The Baserock project has its own infrastructure.git repo (which is forked from definitions.git). You might find it useful to adapt one of the web systems we have running at baserock.org like https://gerrit.baserock.org: almost all infrastructure at baserock.org is deployed from this repo (although not all of it runs on Baserock yet), and we try to follow the process above. You can merge stuff from our infrastructure.git into yours, either by copying the files you need or by adding it as a remote and using git merge (as we did above with 'definitions.git).

Implementation of the reference upgrade mechanism

Morph supports deploying upgrades through the morph upgrade command, very similar to how it supports deploying build systems with the morph deploy command. Each upgrade mechanism is implemented by a '.write extension'. There are some of these built into Morph (run morph help-extensions for a list), and they can also live in a definitions.git repository.

The reference systems that the Baserock project provides, such as the 'build', 'devel', and 'trove' systems, currently support atomic upgrade and rollback. This is implemented using the Btrfs copy-on-write filesystem. Each system-version is kept in a Btrfs subvolume, and snapshotted at deploy time. Certain directories are mounted as separate subvolumes and shared between the system-versions, so that the contents of /home, /root, /opt and /var are shared between versions.

The /etc directory is not shared, but when an upgrade is deployed to a running system, the baserock-system-config-sync tool is used to merge /etc of the running system with the /etc of the system being deployed.

The 'extensions/rawdisk.write' extension, along with some functions built into Morph, handles deploying systems to Btrfs disk images with the layout that the Baserock tooling expects. It can also upgrade disk images, but you must make sure that the machine is powered off if you do this, and if you have made changes in /etc these will not be automatically merged into the new system.

In general, it's better to deploy upgrades to a running system using the 'extensions/ssh-rsync.write' extension. This extension requires access as 'root' over SSH to the system being upgraded. It's fine to use 'localhost' for this, to get a 'build' or 'devel' system to upgrade itself.