How to: Cross-bootstrap using BuildStream

BuildStream is a new build tool which we hope that Baserock will adopt soon. The Baserock definitions currently support BuildStream through an automated conversion script that lives in the definitions.git master branch.

This is the rough process for getting Baserock reference systems to build on a new architecture using BuildStream. There's no "one size fits all" solution -- please proceed with gumption.

Note that BuildStream itself has no defined set of architecture names -- it's up to the definitions authors (i.e. Baserock) to define them. BuildStream's --arch argument defaults to the output of uname -m which doesn't always correspond to the name we use in Baserock -- for example our ppc64b architecture comes up as ppc64 so we need to always pass --arch=ppc64b when building on that architecture. The arches section of our project.conf file gives an idea of the architecture names that Baserock uses.

The Baserock reference definitions are written under the assumption that each element will be compiled on the same platform that it will execute on. They require a prebuilt sysroot to build from, which creates a chicken-and-egg problem on new platforms as a prebuilt sysroot may not be available.

The gnu-toolchain.bst/stage2.bst stack contains specially written elements that can be cross-compiled. To bring up a new platform, we start by cross-building a sysroot and toolchain on an existing platform (usually x86_64), then we compile the remaining components on the target platform. It's simplest to run the sysroot in a container on the target platform, but you can also boot into it directly.

Thus the stages are:

cross-build gnu-toolchain/stage2.bst to create a "stage2" sysroot for the target
boot or chroot into the stage2 sysroot on the target device
do a source-bundle build of gnu-toolchain.bst inside the "stage 2" sysroot to create a "stage 3" base.
set up BuildStream on the target device
cleanly rebuild gnu-toolchain.bst and push the binaries somewhere

You will then be set to run BuildStream builds of the Baserock reference systems on your new device.

Each step is described in more detail below. I'm using the ppc64b architecture as a placeholder which you should replace with whatever is your actual target.

1. Cross-building a stage 2 sysroot

Clone the Baserock definitions.git repo and run the ./convert script, as per the README file.

NOTE: For now you need to use branch sam/buildstream-bootstrap

The Baserock's gnu-toolchain.bst stack builds GCC three times. This is done to ensure that the final builds are independent from the binaries provided by gnu-toolchain/base.bst. Each build is considered a separate stage, hence you will see "stage 2" and "stage 3" referred to in this document. (Note that the elements from stage 3 don't have "stage3" in their names anywhere).

The stage 2 builds can be cross compiled. We start by doing that to produce a "stage 2 sysroot" for the target architecture.

bst --target-arch=ppc64b build bootstrap/stage2-sysroot.bst
bst --target-arch=ppc64b checkout bootstrap/stage2-sysroot.bst ./stage2-ppc64b

We need to fix up the stage2 sysroot so that the source-bundle build we are going to do works correctly. Having a symlink from /usr/bin -> /tools/bin is harmful because the source-bundle will install stuff straight into /, so we replace it with a proper directory and just add a symlink for hardcoded shell shebangs.

cd stage2-ppc64b
rm usr/bin
mkdir usr/bin/
ln -s /tools/bin/sh ./usr/bin/sh
ln -s /tools/bin/bash ./usr/bin/bash
cd ..
tar -cvf ../stage2-ppc64b.tar.gz ./stage2-ppc64b

2. Boot or chroot the stage 2 sysroot

If your target device already has a Linux OS installed on it, you can now copy over the sysroot, extract it somewhere, and use chroot to enter it as a container.

In possible, we recommend using Bubblewrap instead of chroot. This gives a number of benefits such as not requiring 'root' priviliges and automatically mounting special filesystems like /dev. You can do something like this:

bwrap --unshare-user --uid 0 --gid 0 \
      --bind ./stage2-ppc64b / \
      --dev /dev --proc /proc --tmpfs /tmp \
      /tools/bin/sh

If there isn't a Linux OS available for your platform, you will need to find or build a suitable Linux kernel. You will then need to set up the bootloader on your device to load the kernel, and boot into the stage2 sysroot. The exact methods will be device-specific and are out of scope for this guide. Make sure you boot off a large, writeable disk -- the bootstrap build takes lots of space, and it will install stuff directory into /.

If not using Bubblewrap, you will need to mount /dev, /proc and /tmp manually as we are not using any 'init' system that would automatically do it. You can run these commands before building:

mount -t devtmpfs none /dev
mount -t proc none /proc
mount -t tmpfs none /tmp

3. Build a stage 3 sysroot

You may have noticed that our stage 2 sysroot is rather weird-looking and contains most of its files installed into /tools. It's not really usable for anything except building the final versions of GCC and the other tools that we need. (For the curious, the stage2 sysroot corresponds closely to the "Constructing a Temporary System" section of the Linux From Scratch book).

We can't run BuildStream inside the stage 2 sysroot, so instead we get BuildStream to generate a "source bundle" on a different machine. This is a tarball containing the source code for each element and a script that runs each build.

You generate it like this:

bst --target=arch=ppc64b source-bundle bootstrap/stage3-sysroot.bst \
    --except gnu-toolchain/stage2.bst

This creates a file named bootstrap-stage3-sysroot.tar.gz which you now need to copy and extract on the target device inside the stage2 sysroot dir.

You now run the build script contained within the tarball. We need to ensure that it installs the results of each build into an empty directory (the default behaviour is to install to /, but the stage2 sysroot contains symlinks from /usr to /tools that will break stuff). The commands to run are:

export PATH=/usr/bin:/usr/sbin/:/tools/bin:/tools/sbin
cd /bootstrap-stage3-sysroot
./build.sh

This will take time; if you are connected by SSH, consider using nohup so that you don't need to worry about dropped connections stopping your build.

Finally we need to create an archive holding the resulting binaries. The build.sh script installs everything right into / (inside the chroot), so we need to include only the bits we need. Run something like this (either from / in the chroot, or from the root directory of the chroot if you're in the host machine).

cat > ./files-to-exclude <<EOF
./bootstrap-stage3-sysroot
./buildstream
./dev/*
./files-to-exclude
./home/*
./lost+found
./mnt/*
./proc/*
./root/*
./sys/*
./tools
./tmp/*
EOF
tar -X ./files-to-exclude -v -c . -f ./sysroot.tar

4. Set up BuildStream on the target device

If you already have a Linux distribution running on your target device, you can hopefully just install BuildStream and move on to the next section.

If not, you will need to produce a system image that can run BuildStream. The systems/build-system-content.bst element provides just such a thing, but it can only be native-build so we will have to use bst source-bundle again. Run this on your x86_64 machine:

bst --target-arch=ppc64b systems/build-system-content.bst \
    --except systems/gnu-toolchain.bst

Copy the resulting tar.gz over to the target device where you have already built stage3-sysroot.bst and build it in the same manner.

Again, the results are installed to /. You now need to tar up the resulting filesystem image and boot into it on your device. Test that the bst command works and move on to the next step.

5. Cleanly build gnu-toolchain.bst

Now that the bst command is available on the target device, we can produce the final binaries. These can be pushed to a shared artifact cache so that the bootstrapping process never needs to be done again.

Start from a Git clone of the Baserock definitions.git repo with a clean work tree. We are going to modify elements/gnu-toolchain/base.bst to use our temporary sysroot. Again, ppc64b architecture is used as a placeholder.

At time of writing, you will need to modify base-sdk.bst and base-platform.bst instead. In future there will just be a single base.bst element.

Add a new entry to host-arches that imports your sysroot tarball. Something like this:

ppc64b:
  sources:
  - kind: tar
    url: file:///path/to/sysroot.tar

Run bst --arch=ppc64b track gnu-toolchain/base.bst which will update the ref field with the file's checksum.

Now you can do a build of the bootstrap/stage3-sysroot.bst element:

bst --arch=ppc64b build bootstrap/stage3-sysroot.bst

At this point, you have produced sysroot capable of cleanly seeding all future Baserock builds on this platform. If you have the necessary access, push it to releases repo at ostree-releases@ostree.baserock.org/releases/ following the existing naming convention. If you don't have access to do this, contact us.

Once the binaries are available for download, modify gnu-toolchain/base.bst again so that it pulls the sysroot from the https://ostree.baserock.org/releases repo. Test that everything works as expected by building systems/build-system-content.bst. Assuming it all goes well, submit a patch for definitions.git with your changes to base.bst.

If you have done a full cross-bootstrap, now you should write BuildStream elements that can deploy disk images for your device, try them out, and upload the disk images somewhere so that in future people can just download the disk images instead of going through this long-winded process.

How to: Cross-bootstrap using Morph

These are the steps to cross-bootstrap Baserock from one architecture to another.

0. Check architecture support

Unless you know your architecture is supported by morph, you will need to add your architecture to those supported by morph.

To confirm whether your architecture is supported, look at the definition of valid_archs in the morphlib/__init__.py file of the Morph repository.

If your architecture is not listed, add a descriptive string that morph may use to identify the architecture, and also modify the get_host_architecture() method in morphlib/util.py to contain appropriate detection logic, and return your selected string. When selecting a string be sure to use one that uniquely identifies both the instruction set (ISA) and the call interface (ABI) for the new architecture.

To use your modified morph, please refer to the documentation on using the latest morph.

If you have added support for a new architecture and you are intending to build a system based on the linux kernel, you will also need to provide translation logic to convert the morph architecture string to a linux architecture string. This is maintained in the Baserock Linux repository in a file named morph-arch on the baserock/build-essential branch.

Depending on the software in your system, you may wish to configure various additional build tools to use specific flags for your architecture. For example, to pass compilation flags to GCC, one would edit the shell script named morph-arch-config in the build-essential branch of the GCC repository.

1. Select a target architecture to do the cross-bootstrap

We only have morphologies to do cross-bootstrapping for x86_64, ppc64, armv5l, armv7lhf, armv8l64 and armv8b64, so if your target architecture is one of them you only have to pick one of the morphologies.

On the other hand, if your intention is to do a cross-bootstrap to another architecture you have to create a new morphology as follows:

Create a systems/cross-bootstrap-system-TARGET_ARCH-generic.morph file (where TARGET_ARCH is the target architecture) and edit it so that it contains the following (see the most up-to-date example of this here:

name: cross-bootstrap-system-TARGET_ARCH-generic
arch: TARGET_ARCH
description: A system that produces the minimum needed to build a devel system
kind: system
strata:
- name: build-essential
  morph: strata/build-essential.morph
- name: core
  morph: strata/core.morph
- name: python-cliapp
  morph: strata/python-cliapp.morph
- name: python-pygobject
  morph: strata/python-pygobject.morph
- name: libsoup-common
  morph: strata/libsoup-common.morph
- name: ostree-core
  morph: strata/ostree-core.morph
- name: morph-utils
  morph: strata/morph-utils.morph
- name: cross-bootstrap
  morph: strata/cross-bootstrap.morph

You must replace TARGET_ARCH for the target architecture in the morphology. The morphology has to be commited and pushed in a git repository.

2. Generate a bootstrap Baserock tarball

Now is time to Cross-build Baserock. We cross-build the cross-bootstrap-system-TARGET_ARCH-generic using the cross-bootstrap command in morph. With this command morph generates a tarball of the system with a script to do finish the building in the target architecture:

Execute morph cross-bootstrap for the morphology file created:
```
morph cross-bootstrap TARGET_ARCH baserock:baserock/definitions master systems/cross-bootstrap-system-TARGET_ARCH-generic.morph
```
For this example we assume the morphology is in baserock:baserock/definitions repository and in the master branch.
Once morph finishes building, it will tell you where the tarball is.

3. Native building of the cross-bootstrap process

This is the second part of the cross-build process and it has to be done in the target architecture.

First of all you have to use the system created. To do that you have two options:
1. If you have linux running in the target architecture you can copy the tarball there, extract it and then chroot into it with:
```
chroot /path/to/extracted/tarball /bin/sh
```
2. If you do not have a compatible Linux-based OS for the target architecture you can boot the cross-built system directly (e.g. using NFS boot). You will need to build a kernel yourself if you take this approach. Note that the bootstrap tarball doesn't have an init system, you should pass init=/bin/sh to the kernel when you boot into the bootstrap system.
Once you are into the filesystem created, you have to execute the native-bootstrap script (located in '/'). This script ends building everything.

Note: we got a segment fault while directly run the native build script for armv8l64, but pipe the stdout message and background the process will workaround this issue. The reason is still unknown. e.g. ./native-bootstrap > build.log 2>&1 & tail -f build.log

Note2: cmake failed to build on some of the architectures if the environment parameter $PATH is not set to /usr/bin:/bin:/usr/sbin:/sbin.

4. Using the bootstrap system

With the bootstrap system now we can use morph to build systems. Normally you will want to build a devel-system to start using it instead of the bootstrap system.

First of all you have to use the system created. To do that you have two options:
1. If you have linux running in the target architecture you can copy the tarball there, extract it and then chroot into it with:
```
chroot /path/to/extracted/tarball /bin/sh
```
  Note: See 4a for more detailed information.
2. If you prefer, you also can boot the system (e.g. with NFS boot). The problem to do this is that the system doesn't have Kernel.
  
  Note: See 4b for more detailed information.

4a. Extra steps with chroot

You have to mount some special filesystems inside the chroot. To do that, run this outside the chroot:

cd /path/to/uncompressed/tarball
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
mount -o bind /tmp tmp/

And then, you are ready to chroot into it:

chroot . /bin/sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin
LD_LIBRARY_PATH=/lib:/lib64
ldconfig

Morph uses linux-user-chroot when building, and this does not work inside a chroot. There are two options for fixing this:
1. Create a bind mount to hide the chroot: cd /path/to/uncompressed/tarball && mount --bind . .
2. Move the bootstrap filesystem to the top level of another disk. We haven't investigated why it works.
To test that this works you should be able to run the following into the chroot environment.
```
linux-user-chroot / /bin/sh
```
Note: You will not be able to use Morph if this didn't work. If it works, come back out of the linux-user-chroot before building; you are only running it at this stage to check that it works.

4b. Extra steps if you booted the cross-bootstrap system directly

Mount sysfs and proc.

mount -t sysfs none /sys
mount -t proc none /proc

To be able to ssh into it:

/usr/sbin/sshd-keygen
/usr/bin/ssh-keygen -A
/usr/sbin/sshd

4c. Build a devel system

Now you are ready to start using baserock, build and deploy a devel system.

Be warned that we use systemd for the network configuration, but cross-bootstrapped systems don't include systemd. This means that if you try to clone definitions, morph will fail to find the remote repository and will say something like: 'unable to look up git.baserock.org'. To fix this, you will need to make a file named resolv.conf in /etc, and add a nameserver entry to it.