Build Issues

Use the latest versions

Common causes of build errors include

1. Not having the latest version of morph
2. Your Development VM is old and not compatible with the latest version of morph

To fix 1. follow the instuctions on Using latest version of Morph

The easiest way to fix 2. is to create a new development VM using the latest released image as described on Quick Start. The new VM can use the virtual disk containing the '/src' partition from the old VM so you won't lose any work.

Out of disk space

Morph creates quite a lot of artifacts, when building, and leaves them around for traceability and to speed up later builds. However this means your disk may fill up, causing morph build to fail with

ERROR: [Errno 28] No space left on device

Normally this should be resolvable by running morph gc to remove temporary files and old cache items.

If morph gc doesn't clear enough space, try removing old system artefacts with

rm /src/cache/artifacts/*rootfs

You can also safely delete temporary files and directories /src/tmp/*. Don't remove /src/tmp itself.

If you get this error:

ERROR: Insufficient space on disk:
    /tmp/morph_tmp requires 4000000000 bytes free

You should set the tempdir option so Morph places its temporary files somewhere with more space. There are instructions to set up /etc/morph.conf to do this on the Quick start page.

Chunk build fails

When building a chunk fails, you will see an exception such as following from Morph:

ERROR: In staging area /src/tmp/staging/tmp123456 command 'sh \
    autogen.sh --prefix="$PREFIX" --disable-tests' failed.

If the problem is not immediately obvious from the log, use chroot to open a new shell chained to the staging area. Note that it will have been moved from /src/tmp/staging to /src/tmp/failed:

chroot /src/tmp/failed/tmpoeogba

Note that the error message says the staging area is in /src/tmp/staging, but it gets moved to /src/tmp/failed.

You will now be using the commands inside the staging area sysroot and will be unable to access the tools of the host system. You can exit from this by typing exit. You can run the command that failed manually in exactly the same environment that Morph was using. Morph will not reuse this staging area, so you cannot make permanent fixes here -- these must be done in the git repository of the chunk you are building instead.

To rerun the command above, we would do the following:

cd gobject-introspection.build
sh autogen.sh --prefix="/usr" --disable-tests

Warning: If you modify anything outside gobject-introspection.build you can break future builds. If you have inadvertently done this, you need to clean up your hard-link cache, as described below.

Maybe parallel make is unpredictable

Baserock does parallel make by default, for performance. Sometimes this can lead to something failing to build because it has a dependency still building. To rule out this as a possible source of error, you can force make -j1 by setting the following in the chunk.morph file

max-jobs: 1

Broken C compiler

...
checking whether the C compiler works... no
configure: error: in `/gobject-introspection.build':
configure: error: C compiler cannot create executables
See `config.log' for more details

See the config.log file (this may be in a subdirectory, for projects that require out-of-tree builds or use nested configure scripts. You can use find -name config.log to locate it). In this case, the log contains the following:

configure:3611: checking whether the C compiler works
configure:3633: gcc    conftest.c  >&5
ccache: FATAL: distcc: No such file or directory
configure:3637: $? = 1
configure:3675: result: no

Looks like we have manually enabled distcc but it's not available in the staging area, so the compile fails.

C++ Compiler out of memory

The following error message indicates that there is not enough RAM in your Baserock system

g++: internal compiler error: Killed (program cc1plus)

If you are running in a VM, try increasing the amount of RAM allocated to it.

/ is not a mount point

If your morph build fails with

mount(/, MS_PRIVATE | MS_REC): Invalid argument

you're probably in a chroot. Instead of running chroot $path to enter your chroot, run cd $path && mount --bind . . && chroot ..

This is because we use a tool called linux-user-chroot to isolate builds. Part of its setup is to alter mount options of / to make it unable to run SUID binaries. This only works if / is already a mount-point.

Bind-mounting your chroot to itself ensures it becomes a mount-point. This has to be done from outside the chroot however, since it does not work from inside, hence you can't fix up the chroot later, otherwise morph could do this for you.

Cache-related issues

If you have lost power during a build, some rather strage build failures may occur.

1. If your C compiler complains about malformed .a .o or .d files, then you ccache may be corrupt. If re-running the build with --no-ccache works, then follow the instructions for clearing your ccache.
2. If files in your staging area are unexpectedly short, or empty, you may have a corrupted hard-link cache.
3. If you have emptied your hard-link cache and are still encountering this, you will need to check chunks in your local artifact cache.
4. If git complains about short index files, your git repository cache may be corrupt.

Clearing your ccache

Find out where morph has put your cache directory by running morph --dump-config | grep cachedir.

Remove the ccache subdirectory.

Clearing your hard-link cache

Find out where your temp-dir is by running morph --dump-config | grep tempdir.

Remove the chunks subdirectory.

Checking your chunk cache

Find out where morph has put your cache directory by running morph --dump-config | grep cachedir.

We could remove it entirely, but it's worth trying to find the defective chunk first, since it can be expensive to rebuild everything if the artifacts aren't cached on the Trove.

Checking for empty chunks

All valid chunk artifacts have at least 1 file in them, so a chunk of size 0 is not valid.

Running the following command in the artifacts subdirectory will find every artifact that is too small.

find . -name '*.chunk.*' -size 0

To remove all these files, run:

find . -name '*.chunk.*' -size 0 -exec sh -c 'echo ${1%.chunk.*}' - {} \; |
    sort -u | while read f; do rm "$f"*; done

Checking for short chunks

A chunk may be short without being empty, this has to be detected by tar.

find . -name '*.chunk.*' -exec \
    sh -c 'tar -tf "$1" 2>&1 >/dev/null | grep -q "short read"' - {} \; \
    -print

To remove all these files run:

find . -name '*.chunk.*' -exec \
    sh -c 'tar -tf "$1" 2>&1 >/dev/null | grep -q "short read"' - {} \; \
    -exec sh -c 'echo ${1%.chunk.*}' -  {} \; |
    sort -u | while read f; do rm "$f"*; done

Force a clean build

To force Morph doing a clean build comment out the line in your morp.conf

# artifact-cache-server = http://cache.baserock.org:8080/

Then delete all the artifacts (find your cachedir with morph --dump-config | grep cachedir):

rm -rf /src/cache/artifacts

Building on NFS

Using Network File System (NFS) for your /src partition is possible, although it will be slower and more fragile than using local filesystem. You MUST ensure that the clock of the NFS client is synced to the clock of the NFS server.

Many build systems change their behaviour based on the timestamps of files in the repository. For example, Makefiles created by GNU Automake will check whether Makefile.am has an mtime newer than the generated Makefile, and will rerun automake if so. Morph ensures that all 'mtimes' are set to the same time at the start of a build, but this depends on the NFS client and server having matching clocks.

Error: git directory x has no commit at y

This can happen when you try to build definitions from a commit in a local clone of definitions that is not available in Morph's repo cache. It can also happen if you have made commits in your local clone, but they are not pushed to the repo that you cloned. Morph looks in the origin (e.g. git.baserock.org/baserock/baserock/definitions) when building, rather than your local repo.

There are two possible fixes for this. First, you could tell Morph to create a temporary local branch with your uncommitted and unpushed changes: morph build --local-changes=include xx. Or you can push the commits to the remote, if you have push access.

Other Strange Errors

Master of definitions may be broken, leading to all sorts of weird and wonderful errors. You can check for this here: http://mason-x86-64.baserock.org/ . If a commit is red, it's broken.