Baserock/ Using Baserock

Using Baserock

Community

Background

Using and Developing Baserock Systems

  1. Simple build-deploy workflow
  2. Build a Baserock System
  3. Deploy to a raw disk image
  4. Deploy to KVM
  5. Deploy to VirtualBox
  6. Deploy to OpenStack
  7. Updating morph
  8. Upgrading a Baserock installation
  9. Make a change to Baserock
  10. Add something to Baserock
  11. Now what?

This page aims to get you up to speed developing and modifying Baserock systems, after you have established a Baserock development environment in an x_86 virtual machine, Jetson board or Linux chroot.

Simple build-deploy workflow

Currently the simplest way to get going with Baserock on x86 VM or Jetson is to try the simple build-deploy workflow. This process doesn't work yet for chroot.

The following sections on this page describe specific actions in more detail.

Build a Baserock System

A good way to test that your development system is working is to rebuild the smallest commonly useful Baserock system (the "base system"). To do this, you can checkout the current release branch of the Baserock system morphologies and build from there:

git clone git://git.baserock.org/baserock/baserock/definitions --branch baserock-16.13
cd definitions
morph --verbose build systems/base-system-x86_64-generic.morph

NOTE: there used to be morph checkout that you had to use. This command is deprecated now. Morph works in any Git checkout.

Replace x86_64 with x86_32 to build a 32-bit system.

Morph will probably download the components from the artifact cache on git.baserock.org, rather than rebuilding them from scratch. If you want to force it to rebuild everything, you can disable remote artifact fetching with this hack:

morph --verbose --artifact-cache-server=http://example.com/ \
    build systems/base-system-x86_64-generic.morph

Note: also as part of the build, Morph populates its cache; this consists of both the git source trees for each individual chunk in the image (in /src/cache/gits) and the compiled binaries (in /src/cache/artifacts). Subsequent builds will go much more quickly because most things are cached; if the upstream source tree for a chunk hasn't changed, Morph can just re-use the corresponding binary in the artifacts directory. If the source tree has changed then the artifact will be rebuilt, but at least now git only needs to fetch the the latest changes rather than the entire tree.

When the build finishes, Morph will give you a path to the built system image, which will be tar file of the root file system. To be used, this needs to be deployed to a disk image for a virtualisation environment. Deployment is a separate step from building the system as you may wish to deploy the same built system to multiple different targets. For example, the same development system we built above can be deployed to KVM, VirtualBox or even real hardware.

Deploy to a raw disk image

To deploy to a raw disk image you need to create a cluster morphology. Here is an example; put it in a file called clusters/base-system-rawdisk.morph in your definitions.git clone:

name: base-system-rawdisk
kind: cluster
systems:
- morph: systems/base-system-x86_64-generic.morph
  deploy:
      my-raw-disk-image:
          type: extensions/rawdisk
          location: base-system.img
          DISK_SIZE: 4G

Then run the deployment command:

$ morph deploy clusters/base-system-rawdisk.morph

The file base-system.img will be created in the current directory.

A 4 gigabyte disk size will allow some room for upgrading a system.

Once deployment is complete, copy the newly created image out of the virtual machine and test it in a second VM. The fastest way to copy is probably using netcat, since ssh adds a fairly heavy overhead with encryption.

Alternatively, if using KVM, you can shut down your virtual machine, mount the src filesystem directly as a loopback device (with mount -o loop), and simply copy the file out. Be very sure to shut down the VM before mounting the filesystem in the host and conversely to unmount the filesystem before re-starting your VM.

Deploy to KVM

To deploy to KVM you need ssh access to the host you want to deploy to. morph deploy will create a new VM on the host. Below is an example KVM cluster morph.

name: base-system-kvm
kind: cluster
systems:
- morph: systems/base-system-x86_64-generic.morph
  deploy:
    my-kvm-system:
      type: extensions/kvm
      location: kvm+ssh://user@host/my-kvm-system/path/to/vm.img
      DISK_SIZE: 4G
      RAM_SIZE: 2G
      VCPUS: 1
      HOSTNAME: my-kvm-system

Note:
Make sure the location path is correct. It should be the name of your VM followed by a path to the VM's disk image.

To deploy, run the deployment command:

$ morph deploy clusters/base-system-kvm.morph

Morph will ssh into host as user and create a file vm.img at /path/to, user must have permission to create in /path/to.

Deploy to VirtualBox

To deploy to VirtualBox you must be able to access the host that runs VirtualBox over ssh from within your development VM. morph deploy creates a new virtual machine that runs the system image you've just built, and sets up the networking to allow the VM to communicate with the host and with the network

The parameters to morph deploy - as shown in the example below - are as follows:

  • TYPE is virtualbox-ssh
  • SYSTEM is the name of the system you have just built
  • LOCATION is a custom URL constructed from the following
    • How to log in via ssh to the host that runs VirtualBox: alice@192.168.122.1 in the example below. alice is the username, and 192.168.122.1 is the address, as seen from the development VM, of the host. The address may be a private IP address provided by VirtualBox, if your development VM is running on the same physical host.
    • The name of the new virtual machine: devsys in the example. Note that this is not part of the filename on the host machine.
    • The filename on the host machine for the new virtual machine's disk image: /home/alice/devsys.vdi in the example.
  • DISK_SIZE - the size of the disk image to create, which is 2.00 GB in this case

In addition to the TYPE, SYSTEM, LOCATION and DISK_SIZE parameters, you must supply values for the following

  • HOST_IPADDR - the ip address of the host-only network adapter on the VirtualBox host
  • NETMASK - the netmask of the host-only network adapter on the host
  • NETWORK_CONFIG - a string containing configuration information for the eth0 and eth1 interfaces in the VM:
    • eth0 will be connected to the host-only network adapter;
    • eth1 will be connected to the NAT network adapter on the host and will be used by the VM to access the outside world.

Here is a cluster morphology using these parameters:

name: base-system-virtualbox
kind: cluster
systems:
- morph: systems/base-system-x86_64-generic.morph
  deploy:
    my-base-system-virtualbox:
      type: extensions/virtualbox-ssh
      location: vbox+ssh://alice@192.168.122.1/devsys/home/alice/devsys.vdi
      DISK_SIZE: 2G
      HOST_IPADDR: 192.168.100.1
      NETMASK: 255.255.255.0
      NETWORK_CONFIG: lo:loopback;eth0:static,address=192.168.100.2,netmask=255.255.255.0;eth1:dhcp,hostname=$(hostname)

Change the location above to work with your system, and then run it to create and start the VM.

When the VM has finished booting into Baserock booting up, log in as root. Bash isn't the default shell, so you will need to run it manually after logging in by typing bash, after which you'll be greeted by the prompt that you specified in your branch of the Bash source code.

Deploy to OpenStack

To deploy to OpenStack you need to create a cluster morphology with specific parameters:

  • type is openstack.
  • location is the URL to the OpenStack identity service.
  • OPENSTACK_USER is the deployer's OpenStack username.
  • OPENSTACK_TENANT is the OpenStack project for this image.
  • OPENSTACK_IMAGENAME is a name to identify the image in OpenStack.
  • DISK_SIZE - the size of the disk image to create. If you deploy an image with cloud-init (as in the following example), the disk will be resized to fit the maximum space given in OpenStack.

The following field must also be set, but we recommend passing it on the commandline:

  • OPENSTACK_PASSWORD is the password of the OpenStack user.

Here is an example of a cluster morphology to deploy a system with cloud-init integrated, put it in a file called clusters/base-system-openstack.morph in the definitions repository checkout.

name: base-system-openstack
kind: cluster
systems:
- morph: systems/base-system-x86_64-generic.morph
  deploy:
      openstack-image:
          type: extensions/openstack
          location: http://openstack_host:5000/v2.0/
          OPENSTACK_USER: your_user
          OPENSTACK_TENANT: your_project
          OPENSTACK_IMAGENAME: the_imagename
          DISK_SIZE: 3G
          CLOUD_INIT: yes
          KERNEL_ARGS: console=ttyS0 console=tty0

Note:
Make sure DISK_SIZE is enough to fit the system you are trying to deploy. If you are going to deploy a development system you may need 3G.

Then run the deployment command:

$ morph deploy clusters/base-system-openstack.morph openstack-image.OPENSTACK_PASSWORD=the_user_password

Be aware that your shell could save the the full command, including your password, in your shell history file.

Once deployment is complete, it's possible to launch an instance with the new image in OpenStack. Also, if the system deployed has cloud-init you can pass #cloud-config customization scripts using the "Post-creation" tab during the launch of the instance.

WARNING: if your OpenStack authentication token expires during the image upload, the deployment will probably fail. See: https://bugs.launchpad.net/glance/+bug/1371121 and https://bugs.launchpad.net/glance/+bug/1182536. To work around this, you or the administrator of your OpenStack cloud may be able to increase the expiration time of tokens. Otherwise, you will need to either reduce the size of the system you are trying to deploy, or run morph deploy from a machine which has fast enough upload speed to the cloud. A build-system instance within the cloud that you're deploying to works well for this!

Updating morph

To update morph, see this guide: http://wiki.baserock.org/using-latest-morph/

Upgrading a Baserock installation

See the 'upgrading Baserock systems' guide.

Make a change to Baserock

From inside your VM, you can make a change to the code of the system you are running and build a new system image to test your change. To do this, clone the definitions repo (or use your existing clone), and checkout a new feature branch to work in:

$ git clone git://git.baserock.org/baserock/baserock/definitions
$ cd definitions
$ git checkout baserock-16.13 -b test-branch

This Git repo contains the definitions for the Baserock 'build' or 'devel' system that you are running. The system definition will be in the systems/ subfolder. You can find out exactly what system you are running by looking in the /baserock directory, which will have a corresponding .meta file -- for example, a system built from systems/build-system-x86_32.morph will contain the file /baserock/build-system-x86_64-rootfs.meta.

You can edit the system .morph file, or the stratum .morph files that it refers to, in order to add or remove components from your new system. You can also edit code of existing components. For example:

$ morph get-repo bash
$ cd ../bash
$ git status
HEAD detached at 3590145
nothing to commit, working directory clean

This is a normal git repository, probably cloned from git://git.baserock.org/delta/bash. The morph get-repo has checked out the ref that is specified in strata/core.orph, where GNU Bash was defined. You could also use git clone to fetch the repo -- morph get-repo is just a shortcut.

If you want to make a change to GNU Bash and build a system with that change, you should first check out a branch:

$ git checkout -b my-bash-feature

Make your code changes. You probably want to test them locally before rebuilding the whole system with the new version of GNU Bash. Once you are ready to test a Baserock system with the patched version of Bash, you can edit strata/core.morph to look in your local clone of bash.git, by changing the list entry that starts 'name: bash' to this:

- name: bash
  morph: strata/core/bash.morph
  repo: file:///src/bash
  ref: my-bash-feature

This assumes that your bash.git clone is in /src.

You now need to build the new system image, which can be done using Morph. The uncommitted changes in definitions.git will be included on a temporary branch, provided the 'local-changes' setting is set to 'include'. This feature should only be used for local testing, of course! You shouldn't publish definitions containing file:/// URLs, because nobody else will be able to access the repos.

$ morph build systems/base-system-x86_64-generic.morph

After the build completes, you can use morph deploy as above to create a disk image to copy out of your VM and try to boot.

If you want to publish your change, you will need to host your bash.git fork and your definitions.git fork on a public Git server (Github, Gitlab, your own Git server, etc.), and change the repo: field for 'bash' to point to that public repo (e.g. git://github.com/youraccount/bash) instead of your local copy.

Add something to Baserock

This is our current workflow for adding new software into a Baserock system.

The high level view

  • Get the source into git, if it is not already.
  • Add the program as a chunk in (a stratum in) a Baserock system morph file
  • Add the program's build dependencies to the system as required until it builds

More detailed steps

  1. Start with a working VM. Prove it's working by building something, eg base-system.

  2. Fetch the definitions repo (or use an existing clone).

     $ git clone git://git.baserock.org/baserock/baserock/definitions
 $ cd definitions
 $ git checkout -b my-feature-branch

  3. Then retrieve the source you're working with, in another directory.

    • If the source came as git, great.
    • If the upstream is not git, you'll need to get it into git - either by creating your own git repo, or with Lorry, or other magic.

  4. In the definitions clone:

    • create your own system file based on one of the templates in systems/*
    • add a new stratum to it
    • create your new stratum file
    • add new chunk(s) in that for your component(s), specifying repo as "url://to/upstream/repo.git" or "file:///src/your-git-repo"
    • the ref field can be any valid Git ref (such as 'master') while prototyping. Current policy is that in definitions.git master the ref field must point to a fixed commit SHA1 so that it doesn't change unexpectedly.

  5. Now you can try

     $ morph build systems/your-new-system.morph

Unless the build system of the software you're integrating follows a known pattern, you will also need to write a 'chunk morphology' describing how to build it. You will get an error 'couldn't find morphology' if you need to do this. There are plenty of existing chunk morphologies in various subdirectories of the 'strata/' directory in definitions.git.

There is an informal specification of the definitions format.

See build-failures for help on build failures

Now what?

See guides and try adding your favourite software to Baserock. Get in contact.

If you are planning to develop Morph itself, see using-latest-morph.