Moonshot Distbuild Cluster

This documentation follows on from the generic distbuild introduction, and describes the process involved in creating a distbuild network of Jetson-TK1 boards.

Preconditions:

1. You have setup a development environment on a moonshot node as per the instructions at Setup an ARMv8 Baserock system on an HP Moonshot node
2. You have a trove that you have push/pull access to (in this documentation we will refer to this as TROVE_ID and TROVE_HOST), please refer to Deploy Trove to netboot server on Moonshot if a trove is needed within Moonshot.
3. You have another (ideally more) Moonshot nodes to deploy the distbuild image to

Note: We are experiencing some nfs-server collision and network strom issues when having a trove running within Moonshot. The reason is unknown, as set up Trove is not the focus of our current task, we haven't spend too much time to figure out why. You can workaround it by stopping nfs-server, nfsboot-tftp and lorry services on Trove.

Build the Moonshot distbuild image

Followed the instructions on Quick Start onwards to setup morph and your second hard drive correctly.

We'll also need to upgrade to the latest morph version, see http://wiki.baserock.org/using-latest-morph/

Now run following commend to build, assuming you are in definition directory!

morph --verbose build systems/build-system-armv8l64.morph

Once this is done, you'll be ready to deploy.

Deploying a distbuild network netbooting on Moonshot

You'll now want to create a distbuild cluster morph, which will deal with generating the controller and worker nodes needed.

In this example we used 3 node, one as controller, and other two as workers. Make moonshot-distbuild.morph with following content:

name: moonshot-distbuild-armv8l64
kind: cluster
description: |
  Deploy a distbuild network onto HP Moonshot node.

  The distbuild network will contain one controller node and two worker nodes.
systems:
- morph: systems/build-system-armv8l64.morph
  deploy-defaults:
    CONTROLLERHOST: controller
    DISTBUILD_CONTROLLER: false
    DISTBUILD_WORKER: true
    PXEBOOT_MODE: existing-server
    PXEBOOT_CONFIG_TFTP_ADDRESS: sftp://$tftp-server/srv/nfsboot/tftp/
    PXEBOOT_ROOTFS_RSYNC_ADDRESS: rsync://$nfs-server/srv/nfsboot/
    KERNEL_ARGS: console=ttyS0,9600n8r rw
    DTB_PATH: $dtb-path
    MOONSHOT_KERNEL: yes
    FSTAB_SRC: /dev/sda2 /srv/distbuild auto defaults,rw,noatime 0 2
    INSTALL_FILES: "moonshot/manifest distbuild/manifest"
    INSTALLER: no
    TROVE_ID: $trove-id
    TROVE_HOST: $trove-host
    WORKER_SSH_KEY: /src/ssh_keys/worker.key
    WORKER_SSH_PUBKEY: /src/ssh_keys/worker.key.pub
  deploy:
    build-controller:
      type: extensions/pxeboot
      location: $macaddr-controller-node
      DISTBUILD_CONTROLLER: true
      DISTBUILD_WORKER: false
      HOSTNAME: controller
      WORKERS: worker0, worker1
    build-node-1:
      type: extensions/pxeboot
      location: $macaddr-worker0
      HOSTNAME: worker0
    build-node-2:
      type: extensions/pxeboot
      location: $macaddr-worker1
      HOSTNAME: worker1

To deploy use morph deploy:

morph deploy moonshot-distbuild.morph

This will deploy the systems to netboot server, once this is done power on relevant node will make them boot from the right rootfs.

Using the distbuild network

To run a build using your distbuild network:

morph distbuild --controller-initiator-address=$controller-host <system>

Note: you can run distbuild commend from any devel machine which have write access to trove and can access the controller node.

Distbuild places build logs for each chunk in the current working directory, for example the log for a build of pango-misc would be stored in build-step-pango-misc.log, so you could view build progress using:

tailf build-step-pango-misc.log