Baserock/ guides/ Deploying to hardware using pxeboot.write and a Installer system

Using Baserock

Community

Background

Using pxeboot.write and an installer system to deploy to hardware

The pxeboot.write extension exists to make a Computer netboot a Baserock system using PXE. If this extension is used to netboot a installer system, it would be doing a remote deployment in hardware.

This tutorial shows some examples about how you can use the pxeboot.write extension to achieve that goal (on hardware deployment). First of all it would be useful for the reader to execute the following command to get some information about the available modes in this extension:

# morph help pxeboot.write

NOTE: to run this command and follow the tutorial you should be in a definitions checkout (since the write extension is in definitions.git). If you don't have a checkout yet you may find the quick start document useful.

In this tutorial I assume that the target hardware is IPMI accessible and its credentials are:

  • User: Admin
  • Password: PASSWD

And the target machine has:

  • IPMI IP: 172.23.10.123
  • MAC Ethernet: 88:99:AA:BB:CC:DD

Deploying into hardware using the 'spawn-novlan' mode

After reading the help of the pxeboot.write command, you will know that the only option needed in the cluster morphology for this deployment is:

  • PXEBOOT_DEPLOYER_INTERFACE

This deployment assumes that you have exclusive access to the target interface, or that it's vlanned by your infrastructure team.

In my case the target interface is already vlanned in the VLAN7 so as the help suggested, we need to create a pre-vlanned network interface in our Baserock system.

From a Baserock VM

If I want to do this froma Baserock VM, I need to create a network bridge into it to a pre-vlanned interface in my host using macvtap.

First of all I have to get macvtap working (I discovered this later after seen that it wasn't working). In my host (Debian) running the following will enable it:

$ sudo modprobe macvtap

After enabling macvtap I have to create a pre-vlanned interface in the host. To do that you can use vconfig like this:

$ sudo vconfig add eth0 7
Added VLAN with VID == 8 to IF -:eth0:-

To confirm that it has been created you can use ifconfig to check that the interface eth0.7 is present:

$ sudo ifconfig -a
eth0.7    Link encap:Ethernet  HWaddr f0:de:f1:43:3f:94  
          inet6 addr: fe80::f2de:f1ff:fe43:3f94/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:600925 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1392078 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:79128651 (75.4 MiB)  TX bytes:1835481577 (1.7 GiB)

Once I've enabled macvtap and created the pre-vlanned interface, I only have to configure the new interface in my Baserock VM. I'm currently using libvirt. Configuring the new interface is easy to do with virt-manager. To do it I opened the details of the Baserock VM, click on 'Add Hardware', select 'Network', select in 'Host device' the eth0.7 interface, and click on 'Finish'.

After this a new network device should appear in your Baserock VM, you can check that out using ifconfig once again, but this time in the Baserock VM.

# ifconfig -a
ens3      Link encap:Ethernet  HWaddr 52:54:00:31:E8:96  
          inet addr:192.168.122.146  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe31:e896/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:52346 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43128 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8074706 (7.7 MiB)  TX bytes:6054547 (5.7 MiB)

ens6      Link encap:Ethernet  HWaddr 52:54:00:87:93:4F  
          inet6 addr: fe80::5054:ff:fe87:934f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2811 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2833 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:856678 (836.5 KiB)  TX bytes:899042 (877.9 KiB)

In my case I can see a new network interface in my VM called ens6. This is the value that we have to use for PXEBOOT_DEPLOYER_INTERFACE

From Baserock running on Hardware

Doing the deployment from Baserock on Hardware involves much less magic. Assuming that we are in the same case as before (Target vlanned in the VLAN7), we can easily create a network interface in the same vlan with vconfig.

First of all let's figure out the name of our network interface using ifconfig:

# ifconfig -a
enp1s0    Link encap:Ethernet  HWaddr 00:04:4B:25:BE:31  
          inet addr:10.24.2.142  Bcast:10.24.255.255  Mask:255.255.0.0
          inet6 addr: fe80::204:4bff:fe25:be31/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:520978 errors:0 dropped:118 overruns:0 frame:0
          TX packets:1245797 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:79930275 (76.2 MiB)  TX bytes:1654500960 (1.5 GiB)

And now we can create one in the VLAN7 using vconfig, and check it later with ifconfig:

# vconfig add enp1s0 7
# ifconfig -a
...
enp1s0.7  Link encap:Ethernet  HWaddr 00:04:4B:25:BE:31  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:483554 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1238887 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:68894553 (65.7 MiB)  TX bytes:1653753896 (1.5 GiB)

In my case I can see a new network interface called enp1s0.7. This is the value that we would have to use for PXEBOOT_DEPLOYER_INTERFACE/

The deployment using 'spawn-novlan'

At this point we should have everything ready to try a pxeboot deployment. So let's see first the example available in definitions.git and how can we modify it for our deployment.

# cat clusters/hardware-deployment.morph
name: hardware-deployment
kind: cluster
description: |
  Deploy a build-system into hardware using the combination
  of the extensions/pxeboot.write extension and the installer
  system. This examples uses the spawn-novlan mode of
  extensions/pxeboot.write.
systems:
- morph: systems/installer-system-x86_64.morph
  deploy:
    installer:
      type: extensions/pxeboot
      location: AB:CD:EF:12:34:56:78   #MAC address.
      PXEBOOT_MODE: spawn-novlan
      PXEBOOT_DEPLOYER_INTERFACE: ens6
      DISK_SIZE: 6G
      KERNEL_ARGS: console=ttyS1,9600 console=tty0 init=/usr/lib/baserock-installer/installer
      HOSTNAME: installer-system
      IPMI_USER: myipmiuser
      IPMI_PASSWORD: myipmipassword
      IPMI_HOST: 123.34.45.120         #IPMI ip address
      INSTALLER_TARGET_STORAGE_DEVICE: /dev/sda
      INSTALLER_ROOTFS_TO_INSTALL: /rootfs
  subsystems:
  - morph: systems/build-system-x86_64.morph
    deploy:
      to-install:
        type: extensions/sysroot
        location: /rootfs
        INITRAMFS_PATH: boot/initramfs.gz
        KERNEL_ARGS: console=ttyS1,9600 console=tty0
    subsystems:
    - morph: systems/initramfs-x86_64.morph
      deploy:
        initramfs:
          type: extensions/initramfs
          location: boot/initramfs.gz

This cluster is deploying, using extensions/pxeboot.write, an installer system. This installer system has a subsystem which is the system that the installer system will install. In my case I want to install a build system, so I'll leave it as is.

NOTE: to understand better how the installer system works you can see an example in definitions.git: 'clusters/installer-build-system-x86_64.morph'

Let's have a look at what we need to modify to do the on-hardware deployment.

The PXEBOOT_MODE is already set to the one that we want to use (spawn-novlan), and PXEBOOT_DEPLOYER_INTERFACE has already the name of my pre-vlanned interface, so I don't have to change them. I'll only have to change IPMI_USER, IPMI_PASSWORD, IPMI_HOST and location:

name: hardware-deployment
kind: cluster
systems:
- morph: systems/installer-system-x86_64.morph
  deploy:
    installer:
      type: extensions/pxeboot
      location: 88:99:AA:BB:CC:DD
      PXEBOOT_MODE: spawn-novlan
      PXEBOOT_DEPLOYER_INTERFACE: ens6
      DISK_SIZE: 6G
      KERNEL_ARGS: console=ttyS1,9600 console=tty0 init=/usr/lib/baserock-installer/installer
      HOSTNAME: installer-system
      IPMI_USER: Admin
      IPMI_PASSWORD: PASSWD
      IPMI_HOST: 172.23.10.123
      INSTALLER_TARGET_STORAGE_DEVICE: /dev/sda
      INSTALLER_ROOTFS_TO_INSTALL: /rootfs
  subsystems:
  - morph: systems/build-system-x86_64.morph
    deploy:
      to-install:
        type: extensions/sysroot
        location: /rootfs
        INITRAMFS_PATH: boot/initramfs.gz
        KERNEL_ARGS: console=ttyS1,9600 console=tty0
    subsystems:
    - morph: systems/initramfs-x86_64.morph
      deploy:
        initramfs:
          type: extensions/initramfs
          location: boot/initramfs.gz

Now we can try to deploy it but without forgetting to build the systems involved in this deployment first:

# morph build systems/installer-system-x86_64.morph
...
# morph build systems/build-system-x86_64.morph
...
# morph build systems/initramfs-x86_64.morph
...
# morph deploy clusters/hardware-deployment.morph
2014-12-18 16:19:11 [systems/installer-system-x86_64.morph][installer]ERROR: nfs-server.service is not running and is needed for this deployment. Please, run `systemctl start nfs-server` and try `morph deploy` again.
ERROR: pxeboot.check failed with code 1:

And it failed because this deployment needs to use a local nfs server. Reading the error message you can figure out what you need to do to run it.

# systemctl start nfs-server

And now we try again.

# morph deploy clusters/hardware-deployment.morph
2014-12-18 17:15:56 [systems/installer-system-x86_64.morph]Updating cached git repository baserock:baserock/installer-scripts for ref master
2014-12-18 17:16:05 [systems/installer-system-x86_64.morph][installer]Unpacking system for configuration
2014-12-18 17:16:46 [systems/installer-system-x86_64.morph][installer]System unpacked at /src/tmp/deployments/tmpJt3PKp/tmpQf1Fbi
2014-12-18 17:16:46 [systems/installer-system-x86_64.morph][installer]Writing deployment metadata file
2014-12-18 17:17:07 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]Unpacking system for configuration
2014-12-18 17:17:50 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]System unpacked at /src/tmp/deployments/tmpJt3PKp/tmp7wrYBz
2014-12-18 17:17:50 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]Writing deployment metadata file
2014-12-18 17:17:52 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]Unpacking system for configuration
2014-12-18 17:18:01 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]System unpacked at /src/tmp/deployments/tmpJt3PKp/tmp92HtOW
2014-12-18 17:18:01 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]Writing deployment metadata file
2014-12-18 17:18:01 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]Configure system
2014-12-18 17:18:01 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]Writing to device
2014-12-18 17:21:27 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install][systems/initramfs-x86_64.morph][initramfs]Cleaning up
2014-12-18 17:21:27 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]Configure system
2014-12-18 17:21:30 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]Writing to device
2014-12-18 17:21:30 [systems/installer-system-x86_64.morph][installer][systems/build-system-x86_64.morph][to-install]Cleaning up
2014-12-18 17:21:30 [systems/installer-system-x86_64.morph][installer]Configure system
2014-12-18 17:21:30 [systems/installer-system-x86_64.morph][installer]Configuration of the installer system in /src/tmp/deployments/tmpJt3PKp/tmpQf1Fbi/etc/install.conf
2014-12-18 17:21:31 [systems/installer-system-x86_64.morph][installer]Writing to device
2014-12-18 17:21:32 [systems/installer-system-x86_64.morph][installer]Assigning ip address 172.16.0.1/30 to iface ens6
2014-12-18 17:21:33 [systems/installer-system-x86_64.morph][installer]Bringing interface ens6 up
2014-12-18 17:21:33 [systems/installer-system-x86_64.morph][installer]Created tempdir: /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_
2014-12-18 17:21:33 [systems/installer-system-x86_64.morph][installer]Bound listen socket to 172.16.0.1, 69
2014-12-18 17:21:33 [systems/installer-system-x86_64.morph][installer]Serving tftp root /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_, on port 69
2014-12-18 17:21:33 [systems/installer-system-x86_64.morph][installer]Installing /usr/share/syslinux/pxelinux.0 to /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_/pxelinux.0
2014-12-18 17:21:34 [systems/installer-system-x86_64.morph][installer]Installing /src/tmp/deployments/tmpJt3PKp/tmpQf1Fbi/boot/vmlinuz to /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_/kernel
2014-12-18 17:21:35 [systems/installer-system-x86_64.morph][installer]Exporting 172.16.0.2:/src/tmp/deployments/tmpJt3PKp/tmpQf1Fbi as local nfsroot
2014-12-18 17:21:37 [systems/installer-system-x86_64.morph][installer]Set Boot Device to pxe
2014-12-18 17:21:37 [systems/installer-system-x86_64.morph][installer]Chassis Power Control: Reset
2014-12-18 17:21:37 [systems/installer-system-x86_64.morph][installer]Please wait for the system to install, then enter "installed"

At this point the Target system would be netbooting the installer system from my VM. It would be very helpful to see what's going on in the target machine either looking at the screen of it, or using ipmitool:

# ipmitool -I lanplus -H 172.23.10.123 -U Admin -P PASSWS sol activate

Once we know that the installation has finished, we can type installed in the console where we were doing the deployment to finish it:

2014-12-18 17:21:37 [systems/installer-system-x86_64.morph][installer]Please wait for the system to install, then enter "installed"
installed
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Chassis Power Control: Reset
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Created tempdir: /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpt1MlqQ
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Removing 172.16.0.2:/src/tmp/deployments/tmpJt3PKp/tmpQf1Fbi from local nfsroots
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Removing /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_/kernel
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Removing /src/tmp/deployments/tmpJt3PKp/tmpjtInEo/tmpCpKSV_/pxelinux.0
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Killing tftpd listener pid=6919
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Bringing interface ens6 down
2014-12-18 17:31:07 [systems/installer-system-x86_64.morph][installer]Removing ip addresses from iface ens6
2014-12-18 17:31:08 [systems/installer-system-x86_64.morph][installer]Cleaning up
2014-12-18 17:31:18 Finished deployment

Now the target machine will be rebooted and it will hopefully boot a Baserock build system.