A few days ago I decided to create a VPS, known as a “droplet”, with DigitalOcean. They claim a deployment time of 55 seconds. And 55 seconds after hitting the button I had a Debian 7 x64 droplet running. The plan was to migrate my current VPS to this DigitalOcean droplet. The first task I always undertake with any Linux deployment is to create a custom stripped down kernel patched with grsecurity. However, unknown to me, the way DigitalOcean boot your droplet with KVM means that you can only use a kernel of their choice.

But don’t panic because there is a workaround by utilising kexec. kexec is used to speed up reboots. When you power on any machine it goes through a POST procedure to initialise the hardware. When you reboot a machine it goes through this POST procedure again which really isn’t needed if the hardware is already initialised. Normally kexec is invoked during the reboot runlevel to load a kernel in to memory and jump straight in to it, bypassing the hardware initialisation. The trick is to reverse this and have the machine utilise kexec during the startup runlevel, therefore jumping in to the custom kernel straight away and only utilising DigitalOcean’s choice of kernel as a bootstrap.

I’ve seen a lot of hacks where people suggest modifying the init scripts such as /etc/init.d/rcS to kexec the custom kernel on boot before any init scripts are executed. But this is very hacky and you could easily end up in a situation where the custom kernel doesn’t boot but because there’s no way to stop kexec running you could have a completely unusable droplet.

In my opinion the correct way to do this is to write a proper LSB init script. Note that the below is specific to Debian and Ubuntu. The init script should also give you the option to abort the kexec process in case something goes wrong. At least that way the droplet will still boot with DigitalOcean’s kernel and you can get in and fix things.

The LSB init script is as follows:

#!/bin/sh

### BEGIN INIT INFO
# Provides:          droplet-kernel
# Required-Start:
# Required-Stop:
# Should-Start:      glibc
# Default-Start:     S
# Default-Stop:      6
# X-Interactive:     true
# Short-Description: Run kexec on DigitalOcean droplet
# Description:       Runs kexec on a DigitalOcean droplet to boot a custom kernel
### END INIT INFO

PATH=/sbin:/bin:/usr/sbin:/usr/bin

. /lib/lsb/init-functions

test -r /etc/default/droplet-kernel && . /etc/default/droplet-kernel

do_stop() {
        # Don't do anything if kexec-tools are not installed
        # or droplet-kernel is not enabled in defaults file.
        test -x /sbin/kexec || exit 0
        test "$ENABLED" = "true" || exit 0

        # Check 'kexeced' kernel cmdline is present otherwise droplet
        # wasn't booted with a custom kernel via kexec.
        if grep -q ' kexeced$' /proc/cmdline; then

                # Remove 'kexeced' cmdline arguement so that when the droplet
                # is rebooted it will load and boot the custom kernel again.
                cat /proc/cmdline | sed 's/ kexeced$//' > /root/cmdline
                mount --bind -n -o ro /root/cmdline /proc/cmdline >/dev/null
                kexec -u

                log_action_msg "Removed 'kexeced' kernel cmdline from droplet"
        else
                log_action_msg "Droplet was not booted with custom kernel"
        fi
}

do_start() {
        # Don't do anything if kexec-tools are not installed
        # or droplet-kernel is not enabled in defaults file.
        test -x /sbin/kexec || exit 0
        test "$ENABLED" = "true" || exit 0

        do_status

        # Check 'kexeced' kernel cmdline is not present.
        # If it is, the droplet has already booted with kexec. This helps
        # prevent loops.
        if grep -qv ' kexeced$' /proc/cmdline; then

                # Give the option to abort booting the droplet using kexec.
                export KEXEC_ABORT=false
                trap "export KEXEC_ABORT=true" 2
                log_begin_msg "Press Ctrl+C to abort booting droplet with custom kernel"
                sleep 10
                trap - 2
                log_end_msg 0

                REAL_APPEND="$APPEND"
                test -z "$REAL_APPEND" && REAL_APPEND="`cat /proc/cmdline`"

                if [ "$KEXEC_ABORT" = "false" ]; then
                        log_action_begin_msg "Loading new kernel in to droplet memory"
                        if [ -z "$INITRD" ]; then
                                kexec --load "$KERNEL_IMAGE" --append="$REAL_APPEND kexeced"
                        else
                                kexec --load "$KERNEL_IMAGE" --initrd="$INITRD" --append="$REAL_APPEND kexeced"
                        fi
                        log_action_end_msg $?

                        log_action_begin_msg "Attempting to run droplet with custom kernel"
                        kexec -e
                        log_action_end_msg $?
                fi
        fi
}

do_status() {
        if [ "$ENABLED" != "true" ]; then
                log_action_msg "Custom droplet kernel is NOT enabled"
                exit 0
        fi

        log_action_msg "Custom droplet kernel is enabled"

        if grep -q 'kexeced$' /proc/cmdline; then
                log_action_msg "Droplet was booted with a custom kernel"
        else
                log_action_msg "Droplet was NOT booted with a custom kernel"
        fi
}

case "$1" in
  start)
        do_start
        ;;
  restart|reload|force-reload)
        echo "Error: argument '$1' not supported" >&2
        exit 3
        ;;
  stop)
        do_stop
        ;;
  status)
        do_status
        ;;
  *)
        echo "Usage: $0 {start|stop|status}" >&2
        exit 3
        ;;
esac
exit 0

The above should be placed in /etc/init.d/droplet-kernel and chmod 755.

In addition you need /etc/default/droplet-kernel:

# Defaults for droplet-kernel initscript
# sourced by /etc/init.d/droplet-kernel

# Load a custom kernel for the droplet (true/false)
ENABLED=true

# Kernel and initrd image.
# If no initrd image is needed, leave blank.
KERNEL_IMAGE="/vmlinuz"
INITRD="/initrd.img"

# If empty, use current /proc/cmdline
APPEND=""

Now use update-rc.d to install the init script:

$ sudo update-rc.d droplet-kernel defaults

Now when the droplet boots, one of the first thing it does is look at running kexec to load and boot a custom kernel. The /etc/defaults/droplet-kernel file contains all the customisable options, such as easily enabling/disabling this process, where the custom kernel and initrd images are and the option to override the kernel arguments (such as rootfs, verbosity etc).

The script runs interactive and will hold the boot routine for 10 seconds, giving you the option to press Ctrl+C to abort the kexec process. This is especially useful if something has gone wrong with the kernel that will be kexec’d. Pressing Ctrl+C will mean the droplet continues to boot using DigitalOcean’s kernel and you should be able to fix things up.

You’ll notice that when the droplet is kexec’d, it appends “kexeced” to the kernel cmdline argument. It does this to prevent any loops from occurring. The init script checks if “kexeced” is the last argument and if it is won’t try to kexec the kernel. Otherwise the droplet would get in to an endless reboot.

When the droplet is rebooted (runlevel 6), it removes the “kexeced” argument so that we get the option to Ctrl+C during the bootup again. This avoids the need to shutdown the droplet and power it on again for kexec to kick in.

Here it is in action: