Monday, 21 February 2011

Near instant kernel development cycle with KVM

I want to share my setup for rapid kernel development using KVM.

A fast development cycle makes a huge difference to productivity. For firmware and kernel development many areas can be efficiently tested inside virtual machines.

Traditionally physical test machines were used but virtualization lets you take the lab with you. This means working offline without giving up on testing.

In that past I used QEMU when working on the gPXE network bootloader. Now I am using KVM to test Linux kernel changes in less than 30 seconds and it's a really pleasant setup.

What can't be tested under KVM?

A lot of code can be tested in a virtual machine but device drivers or hardware-specific code often require physical machines. But with PCI device assignment, or passing physical PCI devices through into the virtual machine, it is becoming possible to test device drivers in a virtual machine too.

Testing kernels without disk images

Most virtual machines are booted from a disk image or an ISO file, but KVM can directly load a Linux kernel into memory skipping the bootloader. This means you don't need an image file containing the kernel and boot files. Instead, you can run a kernel directly like this:

qemu-kvm -kernel arch/x86/boot/bzImage -initrd initramfs.gz -append "console=ttyS0" -nographic

These flags directly load a kernel and initramfs from the host filesystem without the need to generate a disk image or configure a bootloader.

The optional -initrd flag loads an initramfs for the kernel to use as the root filesystem.

The -append flags adds kernel parameters and can be used to enable the serial console.

The -nographic option restricts the virtual machine to just a serial console and therefore keeps all test kernel output in your terminal rather than in a graphical window.

Building an initramfs

I don't use a distro initramfs generation utility because I like to control which files get included and the init script. Instead I use the linux-2.6/usr/gen_init_cio utility to build an initramfs cpio archive from a specification file. A neat feature of gen_init_cpio is that you don't need to be root in order to create device files or set ownership inside the initramfs. The specification file syntax looks like this:

# a comment
file <name> <location> <mode> <uid> <gid> [<hard links>]
dir <name> <mode> <uid> <gid>
nod <name> <mode> <uid> <gid> <dev_type> <maj> <min>
slink <name> <target> <mode> <uid> <gid>
pipe <name> <mode> <uid> <gid>
sock <name> <mode> <uid> <gid>

The kernel will execute the file at /init. I include busybox in the initramfs and have the following script:

mount -t proc none /proc
mount -t sysfs none /sys
mount -t configfs none /sys/kernel/config
mount -t debugfs none /sys/kernel/debug
mount -t tmpfs none /tmp

# Test setup commands here:
insmod /lib/modules/$(uname -r)/kernel/...

exec /bin/sh -i

Instead of building out a full /lib/modules directory tree I just include those kernel module dependencies that I need. This means I use insmod(8) instead of modprobe(8) because I skip generating depmod(8) dependency metadata.

Tying it all together

Here are the steps I take to build and test a kernel:

cd linux-2.6
[...make some changes...]
usr/gen_init_cpio initramfs | gzip >initramfs.gz
qemu-kvm -kernel arch/x86/boot/bzImage -initrd initramfs.gz -append "console=ttyS0" -nographic

It takes about 28 seconds to the shell prompt inside the virtual machine with ccache and a hot page cache on this laptop. This keeps development fun :)!


  1. Hi Stefan,

    Very useful info, thanks! Can you please paste the entire input to gen_init_cpio ('cat initramfs' or something)? Obviously you create some directories there, but I'd like to see the "include busybox" part and the way you put modules in there..

  2. Creating the initramfs manually is a bit messy. Which files you need depends on your distro (e.g. library dependency paths). I just scraped up busybox from my Debian testing system:

    It should be similar on your system except the detailed library paths and versions can be different.

    Or you could try using a initramfs creator tool like mkinitrd - distros seem to have their own custom tools for this, they all basically do the same thing.

  3. Hi Stefan, I get a kernel panic following your post, it tells me "Ateempt to kill init" ... not sure why this happens or what this means...

    1. Sounds like your /init script is terminating. Linux expects your init process to run forever - otherwise you'll get the error you mentioned.

      In my example the script's final line is "exec /bin/sh -i" so an interactive shell is spawned.

      What's the final line of your /init script? Are you sure the script is running successfully?

  4. Hi Stefan, Can an OS, such as Ubuntu 14, pass its Loadable Kernel Module to a guest OS, using KVM, in order to access restricted hardware such as MSR? I have done the steps to build a custom kernel. But, lsmod or dmesg does not show the new custom kernel. Could you suggest a technique to make sure that the kernel is loaded onto the guest VM please?

    1. You can use qemu -kernel and -initrd options to boot a kernel from the host file system.

      Regarding accessing restricteded hardware (e.g. MSRs), I'm not sure what you're trying to do. Please email if you have questions about this.