Wednesday, September 28, 2011

Preparing and storing patch revisions as git tags using git-publish

This weekend I got down to solving a workflow problem that has been bugging me for some time: preparing and storing patch revisions. Manually managing patch revisions is painful; I often find myself switching between git and my inbox several times to put together a consistent patch series.

git-publish is a script that numbers patch revisions, optionally stores a cover letter, and submits the patches via git-send-email(1). When your tree is in a state that you wish to publish you say:

$ git publish

It creates a git tag that you can refer back to in the future and send out the patch series emails.

No more numbering revisions, copy & pasting cover letters, or running several steps to format and send patch series.

Give it a try if you are tired of manually managing patch revisions with git. git-publish is released under the MIT License at I have provided documentation but you can set it up in just two lines:

$ git clone git://
$ git-publish/git-publish --setup # make available via git alias

Be sure to check out the README - it explains how to install and run it in more detail.

Happy git-publishing!

Monday, September 19, 2011

Enhanced VMDK support now in QEMU

QEMU now has greatly enhanced VMDK VMware disk image file support, thanks to Fam Zheng's hard work during Google Summer of Code 2011. Previously QEMU was only able to handle older VMDK files because it did not support the entire VMDK file format specification. This resulted in qemu-img convert and other tools being unable to open certain VMDK files. As of now, qemu.git has merged code to handle the VMDK specification and work well with modern image files.

If you had trouble in the past manipulating VMDK files with qemu-img, it may be worth another look soon. You can already build the latest and greatest qemu-img from the qemu.git repository and distros will provide packages with full VMDK support in the future:

$ git clone git://
$ cd qemu
$ ./configure
$ make qemu-img
$ ./qemu-img convert ...

It is still recommended to convert VMDK files to QEMU's native formats (raw, qcow2, or qed) in order to get optimal performance for running VMs.

At the end of Fam's Summer of Code project, he put together an article on gotchas and undocumented behavior in the VMDK specification. This will be of great interest to developers writing their own code to manipulate VMDK image files. His experience this summer involved testing a wide range of VMware software, real-world image files, as well as studying existing open-source VMDK code. The VMDK specification is ambiguous in places and does not cover several essential details, so Fam had to figure them out himself and he then documented them.

So with that, happy VMDK-ing...

Sunday, September 11, 2011

How to share files instantly between virtual machines and host

It's pretty common to need to copy files between a virtual machine and the host. This can be drivers or installers from the host into the virtual machine or it could be in order to get some data out of the virtual machine and onto the host.

There are several solutions to sharing files, like network file systems, third-party file hosting, or even the virtfs paravirtualized file system supported by KVM. But my favorite ad-hoc file sharing tool is Python's built-in webserver.

The one-liner that shares files over HTTP

To share some files, simply change into the directory subtree you want to export and start the web server:

$ cd path/to/shared/files && python -m SimpleHTTPServer
Serving HTTP on port 8000 ...

The directory hierarchy at path/to/shared/files is now available over HTTP on all IPs for the machine on port 8000. The web server generates index listings for directories so it is easy to browse.

To access the host from a virtual machine:

  • User networking (slirp):
  • NAT networking: the default gateway from ip route show | grep ^default, for example
  • Bridged or routed networking: the IP address of the host

To access the virtual machine from the host:

  • NAT, bridged, or routed networking: the virtual machine's IP address
  • User networking (slirp): forward a port with the hostfwd_add QEMU monitor command or -net user,hostfwd= option documented in the QEMU man page


There are a couple of reasons why I like Python's built-in web server:

  • No need to install software since Linux and Mac typically already have Python installed.
  • No privileges are required to run the web server.
  • Works for both physical and virtual machines.
  • HTTP is supported by desktop file managers, browsers, and the command-line with wget or curl.
  • Network booting and installation is possible straight from the web server. Be warned that RHEL installs over HTTP have been known to fail with SimpleHTTPServer but I have not encountered problems with other software.

Security tips

This handy HTTP server is only suitable for trusted networks for a couple of reasons:

  • Encryption is not supported so data travels in plain text and can be tampered with in flight.
  • The directory listing feature means you should only export directory subtrees that contain no sensitive data.
  • SimpleHTTPServer is not a production web server and there could be obvious bugs.


Python's built-in web server is one of my favorite tricks that not many people seem to know. I hope this handy command comes in useful to you!

Wednesday, September 7, 2011

QEMU Internals: vhost architecture

This post explains how vhost provides in-kernel virtio devices for KVM. I have been hacking on vhost-scsi and have answered questions about ioeventfd, irqfd, and vhost recently, so I thought this would be a useful QEMU Internals post.

Vhost overview

The vhost drivers in Linux provide in-kernel virtio device emulation. Normally the QEMU userspace process emulates I/O accesses from the guest. Vhost puts virtio emulation code into the kernel, taking QEMU userspace out of the picture. This allows device emulation code to directly call into kernel subsystems instead of performing system calls from userspace.

The vhost-net driver emulates the virtio-net network card in the host kernel. Vhost-net is the oldest vhost device and the only one which is available in mainline Linux. Experimental vhost-blk and vhost-scsi devices have also been developed.

In Linux 3.0 the vhost code lives in drivers/vhost/. Common code that is used by all devices is in drivers/vhost/vhost.c. This includes the virtio vring access functions which all virtio devices need in order to communicate with the guest. The vhost-net code lives in drivers/vhost/net.c.

The vhost driver model

The vhost-net driver creates a /dev/vhost-net character device on the host. This character device serves as the interface for configuring the vhost-net instance.

When QEMU is launched with -netdev tap,vhost=on it opens /dev/vhost-net and initializes the vhost-net instance with several ioctl(2) calls. These are necessary to associate the QEMU process with the vhost-net instance, prepare for virtio feature negotiation, and pass the guest physical memory mapping to the vhost-net driver.

During initialization the vhost driver creates a kernel thread called vhost-$pid, where $pid is the QEMU process pid. This thread is called the "vhost worker thread". The job of the worker thread is to handle I/O events and perform the device emulation.

In-kernel virtio emulation

Vhost does not emulate a complete virtio PCI adapter. Instead it restricts itself to virtqueue operations only. QEMU is still used to perform virtio feature negotiation and live migration, for example. This means a vhost driver is not a self-contained virtio device implementation, it depends on userspace to handle the control plane while the data plane is done in-kernel.

The vhost worker thread waits for virtqueue kicks and then handles buffers that have been placed on the virtqueue. In vhost-net this means taking packets from the tx virtqueue and transmitting them over the tap file descriptor.

File descriptor polling is also done by the vhost worker thread. In vhost-net the worker thread wakes up when packets come in over the tap file descriptor and it places them into the rx virtqueue so the guest can receive them.

Vhost as a userspace interface

One surprising aspect of the vhost architecture is that it is not tied to KVM in any way. Vhost is a userspace interface and has no dependency on the KVM kernel module. This means other userspace code, like libpcap, could in theory use vhost devices if they find them convenient high-performance I/O interfaces.

When a guest kicks the host because it has placed buffers onto a virtqueue, there needs to be a way to signal the vhost worker thread that there is work to do. Since vhost does not depend on the KVM kernel module they cannot communicate directly. Instead vhost instances are set up with an eventfd file descriptor which the vhost worker thread watches for activity. The KVM kernel module has a feature known as ioeventfd for taking an eventfd and hooking it up to a particular guest I/O exit. QEMU userspace registers an ioeventfd for the VIRTIO_PCI_QUEUE_NOTIFY hardware register access which kicks the virtqueue. This is how the vhost worker thread gets notified by the KVM kernel module when the guest kicks the virtqueue.

On the return trip from the vhost worker thread to interrupting the guest a similar approach is used. Vhost takes a "call" file descriptor which it will write to in order to kick the guest. The KVM kernel module has a feature called irqfd which allows an eventfd to trigger guest interrupts. QEMU userspace registers an irqfd for the virtio PCI device interrupt and hands it to the vhost instance. This is how the vhost worker thread can interrupt the guest.

In the end the vhost instance only knows about the guest memory mapping, a kick eventfd, and a call eventfd.

Where to find out more

Here are the main points to begin exploring the code:
  • drivers/vhost/vhost.c - common vhost driver code
  • drivers/vhost/net.c - vhost-net driver
  • virt/kvm/eventfd.c - ioeventfd and irqfd
The QEMU userspace code shows how to initialize the vhost instance:
  • hw/vhost.c - common vhost initialization code
  • hw/vhost_net.c - vhost-net initialization