Saturday, 9 April 2011

How to pass QEMU command-line options through libvirt

An entire virtual machine configuration can be passed on QEMU's extensive
command-line, including everything from PCI slots to CPU features to serial
port settings. While defining a virtual machine from a monster
command-line may seem insane, there are times when QEMU's rich command-line
options come in handy.

And at those times one wishes to side-step libvirt's domain XML and specify
QEMU command-line options directly. Luckily libvirt makes this possible and I
learnt about it from Daniel Berrange and Anthony Liguori on IRC. This libvirt
feature will probably come in handy to others and so I want to share it.

The <qemu:commandline> domain XML tag

There is a special namespace for QEMU-specific tags in libvirt domain XML. You
cannot use QEMU-specific tags without first declaring the namespace. To enable
it use the following:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Now you can add command-line arguments to the QEMU invocation. For example, to load an option ROM with -option-rom:
<qemu:commandline>
   <qemu:arg value='-option-rom'/>
   <qemu:arg value='path/to/my.rom'/>
</qemu:commandline>

It is also possible to add environment variables to the QEMU invocation:
<qemu:commandline>
   <qemu:env name='MY_VAR' value='my_value'/>
</qemu:commandline>

Setting qdev properties through libvirt

Taking this a step further we can set qdev properties through libvirt. There is no domain XML for setting the virtio-blk-pci ioeventfd qdev property. Here is how to set it using <qemu:arg> and the -set QEMU option:
<qemu:commandline>
  <qemu:arg value='-set'/>
  <qemu:arg value='device.virtio-disk0.ioeventfd=off'/>
</qemu:commandline>

The result is that libvirt generates a QEMU command-line that ends with -set device.virtio-disk0.ioeventfd=off. This causes QEMU to go back and set the ioeventfd property of device virtio-disk0 to off.

More information

The following libvirt wiki page documents mappings from QEMU command-line options to libvirt domain XML. This is extremely useful if you know which QEMU option to use but are unsure how to express that in domain XML.

That page also reveals the <qemu:commandline> tag and shows how it can be used to invoke QEMU with the GDB stub (-s).

10 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Hi ,

    I am trying to use the kvm-qemu command line option -incoming , to include a statefile with an existing image . But I cannot add it to the xml file and make it work .

    I tried adding the following ,

    "
    [open_angle_bracket]qemu:commandline[close_angle_bracket]
    [open_angle_bracket]qemu:arg value='-incoming'/[close_angle_bracket]
    [open_angle_bracket]qemu:arg value='\"exec:gzip -c STATEFILE.gz\"'/[close_angle_bracket]
    [open_angle_bracket]/qemu:commandline[close_angle_bracket]


    "

    but when i define it, it disappears. When I saw the wiki page document, I found that ,

    -incoming
    This is used internally by libvirt when performing an incoming migration, or restoring a VM from a save file. As such it is not configurable by the user

    You have any idea , how to get this working ? Thanks !

    - Sethu (sethusubbiah at gmail d.ot com)

    ReplyDelete
  3. Sethu, have you tried the "save" and "restore" virsh commands? They let you specify a save state filename and seem to do what you want.

    Here is how I save the state of my rhel-6 guest:
    virsh # save rhel-6 /var/tmp/rhel-6.save

    The /var/tmp/rhel-6.save file now contains the qemu-kvm save state as well as libvirt metadata. It can be restored later but you need to keep the disk image(s) for this guest untouched (or take a copy of them for restoring).

    Here is how i restore my rhel-6 guest:
    virsh # restore /var/tmp/rhel-6.save

    If your use case is not met by these commands then I suggest getting in touch with the libvirt developers on libvir-list@redhat.com.

    ReplyDelete
  4. G'day Stefan, may thanks for this post! I once again was looking for a way to tell libvirt to pass the -redir options to my VMs and your post was the difference. cheers!

    And for anyone else wanting to do this, see here:

    http://snippets.webaware.com.au/snippets/running-qemu-with-port-redirection-through-libvirt/

    ReplyDelete
  5. Little offline question. I would like to know how qemu acts as a container of guest VM. Any pointer will be of big help. Thanks. Great article.

    ReplyDelete
    Replies
    1. Hi Harish,
      I'm not sure if this is what you are looking for:
      http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html

      Delete
  6. Exactly what I was looking for. Very handy when introducing custom command line options to qemu, but still wanting to use libvirt for all the convenience it provides!

    ReplyDelete
  7. This help me resolve a problem. Thanks.
    Originally, I made a mistake to put those two lines into one line and got "invalid option" error. It took me a lot of time to try to figure out why. Now I know --set and option should be in two xml entries.
    Btw, the example on official libvirt website is not clear about that. That is too bad.

    ReplyDelete
  8. Sorry, doesn't work for me using Centos 6.5.
    Looks like the xml files are validated against schemas (see /usr/share/libvirt/schemas/) and everything not found in the schema is wiped out of the xml cfg.
    Even when I export the xml cfg using 'virsh dumpxml' I get a cleaned up copy without the extra arguments for qemu.
    Thank you Redhat for this versatile system.

    ReplyDelete
    Replies
    1. libvirt is a little peculiar in XML editing. It silently drops XML that it cannot parse and also simplifies the XML (e.g. dropping namespaces that are not used).

      It does work on CentOS 6.5 but it sounds like you're missing some small detail. I can help you get the XML working if you paste it into fpaste.org and post a link here.

      Delete