Saturday, 9 April 2011

How to pass QEMU command-line options through libvirt

An entire virtual machine configuration can be passed on QEMU's extensive
command-line, including everything from PCI slots to CPU features to serial
port settings. While defining a virtual machine from a monster
command-line may seem insane, there are times when QEMU's rich command-line
options come in handy.

And at those times one wishes to side-step libvirt's domain XML and specify
QEMU command-line options directly. Luckily libvirt makes this possible and I
learnt about it from Daniel Berrange and Anthony Liguori on IRC. This libvirt
feature will probably come in handy to others and so I want to share it.

The <qemu:commandline> domain XML tag

There is a special namespace for QEMU-specific tags in libvirt domain XML. You
cannot use QEMU-specific tags without first declaring the namespace. To enable
it use the following:
<domain type='kvm' xmlns:qemu=''>

Now you can add command-line arguments to the QEMU invocation. For example, to load an option ROM with -option-rom:
   <qemu:arg value='-option-rom'/>
   <qemu:arg value='path/to/my.rom'/>

It is also possible to add environment variables to the QEMU invocation:
   <qemu:env name='MY_VAR' value='my_value'/>

Setting qdev properties through libvirt

Taking this a step further we can set qdev properties through libvirt. There is no domain XML for setting the virtio-blk-pci ioeventfd qdev property. Here is how to set it using <qemu:arg> and the -set QEMU option:
  <qemu:arg value='-set'/>
  <qemu:arg value='device.virtio-disk0.ioeventfd=off'/>

The result is that libvirt generates a QEMU command-line that ends with -set device.virtio-disk0.ioeventfd=off. This causes QEMU to go back and set the ioeventfd property of device virtio-disk0 to off.

More information

The following libvirt wiki page documents mappings from QEMU command-line options to libvirt domain XML. This is extremely useful if you know which QEMU option to use but are unsure how to express that in domain XML.

That page also reveals the <qemu:commandline> tag and shows how it can be used to invoke QEMU with the GDB stub (-s).


  1. This comment has been removed by the author.

  2. Hi ,

    I am trying to use the kvm-qemu command line option -incoming , to include a statefile with an existing image . But I cannot add it to the xml file and make it work .

    I tried adding the following ,

    [open_angle_bracket]qemu:arg value='-incoming'/[close_angle_bracket]
    [open_angle_bracket]qemu:arg value='\"exec:gzip -c STATEFILE.gz\"'/[close_angle_bracket]


    but when i define it, it disappears. When I saw the wiki page document, I found that ,

    This is used internally by libvirt when performing an incoming migration, or restoring a VM from a save file. As such it is not configurable by the user

    You have any idea , how to get this working ? Thanks !

    - Sethu (sethusubbiah at gmail d.ot com)

  3. Sethu, have you tried the "save" and "restore" virsh commands? They let you specify a save state filename and seem to do what you want.

    Here is how I save the state of my rhel-6 guest:
    virsh # save rhel-6 /var/tmp/

    The /var/tmp/ file now contains the qemu-kvm save state as well as libvirt metadata. It can be restored later but you need to keep the disk image(s) for this guest untouched (or take a copy of them for restoring).

    Here is how i restore my rhel-6 guest:
    virsh # restore /var/tmp/

    If your use case is not met by these commands then I suggest getting in touch with the libvirt developers on

  4. G'day Stefan, may thanks for this post! I once again was looking for a way to tell libvirt to pass the -redir options to my VMs and your post was the difference. cheers!

    And for anyone else wanting to do this, see here:

  5. Little offline question. I would like to know how qemu acts as a container of guest VM. Any pointer will be of big help. Thanks. Great article.

    1. Hi Harish,
      I'm not sure if this is what you are looking for:

  6. Exactly what I was looking for. Very handy when introducing custom command line options to qemu, but still wanting to use libvirt for all the convenience it provides!

  7. This help me resolve a problem. Thanks.
    Originally, I made a mistake to put those two lines into one line and got "invalid option" error. It took me a lot of time to try to figure out why. Now I know --set and option should be in two xml entries.
    Btw, the example on official libvirt website is not clear about that. That is too bad.

  8. Sorry, doesn't work for me using Centos 6.5.
    Looks like the xml files are validated against schemas (see /usr/share/libvirt/schemas/) and everything not found in the schema is wiped out of the xml cfg.
    Even when I export the xml cfg using 'virsh dumpxml' I get a cleaned up copy without the extra arguments for qemu.
    Thank you Redhat for this versatile system.

    1. libvirt is a little peculiar in XML editing. It silently drops XML that it cannot parse and also simplifies the XML (e.g. dropping namespaces that are not used).

      It does work on CentOS 6.5 but it sounds like you're missing some small detail. I can help you get the XML working if you paste it into and post a link here.

  9. Hello Stefan,
    Thanks for the excellent article.
    After trawling the web trying to solve my query came across your blog.

    If you can provide any hints it'd be really helpful
    Basically i'm trying to set checksum offloading off using one of the 2 following parameters:

    I tried passing it as a argument to qemu-kvm (QEMU emulator version 0.14.1 (qemu-kvm-0.14.1))
    /usr/bin/qemu-kvm ... ... -device virtio-net-pci,csum=off ...
    although the vm is getting launched, the setting is not visible if i check the process running via 'ps'

    Also tried setting it via xml

    Even this is having the same behaviour as above.

    Do you see any issue with the way I have been setting the value.
    Any help would be great.

    Have a nice day.

    P.S. Sorry for the long comment :)

  10. Hi, me again
    it appears the angular brackets dont work so substituting with []

    [interface type='network']
    [mac address='...'/]
    [source network='...'/]
    [target dev='...'/]
    [model type='virtio'/]
    [driver name='vhost' csum='off'/]

    thanks again.

  11. <driver name='vhost' csum='off'/>

    This is not valid libvirt domain XML (see for a reference). Libvirt will drop the unknown csum attribute. Use virsh dumpxml <domain> to view the XML after libvirt has processed and saved it - you'll notice the attribute has been dropped.

    Did you try using ethtool inside the guest to adjust offload features at runtime?

    If that fails I would use the "Setting qdev properties through libvirt" approach from the blog post. Look at the command-line being generated by libvirt and pick the right device name, then set the property using the QEMU -set option.

    1. Hi, Thanks for the reply.
      Yes, we noticed that the xml did not have the attribute in the generated xml and also in the command line via 'ps'.

      Setting the attribute via ethtool in the guest works, but wanted to make it automated via the domainxml.

      Will try the QEMU set option and let you know how it goes..
      - Shishir

  12. hi,

    i'm trying to add environment variables to the QEMU invocation using the example above:
    [[ <]]qemu:env name='MY_VAR' value='my_value'/[[>]]

    when i connect to the vm using console command through virsh and type 'set' (it's linux image), i don't see the 'MY_VAR' variable and value.

    what could be the problem?

    p.s if it's necessary , i could publish the xml configuration file of the current vm.


    1. Libvirt passes environment variables to the QEMU process on the host, not into the guest.

      Look into DHCP Options or cloud-init if you need to send parameters into the guest.

    2. Its turn out that i didn't understand correctly how the QEMU work and what i need to do so i will ask again is explain my goal:

      To have a device emulation which will be able to read a configuration file that its name is provided in runtime?

      I think that the easiest way (maybe the only way) to do so it's by passing the configuration file using QEMU environment variables.

      My question is:
      How the model can access to the QEMU environment variables?

    3. QOM properties or qdev properties are used to pass parameters to emulated devices.

      For example, see e1000_properties[] in hw/net/e1000.c. These parameters are set on the command-line as follows: -device e1000,autonegotiation=on,mitigation=off

      Environment variables are not a good way of passing parameters to emulated devices because you cannot easily distinguish between multiple instances of the same device. Very few devices use environment variables, and if so, it's usually a hack or workaround.

      Using qdev properties is the standard way to pass parameters to devices.

    4. thank you very much for you help.

  13. Hello Stefan!
    Thank you very much for your blog, it helps me a lot in my work with QEMU!

    My question is about setting new option to qemu. Your advice and knowledge would be really very helpful.

    Could you please tell me how to add a command-line property?
    For example, I need to add one to virtio balloon settings, so when I start VM via QEMU in command line, it would look like this:
    "-balloon virtio, my-property=on"

    I have tried several ways to do it.
    In vl.c file there is a function balloon_parse() where qemu_opts_parse() is called (in case we have arguments). In balloon_parse() (after arguments parsing) I have added call for qemu_opt_set to set value for my property. But this way doesn't work.
    I have also tried via object_property_add() and added resperctive setter/getter for it. I have also added my property to :
    static Property virtio_balloon_properties[] = {
    DEFINE_PROP_BOOL("my-property, VirtIOBalloon, my_property, false),
    Of course, I have also added boolean field named "my_property" to VirtIOBalloon structure. But it doesn't work either.
    I have also tried to add a new field in qemu_device_opts structure, but it doesn't help:
    QemuOptsList qemu_device_opts = {
    // other fields //
    .name = "driver",
    .type = QEMU_OPT_STRING,
    .name = "my-property",
    .type = QEMU_OPT_BOOL,
    .def_value_str = "off",
    { /* end of list */ }
    Every time I start VM with my property I get the following error:
    "(qemu) qemu-system-x86_64: -balloon virtio,my-property=on:
    Property '.my-property' not found"

    Could you tell me what I do wrong? I would appreciate any advice.

    Many thanks in advance.
    Best regards

    1. Please send general technical questions to the QEMU mailing list:

  14. I wanted to try this, but as soon as I even insert

    I get

    XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng

    from virsh edit.
    Does anybody have a hint to what I'm doing wrong?

    1. Argh, the comment system eats html-like things…

      I was referring to: domain type='kvm' xmlns:qemu=''

    2. virsh edit parses the XML changes and discards invalid XML. It sounds like you did not enter valid libvirt domain XML.

      Make sure to replace the existing tag with the one from this blog post and add the tag before closing .

    3. Ha, I got caught out by the blog's HTML blocker too :).

      Make sure to replace the existing <domain> tag with the one from this blog post and add the <qemu:commandline> tag before closing </domain>.

    4. So I'm doing this It works perfectly fine on my gentoo, but not on my arch. "Element domain failed to validate content"
      I guess that's a bug and I should report it to the arch bugtracker.

    5. I have replaced the domain tag as shown here but virsh keeps discarding any command line tags. I believe that the schema url is 404 but I am not sure that it has been migrated anywhere else. Anybody know if the url is still valid?

  15. And is there a way to convert XML domain file to a qemu-command-line without starting a node and doing ps aux | grep ?

    1. I could have sworn there is a virsh command to print out the QEMU command-line without launching a guest. Unfortunately I can't find it now.

      Here is another solution: if you have started the guest once already, look at /var/lib/libvirt/qemu/.log to see the full QEMU command-line.