Saturday, 9 April 2011

How to pass QEMU command-line options through libvirt

An entire virtual machine configuration can be passed on QEMU's extensive
command-line, including everything from PCI slots to CPU features to serial
port settings. While defining a virtual machine from a monster
command-line may seem insane, there are times when QEMU's rich command-line
options come in handy.

And at those times one wishes to side-step libvirt's domain XML and specify
QEMU command-line options directly. Luckily libvirt makes this possible and I
learnt about it from Daniel Berrange and Anthony Liguori on IRC. This libvirt
feature will probably come in handy to others and so I want to share it.

The <qemu:commandline> domain XML tag

There is a special namespace for QEMU-specific tags in libvirt domain XML. You
cannot use QEMU-specific tags without first declaring the namespace. To enable
it use the following:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Now you can add command-line arguments to the QEMU invocation. For example, to load an option ROM with -option-rom:
<qemu:commandline>
   <qemu:arg value='-option-rom'/>
   <qemu:arg value='path/to/my.rom'/>
</qemu:commandline>

It is also possible to add environment variables to the QEMU invocation:
<qemu:commandline>
   <qemu:env name='MY_VAR' value='my_value'/>
</qemu:commandline>

Setting qdev properties through libvirt

Taking this a step further we can set qdev properties through libvirt. There is no domain XML for setting the virtio-blk-pci ioeventfd qdev property. Here is how to set it using <qemu:arg> and the -set QEMU option:
<qemu:commandline>
  <qemu:arg value='-set'/>
  <qemu:arg value='device.virtio-disk0.ioeventfd=off'/>
</qemu:commandline>

The result is that libvirt generates a QEMU command-line that ends with -set device.virtio-disk0.ioeventfd=off. This causes QEMU to go back and set the ioeventfd property of device virtio-disk0 to off.

More information

The following libvirt wiki page documents mappings from QEMU command-line options to libvirt domain XML. This is extremely useful if you know which QEMU option to use but are unsure how to express that in domain XML.

That page also reveals the <qemu:commandline> tag and shows how it can be used to invoke QEMU with the GDB stub (-s).

19 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Hi ,

    I am trying to use the kvm-qemu command line option -incoming , to include a statefile with an existing image . But I cannot add it to the xml file and make it work .

    I tried adding the following ,

    "
    [open_angle_bracket]qemu:commandline[close_angle_bracket]
    [open_angle_bracket]qemu:arg value='-incoming'/[close_angle_bracket]
    [open_angle_bracket]qemu:arg value='\"exec:gzip -c STATEFILE.gz\"'/[close_angle_bracket]
    [open_angle_bracket]/qemu:commandline[close_angle_bracket]


    "

    but when i define it, it disappears. When I saw the wiki page document, I found that ,

    -incoming
    This is used internally by libvirt when performing an incoming migration, or restoring a VM from a save file. As such it is not configurable by the user

    You have any idea , how to get this working ? Thanks !

    - Sethu (sethusubbiah at gmail d.ot com)

    ReplyDelete
  3. Sethu, have you tried the "save" and "restore" virsh commands? They let you specify a save state filename and seem to do what you want.

    Here is how I save the state of my rhel-6 guest:
    virsh # save rhel-6 /var/tmp/rhel-6.save

    The /var/tmp/rhel-6.save file now contains the qemu-kvm save state as well as libvirt metadata. It can be restored later but you need to keep the disk image(s) for this guest untouched (or take a copy of them for restoring).

    Here is how i restore my rhel-6 guest:
    virsh # restore /var/tmp/rhel-6.save

    If your use case is not met by these commands then I suggest getting in touch with the libvirt developers on libvir-list@redhat.com.

    ReplyDelete
  4. G'day Stefan, may thanks for this post! I once again was looking for a way to tell libvirt to pass the -redir options to my VMs and your post was the difference. cheers!

    And for anyone else wanting to do this, see here:

    http://snippets.webaware.com.au/snippets/running-qemu-with-port-redirection-through-libvirt/

    ReplyDelete
  5. Little offline question. I would like to know how qemu acts as a container of guest VM. Any pointer will be of big help. Thanks. Great article.

    ReplyDelete
    Replies
    1. Hi Harish,
      I'm not sure if this is what you are looking for:
      http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html

      Delete
  6. Exactly what I was looking for. Very handy when introducing custom command line options to qemu, but still wanting to use libvirt for all the convenience it provides!

    ReplyDelete
  7. This help me resolve a problem. Thanks.
    Originally, I made a mistake to put those two lines into one line and got "invalid option" error. It took me a lot of time to try to figure out why. Now I know --set and option should be in two xml entries.
    Btw, the example on official libvirt website is not clear about that. That is too bad.

    ReplyDelete
  8. Sorry, doesn't work for me using Centos 6.5.
    Looks like the xml files are validated against schemas (see /usr/share/libvirt/schemas/) and everything not found in the schema is wiped out of the xml cfg.
    Even when I export the xml cfg using 'virsh dumpxml' I get a cleaned up copy without the extra arguments for qemu.
    Thank you Redhat for this versatile system.

    ReplyDelete
    Replies
    1. libvirt is a little peculiar in XML editing. It silently drops XML that it cannot parse and also simplifies the XML (e.g. dropping namespaces that are not used).

      It does work on CentOS 6.5 but it sounds like you're missing some small detail. I can help you get the XML working if you paste it into fpaste.org and post a link here.

      Delete
  9. Hello Stefan,
    Thanks for the excellent article.
    After trawling the web trying to solve my query came across your blog.

    If you can provide any hints it'd be really helpful
    Basically i'm trying to set checksum offloading off using one of the 2 following parameters:
    virtio-net-pci.csum=on/off
    virtio-net-pci.guest_csum=on/off

    I tried passing it as a argument to qemu-kvm (QEMU emulator version 0.14.1 (qemu-kvm-0.14.1))
    /usr/bin/qemu-kvm ... ... -device virtio-net-pci,csum=off ...
    although the vm is getting launched, the setting is not visible if i check the process running via 'ps'

    Also tried setting it via xml








    Even this is having the same behaviour as above.

    Do you see any issue with the way I have been setting the value.
    Any help would be great.

    Thanks,
    Have a nice day.
    -Shishir

    P.S. Sorry for the long comment :)

    ReplyDelete
  10. Hi, me again
    it appears the angular brackets dont work so substituting with []

    [interface type='network']
    [mac address='...'/]
    [source network='...'/]
    [target dev='...'/]
    [model type='virtio'/]
    [driver name='vhost' csum='off'/]
    [/interface]

    thanks again.

    ReplyDelete
  11. <driver name='vhost' csum='off'/>

    This is not valid libvirt domain XML (see http://libvirt.org/ for a reference). Libvirt will drop the unknown csum attribute. Use virsh dumpxml <domain> to view the XML after libvirt has processed and saved it - you'll notice the attribute has been dropped.

    Did you try using ethtool inside the guest to adjust offload features at runtime?

    If that fails I would use the "Setting qdev properties through libvirt" approach from the blog post. Look at the command-line being generated by libvirt and pick the right device name, then set the property using the QEMU -set option.

    ReplyDelete
    Replies
    1. Hi, Thanks for the reply.
      Yes, we noticed that the xml did not have the attribute in the generated xml and also in the command line via 'ps'.

      Setting the attribute via ethtool in the guest works, but wanted to make it automated via the domainxml.

      Will try the QEMU set option and let you know how it goes..
      Cheers
      - Shishir

      Delete
  12. hi,

    i'm trying to add environment variables to the QEMU invocation using the example above:
    [[<]]qemu:commandline[[>]]
    [[ <]]qemu:env name='MY_VAR' value='my_value'/[[>]]
    [[<]]/qemu:commandline[[>]]

    when i connect to the vm using console command through virsh and type 'set' (it's linux image), i don't see the 'MY_VAR' variable and value.

    what could be the problem?

    p.s if it's necessary , i could publish the xml configuration file of the current vm.

    thanks,
    lior

    ReplyDelete
    Replies
    1. Libvirt passes environment variables to the QEMU process on the host, not into the guest.

      Look into DHCP Options or cloud-init if you need to send parameters into the guest.

      Delete
    2. Its turn out that i didn't understand correctly how the QEMU work and what i need to do so i will ask again is explain my goal:

      To have a device emulation which will be able to read a configuration file that its name is provided in runtime?

      I think that the easiest way (maybe the only way) to do so it's by passing the configuration file using QEMU environment variables.

      My question is:
      How the model can access to the QEMU environment variables?

      Delete
    3. QOM properties or qdev properties are used to pass parameters to emulated devices.

      For example, see e1000_properties[] in hw/net/e1000.c. These parameters are set on the command-line as follows: -device e1000,autonegotiation=on,mitigation=off

      Environment variables are not a good way of passing parameters to emulated devices because you cannot easily distinguish between multiple instances of the same device. Very few devices use environment variables, and if so, it's usually a hack or workaround.

      Using qdev properties is the standard way to pass parameters to devices.

      Delete
    4. thank you very much for you help.

      Delete