I have been using Linux Containers (LXC) to run lightweight virtual machines for about a year now. I revisited LXC this week because I temporarily lack access to a machine with virtualization extensions for KVM. So I need an alternative for running Linux virtual machines with good performance. Here are my thoughts on LXC and its current status.
What is Linux Containers (LXC)?
The Linux Containers (LXC) project provides container virtualization for Linux, which has been described as "chroot(8) on steroids". The idea of containers is that resources like process identifiers (pids), user identifiers (uids), or memory are no longer managed globally by the kernel. Instead, the kernel accounts each resource against a container and gives them separate namespaces. This way multiple containers can run an init(8) process with pid 1 without conflicting, for example.
LXC can run multiple Linux systems simultaneously on one kernel. Containers are isolated from each other and do not conflict because each resource belongs to a container. The lxc 0.8.0~rc1-8+deb7u1 package I used on Debian comes with "templates" for Fedora, OpenSuSE, Ubuntu, Debian, Arch Linux, and altlinux. It takes about 10 minutes to create a new container pre-installed with one of these Linux distributions.
It's also possible to share resources, like file systems, between containers. It's even possible to run just a single process in a container without booting up a full Linux distribution - this really is "chroot(8) on steroids". But I think these features are useful to fewer users because they require understanding the exact resource dependencies of the application being containerized. It's easier to boot up a full Linux distribution to run the application.
In case you're wondering, Linux Containers (LXC) is purely a software kernel feature and therefore works fine inside KVM, Xen, or VMware. I use it to slice up my virtual server hosted on KVM and it works fine on Amazon EC2, Linode, etc.
How to create a container
First you need to install the lxc package:
sudo aptitude install lxc
There are now a bunch of utilities called lxc(1), lxc-start(1), lxc-create(1), etc available. The lxc(1) utility is a wrapper for the others, similar to git(1) versus git-branch(1).
Create a Debian container from its template:
sudo lxc create mydebian -t debian
There is a configuration file in /var/lib/lxc/mydebian/config that defines resources available to the container. You need to edit this file to set memory limits, set up networking, change the hostname, and more. The specific configuration options are out of scope for this post. Refer to your lxc package documentation, for example, /usr/share/doc/lxc/ on Debian.
Once installation is complete, launch the container:
sudo lxc start mydebian
To shut down the container:
sudo lxc shutdown mydebian
The rough edges
Use virt-manager, if possible
LXC has a few hurdles to get going. They remind me of issues first-time qemu users face. And the answer to configuration complexity is to use a high-level tool like virt-manager. Here are the advantages to using virt-manager:
- Configure the container from a graphical interface. Skip learning configuration file syntax for something you will rarely edit.
- Sets up environment dependencies for cgroups and networking. If you're not familiar with network bridges or IP forwarding this can save time.
- Libvirt API support if you wish to automate or write custom tooling. This is actually a libvirt benefit and not specific to virt-manager itself.
Unfortunately, I wasn't able to use virt-manager because libvirt requires cgroups controllers that are not available on my machine (without recompiling the kernel). Libvirt assumes that you want to fully isolate the container and relies on several cgroups controllers. In my case I don't care about limiting memory usage and it just so happens that the distro kernel I run doesn't have that controller built. I decided to use lxc(1) manually and deal with the configuration file and networking setup.
Templates should indicate what to do after installation
After installing the Debian template, the container booted without networking and no login console. The boot process also spits out a number of errors that are not common on baremetal systems, because the unmodified Debian distro is trying to do things that are not allowed inside the container. Without much in the way of getting started documentation, it took a little while to figure out what was missing.
I ensured the ttys were running getty in /etc/inittab and created /dev/ttyX devices in the container. That took care of login consoles and made lxc console mydebian happy.
For networking I chose to use the veth type together with IP forwarding for external network access. This is fairly standard stuff if you've been playing with KVM or Xen.
Anyway, what is missing from LXC is some information between lxc create and lxc start that gives hints what to do in order to successfully boot and access the container.
Summary
Linux Containers (LXC) allows you to run lightweight Linux virtual machines with good performance. Once they are set up they boot quickly and feel like baremetal. As of September 2012, the container setup experience requires patience and some experience with troubleshooting Linux boot and networking issues. Hopefully the libvirt lxc support will reach the level where it's possible to create containers in virt-manager's wizard without getting familiar with low-level details.
Are you using LXC or have you tried it? Share your comments below.