Tuesday, March 29, 2011

How to use perf-probe

Dynamic tracepoints are an observability feature that allows functions
and even arbitrary lines of source code to be instrumented without recompiling
the program. Armed with a copy of the program's source code, tracepoints can
be placed anywhere at runtime and the value of variables can be dumped each
time execution passes the tracepoint. This is an extremely powerful technique
for instrumenting complex systems or code written by someone else.

I recently investigated issues with CD-ROM media change
inside a KVM guest. Using QEMU tracing I was able to
instrument the ATAPI CD-ROM emulation inside QEMU and observe the commands
being sent to the CD-ROM. I complemented this with perf-probe(1) to
instrument the sr and cdrom drivers in the Linux host and
guest kernels. The power of perf-probe(1) was key to understanding the
state of the CD-ROM drivers without recompiling a custom debugging kernel.

Overview of perf-probe(1)

The probe subcommand of perf(1) allows dynamic tracepoints
to be added or removed inside the Linux kernel
. Both core kernel and modules
can be instrumented.

Besides instrumenting locations in the code, a tracepoint can also fetch
values from local variables, globals, registers, the stack, or memory. You can
combine this with perf-record -g callgraph recording to get stack
traces when execution passes probes. This makes perf-probe(1)
extremely powerful.

Requirements

A word of warning: use a recent kernel and perf(1) tool. I was unable
to use function return probes with Fedora 14's 2.6.35-based kernel. It works
fine on with a 2.6.38 perf(1) tool. In general, I have found the
perf(1) tool to be somewhat unstable but this has improved a lot with
recent versions.

Plain function tracing can be done without installing kernel debuginfo
packages but arbitrary source line probes and variable dumping requires the
kernel debuginfo packages. On Debian-based distros the kernel debuginfo
packages are called linux-image-*-dbg and on Red Hat-based distros
they are called kernel-debuginfo.

Tracing function entry

The following command-line syntax adds a new function entry probe:
sudo perf probe <function-name> [<local-variable> ...]

Tracing function return

The following command-line syntax adds a new function return probe and dumps the return value:
sudo perf probe <funtion-name>%return '$retval'
Notice the arg1 return value in this example output for scsi_test_unit_ready():
$ sudo perf record -e probe:scsi_test_unit_ready -aR sleep 10
[...]
$ sudo perf script
     kworker/0:0-3383  [000]  3546.003235: scsi_test_unit_ready: (ffffffffa0208c6d <- ffffffffa0069306) arg1=8000002
 hald-addon-stor-3025  [001]  3546.004431: scsi_test_unit_ready: (ffffffffa0208c6d <- ffffffffa006a143) arg1=8000002
     kworker/0:0-3383  [000]  3548.004531: scsi_test_unit_ready: (ffffffffa0208c6d <- ffffffffa0069306) arg1=8000002
 hald-addon-stor-3025  [001]  3548.005531: scsi_test_unit_ready: (ffffffffa0208c6d <- ffffffffa006a143) arg1=8000002

Listing probes

Existing probes can be listed:
sudo perf probe -l

Removing probes

A probe can be deleted with:
sudo perf probe -d <probe-name<
All probes can be deleted with:
sudo perf probe -d '*'

Further information

See the perf-probe(1) man page for details on command-line syntax and additional features.

The underlying mechanism of perf-probe(1) is kprobe-based event tracing, which is documented here.

I hope that future Linux kernels will add perf-probe(1) for
userspace processes. Nowadays GDB might already include this feature in its
tracing commands but I haven't had a chance to try them out yet.

Saturday, March 26, 2011

Best practices and tuning tips for KVM

The IBM Best practices for KVM document covers storage, networking, and memory/CPU overcommit tuning. If you are looking for known good configurations and a place to start using KVM for optimized server virtualization, check out this document!

Tuesday, March 22, 2011

How to access the QEMU monitor through libvirt

It is sometimes useful to issue QEMU monitor commands to VMs managed by libvirt. Since libvirt takes control of the monitor socket it is not possible to interact with the QEMU monitor in the same way as when running QEMU or KVM manually.

Daniel Berrange shared the following techniques on IRC a while back. It is actually pretty easy to get at the QEMU monitor even while libvirt is managing the VM:

Method 1: virsh qemu-monitor-command


There is a virsh command available in libvirt ≥0.8.8 that allows you to access the QEMU monitor through virsh:

virsh qemu-monitor-command --hmp <domain> '<command> [...]'

Method 2: Connecting directly to the monitor socket


On older libvirt versions the only option is shutting down libvirt, using the monitor socket directly, and then restarting libvirt:

sudo service libvirt-bin stop  # or "libvirtd" on Red Hat-based distros
sudo nc -U /var/lib/libvirt/qemu/<domain>.monitor
...
sudo service libvirt-bin start

Either way works fine. I hope this is useful for folks troubleshooting QEMU or KVM. In the future I will post more libvirt tips :).

Update: Daniel Berrange adds that using the QEMU monitor essentially voids your libvirt warranty :). Try to only use query commands like info qtree rather than commands that change the state of QEMU like adding/removing devices.

Saturday, March 19, 2011

QEMU.org accepted for Google Summer of Code 2011

Good news for students interested in contributing to open source this summer: QEMU.org has been accepted for Google Summer of Code 2011!

If you are interested or have a friend who is enrolled at a university and would like to get paid to work on open source software this summer, check out the QEMU GSoC 2011 ideas page. The full list of accepted organizations is here.

Take a look at my advice on applying on how to succeed with your student application and get chosen.

It's going to be a fun summer and I'm looking forward to getting to know talented students who want to contribute to open source!

Saturday, March 12, 2011

How to write trace analysis scripts for QEMU

This post shows how to write a Python script that finds overlapping disk writes in a QEMU simple trace file.

Several trace backends, including SystemTap and LTTng Userspace Tracer, are supported by QEMU. The built in "simple" trace backend often gives the best bang for the buck because it does not require installing additional software and is easy to use for developers. See my earlier post for an overview of QEMU tracing.

The simple trace backend has recently been enhanced with a Python module for analyzing trace files. This makes it easy to write scripts that post-process trace files and extract useful information. Take this example from the commit that introduced the simpletrace module:

#!/usr/bin/env python
# Print virtqueue elements that were never returned to the guest.

import simpletrace

class VirtqueueRequestTracker(simpletrace.Analyzer):
    def __init__(self):
        self.elems = set()

    def virtqueue_pop(self, vq, elem, in_num, out_num):
        self.elems.add(elem)

    def virtqueue_fill(self, vq, elem, length, idx):
        self.elems.remove(elem)

    def end(self):
        for elem in self.elems:
            print hex(elem)

simpletrace.run(VirtqueueRequestTracker())

This script tracks virtqueue_pop and virtqueue_fill operations and prints out the elements that were popped but never filled back, which indicates elements have leaked.

The model of an analysis script is similar to awk. Trace records are processed from the input file by invoking methods on the user's simpletrace.Analyzer object. The analyzer object does not have to supply methods for all possible trace events, it can just implement those that it wants to know about. Trace events that have no dedicated method cause the catchall() method to be invoked, if provided.

Tracing disk write operations


Let's write a slightly fancier script that finds disk writes that overlap a given range. I've needed to perform disk write overlap queries in the past when debugging image formats, so this is a useful script to have. QEMU's block layer write function looks like this:

BlockDriverAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                                  QEMUIOVector *iov, int nb_sectors,
                                  BlockDriverCompletionFunc *cb, void *opaque);

The block device is called bs and sectors are 512 bytes. Conveniently there is already a trace event for bdrv_aio_write so we just need to enable it in the trace-events file:

disable bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"

Open the trace-events file and remove the disable keyword from the bdrv_aio_write trace event. Then rebuild QEMU like this:

$ ./configure --enable-trace-backend=simple # ... plus your usual options
$ make

Next time you run QEMU a trace file named trace-$PID will be created in the current working directory. The file is in binary and can be parsed using the simpletrace Python module or pretty-printed like this:

$ scripts/simpletrace.py trace-events trace-12345 # replace "trace-12345" with the actual filename

Finding overlapping disk writes


Here is the usage information for a script to find disk writes that overlap a given range:

usage: find_overlapping_writes.py <trace-events> <trace-file> <bs> <sector-num> <nb-sectors>

The script only considers writes to a specific block device, bs. That means all disk I/O to other block devices is ignored.

For example, let's find writes to a 1 MB region at offset 2 MB of the BlockDriverState 0x29c8180:
$ ./find_overlapping_writes.py trace-events trace-19129 0x29c8180 4096 2048
0xe40+664 opaque=0x2eb4190
0x10d8+3032 opaque=0x2eb41d0

Here is the code:
#!/usr/bin/env python
import sys
import simpletrace

def intersects(a_start, a_len, b_start, b_len):
    return not (a_start + a_len <= b_start or \
                b_start + b_len <= a_start)

class OverlappingWritesAnalyzer(simpletrace.Analyzer):
    def __init__(self, bs, sector_num, nb_sectors):
        self.bs = bs
        self.sector_num = sector_num
        self.nb_sectors = nb_sectors

    def bdrv_aio_writev(self, bs, sector_num, nb_sectors, opaque):
        if bs != self.bs:
            return
        if intersects(self.sector_num, self.nb_sectors, sector_num, nb_sectors):
            print '%#x+%d opaque=%#x' % (sector_num, nb_sectors, opaque)

if len(sys.argv) != 6:
    sys.stderr.write('usage: %s <trace-events> <trace-file> <bs> <sector-num> <nb-sectors>\n' %
                     sys.argv[0])
    sys.exit(1)

trace_events, trace_file, bs, sector_num, nb_sectors = sys.argv[1:]
bs = int(bs, 0)
sector_num = int(sector_num, 0)
nb_sectors = int(nb_sectors, 0)

analyzer = OverlappingWritesAnalyzer(bs, sector_num, nb_sectors)
simpletrace.process(trace_events, trace_file, analyzer)

The core of the script is the OverlappingWritesAnalyzer that checks bdrv_aio_writev events for intersection with the given range.

This script is longer than the virtqueue leak detector example above because it parses command-line arguments. The simpletrace.run() function used by the leak detector handles the default trace events and trace file arguments for you. So scripts that take no special command-line arguments can use simpletrace.run(), which also prints usage information automatically. For overlapping writes we really need our own command-line arguments so the slightly lower-level simpletrace.process() function is used.

Where to find out more


There is more information about the simpletrace module in the doc comments, so the simplest way to get started is:
$ cd qemu
$ PYTHONPATH=scripts python
>>> import simpletrace
>>> help(simpletrace)

Another example of how to use the simpletrace module is the trace file pretty-printer which is included as part of the scripts/simpletrace.py source code itself!

Feel free to leave questions or comments about the simple trace backend and the simpletrace Python module.

Wednesday, March 9, 2011

QEMU Internals: Big picture overview

Last week I started the QEMU Internals series to share knowledge of how QEMU works. I dove straight in to the threading model without a high-level overview. I want to go back and provide the big picture so that the details of the threading model can be understood more easily.

The story of a guest


A guest is created by running the qemu program, also known as qemu-kvm or just kvm. On a host that is running 3 virtual machines there are 3 qemu processes:


When a guest shuts down the qemu process exits. Reboot can be performed without restarting the qemu process for convenience although it would be fine to shut down and then start qemu again.

Guest RAM


Guest RAM is simply allocated when qemu starts up. It is possible to pass in file-backed memory with -mem-path such that hugetlbfs can be used. Either way, the RAM is mapped in to the qemu process' address space and acts as the "physical" memory as seen by the guest:


QEMU supports both big-endian and little-endian target architectures so guest memory needs to be accessed with care from QEMU code. Endian conversion is performed by helper functions instead of accessing guest RAM directly. This makes it possible to run a target with a different endianness from the host.

KVM virtualization


KVM is a virtualization feature in the Linux kernel that lets a program like qemu safely execute guest code directly on the host CPU. This is only possible when the target architecture is supported by the host CPU. Today KVM is available on x86, ARMv8, ppc, s390, and MIPS CPUs.

In order to execute guest code using KVM, the qemu process opens /dev/kvm and issues the KVM_RUN ioctl. The KVM kernel module uses hardware virtualization extensions found on modern Intel and AMD CPUs to directly execute guest code. When the guest accesses a hardware device register, halts the guest CPU, or performs other special operations, KVM exits back to qemu. At that point qemu can emulate the desired outcome of the operation or simply wait for the next guest interrupt in the case of a halted guest CPU.

The basic flow of a guest CPU is as follows:
open("/dev/kvm")
ioctl(KVM_CREATE_VM)
ioctl(KVM_CREATE_VCPU)
for (;;) {
     ioctl(KVM_RUN)
     switch (exit_reason) {
     case KVM_EXIT_IO:  /* ... */
     case KVM_EXIT_HLT: /* ... */
     }
}

The host's view of a running guest


The host kernel schedules qemu like a regular process. Multiple guests run alongside without knowledge of each other. Applications like Firefox or Apache also compete for the same host resources as qemu although resource controls can be used to isolate and prioritize qemu.

Since qemu system emulation provides a full virtual machine inside the qemu userspace process, the details of what processes are running inside the guest are not directly visible from the host. One way of understanding this is that qemu provides a slab of guest RAM, the ability to execute guest code, and emulated hardware devices; therefore any operating system (or no operating system at all) can run inside the guest. There is no ability for the host to peek inside an arbitrary guest.

Guests have a so-called vcpu thread per virtual CPU. A dedicated iothread runs a select(2) event loop to process I/O such as network packets and disk I/O completion. For more details and possible alternate configuration, see the threading model post.

The following diagram illustrates the qemu process as seen from the host:




Further information


Hopefully this gives you an overview of QEMU and KVM architecture. Feel free to leave questions in the comments and check out other QEMU Internals posts for details on these aspects of QEMU.

Here are two presentations on KVM architecture that cover similar areas if you are interested in reading more:

Tuesday, March 8, 2011

How to automatically run checkpatch.pl when developing QEMU

The checkpatch.pl script was recently added to qemu.git as a way to scan patches for coding standard violations. You can automatically run checkpatch.pl when committing changes to git and abort if there are violations:

$ cd qemu
$ cat >.git/hooks/pre-commit
#!/bin/bash
exec git diff --cached | scripts/checkpatch.pl --no-signoff -q -
^D
$ chmod 755 .git/hooks/pre-commit

Any commit that violates the coding standard as checked by checkpatch.pl will be aborted. I am running with this git hook now and will post any tweaks I make to it.

Update: If you encounter a false positive because checkpatch.pl is complaining about code you didn't touch, use git commit --no-verify to override the pre-commit hook. Use this trick sparingly :-).

Saturday, March 5, 2011

QEMU Internals: Overall architecture and threading model

This is the first post in a series on QEMU Internals aimed at developers. It is designed to share knowledge of how QEMU works and make it easier for new contributors to learn about the QEMU codebase.

Running a guest involves executing guest code, handling timers, processing I/O, and responding to monitor commands. Doing all these things at once requires an architecture capable of mediating resources in a safe way without pausing guest execution if a disk I/O or monitor command takes a long time to complete. There are two popular architectures for programs that need to respond to events from multiple sources:
  1. Parallel architecture splits work into processes or threads that can execute simultaneously. I will call this threaded architecture.
  2. Event-driven architecture reacts to events by running a main loop that dispatches to event handlers. This is commonly implemented using the select(2) or poll(2) family of system calls to wait on multiple file descriptors.

QEMU actually uses a hybrid architecture that combines event-driven programming with threads. It makes sense to do this because an event loop cannot take advantage of multiple cores since it only has a single thread of execution. In addition, sometimes it is simpler to write a dedicated thread to offload one specific task rather than integrate it into an event-driven architecture. Nevertheless, the core of QEMU is event-driven and most code executes in that environment.

The event-driven core of QEMU


An event-driven architecture is centered around the event loop which dispatches events to handler functions. QEMU's main event loop is main_loop_wait() and it performs the following tasks:

  1. Waits for file descriptors to become readable or writable. File descriptors play a critical role because files, sockets, pipes, and various other resources are all file descriptors. File descriptors can be added using qemu_set_fd_handler().
  2. Runs expired timers. Timers can be added using qemu_mod_timer().
  3. Runs bottom-halves (BHs), which are like timers that expire immediately. BHs are used to avoid reentrancy and overflowing the call stack. BHs can be added using qemu_bh_schedule().

When a file descriptor becomes ready, a timer expires, or a BH is scheduled, the event loop invokes a callback that responds to the event. Callbacks have two simple rules about their environment:
  1. No other core code is executing at the same time so synchronization is not necessary. Callbacks execute sequentially and atomically with respect to other core code. There is only one thread of control executing core code at any given time.
  2. No blocking system calls or long-running computations should be performed. Since the event loop waits for the callback to return before continuing with other events, it is important to avoid spending an unbounded amount of time in a callback. Breaking this rule causes the guest to pause and the monitor to become unresponsive.

This second rule is sometimes hard to honor and there is code in QEMU which blocks. In fact there is even a nested event loop in qemu_aio_wait() that waits on a subset of the events that the top-level event loop handles. Hopefully these violations will be removed in the future by restructuring the code. New code almost never has a legitimate reason to block and one solution is to use dedicated worker threads to offload long-running or blocking code.

Offloading specific tasks to worker threads


Although many I/O operations can be performed in a non-blocking fashion, there are system calls which have no non-blocking equivalent. Furthermore, sometimes long-running computations simply hog the CPU and are difficult to break up into callbacks. In these cases dedicated worker threads can be used to carefully move these tasks out of core QEMU.

One example user of worker threads is posix-aio-compat.c, an asynchronous file I/O implementation. When core QEMU issues an aio request it is placed on a queue. Worker threads take requests off the queue and execute them outside of core QEMU. They may perform blocking operations since they execute in their own threads and do not block the rest of QEMU. The implementation takes care to perform necessary synchronization and communication between worker threads and core QEMU.

Another example is ui/vnc-jobs-async.c which performs compute-intensive image compression and encoding in worker threads.

Since the majority of core QEMU code is not thread-safe, worker threads cannot call into core QEMU code. Simple utilities like qemu_malloc() are thread-safe but that is the exception rather than the rule. This poses a problem for communicating worker thread events back to core QEMU.

When a worker thread needs to notify core QEMU, a pipe or a qemu_eventfd() file descriptor is added to the event loop. The worker thread can write to the file descriptor and the callback will be invoked by the event loop when the file descriptor becomes readable. In addition, a signal must be used to ensure that the event loop is able to run under all circumstances. This approach is used by posix-aio-compat.c and makes more sense (especially the use of signals) after understanding how guest code is executed.

Executing guest code


So far we have mainly looked at the event loop and its central role in QEMU. Equally as important is the ability to execute guest code, without which QEMU could respond to events but would not be very useful.

There are two mechanism for executing guest code: Tiny Code Generator (TCG) and KVM. TCG emulates the guest using dynamic binary translation, also known as Just-in-Time (JIT) compilation. KVM takes advantage of hardware virtualization extensions present in modern Intel and AMD CPUs for safely executing guest code directly on the host CPU. For the purposes of this post the actual techniques do not matter but what matters is that both TCG and KVM allow us to jump into guest code and execute it.

Jumping into guest code takes away our control of execution and gives control to the guest. While a thread is running guest code it cannot simultaneously be in the event loop because the guest has (safe) control of the CPU. Typically the amount of time spent in guest code is limited because reads and writes to emulated device registers and other exceptions cause us to leave the guest and give control back to QEMU. In extreme cases a guest can spend an unbounded amount of time without giving up control and this would make QEMU unresponsive.

In order to solve the problem of guest code hogging QEMU's thread of control signals are used to break out of the guest. A UNIX signal yanks control away from the current flow of execution and invokes a signal handler function. This allows QEMU to take steps to leave guest code and return to its main loop where the event loop can get a chance to process pending events.

The upshot of this is that new events may not be detected immediately if QEMU is currently in guest code. Most of the time QEMU eventually gets around to processing events but this additional latency is a performance problem in itself. For this reason timers, I/O completion, and notifications from worker threads to core QEMU use signals to ensure that the event loop will be run immediately.

You might be wondering what the overall picture between the event loop and an SMP guest with multiple vcpus looks like. Now that the threading model and guest code has been covered we can discuss the overall architecture.

iothread and non-iothread architecture


The traditional architecture is a single QEMU thread that executes guest code and the event loop. This model is also known as non-iothread or !CONFIG_IOTHREAD and is the default when QEMU is built with ./configure && make. The QEMU thread executes guest code until an exception or signal yields back control. Then it runs one iteration of the event loop without blocking in select(2). Afterwards it dives back into guest code and repeats until QEMU is shut down.

If the guest is started with multiple vcpus using -smp 2, for example, no additional QEMU threads will be created. Instead the single QEMU thread multiplexes between two vcpus executing guest code and the event loop. Therefore non-iothread fails to exploit multicore hosts and can result in poor performance for SMP guests.

Note that despite there being only one QEMU thread there may be zero or more worker threads. These threads may be temporarily or permanent. Remember that they perform specialized tasks and do not execute guest code or process events. I wanted to emphasise this because it is easy to be confused by worker threads when monitoring the host and interpret them as vcpu threads. Remember that non-iothread only ever has one QEMU thread.

The newer architecture is one QEMU thread per vcpu plus a dedicated event loop thread. This model is known as iothread or CONFIG_IOTHREAD and can be enabled with ./configure --enable-io-thread at build time. Each vcpu thread can execute guest code in parallel, offering true SMP support, while the iothread runs the event loop. The rule that core QEMU code never runs simultaneously is maintained through a global mutex that synchronizes core QEMU code across the vcpus and iothread. Most of the time vcpus will be executing guest code and do not need to hold the global mutex. Most of the time the iothread is blocked in select(2) and does not need to hold the global mutex.

Note that TCG is not thread-safe so even under the iothread model it multiplexes vcpus across a single QEMU thread. Only KVM can take advantage of per-vcpu threads.

Conclusion and words about the future

Hopefully this helps communicate the overall architecture of QEMU (which KVM inherits). Feel free to leave questions in the comments below.

In the future the details are likely to change and I hope we will see a move to CONFIG_IOTHREAD by default and maybe even a removal of !CONFIG_IOTHREAD.

I will try to update this post as qemu.git changes.

Thursday, March 3, 2011

Should I use QEMU or KVM?

UPDATE: The qemu-kvm.git fork has been merged back into qemu.git as of QEMU 1.3.0. Always use qemu.git for the latest code. See my full post here.

"What is the difference between QEMU and KVM?" comes up regularly because these two pieces of software share a close relationship. I am going to explain how to choose between the two and the nature of their relationship.

Should I install the qemu or qemu-kvm package?


If you want to run x86 virtual machines on x86 physical machines, install qemu-kvm. It has the fastest and most thoroughly tested support for the common x86 virtualization use case.

If you want to run anything else, install qemu. That includes running non-x86 machines and user level emulation instead of full-system emulation.

If you are still not sure which is right for you, take a look at the QEMU and KVM websites.

How do I check that qemu-kvm is using hardware support?


If qemu-kvm is unable to use hardware virtualization extensions it will fall back to emulation which is much slower. If you are worried this might be the case, run the following check:

grep 'svm\|vmx' /proc/cpuinfo

If you get output then the CPU supports virtualization extensions and KVM should work. Otherwise check that virtualization is enabled in your BIOS.

See the processor support KVM wiki page for more information.

It is also a good idea to use the -enable-kvm command-line option to ensure that KVM is used. The libvirt, virt-manager, virsh stack will do this by default.

What is the difference between qemu.git and qemu-kvm.git?


The QEMU codebase is known as qemu.git. That's the git repository that holds the QEMU source code history. The KVM codebase is known as qemu-kvm.git, the git repository that holds the KVM source code history.

The relationship between qemu.git and qemu-kvm.git is as follows. qemu-kvm.git is a fork of qemu.git and periodically merges updates from qemu.git back into qemu-kvm.git. A lot of code changes are merged into qemu.git and become available in qemu-kvm.git after the next periodic merge. KVM-specific enhancements may be merged into qemu-kvm.git and may be sent back upstream to qemu.git.

Efforts are underway to completely merge qemu-kvm.git into qemu.git. This will make qemu-kvm.git obsolete and result in a single codebase. In the future there may only be a qemu package.

Tuesday, March 1, 2011

Advice for students applying to Google Summer of Code

For the past several years Google has run a program called Summer of Code (GSoC) that funds university students to work on open source projects during the summer. A large number of leading open source projects participate and provide mentorship to students.

Google have announced that organizations can start applying for GSoC 2011. Students will be able to apply for projects once the accepted organizations have been announced. See the timeline for details.

In 2008 I participated as a student and worked on remote GDB debugging for the gPXE network bootloader. I had a great time and stuck around after GSoC ended, continuing to contribute to Etherboot.org. In the following two years of GSoC I participated again, this time as a mentor.

I've seen GSoC from both sides and here is my advice for students who want to apply.

Is it worth doing?


Yes, definitely. GSoC gives you privileged access to an open source community and a mentor who has committed to supporting you. If you have ever been interested in contributing to an open source project then this is the chance!

Choosing a project


In past years there have been many participating organizations to choose from. You can either apply for a listed project idea or suggest your own project idea. If you have your own project idea then make sure to get in touch with the organization well before the student application deadline in order to pitch your idea to them and get their support.

Choose a project idea that you are comfortable with, both in terms of the amount of effort it will take and your current level of skills. You can learn a lot of new things during the summer but make sure you can deliver on what you are promising. The good news is that there are so many project ideas to choose from that you should be able to find something that matches your interests and skills.

The other common factor is the amount of time you will be expected to spend on your project. Many students work full-time from Monday to Friday much like a regular job. Sometimes organizations are happy to accept talented students who can deliver their project with less time commitment. You can probably take some vacation days off but make sure to state your availability upfront when applying.

It's worth keeping in mind that GSoC is quite decentralized. Individual organizations have a large degree of control over how they run their projects. No two organizations work the same so look at their previous years' wiki pages and project archives or ask the mentors to understand how they operate.

I suggest applying to two or three organizations. Since organizations are free to approach GSoC quite differently, it's worth diversifying your applications so you can choose the organization and mentor you are most comfortable with in the end. Remember that just because a piece of software is cool does not mean that their GSoC or community are the right place for you, so look at multiple organizations. Also keep in mind that organizations have limited "slots", or numbers of students that Google will fund, so if you are unlucky an organization may run out of slots and be unable to take you even if they are interested. For these reasons it makes sense to apply to two or three organizations.

Making your application


The bare minimum student application involves filling out a project proposal form. To increase your chance of getting accepted you need to consider how organizations select students.

Organizations will receive more student applications than they have slots. In the past I've seen 5:1 to 10:1 ratios so understand that GSoC is competitive. There are three levels of student applications:
  1. Students who either do not have the skills or did not put in enough time to prepare a decent project proposal. These are easy to spot for the organization and they are not your competition. They do serve as a reminder to double-check the timeline and make sure you fill in appropriate information. If you have questions just ask the organization you are applying to, they'll be glad to help interested students.
  2. Students who might be good candidates but do not stand out from the competition. The majority of applications will be students who probably have the skills to tackle the project but their attitude, personality, and enthusiasm is unknown. These students fail to communicate their abilities and vision clearly enough to stand out. Your main goal when applying should be not to fall into this group.
  3. Students who have shown enthusiasm, ability to communicate, and clearly have the skills not just to complete their project but also to contribute to the community. If you can stand out like this then you're likely to get picked. These students will contact the mentor ahead of time and discuss the project idea. This will arm them with the information to put together a good proposal. They will join mailing lists, forums, or IRC channels to learn about the community. They will even contribute patches before being accepted for GSoC - this is a key action you can take to improve your chances.

Another way of explaining this is that a large number of students will apply. Many of them will be in the ballpark and could potentially complete the project. But due to the high applications to slots ratio, only the best will get chosen.

Perfectly good students will not get a slot so aim to be that top level of student who is interested not just in doing a project over the summer but in diving into the community, helping users, contributing patches, and fixing bugs. If you do that then your chances of being accepted are good.

Final thoughts


I hope this helps you prepare for Google Summer of Code. For more information check out the official FAQ.

This year I'm excited about helping QEMU. A project ideas page has been published, so check that out if you are interested in emulation or virtualization.

Have GSoC questions or advice to share? Post a comment!