Introduction ============ LinuxCon/CloudOpen (Europe), KVM Forum and a bunch of other conferences, including Linux Plumbers, a week long Linux Foundation events just ended last Friday in Düsseldorf, Germany, held at the excellent venue of Congress Center Düsseldorf, on the bank of Rhine river. I spent my most of my last week there. If you're not acquainted, KVM Forum is a developer event that mainly focuses on KVM, QEMU, libvirt projects and their integration work. I spent most of my time at KVM Forum (which for the third year in a row is co-located with the LinuxCon/ClouOpen Europe) with an occasional shuffle to LinuxCon/CloudOpen events to give a talk or to attend a discussion. (If you're into Virtualization development and are just scanning this report, you might also want to take a look at "Other talks that might be interesting to Virtualization folks" to see if you find something of your interest.) Tracing Summit ============== == Day-1 (Mon) == This was the first day of LinuxCon and related events (KVM Forum started on Tuesday). I attended just the below specific talk that I was interested from the Tracing Summit (one of the many topic-focused events) in the morning. User Case Study: Tracing in the QEMU emulator - Stefan Hajnoczi, Red Hat ------------------------------------------------------------------------ Stefan is QEMU's tracing sybsystem maintainer (along with Block and Network subsystems). His talk was about how QEMU project uses tracing and how it integrates different types of tracers. He began by outlining QEMU's architecture: QEMU is a user space process on the host and the guest runs as part of QEMU. The KVM Kernel module switches between host and guest modes. QEMU performs I/O on behalf of the user (e.g. if the virtual machine wants to send a network packet, QEMU will do that for you). Each guest vCPU is a thread on the host when using KVM. Then he described how to observe/profile the KVM/QEMU stack using various tools depending on what specific aspect you want to probe. To examine _inside_ a guest, use tools like perf-kvm(1); to observe QEMU, use static probes, or dynamic probes using perf, strace, etc. To instrument KVM - LTTng, ftrace, perf, netstat, top, etc. Next he discussed the QEMU's 'tracetool', a Python script which generates tracing code for multiple tracer backends (DTrace, LTTng-UST, QEMU's in-built "simple" tracer, etc). The talk was ended with a discussion of tracing TCG (Tiny Code Generator - which compiles machine code from the guest architecture to the host arch) and some Q/A. Slides: http://vmsplice.net/~stefan/stefanha-tracing-summit-2014.pdf More info on QEMU tracing infrastructure, from upstream source: qemu/docs/tracing.txt. KVM Forum/LinuxCon/CloudOpen ============================ === Day-2, Tue === KVM Forum started today. KVM Keynote - Paolo Bonzini, Red Hat ------------------------------------ Traditionally, KVM Forum is kicked off with this keynote by the current KVM maintainer discussing the state of KVM in the last year or so, highlights, architecture support/improvements, optimizations, benchmark details, etc. Slides: http://events.linuxfoundation.org/sites/events/files/slides/kvmforum14-kvm.pdf Security Hardening of KVM - Andrew Honig, Google ------------------------------------------------ This is immediately followed by Paolo's Keynote. Andrew took a deep dive into security aspects of KVM at Kernel level. Google Compute Engine uses KVM, but somewhere in the talk, Andrew mentioned they doesn't use QEMU as user space with KVM, but "something else" without exactly naming what it is. Also in this discussion: different types of KVM vulnerabilities (including specific CVEs), how to reduce attach surface, and approaches to move attack surfaces to user space VMM. He then ended the talk with a couple of new I/O controls (ioctls) that they're experimenting with. If you're into KVM/Kernel internals and security, this might be of interest to you. Slides for this talk: http://events.linuxfoundation.org/sites/events/files/slides/KVM%20Hardening.pdf === Day-3, Wed === On the morning of third day, I went to the LinuxCon Keynote by Joanna Rutkowska, founder of invisiblethingslab.com, she talked about Qubes OS -- an a open source operating system (based on Fedora) designed to provide strong security for desktop/GUI computing ("Security by Compartmentalization"), by using Xen-based virtualization. Linux: Where Are We Going, Dirk Hohndel - Intel; Linus Torvalds - Linux Foundation ----------------------------------------------------------------------- This is one of the customary talks at all Linux Foundation events. It was a small Q/A session, nothing really super interesting. From what I recall, a few topics that were discussed: an audience member asked about shortening (3 to 2 months) Kernel release cycle, on trying to be being sutble on the internet, working with people who you don't get along well with but still respecting the work they do. At the end, when Dirk asked to make a bold (with a sarcastic emphasis on this) prediction for Linux in the next few years, Linus just quipped back saying that "boldest prediction I can say is, I will probably release rc1 in about a week”. Kernel panel, moderated by Jonathan Corbet, LWN editor/Kernel dev ----------------------------------------------------------------- Right after the previous talk followed a Kernel panel discussion moderated by Jonathan Corbet. I only attended the first five minutes where Corbet opened the discussion with the recent remarks by Lennart Poettering about open source communities on G+. Rik Van Riel later on a walk mentioned that the discussion was more interesting after I left. An Update (and Demo) on Techniques to Manage libvirt/QEMU-based Virtual Machine Snapshots and Disk Image Chains ----------------------------------------------------------------------- I gave a talk and a brief demo on managing snapshots/disk image chains with libvirt/QEMU. - Slides: http://events.linuxfoundation.org/sites/events/files/slides/Update-on-QEMU-and-libvirt-snapshots-disk-image-chains-CloudOpen-Eu-2014.pdf - Some examples with a bit of commentary: https://kashyapc.fedorapeople.org/virt/lcco-2014/examples/ (Thanks to Eric Blake and Stefan Hajnoczi for taking time to review some of the above.) === Day-4, Thu === OpenStack Performance Optimization with NUMA, Huge Pages and CPU Pinning - Daniel Berrange, Red Hat ---------------------------------------------------------------- Daniel Berrange began with an overview of Nova, its concepts and architecture (simplified one), focusing on these interactions between Nova <-> libvirt <-> KVM. Then he moved on to discuss about virtual machine scheduling and its goals in the context of Network Functions Virtualization (NFV) and maximising hardware utilization. Next he discussed four specific topics, and their configuration details in the context of Nova and libvirt: NUMA config, Large pages, CPU pinning, I/O devices (assigining PCI devices on a local NUMA node). Slides of the talk: http://people.redhat.com/berrange/kvm-forum-2014/kvm-forum-2014-openstack-perf.pdf Other talks that might be interesting to Virtualization folks ------------------------------------------------------------- Some of these I missed due to other overlapping discussions elsewhere/hallway tracks or I attended only partially, nevertheless I thought can be interesting for those deeply involved in virtualization. (a) Automatic NUMA Balancing - Rik van Riel, Red Hat. - Slides: http://events.linuxfoundation.org/sites/events/files/slides/kvmplumbers2014_riel_automatic_numa_balancing_1.pdf - I came at the end of this talk when Rik was taking Q/A and encouraged the audience to test more. (b) Nested Virtualization - State of the Art and Future Directions - Jan Kiszka, Siemens; Bandan Das, Red Hat - No slides yet. (c) Towards Multi-Threaded Device Emulation in QEMU - Stefan Hajnoczi, Red Hat. - Slides: http://vmsplice.net/~stefan/stefanha-kvm-forum-2014.pdf - This was a pretty low-level discussion of bottlenecks in QEMU's device layer. (d) Memory Externalization With userfaultfd - Andrea Arcangeli & Dr. David Alan Gilbert, Red Hat. - Slides: http://events.linuxfoundation.org/sites/events/files/slides/kvmforum2014.pdf - Memory externalization is about running a virtual machine/or any program with part (or all) of its memory residing on a remote node. Andrea is introducing is two system calls for this, here's his patchset: http://lists.gnu.org/archive/html/qemu-devel/2014-10/msg00386.html - Also, related is Red Hat's David Alan Gilbert's upstream work of post-copy based live migration which uses Andrea's new system calls introduced above: http://wiki.qemu.org/Features/PostCopyLiveMigration If live migration is your subject of interest, you might want to apply Andrea's patch series above and provide some testing feedback. - A related talk: Migration: Trying to make it more robust - Juan Quintela, Red Hat (e) Deterministic Replay and Reverse Debugging in QEMU - Pavel Dovgalyuk, ISP - Slides: http://events.linuxfoundation.org/sites/events/files/slides/presentation_final.pdf (f) More Block Device Configuration - Kevin Wolf, Max Reitz, Red Hat - Slides: http://events.linuxfoundation.org/sites/events/files/slides/blockdev.pdf (g) KVM vs. Valgrind - Christian Bornträger, IBM Germany - http://events.linuxfoundation.org/sites/events/files/slides/valgrind%20and%20KVM.pdf Slides for most of talks should be available, for some they still needs to be uploaded. The recordings of the talks are also being posted to the Youtube's KVM Forum channel, a few of them are up, keep an eye on it for the rest. Hallway/People ============== I had a chance to catch up with several folks from KVM/QEMU/libvirt and Linux communities. Plenty of people to name, in short I met most folks from KVM, QEMU, libvirt contributors and several community contributors from different open source projects. That's it, folks!