Day-1 ----- Network Management and SDN developer room ----------------------------------------- [I myself need to revisit this talk since I didn't take any notes, but thought I'd mention them here.] - Connection tracking and stateful services with Open vSwitch, by Thomas Graf - https://fosdem.org/2015/schedule/event/stateful_open_vswitch/ - DPDK performance (How to not just do a demo with DPDK), by Stephen Hemminger - https://fosdem.org/2015/schedule/event/dpdk_performance/ - A very interesting talk with full of networking wisdom and talks a lot about performance aspects as well. Day-2 ----- What's new in systemd, 2015 edition, by Lennart Poettering ---------------------------------------------------------- This was a no-slides talk (but from time to time Lennart consulted his phone for topics) in a standing-only room. Although Lennart speaks quite fast, this talk had plenty of interesting information with a good summary on a range of topics. I took my notes on a phone while standing in a corner, hence the truncated bullet-style points. - `systemd-nspawn` - Now you can deploy using nspawn - nspawn can boot qcow2 images - nspawn is just a single source file - journalctl can show all the logs from containers - machined - Minimal firewall support in systemd - Features for administrators - `systemctl-edit`, `systemctl-cat` - 'ping gateway' to check if you're connected to a working network - 'systemd-networkd' - Native DHCP, integrates very well with nspawn containers - This does not involve `dnsmasq` - Auditing: is a mess/disaster. In _theory_ has a nice functionality, nice audit client, thanks to `journalctl` indexing everything, and the audit tool itself - Stateless systems - Only /usr no binaries - Unpopulated /etc, system still boots just fine. Every system can be initialized this way - Complexity in /etc/passwd, hence `systemd-sysusers` (reates system users and groups, based on the file format and location specified in sysusers.d) - 'systemd-resolved' - DNS concepts like cache flusing goes away - Name resolution with real DNS - Next on the list: DNSSEC support - Implement Avahi support and get rid of Avahi itself - Journal remoting (`systemd-journal-remote`) - Remoting meanins using HTTP. Uses a pull/push model to recieve journal data - `systemd-coredump` - It is actually *useful* now, withouth having GDB - Very useful for embedded devices - Uses a coredump directory - Tools around coredump are very usfeul: `coredumpctl` - Protection systems for services - Like a mini sandbox - "boot this system normally, but mark /usr as read-onnly." - Read-only or private /home - Private /tmp, private 'devices' - You can use device cgroups controller, but 'private devices' is more 'easy' and is a one-time thing. - `systemd-timesyncd` - Trivial SNTP (Simple Network Time Protocol) client. Does minimal stuff that's necessary. - Has nice integration with `networkd` - Touch a file in /var and read that on boot - Announcement: Merge gummiboot[*] into systemd. "People are going to hate it, but we're going to merge it anyway. We have a strong rationale. If you want to make sure, this the way to go! This is not just about laptops, makes data centers secure. We want to have pluralism." [*] http://freedesktop.org/wiki/Software/gummiboot/ Overall, outlook for systemd in 2015 looks very great. LXD: The container hypervisor, by Serge Hallyn ---------------------------------------------- - Containers - "userspace fiction based on diverse Kernel features" - From the inside, looks ike a VM - From the outside, looks like a process - API - Prior to API, programs had to "shell out"; with API, you can just 'import lxc' (Python bindings) - User namespaces support in 2013 - Goals: - uid separation; container root privileged over container; container root has no privilege outside of container - Uids map 1-1 to kuids - Other namespaces (i.e. network) owned by userns - Summary: root inside container is unprevileged - Download template - images.linuxcontainers.org = base image repository - Solves bootstrap problem - Solves unprivileged container creation problem - What is LXD? - New project under LXC community - Based on Linux Containers (LXC) - Using lxc golang API bindings - Secure by default: user namespaces, cgroups - Daemon - A REST API for managing system containers - Framework fo managing container images - Command workflow Observability in KVM, by Stefan Hajnoczi ---------------------------------------- Quick set of topics the talk touches: - dmesg: OOM killer, seg faults, errors from kvm.ko - Tracing for performance analysis - `strace -f` - kvm.ko kernel module trace events available via `perf` and `trace-cmd` - SystemTap tapset: combines host kernel and QEMU traces - vmstat, tcpdump, ps - Network packet capture - CPU - CPU utilization breakdown on KVM hosts - `mpstat` (from sysstat) - Is my cloud guest getting enough CPU? Reported as %steam time in `mpstat` - vCPU execution - Observe low-level events with `kvm_stat` - Networking - vhost_net with bridged networking: popular config - Troubleshooting bridged networking - tcpdump eth0 inside guest; tcpdump vnet0 on the host; tcpdump virbr0 on the host; tcpdump eth0 on the host - Host-wide interface statistics - `netstat -i` - NAT networking config - virtio-net <-> vhost_net <-> tun <-> bridge <-> eth0 [Guest Kernel] [-----------Host Kernel-------------] - Troubleshoot DHCP and DNS - (host) $ journalctl -r | head # or syslog - Disk I/O - Popular LVM local disk config - Storage provided to guest as virtio-blk PCI adapter - QEMU typically configured with 'cache=none' to bypass hostpage cache - Notes from Rich Jones on QEMU cache modes: http://rwmj.wordpress.com/2013/09/02/new-in-libguestfs-allow-cache-mode-to-be-selected/ - Benchmarking disk performance - Use `fio --direct=1` ($ yum install fio) for benchmarking to bypass page cache - Use `fio --rw=randwrite` for random pattern that avoids QEMU virtio-blk write merging - I/O stats with `iostat` - `iostat -k -x 1` - Compare guest and host to identify unexpected changes - I/O patterns with `blktrace` - To study the exact patterns of I/O requests I definitely recommend this talk if you intend to learn a little more about systematically troubleshooting KVM issues (CPU, Networking, I/O). Abstract: https://fosdem.org/2015/schedule/event/observability/ Slides are here: https://fosdem.org/2015/schedule/event/observability/attachments/slides/657/export/events/attachments/observability/slides/657/stefanha_fosdem_2015.pdf Advanced Snapshots with libvirt and QEMU ---------------------------------------- This was a talk given by yours truly as part of Infrastructure.Next. I had about 40-50 people audience, and being the first talk of the day, I had the most active audience (had at-least 12 questions) and had interesting offline feeback as well from deployers/users. Abstract is here: https://kashyapc.fedorapeople.org/virt/infra.next-2015/Advanced-Snapshots-with-libvirt-and-QEMU.txt Slides: https://kashyapc.fedorapeople.org/virt/infra.next-2015/Advanced-Snapshots-with-libvirt-and-QEMU.pdf