Notes for Day-1 KVMForum
========================

KVM Keynote
-----------

- KVM-RT latency jitter plot
   - `cyclictest`

- Perf. improvements for guests spanning multiple NUMA nodes


Real-time KVM  (Rik Van Riel & Jan Kiszka)
------------------------------------------

- Real-time is about determinism, not speed
- Hardware pitfalls
   - Biggest problems: BIOS, BIOS, and BIOS


Real-Time Virt Drivers

- Control systems 
   - industry, healthcare, transportation


Pushing the limits: 1000 guests per host & beyond
-------------------------------------------------

- Host: adding network devices
  - 2048 macvtap devices

  - System limits hit:
     - Linux bridges limited t0 1024 ports --> add more bridges
     - Max no. of IPv6 routes

- Attaching 4096 disks to the host
   - System limits hit
       - Max no. of inotify watches hit

- Counting file descriptors
   - Mostly eventfds, one per virtqueue
   - For exmple: a virtio-net device is using
      - one virtqueue for TX/RX
      - ?
      - /dev/tap
      - /dev/vhost-net + 3 irqfds
      - 8 FDs in total

- Attaching 4096 disks to the guest

- Using 16384 virtio-block disks

- Starting 4096 guests
   - Many system limits had to be tweated
     - Max. file descriptors for libvirtd
     - Increasing max PTYs
     - Max. processes/threads exceeded
     - Not enough memory -- increased to 2TB

   - "systemd says hello :)"
      - systemd-machined
      - Libvirt calls systemd-machine::CreateMachine() for every guest
        started
      - Message handling in older versions (at least until v208) for
        every guest started
      - CreateMachine() timed outbecause of buggy message handling in
        systemd
         - Fixed in systemd v-219

   - Running CPU and network stress tools

   - Strict overcommit & glibc per-thread arenas
      - malloc() fails much earlier
      - Capped by MALLOC_CAP_ARENA

- Migrating manu guests concurrently -- TCP port range for migration
   - SSHD has a default of 10 sessions
   - When testing migration of > 64 guests increase this limit
   - Change /etc/ssh/sshd_config MaxSessions to a higher number:

- QEMU: Memslot limitation
   - Currently we have only one memslot

- KVM: IRQchip limit
   - The in-kernel irqchip is limited to 1024 pins
   - For s390 already extended to 4906
   - When limit exceeded, devices fall back to QEMU internal irqfd
     handling
   - Problem: one irqfd is set up per virtqueue (mapping to queue
     indicator), so queues per irqchip are exhausted fast
      - virtio-net devices possible = 4096 / 3 virtqueues

- Conclusions
   - Linux (& KVM) scales quite well, most often a matter of adjusting
     ulimits
      - ipv6.sysctl.ip6_rt_max_size
      - sshd: #MaxSessions 128
      - libvirtd: min_port_range, max_port_range
      - systemd: LimitNOFILE
      - sysctl: kernel.pty.max, fs.aui-max-nr,
        fs.inotify.max_user_watches

- In-Kernel IRQchip limits virtio devices
- Apart from some smaller issues, no major code findings in
  libvirt/QEMU/KVM
- No fundamental design problems in our stack: libvirt/QEMU/KVM

Migratable 40GBit/s Ethernet - Alexander Graf
---------------------------------------------

Notes on my phone's notebook.


Status update on KVM-COLO HA/FT solution
----------------------------------------

TO-WATCH


virtio-vsock: Zero-configuration host/guest communication
---------------------------------------------------------

- AF_VSOCK in Linux
   - New socket address family for host/guest communication
   - Supports datagram and stream semantics
   - Addresses are <u32 cid, u32 port>
      - Each guest has unique cid
      - Host has well-known cid
      -

- virtio-vsock device design
   - rx/tx virtqueue pair for data exchange
      - Each packet has a header w/ addressing and state information
      - Packets from all flows use single rx/tx virtqueue pair
      - Credit-based flow control for reliable connections
      - Best-effort for unreliable connections
   -


Guest operating system debugging - David Hildenbrand
----------------------------------------------------

TO-WATCH

Slides:
http://www.linux-kvm.org/images/9/92/01x10-David_Hildebrand-Guest-operating_system_debugging.pdf