Procedure to configure nested virtualization
============================================

Intel-based hosts
-----------------

1. List modules and ensure KVM Kernel modules are enabled on the
   physical host::

      $ lsmod | grep -i kvm
      kvm_intel             133627  0
      kvm                   435079  1 kvm_intel

2. Show information for ``kvm_intel`` module::

      $ modinfo kvm_intel | grep -i nested
      parm:           nested:boolkvm                   435079  1 kvm_intel

3. Ensure nested virt is persistent across reboots by adding it as a
   config directive::

      $ cat /etc/modprobe.d/dist.conf
      options kvm-intel nested=y

4. Temporarily remove the KVM Intel kernel module, enable nested
   virtualization to be persistent across reboots and add the Kernel
   module back::

      $ sudo rmmod kvm-intel
      $ sudo sh -c "echo 'options kvm-intel nested=1' \
          >> /etc/modprobe.d/dist.conf"
      $ sudo modprobe kvm-intel

5. Check if the Nested KVM Kernel module option is enabled::

      $ cat /sys/module/kvm_intel/parameters/nested
      Y

6. Before you boot your level-1 guest (i.e. the guest hypervisor that
   runs the nested guest), expose virtualization extensions to it. The
   following exposes all the CPU features of host to your guest
   unconditionally::

      # This ``virt-xml`` tool is part of 'virt-install' package
      $ virt-xml guest-hyp \
          --edit \
          --cpu host-passthrough,clearxml=yes

7. Start your level-1 guest (i.e. guest hypervisor)::

      $ virsh start guest-hyp --console

8. Ensure KVM extensions are enabled by checking if the character device
   ``/dev/kvm`` is present in the level-1 guest::

      <guest-hyp>$ file /dev/kvm
      /dev/kvm: character special


9. Start your level-2 (or nested) guest::

      <guest-hyp>$ virsh start nested-guest


10. If the Intel hardware is sufficiently advanced (Intel Haswell
    processor or above that has newer hardware virt extensions), you
    might want to enable Shadow VMCS, APIC Virtualization on the
    physical host::

      $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
      Y

      $ cat /sys/module/kvm_intel/parameters/enable_apicv
      Y

      $ cat /sys/module/kvm_intel/parameters/ept
      Y

Instructions for AMD
--------------------

(1) Enable the ``nested`` parameter for AMD Kernel module::

      $ cat /sys/module/kvm_amd/parameters/nested
      0

      $ rmmod kvm-amd
      $ modprobe kvm-amd nested=1

      $ cat /sys/module/kvm_amd/parameters/nested
      1

(2) To make the above value persistent across reboots, add an entry in
    ``/etc/modprobe.dist.conf`` so it looks as below::

      $ cat /etc/modprobe.d/dist.conf
      options kvm-amd nested=y