Setup TLS for NBD and (live) migration streams

QEMU (2.11 onwards) and libvirt (4.4.0 onwards) have gained support for “native TLS”, i.e. TLS built into QEMU. This will secure all data transports, including disks that are not on shared storage, without incurring the limitations of the “tunnelled via libvirtd” transport.

The following is a “from first principles” guide to configure TLS for two guest hypervisors (using nested KVM). This will allow live migration of guests without shared storage setup via TLS.

  1. Setup two guest hypervisors:

    $ virt-builder fedora-28 -o guestHyp1.qcow2 --update \
    --format qcow2 --selinux-relabel --size 80G \
    --root-password password:123456 --install \
    "libguestfs-tools-c libvirt-daemon-kvm virt-install \
    libvirt-daemon-config-network libvirt-daemon-config-nwfilter "

$ virt-builder fedora-28 -o guestHyp2.qcow2 --update \
    --format qcow2 --selinux-relabel --size 80G \
    --root-password password:123456 --install \
    "libguestfs-tools-c libvirt-daemon-kvm virt-install \
    libvirt-daemon-config-network libvirt-daemon-config-nwfilter "

  2. Import them into libvirt:

    $ virt-install --name guestHyp1 --ram 8096 \
  --disk path=./guestHyp2.qcow2,format=qcow2 \
  --machine q35 --os-variant fedora27 \
  --cpu host-passthrough --nographics --import

$ virt-install --name guestHyp1 --ram 8096 \
  --disk path=./guestHyp2.qcow2,format=qcow2 \
  --machine q35 --os-variant fedora27 \
  --cpu host-passthrough --nographics --import

  3. From guestHyp1, start the nested guest:

    $ virt-builder fedora-28 -o nGuest1 --update \
    --format qcow2 --selinux-relabel --size 20G  \
    --root-password password:fedora

$ virt-install --name nGuest1 --ram 2048 \
    --disk path=./nGuest1.qcow2,format=qcow2 --machine q35 \
    --os-variant fedora27 --cpu host-passthrough --nographics \
    --import --print-xml > /tmp/nGuest1.xml

$ virsh define /tmp/nGuest1.xml

Note

The above approach of: virt-install –print-xml, followed by virsh define will let us script it.)

  1. Setup password-less SSH on both guest hypervisors, guestHyp1 and guestHyp2:

    $ ssh-keygen -t rsa
$ ssh-copy-id root@guestHyp$X
$ ssh root@guestHyp$X

  2. Stop firewalld on both guestHyp1 and guestHyp2:: (otherwise you get: error: internal error: unable to execute QEMU command 'drive-mirror': Failed to connect socket: No route to host)

    $ systemctl stop firewalld

Migration without TLS

  1. Migrate ‘nGuest1’ from guestHyp1 to guestHyp2, and vice-versa; without TLS):

    $ virsh migrate --verbose --copy-storage-all \
    --live nguest1 qemu+ssh://guestHyp2/system

$ virsh migrate --verbose --copy-storage-all \
    --live nGuest1 qemu+ssh://guestHyp1/system

Setup TLS

(7.a) On both hosts:

$ mkdir -p /etc/pki/qemu
$ mkdir -p /etc/pki/libvirt/private
$ mkdir -p /etc/pki/CA

(7.b) On guestHyp1, create the CA cert and SCP it to guestHyp2:

$ cd /home
$ mkdir pki && cd pki

$ cat ca.info
cn = JAF Migration TLS Test
ca
cert_signing_key
expiration_days = 700

$ certtool --generate-privkey > cakey.pem

$ certtool --generate-self-signed --load-privkey cakey.pem \
  --template ca.info --outfile cacert.pem
$ cp cacert.pem /etc/pki/CA/
$ scp cacert.pem guestHyp2:/etc/pki/CA/

(7.c) On guestHyp1, generate server certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 cert to its host:

$ cat guestHyp1-server.info
organization = TLS Migration Test
cn = guestHyp1
tls_www_server
encryption_key
signing_key

$ cat guestHyp2-server.info
organization = TLS Migration Test
cn = guestHyp2
tls_www_server
encryption_key
signing_key

$ certtool --generate-privkey > guestHyp1-serverkey.pem
Generating a 3072 bit RSA private key...

$ certtool --generate-privkey > guestHyp2-serverkey.pem
Generating a 3072 bit RSA private key...

$ certtool --generate-certificate --template guestHyp1-server.info \
       --load-privkey guestHyp1-serverkey.pem \
       --load-ca-certificate cacert.pem \
       --load-ca-privkey cakey.pem \
       --outfile guestHyp1-servercert.pem

$ certtool --generate-certificate --template guestHyp2-server.info \
       --load-privkey guestHyp2-serverkey.pem \
       --load-ca-certificate cacert.pem \
       --load-ca-privkey cakey.pem \
       --outfile guestHyp2-servercert.pem

$ cp guestHyp1-servercert.pem /etc/pki/libvirt/servercert.pem
$ cp guestHyp1-serverkey.pem /etc/pki/libvirt/private/serverkey.pem
$ scp guestHyp2-servercert.pem guestHyp2:/etc/pki/libvirt/servercert.pem
$ scp guestHyp2-serverkey.pem guestHyp2:/etc/pki/libvirt/private/serverkey.pem

(7.d) On guestHyp1, modify the permissions:

$ chown root:qemu /etc/pki/libvirt
$ chown root:qemu /etc/pki/libvirt/*
$ chown root:qemu /etc/pki/libvirt/private/*

(7.e) On both hosts, guestHyp1 and guestHyp2, adjust permissoins for their corresponding server certificates:

$ chmod 440 /etc/pki/libvirt/servercert.pem
$ chmod 750 /etc/pki/libvirt/private/
$ chmod 600 /etc/pki/libvirt/private/serverkey.pem

(7.f) On guestHyp1, generate client certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 client cert to its host:

$ certtool --generate-privkey > guestHyp1-clientkey.pem
$ certtool --generate-privkey > guestHyp2-clientkey.pem

$ cat guestHyp1-client.info
country = BE
state = EF
locality = Belgium
organization = TLS Migration Test
cn = guestHyp1
tls_www_client
encryption_key
signing_key

$ cat guestHyp2-client.info
country = BE
state = EF
locality = Belgium
organization = TLS Migration Test
cn = guestHyp1
tls_www_client
encryption_key
signing_key

$ certtool --generate-certificate \
         --template guestHyp1-client.info \
         --load-privkey guestHyp1-clientkey.pem \
         --load-ca-certificate cacert.pem \
         --load-ca-privkey cakey.pem \
         --outfile guestHyp1-clientcert.pem

$ certtool --generate-certificate \
         --template guestHyp2-client.info \
         --load-privkey guestHyp2-clientkey.pem \
         --load-ca-certificate cacert.pem \
         --load-ca-privkey cakey.pem \
         --outfile guestHyp2-clientcert.pem

$ cp guestHyp1-clientcert.pem /etc/pki/libvirt/clientcert.pem
$ cp guestHyp1-clientkey.pem /etc/pki/libvirt/private/clientkey.pem
$ scp guestHyp2-clientcert.pem guestHyp2:/etc/pki/libvirt/clientcert.pem
$ scp guestHyp2-clientkey.pem guestHyp2:/etc/pki/libvirt/private/clientkey.pem

(7.g) On both hosts, guestHyp1 and guestHyp2:

$ chmod 400 /etc/pki/libvirt/clientcert.pem
$ chmod 644 /etc/pki/libvirt/private/clientkey.pem

(7.h) On both, because QEMU requires the /etc/pki/qemu directory contents to be slightly different:

$ cp /etc/pki/CA/cacert.pem /etc/pki/qemu/ca-cert.pem
$ cp /etc/pki/libvirt/servercert.pem /etc/pki/qemu/server-cert.pem
$ cp /etc/pki/libvirt/private/serverkey.pem /etc/pki/qemu/server-key.pem
$ cp /etc/pki/libvirt/clientcert.pem /etc/pki/qemu/client-cert.pem
$ cp /etc/pki/libvirt/private/clientkey.pem /etc/pki/qemu/client-key.pem

(7.i) On both, guestHyp1, and guestHyp2, update ‘x509’ config options in /etc/libvirt/qemu.conf:

  default_tls_x509_cert_dir = "/etc/pki/qemu"
  default_tls_x509_verify = 1

And modify /etc/sysconfig/libvirtd on both (guestHyp1 &
guestHyp2)::

  LIBVIRTD_ARGS="--listen"

And restart libvirt daemon (also on both hosts)::

  $ systemctl restart libvirtd

  1. Run virt-pki-validate on guestHyp1 and guestHyp2:

    $ virt-pki-validate
Found /usr/bin/certtool
Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test
Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp1
Found client private key /etc/pki/libvirt/private/clientkey.pem
Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp1
Found server private key /etc/pki/libvirt/private/serverkey.pem
Make sure /etc/sysconfig/libvirtd is setup to listen to
TCP/IP connections and restart the libvirtd service

$ virt-pki-validate
Found /usr/bin/certtool
Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test
Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp2
Found client private key /etc/pki/libvirt/private/clientkey.pem
Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp2
Found server private key /etc/pki/libvirt/private/serverkey.pem
Make sure /etc/sysconfig/libvirtd is setup to listen to
TCP/IP connections and restart the libvirtd service

  2. IMPORTANT: Ensure the permissions of certificate files and keys in /etc/pki/qemu/* directory on both source and destination to be the following:

    [root@guestHyp1 qemu]$ ls -lasrtZ /etc/pki/qemu/
0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0      110 Dec 10 10:39 ..
4 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:08 ca-cert.pem
4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:08 server-cert.pem
4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:09 client-cert.pem
8 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:09 client-key.pem
4 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 2459 Dec 11 05:32 server-key-stripped.pem
8 -rw----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8177 Dec 11 05:35 server-key.pem
0 drwxr-xr-x.  2 root root unconfined_u:object_r:cert_t:s0  146 Dec 11 06:01 .

[root@guestHyp2 ~]# ls -lasrtZ /etc/pki/qemu/
total 28
0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0      110 Dec 10 10:39 ..
4 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:10 ca-cert.pem
4 -r--r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:10 server-cert.pem
8 -rw-------.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8170 Dec 10 11:10 server-key.pem
4 -r-----r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:10 client-cert.pem
8 -rw-r--r--.  1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:10 client-key.pem
0 drwxr-xr-x.  2 root root unconfined_u:object_r:cert_t:s0  115 Dec 10 11:10 .

Migration with TLS

  1. Migrate ‘nGuest1’ from guestHyp1 to guestHyp2 with TLS:

    $ virsh migrate --tls --verbose --copy-storage-all \
    --migrate-disks vda nGuest1 --live qemu+ssh://guestHyp2/system