Setup TLS for NBD and (live) migration streams ============================================== QEMU (2.11 onwards) and libvirt (4.4.0 onwards) have gained support for "native TLS", i.e. TLS built into QEMU. This will secure all data transports, including disks that are not on shared storage, without incurring the limitations of the "tunnelled via libvirtd" transport. The following is a "from first principles" guide to configure TLS for two guest hypervisors (using nested KVM). This will allow live migration of guests without shared storage setup via TLS. (1) Setup two guest hypervisors:: $ virt-builder fedora-28 -o guestHyp1.qcow2 --update \ --format qcow2 --selinux-relabel --size 80G \ --root-password password:123456 --install \ "libguestfs-tools-c libvirt-daemon-kvm virt-install \ libvirt-daemon-config-network libvirt-daemon-config-nwfilter " $ virt-builder fedora-28 -o guestHyp2.qcow2 --update \ --format qcow2 --selinux-relabel --size 80G \ --root-password password:123456 --install \ "libguestfs-tools-c libvirt-daemon-kvm virt-install \ libvirt-daemon-config-network libvirt-daemon-config-nwfilter " (2) Import them into libvirt:: $ virt-install --name guestHyp1 --ram 8096 \ --disk path=./guestHyp2.qcow2,format=qcow2 \ --machine q35 --os-variant fedora27 \ --cpu host-passthrough --nographics --import $ virt-install --name guestHyp1 --ram 8096 \ --disk path=./guestHyp2.qcow2,format=qcow2 \ --machine q35 --os-variant fedora27 \ --cpu host-passthrough --nographics --import (3) From guestHyp1, start the nested guest:: $ virt-builder fedora-28 -o nGuest1 --update \ --format qcow2 --selinux-relabel --size 20G \ --root-password password:fedora $ virt-install --name nGuest1 --ram 2048 \ --disk path=./nGuest1.qcow2,format=qcow2 --machine q35 \ --os-variant fedora27 --cpu host-passthrough --nographics \ --import --print-xml > /tmp/nGuest1.xml $ virsh define /tmp/nGuest1.xml .. note:: The above approach of: `virt-install --print-xml`, followed by `virsh define` will let us script it.) (4) Setup password-less SSH on both guest hypervisors, guestHyp1 and guestHyp2:: $ ssh-keygen -t rsa $ ssh-copy-id root@guestHyp$X $ ssh root@guestHyp$X (5) Stop ``firewalld`` on both guestHyp1 and guestHyp2:: (otherwise you get: ``error: internal error: unable to execute QEMU command 'drive-mirror': Failed to connect socket: No route to host``) :: $ systemctl stop firewalld Migration without TLS --------------------- (6) Migrate 'nGuest1' from guestHyp1 to guestHyp2, and vice-versa; *without* TLS):: $ virsh migrate --verbose --copy-storage-all \ --live nguest1 qemu+ssh://guestHyp2/system $ virsh migrate --verbose --copy-storage-all \ --live nGuest1 qemu+ssh://guestHyp1/system Setup TLS --------- (7.a) On *both* hosts:: $ mkdir -p /etc/pki/qemu $ mkdir -p /etc/pki/libvirt/private $ mkdir -p /etc/pki/CA (7.b) On guestHyp1, create the CA cert and SCP it to guestHyp2:: $ cd /home $ mkdir pki && cd pki $ cat ca.info cn = JAF Migration TLS Test ca cert_signing_key expiration_days = 700 $ certtool --generate-privkey > cakey.pem $ certtool --generate-self-signed --load-privkey cakey.pem \ --template ca.info --outfile cacert.pem $ cp cacert.pem /etc/pki/CA/ $ scp cacert.pem guestHyp2:/etc/pki/CA/ (7.c) On guestHyp1, generate server certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 cert to its host:: $ cat guestHyp1-server.info organization = TLS Migration Test cn = guestHyp1 tls_www_server encryption_key signing_key $ cat guestHyp2-server.info organization = TLS Migration Test cn = guestHyp2 tls_www_server encryption_key signing_key $ certtool --generate-privkey > guestHyp1-serverkey.pem Generating a 3072 bit RSA private key... $ certtool --generate-privkey > guestHyp2-serverkey.pem Generating a 3072 bit RSA private key... $ certtool --generate-certificate --template guestHyp1-server.info \ --load-privkey guestHyp1-serverkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp1-servercert.pem $ certtool --generate-certificate --template guestHyp2-server.info \ --load-privkey guestHyp2-serverkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp2-servercert.pem $ cp guestHyp1-servercert.pem /etc/pki/libvirt/servercert.pem $ cp guestHyp1-serverkey.pem /etc/pki/libvirt/private/serverkey.pem $ scp guestHyp2-servercert.pem guestHyp2:/etc/pki/libvirt/servercert.pem $ scp guestHyp2-serverkey.pem guestHyp2:/etc/pki/libvirt/private/serverkey.pem (7.d) On guestHyp1, modify the permissions:: $ chown root:qemu /etc/pki/libvirt $ chown root:qemu /etc/pki/libvirt/* $ chown root:qemu /etc/pki/libvirt/private/* (7.e) On *both* hosts, guestHyp1 and guestHyp2, adjust permissoins for their corresponding server certificates:: $ chmod 440 /etc/pki/libvirt/servercert.pem $ chmod 750 /etc/pki/libvirt/private/ $ chmod 600 /etc/pki/libvirt/private/serverkey.pem (7.f) On guestHyp1, generate client certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 client cert to its host:: $ certtool --generate-privkey > guestHyp1-clientkey.pem $ certtool --generate-privkey > guestHyp2-clientkey.pem $ cat guestHyp1-client.info country = BE state = EF locality = Belgium organization = TLS Migration Test cn = guestHyp1 tls_www_client encryption_key signing_key $ cat guestHyp2-client.info country = BE state = EF locality = Belgium organization = TLS Migration Test cn = guestHyp1 tls_www_client encryption_key signing_key $ certtool --generate-certificate \ --template guestHyp1-client.info \ --load-privkey guestHyp1-clientkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp1-clientcert.pem $ certtool --generate-certificate \ --template guestHyp2-client.info \ --load-privkey guestHyp2-clientkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp2-clientcert.pem $ cp guestHyp1-clientcert.pem /etc/pki/libvirt/clientcert.pem $ cp guestHyp1-clientkey.pem /etc/pki/libvirt/private/clientkey.pem $ scp guestHyp2-clientcert.pem guestHyp2:/etc/pki/libvirt/clientcert.pem $ scp guestHyp2-clientkey.pem guestHyp2:/etc/pki/libvirt/private/clientkey.pem (7.g) On *both* hosts, guestHyp1 and guestHyp2:: $ chmod 400 /etc/pki/libvirt/clientcert.pem $ chmod 644 /etc/pki/libvirt/private/clientkey.pem (7.h) On *both*, because QEMU requires the ``/etc/pki/qemu`` directory contents to be slightly different:: $ cp /etc/pki/CA/cacert.pem /etc/pki/qemu/ca-cert.pem $ cp /etc/pki/libvirt/servercert.pem /etc/pki/qemu/server-cert.pem $ cp /etc/pki/libvirt/private/serverkey.pem /etc/pki/qemu/server-key.pem $ cp /etc/pki/libvirt/clientcert.pem /etc/pki/qemu/client-cert.pem $ cp /etc/pki/libvirt/private/clientkey.pem /etc/pki/qemu/client-key.pem (7.i) On *both*, guestHyp1, and guestHyp2, update 'x509' config options in ``/etc/libvirt/qemu.conf``:: default_tls_x509_cert_dir = "/etc/pki/qemu" default_tls_x509_verify = 1 And modify /etc/sysconfig/libvirtd on both (guestHyp1 & guestHyp2):: LIBVIRTD_ARGS="--listen" And restart libvirt daemon (also on both hosts):: $ systemctl restart libvirtd (8) Run ``virt-pki-validate`` on guestHyp1 and guestHyp2:: $ virt-pki-validate Found /usr/bin/certtool Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp1 Found client private key /etc/pki/libvirt/private/clientkey.pem Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp1 Found server private key /etc/pki/libvirt/private/serverkey.pem Make sure /etc/sysconfig/libvirtd is setup to listen to TCP/IP connections and restart the libvirtd service $ virt-pki-validate Found /usr/bin/certtool Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp2 Found client private key /etc/pki/libvirt/private/clientkey.pem Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp2 Found server private key /etc/pki/libvirt/private/serverkey.pem Make sure /etc/sysconfig/libvirtd is setup to listen to TCP/IP connections and restart the libvirtd service (9) **IMPORTANT**: *Ensure* the permissions of certificate files and keys in ``/etc/pki/qemu/*`` directory on both source *and* destination to be the following:: [root@guestHyp1 qemu]$ ls -lasrtZ /etc/pki/qemu/ 0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0 110 Dec 10 10:39 .. 4 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:08 ca-cert.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:08 server-cert.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:09 client-cert.pem 8 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:09 client-key.pem 4 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 2459 Dec 11 05:32 server-key-stripped.pem 8 -rw----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8177 Dec 11 05:35 server-key.pem 0 drwxr-xr-x. 2 root root unconfined_u:object_r:cert_t:s0 146 Dec 11 06:01 . [root@guestHyp2 ~]# ls -lasrtZ /etc/pki/qemu/ total 28 0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0 110 Dec 10 10:39 .. 4 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:10 ca-cert.pem 4 -r--r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:10 server-cert.pem 8 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8170 Dec 10 11:10 server-key.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:10 client-cert.pem 8 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:10 client-key.pem 0 drwxr-xr-x. 2 root root unconfined_u:object_r:cert_t:s0 115 Dec 10 11:10 . Migration with TLS ------------------ (10) Migrate 'nGuest1' from guestHyp1 to guestHyp2 *with* TLS:: $ virsh migrate --tls --verbose --copy-storage-all \ --migrate-disks vda nGuest1 --live qemu+ssh://guestHyp2/system