Full virtual setup for NBD and migration streams over TLS ========================================================= (1) Setup two guest hypervisors:: $ virt-builder fedora-28 -o guestHyp1.qcow2 --update \ --format qcow2 --selinux-relabel --size 80G \ --root-password password:123456 --install \ "libguestfs-tools-c libvirt-daemon-kvm virt-install \ libvirt-daemon-config-network libvirt-daemon-config-nwfilter " $ virt-builder fedora-28 -o guestHyp2.qcow2 --update \ --format qcow2 --selinux-relabel --size 80G \ --root-password password:123456 --install \ "libguestfs-tools-c libvirt-daemon-kvm virt-install \ libvirt-daemon-config-network libvirt-daemon-config-nwfilter " (2) Import them into libvirt:: $ virt-install --name guestHyp1 --ram 8096 \ --disk path=./guestHyp2.qcow2,format=qcow2 \ --machine q35 --os-variant fedora27 \ --cpu host-passthrough --nographics --import $ virt-install --name guestHyp1 --ram 8096 \ --disk path=./guestHyp2.qcow2,format=qcow2 \ --machine q35 --os-variant fedora27 \ --cpu host-passthrough --nographics --import (3) From guestHyp1, start the nested guest:: $ virt-builder fedora-28 -o nGuest1 --update \ --format qcow2 --selinux-relabel --size 20G \ --root-password password:fedora $ virt-install --name nGuest1 --ram 2048 \ --disk path=./nGuest1.qcow2,format=qcow2 --machine q35 \ --os-variant fedora27 --cpu host-passthrough --nographics \ --import --print-xml > /tmp/nGuest1.xml $ virsh define /tmp/nGuest1.xml .. note:: The above approach of: `virt-install --print-xml`, followed by `virsh define` will let us script it.) (4) Setup password-less SSH on both guest hypervisors, guestHyp1 and guestHyp2:: $ ssh-keygen -t rsa $ ssh-copy-id root@guestHyp$X $ ssh root@guestHyp$X (5) Stop ``firewalld`` on both guestHyp1 and guestHyp2:: (otherwise you get: ``error: internal error: unable to execute QEMU command 'drive-mirror': Failed to connect socket: No route to host``) :: $ systemctl stop firewalld (6) Migrate 'nGuest1' from guestHyp1 to guestHyp2, and vice-versa; *without* TLS):: $ virsh migrate --verbose --copy-storage-all \ --live nguest1 qemu+ssh://guestHyp2/system $ virsh migrate --verbose --copy-storage-all \ --live nGuest1 qemu+ssh://guestHyp1/system (7) Setup TLS. (7.a) On _both_ hosts:: $ mkdir -p /etc/pki/qemu $ mkdir -p /etc/pki/libvirt/private $ mkdir -p /etc/pki/CA (7.b) On guestHyp1, create the CA cert and SCP it to guestHyp2:: $ cd /home $ mkdir pki && cd pki $ cat ca.info cn = JAF Migration TLS Test ca cert_signing_key expiration_days = 700 $ certtool --generate-privkey > cakey.pem $ certtool --generate-self-signed --load-privkey cakey.pem \ --template ca.info --outfile cacert.pem $ cp cacert.pem /etc/pki/CA/ $ scp cacert.pem guestHyp2:/etc/pki/CA/ (7.b) On guestHyp1, generate server certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 cert to its host:: $ cat guestHyp1-server.info organization = TLS Migration Test cn = guestHyp1 tls_www_server encryption_key signing_key $ cat guestHyp2-server.info organization = TLS Migration Test cn = guestHyp2 tls_www_server encryption_key signing_key $ certtool --generate-privkey > guestHyp1-serverkey.pem Generating a 3072 bit RSA private key... $ certtool --generate-privkey > guestHyp2-serverkey.pem Generating a 3072 bit RSA private key... $ certtool --generate-certificate --template guestHyp1-server.info \ --load-privkey guestHyp1-serverkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp1-servercert.pem $ certtool --generate-certificate --template guestHyp2-server.info \ --load-privkey guestHyp2-serverkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp2-servercert.pem $ cp guestHyp1-servercert.pem /etc/pki/libvirt/servercert.pem $ cp guestHyp1-serverkey.pem /etc/pki/libvirt/private/serverkey.pem $ scp guestHyp2-servercert.pem guestHyp2:/etc/pki/libvirt/servercert.pem $ scp guestHyp2-serverkey.pem guestHyp2:/etc/pki/libvirt/private/serverkey.pem (7.c) On guestHyp1:: $ chown root:qemu /etc/pki/libvirt $ chown root:qemu /etc/pki/libvirt/* $ chown root:qemu /etc/pki/libvirt/private/* (7.d) On *both* hosts, guestHyp1 and guestHyp2, adjust permissoins for their corresponding server certificates:: $ chmod 440 /etc/pki/libvirt/servercert.pem $ chmod 750 /etc/pki/libvirt/private/ $ chmod 600 /etc/pki/libvirt/private/serverkey.pem (7.e) On guestHyp1, generate client certificates for both, guestHyp1 and guestHyp2 and SCP the guestHyp2 client cert to its host:: $ certtool --generate-privkey > guestHyp1-clientkey.pem $ certtool --generate-privkey > guestHyp2-clientkey.pem $ cat guestHyp1-client.info country = BE state = EF locality = Belgium organization = TLS Migration Test cn = guestHyp1 tls_www_client encryption_key signing_key $ cat guestHyp2-client.info country = BE state = EF locality = Belgium organization = TLS Migration Test cn = guestHyp1 tls_www_client encryption_key signing_key $ certtool --generate-certificate \ --template guestHyp1-client.info \ --load-privkey guestHyp1-clientkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp1-clientcert.pem $ certtool --generate-certificate \ --template guestHyp2-client.info \ --load-privkey guestHyp2-clientkey.pem \ --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem \ --outfile guestHyp2-clientcert.pem $ cp guestHyp1-clientcert.pem /etc/pki/libvirt/clientcert.pem $ cp guestHyp1-clientkey.pem /etc/pki/libvirt/private/clientkey.pem $ scp guestHyp2-clientcert.pem guestHyp2:/etc/pki/libvirt/clientcert.pem $ scp guestHyp2-clientkey.pem guestHyp2:/etc/pki/libvirt/private/clientkey.pem (7.f) On *both* hosts, guestHyp1 and guestHyp2:: $ chmod 400 /etc/pki/libvirt/clientcert.pem $ chmod 644 /etc/pki/libvirt/private/clientkey.pem (7.g) On *both*, because QEMU requires the `/etc/pki/qemu` directory contents to be slightly different:: $ cp /etc/pki/CA/cacert.pem /etc/pki/qemu/ca-cert.pem $ cp /etc/pki/libvirt/servercert.pem /etc/pki/qemu/server-cert.pem $ cp /etc/pki/libvirt/private/serverkey.pem /etc/pki/qemu/server-key.pem $ cp /etc/pki/libvirt/clientcert.pem /etc/pki/qemu/client-cert.pem $ cp /etc/pki/libvirt/private/clientkey.pem /etc/pki/qemu/client-key.pem (7.h) On *both*, guestHyp1, and guestHyp2, update 'x509' config options in /etc/libvirt/qemu.conf:: default_tls_x509_cert_dir = "/etc/pki/qemu" default_tls_x509_verify = 1 And modify /etc/sysconfig/libvirtd on both (guestHyp1 & guestHyp2):: LIBVIRTD_ARGS="--listen" And restart libvirt daemon (also on both hosts):: $ systemctl restart libvirtd (8) Run `virt-pki-validate` on guestHyp1 and guestHyp2:: $ virt-pki-validate Found /usr/bin/certtool Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp1 Found client private key /etc/pki/libvirt/private/clientkey.pem Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp1 Found server private key /etc/pki/libvirt/private/serverkey.pem Make sure /etc/sysconfig/libvirtd is setup to listen to TCP/IP connections and restart the libvirtd service $ virt-pki-validate Found /usr/bin/certtool Found CA certificate /etc/pki/CA/cacert.pem for TLS Migration Test Found client certificate /etc/pki/libvirt/clientcert.pem for guestHyp2 Found client private key /etc/pki/libvirt/private/clientkey.pem Found server certificate /etc/pki/libvirt/servercert.pem for guestHyp2 Found server private key /etc/pki/libvirt/private/serverkey.pem Make sure /etc/sysconfig/libvirtd is setup to listen to TCP/IP connections and restart the libvirtd service (9) **IMPORTANT**: *Ensure* the permissions of certificate files and keys in ``/etc/pki/qemu/*`` directory on both source _and_ destination to be the following:: [root@guestHyp1 qemu]$ ls -lasrtZ /etc/pki/qemu/ total 32 0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0 110 Dec 10 10:39 .. 4 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:08 ca-cert.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:08 server-cert.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:09 client-cert.pem 8 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:09 client-key.pem 4 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 2459 Dec 11 05:32 server-key-stripped.pem 8 -rw----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8177 Dec 11 05:35 server-key.pem 0 drwxr-xr-x. 2 root root unconfined_u:object_r:cert_t:s0 146 Dec 11 06:01 . [root@guestHyp2 ~]# ls -lasrtZ /etc/pki/qemu/ total 28 0 drwxr-xr-x. 10 root root system_u:object_r:cert_t:s0 110 Dec 10 10:39 .. 4 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1464 Dec 10 11:10 ca-cert.pem 4 -r--r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1558 Dec 10 11:10 server-cert.pem 8 -rw-------. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8170 Dec 10 11:10 server-key.pem 4 -r-----r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 1619 Dec 10 11:10 client-cert.pem 8 -rw-r--r--. 1 qemu qemu unconfined_u:object_r:cert_t:s0 8180 Dec 10 11:10 client-key.pem 0 drwxr-xr-x. 2 root root unconfined_u:object_r:cert_t:s0 115 Dec 10 11:10 . (10) Migrate 'nGuest1' from guestHyp1 to guestHyp2 *with* TLS:: $ virsh migrate --tls --verbose --copy-storage-all \ --migrate-disks vda nGuest1 --live qemu+ssh://guestHyp2/system Other notes ----------- There are in total _nine_ TLS-related config options in /etc/libvirt/qemu.conf: default_tls_x509_cert_dir default_tls_x509_verify nbd_tls nbd_tls_x509_cert_dir migrate_tls_x509_cert_dir vnc_tls_x509_cert_dir spice_tls_x509_cert_dir vxhs_tls_x509_cert_dir chardev_tls_x509_cert_dir See how they fits in with ``default_tls_x509_cert_dir`` and ``default_tls_x509_verify``. Answer (thanks: Dan Berrangé, for the IRC chat): If you set the both ``default_*`` parameters for all certificates, then no need to specify any of the other ``*_tls*`` config options. The intention of the upstream libvirt is that you can just use the `default_*` (that way you don't need to set any other ``*_tls*`` parameters, _unless_ you need different certificates for some services. The rationale for that is that some services (e.g. migration / NBD) are only exposed to internal infrastructure; while some sevices (VNC, Spice) might be exposed publically, so might need different certificates. For OpenStack this doesn't matter, though. So we're good