Testing NUMA related hardware setup with libvirt and DevStack
=============================================================

Reference:
https://review.openstack.org/#/c/131818/1/doc/source/devref/testing/libvirt-numa.rst


Ensure to enable services:

    $ sudo systemctl enable postgresql
    $ sudo systemctl enable rabbitmq-server
    $ sudo systemctl enable httpd
    $ sudo systemctl enable openvswitch


Test env
--------

libvirt git (Running on DevStack VM):

    $ git describe
    CVE-2014-7823-193-g6085d91

    $ git rev-parse --short HEAD
    6085d91

Nova:

    nova]$ git describe
    2014.2-995-g5d2ea10
    
    nova]$ git rev-parse --short HEAD
    5d2ea10

DevStack:

    $ git log | head -5
    commit 3b1647310d8b8b975e74d2601ba841e373b97b20
    Merge: ca7aa30 ea70cc9
    Author: Jenkins <jenkins@review.openstack.org>
    Date:   Wed Nov 26 15:42:29 2014 +0000



Testing basic non-NUMA usage
----------------------------

Boot a Nova guest and check the NUMA topology in the database:

$ nova boot --image cirros-0.3.1-x86_64-disk --flavor m1.tiny cirrvm1

$ sudo -u postgres psql
nova=#
nova=# SELECT numa_topology FROM compute_nodes;
                                            numa_topology                                             
------------------------------------------------------------------------------------------------------
 {"cells": [{"mem": {"total": 7794, "used": 0}, "cpu_usage": 0, "cpus": "0,1,2,3,4,5,6,7", "id": 0}]}
(1 row)

nova=# SELECT numa_topology FROM instance_extra;
 numa_topology 
---------------
 
(1 row)

nova=# 


Reconfiguring the DevStack instance to have NUMA topology
---------------------------------------------------------

Ensure the DevStack VM's config has:

$ virsh dumpxml fedvm1 | grep cpu
  <vcpu placement='static' cpuset='0,4,8,12,16,20,24,28,32,36,40,44'>8</vcpu>
  <cpu mode='host-passthrough'>
      <cell id='0' cpus='0-3' memory='4096000'/>
      <cell id='1' cpus='4-5' memory='2048000'/>
      <cell id='2' cpus='6-7' memory='2048000'/>
  </cpu>

And the below in nova.conf:

. . .
[DEFAULT]
scheduler_default_filters=RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter, NUMATopologyFilter

[libvirt]
virt_type = kvm
. . .


Re-join the stack.sh:

$ ./rejoin.sh

(Optionally, ensure all the services have started successfully by
validating all the `screen` windows for OpenStack services.)

$ sudo -u postgres psql nova
nova=# select numa_topology from compute_nodes;
                                                                                                                     numa_topology                    
                                                                                                  
------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------
 {"cells": [{"mem": {"total": 3857, "used": 0}, "cpu_usage": 0, "cpus": "0,1,2,3", "id": 0}, {"mem": {"total": 1969, "used": 0}, "cpu_usage": 0, "cpus
": "4,5", "id": 1}, {"mem": {"total": 1967, "used": 0}, "cpu_usage": 0, "cpus": "6,7", "id": 2}]}
(1 row)

nova=# 

The above indeed shows that there are now 3 NUMA nodes for the “host”
(DevStack, i.e.) machine, the first with 4 GB of RAM and 4 CPUs, and
others with 2 GB of RAM and 2 CPUs each.


Testing instance boot with no NUMA topology requested
-----------------------------------------------------

    $ nova flavor-create m1.numa 999 1024 1 4
    $ nova flavor-key m1.numa set hw:numa_nodes=1
    $ nova flavor-show m1.numa
    +----------------------------+------------------------+
    | Property                   | Value                  |
    +----------------------------+------------------------+
    | OS-FLV-DISABLED:disabled   | False                  |
    | OS-FLV-EXT-DATA:ephemeral  | 0                      |
    | disk                       | 1                      |
    | extra_specs                | {"hw:numa_nodes": "1"} |
    | id                         | 999                    |
    | name                       | m1.numa                |
    | os-flavor-access:is_public | True                   |
    | ram                        | 1024                   |
    | rxtx_factor                | 1.0                    |
    | swap                       |                        |
    | vcpus                      | 4                      |
    +----------------------------+------------------------+


Boot an instance using the new NUMA flavor just created:

  $ nova boot --image cirros-0.3.1-x86_64-disk --flavor m1.numa cirrvm2


Testing instance boot with 1 NUMA cell requested
------------------------------------------------

Works with libvirt RPMs built from git:

  $ git describe
  CVE-2014-7823-193-g6085d91

  $ git rev-parse --short HEAD
  6085d91

which has the commit mentioned in comment #7. 

Re-testing Nova with these RPMs (libvirt-1.2.11 -- yet to be released) version,
Nova instance with NUMA topology boots successfully:

1. Create a Nova flavor with NUMA topology: 

    $ nova flavor-create m1.numa 999 1024 1 4
    $ nova flavor-key m1.numa set hw:numa_nodes=1
    $ nova flavor-show m1.numa
    +----------------------------+------------------------+
    | Property                   | Value                  |
    +----------------------------+------------------------+
    | OS-FLV-DISABLED:disabled   | False                  |
    | OS-FLV-EXT-DATA:ephemeral  | 0                      |
    | disk                       | 1                      |
    | extra_specs                | {"hw:numa_nodes": "1"} |
    | id                         | 999                    |
    | name                       | m1.numa                |
    | os-flavor-access:is_public | True                   |
    | ram                        | 1024                   |
    | rxtx_factor                | 1.0                    |
    | swap                       |                        |
    | vcpus                      | 4                      |
    +----------------------------+------------------------+

2. Boot a Nova guest:

    $ nova boot --image cirros-0.3.1-x86_64-disk --flavor m1.numa cirrvm5

3. Find the Nova instance:

    $ nova list | grep cirrvm5
    | 5d4c50ff-301c-44cb-826f-ffa07266d85f | cirrvm5 | ACTIVE | -          | Running     | public=172.24.4.6 |


4. Find the libvirt ID for the Nova instance:

    $ grep -i 5d4c50ff-301c-44cb-826f-ffa07266d85f /etc/libvirt/qemu/*.xml | grep uuid
    /etc/libvirt/qemu/instance-00000004.xml:  <uuid>5d4c50ff-301c-44cb-826f-ffa07266d85f</uuid>
    [. . .]


5. Nova instance's libvirt XML:

---------------------------
$ sudo virsh dumpxml instance-00000004
<domain type='kvm' id='3'>
  <name>instance-00000004</name>
  <uuid>5d4c50ff-301c-44cb-826f-ffa07266d85f</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1"/>
      <nova:name>cirrvm5</nova:name>
      <nova:creationTime>2014-11-28 18:11:27</nova:creationTime>
      <nova:flavor name="m1.numa">
        <nova:memory>1024</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="f9a0644c9e9540828fbd8249dc9a92a2">admin</nova:user>
        <nova:project uuid="9157c0a4cf194d02bb4aa8023fb9db8b">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="e15c9d88-5de0-4f4b-8d15-708cd32f4ea9"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0-3'/>
    <vcpupin vcpu='1' cpuset='0-3'/>
    <vcpupin vcpu='2' cpuset='0-3'/>
    <vcpupin vcpu='3' cpuset='0-3'/>
    <emulatorpin cpuset='0-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2015.1</entry>
      <entry name='serial'>bf6b5391-2390-df4f-b3dc-aa80d05468bb</entry>
      <entry name='uuid'>5d4c50ff-301c-44cb-826f-ffa07266d85f</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='4' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/kashyapc/src/cloud/data/nova/instances/5d4c50ff-301c-44cb-826f-ffa07266d85f/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/home/kashyapc/src/cloud/data/nova/instances/_base/f187eddcdb76fcfd896c0916b9288e666014ce2b'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/kashyapc/src/cloud/data/nova/instances/5d4c50ff-301c-44cb-826f-ffa07266d85f/disk.config'/>
      <backingStore/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:fd:50:8f'/>
      <source bridge='qbr1ee94b1e-29'/>
      <target dev='tap1ee94b1e-29'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/home/kashyapc/src/cloud/data/nova/instances/5d4c50ff-301c-44cb-826f-ffa07266d85f/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/17'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/home/kashyapc/src/cloud/data/nova/instances/5d4c50ff-301c-44cb-826f-ffa07266d85f/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      <stats period='10'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c176,c615</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c176,c615</imagelabel>
  </seclabel>
</domain>
--------------------

6. Check what Nova recorded in the database (PostgreSQL in this case):

$  sudo -u postgres psql nova
nova=#
nova=# SELECT numa_topology FROM instance_extra;
. . .
 {"nova_object.version": "1.1", "nova_object.changes": ["instance_uuid"], "nova_object.name": "InstanceNUMATopology", "nova_object.data": {"instance_u
uid": "5d4c50ff-301c-44cb-826f-ffa07266d85f", "cells": [{"nova_object.version": "1.1", "nova_object.changes": ["cpuset", "id", "pagesize", "memory"], 
"nova_object.name": "InstanceNUMACell", "nova_object.data": {"cpuset": [0, 1, 2, 3], "id": 0, "pagesize": null, "memory": 1024}, "nova_object.namespac
e": "nova"}]}, "nova_object.namespace": "nova"}

nova=# select numa_topology from compute_nodes;
                                                                                                                       numa_topology

------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
 {"cells": [{"mem": {"total": 3857, "used": 1024}, "cpu_usage": 4, "cpus": "0,1,2,3", "id": 0}, {"mem": {"total": 1969, "used": 0}, "cpu_usage": 0, "c
pus": "4,5", "id": 1}, {"mem": {"total": 1967, "used": 0}, "cpu_usage": 0, "cpus": "6,7", "id": 2}]}
(1 row)

nova=#