Information to collect when debugging Neutron

NOTE:: The below commands are for a 2-node setup (but should apply to other scaled setups too) both running in VMs:

- Controller node:: Nova, Keystone, Cinder, Glance, Neutron (using Open
  vSwitch plugin and GRE tunneling).

- Compute node:: Nova (nova-compute), Neutron (openvswitch-agent)

Neutron networks info

On Controller node:

$ neutron net-list
$ neutron subnet-list
$ neutron net
$ neutron port-list
$ neutron router-list

IP and route info

On Controller node & Compute nodes:

$ ip addr
$ route -n
$ iptables -L
$ cat /etc/sysconfig/iptables
$ iptables -nL

Network Namespaces

On controller node (assuming all Neutron services are running), commands look like:

$ ip netns
$ in netns qrouter-UUID ip a
$ ip netns qrouter-UUID exec ip link
$ ip netns qrouter-UUID exec route -n
$ ip netns qrouter-UUID exec iptables -L -t nat
$ ip netns qdhcp-UUID ip a
$ ip netns qdhcp-UUID ip link
$ ip netns qdhcp-UUID exec route -n

Obvious note, Substitute the 'UUID' value accordingly. I omitted it here for brevity.

Open vSwitch info - OVS db and datapath contents

On both Controller & Compute nodes:

$ ovs-vsctl show
$ ovs-dpctl show
$ ovs-dpctl dump-flows
$ ovs-ofctl dump-flows br-tun
$ ovs-ofctl dump-flows br-tun table=21
$ ovs-ofctl dump-flows br-int

tcpdump diagnostics

On various network devices in play:: OVS bridges, linux bridges, tap devices and veth pairs on compute host, qr, qg interfaces inside network namespaces, physical interfaces, etc.

Some sample commands on various interfaces (not all corner cases are included):

$ tcpdump -envi eth0 | grep -i gre

# Run tcpdump on physical link used by GRE tunnels (on Controller
# node). This may isolate the problem to the compute node or the
# network node.
$ tcpdump -i eth0 -n ip proto gre

$ tcpdump -envi br-int
$ tcpdump -envi br-tun
$ ip netns exec qrouter-UUID tcpdump -nn -i qr-63ea2815-b5 icmp
$ ip netns exec qrouter-UUID tcpdump -nn -i qg-e7110dba-a9 icmp
$ tcpdump -envi 192.168.122.163
$ tcpdump -envi br-ex
$ tcpdump -i eth0 -n arp or icmp
$ tcpdump -i eth0 -ne ip proto 47
$ tcpdump -i br-ex -n icmp
$ tcpdump -i eth0 -n icmp
$ tcpdump -i any  -n icmp
$ tcpdump -i tape7110dba-a9 -n icmp
$ tcpdump -envi qvbb71536f2-dd -n arp or icmp
$ tcpdump -i eth0 -n not port 22
$ tcpdump -i eth0 -n not port 22 and not port amqp

On physical host:

# Check traffic from Compute host (which is running on
# 192.169.142.49)
$ tcpdump -nn -i virbr1 host 192.169.142.49