Information to collect when debugging Neutron 
=============================================

NOTE:: The below commands are for a 2-node setup (but should apply to
other scaled setups too) both running in VMs::

  - Controller node:: Nova, Keystone, Cinder, Glance, Neutron (using Open
    vSwitch plugin and GRE tunneling).

  - Compute node:: Nova (nova-compute), Neutron (openvswitch-agent)


Neutron networks info
---------------------
 
On Controller node::

    $ neutron net-list
    $ neutron subnet-list
    $ neutron net
    $ neutron port-list
    $ neutron router-list


IP and route info 
-----------------

On Controller node & Compute nodes::

    $ ip addr
    $ route -n
    $ iptables -L
    $ cat /etc/sysconfig/iptables
    $ iptables -nL


Network Namespaces
------------------

On controller node (assuming all Neutron services are running),
commands look like::


    $ ip netns
    $ in netns qrouter-UUID ip a
    $ ip netns qrouter-UUID exec ip link
    $ ip netns qrouter-UUID exec route -n
    $ ip netns qrouter-UUID exec iptables -L -t nat
    $ ip netns qdhcp-UUID ip a
    $ ip netns qdhcp-UUID ip link
    $ ip netns qdhcp-UUID exec route -n

Obvious note, Substitute the 'UUID' value accordingly. I omitted it
here for brevity.

Open vSwitch info - OVS db and datapath contents
------------------------------------------------

On both Controller & Compute nodes::

    $ ovs-vsctl show
    $ ovs-dpctl show
    $ ovs-dpctl dump-flows
    $ ovs-ofctl dump-flows br-tun
    $ ovs-ofctl dump-flows br-tun table=21
    $ ovs-ofctl dump-flows br-int



tcpdump diagnostics
-------------------

On various network devices in play:: OVS bridges, linux bridges, tap
devices and veth pairs on compute host, qr, qg interfaces *inside*
network namespaces, physical interfaces, etc.

Some sample commands on various interfaces (not all corner cases are
included)::

    $ tcpdump -envi eth0 | grep -i gre

    # Run tcpdump on physical link used by GRE tunnels (on Controller
    # node). This may isolate the problem to the compute node or the
    # network node.
    $ tcpdump -i eth0 -n ip proto gre

    $ tcpdump -envi br-int
    $ tcpdump -envi br-tun
    $ ip netns exec qrouter-UUID tcpdump -nn -i qr-63ea2815-b5 icmp
    $ ip netns exec qrouter-UUID tcpdump -nn -i qg-e7110dba-a9 icmp
    $ tcpdump -envi 192.168.122.163
    $ tcpdump -envi br-ex
    $ tcpdump -i eth0 -n arp or icmp
    $ tcpdump -i eth0 -ne ip proto 47
    $ tcpdump -i br-ex -n icmp
    $ tcpdump -i eth0 -n icmp
    $ tcpdump -i any  -n icmp
    $ tcpdump -i tape7110dba-a9 -n icmp
    $ tcpdump -envi qvbb71536f2-dd -n arp or icmp
    $ tcpdump -i eth0 -n not port 22
    $ tcpdump -i eth0 -n not port 22 and not port amqp

On physical host::

   # Check traffic from Compute host (which is running on
   # 192.169.142.49)
   $ tcpdump -nn -i virbr1 host 192.169.142.49