Weird stuff we've seen with OpenStack Neutron(And what to do about it)

OpenStack Neutron• So$ware-defined networking component

• Users define their own virtual networks

• Manages IP address assignment

• Floa?ng IP addresses

• Supports many different back-ends - OpenvSwitch, VMware

NSX, Cisco UCS, Midokura....

Neutron usage1

1 Source: OpenStack User Survey, October 2015

Simplified logical architecture

Architecture, con-nued

• neutron-{server,agent}

• OpenvSwitch

• Linux bridging

• Linux network namespaces

• L2

• L3

Namespaces• L2 namespace

• DHCP

• L3 namespace

• Rou4ng

• NAT

• Metadata

Common problems - typical user complaints• VM can't obtain an IP address

• Can't ping / connect to my VM

• Intermi9ent connec:vity

Weirdness #1 - orphaned namespaces• Default (on Ubuntu) is not to delete namespaces at all (!)

• Bug in iproute2 package

• h=ps://bugs.launchpad.net/neutron/+bug/1052535

• Misconfigured sudo rules meant that network namespaces weren't being deleted

• Mismatch between interfaces configured in a namespace and what Neutron expects

Finding out what's supposed to be wherefor netnode in osnet{0..4} ; do echo $netnode for router in $(ssh $netnode 'ip netns list | grep qrouter | cut -d - -f 2-20') ; do neutron router-show $router | grep -i unable donedone

Then delete each invalid namespace and associated OVS port.

• Pro%p: Don't run neutron-ovs-cleanup!

Weirdness #2 - duplicate segmenta4on ID• Customer support ,cket with instances unable to obtain an IP via

DHCP

• Some serious digging required...

Tracing packet flows• tcpdump on compute node and in network namespaces

• Packets not always arriving where you'd expect

• Have to look at OpenFlow rules

DHCP agentneutron dhcp-agent-list-hosting-net 4dc325ed-f141-41d9-8d0a-4f513defacad+--------------------------------------+--------+----------------+-------+| id | host | admin_state_up | alive |+--------------------------------------+--------+----------------+-------+| 1beb99ef-e6f6-4083-8fb6-661f2f61c565 | osnet1 | True | :-) |+--------------------------------------+--------+----------------+-------+

neutron net-show -F provider:segmentation_id 4dc325ed-f141-41d9-8d0a-4f513defacad+--------------------------+-------+| Field | Value |+--------------------------+-------+| provider:segmentation_id | 11 |+--------------------------+-------+

• 11 in hex = 0xb

root@osnet1:~# ovs-ofctl dump-flows br-tun table=2

NXST_FLOW reply (xid=0x4):cookie=0x0, duration=875584.823s, table=2, n_packets=85, n_bytes=10880, idle_age=11560, hard_age=65534, priority=1,tun_id=0x14 actions=mod_vlan_vid:43,resubmit(,10)cookie=0x0, duration=2578615.436s, table=2, n_packets=1345, n_bytes=128202, idle_age=27174, hard_age=65534, priority=1,tun_id=0x10 actions=mod_vlan_vid:2,resubmit(,10)cookie=0x0, duration=2578611.677s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,tun_id=0xd actions=mod_vlan_vid:12,resubmit(,10)cookie=0x0, duration=1806356.959s, table=2, n_packets=5140, n_bytes=364533, idle_age=341, hard_age=65534, priority=1,tun_id=0x21 actions=mod_vlan_vid:35,resubmit(,10)cookie=0x0, duration=2578610.661s, table=2, n_packets=1035919, n_bytes=180430025, idle_age=65534, hard_age=65534, priority=1,tun_id=0x11 actions=mod_vlan_vid:16,resubmit(,10)cookie=0x0, duration=1465355.359s, table=2, n_packets=418252, n_bytes=81112777, idle_age=52, hard_age=65534, priority=1,tun_id=0x13 actions=mod_vlan_vid:42,resubmit(,10)cookie=0x0, duration=1631281.273s, table=2, n_packets=445, n_bytes=52848, idle_age=65534, hard_age=65534, priority=1,tun_id=0x17 actions=mod_vlan_vid:37,resubmit(,10)cookie=0x0, duration=2578609.671s, table=2, n_packets=1821, n_bytes=167272, idle_age=16439, hard_age=65534, priority=1,tun_id=0xc actions=mod_vlan_vid:17,resubmit(,10)cookie=0x0, duration=2574619.932s, table=2, n_packets=490592856, n_bytes=279835052124, idle_age=65534, hard_age=65534, priority=1,tun_id=0x19 actions=mod_vlan_vid:19,resubmit(,10)cookie=0x0, duration=2578613.06s, table=2, n_packets=18, n_bytes=756, idle_age=65534, hard_age=65534, priority=1,tun_id=0xe actions=mod_vlan_vid:8,resubmit(,10)cookie=0x0, duration=1469974.534s, table=2, n_packets=6992536, n_bytes=1567235429, idle_age=9, hard_age=65534, priority=1,tun_id=0x7 actions=mod_vlan_vid:41,resubmit(,10)cookie=0x0, duration=2144082.193s, table=2, n_packets=2583, n_bytes=461773, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1d actions=mod_vlan_vid:32,resubmit(,10)cookie=0x0, duration=2578611.169s, table=2, n_packets=4230304, n_bytes=917966422, idle_age=0, hard_age=65534, priority=1,tun_id=0x5 actions=mod_vlan_vid:14,resubmit(,10)cookie=0x0, duration=85135.825s, table=2, n_packets=1739, n_bytes=130092, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1f actions=mod_vlan_vid:53,resubmit(,10)cookie=0x0, duration=979.195s, table=2, n_packets=123, n_bytes=11895, idle_age=933, priority=1,tun_id=0x22 actions=mod_vlan_vid:54,resubmit(,10)cookie=0x0, duration=1898543.732s, table=2, n_packets=240, n_bytes=30712, idle_age=65534, hard_age=65534, priority=1,tun_id=0x16 actions=mod_vlan_vid:34,resubmit(,10)cookie=0x0, duration=2578614.004s, table=2, n_packets=5595775, n_bytes=5465543420, idle_age=4, hard_age=65534, priority=1,tun_id=0x8 actions=mod_vlan_vid:6,resubmit(,10)cookie=0x0, duration=1473941.345s, table=2, n_packets=4202494, n_bytes=2516931444, idle_age=9, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:40,resubmit(,10)cookie=0x0, duration=2578619.787s, table=2, n_packets=103506, n_bytes=13925984, idle_age=0, hard_age=65534, priority=0 actions=drop

wat.

OpenFlow flows2

2 h$p://assafmuller.com/2013/10/14/gre-tunnels-in-openstack-neutron/

Missing OpenFlow ruleroot@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xbroot@osnet1:~# echo $?1

Try to re-add that network to the responsible agent:

$ neutron dhcp-agent-network-remove 1beb99ef-e6f6-4083-8fb6-661f2f61c565 \ 4dc325ed-f141-41d9-8d0a-4f513defacadRemoved network 4dc325ed-f141-41d9-8d0a-4f513defacad from DHCP agent$ neutron dhcp-agent-network-add 1beb99ef-e6f6-4083-8fb6-661f2f61c565 \ 4dc325ed-f141-41d9-8d0a-4f513defacadAdded network 4dc325ed-f141-41d9-8d0a-4f513defacad to DHCP agentroot@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb cookie=0x0, duration=0.945s, table=2, n_packets=14, n_bytes=588, idle_age=0, priority=1,tun_id=0xb actions=mod_vlan_vid:55,resubmit(,10)

Weirdness #3 - duplicate routers• Intermi)ent connec-vity issues groan

• No DVR or L3-HA enabled

• Routers scheduled and created twice on two network nodes

• Same network configura-on in each namespace

Duplicate routers› neutron l3-agent-list-hosting-router fe79ae7e-debf-44b9-8fd7-601abd5fb928+--------------------------------------+--------+----------------+-------+----------+| id | host | admin_state_up | alive | ha_state |+--------------------------------------+--------+----------------+-------+----------+| 48132c36-b6b1-40fa-b9d9-5474f4f27c3a | osnet0 | True | :-) | || c821a370-b301-40c5-8b7b-25d147ffc904 | osnet1 | True | :-) | |+--------------------------------------+--------+----------------+-------+----------+

› neutron router-show fe79ae7e-debf-44b9-8fd7-601abd5fb928+-----------------------+----------------------------------+| Field | Value |+-----------------------+----------------------------------+| admin_state_up | True || distributed | False || ha | False || status | ACTIVE || tenant_id | 7d718c99276c43d1992d64d061d98f15 |+-----------------------+----------------------------------+

How to approach troubleshoo0ngTroubleshoo*ng checklist

• UUIDs for instance, loca2on, MAC address

• UUIDs for network, subnet, router

• Network node hos2ng L2 and L3 agents

Useful commands - neutron

$ neutron agent-list$ neutron l3-agent-list-hosting-router $router_uuid$ neutron dhcp-agent-list-hosting-net $net_uuid$ neutron router-list-on-l3-agent $agent_uuid$ neutron net-list-on-dhcp-agent $net_uuid$ neutron help

Useful commands - OpenvSwitch

$ ovs-vsctl show$ ovs-ofctl dump-flows $bridge$ ovs-dpctl show

(More) useful commands

Standard network troubleshoo1ng toolkit:

$ tcpdump -enl -i eth1 | grep -i dhcp$ ip netns exec $netns tcpdump port 67 or port 68 -lne$ ip route$ ip address$ iptables-save$ brctl$ mtr

Etc.

Thanks!

Nick JonesDataCentred

h"p://www.datacentred.co.ukh"p://dischord.org

@yankcrime