Edge Virtual Bridging: Introduction and Implementation in ... › images › stories › pdf ›...
Transcript of Edge Virtual Bridging: Introduction and Implementation in ... › images › stories › pdf ›...
© 2009 IBM Corporation
Edge Virtual Bridging: Introduction and Implementation in Linux (Open-LLDP)
Thomas Richter– IBM Research and Development, Linux Technology Center 7-Nov-2012
2
Agenda
■ Virtualization & Bridges/Switches
■ Network Administration Issues
■ IEEE Standard 802.1 T M T M
Qbg
■ Principle of Operation
■ Open-LLDP Design & Configuration
■ Current & Future Work, Related Work
■ References, Acknowledgments, Trademarks
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
3
Virtualization & Switches Today (KVM on Linux)
IBM Presentation Template Full Version
Traffic between VMs within Host
■ Stays inside the host
■ No external traffic analysis in regards to– Security policies (Firewalls, Virus Scans, etc)– Network profiling
■ “Host oriented” approach– Vswitch & SR-IOV switch configuration maintained per host
Switch Locations
Virtual Bridge
SR-IOV (VF/PF)
VM VM VM
vir-br0
Host A
NIC
Switch
Intranet/Internet
Host B
NIC
vir-br0
External Switch
Sys
tem
Adm
in D
om
ain
Net
wor
k A
dmin
Dom
ain
VM
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
4
Network Administration Issues
■ Large number of virtual and physical switches in data centers
■ Host virtual switch offers filtering, ACLs, bandwidth limitations, QoS
– Not integrated in external switch management
– Outside scope of network administrators
– Needs manual configuration/verification per host
■ Migration of VMs
– Network policies and port profiles have to move with VM
– Manually ensure target is correctly configured
IEEE 802.1 Qbg enables
– Configuration and management of bridge services for VMs
– Multiple VMs to share a switch port for relay
“Network oriented” approach: consolidate/automate virtual and physical switch administration
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
5
IEEE 802.1 Qbg (Edge Virtual Bridging)■ All switching done externally
■ VEPA:– Frame relay (forwarding)– Inbound replicate received multi cast packets
■ Switch– Outbound port same as inbound port
■ Simplify physical and virtual switch management
IBM Presentation Template Full Version
VM VM VM
VEB
Host A
NIC
Switch
Host B
NIC
VEPA
VM VM
VSI TypeDatabase
■ Pros:– All traffic forwarded to external switch, host internal VM traffic visible and accountable– Reduces network configuration required by host administrator– No modifications of Ethernet frames
■ Cons:– Additional networking traffic and latency– Requires switch support/configuration– Simultaneous support for VEB and VEPA on same switch port not supported
Hairpin Mode
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
6
VM Attachment and Macvtap Device Options
IBM Presentation Template Full Version
Host B
App
macvtap0 macvtap1
Macvtap
■ Combines tun/tap and macvlan devices
■ Modes:
(1)Bridged: destination MAC address lookup on all macvtap devices defined on NIC
(2)Vepa:Traffic forwarded to external switch
(3)Private: Same as vepa, but ingress traffic blocked
(4)Passthrough: Only 1 macvtap device allowed per NIC (“exclusive” use)
NIC
/dev/tapX I/f to User Space (tuntap)
Virtual I/f with new MAC address
App
3
1
24
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
7
Principle of Operation
(1)Switch announces 802.1Qbg support on configured ports• Access to VSI Type database
(2)Lldpad on Linux host receives switch announcement• Negotiates switch port into “reflective relay”
(3)VM definition contains network information• Used by one or more VMs• Identified by unique ID
(4)Host sends <ID, MAC, VLAN-ID, ...> to switch• Lldpad for LLDP communication with switch• Libvirtd for VM network attachment
(5)Switch receives <ID, MAC, VLAN-ID, ...> on port• Confirms or denies VM connection to network• Enforces ACL and QoS
(6)Libvirtd starts VM
Switch Edge defines port characteristics: ACL, QoS, etc
VM Edge defines connection settings: VLAN-ID, MAC, UUID
Switch
Host B
Eth0
VM VM
VSI TypeDatabase *
1
Eth0.4
macvtap0 macvtap1
4 5
2
3
Lldpad
Libvirtd
A
B
A
B
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
VM Definition (ID, MAC, VLAN-ID)
6
* ID, ACL, QoS
8
IEEE 802.1 Qbg Protocols
IBM Presentation Template Full Version
■ Open-LLDP has been enhanced to support IEEE 802.1 Qbg– Draft 0.2 support available, ratified standard support under work– git://open-lldp.org/lldp/open-lldp
■ EVB Edge Virtual Bridge Protocol– Data Unit carried in LLDP messages– Exchanges information about “reflective relay” mode with switch port
■ CDCP Channel Discovery and Control Protocol– Data Unit carried in LLDP messages– Negotiates service-channels between host and switch port
■ ECP Edge Control Protocol– Simple data carrier protocol with retry and confirmation
■ VDP Virtual station interface Discovery and configuration Protocol– Payload of ECP– Negotiation of VM network data with switch port
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
9
EVB
■ Part of LLDP Messages
– LLDP sends network interface characteristics to neighbors
– 3 Agents for different bridge types (via multi cast MACs 00:80:C2:00:00:0x)
• Nearest bridge, nearest customer bridge, nearest notpmr bridge
■ EVB DU
– Send to nearest customer bridge only
– Exchange information about role, state, max number of retries and wait time
– Host requests reflective relay
– Switch accepts/denies request
SwitchTPMRSVLANSVLANHost.
Nearest Bridge
Nearest no TPMR BridgeNearest Customer Bridge
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
10
ECP
■ LLDP and DCB/DCBX are unacknowledged protocols
■ ECP protocol– Provides acknowledgment, signaling, re-transmit, sequence numbering– TLV formatted payload– Transmit arbitrary payload– Send to neareast customer bridge MAC (Ether type 0x8890)
TLVTLV
ECP-Buffer
Station/Host Bridge
TLVTLV
ECP-BufferTLV
TLV
TLVTLV
ECP-Buffer
Time1 ULP request sendTLV
TLV
TLV
3 set timer4 push to ULP
5 ACK
5a xmit timer?7 timer expired: re-transmit
TLV
ECP-Buffer
8 timer active: drop contents, incr seqno & continue
2 transmit data
Octets
ValueXx bits
Length9 bits
Type7 bits
1 2 3 N
Sample TLV
6 xmit timer?
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
11
VDP
■ Protocol to create, renew and destroy VSI associations
– VSI data consists
• VSI Type ID, VSI Manager ID, VM/IF UUID, <MAC,VLAN>
– Host usually initiator, switch responds with ack/nack
• Pre-associate (RR): send VSI data for switch to check
• Associate: Send VSI data to established association
• Dis-associate: Send VSI data to terminate association
– Associations renewed in regular intervals (keep alive)
– Associations can be terminated by switch
• Reboot, port disabled, etc Keep-Alive
Dis-associate
Association
Ack
Ack
Ack
BridgeStation
Time
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
12
CDCP
■ Share network link for simultaneous VEB/VEPA modes
– Divide link to logical channels using VLAN Tag (Service-VLAN)
• Assign logical channel to VM/VEBVEPA
– Sender inserts 802.1Q VLAN header, receiver removes it
– Requires Q-in-Q support for NIC and switch
■ Not yet implemented
VM
VM
VMVEB
VEPA
VMVM
VMVM
Mul
ti C
hann
el N
IC
Mu
lti C
hann
el S
w
Host
Server Edge
Switch Edge
8100 VLAN #
VLAN Tag
Add Remove
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
13
Example Configuration (with Switch Support for Qbg)LLDPAD Configuration File
eth2 : { tlvid00000001 : { info = "04001B217B3D24"; }; tlvid00000002 : { info = "03001B217B3D24"; }; Tlvid001b3f00 : /* EVB OUI */ { enableTx = true; fmode = "reflectiverelay"; capabilities = "vdp,ecp,rte"; }; adminStatus = 3; vdp : { enableTx = true; }};
VSI Type Database File
<vsi-type> <id> 123 </id> <version> 1 </version> <managerid> 1 </managerid> <vlanid> 4 </vlanid> <name>Thomas4</name> <bandwidth> <txrate> <txcommitedrate>512</txcommitedrate> <txburst>64</txburst> </txrate> <rxrate> <rxcommitedrate>1024</rxcommitedrate> <rxburst>128</rxburst </rxrate> </bandwidth> </vsi-type>
VM Configuration File (Networking Section)
<interface type='direct'> <mac address='08:18:21:63:be:e8'/> <source dev='eth2.4' mode='vepa'/> <virtualport type='802.1Qbg'> <parameters typeid='123' versionid='1' managerid='1' instanceid='a1412857-60f7-4ce1-e95a-2164943f53db'/> </virtualport> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface>
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
14
Related and Future Work
Future:
■ Support for ratified standard (still at draft 0.2)
■ Support for Bonding (active-backup) – Other modes to be done
■ Support for SR-IOV NICs– Kernel and iproute2 support under work (Intel)
■ Support for SNMP
Related:
■ Lldpd (from Vincent Bernat: http://vincent.bernat.im): IEEE 802.1 Qbg not in plan
■ Ladvd (from Sten Spans: http://code.google.com/p/ladvd): IEEE 802.1 Qbg not in plan
■ OpenLLDP (from http://openlldp.sourceforge.net): ?
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
15
Questions?Questions?
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
16
References
(1)Blade Network Technology, Broadcom, Brocade, Citrix, Emulex, Exterme Networks, HP, IBM, Intel, Juniper Networks, Qlogic: “Standardizing Data Center Server-Network Edge Virtualizing”,http://www.extremenetworks.com/libraries/whitepapers/VEPA-EVB_whitepaper.pdf, Oct 2010
(2)IEEE Organization: “http://www.ieee802.org/1/pages/802.1bg.html
(3)Stuart Miniman: “Edge Virtual Bridging”, http://wikibon.org/wiki/v/Edge_Virtual_Bridging, 27 Feb 2012
(4)Vivek Kashyap: “Network Security in the Cloud and Datacenter”, Linux Foundation Collaboration Summit, 7 Apr 2011, San Francisco, Ca, USA
(5)Vivek Kashyap, Arnd Bergman, Stefan Berger, Gerhard Stenzel, Jens Osterkamp: “Automating Virtual Machine Network Profiles”, Linux Symposium, Ottawa, Canada, 13-16 Jul 2010, pp 147-152
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
17
Acknowledgments
■ John Fastabend, Intel, Maintainer of open-lldp
■ Kishore Karolil, Florin Stelian, IBM Systems Networking for switch support
■ Vivek Kashyap, Gerhard Stenzel, Dirk Herrendörfer, Mijo Safradin, Sridhar Sumadrala, IBM
Linux Technology Center, Data Center Networking
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
18
Trademarks
■ This work represents the view of the author and does not necessarily represent the view of IBM.
■ IBM is a registered trademark of International Business Machines Corporation in the United States and/or other countries.
■ UNIX is a registered trademark of The Open Group in the United States and other countries .
■ Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
■ Other company, product, and service names may be trademarks or service marks of others.
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
19
Glossary
CDCP: Channel Discovery and Control Protocol
CEE: Converged Enhanced Ethernet
CNA: Converged Network Adapter
DCB: Data Center Bridging
DCBX: Data Center Bridging Extensions
ECP: Edge Control Protocol
EVB: Edge Virtual Bridge
LLDP: Link Layer Discovery Protocol (IEEE802.1AB)
SR-IOV: Single Resource Input/Output Virtualization
VDP: Virtual station interface Discovery and control Protocol
VEB: Virtual Ethernet Bridge
VEPA: Virtual Ethernet Port Aggregation
VM: Virtual Machine
VSI: Virtual Station Interface
Vswitch: Virtual Switch
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
20
BACKUPBACKUP
IBM Presentation Template Full Version
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
21
Example: VM Creation
IBM Presentation Template Full Version
VSIDatabaseNetwork Admin
creates VSI Types #
System Admin creates VM network withMAC, VLAN-ID, VSI
Switch12 Switch loads database
3
LLDPAD
LIBVIRTDLIBVIRTDLIBVIRTD
VIRT-MANAGER
VM
APP
EthX
EthX.4/Macvtap
4User starts VMSystem Admin
creates VM network withMAC, VLAN-ID, VSI *
5
VM Definitions
LLDPAD negotiates VSIData with Switch Port
86Associate VM VSIData with Switch Port
VM Definitions
Host
7
Libvirt creates & starts VM
VM communicates
* VSI: VSI Type ID,VSI Type Version ID, VSI Manager ID, VSI-IF UUID
# VSI Types: VSI Type ID,VSI Type Version ID, ACL, QoS, etcCourtesy of V. Kashyap [4] page 7
Hairpin Mode
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
22
Example: VM Migration
IBM Presentation Template Full Version
Switch
VIRT-MANAGER
LLDPAD
LIBVIRTDLIBVIRTDLIBVIRTD VM
APP
EthX
EthX.Y/Macvtap
Source Host
8
System Admin Migrates VM
LLDPAD
LIBVIRTDLIBVIRTDLIBVIRTD VM
APP
EthX
Target Host
1
2
SwitchVSI
Database
3
4
56 7VSI Dis-associate
Resume VM
Stop and move VM
Retrieve VSI Data
VSI Pre-associate
VSI Associate
EthX.Y/Macvtap
Courtesy of V. Kashyap [4] page 9
Hairpin ModeHairpin Mode
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
23
Bonding Support
■ Support for Bonding
– Mode: Active-backup
■ “Edge Relay” on bond interface
■ Switches are interconnected for VSI data exchange
Host
eth3
eth2
bond0Bond0.4
Switch2
Switch1VM
libvirtd EVBECP/VDP
lldpad
IntranetInternet
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
24
BACKUP SR-IOV NIC
■ SR-IOV NICS using 1 physical function and several virtual functions
■ Allows one PCIex device to appear as multiple independent PCIex devices– PF can be configured and managed– VF can just move data
■ VF are independent PCIex devices with limited functionality– Appear as individual network interfaces– Requires kernel support– Can be assigned to VMs
■ Integrated internal switch
■ References– http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-
technology-paper.html
IBM Presentation Template Full Version
Eth0
Eth1 EthnEth3Eth2 ....
Physical Function (PF)
Virtual Functions (VF)
NIC
Cable
Integrated switch
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012
25
VM Network Attachment to Host NIC
IBM Presentation Template Full Version
Macvtap
■ TAP Interface to Ethernet NIC
■ Has MAC address of VM
■ Forwards frames from VM to NIC
■ Bypass of virtual switch in host
Host B
VM VM
Eth0
macvtap0 macvtap1
EthYEthX
SR-IOV NIC
VFInternal SwitchIn VEPA PF
Macvtap
■ SR-IOV with VF and VEPA mode
Host B
Eth0
VM VM
Eth0.4
macvtap0 macvtap1
NIC
EthY.4EthX.4
Edge Virtual Bridging: Introduction and Implementation in Linux, Thomas Richter ([email protected]), LinuxCon Barcelona 2012