Paul Polakos (Bel Labs, Alcatel-Lucent, USA) - Network Virtualisation
Network virtualisation using Crossbow Technology · Network virtualisation using Crossbow...
Transcript of Network virtualisation using Crossbow Technology · Network virtualisation using Crossbow...
Network virtualisationusing CrossbowTechnology
Uroš Nedić, MScOpenSolaris Contributor
2
● Crossbow Technical Background ● Network Machines● Network in a Box
3
Crossbow Features
• Virtualization> IP Instances> NIC Virtualization - VNICs > Service Virtualization - Flows
• Resource partitioning> QoS/Diffserv (without performance penalties)> SLA on a per connection basis> Class of Service Support
• Performance & Observability> Dynamic polling> H/W and S/W fanout to multiple cores> Real Time Usage and History
Better Defense against DDOS attacks
4
Virtual Stacks
Services and ProtocolsCompute Resources
NIC 1
CPU 1VIRTUALSQUEUE
CPU 2VIRTUALSQUEUE
CPU 'n'VIRTUALSQUEUE
CPU 1 Virtual Squeue CPU 2 Virtual Squeue
VOIPSQUEUE
HTTPSSQUEUE
DEFAULTSQUEUE
TCPSQUEUE
UDPSQUEUE
DEFAULTSQUEUE
VirtualNIC
VirtualNIC
VirtualNIC
VirtualNIC
VirtualNIC
VirtualNIC
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
Flow Classifier Flow ClassifierNIC 2
5
Virtual Stacks
Services and Protocols
CPU 1VIRTUALSQUEUE
CPU 2VIRTUALSQUEUE
CPU 'n'VIRTUALSQUEUE
CPU 1 Virtual Squeue
VOIPSQUEUE
HTTPSSQUEUE
DEFAULTSQUEUE
VirtualNIC
VirtualNIC
VirtualNIC
The VNICs are in the control path only. The data link layer is bypassed.
The Squeue switches the MSI interrupt perstack between interrupt and polling mode and controls the rate of packet arrival for the virtual stack.
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
MemoryPartition
Flow Classifier Flow ClassifierNIC 1 NIC 2
Compute Resources
6
Crossbow Architecture● Partition the NIC Hardware, kernel queues, and CPU
and allow creation of Virtual NICs● Use dynamic polling on Virtual NICs to schedule rate of
packet arrival per VNIC● Effect of dynamic polling Mpstat (older driver)intr ithr csw icsw migr smtx srw syscl usr sys wt idl10818 8607 4558 1547 161 1797 289 19112 17 69 0 12
Mpstat (Crossbow based driver)intr ithr csw icsw migr smtx srw syscl usr sys wt idl2823 1489 875 151 93 261 1 19825 15 57 0 27
● Use Dynamic polling for B/W partitioning and isolation without any support from switches and routers
7
Parallelized Stack: Made for Cores/Threads
Multi-Core Processor
CrossbowVirtualized Network
Streams & VNICs
1000’s of Logical Connections
Application Threads Or Solaris ContainersOr Virtual Machines
C
L
A
S
S
I
F
I
E
R
VNICRx/TxDMA
Kernel Qsand threads
VNICRx/TxDMA
Kernel Qsand threads
Squeue
Squeue
Neptune2 x 10 Gbe
and/or4 x 1Gbe
ApplicationNICNetwork Kernel CPU
DedicatedLane
VNICRx/TxDMA
Kernel Qsand threads
Squeue
8
Virtual Network Containers
Flow Classifier
Exclusive IPInstance
Rx/TxDMA
Rx/TxDMA
Rx/TxDMA
NIC
bge0 VNIC1 (100Mbps)
VNIC2 (200Mbps)
Exclusive IPInstance
VirtualSQUEUE
VirtualSQUEUE
Zonexb1-z1
Zonexb1-z2
Clientxb2
Clientxb3
SolarisGlobalZone
Virtualization• Exclusive IP Stack• Virtual NICs• Virtual Switches
Resource Control• Bandwidth Limits & Gauarantees• Priority• Dedicated CPUs
Observability• Watch real time usage for each
VNIC• Snmp and Kstat per VNIC• History at no cost
9
Virtual Machines
Solaris Guest OS 1 Solaris Guest OS 2Solaris Host OS
Host OSVIRTUAL SQUEUE
All Traffic
NIC Virtualization Engine NIC Virtualization Engine
NIC Virtualization Engine
Guest OS 1VIRTUAL SQUEUE
Guest OS 2VIRTUAL SQUEUE
All Traffic
Host OS VNICGuest OS 2 VNIC
NICH/W Flow Classifier
HTTPSQUEUE
HTTPSSQUEUE
DEFAULTSQUEUE
VirtualNIC
VirtualNIC
VirtualNIC
Host OSAll traffic
GuestOS 1HTTP
GuestOS 1
HTTPS
GuestOS 1
DEFAULT
GuestOS 2
All Traffic
10
Network Machines
DMZ ApacheSolaris
N2/NIU
Router/Firewall DNS Server
IP IP
TCP/UDP
IP
VirtualNIC A
VirtualNIC A
VirtualNIC B
VirtualNIC B
TCP/UDP
TCP/UDP
Rx/TxDMA
Rx/TxDMA
Rx/TxDMA
Rx/TxDMA
Rx/TxDMA
Rx/TxDMA
Flow Classifier Flow Classifier
NIC A NIC B
Internet Intranet
APIs forISVs at eachlayer
DedicatedCPUs
11
Anatomy of a Typical Router
Major cost paid on bringing the packet in and route lookup etc.
Packets are dropped on Xmit Queues
PORT 0
PORT 1
PORT 2
PORT n
RouteLookupEngine
Xmit Q 1
Xmit Q 2
Xmit Q 3
Xmit Q m
Xmit Port 0
Xmit Port 1
Xmit Port 2
Xmit Port m
12
Anatomy of Crossbow Router
OpenSolaris Network Stack
NetworkInterface
ComputeInterface
Policy-based packet drop on ingress—zero cost drop.• Explicit mapping of Compute Threads or Cores to Network Threads creating virtual
“express lanes”• Dedicated BW, Priority, and CPU/Threads assigned to each lane• High performance packet classification
DMA 0
DMA 1
DMA 2
DMA 16
VNIC
VNIC
VNIC
VNIC
HTTPS
Data
Voice
Video
T/C
T/C
T/C
T/C
Pack
et C
lass
ifier
13
Crossbow Network in a BoxSwitch 3
Router
Port310.0.0.3
Port210.0.0.2
Port110.0.0.1
Port620.0.0.3
Port920.0.0.1
Switch 1
Host 2Host 1Client
100Mbps
Physical Network
1 Gbps1 Gbps1 Gbps1 Gbps
14
Example VNIC Usage• Done using dladm(1M), as with other data-link
interface administration
# dladm create-vnic -d bge1 vnic1
# dladm create-vnic -d bge1 -m random -p maxbw=100M -p cpus=4,5,6 vnic2
# dladm create-etherstub vswitch1
# dladm show-etherstub
LINK
vswitch1
# dladm create-vnic -d vswitch1 -p maxbw=1000M vnic3
# dladm show-vnic
LINK OVER MACTYPE MACVALUE BANDWIDTH CPUS
vnic1 bge1 factory 0:1:2:3:4:5 - -
vnic2 bge1 random 2:5:6:7:8:9 max=100M 4,5,6
vnic3 vswitch1 random 4:3:4:7:0:1 max=1000M -
15
Join Us. ...
• Our communities and projects are open on OpenSolaris.org:> CrossBow: http://opensolaris.org/os/project/crossbow> VNM: http://opensolaris.org/os/project/vnm> Networking:
http://opensolaris.org/os/community/networking
• Where you will find:> Lively discussions, design docs, FAQs, source code
drops, binary releases, etc...