Hosting Virtual Networks on Commodity Hardware

VINI Summer Camp

Decouple Service from Infrastructure

• Service: “slices” of physical infrastructure– Applications and networks that benefit from

• Flexible, custom topologies• Application-specific routing

• Infrastructure: needed to build networks

Fixed Physical Infrastructure

Shared By Many Parties

Network Virtualization: 3 Aspects

• Host: Divide the resources of one physical node into the appearance of multiple distinct hosts

• Network stack: Give each process its own interfaces, routing table, etc.

• Links: Connect two nodes by composing underlying links

Why Virtual Networks

• Sharing amortizes costs– Enterprise network or small ISP does not

have to buy separate routers, switches, etc.– Large ISP can easily expand to new data

center without buying separate equipment

• Programmability and customizability

• Testing in realistic environments

Why Commodity Hardware

• Lower barrier to entry– Servers are inexpensive– Routing (e.g., Quagga), and forwarding (e.g., Click)

software is open source (free)

• No need for specialized hardware– Open-source routing software: Quagga, etc.– Network processors can be hard to program

• Easy adaptation of physical infrastructure– Expansion is easy: buy more servers

Commercial Motivation:Logical Routers

• Consolidation– PoP and Core– Simpler physical topology– Fewer physical interconnection

• Application-Specific Routing– PoP and Core– Simpler physical topology– Fewer physical interconnection

• Wholesale Router Market• Proof-of-Concept Deployment

Other Beneficiaries

• Interactive applications: require application-specific routing protocols– Gaming– VoIP

• Critical services: benefit from custom data plane– Applications that need more debugging info– Applications with stronger security requirements

Requirements

• Speed: Packet forwarding rate that approach that of native, in-kernel

• Flexibility: Support for custom routing protocols and topology

• Isolation: Separation of resource utilization and namespaces

Host Virtualization• Full virtualization: VMWare Server, KVM

– Advantage: No changes to Guest OS, good isolation– Disadvantage: Slow– Paravirtualization: Xen, Viridian

• OS-Level Virtualization: OpenVZ, VServers, Jail– Advantage: Fast– Disadvantage: Requires special kernel, less isolation

Network Stack Virtualization

• Allows each container to have its own – Interfaces– View of IP address space– Routing and ARP tables

• VServer does not provide this function– Solution 1: Patch VServer with NetNS– Solution 2: OpenVZ

• VServer is already used for PlanetLab

Link Virtualization

• Containers need Ethernet connectivity– Routers expect direct Ethernet connections to

neighbors

• Linux GRE tunnels support only IP-in-IP

• Solution: Ethernet GRE (EGRE) tunnel

Synthesis

• Tunnel interface outside of container– Permits traffic shaping outside of container– Easier to create point-to-multipoint topology

• Need to connect tunnel interface to virtual interface

Connecting Interfaces: Bridge

• Linux bridge module: connects virtual interface with the tunnel interface– speed suffers due to bridge table lookup– allows point-to-multipoint topologies

Optimization: ShortBridge

• Kernel module used to join virtual interface inside the container with the tunnel interface

• Achieves high packet forwarding rate

Evaluation

• Forwarding performance– Packets-Per-Second– Source->Node-Under-Test->Sink

• Isolation– Jitter/loss measurements with bursty cross traffic

• Scalability– Forwarding performance as the number of

containers grow• All tests were conducted on Emulab

– 3GHz CPU, 1MB L2 Cache, 800MHz FSB, 2GB 400MHz DDR2 RAM

Forwarding Performance - Click

• Minimal Click configuration– Raw UDP receive->send

• Higher jitter• ~80’000PPS

Forwarding Performance - Bridged

• Allows more flexibility through bridging

• ~250’000PPS

Forwarding Performance – Bridged w/o Tunneling

• Xen: often crashes, ~70’000PPS• OpenVZ: ~300’000PPS• NetNS: ~300’000PPS

Forwarding Performance – Spliced

• Avoids bridging overhead• Point-to-Point topologies only• ~500’000PPS

Forwarding Performance - Direct

• No resource control

• ~580’000PPS

Overall Forwarding Performance

Forwarding for Different Packet Sizes

Isolation

• Setup:– 5 nodes. 2 pairs of source+sink – 2 NetNS containers in spliced mode– pktgen used to generate cross flow– iperf measures jitter on another flow

• Step function– CPU utilization < 99%: no loss, 0.5ms jitter– CPU utilization ~> 100%: loss, 0.5ms jitter for

delivered packets

Scalability Test Setup

Scalability Results

Tradeoffs

• Bridge vs. Shortbridge– Bridge enables point-to-multipoint– Shortbridge is faster

• Data-plane flexibility vs. Performance– Non-IP forwarding requires user-space

processing (Click)

Future Work

• Resource allocation and scheduling– CPU– Interrupts/packet processing

• Long-running deployment on VINI testbed

• Develop applications for the platform

Questions

• Other motivations/applications?

• Other aspects to test?

• Design alternatives?