Post on 21-Dec-2015
RouterVMA High-Level Programming Model and Virtual Machine Architecture for Next-Generation Programmable Routers
Mel Tsaimtsai@eecs.berkeley.edu
2
OutlineThe Changing Landscape of RoutersTechnical goals of RouterVMThe RouterVM ArchitectureGeneralized Packet Filters
PropertiesFormalized ModelGPF Examples
Property Guarantees through RestrictionsExperimental Features
3
Changing Landscape of Routers
Today we see enhanced network functionality as an enterprise requirement
Increased security requirementsManaging and prioritizing trafficBalancing loadOffloading functionality that is too data intensive for servers
To serve these functions, recent hardware advancements allow application-level processing to be incorporated into router-like devices
Routers are no longer “dumb…” Hardware can support wire-speed packet classification, computation, and state management on thousands/millions of flows
4
New Requirements in the Enterprise
ISP
Edge Router
Firewall / VPN
Server LoadBalancer
IP StorageGateway
IntrusionDetection
Content Cache
LinkCompressor Switch
Switch
Switch
Switch
Switch
Server Blades
SAN
ClientWorkstations
200 Mbps
2.5 Gbps
1 Gbps
1 Gbps
1 Gbps
1 Gbps
1 Gbps
1 Gbps
1 Gbps
40 Mbps
Offsite 1-2.5 Gbps2.5 - 10 Gbps
5
New ChallengesHow does the vendor seamlessly integrate a large number of applications onto one device?
How to present an intuitive, unified management interface that properly hides the complexity of integrating multiple functions while simultaneously minimizing potential for errors?
How to achieve end-user flexibility and programmability?Vendors are reluctant to expose interfaces and allow open programmability of devicesCan customers implement new functions, collect new statistics, or reconfigure functions in ways for which the device was not intended?More generally, can programmability be achieved through a high-level interface, without requiring customers to write new code?
Can the vendor’s development framework and integrated application solution effectively target the underlying programmable hardware?
Network processors, ASICs, specialized function units, FPGAs
6
Technical Goals of RouterVM1. Support the “programming” of functionality through a high-level, expressive,
and functionally-complete building block called generalized packet filters (GPFs)
2. Show how an integrated management interface for GPFs on programmable hardware has the potential to reduce management complexity and minimize errors
3. To complement a set of standard out-of-the-box edge router functions, implement an example library of GPFs and show that it is representative of network routing and appliance applications
4. Formally and experimentally analyze the properties of the GPFs and the RouterVM execution model to understand its completeness, expressiveness, ease of specification, and other characteristics
5. Through formal analysis and by building prototype implementations, show that GPFs and the execution model can be effectively mapped onto existing and future programmable router hardware
7
Generalized Packet Filters (1)
GPFs are based on filters found in commercial routers
Packet filter 1
Packet filter 2
Packet filter n
Default filter
FILTER 19 SETUP
NAME - SIP -
SMASK - DIP -
DMASK -PROTO -
SRC PORT -DST PORT -
VLAN - ACTION -
exampleany255.255.255.25510.0.0.0255.255.255.0tcp,udpany80defaultdrop
ClassificationParameters
Action to bePerformed
…simple and easy to use, but not very powerful
Notice how a user configures relatively high-level parameters to specify the filter characteristics. New code and general programmability is not required, because in most cases, users of typical routers and switches don’t need much more flexibility here!
8
Generalized Packet Filters (2)
Key observation: the operation of packet filters can generalize to the following fundamental steps:
IP
TCP
HTTP
iSCSI
FCIP
MPLS
Ethernet
ATM
…?
Intrusion Detect
NAT
Store/Ret. State
TCP/IP lookup
Checksum
Count/Tag
…?
Error Detect
Drop
Route
Load Balance
Replace Fields
Resize Pkt
Encrypt
Forward
Compress
…?
ClassificationParameters
Infer based onpast observations Action(s)
What if all these options were made available in a packet filter?
9
Generalized Packet Filters (3)
While maintaining the parameterized and high-level specification interface of a normal packet filter, a GPF also has:
A widely expanded set of classification criteria
Normal header fieldsApplication-level headers such as URLsRegular ExpressionsMultiple combinations of the above
The ability to maintain information about packet flows
E.g., information sharing among GPFs
A widely expanded set of actionsAllow, drop, load balance, replace fields, encrypt, storage virtualize, NAT, compress, tag
The ability to implement arbitrary sequences of control flow among other GPFs
NAT, traffic shaping and monitoringL7 traffic detection (Kazaa, HTTP, AIM, POP3, etc.)QoS and packet schedulingIntrusion detectionSpam filteringProtocol conversion (e.g. IPv6)Content cachingLoad balancingRouter/server health monitoringStorage, Fibre Channel to IP, iSCSIXML preprocessingTCP offload (TOE)Encryption/compression, VPNsMulticast, Overlays, DHTs
10
GPF Example
Back
pla
ne
Back
pla
ne
ControlProcessor
ControlProcessor
QoS ModuleQoS Module
L2 SwitchingEngine w/ARP
L2 SwitchingEngine w/ARP
IP Router Engine
IP Router Engine
GPF 5:
SLB
GPF 10:
P2P…
Servers
To Clients
A Server Load Balancer and L7 Traffic Detector
10.0.0.1
10.0.0.2
Ext. IP = 24.0.5.6
10.35.x.x
11
GPF Example
Back
pla
ne
Back
pla
ne
ControlProcessor
ControlProcessor
QoS ModuleQoS Module
L2 SwitchingEngine w/ARP
L2 SwitchingEngine w/ARP
IP Router Engine
IP Router Engine
GPF 5:
SLB
GPF 10:
P2P…
Servers
To Clients
A Server Load Balancer and L7 Traffic Detector
10.0.0.1
10.0.0.2
Ext. IP = 24.0.5.6
GPF 5 Setup
name -algorithm -
flowid -sip -
smask - dip -
dmask - proto -
action1 -action2 -action3 -
Server Load Balancerequal flowssip, sportanyany24.0.5.6255.255.255.255udp, tcpslb nat 10.0.0.1, 10.0.0.2log connections, file = log.txttag “skip Yahoo Messenger Filter”
10.35.x.x
12
GPF Example
Back
pla
ne
Back
pla
ne
ControlProcessor
ControlProcessor
QoS ModuleQoS Module
L2 SwitchingEngine w/ARP
L2 SwitchingEngine w/ARP
IP Router Engine
IP Router Engine
GPF 5:
SLB
GPF 10:
P2P…
Servers
To Clients
A Server Load Balancer and L7 Traffic Detector
10.0.0.1
10.0.0.2
Ext. IP = 24.0.5.6
GPF 10 Setup
name - type -
pattern -
timeout - flowid -
sip - smask -
dip - dmask -
proto - action1 -
action2 -
Yahoo Messenger Filteryahoomessenger^(ymsg|ypns|yhoo).?.?.?.?.?.?.?(w|t).*\xc0\x8010 minsip, dip, sport, dportanyany10.35.0.0255.255.0.0tcplimit 1 kbpsemail root 10.35.x.x
13
Formalized GPF Model
Forwardpacket flow
Redirectedpackets
Packetbuffer
Classify Infer
Packetmodification,
tagging,message
generation
External and/orshared state
InternalState
To downstreamcomponent
External computeengines
Config info Config info Config info
GPF statistics and status
Sche
dule
r
14
GPF Execution Sequences
L3Classifier
QoS ModuleQoS Module
L2 SwitchingEngine w/ARP
L2 SwitchingEngine w/ARP
IP Router EngineIP Router Engine
RateLimiter
NetworkAddress
Translate
IntrusionDetection
LoadBalancer
?Many paths exist for packets…
15
Performance and Reliability Guarantees through Restrictions
Goal: be able to understand and analyze the formal properties of the RouterVM paradigm
Difficult with a fully general framework… Guarantees are more readily achieved by restricting the GPFs functionality and control flow flexibility
ExamplesExplicit declarations of shared tables and strict hardware enforcement of access and consistency
Logically or physically restricting packet buffers to one line card?
Allow only well-defined and small packet tags that makes only small adjustments to packet control flow
I.e. references to shared data structures, extracted fields, QoS priorities
16
Limits on Control FlowAllowing only forward jumps eliminates deadlock and livelock
Alternative might be implementing an in-router TTLBonus: achieves the property that multiple packets can be in-flight in the filter chain, but the processing is still analyzable and more-or-less deterministic
L3Classifier
QoS ModuleQoS Module
L2 SwitchingEngine w/ARP
L2 SwitchingEngine w/ARP
IP Router EngineIP Router Engine
RateLimiter
NetworkAddress
Translate
IntrusionDetection
LoadBalancer
?
17
Limits on Regular Expressions
Complex regular expressions may be intractable on most hardware, and may not even be necessary for most apps
Limiting searches to simple/simpler expressions allows you to bound processing time
Complex examples:snmp:
^\\x02\\x01\\x04.+([\\xa0-\\xa3]\\x02[\\x01-\\x04].?.?.?.?\\x02\\x01.?\\x02\\x01.?\\x30)|(\\xa4\\x06.+\\x40\\x04.?.?.?.?\\x02\\x01.?\\x02\\x01.?\\x43)
Java DirectConnect:\\$mynick[\\x09-\\x0d -~]*\\|\\$lock[\\x09-\\x0d -~]*\\||\\$lock[ \\x09-\\x0d -~]*pk=[\\x09-\\x0d -~]*\\|\\$hubname[\\x09-\\x0d -~]*\\||\\$key[\\x09-\\x0d -~]*\\|\\$validatenick[\\x09- \\x0d -~]*\\|
18
Experimental FeaturesMore general and user-friendly interface to look-up tables and databases
Examples: Access to an email white list for a GPF-based spam filterList of “cachable” pages in a content cache
“Define your own” classification fieldA typedef for GPF classification fields?
Conditional actions, conditional classification fieldsConditional execution of one or more actions (potentially AND’ed / OR’ed together) based on classification results
“Switch” actions:A variety of actions can be performed based on a certain field, but you don’t need to write a filter for each caseAlso gives you a “default” action when nothing matches
20
Management ConcernsSeparate appliances create a management headache… If an organization uses 10 appliances, then
Network admins must learn 10 interfacesExpensive rack space is consumed by 10 devicesSoftware updates come from 10 sources under 10 service contractsPinpointing failures involves 10 devicesChanging the network topology involves reorganization of 10 devices
Separate appliances do not communicate or share informationBecause the applications are often similar, they duplicate work waste resources, and can interact with undesirable resultsE.g., firewall is accidentally configured to block “good” traffic that is destined to a load balancer
SNMP-based management approaches (e.g. HP OpenView) may not be enough
Still has the problem of duplicate resourcesCannot always trust vendor interoperabilityHardware does not have consistent capabilities, e.g. ability to report statistics and status
21
…No Existing Solution
Network Processor Support
Supports programmability without writing new code
Considers OtherProgrammable Hardware
Considers multiple apps running simultaneously
Management and usability is a first-class consideration
Dynamic reconfiguration support
Out-of-the-box edgerouter features
Ideas can be easily deployedinto existing routers today
Net
Bin
d
Clic
k
Scou
t
NEP
AL
Library or API based approach
Teja
IP F
abric
s
Act
ive
Net
wor
ks
Gen
esis
Ker
nel
Net
Kit
Inkr
a N
etw
orks
Rou
terV
M
XOR
P
Pron
to
partially or maybeyes
unknown or unable to comment?
?
?
?
?
?
??
?
???
? ??
Management, Configurability, and
Deployment Concerns for Future Programmable
Appliances
22
GPF Hardware Execution Model
QoS ModuleQoS ModuleL2 Switching
Engine w/ ARP
L2 SwitchingEngine w/ ARP
IP Router Engine
IP Router Engine
ControlProcessor
ControlProcessor
Packet filter 1
Packet filter 2
Packet filter n
Default filter
EthernetPort
QoS ModuleQoS Module
BackplaneBackplane
L2 SwitchingEngine w/ ARP
L2 SwitchingEngine w/ ARP
IP Router Engine
IP Router Engine
Packet filter 1
Packet filter 2
Packet filter n
Default filter
EthernetPort
ComputeEngine
ComputeEngine
ComputeEngine
A virtual line card is instantiated for
every port required by the
application
A virtual backplane shuttles packets between line cards
A control CPU handles routing protocols and management tasks
When required, compute engines perform complex, high-latency processing on flows
Blue components are “standard” and are instantiated by
default. GPFs are configured and installed on a per-
application basis
23
A Mapping Example
SWITCHFABRIC
MANAGEMENTCPU
CPU1
PHY
MEM ASIC
CPU1
PHY
MEM ASIC
CPU1
PHY
MEM ASIC
CPU1
PHY
MEM ASIC
8-port RouterVM Configuration
Typical Switch Hardware Configuration
“Mapping distance”is minimized due tostructurally-parallel
organization of RouterVM
24
Limited Hardware Resources
Hardware is not infinitely fast with infinite resources; it is generally limited in the following ways:
Number and type of computational unitsAvailable parallelism (separate CPUs, hardware threads, SIMD, VLIW, etc.)Special-purpose engines (search coprocessors, table lookup, hash units)Communication bandwidth (between computation units, between units and memory, between memory and I/O)Memory (SRAM, SDRAM, local registers, cache)Memory controllers (SDRAM controllers, DMA)
Certain GPFs, and even configurations of GPFs, may not effectively map to certain hardware at the required performance
Future work is to determine where the overheads are, how to determine whether performance can meet hardware abilities, etc.Open question: when and how does RouterVM inform the user when performance cannot be met? By what mechanism does it determine this?
25
Mapping to NPU Linecards
OC-192 NetworkProcessor
SDRAM
Traffic Mangr.
Line card A
<>
GbEMAC
GbEPHY
ChProc
Backplane
SwitchFabric
<>
GbEMAC
GbEPHY
<>
GbEMAC
GbEPHY
<>
GbEMAC
GbEPHY
SearchCoproc
TCAM
SerDes
SerDes
Line card B
SerDes
ChProc
SerDes
10/1
00
Octa
l MA
C1
0/1
00
Octa
l MA
C
NetworkProcessor
X-Bar
SRAM
SDRAM
NetworkProcessor
SRAM
SDRAM
GPF1 GPF2 GPF3 GPF4 GPF5
26
Enriched Features through Packet Tagging
Tags are an interesting way to enhance the functionality of GPFs
Tags can contain extracted link layer, IPv4, TCP, iSCSI, or HTTP fieldsA different way of transferring state…Useful e.g. when hardware filters perform extraction, while software-only filters use it downstream
Tags can be mini-instructions that affect control flow:“This packet is special, ignore downstream filters of type {X, Y, and Z}”“This packet is low-priority, downstream filters can drop this if resources are low”“Ignore this packet’s L2 header and route only based on its IP header”
Tags can contain application-specific data such:Useful statisticsSCSI block numbers in IP storageFlags for an out-of-order streamSymbolic references to downstream router resources