Patrick Crowley and Jon Turnerand John DeHart, Mart Haitjema Fred Kuhns, Jyoti Parwatikar, Ritun Patney, Charlie Wiseman, Mike Wilson, Ken Wong, Dave ZarComputer Science & EngineeringWashington University

www.arl.wustl.edu

Prototype Deployment of Internet Scale Overlay HostingStatus Report – 11/2009

2

Project Objectives

Deploy five experimental overlay hosting platforms » located at Internet 2 PoPs» compatible with PlanetLab, moving to GENI control framework» performance characteristics suitable for service deployment

• integrated system architecture with multiple server blades• shared NP-based server blades for fast-path packet processing

Demonstrate multiple applications

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

Control Processor

General Purpose Processing Engines

Network Processing EngineLine Card

Chassis Switch

External Switch

3

Target Internet 2 Deployment

2

2

2

11

2

2

4

StatusFirst three nodes deployed and operational

» installation went smoothly (mostly)» two hardware issues

• replaced KC linecard, still need to replace one GPE at SALTDemonstration this afternoonReady for a few friendly users

» still need to complete flow stats» version 1 of NPE software» not yet ready to add fastpath code options

• but happy to start working with interested groupsTutorial planned for next GEC

» let me know if you’re interested

5

Current Deployment

I2 Optical Connections

ProtoGENI ProtoGENI ProtoGENI

SPP SPP SPPSPPLCSALT KANS WASHdrn06.arl.wustl.edu

I2 Internet ServiceI2 Router I2 Router I2 Router

64.57.23.210 .214.218

64.57.23.19464.57.23.178 .198.204

.182

.186

6

Photo Gallery (thanks Chris)

7

Hosting Platform Details

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

PLOS

VM VM . . .

General PurposeProcessing Engine

filter

. . .

. . .

filter

Line Cardlookupparse header

format

... ...

queues

...

Network Processing Engine

9

Demonstrations this Afternoon IPv4 network using NP-based fastpaths

» traffic generated from PlanetLab nodes» demonstrates dynamic changes to installed filters

• script in GPE slice issues control messages to modify filters» handling of exception cases by GPE slice» use of traffic monitoring

Forest overlay network – GPE-only implementation» multiple dynamic multicast groups over per-session,

tree-structured communication channels» traffic generated from PlanetLab nodes» traffic monitoring using statistics data generated by

GPE slice

10

IPv4 Demo

Planetlab trafficgenerators

Line Card

SALT-R2NP

E

CP GPE

GPE

WASH-R3

NPE

CP GPE

GPE

Line Card

KANS-R1

NPE

CP GPE

GPE

Line Card

experimentcontrol

SPPLC

CTL

FP CTL

FP

CTL

FP

11

Forest Overlay NetworkFocus on real-time distributed

applications» large online virtual worlds» distributed cyber-physical systems

Large distributed sessions» endpoints issue periodic status reports» and subscribe to dynamically changing sets of reports» requires real-time, non-stop data delivery, even as

communication pattern changesPer-session communication channels (comtrees)

» unicast data delivery with route learning» dynamic multicast subscriptions using multicast core

overlayrouter

client

serveraccess

connection

comtree

12

Forest DemoSALT-R2

NPE

CP GPE

GPE

Line Card

WASH-R3

NPE

CP GPE

GPE

Line Card

KANS-R1

NPE

CP GPE

GPE

Line Card

fRtr

fRtr

fRtr

R2

R3R1

comtree2 R1

R3R2

comtree1

multicastsender multicast

sender

multicastsendermulticast

subscribers

multicastsubscribers

multicastsubscribers

Planetlab nodes

SPPLC

experimentcontrol

13

Developing a new SPP ApplicationCreate reference model as foundation

» work out functional issues» develop and debug control code» build with separate control and datapath processes» implement in Open Network Lab or Emulab

Port reference model to SPPs using GPEs only» initially, just reproduce reference model functionality» extend to use multiple external interfaces, reserved

capacityDesign new NPE code option

» we’ll work with you and do NPE code development» need working GPE version and fastpath design spec

14

Working with SPPsDefine new slice using SPPLC

» much as with PlanetLabLogin to GPE slice to install application codeReserve resources needed for your experiment

» includes interface bandwidth on external ports and NPE fastpath resources

To run experiment during reserved period» configure reserved resources as needed» setup logical endpoints (aka meta-interfaces)» configure fastpath (filters, queues)» setup real-time monitoring» run application and start traffic generators

15

Web Site and SPPLC

drn06.arl.wustl.edu

http://wiki.arl.wustl.edu/index.php/Internet_Scale_Overlay_Hosting

16

Steps in Setting Running ExperimentReserve resources in advanceAt start of session

» configure logical interfaces (aka meta-interfaces)• associate with physical interfaces and port numbers• assign bandwidth

» setup fastpath• assign queues to logical interfaces, set queue parameters• install fastpath filters

» setup traffic monitoringAt end of session

» release resources in use» cancel any remaining reservation time

17

Command Line Toolsresrv – reserve resources in advance

» make_resrv,update_resrv,cancel_resrv,get_resrv,...client – perform variety of operations

» get_ifaces ,get_ifattrs » get_queue_params, set_queue_params, bind_queue » alloc_endpoint, alloc_tunnel» write_fltr, update_result, get_fltr_bykey» read_stats, create_periodic, get_queue_len

create-fp – instantiate fastpathfltr – manipulate fastpath filters for IPv4 code optsliced – monitor traffic and display remotely

18

Reserving Resources in Advanceresrv --cmd make_resrv --xfile resFile

<spp> <rsvRecord> <rDate start="20091117140000" end="20091117160000" /> <fpRSpec name=“jstFastpath"> <fpParams> <copt>1</copt> <bwspec firm="1000000" soft="0" /> <rcnts fltrs="100" queues="10" buffs=“1000" stats=“20" /> ... </fpParams> <ifParams> <ifRec bw=“20000" ip="64.57.23.194" /> ... </ifParams> </fpRSpec> <plRSpec> <ifParams> <ifRec bw="10000" ip="64.57.23.194" /> </ifParams> </plRSpec> </rsvRecord></spp>

reserved time period

aggregate fastpath bandwidth

# of TCAM filters, queues, packet buffers, traffic counters

bandwidth on external interfaces, specified by IP address

external interface bw for GPE slice

19

Configuring Logical InterfacesFor use by GPE slice

» client --cmd alloc_endpoint--bw 1000 --ipaddr 64.57.23.210 --proto 6 --port 3551

For use by fastpath» client --cmd alloc_udp_tunnel

--fpid 0 --bw 16000 --ipaddr 64.57.23.210 --port 20001

setup externally visible port

in Kb/saddress of interface

6 for TCP, 17 for UDP

port number

identifier of fastpath within this slice

20

Configuring FastpathBinding queues to logical interfaces

» client --cmd bind_queue --fpid 0 --miid 1 --qid_list_type 0 --qid_list 4 --qid_list 6

Setting queue parameters» client --cmd set_queue_params --fpid 0

--qid 3 --threshold 1000 --bw 8000 Setting up filters (for IPv4 fastpaths)

» fltr --cmd write_fltr --fpid 0 --fid 10 --key_type 0 --key_rxmi 2 --key_daddr xx --mask_daddr xx --key_dport xx –-txdaddr xx --txdport xx --qid 5

meta-interface id

assigned queues

21

Displaying Real-Time Data

Fastpath maintains traffic counters and queue lengths

To display traffic data» configure an external TCP port» run sliced within your vServer, using configured port

• /sliced --ip 64.57.23.194 --port 3552 &» on remote machine, run SPPmon.jar

• use provided GUI to setup monitoring displays• SPPmon configures sliced, which polls the NPE running the

fastpath• SCD-N reads and returns counter values to sliced• can also display data written to file within your vServer

NPE

CPGPE

GPE

Line Card

SCD-N

sliced

SPPmon

22

SummarySPPs ready for use

» start with a few friendly users, but will open up soon» tutorial at next GEC

To do list» complete flow stats subsystem» version 2 of NPE software

• higher performance and multicast support» resource allocation mechanisms

• enable coordinated reservation across all nodes» work with user groups on new code options