2017 Summer Camp day5 part1 - University of Cambridge€¦ · §OSNT is an open source HW/SW...

NetFPGA SummerCourse

Presented by: Andrew W Moore, Noa Zilberman, Gianni AntichiStephen Ibanez, Marcin Wojcik, Jong Hun Han,

Salvator Galea, Murali Ramanujam, Jingyun Zhang, Yuta Tokusashi

University of CambridgeJuly 24 – July 28, 2017

http://NetFPGA.org

ThepowerofOpenSource:buildyourownproofofconcept!

GianniAntichi

ComputerLaboratoryUniversityofCambridge

[email protected]://www.cl.cam.ac.uk/~ga288

FPGA

Memory

PCI-Express

CPU Memory

NetFPGA Driver

Networking Software

10GbE

10GbE

10GbE

10GbE

10GbE

10GbE

10GbE

10GbE

EDA Tools(Xilinx,

Mentor, etc.)

VerilogmodulesinterconnectedbyFIFOinterfaces

NetFPGA:NetworkedFPGA

My Design

(10GE MAC is soft/replaceable)

BlueSwitch:AMultiTableOpenFlowSwitch

Yourdesigncanlookcompletelydifferent!

Blueswitch:Enablingprovablyconsistentconfigurationofnetworkswitches,HanJ.H.,Mundkur P.,Rotsos C.,Antichi G.,DaveN.,MooreA.W.,NeumannP.G.,ACM/IEEEANCS,Oakland,CA,USA,7-8May,2015

• Opensource,RISCbasedSoC architectures• RISC-V– RISC-VISAsoftprocessor,LinuxOS• CHERI– 64bitMIPSsoftprocessor,BSDOS

NetSoc:NetFPGA +OpenSourceProcessors

PrototypingRISCBased,ReconfigurableNetworkingApplicationsinOpenSource,HanJ.H.,Zilberman N.,Zeeb B.A,Fiessler A.andMoore A.W.,TechnicalReport,CoRR abs/1612.05547,2016

pciebench:anopensourcetoolforbenchmarkingPCIExpress

thereisalimitedunderstandingofPCIe functionality,northetrade-offsthatmustbemadetogetbest-performancefromPCIe systems

• pciebench toolopensourceavailable• ItbuildsonNetFPGA andNetronome boards

400

500

600

700 800 900

1000 1100 1200 1300 1400 1500 1600 1700

8 16 32 64 128 256 512 1024 2048

Late

ncy

(ns)

Transfer Size (Bytes)

LAT_RD (NFP6000-HSW)LAT_RD (NetFPGA-HSW)

LAT_WRRD (NFP6000-HSW)LAT_WRRD (NetFPGA-HSW)

PowerEfficientMAC• APlatformfor100Gb/spower-savingMACdesign(e.g.lights-outMAC)

• PortingMACdesigntoSUMEpermits:– Powermeasurements– Testingprotocol’sresponse– Reconsiderationofpower-savingmechanisms– Evaluatingsuitabilityforcomplexarchitecturesandsystems

• Acceleratingnetworkingservices• Compiling.Net programs

– Tox86– Tosimulationenvironment– TomultipleFPGAtargets

Emu:RapidFPGAPrototypingofNetworkingServicesinC#

Emu:RapidPrototypingofNetworkingServices,SultanaN.,etal.,Usenix AnnualTechnicalConference(ATC),July2017

OpenSourceNetworkTester

• NetFPGAplatformenabledthefirstprototypeofOSNT.

• TheopennatureofNetFPGAecosystemrepresentsthebeststartingpointforopenHW/SWcommunity-orientedprojects.

• OSNTaimstobuildacommunityasNetFPGAdid.OSNT:OpenSourceNetworkTester,Antichi G.etal.,IEEENetworkMagazine,SpecialissueonOpenSourceforNetworking:ToolsandApplications,2014

Opensourcehardwareandsoftwareplatformfornetworkmonitoringandtesting.

https://osnt.orgLowcost,flexibletoupdate,scale-out,noCPUusage,nanosecondresolutionmeasurements

OSNT:OpenSourceNetworkTester

OSNT:OpenSourceNetworkTester,Antichi G.etal.,IEEENetworkMagazine,SpecialissueonOpenSourceforNetworking:ToolsandApplications,2014

NetworkTesterComparison

Cost Flexibility Precision LineRate

DPDK,Moongen

OpenSourceNetworkTester§ OSNTisanopensourceHW/SWplatformfornetworktesting

OSNTHardware

OSKernelNetFPGAKernelDriver

OSNTApplications

nf0 nf1 nf2 nf3

Generator Monitor

x8LanesPCIEGen3

Port0 Port1 Port2 Port3

§ WritteninC,Python§ OpenAPIandregisters

§ WritteninVerilogHDLusingstandardXilinxprotocols

§ Userscanaddandmodifythemodules

OpenSourceNetworkTester§ OSNTisanopensourceHW/SWplatformfornetworktesting

OSNTHardware

OSKernelNetFPGAKernelDriver

OSNTApplications

nf0 nf1 nf2 nf3

Generator Monitor

x8LanesPCIEGen3

Port0 Port1 Port2 Port3

§ WritteninC,Python§ OpenAPIandregisters

§ WritteninVerilogHDLusingstandardXilinxprotocols

§ Userscanaddandmodifythemodules

YOURApplications

YOURLogic

OpenSourceNetworkTesterOSNTcurrently is:

§ 4x10Gbpstrafficgenerator.§ Capturecardwithhighresolutiontimestamp(6.4nsec).§ GPS-readysynchronizedmeasurementkit.

OpenSourceNetworkTesterOSNTcurrently is:

§ 4x10Gbpstrafficgenerator.§ Capturecardwithhighresolutiontimestamp(6.4nsec).§ GPS-readysynchronizedmeasurementkit.

astartingpoint

OSNTArchitecture

InputArbiter Monitor Generator Output

Queues

HOST

PCIe


Queues

HOST

PCIe

OSNTmonitor

OSNTArchitecture


Queues

HOST

PCIe

OSNTmonitor OSNTgenerator

OSNTArchitecture

OSNTRXinterface

MACPCS/PMASFPPacketin

§ TimestamptakenbeforeRXqueuestoreduceFIFO-inducedjitter

OSNTRXinterface

MACPCS/PMASFPPacketin

§ TimestamptakenbeforeRXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset


Queues

HOST

PCIe


OSNTArchitecture

OSNTTXinterface

MACPCS/PMA

SFP Packetout

§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter

OSNTTXinterface

MACPCS/PMA

SFP Packetout

§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset

OSNTTXinterface

MACPCS/PMA

SFP Packetout

§ TimestamptakenafterTXqueuestoreduceFIFO-inducedjitter§ Timestampoverwritespacketdataataconfigurableoffset§ Ifenabled,itwilloverwrite128bitdata:

Dst MAC ... signature pkt count tx timestamp ...

32 bit 32 bit 64 bit

OSNTTimestamp

§ Freerunningcounter?

§ Wecouldusea64-bitcounterdrivenbythe156.25MHzsystemclock(naïvesolution)

OSNTTimestamp



– providesnomeansbywhichtocorrectoscillatorfrequencydrift

OSNTTimestamp



– providesnomeansbywhichtocorrectoscillatorfrequencydrift– producestimestampsexpressedinunitof6.4ns

OSNTTimestamp



– providesnomeansbywhichtocorrectoscillatorfrequencydrift– producestimestampsexpressedinunitof6.4ns– fixed-pointrepresentationoftimeinsecondsmoreusefultohost

OSNTTimestamp

§ DirectDigitalSynthesis(DDS)isthesolution!!

§ DDSisatechniquebywhicharbitraryvariablefrequenciescanbegenerated

OSNTTimestamp



– needatimereferencetocorrectDDSrate(theGPSprovideslong-termstability)

OSNTTimestamp




– allow64bitvalueinfixed-pointrepresentation

OSNTTimestamp




– allow64bitvalueinfixed-pointrepresentation– how Endace DAGcardworks!


Queues

HOST

PCIe


OSNTArchitecture

OSNTGenerator

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

PCAPReplayEngine

QDRController

QDRController

DDR3Controller

DDR3Controller

PCIeDMA

To/FromQDR

Memory

To/FromQDR

Memory

To/FromDDR3

Memory

To/FromDDR3

Memory

DM:DelayModule

RL:RateLimiter

TS:Timestamp

InputArbiter

To/FromHostPC

OSNTGenerator

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

PCAPReplayEngine

QDRController

QDRController

DDR3Controller

DDR3Controller

PCIeDMA

To/FromQDR

Memory

To/FromQDR

Memory

To/FromDDR3

Memory

To/FromDDR3

Memory

DM:DelayModule

RL:RateLimiter

TS:Timestamp

InputArbiter

To/FromHostPC

§ 4x10GPCAPreplayengine

§ SRAM:27MB§ DRAM:8GB

OSNTGenerator



§ DelaymoduleRL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

PCAPReplayEngine

QDRController

QDRController

DDR3Controller

DDR3Controller

PCIeDMA

To/FromQDR

Memory

To/FromQDR

Memory

To/FromDDR3

Memory

To/FromDDR3

Memory

DM:DelayModule

RL:RateLimiter

TS:Timestamp

InputArbiter

To/FromHostPC



§ Delaymodule§ Ratelimiter

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

PCAPReplayEngine

QDRController

QDRController

DDR3Controller

DDR3Controller

PCIeDMA

To/FromQDR

Memory

To/FromQDR

Memory

To/FromDDR3

Memory

To/FromDDR3

Memory

DM:DelayModule

RL:RateLimiter

TS:Timestamp

InputArbiter

To/FromHostPC

OSNTGenerator



§ Delaymodule§ Ratelimiter§ TXtimestamping

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

RL

DM

TS

10GTx

PCAPReplayEngine

QDRController

QDRController

DDR3Controller

DDR3Controller

PCIeDMA

To/FromQDR

Memory

To/FromQDR

Memory

To/FromDDR3

Memory

To/FromDDR3

Memory

DM:DelayModule

RL:RateLimiter

TS:Timestamp

InputArbiter

To/FromHostPC

OSNTGenerator


Queues

HOST

PCIe


OSNTArchiteture

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter

Aggregatestatisticstohostsoftware

TCAMrulemanager

Cut/Hashsettingfromhost

Hostanalysissoftware

CoreMonitoring

FilteringStage

PCIe DMA

OSNTMonitor

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter


TCAMrulemanager



CoreMonitoring

FilteringStage

PCIe DMA

packetenteringtheboard

OSNTMonitor

§ RXtimestamp

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter


TCAMrulemanager



CoreMonitoring

FilteringStage

PCIe DMA

OSNTMonitor

§ RXtimestamp

§ Statscollector

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter


TCAMrulemanager



CoreMonitoring

FilteringStage

PCIe DMA

OSNTMonitor

§ RXtimestamp

§ Statscollector

§ TCAM-basedPacketFilter(5-tuple)

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter


TCAMrulemanager



CoreMonitoring

FilteringStage

PCIe DMA

OSNTMonitor

§ RXtimestamp

§ Statscollector

§ TCAM-basedPacketFilter(5-tuple)

§ Cut-Hashfunction

10GRx

10GRx

10GRx

10GRx

RxQ RxQ RxQ RxQ

StatisticsCollector

HeaderExtractor

TCAM

DecisionModule

PacketsFIFO

Cut/Hash

TimeStamp

InputArbiter


TCAMrulemanager



CoreMonitoring

FilteringStage

PCIe DMA

OSNTMonitor

OSNTGUI§ OSNTGUI– ExtensibleGeneratorandMonitorGUIinPython.§ Command-Line-Interfaceisalsoavailable.

GeneratorGUI

MonitorGUI

49

OSNTcommandline§ Command-Line-Interfaceisavailabletocreateascriptautomatingthetestprocess.

OSNTinaction

Enablingnetworkinnovationwithaccuratenetworkingsystemscharacterization

Forwardinglatencymeasurement§ Unloadedswitchesbaselinelatencynocrosstraffic

TS-Tx

OSNT

OSKernel

Application

TS-Rx

deviceundertest

Forwardinglatencymeasurement


SDNTestingSuite

• SDNenablesunprecedentedflexibleandextensiblenetworkcontrol

• OpenFlow specificationslackperformancesemantics– Whatdoesabarrierreplysignifies?

• OpenFlow performanceaspectsareyettobeexplored– HowdoyoucomparetwoOpenFlow switches?

• OpenFlow flexibilityisnotalwaysportableonswitchASIC

OpenFlow toolstack X-Ray

ControlApplicationComplexity



ControlChannelCapacity




Scarceco-processorresourcesSwitchOSschedulingisnontrivial




Scarceco-processorresourcesSwitchOSschedulingisnontrivial

ASICdriver->policyconfiguration


SDNController

SDNnetworksperformances

THEPROACTIVECASE

01.Barrierrequest02.Setofnewrules

SDNController

THEPROACTIVECASE

Barrierreply01.Barrierrequest02.Setofnewrules


SDNController

THEPROACTIVECASE


HWdatapathSWcontroller


SDNController

THEPROACTIVECASE


HWdatapath

SWcontroller?


Control/DataplaneconsistencyConsistentpolicyupdateaffectssecurityinSDN.

SW0Untrusted

Port1

Untrusted

SW1

SW2

Trusted

Port2

U->SW1T->SW2

SwitchController

Targetstateneededtoupdate

T->SW1U->SW2

OSNT

OSKernel

Application

NIC

0 1 2 3

ControlChannel

Control/Dataplaneconsistency

OSNT

OSKernel

Application

NIC

0 1 2 3

ControlChannel

1.Initialrule:0® 1

2.Ruleupdate :0® 2

(asasetofdifferentIPrules)


OSNT

OSKernel

Application

NIC

0 1 2 3

ControlChannel

1.Initialrule:0® 1

2.Ruleupdate :0® 2

(asasetofdifferentIPrules)

Wegenerateanaggregate2Gbpswith150Bpackets



TCAM+RAM



RAM




Aflowmodificationrequiresonlytochangethe“action”intheRAM.TheflowisalreadypresentintheTCAM.



AflowinsertionrequiresalsotowritethematchingfieldsintheTCAM.

Dataplane performances

0

5

10

15

20

0 200 400 600 800 1000

inse

rtio

n d

ela

y (s

ec)

number of flows

Pica8 OVSPica8 L2/L3

Force10

Flowadditiondelay

OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015

0

0.5

1

1.5

2

2.5

3

0 200 400 600 800 1000

inse

rtio

n d

ela

y (s

ec)

number of flows

Pica8 OVSPica8 L2/L3

Force10

Dataplane performancesFlowmodificationdelay

OFLOPS-Turbo:TestingtheNext-GenerationOpenFlow Switch,Rostos C.,Antichi G.,Bruyere M.,OwezarskiP.,MooreA.W.,IEEEICC,London,UK,8-12June,2015


THEREACTIVECASE

SDNController

packet

THEREACTIVECASE

SDNController

packet modifiedpacket


ControllerperformancesinSDNnetworks

Controlchannel

DataTrafficLink

OSNT

OSKernel

OFController OFSwitch

OFControllers

ControllerperformancesinSDNnetworks

OSNT-SUME-liveGithub§ OSNT-SUME-liveispubliclyavailable.

Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng,

Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo, Stephen Gabriel Ibanez

Acknowledgments(I)

NetFPGA Team at Stanford University (Past and Present):

NetFPGA Team at University of Cambridge (Past and Present):Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik, Matthew Grosvenor, Yury

Audzevich, Neelakandan Manihatty-Bojan, Georgina Kalogeridou, Jong Hun Han, Noa Zilberman, Gianni Antichi, Charalampos Rotsos, Hwanju Kim, Marco Forconesi, Jinyun Zhang,

Bjoern Zeeb, Robert Watson, Salvator Galea, Marcin Wojcik, Diana Andreea Popescu, Murali Ramanujam

All Community members (including but not limited to):Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff

Shafer, Eric Keller, Tatsuya Yabe, Bilal Anwer, Yashar Ganjali, Martin Labrecque, Lisa Donatini, Sergio Lopez-Buedo , Andreas Fiessler, Robert Soule, Pietro Bressana, Yuta Tokusashi

Patrick Lysaght, Kees Vissers, Michaela Blott, Shep Siegel, Cathal McCabe

Steve Wang, Erik Cengar, Michael Alexander, Sam Bobrowicz, Garrett Aufdemberg,Patrick Kane, Tom Weldon

Acknowledgements (II)

2017 Summer Camp day5 part1 - University of Cambridge€¦ · §OSNT is an open source HW/SW...

Documents

Transcript of 2017 Summer Camp day5 part1 - University of Cambridge€¦ · §OSNT is an open source HW/SW...