The BLU Project - ITNOG · Quagga (BGP) The BLU project BLUos EOLO customizations •...

TheBLUProjectPast,Present,andFuture

2

The“BLUpeople”

F.Alberti P.Biasoli S.Ceccato M.Citterio A.Milani

3

Outline

1.Challenges

2.Tools

3.Solutions

4

About EOLO

• Atthebeginning(2006):• Broadbandforruralareas(innorthernItaly)

• Today:• Coverage:>90%northernItaly,>50%centralItaly• 230Kcustomers(4%marketshare)• From30Mbpsto1Gbpsandwhite-labelwholesale

• Eachmonth:• 13Knewcustomers

5

FWA– AnatomyofaBaseTransceiverStation

Tothetop• Backhaul(PTPlinks)• Clientaccess(PMPlinks)

Totheground• Routers,switches• Energystations

6

About EOLO

• FixedWirelessAccess(FWA)ISP

• CoreinMilan

• Backhaulconnectionsbymeansofwireless linksestablishedbetween BaseTransceiverStation

7

(Avery)Distinguishing feature

Opportunisticapproach• NewlinkestablishedwhenevertwoBTSs“seeeachothers”

8

ChallengesDiameter ofthenetwork

12hops

9

ChallengesTopology changes

2Days

Radiolinkssufferbadweatherconditions

MAX16minuteswithoutTC

10

Oursolution:theBLUproject

BLUrouters BLUos BLU-GW

11

TheBLUprojectRouters

• TileGX architecture• 72coreCPU• 16GBDDR3RAM• 128GBSSD• Lowpower:90Wavg,150Wmax• 24x1Gports,2x10Gports

• IEEE1613compliant• Electromagneticcompatibility• Error-freeoperationin-15C/+75Cenvironment• EMIfieldsofupto180V/m

12

Basedon6windgateby6WINDS.A.• Linux3.10.61

• OpenVSwitch

• 6WINDuserspace networkstack• BasedonDPDK• Associatecorestoports(differentconfigurationbasedontherouterstasks/position)

• Userspace dataplane – HIGHLYEFFICIENT

• PPPdaemon

• DHCPrelay

• Quagga(BGP)

TheBLUprojectBLUos

EOLOcustomizations• OpenVSwitch mac-learningimproved

• RFC3107insideQuaggaBGP

• Automaticset-up/rescueprocedure• Self-managementstrategies• Corruptedconfigurations• Hosttemporarilydisconnected

• Monitoringdaemons• LLDPneighboringdetails• Bandwidth(radiolinkssufferbadweather,interferences,etc.)

• Tx/Rxtrafficoneachinterface(bothBHandAccess)

13

• Avoid(or,atleast,limit)humanerror

• FullmanagementofourBLUrouters• Parallelexecutionofcommands• Schedulerfornightupdater• Success/failure/rollback

• CollectionandanalysisofdatafromtheBLUrouters• Topologyupdates• Statistics,logsanderrormessages

• Statisticsvisualization

TheBLUprojectBLU-GW

14

TheBLUprojectBLU-GW- Management

15

TheBLUprojectBLU-GW- Management

Openflowtemplateeditor

16

TheBLUprojectBLU-GW- Monitoring

Auto-completionSQL-likesyntax

17


Interfaceconf.

Tx/Rx

18


Plentyofstatistics(e.g.,CPUusage)

19


Plentyofstatistics(e.g.,temperatures)

20

TheBLUprojectPhase1- Challenges

• Backhaul:• Wirelesslinks• PVST+L2

• Veryhighnumberofmac-address!

• PPP?DHCP?

• Trafficbalancing?Overloadedlinks?

21

TheBLUprojectPhase1- Solution

• Backhaul:• Wirelesslinks• PVST+L2

• Veryhighnumberofmac-address!• PPPserver/DHCPrelayontheBLU• EachBLUadvertisestothecoreitsPPP/DHCPIPpools

• Trafficbalancing?Overloadedlinks?• MultipleBHvlans (costchangesdeterminedifferentpaths)

• Costchangesperformedmanually!

• OVSstillsuffersthehighnumber(>1KperBLU)ofmac-addresses!• Flows-handlingperformancedecreases– trafficgetslost

22

TheBLUprojectBLUos – HackingOVS

in_port(2),eth(src=00:00:00:00:00:01,dst=00:00:00:00:05:03),…,actions:9in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:00:01),…,actions:2

00:00:00:00:00:01

00:00:00:00:00:02

00:00:00:00:05:03

eth2 eth9

23


in_port(2),eth(src=00:00:00:00:00:01,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:01:01,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:02:01,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:03:01,dst=00:00:00:00:05:03),…,actions:9

in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:00:01),…,actions:2in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:01:01),…,actions:2in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:02:01),…,actions:2

in_port(9),eth(src=00:00:00:00:05:03,dst=ff:ff:ff:ff:ff:ff),…,actions:2,3,4,5in_port(9),eth(src=00:00:00:00:01:03,dst=ff:ff:ff:ff:ff:ff),…,actions:2,3,4,5

00:00:00:00:00:01

00:00:00:00:00:02

00:00:00:00:05:03

00:00:00:00:01:01

00:00:00:00:03:01

00:00:00:00:02:01

00:00:00:00:04:03

00:00:00:00:06:03

eth2 eth9

24


in_port(2),eth(src=*,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:01:01,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:02:01,dst=00:00:00:00:05:03),…,actions:9in_port(2),eth(src=00:00:00:00:03:01,dst=00:00:00:00:05:03),…,actions:9

in_port(9),eth(src=*,dst=00:00:00:00:00:01),…,actions:2in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:01:01),…,actions:2in_port(9),eth(src=00:00:00:00:05:03,dst=00:00:00:00:02:01),…,actions:2

in_port(9),eth(src=*,dst=ff:ff:ff:ff:ff:ff),…,actions:2,3,4,5in_port(9),eth(src=00:00:00:00:01:03,dst=ff:ff:ff:ff:ff:ff),…,actions:2,3,4,5

00:00:00:00:00:01

00:00:00:00:00:02

00:00:00:00:05:03

00:00:00:00:01:01

00:00:00:00:03:01

00:00:00:00:02:01

00:00:00:00:04:03

00:00:00:00:06:03

eth2 eth9

25


26

TheBLUprojectPhase2

• BLUGWknows• Linkcapacity• ResidualBHlinkcapacity• BLUs’“desiredbandwidth”(accessifaces bandwidth)

• AcentralizedSDNcontrollerdecidestherouting• Residualcapacitygetsoptimized• InformstheBLUonhowtohandletraffic

• Openflow rulesforMPLSlabelswitching

• RFC3107for(downlink)trafficlabeling

27

TheBLUprojectPhase2

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.75

0.35 0.45

0.5

6

0.15

Internet

28

TheBLUprojectPhase2

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.15

0.75

0.35 0.45

0.5

6

Internet

BLUGW

29

TheBLUprojectPhase2

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.15

0.75

0.35 0.45

0.5

6

in_port(2),mpls=L7,out_port(3) in_port(5),mpls=L7,out_port(9)

Internet

Dst=R7,Label=L7

BLUGW

30

TheBLUprojectPhase2

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.15

0.15

0.35 0.45

0.5

6

Internet

Dst=R7,Label=L7

BLUGW

31

TheBLUprojectPhase2

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.15

0.15

0.35 0.45

0.5

6

in_port(2),mpls=L7,out_port(3)



Internet


Dst=R7,Label=L7

BLUGW

in_port(2),mpls=L7,out_port(3) in_port(5),mpls=L7,out_port(9)

32

TheBLUprojectPhase2- Issues

• OpenFlow granularityisrathersmall• AsingleflowthroughthenetworkrequiresalltheBLUsalongthepathtobe(re)programmed

• Everytimethenetworkchanges(!!!)thecontrollerneedstore-computetheroutesandupgradetheBLU

• Upgradesneedtobeatomic,highriskofinconsistentstateofthenetwork

• Thereisnoa“BIGREDBUTTON”tobepressedincaseofcriticalissues!• Howtogobacktoasafeconfigurationincaseoftroubles?

33

TheBLUprojectPhase3

• SegmentRoutingphilosophy

• IGPrunningontheBLUrouters• eBGP,RFC3107

• RFC3107+SRpolicyfordownlinktrafficlabeling

• Labelswitching(OVS)• eBGP governsOpenvSwitch:activeroutesdeterminelabelswitchingflows

34

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.75

0.35 0.45

0.5

6

0.15

Internet

35

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=L7

36

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=L7

37

TheBLUprojectPhase3– TrafficEngineering

• Howtooptimizethetrafficflows?

• BLUGWstillknows• (quasi-)Realtimetopology• Residualbandwidth• Desiredaccessifaces bandwidth• NEW:BLUs’IGProutingtables

• Optimizationgoesthroughthesepoints:• Findawaypointwheretoroutethetraffictobeoptimized• InstructtheCOREontherequiredlabelstack

38

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.5

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=L7

mpls=L6,out_port(2)

mpls=L6,out_port(4)mpls=L6,out_port(5)


mpls=L6,out_port(7)

39

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.25

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=L7

mpls=L6,out_port(2)



mpls=L6,out_port(7)

40

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.25

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=[L6,L7]

mpls=L6,out_port(2)



mpls=L6,out_port(7)

41

TheBLUprojectPhase3

R7

R6

R5R3

R4

R2

Core

0.4Gbps

10

1

0.5

0.25

0.75

0.35 0.45

0.5

6

0.15

Internet

mpls=L7,out_port(3)

mpls=L7,out_port(4)

mpls=L7,out_port(9)

mpls=L7,out_port(2)

mpls=L7,out_port(5)

mpls=L7,out_port(3)

Dst=R7,Label=[L6,L7]

mpls=L6,out_port(2)



mpls=L6,out_port(7)

42

TheBLUprojectPhase3- Pros

• Noneedtore-programtheBLUsfromBLUGW• TheyautomaticallylearnhowtoforwardMPLSlabelsbasedonIGProutes

• Optimizer:• Comesintoplayonlyincaseofcongestion• Manytoolstotackletheoptimizationproblem

• Operationsresearch,LP,SMT,etc.+alotofheuristics

• Wedohavea“BigRedButton” now!• Incaseoftroubles,turningofftheoptimizerrevertstoshortestpathforwarding

43

TheBLUprojectTheFuture

• Newhardware()• Motivation:needofmorethan2x10Ginterfacesonimportantsites• SingleinsteadofmanyBLUs(powerefficiency)• Intelbased• BLUos compliant

• Latency-basedoptimizationsfor“delay-aware”traffic(e.g,voice,videostreaming,etc.)• BLUscanmeasurelatencyfortheirBHlinks• Optimizercantakethesevaluesintoconsideration

• BLU2BLU(almost)forfree!• src BLUtagstrafficwiththedst BLU’slabel• …orstackoflabels(dependingonwhattheoptimizersays)

44

TheBLUproject

Thankyou!

[email protected]

IfyouareinterestedintheBLUproject,donothesitatetocontactus!

The BLU Project - ITNOG · Quagga (BGP) The BLU project BLUos EOLO customizations •...

Documents

Transcript of The BLU Project - ITNOG · Quagga (BGP) The BLU project BLUos EOLO customizations •...