pmacct · 2017-05-28 · LinkedIn: plucente Digging data out of networks worldwide for fun and...
28
GRNOG Workshop #1, Athens – May 2017 Introduc?on to pmacct Paolo Lucente pmacct
Transcript of pmacct · 2017-05-28 · LinkedIn: plucente Digging data out of networks worldwide for fun and...
GRNOGWorkshop#1,Athens–May2017
Introduc?ontopmacct
PaoloLucentepmacct
whoami
PaoloLucenteGitHub:paololucenteLinkedIn:plucenteDiggingdataoutofnetworksworldwideforfunandprofitformorethan10years
libpcap
pmacctisopen-source,free,GPL’edsoOware
sFlow
BGP
maps
IGP
MySQL PgSQL SQLite
MongoDB BerkeleyDB
flat-files
RabbitMQ Kafka
memory tables
sFlow
tee
NetFlow IPFIX
NetFlow IPFIX
hPp://www.pmacct.net/
Streaming Telemetry
BMP GeoIP
pmacct:afewsimpleuse-cases
BMP
flat-files
tee
NetFlow IPFIX
sFlow
Kafka
IPFIX libpcap
pmacct:aslightlymorecomplexuse-case
BGP
flat-files
tee
NetFlow IPFIX
Kafka
MySQL
aggregation method #1
aggregation method #2
nfacctd
Theuse-caseformessagebrokers
pmacct-to-elas?csearch0.3.0
Creditsto:PierCarloChiodi,hPps://github.com/pierky/pmacct-to-elas?csearch
Usecasesbyindustry
Keypmacctnon-technicalfacts
§ 10+yearsoldproject§ Can’tspellthenameaOertheseconddrink§ Free,open-source,independent§ Underac?vedevelopment§ Innova?onbeingintroduced§ Welldeployedaround,alsolargeSPs§ Aimstobethetrafficaccoun?ngtoolclosertotheSPcommunityneeds
Sometechnicalfacts(1/2)
§ Pluggablearchitecture:• Caneasilyaddsupportfornewdatasourcesandbackends
§ Correla?onofdatasources:• Na?velysupporteddatasources(ie.BGP,BMP,IGP,StreamingTelemetry)
• Externaldatasourcesviatagsandlabels§ Pervasivedata-reduc?ontechniques,ie.:• Dataaggrega?on• Filtering• Sampling
Sometechnicalfacts(2/2)
§ Buildmul?pleviewsoutoftheverysamecollectednetworktrafficdataset,ie.:• Unaggregatedtoflat-filesforsecurityandforensics;ortomessagebrokers(RabbitMQ,Kaga)forBigData
• Aggregatedas[<ingressrouter>,<ingressinterface>,<BGPnext-hop>,<peerdes?na?onASN>]andsenttoaSQLDBtobuildaninternaltrafficmatrixforcapacityplanningpurposes
§ Enableanaly?csagainstthecollecteddatasources(ie.BGP,BMP,StreamingTelemetry):• Streamreal-?me• Dumpatregular?meintervals(possiblestatecompression)
Furtherinforma?onaboutpmacct
§ hPps://github.com/pmacct/pmacct• OfficialGitHubrepository,wherestarandwatchusJ
§ hPp://www.pmacct.net/lucente_pmacct_uknof14.pdf• MoreaboutcouplingtelemetryandBGP
§ hPp://ripe61.ripe.net/presenta?ons/156-ripe61-bcp-planning-and-te.pdf• Moreabouttrafficmatrices,capacityplanning&TE
§ hPps://github.com/pmacct/pmacct/wiki/• Wiki:docs,implementa?onnotes,ecosystem,etc.
ItjustseemedGreecewastherightplacewheretoshareoctopusstories..
GRNOGWorkshop#1,Athens–May2017
Thanks!Ques?ons?
PaoloLucente<[email protected]>
hPp://www.pmacct.net/|hPps://github.com/pmacct/pmacct
Introduc?ontopmacct
GRNOGWorkshop#1,Athens–May2017
GRNOGWorkshop#1,Athens–May2017
LatestsonBGPmonitoring
PaoloLucenteNTTCommunica?ons|pmacct
BGP
§ Protocoltoadver?seReachabilityInforma?on:• TheNetworkLayerpartofthestory,whiles?lldominant,is“old”:BGPisusedastransportforavarietyofdifferentinfo
§ Goodatpolicycontrol:• Althoughqualityfactors,ie.latency,jiPerandpacketloss,increasinglypopularforcontentdeliveryinplaceofthetradi?onalBGPselec?onalgorithm
§ Goodatinforma?onhiding:• But,thenagain,thisistherecipeforscalingtothecurrentInternetsizeandbeyond
EarlyaPemptsatgainingvisibility
Creditsto:E.Jasinska(Nezlix),P.Lucente(pmacct)@NANOG61
§ Circa2013§ Goal:seeallpathsinaBGPmul?-pathscenario,avoidingscreenscraping
BMP§ BGPMonitoringProtocol§ RFC7854:• firstdraOin2008,sparseworkun?l2012;• stallbetween2012and2015;• realtrac?onkicksin:10draOsbetween2015and2016;• RFCawardinJun2016
§ Uncomplicatedprotocoldesign§ Greateffortbut..• ..industryevolvedalltheseyears• increasedhungerfordata
ADevOpsguyduringlunchbreak
Tradi?onalBGPmonitoring
Peers
VantagePoint(VP)
Best
Path
BGPistheworld’sbestinforma?onhidingprotocol<tm>.Itonlygivesme
therouter’sbestpath
PeeringRouter
P0
P1
P2
P3
P4
P3
AndVPdoesnotevenknowwhyP3waschosen
Creditsto:R.Bush(IIJ)@BMPBoF,RIPE74
BGPmonitoringwithBMP(1/2)
Creditsto:R.Bush(IIJ)@BMPBoF,RIPE74
All
Paths
VantagePoint
WithBMP,Ilearnallthepathsthepeeringrouterheard
PeeringRouter
P0-4
Peers
P0
P1
P2
P3
P4
BGPmonitoringwithBMP(2/2)
BMP:problemstatement
§ TheBGPprotocolisoneoftheveryfewprotocolsrunningontheInternetthathasastandardized,cleanandseparatemonitoringplane,BMP(think,forexample,toDNS..)
§ S?llBMP,initscurrentshape,doescoveronlypre-andpost-policiesAdj-RIB-In;anoperatorwouldprobablyneed:• ActualBGPpeering(s)forloc-RIB• Worse-case,screenscrapingforAdj-RIB-Out
BGPPeer-BAdj-Rib-In(Pre)
Adj-Rib-In(Post)
Filters/Policy Filters/Policy
Adj-Rib-In(Post)
BGPPeer-AAdj-Rib-In(Pre)Wecanseethis
Andthis
Problemstatementvisualized
Creditsto:T.Evens(Cisco),S.Bayraktar(Cisco),P.Lucente(NTT)@GROWWG,IETF98
Sta?cBGPPeer-BAdj-Rib-In(Pre)
Adj-Rib-In(Post)
Filters/Policy Filters/Policy
Accepted Accepted
Adj-Rib-In(Post)
Adj-Rib-Out(Pre)
Adj-Rib-Out(Post)
Filters/Policy
BGPPeer-AAdj-Rib-In(Pre) ISIS
Wealsowanttoseethis
Andthis
Andthis
Loc-Rib
Creditsto:T.Evens(Cisco),S.Bayraktar(Cisco),P.Lucente(NTT)@GROWWG,IETF98
Proposal:extendBMPtoloc-RIBandAdj-RIB-Out(1/3)
Proposal:extendBMPtoloc-RIBandAdj-RIB-Out(2/3)
Proposal:extendBMPtoloc-RIBandAdj-RIB-Out(3/3)
Creditsto:T.Evens(Cisco),S.Bayraktar(Cisco),P.Lucente(NTT)@GROWWG,IETF98
§ Loc-RIB:§ Monitorroutesselectedanduserbytherouter:
o ECMPo Correla?onwithNetFlow/IPFIXo Next-hoppreserva?on
§ MonitorlocallyoriginatedandBGProuteswithoutrequiringpeering
§ Policyverifica?on§ Adj-RIB-Out:
§ Monitorroutesadver?sedtopeers§ Policyverifica?on
draO-evens-grow-bmp-{local-rib,adj-rib-out}use-cases
Thanks!Ques?ons?
PaoloLucenteNTTCommunica?ons|pmacct
LatestsonBGPmonitoring
GRNOGWorkshop#1,Athens–May2017
