1

ESnet Network MeasurementCurrent Status

Joe Metzger

Jan 24th 2008ESCC meeting

Energy Sciences NetworkLawrence Berkeley National Laboratory

Networking for the Future of Science

2

Bandwidth Test Platforms• Deployment

– MANs: BNL, ANL, FNAL, LBL– Hubs: WASH, Starlight, ALBU, SUNN, AOFA, CHIC, ATLA– 20 more will be ordered, and most will be installed in FY08

• Current Uses– Acceptance Testing– Debugging network problems

• Additional uses once we resolve some problems and get more systems deployed– Regular scheduled bandwidth testing

• Internal to ESnet IP & SDN Network• External to important peers, and ESnet sites that are interested

– Dynamic end-user testing via• perfSONAR• ESnet Performance Center

3

Latency Test Platforms

• Deployment Status– 11 in field– ~20 more to be ordered & deployed

• Most are using CDMA clocks– Signal not available at all locations– Signal degrades over time as POPs are populated

• IRIG– Original plan to get IRIG sources from Level 3 at several

locations is not going smoothly.

4

ESnet Statistics Collection Systems• SNMP collection Systems

– Raw Collectors• Used for monthly statistics reports

– MRTG• Several different collections for:

– ESnet Info– NetInfo

– ESxSNMP & TSDB• ESxSNMP - SNMP collection system• TSDB Time Series Data Base• Developed in-house by Jon Dugan• Main focus is improved meta-data management• Open source (code.google.com)• Should replace other ESnet SNMP stats collections systems in the next 6

months or so.

• NetFlow collection system

5

perfSONAR Collaboration Status• perfSONAR MDM 3.0

– A bundle of all perfSONAR applications that complete the release management process

– One goal is to include all of the parts necessary to support the GEANT MDM pilot

– Substantial completion in February or March

• perfSONAR PS 0.06 Beta– A bundle of perfSONAR applications– Written in Perl, primarily in the US– Officially released Monday

• But I have been running it for months.

• LHCOPN MDM– A proposal by Dante to deploy perfSONAR appliances at all of the

LHC Tier1 sites.

6

ESnet perfSONAR near-term goals

• NetInfo– Convert to use perfSONAR data sources– Dynamically generate all maps and navigation– Expand to utilize perfSONAR data from other domains

• Work with the LHCOPN community to deploy perfSONAR measurement services at the Tier 1 and 2 centers

• Work to get the perfSONAR collaboration focused on dynamic hybrid network issues– publication, visualization, and analysis of dynamic circuits

7

What is perfSONAR• A collaboration

– Production network operators focused on designing and building tools that they will deploy and use on their networks to provide monitoring and diagnostic capabilites to themselves and their user comunities.

• An architecture & a set of protocols– Web Services Architecture– Protocols based on the Open Grid Forum Network Measurement

Working Group Schemas

• Several interoperable software implementations– Java & Perl

• A growing deployed measurement infrastructure

8

perfSONAR Collaborators

•GRNET•HEAnet•Internet2•ISTF•POZNAN•UNINETT•University of Delaware •Renater•RedIRIS•SLAC•SWITCH•SURFnet

•RNP•ARNES•BELNET•CARNET•CESNET•CYNET•DANTE•DFN•ESnet•FCCN•FERMI•GARR•GEANT

And anybody else I missed

9

Widely Deployed perfSONAR ServicesUtilization MA:• Internet2

• ISTF/ACAD

• CARNet

• Cesnet

• CYnet

• ESnet

• FERMI

• GARR

• GEANT2

• GRNET

• MREN

• PIONIER

• RedCLARA

• RENATER

• RNP

• SEEREN

• SWITCH

• Uninett             

Layer 2 Circuit Status for LHCOPN Circuits

• ASCC

• CERN

• ESnet

• GARR

• GEANT2

• NORDUnet

• Renater

• REDIris

• Sara/SURFnet

• SWITCH

• CANARIE

• USLHCnet

• UKERNA

10

perfSONAR Architecture• Interoperable network measurement middleware:

– Modular– Web services-based– Decentralized– Locally controlled

• Integrates:• Network measurement tools• Network measurement archives• Discovery• Authentication and authorization• Data manipulation• Resource protection• Topology

• Based on:• Open Grid Forum Network Measurement Working Group schema.

• Each Domain• Collects measurement data internal to their domain• May collect cross-domain measurement data• Controls publication of the information they collect and manage

11

perfSONAR codebases

• 2 Interoperable implementations– Java– Perl

12

perfSONAR Services (1)• Measurement Point Service

– Enables the initiation of performance tests– Currently in use: bandwidth, latency, looking glass, circuit status

• Measurement Archive Service– Stores and publishes performance monitoring results– Currently in use: interface utilization and errors, bandwidth, latency, circuit

status

• Lookup Service– Allows the client to discover the existing services and other LS services.– Dynamic: services registration themselves to the LS and mention their

capabilities, they can also leave or be removed if a service goes down.– Currently available: service discovery in a single domain. Global integration

getting close.

• Authentication Service– Based on Existing efforts: Internet2 MAT, GN2-JRA5– Authentication & Authorization functionality for the framework– Users can have several roles, the authorisation is done based on the user role.– Trust relationship between networks– Currently status: Some services work with Eduroam identities

13

perfSONAR Services (2)• Transformation Service

– Transform the data (aggregation, concatenation, correlation, translation, etc).

– Current status: perfSONAR_PS includes a transform service that determines circuit status based on info from several other services

• Topology Service– Publish the network topology information via the framework.– Currently status: perfSONAR_PS includes a useful server. CNIS

exports a perfSONAR topology interface

• Resource protector– Arbitrate the consumption of limited resources.– Current status: not in perfSONAR, but some capabilities in underlying

tools.

14

Weather Maps - Internet2

15

Gmaps from SLAC

16

CNM from DFN

17

CNM from DFN

18

perfSONARUI from acad.bg

19

PerfsonarUI 1

20

PerfsonarUI 2

21

PerfsonarUI 3

22

Oscars Circuit plugin - Internet2

23

Oscars circuit plugin

24

E2Emon - Monitoring Circuits

E2Emon: Status of E2E link CERN-LHCOPN-FNAL-001

E2Emon generated view of the data for one OPN link [E2EMON]

26

Traceroute Visualizer

• Forward direction bandwidth utilization on application path from LBNL to INFN-Frascati (Italy)– traffic shown as bars on those network device interfaces that have an associated MP

services (the first 4 graphs are normalized to 2000 Mb/s, the last to 500 Mb/s)

1 ir1000gw (131.243.2.1)2 er1kgw 3 lbl2-ge-lbnl.es.net

4 slacmr1-sdn-lblmr1.es.net (GRAPH OMITTED)5 snv2mr1-slacmr1.es.net (GRAPH OMITTED)6 snv2sdn1-snv2mr1.es.net

7 chislsdn1-oc192-snv2sdn1.es.net (GRAPH OMITTED)8 chiccr1-chislsdn1.es.net

9 aofacr1-chicsdn1.es.net (GRAPH OMITTED)

10 esnet.rt1.nyc.us.geant2.net (NO DATA)11 so-7-0-0.rt1.ams.nl.geant2.net (NO DATA)12 so-6-2-0.rt1.fra.de.geant2.net (NO DATA)13 so-6-2-0.rt1.gen.ch.geant2.net (NO DATA)14 so-2-0-0.rt1.mil.it.geant2.net (NO DATA)15 garr-gw.rt1.mil.it.geant2.net (NO DATA)16 rt1-mi1-rt-mi2.mi2.garr.net

17 rt-mi2-rt-rm2.rm2.garr.net (GRAPH OMITTED)18 rt-rm2-rc-fra.fra.garr.net (GRAPH OMITTED)19 rc-fra-ru-lnf.fra.garr.net (GRAPH OMITTED)

2021 www6.lnf.infn.it (193.206.84.223) 189.908 ms 189.596 ms 189.684 ms

link capacity is also provided

27

In Summary, perfSONAR is:

• A global collaboration

• An architecture and a set of protocols

• Several interoperable software implementations

• A measurement infrastructure that is deployed in over a dozen different networks