ESnet and ANI Testbed Update - indico.bnl.gov...Provides world's largest collection (32) of...

ESnet and ANI Testbed Update

Brian Tierney

DOE PI Meeting

Bethesda, MD

March 2, 2011

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Outline

1.  ESnet Overview 2.  ESnet R&D projects: OSCARS and perfSONAR 3.  ANI Testbeds

2


ESnet is a Mission Organization

3

DOE Office of Science mission: to deliver scientific discoveries and major scientific tools for transforming our understanding of nature, and to advance the energy security, economic security, and national security of the United States.

ESnet mission: to accelerate scientific discovery for the DOE Office of Science by delivering unparalleled network infrastructure, services, tools, and innovation.


ESnet Supports DOE Office of Science

SC provides broad support for Labs, Facilities, people

•  almost $5B/year in funding

•  45% of Federal support for physical sciences research

•  key funding for basic research in biology, computing, energy, climate

•  over 100 Nobel Prizes in past 60 years

Supporting > 27,000 PhDs, grad students, engineers at >300 institutions.

Provides world's largest collection (32) of scientific user facilities:

•  supercomputer centers, accelerators, light sources, neutron sources, electron microscopes, nano-scale centers, a sequencing center, fusion facilities.

ESnet is one of them – connecting DOE sites, facilities, scientists and collaborators.

•  optimized for science data transport

•  every service exists to support scientific discovery

4


ESnet Supports DOE Office of Science

8

UniversitiesDOE laboratories

The Office of Science supports:27,000 Ph.D.s, graduate students, undergraduates, engineers, and technicians26,000 users of open-access facilities300 leading academic institutions17 DOE laboratories

SC Supports Research at More than 300 Institutions Across the U.S.

5


ESnet is Embedded in a Global Research Networking Ecosystem

3/2/12 6


ESnet is Embedded in a Global Research Networking Ecosystem

Most important implication: requirement for multi-domain coordination.

•  Large-scale science is a multi-domain, end-to-end endeavor.

•  We must build services across networks, exchanges, campuses. •  Problem resolution, data mobility, measurement / monitoring,

guaranteed services, collaboration – all multi-domain!

•  We spend much time coordinating, communicating •  International: GLIF, DICE, LHCOPN, LHCONE, TERENA •  US: Joint Techs, ESCC, Supercomputing, many more

Our drivers are different from those of commercial providers.

3/2/12 7


ESnet4 Topology (n x 10G core)

3/2/12 8

ESnet4 NetworkDrawn by Mark Redman

Revised:Revised by Mark Redman

!"#$%&'(%#$%)*+%,-.%/0$#12$$%3$2%-1456%,*%7*8%92:#$8+#;0826%<*+%=*(#2$%=*18'=8%#1)*>2$6128

US R&EInternationalR&E Peering

CommercialCommercial PeeringInternational Peering

OC12-Link(622Mb)NLR 10 GE Circuit

Internet 210GE-SDN-Circuit

Internet210GE-IP-Circuit

1GE-LinkSDN 10GE-Link

IP

MAN SITE

MAN 10 GE RINGSITE OC48-Link(2.48Gb) OC3-Link(155Mb) FE-Link(100Mb) DS3-Link(45Mb) T1-Link(1.45Mb)Ether-Link(10Mb)R&E and International

10G Peering ConnectionsR&E and International1G Peering ConnectionsPerfSonar enabled

perfSONAR4/14/2011

CANADA(CANARIE)

CENIC/DREN/INTERNET2/

LOS NETTOS/NASA/NLR/UW

ASIA-PACIFIC(KAREN/KREONET2/NUS-GP/ODN/REANNZ/SINET/TANET2/

TRANSPAC2)

AUSTRALIA(AARnet)

CENIC/INTERNET2/LOS NETTOS/NASA

ASIA-PACIFIC(KAREN/KREONET2/NUS-GP/

ODN/REANNZ/SINET/TRANSPAC2)

AUSTRALIA(AARNET)

LATIN AMERICA(AMPATH/CLARA)

EUROPE(GEANT)

MAN LANINTERNET2 / NLR /

NYSERNET

ASIA-PACIFIC(BINP/HEPNET)

CERNUSLHCNet

FRANCE(OPEN TRANSIT)

CANADA(CANARIE)

RUSSIA & CHINA(GLORIAD)

ASIA-PACIFIC(ASGC/KREONET2/TANET2)

CICNET-OMNIPOP/DREN/INTERNET2/MREN/NISN/NLR/NWU/STARLIGHT/UCHICAGO/

UIOWA/UWMADISON/WISCNET

SOX

FRGP

ABQ-GIGAPOPOSOGRANDE

TECHNET

MERIT/ULTRALIGHT

CERNUSLHCNet

CHINA(GLORIAD)

MAXGIGAPOP

NASA / NISNUS-DOI

3ROX/PSC

CANADA(CANARIE)

NOX

ASIA-PACIFIC(SINET)

INDIANA-GIGAPOPUI-ICCN

DREN/INTERNET2/

NASA

SANFORD

ONENET

LATIN AMERICA(CLARA / CUDI)

LATIN AMERICA(CLARA / CUDI)

INTERNET2

SDSC

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

perfSONAR

SNV-MR2

SDN1KANS

ATLA-CR1

CLEV-CR1

SDN1CLEV

BOST-CR1

SDN2SDSC

AOFA-CR2

ELPA-CR1

SDN1PNWG

SDN1ALBU

SDN1DENV

SDN1ATLA

SDN2CHIC

GA-RT2

PNWG-CR1

SDN1HOUS

HOUS-CR1

STAR-CR1SDN1STAR

SDN2WASH

BOIS-CR1

SDN2SUNN

SDN1ELPA

SDN1AOFA

SDN1NEWY

ALBU-CR1

DENV-CR2

NASH-CR1 SDN1NASH

ORNL-RT2

LASV-RT1WISCNET

KANS-CR1

SUNN-CR1 CHIC-CR1

WASH-CR1

CERN

EUROPE(GEANT/

NORDUNET)

SLA

C

SNLLLLNL

LBNL

JGI

NERSC

ANL

FNAL

EM

ORO

BJC

SRELSRS

LANL

KIR

TLA

ND

-AFB

DO

E-A

BLQ

INL

Y12

AMES

PU-HEP

GFDL

PPPL

NREL

IARC

GA

PAIX-PA

LIGO

PNWG

PNNL

MIT

JLAB

PNNL

PANTEXSNLA

ORNL

FORR

DOE-GTN

NNSA

NSO

EQX-SJ

EQX-ASH

BNL

EQX-CHI

ORAUNOAA OSTI

PNNL

SNL-DC

LLNL & LANL-DC

KCP


ESnet5 Optical Infrastructure Footprint

3/2/12 9

!"#$

%&!#

'#%&

(&)!

"*+"

'",-

'&.$

!$%&

),-)

"/01

01)0

!$#.

1&*!

-#%%

$*%!

2#,!

!#,#

3#.#

+&&-

"%'#

#%(*

-",4

'1&"'),"

%#!4

"01&

!#%$

"*."

.",&

(&!$#%(#

(*55

')$$

0%"4

!&*$

6#02

#$%#

01#$

'",!(#$&1&*%

623!

3"3'

,#!1

%&*)

0),0

01#.

.#%"

7#!1

'1)%

,"78

,"7+

#!1(

#,%

5,#%

&.,%

!,%%

,".!0

%(,%

6+)

!%#0

"!9:;<=&>;?@AB=,:;CDEF

4:EG?D9=<HI

JKLKK=JMNMOIJO

KI=%APQRAG

SJ=%APQRAG

)9T?9:EA=C?;U=JII+=%?;=VWII+=PAXY

#RR=-ED>=,DR:=V0?:9AY

&.,%="X>E:GG=,DR:=V0?:9AY

(,%

WW=%APQRAG

%)3#,=,DR:=V)9T?9:EAY

!*,,

!#0

Figure 1.

Figure 2.


ESnet5 L2/3 Architecture (n x 100G core)

3/2/12 10

!"#$

%&!#

'#%&

(&)!

"*+"

'",-

'&.$

!$%&

),-)

"/01

01)0

!$#.

1&*!

-#%%

$*%!

2#,!

!#,#

3#.#

+&&-

"%'#

#%(*

-",4

'1&"'),"

%#!4

"01&

!#%$

"*."

.",&

(&!$#%(#

(*55

')$$

0%"4

!&*$

6#02

#$%#

01#$

'",!(#$&1&*%

623!

3"3'

,#!1

%&*)

0),0

01#.

.#%"

7#!1

'1)%

,"78

,"7+

#!1(

#,%

5,#%

&.,%

!,%%

,".!0

%(,%

6+)

!%#0

"!9:;<=&>;?@AB=,:;CDEF

4:EG?D9=<HI

JKLKK=JMNMOIJO

KI=%APQRAG

SJ=%APQRAG

)9T?9:EA=C?;U=JII+=%?;=VWII+=PAXY

#RR=-ED>=,DR:=V0?:9AY

&.,%="X>E:GG=,DR:=V0?:9AY

(,%

WW=%APQRAG

%)3#,=,DR:=V)9T?9:EAY

!*,,

!#0

Figure 1.

Figure 2.


12,924 miles of Long-Haul Dark Fiber, ESnet5+

3/2/12 11 Figure 3.


One Important Driver for Innovation

3/2/12 12

1990 1995 2000 2005 2010

Total Yearly Traffic

Year

Traf

fic (P

B)

0.001

0.01

0.1

1.0

10.0

100.0

Although ESnet recently celebrated its 25th anniversary, its roots go back to the early 1980s, when two independent DOE-funded networks (HEPnet and MFEnet) merged. Those networks, which each served a single science – High Energy Physics and Magnetic Fusion – were on separate and incompatible technological paths. ESnet, designed to be protocol-stack agnostic, met the needs of both communities with a common infrastructure. In time, as IP became the lingua franca of the Internet, other protocols faded. Appendix (#), the ESnet Technical History, provides in-depth information on these early years.

Since its inception, ESnet has delivered services required by DOE science, many of which are cost-prohibitive (or simply unavailable) on the commercial market. Today, such services include

engineering, IPv6 service and performance parity, robust inter-domain multicast, continuous active performance monitoring, and

provides full operational transparency providing the community with valuable real-time information on network topology and utilization.

than the commercial Internet core, which sees millions of small

may last for days (as when data sets are transferred from the Large Hadron Collider [LHC] to Brookhaven and Fermi National Labs in the US). The price advantage ESnet customers enjoy was recently

under the stimulus-funded ANI project. Once this infrastructure is

deployed, the cost of adding new capacity will be a simple function of procuring new optical interfaces.

ESnet Production Services

In all its activities, ESnet strives to be at the leading edge of

solutions that advance the state of the art in distributed science. Leveraging its requirements workshops, collaboration with ESnet site technical coordinators and its peers around the world, ESnet has developed a portfolio of production services to meet users’ current and near future needs. The model of a consistent service set for science collaborations, delivered by independent but cooperating organizations worldwide, is a critical feature of ESnet’s research ecosystem and for which all of its services are engineered.

ESnet services include:

Core Services

» IP services

» Virtual circuits services

» Performance measurement and monitoring

Community Services

» Visualization tools

» Audio, video and desktop collaboration services

» Consulting support in data transfer, network architecture, and performance measurement

Core Services

IP Services

Network connectivity is oxygen for data-intensive science. A high-performance, feature-rich network is the foundation for

facilities. Routed IP is the most basic of Internet services, yet the effective support of science requires not only basic connectivity, but connectivity that enables high-performance applications. ESnet’s routed service is engineered to provide the highest level of support for demanding science applications, in particular high-performance TCP-based data transfers that are intolerant of packet loss.

ESnet has been an early adopter of new technologies, both to

applications. Two examples of this are IP multicast and the next-generation Internet protocol, IPv6. The DOE science programs used IP multicast for its collaboratories research, and ESnet was a key enabler of that research through its support of robust interdomain

allocation in North America, and the ESnet core network has

Perc

ent G

row

th

InternetESnet

Figure 2. CAGRPB Accepted per Compound Annual Growth Rates for ESnet [measured] and Internet core [estimated and predicted, see "Power Trends in Communication Networks", IEEE Journal of Selected Topics in Quantum Electronics, VOL. 17, NO. 2].

ESnet ESnet vs Internet


ESnet is Engineered for the Data Explosion Flood Tsunami Revolution

3/2/12 13

Cost of Sequencing per Genome

Date

Dolla

rs

Sep

2001

Mar

200

2Se

p 20

02M

ar 2

003

Oct

200

3Ja

n 20

04Ap

r 200

4Ju

l 200

4O

ct 2

004

Jan

2005

Apr 2

005

Jul 2

005

Oct

200

5Ja

n 20

06Ap

r 200

6Ju

l 200

6O

ct 2

006

Jan

2007

Apr 2

007

Jul 2

007

Oct

200

7Ja

n 20

08Ap

r 200

8Ju

l 200

8O

ct 2

008

Jan

2009

Apr 2

009

Jul 2

009

Oct

200

9Ja

n 20

10Ap

r 201

0Ju

l 201

0O

ct 2

010

Jan

2011

Apr 2

011

Jul 2

011

10,000

100,000

1,000,000

10,000,000

100,000,000

source: http://www.genome.gov/sequencingcosts/

Crushing data volumes producing qualitative changes in: •  instrument design •  workflows •  collaborations •  mission statements •  cyberinfrastructure •  discovery processes


Elephant (or Alpha) Flows

3/2/12 14

Flow-Type Zoology, by Dimension: •  size: elephant and mouse •  rate: cheetah and snail •  duration: tortoise and dragonfly •  burstiness: porcupine and stingray

Kun-chan Lan and John Heidemann, “A Measurement Study of Correlations of Internet Flow Characteristics.” ACM Comput. Netw. 50, 1 (January 2006), 46-6.

Or: •  Alpha flows (> H byes in < T sec) •  caused by transfers of large files

over fast links

S. Sarvotham, R. Riedi, and R. Baraniuk, “Connection-Level Analysis and Modeling of Network Traffic,” ACM SIGCOMM Internet Measurement Workshop, November 2001, 99–104.


Science Traffic is Different

3/2/12 15

and has already enabled researchers to make progress in multiple domains.

A few efforts that showcase ESnet’s successful transition of RDI concepts into production include:

typically had large bandwidth-delay products, and were extremely

dynamic virtual circuit services, which were deployed to supplement best-effort IP. The underlying technologies developed by ESnet — the OSCARS protocol and code base — have been adopted by over a dozen networks around the world including Internet2f in the US; RNP in Brazil; and NORDUnet, which serves the Nordic region of Europe. OSCARS also interoperates with similar services

f In addition, Internet2’s NSF-funded DyGIR and DYNES projects will in 2012 deploy over 60 more instances of OSCARS at university campuses and regional networks in the United States.

provided by GÉANT, the pan-European R&E network.

The latest release of OSCARS 0.6 was developed with a modular architecture that provides a consistent platform for researchers to experiment with more complex services, path computation and scheduling algorithms thus greatly easing the transition of their research from algorithm to a deployed service.

Understanding the health of the network is key for R&E networks like ESnet. A seemingly minor network fault, such as a dirty

performance. Performance issues can be challenging, especially

numerous organizations and time zones must collaborate on the debugging process. Poor performance not only hinders science productivity, but can lead collaborations to abandon networks and rely on physical shipment of disks or other media.

To address these issues, ESnet and several R&E partners have worked for over a decade to develop and standardize perfSONAR, which serves as an infrastructure for network performance monitoring, making it easier to solve end-to-end performance problems on paths crossing several networks. It contains a set of services delivering performance measurements in a federated

Tra!

c (T

B)

Figure 6.


Science Traffic is Different

3/2/12 16

ESnet Accepted Traffic on January 28, 2012 (one day) http://www.es.net

http://www.ams-ix.net/statistics/


How do we Know what our Customers Need?

•  Each Program Office at Office of Science has a dedicated requirements workshop devoted to its needs

•  Two requirements workshops per year, with attendees chosen by science programs

•  Discussion centered on science case studies

•  Requirements derived from case studies + discussions

•  Reports contain requirements, case study text

3/2/12 17


ESnet R&D

18


ESnet Research Success: OSCARS

OSCARS: an application and protocol suite for dynamic, inter-domain circuit management

• notable ESnet innovation

• used for service guarantees, traffic engineering, overlays

• carries almost ~50% of ESnet traffic (alpha flows)

• a customer-facing service: web interface for people, API for apps

• deployed in 21+ networks

3/2/12 19


ESnet Services: Core

OSCARS: an application and protocol suite for dynamic, inter-domain circuit management

•  notable ESnet innovation •  used for service guarantees,

traffic engineering, overlays •  carries almost ~50% of

ESnet traffic (alpha flows) •  a customer-facing service:

web interface for people, API for apps

•  deployed in 21+ networks

2/1/12 40


Production services OSCARS

Research Motivation •  Network guarantees needed by service sensitive applications & large flows

Objective •  Provide secure end-to-end guaranteed circuits across ESnet & multiple domains.

Impact •  Carries 50% of ESnet’s 100+PB yearly data •  Deployed in about 21 networks •  Under evaluation by an additional 26 greenfield deployments

2003: Flow analysis and identification of large flows

2004: DOE funded

research project

2007: OSCARS production

service

2008: DICE collaboration

completes IDC protocol v1.0

2008: GLIF interops multiple implementations using FENIUS (ESnet driven)

2009: ARCHStone

research project funded

2011: Standards based NSI protocol defined in

OGF, successful interop with multiple

independent implementations


OSCARS Deployments Worldwide WAN, Regional, Campus, Testbeds

OSCARS

US: 13 Deployed 26 in progress

RNP

NORDUnet KOREN

JGN2


ESnet Research Success: perfSONAR

Distributed platform and protocol suite for performance measurement, monitoring, and visualization.

Assures end-to-end health of global networks.

400+ nodes deployed worldwide.

Developed in a global collaboration of R&E networking organizations.

Helps us find and fix problems!

3/2/12 22


Network tools perfSONAR Research Motivation

•  Certain performance problems only apparent to long-latency TCP flows, testing those requires source/sink in other domains

Objective •  Provide an infrastructure that identifies and isolates performance problems in a

multi-domain environment.

Impact •  Almost every ESnet site has fixed network performance problems •  LHC collaboration has deployed extensively •  400+ perfSONAR hosts deployed on 100 networks and in 14 countries

2000: Formation

of OGF group

2003: First OGF doc on

network measurements

2005: non-standard GE performance

testers

2006: 1st release of software. perfSONAR

system 10GE b/w and GE OWAMP

2007:perfSONAR schema published

2008: 1st release of

perfSONAR-PS. 20 nodes

deployed

2009-11: Deployment at all 10G ESnet sites.

perfSONAR release every 6

months


perfSONAR in Use

Effect of full routing table, followed by reboot.

Effect of gradual optical line card failure.

1/29/2012 24


Creating VO-Specific Dashboards from perfSONAR Data

1/29/2012

Joint Techs, Winter 2012 25


DOE’s Advanced Network Initiative (ANI) Testbed Project Update


Testbed Value to Network Researchers

A realistic national-scale environment for innovative network research that would be impossible for most researchers to create in their labs

•  Many types of network research require realistic high-latency environments

Maximum flexibility: researchers get “super-user” access to everything −  ability to install custom OS on hosts −  ability to do full routers/switch configuration −  ability to install custom router software

A controlled environment that supports reproducible results

Priority is given to research that aligns with ESnet strategic focus areas, such as:

High performance network protocols, data transfer middleware, Openflow + MPLS integration strategies, energy efficient networking

3/2/12 27


ESnet Research Testbeds

Fertile environment for researchers to test and prove concepts

Control Plane Testbed •  Long Island fiber, Infinera transport •  Juniper, NEC OpenFlow,

Data transfer nodes, VMs

100G Testbed •  High-speed protocol research •  Routed network from Argonne to

NERSC

Dark Fiber Testbed •  Disruptive network approaches •  Quantum-key encryption

ANI 100G Router

nersc-diskpt-2

nersc-diskpt-3

nersc-diskpt-1

nersc-C2940 switch

4x10GE (MM)

4x 10GE (MM)

Site Router(nersc-mr2)

anl-mempt-2

anl-mempt-1

anl-app

nersc-app

NERSC ANL

Updated December 11, 2011

ANI Middleware Testbed

ANL Site Router

4x10GE (MM)

4x10GE (MM)

100G100G

1GE

1 GE

1 GE

1 GE

1GE

1 GE

1 GE1 GE

10G

10G

To ESnet

ANI 100G Router

4x10GE (MM)

100G 100G

ANI 100G Network

anl-mempt-1 NICs:2: 2x10G Myricom


nersc-diskpt-1 NICs:2: 2x10G Myricom1: 4x10G HotLava

nersc-diskpt-2 NICs:1: 2x10G Myricom1: 2x10G Chelsio1: 6x10G HotLava

nersc-diskpt-3 NICs:1: 2x10G Myricom1: 2x10G Mellanox1: 6x10G HotLava

Note: ANI 100G routers and 100G wave available till summer 2012; Testbed resources after that subject funding availability.

nersc-asw1

anl-C2940 switch

1 GE

anl-asw1

1 GE

To ESnet

eth0

eth0

eth0

eth0

eth0

eth0

eth2-5

eth2-5

eth2-5

eth2-5

eth2-5

eth0

anl-mempt-3

4x10GE (MM)

eth2-5 eth0

1 GE

anl-mempt-3 NICs:1: 2x10G Myricom1: 2x10G Mellanox

4x10GE (MM)

10GE (MM)10GE (MM)

10GE (MM)

3x10GE (MM)

10GE (MM)

2x10GE (SM)[GT-TAM]

2x10GE (MM)

10GE (MM)

newy-tb-rt-1 (MX80)

newy-tb-of-1 (NEC switch)

bnl-tb-wdm-3newy-tb-wdm-1

bnl-tb-of-3 (NEC switch)

bnl-tb-rt-2 (MX80)

bnl-tb-wdm-4

3x10GE (MM)

1GE

1GE

bnl-diskpt-1 bnl-diskpt-2

newy-diskpt-1

bnl-app

1GE

1GE

1GE

2x10GE (SM)[GT-TAM]

newy-app

1GE

1GE

2x10GE (SM)

NEWY

BNL

AofA

100G Prototype Network

Updated July 8, 2011

2x10GE (SM)

bnl-mon

1GE

Long Island MAN (LIMAN) Testbed Architecture

2x10GE (SM)

10GE (MM)

2x10GE (MM)

1GE

1GE

(dashed line = planned)

ESnet Production Network

bnl-tb-of-2 (NEC switch)

10GE (MM)

!

Selma

Jackson

Houston

Dallas

Tulsa

Kansas CitySt. Louis

Peoria

Nashville

Louisville

Indianapolis

ChicagoSouth Bend

Cleveland

Buffalo

Albany

Cambridge

New York

Pittsburgh

Washington DCDenverGoodland

Albuquerque

El Paso

Los AngelesPhoenix

Echo Springs

Salt Lake CityReno

Sacramento

Sunnyvale

Eugene

Seattle

Boise

Raleigh

Philadelphia

Atlanta

Charlotte

Chattanooga

317 miles20

5 mile

s130 miles

152 miles 259 miles

264 miles

295 m

iles

179 miles

137 m

iles

228 miles255 miles

212

mile

s

150 miles 228 miles27

5 mile

s

212 m

iles

276

mile

s

116 miles

95 miles

560

mile

s

284 miles

618 miles

317 miles

422 miles

551 miles325 miles

500

mile

s

249 miles

863 miles

257

mile

s

278

mile

s

248 miles

172 miles

306 miles

275 miles

147 miles246 miles

198 miles 460 miles

336 miles

204 miles

532 miles

138 miles

LBNL Long Haul Dark Fiber Routes12,924 miles

BayExpres Metro Fibers: 432 milesChiExpress Metro Fibers: 167 milesNYExpress Metro Fibers: 6 miles

74 milesChepachet

Stamford61 miles

119 m

iles

Silver City

119 milesSeminary


Community impact so far: Peer-reviewed Testbed projects

•  25 projects accepted: -  17: testbed proposal review -  8: direct DOE/ASCR funding -  5 from Industry; 8 from DOE

labs; 3 from NASA; 1 from DISA; 7 from Universities

•  Strong diversity in research topics

•  Four papers accepted or submitted for publishing, more to follow

3/2/12 29


Control Plane Testbed

Control Plane Testbed

700 KM loop on Long Island Between Manhattan and Brookhaven National Lab

Infinera DWDM switches, NEC OpenFlow switches, Juniper router (soon with OpenFlow support), 6 hosts

Research Examples •  OpenFlow experiments •  Path Computation algorithms •  MPLS/OpenFlow Integration

3/2/12 30


Notes: - “App Host”: can be used for researcher application, control plane control software, etc. Can support up to 8 simultaneous VMs - “I/O Testers” are capable of 15 G disk-to-disk or 35G memory-to-memory - Other infrastructure not shown: VPN Server, file server (NFS, webdav, svn, etc.)

Control Plane Testbed, Long Island, NY

3/2/12 31


100G Testbed

100G Testbed

100Gbps path from Oakland, CA (NERSC) to Argonne, IL (ANL)

3 high-end hosts at each site, capable of >39 Gbps each

Research Examples •  100G data transfer tools −  FTP100, GridFTP, BestMan, Saratoga

•  TCP alternatives −  RDMA, RoCE

•  TCP enhancements −  PERT, RAPID TCP

3/2/12 32


Available as of Jan 3, 2012

3/2/12 33

ANI 100G Router

nersc-diskpt-2

nersc-diskpt-3

nersc-diskpt-1

nersc-C2940 switch

4x10GE (MM)

4x 10GE (MM)

Site Router(nersc-mr2)

anl-mempt-2

anl-mempt-1

anl-app

nersc-app

NERSC ANL

Updated March 1, 2012

ANI 100G Testbed

ANL Site Router

5x10GE (MM)

4x10GE (MM)

100G100G

1GE

1 GE

1 GE

1 GE

1GE

1 GE

1 GE1 GE

10G

10G

To ESnet

ANI 100G Router

4x10GE (MM)

100G 100G

ANI 100G Network





nersc-diskpt-3 NICs:2: 2x10G Myricom1: 2x10G Mellanox

Note: ANI 100G routers and 100G wave available till summer 2012; Testbed resources after that subject funding availability.

nersc-asw1

anl-C2940 switch

1 GE

anl-asw1

1 GE

To ESnet

eth0

eth0

eth0

eth0

eth0

eth0

eth2-5

eth2-5,20

eth2-5,20

eth2-5

eth2-5

eth0

anl-mempt-3

5x10GE (MM)

eth2-5 eth0

1 GE

anl-mempt-3 NICs:2: 2x10G Myricom1: 2x10G Mellanox

4x10GE (MM)

10GE (MM)


Related Resources

Magellan at NERSC

currently 15 nodes with 1 10GE, GPFS backend

soon: 8 nodes with 2x10GE, GPFS backend

Magellan at ANL

went away in January

ORNL test cluster

12 nodes, each with flash-based scratch disk

FNAL test cluster

in planning stage

3/2/12 34


Dark Fiber Testbed

Dark Fiber Testbed

Nationwide footprint (13,000 miles)

Researchers must pay to light the fiber, but no charge for access if results will be public

Research Examples •  Greentouch Energy experiments •  JPL quantum networking / quantum cryptography experiments

3/2/12 35


3/2/12 36


Dark Fiber Network

10/18/11 5

Selma

Jackson

Houston

Dallas

Tulsa

Kansas CitySt. Louis

Peoria

Nashville

Louisville

Indianapolis

ChicagoSouth Bend

Cleveland

Buffalo

Albany

Cambridge

New York

Pittsburgh

Washington DCDenverGoodland

Albuquerque

El Paso

Los AngelesPhoenix

Echo Springs

Salt Lake CityReno

Sacramento

Sunnyvale

Eugene

Seattle

Boise

Raleigh

Philadelphia

Atlanta

Charlotte

Chattanooga

317 miles20

5 mile

s130 miles

152 miles 259 miles

264 miles

295 m

iles

179 miles

137 m

iles

228 miles

255 miles

212

mile

s

150 miles 228 miles27

5 mile

s

212

mile

s27

6 m

iles

116 miles

95 miles

560

mile

s

284 miles

618 miles

317 miles

422 miles

551 miles325 miles

500

mile

s

249 miles

863 miles

257

mile

s

278

mile

s

248 miles

172 miles

306 miles

275 miles

147 miles246 miles

198 miles 460 miles

336 miles

204 miles

532 miles

138 miles

LBNL Long Haul Dark Fiber Routes12,924 miles

BayExpres Metro Fibers: 432 milesChiExpress Metro Fibers: 167 milesNYExpress Metro Fibers: 6 miles

74 milesChepachet

Stamford61 miles

119 m

iles

Silver City

119 milesSeminary


Current ANI Testbed Research

25 projects have been accepted to the testbed so far:

8 via direct DOE/ASCR funding

17 via testbed proposal review process

5 from Industry; 8 from DOE labs; 3 from NASA, 1 from DISA, 7 from Universities

Wide range of projects:

9: high-speed middleware

3: OpenFlow

4: other network control plane

1: 100Gbps end host hardware

1: network flow classification

http://www.es.net/RandD/advanced-networking-initiative/current-testbed-research/

3/2/12 37

1: quantum communications

2: Wide Area RDMA

2: TCP congestion control

1: security

1: energy efficiency


Potential Breakthroughs from Current Testbed Research

Unconditionally secure continuous variable quantum communications

Ability to scale TCP well beyond 10Gbps

Alternative transport protocols that scale better than TCP (e.g.: RDMA)

Ability to easily create end-to-end circuits

Ability to scale data transfer tools/middleware to 100Gbps and beyond

100Gbps host interface

For more information see:

http://www.es.net/RandD/advanced-networking-initiative/current-testbed-research/

3/2/12 38


Results to Date: Publications Submitted

After only 7 months of being operational, there were already several publications submitted based on ESnet’s Testbed results:

Energy Profiling of Network Elements for Rate Adaptation Technologies, M. Ricca, A. Francini, S. Fortune, T. Klein, submitted to the ACM/IEEE 3rd International Conference on Future Energy Systems (e-Energy 2012).

Traffic engineering in hybrid IP/optical circuit networks. Zhenzhen Yan, Chris Tracy, and Malathi Veeraraghavan, submitted to the IEEE 13th Conference on High Performance Switching and Routing (HPSR)

Middleware Support for RDMA-based Data Transfer in Cloud Computing, Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, and Thomas Robertazzi, submitted to IEEE International Parallel & Distributed Processing Symposium (IPDPS)

Identifying gaps in Grid middleware on fast networks with the Advanced Network Initiative, G. Garzolglio and Dykstra, accepted at International Conference on Computing in High Energy and Nuclear Physics (CHEP 2012)

3/2/12 39


Results to Date: Press Releases

Joint Press Release by Orange Silicon Valley, Bay Microsystems and ESnet on the first wide area 40G RDMA results (http://goo.gl/57gSJ)

•  Showed that RDMA moves data at up to 96 percent of the peak capacity of the network −  Much more efficient than TCP

•  Replicated at SC11 over a 10,000KM path (http://goo.gl/ZE0w8)

Greentouch press release on plans to use ESnet dark fiber testbed for energy experiments: http://goo.gl/OB6nD

3/2/12 40


Results to Date: Demonstrations

•  ESnet Demonstration of OSCARS/OpenFlow integration on the ANI testbed at the Open Networking Summit held at Stanford. −  http://goo.gl/4FFBg

•  ESnet Demonstration of 9.9Gbps RDMA transfers from the ANI Testbed at BNL to Seattle at SC11

3/2/12 41


Results to Date: Industry Involvement

•  Loaner hardware for 40GE testing −  Infinera, Alcatel-Lucent (ALU), Bay Microsystems, Mellanox

3/2/12 42


100G Testbed Performance

3/2/12 43

48.6ms RTT, 97.9Gbps aggregate TCP throughput* with 10 TCP streams


Per-Stream Results

3/2/12 44


Testbed Usage: Fully booked 24x7!

Feb 2012

March 2012

45


Testbed Access

Proposal process to gain access described at:

https://sites.google.com/a/lbl.gov/ani-testbed/ Testbed is available to anyone:

•  DOE researchers •  Other government agencies •  Industry

Must submit a short proposal to the testbed review committee •  Committee is made up of members from the R&E community and

industry Goal is to accept roughly five proposals every 6 month review cycle

•  Next round of proposals is due April 1, 2012

46


Future Testbed Challenges

Long term operational support of the Testbed •  Current testbed funding will run out in August of this year −  Currently funded at 2FTE for testbed support

•  This is essential for continued success

Hope to secure funding to ensure persistent/indefinite availability •  100G wave from NERSC to ANL will transition to production use

starting this summer, so need to identify funds to light a 2nd wave for testbed use

Dark Fiber will be available to the testbed until it is needed for production use

•  For some segments this may not be for several years.

3/2/12 47


More Information

http://sites.google.com/a/lbl.gov/ani-testbed/

email: [email protected]

[email protected]

ESnet and ANI Testbed Update - indico.bnl.gov...Provides world's largest collection (32) of...

Documents

Transcript of ESnet and ANI Testbed Update - indico.bnl.gov...Provides world's largest collection (32) of...