Post on 13-Jan-2016
description
1
ESnet UpdateWinter 2008
Joint Techs Workshop
Joe BurresciaESnet General Manager
January 21, 2008
Energy Sciences NetworkLawrence Berkeley National Laboratory
Networking for the Future of Science
2
TWC
SNLL
YUCCA MT
BECHTEL-NV
PNNLLIGO
INEEL
LANL
SNLAAlliedSignal
PANTEX
ARM
KCP
NOAA
OSTI ORAU
SRS
JLAB
PPPL
Lab DCOffices
MIT
ANL
BNL
FNALAMES
NR
EL
LLNL
GA
DOE-ALB
OSC GTNNNSA
International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less
NNSA Sponsored (12)Joint Sponsored (3)
Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)
42 end user sites
SINet (Japan)Russia (BINP)CA*net4
FranceGLORIAD (Russia, China)Korea (Kreonet2
Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren
ESnet IP core: Packet over
SONET Optical Ring and Hubs
ELP
DC
commercial peering points
MAE-E
PAIX-PAEquinix, etc.
PN
WG
Po
P/
PA
cifi
cWav
e
ESnet 3 with Sites and Peers (Early 2007)
ESnet core hubs IP
Abilene high-speed peering points with Internet2/Abilene
Abilene
Ab
ilen
e
CERN(USLHCnet
DOE+CERN funded)
GÉANT - France, Germany, Italy, UK, etc
NYC
Starlight
SNV
Ab
ilene
JGI
LBNL
SLACNERSC
SNV SDN
SDSC
Equinix
SNV
ALB
ORNL
CHI
MRENNetherlandsStarTapTaiwan (TANet2, ASCC)
NA
SA
Am
es
AU
AU
SEA
CH
I-SL M
AN
LA
NA
bile
ne
Specific R&E network peers
Other R&E peering points
UNM
MAXGPoP
AMPATH(S. America)
AMPATH(S. America)
ES
net
Scie
nce D
ata
Netw
ork
(S
DN
) core
R&Enetworks
Office Of Science Sponsored (22)
ATL
NSF/IRNCfunded
Equinix
3
ESnet 3 Backbone as of January 1, 2007
Sunnyvale
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)MAN rings (≥ 10 G/s)Lab supplied links
Seattle
San Diego Albuquerque
El Paso
Chicago New York
City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
4
ESnet 4 Backbone as of April 15, 2007
Clev.
Boston
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
5
ESnet 4 Backbone as of May 15, 2007
SNV
Clev.
Boston
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
6
ESnet 4 Backbone as of June 20, 2007
Clev.
Boston
Houston
Kansas City
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
7
ESnet 4 Backbone August 1, 2007 (Last JT meeting at FNAL)
Clev.
Boston
Houston
Los Angeles
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
8
ESnet 4 Backbone September 30, 2007
Clev.
Boston
Houston
Boise
Los Angeles
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
9
ESnet 4 Backbone December 2007
Clev.
Boston
Houston
Boise
Los Angeles
10 Gb/s SDN core (NLR)2.5 Gb/s IP Tail (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
New York City
Washington DC
Atlanta
Denver
Nashville
Future ESnet Hub
ESnet Hub
Chicago
10
DC
ESnet 4 Backbone December, 2008
Houston 10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Los Angeles
Nashville
Future ESnet Hub
ESnet Hub
X2
X2
X2
X2
X2
X2
X2
LVK
SNLL
YUCCA MT
BECHTEL-NV
PNNLLIGO
LANL
SNLAAlliedSignal
PANTEX
ARM
KCP
NOAA
OSTI
ORAU
SRS
JLAB
PPPL
Lab DCOffices
MIT/PSFC
BNL
AMES
NREL
LLNL
GA
DOE-ALB
DOE GTNNNSA
NNSA Sponsored (13+)Joint Sponsored (3)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)
~45 end user sites
SINet (Japan)Russia (BINP)CA*net4
FranceGLORIAD (Russia, China)Korea (Kreonet2
Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren
ELPA
WA
SH
commercial peering points
PAIX-PAEquinix, etc.
ESnet Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators (12/2007)
ESnet core hubs
CERN(USLHCnet:
DOE+CERN funded)
GÉANT - France, Germany, Italy, UK, etc
NEWY
SUNN
Ab
ilen
e
JGI
LBNL
SLACNERSC
SNV1
SDSC
Equinix
ALBU
OR
NL
CHIC
MRENStarTapTaiwan (TANet2, ASCC)
NA
SA
Am
es
AU
AU
SEA
CH
I-SL
Specific R&E network peers
UNM
MAXGPoPNLR
AMPATH(S. America)
AMPATH(S. America)
R&Enetworks
Office Of Science Sponsored (22)
ATLA
NSF/IRNCfunded
IARC
Pac
Wav
e
KAREN/REANNZODN Japan Telecom AmericaNLR-PacketnetAbilene/I2
KAREN / REANNZInternet2SINGARENODN Japan Telecom America
NETL
ANLFNAL
Starlight
USLH
CN
et
NLR
International (1-10 Gb/s)10 Gb/s SDN core (I2, NLR)10Gb/s IP coreMAN rings (≥ 10 Gb/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less
Salt Lake
PacWave
Ab
ilen
e
Equinix DENV
DOE
SUNN
NASH
Geography isonly representational
Inte
rnet
2N
YS
ER
Net
MA
N L
AN
Other R&E peering points
US
HL
CN
et
to G
ÉA
NT
INL
12ESnet4Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),
500-600 Gbps by 2011-2012 (100 Gb/s circuits)
Cle
vela
nd
Europe(GEANT)
Asia-Pacific
New York
Chicago
Washington DC
Atl
anta
CERN (30+ Gbps)
Seattle
Albuquerque
Au
str
ali
a
San Diego
LA
Denver
South America(AMPATH)
South America(AMPATH)
Canada(CANARIE)
CERN (30+ Gbps)Canada(CANARIE)
Asi
a-Pac
ific
Asia Pacific
GLORIAD (Russia and
China)
Boise
HoustonJacksonville
Tulsa
Boston
Science Data Network Core
IP Core
Kansa
s
City
Au
str
ali
a
Core network fiber path is~ 14,000 miles / 24,000 km
162
5 m
iles
/ 2
545
km
2700 miles / 4300 km
Sunnyvale
Production IP core (10Gbps)
SDN core (20-30-40-50 Gbps)
MANs (20-60 Gbps) or backbone loops for site access
International connections
IP core hubs
Primary DOE LabsSDN hubs
High speed cross-connectswith Ineternet2/AbilenePossible hubs
USLHCNet
13A Tail of Two ESnet4 Hubs
MX960 Switch
T320 Router
6509 Switch
T320 Routers
Sunnyvale Ca Hub Chicago HubESnet’s SDN backbone is implemented with Layer2 switches; Cisco 6509s and Juniper MX960s each present their own unique challenges.
14ESnet 4 Factoids as of January 21, 2008
• ESnet4 installation to date:o 32 new 10Gb/s backbone circuits
- Over 3 times the number from last JT meeting
o 20,284 10Gb/s backbone Route Miles - More than doubled from last JT meeting
o 10 new hubs- Since last meeting
– Seattle– Sunnyvale– Nashville
o 7 new routers 4 new switcheso Chicago MAN now connected to Level3 POP
- 2 x 10GE to ANL
- 2 x 10GE to FNAL
- 3 x 10GE to Starlight
15ESnet Traffic Continues to Exceed 2 PetaBytes/Month
Bytes Accepted
0.00E+00
5.00E+14
1.00E+15
1.50E+15
2.00E+15
2.50E+15
3.00E+15
Jan, 00Apr, 00Jul, 00Oct, 00Jan, 01Apr, 01Jul, 01Oct, 01Jan, 02Apr, 02Jul, 02Oct, 02Jan, 03Apr, 03Jul, 03Oct, 03Jan, 04Apr, 04Jul, 04Oct, 04Jan, 05Apr, 05Jul, 05Oct, 05Jan, 06Apr, 06Jul, 06Oct, 06Jan, 07Apr, 07Jul, 07Oct, 07
2.7 PBytes in July 2007
1 PBytes in April 2006
ESnet traffic historically has increased 10x every 47 months
Overall traffic tracks the very large
science use of the network
When A Few Large Data Sources/Sinks Dominate Trafficit is Not Surprising that Overall Network Usage Follows the
Patterns of the Very Large Users - This Trend Will Reverse in the Next Few Weeks as the Next Round of LHC Data Challenges Kicks Off
17ESnet Continues to be Highly Reliable; Even During the Transition
ESnet Availability 2/2007 through 1/2008
0
200
400
600
800
1000
1200
1400
1600
1800
ANL 100.000ORNL 100.000SLAC 100.000
FNAL 99.999LIGO 99.998LLNL 99.998NERSC 99.998
PNNL 99.998
DOE-GTN 99.997
SNLL 99.997LBL 99.996
MSRI 99.994
LLNL-DC 99.991NSTEC 99.991
LANL-DC 99.990
JGI 99.988IARC 99.985PPPL 99.985JLab 99.984
DOE-ALB 99.973
LANL 99.972SNLA 99.971Pantex 99.967
BNL 99.966NREL 99.965
MIT 99.947
DOE-NNSA 99.917
Yucca 99.917
GA 99.916INL 99.909
Bechtel 99.885
KCP 99.871Y12 99.863BJC 99.862ORAU 99.857
Ames-Lab 99.852
OSTI 99.851NOAA 99.756Lamont 99.754
SRS 99.704
Outage Minutes
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
“5 nines” (>99.995%) “3 nines” (>99.5%)“4 nines” (>99.95%)
Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.
18OSCARS Overview
Path Computation• Topology
• Reachability• Contraints
Scheduling• AAA
• Availability
Provisioning• Signalling• Security
• Resiliency/Redundancy
OSCARSGuaranteedBandwidth
Virtual Circuit Services
On-demand Secure Circuits and Advance Reservation System
19OSCARS Status Update
• ESnet Centric Deploymento Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet
(1Q05)o Prototype layer 2 (Ethernet VLAN) virtual circuit service deployed in ESnet (3Q07)
• Inter-Domain Collaborative Effortso Terapaths (BNL)
- Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06)- Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)
o LambdaStation (FNAL)- Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)
o HOPI/DRAGON- Inter-domain exchange of control messages demonstrated (1Q07)- Integration of OSCARS and DRAGON has been successful (1Q07)
o DICE- First draft of topology exchange schema has been formalized (in collaboration with NMWG)
(2Q07), interoperability test demonstrated 3Q07- Initial implementation of reservation and signaling messages demonstrated at SC07 (4Q07)
o UVA- Integration of Token based authorization in OSCARS under testing
o Nortel- Topology exchange demonstrated successfully 3Q07 - Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)
20
• ESneto About 1/3 of the 10GE bandwidth test platforms & 1/2 of the
latency test platforms for ESnet 4 have been deployed.- 10GE test systems are being used extensively for acceptance
testing and debugging- Structured & ad-hoc external testing capabilities have not been
enabled yet.- Clocking issues at a couple POPS are not resolved.
o Work is progressing on revamping the ESnet statistics collection, management & publication systems
- ESxSNMP & TSDB & PerfSONAR Measurement Archive (MA)- PerfSONAR TS & OSCARS Topology DB- NetInfo being restructured to be PerfSONAR based
• LHC and PerfSONARo PerfSONAR based network measurement solutions for the
Tier 1/Tier 2 community are nearing completion.o A proposal from DANTE to deploy a perfSONAR based
network measurement service across the LHCOPN at all Tier1 sites is being evaluated by the Tier 1 centers
Network Measurement Update
21
End