PPT

1

ESnet4IP Network and Science Data Network

Configuration and Roll OutProgress and Projected Schedule as of May 28, 2007

For more information contactWilliam E. Johnston ([email protected]), ESnet Dept. Head,

Joe Burrescia ([email protected]), ESnet General Manager, orJim Gagliardi ([email protected]), ESnet Operations and Deployment Manager

04/08/23 00:25

This talk is available at www.es.net/ESnet4

2ESnet4

• Internet2 has partnered with Level 3 Communications Co. and Infinera Corp. for a dedicated optical fiber infrastructure with a national footprint and a rich topology - the “Internet2 Network”

o The fiber will be provisioned with Infinera Dense Wave Division Multiplexing equipment that uses an advanced, integrated optical-electrical design

o Level 3 will maintain the fiber and the DWDM equipment

o The DWDM equipment will initially be provisioned to provide10 optical circuits (lambdas - s) across the entire fiber footprint (80 s is max.)

• ESnet has partnered with Internet2 to:o Share the optical infrastructure

o Develop new circuit-oriented network services

o Explore mechanisms that could be used for the ESnet Network Operations Center (NOC) and the Internet2/Indiana University NOC to back each other up for disaster recovery purposes

3ESnet4

• ESnet is building its next generation IP network and its new circuit-oriented Science Data Network primarily on the Internet2 circuits (s) that are dedicated to ESnet, together with a few National Lambda Rail and other circuitso ESnet will provision and operate its own routing and

switching hardware that is installed in various commercial telecom hubs around the country, as it has done for the past 20 years

o ESnet’s peering relationships with the commercial Internet, various US research and education networks, and numerous international networks will continue and evolve as they have for the past 20 years

4ESnet4

• ESnet4 will also involve an expansion of the multi-10Gb/s Metropolitan Area Rings in the San Francisco Bay Area, Chicago, Long Island, Newport News (VA/Washington, DC area), and Atlantao provide multiple, independent connections for ESnet sites

to the ESnet core network

o expandable

• Several 10Gb/s links provided by the Labs that will be used to establish multiple, independent connections to the ESnet coreo currently PNNL and ORNL

5

DenverFront Range GP

1850 PearlLevel 3 Kansas City

GPN1100 WalnutLevel 3

Seattle1000 Denny WayLevel 3

Portland1335 NW NorthropLevel 3

SunnyvaleCENIC

1380 KiferLevel 3

Pacific Northwest GP2001 6th AveWestin Bldg

San Diego8929 Aero DriveKearny Mesa, CA

PortlandOregon GP

707 SW WashingtonQwest

San Francisco

Oakland

Sacramento

Los AngelesCENIC

600 W 7thEquinix

Santa Barbara Los Angeles818 W 7th

Level 3

Salt LakeInter-Mountain GP

572 S DeLongLevel 3

Albuquerque

New Mexico GP

104 Gold Ave SELevel 3

TulsaOneNet

18 W ArcherLevel 3

HoustonLEARN1201 N I-45 Level 3

Dallas

San AntonioBaton RougeLONI9987 BurbankLevel 3

New Orleans

Chicago600 W Chicago

Level 3 MC

Detroit

ChicagoCIC/MREN

MERITBOREASInternet2

710 N LakeshoreStarlight

Atlanta180 Peachtree St NELevel 3 MC

Birmingham

CharlotteRaleighNCREN5301 Departure DrLevel 3

PittsburghPittsburgh GP143 S 25thLevel 3 Washington

MAX1755 Old Meadow LaneMcLean, VALevel 3

PhiladelphiaMAGPI401 N BroadLevel 3

New YorkNYSERNET32 Ave of the Americas

New York111 8th

Level 3

CambridgeNOX300 Bent StLevel 3

Tampa

MiamiSouth Florida GP

45 NW 5thLevel 3

Orlando

JacksonvilleFLR4814 Phillips HwyLevel 3

Indianapolis1902 S East StLevel 3

Louisville848 S 8th StLevel 3

El Paso501 W Overland

Level 3

NashvilleTennessee GP2990 Sidco Dr

Level 3

ClevelandTFN4000 ChesterLevel 3

Albany316 N PearlLevel 3

Ogden

Rieth

Boise

435 W M

cGregor D

rive

Tionesta

Rawlins Omaha

St. Louis

EurekaReno

Valentine

Sanderson

Raton

Phoenix

San Luis Obispo

Mobile

Tallahassee

SyracuseRochester

Cincinnati

Austin

Tucson

Buffalo

Rancho De La Fe(tentative)

Edison

Atlanta, SLR345 Courtland

Infinera equipment sites

regen site

Internet2 Core Node

Internet2 networkDWDM node (common)

DWDM node(ESnet only)

other Level 3 node

Level 3 Fiber

extension from Internet2 network to a RON/Core

connector node

Internet2 / Level3 / Infinera Optical Infrastructure

6ESnet IP + SDN 2007 Configuration and Roll OutJan 1, 2007

Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. First complement of circuits and PoP co-location become available2. NLR CHIC-WASH link operational

All circuits are 10Gb/s.

Existing site supplied circuit

Existing NLR

circuits

Existing site supplied circuit

(0)

(1)

ESnet IP network (Internet2 circuits)ESnet Science Data Network (Internet2)

ESnet SDN (NLR circuits)Lab supplied linkLHC related linkMAN linkInternational IP ConnectionsStatus indefinite / not installedInternet2 circuit number(20)

Layer 1 optical nodes at eventual ESnet Points of Presence

ESnet IP router hubs

Layer 1 optical nodes not currently in ESnet plans

Lab site

ESnet IP internal switch hubs

ESnet SDN OSCARS/MPLS switch hubs

ESnet SDN internal switch hubs

Corrected Map

7ESnet IP + SDN 2007 Configuration and Roll OutTarget: Apr. 15, 2007

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Washington, DC, Jan. 31, 2006. COMPLETED, Feb. 27, 2007

2. ESnet equipment installed in NYC, Jan. 15, 2007 COMPLETED, Jan. 93. WDC-NYC circuits tested and accepted, Feb. 15, 2007. COMPLETED, Mar. 154. ESnet equipment installed in Chicago, Feb. 28, 2007 COMPLETED5. NYC-BOS-CLE-CHI circuits tested and accepted, April 15, 2007 COMPLETED, Mar. 156. WDC-CLE-CHI circuits tested and accepted, April 15, 2007. COMPLETED, Apr. 18


(0)

(1)

Ch

icag

o

StarLight

LA

San DiegoUCSD

Corrected Map






Lab site




8ESnet IP + SDN 2007 Configuration and Roll Out Target: May 15, 2007

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Atlanta, Apr. 30, 2007 COMPLETED, Apr. 232. Atlanta – Wash. circuit tested and accepted, May 15, 2007. COMPLETED, May 163. Build ATL MAN, May 1. COMPLETED4. Install ORNL, OC48 backup circuit, May 1. Delayed by Qwest install issues


OC48

(0)

(1)

Ch

icag

o

StarLight

LA

San DiegoUCSD






Lab site




Corrected Map

9ESnet IP + SDN 2007 Configuration and Roll Out Target: Jun. 15, 2007

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Albuquerque, May 31, 2007; revised est. June 4.2. ESnet equipment installed in Denver, May 31, 2007; revised est. June 11.3. ATL-ALB circuits tested and accepted, June 15, 2007.4. ALB-DEN-CHI circuits tested and accepted, June 15, 2007.

All circuits are 10Gb/s, unless noted.

OC48

(0)

(1)

Ch

icag

o

StarLight

LA

San DiegoUCSD






Lab site




Corrected Map

10ESnet IP + SDN 2007 Configuration and Roll Out Target: July 15, 2007; revised target: August 1

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Sunnyvale, CA, June, 30, 2007; revised est. July 25.2. ALB-ELP-SD-SNV circuit tested and accepted, July 15, 2007; revised est. August 1.3. SNV-SLC-DEN circuit tested and accepted; revised est. August 1.

OC48

(0)

(1)

Ch

icag

o

StarLight

LA

San DiegoUCSD






Lab site




Corrected Map

11ESnet IP + SDN Configuration and Roll Out Target: August 15, 2007

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville(1(3))

(1(12))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(30)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Seattle, July, 31, 20072. DEN-BOI-SEA-SNV circuits tested and accepted, August 15, 2007.

OC48

(0)

(1)

Ch

icag

o

StarLight

LA

San DiegoUCSD






Lab site




Corrected Map

12ESnet IP + SDN Configuration and Roll Out Target: Sept. 15, 2007 (final configuration for 2007)

Denver

Seattle

Su

nn

yv

ale

Chicago

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Nashville(1(3))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

1. ESnet equipment installed in Nashville, Aug. 31, 20072. ATL-NAS-CHI circuits tested and accepted, Sept. 15, 2007.

Atlanta

OC48

(0)

(1)

(30)




Lab site




LA

San DiegoUCSD

Corrected Map



13


ESnet IP switch only hubs

ESnet IP switch/router hubs

ESnet SDN switch hubs


Lab site

ESnet IP coreESnet Science Data Network coreESnet SDN core, NLR links (existing)Lab supplied linkLHC related linkMAN linkInternational IP Connections

ESnet4 Metro Area Rings, 2007 Configurations

Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitts.

NYC

Boston

Philadelphia

Indianapolis

Atlanta

Nashville


MAX

West Chicago MAN Long Island MAN

Newport News - Elite

San FranciscoBay Area MAN

LBNL

SLAC

JGI

LLNL

SNLL

NERSC

JLab

ELITE

ODU

MATP

Wash., DC

FNAL

600 W. Chicago

Starlight

ANL

USLHCNet

OC48(1(3))

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

(0)

(1)

(30)

BNL

32 AoA, NYC

USLHCNet

Uncorrected Map

14ESnet4 IP + SDN, 2008 Configuration (Estimated)

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DC

Salt Lake City

Portland

BatonRougeHouston

Pitts. NYC

Boston

Philadelphia

Indianapolis

(? )

Atlanta

NashvilleOC48

(0)

(1)

1. Contract for 2008-2011 completed by mid-FY2007.2. Last three IP circuits (Kansas City-Houston, Chicago-Nashville-Atlanta, and Cleveland-Washington) and associated

ESnet equipment, are installed, tested, and accepted.3. First SDN ring is completed (Sunnyvale-Atlanta) and associated ESnet equipment is installed, tested, and accepted.4. Second SDN northern tier circuits installed, tested, and accepted.

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(3)

(21)

(27)

(14)

(24)

(15)

(30)

1

2

1

2

2

1

2

2

2

1

1

21

1

1

2

1

2

2

1

22




Lab site



ESnet SDN internal switch hubsC

hic

ago

StarLight

LA

San DiegoUCSD

Corrected Map



15






Lab site

ESnet4 Metro Area Rings, 2008 Configurations (Est.)

Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash DC

Salt Lake City

Portland

BatonRougeHouston

Pitt

s.

NYC

Boston

Philadelphia

(1 )

Indianapolis

(? )

Atlanta

Nashville

(2)

(13)

(21)

All circuits are 10Gb/s, or multiples thereof, as indicated. (E.g. 2 = 20 Gb/s., etc.)

ESnet IP core (1 )ESnet Science Data Network coreESnet SDN core, NLR linksLab supplied linkLHC related linkMAN linkInternational IP Connections


LBNL

SLAC

JGI

LLNL

SNLL

NERSC

West Chicago MAN

FNAL

600 W. Chicago

Starlight

ANL

USLHCNet

Long Island MAN

BNL

111-8th

32 AoA, NYC

USLHCNet


JLab

ELITE

ODU

MATP

Wash., DC

OC48(3)

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(21)

(27)

(14)

(24)

(15)

(1)

(30)

Uncorrected Map

16ESnet4 IP + SDN, 2009 Configuration (Est.)

Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DC

Salt Lake City

Portland

BatonRougeHouston

Pitts. NYC

Boston

Philadelphia

Indianapolis

(? )

Atlanta

NashvilleOC48

(0)

(1)(3)

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(21)

(27)

(14)

(24)

(15)

(30)

LA

San DiegoUCSD

Ch

icag

o

StarLight

2

3

2

3

3

2

3

3

3

2

2

32

2

2

3

1

2

2

2

33

Corrected Map




Lab site







Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DC

Salt Lake City

Portland

BatonRougeHouston

Pitt

s. NYC

Boston

Philadelphia

Indianapolis

(? )

Atlanta

Nashville






Lab site

OC48

(0)

(1)

ESnet IP coreESnet Science Data Network coreESnet SDN core, NLR links (existing)Lab supplied linkLHC related linkMAN linkInternational IP ConnectionsInternet2 circuit number(20)

3

2

2

2

2 2

2

3

3

3

1

22

3

3

3

3

32

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(27)

(14)

(24)

(15)

(30)

West Chicago MAN Long Island MAN



LBNL

SLAC

JGI

LLNL

SNLL

NERSC

FNAL

600 W. Chicago

Starlight

ANL

USLHCNet

BNL

111-8th

32 AoA, NYC

USLHCNet

JLab

ELITE

ODU

MATP

Wash., DC

2

2(3)

(21)

Uncorrected Map


Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DC

Salt Lake City

Portland

BatonRougeHouston

Pitt

s. NYC

Boston

Philadelphia

Indianapolis

(>1 )

Atlanta

NashvilleOC48

(0)

(1)

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(27)

(14)

(24)

(15)

(30)

(3)

(21)

3

4

3

4

4

3

4

4

4

3

3

43

3

3

4

2

3

3

3

44

LA

San DiegoUCSD

Ch

icag

o

StarLight




Lab site




Corrected Map




Denver

Seattle

Su

nn

yv

ale

Raleigh

Jacksonville

KC

El Paso

Albuq.Tulsa

Clev.

Boise

Wash. DC

Salt Lake City

Portland

BatonRougeHouston

Pitt

s. NYC

Boston

Philadelphia

Indianapolis

(>1 )

Atlanta

NashvilleOC48

(0)

(1)

(7)

(17)

(19) (20)

(22)(23)

(29)

(28)

(8)

(16)

(32)

(2)

(4)

(5)

(6)

(9)

(11)

(13) (25)

(26)

(10)

(12)

(27)

(14)

(24)

(15)

(30)

(3)

(21)

4

5

4

5

5

4

5

5

5

4

4

54

4

4

5

2

3

3

4

55

LA

San DiegoUCSD

Ch

icag

o

StarLight




Lab site




Corrected Map



20ESnet4Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe(GEANT)

Asia-Pacific

New York

Chicago

Washington DC

Atl

anta

CERN (30+ Gbps)

Seattle

Albuquerque

Au

str

ali

a

San Diego

LA

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN (30+ Gbps)Canada(CANARIE)

Europe(GEANT)

Asi

a-Pac

ific

Asia Pacific

GLORIAD (Russia and

China)

Boise

HoustonJacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s

City

Au

str

ali

a

Core network fiber path is~ 14,000 miles / 24,000 km

162

5 m

iles

/ 2

545

km

2700 miles / 4300 km

Sunnyvale

Production IP core (10Gbps)

SDN core (20-30-40-50 Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

IP core hubs

Primary DOE LabsSDN hubs

High speed cross-connectswith Ineternet2/AbilenePossible hubs

USLHCNet

PPT

Documents

Transcript of PPT