OptIPuter Infostructure: East of I-5*

Tom DeFanti, Maxine Brown, Jason Leigh, Oliver Yu, Tom Moher, Bob Grossman, Joe Mambretti, Valerie Taylor, Cees de Laat

“East of I-5” OptIPuter

• “East of I-5” OptIPuter is focused on large-data applications using experimental technology

• Experimental means that – It is obtainable and affordable– It works 99% of the time– It is programmable– Brute force is acceptable

• Our goal: to make photonics controllable by Grid middleware as soon as possible

What is a Lambda?

• A lambda, in networking, is a fully dedicated wavelength of light in an optical network, typically used today for 1-10Gbps.

• We are now working with 1Gb dedicated layer2 circuits that act like lambdas

• We need enough to schedule and manipulate: up to 40 1Gb “sub” lambdas will be available to the OptIPuter locally, regionally, internationally

• We expect 10Gb lambdas to be available to the OptIPuter in a few years; first locally, then regionally, then (inter)nationally

Gross Optical Burst Switching (GOBS)

• Move a terabyte or petabyte on a schedule• Roam a 375600 x 375600 pixel remote database• Applications will be able to request dedicated

Lambdas using the routed infrastructure• Bypass the routers in between

PC

PC

PC

PC

PC

PC

Photonic Switch(Glimmerglass, Calient)

Cluster Visualization

5x3 Grid of 1280x1024 Pixel LCD Panels Driven by 16-PC Cluster Resolution=6400x3072 Pixels, or ~3000x1500 pixels in Autostereo

NTT 4Kx2K Compressed Video from Chicago to Los Angeles

• Pre-compressed to 300 Mbps in Chicago using an experimental JPEG 2000 SHD codec

• Received in LA at USC Zemeckis Center by an NTT real-time decoder, and fed to NTT's prototype SHD frame-buffer and 8-megapixel full-color D-ILA projector for display on a large screen.

• SHD= 4xHDTV or 16xDVD

http://www.ntt.co.jp/news/news02e/0211/021113.html

• Lambdas for high bandwidth applications:– Bypass production network– Middleware request for

optical pipes• Rationale:

– Lower the cost of transport per packet

Application

Middleware

Transport

Application

Middleware

Transport

Router

Router

UvA

Router

Router

CA*net4

Router

ams

chi

SURFnet5

UBC Vancouver

Switch

GbE

GbE

GbE

Lambda

Switch

Lambda

Switch

Switch

Router

High bandwidth application

Scale 2-20-200

EVLEVL

OMNInet Chicago 2x10GE LambdaGrid Scale: 2ms

• A four-site network in Chicago -- a 10GE service trial• A test bed for all-optical switching and advanced middleware• Partners: SBC, Nortel, iCAIR at Northwestern, EVL, CANARIE, ANL

2x10GE

Northwestern U

OpticalSwitchingPlatform

Passport8600

ApplicationCluster

ApplicationCluster

OpticalSwitchingPlatform

Passport8600

2x10GE

StarLight

OPTera Metro5200

ApplicationCluster

OpticalSwitchingPlatform

Passport8600

2x10GE8x1GE

UIC

Carrier Hotel

OpticalSwitchingPlatform

Passport8600

2x10GE8x1GE

8x1GE

8x1GELoop back

UIUC/NCSA

Starlight(NU-Chicago)Argonne

UChicagoIIT

UIC

Illinois Century NetworkJames R. Thompson CtrCity HallState of IL Bldg

4 pair

12 pair

4 pair

2 pair 2 pair

4 pair

18 pair

4 10 pair

12 pair

2 pair

Level(3)111 N. Canal

McLeodUSA151/155 N. MichiganDoral Plaza

Qwest455 N. Cityfront

UC Gleacher450 N. Cityfront

Illinois’ I-WIRE

TransPAC/APAN

Euro-LinkSurfNET

Euro-LinkCERN

GÉANT

CA*net4

Abilene

Euro-Link Funded 10Gb LinkEuro-Link Co-Funded 10Gb LinkTransPAC Co-Funded 10Gb LinkPartner Matching 10Gb LinkGÉANT Matching 2.5Gb Links

Euro-Link/Trans-Light andTransPAC/Trans-Light SupplementFunded, Co-Funded and Matching Links

I-5/PLR

• A Production Network 1GigE and 10GigE exchange• An Experimental Network lambda exchange• A Research Network 1GigE and 10GigE MEMS-switched

exchange• Host to DTFnet, the TeraGrid’s 4x10Gb T640-based

Experimental Network, perhaps for future collaborations• A co-location space with 66 racks for networking and

computing and data-management equipment • An OIX with fiber and/or circuits from SBC/Ameritech,

Qwest, AT&T, Global Crossing, Looking Glass Networks, Level 3, RCN, Deutsche Telekom/T-Systems, I-WIRE

• A facility for links coming from NetherLight, CERN/DataTAG, CA*net4, and proposed from UK-Light and APAN forming Trans-Light

StarLight supports the OptIPuter as

Clusters East of I-5

• Each site (StarLight, UIC, NetherLight) has – Several clusters with dual processors and dual GigE

cards– Electronic switching and routing– Optical switching– Specialized Clusters

• Computing • Data Mining and Serving• Visualization

• Upgrade to Itanium Clusters in progress• Upgrade to 10GigE NICs next year

Lambdas East of I-5, 2003

• Illinois– 16 GigEs to UIC– 8 GigEs to NorthWestern/Evanston– GigEs to ANL, NCSA

• Canada– 8 GigEs Chicago to NYC– 8 GigEs Chicago to Seattle

• Europe– 16 GigEs Chicago to Amsterdam– 4 GigEs Chicago to CERN– 2 GigEs CERN to Amsterdam

Optical MEMS Switching

Why Optical Switching?• No need to look at every packet when transferring

a terabyte of information– 1% the cost of routing– 10% the cost of switching– 64x64 10Gb:

• $100,000 O-O-O switched• $1,000,000 O-E-O switched• $10,000,000 O-E-O Routed

• Spend the savings on links, computing and collaboration systems instead!

• Replaces patch panels; allows rapid reconfiguration of 1 and 10Gb experiments

UIC Procurement of 3D MEMS Switches

• Sent out bid request for (2) 64x64 3D MEMS Switches

• GlimmerGlass Networks and Calient switches were tested with a small cluster– Using both MMF (with adapters) and SMF NICs– Computer controlled– No problems encountered

• Calient won the bid for 2 switches; UIC is upgrading one switch to 128x128

• GlimmerGlass Networks will provide EVL a loaner 64x64 switch

Optical Switches at StarLight and NetherLight

N E T H E R L I G H T

A “groomer” is a box that accepts multiple circuits of varying types (e.g., 1GigE, 10GigE) and aggregates and/or disseminates over the 10Gbps transoceanic link. As the amount of transoceanic connectivity increases, we aim to “bandwidth match” the amount of data being sent and/or received by clusters across continents.

GigE = Gigabit Ethernet (Gbps connection type)

8-processor cluster

16-processor cluster

Switch/Router

8 GigE16 GigE

8 GigE16 GigE

Control plane

Data plane

“Groomer” at StarLight

8 GigE

2 GigE

128x128MEMS

Optical Switch

N-processor cluster

8 GigE

N GigE

“Groomer” at NetherLight

Control plane

Data plane

2 GigE

OC-192

(10Gbps)

N GigE

64x64MEMS

Optical Switch

Switch/Router

SL 650916-processor cluster

8-processor cluster

ONI DWDM

SL OPSW

NL 15454

16

8

2

6

6

2

8

OC-192

128x128 MEMSOptical Switch

OMNInet8

4

10Gb

10Gb

Layer2/3 Switch/Router

EVL 6509

16-processor cluster

16

16

EVL OPSW

64x64 MEMS

16

ONI DWDM

OMNInet

6

8

2

Layer2/3 Switch/Router 16

UIC/EVL LAN

StarLightStarLight LAN

NetherLight/UvA

NL 15454

NL OPSWNL 6509

Layer2/3 Switch/Router

6

2

8-processor cluster

8

8

Fiber Use in Pairs

GE ElectronicallySwitched

GE OpticallySwitched

64x64 MEMSOptical Switch

OptIPuterEVL/StarLight/NetherLight/UvA

Summer 2003 Plan

Hard Infostructure Problems

• Internet is not designed for single large-scale users—TCP is not usable for long fat applications

• Circuits are not scalable, but neither are router$• All intelligence has to be on the edge• Tuning compute, data, visualization, networking

using clusters to get order of magnitude improvement

• Security at 10Gb line speed

Thanks to…• StarLight planning, research, collaborations, and outreach

efforts are made possible, in major part, by funding from: – National Science Foundation (NSF) awards ANI-9980480, ANI-9730202,

EIA-9802090, EIA-9871058, ANI-0225642, and EIA-0115809– NSF Partnerships for Advanced Computational Infrastructure (PACI)

cooperative agreement ACI-9619019 to NCSA– State of Illinois I-WIRE Program, and major UIC cost sharing– Northwestern University for providing space, engineering and

management• NSF/CISE/ANIR and DoE/Argonne National Laboratory for StarLight and

I-WIRE network engineering and planning leadership• NSF/CISE/ACIR and NCSA/SDSC for DTF/TeraGrid/ETF opportunities• The OMNInet Initiative• UCAID/Abilene for Internet2 and ITN transit; IU for the GlobalNOC • Bill St. Arnaud of CANARIE, Kees Neggers of SURFnet, Olivier Martin of

CERN, Michael McRobbie of IU, and Harvey Newman of CalTech

OptIPuter: East of I-5