The OptIPuter—Lambda Coupled Distributed Computing,

Peer-to-Peer Storage, and Volume Visualization

Dr. Larry Smarr,

Director, California Institute for Telecommunications and Information Technology

Professor, Computer Science and Engineering,

Jacobs School of Engineering, UCSD

Two New Cal-(IT)2 Buildings Approved by Legislature This Week!

• Will Create New Laboratory Facilities– Clean Rooms for Nanotech and BioMEMS– Computer Arts Virtual Reality– Wireless and Optical Networking– Interdisciplinary Teams

Bioengineering

UC San Diego

UC Irvine

Future Systems Will “Waste” Bandwidth and Storage, While Conserving Computing

Scientific American, January 2001

S-Curves of Exponential Technology Growth

0%

100%

Research

Experimental/Early Adopters

Production/Mass Market

Time

Technology S-Curve

Gigabit Testbeds

Connections Program

Internet2 Abilene

DWDM

Experimental Networks

Lambda Grids

~1990s 2000 2010

Networking Technology S-Curves

Tec

hn

olog

y P

enet

rati

on

CONTROL

PLANE

Clusters

DynamicallyAllocatedLightpaths

Switch Fabrics

PhysicalMonitoring

Apps Middleware

A LambdaGrid Will Be the Backbone for an e-Science Network

• Metro Area Laboratories Springing Up Worldwide

• Developing GigE and 10GigE Applications and Services

• Testing Optical Switches

• Metro Optical Testbeds-the next GigaPOP?

Source: Joe Mambretti, NU

NSF Defines Three Classes of Networks Beyond the Commodity Internet

• Production Networks (e.g. Internet2) – High-Performance Networks– Reaches All US Researchers– 24 / 7 Reliable

• Experimental Networks– Trials of Cutting-Edge High-Performance Networks – Deliver Advanced Application Needs Unsupported by Production Networks– Robust Enough to Support Application-Dictated Development:

– Software Application Toolkits, – Middleware, – Computing and Networking

• Research Networks– Smaller-Scale Network Prototypes – Enable Basic Scientific and Engineering Network Research– Testing of Component Technologies, Protocols, Network Architectures– Not Expected to Be Persistent– Not Expected to Support Production Applications

www.evl.uic.edu/activity/NSF/index.html

Local and Regional Lambda Experimental Networks Are Achievable and Practical

• Several GigaPOPs Are Building Multi-Lambda Metropolitan Experimental Networks by Lighting up Their Own Dark Fiber – With Hundreds of Lambdas by 2010

• LambdaGrid Software to Dynamically Couple Applications to Metro and Resources Is Possible

• Metro Experimental Networks Will Be Sensitized to: – End Application Needs – Local Connectivity Issues – Current National Production Networks Have Not Begun to Satisfy

• Substantial State and Local Funds Can Be Heavily Leveraged by an NSF Experimental Networks Program – Cross-country Inter-Connection– Persistent Support of Emerging Experimental Networks– First NSF Workshop UIC December 2001– Second NSF Workshop UCI May 2002– Expected NSF RFP by Fall 2002

Cal-(IT)2 is Developing a Metro-ScaleExperimental Optical Network

• Driven by Data-Intensive Applications– Real Time Seismic – Emergency Response– Medical Imaging

• Linked UCSD and SDSU– Dedication March 4, 2002

Linking Control Rooms

Cox, Panoram,SAIC, SGI, IBM,

TeraBurst NetworksSD Telecom Council

UCSD SDSU44 Miles of Cox Fiber

Beginning of Cal-(IT)2 Metro Optical Networking Laboratory

TeraBurst

TeraBurst

Next Step– California Must Have a State-Wide

Experimental Optical Network• The Institutes are Creating a Joint Plan

– Led by Cal-(IT)2 & CITRIS

– Involving QB3 and CNSI

• Leveraging Today’s CENIC Investment– Provides California Internet Connectivity

– K-12 and Universities

• Necessary for Data-Intensive Science– Widely Available to Many Disciplines

• California is Not the Leader Today!– Illinois, Indiana, and Canada Already There

CENIC and CISI May Create a Dark FiberExperimental and Research Network

Figure 2: SoCal Optical Research and Experimental Network

The SoCal Component

Some Scientific Applications Require Experimental Optical Networks

• Challenges– Large Data Challenges in Neuro and Earth Sciences– Each Data Object is 3D and Gigabytes– Data are Generated and Stored in Distributed Archives– Research is Carried Out on Federated Repository

• Requirements– Computing Requirements PC Clusters– Communications Dedicated Lambdas– Data Large Peer-to-Peer Lambda Attached Storage – Visualization Collaborative Volume Algorithms

• Response– Cal-(IT)2 OptIPuter Research Project

NIH is Funding a National-Scale Grid Which is an OptIPuter Application Driver

National Partnership for Advanced Computational Infrastructure

Part of the UCSD CRBS Center for Research on Biological Structure

Biomedical Informatics Research Network

(BIRN)NIH Plans to Expand

to Other Organs and Many Laboratories

The OptIPuter is an Experimental Network Research Project

• Multiple Lambdas Linking Clusters and Storage– Integration with InfiniBand PC Clusters– Rethink TCP/IP Protocols– Peer to Peer Storage– LambdaGrid Software Stack– Interactive Collaborative Volume Visualization

• NSF Large Information Technology Research Proposal– Larry Smarr, PI

– UCSD and UIC Lead Campuses– USC, UCI, SDSU, NW Partnering Campuses– Industrial Partners: IBM, Telcordia/SAIC, Chiaro

Networks, CENIC– San Diego Telecom Council Letter of Support

– Submitted April 4, 2002– Seeking $15M over 5 Years

The OptIPuter Research Team and ItsRegional, National, Int’l Extensions

Vancouver

Seattle

Portland

San Francisco

Los Angeles

San Diego(SDSC)

NCSA

SURFnet CERNCA*net4

AsiaPacific

AsiaPacific

AMPATH

PSC

Atlanta

CA*net4

Source: Tom DeFanti and Maxine Brown, UIC

NYC

TeraGrid DTFnet

CENIC

Pacific LightRail

Chicago

UICNU

USC

UCSD, SDSUUCI

OMNInet Optical Switching TrialNow Underway in Chicago Metro Area

• A Four-Site Network in Chicago -- The First 10GE Service Trial!

• A Test Bed for All-Optical Switching and Advanced High-speed Services

• Partners: SBC, Nortel, iCAIR at Northwestern, UIC, CANARIE, ANL

2x10GE

Northwestern Univ.

OpticalSwitchingPlatform

Passport8600

ApplicationCluster

ApplicationCluster

OpticalSwitchingPlatform

Passport8600

2x10GE

StarLight

OPTera Metro5200

ApplicationCluster

OpticalSwitchingPlatform

Passport8600

2x10GE8x1GE

Univ. Illinois at Chicago

CA*net3/4--Chicago

OpticalSwitchingPlatform

Passport8600

ApplicationCluster

2x10GE8x1GE

8x1GE

8x1GE

Planned Chicago Metro Electronic Switching OptIPuter Laboratory

Int’l GE, 10GE

Nat’l GE, 10GE

Metro GE, 10GE

16x1 GE 16x10 GE

16-Processor McKinley at University of Illinois at Chicago

16-Processor Montecito/Chivano

at Northwestern StarLight

10x1 GE

+

1x10GE

Nationals: Illinois, California, Wisconsin, Indiana, Abilene, FedNets. Washington, Pennsylvania…

Internationals: Canada, Holland, CERN, GTRN, AmPATH, Asia…

Source: Tom DeFanti

Plan for UCSD Campus-Scale OptIPuter

Cluster orStorage

32GigEfiber pairs

Fiber Pair.DWDM

All OpticalSwitch

...

DWDM Lambdas

GigE Switchmultipleuplinks

To Distant Optiputer Locations

GigE/LambdaConversion

Philip Papadopoulos, SDSC

Initially Connect:Schools of Medicine

and Engineering with SIO and SDSC

OptIPuter Optical Research Topics

• Integrate InfiniBand and IP Over Lambdas – Connectivity, Addressing, Configuration, Recovery and Resource Discovery

• Design Evaluation, Monitoring and Analysis Tools for – Optimizing InfiniBand Over Dedicated Lambdas

• Devise Network Control and Management (NC&M) and Traffic Engineering– Gather Cluster Demand and Signal the Reconfiguration of the Network

• Extend Signaling – Permit OptIPuter Clusters to Establish and Teardown Connections to Peer

Clusters Across the IP/Lambda Network– Integrate This Signaling With Network-based NC&M

• Extend the IP/Lambda Network to Support Very Fast Switches – Participate in the Optical Switching of Label Switched Paths (LSPS) That

Transport InfiniBand Messages• Develop Common-Control-Plane Optical Transport Architectures

– With Generalized Multi-Protocol Label Switching (GMPLS) – Transport Traffic Over Multiple User Planes With Variable Switching Modes,

Namely Lambda Switching – Burst Switching– Inverse Multiplexing (Enabling One Application to Use Multiple Lambdas)

Other OptIPuter Research Topics

• Software Architecture– LambdaGrid Middleware Architecture – Application Communication Abstractions and Cluster Bundle

Composition– Real-Time, Dependable Execution Environment

• Distribute Storage While Optimizing Storewidth:– Distribute Massive Pools of Physical RAM (Network Memory)– Develop Visual TeraMining Techniques to Mine Petabytes of Data – Enable Ultrafast Image Rendering – Create for Optical Storage Area Networks (OSANs)

• Enhance Security Mechanisms:– End-to-End Integrity Check of Data Streams– Access Multiple Locations With Trusted Authentication Mechanisms– Use LambdaGrid Middleware for Authentication, Authorization,

Validation, Encryption and Forensic Analysis of Multiple Systems and Administrative Domains