Genomics at the Speed of Light: Understanding the Living Ocean

The Gordon and Betty Moore Foundation

2nd Annual Marine Microbiology Investigator Symposium

The Golden Gate Club, The Presidio of San Francisco

July 17-19, 2006

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

fc *

Dedicated Optical Channels Makes High Performance Cyberinfrastructure Possible

(WDM)

Source: Steve Wallach, Chiaro Networks

“Lambdas”Parallel Lambdas are Driving Optical Networking

The Way Parallel Processors Drove 1990s Computing

National Lambda Rail (NLR) and TeraGrid Provides Cyberinfrastructure Backbone for U.S. Researchers

NLR 4 x 10Gb Lambdas Initially Capable of 40 x 10Gb wavelengths at Buildout

Links Two Dozen State and Regional Optical

Networks

DOE, NSF, & NASA

Using NLR

San Francisco Pittsburgh

Cleveland

San Diego

Los Angeles

Portland

Seattle

Pensacola

Baton Rouge

HoustonSan Antonio

Las Cruces /El Paso

Phoenix

New York City

Washington, DC

Raleigh

Jacksonville

Dallas

Tulsa

Atlanta

Kansas City

Denver

Ogden/Salt Lake City

Boise

Albuquerque

UC-TeraGridUIC/NW-Starlight

Chicago

International Collaborators

NSF’s TeraGrid Has 4 x 10Gb Lambda Backbone

www.glif.is

Created in Reykjavik, Iceland 2003

Countries are Aggressively Creating Gigabit Services:Interactive Access to CAMERA and LOOKING Systems

Visualization courtesy of Bob Patterson, NCSA.

Flat FileServerFarm

W E

B P

OR

TA

L

TraditionalUser

Response

Request

DedicatedCompute Farm(1000 CPUs)

TeraGrid: Cyberinfrastructure Backplane(scheduled activities, e.g. all by all comparison)

(10000s of CPUs)

Web(other service)

Local Cluster

LocalEnvironment

DirectAccess LambdaCnxns

Data-BaseFarm

10 GigE Fabric

Calit2’s Direct Access Core Architecture Will Create Next Generation Metagenomics Server

Source: Phil Papadopoulos, SDSC, Calit2+

We

b S

erv

ice

s

Sargasso Sea Data

Sorcerer II Expedition (GOS)

JGI Community Sequencing Project

Moore Marine Microbial Project

NASA Goddard Satellite Data

Community Microbial Metagenomics Data

The Future Home of the Moore Foundation Funded Marine Microbial Ecology Metagenomics Complex

First Implementation of the CAMERA Complex

Photo Courtesy Joe Keefe, Calit2

Major Buildout of Calit2 Server Room Underway

Moore CAMERA Server Facility Environmental Upgrade Underway

• Addition of 1000 sf to server facility underway– Adding 500kva Transformer, 6 225A Breakers, Starline Power Grid– Adding 66 Ton of Cooling, Ducting and Registers– Adding Plumbing, Electrical Raceway, Overhead Cable Trays– Supporting an Additional 20 Racks of Equipment– Completion September 2006

Source: Greg Hidley, Calit2

Moore CAMERAProduction Environment

• Creation of Initial Production Environment – September 2006– Status – RFP to Appear August 1 – Targeting September Installation– Hardware (Best Approximation Today)

– Compute Nodes –– ~200 4 CPU Nodes = ~800 Processing Cores

– Storage Servers –– 10 systems = ¼ Petabyte raw (100-125TB usable raided and replicated)

storage– Database Servers

– Larger 20-40TB– Smaller 5-10TB

– Network Management – – Force10 E1200 Router w/12 10GigE Interfaces to Each System Ports

• Expansion Late 2007—Will be Tuned by Initial Usage and Load– Compute Nodes –

– Towards 1600+ Processing Cores– Storage and Databases –

– Towards 200+ TB of Storage

Source: Greg Hidley, Calit2; Phil Papadopoulos, SDSC, Calit2

OptIPuter Scalable Adaptive Graphics Environment (SAGE) Allows Integration of HD Streams

OptIPortal– Termination

Device for the

OptIPuter Global

Backplane

Calit2 and the Venter Institute Will Combine Telepresence with Remote Interactive Analysis

OptIPuter Visualized

Data

HDTV Over

Lambda

Live Demonstration

of 21st Century National-Scale Team Science 25 Miles

Venter Institute

Calit2 and the Venter Institute Test CineGrid™ with HDTV Movie by John Carter

Live Demonstration of 21st Century Entertainment Delivery June 14, 2006

JCVI

Sony HDTV JH-3

JC Venter Institute Rockville, MD

Calit2 Auditorium

StarLight

Chicago

OptIPortal– Termination Device for the OptIPuter Global Backplane

• 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000• 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC!• Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC

Source: Phil Papadopoulos SDSC, Calit2

UIC/UCSD 10GE CAVEWave on the National LambdaRailEmerging OptIPortal Sites

CAVEWave Connects Chicago to Seattle to San Diego…and Washington D.C. as of 4/1/06

and JCVI as of 5/15/06

NEW!

NEW!

SunLight

CICESE

UW

JCVI

MIT

SIO UCSD

SDSU

UIC EVL

UCI

OptIPortals

First Remote Interactive High Definition Video Exploration of Deep Sea Vents

Source John Delaney & Deborah Kelley, UWash

Canadian-U.S. Collaboration

High Definition Still Frame of Hydrothermal Vent Ecology 2.3 Km Deep

White Filamentous Bacteria on 'Pill Bug' Outer Carapace

1 cm.

Source: John Delaney and

Research Channel, U Washington