The Missing Link: Dedicated End-to-End 10Gbps Optical Lightpaths for Clusters, Grids, and Clouds...
-
Upload
eric-rollins -
Category
Documents
-
view
212 -
download
0
Transcript of The Missing Link: Dedicated End-to-End 10Gbps Optical Lightpaths for Clusters, Grids, and Clouds...
The Missing Link: Dedicated End-to-End 10Gbps Optical Lightpaths
for Clusters, Grids, and Clouds
Invited Keynote Presentation
11th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing
Newport Beach, CA
May 24, 2011
Dr. Larry Smarr
Director, California Institute for Telecommunications and Information Technology
Harry E. Gruber Professor,
Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
Follow me on Twitter: lsmarr
1
Abstract
Today we are living in a data-dominated world where distributed scientific instruments, as well as clusters, generate terabytes to petabytes of data which are stored increasingly in specialized campus facilities or in the Cloud. It was in response to this challenge that the NSF funded the OptIPuter project to research how user-controlled 10Gbps dedicated lightpaths (or "lambdas") could transform the Grid into a LambdaGrid. This provides direct access to global data repositories, scientific instruments, and computational resources from "OptIPortals," PC clusters which provide scalable visualization, computing, and storage in the user's campus laboratory. The use of dedicated lightpaths over fiber optic cables enables individual researchers to experience "clear channel" 10,000 megabits/sec, 100-1000 times faster than over today's shared Internet-a critical capability for data-intensive science. The seven-year OptIPuter computer science research project is now over, but it stimulated a national and global build-out of dedicated fiber optic networks. U.S. universities now have access to high bandwidth lambdas through the National LambdaRail, Internet2's WaveCo, and the Global Lambda Integrated Facility. A few pioneering campuses are now building on-campus lightpaths to connect the data-intensive researchers, data generators, and vast storage systems to each other on campus, as well as to the national network campus gateways. I will give examples of the application use of this emerging high performance cyberinfrastructure in genomics, ocean observatories, radio astronomy, and cosmology.
Large Data Challenge: Average Throughput to End User on Shared Internet is 10-100 Mbps
http://ensight.eos.nasa.gov/Missions/terra/index.shtml
Transferring 1 TB:--50 Mbps = 2 Days--10 Gbps = 15 Minutes
TestedJanuary 2011
fc *
OptIPuter Solution: Give Dedicated Optical Channels to Data-Intensive Users
(WDM)
Source: Steve Wallach, Chiaro Networks
“Lambdas”Parallel Lambdas are Driving Optical Networking
The Way Parallel Processors Drove 1990s Computing
10 Gbps per User >100x Shared Internet Throughput
Dedicated 10Gbps Lightpaths Tie Together State and Regional Fiber Infrastructure
Interconnects Two Dozen
State and Regional Optical Networks
Internet2 WaveCo Circuit Network Is Now Available
Visualization courtesy of Bob Patterson, NCSA.
www.glif.is
Created in Reykjavik, Iceland 2003
The Global Lambda Integrated Facility--Creating a Planetary-Scale High Bandwidth Collaboratory
Research Innovation Labs Linked by 10G Dedicated Lambdas
High Resolution Uncompressed HD StreamsRequire Multi-Gigabit/s Lambdas
U. Washington
JGN II WorkshopOsaka, Japan
Jan 2005
Prof. Osaka Prof. Aoyama
Prof. Smarr
Source: U Washington Research Channel
Telepresence Using Uncompressed 1.5 Gbps HDTV Streaming Over IP on Fiber
Optics--75x Home Cable “HDTV” Bandwidth!
“I can see every hair on your head!”—Prof. Aoyama
September 26-30, 2005Calit2 @ University of California, San Diego
California Institute for Telecommunications and Information Technology
Borderless CollaborationBetween Global University Research Centers at 10Gbps
iGrid
2005T H E G L O B A L L A M B D A I N T E G R A T E D F A C I L I T Y
Maxine Brown, Tom DeFanti, Co-Chairs
www.igrid2005.org
100Gb of Bandwidth into the Calit2@UCSD BuildingMore than 150Gb GLIF Transoceanic Bandwidth!450 Attendees, 130 Participating Organizations
20 Countries Driving 49 Demonstrations1- or 10- Gbps Per Demo
Telepresence Meeting Using Digital Cinema 4k Streams
Keio University President Anzai
UCSD Chancellor Fox
Lays Technical Basis for
Global Digital
Cinema
Sony NTT SGI
Streaming 4k with JPEG
2000 Compression
½ Gbit/sec
100 Times the Resolution
of YouTube!
Calit2@UCSD Auditorium
4k = 4000x2000 Pixels = 4xHD
iGrid Lambda High Performance Computing Services:Distributing AMR Cosmology Simulations
• Uses ENZO Computational Cosmology Code– Grid-Based Adaptive Mesh
Refinement Simulation Code– Developed by Mike Norman, UCSD
• Can One Distribute the Computing?– iGrid2005 to Chicago to Amsterdam
• Distributing Code Using Layer 3 Routers Fails
• Instead Using Layer 2, Essentially Same Performance as Running on Single Supercomputer– Using Dynamic Lightpath
Provisioning
Source: Joe Mambretti, Northwestern U
iGrid Lambda Control Services: Transform Batch to Real-Time Global e-Very Long Baseline Interferometry
• Goal: Real-Time VLBI Radio Telescope Data Correlation • Achieved 512Mb Transfers from USA and Sweden to MIT• Results Streamed to iGrid2005 in San Diego
Optical Connections Dynamically Managed Using the DRAGON Control Plane and Internet2 HOPI Network
Source: Jerry Sobieski, DRAGON
The OptIPuter Project: Creating High Resolution Portals Over Dedicated Optical Channels to Global Science Data
Picture Source: Mark Ellisman, David Lee, Jason Leigh
Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PIUniv. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AISTIndustry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
Scalable Adaptive Graphics Environment (SAGE)
OptIPortal
What is the OptIPuter?
• Applications Drivers Interactive Analysis of Large Data Sets
• OptIPuter Nodes Scalable PC Clusters with Graphics Cards
• IP over Lambda Connectivity Predictable Backplane
• Open Source LambdaGrid Middleware Network is Reservable
• Data Retrieval and Mining Lambda Attached Data Servers
• High Defn. Vis., Collab. SW High Performance Collaboratory
See Nov 2003 Communications of the ACM for Articles on OptIPuter Technologies
www.optiputer.net
OptIPuter Software Architecture--a Service-Oriented Architecture Integrating Lambdas Into the Grid
GTP XCP UDT
LambdaStreamCEP RBUDP
DVC Configuration
Distributed Virtual Computer (DVC) API
DVC Runtime Library
Globus
XIOGRAM GSI
Distributed Applications/ Web ServicesTelescience
Vol-a-Tile
SAGE JuxtaView
Visualization
Data Services
LambdaRAM
DVC ServicesDVC Core Services
DVC Job Scheduling
DVCCommunication
Resource Identify/Acquire
NamespaceManagement
Security Management
High SpeedCommunication
Storage Services
IPLambdas
Discovery and Control
PIN/PDC RobuStore
OptIPortals Scale to 1/3 Billion Pixels Enabling Viewing of Very Large Images or Many Simultaneous Images
Spitzer Space Telescope (Infrared)
Source: Falko Kuester, Calit2@UCSD
NASA Earth Satellite Images
Bushfires October 2007
San Diego
The Latest OptIPuter Innovation:Quickly Deployable Nearly Seamless OptIPortables
45 minute setup, 15 minute tear-down with two people (possible with one)
Shipping Case
Calit2 3D Immersive StarCAVE OptIPortal
Cluster with 30 Nvidia 5600 cards-60 GB Texture Memory
Source: Tom DeFanti, Greg Dawe, Calit2
Connected at 50 Gb/s to Quartzite
30 HD Projectors!
15 Meyer Sound Speakers + Subwoofer
Passive Polarization--Optimized the
Polarization Separation and Minimized Attenuation
3D Stereo Head Tracked OptIPortal:NexCAVE
Source: Tom DeFanti, Calit2@UCSD
www.calit2.net/newsroom/article.php?id=1584
Array of JVC HDTV 3D LCD ScreensKAUST NexCAVE = 22.5MPixels
High Definition Video Connected OptIPortals:Virtual Working Spaces for Data Intensive Research
Source: Falko Kuester, Kai Doerr Calit2; Michael Sims, Larry Edwards, Estelle Dodson NASA
Calit2@UCSD 10Gbps Link to NASA Ames Lunar Science Institute, Mountain View, CA
NASA SupportsTwo Virtual Institutes
LifeSize HD
2010
EVL’s SAGE OptIPortal VisualCastingMulti-Site OptIPuter Collaboratory
CENIC CalREN-XD Workshop Sept. 15, 2008
EVL-UI Chicago
U Michigan
Streaming 4k
Source: Jason Leigh, Luc Renambot, EVL, UI Chicago
At Supercomputing 2008 Austin, TexasNovember, 2008SC08 Bandwidth Challenge Entry
Requires 10 Gbps Lightpath to Each Site
Total Aggregate VisualCasting Bandwidth for Nov. 18, 2008Sustained 10,000-20,000 Mbps!
NICSORNL
NSF TeraGrid KrakenCray XT5
8,256 Compute Nodes99,072 Compute Cores
129 TB RAM
simulation
Argonne NLDOE Eureka
100 Dual Quad Core Xeon Servers200 NVIDIA Quadro FX GPUs in 50
Quadro Plex S4 1U enclosures3.2 TB RAM rendering
SDSC
Calit2/SDSC OptIPortal120 30” (2560 x 1600 pixel) LCD panels10 NVIDIA Quadro FX 4600 graphics cards > 80 megapixels10 Gb/s network throughout
visualization
ESnet10 Gb/s fiber optic network
*ANL * Calit2 * LBNL * NICS * ORNL * SDSC
Using Supernetworks to Couple End User’s OptIPortal to Remote Supercomputers and Visualization Servers
Source: Mike Norman, Rick Wagner, SDSC
Eureka100 Dual Quad Core Xeon Servers
200 NVIDIA FX GPUs 3.2 TB RAM
ALCF
Rendering
Science Data Network (SDN)> 10 Gb/s Fiber Optic NetworkDynamic VLANs ConfiguredUsing OSCARS
ESnetSDSC
OptIPortal (40M pixels LCDs)10 NVIDIA FX 4600 Cards10 Gb/s Network Throughout
Visualization
Last Year NowHigh-Resolution (4K+, 15+ FPS)—But:• Command-Line Driven• Fixed Color Maps, Transfer Functions• Slow Exploration of Data
Driven by a Simple Web GUI:•Rotate, Pan, Zoom •GUI Works from Most Browsers• Manipulate Colors and Opacity• Fast Renderer Response Time
National-Scale Interactive Remote Renderingof Large Datasets
Interactive Remote Rendering
Real-Time Volume Rendering Streamed from ANL to SDSC
Source: Rick Wagner, SDSC
NSF OOI is a $400M Program -OOI CI is $34M Part of This
Source: Matthew Arrott, Calit2 Program Manager for OOI CI
30-40 Software EngineersHoused at Calit2@UCSD
OOI CIPhysical Network Implementation
Source: John Orcutt, Matthew Arrott, SIO/Calit2
OOI CI is Built on NLR/I2 Optical Infrastructure
CWave core PoP
10GE waves on NLR and CENIC (LA to SD)
Equinix818 W. 7th St.Los Angeles
PacificWave1000 Denny Way(Westin Bldg.)Seattle
Level31360 Kifer Rd.Sunnyvale
StarLightNorthwestern UnivChicago
Calit2San Diego
McLean
CENIC Wave Cisco Has Built 10 GigE Waves on CENIC, PW, & NLR and Installed Large 6506 Switches for
Access Points in San Diego, Los Angeles, Sunnyvale, Seattle, Chicago and McLean
for CineGrid MembersSome of These Points are also GLIF GOLEs
Source: John (JJ) Jamison, Cisco
Cisco CWave for CineGrid: A New Cyberinfrastructurefor High Resolution Media Streaming*
May 2007*
2007
CineGrid 4K Digital Cinema Projects: “Learning by Doing”
CineGrid @ iGrid 2005 CineGrid @ AES 2006
CineGrid @ GLIF 2007
Laurin Herr, Pacific Interface; Tom DeFanti, Calit2
CineGrid @ Holland Festival 2007
First Tri-Continental Premier of a Streamed 4K Feature Film With Global HD Discussion
San Paulo, Brazil Auditorium
Keio Univ., Japan Calit2@UCSD
4K Transmission Over 10Gbps--4 HD Projections from One 4K Projector
4K Film Director, Beto Souza
Source: Sheldon Brown, CRCA, Calit2
July 30, 2009
CineGrid 4K Remote Microscopy Collaboratory:USC to Calit2
Richard Weinberg, USC
Photo: Alan Decker December 8, 2009
Open Cloud OptIPuter Testbed--Manage and Compute Large Datasets Over 10Gbps Lambdas
29
NLR C-Wave
MREN
CENIC Dragon
Open Source SW Hadoop Sector/Sphere Nebula Thrift, GPB Eucalyptus Benchmarks
Source: Robert Grossman, UChicago
• 9 Racks• 500 Nodes• 1000+ Cores• 10+ Gb/s Now• Upgrading Portions to
100 Gb/s in 2010/2011
Terasort on Open Cloud TestbedSustains >5 Gbps--Only 5% Distance Penalty!
Sorting 10 Billion Records (1.2 TB) at 4 Sites (120 Nodes)
Source: Robert Grossman, UChicago
“Blueprint for the Digital University”--Report of the UCSD Research Cyberinfrastructure Design Team
• Focus on Data-Intensive Cyberinfrastructure
research.ucsd.edu/documents/rcidt/RCIDTReportFinal2009.pdf
No Data Bottlenecks--Design for Gigabit/s Data Flows
April 2009
Source: Jim Dolgonas, CENIC
Campus Preparations Needed to Accept CENIC CalREN Handoff to Campus
Current UCSD Prototype Optical Core:Bridging End-Users to CENIC L1, L2, L3 Services
Source: Phil Papadopoulos, SDSC/Calit2 (Quartzite PI, OptIPuter co-PI)Quartzite Network MRI #CNS-0421555; OptIPuter #ANI-0225642
Lucent
Glimmerglass
Force10
Enpoints:
>= 60 endpoints at 10 GigE
>= 32 Packet switched
>= 32 Switched wavelengths
>= 300 Connected endpoints
Approximately 0.5 TBit/s Arrive at the “Optical” Center of Campus.Switching is a Hybrid of: Packet, Lambda, Circuit --OOO and Packet Switches
Calit2 SunlightOptical Exchange Contains Quartzite
Maxine Brown,
EVL, UICOptIPuter
Project Manager
UCSD Campus Investment in Fiber Enables Consolidation of Energy Efficient Computing & Storage
Source: Philip Papadopoulos, SDSC, UCSD
OptIPortalTiled Display Wall
Campus Lab Cluster
Digital Data Collections
N x 10Gb/sN x 10Gb/s
Triton – Petascale
Data Analysis
Gordon – HPD System
Cluster Condo
WAN 10Gb: WAN 10Gb: CENIC, NLR, I2CENIC, NLR, I2
Scientific Instruments
DataOasis (Central) Storage
GreenLightData Center
National Center for Microscopy and Imaging Research: Integrated Infrastructure of Shared Resources
Source: Steve Peltier, NCMIR
Local SOM Infrastructure
Scientific Instruments
End UserWorkstations
Shared Infrastructure
Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis
http://camera.calit2.net/
Calit2 Microbial Metagenomics Cluster-Lambda Direct Connect Science Data Server
512 Processors ~5 Teraflops
~ 200 Terabytes Storage 1GbE and
10GbESwitched/ Routed
Core
~200TB Sun
X4500 Storage
10GbE
Source: Phil Papadopoulos, SDSC, Calit2
4000 UsersFrom 90 Countries
Creating CAMERA 2.0 -Advanced Cyberinfrastructure Service Oriented Architecture
Source: CAMERA CTO Mark Ellisman
OptIPuter Persistent Infrastructure EnablesCalit2 and U Washington CAMERA Collaboratory
Ginger Armbrust’s Diatoms:
Micrographs, Chromosomes,
Genetic Assembly
Photo Credit: Alan Decker Feb. 29, 2008
iHDTV: 1500 Mbits/sec Calit2 to UW Research Channel Over NLR
NSF Funds a Data-Intensive Track 2 Supercomputer:SDSC’s Gordon-Coming Summer 2011
• Data-Intensive Supercomputer Based on SSD Flash Memory and Virtual Shared Memory SW– Emphasizes MEM and IOPS over FLOPS– Supernode has Virtual Shared Memory:
– 2 TB RAM Aggregate– 8 TB SSD Aggregate– Total Machine = 32 Supernodes– 4 PB Disk Parallel File System >100 GB/s I/O
• System Designed to Accelerate Access to Massive Data Bases being Generated in Many Fields of Science, Engineering, Medicine, and Social Science
Source: Mike Norman, Allan Snavely SDSC
Rapid Evolution of 10GbE Port PricesMakes Campus-Scale 10Gbps CI Affordable
2005 2007 2009 2010
$80K/port Chiaro(60 Max)
$ 5KForce 10(40 max)
$ 500Arista48 ports
~$1000(300+ Max)
$ 400Arista48 ports
• Port Pricing is Falling • Density is Rising – Dramatically• Cost of 10GbE Approaching Cluster HPC Interconnects
Source: Philip Papadopoulos, SDSC/Calit2
Arista Enables SDSC’s Massive Parallel 10G Switched Data Analysis Resource
212
OptIPuterOptIPuter
32
Co-LoCo-Lo
UCSD RCI
UCSD RCI
CENIC/NLR
CENIC/NLR
Trestles100 TF
8Dash
128Gordon
Oasis Procurement (RFP)
• Phase0: > 8GB/s Sustained Today • Phase I: > 50 GB/sec for Lustre (May 2011) :Phase II: >100 GB/s (Feb 2012)
40128
Source: Philip Papadopoulos, SDSC/Calit2
Triton32
Radical Change Enabled by Arista 7508 10G Switch
384 10G Capable
8Existing
Commodity Storage1/3 PB
2000 TB> 50 GB/s
10Gbps
58 2
4
Data Oasis – 3 Different Types of Storage
Calit2 CAMERA Automatic Overflows into SDSC Triton
Triton Resource
CAMERA
DATA
@ CALIT2
@ SDSC
CAMERA -Managed
Job Submit Portal (VM)
10Gbps
Transparently Sends Jobs to Submit Portal
on Triton
Direct Mount
== No Data Staging
California and Washington Universities Are Testing a 10Gbps Lambda-Connected Commercial Data Cloud
• Amazon Experiment for Big Data– Only Available Through CENIC & Pacific NW
GigaPOP– Private 10Gbps Peering Paths
– Includes Amazon EC2 Computing & S3 Storage Services
• Early Experiments Underway– Phil Papadopoulos, Calit2/SDSC Rocks– Robert Grossman, Open Cloud Consortium
Using Condor and Amazon EC2 onAdaptive Poisson-Boltzmann Solver (APBS)
• APBS Rocks Roll (NBCR) + EC2 Roll + Condor Roll = Amazon VM
• Cluster extension into Amazon using Condor
Running in Amazon Cloud
APBS + EC2 + Condor
EC2 CloudEC2 CloudLocal Cluster
NBCR VM
NBCR VM
NBCR VM
Source: Phil Papadopoulos, SDSC/Calit2
Hybrid Cloud Computing with modENCODE Data
• Computations in Bionimbus Can Span the Community Cloud & the Amazon Public Cloud to Form a Hybrid Cloud
• Sector was used to Support the Data Transfer between Two Virtual Machines – One VM was at UIC and One VM was an Amazon EC2 Instance
• Graph Illustrates How the Throughput between Two Virtual Machines in a Wide Area Cloud Depends upon the File Size
Source: Robert Grossman, UChicago
Biological data (Bionimbus)
OptIPlanet Collaboratory:Enabled by 10Gbps “End-to-End” Lightpaths
National LambdaRail
CampusOptical Switch
Data Repositories & Clusters
HPC
HD/4k Video Repositories
End User OptIPortal
10G Lightpaths
HD/4k Live Video
Local or Remote Instruments
You Can Download This Presentation at lsmarr.calit2.net