The Pacific Research Platform: Leading Up to the National Research Platform
-
Upload
larry-smarr -
Category
Technology
-
view
483 -
download
1
Transcript of The Pacific Research Platform: Leading Up to the National Research Platform
“The Pacific Research Platform:
Leading Up to the National Research Platform”
Opening Keynote
The National Research Platform Workshop
Montana State University
Bozeman, MT
August 7, 2017
Dr. Larry Smarr
Director, California Institute for Telecommunications and Information Technology
Harry E. Gruber Professor,
Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
http://lsmarr.calit2.net1
30 Years Ago NSF Brought to University Researchers
a DOE HPC Center Model
NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet
1985/6
Launching the Nation’s Information Infrastructure:
NSFnet Supernetwork and the Six NSF Supercomputers
NCSA
NSFNET 56 Kb/s Backbone (1986-8)
PSCNCAR
CTC
JVNC
SDSC
PRP’s Backbone is One Million Times Faster!
I-WAY: Information Wide Area YearSupercomputing ‘95
• The First National 155 Mbps Research Network
– 65 Science Projects
– Into the San Diego Convention Center
• I-Way Featured:
– Networked Visualization Applications
– Large-Scale Immersive Displays
– I-Soft Programming Environment
– Led to the Globus Project
UIChttp://archive.ncsa.uiuc.edu/General/Training/SC95/GII.HPCC.html
See Session 1
Talk by
Ian Foster
NSF’s PACI Program was Built on the vBNS
to Prototype America’s 21st Century Information Infrastructure
The PACI Grid Testbed
National Computational Science
1997
vBNS
led to
NSF’s OptIPuter Project: Using Supernetworks
to Meet the Needs of Data-Intensive Researchers
OptIPortal–
Termination
Device
for the
OptIPuter
Global
Backplane
Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PIUniv. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST
Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
2003-2009
$13,500,000
In August 2003,
Jason Leigh and his
students used
RBUDP to blast
data from NCSA to
SDSC over the
TeraGrid DTFnet,
achieving18Gbps
file transfer out of
the available
20Gbps
LS Slide 2005
We Have Been Working Toward PRP for 15 Years:
NSF OptIPuter, Quartzite, Prism Awards
PI Papadopoulos,
2013-2015
PI Smarr,
2002-2009PI Papadopoulos,
2004-2007
Thirty Years After NSF Adopts DOE Supercomputer Center Model
NSF Adopts DOE ESnet’s Science DMZ for High Performance Applications
• A Science DMZ integrates 4 key concepts into a unified whole:
– A network architecture designed for high-performance applications,
with the science network distinct from the general-purpose network
– The use of dedicated systems as data transfer nodes (DTNs)
– Performance measurement and network testing systems that are
regularly used to characterize and troubleshoot the network
– Security policies and enforcement mechanisms that are tailored for
high performance science environments
http://fasterdata.es.net/science-dmz/
Science DMZ
Coined 2010
The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis
for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program
See Deep Dive 6
On High-Perf
Networking
Based on Community Input and on ESnet’s Science DMZ Concept,
NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways
Red 2012 CC-NIE Awardees
Yellow 2013 CC-NIE Awardees
Green 2014 CC*IIE Awardees
Blue 2015 CC*DNI Awardees
Purple Multiple Time Awardees
Source: NSF
(GDC)
Logical Next Step: The Pacific Research Platform Creates
a Regional End-to-End Science-Driven “Big Data Superhighway” System
NSF CC*DNI Grant
$5M 10/2015-10/2020
PI: Larry Smarr, UC San Diego Calit2
Co-Pis:
• Camille Crittenden, UC Berkeley CITRIS,
• Tom DeFanti, UC San Diego Calit2,
• Philip Papadopoulos, UCSD SDSC,
• Frank Wuerthwein, UCSD Physics and SDSC
Letters of Commitment from:
• 50 Researchers from 15 Campuses
• 32 IT/Network Organization Leaders
See Deep Dive 3
on NSF Research CI
Developing Regional DMZs
Require a Lot of Consultation-Sociotechnilogical Problem
• May 2014 LS Gives Invited Presentation to UC IT Leadership Council
– Strong Support from UC and UCOP CIOs
• July 2014 LS Gives Invited Talk to CENIC Annual Retreat
– CENIC/PW Agrees to Act as Backplane
– CIO Support Extends to CA Private Research Universities
• December 2014 UCOP CIO and VPR’s Provide PRP “Momentum Money”
• January 2015 Kickoff of PRPv0 by Network Engineers
– Begins Every Two Week Conference Calls, Now Weekly
• March 2015 LS Invited “Blue Sky” Presentation to UC VCR/CIO Summit
– NSF PRP Proposal Submitted With Letters of Commitment From:
– 50 Researchers from 15 Campuses
– 32 IT/Network Organization Leaders
• July 2015 NSF Announces Funding for PRP
• October 2015 PRP Grant Begins
Big Data Science Data Transfer Nodes (DTNs)-
Flash I/O Network Appliances (FIONAs)
UCSD Designed FIONAs
To Solve the Disk-to-Disk
Data Transfer Problem
at Full Speed
on 10G, 40G and 100G Networks
FIONAS—10/40G, $8,000FIONette—1G, $1,000
Phil Papadopoulos, SDSC &
Tom DeFanti, Joe Keefe & John Graham, Calit2
John Graham, Calit2
See Deep Dive 4
How Does PRPv1 Work?
We Measure Disk-to-Disk Throughput with 10GB File Transfer
4 Times Per Day in Both Directions for All PRP Sites
January 29, 2016
From Start of Monitoring 12 DTNs
to 24 DTNs Connected at 10-40G
in 1 ½ Years
July 21, 2017
Source: John Graham, Calit2
Increasing Participation Through
PRP Science Engagement Workshops
Source: Camille Crittenden, UC Berkeley
UC San DiegoUC Merced
UC Davis UC Berkeley
See Session 3
on Engagement
The First National Research Platform Workshop
on August 7-8, 2017
Announced in I2 Closing Keynote:
Larry Smarr “Toward a National Big Data Superhighway”
on Wednesday, April 26.
Co-Chairs:
Larry Smarr, Calit2
& Jim Bottum, Internet2
150 Attendees
PRP’s First 1.5 Years:
Connecting Multi-Campus Application Teams and Devices
Particle Physics: Using the PRP to Fix
the “Last Mile Problem” of the LHC in California
ATLASCMS
U.S. Institutions
Participating in LHC
LHC Data
Generated by
CMS &
ATLAS
Detectors
Analyzed
on OSG
Maps from www.uslhc.us
See Talks by
Harvey Newman,
Frank Wuerthwein,
& Rob Gardner
Global Scientific Instruments Will Produce Ultralarge Datasets Continuously
Requiring Dedicated Optic Fiber and Supercomputers
Square Kilometer Array Large Synoptic Survey Telescope
https://tnc15.terena.org/getfile/1939
3.2 Gpixel Camera
Tracks ~40B Objects,
Creates 10M Alerts/Night
Within 1 Minute of Observing
2x100Gb/s
See Session 5 Talk
By Heidi Morgan
“First Light”
In 2019
100 Gbps FIONA at UCSC Allows for Downloads to the UCSC Hyades Cluster
from the LBNL NERSC Supercomputer for DESI Science Analysis
300 images per night.
100MB per raw image
30GB per night
120GB per night
250 images per night.
530MB per raw image
150 GB per night
800GB per night
Source: Peter Nugent, LBNL
Professor of Astronomy, UC Berkeley
Precursors to
LSST and NCSA
NSF-Funded Cyberengineer
Shaw Dong @UCSC
Receiving FIONA
Feb 7, 2017
Cancer Genomics Hub (UCSC) Was Housed in SDSC, But NIH Moved Dataset
From SDSC to Uchicago - So the PRP Deployed a FIONA to Chicago’s MREN
1G
8G
Data Source: David Haussler,
Brad Smith, UCSC
15GJan 2016
See Deep Dive 5
on Community
Data Resources
PRP Is Linking the Laboratories of
the Pacific Earthquake Engineering Research Center
http://peer.berkeley.edu/
PEER Labs: UC Berkeley, Caltech, Stanford,
UC Davis, UC San Diego, and UC Los Angeles
John Graham Installing FIONette at PEER Feb 10, 2017
40G FIONAs
20x40G PRP-connected
WAVE@UC San Diego
PRP Now Enables
Distributed Virtual Reality (and Distributed Machine Learning)
PRP
WAVE @UC Merced
Transferring 5 CAVEcam Images from UCSD to UC Merced:
2 Gigabytes now takes 2 Seconds (8 Gb/sec)
Visualizing Enormous Datasets Resulting from Supercomputer Simulations
Using EVL’s DTN GPU Services (PRP Provided EVL DTN)
Cosmological Volume Simulation
• Computed on the ALCF Mira – 29 Billion Particles
– 40 TB per Snapshot and 500 Snapshots
• Then Sent to NCSA Blue Waters – 1st -Level Data Analytics & Visualization
– Using “yt”
• Web Services on EVL’s DTN – Used to Convert a Snapshot
– To a DZI Multi-Resolution Image
– Displayed Using SAGE2
– In the SC16 NCSA Booth
Cosmologist Katrin Heitmann of ANL
Standing in Front of the SAGE2 Display at SC16
Slide from Maxine Brown, EVL Director
See Deep Dive 8
on SC17 Demos
The Prototype PRP Has Attracted
New Application Drivers
Scott Sellars, Marty Ralph
Center for Western Weather and Water Extremes
Frank Vernon - Expansion of HPWREN
Tom Levy, Cultural Heritage
Cryo EM
GPU JupyterHub:
2 x 14-core CPUs
256GB RAM
1.2TB FLASH
3.8TB SSD
Nvidia K80 GPU
Dual 40GbE NICs
And a Trusted Platform
Module
GPU JupyterHub:
1 x 18-core CPUs
128GB RAM
3.8TB SSD
Nvidia K80 GPU
Dual 40GbE NICs
And a Trusted Platform
Module
PRP UC-JupyterHub Backbone
Next Step: Deploy Across PRP
Source: John Graham, Calit2
UCB UCSD
Cryo-electron Microscopy (cryo-EM)
Has Driven a “Resolution Revolution” in the Last Five Years
Exposure (every 60 seconds):
X & Y dimensions: 7420 x 7676 Pixels
Frames per Movie: 10 - 50
Size: 3 - 10 GB per Movie
Every 24 hours:
Number of Movies: ~1400
Data Size: ~5 TB
Typical Datasets:
Length of Time: 2 - 6 Days
Total size: 10 - 30 TB
Each Cryo-EM ‘Image’ is Actually a Movie
Source: Michael A. Cianfrocco,
Elizabeth Villa, & Andres Leschziner, UCSD
Using PRP to Connect Cryo-EM across California
With End Users and Computational Facilities
Long term:
‣Partner with Cryo-EM Facilities to Stream Data
Straight from Microscopes (over PRP) to SDSC
‣Perform All Cryo-EM Analysis (from Micrographs
to 3D Models) via Web Browser on SDSC
‣Expand Computing to Other XSEDE Resources
(e.g. Xstream) and DOE’s NERSC
Short term:
‣Provide 2D and 3D Analysis on Particle Stacks on
Comet at SDSC
Source: Michael A. Cianfrocco, UCSD
**
SDSC
NERSC
Xstream
3 Supercomputer Centers
cosmic-cryoem.org
~20 Microscopes in CA
UCLA
UC Davis
UC Santa Cruz
SF Bay
UC Berkeley, LBNL,
UCSF, Stanford
San Diego
UCSD, TSRI, Salk*
Extending
to MSU
HPWREN Real-Time Network Cameras on Mountains
for Environmental Observations and Fires
San Diego County Red Mountain Fire Cameras
• Southeast (left) “Highway” Fire
• Southwest (center rear) “Poinsettia” Fire
• West (right) “Tomahawk” Fire
Source: Frank Vernon,
Hans Werner Braun HPWREN
May 14, 2014
PRP Backbone Sets Stage for 2017 Expansion
of HPWREN, Connected to CENIC, into Orange and Riverside Counties
• PRP CENIC 100G Link
UCSD to SDSU
– DTN FIONAs Endpoints
– Data Redundancy
– Disaster Recovery
– High Availability
– Network Redundancy
• CENIC Enables PRP
10G Links Between
UCSD, SDSU, & UCI
HPWREN Servers
• Potential Future UCR
CENIC Anchor
UCR
UCI
UCSD
SDSU
Source: Frank Vernon,
Hans Werner Braun HPWREN UCI Antenna Dedicated
June 27, 2017
UC Catalyst Project: Linking Cultural Heritage and Archaeology Datasets
at UCB, UCLA, UCM and UCSD with CAVEkiosks
48 Megapixel CAVEkiosk
UCSD Library
48 Megapixel CAVEkiosk
UCB Library24 Megapixel CAVEkiosk
UCM Library
See Session 2 Talk by
Chris Hoffman
Director: F. Martin Ralph Website: cw3e.ucsd.edu
Big Data Collaboration with:
Source: Scott Sellers, CW3E
Collaboration on Atmospheric Water in the West
Between UC San Diego and UC Irvine
Director, Soroosh Sorooshian, UCSD Website http://chrs.web.uci.edu
Calit2’s FIONA
SDSC’s COMET
Calit2’s FIONA
Pacific Research Platform (10-100 Gb/s)
GPUsGPUs
Complete workflow time: 20 days20 hrs20 Minutes!
UC, Irvine UC, San Diego
Major Speedup in Scientific Work Flow
Using the PRP
Source: Scott Sellers, CW3E
New NSF CHASE-CI Grant Creates a Community Cyberinfrastructure
Adding a Machine Learning Layer Built on Top of the Pacific Research Platform
Caltech
UCB
UCI UCR
UCSD
UCSC
Stanford
MSU
UCM
SDSU
NSF Grant for High Speed “Cloud” of 256 GPUs
For 30 ML Faculty & Their Students at 10 Campuses
for Training AI Algorithms on Big Data
See Session 3 Talk by
Tom DeFanti
We are Now Investigating
How the PRP Prototype Might Be Extended to National-Scale
From the text of the PRP cooperative agreement:
After approximately 18 (or TBD) months, a site visit and comprehensive review of
progress towards meeting project milestones and goals and overall performance and
management processes will take place, including user community relationships,
scientific impacts, and the status of the project as a model for potential future
national-scale, network-aware, data-focused cyberinfrastructure attributes,
approaches, and capabilities.
PRP is Partnering with the Advanced CyberInfrastructure –
Research and Education Facilitators (ACI-REF) NSF Grant to Explore Extension
PRP Connected
ACI-REF has also spawned the 28-member Campus Research Computing consortium (CaRC), funded by the NSF as a Research Coordination Network (RCN).
CaRC is dedicated to sharing best practices, expertise, and resources, enabling the advancement of campus- based research computing activities around the nation.
Jim Bottum, Principal Investigator
ACI-REF
CaRC
See Session 1 Talk by
Tom Cheatham
Expanding to the Global Research Platform
Via CENIC/Pacific Wave, Internet2, and International Links
See
Session 5
on GRP
PRP
PRP’s Current
International
Partners
Korea Shows Distance is Not the Barrier
to Above 5Gb/s Disk-to-Disk Performance
Netherlands
Guam
Australia
Korea
Japan
Our Support:
• US National Science Foundation (NSF) awards
CNS 0821155, CNS-1338192, CNS-1456638, CNS-1730158,
ACI-1540112, & ACI-1541349
• University of California Office of the President CIO
• UCSD Chancellor’s Integrated Digital Infrastructure Program
• UCSD Next Generation Networking initiative
• Calit2 and Calit2 Qualcomm Institute
• CENIC, PacificWave and StarLight
• DOE ESnet