Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.
-
Upload
myron-hood -
Category
Documents
-
view
215 -
download
0
Transcript of Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.
![Page 1: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/1.jpg)
Development Timelines
Ken Kennedy
Andrew ChienKeith Cooper
Ian FosterJohn Mellor-Curmmey
Dan Reed
![Page 2: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/2.jpg)
What We Have Today
• Prototype Testbeds with middleware
• Prototype of execution components in ScaLAPACK and Cactus
• Design for the execution environment—Implementation of resource specification (AART)—Interfaces for Scheduler/Resource Negotiator
– Prototype resource scheduler (Dail)– Prototype renegotiator (Sievert)
—Prototype Contract Monitoring System
• Grid ready libraries with performance models
• Tools for extracting information from program executables—For performance estimation on single nodes
• DSL design for Signal/Image Processing
![Page 3: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/3.jpg)
What We Will Have on CGrADS Day One
• Prototype Execution System for Heterogeneous Clusters—Prototype Scheduler/Resource Negotiator—Binder—Run-time system—Contract monitor (working together)
• ScaLAPACK re-implemented using the prototype
• Cactus using generic resource selector
• Prototype automatic performance modeling for black boxes
• Script-based application composition (without optimization)
• Testbeds—Microgrid running ScaLAPACK and Cactus—Integrated NWS and prediction in MacroGrid and
configuration tools
![Page 4: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/4.jpg)
Currently Targeted Application Milestones
• Three codes concurrently, each engaged for two-three years
• 2002—Cactus: traditional PDE solver, aggressive application scenarios—CAPS: dynamic data acquisition and real-time data ingest—ChemEng Workbench: application service scenarios prototypes
• 2003—Cactus: by now transitioned to operational use by application
group—CAPS: adaptive execution for high-speed prediction—ChemEng Workbench: application service scenarios operational—CMS/GriPhyN: query estimation and dynamic scheduling—BIRN-like distributed bioscience: emergent behavior issues
• 2004—CAPS: by now transitioned to operational use by application group—CMS/GriPhyN: large-scale experimentation in production settings—NEES: application service and real-time data analysis scenarios
![Page 5: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/5.jpg)
NCSA Linux Cluster
5) Secondary reports complete to master
Master Condor job running at
Caltech
7) GridFTP fetches data from UniTree
NCSA UniTree - GridFTP-enabled FTP server
4) 100 data files transferred via GridFTP, ~ 1 GB each
Secondary Condor job on
WI pool
3) 100 Monte Carlo jobs on Wisconsin Condor pool
2) Launch secondary job on WI pool; input files via Globus GASS
Caltech workstation
6) Master starts reconstruction jobs via Globus jobmanager on cluster
8) Processed objectivity database stored to UniTree
9) Reconstruction job reports complete to master
CMS Data Reconstruction Example
Scott Koranda, Miron Livny, others
![Page 6: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/6.jpg)
Program Preparation System Milestones
2002—Preliminary automated support for performance models
(black boxes)—Binder includes local optimization, inserts probes and
actuators—Prototype DSL for signal processing—Evaluate original COP design
2003—Binder support for global optimization—Experiment with contract monitoring/reporting in
applications—Evaluate and extend DSL support for signal processing
2004—First dynamic optimizer prototype, plan for retargeting
Binder—Initial telescoping language prototype based on experience
with signal processing DSL—Whole Program Compiler generates initial COPs
![Page 7: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/7.jpg)
Program Execution System Milestones
• 2002—Virtual organization management tools—Resource selector and application manager prototypes—Temporal contract violation specification—Rescheduling models including distribution costs & info
quality
• 2003—Integrated resource monitoring and prediction prototype—Resource selector and application manager tools—Composable contract specification and tools—Reconfigurable object program specification—Scheduling models for highly parallel and data Grid
applications
• 2004—Enhanced application and resource measurement
infrastructure—Enhanced resource scheduling infrastructure with adv.
resv., etc.—Performance economics for global resource scheduling
![Page 8: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/8.jpg)
Execution Environment Milestones
GrADSoftv1
GrADSoftv2
GrADSoftv3
Dist. Comp.:Cactus
App Server:ChemEng
Real-time:CAPS
Data Grid:CMS
Real-time:CAPS
Data Grid:CMS
AutoPilotGlobus
MPICH-GNWS
NetSolveAppLeS
Cactus
Virt. OrgMgmt Tools
ResourceSelector
ApplnManager
ContractViolation
ChemEng
MacroGridMicroGrid
Year 1-2 Year 2-3 Year 3-4
ComposableContracts
SchedulingModels
ReconfigurableObject
Programs
ResourceMonitoring
App Server:NEES
PerformanceEconomies
EnhancedResource Mgt
ResourceMonitoring
![Page 9: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/9.jpg)
Software Development Process
Andrew A. ChienDepartment of Computer Science and Engineering
University of California, San Diego
http://hipersoft.rice.edu/stc_site_visit/talks/SWProcess.pdf
![Page 10: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/10.jpg)
Software Development Process
• Development Efficiency versus Robust Output—Key is choosing where to live on this spectrum
• CGrADS Objectives—Ground breaking research—Rapid development of functionality and experimentation—Difficult to know what will become widely used
– Hardening of selected components
![Page 11: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/11.jpg)
Software Development Approach
• Experimental development with lightweight process
• Core development with close to full process
• Targeted hardening of what becomes core—Enable researchers inside and outside CGrADS to build on
core—Software output not supported as a product
![Page 12: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/12.jpg)
Software Development Process
• Project Software Manager defines and enforces process; drives progress
• Infrastructure—Revision control system (CVS,SourceSafe,Clearcase)
– Code, documents—Defect tracking system (ClearQuest)—Design, documentation, coding guidelines—Software infrastructure
• Process—Lightweight Process—Industrial Process
![Page 13: Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.](https://reader036.fdocuments.in/reader036/viewer/2022082611/56649ed35503460f94be2f01/html5/thumbnails/13.jpg)
Two Flavors of Process
• “Lightweight” Process (Research Strength)—Requirements and Design; Review—Implementation and Test; Review—Integration; Test—Iterative Improvement
• Full Process (Industrial Strength)—Requirements; Review—Design; Review—Implementation; Test and Review—Integration; Test—Iterative Improvement