Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan...
-
Upload
lindsey-french -
Category
Documents
-
view
218 -
download
1
Transcript of Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan...
![Page 1: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/1.jpg)
Toward interactive visualization in a distributed workflow
Toward interactive visualization in a distributed workflow
Steven G. Parker
Oscar Barney
Ayla Khan
Thiago Ize
Steven G. Parker
Oscar Barney
Ayla Khan
Thiago Ize
![Page 2: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/2.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Component-Based Architectures
Experience with numerous component-based architectures
• CCA (Parallel, Method Invocation, multi-language)
• SCIRun (Shared memory, Dataflow, C++)• Uintah (Parallel, Method Invocation, C++)• Kepler (Single process + web services,
Generalized dataflow, Java +)• SCIRun2 (Distributed/Parallel, Multi-model,
mutli-language)
![Page 3: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/3.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
DOE Common Component Architecture Project
A CA for large-scale Scientific Computation• Component Characteristics
May be SPMD or multi-threaded parallel objects• Heterogeneity
Parallel platforms to desktops and any language• Local and Remote
Parallel communication for remote parallel interfaces and 0-copy in-process connection
• Dynamic Composition and Integration Hot-swapable components, shared instances
• www.cca-forum.org
Open forum involving DOE labs, Universities, others
![Page 4: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/4.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Uintah• CCA-ish component
architecture (C++ only)• Plus components for
multiphysics structured AMR simulations
• Scales to 2000+ processors
SimulationController
SimulationController
ProblemSpecification
ProblemSpecification
XMLSimulation
(One of Arches, ICE, MPM,
MPMICE, MPMArches
, …)
Simulation(One of Arches, ICE,
MPM, MPMICE,
MPMArches, …)
SchedulerScheduler
Tasks
DataArchiver
DataArchiver Tasks
Callbacks
CallbacksMPI
Assignments
LoadBalancer
LoadBalancer
Configuration
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.QuickTime™ and a
YUV420 codec decompressorare needed to see this picture.
![Page 5: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/5.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SCIRun
![Page 6: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/6.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SCIRun PowerApps: BioImage
![Page 7: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/7.jpg)
2/20/2004 Building a KEPLER Extension Using Ptolemy II
The KEPLER Systemfor Scientific Workflows …
A framework for design, execution and deployment of scientific workflows
Caters specifically to the domain scientist Builds on Ptolemy II Application pull from various projectshttp://kepler.ecoinformatics.org
Slide thanks to: Ilkay Altintas and Efrat JeagerSDSC UCSD
![Page 8: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/8.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Kepler Workflow
![Page 9: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/9.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Component Architecture Design Choices
• Degree of isolation: processes, threads, single address space?
• Mechanism for communication: dataflow, process networks, method invocation
• Synchronization• Programming languages: expressiveness tradeoffs• Data types explicitly supported• Performance requirements• Extra tools required?• Explicit support for parallelism?
Multiple designs for component architectures+ Tailored to application needs- Islands of functionality
![Page 10: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/10.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SCIRun2• SCIRun2 provides a
component model for component models (metacomponents)
• Plug-ins provide support for:• CCA• SCIRun• Vtk• Others
• Components use “native” communication mechanisms to connect to similar components
• Bridges connect models
SCIRun2
![Page 11: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/11.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Meta-components example
Common Framework
Driver Function
Integrator
Function
CORBACCA
Integrator
Driver Bridge
![Page 12: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/12.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Application
![Page 13: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/13.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SDM Requirements
Distributed Workflow• Repetitive• Shared resources• Automatically driven• Coarse-grained (seconds to minute per operation)
Interactive Visualization• Exploratory• Dedicated resources• User-driven• Fine-grained (milliseconds to seconds per operation)
![Page 14: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/14.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Goal
What the user wants• To get work done• Make hard things easy
How to do this1. Combine tools with disparate strengths2. Make them work efficiently3. Focus on interfaces4. Enable consistent user interfaces
![Page 15: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/15.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Utah's Contibution To the SPA Group
• SCIRun can now be controlled from SPA/Kepler workflows Server interface JNI interface
• “Smart” Re-run capability• Provenance framework
![Page 16: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/16.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Kepler Workflow
![Page 17: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/17.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Workflow Requirements and “Wants” We Address
• Seamless access to resources and services• “Smart” re-runs• Data provenance• Reliability and fault-tolerance• Detached execution
From: B. Ludäscher, et al. Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice & Experience, Special Issue on Scientific Workflows, to appear, 2005.
![Page 18: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/18.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SCIRun With SPA/Kepler
• Kepler actor sends requests to a SCIRun server
• Useful for processing batch jobs or iterating through the parameter space of a SCIRun module (actor)
• Requires existing SCIRun network, which the workflow actor will tell SCIRun to load
• JNI interface to SCIRun
![Page 19: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/19.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
SCIRun Server
• Simple TCP/IP server that can be started remotely by Kepler
• Accepts requests from client actor in the workflow and then sends back location of results when it has finished
• Allows for the possibility of remote or/and detached execution of SCIRun
![Page 20: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/20.jpg)
SCIRun and Kepler Dataflow Integration
Automate SCIRun network execution with a Kepler actor driving execution through a JNI interface or a remote connection to a SCIRun server
Incorporate SCIRun computation and visualization with the SPA workflow engine
![Page 21: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/21.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
JNI interface with workflow
![Page 22: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/22.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
What is provenance data?
• In general: steps taken to get a result• Information about computational
experiments or runs of scientific workflows that is needed to reproduce results
• We want to log metadata, steps applied to data, tools used to create data products
• Useful when you want to share/publish results
![Page 23: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/23.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
The Standalone Provenance Frameworkhttp://kepler-project.org/Wiki.jsp?page=KeplerProvenanceFramework
![Page 24: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/24.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
“Smart” re-runs
• Instead of running a workflow from scratch we only re-run parts of the workflow that have not been done before
• Example: we change a parameter downstream and dont want to re-run the actors that lead up to the one with the parameter change
• Especially useful in visualization pipelines and long running workflows
![Page 25: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/25.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Utah and “Smart” Re-runs
• Uses VisTrails’ cache manager algorithm*• Idea is to re-run as little of the network as
possible by combining intermediate results from different workflow runs
• Recreates input to actors that need to be re-fired
* L. Bavoil, et al. VisTrails: Enabling Interactive Multiple-View Visualizations. IEEE Visualization, 2005.
![Page 26: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/26.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
![Page 27: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/27.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
What is needed for “Smart” Re-runs
• We need to keep track of what we have done before
• Specifically we need to know what actors have been given what inputs with what outputs
• Stored provenance data can give us the information we need
![Page 28: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/28.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Other uses for provenance data
• Recreate results• Recover from a system failure• Checkpoint a workflow• Create semantic links
![Page 29: Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.](https://reader036.fdocuments.in/reader036/viewer/2022062423/5697bfd41a28abf838cacf82/html5/thumbnails/29.jpg)
Scientific Computing and Imaging Institute, University of UtahScientific Computing and Imaging Institute, University of Utah
Future work
• Continue work on “Smart” Re-runs system• Help workflow users integrate SCIRun with
their workflows• Get provenance framework checked into
Ptolemy CVS• Work on other provenance issues • Help SCIRun users take advantage of
workflow technology• Develop CCA to Kepler bridging mechanisms