A Framework for Particle Advection for Very Large Data
description
Transcript of A Framework for Particle Advection for Very Large Data
![Page 1: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/1.jpg)
A Framework for Particle Advection for
Very Large DataHank Childs, LBNL/UCDavisDavid Pugmire, ORNL
Christoph Garth, Kaiserslautern
David Camp, LBNL/UCDavisSean Ahern, ORNL
Gunther Weber, LBNLAllen Sanderson, Univ. of
Utah
![Page 2: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/2.jpg)
Particle advection is a foundational visualization algorithm
• Advecting particles creates integral curves
![Page 3: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/3.jpg)
Particle advection is the duct tape of the visualization world
Advecting particles is essential to understanding flow and other
phenomena (e.g. magnetic fields)!
![Page 4: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/4.jpg)
Outline A general system for particle-advection
based analysis Efficient advection of particles
![Page 5: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/5.jpg)
Outline A general system for particle-advection
based analysis Efficient advection of particles
![Page 6: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/6.jpg)
Goal Efficient code for a variety of particle
advection workloads and techniques Cognizant of use cases from 1 particle to
>>10K particles. Need handling of every particle, every
evaluation to be efficient. Want to support many diverse flow
techniques: flexibility/extensibility is key.
Fit within data flow network design (i.e. a filter)
![Page 7: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/7.jpg)
Design PICS filter: parallel integral curve system Execution:
Instantiate particles at seed locations Step particles to form integral curves
Analysis performed at each step Termination criteria evaluated for each step
When all integral curves have completed, create final output
![Page 8: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/8.jpg)
Design Five major types of extensibility:
How to parallelize? How do you evaluate velocity field? How do you advect particles? Initial particle locations? How do you analyze the particle paths?
![Page 9: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/9.jpg)
Inheritance hierarchy
avtPICSFilter
Streamline Filter
Your derived type of PICS
filter
avtIntegralCurve
avtStreamlineIC
Your derived type of integral
curve
We disliked the “matching inheritance” scheme, but this achieved all of our design goals cleanly.
![Page 10: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/10.jpg)
#1: How to parallelize?
avtICAlgorithm
avtParDomIC-Algorithm
(parallel over data)
avtSerialIC-Algorithm
(parallel over seeds)
avtMasterSlave-
ICAlgorithm
![Page 11: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/11.jpg)
#2: Evaluating velocity field
avtIVPField
avtIVPVTKField
avtIVPVTK- TimeVarying-
FieldavtIVPM3DC1
Field
avtIVP-<YOUR>HigherOrder-Field
IVP = initial value problem
![Page 12: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/12.jpg)
#3: How do you advect particles?
avtIVPSolver
avtIVPDopri5 avtIVPEuler avtIVPLeapfrog
avtIVP-M3DC1Integrato
r
IVP = initial value problem
avtIVPAdams-Bashforth
![Page 13: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/13.jpg)
#4: Initial particle locations avtPICSFilter::GetInitialLocations() = 0;
![Page 14: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/14.jpg)
#5: How do you analyze particle path? avtIntegralCurve::AnalyzeStep() = 0;
All AnalyzeStep will evaluate termination criteria avtPICSFilter::CreateIntegralCurveOutput(
std::vector<avtIntegralCurve*> &) = 0;
Examples: Streamline: store location and scalars for current
step in data members Poincare: store location for current step in data
members FTLE: only store location of final step, no-op for
preceding steps NOTE: these derived types create very
different types of outputs.
![Page 15: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/15.jpg)
Putting it all togetherPICS Filter
avtICAlgorithmavtIVPSolv
er
avtIVPFieldVector<
avtIntegral-Curve>
Integral curves sent to other processors with some derived types of avtICAlgorithm.
::CreateInitialLocations() = 0;
::AnalyzeStep() = 0;
http://www.visitusers.org/index.php?title=Pics_dev
![Page 16: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/16.jpg)
Outline A general system for particle-advection
based analysis Efficient advection of particles
![Page 17: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/17.jpg)
Advecting particles
Decomposition of large data set into blocks on filesystem
?
What is the right strategy for getting particle and data together?
![Page 18: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/18.jpg)
Strategy: load blocks necessary for advection
Decomposition of large data set into blocks on filesystem
Go to filesystem and read block
![Page 19: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/19.jpg)
Decomposition of large data set into blocks on filesystem
Strategy: load blocks necessary for advection
![Page 20: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/20.jpg)
“Parallelize over Particles” Basic idea: particles are partitioned over
PEs, blocks of data are loaded as needed. Positives:
Indifferent to data size Trivial parallelization (partition particles over
processors) Negative:
Redundant I/O (both across PEs and within a PE) is a significant problem.
![Page 21: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/21.jpg)
“Parallelize over data” strategy:parallelize over blocks and pass particles
PE1 PE2
PE4PE3
![Page 22: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/22.jpg)
“Parallelize over Data” Basic idea: data is partitioned over PEs,
particles are communicated as needed. Positives:
Only load data one time Negative:
Starvation!
![Page 23: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/23.jpg)
Contracts are needed to enable these different processing techniques
Parallelize-over-seeds One execution per block Only limited data available at one time
Parallelize-over-data One execution total Entire data set available at one time
![Page 24: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/24.jpg)
Both parallelization schemes have serious flaws.
Two approaches:
Parallelizing Over I/O EfficiencyData Good BadParticles Bad Good
Parallelizeover particles
Parallelizeover dataHybrid algorithms
![Page 25: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/25.jpg)
The master-slave algorithm is an example of a hybrid technique.
Uses “master-slave” model of communication One process has unidirectional control over
other processes Algorithm adapts during runtime to avoid
pitfalls of parallelize-over-data and parallelize-over-particles. Nice property for production visualization tools. (Implemented in VisIt)
D. Pugmire, H. Childs, C. Garth, S. Ahern, G. Weber, “Scalable Computation of
Streamlines on Very Large Datasets.” SC09, Portland, OR, November, 2009
![Page 26: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/26.jpg)
Master-Slave Hybrid Algorithm• Divide PEs into groups of N
• Uniformly distribute seed points to each group
Master:- Monitor workload- Make decisions to optimize resource
utilization
Slaves:- Respond to commands from
Master- Report status when work
complete
SlaveSlaveSlave
Master
SlaveSlaveSlave
Master
SlaveSlaveSlave
Master
SlaveSlaveSlave
MasterP0P1P2P3
P4P5P6P7
P8P9P10P11
P12P13P14P15
![Page 27: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/27.jpg)
Master Process Pseudocode
Master(){ while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand()
for cmd in commands SendCommandToSlaves( cmd ) } }}
What are the possible commands?
![Page 28: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/28.jpg)
Commands that can be issued by master
Master Slave
Slave is given a particle that is contained in a block that is already loaded
1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send
OOB = out of bounds
![Page 29: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/29.jpg)
Master Slave
Slave is given a particle and loads the block
Commands that can be issued by master
1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send
OOB = out of bounds
![Page 30: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/30.jpg)
Master Slave
Load
Slave is instructed to load a block. The particle advection in that block can then proceed.
Commands that can be issued by master
1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send
OOB = out of bounds
![Page 31: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/31.jpg)
Master Slave
Send to J
Slave J
Slave is instructed to send the particle to another slave that has loaded the block
Commands that can be issued by master
1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send
OOB = out of bounds
![Page 32: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/32.jpg)
Master Process Pseudocode
Master(){ while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand()
for cmd in commands SendCommandToSlaves( cmd ) } }}
![Page 33: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/33.jpg)
Driving factors… You don’t want PEs to sit idle
(The problem with parallelize-over-data) You don’t want PEs to spend all their
time doing I/O (The problem with parallelize-over-
particles) you want to do “least amount” of I/O
that will prevent “excessive” idleness.
![Page 34: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/34.jpg)
Heuristics If no Slave has the block, then load that
block! If some Slave does have the block…
If that Slave is not busy, then have it process the particle.
If that Slave is busy, then wait “a while.” If you’ve waited a “long time,” have another
Slave load the block and process the particle.
![Page 35: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/35.jpg)
Master-slave in action
S0
S0
S1
S1S2
S3
S4
Iteration
Action
0 S0 reads B0,S3 reads B1
1 S1 passes points to S0,S4 passes points to S3,S2 reads B0
0: Read
0: Read1: Pass
1: Pass1: Read
![Page 36: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/36.jpg)
Algorithm Test Cases
- Core collapse supernova simulation- Magnetic confinement fusion simulation- Hydraulic flow simulation
![Page 37: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/37.jpg)
Workload distribution in parallelize-over-data
Starvation
![Page 38: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/38.jpg)
Workload distribution in parallelize-over-particles
Too much I/O
![Page 39: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/39.jpg)
Workload distribution in master-slave algorithm
Just right
![Page 40: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/40.jpg)
ParticlesData Hybrid
Workload distribution in supernova simulation
Parallelization by:
Colored by PE doing integration
![Page 41: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/41.jpg)
Astrophysics Test Case: Total time to compute 20,000 Streamlines
Sec
onds
Sec
onds
Number of procs Number of procs
Uniform Seeding
Non-uniform Seeding
Data HybridParticles
![Page 42: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/42.jpg)
Astrophysics Test Case: Number of blocks loaded
Blo
cks
load
ed
Blo
cks
load
ed
Number of procs Number of procs
Data Hybrid
Uniform Seeding
Non-uniform Seeding
Particles
![Page 43: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/43.jpg)
Summary: Master-Slave Algorithm
First ever attempt at a hybrid algorithm for particle advection
Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over-particles. Nice property for production visualization tools.
Implemented inside VisIt visualization and analysis package.
![Page 44: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/44.jpg)
Final thoughts… Summary:
Particle advection is important for understanding flow and efficiently parallelizing this computation is difficult.
We have developed a freely available system for doing this analysis for large data.
Documentation: (PICS)
http://www.visitusers.org/index.php?title=Pics_dev
(VisIt) http://www.llnl.gov/visit Future work:
UI extensions, including Python Additional analysis techniques (FTLE & more)
![Page 45: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/45.jpg)
Acknowledgements Funding: This work was supported by the
Director, Office of Science, Office and Advanced Scientific Computing Research, of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 through the Scientific Discovery through Advanced Computing (SciDAC) program's Visualization and Analytics Center for Enabling Technologies (VACET).
Program Manager: Lucy Nowell Master-Slave Algorithm: Dave Pugmire (ORNL),
Hank Childs (LBNL/UCD), Christoph Garth (Kaiserslautern), Sean Ahern (ORNL), and Gunther Weber (LBNL)
PICS framework: Hank Childs (LBNL/UCD), Dave Pugmire (ORNL), Christoph Garth (Kaiserslautern), David Camp (LBNL/UCD), Allen Sanderson (Univ of Utah)
![Page 46: A Framework for Particle Advection for Very Large Data](https://reader030.fdocuments.in/reader030/viewer/2022012917/56816276550346895dd2e786/html5/thumbnails/46.jpg)
A Framework for Particle Advection for
Very Large DataHank Childs, LBNL/UCDavisDavid Pugmire, ORNL
Christoph Garth, Kaiserslautern
David Camp, LBNL/UCDavisSean Ahern, ORNL
Gunther Weber, LBNLAllen Sanderson, Univ. of
Utah
Thank you!!