Particle Advection Defined & Usage

Particle advection is a foundational visualization algorithm

• Advecting particles creates integral curves

Particle advection is a foundational visualization algorithm

Courtesy C. Garth, Kaiserslautern

Particle advection is a foundational visualization algorithm

Streamlines in the “fish tank”

Image credit:A. Sanderson, Univ. Utah

Particle advection is a foundational visualization algorithmPoincaré analysis

of a tokamak

Particle advection is a foundational visualization algorithm

Stream surface

around a tokamak

Lagrangian Methods

Visualize manifolds of maximal stretching in a flow, as indicated by dense particles

Finite-Time Lyapunov Exponent (FTLE)

Courtesy Garth

www.vacet.org

Lagrangian Methods Visualize manifolds of maximal

stretching in a flow, as indicated by dense particles

Forward in time: FTLE+ indicates divergence

Backward in time: FTLE- indicates convergence

Courtesy Garth

Particle Advection In A Parallel Setting

Parallel Particle Advection: How To Load Balance?

N particles (P1, P2, … Pn), M Processing Elements/PEs (PE1, …, PEm) (Processing Element = instance of your program)

Each particle takes a variable number of steps, S1, S2, … Sn

Total number of steps is ΣSi We cannot do less work than this (ΣSi)

Goal: Distribute the ΣSi steps over M PEs such that problem finishes in minimal time

Goal: Distribute the ΣSi steps over M PEs such that problem finishes in minimal time

Sounds sort of like a bin-packing problem, but… particles can move from PE to PE (i.e. work

can move from bin to bin) path of particle is data dependent and

unknown a priori (i.e. we don’t know Si beforehand)

big data significantly complicates this picture…. … data may not be readily available, introducing

starvation

Parallel Particle Advection: How To Load Balance?

Advecting particles

Decomposition of large data set into blocks on filesystem

?

What is the right strategy for getting particle and data together?

Strategy: load blocks necessary for advection

Decomposition of large data set into blocks on filesystem

Go to filesystem and read block

Decomposition of large data set into blocks on filesystem

Strategy: load blocks necessary for advection

“Parallelize over Particles” Basic idea: particles are partitioned over

PEs, blocks of data are loaded as needed. Positives:

Indifferent to data size: a serial program can process data of any size

Trivial parallelization (partition particles over processors)

Negative: Redundant I/O (both across PEs and within a

PE) is a significant problem.

“Parallelize over data” strategy:parallelize over blocks and pass particles

PE1 PE2

PE4PE3

“Parallelize over Data” Basic idea: data is partitioned over PEs,

particles are communicated as needed. Positives:

Only load data one time Great for in situ processing

Negative: Starvation!

Contracts are needed to enable these different processing techniques

Parallelize-over-seeds One execution per block Only limited data available at one time

Parallelize-over-data One execution total Entire data set available at one time

Both parallelization schemes have serious flaws.

Two approaches:

Parallelizing Over I/O EfficiencyData Good BadParticles Bad Good

Parallelizeover particles

Parallelizeover dataHybrid algorithms

The master-slave algorithm is an example of a hybrid technique.

Uses “master-slave” model of communication One process has unidirectional control over

other processes Algorithm adapts during runtime to avoid

pitfalls of parallelize-over-data and parallelize-over-particles. Nice property for production visualization tools. (Implemented in VisIt)

D. Pugmire, H. Childs, C. Garth, S. Ahern, G. Weber, “Scalable Computation of

Streamlines on Very Large Datasets.” SC09, Portland, OR, November, 2009

Master-Slave Hybrid Algorithm• Divide processors into groups of N

• Uniformly distribute seed points to each group

Master:- Monitor workload- Make decisions to optimize resource

utilization

Slaves:- Respond to commands from

Master- Report status when work

complete

SlaveSlaveSlave

Master

SlaveSlaveSlave

Master

SlaveSlaveSlave

Master

SlaveSlaveSlave

MasterP0P1P2P3

P4P5P6P7

P8P9P10P11

P12P13P14P15

Master Process Pseudocode

Master(){ while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand()

for cmd in commands SendCommandToSlaves( cmd ) } }}

What are the possible commands?

Commands that can be issued by master

Master Slave

Slave is given a particle that is contained in a block that is already loaded

1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send

OOB = out of bounds

Master Slave

Slave is given a particle and loads the block

Commands that can be issued by master

1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send

OOB = out of bounds

Master Slave

Load

Slave is instructed to load a block. The particle advection in that block can then proceed.

Commands that can be issued by master

1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send

OOB = out of bounds

Master Slave

Send to J

Slave J

Slave is instructed to send the particle to another slave that has loaded the block

Commands that can be issued by master

1. Assign / Loaded Block2. Assign / Unloaded Block3. Handle OOB / Load4. Handle OOB / Send

OOB = out of bounds

Master Process Pseudocode

Master(){ while ( ! done ) { if ( NewStatusFromAnySlave() ) { commands = DetermineMostEfficientCommand()

for cmd in commands SendCommandToSlaves( cmd ) } }}

Driving factors… You don’t want PEs to sit idle

(The problem with parallelize-over-data) You don’t want PEs to spend all their

time doing I/O (The problem with parallelize-over-seeds)

you want to do “least amount” of I/O that will prevent “excessive” idleness.

The heuristics behind the heuristics Rules:

If no PE has the block, then load it! If a PE does have the block…

If it is not busy, then give it the particle If it is busy, then wait “a while”. If you’ve waited a “long time”, then load it

yourself.

Master-slave in action

S0

S0

S1

S1S2

S3

S4

Iteration

Action

0 S0 reads B0,S3 reads B1

1 S1 passes points to S0,S4 passes points to S3,S2 reads B0

0: Read

0: Read1: Pass

1: Pass1: Read

Algorithm Test Cases

- Core collapse supernova simulation- Magnetic confinement fusion simulation- Hydraulic flow simulation

Workload distribution in parallelize-over-data

Starvation

Workload distribution in parallelize-over-particles

Too much I/O

Workload distribution in master-slave algorithm

Just right

ParticlesData Hybrid

Workload distribution in supernova simulation

Parallelization by:

Colored by PE doing integration

Astrophysics Test Case: Total time to compute 20,000 Streamlines

Sec

onds

Sec

onds

Number of procs Number of procs

Uniform Seeding

Non-uniform Seeding

Data HybridParticles

Astrophysics Test Case: Number of blocks loaded

Blo

cks

load

ed

Blo

cks

load

ed

Number of procs Number of procs

Data Hybrid

Uniform Seeding

Non-uniform Seeding

Particles

Summary: Master-Slave Algorithm

First ever attempt at a hybrid algorithm for particle advection

Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over-particles. Nice property for production visualization tools.

Implemented inside VisIt visualization and analysis package.

Benefits of an Extended Memory Hierarchy

Emerging I/O Configuration

Emerging I/O ConfigurationCommon I/O Configuration

Simulation Codes

SimulationCode

GlobalFile System

Node

Checkpoint/Restart

SimulationCode

Node

Checkpoint/Restart

SSD

Parallelize-over-Seeds Algorithm

IntegrationI/O

Memory Cache

GlobalFile System

Streamlines

Node

Parallelize-over-Seeds Algorithm with the Extended Memory Hierarchy

IntegrationI/O

Memory Cache

GlobalFile System

Streamlines

Local File SystemNode

I/O Configurations

A Real World Particle Advection System

Goal Efficient code for a variety of particle

advection based techniques Cognizant of use cases with >>10K

particles. Need handling of every particle, every

evaluation to be efficient. Want to support diverse flow techniques:

flexibility/extensibility is key. Fit within data flow network design (i.e. a

filter)

Motivating examples of system FTLE Stream surfaces Streamline Dynamical Systems (e.g. Poincaré Maps) Statistics based analysis + more

Design PICS filter: parallel integral curve system Execution:

Instantiate particles at seed locations Step particles to form integral curves

Analysis performed at each step Termination criteria evaluated for each step

When all integral curves have completed, create final output

Design Five major types of extensibility:

How to parallelize? How do you evaluate velocity field? How do you advect particles? Initial particle locations? How do you analyze the particle paths?

Inheritance hierarchy

avtPICSFilter

Streamline Filter

Your derived type of PICS

filter

avtIntegralCurve

avtStreamlineIC

Your derived type of integral

curve

We disliked the “matching inheritance” scheme, but this achieved all of our design goals cleanly.

#1: How to parallelize?

avtICAlgorithm

avtParDomIC-Algorithm

(parallel over data)

avtSerialIC-Algorithm

(parallel over seeds)

avtMasterSlave-

ICAlgorithm

#2: Evaluating velocity field

avtIVPField

avtIVPVTKField

avtIVPVTK- TimeVarying-

FieldavtIVPM3DC1

Field

avtIVP-<YOUR>HigherOrder-Field

IVP = initial value problem

#3: How do you advect particles?

avtIVPSolver

avtIVPDopri5 avtIVPEuler avtIVPLeapfrog

avtIVP-M3DC1Integrato

r

IVP = initial value problem

avtIVPAdams-Bashforth

#4: Initial particle locations avtPICSFilter::GetInitialLocations() = 0;

#5: How do you analyze particle path? avtIntegralCurve::AnalyzeStep() = 0;

All AnalyzeStep will evaluate termination criteria avtPICSFilter::CreateIntegralCurveOutput(

std::vector<avtIntegralCurve*> &) = 0;

Examples: Streamline: store location and scalars for current

step in data members Poincare: store location for current step in data

members FTLE: only store location of final step, no-op for

preceding steps NOTE: these derived types create very

different types of outputs.

Putting it all togetherPICS Filter

avtICAlgorithmavtIVPSolv

er

avtIVPFieldVector<

avtIntegral-Curve>

Integral curves sent to other processors with some derived types of avtICAlgorithm.

::CreateInitialLocations() = 0;

::AnalyzeStep() = 0;