ADaPT: An Event-Passing Protocol for Reducing Delivery Costs in Scatter-Gather Parallel Processes

David M. Woollard†§

Nenad Medvidovic†

Walter M. Yamada§

Theodore W. Berger§

†Center for Software Engineering

University of Southern CaliforniaLos Angeles, California 90089-0781

§Laboratory for Neural Dynamics

University of Southern CaliforniaLos Angeles, California 90089-1451

Outline

Motivation

Established Techniques

ADaPT

Performance Comparison

Conclusions

ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in

Scatter-Gather Parallel Processes

Motivation

• A computational-science section of the Center for Neural Engineering

• Part of a National Science Foundation engineering research center dedicated to biomimetic microelectronic systems

• Combines computational electrophysiology, engineering, pharmacology, and other disciplines

• Integrates empirically-measured, realistic, and biologically-inspired synaptic models for the purposeof temporal signals processing

Motivation

What is the Laboratory for Neural Dynamics?

• Biologically-inspired rather than realistic• Computationally-complex and non-linear• Signals processing application was originally a

proof of concept• Now a synergistic field for the Center

Motivation

The Dynamic Synapse

Presynapse PostsynapseNa+

Ca2+Ca2+

Action Potential input Glutamate release Action Potential output

threshold threshold

Figure 1: Electro-chemical synaptic transmission

Neuron

Synaptic Potential Summation

Feedback

• Classical NN structure

• Increased synaptic functionality

• Parameter trainingvia genetic algorithms

Motivation

Dynamic Synapse Neural Networks

K length filter b

ank

Array of K input neurons

12

3

K

APAP

AP

AP

3xK Pre-synaptic Matrix

3xK Post-synaptic Matrix

LAYER 1

LAYER 2

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

EPSP

AP

AP

AP

Feedback Modulation

AA

Output

Neurons

Microphone Array

Captured Sound

Figure 2: 3xK 2-Layer DSNN Single Word Classifier.

Outline

Motivation

Established Techniques

ADaPT

Performance Comparison

Conclusions

ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in

Scatter-Gather Parallel Processes

Motivation

• Naïve Approach• Multiple Sequential

Scatter-Gathers• With uniform computation time,

exhibits decent parallelism• With variable computation times,

significant idle time

Established Techniques

Scatter-Gather

Worke

r 1

Worke

r 2

Worke

r 3

Worke

r 4

Worke

r 5

Worke

r 6

Worke

r n -

2 Worke

r n -

1

Worke

r n

Maste

r

Worke

r 1

Worke

r 2

Worke

r 3

Worke

r 4

Worke

r 5

Worke

r 6

Worke

r n -

2 Worke

r n -

1

Worke

r n

Maste

r

Worke

r 1

Worke

r 2

Worke

r 3

Worke

r 4

Worke

r 5

Worke

r 6

Worke

r n -

2 Worke

r n -

1

Worke

r n

Maste

rM

aste

r

Co

mp

uta

tion

T

ime

Figure 3: Multi-phase evaluation of 3n genomes by n workers using naïve scatter-gathering.

• Asynchronous scattering• Reduced idle time for workers• Closer to optimal time to solution• Dynamic allocation of resources

– More difficult

Established Techniques

A More Efficient Mapping

Maste

rCo

mp

uta

tion

T

ime

Worke

r 1

Worke

r 2

Worke

r 3

Worke

r 4

Worke

r 5

Worke

r 6

Worke

r n -

2 Worke

r n -

1

Worke

r n

Maste

r

Figure 4: Multi-phase evaluation of 3n genomes by n workers using a more efficient mapping.

Outline

Motivation

Established Techniques

ADaPT

Performance Comparison

Conclusions

ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in

Scatter-Gather Parallel Processes

Motivation

• Publish/Subscribe– Worker-centric, i.e. processes subscribe to the master– Data is transported (published) to workers as events– Unsubscription is possible

• Two-phase adaptive protocol– Learning phase: request-reply, monitoring of time

between requests– Aggressive phase: events are pushed to workers at regular

intervals

ADaPT

Adaptive Data-parallel Publish/Subscribe Transport Protocol

Outline

Motivation

Established Techniques

ADaPT

Performance Comparison

Conclusions

ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in

Scatter-Gather Parallel Processes

Motivation

• Two protocols– Aggressive & Conservative

• Scatter/Gathers in most implementations use conservative protocols

• Analysis due to Gropp, et. al.

Performance Comparison

Message-passing costs for MPI scatters

C(MPI Scatter) = (# pop.)[3s + r(n+3e)]

Where n = event payloade = enveloper = network bandwidths = latency

Equation 1: Computation time cost in of scatters In MPI.

Performance Comparison

• Our assumption is a normally distributed population of compute times

• An ideal ordering of computations would be sortedby compute time

• How much idle time is present?

Computational Costs for Multiple scatters in MPI

C(Computation) = (# pop.)(avg. compute time) + (# workers)(avg. compute time)Equation 2: Computation time of a normally-distributed population using scatters in MPI.

Figure 5: Graph of sorted compute times of anormal distribution illustrating idle time.

• Three different costs of event-passing in ADaPT:– Subscription– Learning Phase– Aggressive Phase

Performance Comparison

Message-Passing costs for ADaPT

C(subscription) = (# workers) x [s + re]

C(learning) = (# samples) x [2s + r(n+2e)]

C(aggressive) = (# pop - # samples) x[s + r(n+e)]

Note: we assume control events to be of size e

Equation 3: Event-passing costs of ADaPT.

• An unsubscribe occurs when a worker’s event buffer is in danger of overflowing

• With ADaPT, an overflowoccurs when a worker receivesm-1 events triggering computetimes greater than the estimatedaverage (assuming a worker buffers m events)

• Conservatively, we have decidedthat workers should clear theirbuffers before resubscribing

Performance Comparison

Unsubscribe Costs for ADaPT

- We used a Monte Carlo simulation (details in paper) to determine E,the % pop with compute times > than the estimated mean given error as a function of % pop. sampled

P(unsubscribe) =

E*Pop C m-1

Pop C m-1

C(unsubscribe) = P(unsubscribe) x [2(s+re) + (m-1)(avg. compute+ Δ)]

Equation 4: Costs of worker unsubscription in ADaPT.

Performance Comparison

• Which protocol is more appropriate?

• For simplicity of comparisonwe will drop the latency termand assume the number of samples to be equal to thenumber of workers– i.e. each worker’s first

computation is monitored

Analysis (# pop - # samples)2re +

(# samples)(avg. compute time) > (# pop / m) x P(unsubscribe) x[2re + (m-1)(avg. compute time)]

Equation 5: Cost comparison of MPI vs. ADaPT.

Figure 6: Graph of inequality in Equation 5.

Outline

Motivation

Established Techniques

ADaPT

Performance Comparison

Conclusions

ADaPT: An Event-Passing Protocol For Reducing Delivery Costs in

Scatter-Gather Parallel Processes

Motivation

• ADaPT is useful when multiple scattering of data must occur due to natural aggregation– An example is the training of the DSNN using genetic

algorithms

• Worker-centric approach for reduced processor idle time

• Unsubscription is expensive but can be avoided withgreater event-buffering capabilities

• ADaPT exploits an event pattern which emerges fromthe application of a well-known architectural pattern

Conclusions

What have we shown?

Thank You