Chapter ∞: Parallel Algorithms
description
Transcript of Chapter ∞: Parallel Algorithms
Chapter ∞: Parallel Algorithms• PCAM• SIMD & MIMD
CS 340 Page 1
• Designing Parallel Programs
CS 340 Page 2
PCAMWhen designing algorithms for systems of parallel processors, two primary problems must be addressed:• Avoid processor idle time by balancing the
processing load between all of the processors.• Avoid time-consuming communication between
processors by allocating interdependent tasks to the same processor.
Problem
Partition
Communicate
Agglomerate
Map
One common approach: PCAM• Partition: Split the problem into a large number of
small tasks to expose opportunities for parallel execution.• Communicate: Identify the data flow that is required between separate tasks.
• Agglomerate: Combine tasks to minimize communication and balance loads; evaluate data replication pros and cons.• Map: Assign tasks to processors in a manner that improves concurrency while maintaining locality.
CS 340 Page 3
PCAM Example: VLSI FloorplanAssume that you are designing a microchip with several layout restrictions:
Possibilities (among many others):
• The northwest corner of the chip must contain a 55 component of type A.
• Right next to the type-A component on the north side of the chip, a 24 square unit component of type B is needed.• Adjacent to the type-B component’s east or south side, a 40-square unit component of type C is needed.
The idea is to minimize the overall area of the chip.
CS 340 Page 4
VLSI Floorplan: Algorithm OutlineThe common sequential approach for this problem is a tree-based, depth-first technique called branch-and-bound, in which subtrees are pruned whenever it becomes clear that they become too costly.
Chip Size: 25
Chip Size: 64 Chip Size: 65Chip Size: 54 Chip Size: 55 Chip Size: 85 Chip Size: 84
Chip Size: 114
Chip Size: 143
Chip Size: 174
Chip Size: 200
Chip Size: 144
Chip Size: 112
Chip Size: 130
Chip Size: 234
Chip Size: 140
Chip Size: 150
Chip Size: 102
Chip Size: 220
Component A Placement
Component B Placement:
Six Possibilities
Component C Placement (B1
Version): Twelve Possibilities
CS 340 Page 5
Parallelizing VLSI Floorplan: PartitioningThere is no obvious data structure that could be used to perform a decomposition of this problem’s domain into components that could be mapped to separate processors.
A fine-grained functional decomposition is therefore needed, where the exploration of each search tree node is handled by a separate task.This means that new tasks will be created in a wavefront as the search progresses down the search tree, which will be explored in a breadth-first fashion. Notice that only tasks on the wavefront will be able to execute concurrently.
Chip Size: 25
Chip Size: 64 Chip Size: 65Chip Size: 54 Chip Size: 55 Chip Size: 85 Chip Size: 84
Chip Size: 114
Chip Size: 143Chip Size: 174
Chip Size: 200
Chip Size: 144
Chip Size: 112
Chip Size: 130
Chip Size: 234
Chip Size: 140
Chip Size: 150 Chip Size: 102
Chip Size: 220
CS 340 Page 6
Parallelizing VLSI Floorplan: CommunicationIn a parallel implementation of simple search, tasks can execute independently and need communicate only to report solutions.The parallel algorithm for this problem will also need to keep track of the bounding value (i.e., the smallest chip area found so far), which must be accessed by every task. One possibility would be to encapsulate the bounding value maintenance in a single centralized task with which the other tasks will communicate.
Chip Size: 25
Chip Size: 64 Chip Size: 65Chip Size: 54 Chip Size: 55 Chip Size: 85 Chip Size: 84
Chip Size: 114
Chip Size: 143Chip Size: 174
Chip Size: 200
Chip Size: 144
Chip Size: 112
Chip Size: 130
Chip Size: 234
Chip Size: 140
Chip Size: 150 Chip Size: 102
Chip Size: 220
This approach is inherently unscalable, since the processor handling the centralized task can only service requests from the other tasks at a particular rate, thus bounding the number of tasks that can execute concurrently.
CS 340 Page 7
Parallelizing VLSI Floorplan: AgglomerationPractically speaking, parallelism will have to be constrained by the number of processors being employed on the program.For this problem, an obvious agglomeration would be to split the breadth first search into tasks at each node until a certain level of the tree is reached, at which point each subtree can be handled as a sequential depth-first search by its assigned processor.
Of course, this step is more significant if the number of components being placed on the chip, and the number of options for placing them, is substantially increased.
Chip Size: 25
Chip Size: 64
Chip Size: 65
Chip Size: 54
Chip Size: 55
Chip Size: 85
Chip Size: 84
B1 Subtre
e
B2 Subtre
e
B3 Subtre
e
B4 Subtre
e
B5 Subtre
e
B6 Subtre
e
CS 340 Page 8
Parallelizing VLSI Floorplan: MappingOne priority when mapping tasks to processors is determining how to avoid lengthy idle periods for the processors.For this problem, one processor can be assigned a supervisory role by handling the root node and generating tasks associated with the subtrees.
The agglomeration step ensured that the tasks are handled in a depth-first manner, which facilitates the pruning of unnecessary tasks.
Chip Size: 25
Chip Size: 64
Chip Size: 65
Chip Size: 54
Chip Size: 55
Chip Size: 85
Chip Size: 84
B1 Subtre
e
B2 Subtre
e
B3 Subtre
e
B4 Subtre
e
B5 Subtre
e
B6 Subtre
e
The other nodes maintain a queue of assigned tasks and, whenever their queues are depleted, they request tasks from other nodes, which facilitates load balancing.
In order to keep the pruning rate as high as possible, processors keep tasks that are far from the root (i.e., more likely to cause pruning soon) and hand tasks that are near the root to idle processors requesting tasks.
CS 340 Page 9
SIMDParallel processing on SIMD (Single Instruction, Multiple Data) machines is accomplished by having all processors perform the same instruction on different data in a vectorized manner.For applications that require huge blocks of data upon which the same set of operations must be performed, both the data retrieval and the data processing can experience significant performance benefits.
Processor 1
Processor 2
Processor 3
Processor 4
Instruction:
Retrieve Data
Block A[ ] = A[0…
99]Instruction:
Retrieve Data Block A[ ] = A[100…
199]Instruction:
Retrieve Data Block
A[ ] = A[200…
299]Instruction:
Retrieve Data Block A[ ] = A[300…
399]
Instruction: Retrieve
Data Block B[ ] = B[0…
99]Instruction:
Retrieve Data Block
B[ ] = B[100…
199]Instruction:
Retrieve Data Block
B[ ] = B[200…
299]
Instruction: Retrieve Data Block B[ ] = B[300…399]
Instruction: A[0] =
A[0] * B[0]
Instruction: A[0] =
A[0] * B[0]
Instruction: A[0] =
A[0] * B[0]
Instruction: A[0] =
A[0] * B[0]
Instruction: A[19] =
A[3] * B[6]
Instruction: A[19] =
A[3] * B[6]
Instruction: A[19]
=A[3] * B[6]
Instruction: A[19] =
A[3] * B[6]
CS 340 Page 10
SIMD InefficiencyParallelism in a SIMD machine does not necessarily eliminate all inefficiency and waste.For example, consider what happens when multiple processors attempt to execute the conditional at right with their respective data:Process
or 1
Processor 2
Processor 3
Processor 4
Compare u1
(10) tov1 (20)
Compare u2
(16) tov2 (12)
Compare u3
(14) tov3 (21)
Compare u4
(15) tov4 (13)
Multiply x1 and
y1
Multiply x2 and
y2
Multiply x3 and
y3
Multiply x4 and
y4
if (u < v) z = x * y;else z = x + y;
Add x1 and y1
Add x2 and y2
Add x3 and y3
Add x4 and y4
Store product
in z1
Store sumin z2
Store product
in z3
Store sumin z4
Because the processors handle the instructions in lockstep, both condition cases are executed, regardless of the condition’s evaluation.
CS 340 Page 11
SIMD ApplicationsThe limitations of the SIMD approach (i.e., large data sets to which uniform instructions are applied in lockstep fashion) have restricted its applicability to such areas as graphics (e.g., animation, games) and video processing (e.g., image enhancement, video compression, format conversion).Note that divide-
and-conquer algorithms for sequential machines tend to have linear speedups on SIMD machines.Mergesort, for example, takes O() time on a p-processor MIMD machine, which translates to O(n) time on an n-processor machine.
CS 340 Page 12
MIMDParallel processing on MIMD (Multiple Instruction, Multiple Data) machines is accomplished by having all processors work independently, performing different instructions on different data. Applications for this type of
system involve tasks that can be easily separated from each other, such as CAD/CAM, modeling and simulation, and network processing, thus minimizing the time-consuming communication between processors.
Processor 1
Processor 2
Processor 3
Processor 4
…z = x * y;alpha(z, n);……for (i = 1; i < n; i++){ A[i] = 0.0; B[i] = 14.8;}…
…z = x * y;alpha(z, n);…
…w = u * v;t = 2 * w + v / 10;…
Most current supercomputers, networked parallel computer clusters and “grids”, and multi-core PCs use the MIMD approach.
CS 340 Page 13
Designing Parallel ProgramsDesigning and developing parallel programs has characteristically been a very manual process with the programmer typically responsible for both identifying and actually implementing parallelism. Very often, manually developing parallel codes is a time-consuming, complex, error-prone, and iterative process. The first step is to determine whether or not a problem is one that can actually be parallelized.
Example of Parallelizable Problem:
Calculate the potential energy for each of several thousand independent conformations of a molecule; when done, find the minimum energy conformation. This problem is able to be solved in parallel. Each of the molecular conformations is independently determinable. The calculation of the minimum energy conformation is also a parallelizable problem.
Example of Non-Parallelizable Problem:
Calculation of the Fibonacci series (1, 1, 2, 3, 5, 8, 13, 21, ...) by use of the formula:
F(n) = F(n-1) + F(n-2) This is a non-parallelizable problem because the calculation of the Fibonacci sequence as shown would entail dependent calculations rather than independent ones. The calculation of the F(n) value uses those of both F(n-1) and F(n-2). These three terms cannot be calculated independently and therefore, not in parallel.
CS 340 Page 14
Problem DecompositionOne of the first steps in designing a parallel program is to break the problem into discrete “chunks” of work that can be distributed to multiple tasks. Method 1: Domain Decomposition
In this type of partitioning, the data associated with the problem is decomposed. Each parallel task then works on a portion of the data.
Problem Data Set
Task 1 Task 2 Task 3 Task 4
CS 340 Page 15
Method 2: Functional DecompositionIn this approach, the partitioning is based on the computation that is to be performed rather than on the data manipulated by the computation. The problem is decomposed according to the work that must be done. Each task then performs a portion of the overall work.
Task 1 Task 2 Task 3 Task 4
Problem Instruction Set
CS 340 Page 16
Functional Decomposition ExamplesFunctional decomposition lends itself to problems that can be split into different tasks, such as:
Ecosystem Modeling Each program calculates the population of a given group, where each group's growth depends on that of its neighbors. As time progresses, each process calculates its current state, then exchanges information with the neighbor populations. All tasks then progress to calculate the state at the next time step. Signal Processing
An audio signal data set is passed through four distinct computational filters. Each filter is a separate process. The first
segment of data must pass through the first filter before progressing to the
second. When it does, the second segment of data passes through the first filter. By the time the fourth segment of
data is in the first filter, all four tasks are busy.
Climate Modeling Each model component can be thought of as a separate task. Arrows represent exchanges of data between components during computation: the atmosphere model generates wind velocity data that are used by the ocean model, the ocean model generates sea surface temperature data that are used by the atmosphere model, and so on.