Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel...
description
Transcript of Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel...
![Page 1: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/1.jpg)
Lecture 3:Lecture 3:
Designing Parallel Designing Parallel ProgramsPrograms
![Page 2: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/2.jpg)
Methodological Design
Designing and Building Parallel Programs
by Ian Foster
www-unix.mcs.anl.gov/dbpp
![Page 3: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/3.jpg)
Methodological Design
Partitioning Communication Agglomeration Mapping
![Page 4: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/4.jpg)
Methodological Design
PROBLEM
![Page 5: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/5.jpg)
Methodological Design Partitioning
The computation that is to be performed and the data operated on by this computation are decomposed into small tasks.
Practical issues such as the number of processors in the target computer are ignored, and attention is focused on recognizing opportunities for parallel execution.
PROBLEMPartitioning
![Page 6: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/6.jpg)
Partitioning
Functional Decomposition
Data Decomposition
![Page 7: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/7.jpg)
Partitioning
Functional Decomposition
![Page 8: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/8.jpg)
Partitioning
Data Decomposition
![Page 9: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/9.jpg)
Methodological DesignSingle Program Multiple Data (SPMD) programming model
A fixed number of identical tasks are created at program startup. Each task executes the same program but operates on different data.
![Page 10: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/10.jpg)
Methodological Design Communication
The communication required to coordinate task execution is determined, and appropriate communication structures and algorithms are defined.
PROBLEMPartitioning
Communication
![Page 11: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/11.jpg)
Communication
Latency (L): How long does it take to start sending a "message"? (in microseconds)
Bandwidth (B): What data rate can be sustained once the message is started? (in Mbytes/sec)
Transfer time = L + (1/B) * data size
![Page 12: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/12.jpg)
Communication Local Communication
Xi,j = (4 Xi,j + Xi-1,j + Xi+1,j + Xi,j-1 + Xi,j+1 )/8
Xi-1,j
Xi,jXi,j-1 Xi,j+1
Xi+1,j
5-point Stencil
![Page 13: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/13.jpg)
Communication Global Communication
n
∑Xi i=0
X0 X1 X2 X3 X4 X5 X6 X7
∑01 ∑2
3 ∑45 ∑6
7
∑07
∑03 ∑4
7
![Page 14: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/14.jpg)
Communication
Static CommunicationCommunicating tasks do not change over time.
Dynamic CommunicationCommunicating tasks are determined by data computed at run time.
![Page 15: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/15.jpg)
Communication
Dynamic Communication
B[i]=A[ k[i] ]k 1 2 1 3 4
A 10 15 20 25 30
B 10 15 10 20 25
![Page 16: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/16.jpg)
Methodological Design Agglomeration
The task and communication structures defined in the first two stages of a design are evaluated with respect to performance requirements and implementation costs.
If necessary, tasks are combined into larger tasks to improve performance or to reduce development costs.
PROBLEMPartitioning
Communication
Agglomeration
![Page 17: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/17.jpg)
Agglomeration
![Page 18: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/18.jpg)
Agglomeration
![Page 19: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/19.jpg)
Agglomeration
Goals are:
Increasing Granularity (fine grain / course grain)
Decreasing communication/computation ratio
![Page 20: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/20.jpg)
AgglomerationXi,j = (4 Xi,j + Xi-1,j + Xi+1,j + Xi,j-1 + Xi,j+1 )/8
![Page 21: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/21.jpg)
AgglomerationXi,j = (4 Xi,j + Xi-1,j + Xi+1,j + Xi,j-1 + Xi,j+1 )/8
![Page 22: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/22.jpg)
Agglomeration
![Page 23: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/23.jpg)
Methodological Design Mapping
Each task is assigned to a processor in a manner that minimizes the total execution time.
Mapping can be specified statically or determined at runtime by load-balancing algorithms.
PROBLEMPartitioning
Communication
Agglomeration
Mapping
![Page 24: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/24.jpg)
MappingIf each task performs the same amount of computation
and communicates:
Cyclic Mapping
![Page 25: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/25.jpg)
MappingThe goal of mapping algorithms is to minimize total
execution time.
Mapping algorithms attempt to satisfy the competing goals of:
maximizing processor utilization minimizing communication costs.
Mapping is NP-complete
![Page 26: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/26.jpg)
MappingTwo strategies to achieve this goal:
Place tasks that are able to execute concurrently on different processors
Place tasks that communicate frequently on the same processor
![Page 27: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/27.jpg)
Mapping
Consider the topology
![Page 28: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/28.jpg)
Methodological Design
Case Study: Atmosphere
Model
![Page 29: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/29.jpg)
Methodological Design
Case Study: Atmosphere
Model
![Page 30: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/30.jpg)
Methodological Design
Case Study: Atmosphere
Model
![Page 31: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/31.jpg)
Methodological Design
Case Study: Atmosphere
Model
PartitioningNx x Ny x Nz tasks
![Page 32: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/32.jpg)
Methodological Design
Case Study: Atmosphere
Model
Communication
9-point stencil in horizontal dimension3-point stencil in vertical dimension
![Page 33: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/33.jpg)
Methodological Design
Case Study: Atmosphere
Model
Communication
Computations
1. Global operations
2. Physics calculations
where denotes the mass at grid point (i,j,k) .
The total clear sky (TCS) at level k≥1 is defined as
where level 0 is the top of the atmosphere and
cldi is the cloud fraction at level i . In total, the physics component of the model requires on the order of 30 communications per grid point and per time step
![Page 34: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/34.jpg)
Methodological Design
Case Study: Atmosphere
Model
Communicationobtains data from eight other tasks
![Page 35: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/35.jpg)
Methodological Design
Case Study: Atmosphere
Model
Communicationobtains data from eight other tasks
only 4 communications are required per task
![Page 36: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/36.jpg)
Methodological Design
Case Study: Atmosphere
Model
AgglomerationVertical dimension requires communication (2 messages) also (30 messages) for various ``physics'' computations
These communications can be avoided by agglomerating tasks within each vertical column.
Therefore tasks will be created.
![Page 37: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/37.jpg)
Methodological Design
Case Study: Atmosphere
Model
Mapping
![Page 38: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/38.jpg)
Methodological Design
![Page 39: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/39.jpg)
Methodological Design
Agglomeration
![Page 40: Lecture 3: Designing Parallel Programs. Methodological Design Designing and Building Parallel Programs by Ian Foster .](https://reader035.fdocuments.in/reader035/viewer/2022062401/5a4d1b7f7f8b9ab0599ba5d8/html5/thumbnails/40.jpg)
Methodological Design
Increase Granularity Decrease communication/computation ratio