Aspects of practical parallel programming Parallel programming models Data parallel
The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing...
Transcript of The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing...
![Page 1: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/1.jpg)
1
The Two Main Models of Parallel Processing
Distributed Memory (MPI) and Shared Memory (OpenMP)
Two different HW arch. models led to two different SW Programming models
o 5 years ago, MPI was standard; Now, in 2010, OpenMP is more popular
![Page 2: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/2.jpg)
2
![Page 3: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/3.jpg)
3
![Page 4: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/4.jpg)
4
![Page 5: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/5.jpg)
5
![Page 6: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/6.jpg)
6
![Page 7: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/7.jpg)
7
![Page 8: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/8.jpg)
8
![Page 9: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/9.jpg)
9
![Page 10: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/10.jpg)
10
![Page 11: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/11.jpg)
11
![Page 12: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/12.jpg)
12
![Page 13: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/13.jpg)
13
In the “old days”, each processor was in a separate computer
o So parallel processing was accomplished using a collection of these computers
Many computers were put together into a Cluster or a Large SuperComputer
o Each had its own Memory Distributed Memory Message Passing Needed
Now (2010) Multi-Core and Many-Core Designs put several processors on same chip
o So cores are likely to Share Memory Shared Mem Shared Mem Programming
![Page 14: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/14.jpg)
14
Disadvantages:
![Page 15: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/15.jpg)
15
OpenMP
![Page 16: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/16.jpg)
16
General Philosophy behind OpenMP is that the compiler doesn’t have enough
Information at the Source Code Level to do effective Parallelization
Therefore, Programmer must add extra “Comments” to help in Parallelization
Comments, or “Directives”, are typically focused on Loops in the Code
Loops (that are data independent per iteration) are prime targets for parallelization
Loops are typically where the majority of execution time is spent
Loops are typically modular and operate on Arrays or Matrices
By focusing on Loops, we can get the best bang per buck
Minimal extra coding (directives) to get maximum speedup in runtime hotspots
![Page 17: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/17.jpg)
17
![Page 18: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/18.jpg)
18
Basic Idea Behind OpenMP
![Page 19: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/19.jpg)
19
![Page 20: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/20.jpg)
20
Sample OpenMP Directives:
Directives look like comments to a non-OpenMP savvy compiler (OpenMP disabled)
o So by not using the –fopenmp compiler option, parallelization is switched off
![Page 21: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/21.jpg)
21
OpenMP can use the same source code for both Sequential and Parallel Execution
o Provides Tremendous Advantages
Ultimate Scaling (from 1 core sequential up to N cores of parallelism)
Simplifies Debugging and Optimization
Optimized sequential program will more likely be an optimized parallel one
Allows Debugging to occur in simpler, Sequential mode first
(But a correct sequential program is not necessarily a correct parallel one)
Allows Incremental Parallelization
![Page 22: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/22.jpg)
22
If Parallelization switch of Compiler is turned on (using the –fopenmp option) then:
Programmer must ensure Code + Directives are correct
![Page 23: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/23.jpg)
23
OpenMP’s Memory Model
![Page 24: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/24.jpg)
24
OpenMP’s Execution Model is based on Forks
o Similar to Unix Fork
o Instead of Manually using Fork, Wait and Signal, Programmer uses Directives
When work can be done in Parallel, Programmer Bounds Code with OpenMP Directives
Prime Candidates for Parallelization (and forking of worker threads) are Loops
o Code not in loops are not executed much, so can proceed in Sequential mode
![Page 25: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/25.jpg)
25
![Page 26: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/26.jpg)
26
![Page 27: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/27.jpg)
27
Gives the Following Result for N = 1000
200 Iterations are assigned to each of the 5 Threads
![Page 28: The Two Main Models of Parallel Processing …...1 The Two Main Models of Parallel Processing Distributed Memory (MPI) and Shared Memory (OpenMP) Two different HW arch. models led](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6ee1a7f024f44de0d7ab9/html5/thumbnails/28.jpg)
28