Parallel Programming: Speedups and Amdahl’s...
Transcript of Parallel Programming: Speedups and Amdahl’s...
mjb – March 15, 2020
1
Computer Graphics
Parallel Programming:Speedups and Amdahl’s law
speedups.and.amdahls.law.pptx
Mike [email protected]
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
mjb – March 15, 2020
2
Computer Graphics
Definition of Speedup
1n
n
TSpeedupT
If you are using n processors, your Speedupn is:
nn
SpeedupEfficiencyn
And your Speedup Efficiencyn is:
where T1 is the execution time on one core and Tn is the execution time on n cores.Note that Speedupn should be > 1.
which could be as high as 1., but probably never will be.
mjb – March 15, 2020
3
Computer Graphics
However, Multicore is not a Free Lunch:Amdahl’s Law
1 1 1
(1 )n
parallel parallelnsequential parallel
TSpeedup F FT F Fn n
If you think of all the operations that a program needs to do as being divided between a fraction that is parallelizable and a fraction that isn’t (i.e., is stuck at being sequential), then Amdahl’s Law says:
If you put in n processors, you should get n times Speedup(and 100% Speedup Efficiency), right? Wrong!
There are always some fraction of the total operation that is inherently sequential and cannot be parallelized no matter what you do. This includes reading data, setting up calculations, control logic, storing results, etc.
This fraction can be reduced by deploying multiple processors. This fraction can’t.
mjb – March 15, 2020
4
Computer Graphics
A Visual Explanation of Amdahl’s Law
# of Cores
1 2 4
Exec
utio
n Ti
me
ParallelPortion
SequentialPortion
SequentialPortion
SequentialPortion
ParallelPortion Parallel
Portion
SequentialPortion
~∞
The Sequential Portion doesn’t go away, and it also doesn’t get any smaller. It just gets more and more dominant.
≈
mjb – March 15, 2020
5
Computer Graphics
Amdahl’s Law as a Function of Number of Processors and Fparallel
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29# Processors
x Sp
eedu
p
60%
40%
90%
80%
20%
Fparallel :
mjb – March 15, 2020
6
Computer Graphics
Parallel Fraction
# processors
X Sp
eedu
p
mjb – March 15, 2020
7
Computer Graphics
Amdahl’s Law
1 1max lim 1n sequential parallel
Speedup SpeedupF F
Note that these fractions put an upper bound on how much benefit you will get from adding more processors:
Fparallel maxSpeedup
0.00 1.00
0.10 1.11
0.20 1.25
0.30 1.43
0.40 1.67
0.50 2.00
0.60 2.50
0.70 3.33
0.80 5.00
0.90 10.00
0.95 20.00
0.99 100.00
mjb – March 15, 2020
8
Computer Graphics
1 1
1 11 1 1
1 1
1 11 ( ) 111 1 1 1 ( 1) ( 1) ( 1)
n n n
n n
T T T T Tn T T T TT T T n nSF n n n n T n n T n Speedup
n n n n
Solving for F:
If you’ve got several (n,S) values, you can take the average (which is actually a least squares fit):
1 1
(1 )n
TS FT Fn
Amdahl’s law says:
1
1
2
, 2..( 1)
1
inii
iN
ii
T TnF i Nn T
FF
N
1 (1 ) 1F F nFFS n n
1 (1 )1 nFS n
1niT Tnote that when i=1,
You can also solve for Fparallel using Amdahl’s Law if youknow your speedup and the number of processors
Use this if you know the timing
Use this if you know the speedup
mjb – March 15, 2020
9
Computer Graphics
A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation
Gustafson observed that as you increase the number of processors, you have a tendency to attack larger and larger versions of the problem. He also observed that when you use the same parallel program on larger datasets, the parallel fraction, Fp, increases.
Let P be the amount of time spent on the parallel portion of an original task and S spent on the serial portion. Then
orp
PFP S
p
p
P PFS
F
Without loss of generality, we can set P=1 so that, really, S is now a fraction of P. We now have:
1 p
p
FS
F
ParallelTime
SerialTime
mjb – March 15, 2020
10
Computer Graphics
A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation
' ''pP NPF
P S NP S
'
1pp
p
N NF FN S NF
We know that if we multiply the amount of data to process by N, then the amount of parallel work becomes NP. Surely the serial work must increase too, but we don’t know how much. Let’s say it doesn’t increase at all, so that we know we are getting an upper bound answer.
In that case, the new parallel fraction is:
And substituting for P (=1) and for S, we have:
mjb – March 15, 2020
11
Computer Graphics
A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation
If we tabulate this, we get a table of Fp’ values:
mjb – March 15, 2020
12
Computer Graphics
A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation
Or, graphing it:
mjb – March 15, 2020
13
Computer Graphics
A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation
We can also turn Fp’ into a Maximum Speedup: