01~Chapter 01

8/2/2019 01~Chapter 01

1/118

Introduction

Companion slides forThe Art of Multiprocessor

Programmingby Maurice Herlihy & Nir Shavit

8/2/2019 01~Chapter 01

2/118

Art of Multiprocessor Programming 2

Moores Law

Clock speed

flatteningsharply

Transistorcount stillrising

8/2/2019 01~Chapter 01

3/118


Still on some of your desktops:

The Uniprocesor

memory

cpu

8/2/2019 01~Chapter 01

4/118


In the Enterprise:The Shared Memory Multiprocessor

(SMP)

cache

Bus

Bus

shared memory

cachecache

8/2/2019 01~Chapter 01

5/118


Your New Desktop:The Multicore Processor

(CMP)

cache

Bus

Bus

shared memory

cachecacheAll on thesame chip

SunT2000Niagara

8/2/2019 01~Chapter 01

6/118


Multicores Are Here

Intel's Intel ups ante with 4-core chip.New microprocessor, due this year, will befaster, use less electricity... [San FranChronicle]

AMD will launch a dual-core version of itsOpteron server processor at an event inNew York on April 21. [PC World]

SunsNiagarawill have eight cores, eachcore capable of running 4 threads inparallel, for 32 concurrently runningthreads. . [The Inquierer]

8/2/2019 01~Chapter 01

7/118


Why do we care?

Time no longer cures software bloat The free ride is over

When you double your programs pathlength You cant just wait 6 months

Your software must somehow exploittwice as much concurrency

8/2/2019 01~Chapter 01

8/118


Traditional Scaling Process

User code

TraditionalUniprocessor

Speedup1.8x

7x

3.6x

Time: Moores law

8/2/2019 01~Chapter 01

9/118


Multicore Scaling Process

User code

Multicore

Speedup1.8x

7x

3.6x

Unfortunately, not so simple

8/2/2019 01~Chapter 01

10/118


Real-World Scaling Process

1.8x 2x2.9x

User code

Multicore

Speedup

Parallelization and Synchronizationrequire great care

8/2/2019 01~Chapter 01

11/118


Multicore Programming:

Course Overview

Fundamentals

Models, algorithms, impossibility

Real-World programming

Architectures Techniques

8/2/2019 01~Chapter 01

12/118


Multicore Programming:

Course Overview

Fundamentals

Models, algorithms, impossibility

Real-World programming

Architectures Techniques

8/2/2019 01~Chapter 01

13/118

Art of MultiprocessorProgramming13

Sequential Computation

memory

object object

thread

8/2/2019 01~Chapter 01

14/118


Concurrent Computation

memory

object object

8/2/2019 01~Chapter 01

15/118


Asynchrony

Sudden unpredictable delays Cache misses (short)

Page faults (long) Scheduling quantum used up (really long)

8/2/2019 01~Chapter 01

16/118


Model Summary

Multiple threads Sometimes calledprocesses

Single shared memory Objectslive in memory

Unpredictable asynchronous delays

8/2/2019 01~Chapter 01

17/118


Road Map

We are going to focus on principlesfirst, then practice

Start with idealized models Look at simplistic problems

Emphasize correctness over pragmatism

Correctness may be theoretical, butincorrectness has practical impact

8/2/2019 01~Chapter 01

18/118


Concurrency Jargon

Hardware Processors

Software Threads, processes

Sometimes OK to confuse them,

sometimes not.

8/2/2019 01~Chapter 01

19/118


Parallel Primality Testing

Challenge Print primes from 1 to 1010

Given Ten-processor multiprocessor

One thread per processor

Goal Get ten-fold speedup (or close)

8/2/2019 01~Chapter 01

20/118


Load Balancing

Split the work evenly

Each thread tests range of 109

109 101021091

P0 P1 P9

8/2/2019 01~Chapter 01

21/118

Art of MultiprocessorProgramming 21

Procedure for Thread i

void primePrint {int i = ThreadID.get(); // IDs in {0..9}for(j = i*109+1, j

8/2/2019 01~Chapter 01

22/118


Issues

Higher ranges have fewer primes

Yet larger numbers harder to test

Thread workloads Uneven

Hard to predict

8/2/2019 01~Chapter 01

23/118


Issues

Higher ranges have fewer primes

Yet larger numbers harder to test

Thread workloads Uneven

Hard to predict

Need dynamicload balancing

8/2/2019 01~Chapter 01

24/118


17

1819

Shared Counter

each thread

takes a number

8/2/2019 01~Chapter 01

25/118



int counter = new Counter(1);void primePrint {

long j = 0;while (j < 1010) {j = counter.getAndIncrement();if (isPrime(j))print(j);}}

8/2/2019 01~Chapter 01

26/118


Counter counter = new Counter(1);void primePrint {long j = 0;while (j < 1010) {j = counter.getAndIncrement();if (isPrime(j))

print(j);}}


Shared counter

object

8/2/2019 01~Chapter 01

27/118


Where Things Reside

cache

BusBus

cachecache

1

shared counter

shared

memory

void primePrint {int i =ThreadID.get(); // IDsin {0..9}for(j = i*109+1,j

8/2/2019 01~Chapter 01

28/118



Counter counter = new Counter(1);void primePrint {

long j = 0;while (j < 1010) {j = counter.getAndIncrement();if (isPrime(j))print(j);}}

Stop when everyvalue taken

8/2/2019 01~Chapter 01

29/118


Counter counter = new Counter(1);void primePrint {long j = 0;while (j < 1010) {j =counter.getAndIncrement();if (isPrime(j))

print(j);}}


Increment & returneach new value

8/2/2019 01~Chapter 01

30/118


Counter Implementation

public class Counter{private long value;public long getAndIncrement() {return value++;}}

8/2/2019 01~Chapter 01

31/118


Counter Implementation

public class Counter {private long value;public long getAndIncrement() {return value++;}}

8/2/2019 01~Chapter 01

32/118


What It Means


8/2/2019 01~Chapter 01

33/118


What It Means


temp = value;

value = value + 1;

return temp;

8/2/2019 01~Chapter 01

34/118


time

Not so good

Value 1

read1

read1

write2 read2 write3

write2

2 3 2

8/2/2019 01~Chapter 01

35/118


Is this problem inherent?

If we could only glue reads and writes

read

write read

write

8/2/2019 01~Chapter 01

36/118


Challenge

public class Counter {private long value;public long getAndIncrement() {temp = value;value = temp + 1;return temp;}}

8/2/2019 01~Chapter 01

37/118


Challenge

public class Counter {private long value;public long getAndIncrement() {temp = value;value = temp + 1;return temp;}} Make these stepsatomic(indivisible)

8/2/2019 01~Chapter 01

38/118


Hardware Solution

public class Counter {private long value;public long getAndIncrement() {temp = value;value = temp + 1;return temp;}} ReadModifyWrite()

instruction

8/2/2019 01~Chapter 01

39/118


An Aside: Java

public class Counter {private long value;public long getAndIncrement() {synchronized{temp = value;value = temp + 1;

}return temp;}}

8/2/2019 01~Chapter 01

40/118


An Aside: Java

public class Counter {private long value;public long getAndIncrement() {synchronized{temp = value;value = temp + 1;

}return temp;}} Synchronized block

8/2/2019 01~Chapter 01

41/118


An Aside: Java

public class Counter {private long value;public long getAndIncrement() {synchronized {temp = value;value = temp + 1;

}return temp;}}

Mutual Exclusion

8/2/2019 01~Chapter 01

42/118


Mutual Exclusion or Alice &

Bob share a pondA B

8/2/2019 01~Chapter 01

43/118


Alice has a pet

A B

8/2/2019 01~Chapter 01

44/118


Bob has a pet

A B

8/2/2019 01~Chapter 01

45/118


The Problem

A B

The pets dont

get along

8/2/2019 01~Chapter 01

46/118

Art of Multiprocessor Programming46

Formalizing the Problem

Two types of formal properties inasynchronous computation:

Safety Properties Nothing bad happens ever

Liveness Properties

Something good happens eventually

8/2/2019 01~Chapter 01

47/118


Formalizing our Problem

Mutual Exclusion Both pets never in pond simultaneously

This is a safetyproperty No Deadlock

if only one wants in, it gets in

if both want in, one gets in. This is a livenessproperty

8/2/2019 01~Chapter 01

48/118


Simple Protocol

Idea Just look at the pond

Gotcha Trees obscure the view

8/2/2019 01~Chapter 01

49/118


Interpretation

Threads cant see what otherthreads are doing

Explicit communication required forcoordination

8/2/2019 01~Chapter 01

50/118


Cell Phone Protocol

Idea Bob calls Alice (or vice-versa)

Gotcha Bob takes shower

Alice recharges battery

Bob out shopping for pet food

8/2/2019 01~Chapter 01

51/118


Interpretation

Message-passing doesnt work

Recipient might not be

Listening There at all

Communication must be

Persistent (like writing) Not transient (like speaking)

8/2/2019 01~Chapter 01

52/118


Can Protocol

col

a

cola

8/2/2019 01~Chapter 01

53/118

Art of MultiprocessorProgramming

53

Bob conveys a bit

A B

cola

8/2/2019 01~Chapter 01

54/118


54

Bob conveys a bit

A B

8/2/2019 01~Chapter 01

55/118


Can Protocol

Idea Cans on Alices windowsill

Strings lead to Bobs house Bob pulls strings, knocks over cans

Gotcha

Cans cannot be reused Bob runs out of cans

8/2/2019 01~Chapter 01

56/118


Interpretation

Cannot solve mutual exclusion withinterrupts

Sender sets fixed bit in receivers space Receiver resets bit when ready

Requires unbounded number of inturrupt

bits

8/2/2019 01~Chapter 01

57/118


57

Flag Protocol

A B

8/2/2019 01~Chapter 01

58/118


58

Alices Protocol (sort of)

A B

8/2/2019 01~Chapter 01

59/118


59

Bobs Protocol (sort of)

A B

8/2/2019 01~Chapter 01

60/118


Alices Protocol

Raise flag

Wait until Bobs flag is down Unleash pet

Lower flag when pet returns

8/2/2019 01~Chapter 01

61/118


Bobs Protocol

Raise flag

Wait until Alices flag is down Unleash pet

Lower flag when pet returns

8/2/2019 01~Chapter 01

62/118


Bobs Protocol (2nd try)

Raise flag

While Alices flag is up Lower flag Wait for Alices flag to go down

Raise flag Unleash pet Lower flag when pet returns

8/2/2019 01~Chapter 01

63/118


Bobs Protocol

Raise flag

While Alices flag is up Lower flag Wait for Alices flag to go down

Raise flag Unleash pet Lower flag when pet returns

Bob defersto Alice

8/2/2019 01~Chapter 01

64/118


The Flag Principle

Raise the flag

Look at others flag

Flag Principle: If each raises and looks, then

Last to look must see both flags up

8/2/2019 01~Chapter 01

65/118


Proof of Mutual Exclusion

Assume both pets in pond Derive a contradiction

By reasoning backwards Consider the last time Alice and Bob

each looked before letting the pets in

Without loss of generality assumeAlice was the last to look

P f

8/2/2019 01~Chapter 01

66/118


66

Proof

time

Alices last look

Alice last raised her flag

Bobs lastlook

Alice must have seen Bobs Flag. A Contradiction

Bob last raisedflag

8/2/2019 01~Chapter 01

67/118


Proof of No Deadlock

If only one pet wants in, it gets in.

8/2/2019 01~Chapter 01

68/118




Deadlock requires both continually

trying to get in.

8/2/2019 01~Chapter 01

69/118




Deadlock requires both continually

trying to get in. If Bob sees Alices flag, he gives her

priority (a gentleman)

8/2/2019 01~Chapter 01

70/118


Remarks

Protocol is unfair Bobs pet might never get in

Protocol uses waiting If Bob is eaten by his pet, Alices pet

might never get in

8/2/2019 01~Chapter 01

71/118


Moral of Story

Mutual Exclusion cannot be solved by transient communication (cell phones)

interrupts (cans) It can be solved by one-bit shared variables that can be read or written

8/2/2019 01~Chapter 01

72/118


The Arbiter Problem (an aside)

Pick apoint

Pick apoint

8/2/2019 01~Chapter 01

73/118


The Fable Continues

Alice and Bob fall in love & marry

8/2/2019 01~Chapter 01

74/118


The Fable Continues


Then they fall out of love & divorce

She gets the pets He has to feed them

8/2/2019 01~Chapter 01

75/118


The Fable Continues


Then they fall out of love & divorce

She gets the pets He has to feed them

Leading to a new coordination

problem: Producer-Consumer

8/2/2019 01~Chapter 01

76/118


76

Bob Puts Food in the Pond

A

8/2/2019 01~Chapter 01

77/118


77

mmm

Alice releases her pets to Feed

Bmmm

8/2/2019 01~Chapter 01

78/118


Producer/Consumer

Alice and Bob cant meet Each has restraining order on other

So he puts food in the pond And later, she releases the pets

Avoid

Releasing pets when theres no food Putting out food if uneaten food remains

8/2/2019 01~Chapter 01

79/118


Producer/Consumer

Need a mechanism so that Bob lets Alice know when food has been

put out Alice lets Bob know when to put out

more food

8/2/2019 01~Chapter 01

80/118

Art of Multiprocessor

Programming

80

Surprise Solution

A B

cola

8/2/2019 01~Chapter 01

81/118


Programming

81

Bob puts food in Pond

A B

cola

8/2/2019 01~Chapter 01

82/118


Programming

82

Bob knocks over Can

A B

8/2/2019 01~Chapter 01

83/118


Programming

83

Alice Releases Pets

A Byum Byum

Ali R C h P

8/2/2019 01~Chapter 01

84/118


Programming

84

Alice Resets Can when Pets are

FedA B

cola

8/2/2019 01~Chapter 01

85/118


Programming

85

Pseudocode

while (true) {while (can.isUp()){};pet.release();pet.recapture();can.reset();}

Alices code

8/2/2019 01~Chapter 01

86/118


Programming

86

Pseudocode

while (true) {while (can.isUp()){};pet.release();pet.recapture();can.reset();}

Alices code

while (true) {while (can.isDown()){};pond.stockWithFood();can.knockOver();}

Bobs code

C t

8/2/2019 01~Chapter 01

87/118


Correctness

Mutual Exclusion Pets and Bob never together in pond

8/2/2019 01~Chapter 01

88/118


Correctness


No Starvation

if Bob always willing to feed, and petsalways famished, then pets eat infinitelyoften.

C t ss

8/2/2019 01~Chapter 01

89/118


Correctness


No Starvation

if Bob always willing to feed, and petsalways famished, then pets eat infinitelyoften.

Producer/Consumer

The pets never enter pond unless there isfood, and Bob never provides food ifthere is unconsumed food.

safetyliveness

safety

C ld Al S l U i Fl

8/2/2019 01~Chapter 01

90/118


Programming

90

Could Also Solve Using Flags

A B

8/2/2019 01~Chapter 01

91/118


Waiting

Both solutions use waiting while(mumble){}

Waiting isproblematic If one participant is delayed

So is everyone else

But delays are common & unpredictable

8/2/2019 01~Chapter 01

92/118


The Fable drags on

Bob and Alice still have issues

8/2/2019 01~Chapter 01

93/118


The Fable drags on


So they need to communicate

8/2/2019 01~Chapter 01

94/118


The Fable drags on


So they need to communicate

So they agree to use billboards

8/2/2019 01~Chapter 01

95/118


Programming

95

E1

D2

C3

Billboards are Large

B3A

1

LetterTiles

From Scrabble box

8/2/2019 01~Chapter 01

96/118


Programming

96

E1

D2

C3

Write One Letter at a Time

B3A

1

W4 A1 S1

H4

8/2/2019 01~Chapter 01

97/118


Programming

97

To post a message

W4 A1 S1 H4 A1C3 R1T1 H4 E1

whe

w

L t d th

8/2/2019 01~Chapter 01

98/118


Programming

98

S1

Lets send another message

S1

E1

L1

L1

L1

V4

L1 A

1

M3

A1

A1

P3

8/2/2019 01~Chapter 01

99/118


Programming

99

Uh-Oh

A1C3 R1T1 H4 E1S1 E1 L1 L1

L1

OK

8/2/2019 01~Chapter 01

100/118


Readers/Writers

Devise a protocol so that Writer writes one letter at a time

Reader reads one letter at a time Reader sees

Old message or new message

No mixed messages

8/2/2019 01~Chapter 01

101/118


Readers/Writers (continued)

Easy with mutual exclusion

But mutual exclusion requires waiting

One waits for the other Everyone executes sequentially

Remarkably

We can solve R/W without mutualexclusion

8/2/2019 01~Chapter 01

102/118


Why do we care?

We want as much of the code aspossible to execute concurrently (in

parallel) A larger sequential part implies

reduced performance

Amdahls law: this relation is notlinear

8/2/2019 01~Chapter 01

103/118


Amdahls Law

OldExecutionTime

NewExecutionTimeSpeedup=

of computation given nCPUs instead of 1

8/2/2019 01~Chapter 01

104/118


Amdahls Law

pp

n

1

1

Speedup=

8/2/2019 01~Chapter 01

105/118


Amdahls Law

pp

n

1

1

Speedup=

Parallelfraction

8/2/2019 01~Chapter 01

106/118


Amdahls Law

pp

n

1

1

Speedup=

Parallelfraction

Sequentialfraction

8/2/2019 01~Chapter 01

107/118


Amdahls Law

pp

n

1

1

Speedup=

Parallelfraction

Number ofprocessors

Sequentialfraction

8/2/2019 01~Chapter 01

108/118


Programming

108

Example

Ten processors

60% concurrent, 40% sequential

How close to 10-fold speedup?

8/2/2019 01~Chapter 01

109/118


Programming

109

Example

Ten processors



10

6.0

6.01

1

Speedup=2.17=

8/2/2019 01~Chapter 01

110/118


Programming

110

Example

Ten processors



E l

8/2/2019 01~Chapter 01

111/118


Programming

111

Example

Ten processors



10

8.0

8.01

1

Speedup=3.57=

E l

8/2/2019 01~Chapter 01

112/118


Programming

112

Example

Ten processors



E l

8/2/2019 01~Chapter 01

113/118


Programming

113

Example

Ten processors



10

9.0

9.01

1

Speedup=5.26=

E l

8/2/2019 01~Chapter 01

114/118


Programming

114

Example

Ten processors



E l

8/2/2019 01~Chapter 01

115/118


Programming

115

Example

Ten processors



10

99.0

99.01

1

Speedup=9.17=

Th M l

8/2/2019 01~Chapter 01

116/118


The Moral

Making good use of our multipleprocessors (cores) means

Finding ways to effectively parallelizeour code Minimize sequential parts

Reduce idle time in which threads waitwithout

M l i P i

8/2/2019 01~Chapter 01

117/118


Multicore Programming

This is what this course is about The % that is not easy to make

concurrent yet may have a large impacton overall speedup

Next week: A more serious look at mutual exclusion

This work is licensed under a Creative Commons Attribution-ShareAlike 2 5 License
http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/

8/2/2019 01~Chapter 01

118/118

ShareAlike 2.5 License.

You are free:

to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions:

Attribution. You must attribute the work to The Art ofMultiprocessor Programming (but not in any way that suggests thatthe authors endorse you or your use of the work).

Share Alike. If you alter, transform, or build upon this work, youmay distribute the resulting work only under the same, similar or acompatible license.

For any reuse or distribution, you must make clear to others the licenseterms of this work. The best way to do this is with a link to

http://creativecommons.org/licenses/by-sa/3.0/.

Any of the above conditions can be waived if you get permission from

the copyright holder. Nothing in this license impairs or restricts the author's moral rights.
http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/http://creativecommons.org/licenses/by-sa/2.5/

01~Chapter 01

Documents

Transcript of 01~Chapter 01