ParallelismMarco Serafini

COMPSCI 590SLecture 3

2

Announcements• Reviews

• First paper posted on website

• Review due by this Wednesday 11 PM (hard deadline)

• Data Science Career Mixer (save the date!)• November 5, 4-7 pm

• Campus Center Auditorium

• Recruiting and industry engagement event

3

Why multi-core architectures?

4

Multi-Cores• We have talked about multi-core architectures• Why do we actually use multi-cores?• Why not a single core?

5

Maximum Clock Rate is Stagnating

Source: https://queue.acm.org/detail.cfm?id=2181798

Two major “laws” are collapsing• Moore’s law• Dennard scaling

6

Moore’s Law• “Density of transistors in an integrated circuit doubles every two years”. Smaller à changes propagate faster

So far so good, but the trend is slowing down and it won’t last for long (Intel’s prediction: until 2021 unless new technologies arise) [1]

[1] https://www.technologyreview.com/s/601441/moores-law-is-dead-now-what/

Expo

nent

ial a

xis

7

Dennard Scaling• “Reducing transistor size does not increase power

density à power consumption proportional to chip

area”

• Stopped holding around 2006

• Assumptions break when physical system close to limit

• Post-Dennard-scaling world of today

• Huge cooling and power consumption issues

• If we kept the same clock frequency trends, today a CPU

would have the power density of a nuclear reactor

8

Heat Dissipation Problem• Large datacenters consume energy like large cities• Cooling is the main cost factor

Google @ Columbia River valley (2006) Facebook @ Luleå (2015)

9

Where is Luleå?

10

Possible Solutions• Dynamic Voltage and Frequency Scaling (DVFS)

• E.g. Intel’s TurboBoost• Only works under low load

• Use part of the chip for coprocessors (e.g. graphics)• Lower power consumption• Limited number of generic functionalities to offload

11

More Solutions• Multicores

• Replace 1 powerful core with multiple weaker cores on a chip• SIMD

• Single Instruction Multiple Data• A massive number of cores with reduced flexibility

• FPGAs• Dedicated hardware designed for a specific task

12

Multi-Core processors• Idea: scale computational power linearly

• Instead of a single 5 GHz core, 2 * 2.5 GHz cores• Scale heat dissipation linearly

• k cores have ~ k times the heat dissipation of a single core• Increasing frequency of a single core by k times creates superlinear heat dissipation increase

13

Memory Bandwidth Bottleneck• Cores compete for the same main memory bus• Caches help in two ways

• They reduce latency (as we have discussed)• They also increase throughput by avoiding bus contention

14

How to Leverage Multicores• Run multiple tasks in parallel

• Multiprocessing• Multithreading

• E.g. PCs have many parallel background apps• OS, music, antivirus, web browser, …

• How to parallelize one app is not trivial• Embarrassingly parallel tasks

• Can be run by multiple threads• No coordination

15

SIMD Processors• Single Instruction Multiple Data (SIMD) processors• Example

• Graphical Processing Units (GPUs)• Intel Phi coprocessors

• Q: Possible SIMD snippets

for i in [0,n-1] dov[i] = v[i] * pi

for i in [0,n-1] doif v[i] < 0.01 then

v[i] = 0

16

Automatic Parallelization?• Holy grail in the multi-processor era• Approaches

• Programming languages• Systems with APIs that help express parallelism• Efficient coordination mechanisms

17

Processes vs. Threads

18

Processes & Threads• We have discussed that multicores is the future• How to make use of parallelism?• OS/PL support for parallel programming

• Processes• Threads

19

Processes vs. Threads• Process: separate memory space• Thread: shared memory space (except stack)

Processes ThreadsHeap not shared sharedGlobal variables not shared sharedLocal variables (Stack) not shared not sharedCode shared sharedFile handles not shared shared

20

Parallel Programming• Shared memory

• Threads • Access same memory locations (in heap & global variables)

• Message-Passing• Processes• Explicit communication: message-passing

Shared Memory

22

Shared Memory Examplevoid main (){

x = 12; // assume that x is a global variablet = new ThreadX();t.start(); // starts thread ty = 12/x;System.out.println(y);t.join(); // wait until t completes

}

class ThreadX extends Thread{void run (){

x = 0;}

}

• Question: What is printed as output?

This is “pseudo-Java”

in C++:pthread_createpthread_join

23

Desired: AtomicityThread a…foo()…

Thread b…foo()…

void foo (){x = 0;x = 1;y = 1/x;

}

x = 0x = 1y = 1

x = 0x = 1y = 1

Thread a Thread b

time

happens-before

changes become visible

DESIRED

x = 0x = 1

y = 1/0

x = 0

Thread a Thread bPOSSIBLE

foo should be atomic, in the sense of indivisible (ancient Greek)

24

Race Condition• Non-deterministic access to shared variables

• Correctness requires specific sequence of accesses

• But we cannot rely on it because of non-determinism!

• Solutions

• Enforce a specific order using synchronization• Enforce a sequence of happen-before relationships

• Locks, mutexes, semaphores: threads block each other

• Lock-free algorithms: threads do not wait for each other• Hard to implement correctly! Typical programmer uses locks

• Java has optimized data structures with thread-safety, e.g., ConcurrentHashMap

25

LocksThread a…l.lock()foo()l.unlock()

Thread b…l.lock()foo()l.unlock()

void foo (){x = 0;x ++;y = 1/x;

}

x = 0x = 1

x = 0

Thread a Thread bImpossible now

l.lock()foo()

Thread a Thread b

time

Possible

l.lock() - waits

foo()l.unlock()

l.unlock()

l.lock() - acquires

We use a lock variable land use it to synchronize

Equivalent: declarevoid synchronized foo()

26

Deadlock

• Question: What can go wrong?

Thread a…l1.lock()l2.lock()foo()l1.unlock()l2.unlock()

Thread b…l2.lock()l1.lock()foo()l2.unlock()l1.unlock()

27

Requirements for a Deadlock• Mutual exclusion: resources (locks) held and non-shareable• Hold and wait: hold a resource and request another• No preemption: can unlock only when holding• Circular wait: chain of threads waiting each other

• Question: Simple solution? • All threads acquire locks in same order

28

Notify / WaitThread a…synchronized(o){

o.wait();foo();

}

Thread b…synchronized(o){

foo();o.notify();

}

o.wait()…

Thread a waits…

Thread a Thread b

foo()o.notify()

o.wait()foo()

Notify on an object sends a signal that activates other threads waiting on that object

This code guarantees that Thread b executes foo before Thread a

29

What About Cache Coherency?• Cache coherency ensures atomicity for

• Single instructions• Single cache lines

• In reality• Different variables may reside on different cache lines• A variable may be accessed across multiple instructions

• Single high-level instructions may compile to multiple low-level ones• Example: a++ in C may compile to load (a, r0); r0 = r0 + 1; store(r0, a)

• That’s why we need locks• Main lesson learned from cache coherency discussion: you should partition data

30

Challenges with Multi-Threading• Correctness

• Heisenbugs: Non-deterministic bugs that appear only in certain conditions.• Hard to reproduce à Hard to debug

• Performance• Understanding concurrency bottlenecks is hard!• “Waiting time” does not show up in profilers (only CPU time)

• Load-balance• Make sure all cores work all the time and do not wait

31

Critical Path• Coordination (barrier) makes load balancing harder• Critical path: Maximum sequential path (thread t1, 10 steps)

t1

t1 t2 t3

start multiple threads

t1wait for all threads

to complete (barrier)

t1

9 extrasteps

one step each

Message Passing

33

Message Passing• Processes communicate by exchanging messages• Sockets: Communication endpoints

• On a network: UDP sockets, TCP sockets• Internal to a node: Inter-Process Communication (IPC)• Different technologies but similar abstractions

34

Building a Message• Serialization

• Message content stored at random locations in RAM • They need to be packed into a byte array to be sent

• Deserialization• Receive the byte array• Rebuild the original variable

• Pointers do not make sense anymore across nodes!

35

Example: Serializing a Binary Tree• Question: How to serialize it?• Possible solution

• DFS• Mark null pointers with -1

• How to deserialize?

10

12null null

5null null 10 -15 -1 12 -1 -1

36

Threads + Message Passing• Client-server model

• Client sends requests• Server computes replies and sends them back

• Threads often used to hide latency• Each client request is handled by a thread• The request might wait for resources (e.g. I/O)• Other threads execute other requests in the meanwhile

37

Processes in Different Languages• Java (interpreted)

• The Java Virtual Machine (interpreter) is a process• Creating a new process entails creating a new JVM

• ProcessBuilder

• C/C++ (compiled)• OS-specific details of how processes can be generated• Typical command: fork()

• Creates a child process, which executes instruction after fork()• Child process is a full copy of the parent

• More on forking later