Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor...

286
Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Transcript of Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor...

Page 1: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Linked Lists: Locking, Lock-Free, and Beyond …

Companion slides forThe Art of Multiprocessor Programming

by Maurice Herlihy & Nir Shavit

Page 2: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 2

Last Lecture: Spin-Locks

CS

Resets lock upon exit

spin lock

critical section

...

Page 3: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 3

Today: Concurrent Objects

• Adding threads should not lower throughput– Contention effects– Mostly fixed by Queue locks

Page 4: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 4

Today: Concurrent Objects

• Adding threads should not lower throughput– Contention effects– Mostly fixed by Queue locks

• Should increase throughput– Not possible if inherently sequential– Surprising things are parallelizable

Page 5: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 5

Coarse-Grained Synchronization

• Each method locks the object– Avoid contention using queue locks

Page 6: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 6

Coarse-Grained Synchronization

• Each method locks the object– Avoid contention using queue locks – Easy to reason about

• In simple cases

Page 7: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 7

Coarse-Grained Synchronization

• Each method locks the object– Avoid contention using queue locks – Easy to reason about

• In simple cases

• So, are we done?

Page 8: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 8

Coarse-Grained Synchronization

• Sequential bottleneck– Threads “stand in line”

Page 9: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 9

Coarse-Grained Synchronization

• Sequential bottleneck– Threads “stand in line”

• Adding more threads– Does not improve throughput– Struggle to keep it from getting worse

Page 10: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 10

Coarse-Grained Synchronization

• Sequential bottleneck– Threads “stand in line”

• Adding more threads– Does not improve throughput– Struggle to keep it from getting worse

• So why even use a multiprocessor?– Well, some apps inherently parallel …

Page 11: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 11

This Lecture

• Introduce four “patterns”– Bag of tricks …– Methods that work more than once …

Page 12: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 12

This Lecture

• Introduce four “patterns”– Bag of tricks …– Methods that work more than once …

• For highly-concurrent objects– Concurrent access– More threads, more throughput

Page 13: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 13

First:Fine-Grained Synchronization

• Instead of using a single lock …

• Split object into– Independently-synchronized components

• Methods conflict when they access– The same component …– At the same time

Page 14: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 14

Second:Optimistic Synchronization

• Search without locking …

Page 15: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 15

Second:Optimistic Synchronization

• Search without locking …

• If you find it, lock and check …– OK: we are done– Oops: start over

Page 16: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 16

Second:Optimistic Synchronization

• Search without locking …

• If you find it, lock and check …– OK: we are done– Oops: start over

• Evaluation– Usually cheaper than locking, but– Mistakes are expensive

Page 17: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 17

Third:Lazy Synchronization

• Postpone hard work

• Removing components is tricky– Logical removal

• Mark component to be deleted

– Physical removal• Do what needs to be done

Page 18: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 18

Fourth:Lock-Free Synchronization

• Don’t use locks at all– Use compareAndSet() & relatives …

Page 19: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 19

Fourth:Lock-Free Synchronization

• Don’t use locks at all– Use compareAndSet() & relatives …

• Advantages– No Scheduler Assumptions/Support

Page 20: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 20

Fourth:Lock-Free Synchronization

• Don’t use locks at all– Use compareAndSet() & relatives …

• Advantages– No Scheduler Assumptions/Support

• Disadvantages– Complex– Sometimes high overhead

Page 21: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 21

Linked List

• Illustrate these patterns …

• Using a list-based Set– Common application– Building block for other apps

Page 22: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 22

Set Interface

• Unordered collection of items

Page 23: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 23

Set Interface

• Unordered collection of items

• No duplicates

Page 24: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 24

Set Interface

• Unordered collection of items

• No duplicates

• Methods– add(x) put x in set– remove(x) take x out of set– contains(x) tests if x in set

Page 25: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 25

List-Based Sets

public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x);}

Page 26: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 26

List-Based Sets

public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x);}

Add item to set

Page 27: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 27

List-Based Sets

public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(Tt x);}

Remove item from set

Page 28: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 28

List-Based Sets

public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x);}

Is item in set?

Page 29: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 29

List Node

public class Node { public T item; public int key; public Node next;}

Page 30: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 30

List Node

public class Node { public T item; public int key; public Node next;}

item of interest

Page 31: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 31

List Node

public class Node { public T item; public int key; public Node next;}

Usually hash code

Page 32: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 32

List Node

public class Node { public T item; public int key; public Node next;}

Reference to next node

Page 33: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 33

The List-Based Set

a b c

Sorted with Sentinel nodes(min & max possible keys)

-∞

+∞

Page 34: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 34

Reasoning about Concurrent Objects

• Invariant– Property that always holds

Page 35: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 35

Reasoning about Concurrent Objects

• Invariant– Property that always holds

• Established because– True when object is created– Truth preserved by each method

• Each step of each method

Page 36: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 36

Specifically …

• Invariants preserved by– add()– remove()– contains()

Page 37: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 37

Specifically …

• Invariants preserved by– add()– remove()– contains()

• Most steps are trivial– Usually one step tricky– Often linearization point

Page 38: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 38

Interference

• Invariants make sense only if – methods considered– are the only modifiers

Page 39: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 39

Interference

• Invariants make sense only if – methods considered– are the only modifiers

• Language encapsulation helps– List nodes not visible outside class

Page 40: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 40

Interference

• Invariants make sense only if – methods considered– are the only modifiers

• Language encapsulation helps– List nodes not visible outside class

Page 41: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 41

Interference

• Freedom from interference needed even for removed nodes– Some algorithms traverse removed nodes– Careful with malloc() & free()!

• Garbage collection helps here

Page 42: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 42

Abstract Data Types

• Concrete representation:

• Abstract Type:– {a, b}

a b

Page 43: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 43

Abstract Data Types

• Meaning of rep given by abstraction map

– S( ) = {a,b}a b

Page 44: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 44

Rep Invariant

• Which concrete values meaningful?– Sorted?– Duplicates?

• Rep invariant– Characterizes legal concrete reps– Preserved by methods– Relied on by methods

Page 45: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 45

Blame Game

• Rep invariant is a contract

• Suppose– add() leaves behind 2 copies of x– remove() removes only 1

• Which is incorrect?

Page 46: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 46

Blame Game

• Suppose– add() leaves behind 2 copies of x– remove() removes only 1

Page 47: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 47

Blame Game

• Suppose– add() leaves behind 2 copies of x– remove() removes only 1

• Which is incorrect?– If rep invariant says no duplicates

• add() is incorrect

– Otherwise• remove() is incorrect

Page 48: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 48

Rep Invariant (partly)

• Sentinel nodes– tail reachable from head

• Sorted

• No duplicates

Page 49: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 49

Abstraction Map

• S(head) =– { x | there exists a such that

• a reachable from head and• a.item = x

– }

Page 50: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 50

Sequential List Based Set

a c d

a b c

Add()

Remove()

Page 51: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 51

Sequential List Based Set

a c d

b

a b c

add()

remove()

Page 52: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 52

Coarse-Grained Locking

a b d

Page 53: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 53

Coarse-Grained Locking

a b d

c

Page 54: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 54

honk!

Coarse-Grained Locking

a b d

c

Simple but hotspot + bottleneck

honk!

Page 55: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 55

Coarse-Grained Locking

• Easy, same as synchronized methods– “One lock to rule them all …”

Page 56: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 56

Coarse-Grained Locking

• Easy, same as synchronized methods– “One lock to rule them all …”

• Simple, clearly correct– Deserves respect!

• Works poorly with contention– Queue locks help– But bottleneck still an issue

Page 57: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 57

Fine-grained Locking

• Requires careful thought– “Do not meddle in the affairs of wizards, for

they are subtle and quick to anger”

Page 58: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 58

Fine-grained Locking

• Requires careful thought– “Do not meddle in the affairs of wizards, for

they are subtle and quick to anger”

• Split object into pieces– Each piece has own lock– Methods that work on disjoint pieces need

not exclude each other

Page 59: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 59

Hand-over-Hand locking

a b c

Page 60: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 60

Hand-over-Hand locking

a b c

Page 61: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 61

Hand-over-Hand locking

a b c

Page 62: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 62

Hand-over-Hand locking

a b c

Page 63: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 63

Hand-over-Hand locking

a b c

Page 64: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 64

Removing a Node

a b c d

remove(b)

Page 65: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 65

Removing a Node

a b c d

remove(b)

Page 66: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 66

Removing a Node

a b c d

remove(b)

Page 67: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 67

Removing a Node

a b c d

remove(b)

Page 68: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 68

Removing a Node

a b c d

remove(b)

Page 69: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 69

Removing a Node

a c d

remove(b)Why hold 2 locks?

Page 70: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 70

Concurrent Removes

a b c d

remove(c)remove(b)

Page 71: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 71

Concurrent Removes

a b c d

remove(b)remove(c)

Page 72: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 72

Concurrent Removes

a b c d

remove(b)remove(c)

Page 73: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 73

Concurrent Removes

a b c d

remove(b)remove(c)

Page 74: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 74

Concurrent Removes

a b c d

remove(b)remove(c)

Page 75: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 75

Concurrent Removes

a b c d

remove(b)remove(c)

Page 76: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 76

Concurrent Removes

a b c d

remove(b)remove(c)

Page 77: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 77

Concurrent Removes

a b c d

remove(b)remove(c)

Page 78: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 78

Concurrent Removes

a b c d

remove(b)remove(c)

Page 79: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 79

Concurrent Removes

a b c d

remove(b)remove(c)

Page 80: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 80

Uh, Oh

a c d

remove(b)remove(c)

Page 81: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 81

Uh, Oh

a c d

Bad news, c not removed

remove(b)remove(c)

Page 82: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 82

Problem

• To delete node c– Swing node b’s next field to d

• Problem is,– Someone deleting b concurrently could

direct a pointer

to c

ba c

ba c

Page 83: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 83

Insight

• If a node is locked– No one can delete node’s successor

• If a thread locks– Node to be deleted– And its predecessor– Then it works

Page 84: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 84

Hand-Over-Hand Again

a b c d

remove(b)

Page 85: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 85

Hand-Over-Hand Again

a b c d

remove(b)

Page 86: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 86

Hand-Over-Hand Again

a b c d

remove(b)

Page 87: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 87

Hand-Over-Hand Again

a b c d

remove(b)Found

it!

Page 88: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 88

Hand-Over-Hand Again

a b c d

remove(b)Found

it!

Page 89: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 89

Hand-Over-Hand Again

a c d

remove(b)

Page 90: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 90

Removing a Node

a b c d

remove(b)remove(c)

Page 91: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 91

Removing a Node

a b c d

remove(b)remove(c)

Page 92: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 92

Removing a Node

a b c d

remove(b)remove(c)

Page 93: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 93

Removing a Node

a b c d

remove(b)remove(c)

Page 94: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 94

Removing a Node

a b c d

remove(b)remove(c)

Page 95: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 95

Removing a Node

a b c d

remove(b)remove(c)

Page 96: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 96

Removing a Node

a b c d

remove(b)remove(c)

Page 97: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 97

Removing a Node

a b c d

remove(b)remove(c)

Page 98: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 98

Removing a Node

a b c d

Must acquire Lock for

b

remove(c)

Page 99: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 99

Removing a Node

a b c d

Cannot acquire

lock for b

remove(c)

Page 100: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 100

Removing a Node

a b c d

Wait!remove(c)

Page 101: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 101

Removing a Node

a b d

Proceed to

remove(b)

Page 102: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 102

Removing a Node

a b d

remove(b)

Page 103: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 103

Removing a Node

a b d

remove(b)

Page 104: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 104

Removing a Node

a d

remove(b)

Page 105: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 105

Removing a Node

a d

Page 106: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 106

Remove method

public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Page 107: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 107

Remove method

public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Key used to order node

Page 108: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 108

Remove method

public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }}

Predecessor and current nodes

Page 109: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 109

Remove method

public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Make sure locks released

Page 110: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 110

Remove method

public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

Everything else

Page 111: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 111

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

Page 112: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 112

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

lock pred == head

Page 113: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 113

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

Lock current

Page 114: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 114

Remove method

try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); …} finally { … }

Traversing list

Page 115: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 115

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Page 116: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 116

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Search key range

Page 117: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 117

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

At start of each loop: curr and pred locked

Page 118: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 118

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;If item found, remove node

Page 119: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 119

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;If node found, remove it

Page 120: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 120

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Unlock predecessor

Page 121: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 121

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Only one node locked!

Page 122: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 122

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

demote current

Page 123: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 123

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false;

Find and lock new current

Page 124: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 124

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false;

Lock invariant restored

Page 125: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 125

Remove: searching

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Otherwise, not present

Page 126: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 127

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

•pred reachable from head•curr is pred.next•So curr.item is in the set

Page 127: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 128

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Linearization point ifitem is present

Page 128: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 129

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Node locked, so no other thread can remove it ….

Page 129: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 130

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Item not present

Page 130: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 131

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

•pred reachable from head•curr is pred.next•pred.key < key •key < curr.key

Page 131: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 132

while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

Why remove() is linearizable

Linearization point

Page 132: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 133

Adding Nodes

• To add node e– Must lock predecessor– Must lock successor

• Neither can be deleted– (Is successor lock actually required?)

Page 133: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 134

Same Abstraction Map

• S(head) =– { x | there exists a such that

• a reachable from head and• a.item = x

– }

Page 134: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 135

Rep Invariant

• Easy to check that– tail always reachable from head– Nodes sorted, no duplicates

Page 135: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 136

Drawbacks

• Better than coarse-grained lock– Threads can traverse in parallel

• Still not ideal– Long chain of acquire/release– Inefficient

Page 136: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 137

Optimistic Synchronization

• Find nodes without locking

• Lock nodes

• Check that everything is OK

Page 137: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 138

Optimistic: Traverse without Locking

b d ea

add(c) Aha!

Page 138: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 139

Optimistic: Lock and Load

b d ea

add(c)

Page 139: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 140

Optimistic: Lock and Load

b d ea

add(c)

c

Page 140: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 141

What could go wrong?

b d ea

add(c) Aha!

Page 141: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 142

What could go wrong?

b d ea

add(c)

Page 142: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 143

What could go wrong?

b d ea

remove(b)

Page 143: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 144

What could go wrong?

b d ea

remove(b)

Page 144: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 145

What could go wrong?

b d ea

add(c)

Page 145: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 146

What could go wrong?

b d ea

add(c)

c

Page 146: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 147

What could go wrong?

d ea

add(c) Uh-oh

Page 147: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 148

Validate – Part 1

b d ea

add(c) Yes, b still reachable from head

Page 148: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 149

What Else Could Go Wrong?

b d ea

add(c) Aha!

Page 149: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 150

What Else Coould Go Wrong?

b d ea

add(c)

add(b’)

Page 150: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 151

What Else Coould Go Wrong?

b d ea

add(c)

add(b’)b’

Page 151: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 152

What Else Could Go Wrong?

b d ea

add(c)b’

Page 152: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 153

What Else Could Go Wrong?

b d ea

add(c)

c

Page 153: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 154

Validate Part 2(while holding locks)

b d ea

add(c) Yes, b still points to d

Page 154: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 155

Optimistic: Linearization Point

b d ea

add(c)

c

Page 155: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 156

Same Abstraction Map

• S(head) =– { x | there exists a such that

• a reachable from head and• a.item = x

– }

Page 156: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 157

Invariants

• Careful: we may traverse deleted nodes

• But we establish properties by– Validation– After we lock target nodes

Page 157: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 158

Correctness

• If– Nodes b and c both locked– Node b still accessible– Node c still successor to b

• Then– Neither will be deleted– OK to delete and return true

Page 158: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 159

Unsuccessful Remove

a b d e

remove(c)

Aha!

Page 159: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 160

Validate (1)

a b d e

Yes, b still reachable from head

remove(c)

Page 160: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 161

Validate (2)

a b d e

remove(c) Yes, b still points to d

Page 161: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 162

OK Computer

a b d e

remove(c) return false

Page 162: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 163

Correctness

• If– Nodes b and d both locked– Node b still accessible– Node d still successor to b

• Then– Neither will be deleted– No thread can add c after b– OK to return false

Page 163: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 164

Validationprivate boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Page 164: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 165

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Predecessor & current nodes

Page 165: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 166

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Begin at the beginning

Page 166: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 167

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Search range of keys

Page 167: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 168

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Predecessor reachable

Page 168: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 169

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Is current node next?

Page 169: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 170

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

Validation

Otherwise move on

Page 170: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 171

private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false;}

ValidationPredecessor not reachable

Page 171: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 172

Remove: searchingpublic boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Page 172: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 173

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Search key

Page 173: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 174

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Retry on synchronization conflict

Page 174: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 175

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Examine predecessor and current nodes

Page 175: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 176

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Search by key

Page 176: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 177

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Stop if we find item

Page 177: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 178

public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

Remove: searching

Move along

Page 178: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 179

On Exit from Loop

• If item is present– curr holds item– pred just before curr

• If item is absent– curr has first higher key– pred just before curr

• Assuming no synchronization problems

Page 179: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 180

Remove Methodtry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Page 180: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 181

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Always unlock

Page 181: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 182

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Lock both nodes

Page 182: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 183

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

Check for synchronization conflicts

Page 183: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 184

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

target found, remove node

Page 184: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 185

try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Remove Method

target not found

Page 185: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 186

Optimistic List

• Limited hot-spots– Targets of add(), remove(), contains()– No contention on traversals

• Moreover– Traversals are wait-free– Food for thought …

Page 186: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 187

So Far, So Good

• Much less lock acquisition/release– Performance– Concurrency

• Problems– Need to traverse list twice– contains() method acquires locks

Page 187: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 188

Evaluation

• Optimistic is effective if– cost of scanning twice without locks

is less than– cost of scanning once with locks

• Drawback– contains() acquires locks– 90% of calls in many apps

Page 188: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 189

Lazy List

• Like optimistic, except– Scan once– contains(x) never locks …

• Key insight– Removing nodes causes trouble– Do it “lazily”

Page 189: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 190

Lazy List

• remove()– Scans list (as before)– Locks predecessor & current (as before)

• Logical delete– Marks current node as removed (new!)

• Physical delete– Redirects predecessor’s next (as before)

Page 190: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 191

Lazy Removal

aa b c d

Page 191: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

c

Art of Multiprocessor Programming 192

Lazy Removal

aa b d

Present in list

Page 192: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

c

Art of Multiprocessor Programming 193

Lazy Removal

aa b d

Logically deleted

Page 193: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 194

Lazy Removal

aa b c d

Physically deleted

Page 194: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 195

Lazy Removal

aa b d

Physically deleted

Page 195: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 196

Lazy List

• All Methods– Scan through locked and marked nodes– Removing a node doesn’t slow down other

method calls …

• Must still lock pred and curr nodes.

Page 196: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 197

Validation

• No need to rescan list!

• Check that pred is not marked

• Check that curr is not marked

• Check that pred points to curr

Page 197: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 198

Business as Usual

a b c

Page 198: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 199

Business as Usual

a b c

Page 199: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 200

Business as Usual

a b c

Page 200: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 201

Business as Usual

a b c

remove(b)

Page 201: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 202

Business as Usual

a b c

a not marked

Page 202: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 203

Business as Usual

a b c

a still points

to b

Page 203: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 204

Business as Usual

a b c

Logical delete

Page 204: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 205

Business as Usual

a b c

physical delete

Page 205: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 206

Business as Usual

a b c

Page 206: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 207

New Abstraction Map

• S(head) =– { x | there exists node a such that

• a reachable from head and• a.item = x and• a is unmarked

– }

Page 207: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 208

Invariant

• If not marked then item in the set

• and reachable from head

• and if not yet traversed it is reachable from pred

Page 208: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 209

Validationprivate boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

Page 209: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 210

private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

List Validate Method

Predecessor not Logically removed

Page 210: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 211

private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

List Validate Method

Current not Logically removed

Page 211: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 212

private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

List Validate Method

Predecessor stillPoints to current

Page 212: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 213

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Page 213: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 214

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Validate as before

Page 214: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 215

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Key found

Page 215: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 216

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

Logical remove

Page 216: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 217

Removetry { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally {

pred.unlock();curr.unlock();

}}}

physical remove

Page 217: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 218

Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Page 218: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 219

Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Start at the head

Page 219: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 220

Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Search key range

Page 220: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 221

Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Traverse without locking(nodes may have been removed)

Page 221: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 222

Containspublic boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked;}

Present and undeleted?

Page 222: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 223

Summary: Wait-free Contains

a 0 0 0a b c 0e1d

Use Mark bit + list ordering 1. Not marked in the set2. Marked or missing not in the set

Page 223: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 224

Lazy List

a 0 0 0a b c 0e1d

Lazy add() and remove() + Wait-free contains()

Page 224: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 225

Evaluation

• Good:– contains() doesn’t lock– In fact, its wait-free! – Good because typically high % contains()– Uncontended calls don’t re-traverse

• Bad– Contended add() and remove() calls do re-

traverse– Traffic jam if one thread delays

Page 225: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 226

Traffic Jam

• Any concurrent data structure based on mutual exclusion has a weakness

• If one thread– Enters critical section– And “eats the big muffin”

• Cache miss, page fault, descheduled …

– Everyone else using that lock is stuck!– Need to trust the scheduler….

Page 226: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 227

Reminder: Lock-Free Data Structures

• No matter what …– Guarantees minimal progress in any

execution– i.e. Some thread will always complete a

method call– Even if others halt at malicious times– Implies that implementation can’t use locks

Page 227: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 228

Lock-free Lists

• Next logical step– Wait-free contains()– lock-free add() and remove()

• Use only compareAndSet()– What could go wrong?

Page 228: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 229

a 0 0 0a b c 0e1c

Logical Removal

Physical RemovalUse CAS to verify pointer is correct

Not enough!

Lock-free Lists

Page 229: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 230

Problem…

a 0 0 0a b c 0e1c

Logical Removal

Physical Removal0d

Node added

Page 230: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 231

The Solution: Combine Bit and Pointer

a 0 0 0a b c 0e1c

Logical Removal =Set Mark Bit

PhysicalRemovalCAS

0d

Mark-Bit and Pointerare CASed together(AtomicMarkableReference)

Fail CAS: Node not added after logical Removal

Page 231: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 232

Solution

• Use AtomicMarkableReference

• Atomically– Swing reference and– Update flag

• Remove in two steps– Set mark bit in next field– Redirect predecessor’s pointer

Page 232: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 233

Marking a Node

• AtomicMarkableReference class– Java.util.concurrent.atomic package

address F

mark bit

Reference

Page 233: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 234

Extracting Reference & Mark

Public Object get(boolean[] marked);

Page 234: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 235

Extracting Reference & Mark

Public Object get(boolean[] marked);

Returns reference

Returns mark at array index 0!

Page 235: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 236

Extracting Mark Only

public boolean isMarked();

Value of mark

Page 236: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 237

Changing State

Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark);

Page 237: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 238

Changing State

Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark);

If this is the current reference …

And this is the current mark …

Page 238: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 239

Changing State

Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark);

…then change to this new reference …

… and this new mark

Page 239: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 240

Changing State

public boolean attemptMark( Object expectedRef, boolean updateMark);

Page 240: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 241

Changing State

public boolean attemptMark( Object expectedRef, boolean updateMark);

If this is the current reference …

Page 241: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 242

Changing State

public boolean attemptMark( Object expectedRef, boolean updateMark);

.. then change to this new mark.

Page 242: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

bCAS

Art of Multiprocessor Programming 243

Removing a Node

a c d

remove c

Page 243: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 244

Removing a Node

a b d

remove b

remove c

c

failed

CAS CAS

Page 244: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 245

Removing a Node

a b d

remove b

remove c

c

Page 245: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 246

Removing a Node

a d

remove b

remove c

Page 246: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 247

Traversing the List

• Q: what do you do when you find a “logically” deleted node in your path?

• A: finish the job.– CAS the predecessor’s next field– Proceed (repeat as needed)

Page 247: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 248

Lock-Free Traversal(only Add and Remove)

a b c dCAS

Uh-oh

pred currpred curr

Page 248: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 249

The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}

Page 249: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 250

The Window Class

class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }}

A container for pred and current values

Page 250: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 251

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

Page 251: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 252

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

Find returns window

Page 252: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 253

Using the Find Method

Window window = find(head, key); Node pred = window.pred; curr = window.curr;

Extract pred and curr

Page 253: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming© Herlihy-Shavit 2007

254

The Find Method

Window window = find(item);

At some instant,

pred curr succ

itemor …

Page 254: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming© Herlihy-Shavit 2007

255

The Find Method

Window window = find(item);

At some instant,

predcurr= null

succ

item not in list

Page 255: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 256

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Page 256: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 257

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet (succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Keep trying

Page 257: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 258

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet (succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} Find neighbors

Page 258: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 259

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}} She’s not there …

Page 259: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 260

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Try to mark node as deleted

Page 260: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 261

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

If it doesn’t work, just retry, if it

does, job essentially done

Page 261: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 262

Removepublic boolean remove(T item) {Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true;}}}

Try to advance reference(if we don’t succeed, someone else did or will).

a

Page 262: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 263

Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}

Page 263: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 264

Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}} Item already there.

Page 264: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 265

Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}

create new node

Page 265: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 266

Addpublic boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;}}}}

Install new node, else retry loop

Page 266: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 267

Wait-free Contains

public boolean contains(T item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

Page 267: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 268

Wait-free Contains

public boolean contains(T item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

Only diff is that we get and check

marked

Page 268: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 269

Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Page 269: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 270

Lock-free Findpublic Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

If list changes while traversed,

start over

Page 270: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 271

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Start looking from head

Page 271: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 272

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Move down the list

Page 272: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 273

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Get ref to successor and current deleted bit

Page 273: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 274

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Try to remove deleted nodes in path…code details soon

Page 274: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 275

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

If curr key that is greater or equal, return pred and curr

Page 275: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 276

public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; }}}

Lock-free Find

Otherwise advance window and loop again

Page 276: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 277

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

Page 277: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 278

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

Try to snip out node

Page 278: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 279

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

if predecessor’s next field changed, retry whole traversal

Page 279: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 280

Lock-free Find

retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); }…

Otherwise move on to check if next node deleted

Page 280: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Performance

• Different list-based set implementaions

• 16-node machine

• Vary percentage of contains() calls

Art of Multiprocessor Programming 281

Page 281: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 282

High Contains Ratio

Lock-free Lazy list

Coarse GrainedFine Lock-coupling

Page 282: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 283

Low Contains Ratio

Lock-free

Lazy list

Coarse GrainedFine Lock-coupling

Page 283: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 284

As Contains Ratio Increases

Lock-free

Lazy list

Coarse GrainedFine Lock-coupling

% Contains()

Page 284: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 285

Summary

• Coarse-grained locking

• Fine-grained locking

• Optimistic synchronization

• Lazy synchronization

• Lock-free synchronization

Page 285: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 286

“To Lock or Not to Lock”

• Locking vs. Non-blocking:– Extremist views on both sides

• The answer: nobler to compromise– Example: Lazy list combines blocking add()

and remove()and a wait-free contains()– Remember: Blocking/non-blocking is a property

of a method

Page 286: Linked Lists: Locking, Lock-Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit.

Art of Multiprocessor Programming 287

         This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.

• You are free:– to Share — to copy, distribute and transmit the work – to Remix — to adapt the work

• Under the following conditions:– Attribution. You must attribute the work to “The Art of

Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work).

– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.

• For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to– http://creativecommons.org/licenses/by-sa/3.0/.

• Any of the above conditions can be waived if you get permission from the copyright holder.

• Nothing in this license impairs or restricts the author's moral rights.