Calvin:Deterministic or Not?Free Will to Choose

Derek R. Hower, Polina Dudnik,Mark D. Hill, David A. Wood

Executive Summary• Determinism Valuable:

– Same inputs Same multithreaded execution– Debugging, Fault Tolerance, Security

• Performance Required:– Slow & deterministic not enough

• Propose: Calvin– Leverages Total Store Order (TSO) in hardware to... – … deterministically order memory operations

• Multiple modes w/o speculation– 20% Deterministic (vs. software 1-11X)– 8% Conventional

Determinism @ Good Performance

Outline• Motivation & Goals

• Model

• Implementation

• Evaluation

• Conclusion

• Related Work (optional)

Want Deterministic Execution

if (account >= sum)

account -= sum; account -= sum;

if (account >= sum)account = 100

account = 0

account = 0

account = 0

Bug: unprotected account update

thread 0

Bug: unprotected account update

Want Deterministic Execution

thread 0if (account >= sum)

account -= sum;

account -= sum;

if (account >= sum)account = 100

account = 100

account = 0

account = -100

Specific Goals• Strong Determinism:

– Make no assumptions about program behavior

– Help debug racey programs

• Performance:– Small enough overhead to

be on all the time

• Compatibility:– Complex speculative cores– Non-speculative cores

Strong Determinism Performance

Compatibility

Outline• Motivation & Goals

• Model

• Implementation

• Evaluation

• Conclusion

• Related Work (optional)

Proc

1

Proc

0

Calvin: The Big Picture

Load A

Load C

Store B

Store D

Mem

ory

Orde

r

Load D

Store B

Store ALoad A

Recall Total Store Order (TSO)…• TSO is a Relaxed memory model• Key point: write completion can be delayed

processor 0ST A <- 1

R1 <- LD B

ST A <- 1

R1 <- LD B

ST A <- 1

R1 <- LD B

ST A <- 1

R1 <- LD B

Mem

ory

Orde

rPC ->local

buffering

R2 <- LD AR2 <- LD A

Buff

erBuff

er Proc

1

Proc

0

Calvin Model: One InterleavingM

emor

y Or

der

Load A

Load C

Store B

Store DLoad D

Store B

Store ALoad A

Load A

Load C

Store B

Store DLoad D

Store B

Store ALoad A

Exec

ute

Publ

is h

1) all loads before all stores (execute)

2) all stores in processor order (publish)

Exec

ute

Publ

is hPRO

CESS

OR

0

PRO

CESS

OR

1

Calvin Model: Reduce Scope• Temporally divide multithreaded execution into global strata

Stratum S

Stratum S + 1Begin Stratum

Begin Stratum

Tim

e

Load

Load

Load

Store

StoreLoad

Store

StoreLoad

Load

Store

StoreLoad

Load

Store

Store

Load

Store

Load

Store

Store

Load

Load

Load LoadStore

Exec

ute

Publ

is h

End Stratum and Synchronize

End Stratum and Synchronize

Stratum Termination Function (3 Modes)

1. Unbounded deterministic:– determinism architectural events only, e.g. instructions– (#instructions == threshold) OR synchronization

2. Conventional:– performance reduce load imbalance, e.g. cycle count– (#cycles == threshold) OR synchronization

2. Bounded deterministic:– determinism architectural events only, e.g. instructions– (#instructions == threshold) OR (synchronization) OR (resource exhaustion)

Outline• Motivation & Goals

• Model

• Implementation– Write Cache– MIST Protocol– Stratum Size Predictor

• Evaluation

• Conclusion

• Related Work (optional)

Implementation: Overview• Implementation Challenges:– Stratification Load imbalance due to barriers– Buffering Conventional store buffers do not

scale– Ordering Serial flush is sloooooooow

• Calvin-MIST Implementation:– Store buffers Unordered write cache– Load imbalance Stratum Size Predictor (in

paper)– Fast flush MIST Coherence Protocol

Proc

1Pr

oc 0Load

ALoad C

Load B

Load A

Exec

ute

Publ

is h

Unordered Write Cache• Behavior:

– drops program store ordering– coalesces stores– prohibits loads in publish phase

• Replacements/overflow:1. End stratum

– Bounded Deterministic Mode– Repeatable only on same HW

2. Log (TM-like)– Unbounded Deterministic

Mode– Repeatable on any HW

Store BStore D

Store A

Atomic Flush

Store D

MIST Protocol• Goal: speed up publish phase– delayed “timebomb” invalidate (in paper) – write caches flush in parallel

Proc

1

Proc

0

Load ALoad C Load B

Load A

Exec

ute

Publ

ishStore B

Store DStore AStore D

Outline• Motivation & Goals

• Model

• Implementation

• Evaluation

• Conclusion

• Related Work (optional)

Evaluation Methodology• Infrastructure

– Bochs– GEMS

• Workloads

– Parsec– Mantevo

Base Calvin-MIST

Cores 8, 2.0 Ghz in-order pipelined

Write Cache N/A 64 entry, 8 way

L1 Cache Private, Split L1 I&D, 32K 8-way, 1 cycle

Coherence Protocol

Conventional MOESI Multiple Writer MIST

Barrier N/A 16 cycle latency

L2 Cache Shared, 8MB, 16-way, 8 banks, 12 cycles

Directory Distributed at the L2 banks

Unbounded Deterministic Mode

0

0.5

1

1.5

2

2.5 UDBDC

Nor

mal

ized

Exe

cuti

on T

ime publis

h~20%

slowdown

fine-grained locking

frequent overflow

Bounded Deterministic Mode

0

0.5

1

1.5

2

2.5 phase2UDBDC

Nor

mal

ized

Exe

cuti

on T

ime publis

h~20% simpler HW

better stratum

size

Conventional Mode

0

0.5

1

1.5

2

2.5 logphase2UDBDC

Nor

mal

ized

Exe

cuti

on T

ime publis

h~8% slowdown

bad stratum

size

Outline• Motivation & Goals

• Model

• Implementation

• Evaluation

• Conclusion

• Related Work (optional)

Conclusion• Determinism Valuable:

– Same inputs Same multithreaded execution– Debugging, Fault Tolerance, Security

• Performance Required:– Uninteresting to be slow & deterministic

• Propose: Calvin– Leverages TSO in hardware to... – … deterministically order memory operations

• Multiple modes w/o speculation– 20% Deterministic– 8% Conventional

Determinism @ Good Performance

Outline• Motivation & Goals

• Model

• Implementation

• Evaluation

• Conclusion

• Related Work (optional)

Related Work• DMP [Devietti, J. et al., ASPLOS ‘09]

– First hardware solution for strong determinism– Good performance through TM-like speculation– Calvin seeks good performance with less speculation (power?)

• Kendo [Olszewski, M. et. al., ASPLOS ‘09]– First software solution for weak determinism– Good performance, but not as general (e.g., debugging data races)– Calvin seeks good performance for strong determinism

• CoreDet [Bergan, T. et al., ASPLOS ‘10]– First software solution for strong determinism– Exploits relaxed model, e.g., TSO with software store buffer– Performance left room for improvement– Calvin implements similar ideas in hardware to be fast

Questions?

Backup Slides Follow

R0 = 2R1 = 1

R2 = 0

Calvin Model

Stratum S

Mem

ory

Orde

r

processor 0ST A <- 1

R2 <- LD A

R1 <- LD B

ST A <- 2processor 1

ST B <- 3

R0 <- LD A

BufferBuffer

A = 1 A = 2B = 3

Exec

ute

Publ

is h

• Deterministically order memory operations within stratum• All loads before all stores• All stores are ordered by processor

Coherence Protocol• Write-back protocol• Allows parallel write cache flush• Allows fast reader invalidate

# states MIST MESI MOESIStable @

L1 6 4 7Transient @

L1 12 6 8Stable @

L2 5 3 13Transient @

L2 17 14 46Total 40 27 74

L1 Cache States

State Meaning Global Invariant

I Not Present/Invalid 0 or more readers, 0 or more writers

S Read Permission, no other writers in the system

1 or more readers, 0 writers

M Write permission, didn’t write in current stratum

0 readers, 1 writer

Ts Read permission until the end of the stratum

1 or more readers, 1 or more writers

Mw Write permission, wrote in current stratum 0 readers, 1 writer

MMw Write permission until the end of the stratum

2 or more writers, 0 or more readers

Directory States

State Meaning Global Invariant Valid Copy @

I Not Present/Invalid 0 readers, 0 writers Memory

S One or more readers

1 or more readers, 0 writers L2 Cache

M Only one writer 0 or more readers, 1 writer Processor

MM No readers/writers 0 readers, 0 writers L2 Cache

MS Multiple writers 0 or more readers, 1 or more writers L2 Cache

Stratum Size Predictor• Stratum Size

Predictor:– optimizes stratum size– adopts to loads

imbalance

• Large stratum:– reduce instruction mix

variability

• Small stratum:– adopt to synchronization

Proc

1

Proc

0

L1

Cach

e

L1

Cach

e

Reader Self-InvalidationTi

me

Exec

ute

Publ

is h

L2 C

acheB:

Shared

Processor 0 Processor 1

B: Shared B: Shared

LDST

Intent

B: Shared B: ModifiedB: Modified

B: Shared B: ModifiedB: Modified

Predictor

MemBar?C&BD:

Overflow?

Stratum Ends

Saturated?

Decrement Predictor

Increment Predictor

Size*2 Size/2

No Yes

Yes/Low

Yes/High

Stratum Ends

No

Predictor Helps Improve Performance

beam blc

kbd

trde

dup

epetr

aflu

id freq

hpccg

minimd

phpcc

g ray swap vip

sx26

4mea

n

-0.1

-0.05

0

0.05

0.1

0.15 CBDUD

Spee

dup

Write Cache Size Affects Performance

.0

0.5

1

1.5

2

2.5

log phase2 64E_8W 32E_8W16E_8W

Nor

mal

ized

Exe

cuti

on T

ime

Bottom Line

0

0.5

1

1.5

2

2.5 logphase2UDBDC

Nor

mal

ized

Exe

cuti

on T

ime publis

h

Mantevo

Calvin-MIST Operation

Example Protocol Operation

Atomic Operations• Ensure that only one atomic operation executes

per stratum• Logically place the atomic operation at the end of

the stratum

• Terminate stratum on atomic operation• Execute both R and W parts of RMW as

processor’s last store• Allows processors to communicate within a

stratum

Multi-Writer Example

Core 2Core 1L1 Cache L1 Cache

Write Cache Write CacheExecution PhasePublish Phase

FWD FWD

L2 CacheACKNAC

KACK

43

Atomic Operations• TSO atomic ordering rules:

1) All previous loads and stores2) Atomic (both load and store portion)3) All subsequent loads and stores

• Calvin satisfies rules by:1) Ending strata on atomics2) Executing atomic op entirely in publish phase3) Executing next instruction in next strata

44

Atomic Example

Proc

1

Proc

0

Load A

Load A

Store A

Store L

Load C

Store C

Store B

Load B

Mem

ory

Ord

er

RMW LLoad A

Store C

Stall

45

Deterministic Input• Program’s repeatability depends on deterministic

input

• Input:– Use mechanisms from uniprocessor deterministic replay,

e.g.:• Revirt• VMware Replay• FDR

• Interrupts:– Delivered only on strata boundaries

• Makes for easy logging (e.g., <vector #, strata #>)

46

Conventional Mode Slowdown• Sources:

– Barrier latency (16 cycle)• Results indicate 4 cycle barrier largely eliminates overhead

– Load imbalance• Especially in presence of fine-grained communication

– Slow inter-thread communication• Threads cannot communicate within a stratum

With Average Stratum Size

. beam blck bdtr dedup epetra fluid freq hpccg minimd phpccg ray swap vips x264 mean0

0.5

1

1.5

2

2.5

1312

6

8984

5135

1071

1521

5

571

5948

1214

8

5476

1206

2

4584 1363

8

1235

7

1203

4

1.03

9012

4293

4226

3257

3132

1503 540

3568

105

2542 25

02

1938

2386

1254

2849

3001

3153

1.16

5101

5072

3813

3269

3132

1497

534

3574

104

2855 25

60

2307

2426

1453

3378

3035

3229

1.17

6691

4707

0772

logphase2UDBDC