Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015...
Transcript of Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015...
![Page 1: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/1.jpg)
Reproducibility of Experiments in Concurrent Programming
Vincent Gramoli
1
![Page 2: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/2.jpg)
Roadmap• Why reproducibility?
• Towards the reproducibility of an experiment?
• Concurrent experiments cannot be reproduced
• How to do better?
• Understanding the impact of the workload
• Hands-on Session
2
![Page 3: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/3.jpg)
Corroboration • Science moves forward by corroboration
3
![Page 4: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/4.jpg)
Responsibility
http://www.nature.com/news/reproducibility-1.17552
Science advances faster when people waste less time pursuing false leads. […] There is growing alarm about results that cannot be reproduced.
Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers.
Journals, scientists, institutions and funders all have a part in tackling reproducibility.
4
![Page 5: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/5.jpg)
Software and System EvaluationSince
OOPSLA 2013ECOOP 2013POPL 2014PLDI 2014SAS 2014
PPoPP 2015CGO 2015CAV 2015
ADAPT 2015The ‘Since’ column indicates the year the conference is held
5
![Page 6: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/6.jpg)
Evaluated Artifact1. Consistent with the paper: does the artifact substantiate
and help reproduce the claims of the paper
2. Complete. What is the fraction of the results that can be reproduced
3. Well documented. Does the artifact describe and demonstrate how to apply the presented method to a new input?
4. Easy to reuse. How easy is it to reuse the provided artifact?
6
![Page 7: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/7.jpg)
Evaluated Artifact
7
![Page 8: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/8.jpg)
Ok, I want my experiments to be reproducible, but
how can I do?
8
![Page 9: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/9.jpg)
Experimenting
Makefile run.sh
avg.datplot.gp
Computing logs.dat
Aggregating
Visualizing
9
![Page 10: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/10.jpg)
Experimenting cont’d
Makefile
process.sh
benchmark.java run.sh log.dat
stats.datplot.gpgraph.pdf
10
![Page 11: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/11.jpg)
List experimental settings• Compile flags:
• gcc -O3 -std=gnu11
• Environment variables:
• LD_PRELOAD: what memory allocator is called?
• JAVA_HOME: what version of Java?
• Runtime options (JVM HotSpot):
• -server
• -XX:+UseBoundThreads (Bind user threads to Solaris kernel threads)
• -Xoptimize (Use the optimizing JIT compiler)
• Median, average, min, max, standard deviation, 99%-ile
Makefile
run.sh
process.sh
11
![Page 12: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/12.jpg)
Example: Script ‘run.sh’#!/bin/bash DIR=.. OUT=../log BIN=../bin OPTS=-server
# settings THREADS="1 2 4 8 16 32 64" SIZES=“1024 4096 16384 65536” UPDATES=“0 10 20 30 40 50 60 70 80 90 100” LENGTH=“5000” ITERATIONS=5 WARMUP=2 DATE=`date` COMP=`uname -a`
# benchmarks SYNCS="Sequential LockBased LockFree Transactional” ALGOS=“Tree Queue SkipList HashTable Array LinkedList” TYPES=“Map Set Collection” MAINCLASS=contention.benchmark.Test
for sync in ${SYNCS}; do for algo in ${ALGOS}; do for type in ${TYPES}; do for update in ${UPDATES}; do for thread in ${THREADS}; do for size in ${SIZES}; do
r=`echo "2*${i}" | bc` out=${output}/log/${bench}-lockfree-i${i}-u${write}-t${t}.log bench=${sync}${algo}${type} echo “# “ ${TIME} “-“ ${COMP} “-“ ${JAVA_HOME} “-“ ${PATH} > ${out} for (( j=1; j<=${ITERATIONS}; j++ )); do java ${OPTS} -cp ${CP} ${MAINCLASS} -W ${WARMUP} \ -u ${update} -d ${l} -t ${t} -i ${i} \
-r ${r} -b ${bench} 2>&1 >> ${out} done done done done done
12
![Page 13: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/13.jpg)
Backup everything• Output:
• Keep the output logs even though you are interested in the mean/median/min/max
• Aggregation:
• Keep the intermediary data (after aggregation)
• Visualization:
• Keep the visualization scripts (gnuplot)
• Use version management system
• git, svn, blockchain, etc.
13
![Page 14: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/14.jpg)
Take existing benchmarks
1
2
3
14
Write README.md file, INSTALL file, doc, HOWTOs…
Make a screencast available
Package into virtualized environment (virtual box, docker), with OS, libraries, benchmarks…
![Page 15: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/15.jpg)
Experiments of Concurrent Programs cannot be
Reproduced
15
![Page 16: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/16.jpg)
Reproducing Concurrency• Multi-threading is non-deterministic
• Replaying an execution leads to a different interleaving
• The notion of “Thread” is tied to the OS
• Lightweight processes, Java threads, POSIX threads…
• The notion of core/memory is tied to the architecture
• single-/multi-socket, memory controllers…
• Have you tried to reproduce concurrency bugs?
16
![Page 17: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/17.jpg)
Reproducing Concurrency• Multi-threading is non-deterministic
• Replaying an execution leads to a different interleaving
• The notion of “Thread” is tied to the OS
• Lightweight processes, Java threads, POSIX threads…
• The notion of core/memory is tied to the architecture
• single-/multi-socket, memory controllers…
• Have you tried to reproduce concurrency bugs?
17
Not an
excus
e!
Identif
y the
causes
of yo
ur res
ults
![Page 18: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/18.jpg)
The List-Based Set• Sorted linked list with sentinel head “H” and tail “T”
• Implements a integer set (insert/remove/contains)
• A value belongs to the set if it is the key of a node reachable from the head, e.g., {1, 3, 5}
H" 1" 3" 5" T"
18
![Page 19: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/19.jpg)
The List-Based Set (con’t)• Multiple threads execute operations concurrently
• Intuitively, modifications by concurrent threads of non-adjacent nodes should not conflict
19
![Page 20: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/20.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
20
![Page 21: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/21.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
insert/removeinsert/remove/contains ops
21
![Page 22: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/22.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
what if insert(v) fails updating because v ∈ set?
22
![Page 23: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/23.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
same JVM iterations?
23
![Page 24: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/24.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
how much time to warmup the JIT compiler?
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
24
![Page 25: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/25.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
when does hyperthreading kicks in? at 2 or 5?
25
![Page 26: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/26.jpg)
Lack of documentation
List-based set, Java8, 10% updates, avg 10 runs of 10s
Thro
ught
put (
ops
/ ms)
0
25
50
75
100
Number of threads
1 2 3 4 5 6 7 8
ConcurrentSequential
how does it compare to sequential code performance?
26
![Page 27: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/27.jpg)
How can we do better?
27
![Page 28: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/28.jpg)
Take existing benchmarks
Do not reinvent the wheel
List the parameters
Explain the experimental settings
1
2
3
28
![Page 29: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/29.jpg)
Synchrobench [PPoPP’15]
29
![Page 30: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/30.jpg)
Synchrobench (con’t)
Written in Java and C/C++
Run on x86(-64), SPARC, Power8, Tilera…
Compare CAS, TM, Mutex, CopyOnWrite, RCU, Spinlock, no synchronization…
Benchmark w/ trees, skip lists, linked lists, queues, arrays, hash tables…
1
2
3
4
30
![Page 31: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/31.jpg)
Synchrobench (con’t)Main parameters:
• b: benchmark to run
• u: update ratio (%)(#insert+#delete)/#contains
• f: are updates effective?
• i: initial size
• t: #threads
• d: duration (ms) of the benchmark
• n: #iterations in one JVM instance
• W: warmup (seconds)
31
![Page 32: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/32.jpg)
Architecture Impact• AMD Turbo Core
• Intel Turbo Boost
32
![Page 33: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/33.jpg)
Architecture ImpactReport the throughput with 1 thread in different JVM
1. without Turbo Boost
2. withTurbo Boost
Thro
ughp
ut (o
p/s)
6,000,000
7,500,000
9,000,000
10,500,000
12,000,000
Iteration number
1 2 3 4 5
w/o TB w/ TB
2x8-HT-Intel-Xeon—W0-t1-d5000-u40-i0-r50-bVersionedListSet33
![Page 34: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/34.jpg)
Architecture ImpactSimultaneous Multi-threading: allows to have threads sharing pipeline, CPU, and caches and execute multiple instructions per cycle on a processor.
34
![Page 35: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/35.jpg)
Architecture ImpactSimultaneous Multi-threading [CACM’11]
Performance*of*hyperthreading*over*no*hyperthreading*on*Intel*Xeon*CPUs*
35
![Page 36: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/36.jpg)
Operating System ImpactThread can be bound to Solaris lightweight processes (LWP) on the HotSpot JVM. When a LWP for thread is created, the kernel assigns a thread to a locality group (or lgroup)
• In Solaris 10 threads balanced over dies, then over cores, then over pipelines
• In Solaris 11 threads are grouped on a unique process on the same lgroup until the workload exhausts half of the resources of this lgroup.
https://blogs.oracle.com/dave/entry/thread_placement_policies_on_numa36
![Page 37: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/37.jpg)
Operating System Impact
• Thread pinning [PPoPP’15]AMD$Params:$u10.f1.l5000.s0.a0.i[16384..65536].r[32768..132072].W20.$$$COMP:$compact$pinning$policy$(same$NUMA.node$first)$SCAT:$scaOer$pinning$policy$(different$NUMA$node$first)$
4x16-AMD-Opteron—W20-d5000-u40-f137
![Page 38: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/38.jpg)
Prog. Lang. ImpactTo compare-and-set two values atomically:
• sun.misc.Unsafe (Java 8-)
• Java AtomicMarkableReference stores the mark and reference separately, requiring an extra read to return the reference.
• Bitmask in C/C++:
• ref = word & ~(uintptr_t) 1
• mark = word & (uintptr_t) 1
38
![Page 39: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/39.jpg)
Prog. Lang. Impact
Harris linked list (Algo.21) [DISC’01]
• Use Java version using AtomicMarkableReference
• C/C++ version usingbitmask
2x8-HT-Intel-Xeon—d5000-u50-f1-i65536-r132768-bNonBlockingLinkedListSet39
![Page 40: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/40.jpg)
Workload Impact
40
![Page 41: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/41.jpg)
Grab 2 fine-grained locks per update operation [OPODIS’05]
• The lazy linked list grabs no lock during traversal
• But its insertion (resp. deletion) grabs locks before detecting that an element is present (resp. absent)
• This rejects schedules in read-only executions
Lazy list-based set
Valid?'
41
![Page 42: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/42.jpg)
Traverse; lock; validate; try-update; unlock
1. Find the position to insert/delete
2. lock the pred and current node
3. validate: - check neither pred nor curr are deleted- check that pred points to curr
4. try inserting or deleting the node
5. unlock
Lazy list-based set
Valid?'
42
![Page 43: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/43.jpg)
Lazy list-based setinsert(4): traverse
3" 5"
First"value">"4"
43
![Page 44: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/44.jpg)
Lazy list-based setinsert(4): traverse; lock
3" 5"
44
![Page 45: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/45.jpg)
Lazy list-based setinsert(4): traverse; lock; validate
3" 5"
Not"deleted?"
S,ll"linked?"
45
![Page 46: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/46.jpg)
Lazy list-based setinsert(4): traverse; lock; validate; try-update
3" 5"
value"!="4?"
4"
46
![Page 47: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/47.jpg)
Lazy list-based setinsert(4): traverse; lock; validate; try-update; unlock
3" 4" 5"
47
![Page 48: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/48.jpg)
Lazy list-based setWhy is the validation necessary after locking?
48
![Page 49: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/49.jpg)
Lazy list-based setWhy is the validation necessary after locking?
Between the time pred/curr are found and the time they are locked…
…some other threads could:
• delete pred,
• delete curr or
• insert between pred and curr
49
![Page 50: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/50.jpg)
Lazy list-based setWhy is the validation necessary after locking?
3" 5"
First"value">"4"
50
![Page 51: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/51.jpg)
Lazy list-based setWhy is the validation necessary after locking?
3" 5"
4"
51
![Page 52: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/52.jpg)
Lazy list-based setWhy is the validation necessary after locking?
3" 5"
4"
52
![Page 53: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/53.jpg)
Lazy list-based setWhy is the validation necessary after locking?
3" 5"
4"
4"
53
![Page 54: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/54.jpg)
Lazy list-based setWhat happen if an update returns unsuccessfully?
54
![Page 55: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/55.jpg)
Lazy list-based setWhat happen if an update returns unsuccessfully?
4" 5"
First"value">"4"
55
![Page 56: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/56.jpg)
Lazy list-based setWhat happen if an update returns unsuccessfully?
4" 5"
56
![Page 57: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/57.jpg)
Lazy list-based setWhat happen if an update returns unsuccessfully?
delete(5)(is(blocked(
4( 5(
delete(4)(is(blocked(
insert(5)(is(blocked(
insert(4)(is(blocked(
Lost of stalls: reduced concurrency57
![Page 58: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/58.jpg)
Lazy list-based setWhat happen if an update returns unsuccessfully?
4" 5"
No,"value=4?"
Is this necessary to lock without effective update?58
![Page 59: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/59.jpg)
Traverse; lock; validate; try-update; unlock
1. Find the position to insert/delete
2. lock the pred and current node
3. validate: - check neither pred nor curr are deleted- check that pred points to curr
4. try inserting or deleting a node
5. unlock
Lazy list-based set
Valid?'
59
![Page 60: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/60.jpg)
Traverse; lock; validate; try-update; unlock
1. Find the position to insert/delete
3. validate: - check neither pred nor curr are deleted- check that pred points to curr
2. lock the pred and current node
4. try inserting or deleting a node
5. unlock
Lazy list-based set
Valid!'
60
![Page 61: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/61.jpg)
Traverse; lock; validate; try-update; unlock
1. Find the position to insert/delete
3. validate: - check neither pred nor curr are deleted- check that pred points to curr
2. lock the pred and current node
4. try inserting or deleting a node
5. unlock
Lazy list-based set
Valid!'
Cannot
work with
traditi
onal lo
cks
61
![Page 62: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/62.jpg)
• The versioned list uses a pre-locking validation [DISC’15]
• It uses a versioned try-lock <ver, lock> per node w/ ver = #acquirements
• Traversals do not acquire versioned try-locks
• Once the position is found:
• The update validates by reading the version
• An insert(v) (resp. delete(v)) grabs the lock only if v is absent (resp. present) and version is unchanged
Versioned list-based set
6
Valid!(
62
![Page 63: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/63.jpg)
Versioned list-based setinsert(4)
30# 50#
First#value#>#4#
63
![Page 64: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/64.jpg)
Versioned list-based setinsert(4)
30# 50#
Record#version:#0#
64
![Page 65: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/65.jpg)
Versioned list-based setinsert(4)
30# 50#
value#!=#4?#
version(of(pred:(0(
65
![Page 66: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/66.jpg)
Versioned list-based setinsert(4)
30# 50#
Not#deleted?#
S-ll#linked?#
version(of(prev:(0(
version(of(pred:(0(
66
![Page 67: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/67.jpg)
Versioned list-based setinsert(4)
3" 5"0 0
version(of(pred:(0(
67
![Page 68: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/68.jpg)
version(of(pred:(0(
Versioned list-based setinsert(4)
3" 5"0 0
compare…
68
![Page 69: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/69.jpg)
Versioned list-based setinsert(4)
3" 5"
40"
1 0…and s
wap
69
![Page 70: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/70.jpg)
Versioned list-based setinsert(4)
31# 50#
40#
70
![Page 71: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/71.jpg)
Traverse; validate; try-lock; update; unlock
1. Find the position to insert/delete
2. record version of pred
3. validate: - check neither pred nor curr are deleted- check that pred points to curr
4. lock the pred and current node
5. try inserting or deleting a node
6. unlock
Versioned list-based set
6
Valid!(
71
![Page 72: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/72.jpg)
Versioned list-based setWhat happen if an update returns unsuccessfully?
40# 50#
First#value#>#4#
72
![Page 73: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/73.jpg)
Versioned list-based setWhat happen if an update returns unsuccessfully?
40# 50#
Record#version:#0#
73
![Page 74: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/74.jpg)
Versioned list-based setWhat happen if an update returns unsuccessfully?
40# 50#
value#!=#4?#
74
![Page 75: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/75.jpg)
Versioned list-based setWhat happen if an update returns unsuccessfully?
40# 50#
No,#value=4#
75
![Page 76: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/76.jpg)
In the lazy list:
All attempted updates grab locks
This is necessary to guarantee correctness
In the versioned list:
Only effective updates grab locks
The versioned list is concurrency-optimal [DISC’15]: There is no list that accepts more correct schedules of memory accesses
Effective Updates
6
Valid!(
Valid?'
76
![Page 77: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/77.jpg)
Hands-on Session
77
![Page 78: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/78.jpg)
Install Synchrobenchwget https://github.com/gramoli/synchrobench/archive/master.zipunzip master.zipcd synchrobench-‐master/javaant jarbench
1
2
3
4
78
![Page 79: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/79.jpg)
Compute
java -server -cp bin \ contention.benchmark.Test -W 0 -t 2 -d 5000 -u 40 -i 0 -r 50 -b \ linkedlists.lockbased.VersionedListSet -n 0
• Run the following code 5 times
java -server -cp bin \ contention.benchmark.Test -W 0 -t 2 -d 5000 -u 40 -i 0 -r 50 -b \ linkedlists.lockbased.VersionedListSet -n 5
• Run the following code once with -n 5
• What do you observe, regarding the ‘Throughput’
79
![Page 80: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/80.jpg)
JIT optimizations Report the throughput with 2 threads & 5 iterations:
1. as part of the same JVM instance (-n 5)
2. as part of separate instances
Thro
ughp
ut (o
p/s)
5,100,000
5,175,000
5,250,000
5,325,000
5,400,000
Iteration number
1 2 3 4 5
5 x java-n01 x java-n5
2x8-HT-Intel-Xeon—W0-t2-d5000-u40-i0-r50-bVersionedListSet80
![Page 81: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/81.jpg)
Compute
java -server -cp lib/compositional-deucestm-0.1.jar \ contention.benchmark.Test -W 0 -t 1 -d 5000 -u 40 -i 0 -r 50 -b \ linkedlists.lockbased.LazyLinkedListSortedSet
• Run the following code with different thread counts 1, 2, 4, 8.
java -server -cp lib/compositional-deucestm-0.1.jar \ contention.benchmark.Test -W 0 -t 1 -d 5000 -u 40 -i 0 -r 50 -b \ linkedlists.lockbased.VersionedListSet
81
![Page 82: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/82.jpg)
Aggregate • Report the “Throughput (ops/s): 1731921.1783” in
a table where columns are <VersionedList, LazyList> and rows are <1,2,4,8> threads
#threads Versioned List Lazy List1 4523706.846 2940885.842
2 4895072.451 2415960.232
4 5661667.679 2266188.053
8 6837793.548 2026012.515
82
![Page 83: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/83.jpg)
Visualize
• Using gnuplot, excel, open office or google sheets, draw a graph to compare the performance
• What do you observe? Why?
83
![Page 84: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/84.jpg)
Results• The lazy list is overly conservative while the
versioned list is concurrency-optimal [DISC’15]
• Difference in performance is exacerbated with the contention (4x16-core 2.4GHz AMD Proc. 6378, Ubuntu 14.04.3 LTS, Java 1.7.0_95 OpenJDK 64-Bit Server) => documenting workloads is crucial84
![Page 85: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/85.jpg)
The Slide to Remember
Use well-documented benchmarking tools
Gather your sources (code, scripts, logs and aggregated dataset), list the settings (OS, architecture, environment, runtime observations, workloads)
Package everything, share your code and submit your artifact
1
2
3
85
![Page 86: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/86.jpg)
See you in Sydney
86
![Page 87: Reproducibility of Experiments in Concurrent Programming · PPoPP 2015 CGO 2015 CAV 2015 ADAPT 2015 The ‘Since’ column indicates the year the conference is held 5. Evaluated Artifact](https://reader034.fdocuments.in/reader034/viewer/2022050419/5f8eb08ac5687310670fb507/html5/thumbnails/87.jpg)
References• [TR’12] Can you trust your experimental results? Blackburn, Diwan, Hauswirth, Sweeney, Amaral, Babka, Binder,
Brecht, Bulej, Eeckhout, Fischmeister, Frampton, Garner, Georges, Hendren, Hind, Hosking, Jones, Kalibera, Moret, Nystrom, Pankratius, Tuma. Evaluate TR 2012.
• [T. Harris’16] Do not believe everything you read in the papers. Tim Harris. Personal communication, NICTA SSRG 4th Summer School, Feb. 2016
• [OPODIS’05] A lazy concurrent list-based set algorithm. Heller, Herlihy, Luchangco, Moir, Sherer, Shavit. OPODIS 2005.
• [DISC’15] A Concurrency-Optimal List-Based Set. Gramoli, Kuznetsov, Ravi, Shang. arXiv:1502.01633v1, Feb. 2015 and DISC 2015.
• [IHG12] The case for open computer programs. Darrel C. Ince, Leslie Hatton & John Graham-Cumming. Nature 482(10836):485–488, Feb. 2012.
• [DISC’01] A Pragmatic Implementation of Non-blocking Linked-Lists. Tim Harris, DISC 2001.
• [ECOOP’14] Reusable Data Types. Gramoli and Guerraoui, ECOOP 2014.
• [CACM’11] Why STM can be more than a research toy. Dragojevic, Felber, Gramoli, Guerraoui. Commun. ACM 2011.
• [PPoPP’15] More Than You Ever Wanted to Know about Synchronization: Synchrobench, measuring the Impact of the Synchronization on Concurrent Algorithms. Vincent Gramoli. PPoPP 2015.
87