Abstraction Without Guilt Steve Blackburn Australian National University.
-
Upload
logan-melton -
Category
Documents
-
view
215 -
download
0
Transcript of Abstraction Without Guilt Steve Blackburn Australian National University.
![Page 1: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/1.jpg)
AbstractionWithout
Guilt
Steve BlackburnAustralian National University
![Page 2: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/2.jpg)
Astraction Without Guilt 2
Abstraction
Steve Blackburn
![Page 3: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/3.jpg)
Astraction Without Guilt 3
September 1999Tibet
Steve Blackburn
A Month LaterAmherst, MA
A Few Days LaterAmherst, MA
Meanwhile…Denver, CO
A Few Days LaterAmherst, MA
![Page 4: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/4.jpg)
Astraction Without Guilt 4
Part I
MMTk(Flexibility & performance: oil & water?)
Steve Blackburn
![Page 5: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/5.jpg)
Astraction Without Guilt 5Steve Blackburn
![Page 6: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/6.jpg)
Astraction Without Guilt 6Steve Blackburn
![Page 7: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/7.jpg)
Astraction Without Guilt 7
The Researcher’s Dream
• Rapid realization of ideas– Express ideas as abstractly as possible– Reuse thoroughly tested building blocks
• Apples to apples comparisons– Compare ideas in common framework
• Credible artifacts– Performance comparable to production
Steve Blackburn
![Page 8: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/8.jpg)
Astraction Without Guilt 8
State of The Art 1999
Stefanović et al• Detailed study of
mechanisms• Detailed cost model• Done in simulation
✘ Credible artifact✔ Apples to apples✔ Rapid realization
Attanasio et al• Detailed perf. study• Studied in real JVM• Credible performance• Monolithic collectors
✘ Rapid realization✘ Apples to apples✔ Credible artifact
Steve Blackburn
![Page 9: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/9.jpg)
Astraction Without Guilt 9
(Fast, correct)
GC is Hard
Steve Blackburn
1665 gc_checkWriteBuffers();1666 1667 // all GC threads return, having completed Major collection1668 return;1669 } // collect
1035 static void1036 collect () {1037 int i,temp,bytes;1038 boolean selectedGCThread = false; // indicates 1 thread to generate output
1121 // set Done flag to allow other GC threads to begin processing1122 initGCDone = true;1123 1124 } // END SINGLE GC THREAD SECTION - GC INITIALIZATION1125 1126 else {1127 // Each GC thread must wait here until initialization is complete
Watson 2.0 GenCopy
![Page 10: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/10.jpg)
Astraction Without Guilt 10
Our Approach
• Premature optimization is evil– Very hard to intuit performance
• Have faith!– Vivek’s compiler team is brilliant
• Identify and optimize hot paths–Most code is cold, non-critical
• Be fearless!– Just assume that things will work OK :-)
Steve Blackburn
![Page 11: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/11.jpg)
Astraction Without Guilt 11
Three months later…
Steve Blackburn
![Page 12: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/12.jpg)
Astraction Without Guilt 12
Code Complexity
methods NCSS NCSS/m ΣCCN
Watson (Jikes RVM 2.2 circa 2003)
GenCopy 42 659 15.7 220
GenMS 40 531 13.3 171
MMTk (Jikes RVM 2.2 circa 2003)
GenCopy 19 117 6.2 49
GenMS 18 104 5.8 158
Generational
36 279 7.4 102
Steve Blackburn
![Page 13: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/13.jpg)
Astraction Without Guilt 13
Performance
Steve Blackburn
Allocation Rate(MB/sec)
Tracing Rate(MB/sec)
Watson SemiSpace 690 58
MMTk SemiSpace 610 55
Watson MarkSweep 600 93
MMTk MarkSweep 575 69
![Page 14: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/14.jpg)
Astraction Without Guilt 14
Performance
Steve Blackburn
No Inlining(MB/sec)
Inlining(MB/sec)
MMTk SemiSpace 396 610
MMTk MarkSweep 315 575
C malloc() 478 n/a
C calloc() 338 n/a
![Page 15: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/15.jpg)
Astraction Without Guilt 15
Since Then…
• Heavily used• Basis for many new GC algorithms• Basis for comparative studies• Constantly evolving• Constantly pushing limits• Now 12 or so supported GCs• High performance
Steve Blackburn
![Page 16: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/16.jpg)
Astraction Without Guilt 16Steve Blackburn
The Lessons…
(Building good infrastructure is hard.)
Flexibility & performance ≠ Oil & water.
Appropriate abstraction is a game changer.
![Page 17: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/17.jpg)
Astraction Without Guilt 17
Part II
vmmagic(Having your cake and eating it.)
Steve Blackburn
![Page 18: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/18.jpg)
Astraction Without Guilt 18
Why Java?(Why not C?)
• Type safety• Memory safety• Encapsulation, composition, reuse• Concurrency (language supported)• etc…
Steve Blackburn
![Page 19: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/19.jpg)
Astraction Without Guilt 19Steve Blackburn
![Page 20: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/20.jpg)
Astraction Without Guilt 20
(Fast, correct)
GC is Hard
Steve Blackburn
1665 gc_checkWriteBuffers();1666 1667 // all GC threads return, having completed Major collection1668 return;1669 } // collect
1035 static void1036 collect () {1037 int i,temp,bytes;1038 boolean selectedGCThread = false; // indicates 1 thread to generate output
1121 // set Done flag to allow other GC threads to begin processing1122 initGCDone = true;1123 1124 } // END SINGLE GC THREAD SECTION - GC INITIALIZATION1125 1126 else {1127 // Each GC thread must wait here until initialization is complete
Watson 2.0 GenCopy
![Page 21: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/21.jpg)
Astraction Without Guilt 21
A Dirty Little Secret
Steve Blackburn
+kADDRESS=intand you can’t use new()avoid method callsavoid inheritance
![Page 22: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/22.jpg)
Astraction Without Guilt 22
Why Java?(Why not C?)
• Type safety• Memory safety• Encapsulation, composition, reuse• Concurrency (language supported)• etc…
Steve Blackburn
![Page 23: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/23.jpg)
Astraction Without Guilt 23
The Problem
• Systems programmers:– Strive for reliability, security,
maintainability– Depend on performance and transparent
access to low-level primitives
• Abstraction:– Enables the former– Typically obstructs the latter
Steve Blackburn
![Page 24: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/24.jpg)
Astraction Without Guilt 24
Why (1975)
Steve Blackburn
James J. Horning, Yes! High level languages should be used to write systems software. ACM Annual Conference/Annual MeetingProceedings of the 1975 Annual Conference.
![Page 25: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/25.jpg)
Astraction Without Guilt 25
Why (1975)
Steve Blackburn
James J. Horning, Yes! High level languages should be used to write systems software. ACM Annual Conference/Annual MeetingProceedings of the 1975 Annual Conference.
![Page 26: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/26.jpg)
Astraction Without Guilt 26
Why Not (1972)
Steve Blackburn
J.G. Fletcher et al, On the appropriate language for system programming. ACM SIGPLAN Notices, Volume 7, Number 7, 1972
![Page 27: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/27.jpg)
Astraction Without Guilt 27
Why The 70s Shift to C?
• Better language design• Better language implementation• More complex software• More complex hardware• More hardware (portability)
Steve Blackburn
Charles Landau, On high-level languages for system programmingACM SIGPLAN Notices, Volume 11, Issue 1,1976
![Page 28: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/28.jpg)
Astraction Without Guilt 28
Time for a Second Take?
• Change continues– heterogeneous multi-core?–more complex software
• Higher-level languages– type safe–memory safe– strong abstractions over hardware
Steve Blackburn
![Page 29: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/29.jpg)
Astraction Without Guilt 29
Solutions?
1. Fortify low-level languages (C/C++)– Memory safety (e.g., cons. GC, talloc)– Idioms (e.g., restrictive use of types)– Tools (e.g., Valgrind’s memcheck)
2. Use a Systems PL– BLISS, Modula-3, Cyclone
3. Use two languages– FFI’s such as JNI & PInvoke
4. Extend a high-level language– Jikes RVM extends Java
Steve Blackburn
![Page 30: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/30.jpg)
Astraction Without Guilt 30
Our Approach
• Key Principle: ContainmentMinimize exposure to low-level coding
– Extensibility• Requirements change quickly• Languages change slowly
– Encapsulation• Contained low-level semantics
– Fine-grained lowering of semantics• Minimize impedance• Separation of concerns
Steve Blackburn
![Page 31: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/31.jpg)
Astraction Without Guilt 31
Our Framework
• Extend semantics– Intrinsic methods
• Control semantics– Scoped semantic changes
• Extend types– box/unbox, ref/value, arch. sizes, etc
• Prefer to retain syntax (pragmatism)– Existing front-end tools useable
Steve Blackburn
![Page 32: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/32.jpg)
Astraction Without Guilt 32
A Concrete Example
void prefetchObjects(OOP *buffer, int size) {
for(int i=0; i < size; i++) { OOP current = buffer[i];
asm volatile("prefetchnta (%0)" :: "r" (current)); }}
Steve Blackburn
![Page 33: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/33.jpg)
Astraction Without Guilt 33
A Concrete Example
void prefetchObjects(OOP *buffer, int size) {
for(int i=0; i < size; i++) { OOP current = buffer[i];
asm volatile("prefetchnta (%0)" :: "r" (current)); }}
Steve Blackburn
![Page 34: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/34.jpg)
Astraction Without Guilt 34
A Concrete Example
void prefetchObjects(OOP *buffer, int size) {
for(int i=0; i < size; i++) { OOP current = buffer[i];
asm volatile("prefetchnta (%0)" :: "r" (current)); }}
Steve Blackburn
![Page 35: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/35.jpg)
Astraction Without Guilt 35
Java Version
void prefetchObjects( ?!? buffer) { for(int i=0;i<buffer.length;i++)
{ ?!? current = buffer[i];
?!?
}}Steve Blackburn
void prefetchObjects( OOP *buffer, int size) {
for(int i=0;i < size;i++){ OOP o = buffer[i];
asm volatile( "prefetchnta (%0)" :: "r" (o)); }}
![Page 36: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/36.jpg)
Astraction Without Guilt 36
“Magic” in JikesRVM
• Raw access to memory?• Use int and “magic” (peek & poke)
int ref;int value = VM_Magic.loadIntAtOffset(ref, offset);
Steve Blackburn
![Page 37: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/37.jpg)
Astraction Without Guilt 37
“Magic” in JikesRVM
• Raw access to memory?• Use int and “magic” (peek & poke)• Use ADDRESS macro (as int or long)
ADDRESS ref;int value = VM_Magic.loadIntAtOffset(ref, offset);
Steve Blackburn
![Page 38: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/38.jpg)
Astraction Without Guilt 38
“Magic” in JikesRVM
• Raw access to memory?• Use int and “magic” (peek & poke)• Use ADDRESS macro (as int or long)• Use magical Address type (~= void*)–Typed; magic on “instance”
Address ref;int value = ref.loadInt(offset);
Steve Blackburn
![Page 39: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/39.jpg)
Astraction Without Guilt 39
“Magic” in JikesRVM
• Raw access to memory?• Use int and “magic” (peek & poke)• Use ADDRESS macro (as int or long)• Use magical Address type (~= void*)• Use ObjectReference magic type–More strongly typed– Abstracts over mechanism (handle/pointer)
Steve Blackburn
![Page 40: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/40.jpg)
Astraction Without Guilt 40
Java Version
void prefetchObjects( ObjectReference[] buffer) { for(int i=0;i<buffer.length;i++) { ObjectReference current = buffer[i];
?!?
}}Steve Blackburn
void prefetchObjects( OOP *buffer, int size) {
for(int i=0;i < size;i++){ OOP o = buffer[i];
asm volatile( "prefetchnta (%0)" :: "r" (o)); }}
![Page 41: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/41.jpg)
Astraction Without Guilt 41
Intrinsics
• Contract between user and compiler– Implement semantics beyond language– Requires co-operation of compiler writer
• Canonical intrinsicsclass ObjectReference { ... @Intrinsic{Prefetch} void prefetch() {} // empty ...}
• Express intrinsic abstractly– e.g., Intermediate language
Steve Blackburn
Intrinsics
Prefetch
LoadInt...
![Page 42: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/42.jpg)
Astraction Without Guilt 42
Java Version
void prefetchObjects( ObjectReference[] buffer) { for(int i=0;i<buffer.length;i++) { ObjectReference current = buffer[i];
current.prefetch();
}}Steve Blackburn
void prefetchObjects( OOP *buffer, int size) {
for(int i=0;i < size;i++){ OOP o = buffer[i];
asm volatile( "prefetchnta (%0)" :: "r" (o)); }}
Other performance overheads?
![Page 43: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/43.jpg)
Astraction Without Guilt 43
Semantic Regimes
• Scoped semantic change– Additive• UncheckedCast (allow certain intrinsics)
– Subtractive• NoNew (allocation via new() not permitted)
– Other• SpillAllRegisters• NoGCYield• NoBoundsChecks
Steve Blackburn
![Page 44: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/44.jpg)
Astraction Without Guilt 44
Both Versions@NoBoundsCheckvoid prefetchObjects( ObjectReference[] buffer) {
for(int i=0;i<buffer.length;i++) { ObjectReference current = buffer[i];
current.prefetch();
}}Steve Blackburn
void prefetchObjects( OOP *buffer, int size) {
for(int i=0;i < size;i++){ OOP o = buffer[i];
asm volatile( "prefetchnta (%0)" :: "r" (o)); }}
![Page 45: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/45.jpg)
Astraction Without Guilt 45
ObjectReference Abstraction
• Insulates from implementation• Handles?class ObjectReference { int handle; void prefetch() { getPayload().prefetch(); }
Address getPayload() { ... }}
Steve Blackburn
![Page 46: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/46.jpg)
Astraction Without Guilt 46
ObjectReference Abstraction
• Insulates from implementation• Handles?class ObjectReference { int handle; void prefetch() { getPayload().prefetch(); }
Address getPayload() { ... }}
Steve Blackburn
![Page 47: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/47.jpg)
Astraction Without Guilt 47
Summary
• Higher-languages for low-level coding• Our framework– Extensible semantics (don’t bake in)– Containment of semantic change– Extend types
• Much implemented• Interested in feedback, thoughts
“You mean: use sudo instead of root?”
Steve Blackburn
![Page 48: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/48.jpg)
Astraction Without Guilt 48
Part III
DaCapo Benchmarks(Just how guilty should I feel?)
Steve Blackburn
![Page 49: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/49.jpg)
Astraction Without Guilt 49Steve Blackburn
![Page 50: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/50.jpg)
Astraction Without Guilt 50
The success of most systems innovationhinges on benchmark performance.
Predicate 2. Methodology is appropriate.
Predicate 1. Benchmarks reflect current(and ideally, future) reality.
Steve Blackburn
![Page 51: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/51.jpg)
Astraction Without Guilt 51
Benchmarks & Reality
1. JVM design & implementation– SPECjvm98 is small and jbb is relatively simple
• Q: What has this done to GC research?• Q: What has this done to compiler research?
2. Computer architecture– ISCA & Micro rely on SPEC CPU
• Q: What does this mean for Java and C# performance on modern architectures?
3. C#– Public benchmarks are almost non-existent
• Q: How has this impacted research?
Steve Blackburn
![Page 52: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/52.jpg)
Astraction Without Guilt 52
Benchmarks & Methodology
• We’re not in Kansas anymore!– JIT compilation, GC, dynamic checks, etc
• Methodology has not adapted– Needs to be codified and mandated
“…this sophistication provides a significant challenge to understanding complete system performance, not found in traditional languages such as C or C++”
[Hauswirth et al OOPSLA ’04]
Steve Blackburn
![Page 53: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/53.jpg)
Astraction Without Guilt 53
• Comprehensive comparison– 3 state-of-the-art JVMs– Best of 5 executions– 19 benchmarks– 1 platform
Benchmarks & Methodology
compress jess
raytrace db
javac
mpegaudiomtrt jack
antlrbloat
chart
eclipse fop
hsqldb
luindex
lusearch
jython
pmdxa
lan
geomean
00.20.40.60.8
11.21.41.61.8
System ASystem BSystem C
No
rma
lize
d T
ime
compress jess
raytrace db
javac
mpegaudiomtrt jack
antlrbloat
chart
eclipse fop
hsqldb
luindex
lusearch
jython
pmdxa
lan
geomean
0
0.4
0.8
1.2
1.6
2System ASystem BSystem C
No
rma
lize
d T
ime
compress jess
raytrace db
javac
mpegaudiomtrt jack
antlrbloat
chart
eclipse fop
hsqldb
luindex
lusearch
jython
pmdxa
lan
geomean
0
0.4
0.8
1.2
1.6
2System ASystem BSystem C
No
rma
lize
d T
ime
1st iteration
2nd iteration
3rd iteration
Steve Blackburn
![Page 54: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/54.jpg)
Astraction Without Guilt 54
System A System B1
1.05
1.1
1.15
1.2
1.25
1.3
1.35SPEC _209_db
Norm
alize
d T
ime
System A System B0.95
1
1.05
1.1
1.15
1.2SPEC _209_db
Norm
alize
d T
ime
Benchmarks & Methodology
20 40 60 80 100 1201
1.05
1.1
1.15
1.2
1.25
1.3SPEC _209_db
System ASystem B
Heap Size (MB)
Norm
alize
d T
ime
System A System B
1
1.05
1.1
1.15
1.2
1.25
1.3
1.35SPEC _209_db
No
rmali
zed
Tim
e
System A System B
0.95
1
1.05
1.1
1.15
1.2SPEC _209_db
No
rmali
zed
Tim
e
Steve Blackburn
![Page 55: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/55.jpg)
Astraction Without Guilt 55
antlr bloat chart eclipse fop hsqldb jython lusearch luindex pmd xalan min max ge-omean
0.50
1.00
1.50
2.00
2.50
3.00
3.501st JVM A 2nd JVM A 3rd JVM A1st JVM B 2nd JVM B 3rd JVM B
Tim
e (
No
rmali
zed
)
antlr bloat chart eclipse fop hsqldb jython lusearch luindex pmd xalan min max ge-omean
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.501st JVM A 2nd JVM A 3rd JVM A1st JVM B 2nd JVM B 3rd JVM B
Tim
e (
No
rmali
zed
)
antlr bloat chart eclipse fop hsqldb jython lusearch luindex pmd xalan min max ge-omean
0.501.001.50
2.002.50
3.003.504.00
4.505.005.50
1st JVM A 2nd JVM A 3rd JVM A1st JVM B 2nd JVM B 3rd JVM B
Tim
e (
No
rmali
zed
)
Benchmarks & Methodology
Pentium M
AMD Athlon
SPARC
Steve Blackburn
![Page 56: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/56.jpg)
Astraction Without Guilt 56
The success of most systems innovationhinges on benchmark performance.
Predicate 2. Methodology is appropriate.
Predicate 1. Benchmarks reflect current(and ideally, future) reality.
?
Steve Blackburn
![Page 57: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/57.jpg)
Astraction Without Guilt 57
Innovation Trap
• Innovation is gated by benchmarks• Poor benchmarking retards innovation
& misdirects energy– Reality: inappropriate, unrealistic benchmarks– Reality: poor methodology
• Examples– GC avoided in SPEC performance runs– Managed languages largely ignored by
architects– C# is largely ignored by all researchers
Steve Blackburn
![Page 58: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/58.jpg)
Astraction Without Guilt 58
How Did This Happen?
• Researchers depend on SPEC– Primary purveyor & de facto guardian– Industry body concerned with product
comparison• Little involvement from researchers• Not concerned with research analysis/methodology
– Historically C & Fortran benchmarks • Did not update/adapt methodology for Java
– But the research community depends on them!• Researchers tend not to create their own
suites– Enormously expensive exercise
Steve Blackburn
![Page 59: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/59.jpg)
Astraction Without Guilt 59
The DaCapo Suite: Goals
• Open source– Encourage (& leverage) community feedback– Enable analysis of benchmark sources– Freely available, avoid intellectual property
restrictions
• Real, non-trivial applications– Popular, non-contrived, active applications– Use analysis to ensure non-trivial, good coverage
• Responsive, not static– Adapt the suite as circumstances change
• Easy to use
Steve Blackburn
![Page 60: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/60.jpg)
Astraction Without Guilt 60
The DaCapo Suite: Today
• Open source (www.dacapobench.org)
• Second major release imminent(beta2 out now!)
• Real, non-trivial applications– Uncovered bugs in product JVMs
• Responsive, not static– Have adapted the suite enormously
• Easy to use– Single jar file, OS-independent, validation
Steve Blackburn
![Page 61: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/61.jpg)
Astraction Without Guilt 61
Broader Impact
• Just the tip of the iceberg?– Q: How many good ideas did not see light of day because of
jvm98?
• A problem unique to Java?– Q: How has the lack of C# benchmarks impacted research?
• What’s next?– Multicore architectures, transactional memory,
Fortress, dynamic languages, …– Q: Can we evaluate TM versus locking?– Q: Can we evaluate TM implementations? (SPLASH & JBB???)
• Are we prepared to let major directions in our field unfold at the whim of inadequate methodology?
Steve Blackburn
![Page 62: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/62.jpg)
Astraction Without Guilt 62
Lessons
• Systems innovation is gated by benchmarks– Benchmarks & methodology can retard or accelerate
innovation, focus or misdirect energy.• As a community, we have failed
– We have unrealistic benchmarks and poor methodology• We have a unique opportunity
– Transactional memory, multicore performance, dynamic languages, etc…
• We need to take responsibility for benchmarks & methodology– Formally (eg SIGPLAN) or via ad hoc consortia (eg
DaCapo)
Steve Blackburn
![Page 63: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/63.jpg)
Astraction Without Guilt 63
Conclusions(So what did I learn in those 10 years??)
Steve Blackburn
![Page 64: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/64.jpg)
Astraction Without Guilt 64
Conclusions
• HL-LL programming is productive• Building HL-LL frameworks is hard• Evaluating modern languages is hard• It’s time for a cultural shift– In how we write systems code– In how we evaluate systems
• Exciting times for language implementers
Abstraction need not be a guilty pleasure(…and take time out to climb a mountain!)
Steve Blackburn
![Page 65: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/65.jpg)
Astraction Without Guilt 65Steve Blackburn
![Page 66: Abstraction Without Guilt Steve Blackburn Australian National University.](https://reader036.fdocuments.in/reader036/viewer/2022062719/56649eef5503460f94bfebde/html5/thumbnails/66.jpg)
Astraction Without Guilt 66
Acknowledgements
• Jikes RVM and Jalapeno teams• DaCapo research group• Moxie team• My co-authors, particularly:– Kathryn McKinley, Perry Cheng, Daniel
Frampton, Robin Garner, David Grove, Eliot Moss, Sergey Salishev,
Steve Blackburn