Post on 20-Dec-2015
Reducing Garbage Collector Cache Misses
Shachar Rubinstein
Garbage Collection Seminar
The End!
The general problem
CPU’s are getting fast faster and faster Main memory speed lags behind Result: The cost to access main memory is
increasing
Solutions
Hardware and software techniques:– Memory hierarchy– Prefetcing– Multithreading– Non-blocking caches– Dynamic instruction scheduling– Speculative execution
Great Solutions?
Complex hardware and compilers Ineffective for many programs Attack the manifestation (= memory latency)
and not the source (=poor reference locality)
Not exactly…
Previous work
Improving cache locality in dense matrices using loop transformation
Other profile-driven, compiler directed approach
The GC problem
Little temporal locality. Each live object is usually read only once
during mark phase. Most reads are likely to miss. The new contents are unlikely to be used
more than once.
The GC problem – cont.
The sweep phase, like the mark phase, also touches each object once
That’s since the free list pointers are maintained in the objects themselves
Unlike the mark phase, the sweep phase is more sequential
The GC problem – cont.
The sweep is less likely to use cache contents left by the marker
The allocator is likely to miss again, when the object is allocated
The GC problem - previous work
Older work concentrated on paging performance.
Memory size increase lead to abandoning this goal.
But memory size also lead to huge cache miss penalties.
The largest cache size < heap size This problem is unavoidable.
Previous work
Reducing sweep time for a nearly empty heap
Compiler-based prefetching for recursive data structures
How am I going to improve the situation?
Do some magic! Well no… Use real-time information to improve program
cache locality. The mark and sweep phases offers
invaluable opportunities for improvements– Bring objects earlier to the cache– Reuse freed objects for reallocation
Some numbers
Relative to copy GC– Cache miss rates reduced by 21-42%– Program performance improved by 14-37%
Relative to a page level GC– Cache miss rates reduced by 20-41%– Program performance improved by 18-31%
Road map
Cache conscious data placement using generational GC
– Overview– Short generational GC reminder– Real-time data profiling– Object affinity graph– Combining the affinity graph with GC– Experimental evaluation
Other methods and their experimental results
Overview
A program is instrumented to profile its access patterns
The data is used in the same execution and not the next one.
The data -> affinity graph A new copy algorithm uses the graph to
layout the data while copying.
Generational GC – A reminder
The heap is divided to generations GC activity concentrates on young objects,
which typically die faster. Objects that survive one or more scavenges
are moved to the next generation
Implementation notes
The authors used the UM GC toolkit The toolkit has several steps per generation The authors used a single step for each
generation for simplicity. Each step consists of fixed size blocks The blocks are not necessarily contiguous in
memory
Implementation notes - steps
Implementation notes - steps
The steps are used to encode the objects’ age
An object which survives a scavenge is moved to the next step
Implementation notes – moving between generations
The scavenger collects a generation g and all its younger generations
It starts from objects that are:– In g– Reachable from the roots.
Moving an object is copying it into a TO space.
The FROM space can be reused
Copying algorithm – a reminder
Cheney’s algorithm TO and FROM spaces are
switched Starts from the root set Objects are traversed
breadth-first using a queue Objects are copied to TO
space Terminates when the
queue is empty
Copying algorithm – the queue trick
The algorithm
Did you get it?
Real time data profiling
Earlier program run profile is not good enough
Real time data eliminates:– Profile execution run– Finding inputs
But the overhead must be low!
Great!
Profiling data access patterns
Trace every load and store to heap
Huge overhead (factor of 10!)
Reducing overhead
Use object oriented programs properties
1. Most objects are small, often less than 32 bytes
– No need to distinguish between fields, since cache blocks are bigger
Reducing overhead – cont.
2. Most object accesses are not lightweight– Profiling instrumentation will not incur large
overhead
Don’t believe? Stay awake
Collecting profiling data
“Load”s of base object addresses Uses a modified compiler The compiler retains object type information
for selective loads
Code instrumentation
Collecting profiling data - cont
The base object address is entered to an object access buffer
Implementation note
Uses a page trap for buffer overflow A trap causes a graph to be built Recommended buffer size: 15000 (60KB)
Affinity?
Main Entry: af·fin·i·ty Pronunciation: &-'fi-n&-tEFunction: nounInflected Form(s): plural -tiesEtymology: Middle English affinite, from Middle French or Latin; Middle French afinité, from Latin affinitas, from affinis bordering on, related by marriage, from ad- + finis end, borderDate: 14th century1 : relationship by marriage2 a : sympathy marked by community of interest : KINSHIP b (1) : an attraction to or liking for something <people with an affinity to darkness -- Mark Twain> <pork and fennel have a natural affinity for each other -- Abby Mandel> (2) : an attractive force between substances or particles that causes them to enter into and remain in chemical combination c : a person especially of the opposite sex having a particular attraction for one3 a : likeness based on relationship or causal connection <found an affinity between the teller of a tale and the craftsman -- Mary McCarthy> <this investigation, with affinities to a case history, a psychoanalysis, a detective story -- Oliver Sacks> b : a relation between biological groups involving resemblance in structural plan and indicating a common origin
The object affinity graph
The object affinity graph
Nodes – objects Edges – Temporal affinity between objects An undirected graph
Building the graph
Inserting an object to the queue
Incrementing edges’ weight
All clear?
Demonstration
A
B
A
C
D
D
A
Locality queueObject access buffer Graph
Queue tail
Queue tail
Demonstration
A
B
A
C
D
D
A
Locality queueObject access buffer Graph
A
Queue tail
Queue tail
Demonstration
A
B
A
C
D
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
1
Queue tail
Demonstration
A
B
C
D
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
12
Queue tail
Demonstration
A
B
C
D
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C1
1
Queue tail
Demonstration
A
C
D
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C1
1D
11
Queue tail
Demonstration
A
C
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C1
1D
11
Queue tail
Demonstration
A
C
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C1
1D
22
Queue tail
Demonstration
C
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C1
1D
22
Queue tail
Demonstration
C
D
A
Locality queueObject access buffer Graph
Queue tail
A
B
2
C2
1D
23
Queue tail
Implementation notes
A separate affinity graph is built for each generation, except the first.
It uses the fact that the object generation is encoded in its address.
This method prevents placing objects from different generations in the same cache block. (Explanations later on)
Implementation notes – queue size
The locality queue size is important Too small -> Miss temporal relationships Too big -> huge graph, long processing time Recommended: 3.
Implementation notes
Re-create or update the graph? Depends on the application
– Access phases should re-create– Uniform behavior should update
In this article – re-create before each scavange
Stop!
Our goal: Produce a cache conscious data layout, so that objects are likely to reside in the same cache block
In English: place objects with high temporal affinity next to each other.
The method: Use the profiling information we’ve collected in the copying process.
GC + Real-time profiling
Use the object affinity graph in the Copying algorithm.
Example – object affinity graph
Example – before step 1
Step 1 – using the graph
Flip roles (TO and FROM) Initialize free and unprocessed to the
beginning of the TO space. Pick a node that is in:
– The root set– and– the affinity graph and has the highest edge weight
Perform a greedy DFS on the graph
Step 1 – cont.
Copy each visited object to the TO space Increment the free pointer Store a forwarding address in the FROM
space
Example – After step 1
Step 2 – continues Cheney’s way
Process all objects between the unprocessed and the free pointers, as in Cheney’s algorithm
Example – After step 2
Step 3 - cleanup
Ensure all roots are in the TO space If not, process them using Cheney’s
algorithm
Example – After step 3
Implementation notes
The object access buffer can be used as a stack for the DFS
Inaccurate results(?)
The object affinity graph may retain objects not reachable = garbage
They will be incorrectly promoted at most once
Efforts are focused on longer lived objects and not on the youngest generation
Experimental evaluation
Methodology – If we have the time Object oriented programs manipulate small
objects Real-time data profiling overhead The algorithm impact on performance
Size of heap objects
But that’s not the point!
Small objects often die fast
Surviving heap objects
Real-time data profiling overhead
Overall execution time
Overall execution time - notes
No impact on L1 cache because its blocks are 16B
Compared to WLM algorithm
Comparison notes
WLM (Wilson-Lam-Moher) improves program’s virtual memory locality.
It performed worse or close to Cheney’s because of the 2GB memory
What else?
Other methods
Two methods that can be used with the previous one– Prefetch on grey– Lazy sweeping
Assumptions
Non moving mark-sweep collector For simplicity, the collector segregates
objects by size. Each block contains objects of a single size
The collector’s data structure are outside the user-visible heap
A mark bit is reserved for each word in the block
Advantages of “outside the heap” data
The mark phase does not need to examine (=bring to the cache) pointer-free objects
Sequences of small unreachable objects can be reclaimed as a group– A single instruction is needed to examine their
sequence of mark bits– It is used when a heap block turns out to be
empty
The mark phase – a reminder
Ensure that all objects are white. Grey all objects pointed to by a root. while there is a grey object g
– blacken g– For each pointer p in g
if p points to a white object– grey that object.
The mark phase – colors
1 mark bit– 0 is white– 1 is grey/black
Stack– In the stack – grey– Removed from stack - black
The mark GC problem
A significant fraction of time is spent to retrieve the first pointer p from each grey object
About third of the marker’s execution time is spent
This time is expected to increase on future machines
Prefetching
A modern CPU instruction A program can prefetch data into the cache
for future use
Prefetching – cont.
But object reference must be predicted soon enough
For example, if the object is in main memory, it must be prefetched hundred of cycles before its use
Prefetching instructions are mostly inserted by compiler optimizations
Prefetch on grey
When? Prefetch as soon as p is found likely to be a pointer
What? Prefetch the first cache line of the object
To improve performance
The last pointer to be pushed on the mark stack is prefetched first
It minimizes the cases in which a just grayed object is immediately examined
And to improve more
Prefetch a few cache lines ahead when scanning an object
It helps with large objects It prefetches more objects if it isn’t that large
The sweep GC problem
If (reclaimed memory > cache size)– Objects are likely to be evicted from the cache by
the allocator or mutator
Thus, the allocator will miss again when reusing the reclaimed memory
Lazy sweeping
Originally used to reduce page faults Delay the sweeping for the allocator Pages will be reused instead of evicted from
the cache
A reminder
A mark bit is saved for each word in a cache block.
A mark bit is used only if its word is the beginning of an object
Cache lazy sweeping – the collector
Scans for each block its mark bits If all bits are unmarked, the block is added to
the free blocks’ pool without touching it If some bits are marked, it’s added to a
queue of blocks waiting to be swept There are several queues, one or more for
each object size
Cache lazy sweeping – the allocator
Maps the request to the appropriate object free list
Returns the first object from the list If the list is empty
– It sweeps the queue of the right size for a block with some available objects
Experimental results
Measured on two platforms Second platform is to get some calibration on
architecture variation
Pentium III/500 results
HP PA-8000/180 based results
Results conclusions
Prefetch on grey eliminates a third to almost all cache miss overhead in the marker.
But it is dependent on data structures used in the program
Results conclusions – cont.
Collector performance is determined by the marker
The sweep performance is architecture dependent
Conclusions
Be concerned about cache locality or Have a method that does it for you
Conclusions – cont.
Real-time data profiling is feasible Produce cache conscious data layout using
that information May help reduce the performance gap
between high-level to low-level languages
Conclusions – cont.
Prefetch on grey and lazy sweeping are cheap to implement and should be in future garbage collectors
Bibliography
Using Generational Garbage Collection To Implement Cache-Conscious Data Placement - Trishul M. Chilimbi and James R. Larus
Reducing Garbage Collector Cache Misses - Hans-J. Boehm
Further reading
Look at the articles Garbage collection – algorithms for automatic
dynamic memory management – Richard Jones & Rafael Lins
Further reading – cont.
Cecil – – Craig Chambers. “Object-oriented multi-methods
in Cecil.” In Proceedings ECOOP’92, LNCS 615, Springer-Verlag, pages 33–56, June 1992.
– Craig Chambers. “The Cecil language: Specification and rationale.” University of Washington Seattle, Technical Report TR-93-03-05, Mar. 1993.
Hyperion by Dan Simmons
Items
Large objects Inter-generational objects placement Why explicitly build free lists? Experimental methodology Second experimental methodology
Large objects
Ungar and Jackson : – There’s an advantage from not copying large
objects (>= 256 bytes) with the same age
A large object is never copied Each step has an associated set of large
objects
Large objects – cont.
A large object is linked in a doubly linked list. If it survives a collection, it’s removed from its
list and inserted to the TO space list. No compaction is done on large objects.
Large objects – cont.
Read more in David Ungar and Frank Jackson. “An adaptive tenuring policy for generation scavengers.” ACM Transactions on Programming Languages and Systems, 14(1):1–27, January 1992
Two generations, one cache block
How important is co-location of inter-generation objects?
The way to achieve this is to demote or promote.
Two generations, one cache block – cont.
Intra-generation pointers are not tracked. In order to demote safely, it’s needed to
collect its original generation Result: Long collection time
Two generations, one cache block – cont.
Promote can be done safely– The young generation is being collected and its
pointers updated– Pointers from old to young are tracked
The locality benefit will start only when the old generation is collected
Premature promotion
Why explicitly build free lists?
Allocation is fast Heap scanning for unmarked objects can be
fast using mark bits Little additional space overhead is required
Experimental methodology
Vortex compiler infrastructure Vortex supports GGC only for Cecil Cecil – A dynamically typed, purely object-
oriented language. Used Cecil benchmarks Repeated each experiment 5 times and
reported the average
Cecil benchmarks
Cecil benchmarks – cont.
Compiled at highest (o2) optimization level
The platform
Sun Ultraserver E5000 12 167Mhz UltraSPARC processors 2GB memory – To prevent page faults Solaris 2.5.1
The platform - memory
L1 – 16KB direct-mapped, 16B blocks L2 – 1MB unified direct-mapped, 64B blocks 64 entry iTLB and 64 entry dTLB, fully
associative
The platform – memory costs
L1, data cache hit – 1 cycle L1 miss, L2 hit – 6 cycles L2 miss – additional 64 cycles
Second experimental methodology
Two platforms All benchmarks except one are C programs
Pentium measurements
Dual processor 500Mhz Pentium III (but only one used)
100Mhz bus 512KB L2 cache Physical memory > 300MB (why keep it a secret?),
which prevented paging and allowed the whole executable in memory
RedHat 6.1 Benchmarks compiled using gcc with –O2
RISC measurements
A single PA-8000/180 MHz processor Running HP/UX 11 Single level I and D caches, 1MB each
Benchmarks
Execution time measurements are a five runs average
The division between sweep and mark times is arbitrary
Pentium III prefetcht0 introduced a new overhead, so prefetchnta was used. It was less effective eliminating cache miss, though
?
The end
Lectured by: Shachar Rubinstein
shachar1@post.tau.ac.il
GC seminarMolley Sagiv
Audience:
You
Thanks:
For your patience
The Powerpoint XP effects
My parents
No animals were harmed during this production (except for annoying mosquitoes)
Thank you for listening! (and staying awake…)