Keeping Infinispan In Shape: Highly-Precise, Scalable Data Eviction

[email protected] | twitter.com/galderz | zamarreno.comThursday, October 7, 2010

mailto:[email protected]


http://twitter.com/maniksurtani


[email protected] | twitter.com/galderz | zamarreno.com

Keeping Infinispan In Shape: Highly-Precise, Scalable

Data Eviction

Galder Zamarreño & Vladimir BlagojevicSenior Engineer, Red Hat7th October 2010, JUDCon - Berlin

Thursday, October 7, 2010


Who is Galder?

• R&D engineer (Red Hat Inc):• Infinispan developer• JBoss Cache developer

• Contributor and committer:• JBoss AS, Hibernate, JGroups, JBoss Portal,...etc

• Blog: zamarreno.com• Twitter: @galderz







Agenda

• Eviction in:• Java Collections Framework• JBoss Cache• Infinispan 4.0

• New in Infinispan 4.1:• LIRS• Batching Updates







Java Collections Framework

• Key building block of any Java app

• Introduced in Java 1.2

• Extended with concurrent collections in Java 1.5

• Collection element eviction ??







Collections cannot grow forever







If using a collection as cache

• Forces clients to either:

• Remove elements proactively

• Or run a periodic cleanup process

• Which can be a PITA...







Eviction in JBoss Cache days







JBoss Cache eviction issues

• Under heavy load, eviction queues could fill up - bottleneck

• Possibly a side effect of MVCC’s non-blocking reads

• Separating data into regions could alleviate the issue

• But it’s really a hack!







Original Infinispan 4.0 eviction

• Tried a different approach:

• Avoid using queues to hold events

• Taking advantage of tree to map change:

• Attempt to maintain map ordered as per eviction rules

• Could we use ConcurrentSkipListMap ?

• No. O(1) desired for all map operations







Doubly Linked ConcurrentHashMap

• Based on H.Sundell and P.Tsigas paper:

• Lock-Free Deques and Doubly Linked Lists (2008)

• With each cache access or update update links

• Eviction just a matter of walking the linked list one way







Issues under heavy load

• Algorithm always uses same node (the tail) to append stuff

• With high concurrency, CAS stress lead to loads of retrying

• So much retrying, we’re getting infinite loops

• Before 4.0.0.Final, reverted to a more conservative algorithm







4.0.0.Final eviction







LRU eviction algorithm issues

• Weak access locality

• One-time accessed keys not evicted timely

• In loops, soon to be accessed keys might get evicted first

• With distinct access frequencies, frequently accessed keys can unfortunately get evicted

• LRU’s working set limited to cache size







Enter the room... LIRS

• Eviction algorithm that can cope with weak access locality

• Based on S.Jiang and X.Zhang’s 2002 paper:

• LIRS: An efficient low inter-reference recency set replacement policy to improve buffer cache performance

• LIRS based around two concepts: IRR and Recency







IRR and Recency• Inter-Reference Recency (IRR):

• Number of other unique keys accessed between two consecutive accesses to same key

• Recency (R)

• Number of other unique keys accessed from last reference until now







How does LIRS work?

• If key has a high IRR, it’s next IRR is likely to be high again

• Keys with highest IRR are considered for eviction

• Once IRR is out of date, we start relying on Recency

• LIRS = Low Inter-reference Recency Set







How is LIRS implemented?

• Low IRR (LIR) area

• Holds hot keys!

• High IRR (HIR) area

• Holds recently accessed keys

• Keys here might get promoted to LIR area







Cache Hit - LIR area







Hit - HIR area and in LIR Q ...







Hit - HIR area and in LIR Q







Hit - HIR area and not in LIR Q







Cache Miss







LIRS implementation hurdles

• LIRS requires a lot of key shifting around

• It can lead to high contention

• Unless you can implement it in scalable way, it’s useless

• Low contended way to implement a high precision eviction algorithm? Is it possible?







Batching Eviction Updates

• Original idea: X.Ding, S.Jiang and X.Zhang’s 2009 paper:

• BP-Wrapper: A System Framework Making Any Replacement Algorithms (Almost) Lock Contention Free

• Keeping cache access per thread in a queue

• If queue reaches a threshold:

• Acquire locks and execute eviction as per algorithm

• Batching updates significantly lowers lock contention







Batching Updates in Infinispan

• Decided against recording access per thread

• 100s threads could be hitting cache; some short lived

• Created BoundedConcurrentHashMap

• Based on Doug Lea's ConcurrentHashMap

• Records accesses in a lock-free queue in each segment

• When threshold passed, acquire lock for segment and evict







Precision and Performance

• Segment level eviction does not affect overall precision

• Community member run some performance tests:

• After swapping their own LRU cache with BoundedCHM:

• Berlin SPARQL Benchmark performance increased 55-60% for both cold and hot caches







Summary

• In JBoss Cache, eviction can become a bottleneck

• Infinispan 4.0 uses conservative eviction

• Infinispan 4.1 has more precise eviction algorithm (LIRS)

• Batching updates, present in 4.1, significantly lowers lock contention

• Result = Highly-concurrent, highly-precise implicit eviction







Questions?

infinispan.org

blog.infinispan.org

twitter.com/infinispan

#infinispan

#judcon












Keeping Infinispan In Shape: Highly-Precise, Scalable Data Eviction

Technology

Transcript of Keeping Infinispan In Shape: Highly-Precise, Scalable Data Eviction