How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd 2014-09-30.
-
Upload
angel-willa-jefferson -
Category
Documents
-
view
224 -
download
0
Transcript of How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd 2014-09-30.
How not to do Java ConcurrencyAnd how to find if you did it wrong
Mark Winterrowd
2014-09-30
Introductions
• Coverity
• Me
• You!
• Bob
Copyright Coverity, Inc., 20142
Copyright Coverity, Inc., 20143
Meet Bob
Bob's development philosophy:
• Speed is everything
• Clever tweaks are their own reward
• Trying it out once or twice is plenty testing
Your manager has assigned him to help you out!
Your Mission: Keep your code correct!• Bob will write code mirroring mistakes seen in the
wild
• For novices: preparation for handling your own Bob
• For experts: Examples occurring in real code
• Modeled off of code seen in Tomcat, Eclipse and Android
Copyright Coverity, Inc., 20144
Patterns discussed here are low level• Prefer to use higher level structures
• e.g., java.util.concurrent
• Hide uses behind higher level calls or objects
• Sometimes unavoidable
Copyright Coverity, Inc., 20145
Multithreaded Lazy Initialization
Or, sacrificing correctness for speed
Copyright Coverity, Inc., 20147
Task: Lazily initialize a singleton
In a single-threaded program:
private static MyObj inst = null;
public static MyObj getInst() {
if(inst == null) {
inst = new MyObj();
}
return inst;
}
This won't work in a multithreaded program.
Copyright Coverity, Inc., 20148
Simple Fix:
private static MyObj inst = null;
static final Object initLock = new Object();
public static MyObj getInst() {
synchronized(initLock) {
if(inst == null) {
inst = new MyObj();
}
return inst;
}
}
Bob objects: "This requires you to wait to get the lock every time!"
Copyright Coverity, Inc., 20149
Bob's Performance Optimization
public static MyObj getInst() {
inst = new MyObj();
}
}
return inst;
}
"Since we only need to set inst the first time, synchronizing to check for null is unnecessary!"
if(inst == null) {
synchronized(initLock) {
Is Bob right?Is his code correct?
10
Bob's optimized code in action
Copyright Coverity, Inc., 2014
Thread 1 Thread 2
if(inst == null)
if(inst == null)
synchronized(initLock)inst = new MyObj();return inst;
synchronized(initLock)inst = new MyObj();return inst;
No longer a singleton!
The JVM can start and stop threads at will• Holding a lock prevents another thread from
executing code guarded by the same lock object
• Bob says "In that case, I have an idea!"
Copyright Coverity, Inc., 201411
Copyright Coverity, Inc., 201412
"Also check in the synchronized block!"private static MyObj inst = null;
static final Object initLock = new Object();
public static MyObj getInst() {
if(inst == null)
synchronized(initLock) {
"Now, we can't have inst be initialized multiple times!"
} return inst;}
if(inst == null)
inst = new MyObj();
Is Bob right? Is his code correct?
13
Let's take a look at this line
Copyright Coverity, Inc., 2014
inst = new MyObj();
MyObj() {
this.field1 = someVal1;
this.field2 = someVal2;
}
A raw MyObj is created before running the constructor
(raw MyObj).field1 = someVal1;(raw MyObj).field2 = someVal2;inst = (raw MyObj);
Copyright Coverity, Inc., 201414
JVM re-orders code at runtime!• Must have same result in current thread
• Cannot move code out of or through a synchronized block• Reordering inside or into a synchronized is fine!
public static MyObj getInst() { if(inst == null) synchronized(initLock) { if(inst == null) {
inst = new MyObj();
} } return inst;}
inst = (raw MyObj)inst.field1 = someValinst.field2 = someVal2
This instruction ordering in action:
Copyright Coverity, Inc., 201415
Thread 1 Thread 2
if(inst == null)
if(inst == null)
synchronized(initLock)
inst = (raw MyObj);
return inst;Returned an uninitialized MyObj!
What is the name of this bad pattern?
if(inst == null)
Copyright Coverity, Inc., 201416
Preventing reordering bugs
• Hold the lock while checking the field for null
public static MyObj getInst() {
synchronized(initLock) {
if(inst == null)
inst = new MyObj();
}
return inst;
}
if(inst == null)Back to our original solution!
"This problem seems unlikely. Is fixing it
worth the slowdown?"
Copyright Coverity, Inc., 201417
What slowdown? How unlikely?
• Uncontested lock acquisition is fast in modern JVMs
• In computing, unlikely can happen often
• In 2004, a developer opened Eclipse bug 50178 with this comment:
"I know I'm picky, but [...] getManifest has an example of what is known as "Double-Checked Locking problem [...] BundleLoader.createClassLoader has the same problem"
Copyright Coverity, Inc., 201418
In the wild: Eclipse bug 50178Reply in 2006:"Synchronizing the places where we lazily create ClassLoader/BundleLoaderProxy objects will cause measurable slowdown. To fix this properly would require at least three additional syncs [...] I added todo comments in the code where double checks are currently being done."A month later:"We are currently having a very hard to reproduce bug when loading preference pages of our RCP application [...] I traced the problem to getBundleLoader [...] When looking at [two methods], I saw they both contained the TODO for this bug.
Isn't it better to have slightly less performance instead of possible threading bugs?"
Copyright Coverity, Inc., 201419
Bug 50178 as reported in Coverity Connect
Copyright Coverity, Inc., 201420
Premature Optimization...
• Especially dangerous in concurrency
• Rare bugs may be likely in a user's workflow
• Slow locking may be unnecessary contention• Is the lock protecting too much data?
• Is the lock protecting unrelated critical sections?
• Can some expensive operations be safely moved outside of a locked region?
Choosing your locks
Or rather, what not to choose
Copyright Coverity, Inc., 201422
Make this code thread safe
static Object[] items;
void update(Object newItem) {
"Synchronize on items so another
thread can't change items." Object[] oldArr = items;
items = new Object[items.length + 1]
items[items.length - 1] = newItem;
for(int i=0; i < oldArr.length; i++){
items[i] = oldArr[i];
}
}
}
synchronized(items) {
23
What does synchronizing on a field do? • Does not block accesses from other threads to that
field
• Acquires a lock on the field's contents
Copyright Coverity, Inc., 2014
Thread 1
synchronized(items)
items = new Object[...]
synchronized(items)items = new Object[...]
Different contents!
Object[ ] oldArr = items
items[...] = newItem
Thread 2
Copyright Coverity, Inc., 201424
Don't lock on mutable fields
static Object[] items;
void update(Object newItem) {
synchronized(
Object[] newAr = new Object[items.length+1];
newAr[newAr.length - 1] = newItem;
for(int i=0; i < items.length; i++){
newArr[i] = items[i]
}
items = newAr;
}
}
) {
private static final Object lock = new Object();
itemslock
Copyright Coverity, Inc., 201425
Tomcat bug 46990: Locking a mutable field
"Whilst there aren't any
explicit bugs caused by this,
it may be behind some of the harder to
reproduce bugs."
26
Bob sends you the following code for review
Copyright Coverity, Inc., 2014
class AppCtx { private static SysCtx sCtx;
public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; }}
"synchronized ensures this
code is thread-safe!"
27
What does the synchronized modifier do?
Copyright Coverity, Inc., 2014
class AppCtx { private static SysCtx sCtx;
public synchronized SysCtx getSysCtx() {
if(sCtx == null) { sCtx = new SysCtx();}return sCtx;
}}
synchronized(this) {
28
Two AppCtx, each in their own thread
Copyright Coverity, Inc., 2014
this
AppCtx 1 AppCtx 2
this
sCtxnull
unlocked
unlocked
synchronized(this)Static AppCtx
Members
AppCtx1 AppCtx2
if(sCtx == null)
synchronized(this)
if(sCtx == null)
sCtx = new SysCtx()
InitializedInitialized
return sCtx;
sCtx = new SysCtx()return sCtx;
No longer a singleton!
29
Guard static members with static locks
Copyright Coverity, Inc., 2014
class AppCtx { private static SysCtx sCtx;
public SysCtx getSysCtx() { synchronized( ) { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } }}
private static final Object lock=new Object();
thislock
Copyright Coverity, Inc., 201430
In the wild: Android bug 12015587
Caused loss of display info, user name, and permissions
Wait and Notify
and threads communicating poorly
Copyright Coverity, Inc., 201432
Thread 1 consumes results from Thread 2LinkedList<QueueItem> jobQueue;
// Called in Thread 1
void processItemFromQueue() {
QueueItem item = null;
synchronized(qLock) {
item = jobQueue.remove();
}
processItem(item);
}
// Called in Thread 2void putItemInQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); }}
What if jobQueue is empty?
Copyright Coverity, Inc., 201433
The wait/notify pattern
• lock.wait( ): current thread stops running, releases lock, and is placed on lock's wait set
• lock.notifyAll( ): Starts up all threads in the wait set, which attempt to re-acquire lock.
• Call wait when your thread needs another thread to satisfy some condition
• Call notifyAll when your thread has satisfied a condition for another thread
Copyright Coverity, Inc., 201434
wait and notify in actionExecuting
Instructionslock
Locked By
Unlocked
Wait Set
synchronized(lock)
lock.wait( )
Blocked on lock
synchronized(lock)
lock.wait( )
synchronized(lock)
lock.wait( )
lock.notifyAll( )
release lock
synchronized(lock)
release lock
Copyright Coverity, Inc., 201435
You see this in a review of Bob's code:void processItemFromQueue() {
QueueItem item = null;
// Hold the lock as briefly as possible!
if(jobQueue.empty())
synchronized(qLock) {
qLock.wait();
}
item = jobQueue.remove();
processItem(item);
}
void addItemToQueue(QueueItem item) {
synchronized(qLock) {
jobQueue.add(item);
qLock.notifyAll();
}
}
Is Bob's code correct?
Copyright Coverity, Inc., 201436
Bob's code in action:Executing
Instructions
qLock
Locked By
Unlocked
Wait Set
if(jobQueue.empty())
Consumer
qLock.wait()
C
Blocked on qLocksynchronized(qLock)
Consumer Producer
synchronized(qLock)
ProducerjobQueue.add(item)
release qLock
qLock.notifyAll()
No waiting threads to notify
returns true
Consumer is waiting while items are in the queue!
Copyright Coverity, Inc., 201437
Waiting on a stale condition is a bug• Another thread can change the state between the
check and the wait
• At best, causes unnecessary delays• What if future notifications were blocked on this task?
• Always check wait condition while holding lock
Bob comes back with a second version...
38
item = jobQueue.remove();
Code review round #2
void processItemFromQueue() {
QueueItem item;
Copyright Coverity, Inc., 2014
Bob: "Now the wait will only occur when the queue is empty!"
if(jobQueue.empty()) {
processItem(item);
qLock.wait();
synchronized(qLock) {
Is Bob right?Is his code correct?
"A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup."-Object.wait() documentation
}}
}
Bob's code in action:
Copyright Coverity, Inc., 201439
Executing Instructions
qLock
Locked By
Unlocked
Wait Set
if(jobQueue.empty())
ConsumerqLock.wait()
C
Blocked on qLock
synchronized(qLock)
Consumer Producer
A spurious wakeup occurs!
jobQueue.remove()
jobQueue empty!
Copyright Coverity, Inc., 201440
Check your wait condition inside a loop!
void processItemFromQueue() {
QueueItem item;
synchronized(qLock) {
qLock.wait();
item = jobQueue.remove();
}
processItem(item);
}
whileif(jobQueue.empty())
Copyright Coverity, Inc., 201441
In the wild: Eclipse bug 366048
• Deadlock when ctrl-x was used to cut text.
• Diagnosed as a wait occurring and never waking up.
• Remains unfixed, timeout to wait was set to 30s.
Copyright Coverity, Inc., 201442
The bug (in 2004 code)
In code from 2005This bug was "fixed" with a 30 second wait timeout in 2013.
Preventing new concurrency bugs
and fixing the ones you have
Copyright Coverity, Inc., 201444
Don't get overly clever!
• Do you need a multithreaded program?
• Do you need lazy initialization?• Initialize using static{}
• Is there a higher level structure to replace wait/notify?• Our example: java.util.concurrent.BlockingQueue
• Sometimes it's unavoidable, or hard to remove.
Standard methods help somewhat
• Education
• Code reviews
• Testing
Copyright Coverity, Inc., 201445
Nondeterminism limits effectiveness
Is there anything else we can do?
Don’t fix pre-existing problems
Copyright Coverity, Inc., 201446
Manual Inspection• Responsible for finding many of our sampled
fixed defects
• One tip we found in Bugzilla:
• No guarantee anyone will notice right away• Most of the fixed bugs found by inspection had
been present for over 3 years.
• Why not have a machine inspect your code instead?
is Slow
"Search[ing] for the 'synchronized' keyword goes a long way"
Copyright Coverity, Inc., 201447
Static Analysis: Automatic code inspection• Open source static analysis FindBugs™ finds many
issues
• But Coverity's static analysis finds• more real bugs
• with better explanations
• and a lower rate of false reports
• Also, we integrate with FindBugs
Copyright Coverity, Inc., 201448
Analyze your code for free!
• Coverity's SCAN program provides free defect reports for open source programs.• http://scan.coverity.com
• Coverity's CodeSpotter: free-to-use SaaS analysis• http://www.code-spotter.com
• Also a commercial enterprise solution for proprietary software• http://www.coverity.com
Copyright Coverity, Inc., 201449
More about Java Concurrency
• Java Language Specification
• "Java Concurrency in Practice" by Goetz et al
• "Java theory and practice: Are all stateful Web applications broken?" by Goetz• http://www.ibm.com/developerworks/library/j-jtp09238/
• "Double checked locking is broken", many signers.• http://www.cs.umd.edu/~
pugh/java/memoryModel/DoubleCheckedLocking.html
Copyright 2014 Coverity, Inc.