Introduction Previous Work Proposed Solution Evaluation Conclusions

Deadlock Immunity: Enabling Systems To Defend AgainstDeadlocks

Authors: Horatiu Jula1, Daniel Tralamazza1,Cristian Zamfir1,George Candea1

Presenter: Mihaela-Andreea Vasile2

1Dependable Systems LaboratoryEPFL

Switzerland

2Computer Science Department, Faculty of Automatic Control and ComputersUniversity Politehnica of Bucharest

Bucharest, Romania

December 4, 2014

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

1 Introduction

2 Previous Work

3 Proposed Solution

4 Evaluation

5 Conclusions

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Deadlocks

Parallel applications- behaviour of concurrently running threads- great number of bugs

Deadlocks- synchronization mutex locks- incorrect coordination in using locks

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Deadlocks

Sources for deadlocks- bugs in the source codes- threads scheduling at runtime- using third party code

Deadlock immunity- develop resistance against deadlocks- automatically- at runtime

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Techniques for avoiding deadlocks

Language-level approaches- new type systems: simplify

writing concurrent code- + no runtime overhead- - new languages/constructs

Transactional memory- threads execute transactions

instead of synchronized blocks- locking thread scheduling- improve performance use

locks

Static analysis tools- detect deadlocks at compile

time- programmers involvement

Ownership types for safe programming: preventing data races anddeadlocks (Boyapati, Lee, Rinard)

Transactional memory: Architectural support for lock-free datastructures (Herlihy, Moss)

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Techniques for avoiding deadlocks

Toward dynamic approaches- detect deadlocks at runtime gate locks- serialize the access to locks that could cause deadlocks (JVM)- large amount of false positives

Pure dynamic approaches- create checkpoints and revert execution- re-execute in modified environment- program does not improve- performance overhead

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

System overview

RAG- allocation graph- built by Dimmunix based on

runtime events

Avoidance code- intercept lock/unlock- decide if GO/YIELD based on

cached RAG- place events in queue for

Dimmunix

Monitor Thread- build RAG- update cached RAG for

Avoidance code- cycle detection- save deadlock/starvation

signatures

Figure: Dimmunix architecture

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

RAG

Built based on runtime vents provided by Avoidance component

Updated every miliseconds

Vertices- threads- locks

Edges- request - T wants to acquire L

(T L)- allow - T waits for L (T L)- hold - T acquired L (L T)- yield - T cannot acquire L

because of T1(T T1) Figure: RAG example

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Detect Deadlocks/Starvation

Detect cycles in RAG

Deadlock cycles- T in cycle with only hold, allow, request edges

Starvation cycles- yield cycles- T1 is reachable from T through yield

edges- every T1 can reach T

Figure: Starvation cycles

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Recover from Deadlocks/Starvation

Deadlocks- not part of Dimmunix- program restart- define recovery handlers, called after saving the signature

Starvation- T - starved, holds most locks- cancel yield for T- go for the most recent requested lock

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Avoid Previous Patterns

Save signatures- for detected deadlock/starvation cycles- stack labels for all hold/yield edges- general patterns - only call flow

Avoid Deadlocks- request lock

- add allow edge in cached RAG

- check signatures in history

- potential deadlock: yield T (cached RAG, event, response)

- no deadlock: allow T (cached RAG, event, response)

- acquire lock- allow edge hold edge- acquired event to monitor

- release lock- remove hold edge

- release event to monitor

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Avoid Previous Patterns

Avoid Deadlocks- save a matching depth- importance: miss deadlock/false positives- fixed value/ heuristic

Avoid Starvation- weak immunity

- default

- break starvation

- ignore Dimmunix

- miss some deadlocks

- strong immunity- restart program

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Properties and Limitations

Deadlock-free programs- not affected by Dimmunix- signatures saved on deadlock occurrences

Correct outputs- alter thread scheduling- non-real-time systems: the output does not depend on the scheduler

Lose functionality- all execution paths are deadlocked

- user-disabled signatures

- automatically disable frequent patterns

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Implementations and Evaluation

Implementations- Java - byte code instrumentation- POSIX threads: FreeBSD, LinuxNPTL - modified libraries- no need to modify source code nor programming language

Experiments conditions- real-systems integration: MySQL, Apache Active MQ, Java JDK- strong immunity- = 100 msec

Considered Topics- Real deadlocks efficiency- Performance Overhead- False Positives

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Real Deadlocks Efficiency

- Deadlock bugs: test cases to reproduce reported bugs

- Third party libraries: Java JDK - synchronized classes

Figure: Reported bugs avoided byDimmunix

Figure: Java JDK 1.6 deadlocks avoidedby Dimmunix

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Performance Overhead

- real-applications: various historysize

- micro benchmarks- Nt threads, Nl locks- held time, and request timein, out

- synthetic deadlock history

Figure: Overhead as function of historysize

Figure: Overhead as function of nothreads

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Performance Overhead

Figure: Overhead as function of in,out, Java

Figure: Overhead as function of historysize, pthreads

Figure: Breakdown overhead, Java

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

False Positives and Resource Utilization

- False Positives- too shallow signature match- the patterns depend on inputs- the patterns depend on the program state

- Resource utilization- 2-1024 threads, 8-32 locks, 64 signatures- pthreads: 6-25 MB- Java: 79-127 MB

Figure: False positives overhead

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Further work

Optimizations for Dimmunix- deadlock detection: performed oine- deadlock avoidance

- monitor statements previously envolved in deadlocks

- reduce the number of false positives

- reduce yielding time

Communix- collaborative immunity framework for Java- complementary for Dimmunix- sends signatures to a server- select new signatures appropriate to the Java application

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

Conclusions

- programs: develop resistance against deadlocks

- complementary tool for software systems

- may be used when fixing is expensive/risky

- may trigger latent bugs

- Dimmunix has been improved and extended

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

Introduction Previous Work Proposed Solution Evaluation Conclusions

H. Jula, D. Tralamazza, C. Zamfir, G. Candea Dependable Systems Laboratory EPFL

Deadlock Immunity: Enabling Systems To Defend Against Deadlocks

IntroductionPrevious WorkProposed SolutionEvaluationConclusions