Falcon: Fault Localization in Concurrent Programs

1

Background Technique Studies Conclusion

Sangmin Park, Richard Vuduc, and Mary Jean Harrold

College of ComputingGeorgia Institute of Technology

Falcon: Fault Localization inConcurrent Programs

2


• Concurrency bugs are difficult to find• Non-deterministic behavior• Large thread interleaving space

• Godefroid, Nagappan [MSR-TR08] found that

Developers

Handle concurrencyissues

Motivation

3


• Bug finding for concurrency bugs• Static/dynamic data race detection

(e.g., [Ronesse99, Savage97, O’Callanhan03])

• Static/dynamic atomicity violation detection

(e.g., [Flanagan03, Wang06, Flanagan08])

• Dynamic pattern analysis (e.g., [Lu06, Hammer08])

• Fault localization• For sequential programs

(e.g., [Jones02, Liblit03, Liu05, Baah08])Techniques have not been applied to concurrent programsTechniques have not been applied to concurrent programs

1. Techniques target only one type of concurrency bug

2. Techniques report benign (harmless) results along with harmful results

1. Techniques target only one type of concurrency bug

2. Techniques report benign (harmless) results along with harmful results

Existing Techniques

4


Falcon Research Outline

• Background• Technique: patterns + suspiciousness• Empirical Studies• Conclusion

5


Concurrency ViolationsLu et al.’s [ASPLOS08] found that

Deadlock: 30%Deadlock: 30%Order andAtomicity violations:67%

Both occur due to undesirable memory-access sequences

Others: 3%

Both occur due to undesirable memory-access sequences

6


Order ViolationWhen desired order of accesses is reversed

Unintended program behavior

Thread T1

11: void M1()

12: {

13: lock(L);

14: A=new Object();

15: if (A) { /*…*/ }

16: unlock(L);

17: }

Thread T1

11: void M1()

12: {

13: lock(L);

14: A=new Object();

15: if (A) { /*…*/ }

16: unlock(L);

17: }

Thread T2

21: void M2()

22: {

23: lock(L);

24: x=A.getX();

25: unlock(L);

26: }

Thread T2

21: void M2()

22: {

23: lock(L);

24: x=A.getX();

25: unlock(L);

26: }

For A access Correct order Incorrect order

W14

R24 W14

R24

RT2 WT1

7


Atomicity ViolationWhen atomic region is interfered by another thread

Unintended program behavior

Thread T3

31: copy(Obj o)32: {33: …34: int sz=o.sz();35: …36: copy(sz, o, this.buffer);37: …38: }

Thread T3

31: copy(Obj o)32: {33: …34: int sz=o.sz();35: …36: copy(sz, o, this.buffer);37: …38: }

For o access Correct order

Incorrect order

RT3 WT4 RT3

Thread T4

41: access()42: {43: …44: o.clear();45: …46: }

Thread T4

41: access()42: {43: …44: o.clear();45: …46: }

R36

R36

W44

W44

R34

R34

8


Patterns for Concurrency Violations

1 RT1 * WT2

2 WT1 * RT2

3 WT1 * WT2

Patterns

for order violation

1 RT1 * WT2 * RT1

2 WT1 * WT2 * RT1

3 WT1 * RT2 * WT1

4 RT1 * WT2 * WT1

5 WT1 * WT2 * WT1

Patterns for atomicity violation[PPoPP06, ASPLOS06]

Note• Concurrency violations represented as patterns• Patterns not always concurrency violations

Note• Concurrency violations represented as patterns• Patterns not always concurrency violations

1 RT1 * WT2

2 WT1 * RT2

3 WT1 * WT2

1 RT1 * WT2 * RT1

2 WT1 * WT2 * RT1

3 WT1 * RT2 * WT1

4 RT1 * WT2 * WT1

5 WT1 * WT2 * WT1

9


Phase 1. Dynamic pattern recording• Input: a set of executions• Output: patterns, outcome (pass/fail)

Phase 2. Statistical analysis• Compute suspiciousness of each pattern

Technique For Concurrency—Falcon

Intuition Patterns appearing in failing executions are more suspicious than patterns appearing in passing executions

10


R11-W11 -R31

-31

y-Access : W12-W22-R32-R13

x-Access : W11-W21-R31-R13x-Access :

Falcon Example on Atomicity Violation

Initially x=0; y=0;

Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);

Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;

Phase 1. Dynamic pattern recording• Monitor accesses per variable• Monitor accesses per thread• Create window per variable • Detect patterns inside window

x-Access :

y-Access :

Statements:

y-Access :

-21 -12-22-32-1311

-R21-W21 -R13

R12-W12-R22-W22-R32-R13

W11-W21-R31

W12-W22-R32-R13

-R13

x=0 y=0

x=1y=1x=2y=2

x=2y=2

x=2y=2

11



Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;

Run 1 (r1):

x-access:

y-access:

r1 r2 r3 r4 Suspicious-ness

Pattern 1:W11-W31-R13




Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousness

12







P


Run 1 (r1): 11 31 21 12 32 22 13

x-access: W11 W31 R21 R13

y-access: W12 W32 R22 R13






Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;

Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousnessRun 1 (r1): 11 31 21 12 32 22 13



13



Run 2 (r2): 11 31 21 12 22 32 13








P P

Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;


14



Run 3 (r3): 11 21 31 12 32 22 13








P P P

Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;


15



Run 4 (r4): 11 21 31 12 22 32 13








P P P F

Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;


16




0.0


0.5


0.0


0.5

P P P F



0.0



0.0


P P P F







P P P F

Jaccard formula [Abreu07]

suspiciousness(P) =failed (P)

totalfailed + passed(P)

Initially x=0; y=0;


Thread 221:if(x==1) x=2;22:if(y==1) y=2;

Thread 331:if(x==1) x=3;32:if(y==1) y=3;

17


Empirical Studies• Studies

1. Determine window size needed and its relationship to number of threads

2. Evaluate technique’s effectiveness

3. Compare technique with Cooperative Crug Isolation (CCI) [Thakur et al.,WODA09]

4. Evaluate technique’s efficiency (see paper)

• Empirical setup• Implemented in Java• Evaluated on a set of subjects

18


Subjects

Empirical SetupProgram LOC Failure Rate (%) Violation Type

Account 115 3 Atomicity

AirlineTickets 95 54 Atomicity

BubbleSort2 130 69 Atomicity

BufWriter 255 14 Atomicity

Lottery 359 43 Atomicity

MergeSort 375 84 Atomicity

Shop 273 2 Atomicity

ArrayList 5866 2 Atomicity

HashSet 7086 3 Atomicity

StringBuffer 1320 3 Atomicity

TreeSet 7532 3 Atomicity

Vector 709 2 Atomicity

Cache4j 3897 3 Order

RayTracer 1924 14 Atomicity

ContestBenchmark

Java Collection

Account 115 3 Atomicity

19


Goals To determine

• Window size to detect patterns for atomicity violations• Correlation of window size and number of threads

Method• For test case with 100 runs and subjects running with

p threads, where p={4,8,16,32,64}• recorded patterns during execution• computed window size needed to detect each

pattern• Aggregated data from all subjects

Empirical Study 1 – Window Size

20


4 8 16 32 640

10

20

30

40

50

60

Number of Threads

Win

do

w S

ize

to D

etec

t P

atte

rn

• Median window size 3

• Most patterns with window size < 10

• Window size and number of threads not positively correlated

Result – Window Size

21


Goal• To measure effectiveness of suspiciousness ranking

Techniques compared• Conflict Serializability (e..g, [Wang and Stoller,

PPoPP06])• AVIO [Lu et al., ASPLOS06]

Method• For test case with 100 runs and window size 5

• Computed suspiciousness of each pattern• Computed fault-localization expense (number of

patterns that must be examined to find fault)

Empirical Study 2 – Effectiveness

22


Result - Effectiveness

BufWriter 105 25 1

SubjectsConflict-Serializability AVIO Falcon

Patterns (Ranking) Patterns (Ranking) Patterns (Ranking)

Account 11 (-) 7 (-) 11 (2)

AirlineTickets 4 (-) 0 (-) 4 (1)

BubbleSort2 4 (-) 1 (-) 4 (1)

BufWriter 105 (-) 25 (-) 105 (1)

Lottery 4 (-) 4 (-) 4 (1)

MergeSort 76 (-) 42 (-) 76 (1)

Shop 10 (-) 0 (-) 10 (2)

ArrayList 1 (-) 1 (-) 1 (1)

HashSet 7 (-) 3 (-) 7 (1)

StringBuffer 2 (-) 1 (-) 2 (1)

TreeSet 9 (-) 3 (-) 9 (2)

Vector 1 (-) 1 (-) 1 (1)

Cache4j 23 (1)

RayTracer 14 (-) 14 (-) 14 (2)

Conflict-Serializability

Patterns (Ranking)

11 (-)

4 (-)

4 (-)

105 (-)

4 (-)

76 (-)

10 (-)

1 (-)

7 (-)

2 (-)

9 (-)

1 (-)

14 (-)

AVIO

Patterns (Ranking)

7 (-)

0 (-)

1 (-)

25 (-)

4 (-)

42 (-)

0 (-)

1 (-)

3 (-)

1 (-)

3 (-)

1 (-)

14 (-)

Falcon

Patterns (Ranking)

11 (2)

4 (1)

4 (1)

105 (1)

4 (1)

76 (1)

10 (2)

1 (1)

7 (1)

2 (1)

9 (2)

1 (1)

23 (1)

14 (2)

BufWriter 105 (-) 25 (-) 105 (1)

* (-) : Patterns are not ranked

23


Goal• To compare effectiveness of Falcon to CCI*

• CCI observes memory accesses, outcome (pass/fail)• CCI reports suspiciousness of each memory access

predicate (i.e., memory access and previous access type pair)

Method• Implemented the CCI technique in the Falcon toolset• For test case with 100 runs and subjects

• Executed the tool• Found the access and rank of actual bug

Empirical Study 3 – Case Study

* Cooperative Crug Isolation [Thakur, Sen, Liblit, Lu, WODA09]

24


Result – Case StudySubjects

CCI Falcon

Accesses of Bug (Rank) Pattern of Bug (Rank)

Account R (6), W (4), R (6) R-W-R (2)

AirlineTickets R (2), W (5), R (4) R-W-R (1)

BubbleSort2 W (1), R (7), W (2) W-R-W (1)

BufWriter R (5), W (1), R (5) R-W-R (1)

Lottery R (1), W (7), R (4) R-W-R (1)

MergeSort W (1), R (3), W (12) W-R-W (1)

Shop R (15), W (12), R (15) R-W-R (2)

ArrayList R (4), W (4), R (4) R-W-R (1)

HashSet R (15), W (15), R (1) R-W-R (1)

StringBuffer R (14), W (2), R (1) R-W-R (1)

TreeSet R (10), W (5), R (10) R-W-R (2)

Vector R (2), W (2), R (2) R-W-R (1)

Cache4j W (79), R (2) W-R (1)

RayTracer R (4), W (4), W (3) R-W-W (2)

HashSet R (15), W (15), R (1) R-W-R (1)

Shop R (15), W (12), R (15) R-W-R (2)

CCI

Accesses of Bug (Rank)

R (6), W (4), R (6)

R (2), W (5), R (4)

W (1), R (7), W (2)

R (5), W (1), R (5)

R (1), W (7), R (4)

W (1), R (3), W (12)

R (15), W (12), R (15)

R (4), W (4), R (4)

R (15), W (15), R (1)

R (14), W (2), R (1)

R (10), W (5), R (10)

R (2), W (2), R (2)

W (79), R (2)

R (4), W (4), W (3)

Falcon

Pattern of Bug (Rank)

R-W-R (2)

R-W-R (1)

W-R-W (1)

R-W-R (1)

R-W-R (1)

W-R-W (1)

R-W-R (2)

R-W-R (1)

R-W-R (1)

R-W-R (1)

R-W-R (2)

R-W-R (1)

W-R (1)

R-W-W (2)

Account R (6), W (4), R (6) R-W-R (2)

Falcon ranking1. WS1-WS3-RS2

2. RS6-WS2-RS6

3. WS4-RS6-WS2

4. RS1-WS4-WS6

5. WS6-WS4-RS6

6. RS2-WS4-WS6

7. …

Falcon ranking1. RS2-WS3-RS1

2. RS6-WS2-RS6

3. RS4-WS6-WS2

4. WS1-WS3-WS6

5. RS6-WS3-RS6

6. RS2-WS3-RS3

7. RS1-WS4-WS6

• CCI reports the actual bug with lower ranks• CCI does not provide an association of buggy

accesses

• CCI reports the actual bug with lower ranks• CCI does not provide an association of buggy

accesses

CCI ranking1. RS1

2. RS2

3. WS3

4. WS4

5. RS5

6. RS6

7. …

CCI ranking1. RS1

…

12. WS2

13. …

14. …

15. RS6

…

CCI ranking1. RS1

…

…

…

15. RS2

15. WS3

…

Falcon ranking1. WS1-WS3-RS2

2. RS6-WS4-RS6

3. WS4-WS6-RS2

4. RS1-WS4-RS6

5. WS6-WS4-WS6

6. RS2-WS4-WS6

7. …

25


Future Work

• Perform more empirical studies• With larger subjects• With more subjects especially with order violations• With considering multiple test cases

• Apply to other languages and systems• Other than Java language

• Extend the technique to support• Handling multi-variable related patterns for

atomicity violations• Localizing deadlocks

26


Contributions

• Falcon is the first technique• that reports and ranks patterns by

suspiciousness• that addresses order violations and

atomicity violations with patterns• Studies show that

• Falcon’s window size• Small constant • Not correlated with number of threads

• Falcon’s suspiciousness ranking technique• Effective in identifying faulty interleavings• More effective than ranking individual accesses

Questions?Questions?

27


Backup Slides

28


Result – EfficiencyConflict-Serializability (PPoPP 06)

AVIO(ASPLOS 06)

Atomic-Set-Serializability(ICSE 08)

Falcon(ICSE10)

Efficiency (Average slowdown)

38X (Median slowdown)

25X 13X(Used almost same set of subjects as Falcon)

9.9X

Falcon: Fault Localization in Concurrent Programs

Technology

Transcript of Falcon: Fault Localization in Concurrent Programs