Automatic Program Correction

Anton AkhiFriday, July 08, 2011

2

Plan

• Introduction• Survey of existing tools

3

Why do we need automatic program correction?

• Even if bug has been found it is still programmer’s job to think of fix

• Allows to fix bug automatically or provide high-quality fix suggestions

4

Isn’t it a magic?

• Tools require failing and passing runs• Oracle to check if test passes• Program with bug is not far away from the

right one

5

Existing tools

• Genetic programming:– Automatic Program Repair with Evolutionary Computation– A Novel Co-evolutionary Approach to Automatic Software

Bug Fixing• Machine learning:

– BugFix• Contract usage:

– Automated Debugging using Path-Based Weakest Preconditions

– AutoFix-E– AutoFix-E2

6

Automatic Program Repair with Evolutionary Computation

• W. Weimer, S. Forrest, C. Le Goues, T. Nguyen• Usage of GP to fix bugs:– Individuals– Genetic Operators– Fitness function

7

Genetic programming: individuals

• Individual is represented as an abstract syntax tree and weighted path

• Weighted path is a list of statements visited on negative test case with weight of every statement:– 1 if not visited by positive tests– 0.1 if visited by positive tests

8

Genetic programming: operators

• Mutation:– statement on a weighted pass is considered for

mutation with probability proportional to its weight

– deletion, insertion, swap• Crossover:– exchange of subtrees chosen at random between

two individuals

9

Genetic programming: fitness function

• Weighted sum of test cases passed• Weight of a negative test is at least as much as

positive test

10

Minimizing changes

• Trim unnecessary edits• One-minimal subset of changes is a subset

such that without any of the changes program will stop passing all the tests

• Use delta debugging to compute one-minimal subset of changes

11

Automatic Program Repair with Evolutionary Computation: Conclusions

• Needs:– negative and positive test cases– oracle– fault localization

• Uses:– genetic programming– Delta debugging

• Method has a lot of criticism

12

A Novel Co-evolutionary Approach to Automatic Software Bug Fixing

• A. Arcuri and X. Yao• Genetic Programming• Distance Functions• Search Based Software Testing• Co-evolution

13

Genetic Programming

• Genetic Program consists of primitives• Program is represented in a tree form• Fitness function to be minimized:

• is a number of nodes in a program• is a number of raised exceptions• is a special distance function

iTt

P tgtdgE

gE

gN

gNgf ,

11

gN

gE

tgtd P ,

14

Distance Functions

• Works fine with numbers and boolean expressions

• Predicates involving and can be handled only in cases small

Q

15

Search Based Software Testing

• Find tests that make evolutionary programs fail

• Fitness function for test case to be maximized:

iGg

P tgtdtf ,

16

Co-evolution

• Competitive co-evolution• First generation: copies of buggy program and

unit tests• Mutations, crossover, replacement by the

original program• Penalty for short programs

17

A Novel Co-evolutionary Approach to Automatic Software Bug Fixing: Conclusions

• Needs:– starting set of unit tests– oracle

• Uses:– genetic programming– co-evolution– search based software testing

• Some bugs are too difficult to solve• Bug that is difficult to be fixed by a human

might be very easy for program

18

BugFix: A Learning-Based Tool to Assist Developers in Fixing Bugs

• D. Jeffrey, M. Feng, N. Gupta, R. Gupta• Association rule learning• Interesting value mapping pairs (IVMP)• Situation descriptors• Knowledgebase of rules

19

Association rule learning

• Association rule learning is a popular method for discovering the relationship between variables in database

• Association rule where X, Y are sets of attributes and means that if the items in set X are present then it is probable that the items in set Y are also present

• The confidence of the rule is

where supp(X) is the fraction of transactions containing X

YX YX

XsuppYXsuppYXconf

20

Interesting Value Mapping Pairs

• An Interesting Value Mapping Pair (IVMP) is a pair of value mappings (original, alternate) associated with a particular statement instance in a failing run, such that: (1) original is the original value mapping used by the failing run at that instance; and (2) alternate is an alternate (different) value mapping such that if the values in original are replaced by the values in alternate at that instance during re-execution of the failing run, then the incorrect output of the failing run becomes correct

21

Situation descriptors

• Statement structure situation descriptors• IVMP pattern situation descriptors• Value pattern situation descriptors

22

Statement structure situation descriptors

• Unordered tokens comprising the statement

23

Statement structure situation descriptors: examples

24

IVMP pattern situation descriptors

• Consider pattern to occur when corresponding values in the IVMPs compare to each other in the same way across all IVMPs at a statement

• Compare in terms less than, greater than, or equal to

• Look at the pairs of values:– within original sets of values– within alternate sets of values– between corresponding values in sets

25

IVMP pattern situation descriptors: examples

26

Value pattern situation descriptors

• Similar to IVMP patterns

27

Knowledgebase of rules

• Database of bug-fix scenarios• Initially created through training data of

known debugging situations

28

How it works

• Identify rules to consider– rules in which debugging situation is subset of the

current debugging situation• Sort rules by confidence values• Report prioritized bug-fix descriptions• Learn from current debugging situation and

the corresponding bug fix

29

BugFix: A Learning-Based Tool to Assist Developers in Fixing Bugs: Conclusions

• BugFix assists in fixing bugs by producing a list of bug-fix suggestions

• Tool learns through new situations• Is not very good with new and logically

difficult bugs

30

Automated Debugging using Path Based ‑Weakest Preconditions

• H. He, N. Gupta• Representation of an error trace• Path-based weakest precondition• Hypothesized program state• Actual program state• Detection of evidence• Location and modification of likely errorneous

statement

31

Representation of an error trace

• is an instance of an executed statement in an error trace; i is an execution point and j is the line number of statement

• Branch predicates:– atomic predicate: (expr relop const)– compound predicates are in disjunctive normal form:

where and is an atomic predicate

• Precondition and postcondition for failing run are transformed in DNF:– replace and by disjunction and conjunction

ji,

neeE 0 ni gge 0 ig

32

Path based weakest precondition‑

• pwp(T, R) is the set of all states such that an execution of function F that flows execution trace T begun in any of them is guaranteed to terminate is state satisfying R

• where means substituting every occurrence of x in R with a

• where B is branch predicate•

axRRaxpwp ),( ax

RRBpwp ),(

)),(,(),;( RDpwpCpwpRDCpwp

33

Hypothesized program state

• Let where is a trace from point i to the end of trace and R is postcondition

• defines the set of hypothesized program states at an execution point i

• and

RTpwpR nii ,,niT ,

iR

1, iii RStmtpwpR RStmtpwpR nn ,

34

Actual program state

• Represented by predicates in DNF which are true for given input

• Consists of forward program states and backward program states

•

FiQ

BiQ

Bi

Fii QQQ

35

Forward and backward program states

• Forward program states are defined as:– positive conjunctions in precondition– – – and are sets of predicates killed by and

derived from statement • Backward program states are defined as:– if is an assignment statement– if is a branch predicate–

FQ1 111 ii

Fi

Fi GenKillQQ

nnFn

Fbottom GenKillQQ

iKill iGen

iStmt

Bii

Bi QStmtpwpQ 1,

iStmtBii

Bi QStmtQ 1 iStmt

BbottomQ

36

Detection of evidence

• A is less restrictive than B if is false• Evidence at point i is situation when is less

restrictive than• Two types of evidence:– Explicit• Type I• Type II

– Implicit

BA

iQ

iR

37

Types of evidence

• Explicit– Type I: if at point i in appears negative r in form

(0 relop const) then r forms an explicit evidence of Type I

– Type II: let an atomic predicate in and a negative atomic predicate in If then q and r form an explicit evidence of Type II

• Implicit: negative predicate r in that is not present in an explicit evidence

iR iranyEexplicit ,,

1: constrelopexprq q iQ2: constrelopexprr r

rq exprexpr irqEexplicit ,,iR

1R

38

Location and modification of likely erroneous statement

• Use transitivity and equality to deduce new predicates. New states are and

• Explicit Type I– If = then match r to 0=0. Consider every

assignment statement between i and bottom of the trace as a possible candidate for modification

– Let from be a corresponding predicate to r, and and

– Goal is to make , so – If then problem is solved

*iQ

*iR

kr*kR

rhslhsStmtk : 0 rhslhse 1 krc

00kr 00, 1 krrhslhspwp

01 rhslhsrk

39

Location and modification of likely erroneous statement: Explicit Type I

• Consider e and c as a set of strings of characters• Let and be difference between e and c and

between c and e correspondingly• If appears in rhs than replace it with • If appears in lhs than replace it with only if

is a single variable• If none of the above works than select r as e and

try every q from as c

ed cd

ed cd

ed cd cd

*iQ

40

Location and modification of likely erroneous statement: Explicit Type II

• Consider • Either q or r could be in error• Change the form of r to q at i– Same manner as above

• Change the form of q to r at i– Change original branch predicate from which q

may be derived or some assignment statement– Change of an assignment statement does not

change relop in q

irqEexplicit ,,

41

Location and modification of likely erroneous statement: Implicit

• If there is a loop in the trace, which contributes some constraints on , and missed constraints have similarity with constraints added by the loop then try to derive the possible missing iterations in the loop

• Try to match negative r from to some q from *1Q

*1R

*1R

42

Automated Debugging using Path Based ‑Weakest Preconditions: Conclusions

• Uses contracts• Can handle only one error• Cannot handle loops well

43

Automated Fixing of Programs with Contracts

• Yi Wei, Yu Pei, C.A. Furia, L.S. Silva, S. Buchholz, B. Meyer, A. Zeller

• Assessing Object State• Fault Analysis• Behavioral Models• Generating Candidate Fixes• Linearly Constrained Assertions• Validating and Ranking Fixes

44

Assessing Object State

• Argument-less Boolean Queries– Absolutely describe state– Seldom preconditions– Widely used in Eiffel contracts

• Complex Predicates– Boolean queries are often correlated– Implication expresses correlation– Mine the contracts– Mutate implications

45

Fault Analysis

• Find state invariants: passing and failing states – sets of predicates that hold during all passing and failing runs respectively

• Fault profile contains all predicates that hold in the passing run but not in the failing run

• Find the strongest predicate that implies the negation of violated assertion

46

Behavioral models

• Finite state automaton– States are predicates that hold– Transitions are routines

• Automaton is built based on test runs• Determine a sequence of routine calls that

change the object state appropriately• A snippet for a set of predicates is any

sequence of routines that drive object from a state where none of predicates hold to one where all of them hold

47

Generating Candidate Fixes

• Fix Schemas• snippet is a sequence of routine calls• old_stmt – some statements in the original

program• fail:– – –

pnot

andnotandnot 21 pp

clauseviolated _not

48

Linearly Constrained Assertions

• Determine what is variable and what is constant

• Assign weights:– Arguments in precondition receive lower weights– In assertion weight is inversely proportional to the

number of occurrences– Identifiers that routine can assign receive less

weight

49

Linearly Constrained Assertions: Generating fixes

• Select a value for variable that satisfies the constraint– Look for extremal values

• Plug the value into a fix schema– if not constraint then new_stmt else old_stmt end

50

Validating and Ranking Fixes

• Candidate is valid if it passes all the tests• Two metrics for ranking:– Dynamic: estimates the difference in runtime

behavior between the fix and the original based on state distance

– Static: • OS: 0 for schemas (a) and (b) and number of

statements in old_stmt for (c) and (d)• SN: number of statements in snippet• BF: number of branches to reach old_stmt from the

point of injection of the instantiated fix schema

BFSNOS 5.25

51

Automated Fixing of Programs with Contracts: Conclusions

• Uses contracts, passing and failing tests to deduce and suggest bug fixes

• Successfully proposed fixes for 16 out of 42 found bugs in EiffelBase library

52

Evidence-Based Automated Program Fixing

• Yu Pei, Yi Wei, C.A. Furia, M. Nordio, B. Meyer• Predicates, Expressions, and States• Static Analysis• Dynamic Analysis• Fixing Actions• Fix Candidate Generation• Validation of Candidates

53

Predicates, Expressions

• is a set of all non-constant expressions in routine r

• is a set of boolean predicates:– Boolean expressions: every– Voidness checks: for every– Integer comparisons: for every and

every and in – Complements for every

rE

rP

rEb

rEeVoide ee ~ rEe

0\ eEe r ~ ,,

p rPp

54

States

• State Components– - a triple where v is a value of predicate p for

some test case t which riches l– comp(T) denotes all the triples defined by

the tests in the set

vpl ,,

vpl ,,

Tt

55

Static Analysis

• sub(e) is the set of all sub-expressions of e• Expression proximity • Expression dependence between predicate

and a contract clause c• Control distance is the length of the

shortest directed path from to on the control-flow graph

• Control dependence

2121, esubesubeeeprox

rPp

rPceprox

cpeproxcpedep

|,max

,,

21 , llcdist

1l 2l

jrjcdist

jlcdistjlcdep

and|,max

,1,

56

Dynamic Analysis

• is a score for tests• for i-th failing test and for i-th

passing test; • The evidence provided by each test case:

t it it

1,0

rcj

r PvvFuuvpldyn ||,, ,

57

Combining Static and Dynamic Analysis

• Combined evidence score:

111 ,,,,

3,,

vpldynjlcdepcpedep

vplfixme

58

Fixing Actions

• A component with a high evidence score induces a number of possible actions:– Derived Expressions– Expression Modification– Expression Replacement

vpl ,,

59

Fix Candidate Generation

• Candidates are generated in a way similar to previous method

• Candidate is valid if it passes all the tests

60

Evidence-Based Automated Program Fixing: Conclusions

• Uses contracts and sets of passing and failing tests

• Combines static and dynamic approaches

61

Conclusion

• Automated Bug Fixing is real!• Some bugs are still too difficult to fix or even

localize• All approaches need some kind of oracle;

sometimes contracts are the oracle

62

References

• W. Weimer, S. Forrest, C. Le Goues, T. Nguyen, Automatic Program Repair with Evolutionary Computation

• A. Arcuri and X. Yao, A Novel Co-evolutionary Approach to Automatic Software Bug Fixing

• D. Jeffrey, M. Feng, N. Gupta, R. Gupta, BugFix: A Learning-Based Tool to Assist Developers in Fixing Bugs

• H. He, N. Gupta, Automated Debugging using Path Based ‑Weakest Preconditions

• Yi Wei, Yu Pei, C.A. Furia, L.S. Silva, S. Buchholz, B. Meyer, A. Zeller, Automated Fixing of Programs with Contracts

• Yu Pei, Yi Wei, C.A. Furia, M. Nordio, B. Meyer, Evidence-Based Automated Program Fixing