Empirical Study on Synthesis Engines for Semantics-based...

14
Empirical Study on Synthesis Engines for Semantics-based Program Repair Xuan Bach D. Le 1 , David Lo 1 , Claire Le Goues 2 1 School of Information Systems, Singapore Management University 2 School of Computer Science, Carnegie Mellon University {dxb.le.2013,davidlo}@smu.edu.sg [email protected] 1

Transcript of Empirical Study on Synthesis Engines for Semantics-based...

Page 1: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Empirical Study on Synthesis Engines for Semantics-based

Program Repair

Xuan Bach D. Le1, David Lo1, Claire Le Goues2 1School of Information Systems,

Singapore Management University 2School of Computer Science,

Carnegie Mellon University {dxb.le.2013,davidlo}@smu.edu.sg

[email protected] 1

Page 2: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

SE@CMU is recruiting graduate students!

•  Send us your awesome undergrads! •  Tell them to check the box next to “PhD in

Software Engineering”, so we’re certain to see them!

•  Early deadline: December 1 •  Final, we-mean-it deadline: December 15.

2

Page 3: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Automatic patch generation seeks to improve software quality. •  Bugs in software incur tremendous

maintenance cost.

•  Developers presently debug and fix bugs manually.

•  Automated program repair:

In 2006, everyday, almost 300 bugs appear in Mozilla […] far too much for programmers to handle

3

APR = Fault Localization + Repair Strategies

Page 4: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Semantics-based repair extracts value-based specifications using tests + symbolic execution, constructs patches using synthesis.

Program Synthesis

Specs Pases all

tests?

Yes

No

Mechtaev et al.. Angelix: scalable multiline program patch synthesis via symbolic analysis. In ICSE '16.

Page 5: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Our contributions:

•  Pluggable framework to assess many types of syntax-guided program synthesis in the core of Angelix.

•  Evaluation of different synthesis techniques for semantic program repair. – Key finding: effectiveness of different

synthesis engines varies!

5

Page 6: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Syntax Guided Program Synthesis (SyGuS)

: use a restricted grammar to describe syntactic space of possible solutions

•  Use different search techniques to search for solutions conforming to provided grammar

•  We evaluate: Enumerative, Stochastic, Symbolic, and CVC4

Page 7: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Example of Buggy Program

bool min(int x, int y){ bool cond = x < y; if(cond){

return true; }else{

return false; } }

Test # Value of x

Value of y

Expected output

1 1 2 true 2 2 2 true 3 5 2 false

FAILED

7

Page 8: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Selective Symbolic Execution

bool min(int x, int y){ bool cond = α; if(cond){

return true; }else{

return false; } }

Replace buggy expression by

symbolic variable

ü  Switch to symbolic execution when necessary, collect path conditions.

ü  Infer specs for each test

x y expected

2 2 true

PC1: α & x = 2 & y = 2 & expected output[true] = actual output[true]

PC2: (not α) & x = 2 & y = 2 & expected output[true] = actual output[false]

8

Page 9: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Extract Value-based Specifications

PC1: α & x = 2 & y = 2 & expected output[true] = actual output[true]

PC2: (not α) & x = 2 & y = 2 & expected output[true] = actual output[false]

Satisfiable

Unsatisfiable

Model: x = 2 & y = 2 & α = true

Pre-condition: x = 2 & y = 2 Post-condition: α = true

Synthesize α over x and y, permitting a restricted set of components, satisfying spec

9

Page 10: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Hard constraints on functionality + soft constraints on form + PartialMax SMT =

minimal repair 10

(mathy details elided for brevity.)

Public domain, CC0,

Page 11: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Our framework converts specs inferred by Angelix to generic SyGuS format.

Value-based Specs

SyGuS Engine 1

SyGuS format Patch SyGuS

Engine 2

Optimizations for generating minimal patch

11

Page 12: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Experiments

•  188 programs from IntroClass benchmark – Use black-box tests for repair, white-box tests

for testing quality of generated patches. – Only report the patches that generalize to the

held-out tests! •  Synthesis techniques that help generate

more correct patches are better •  Evaluate on: Enumerative, Stochastic,

Symbolic, CVC4, and Angelix’s synthesis engine

12

Page 13: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Synthesis techniques vary in the bugs they can correctly address.

13

Page 14: Empirical Study on Synthesis Engines for Semantics-based ...clegoues/docs/slides/presentation-SyGuS_repair.pdf · Empirical Study on Synthesis Engines for Semantics-based Program

Summary

•  We evaluate the effectiveness of different synthesis techniques in the context of program repair, finding: – Performance of synthesis techniques varies – Forging results of synthesis techniques

increases effectiveness of program repair, e.g., fix more bugs

•  We plan to develop more effective synthesis techniques for repair.

14

Code: https://github.com/xuanbachle/syntax-guided-synthesis-repair