Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage:...
Transcript of Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage:...
![Page 1: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/1.jpg)
CSE P 590Beyond Coverage: Modern Testing and Debugging
Spring 2019
Course introduction
April 02, 2019
![Page 2: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/2.jpg)
The CSE P 590 teamInstructor
● René Just● Office: CSE2 338● Office hours: After class and by appointment● [email protected]
Teaching assistant● Martin Kellogg● Office: CSE2 253● Office hours: Before/after class and by appointment● [email protected]
![Page 3: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/3.jpg)
Today
● Logistics● Brief introduction● Your background and expectations● Course overview● Static vs. dynamic program analysis● Class projects
![Page 4: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/4.jpg)
Logistics
● CSE2 G10, Tue, 6:30pm – 9:20pm.● Lectures, discussions, and lab session.
● Course material, schedule, etc. on website: https://homes.cs.washington.edu/~rjust/courses/2019Spring/CSEP590
● Submission of assignments via Canvas: https://canvas.uw.edu
● Discussions on Piazza:piazza.com/washington/spring2019/csep590
![Page 5: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/5.jpg)
My background
![Page 6: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/6.jpg)
My background
![Page 7: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/7.jpg)
My backgroundMy research areas● Software testing and verification● Software debugging● Software security
![Page 8: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/8.jpg)
My backgroundMy research areas● Software testing and verification● Software debugging● Software security
● Empirical software engineering● Data science / Applied ML
![Page 9: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/9.jpg)
My backgroundMy research areas● Software testing and verification● Software debugging● Software security
● Empirical software engineering● Data science / Applied ML
![Page 10: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/10.jpg)
The Role of Software Engineering in Research
Experimental infrastructure is software, too!
Example (automated debugging)● 150 configurations, 1000+ benchmarks● 1-85 hours per execution● 200,000+ CPU hours (~23 CPU years)
1 0.34 0.81
2 0.52 0.32
3 0.21 0.53
4 0.81 0.22
... ... ...
Infrastructure
Design space exploration
![Page 11: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/11.jpg)
Your background and expectations
Introduction and a very brief survey
● Role: What is your current role?● Background: What is your SE background?● Top-2 expectations: What do you expect from this course?
![Page 12: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/12.jpg)
Today
● Logistics● Brief introduction● Your background and expectations● Course overview● Static vs. dynamic program analysis● Class projects
![Page 13: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/13.jpg)
Course overview: the big picture● 04/02: Course introduction
● 04/09: Best practices and version control
● 04/16: Coverage-based testing
● 04/23: Automated test generation
● 04/30: Mutation-based testing
● 05/07: Mutation-based testing
● 05/14: Formal methods/constraint-based testing
● 05/21: Fault localization
● 05/28: Defect prediction
● 06/04: Type checking and pluggable types
![Page 14: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/14.jpg)
Course overview: the big picture● 04/02: Course introduction
● 04/09: Best practices and version control
● 04/16: Coverage-based testing
● 04/23: Automated test generation
● 04/30: Mutation-based testing
● 05/07: Mutation-based testing
● 05/14: Formal methods/constraint-based testing
● 05/21: Fault localization
● 05/28: Defect prediction
● 06/04: Type checking and pluggable types
In-class exercise
In-class exercise
In-class exercise
Project presentation
In-class exercise
In-class exercise
In-class exercise
Project presentation
Questions?
![Page 15: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/15.jpg)
Course overview: grading
● 30% Class project● 60% In-class exercises (6 sessions)● 10% Participation
Questions?
![Page 16: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/16.jpg)
Course overview: expectations
● Conduct a quarter-long group project.● Some programming (and OO) experience.● Read a few research papers.
● Have fun!
![Page 17: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/17.jpg)
Today
● Logistics● Brief introduction● Your background and expectations● Course overview● Static vs. dynamic program analysis● Class projects
![Page 18: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/18.jpg)
What is Software Engineering?
![Page 19: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/19.jpg)
What is Software Engineering?
● Developing in an IDEand software ecosystem?
● Coding and debugging?
● Deploying and runninga software system?
● Empirical evaluations?
● Modeling and designing?
![Page 20: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/20.jpg)
What is Software Engineering?
● Developing in an IDEand software ecosystem?
● Coding and debugging?
● Deploying and runninga software system?
● Empirical evaluations?
● Modeling and designing?
All of the above -- much more than just writing code!
![Page 21: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/21.jpg)
What is Software Engineering?
More than just writing codeThe complete process of specifying, designing, developing, analyzing, deploying, and maintaining a software system.
● Common Software Engineering tasks include:○ Requirements engineering○ Specification writing and documentation○ Software architecture and design○ Programming○ Software testing and debugging○ Refactoring
![Page 22: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/22.jpg)
What is Software Engineering?
More than just writing codeThe complete process of specifying, designing, developing, analyzing, deploying, and maintaining a software system.
● Common Software Engineering tasks include:○ Requirements engineering○ Specification writing and documentation○ Software architecture and design○ Programming Just one out of many important tasks!○ Software testing and debugging○ Refactoring
![Page 23: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/23.jpg)
What is Software Engineering?
More than just writing codeThe complete process of specifying, designing, developing, analyzing, deploying, and maintaining a software system.
● Common Software Engineering tasks include:○ Requirements engineering○ Specification writing and documentation○ Software architecture and design○ Programming○ Software testing and debugging○ Refactoring
Program analysis is a crucial task in Software Engineering!
![Page 24: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/24.jpg)
What is program analysis?
![Page 25: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/25.jpg)
What is program analysis?
● (Automatically) analyze the behavior of a program○ optimize the program or○ check program’s behavior (against its specification)
● Concerned with properties such as○ Correctness○ Safety○ Liveness○ Performance
● Can be static or dynamic or a combination of both
What’s the difference betweena static analysis and a dynamic analysis?
![Page 26: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/26.jpg)
Static vs. dynamic analysis
Static analysis● Reason about a program without executing it● Build an abstraction of run-time states
(and prove a property of the program)
Dynamic analysis● Reason about a program by executing it
(with some inputs)● Observe actual behavior
![Page 27: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/27.jpg)
Why do we need program analysis?
![Page 28: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/28.jpg)
Why do we need program analysis?
● ~15 million lines of code
Let’s say 50 lines per page (0.05 mm)
![Page 29: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/29.jpg)
Why do we need program analysis?
● ~15 million lines of code
Let’s say 50 lines per page (0.05 mm)● 300000 pages● 15 m (49 ft)
![Page 30: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/30.jpg)
Why do we need program analysis?
![Page 31: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/31.jpg)
Why do we need program analysis?
![Page 32: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/32.jpg)
Why do we need program analysis?
![Page 33: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/33.jpg)
● Increase confidence in program correctness● Understand the program’s behavior● Prove properties about the program
Why do we need program analysis?
![Page 34: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/34.jpg)
Code review/inspection
Different types of reviews● Code/design review● Informal walkthrough● Formal inspection
![Page 35: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/35.jpg)
Code review/inspection
Different types of reviews● Code/design review● Informal walkthrough● Formal inspection
Let’s do an informal code review.Anything that could be improved in this code?
double foo(double[] d) { int n = d.length; double s = 0; int i = 0; while (i<n) s = s + d[i]; i = i + 1; double a = s / n; return a;}
![Page 36: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/36.jpg)
Code review/inspection
Different types of reviews● Code/design review● Informal walkthrough● Formal inspection
Anything wrong with that code?
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
![Page 37: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/37.jpg)
Code review/inspection
Different types of reviews● Code/design review● Informal walkthrough● Formal inspection
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
![Page 38: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/38.jpg)
static OSStatusSSLVerifySignedServerKeyExchange(...) {
OSStatus err;...if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
goto fail;if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
goto fail;if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)goto fail;
err = sslRawVerify(ctx, ctx->peerPubKey, dataToSign, dataToSignLen, signature, signatureLen);if(err) {
sslErrorLog("SSLDecodeSignedServerKeyExchange: sslRawVerify returned %d\n", (int)err);goto fail;
}fail:
SSLFreeBuffer(&signedHashes);SSLFreeBuffer(&hashCtx);return err;
}
Anything wrong with that code?
![Page 39: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/39.jpg)
static OSStatusSSLVerifySignedServerKeyExchange(...) {
OSStatus err;...if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
goto fail;if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
goto fail;if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)goto fail;
err = sslRawVerify(ctx, ctx->peerPubKey, dataToSign, dataToSignLen, signature, signatureLen);if(err) {
sslErrorLog("SSLDecodeSignedServerKeyExchange: sslRawVerify returned %d\n", (int)err);goto fail;
}fail:
SSLFreeBuffer(&signedHashes);SSLFreeBuffer(&hashCtx);return err;
}
Anything wrong with that code?
Apple’s “goto fail” bug:a security vulnerability for 2 years!
![Page 40: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/40.jpg)
Code review/inspection
Pros● Can be applied at any step in the development process● Improves confidence and communication
Cons● Time-consuming● Mostly informal● Not replicable
![Page 41: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/41.jpg)
Static vs. dynamic analysis
Static analysis● Reason about a program without executing it● Build an abstraction of run-time states
(and prove a property of the program)
Dynamic analysis● Reason about a program by executing it
(with some inputs)● Observe actual behavior
![Page 42: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/42.jpg)
Static analysis: examples
● Type checking of a compiler● Rule/pattern-based analysis (PMD, Findbugs, etc.).
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) { sum = sum + nums[i]; i = i + 1; } double avg = sum / n;
return avg;}
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
![Page 43: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/43.jpg)
Static analysis: examples
● Type checking of a compiler● Rule/pattern-based analysis (PMD, Findbugs, etc.).
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) { sum = sum + nums[i]; i = i + 1; } double avg = sum / n;
return avg;}
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
![Page 44: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/44.jpg)
Static analysis: examples
● Control-flow analysis● Data-flow analysis
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
What is the control flow graph (CFG) for this avg function?
![Page 45: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/45.jpg)
Static analysis: examples
● Control-flow analysis● Data-flow analysis
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
Entrypoint
n = nums.length
sum = 0
i = 0
i < n
Normalexit
true
false
sum = sum + nums[i]
i = i + 1
avg = sum / n
return avg
![Page 46: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/46.jpg)
Static analysis: examples
● Control-flow analysis● Data-flow analysis
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
Entrypoint
n = nums.length
sum = 0
i = 0
i < n
Normalexit
true
false
sum = sum + nums[i]
i = i + 1
avg = sum / n
return avg
Can we conclude that this is an infinite loop?
Why or why not?
![Page 47: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/47.jpg)
Dynamic analysis: examples
● Software testing● Software monitoring or profiling
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
A test for the avg function:
@Test public void testAvg() { double nums =
new double[]{1.0, 2.0, 3.0}); double actual = Math.avg(nums); double expected = 2.0; assertEquals(expected,actual,EPS); }
![Page 48: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/48.jpg)
Dynamic analysis: examples
● Software testing● Software monitoring or profiling
double avg(double[] nums) { int n = nums.length; double sum = 0;
int i = 0; while (i<n) sum = sum + nums[i]; i = i + 1; double avg = sum / n;
return avg;}
A test for the avg function:
@Test public void testAvg() { double nums =
new double[]{1.0, 2.0, 3.0}); double actual = Math.avg(nums); double expected = 2.0; assertEquals(expected,actual,EPS); }
What happens if we execute this test? What can we conclude?
![Page 49: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/49.jpg)
Static analysis vs. dynamic analysis
● Can these analyses pinpoint a problem in the code?
● Does a reported error always indicate thatsomething is wrong with the code (no false positives)?
● Does no reported error indicate that there isnothing wrong with the code (no false negatives)?
![Page 50: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/50.jpg)
Static analysis vs. dynamic analysis
● Can these analyses pinpoint a problem in the code?
● Does a reported error always indicate thatsomething is wrong with the code (no false positives)?
● Does no reported error indicate that there isnothing wrong with the code (no false negatives)?
Should we use static or dynamic analysis?
![Page 51: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/51.jpg)
Today
● Logistics● Brief introduction● Your background and expectations● Course overview● Static vs. dynamic program analysis● Class projects
![Page 52: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/52.jpg)
Logistics● 3-5 students per project group.● Group selection until 04/09 (further discussion on Piazza).● 2 informal (in-class) presentations (~10min + Q&A).
High-level topics (suggestions)1. Code coverage
a. A new code coverage tool for Javab. API for existing code coverage tools (Cobertura, JaCoCo)
2. Mutation testing (Major)a. Compiler-integrated mutator (compiler plugin)b. Mutation analyzer (standalone or IDE plugin)c. Visualization for mutation testing results
3. Fault database/benchmark (Defects4J)a. Build system inferenceb. Commit minimization
4. Static analysis: pluggable type checker
Class projects: overview
![Page 53: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/53.jpg)
Project: New code coverage tool for Java
Goal:Design and implement a new code coverage tool for Java programs (source-code, AST, or byte-code level).Support queries such as:
● Is line x covered in method y?● How often is it covered?● How many lines are covered overall?● How many lines exist in method y?● ...
![Page 54: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/54.jpg)
Project: API for existing code coverage tools
Goal:Design a Java API that defines a common abstraction for code coverage tools, and support existing tools (e.g., Cobertura, JaCoCo). Support queries such as:
● Is line x covered in method y?● How often is it covered?● How many lines are covered overall?● How many lines exist in method y?● ...
![Page 55: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/55.jpg)
Projects: Mutation testing
Goal (project 1):Develop a program mutator (e.g., Java compiler plugin).
Goal (project 2):Develop a mutation analyzer (standalone, IDE plugin) for an existing program mutator.
Goal (project 3):Develop a visualization for the output of an existing mutation analyzer.
![Page 56: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/56.jpg)
Project: Build system inference
Goal:Given a project's build file (e.g., Apache Ant's build.xml),automatically determine (infer) relevant properties.
<project name="Example" default="compile" basedir="."> <!-- Compile the project --> <target name="compile" depends="init" description="Compile"> <javac includeantruntime="true" srcdir="src" destdir="bin" debug="yes"> <classpath location="lib/junit.jar"/> </javac> </target>
● Where are the sources?● Where are the tests?● What’s the classpath?● ...
![Page 57: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/57.jpg)
Goal:Given a bug-fixing commit and a test suite that failed before and passes after that commit, automatically minimize the changes in that commit such that only changes relevant to the bug fix remain.
Project: Commit minimization
![Page 58: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/58.jpg)
Logistics● 3-5 students per project group.● Group selection until 04/09 (further discussion on Piazza).● 2 informal (in-class) presentations (~10min + Q&A).
High-level topics (suggestions)1. Code coverage
a. A new code coverage tool for Javab. API for existing code coverage tools (Cobertura, JaCoCo)
2. Mutation testing (Major)a. Compiler-integrated mutator (compiler plugin)b. Mutation analyzer (standalone or IDE plugin)c. Visualization for mutation testing results
3. Fault database/benchmark (Defects4J)a. Build system inferenceb. Commit minimization
4. Static analysis: pluggable type checker
Class projects: overview
![Page 59: Spring 2019 Beyond Coverage: Modern Testing and Debugging ...rjust/courses/... · Beyond Coverage: Modern Testing and Debugging Spring 2019 Course introduction April 02, 2019. The](https://reader034.fdocuments.in/reader034/viewer/2022042403/5f14fa4e1b7f5e36e163c009/html5/thumbnails/59.jpg)
Class projects: brainstorming session
Group by high-level interest● Code coverage● Mutation testing● Applied machine learning● Static analysis● Fault database/benchmarks
Goals● What high-level project ideas should we add to the list?● Pitch a brief project proposal for new ideas.