Spring 2020 Software Engineering Delta Debugging April 27 ...rjust/courses/... · CSE 403 Software...
36
CSE 403 Software Engineering Spring 2020 Delta Debugging April 27, 2020
Transcript of Spring 2020 Software Engineering Delta Debugging April 27 ...rjust/courses/... · CSE 403 Software...
CSE 403Software Engineering
Spring 2020
Delta Debugging
April 27, 2020
Logistics
In-class exercise 2: Wednesday 04/29● Set up your machine: make and cc (GNU or LLVM).
● ssh into Attu is always an option.
● (Optional) pair up by Tuesday night (group In-class-2 on Canvas).
Recap: In-class exercise and git bisect
Git bisect: mostly binary search● What’s the best, worst, and average case complexity of git bisect?
Undoing a commit vs. rewriting history● Which git command can you use to undo a defect-inducing commit?
Briefly explain what problem may generally occur when undoing a commit and what best practices mitigate this problem.
Delta Debugging● Motivating example● The core algorithm● Live demo● A little quiz
Today
This is a crashing test case<td align=left valign=top> <SELECT NAME="op sys" MULTIPLE SIZE=7> <OPTION VALUE="All">All <OPTION VALUE="Windows 3.1">Windows 3.1 <OPTION VALUE="Windows 95">Windows 95 <OPTION VALUE="Windows 98">Windows 98 <OPTION VALUE="Windows ME">Windows ME <OPTION VALUE="Windows 2000">Windows 2000 <OPTION VALUE="Windows NT">Windows NT <OPTION VALUE="Mac System 7">Mac System 7 <OPTION VALUE="Mac System 7.5">Mac System 7.5 <OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1 <OPTION VALUE="Mac System 8.0">Mac System 8.0 <OPTION VALUE="Mac System 8.5">Mac System 8.5 <OPTION VALUE="Mac System 8.6">Mac System 8.6 <OPTION VALUE="Mac System 9.x">Mac System 9.x <OPTION VALUE="MacOS X">MacOS X <OPTION VALUE="Linux">Linux <OPTION VALUE="BSDI">BSDI <OPTION VALUE="FreeBSD">FreeBSD <OPTION VALUE="NetBSD">NetBSD <OPTION VALUE="OpenBSD">OpenBSD <OPTION VALUE="AIX">AIX <OPTION VALUE="BeOS">BeOS <OPTION VALUE="HP-UX">HP-UX <OPTION VALUE="IRIX">IRIX <OPTION VALUE="Neutrino">Neutrino <OPTION VALUE="OpenVMS">OpenVMS <OPTION VALUE="OS/2">OS/2 <OPTION VALUE="OSF/1">OSF/1 <OPTION VALUE="Solaris">Solaris <OPTION VALUE="SunOS">SunOS <OPTION VALUE="other">other</SELECT></td> <td align=left valign=top> <SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT> </td> <td align=left valign=top> <SELECT NAME="bug severity" MULTIPLE SIZE=7> <OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT> </tr> </table>
● Crashed Mozilla
● How would you debugthe problem?
This is a crashing test case<td align=left valign=top> <SELECT NAME="op sys" MULTIPLE SIZE=7> <OPTION VALUE="All">All <OPTION VALUE="Windows 3.1">Windows 3.1 <OPTION VALUE="Windows 95">Windows 95 <OPTION VALUE="Windows 98">Windows 98 <OPTION VALUE="Windows ME">Windows ME <OPTION VALUE="Windows 2000">Windows 2000 <OPTION VALUE="Windows NT">Windows NT <OPTION VALUE="Mac System 7">Mac System 7 <OPTION VALUE="Mac System 7.5">Mac System 7.5 <OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1 <OPTION VALUE="Mac System 8.0">Mac System 8.0 <OPTION VALUE="Mac System 8.5">Mac System 8.5 <OPTION VALUE="Mac System 8.6">Mac System 8.6 <OPTION VALUE="Mac System 9.x">Mac System 9.x <OPTION VALUE="MacOS X">MacOS X <OPTION VALUE="Linux">Linux <OPTION VALUE="BSDI">BSDI <OPTION VALUE="FreeBSD">FreeBSD <OPTION VALUE="NetBSD">NetBSD <OPTION VALUE="OpenBSD">OpenBSD <OPTION VALUE="AIX">AIX <OPTION VALUE="BeOS">BeOS <OPTION VALUE="HP-UX">HP-UX <OPTION VALUE="IRIX">IRIX <OPTION VALUE="Neutrino">Neutrino <OPTION VALUE="OpenVMS">OpenVMS <OPTION VALUE="OS/2">OS/2 <OPTION VALUE="OSF/1">OSF/1 <OPTION VALUE="Solaris">Solaris <OPTION VALUE="SunOS">SunOS <OPTION VALUE="other">other</SELECT></td> <td align=left valign=top> <SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT> </td> <td align=left valign=top> <SELECT NAME="bug severity" MULTIPLE SIZE=7> <OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT> </tr> </table>
● Crashed Mozilla
● How would you debugthe problem?
● A minimal test case is:<SELECT>
● Can we automate theprocess of minimizingtest cases?
● What’s the naiveapproach for anoptimal solution?
Minimizing test cases
Test case Test case Test case
Minimizing test cases
Test case Test case
Failing Passing
Test case
Passing
Minimizing test cases
Test case Test case
Failing Passing
Test case
Passing
Goal: Minimize the failing test case
The happy path: binary search
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Failing test with 16 lines.The minimal test has 2 lines.
F
The happy path: binary search
F P
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
The happy path: binary search
F PP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
The happy path: binary search
P PPF
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
The happy path: binary search
P PPF
P P
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
The happy path: binary search
P PPF
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
Successfully minimized the failing test to 2 lines
The not so happy path...
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
F
Suppose the failure pattern is more complex.
The not so happy path...
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
Binary search does not give optimal results.
P P
Delta debugging: binary search + X
The DD algorithm
Four basic steps:1. Test each subset2. Test each complement3. Increase granularity4. Reduce
Delta Debugging: mostly binary search
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
F
Delta Debugging: mostly binary search
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PP
Delta Debugging: granularity
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PP P P
Delta Debugging: complements
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PF
Delta Debugging: reduce
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PP P PP
Delta Debugging: reduce
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PF P
Delta Debugging: 1-minimality
F
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F
PPF FP P P
Failure inputs must be deterministic and monotone.
Delta Debugging: concrete example
Program and initial test case● Program P crashes whenever the input contains 1 7 8● Initial crashing test input is: 1 2 3 4 5 6 7 8
Recall the basic approach:1. Test each subset2. Test each complement3. Increase granularity4. Reduce
Delta Debugging: concrete example
Delta Debugging: concrete example
Delta Debugging: concrete example
Delta Debugging: concrete example
Delta Debugging: concrete example
Delta Debugging: concrete example
Delta debugging: live example
A little quiz
Program and initial test case● Program P takes as input an Array of integers a.● P crashes whenever a contains 42.● Initial crashing test case is: 2424
Complete the following table
Iteration n input △1, …, △n
▽1, …, ▽n
1 2 2424
A little quiz
Program and initial test case● Program P takes as input an Array of integers a.● P crashes whenever a contains 42.● Initial crashing test case is: 2424
Complete the following table
Iteration n input △1, …, △n
▽1, …, ▽n
1 2 2424 24, (24)
2 4 2424 2, 4, (2), (4),424, 224, 244, 242
3 3 424 (4), (2), (4),(24), 44, 42
4 2 42 (4),( 2)
