Software Logic Mutation Testing Presented by Gary Kaminski.
-
Upload
christopher-george -
Category
Documents
-
view
221 -
download
0
Transcript of Software Logic Mutation Testing Presented by Gary Kaminski.
![Page 1: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/1.jpg)
Software Logic Mutation Testing
Presented by
Gary Kaminski
![Page 2: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/2.jpg)
Overview
• Introduction and Motivation
• Logic Faults and Detection
• Case Study Comparison
• Context
• Conclusion
![Page 3: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/3.jpg)
Logic Mutation Testing Process
1) Systematically create program variations (mutants) each containing a single typical logic fault
if (a && b) if (a || b)
2) For each mutant, find an input that distinguishes the original program from the mutant program (killing)
- a && b and a || b evaluate to different truth values
• Focus is on step 1)
![Page 4: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/4.jpg)
Logic Mutation Testing
• Logic mutants are created by applying mutation operators to Boolean predicates
• Boolean predicates may come from:
1) UML Diagrams
2) Security Requirements
3) Looping constructs (while / do-while / for)
4) Preconditions for methods (if - then)
![Page 5: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/5.jpg)
MotivationCurrent mutation systems:
– generate excess logic mutants– do not detect certain classes of logic faults
Goal:
- generate fewer logic mutants
- detect more logic faults
Assumption:
- formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates
![Page 6: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/6.jpg)
Minimal DNF
• Terms separated by OR, literals by AND
ab + a!c vs. a(b + !c)
• Make each term true and other terms false
ab + ac vs. ab + abc
• Impossible to remove a literal or term without changing the predicate
ab vs. abc + ab!c
![Page 7: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/7.jpg)
Excess Logic Mutants - Example
Original: ab + b!c + !bc
1) mutant generated multiple times: ab + _ + !bc
2) equivalent mutants: ac + b!c + !bc
3) mutants guaranteed to be killed by an input that kills some other mutant:
!ab + b!c + !bc and cb + b!c + !bc
4) highly effective mutation operators not used: abc + b!c + !bc
![Page 8: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/8.jpg)
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Expression Negation Fault: !(ab + b!c + !bc)• Expression Stuck At Fault 0: FALSE• Expression Stuck At Fault 1: TRUE• Term Negation Fault: !(ab) + b!c + !bc• Term Omission Fault: ab + _ + !bc• Scalar Variable Replacement TOF: ab + c!c + !bc
• TOF generates same mutant twice: ab + _ + !bc
![Page 9: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/9.jpg)
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Operator Reference Fault+: abb!c + !bc• Operator Reference Fault .: a + b + b!c + !bc• Operator Reference Fault+0: FALSE + !bc• Operator Reference Fault+1: TRUE + !bc• Operator Reference Fault.0: FALSE + b!c + !bc• Operator Reference Fault.1: TRUE + b!c + !bc
![Page 10: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/10.jpg)
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Literal Negation Fault: !ab + b!c + !bc• Literal Stuck At Fault 0: FALSEb + b!c + !bc• Literal Stuck At Fault 1: TRUEb + b!c + !bc• Literal Omission Fault: _b + b!c + !bc• SVR LOF: bb + b!c + !bc
• LOF generates same mutant twice original: abc _bc, a_c, a_c, ab_
![Page 11: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/11.jpg)
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Literal Reference Fault: ac + b!c + !bc • Literal Reference Fault: a!c + b!c + !bc• Literal Insertion Fault: abc + b!c + !bc• Literal Insertion Fault: ab!c + b!c + !bc
• LRF and LIF generate equivalent mutants
![Page 12: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/12.jpg)
Logic Fault Minimal DNF Detection
Green Arrow – Detection may not holdBlack Arrow – Detection always holds
Red Box – Mutation Operator does not existBlue Box – Mutation Operator partially existsBlack Box – Mutation Operator exists
LOF
ORF.
LRF
LNF
TNF
LIF
TOF
LSTF0
ORF.0
ESTF0
ORF+
ORF+0
ENF
LSTF1
ORF.1
ORF+1
ESTF1
SVRTOF
SVRLOF
![Page 13: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/13.jpg)
Logic Mutation Comparison
Original: ab + b!c + !bc• 61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs
• Equivalent LIF: abc + b!c + !bc• LRF: cb + b!c + !bc & ac + b!c + !bc• LOF: a_ + b!c + !bc
• Equivalent LIF: ab + b!c!a + !bc• LRFs: ab + !a!c + !bc & ab + b!a + !bc
• No Equivalent LIFs for term !bc• LOFs: ab + b!c + _c & ab + b!c + !b_
![Page 14: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/14.jpg)
Mutation Efficiency Measures Mutation Efficiency Score = Faults Detected - Mutants GeneratedMutation Efficiency Ratio = Faults Detected / Mutants Generated• FD assumes an input is found to kill the mutant• FD includes equivalent mutants detected automatically
ab + b!c + !bc
Current Mutation Systems:• MES: 60 FD - 61 MG = -1• MER: 60 FD / 61 MG = .98
LIF-LRF-LOF• MES: 73 FD – 10 MG = 63 • MER: 73 FD / 10 MG = 7.30
![Page 15: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/15.jpg)
Case Study
• Analyzed 19 Boolean predicates in an avionics software system
• Number of unique literals range: 5 to 13
• Compared MES and MER for current mutation systems vs. LIF-LRF-LOF
• Examined how MES and MER vary based on number of unique literals
![Page 16: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/16.jpg)
Case Study Results
• LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819)
• Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082)
• Number of literals matters for MES but not for MER• MES = FD – MG and MER = FD / MG
Current LIF-LRF-LOF
MES 0 932MER 1.00 7.22
![Page 17: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/17.jpg)
Mutation Efficiency Score (FD – MG) vs.
Number of Unique Literals
0
500
1000
1500
2000
2500
5 6 7 8 9 10 11 12 13
LIF-LRF-LOF
CurrentMutation
![Page 18: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/18.jpg)
Mutation Efficiency Ratio (FD/MG) vs.
Number of Unique Literals
0
2
4
6
8
10
12
5 6 7 8 9 10 11 12 13
LIF-LRF-LOF
CurrentMutation
![Page 19: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/19.jpg)
Context: Internal Variable Problem
• What input values kill a mutant?
• Mutation may be buried deep in the code
• Must reach predicate and have mutant predicate and original predicate evaluate to different truth values
• Partial solutions using constraints exist
![Page 20: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/20.jpg)
Context: Minimal DNF in Practice
1) 95% of 20,256 Boolean predicates in avionics software were in minimal DNF*
2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates*
*Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages 14-21.
![Page 21: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/21.jpg)
Context: Industry Audience
• LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals
• Exhaustive coverage for < 5 unique literals
• ab + b!c + bc (8 tests vs. 10 mutants)
• Avionics software*
*Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal, 9(5): 193-200, September 1994.
![Page 22: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/22.jpg)
Related Work1) J.J. Chilenski. An Investigation of Three Forms of the Modified Condition
Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April 2001.
• Non-Minimal DNF based logic coverage criteria
2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, 1999. (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages 606-613.
• Minimal DNF based logic coverage criteria and Double Faults
3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): 411-424, October 1999.
• Seminal work in logic fault hierarchy
![Page 23: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/23.jpg)
Conclusion
• Current logic mutation testing is inefficient• Excess mutants generated and faults are missed • Fault hierarchy approach based on minimal
DNF: detect more faults with less mutants• Applications in avionics software
![Page 24: Software Logic Mutation Testing Presented by Gary Kaminski.](https://reader035.fdocuments.in/reader035/viewer/2022062517/56649e965503460f94b9a19c/html5/thumbnails/24.jpg)
Other Research and Publications
Logic Coverage Criteria:• Reduce test set size and maintain fault detection • Use feasibility of smaller, less expensive component criteria
Internal variable problem• CoJava and constraint programming
Publications:G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives of Logic Testing for Software. Submitted to Journal of Software Testing, Verification, and Reliability, January 2008. Online.