Coverage and Mutation testing tools
So#ware Analysis and Tes1ngCu Nguyen Duy (cunduy at :k dot eu)
Alessandro Marche?o (marche?o at :k dot eu)
Paolo Tonella (tonella at :k dot eu)
Mariano Ceccato (ceccato at :k dot eu)
Academic Year 2009-‐2010
Coverage testing (recall)
Coverage measures describe the degree to which a program has been testedtell the effectiveness in terms of coverage of the test
set, help improve software qualityinform quantitatively the project manager about the
progress of testingMany type of coverage measures
statementspathsmethods, classesrequirement specifications, etc. 2
Coverage tool features
Compile time: automatic instrument code for coverage recording
Runtime: structure-based coverage recording lines, blocks, conditions, methods and classes
Reportingquantitative coverage measure
percentage of code executedmost executed v.s. never executed code
visual navigationquickly navigate to code that is not executed to improve
the test set 3
Coverage tools
4source code in not-widely-used languages on a variety ofplatforms [30].
3.2. Instrumentation overhead
Coverage-testing tools capture coverage information by moni-toring program execution. Execution is monitored by insertingprobes into the program before or during its execution.A probe is typically a few lines of code that, when executed,generate a record or event that indicates that programexecution has passed through the point where the probe islocated. There are two kinds of overhead associated withinstrumenting a program with probes: the off-line overheadof inserting probes into the program, and the run time over-head of executing the probes to record the execution trace.
3.2.1. Off-line program analysis and instrumentationoverhead
Source code instrumentation, used by most of the toolsincluding BullseyeCoverage, Parasoft Insure!!, Intel CodeCoverage Tool, Semantic Designs and TestWork, requiresrecompilation, but provides more direct results and is moreadaptable to a wide variety of processors and platforms. Itcannot be used when the source code is not available, as isoften the case for third party code. C/C!! tools such asDynamic Memory Systems’ Dynamics, use runtime instru-mentation, which makes them feasible in a productionenvironment. They may be more efficient in terms of compi-lation time, but less portable. The Java coverage tool Koalog
Code Coverage does not require instrumentation, and there-fore no recompilation is needed [25]. It operates with the pro-duction binaries using the Java Debug Interface, which is partof the Java Platform Debugger Architecture (JPDA). KoalogCode Coverage is platform independent, but requires aJPDA compliant Java Virtual Machine (JVM). Agitar’s Agita-tor runs the code in a modified JVM, also using a dynamicinstrumentation approach. eXVantage uses source code instru-mentation for C/C!! and bytecode instrumentation for Java.As compared to the other 16 tools, it has the highest off-lineinstrumentation overhead because it analyzes the program insuch a way that it can select the least number of probes tobe inserted into the target program.
3.2.2. Run-time instrumentation overheadCompanies that provide tools for system software or embeddedsoftware tend to focus more on reducing run-time overhead, sothat their tools can be usable in real-time environments, e.g.CodeTEST [18]. TCAT claims that its TCAT C/C!!Version 3.2 maintains its overhead for execution size ratio at1.1–1.8 and execution speed ratio at 1.1–1.5 [29]; SemanticDesigns claims 1.1–1.3, varying according to language andcompiler, among the best in our survey. Clover claims thattheir execution speed overhead is highly variable, dependingon the nature of the application under test, and the nature ofthe tests. Typical execution speed ratio is 1.2–1.5. eXVantagehas different versions for different platforms, but claims aratio of 1.01 for versions optimized real-time, in some environ-ments, based on initial trials (Table 2) [24].
3.3. Additional features
Coverage testing tools can be used to assist in debugging, andsome of the coverage tools provide debugging assistance, suchas Agitar, Dynamic, JCover, Jtest and Semantic Designs. Eachuses a different solution. For example, Agitar provides a snap-shot and stack trace to help developers to track the cause ofbugs. JCover has the ability to do coverage differencing andcomparison to expose the erroneous code. Semantic Designsprovides slicing and dicing operations on test coverage datavia the GUI to allow code executed/not executed by arbitrarycombinations of test runs to be easily isolated visually.eXVantage uses a dynamic execution slicing approach. Itcreates an execution slice for each test case and reads resultsfrom a testing oracle to generate a bug localization reportautomatically whenever a failed test is detected [8].Coverage testing tools can also be used for program profil-
ing to identify heavily executed parts of programs. Profilingdata can be used in compiler optimization, program refactor-ing, performance-related debugging, etc. Many tools, includ-ing eXVantage, CodeTEST, Dynamic Code Coverage,JCover, PurifyPlus and Semantic Designs, support this feature.
TABLE 1: coverage tools and the languages to which they apply(alphabetical by tool name)
Tool name C!!/C Java Other
Agitar [14] X
Bullseye [15] X
Clover [16] X .net
Cobertura [17] X
CodeTEST [18] X
Dynamic [19] X
EMMA [20] X
eXVantage [21] X X
Gcov [22] X
Intel [23] X FORTRAN
JCover [24] X
Koalog [25] X
Parasoft (C!!test) [26] X
Parasoft (Jtest) [26] X
PurifyPlus [27] X X Basic, .net
Semantic Designs (SD)
[28]
X X C#, PHP,
COBOL,
PARLANSE
TCAT [29] X X
Page 4 of 9 YANG et al.
THE COMPUTER JOURNAL, 2007
source: doi:10.1093/comjnl/bxm021
Clover
Is a commercial coverage toolIntegration with a wide range of development
environment: Ant, Eclipse, Maven, IntelliJInstallation
Eclipse update: http://update.atlassian.com/eclipse/clover/
Get an evaluation license
5
Enable Clover to an Eclipse project
Right click on the project
Select Clover > Enable on this project
7
Run Unit test with Clover
In the Coverage Explorer view
Right click on the project
Select Run With Clover As > JUnit Test
9
Clover coverage report
10
report menu
projectinfo
overall coverage
testresults
tree mapcoverage by package
Exercise 1
Open project: Exercise-4-CurrentAccountEnable Clover on this projectExecute Unit TestGenerate coverage reportReport the test coverage of each class,
identify which lines of code have not executed
Task: improve the test cases to reach 100% coverage
11
Introduction
14
program
Test CaseTest CaseTest
CaseTest CaseTest
Case
How good these test cases are?- fault detection - coverage
Introduction
14
program
Test CaseTest CaseTest
CaseTest CaseTest
Case
How good these test cases are?- fault detection - coverage
what if no fault foundwhat if 100% covered
Introduction
14
program
Test CaseTest CaseTest
CaseTest CaseTest
Case
How good these test cases are?- fault detection - coverage
what if no fault foundwhat if 100% covered
- mutant core with mutation testing
Mutation testing terms
Mutant: a copy of the original program with a small change (seeded fault)
Mutation operator: applied to make change (automatically) to the original program
Mutant killed: if its behaviors/outputs differ from those of the original program
Mutant score: of a test case = number of killed mutants / total number of mutants
15
Mutation operators
Method-level operators, example:Arithmetic operator, e.g. “-” -> “+”Relational operator, e.g. “ > ” -> “ < ”Conditional operator, e.g. “ && ” -> “ || ”
Class-level operators, example: EncapsulationInheritancePolymorphism
16http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf
http://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf
Mutant creation
17
program
mutant 1
mutant 2
mutant N
Mutation operator
package math;
public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
Mutant creation
17
program
mutant 1
mutant 2
mutant N
Mutation operator
package math;
public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
package math;
public class Add {
public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}
Example
19
public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}
public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}
public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}
Example
19
public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}
public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}
public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}
public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}
TC1:Add o = new Add();print(o.sum(1,2));print(o.sum(1.0,2.0));
MutantScore(TC1) = ?
MuClipse
An Eclipse plugin for MuJava (http://cs.gmu.edu/~offutt/mujava/)
Main features:GGenerate mutantsRun testsReport mutant score
20
Exercise 1
Install MuClipsehttp://selab.fbk.eu/swat/projects/muclipse.tgz
Add java library to Exercise-3-MathOperatorshttp://selab.fbk.eu/swat/projects/mu-lib.tgz
Report the mutant score of the test case: TestCase1_add_Junit3_a of the project use only traditional mutation operators
Improve the mutant score of the test case23
Top Related