Craft-Conf - Improve your Tests with Mutation Testing
-
Upload
nicolas-frankel -
Category
Software
-
view
274 -
download
2
Transcript of Craft-Conf - Improve your Tests with Mutation Testing
![Page 1: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/1.jpg)
MUTATION TESTINGTO THE RESCUE OF YOUR TESTSNICOLAS FRÄNKEL
![Page 2: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/2.jpg)
ME, MYSELF AND I
@nicolas_frankel #mutationtesting
(Java) Developer & Architect
•As Consultant
![Page 3: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/3.jpg)
HYBRIS, AN SAP COMPANY
3
![Page 4: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/4.jpg)
MANY KINDS OF TESTING
@nicolas_frankel #mutationtesting
Unit Testing
Integration Testing
End-to-end Testing
Performance Testing
Penetration Testing
Exploratory Testing
etc.
![Page 5: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/5.jpg)
THEIR ONLY SINGLE GOAL
@nicolas_frankel #mutationtesting
Ensure the quality of the production code
![Page 6: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/6.jpg)
THE PROBLEM
@nicolas_frankel #mutationtesting
How to check the quality of the testing code?
![Page 7: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/7.jpg)
CODE COVERAGE
@nicolas_frankel #mutationtesting
“Code coverage is a measure used to describe the degree to which the source code of a program is tested”
--Wikipediahttp://en.wikipedia.org/wiki/Co
de_coverage
![Page 8: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/8.jpg)
MEASURING CODE COVERAGE
@nicolas_frankel #mutationtesting
Check whether a source code line is executed during a test
•Or Branch Coverage
![Page 9: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/9.jpg)
COMPUTING CODE COVERAGE
CC =Lexecuted
Ltotal*100
CC: Code Coverage(in percent)
Lexecuted: Number of executed lines of code
Ltotal: Number of total lines of code
@nicolas_frankel #mutationtesting
![Page 10: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/10.jpg)
JAVA TOOLS FOR CODE COVERAGE
@nicolas_frankel #mutationtesting
JaCoCo
Clover
Cobertura
etc.
![Page 11: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/11.jpg)
100% CODE COVERAGE?
@nicolas_frankel #mutationtesting
“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebobmarti
n/status/55966620509667328
![Page 12: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/12.jpg)
ASSERT-LESS TESTING
@Test
public void add_should_add() {
new Math().add(1, 1);
}
@nicolas_frankel #mutationtesting
But, where is the
assert?As long as the Code Coverage is
OK…
![Page 13: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/13.jpg)
CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Any metric can be gamed!
Code coverage is a metric…
⇒ Code coverage can be gamed
• On purpose
• Or by accident
![Page 14: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/14.jpg)
CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Code Coverage lulls you into a false sense of security…
![Page 15: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/15.jpg)
THE PROBLEM STILL STANDS
@nicolas_frankel #mutationtesting
Code coverage cannot ensure test quality
• Is there another way?
Mutation Testing to the rescue!
![Page 16: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/16.jpg)
THE CAST
@nicolas_frankel #mutationtesting
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”
![Page 17: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/17.jpg)
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
![Page 18: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/18.jpg)
STANDARD TESTING
@nicolas_frankel #mutationtesting
✔Execute Test
![Page 19: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/19.jpg)
MUTATION TESTING
@nicolas_frankel #mutationtesting
?Execute SAME Test
MUTATION
![Page 20: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/20.jpg)
MUTATION TESTING
@nicolas_frankel #mutationtesting
✗
✔Execute SAME Test
Execute SAME Test
Mutant
Killed
Mutant
Survived
![Page 21: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/21.jpg)
KILLED OR SURVIVING?
@nicolas_frankel #mutationtesting
Surviving means changing the source code did not change the test result
• It’s bad!
Killed means changing the source code changed the test result
• It’s good
![Page 22: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/22.jpg)
TEST THE CODE
@nicolas_frankel #mutationtesting
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
![Page 23: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/23.jpg)
✔Execute Test
SURVIVING MUTANT
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
![Page 24: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/24.jpg)
TEST THE CODE
@nicolas_frankel #mutationtesting
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
![Page 25: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/25.jpg)
KILLED MUTANT
@nicolas_frankel #mutationtesting
✗Execute SAME Test
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
![Page 26: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/26.jpg)
MUTATION TESTING IN JAVA
@nicolas_frankel #mutationtesting
PIT is a tool for Mutation testing
Available as
• Command-line tool
•Ant target
•Maven plugin
![Page 27: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/27.jpg)
MUTATORS
@nicolas_frankel #mutationtesting
Mutators are patterns applied to source code to produce mutations
![Page 28: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/28.jpg)
PIT MUTATORS SAMPLEName Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
@nicolas_frankel #mutationtesting
![Page 29: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/29.jpg)
IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Conditionals Boundary
• Potential serious bug hiding there
•if (foo > bar)
![Page 30: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/30.jpg)
IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Void Method Call• Assert.checkNotNull()
• connection.close()
![Page 31: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/31.jpg)
REMEMBER
@nicolas_frankel #mutationtesting
It’s not because the IDE generates code safely that it will never change
• equals()
• hashCode()
![Page 32: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/32.jpg)
ENOUGH TALK…
@nicolas_frankel #mutationtesting
![Page 33: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/33.jpg)
SH… HAPPENS
@nicolas_frankel #mutationtesting
False positives Imperfect Sloooooooooooooooow
![Page 34: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/34.jpg)
PIT IS IMPERFECT
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
@nicolas_frankel #mutationtesting
![Page 35: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/35.jpg)
PIT IS IMPERFECT
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
@nicolas_frankel #mutationtesting
![Page 36: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/36.jpg)
WHY SO SLOW?
@nicolas_frankel #mutationtesting
Analyze test code
For each class under test
• For each mutator• Create mutation
• For each mutation• Run test• Analyze result• Aggregate results
![Page 37: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/37.jpg)
WORKAROUNDS
Increase number of threads
Set a limited a set of mutators
Limit scope of target classes
Limit number of tests
Limit dependency distance
Use incremental analysis
Don’t bind to the test phase
Use scmMutationCoverage
![Page 38: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/38.jpg)
INCREMENTAL ANALYSIS
Metadata stored between runs
Mutant will not be checked again, if:
• timed out, and class has not changed• killed, and neither class nor test have changed• survived, and there are no new/changed tests for it
@nicolas_frankel #mutationtesting
![Page 39: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/39.jpg)
THE SILVER BULLET?
@nicolas_frankel #mutationtesting
Checks the relevance of your unit tests
Points out potential bugs
![Page 40: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/40.jpg)
THE REST IS UP TO YOU
@nicolas_frankel #mutationtesting
Validate the assembled application
• Integration Testing
Check the performance
• Performance Testing
Look out for display bugs
• End-to-end testing
Etc.
![Page 41: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/41.jpg)
TESTING IS ABOUT ROI
@nicolas_frankel #mutationtesting
Don’t test to achieve 100% coverage
Test because it saves money in the long run
Prioritize:
• Business-critical code• Complex code
![Page 42: Craft-Conf - Improve your Tests with Mutation Testing](https://reader031.fdocuments.in/reader031/viewer/2022030318/5a6ecf6c7f8b9a70728b65e5/html5/thumbnails/42.jpg)
Q&A
@nicolas_frankel #springboot
http://blog.frankel.ch/@nicolas_frankelhttp://frankel.in/https://git.io/vznQK