Kill the mutants and test your tests - Roy van Rijn

55
KILL THE MUTANTS a better way to test your tests

Transcript of Kill the mutants and test your tests - Roy van Rijn

Page 1: Kill the mutants and test your tests - Roy van Rijn

KILL THE MUTANTSa better way to test your tests

Page 2: Kill the mutants and test your tests - Roy van Rijn

ABOUT ME

• Roy van Rijn

• Mutants:

• Nora

• Lucas

• Works for

Page 3: Kill the mutants and test your tests - Roy van Rijn

SHOW OF HANDSlet's do a

Page 4: Kill the mutants and test your tests - Roy van Rijn

WHO DOES

• Unit testing

• Test-driven development (TDD)

• Continuous integration

• Measure code coverage

• Mutation testing

Page 5: Kill the mutants and test your tests - Roy van Rijn

UNIT TESTING

• Prove your code works

• Instant regression tests

• Improve code design

• Has become a mainstream practise over the last 10 years

Page 6: Kill the mutants and test your tests - Roy van Rijn

CONTINUOUS INTEGRATION

• Automate testing

• Maintain a single source repository

• Collect statistics

Page 7: Kill the mutants and test your tests - Roy van Rijn

CODE COVERAGE

• Measure the lines (or branches) that are executed during

testing

Page 8: Kill the mutants and test your tests - Roy van Rijn

CODE COVERAGE

• How did they test your car?

Page 9: Kill the mutants and test your tests - Roy van Rijn

CODE COVERAGE

• Who has seen (or written?) tests

• without verifications or assertions?

• just to fake and boost coverage?

• 100% branch coverage proves nothing

Page 10: Kill the mutants and test your tests - Roy van Rijn

QUIS CUSTODIET IPSOS

CUSTODES?Who watches the watchmen?

Page 11: Kill the mutants and test your tests - Roy van Rijn

MUTATION TESTING

• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth

Prize)

• A better way to measure the quality of your tests

• Surge of interest in the 1980s

• Time to revive this interest!

Page 12: Kill the mutants and test your tests - Roy van Rijn

TERMINOLOGY: MUTATION

• A mutation is a (small) change in your codebase, for example:

Page 13: Kill the mutants and test your tests - Roy van Rijn

TERMINOLOGY: MUTANT

• A mutant is a mutated version of your class

Page 14: Kill the mutants and test your tests - Roy van Rijn

MUTATION TESTING

• Generate (a lot of) mutants of your codebase

• Run (some of) your unit tests

• Check the outcome!

Page 15: Kill the mutants and test your tests - Roy van Rijn

OUTCOME #1: KILLED

• A mutant is killed if a test fails (detecting the mutated code)

• This proves the mutated code is properly tested

Page 16: Kill the mutants and test your tests - Roy van Rijn

OUTCOME #2: LIVED

• A mutant didn’t trigger a failing test…

Page 17: Kill the mutants and test your tests - Roy van Rijn

OUTCOME #3: TIMED OUT

• The mutant caused the program loop, get stuck

Page 18: Kill the mutants and test your tests - Roy van Rijn

OTHER OUTCOMES

• NON-VIABLE

• JVM could not load the mutant bytecode

• MEMORY ERROR

• JVM ran out of memory during test

• RUN ERROR

• An error but none of the above.

Page 19: Kill the mutants and test your tests - Roy van Rijn

FAULT INJECTION?

• With fault injection you test code

• Inject faults/mutations and see how the system reacts

• With mutation testing you test your tests

• Inject faults/mutations and see how the tests react

Page 20: Kill the mutants and test your tests - Roy van Rijn

TOOLING

• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)

• Jester: http://jester.sourceforge.net/ (inactive)

• Jumble: http://jumble.sourceforge.net/ (inactive)

• javaLanche: http://www.st.cs.uni-saarland.de/mutation/

(inactive)

• PIT: http://pitest.org/

Page 21: Kill the mutants and test your tests - Roy van Rijn

USING PIT

• PIT uses configurable ‘mutators'

• ASM (bytecode manipulation) is used to mutate your code

• No mutated code is stored, it can't interfere with your code

• Generates reports with test results

Page 22: Kill the mutants and test your tests - Roy van Rijn

MUTATORS: CONDITION BOUNDARY

> into >=

< into <=

>= into >

<= into <

Page 23: Kill the mutants and test your tests - Roy van Rijn

MUTATORS: NEGATE CONDITIONALS

== into !=

!= into ==

<= into >

>= into <

< into >=

> into <=

Page 24: Kill the mutants and test your tests - Roy van Rijn

MUTATORS: REMOVE

CONDITIONALS

into

if(true) {

//something

}

if(a == b) {

//something

}

Page 25: Kill the mutants and test your tests - Roy van Rijn

MUTATORS: MATH

+ into -

- into +

* into /

/ into *

% into *

& into |

<< into >>

>> into <<

>>> into <<<

a++ into a--

a-- into a++

Page 26: Kill the mutants and test your tests - Roy van Rijn

MUTATORS: MANY MORE

• Replacing return values (return a; becomes return 0;)

• Removal of void invocations (doSomething(); is removed)

• Some enabled by default, others are optional/configurable

Page 27: Kill the mutants and test your tests - Roy van Rijn

MUTATION TESTING IS SLOW?

• Speed was unacceptable in the 80's

• Mutation testing is still CPU intensive

• But PIT has a lot of methods to speed it up!

Page 28: Kill the mutants and test your tests - Roy van Rijn

WHICH TESTS TO RUN?

• PIT uses code coverage to decide which tests to run:

• A mutation is on a line covered by 3 tests? Only run those.

Page 29: Kill the mutants and test your tests - Roy van Rijn

SIMPLE EXAMPLE

• 100 classes

• 10 unit tests per class

• 2 ms per unit test

• Total time (all tests): 100 x 10 x 2ms = 2s

Page 30: Kill the mutants and test your tests - Roy van Rijn

SIMPLE EXAMPLE

• Total time (all tests): 100 x 10 x 2ms = 2s

• 8 mutants per class, 100 classes x 8 = 800 mutants

• Brute force: 800 x 2s = 26m40s

• Smart testing: 800 x 10 x 2ms = 16s

Page 31: Kill the mutants and test your tests - Roy van Rijn

LONGER EXAMPLE

• Total time (all tests): 1000 x 10 x 2ms = 20s

• 8 mutants per class, 1000 classes x 8 = 8000 mutants

• Brute force: 8000 x 20s = 1d20h26m40s…!!!

• Smart testing: 8000 x 10 x 2ms = 2m40s

Page 32: Kill the mutants and test your tests - Roy van Rijn

PERFORMANCE TIPS

• Write fast tests

• Good separation or concerns

• Use small classes, keep amount of unit tests per class low

Page 33: Kill the mutants and test your tests - Roy van Rijn

INCREMENTAL ANALYSIS

• Experimental feature

• Incremental analysis keeps track of:

• Changes in the codebase

• Previous results

Page 34: Kill the mutants and test your tests - Roy van Rijn

HOW ABOUT MOCKING?

• PIT has support for:

• Mockito, EasyMock, JMock, PowerMock and JMockit

Page 35: Kill the mutants and test your tests - Roy van Rijn

HOW TO USE PIT?

• Standalone Java process

• Build: Ant task, Maven plugin

• CI: Sonarqube plugin, Gradle plugin

• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin

Page 36: Kill the mutants and test your tests - Roy van Rijn

STANDALONE JAVA

java -cp <your classpath including pit jar and dependencies>

org.pitest.mutationtest.commandline.MutationCoverageReport

--reportDir /somePath/

--targetClasses com.your.package.tobemutated*

--targetTests com.your.package.*

--sourceDirs /sourcePath/

Page 37: Kill the mutants and test your tests - Roy van Rijn

MAVEN PLUGIN

Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage

<plugin>

<groupId>org.pitest</groupId>

<artifactId>pitest-maven</artifactId>

<version>1.0.0</version>

<configuration>

<targetClasses>

<param>com.your.package.tobemutated*</param>

</targetClasses>

<jvmArgs>…</jvmArgs>

</configuration>

</plugin>

Page 38: Kill the mutants and test your tests - Roy van Rijn

EXAMPLELet’s kill some mutants… or be killed.

Page 39: Kill the mutants and test your tests - Roy van Rijn

USE CASE

The price of an item is 17 euro

If you buy 20 or more, all items cost 15 euro

If you have a coupon, all items cost 15 euro

Page 40: Kill the mutants and test your tests - Roy van Rijn

CODE

public int getPrice(int amountOfThings, boolean coupon) {

if (amountOfThings >= 20 || coupon) {

return amountOfThings * 15;

}

return amountOfThings * 17;

}

Page 41: Kill the mutants and test your tests - Roy van Rijn

TEST #1

@Test

public void testNormalPricing() {

//Not enough for discount:

int amount = 1;

Assert.assertEquals(17, businessLogic.getPrice(amount, false));

}

Page 42: Kill the mutants and test your tests - Roy van Rijn

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {

if (amountOfThings >= 20 || coupon) {

return amountOfThings * 15;

}

return amountOfThings * 17;

}

Page 43: Kill the mutants and test your tests - Roy van Rijn

TEST #2

@Test

public void testDiscountPricingByAmount() {

//Enough for discount:

int amount = 100;

Assert.assertEquals(1500, businessLogic.getPrice(amount, false));

}

Page 44: Kill the mutants and test your tests - Roy van Rijn

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {

if (amountOfThings >= 20 || coupon) {

return amountOfThings * 15;

}

return amountOfThings * 17;

}

Page 45: Kill the mutants and test your tests - Roy van Rijn

TEST #3

@Test

public void testDiscountWithCoupon() {

//Not enough for discount, but coupon:

int amount = 1;

Assert.assertEquals(15, businessLogic.getPrice(amount, true));

}

Page 46: Kill the mutants and test your tests - Roy van Rijn

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {

if (amountOfThings >= 20 || coupon) {

return amountOfThings * 15;

}

return amountOfThings * 17;

}

Page 47: Kill the mutants and test your tests - Roy van Rijn

PIT RESULT

Page 48: Kill the mutants and test your tests - Roy van Rijn

PIT RESULT

> org.pitest.mutationtest…ConditionalsBoundaryMutator

>> Generated 1 Killed 0 (0%)

> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0

> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0

> NO_COVERAGE 0

PIT tells us: Changing >= into > doesn’t trigger a failing test

Page 49: Kill the mutants and test your tests - Roy van Rijn

TEST #4

@Test

public void testDiscountAmountCornerCase() {

//Just enough for discount, mutation into > should fail this test

int amount = 20;

Assert.assertEquals(300, businessLogic.getPrice(amount, true));

}

Page 50: Kill the mutants and test your tests - Roy van Rijn

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {

if (amountOfThings >= 20 || coupon) {

return amountOfThings * 15;

}

return amountOfThings * 17;

}

Page 51: Kill the mutants and test your tests - Roy van Rijn

PIT RESULT

Page 52: Kill the mutants and test your tests - Roy van Rijn

PIT RESULT

> org.pitest.mutationtest…ConditionalsBoundaryMutator

>> Generated 1 Killed 0 (0%)

> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0

> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0

> NO_COVERAGE 0

STILL WRONG!?

Page 53: Kill the mutants and test your tests - Roy van Rijn

DID YOU SPOT THE BUG?

@Test

public void testDiscountAmountCornerCase() {

//Just enough for discount, mutation into > should fail this test

int amount = 20;

Assert.assertEquals(300, businessLogic.getPrice(amount, true));

}

Page 54: Kill the mutants and test your tests - Roy van Rijn

SUMMARY

• Mutation testing automatically tests your tests

• Mutation testing can find bugs in your tests

• Code coverage is wrong, gives a false sense of security

• Mutation testing with PIT is easy to implement

Page 55: Kill the mutants and test your tests - Roy van Rijn

QUESTIONS?