What Test Coverage Mean to us?

Joseph Yao

Tuesday, July 9, 13

Who I am?

n Agile Coach at Odd-e

n Father, Husband...

n Software Craftsman

n Basketball, Board Game...

Tuesday, July 9, 13

Agenda

n Bias and Truth of Test Coverage

n Classical Test Coverage

n Mutation Test Coverage

n TDD and Test Coverage

Tuesday, July 9, 13

Bias of Test Coverage

Tuesday, July 9, 13

Truth of Test Coverage

I expect a high level of coverage. Sometimes managers require one. There's a subtle difference.-- Brian Marick

http://martinfowler.com/bliki/TestCoverage.html

Tuesday, July 9, 13

Truth of Test Coverage

Tuesday, July 9, 13

Classical Test Coverage

Tuesday, July 9, 13

Function Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

Satisfied Test: foo(2, 2)Tuesday, July 9, 13

Function Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

Tuesday, July 9, 13

Statement Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

Satisfied Test: foo(2, 2)Tuesday, July 9, 13

Statement Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

Unsatisfied Test: foo(2, -1)Tuesday, July 9, 13

Decision Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

T/F

Satisfied Test: foo(2, 2), foo(-1, 2)Tuesday, July 9, 13

Decision Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

T/F

Unsatisfied Test: foo(2, 2)Tuesday, July 9, 13

Condition Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

T/F

Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1)

T/F

Tuesday, July 9, 13

Condition Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

T/F

Unsatisfied Test: foo(2, 2), foo(-1, 2)

T/F

Tuesday, July 9, 13

Classical Test Coverage

Tuesday, July 9, 13

Classical Test Coverage

Tuesday, July 9, 13

Expectation of Automated Test

Tuesday, July 9, 13

Expectation of Automated Test

Any Possible “Bad” Code

Change can be Detected

Tuesday, July 9, 13

Mutation Test Coverage

Tuesday, July 9, 13

Why Mutation Test

Tuesday, July 9, 13

Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}

Tuesday, July 9, 13

Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}

int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}

Tuesday, July 9, 13

Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}

int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}

n Missing test to prove the sideEffect call is needed

n If code is redundant, it can’t be learnt via any classical test coverage

Tuesday, July 9, 13

Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

Condition Coverage Satisfied Test:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, -1))

assertEquals(0, foo(-1, 2))

Tuesday, July 9, 13

Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}

Condition Coverage Satisfied Test:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, -1))

assertEquals(0, foo(-1, 2))

Tuesday, July 9, 13

Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}

Condition Coverage Satisfied Test:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, -1))

assertEquals(0, foo(-1, 2))

Test to cover this Mutator:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, 0))

assertEquals(0, foo(-1, 2))

Tuesday, July 9, 13

Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}

int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}

Condition Coverage Satisfied Test:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, -1))

assertEquals(0, foo(-1, 2))

n Test not prevent “bad” code change to happen

Test to cover this Mutator:

assertEquals(2, foo(2, 2))

assertEquals(0, foo(2, 0))

assertEquals(0, foo(-1, 2))

Tuesday, July 9, 13

Other Mutatorsn Negate Conditionals Mutator

n Math Mutator

n Increments Mutator

n Invert Negatives Mutator

n Inline Constant Mutator

n Return Values Mutator

n Void Method Calls Mutator

n Non Void Method Calls Mutator

n Constructor Calls Mutator

Tuesday, July 9, 13

Test Driven Development

Tuesday, July 9, 13

TDD and Test Coverage

n You can most likely ignore Classical Test Coverage

n You can’t ignore Mutation Test Coverage

n If not TDD, then you can learn a lot from both

Classical and Mutation Test Coverage

Tuesday, July 9, 13

PokerHands Kata

Tuesday, July 9, 13

True Mutation

Tuesday, July 9, 13

Problematic Test

Tuesday, July 9, 13

False Mutation

Tuesday, July 9, 13

Reference

n Martin Fowler’s blog http://martinfowler.com/bliki/TestCoverage.html

n Mutation Test Wikipedia http://en.wikipedia.org/wiki/Mutation_testing

n Mutation Test Analysis Report http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository/TR-09-06.pdf

Tuesday, July 9, 13

Q & A

新浪微博：姚若舟微信：yaoruozhou

土豆（Kata视频）：姚若舟Github: JosephYao

Tuesday, July 9, 13