Android Mutation Testing

Hazem SalehMobile Architect @Nickelodeon

About

More than twelve years of experience in Java and mobile solutions.

Apache Open Source Committer.

Author of four technical books.

DeveloperWorks Contributing Author and DZone MVB.

Technical Speaker (JavaOne, ApacheCon, Geecon, JavaLand …etc).

JSR Experts Group Member.

An X-IBMer and Currently a Mobile Software Architect in Nickelodeon.

Agenda

Developers Life without Unit Testing.

Unit Testing 101.

Unit Testing in Android.

Code Coverage and Jacoco.

Code Coverage Demo.

Current challenges of traditional code coverage.

Mutation Testing 101.

What and Why PIT?

PIT Android Integration.

Mutation Testing Demo.

Quiz.

Q & A.

Developers Life without Unit Testing

Complex integration between the system components.

Developers Life without Unit Testing

Unmanaged number of new/regression defects especially when the system complexity increases.

Developers Life without Unit Testing

Low application quality.

Longer testing cycle.

test

fix

test

fix

test

fix

testfix

test

fix

test

fix

test

fix

Unit Testing 101

A unit testing is a piece of code (usually a method) that invokes another piece of code and later checks the correctness of some assumptions.

Unit testing helps in detecting BUGGY components in the early stages of the project.

A test suite is a set of test cases, and a test case is a set of tests which verifies the system components.

AutomatedRepeatable

Easy to run.

Fast

Easy to understand Incremental

Unit Testing 101

Unit Testing in ANDROID

Generally, Unit tests in Android are implemented using:

– JUnit.

– Mockito (for mocking testable class dependencies).

However, there are some challenges:– Mocking Static methods.

– Mocking final methods or classes.

– Mocking private methods.

Thanks to PowerMock, the previous challenges are met.

Unit Testing in ANDROID

Other challenge:

–Unit testing classes with huge dependency on Android SDK is hard (such as activities / fragments).

Thanks to Robolectric:–This challenge is also met:http://robolectric.org/

–Robolectric mocks all Android classes by Shadow classes.

–Robolectric adds a great power to Android apps unit testing.

Code Coverage and Jacoco

Code Coverage represents the amount of source code which will be executed when test cases run.

In order to measure the amount of tested source code, there are popular coverage criteria:

–Statement coverage

–Function coverage

–Branch coverage, which represents the amount of branches of each control structure (such as if) covered.

Code Coverage and Jacoco

JaCoCo is a free code coverage library for Java.

It includes the popular code coverage metrics: statement coverage, branch coverage, and method coverage.

Code Coverage and Jacoco

Fortunately, JaCoCo is fully Integrated with Gradle.

JaCoCo works fine with Android projects.

In order to configure JaCoCo with Gradle:

–Enable code coverage for the build type(s) that you will be testing with.

–Import JaCoCo in your gradle build file.

–Configure JaCoCo with your Java sources.

Code Coverage and Jacoco

Demo

Sample URL:https://github.com/hazems/Dagger-Sample

Challenges

Traditional code coverage only measure the amount of executed code.

Traditional code coverage does not detect code faults.

Traditional code coverage does not show how strong the code is.

Mutation testing 101

Mutation testing is about seeding app source code with faults (mutations).

After seeding, unit tests then run.

If a unit test fails, then a mutation is killed (and means that your unit test is strong enough to face this mutation).

If a unit test succeeds, then a mutation is lived (and means that your unit test needs modification to be stronger).

In mutation testing, the quality of the test can be measured by the percentage of the killed mutations.

Mutation testing 101

Mutations can be for example:–Negate Conditionals Mutator (<= TO > and < TO >=)

–Increment Mutator (i++ TO i--)

–Invert Negatives Mutator (-I to I)

–Math Mutator (+ to – and – to + and * to / and * to /)

–Return Values Mutator (boolean true to false)

–Void Method Call Mutator (removing call).

What and Why PIT?

One of the mutation testing tools for Java.

It has advantages–Fast to execute (unlike most of the Java mutation testing tools).

–Compatible with:

•Ant

•Maven

•Gradle

–Active.

–Provides easy to read test reports.

PIT Android Integration

Gradle Plugin for PIT is available:http://gradle-pitest-plugin.solidsoft.info/ https://github.com/szpak/gradle-pitest-plugin

Gradle Plugin works perfect with Java projects.

However, for Android Gradle projects, you need to use this fork:https://github.com/koral--/gradle-pitest-plugin

PIT Android Integration

For using PIT in your Android apps, add PIT plugin to your top-level build.gradle file as followsbuildscript {

repositories { mavenCentral()

}

dependencies { classpath

'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.3'

}}

Apply plugin: 'com.android.application’apply plugin: 'pl.droidsonroids.pitest'

PIT Android Integration

Configure PIT plugin basically as follows

pitest { targetClasses = ['com.test.xyz.daggersample.presenter.*']

/* Specify target classes to be mutated */

excludedClasses = ['**Factory*'] /* Exclude the unnecessary classes */

pitestVersion = "1.1.0"

threads = 4 /* Specify number of threads to execute mutation testing */

outputFormats = ['XML', 'HTML'] /* Specify output format */

}

PIT Android Integration

Check the mutation results by executing the following command:>./gradlew clean pitest

Mutation Testing Demo

Demo

Quiz

Q & A

Twitter: @hazemsEmail: hazems@apache.org