Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test...

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in

Applications without Test Oracles

Christian Murphy, Kuang Shen, Gail Kaiser

Columbia University

Chris Murphy – Columbia University 2

Problem Statement Some applications (e.g. machine learning,

simulation) do not have test oracles that indicate whether the output is correct for arbitrary input

Oracles may exist for a limited subset of the input domain, and gross errors (e.g. crashes) can be detected with certain inputs or other techniques

However, it is difficult to detect subtle (computational) errors for arbitrary inputs


Observation Even when there is no oracle in the general

case (i.e. knowing the relationship between a particular input and its output), it may be possible to know relationships between sets of inputs and their corresponding outputs

One such approach that takes advantage of this is “Metamorphic Testing” [Chen ’98]


Metamorphic Testing An approach for creating follow-up test cases

based on those that have not revealed any defects

If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce a new test case, t(x)

We can then predict the expected value of f(t(x)) based on the value of f(x) obtained from the actual execution


Metamorphic Testing Example Consider a function std_dev(A[ ])

If we permute the elements in A, the result should stay the same

If we multiply each element in A by 2, the result should be doubled

These properties can be used to create a “pseudo-oracle” for the function


Metamorphic Testing without an Oracle When a test oracle exists, we can know

whether f(t(x)) is correctBecause we have an oracle for f(x)So if f(t(x)) is as expected, then it is correct

When there is no test oracle, f(x) acts as a “pseudo-oracle” for f(t(x)) If f(t(x)) is as expected, it is not necessarily

correctHowever, if f(t(x)) is not as expected, either f(x) or f(t(x)) (or both) is wrong


Our Solution To address the problem of applications that have

no test oracle, we use Metamorphic Testing at the function level

To specify the metamorphic properties, we use extensions to the Java Modeling Language (JML) and a pre-processor called Corduroy

To automate the process and address the need for initial input values, we check the properties at runtime


Related Work Applying Metamorphic Testing to “non-testable

programs” (applications without test oracles) Chen IST’02, Chen ISSTA’02, Chan JWSR’07

Checking algebraic specifications at runtime Sankar ISSTA’91, Nunes ICFEM’06

Checking program invariants at runtime Flanagan FME’01, Hangal ICSE’02


JML Basics Behavioral interface specification language that

embraces the Design by Contract approach

Developers can specify: preconditions (“requires”) postconditions (“ensures”) arbitrary assertions

Specifications can be checked dynamically using JML Runtime Assertion Checking tools


ProcessSpecify functions’ metamorphic

properties using extension to JML

Specify functions’ metamorphicproperties using extension to JML

Methods that checkmetamorphic properties

Methods that checkmetamorphic properties

Pure JML specificationsthat invoke test methods

Pure JML specificationsthat invoke test methods

CorduroyCorduroy


Metamorphic Properties in JML

/*@ @ensures \result <= 1 && \result >= -1; @meta sine(x + 2 * PI) == \result @meta -1 * sine(-x) == \result */public double sine (double x) { ...}


Extension to JML: Array Functions/*@ @ensures \result >= 0; @meta std_dev(\shuffle(A)) ==

\result @meta std_dev(\multiply(A, 2)) ==

\result * 2 */public double std_dev(double[] A) { ...}


Extensions to JML Array/Collection Functions

Shuffle/permute the order of the elementsMultiply each element by a constantAdd a constant to each element Include a new elementExclude an element

Addressing non-determinismCheck in range of possible valuesCheck in a set of possible values


Model

CallerFunction

CallerFunction

Input(s)Input(s)

CalleeFunction

CalleeFunction

OutputOutput

CheckMetamorphic

Properties

CheckMetamorphic

Properties

Output

Input(s)


Implementation/*@ @meta average(\multiply(A, 2)) == \result * 2 */public double average(double[] A) { ...}

protected boolean __metaTest_average (double[] A, double result) { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), result * 2);}

@ensures __metaTest_average(A, \result);


Case Studies We investigated the WEKA and RapidMiner

toolkits for Machine Learning in Java

For WEKA, we tested four apps:Naïve Bayes, Support Vector Machines (SVM),

C4.5 Decision Tree, and k-Nearest Neighbors For RapidMiner, we tested one app:

Naïve Bayes


Experimental Setup For each of the five apps, we specified 4-6

metamorphic properties of selected methods (based on our knowledge of the expected behavior of the overall application)

Testing was conducted using data sets from UCI Machine Learning Repository

Goal was to determine whether the properties held as expected


Findings Discovered defects in WEKA k-NN and

WEKA Naïve Bayes related to modifying the machine learning “model”This was the result of a variable not being

updated appropriately

Discovered a defect in RapidMiner Naïve Bayes related to determining confidence There was an error in the calculation


Limitations Specifying and checking the properties was

simplified by use of the tool, but still requires some knowledge of the app to determine the properties in the first place

The approach works well for functions that take input and produce output, but in some case requires more complex properties than can be expressed in JML


Future Work Reducing testing time by checking

properties in parallel

Implementations for other languages

More empirical studies to determine effectiveness and applicability to other domains


Summary We have presented a testing approach that uses

metamorphic testing of individual functions in applications that do not have test oracles

These are specified via an extension to the JML specification language

We have also presented an implementation framework called Corduroy, which converts the specification of metamorphic properties into test methods

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in

Applications without Test Oracles

Chris Murphy

[email protected]


Categories of Metamorphic Properties Additive: Increase (or decrease) numerical values

by a constant Multiplicative: Multiply numerical values by a

constant Permutative: Randomly permute the order of

elements in a set Invertive: Reverse the order of elements in a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set

ML apps such as ranking, classification, and anomaly detection exhibit these properties [Murphy SEKE’08]


Specifying More Complex Properties/*@ @meta test_myFunc(x, \result)*/public double myFunc (int x) { ...}

private boolean test_myFunc (int x, double result){ ...}


Addressing Side Effects// holds result of last call to “average”private double value;

/*@ @assignable value; @meta average(\multiply(A, 2)) == value * 2 */public void average(double[] A) { ... value = ... // no return value}


Addressing Side Effectsprotected boolean __metaTest_average (double[] A) { double __value = value; // backup try { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), value * 2); } finally { value = __value; // restore }}

__value

Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test...

Documents

Transcript of Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test...