Quick Checking Your Java Classes With T2

Wishnu PrasetyaDept. of Information & Computing Sciences

Utrecht University

www.cs.uu.nl/~wishnu

Overview

What is T2, and its key ideas Implementation Programming Specifications Related Work Result & Research Status Conclusion

2

3

T2 = Testing Tool

A tool to automatically test a Java class

with some interesting features ...

Unit testing.

Inspired by QuickCheck

4

QuickCheck and T2

Both generates tests randomly

Not pure random, but with some direction

passive direction from specs.

One is for functional, the other for OO

5

Example : my own sorted list

public class SortedList {

private LinkedList<Comparable> s

private Comparable max

public SortedList() // initialize s to empty

public void insert(Comparable x)

public Comparable get()

}

6

In-code Specification

public class SortedList {

private LinkedList<Comparable> s

private Comparable max

public SortedList()

public void insert(Comparable x)

public Comparable get() // return and remove max

}

public boolean classinv() { return s.isEmpty() s.contains(max)

}

public Comparable get_spec() {

assert s.isEmpty() : "PRE"

Comparable ret = get()

assert … ret.compareTo(s.getLast()) 0 : "POST"

return ret }

Less abstract Cannot guarantee non-interference Declarative Powerful Minimum maintenance

7

What if my specification is partial ?

private s private max

// classinv : s.isEmpty() s.contains(max)

public void insert(Comparable x) { int i = 0 for (Comparable y : s) { if (y.compareTo(x) > 0) break i++ } s.add(i,x) if (max==null || x.compareTo(max) < 0) max = x }

No pre/post-conditions Class inv. Is partial

Most tools do isolated testing; it will Leak faults produce losts of false positives

8

Sequences as Tests

Simplification We only have private and public members Updating a field F is represented by a call to setF

We’ll talk about a class C

A test on C is: a (finite) sequence of calls to pubmeth(C)

9

private s private max

// classinv : s.isEmpty() s.contains(max) public void insert(Comparable x) // faulty

public Comparable get() // return and remove max

public Comparable get_spec() {

assert !s.isEmpty() : "PRE"

Comparable ret = get()

assert … ret.compareTo(s.getLast()) 0 : "POST"

return ret }

Back to example

u = new SortedList<Integer>

u.insert(1) ; u.insert(10)

u.get()

Hey, with sequence I can catch the fault, despite my partial spec!

It also autogenerates ‘interesting’ target objects.

10

Functions vs Objects

f :: A B Its behavior is fully determined by the relation between

its parameter and result. Deterministic.

An object u:C Its behavior (transition from state to state) is

determined by all C’s methods. Non-deterministic. From this persperctive it is natural to use sequence.

Implementation

1. Representation of tests

2. T2 algorithm

Representing a test

As a Java (Jtest) code

Most testing tools do this...

Alternatively, represent a test as ordinary data

In Haskell you can exploit higher order functions

In Java object. Possible via “meta” objects from Reflection library.

12

void test1 () {

try { u = new SortedList<Integer> u.insert(1) ; u.insert(10) u.get_spec() }

catch (AssertError e) …

13

Meta representation of a test

class CALL_METHOD

Method m

MkValStep[] params

class CONST Object val

A test is a list of :

class CREATE_TARGET_OBJECT

Constructor con

MkValStep[] params

class REF

int index

Object pool

This is just a list of objects; you can program over it! E.g. :

• Producing the corresponding execution• On the fly checking• Save, reload, regress• Pretty print, including intermediate states• Mutate

Difficult, if we are to do these directly over Junit code.

14

Simple Algorithm

Top level: Generate (and check) N number of tests Save violating tests

Generating a test

Create a target object u ; =

Repeat until violation, or maxlegth :

choose a method m pubmeth(C)

extend with a call to m

check classinv and m_spec

Checking a step u.m ( ... )

1. If m_spec exists, call it instead. Catch these:

1. PRE AssertError discard 2. Other AssertError spec violation

3. Other Error or RunTimeException internal violation

4. Other Exception no violation (expected)

2. Check classinv on u, catch these:

1. APPMODEL AssertError discard 2. Other Throwable violation + warning

3. Nothing thrown, but returns false violation

15

16

Every step in a test, is a call m( …. )

Create a fresh object by calling a constructor (via reflection).

Reuse a previously created object. Also cover over side effects! Require you to maintain a pool of objects.

T2 additionally allows you to supply a base domain: Is searched first can force range Use cloning side effect free

Whatever the decision, it is represented as a MkValStep object

17

Main Testing API

Class RndEngine

static void RndTest (Pool p,

BaseDomain d,

InterfaceMap m,

Class C,

String[] options )

18

** Error trace [1] : ** CREATING target object: (U2.T2.examples.SortedList) @ 0 s (LinkedList) @ 1 max NULL ** STEP 1. ** Calling method insert with: ** Receiver: target-obj ** Arg [0:] (Integer) : 29 ** Target object after the step: (U2.T2.examples.SortedList) @ 0 s (LinkedList) @ 3 [0] (Integer) : 29 max (Integer) : 29 ** STEP 2. ** Calling method insert with: ** Receiver: target-obj ** Arg [0:] (Integer) : 0 ** Target object after the step: (U2.T2.examples.SortedList) @ 0 s (LinkedList) @ 5 [0] (Integer) : 0 [1] (Integer) : 29 max (Integer) : 0

** STEP 3. ** Calling method get_spec with: ** Receiver: target-obj ** Throwing java.lang.AssertionError: POST ** Target object after the step: (U2.T2.examples.SortedList) @ 0 s (LinkedList) @ 3 [0] (Integer) : 29 max (Integer) : 29 ** Assertion VIOLATED! ** Strack trace: java.lang.AssertionError: POST at U2.T2.examples.SortedList.get_spec(SortedList.java:44) ...

Programming Specifications

1. Application model

2. Temporal properties

Abstract View

20

C Application

m1

m2

-step....

....skip

the state is the object pool

Automaton ANY(C).

Implicitly assume App respects C’s ownership claim.

A behavior of C is an (infinite) path in ANY.

A property is a predicate over paths.

A test is a finite prefix of a path.

Btw, a single threaded model !

Application Model

21

m1

m2

-step....

....skip

What if we know something about App?

Model this assumption as constraints over ANY.

Don’t constrain -steps.

You can still constrain e.g. order in which methods of C are called.

T2 only cheks the m-steps!

To force T2 to check the constraint, put it in the the classinv.

Example

22

private String aux_laststep = null

private int aux_delta = 0

public boolean appmodel() {

if (aux_laststep.equals("insert")) aux_delta++

if (aux_laststep.equals("get")) aux_delta--

return aux_delta>=0 ;

}

will be filled by T2

boolean classinv() { assert appmodel() : “APPMODEL”

return …. // old spec

}

Temporal Property

Properties are predicates over paths so they are all temporal.

Example. For all M,S we require:

s == S && max == M unless open

Checking a predicate over paths implies we have to maintain paths.

Complicated! T2 only maintains current state. If the the state-preds are closed in u, the prop is -

insensiteve!

23

Add it to SortedList;change methods etc.

Rephrase the property

Introduce aux-vars to maintain past states.

Now we can express the property as a classinv :

aux_open_frozen

|| (aux_s_frozen.equals(s) && aux_max_frozen==max)

|| open

24

Also extends classinv to update the aux-vars accordingly.

Some Related Work

1. Verification

2. Other testing approaches

How is T2 related to Verification ?

Checking temporal property essentially as in explicit-state model checking (SPIN) MC generates paths from models Finite states can generate all paths!

JPAX : runtime verification tool Generates finite paths from a real application Obviously you can’t generate all paths; so just pick some. Problem: no control over execution fault may only surface

after a long execution

T2 generate finite paths from a real unit Full control over the generation of paths

26

v

v

Other Path-based Testing Approaches

TestLog (Ducasse et al) Mine paths from execution of a real application Use Prolog to ‘query’ properties over those paths

Palulu (Artzi et al) Mine paths from execution of a real application Use them to construct an application model

CnC Isolated (not path-based) approach Use a combo of Hoare-logic/theorem prover/constraint solver to

calculate tests data

27

Some Results

Class #MethAver.Cyclo

Max.Cyclo

#Inst

MethCov. (%)

BlockCov. (%)

Time (ms) Fault #step

BinarySearchTree 18 2.4 7 359 100 94 907 No 20 K

SortedList 7 3.6 6 212 100 98 16 yes 0.5 K

Show 13 4.4 33 844 88 84 2500 no 5 K

Pool 10 2.9 7 340 100 91 531 no 5 K

Not a Swiss Knife…

Some classes are not suitable for T2. E.g. two main classes in T2 :

RndEngine only has one main API

Its internal behavior (e.g. loop invariant) is more interesting.

Trace can’t generate meaningful instances on its own.

.

Class #MethAver.Cyclo

Max.Cyclo #instr

Trace 14 6,3 22 1133

RndEngine 15 7,8 21 2117

Research Status

On going. Tool prototype, but fully functional

Paper and documentation provided

Open issues (Master thesis topics) Is it really useful? Can it scale up?

Making the random testing more directed, e.g. by using WP-logic

Dealing with ownership issue Testing User Interface Applying complementary ideas from other approaches

30

Conclusion

On-going. S/W :

Not an all-purpose solution Cannot handle concurrency Strong features Preliminary results seem promising

www.cs.uu.nl/wiki/WP/T2Framework

31