Parameterized Unit Testing - ICTSS 2011ictss2011.lri.fr/ictss11tutorial.pdfParameterized Unit...

Parameterized Unit Testing ICTSS’11 Tutorial

Nikolai Tillmann Microsoft Research http://research.microsoft.com/pex/ictss11tutorial.pptx

Outline

First part: 14:00 - 15:00pm Second part: 15:30 - 17:00pm

Outline

Unit Testing Isolation Parameterized Unit Testing

Data Generation by Dynamic Symbolic Execution

Patterns Limitations and other Details

About the exercises…

Interactive: http://pexforfun.com

Demos: Pex for Visual Studio

http://research.microsoft.com/pex

Requires Windows, .NET 2.0 or 4.0,

ideally Visual Studio 2008 / 2010

http://pexforfun.com/

http://research.microsoft.com/pex/downloads.aspx

http://research.microsoft.com/pex/downloads.aspx

Unit Testing

Quiz: Unit testing

What is a unit test?

Unit Testing

void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); }

A unit test is a small program with assertions Test a single (small) unit of code

Let’s play a game!

The Code Under Test

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }

t:\myapp.ini A=B Foo=C C=D

Quiz: Coverage

How much block coverage do we need?

1. 50%

2. 80%

3. 100%

4. Block coverage alone is not enough

Quiz: Coverage

Do we need more tests to get 100% cov.?

[TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue(); }

Quiz: Coverage



Quiz: Coverage

Do we need more tests to get 100% cov.?

[TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue(); }


Quiz: Assertions

Why write Assertions (or not)?

1. Documentation

2. Double check your program

3. Please your manager

4. Prevent future bugs

5. Validate user inputs

6. Catch errors early

Quiz: Assertions

Which Assertions should you write? 1. Assert.IsTrue(value == “b”);

2. Assert.IsTrue(value == null);

3. Assert.IsTrue(String.IsNullOrEmpty(value))

4. Assert.IsTrue(false);

5. No assertions

[TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); }

Quiz: Coverage + Assertions

What gives you confidence in the code?

1. High coverage, no assertions

2. Low coverage, many assertions

3. High coverage, many assertions

4. Low coverage, no assertions

5. I wrote it

The Code Under Test


Quiz: Isolation

In the example, what are the external dependencies?

1. Network Share

2. Local Disk

3. No file system, all in memory

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... }

Quiz: Isolation

What is the problem with a Local Disk?

1. Mapping already exists

2. Cannot run tests concurrently

3. Disk full

4. Access rights

Quiz: Isolation

How do you mitigate the Local Disk issues?

1. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration

2. Refactoring: use Streams

3. Refactoring: introduce interface IFileSystem

4. Refactoring: pass the lines as parameter

5. Change implementation of File.ReadAllLines

Quiz: Definition of Unit Test

What is a Unit Test?

A Unit Test is a program that runs fast the code under test, without environment dependencies, with assertions.

What is a Unit Test Suite?

A set of Unit Tests which achieves high code coverage

Isolation

How do you change File.ReadAllLines?

1. Override static method

2. Changing the .NET Framework source code (and recompiling it)

3. Rewrite application in JavaScript

4. Code instrumentation

Dependency hell

var lines = File.ReadAllLines(@"t:\myapp.ini");

Replace any .NET method with a delegate

var lines = File.ReadAllLines(@"t:\myapp.ini");

File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn) { return new string[0]; };

What if we could replace File.ReadAllLines?

Moles

Motivation for Moles

Why another isolation framework?

Specifically designed to enable Pex

Simple, well-defined semantics

▪ “Replace any .NET method”

Mole Types Code Generation

// System.IO public static class File { public static string[] ReadAllLines(string fn); }

// System.IO.Moles public class MFile { public static Func<string, string[]> ReadAllLinesString; } // delegate R Func<T, R>(T t);

Injecting Detours at Runtime

// System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLinesString(fn); … original code } } Automatically injected

at runtime

Moles Demo

Start from ReadFooValue problem Create test project Add moles for mscorlib Write test using moles Run test Debug test

Quiz: Moles Usage

When should you use Moles (and not)? 1. Always use Moles to solve isolation issues

2. With Moles, one does not need to use interfaces anymore

3. Moles only should be used for 3rd party API, use interfaces for isolation in your APIs

4. Moles can be used in production code

5. Moles lets you get away with untestable APIs

6. Moles make test cases more robust

7. With Moles, you do not need integration tests anymore

8. Moles make tests easier to understand

9. Moles is for poor programmers, real programmers rely on interfaces

Exercise (Optional / homework)

Step-by-step tutorial: research.microsoft.com/documentation.aspx “Unit Testing with Microsoft Moles” “Unit Testing SharePoint Foundation with

Microsoft Pex and Moles”

http://research.microsoft.com/en-us/projects/pex/documentation.aspx


What you learned so far

The Definition of Unit Testing

Unit Test Isolation

1. Identify External Dependencies

2. Replace each External API with your delegate

Parameterized Unit Testing

The Recipe of Unit Testing

var list = new List(); list.Add(item); var count = list.Count;

Assert.AreEqual(1, count); }

Three essential ingredients: Data Method Sequence Assertions void AddAndCount() { int item = 3;

Quiz: list.Add(???)

Which value matters? 1. 0

2. 1

3. int.MaxValue, int.MinValue

4. -1

5. 1000

6. it does not matter

7. I don’t know until I read the code

list.Add(???);

Parameterized Unit Test =

Unit Test with Parameters Separation of concerns

Specification of behavior

Data to achieve coverage


void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

for any list, for any item,

… adding 1 item increments Count by 1

Parameterized Unit Tests are Algebraic Specifications

A Parameterized Unit Test can be read as a universally quantified, conditional axiom.

void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); }

forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)

Parameterized Unit Testing is going mainstream Parameterized Unit Tests (PUTs) commonly supported by

various test frameworks .NET: Supported by .NET test frameworks

http://www.mbunit.com/

http://www.nunit.org/

…

Java: Supported by JUnit 4.X

http://www.junit.org/

Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex

http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne

http://www.agitar.com/

Problem

We cannot execute parameterized unit tests without data.

Where does the data come from?

Random data generator Real customer data Ranges Some values hand-picked by dev/tester

47

Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible

How would you do it?

Data Generation Challenge

Combines ideas from symbolic execution

to capture behavior symbolically

model checking

to perform systematic bounded state space exploration

testing

to monitor and check concrete behavior

constraint solving

to find inputs

Dynamic Symbolic Execution Combines many ideas

49

Combines concrete and symbolic execution Algorithm: Set J := ∅ (J is set of already analyzed program inputs) Loop Choose program input i ∉ J (stop if no such i can be found) Output i (test case) Execute P(i); record path condition C (in particular, C(i) holds) Set J := J ∪ C (viewing C as the set { i | C(i ) } ) End loop

Loop does not terminate if number of execution paths is infinite (in the presence of loops/recursion)

Dynamic Symbolic Execution The high-level algorithm • This choice decides search order • Search order decides how quick we

can achieve high code coverage! • Incomplete constraint-solver leads

to under-approximation

Code to generate inputs for:

Constraints to solve

a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); }

Observed constraints

a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890

Data

null {} {0} {123…}

a==null

a.Length>0

a[0]==123… T

T F

T

F

F

Execute&Monitor Solve

Choose next path

Done: There is no path left.

Dynamic Symbolic Execution Example

http://pex4fun.com/CoverMe



Dynamic Symbolic Execution Exercises

ArrayIndexLength Pex knows about all implicit, exception-throwing control-flow branches

ArrayHeap Pex models the heap

Assert, Assert123 Assertions connect code coverage and correctness

http://pex4fun.com/DynamicSymbolicExecutionExercises



Assertions vs. Coverage

What about finding faults? void AssertIsTrue(bool condition) { if (!condition) throw new AssertionException(); }

Assertions induce branches

Pex tries to cover branches

▪ Pex tries to fail assertions -> finds bug!

Create new project in Visual Studio. Insert CoverMe method below. Right-click on CoverMe method, and select “Run Pex”. Inspect results in table. Save tests, …, have fun!

public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); }

Dynamic Symbolic Execution with Pex in Visual Studio


Generated Test Inputs

are persisted as C# Unit

Tests

Exercise (Optional / homework)

Step-by-step tutorial: research.microsoft.com/documentation.aspx “Exploring Code with Microsoft Pex”



How to test this code?

(Actual code from .NET base class libraries)

Motivation: Unit Testing Hell ResourceReader

59

Motivation: Unit Testing Hell ResourceReader

Demo ResourceReader

[PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } } }

Defined by execution environment / programming language, symbolic execution precision, and constraint solving Execution environment: C, Java, x86, .NET,… Precision: linear vs. non-linear arithmetic, “gods integers” vs.

bitvectors, concrete heap vs. symbolic heap., floating-point values, etc. Solvers: lp_solve, CVCLite, STP, Disolver, Z3,…

Examples of DSE implementations: DART (Bell Labs), and also CUTE “concolic execution” EXE/EGT/KLEE (Stanford) “constraint-based execution” Vigilante (Microsoft) to generate worm filters BitScope (CMU/Berkeley) for malware analysis Sage (Microsoft) for security testing of X86 code Yogi (Microsoft) to verify device drivers (integrated in SLAM) Pex (Microsoft) for parameterized unit testing of .NET code CREST, jCUTE, jFuzz, …

Dynamic Symbolic Execution Many implementations

Constraint Solving vs. Search Strategies

Test Inputs

Constraint System

Execution Path

Known Paths

Initially, choose Arbitrary

Path-constraint solving is just

hard.

Reachability is undecidable!

(Loops)

Representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, …

Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references to values" Elements of arrays, memory accessed through unsafe pointers

by mutable “mapping of integers to values"

Efficiency by Many reduction rules, including reduction of ground terms to constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical operations Patricia Trees to represent AC1 operators

(including parallel array updates)

Symbolic State Representation

SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to theories

SAT solver for Boolean structure

Decision procedures for relevant theories: uninterpreted functions with equalities, linear integer arithmetic, bitvector arithmetic, arrays, tuples

Model generation for satisfiable formulas Models used as test inputs

Limitations No decision procedure for floating point arithmetic

Pex uses Z3: http://research.microsoft.com/z3

Constraint Solving

http://research.microsoft.com/z3

)1()2),3,,(((2 xyfyxawritereadfyx

Arithmetic Array Theory Uninterpreted

Functions

Satisfiability Modulo Theories

void Foo(int x, int y, int[] a) { if (x + 2 == y) { a[x] = 3; Debug.Assert(f(a[y - 2]) == f(y – x + 1)); } }

Pex uses some heuristic approaches for domain-specific problems.

Strings [TACAS’09, ICST’10] “uninterpreted functions” for string operations

two-phase solving: first solve integer constraints involving indices and lengths, then character constraints

Floating-point arithmetic Search-based approach: minimization of fitness function

… (more to come)

Domain-specific Constraint Solving

Writing Parameterized Unit Tests


Exercise Write PUT for ReadFooValue

Can you come up with Parameterized Unit Tests?

Create new solution with C# class library Write code for ReadFooValue Create Test Project Add moles for mscorlib Write parameterized unit tests.

Or get result from ReadFooValuePUT\ReadFooValuePUT.sln

Exercise Write PUT for ReadFooValue

[PexMethod] // attribute marks pex tests public void Test(...) { ... }

Exercise Crash Test

[PexMethod] void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); }

Execute the code without assertions

Exercise Assert Observed Results

[PexMethod] public string Regression(string[] lines) { MFile.ReadAllLines = () => lines; return Reader.ReadFooValue(); }

Manually review outputs Pex inserts Assertions automatically

Exercise

[PexMethod] void FooExist(string value1) { PexAssume.IsTrue(value1 != null); var lines = new string[]{“Foo=“ + value1}; MFile.ReadAllLines = () => lines; var value2 = Reader.ReadFooValue();

PexAssert.AreEqual(value1, value2); }

Write and read property

Test Project

Test Generation Work-Flow

Code-Under-Test Project

Parameterized Unit Tests

Pex Generated Tests

Patterns

Patterns how to create objects

[PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); }

What are the test inputs? Only null An array list created by calling a constructor ... and also calling Add(…) many times … and also calling Clear() many times …

Quiz: Complex Objects As Inputs

[PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); }

Other possible problems: What if… ArrayList has no public constructor ArrayList is an abstract class (or interface)

Quiz: Complex Objects As Inputs

Problem: Test parameters may be classes Symbolic execution tracks constraints over (private)

fields of objects Constraint solver determines solution, assigning

concrete values to (private) fields However, in practice objects must be initialized by

constructor, and can be mutated only by calling certain methods

What sequence of method calls reaches a desired field assignment? There may be no such sequence

In general, undecidable

Creating complex objects The Problem

Factory methods

Parameterized factories create and configure objects

Explicit: Public static method annotated with [PexFactoryMethod]

Implicit: Pex guesses “best” constructor/setters

Result: Exploration of reachable states

Reachable using factory method

Reachable within configured bounds

Under-approximation of possible states

Object creation: Factory methods

Hand-written factory method: [PexFactoryMethod(ArrayList)] public static ArrayList Create( int capacity, object[] values) { var a = new ArrayList(capacity); foreach (value in values) a.Add(value); return a; }

Pex will explore both factory method and PUT!

Factory methods ArrayList

Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit

Inspect generated test cases

Notice constructor implicitly used by Pex

Save factory method

Edit factory method

Explore again

Demo: Factory methods

Write class invariant as boolean-valued parameterless method

Refers to private fields

Must be placed in implementation code

Write special constructor/factory method for testing only

Sets fields, and assumes invariant

Result: Exploration of feasible states

May include states that are not reachable

Alternative Object creation via class invariants

Patterns for Parameterized Unit Tests

Pattern 4A

Assume, Arrange, Act, Assert

[PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act // assert quiz 1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }

Pattern Regression Tests

Generated test asserts any observed value

Return value, out parameters

When code evolves, breaking changes in observable will be discovered

int AddTest(int a, int b) { return a + b; }

void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); }

Pattern Roundtrip

For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); // assert quiz 1. Assert.AreEqual(value, s); 2. Assert.AreEqual(s, parsed); 3. Assert.AreEqual(parsed, value); }

Pattern Normalized Roundtrip

For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x

void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); // assert quiz 1. Assert(x == intermediate); 2. Assert(intermediate == roundtripped); 3. Assert(normalized == roundtripped); 4. Assert(x == roundtripped); }

Pattern Reachability

Indicate which portions of a PUT should be reachable.

[PexAssertReachEventually] public void Constructor(object input) { new Foo(input); PexAssert.ReachEventually(); }

Pattern Same Observable Behavior

Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.

public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); }

Pattern Commutative Diagram

Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold that f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.

string Multiply(string x, string y); int Multiply(int x, int y);

void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); }

Homework Patterns

Read patterns paper: research.microsoft.com/Pex/documentation.aspx “Parameterized Test Patterns for Microsoft Pex”

http://research.microsoft.com/Pex/documentation.aspx



Limitations It’s called Parameterized Unit Testing

Pex understand managed .NET code only

Pex does not understand native code.

Problem if branching over values obtained from the environment

Pex may not automatically detect all such cases.

Testability

if (!File.Exists(f)) throw ...

File System?

Hidden Complexity

Pex analyzes every executed .NET instruction Some used libraries may be surprisingly expensive

to analyze

XML parsing

repeatedly converting data between different representations

void Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }

Don’t do this.

Exploration Boundaries

Configurable bounds include:

TimeOut

MaxBranches

MaxCalls

MaxConditions

▪ Number of conditions that depend on test inputs

MaxRuns

ConstraintSolverTimeOut

ConstraintSolverMemoryLimit

Multi-threaded code

Unlike test inputs, thread-interleavings can normally not be controlled

Thus, Pex can only explore single-threaded code

Related approach to explore thread-schedules (but not input parameters) by controlling thread-scheduler: CHESS http://research.microsoft.com/CHESS

http://research.microsoft.com/CHESS

Lack of Test Oracle

Unit Testing != Test Generation

Wrapping up

Conclusion What you learned today

The Definition of Unit Testing

Unit Tests are not Integration Tests

Unit Test Isolation

Introduction of Abstraction Layers


Separation of concerns: Coverage/Specification

Patterns to write Parameterized Unit Tests


Availability

Pex and Moles are Visual Studio 2010 Power Tools

(academic/commercial license available)

Minimum: Windows XP, .NET 2.0 Ideally: Visual Studio 2010 Professional http://research.microsoft.com/pex/

http://pexforfun.com

http://research.microsoft.com/pex/

http://pexforfun.com/

Thank you

http://research.microsoft.com/pex http://sites.google.com/site/asergrp/

Parameterized Unit Testing - ICTSS 2011ictss2011.lri.fr/ictss11tutorial.pdfParameterized Unit...

Documents

Transcript of Parameterized Unit Testing - ICTSS 2011ictss2011.lri.fr/ictss11tutorial.pdfParameterized Unit...