Parameterized Unit Tests By Nikolai Tillmann and Wolfram Schulte Proc. of ESEC/FSE 2005 Presented by...
-
Upload
jasmine-wheeler -
Category
Documents
-
view
220 -
download
1
Transcript of Parameterized Unit Tests By Nikolai Tillmann and Wolfram Schulte Proc. of ESEC/FSE 2005 Presented by...
Parameterized Unit Tests
By Nikolai Tillmann and Wolfram SchulteProc. of ESEC/FSE 2005
Presented by Yunho KimProvable Software Lab, KAIST
Contents
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 2/27
• Introduction
• Parameterized unit tests
• Framework
• Conclusion
• Unit testing is a method of testing that verifies the individual units of source code are working properly
From http://en.wikipedia.org/wiki/Unit_testing
• A unit test consists of a sequence of method calls with assertions
• Unit tests are often served as simple specifica-tions
IntroductionWhat Is Unit Testing?
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 3/27
[TestMethod]void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added}
• Unit tests tools do not provide any guidance for– which tests should be written for high code coverage– how to come up with a minimal number of test cases– what guarantees the test cases provide
• What about other array lists and other objects?
IntroductionWhat Is the Problem?
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 4/27
[TestMethod]void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added}
• Parameterized unit tests are generalized unit tests by allowing parameters– Parameterized test methods are specifications of the
behavior of the methods-under-tests– PUTs describe a set of traditional unit tests which can be
obtained by instantiating the parameterized test methods
IntroductionParameterized Unit Tests(PUTs)
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 5/27
[PexMethod]void TestAdd(ArrayList a, object o){ // Given arbitrary array // lists and objects Assume.IsTrue(a!=null); // Assume a null int i = a.Count; a.Add(o); // Push a value into it Assert.IsTrue(a[i] == o); // Check that value was added}
Contents
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 6/27
• Introduction
• Parameterized unit tests
• Framework
• Conclusion
• Pex generates Unit Tests from hand-written Parameter-ized Unit Tests through Automated Exploratory Test-ing based on Dynamic Symbolic Execution.
Parameterized Unit TestsPex: a PUTs framework
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 7/27
From http://research.microsoft.com/pex/default-.aspx
• Symbolic execution is used to automatically and system-atically produce the minimal set of actual parameters(test inputs)
Parameterized Unit TestsTest Case Generation
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 8/27
Algorithms1. For each formal parameter a symbolic variable is intro-
duced2. For each code path, a path condition is built over symbolic
variables1. Loops and recursions are approximated up to a speci-
fied number of unfoldings3. After all constraints are collected, find solutions for the
constraints4. Finally, deduce what inputs cause a code path to be taken
• The number of possible paths to consider can grow expo-nentially with the number of control flow decisions
Parameterized Unit TestsReusing Parameterized Unit Tests
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 9/27
1 class Bag{ 2 Hashtable h = new Hashtable(); 3 4 void Add(object o){ 5 if (h.ContainsKey(o)) h[o] = (int)h[o] + 1; 6 else h[o] = 1; 7 } 8 int multiplicity(object o){ 9 if (h.ContainsKey(o)) return (int)h[o];10 else return 0;11 }12 }1 [PexMethod]2 Void AddMultiplicityTest(Bag b, objectx){3 Assume.IsTrue(b!=null & x!=nul);4 int m = b.multiplicity(x);5 b.Add(x);6 Assert.IsTrue(m+1 == b.Multiplicity(x));
During symbolic execution, ContainsKey method is symbolically run twice
• If we have axioms summarizing the hash table’s operations, we can avoid execution of the hash table
Parameterized Unit TestsReusing Parameterized Unit Tests
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 10/27
[TestClass, UsingAxioms(typeof(HashtableTests))]Class BagTests{ [PexMethod] void AddMultiplicityTest() { } }
[TestClass, ProvidingAxioms(typeof(Hashtable))]Class HashtableTests{ [TestAxiom] void NothingContainedInitially(object key){ } [TestAxiom] void SetImpliesContainsAndGet(){ } }
Provided axioms are re-written as con-straints
Contents
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 11/27
• Introduction
• Parameterized unit tests
• Framework– Symbolic state–Constraints– Symbolic Evaluation– Axioms
• Conclusion
• Symbolic states are like concrete states except that sym-bolic states can contain symbolic expressions
• Symbolic expressions E is described by the following grammar
• ObjectId is an infinite set of potential object identifiers• VarId, TypeId and FuncId are a set of variable, type and
function identifiers respectively
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 12/27
E = o object ids 2 ObjectId | v variables 2 VarId | t types 2 TypeId | f( ) function application 2 Fun-cId | 8 .E universal quantification
¹E¹v
• There are two types of function symbols
• Predefined function symbols have a fixed interpreta-tion in the theorem provers used– add(x, y), equals(x, y), subtype(x, y)– null, void, 0, 1, 2, … (constants are considered as nullary functions)– Storage function symbols operating on maps
• update(m, x, y) denotes the update of map m at index x to the new value y• select (m, x) selects the value of map m at index x
• Uninterpreted function symbols represent properties of ob-jects and method calls appearing in axioms
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 13/27
• Uninterpreted function symbols represent properties of objects and method calls appearing in axioms– For example, type(x) denotes the type of object x, fieldf(x) denotes the
address of field f of object x
• For each method m of the program with n parameters, up to three uninterpreted function symbols are introduced, ms, mx, mr
– They are used to summarize different dynamic aspects of m
• Let h be an expression denoting the state of the heap in which m( ) is called– ms(h, ) denotes the resulting state of the heap
– mr(h, ) denotes the return value of the call
– mx(h, ) represents the type of an exception
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 14/27
¹x¹x¹x¹x
• There are two kinds of heaps, ‘extensional’ and ‘intensional’ heap
• The extensional heap is represented by nested applica-tions of the map update function
• For example, the execution of the code fragment
turns a given extensional heap He into H’e assuming the lo-cations p, q hold the expressions t, u
• Extensional heap operations have fixed interpretations in the theorem provers
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 15/27
p.f=1; q.g=2
H’e = update(update(He, fieldf(t), 1), fieldg(u), 2)
• The intensional heap is described by a history of method invocations
• ms(Hi, ) represents the sequence of method calls followed by a call to m( )
• Consider for example the execution of the following code fragment assuming that t is the expression held in location o
• Intensional heap operations are uninterpreted functions
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 16/27
¹x¹x
ArrayList a = new ArrayList(); a.Add(o);
H’i = Adds (ArrayLists(Hi,a),a,t)
• A symbolic state is a 5-tuple S = (O, A, He, Hi, X) – O ½ ObjectId– A is a stack of activation records associated with a method, a program
counter, as well as a store for the formal parameters and local vari-ables
– He and Hi are expression denoting the extensional heap and the inten-sional heap respectively
– X is an object expression denoting the current exception
• S+1 is like S except that the program counter has been in-cremented
• The set of all symbolic states is called State
FrameworkSymbolic State
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 17/27
• A constraint on a symbolic state is a pair C = (BG, PC)
• BG is the static background only depending on the pro-gram declarations
• PC is the dynamic path condition built up during symbolic execution
• The set of all constraints is called Constraints
• A constrained state is a pair (S, C)
Framework Constraints
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 18/27
• The one-step relation
describes the effect of the current instruction from a given constrained state (S, C)
• Most instructions are handled in the standard way– A method call instruction pushes a new activation record onto the pro-
gram stack
• The interesting cases are object creation, conditional branch, member access, assume and assert
Framework Symbolic Evaluation
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 19/27
! µ (State £ Constraints) £ (State £ Constraints)
• New object creation of type T– Let o 2 ObjectId – O(S) be a fresh object identifier– (S, C) ! (S’, C Æ type(o) = T) where S’ is like S+1 except that He(S) is re-
placed with
– Where f1, , fn are the fields of T and v1, , vn default values
• Conditional branch with a condition c and label l– If (S, C Æ c) is feasible, then (S, C) ! (S’, C Æ c) where the program
counter in S’ is set to l– If (S, C Æ : c) is feasible, then (S, C) ! (S+1, C Æ : c)
Framework Symbolic Evaluation
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 20/27
update( update (h,fieldf1 (o), v1) , fieldfn (o), vn)
• Member access with target expression t and result loca-tion r – If (S, C Æ t null) is feasible, then (S, C) ! (S’, C Æ t null) where S’ is
like S+1 except that the location r holds (He (S), fieldf(t))
– If (S, C Æ t = null) is feasible, then (S, C) ! (S’’, C Æ type(e) = Null-ReferenceException Æ t = null) where S’’ is like S except that the current exception X = e
• Assume.IsTrue with condition c– If (S, C Æ c ) is feasible, then (S, C) ! (S+1, C Æ c)
• Assert.IsTrue with condition c is semantically equivalent to
Framework Symbolic Evaluation
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 21/27
If (!c) throw new AssertFailedException();
• PUT can be seen as a summary, an axiom of external be-havior of method-under-test
• To summarize the set of methods M, we refine the behavior of ! relation for method calls to M
• Call to a method m 2 M– Let H’i = ms (Hi (S), ).
– If (S, C Æ mx (Hi(S), ) = void) is feasible, (S, C) ! (S’, C Æ mx(hi(S), ) = void) where S’ is S+1 except that Hi is replaced with H’i
– IF (S, C Æ mx (Hi(S), ) void) is feasible, (S, C) ! (S’’, C Æ mx(Hi (S), ) void) where S’’ is S+1 except that Hi is replaced with H’i and X = mx(hi(S), )
Framework Axioms
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 22/27
¹x¹x
¹x
¹x
¹x¹x
• PUTs generate axiom formulas for by exploring a test axiom method like we do for test case generation
• PUTs explore the method with a modified one-step relation !‘
• !‘ is like except that Assert.IsTrue(c) in a state (S, C) is treated like As-sume.IsTrue(c) and a new axiom formula 8h, .PC(C) ! c is generated and conjoined BG(C)
Parameterized Unit TestsAxioms
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 23/27
¹x
[PexMethod]void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4}
• For example
Parameterized Unit TestsAxioms
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 24/27
[PexMethod]void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4}
• For example
• Exploring TestAdd generates the following universally quantified formula as an axiom
Parameterized Unit TestsAxioms
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 25/27
[PexMethod]void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4}
8 h,a,o. (a = null Ç subtype(type(a), ArrayList)) Æ a null ! getItemr ( Adds (getCounts (h,a),a,o), a, getCountr (h,a)) = o
Conclusion
Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 26/27
• The authors presented the concept of parameter-ized unit tests, a generalization of unit tests
• PUTs can be turned into axioms, which summarize three different aspects of the methods’ behavior– The axioms can be reused during symbolic execution