Next Generation Developer Testing: Parameterized Testing
Transcript of Next Generation Developer Testing: Parameterized Testing
Next Generation Developer Testing: Parameterized Testing
Tao Xie Department of Computer Science
University of Illinois at Urbana-Champaign Email: [email protected]
http://taoxie.cs.illinois.edu/
In Collaboration with Microsoft Research
2
Introduction
Developer (aka unit) testing – a widely adopted practice forensuring high quality software
A conventional unit test (CUT): small program with test inputs and test assertions
void AddTest() {
HashSet set = new HashSet();
set.Add(7);
set.Add(3);
Assert.IsTrue(set.Count == 2);
}
Test Scenario
Test Assertions
Test Data
3
Parameterized Unit Tests (PUTs)
Recent advances in unit testing introduced PUTs
void AddSpec(int x, int y)
{
HashSet set = new HashSet();
set.Add(x);
set.Add(y);
Assert.AreEqual(x == y, set.Count == 1);
Assert.AreEqual(x != y, set.Count == 2);
}
PUTs separate two concerns:• Specification of externally visible behavior (assertions)
• Selection of internally relevant test inputs (coverage)
4
Parameterized Unit Tests (PUTs)
More beneficial than CUTs
• Help describe behaviors for all test arguments
Address two main issues with CUTs
• Missing test data required for exercising importantbehaviors
Low fault-detection capability
• Including test data that exercises the same behaviour
Redundant unit tests
5
An Example using IntStack
public void CUT1() {int elem = 1;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {int elem = 30;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
public void CUT3() {int elem1 = 1, elem2 = 30;IntStack stk = new IntStack();stk.Push(elem1);stk.Push(elem2);Assert.AreEqual(2, stk.Count());
}
CUT1 and CUT2 exercise push with different test data
CUT3 exercises push when stack is not empty
Two main issues with CUTs:
Fault-detection capability issue: undetected defect where things go wrong when passing a negative value to push
Redundant test issue: CUT2 is redundant with respect to CUT1
6
An Example using IntStack
public void CUT1() {int elem = 1;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {int elem = 30;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
public void CUT3() {int elem1 = 1, elem2 = 30;IntStack stk = new IntStack();stk.Push(elem1);stk.Push(elem2);Assert.AreEqual(2, stk.Count());
}
No need to describe test data
• Generated automatically
Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {Assume.IsTrue(elem != null);IntStack stk = new IntStack();for(int i in elem)
stk.push(elem);Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT
Parameterized Unit Tests areAlgebraic Specifications
• A Parameterized Unit Test can be read as a universally quantified, conditional axiom.
void ReadWrite(Storage s, string name, string data) {Assume.IsTrue(s!=null && name!=null && data!=null);s.WriteResource(name, data);var readData = s.ReadResource(name);Assert.AreEqual(data, readData);
}
Storage s, string name, string data:s ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒equals(ReadResource(WriteResource(s,name,data).state, name).retval,data)
Parameterized Unit Testingis going mainstream
Parameterized Unit Tests (PUTs) commonly supported by various test frameworks
• .NET: Supported by .NET test frameworks
– http://www.mbunit.com/
– http://www.nunit.org/ …
• Java: Supported by JUnit 4.X
– http://www.junit.org/
Generating test inputs for PUTs supported by tools
• .NET: Supported by Microsoft Visual Studio 2015 IntelliTest
– Formerly Microsoft Research Pex: http://research.microsoft.com/pex/
• Java: Supported by Agitar AgitarOne
– http://www.agitar.com/
Parameterized Tests in JUnit
9https://github.com/junit-team/junit/wiki/Parameterized-tests
JUnit Theories
10https://github.com/junit-team/junit/wiki/Theories
Assumptions and Assertions
void PexAssume.IsTrue(bool c) {if (!c)
throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) {
if (!c)throw new AssertionViolationException();
}
• Assumptions and assertions induce branches
• Executions which cause assumption violations are ignored, not reported as errors or test cases
Test Data Generation
• Human– Expensive, incomplete, …
• Brute Force– Pairwise, predefined data, etc…
• Semi - Random– Cheap, Fast– “It passed a thousand tests” feeling
• Dynamic Symbolic Execution: IntelliTest/Pex, SAGE, CUTE, …
– Automated white-box– Not random – Constraint Solving
13
void CoverMe(int[] a){
if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)
throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Generates test data systematically
Done: There is no path left.
Background: DSE
Pex4Funhttp://pex4fun.com/
Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
1,703,247 clicked 'Ask Pex!'
Pex4Fun
• Click http://pex4fun.com/default.aspx?language=CSharp&sa
mple=_Template
• Copy and modify the following code snippet in the code
editing box (or simply click here)using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;
[PexClass]
public class Program
{
[PexMethod]//[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]
public static string testMethod(int x, int y)
{
PexAssume.IsTrue(y >= 0);//replace here with your assumption
//... enter your code under test here
//if (x == 10000) throw new Exception();
PexAssert.IsTrue(y >= 0);//replace here with your assertion
return PexSymbolicValue.GetPathConditionString();
}
}
https://sites.google.com/site/teachpex/Home/pex-usage-tips
Microsoft Visual Studio 2015 IntelliTest
16https://msdn.microsoft.com/en-us/library/dn823749.aspx
Microsoft Visual Studio 2015 IntelliTest
17https://msdn.microsoft.com/en-us/library/dn823749.aspx
18
Recall: An Example using IntStack
public void CUT1() {int elem = 1;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
Three CUTs
public void CUT2() {int elem = 30;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());
}
public void CUT3() {int elem1 = 1, elem2 = 30;IntStack stk = new IntStack();stk.Push(elem1);stk.Push(elem2);Assert.AreEqual(2, stk.Count());
}
No need to describe test data
• Generated automatically
Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {Assume.IsTrue(elem != null);IntStack stk = new IntStack();for(int i in elem)
stk.push(elem);Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT
19
Test Generalization: CUTs PUT
Major Steps
• S1: Parameterize
• S2: Generalize Test Oracle
• S3: Add Assumptions
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
Example00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
20
SettingsGroup class of NUnit with the SaveSetting method under test
An Existing CUT00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
21
00: //tg is of type SettingsGroup
01: [Test]
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
07: }
Existing CUT
SettingsGroup class of NUnit with the SaveSetting method under test
?
Issues with Existing CUT
22
Only CUT for verifying SaveSetting method• Does not verify the behavior for the types bool and enum
• Does not cover the true branch in Line 8
Does test generalization addresses these two issues
…
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) { …
23
S1 - Parameterize Promote all primitive values as arguments
• name of setting as a parameter of type string
• string “TOM” and int 5 as a parameter of type object
helping IntelliTest/Pex generate concrete values based on theconstraints encountered in different paths
Promote non-primitive objects such as receiver objectsas arguments
helping IntelliTest/Pex generate object states for the receiverobjects that can cover additional paths
//Original CUT
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");….
//New PUT
02: public void TestSave(
SettingsGroup st,
string sn, object sv){
….
24
S2 – Generalize Test Oracle
Replace the constant value, “TOM” and 5, withthe relevant parameter of the PUT
//Original CUT
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
….
//New PUT
02: public void TestSave(SettingsGroup st,
string sn, object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
//New PUT
02: public void TestSave(SettingsGroup st,
string sn, object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
25
S3 – Add Assumptions
IntelliTest/Pex requires guidance in generatinglegal values for the parametersE.g., Add the tag PexAssumeUnderTest (PAUT) with the parameter, i.e.,
generated value should not be null
//New PUT
02: public void TestSave([PAUT]SettingsGroup
st, [PAUT]string sn, [PAUT] object sv){
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
26
Example Summary
Resulting PUT
Result of S1 Parameters “sn” and “sv”
Result of S2 Line 4: “st.GetSetting(sn)”
Result of S3 Parameter “st”
00: //PAUT: PexAssumeUnderTest
01: [PexMethod]
02: public void TestSave([PAUT]SettingsGroup st,
[PAUT] string sn, [PAUT] object sv) {
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));
05: }
27
Example - Test Generalization Results
Achieved 10% branch cov
Required 2 method calls
with values of type string and
int
2: public void
TestSave([PAUT]SettingsGroup st,
[PAUT]string sn, [PAUT] object sv){
3: st.SaveSetting(sn, sv);
4: PexAssert.AreEqual
(sv,st.GetSetting(sn));
2: public void TestSettingsGroup() {
3: tg.SaveSetting("X",5);
4: tg.SaveSetting("NAME","Tom");….
Achieved 90% branch cov
Required only 1 method call,
sufficient to test for all types
(parameter of type object)
Original CUT New PUT
EXAMPLE PATTERNSFOR PARAMETERIZED UNIT TESTS
http://research.microsoft.com/pex/patterns.pdf
Pattern4A
• Assume, Arrange, Act, Assert
[PexMethod]void Add(List target, T value) {
PexAssume.IsNotNull(target); // assumevar count = target.Count; // arrangetarget.Add(value); // actAssert(target.Count == count + 1)//assert
}
PatternRoundtrip
• For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) {string s = value.ToString();int parsed = int.Parse(s);Assert.AreEqual(value, parsed);
}
PatternState Relation
• Observe a state change
void ContainedAfterAdd(string value) {var list = new List<string>();list.Add(value);Assert.IsTrue(list.Contains(value));
}
PatternCommutative Diagram
• Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) {string z1 = Multiply(x, y).ToString();string z2 = Multiply(x.ToString(), y.ToString());PexAssert.AreEqual(z1, z2);
}
Behind the Scene of Code Hunt
Secret Implementation
class Secret {public static int Puzzle(int x) {
if (x <= 0) return 1;return x * Puzzle(x-1);
}}
Player Implementation
class Player {public static int Puzzle(int x) {
return x;}
}
class Test {public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);
}}
behaviorSecret Impl == Player Impl
34
It’s a game!
• iterative gameplay• adaptive• personalized• no cheating• clear winning criterion
code
test cases
Bishop et al. Code Hunt: Experience with Coding Contests at Scale. ICSE 2015, JSEEThttp://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf
Next Generation Developer Testing: Parameterized Testing
Microsoft Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test GeneralizationCUTs PUTs
PUT Patterns PUTs
…
Thank You
37
https://sites.google.com/site/teachpex/http://research.microsoft.com/pexhttp://taoxie.cs.illinois.edu/
Collaboration is Welcome!
Email: [email protected]
Next Generation Developer Testing: Parameterized Testing
Microsoft Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test GeneralizationCUTs PUTs
PUT Patterns PUTs
…
http://taoxie.cs.illinois.edu/[email protected]
https://sites.google.com/site/teachpex/
PatternRoundtrip
• For an API f(x), f-1(f(x)) = x for all x
void PropertyRoundtrip(string value) {var target = new Foo();target.Name = value;var roundtripped = target.Name;Assert.AreEqual(value, roundtripped);
}
http://research.microsoft.com/pex/patterns.pdf
PatternSanitized Roundtrip
• For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) {var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);Assert(normalized == roundtripped);
}
http://research.microsoft.com/pex/patterns.pdf
PatternReachability
• Indicate which portions of a PUT should be reachable.
[PexExpectedGoals]public void InvariantAfterParsing(string input){
ComplexDataStructure x;bool success = ComplexDataStructure.TryParse(
input, out x);PexAssume.IsTrue(success);PexGoal.Reached(); x.AssertInvariant();
}
http://research.microsoft.com/pex/patterns.pdf
PatternRegression Tests
• Generated test asserts any observed value
– Return value, out parameters, PexGoal
• When code evolves, breaking changes in observable will be discovered
int AddTest(int a, int b) {return a + b; }
void AddTest01() {var result = AddTest(0, 0);Assert.AreEqual(0, result);
}http://research.microsoft.com/pex/patterns.pdf
PatternSame Observable Behavior
• Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.
public void ConcatsBehaveTheSame(string left, string right)
{PexAssert.AreBehaviorsEqual(
() => StringFormatter.ConcatV1(left, right),() => StringFormatter.ConcatV2(left, right));
}
http://research.microsoft.com/pex/patterns.pdf
PatternAllowed Exception
• Allowed exception -> negative test case
[PexAllowedException(typeof(ArgumentException))]void Test(object item) {
var foo = new Foo(item) // validates item
// generated test (C#)[ExpectedException(typeof(ArgumentException))]void Test01() {
Test(null); // argument check}
http://research.microsoft.com/pex/patterns.pdf
45
S4 – Add Factory Method IntelliTest/Pex requires guidance handling non-
primitive objectse.g., Add factory methods that generate instances of non-primitive objects
00: //PAUT: PexAssumeUnderTest
01: //MSS: MemorySettingsStorage (class)
01: [PexFactoryMethod(typeof(MSS))]
02: public static MSS Create([PAUT] string[] sn, [PAUT]object[] sv) {
03: PexAssume.IsTrue(sn.Length == sv.Length);
04: PexAssume.IsTrue(sn.Length > 0);
05: MSS mss = new MSS();
06: for(int count = 0; count < sn.Length; count++) {
07: mss.SaveSetting(sn[count], sv[count]);
08: }
09: return mss;
10: }
• Help create different object states for MSSThummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
46
S5 – Add Mock Object
Pex faces challenges in handling code that
interacts with external environment, e.g., file system• Write mock objects to assist Pex and to test features in isolation
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
47
Empirical Study
RQ1: Branch coverage• How much higher percentage of branch coverage is achieved
by retrofitted PUTs compared to existing CUTs?
RQ2: Defect detection• How many new defects (that are not detected by existing
CUTs) are detected by PUTs and vice-versa?
RQ3: Generalization effort• How much effort is required for generalizing CUTs to PUTs?
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
Study Setup Three Subject Applications
48
Subjects Downloads Code Under Test
# Classes # Methods KLOC
Test Code
# Classes # CUTs KLOC
NUnit 193,563 9 87 1.4 9 49 0.9
DSA 3,239 27 259 2.4 20 337 2.5
QuickGraph 7,969 56 463 6.2 9 21 1.2
TOTAL 92 809 10 38 407 4.6
High downloads count
Lines of code under test: 10 KLOC
Number of CUTs: 407Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
Study Setup – cont.
49
Conducted by the first and second authors
• No prior knowledge of subjects
• Two years of experience with PUTs and Pex
• Retrofitted 407 CUTs (4.6 KLOC) as 224 PUTs (4.0 KLOC)
Generated three categories of CUTs• C1: Existing CUTs
• C2: CUTs generated from PUTs
• C3: Existing CUTs + RTs (tests generated using Randoop) Help show that benefits of generalization cannot be achieved by
generating tests randomly
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
RQ1: Branch Coverage
50
Subjects # RTs Branch Coverage
CUTs (%) CUTs + RTs (%) PUTs (%)
Overall Increase
(%)
MaximumIncrease
(%)
NUnit 144 78 78 88 10 52
DSA 615 91 91 92 1 1
QuickGraph
3628 87 88 89 2 11
CUTs + RTs: Added 144, 615, and 3628 tests using Randoop
Branch coverage increased by 0%, 0%, and 1%
CUTs generated from PUTs: Branch coverage increased by 10%, 1%, and 2%
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
RQ2: Defect Detection
51
RTs: CUTs generated using Randoop
Pex without generalized PUTs: CUTs generated by applying Pex on public methods
CUTs generated from PUTs: CUTs generated by applying Pex on generalized PUTs
• Detected all defects detected by the first two categories
CUTs category #Failing Tests
DSA NUnit QuickGraph
# Real Defects
Basic CUTs 0 0 0 0
RTs 90 25 738 4
Pex without generalized PUTs
23 170 17 2
CUTs generated from PUTs
15 4 0 19
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
RQ3: Generalization Effort
52
Both authors conducted comparable amount of generalization
Effort spent in hours
• NUnit: 2.8 hrs
• DSA: 13.8 hrs
• QuickGraph: 1.5 hrs
Effort is worthwhile compared to benefits of test generalization
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf