Post on 29-Mar-2015
Contract Checking and Automated Test Generation With Pex
Mike BarnettRSDEMicrosoft Research
Nikolai TillmannRSDEMicrosoft Research
TL51
Code ContractsFor .NET
Mike BarnettRSDEMicrosoft Research
Nikolai TillmannRSDEMicrosoft Research
AutomatedTest GenerationWith Pex
Two Tools.One Talk.
Code Contracts for .NET
announcingContract
Library in .NET 4.0
Coming soon: Tools for contracts at
string GetDescription(int x);
/// <param name=“x”>/// should be greater than zero/// </param>/// <returns>/// non-null string/// </returns>string GetDescription(int x);
How Do You Know What It Does?
string s = o.GetDescription(i);… s.Length …
Let’s Fix It For Once And All!
string GetDescription(int x) { Contract.Requires( 0 < x ); Contract.Ensures( Contract.Result<string>() != null ); …}
Design-by-Contract meets .NET!
Contracts document design decisions Preconditions, postconditions, invariants Facilitate code evolution
Contracts are executable Checked documentation Test oracles
Contracts help static verification Early error detection Clarify responsibility at API boundaries
Assertions help, but everyone has their own flavor Difficult to tool against Not seen as a contract between 2 parties
Contracts
Introduction
Mike BarnettRSDEResearch in Software Engineering
demo
class Rational {
public Rational(int n, int d) { Contract.Requires( 0 < d ); this.N = n; this.D = d; }}
What Can I Do With Contracts?
Documentation
Runtime Checking
Static Checking
Test Generation
(Pex)
Requires What must be true at method entry
Ensures What must be true at method exit
Invariants What must be true at all method exits
Assertions What must be true at a particular point
Assumptions What should be true at a particular point
What Contracts Can I Write?
Any boolean expression In your favorite programming language! Including method calls (but must
be marked Pure) Quantifiers
Contract.ForAll(0,A.Length,i => A[i] > 0); Contract.Exists(0,A.Length,i => A[i] > 0);
Contract.Result refer to the return value of the method
Contract.OldValue refer to values at method entry
What Can I Put In A Contract?
void ObjectInvariant() { Contract.Invariant( items != null );}
How Do I Write A Contract?
Features• Declarative• Language expression syntax• Type checking / IDE
• Special Encodings• Result and OldValue
public virtual int Add(object value){
Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (count == items.Length) EnsureCapacity(count + 1); items[count] = value; return count++;}
Runtime Checking
Mike BarnettRSDEResearch in Software Engineering
demo
.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret} // end of method BaseList::Add
csc/vbc/…
csc/vbc/…
.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0029 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_004b ldloc.0 ret}
How Does It Work?
ReleaseCompile
/d:CONTRACTS_FULL
ccrewrite
• Inheritance• Style checks
Executable Runtime Contract Checking
public virtual int Add(object value){ Contract.Requires( value != null );
Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );
if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++;}
.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ .locals init (int32 'Contract.Old(Count)', int32 'Contract.Result<int>()') ldarg.0 call instance int32 TabDemo.BaseList::get_Count() stloc.3 ldarg.1 ldnull ceq ldc.i4.0 ceq ldstr "value != null" call void __RewriterMethods::RewriterRequires$PST06000009(bool, string) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue IL_004d nop ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) nop nop ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br IL_0072 ldloc.0 stloc.s 'Contract.Result<int>()' br IL_007a ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldloc.3 ldc.i4.1 add ceq ldstr "Count == Contract.Old(Count) + 1" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ldloc.s V_4 ceq ldstr "Contract.Result<int>() == Contract.Old(Count)" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ret}
No silver bullet But helps catch errors earliest Best used in a focused manner
Guides development Discovers implicit assumptions Propagates assumptions
Not only explicit contracts Dereferencing null Indexing arrays Arithmetic exceptions
Static Contract Checking
What Do You Ship?
srcsrc
srcsrc
PowerLib.dll
(minimal runtime checks)
PowerLib.Contracts.dll
All contracts, no code
+
ReleaseAssemblies
Contract ReferenceAssemblies
Contract Reference Assemblies
Compile/d:CONTRACTS_FULL
ccRefGen
.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret} // end of method BaseList::Add
Contract ReferenceAssembly
.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool)} // end of method BaseList::Add
public virtual int Add(object value){ Contract.Requires( value != null );
Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );
if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++;}
[ContractClass(typeof(CloneableContract))]public interface ICloneable {
object Clone();
}
Interface Contracts
[ContractClassFor(typeof(ICloneable))]public class CloneableContract : ICloneable {
public object Clone() { Contract.Ensures( Contract.Result<object>() != null );
… }
}All classes implementing the interface inherit the contract
Contract library class enables contracts in all .NET languages No restrictions on what can be expressed
Contracts are being used in the BCL today Contract library is a core component of .NET
4.0 Same contracts used for
Runtime checking Static checking Documentation generation
Code Contracts Summary
Get Pex for Visual Studio 2010 at
Do you care about quality?Shake your code with Pex.
announcing Pex
Testing is tedious Too easy to miss cases Old tests get stale Too much legacy code −
what does it do?
Testing Before Pex
Don’t give up, Pex has arrived.
Tests for Freewith Pex
Nikolai TillmannRSDEMicrosoft Research
demo Pex
It’s painless: Just right-click your code It’s effective: Test cases for free It’s fun:
Inspect table, Save tests, file work items, or debug right away
What The Demo Showed
Code Coverage for Free with Pex Nikolai Tillmann
RSDEMicrosoft Research
demo Pex
Pex automatically generatedcomprehensive test suitewith high code coverage
Pex finds contract violationsThe generated test suite
integrates withVisual Studio Team Test
can be used asBuild Verification Test (BVT)
What The Demo Showed
A traditional Unit Test is amethod without parameters.
There Is More…
[TestMethod]void AddZeroTest(){ var r = new Rational(1,2); var q = r + 0; Assert(r == p);}
Handwritten
A Parameterized Unit Test is simply atest method that takes parameters
Pex generates traditional Unit Tests (in C#) which fix particular values
There Is More…Parameterized unit testing with pex
[PexMethod]void AddZeroTest( Rational r){ var q = r + 0; Assert(r == p);}
[TestMethod]void AddZeroTest01(){ var r = new Rational(1,2); this.AddZeroTest(r);}
Pex
calls
Automatically generatedHandwritten
How to test this code?(Actual code from .NET base class libraries)
A Tester's Nightmare: ResourceReader
A Tester's Nightmare: ResourceReader
Parameterized Unit Tests
[PexAllowedException(typeof(ArgumentException))][TestClass]public partial class ResourceReaderTest { [PexMethod] public void ReadEntries(byte[] data) { if (data == null) return; fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}
Parameterized Unit Testing
Nikolai TillmannRSDEMicrosoft Research
demo Pex
Pex does not guess. Pex does not generate random inputs, enumerate all possible values, or make you write test input generators
Instead, Pex analyzes your .NET code. Pex partitions inputs into equivalence classes One equivalence class per branching behavior Test inputs computed by Z3, the world class
constraint solver for program analysisfrom Microsoft Research
Precise inter-procedural, path-sensitive analysis
As a result, you geta small test suite with high code coverage
Pex Understands The Code
Pex generates small test suites with high code coverage and bug reports for free
Use previously generated test suites asBuild Verification Test (BVT)
Reduce test maintenance costsby parameterized unit testing
Pex has been used in Microsoftto test core .NET components
Download Pex for Visual Studio 2010 CTP on
http://msdn.microsoft.com/DevLabs
Pex Summary
Code Contracts for .NET:preconditions, postconditions, invariants http://research.microsoft.com/Contracts/
Pex: test generation for .NET http://research.microsoft.com/Pex/
Summary
Research in Software Engineering
Evals & Recordings
Please fill
out your
evaluation for
this session at:
This session will be available as a recording at:
www.microsoftpdc.com
Please use the microphones provided
Q&A
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.