Post on 11-Jan-2017
Your systems. Working as one.
Property-based Testing and Generators
Sumant Tambe7/29/2015
Agenda
• Property-based Testing
• The need for generators
• Design/Implementation of the generator library
• Using mathematics for API design
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
8/25/2015 © 2015 RTI 2
Why should you care?
• If you write software for living and if you are serious about any of the following– Bug-free code
– Productivity• Modularity
• Reusability
• Extensibility
• Maintainability
– Performance/latency• Multi-core scalability
– and having fun while doing that!
8/25/2015 © 2015 RTI 3
8/25/2015 © 2015 RTI 4Source Google images
What’s a Property
• Reversing a linked-list twice has no effect– reverse(reverse(list)) == list
• Compression can make things smaller– compress(input).size <= input.size
• Round-trip serialization/deserialization has no effect– deserialize(serialize(input)) == input
8/25/2015 © 2015 RTI 5
Property-based Testing
• Complementary to traditional example-based testing
• Specify post-conditions that must hold no matter what
• Encourages the programmer to think harder about the code
• More declarative
• Need data generators to produce inputs
• Free reproducers! – Good property-based testing frameworks shrink inputs to
minimal automatically
• Famous: Haskell’s QuickCheck, Scala’s ScalaTest
8/25/2015 © 2015 RTI 6
A property-test in RefleX
8/25/2015 © 2015 RTI 7
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What’s the input data?
• What are the input types?
Data Generators
8/25/2015 © 2015 RTI 8
Lua has less syntactic noise than C++. So, code examples are in Lua
Data Generators
8/25/2015 © 2015 RTI 9
• Simplest data generator = math.random
math.randomseed(os.time())
local r = math.random(6)
print(r) -- one of 1, 2, 3, 4, 5, 6
r = math.random(20, 25)
print(r) -- one of 20, 21, 22, 23, 24, 25
Data Generators
8/25/2015 © 2015 RTI 10
• A boolean generatorfunction boolGen()
return math.random(2) > 1
end
• An uppercase character generator
function upperCaseGen() -- A=65, Z=90
return string.char(math.random(65, 90))
end
• An int16 generatorfunction int16Gen()
return math.random(MIN_INT16, MAX_INT16)
end
Data Generators
8/25/2015 © 2015 RTI 11
• A range generatorfunction rangeGen(lo, hi)
return math.random(lo, hi)
end
• Inconvenient because args must be passed every time
Use state!
Closure Recap
8/25/2015 © 2015 RTI 12
• A simple closurelocal f = function ()
return boolGen()
end
print(f()) -- true or false
• Another closurelocal a = 20
local f = function ()
print(a)
end
f() -- 20
a = 30
f() -- 30
Closures capture state
8/25/2015 © 2015 RTI 13
• Back to the range generator
function rangeGen(lo, hi)
return function()
return math.random(lo, hi)
end
end
local f = rangeGen(20, 25)
print(f()) -- one of 20, 21, 22, 23, 24, 25
Closures capture state
8/25/2015 © 2015 RTI 14
• The oneOf generator
function oneOfGen(array)
local len = #array –- get length of the array
return function()
return array[math.random(1, len)]
end
end
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local f = oneOfGen(days)
print(f()) -- one of Sun, Mon, Tue, Wed, Thu, Fri, Sat
8/25/2015 © 2015 RTI 15
“Closures are poor man's objects; Objects are poor
man's closure”
-- The venerable master Qc Na Source Google images
Closure vs Object
• Is function a sufficient abstraction for anygenerator?
function () {
return blah;
}
• Answer depends upon your language– No. In Lua, C++, …
– Yes. In Haskell
8/25/2015 © 2015 RTI 16
The Generator Library
8/25/2015 17
local Gen = require(“generator”)
local rangeGen = Gen.rangeGen(20,25)
for i=1, 5 do
print(rangeGen:generate()) –- 5 values in range(20, 25)
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
local stepperGen = Gen.stepperGen(1, 10)
for i=1, 5 do
print(stepperGen:generate()) –- 1, 2, 3, 4, 5
end
local infiniteGen = Gen.stepperGen(1, 10, 1, true)
while true do
print(infiniteGen:generate()) –- 1, 2, 3, 4, ...
end
The Generator Library
8/25/2015 18
local Gen = require(“generator”)
local charGen = Gen.uppercaseGen()
local maxLen = 64
local strGen = Gen.stringGen(maxLen, charGen)
for i=1, 5 do
print(strGen:generate()) -- 5 uppercase strings upto size 64
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
sequenceGen is quite similar
The Generator library (batteries included)
8/25/2015 © 2015 RTI 19
boolGen posFloatGen emptyGen
charGen posDoubleGen singleGen
wcharGen floatGen constantGen
octetGen doubleGen oneOfGen
shortGen lowercaseGen inOrderGen
int16Gen uppercaseGen stepperGen
int32Gen alphaGen rangeGen
int64Gen alphaNumGen seqGen
uint16Gen printableGen arrayGen
uint32Gen stringGen aggregateGen
uint64Gen nonEmptyStringGen enumGen
The Generator “Class”• Lua does not have “classes” but that’s not important
8/25/2015 © 2015 RTI 20
local Generator = {} –- an object that will serve as a class
function Generator:new(generateImpl)
local o = {}
o.genImpl = generateImpl -- A method assignment
-- some additional metatable manipulation here
return o
end
function Generator:generate()
return self.genImpl()
end
function rangeGen(lo, hi)
return Generator:new(function ()
return math.random(lo, hi)
end)
end
8/25/2015 © 2015 RTI 21Source Google images
Composing Generators
• Producing more complex Generators from one or more simpler Generators
8/25/2015 © 2015 RTI 22
Generator
map
reduce
append
where
zipMany
concatMap
take
scan
Mapping over a generator
8/25/2015 © 2015 RTI 23
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local fiboGen = Gen.fibonacciGen()
-- 0 1 1 2 3 5 8 ...
local plus1Gen = fiboGen:map(function (i) return i+1 end)
-- 1 2 2 3 4 6 9 ...
local dayGen = plus1Gen:map(function (j) return days[j] end)
-- Sun, Mon, Mon, Tue, Wed, Fri, nil, ...
for i=1, 6 do
print(dayGen:generate()) -- Sun, Mon, Mon, Tue, Wed, Fri
end
Zipping generators
8/25/2015 © 2015 RTI 24
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local pointGen =
Gen.zipMany(xGen, yGen, function (x, y)
return { x, y }
end)
for i=1, 5 do
print(pointGen:generate()) –- {0,0}, {1,2}, {3,4}, ...
end
Zipping generators
8/25/2015 © 2015 RTI 25
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local colorGen = Gen.oneOfGen({ “RED”, “GREEN”, “BLUE” })
local sizeGen = Gen.constantGen(30)
local shapeGen =
Gen.zipMany(xGen, yGen, colorGen, sizeGen,
function (x, y, color, size)
return { x = x,
y = y,
color = color,
shapesize = size }
end)
• The mandatory Shapes example
Using Data Domain Specific Language (DDSL)
8/25/2015 © 2015 RTI 26
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGen = Gen.aggregateGen(ShapeType)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end
• Use Lua DDSL to define types
Five random shapes
8/25/2015 © 2015 RTI 27
COLOR?!
Use a Collection of Generators
8/25/2015 © 2015 RTI 28
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGenLib = {}
shapeGenLib.x = Gen.stepperGen(0, 100)
shapeGenLib.y = Gen.stepperGen(0, 200, 2)
shapeGenLib.color = Gen.oneOf({ "RED", "GREEN", "BLUE" })
shapeGenLib.shapesize = Gen.rangeGen(20, 30)
local shapeGen = Gen.aggregateGen(ShapeType, shapeGenLib)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end
Five sensible shapes
8/25/2015 © 2015 RTI 29
In not so distant future…
8/25/2015 © 2015 RTI 30
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in • MEP Message Emulation Processes (i.e., generate data)• MCT Generator expressions may be specified directly in the UI
8/25/2015 © 2015 RTI 31
The Mathematics of the Generator API
Generators
• General design principal
– Create a language to solve a problem
– API is the language
– Reuse parsing/compilation of the host language (duh!)
• The “language” of Generators
– Primitive values are basic generators
– Complex values are composite generators
– All APIs result in some generator• Especially, map, zip, reduce, etc.
– I.e., generators form an algebra
8/25/2015 © 2015 RTI 32
Algebra
A first-class abstraction
+Compositional* API
8/25/2015 © 2015 RTI 33
*Composition based on mathematical concepts
8/25/2015 © 2015 RTI 34
What Math Concepts?
• Algebraic Structures from Category Theory–Monoid
–Monad
– Functor
–Applicative
–Group
– Endofunctor
• Where to begin?–Haskell
– But innocent should start with Scala perhaps
8/25/2015 © 2015 RTI 35
Does it have anything to do with DDS?
Yes!
Composable API for DDS exists. It’s Rx4DDSAlso, RPC-over-DDS (futures). Both are humble beginnings
And remember, sky is the limit
Next Items
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
• (Back to) Testing RefleX API properties
8/25/2015 © 2015 RTI 36
Constraints in Data Models
• Database Schemas
• XSD Restrictions
8/25/2015 © 2015 RTI 37
CREATE TABLE products (
product_no integer UNIQUE NOT NULL,
name text,
price numeric CHECK (price > 0)
);
<xs:element name="age">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:minInclusive value="0"/>
<xs:maxInclusive value="120"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="prodid">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:pattern value="[0-9][0-9][0-9]"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
Constraints in Data Models
• STANAG 4607– Ground Moving Target Indicator
• Object Constraint Language– Defined by OMG: part of the UML standard
– Huge!
8/25/2015 © 2015 RTI 38
In not so distant future…
8/25/2015 © 2015 RTI 39
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in • MEP Message Emulation Processes (i.e., generate data)• MCT Generator expressions may be specified directly in the UI
General Syntax for Data Generation
• Mathematical Set Builder Notation
• Suggested Generator Syntax
– [ x | x <- stepperGen() ]
– [ 2*x | x <- fibonacciGen() ]
– [ “ID” .. i | i <- stepperGen(1, 256) ]
• Translated mechanically to the Generator API– map, flatMap, etc.
– List Comprehension in disguise• Python, Haskell has it. Lua does not.
8/25/2015 © 2015 RTI 40
Value Dependencies
• List Comprehension alone is not sufficient
• What if data members have value dependencies– o.y = o.x * o.x
– o.len = length(o.somearray)
– o.min = minimum(o.somearray)
– o.max = maximum(o.somearray)
8/25/2015 © 2015 RTI 41
Syntax for Value Dependencies
• Generator Syntax seems to work for value dependencies too
• Just refer to the generator of the parent attribute
– [ a*a | a <- $(x) ]
– [ len(a) | a <- $(somearray) ]
– [ min(point.x) | point <- t,
t <- $(trajectory) ]
(Assuming trajectory is an array of points )
8/25/2015 © 2015 RTI 42
Pull Generators aren’t enough
• Pull Generators don’t handle dependencies well– At least not without state
– Requires multiple passes through state for every generated sample • Seems expensive
– Dependencies must be “interpreted” for every sample• I.e., needs to refer type description to do its job
• Dependencies can’t be “compiled” for performance
– A relatively simple and powerful alternative exists
8/25/2015 © 2015 RTI 43
How dependencies break
• The ShapeType generator asks leaf generators to produce a new value.
• Dependencies are immediately lost
8/25/2015 © 2015 RTI 44
X
Y
ShapeType
Y = X2
Circle represents a generator not state
Reactive Generators
• Work “just like DDS!”• Propagate values “downstream”
– Push-based instead of pull-based
• Subject-Observer (pub/sub) pattern– A single value is propagated to one or more dependent
attributes
• Dependent generators take the “right” action– Such as doubling the value, counting min, etc.
• map, flatmap, reduce etc. work exactly like pull generators
• Composable– Generator expressions can be mechanically translated to
reactive generator API
8/25/2015 © 2015 RTI 45
Basics of Reactive Generators
8/25/2015 © 2015 RTI 46
class Generator<T>
{
void listen(callback);
};
Where callback is a closure
callback = function (t) {
// provided by the user
}
You can register multiple callbacks
Create Listen Push Dispose
Lifecycle of a reactive (push) generator
n n
Create aPull
GeneratorSubject
transform, …
Optional
Composing Reactive Generators
• Producing more complex Generators from one or more simpler Generators
8/25/2015 © 2015 RTI 48
ReactGen
map
reduce
append
where
zipMany
concatMap
take
scan
Architecture
8/25/2015 © 2015 RTI 49
Back to Property-based Testing in RefleX
8/25/2015 © 2015 RTI 50
Round-trip property-test for ShapeType
8/25/2015 © 2015 RTI 51
void test_roundtrip_shape(const ShapeType & in)
{
ShapeType out; // empty
DDS::DynamicData dd(...); // empty
reflex::write_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• The assertion must hold for all instances.
void test_many_shapes()
{
auto generator = gen::make_aggregate_gen<ShapeType>();
for (int i = 0;i < 100; ++i) {
test_roundtrip_shapetype(gen.generate());
}
}
A generic property-test in RefleX
8/25/2015 © 2015 RTI 52
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What are the input types?
• Given a type, how to produce data?
Random Types at Compile-time
8/25/2015 © 2015 RTI 53
Random Numbers at
Compile-time
if we can generate
Random Numbers at Compile-time
• Demo
– In C
– Then in C++
8/25/2015 © 2015 RTI 54
Live code
Linear Feedback Shift Register (LFSR)
• Simple random number generator• 4 bit Fibonacci LFSR• Feedback polynomial = X4 + X3 + 1• Taps 4 and 3
8/25/2015 © 2015 RTI 55
Linear Feedback Shift Register (LFSR)
• Live code feedback polynomial (16 bit)– X16 + X14 + X13 + X11 + 1
• Taps: 16, 14, 13, and 11
8/25/2015 © 2015 RTI 56
Type Generators Demystified
• Random seed as command line #define
• Compile-time random numbers using LFSR
• C++ meta-programming for type synthesis
• std::string generator for type names and member names
• RefleX to map types to typecode and dynamic data
8/25/2015 © 2015 RTI 57
RefleX Property-test Type Generator
• “Good” Numbers– 31415 (3 nested structs)
– 32415 (51 nested structs)
– 2626 (12 nested structs)
– 10295 (15 nested structs)
– 21525 (41 nested structs)
– 7937 ( 5 nested structs)
– ...
• “Bad” Numbers– 2152 (test seg-faults) sizeof(tuple) > 2750 KB!
8/25/2015 © 2015 RTI 58
Github: https://github.com/rticommunity/rticonnextdds-reflex/blob/develop/test/generator
Thank You!
8/25/2015 © 2015 RTI 59