A Generic Method for Statistical Testing
-
Upload
rajah-decker -
Category
Documents
-
view
28 -
download
0
description
Transcript of A Generic Method for Statistical Testing
![Page 1: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/1.jpg)
ISSRE 2004 1
A Generic Method for Statistical Testing
A. Denise, M.-C. Gaudel and S.-D. Gouraud
{denise, mcg, gouraud}@lri.fr
L.R.I, Université Paris XI, 91400 ORSAY, FRANCE
![Page 2: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/2.jpg)
2
Outline
Context Statistical testing and test quality Combinatorial Structures
Our new approach of Statistical Testing Draw paths Optimise test quality
Validation of our approach Application to structural statistical testing Experimental results
Conclusion and Futures
![Page 3: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/3.jpg)
3
Combinatorial structures specification The specification of a class of Combinatorial
Structures is a set of production made from basic objects: ε and Atom resp. of size 0 and 1 constructions: union(+), product(x), sequence, etc. cardinality constraints
Example: Complete (non empty) binary tree: T= L+ TxT
where L is an Atom that represents some Leaf
Complexity of counting and generating Linear in our case n log n in general for combinatorial structures of size n
![Page 4: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/4.jpg)
4
Statistical (or random) Testing
Selecting test data uniformly (or based on an operational profile) at random from the input domain of the program
It is possible to test more intensively than with the other methods
Bad coverage of particular cases like exception cases
Solution? A combination with another testing method [Thévenod-Fosse,Waeselynck, LAAS,1991].
![Page 5: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/5.jpg)
5
Quality of statistical testing [TF,Wa]
Let E the set of elements to be covered N the number of tests
The test quality qN is the weakest probability that any element of E has to be covered when N tests are exercised
qqNN =1-(1- p =1-(1- pminmin))NN
where pmin = min{p(e), eE}
To maximise qN, we need to maximise pmin A solution (not always possible):
Uniform drawing among E
![Page 6: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/6.jpg)
6
Our approach of Statistical Testing
Random drawing of pathsRandom drawing of paths
The set of paths of a graph can be easily represented by a combinatorial structure specification
Random generation with a linear complexity
2 steps:1) Draw an adequate set of paths
2) Find input data which ensure the execution of these paths
(of length ≤ n)
![Page 7: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/7.jpg)
7
Graph and Combinatorial Structures
Atoms= edges
Sequence of edges= paths
S= v.S + v.e0.C.e7
C= e1.e2 + e3.B.e6
B= e4.I + ε
I= e5.B
INIT
EXIT
I0
C1
I2
I5
I4B3
v
e1
e2
v
e0
e3e5
e4
e6
e7
S C
![Page 8: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/8.jpg)
8
Generation: counting
0 1 2 3 4 5 6 7 8 9 … n
C 0 0 2 0 1 0 1 0 1 0 … …
B 1 0 1 0 1 0 1 0 1 0 … …
I 0 1 0 1 0 1 0 1 0 1 … …
S 0 0 0 0 0 2 2 3 3 4 … …
From S, there are 3 paths of length 7
![Page 9: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/9.jpg)
9
Generation: drawing
Length=7
INIT
EXIT
I0
C1
I2
I5
I4B3
v
e1
e2
v
e0
e3e5
e4
e6
e7vvve0e1e2e7 vvve0e3e6e7 ve0e3e4e5e6e7
S7
vS6 ve0C4e7
ve0e3B2e6e7vvS5
vvve0C2e7
S= v.S + v.e0.C.e7
C= e1.e2 + e3.B.e6
B= e4.I + εI= e5.B
vve0C3e7
vvvS3
? ?
1
1
0
0
1
2/3 1/3
1/21/2
![Page 10: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/10.jpg)
10
Combinatorial Structures & Statistical Testing
If the criterion consists in covering a set of paths: the corresponding combinatorial structure specification is built
Examples: all paths passing through the edge a,all paths passing through the node B3 then the node I4…
If the criterion consists in covering a set of elements: ???Examples:
all nodes, all edges …
How a uniform drawing among paths can ensure a good test quality for the coverage of elements of the criterion?
![Page 11: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/11.jpg)
11
Drawing paths
Let be N the number of tests, we want to:
1. Pick, with a suitable distribution, N elements e1,…,eN among the elements to be covered
2. For each ei, draw uniformly a path (of length ≤ n) among those which pass through this element ei.
![Page 12: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/12.jpg)
12
Example: all red nodes
E={I2,I0,I4,I5} 5 paths of length 11. Uniform distribution
p(I2)= 1/4 +1/41/5 +1/40 +1/41/5 =7/20 = 0.35
And: p(I4)=11/20, p(I0)=1, p(I5)=1
pmin=p(I2)= 0.35
pmin is not optimal!
INIT
EXIT
I0
C1
I2
I5
I4B3
![Page 13: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/13.jpg)
13
Example: all red nodes
E={I2,I0,I4,I5}5 paths of length 11.Distribution
p1(I0)=p1(I5)=0
p1(I2)=p1(I4)=0.5
pmin=0.5
How could we automaticallymaximise pmin?
INIT
EXIT
I0
C1
I2
I5
I4B3
![Page 14: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/14.jpg)
14
Probability of an element
The probability p(e) of the element e to be exercised by one run is:
p(e)=p1(e)+p2(e) Probability to draw this element (step 1): p1(e) Probability to draw a path passing through this
element (step 2):
where e’ is the element drawn in step 1,
c(e’) is the number of paths passing through e’
c(e,e’) is the number of paths passing through e and e’
e'e)'(1)'c(
)',c()(2 ep
eee
ep
![Page 15: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/15.jpg)
15
A method to calculate the distribution
To optimise the test quality, we have to maximise pmin.
However for all e in E,
e'e)'(1)'c(
)',c()(1min ep
eee
epp
![Page 16: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/16.jpg)
16
Maximise pmin under these constraints
This optimisation problem is solved by a Simplex and the p1(ei) are deduced.
)(1...)2(1)1(11
)(1)(
),(...)1(1)1(
)1,(min
...
)(1)(
),1(...)1(1)1(
)1,1(min
e Epepep
e Epe Ec
e Ee Ecep
ec
ee Ecp
e Epe Ec
e Eecep
eceecp
Spmin=
![Page 17: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/17.jpg)
17
Example: all red nodes
Uniform distribution on paths5 paths of length 11
p1(I2)=p1(I4)=0.5
p1(I0)=p1(I5)=0
pmin=0.5
INIT
EXIT
I0
C1
I2
I5
I4B3
![Page 18: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/18.jpg)
18
From paths to Input Data
Find the input data that will
cause the execution of each drawn paths
In case of finite models: FSA, FSM Input data= sequence of inputs labelling the
edges + additional inputs (for observation)
![Page 19: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/19.jpg)
19
From paths to Input Data
General case of infinite models: any description including non-trivial data types and guards like EFSM, state-charts, CFG
Build the path predicates Solve them (semi-decidable problem)
Input data = any data satisfying the predicate
![Page 20: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/20.jpg)
20
Experimental validation
Application to Structural Statistical testing Prototype AuGuSTe
Same set of programs and mutants as Thévenod-Fosse, Waeselynck and Crouzet
Compare the detection power of our approach (drawing paths and solving the predicates) with theirs (building the input distribution explicitly)
Experimental results are quite similar
![Page 21: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/21.jpg)
21
Experiences with qN=0.9999
#lines #pathsCoverage
criterion CCardinality of C
Fct1 30 17 All paths 17
Fct2 43 9 All paths 9
Fct3 135 33 All paths 33
Fct4 77 ∞ All branches 41
Fct4: more than 1016 paths of length ≤ 234 long predicates
![Page 22: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/22.jpg)
22
Mutation scores
FCT1 FCT2 FCT3 FCT4
TFWaCr
Min
1 1
1 0.9898
Ave 1 0.9901
Max 1 0.9915
AuGuSTe
Min
1 1
0.9951 0.9854
Ave 0.9989 0.9854
Max 1 0.9854
More than 10000 runs performed on 2914 mutants Fct3: non independence of the test experiments
![Page 23: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/23.jpg)
23
Conclusion
New application of some results in combinatorics to testing
A generic and automated statistical testing method
Good experimental results for structural statistical testing
Quite similar than [TFWaCr] It is very likely that the method scales up well
![Page 24: A Generic Method for Statistical Testing](https://reader035.fdocuments.in/reader035/viewer/2022062314/5681352f550346895d9c97b0/html5/thumbnails/24.jpg)
24
Future Work
Combinatorial structures tools can deal with more complex languages
More elaborated combinatorial structures
Elimination of some major source of unfeasible paths
Random generation based on Boltzmann models [DuFlLoSc]
The bound of the length of considered paths can be avoided