Test Generation based on Abstraction and Test purposes to ...€¦ · 3 Process application The...
Transcript of Test Generation based on Abstraction and Test purposes to ...€¦ · 3 Process application The...
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Test Generation based on Abstraction and Test
purposes to complement Structural tests
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson
A-MOST 2010 - April 6, 2010 - Paris
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 1/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Context
Model-Based Testing
MBT from static selection criterion
MBT from dynamic selection criterion
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 2/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Context
Model-Based Testing
MBT from static selection criterion
MBT from dynamic selection criterion
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 2/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Problem and Motivations
Motivations
Complete the tests from static criteria with tests fromdynamic criteria
Evaluate the coverage of the generated tests
Problem
Combinatorial explosion of the size of an LTS
Solution
Abstraction to reduce the size of the state space
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 3/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Outline
1 Problem and Motivations
2 Process overview
3 Process applicationThe robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
4 Experimental resultsAbstraction computationTest generation
5 Conclusion and Future works
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 4/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Our process
Differences
Introduction of an abstraction computation
Replace the model by an abstraction
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 5/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
A conveying system : Robot
Specifications
A gripper moves parts in two dimensions
3 sort of parts : T1, T2, T3
Operations : Arrival, Load, Unload, Up, Down, Left, Right,LeftEvac, RightEvac
Unloading rules : T1 left, T2 right, T3 regardless
T3 parts : Left unloading priority
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 6/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
A test purpose for Robot
What is a test purpose ?
Description of a test intention
Combine operation calls and states to reach
Formalization : state machine or regular expression
Left unloading priority
true G = T3 ∧ LE 6= free true true G = T3 trueLeftEvac $op Unload
$op $op
$op Right
q0 q1 q2 q3 q4 qf
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 7/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Definition of an abstraction
What is an abstraction ?
Gives a thumbnail of the model
Simulates executions of the model
Abstraction computation with GeneSyst (Bert,Potet,Stouls)
Disjunction of state predicates defined by user
One predicate ⇐⇒ one symbolic state of the abstraction
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 8/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
The GeneSyst tool
Principles
; How does GeneSyst compute an abstraction ?
Predicate abstractionCompuation of weakest precondition and resolution ofsatisfiability problems with prover technologyProof failure for a transition ⇒ Transition is kept in theabstractionFilter transitions with CLP
; How to define state predicates ?
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 9/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Defining a set of abstraction predicates
Predicates which reveal changes in test purposeWhich ones do we propose ?
Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space
Left unloading priority
($op)+; (LE ∈ {T1,T3} ∧ G = T3)
. Right
. LeftEvac
. ($op)+; (G = T3)
. Unload
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Defining a set of abstraction predicates
Predicates which reveal changes in test purposeWhich ones do we propose ?
Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space
LE ∈ {T1,T3}G ∈ T3
Left unloading priority
($op)+; (LE ∈ {T1,T3} ∧ G = T3)
. Right
. LeftEvac
. ($op)+; (G = T3)
. Unload
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Defining a set of abstraction predicates
Predicates which reveal changes in test purposeWhich ones do we propose ?
Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space
LE = free
G = free
Left unloading priority
($op)+; (LE ∈ {T1,T3} ∧ G = T3)
. Right
. LeftEvac
. ($op)+; (G = T3)
. Unload
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Defining a set of abstraction predicates
Predicates which reveal changes in test purposeWhich ones do we propose ?
Present used to define target states in test purposeInvolving on variables modified by all operations of test purposeDefining a partition of the state space
LE {free},{T2},{T1,T3}G {free},{T1,T2},{T3}
Left unloading priority
($op)+; (LE ∈ {T1,T3} ∧ G = T3)
. Right
. LeftEvac
. ($op)+; (G = T3)
. Unload
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 10/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Abstraction computation
LE {free},{T2},{T1,T3}G {free},{T1,T2},{T3}
QInit
(Deg = l & Dt = l)
Init
ApEdD
Rg
(Deg = l & Dt = T3)
C
(Deg = l & Dt : {T1, T2})
C
ApEdMRg
(Deg : {T1, T3} & Dt = l)
Dc_g
Dc_d
ApEdMRd
Dc_g
Eg
ApEdD
Rg
(Deg : {T1, T3} & Dt = T3)
C
(Deg : {T1, T3} & Dt : {T1, T2})
C
Eg
Dc_d
ApEdMRd
Eg
Dc_d
ApEdMRd
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 11/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Synchronization between the abstraction and the TP
Abstraction
QInit
(Deg = l & Dt = l)
Init
ApEdD
Rg
(Deg = l & Dt = T3)
C
(Deg = l & Dt : {T1, T2})
C
ApEdMRg
(Deg : {T1, T3} & Dt = l)
Dc_g
Dc_d
ApEdMRd
Dc_g
Eg
ApEdD
Rg
(Deg : {T1, T3} & Dt = T3)
C
(Deg : {T1, T3} & Dt : {T1, T2})
C
Eg
Dc_d
ApEdMRd
Eg
Dc_d
ApEdMRd
Test purpose
true
G = T3 ∧ LE 6= free
true
true
G = T3
true
LeftEvac
$op
Unload
$op
Right
$op
$op
q0
q1
q2
q3
q4
qf
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 12/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Synchronization between the abstraction and the TP
Synchronization
Instantiation of ($op)+
($op)+ ; (LE ∈ {T1,T3} ∧ G = T3)
. Right
. . .
(Deg = l & Dt = l), q0
ApEdDRg
(Deg = l & Dt = T3), q0
C
(Deg = l & Dt : {T1, T2}), q0
C
ApEdMRg
(Deg : {T1, T3} & Dt = l), q0
Dc_g
Dc_d
ApEdMRd
Dc_g
Eg
ApEdDRg
(Deg : {T1, T3} & Dt = T3), q0
C
(Deg : {T1, T3} & Dt = T3), q1
C(Deg : {T1, T3} & Dt : {T1, T2}), q0
C
Eg
Dc_d
ApEdMRd
ApEdMRd
(Deg : {T1, T3} & Dt = T3), q2
Rd
Eg
Dc_d
ApEdMRd
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 13/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Generation and Instantiation of tests
Generation of a set of symbolic tests
All-edges criterion on synchronized product
Application of the Chinese Postman Algorithm
Instantiation of a set of symbolic tests
Why isn’t instantiation complete ?
; Tests use over-approximated transitions in abstraction
How do we deal with that ?
; Interleaving of reflexive operations calls in tests
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 14/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Instantiation of symbolic tests
G = free G = T3. . . . . .
Arrival, RightEvac,Down, Left . . .
Load
Symbolic test
. . . ; (G = free)
. Load ; (G = T3)
Instantiated test
. . . ; (G = free)
. Arrival ; (G = free)
. Load ; (G = T3). . . . . .
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Instantiation of symbolic tests
G = free G = T3. . . . . .
Arrival, RightEvac,Down, Left . . .
Load
Symbolic test
. . . ; (G = free)
. Load ; (G = T3)
Instantiated test
. . . ; (G = free)
. Arrival ; (G = free)
. Load ; (G = T3). . . . . .
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
The robot exampleDefinition of an abstractionSynchronization between the abstraction and the test purposeTest generationInstantiation of tests
Instantiation of symbolic tests
G = free G = T3. . . . . .
Arrival, RightEvac,Down, Left . . .
Load
. . . . . .
Symbolic test
. . . ; (G = free)
. Load ; (G = T3)
Instantiated test
. . . ; (G = free)
. Arrival ; (G = free)
. Load ; (G = T3). . . . . .
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 15/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Abstraction computationTest generation
Experimental results
Abstraction computation - Synchronization
, Reduction of the size of the models
,/ 1 abstraction per 1 test purpose
/ Abstraction computation : time consuming
B Model ♯StatesAbstraction computation Synchronization
♯States ♯Trans. Time ♯States ♯Trans.
Qui-Donc 135 24 2 min. 9 365 18 1.5 min. 12 54
Robot 3726 37 5 min. 17 1068 50 8 min. 23 214
DeMoney 1030 3 148 35 min. 7 3327 228 1.5 h. 18 422
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 16/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Abstraction computationTest generation
Experimental results
Test generation - Instantiation
, Number of instantiated tests increase with filtering
/ Instantiation and filtering are time consuming with largeexample
B ModelWithout filtering With filtering
♯Instant. tests / Time of ♯Filter. Filter ♯Instant. tests / Time of♯Symbolic tests instantiation trans. time ♯Symbolic tests instantiation
Qui-Donc4/10 (40%)
≤ 1 s.2
≤ 1 s.
6/10 (60%)≤ 1 s.
5/15 (33%) 2 6/11 (55%)
Robot7/12 (58%) 3 min. 1 7/11 (64%) 2 min.8/23 (35%) 5 min. 0 8/23 (35%) 5 min.
DeMoney0/32 (0%)
2 h.49
2 h.0/19 (0%) 2 h.
0/32 (0%) 56 17/18 (94%) 1 h.
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 17/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Conclusion and Future works
Conclusion
, Automatic extraction of abstraction predicates
, Abstraction related to test purpose
,/ 1 test purpose per 1 abstraction
/ Computation time of abstraction
/ Too much over-approximation for efficient test instantiation
Future works
Common abstraction (1 abstraction for N tests purposes)
Under-approximations rather than over-approximations
Tri-modal Transitions System generation (Godefroid, Ball)
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 18/19
Problem and MotivationsProcess overview
Process applicationExperimental results
Conclusion and Future works
Thanks for your attention
Questions ?
F. Bouquet, P.-C. Bué, J. Julliand, P.-A. Masson Abstractions of models for test 19/19