Delta-oriented Model-based SPL Regression Testing · Sascha Lity j Delta-oriented Model-based SPL...
Transcript of Delta-oriented Model-based SPL Regression Testing · Sascha Lity j Delta-oriented Model-based SPL...
Institut für Programmierungund Reaktive Systeme
Delta-oriented Model-based SPL Regression Testing
Sascha LityFOSD-Treffen – 22.03.2012
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Outline
Motivation
Delta State Machines
Delta-oriented SPL Regression Testing
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 2Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Motivation
Model-based TestingTest model describes system behaviorTest case generationConformance testing
Regression TestingTesting of software versionsReuse of test casesReuse of test results
⇒ Combination for reasoning about reuse potentials
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 3Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Motivation
Model-based TestingTest model describes system behaviorTest case generationConformance testing
Regression TestingTesting of software versionsReuse of test casesReuse of test results
⇒ Combination for reasoning about reuse potentials
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 3Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Motivation
Model-based TestingTest model describes system behaviorTest case generationConformance testing
Regression TestingTesting of software versionsReuse of test casesReuse of test results
⇒ Combination for reasoning about reuse potentials
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 3Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Variability in Test Models
How to encode variability in test models?Annotative/Selective [WSS08, LOGS11, COLS11]Compositional [UKB10, SV08]Transformational [LSKL12]
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 4Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta State Machines
Adaption of Delta Modeling [Sch10] approach
TransformationalModeling and Generation of variable state machines
Model-based TestingModel-driven Development
Deltas define transformationsAdd/Remove of basic state machinesAdd/Remove of hierarchyAdd/Remove of statesAdd/Remove of transitionsModification of transition labels
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 5Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta State Machines
Adaption of Delta Modeling [Sch10] approachTransformational
Modeling and Generation of variable state machinesModel-based TestingModel-driven Development
Deltas define transformationsAdd/Remove of basic state machinesAdd/Remove of hierarchyAdd/Remove of statesAdd/Remove of transitionsModification of transition labels
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 5Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta State Machines
Adaption of Delta Modeling [Sch10] approachTransformationalModeling and Generation of variable state machines
Model-based TestingModel-driven Development
Deltas define transformationsAdd/Remove of basic state machinesAdd/Remove of hierarchyAdd/Remove of statesAdd/Remove of transitionsModification of transition labels
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 5Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta State Machines
Adaption of Delta Modeling [Sch10] approachTransformationalModeling and Generation of variable state machines
Model-based TestingModel-driven Development
Deltas define transformationsAdd/Remove of basic state machinesAdd/Remove of hierarchyAdd/Remove of statesAdd/Remove of transitionsModification of transition labels
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 5Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Transformations (1/2)
s1
s3
Δ Bsm_Root
s2
s1
s3
Bsm_Root
s2
s4
s6
Bsm_Sub
s5
Add a hierarchy and add a basic state machine
s1
s3
Δ Bsm_Root
s2 s1
s3
Bsm_Root
s2
s4
Add a state and a transition
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 6Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Transformations (1/2)
s1
s3
Δ Bsm_Root
s2
s1
s3
Bsm_Root
s2
s4
s6
Bsm_Sub
s5
Add a hierarchy and add a basic state machine
s1
s3
Δ Bsm_Root
s2 s1
s3
Bsm_Root
s2
s4
Add a state and a transition
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 6Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Transformations (2/2)
s1 s2t1:e1/e2;e3
Δ
s1 s2t1:e5/e2;e3
s1 s2t1:e1/e2;e3
Δ
s1 s2t1:e1/e3;e4;e5
Modification of transition labels
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 7Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Example BCS Delta
FP_on
Name: DRemFPTransManPWAddTransAutoPW Apply to: FP_BSM After: When: Automatic Power Window
FP_off
fp_finger_detected/fp_on = true;
fp_finger_detected/ pw_auto_mv = 0; fp_on = true +
-
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 8Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Generation of State Machines
Core + Delta 1 +
Delta 2 + Delta 3
Core + Delta 1 +
Delta 2
Core + Delta 1
T6: e3/ T7: e1/
Application Condition
Feature Configuration
Feature Model
S1 S2
S3
T1: e1/e2
T2: e3/ T3: e2/
T5: e4/e6 T4: e5/e6
Core Product
Delta 1 Add
T1: e1/e2
Delta 2 Rem
Delta 3 Mod
T5: e5/e2;e3;e4
+
+
+
S1 S2
S3
T1: e1/e2
T2: e3/ T3: e2/
T5: e4/e6 T4: e5/e6
T6: e3/ T7: e1/ S4
S1 S2
S3
T2: e3/ T3: e2/
T5: e4/e6 T4: e5/e6
T6: e3/ T7: e1/ S4
S1 S2
S3
T2: e3/ T3: e2/
T4: e5/e6
T6: e3/ T7: e1/ S4
T5: e5/e2;e3;e4
Product
S1 S2
S1 S2
S3
+ + +
-
*
+ +
-
* S2
S4
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 9Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Outline
Motivation
Delta State Machines
Delta-oriented SPL Regression Testing
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 10Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Product Test Artifacts
DefinitionThe collection of test artifacts for a product p is a 4-tupletap = (tmp, tgp, tsp, tpp) consisting of a test model tmp, a finite set tgpof test goals in tmp for a criterion C , a test suite tsp, and a test plantpp.
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 11Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Product Test Model Generation
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
t2 : rck lock/cls locked;pw disabled; pw but up
t3 : rck unlock/cls unlocked;pw enabled
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
t4 : car drives/cls locked
δtm = {add t2, add t3}δ−1tm = {rem t2, rem t3} δtm′ = {add t4}
δtm,tm′ = {rem t2,rem t3, add t4}
1
δtm,tm′ = (δtm \ δtm′)−1 ∪ (δtm′ \ δtm)
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 12Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Product Test Model Generation
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
t2 : rck lock/cls locked;pw disabled; pw but up
t3 : rck unlock/cls unlocked;pw enabled
unlock lock
t0 : key pos lock/cls locked;pw disabled; pw but up
t1 : key pos unlock/cls unlocked;pw enabled
t4 : car drives/cls locked
δtm = {add t2, add t3}δ−1tm = {rem t2, rem t3} δtm′ = {add t4}
δtm,tm′ = {rem t2,rem t3, add t4}
1
δtm,tm′ = (δtm \ δtm′)−1 ∪ (δtm′ \ δtm)
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 12Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta-oriented SPL Regression Testing (1/2)
s1 s2
s3
t1: e1/e2
t2: e3/ t3: e2/
t5: e4/e6 t4: e5/e6
TM
TG
s1 s2
s3
t2: e3/ t3: e2/
t5: e4/e6 t4: e5/e6
t6: e3/ t7: e1/ s4
t1
t2
t3
t4 t5
t2
t3
t4
t5
t6 t7
C1 C1
t1
TS
s1 s2
s3
t1
t3 t4
t5
s1 s2
s3
t1 t3
s1
s3 t4
t2
t5
s1 s2
s3
t1 t3
tc1
s1
s3 t4
t2
tc2
tc3
tc4
tc1
tc2
tc3
Generation Generation
TP t5
s1 s2
s3
t1 t3
s1
s3 t4
t2
t5 s1
s2
s3
t3
t6 t7 s4 tc2
tc4 tc3
Selection Selection
Δ
Δ
Δ
Δ
t5 s1
s2
s3
t3
t6 t7 s4
s1 s2
s3
t1
t3 t4
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 13Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Delta-oriented SPL Regression Testing (2/2)
tmi
δtmi,tmi+1 +3 tmi+1
tgiδtgi,tgi+1 +3 tgi+1
tpiδtpi,tpi+1 +3 tpi+1
tsiδtsi,tsi+1 +3 tsi+1
1
δta,ta′ = (δtm,tm′ , δtg,tg′ , δts,ts′ , δtp,tp′)
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 14Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Body Comfort System Case Study
Finger Protection
Body Comfort System
Power Window
Central Locking System
Automatic Locking
Remote Control
Key
Human Machine Interface
Manual Power
Window
Automatic Power
Window
Control Alarm System
Safety Function
Interior Monitoring
Alarm System
Exterior Mirror
Electric Heatable
require
require
require
Mandatory Or
require
exclude Optional Alternative
Status LED
Security Door
System
LED Central Locking System
LED Power
Window
LED Exterior Mirror
LED Heatable
LED Alarm System
LED Finger Protection
Control Automatic
Power Window
require
exclude
require
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 15Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Case Study Statistics
Previous results obtained in evaluation of MoSo-PoLiTe [OLZG11]Representative subset of 17 product variants150% state machine test model
105 states107 transitions26 input events33 output events
64 test cases executed in average
Remodelling of test model1 core40 deltas
Testing of 18 variants10 test cases generated in average9 test cases executed in average
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 16Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Case Study Statistics
Previous results obtained in evaluation of MoSo-PoLiTe [OLZG11]Representative subset of 17 product variants150% state machine test model
105 states107 transitions26 input events33 output events
64 test cases executed in averageRemodelling of test model
1 core40 deltas
Testing of 18 variants10 test cases generated in average9 test cases executed in average
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 16Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Case Study Statistics
Previous results obtained in evaluation of MoSo-PoLiTe [OLZG11]Representative subset of 17 product variants150% state machine test model
105 states107 transitions26 input events33 output events
64 test cases executed in averageRemodelling of test model
1 core40 deltas
Testing of 18 variants10 test cases generated in average9 test cases executed in average
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 16Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Case Study Results
30
40
50
60
70
80
New
Selection
MoSo-PoLiTe
0
10
20
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 17Institut für Programmierung
und Reaktive Systeme
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 18Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Literature I
H. Cichos, S. Oster, M. Lochau, and A. Schurr.Model-based Coverage-Driven Test Suite Generation for Software Product Lines.In MODELS’11, 2011.
Malte Lochau, Sebastian Oster, Ursula Goltz, and Andy Schurr.Model-based Pairwise Testing for Feature Interaction Coverage in Software ProductLine Engineering.Software Quality Journal, pages 1–38, 2011.
Malte Lochau, Ina Schaefer, Jochen Kamischke, and Sascha Lity.Incremental model-based testing of delta-oriented software product lines.In 6th International Conference on Tests & Proofs (TAP 2012), Prague, CzechRepublic, 2012.accepted to appear.
S. Oster, M. Lochau, M. Zink, and M. Grechanik.Pairwise Feature-Interaction Testing for SPLs: Potentials and Limitations.In FOSD’11, 2011.
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 19Institut für Programmierung
und Reaktive Systeme
Motivation Delta State Machines Delta-oriented SPL Regression Testing
Literature II
Ina Schaefer.Variability modeling for model-driven development of software product lines.Fourth International Workshop on Variability Modelling of Software-intensive Systems(VaMoS 2010), January 2010.
N. Szasz and P. Vilanova.Statecharts and Variabilities.In VAMOS’08, pages 131–140, 2008.
Engin Uzuncaova, Sarfraz Khurshid, and Don S. Batory.Incremental test generation for software product lines.IEEE Trans. Software Eng., 36(3):309–322, 2010.
S. Weißleder, D. Sokenou, and H. Schlingloff.Reusing State Machines for Automatic Test Generation in ProductLines.In MoTiP’2008, 2008.
Sascha Lity | Delta-oriented Model-based SPL Regression Testing | 20Institut für Programmierung
und Reaktive Systeme