www.unamur.be
Covering SPL Behaviour with Sampled Configurations: An Initial Assessment
Xavier Devroey <[email protected]> Gilles Perrouin <[email protected]>Axel Legay <[email protected]>Pierre-Yves Schobbens <[email protected]>Patrick Heymans <[email protected]>
9th International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ‘15
Hildesheim, Germany
2
Introduction
• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise
• Behavioural approaches– Select test cases in a behavioural model of the SPL
VaMoS '15 [email protected]
Devroey et al. (2014). Towards Statistical Prioritization for Software Product Lines Testing. In VaMoS ’14.
3
Introduction
• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise
• Behavioural approaches– Select test cases in a behavioural model of the SPL
Practically, which behavioural coverage similarity and pairwise sampling do achieve ?
VaMoS '15 [email protected]
SPECIFYING A PRODUCT LINE
Featured Transition Systems (FTSs) [Classen et al. 2011]
[email protected] 5VaMoS '15
Images credit: http://hedbonstudios.deviantart.com
[email protected] 6VaMoS '15
Images credit: http://hedbonstudios.deviantart.com
STRUCTURAL VS BEHAVIOURAL COVERAGE CRITERIA
Image credits: http://domonated.deviantart.com
Structural Coverage
• Which configurations do I select in the FD ?• Pairwise criterion– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]
VaMoS '15
Structural Coverage
• Which configurations do I select in the FD ?• Pairwise criteria– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]
• (Di)Similarity criteria– Dissimilar configurations to test– With few selected/deselected features in common– Uses a distance function (easy to compute)– Empirically validated [Henard et al. 2013]
VaMoS '15
Steffens et al. (2012). Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VaMoS ‘12
Behavioural Coverage
• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL
VaMoS '15
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’1
Behavioural Coverage
• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL
• Coverage criteria– Executable set of test cases, FTS → [0, 1]– State coverage– Transition coverage – Transition-pairs coverage
VaMoS '15
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’14
Behavioural Coverage
Exec. Test-Suite TS is Minimal for cov iff
• There is no exec. TS’ with a better coverage cov
• There is no exec. TS’ with fewer Test-Cases
Exec. Test-Suite TS is P-Minimal for cov iff
• There is no exec. TS’ with a better coverage cov
• There is no exec. TS’ which request fewer products to execute TS’
SPLat @ SPLC'14
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’14
FTS Test-Case Selection
Number of Exec T.
C.
Number of
Products
Test Case Selection
Coverage
SPLat @ SPLC'14
EXPERIMENT
… using PLEDGE and SPLCAT
Setup
• SPLCAT– T-wise criterion, with T in [1,3]
• PLEDGE– (Di)Similarity criterion– Uses a genetic algorithm
• Number of configurations (population) given by SPLCAT• Time = 60 sec. (default time)
VaMoS '15
Coverage computation process
Compute coverage and cumulated coverage
Projects the FTS Model
Generate a set of configurations from the FD
VaMoS '15
Coverage computation process
Compute coverage and cumulated coverage
Project the FTS Model
Generate a set of configurations from the FD
VaMoS '15
Coverage computation process
Compute coverage and cumulated coverage on FTS
Project the FTS Model
Generate a set of configurations from the FD
VaMoS '15
Observations
• Relatively few configurations are needed to cover 100% of states, transitions and actions– Tendency is confirmed both on academic and Claroline
• T-Wise coverage (SPLCAT) provides better coverage (for all our behavioural criteria) for T = 3
VaMoS '15
Observations
• For both approaches, there is a need for prioritization and optimal behavioural coverage – E.g., from the 73 config. Generated for Claroline, only 1 is
needed to achieve state coverage
• Threat: Considered FDs have few features and cross-tree constraints
VaMoS '15
CONCLUSION
… and Future Works
Conclusion
• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage
VaMoS '15
Conclusion
• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage
• Maximising Behavioural coverage cannot be done only using structural coverage criteria
VaMoS '15
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
VaMoS '15
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
• … in order to define hybrid behavioural/structural criteria
VaMoS '15
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
• … in order to define hybrid behavioural/structural criteria
• Guided by replication of the experiment on more complex case studies (more features, more constraints, more complex behaviour, etc.)
VaMoS '15
THANK YOU FOR YOUR ATTENTION !
Models and tools available at https://projects.info.unamur.be/vibes/
Top Related