State Machine Flattening, a Mapping Study and Tools Assessment
41
www.unamur.be State Machine Flattening, a Mapping Study and Tools Assessment Xavier Devroey <[email protected]> Maxime Cordy <[email protected]> Axel Legay <[email protected]> Pierre-Yves Schobbens <[email protected]> Patrick Heymans <[email protected]> 11th Workshop on Advances in Model Based Testing, A- MOST@ICST ’15 Graz, Austria
-
Upload
xavier-devroey -
Category
Science
-
view
36 -
download
3
Transcript of State Machine Flattening, a Mapping Study and Tools Assessment
www.unamur.be
State Machine Flattening, a Mapping Study and Tools Assessment
Xavier Devroey <[email protected]> Maxime Cordy <[email protected]>Axel Legay <[email protected]>Pierre-Yves Schobbens <[email protected]>Patrick Heymans <[email protected]>
11th Workshop on Advances in Model Based Testing, A-MOST@ICST ’15Graz, Austria
THE CONTEXT Behavioural Model-Based
Testing (of Software
Product Lines)Framework
THE CONTEXT
THE CONTEXT
THE CONTEXT
Let’s check the literature!Let’s do a
Systematic Mapping
Study!
A SYSTEMATIC MAPPING STUDY ???
SMS: Small Introduction
• Largely used in medical research• … but quite new in Software Engineering• Goal: categorise published documents • “It requires less effort [than a systematic literature
review] while providing a more coarse-grained overview.” [Petersen, 2008]
A-MOST '15
Phase 1: Definition of RQs
• (RQ1) What are the input and output models used in the different flattening approaches?
• (RQ2) Do the different approaches support hierarchy (composite states) and parallelism (orthogonal states) in the input model?
• (RQ3) In which context are the different flattening approaches performed (e.g., code generation, test-case generation, semantic definition, etc.)?
A-MOST '15
Phase 2: Search of Publications
• Exploratory search– Google Scholar (http://scholar.google.be/)– Science Direct (http://www.sciencedirect.com)– Computer Science Bibliographies (http://liinwww.ira.uka.
de/bibliography/)
• Inclusion criteria:– Books, articles, proceedings, technical reports and grey literature
• Exclusion criteria: – Slides
• Extension by related work– By word search in the documents (flat.* or hierarch.* or
orthogonal.*)A-MOST '15
Set of Publications
A-MOST '15
• Initial set: 24 publications
• After step 3 : 12 new publications
• After step 3 (2nd): 1 new publication which did not match the inclusion and exclusion criteria
• Set of publications: 36 publications
Phase 3: Keywording
• Step 1: tag papers using keywords– Adaptive reading depth– Guide the keywording process using research questions
• The purpose of the research (RQ3)• The input/outpout models of the transformation (RQ1)• Hierarchy/orthogonality support (RQ2)• Implementation (RQ3)
– In parallel by two reviewers + arbitrator
• Step 2: group keywords to build classification scheme
A-MOST '15
Phase 3 bis: Quality Assessment
• Quality of the description of flattening in the publications• Based on Systematic Litterature Review protocol• Usability
– G 1.1 Is there a tool implementation?– G 1.2 Is there a small example?– G 1.3 Is there a more significant case study (even if not fully detailed)?– G 1.4 Are the input and output models described?– G 1.5 Does the publication present the limitations of the transformation?
• Degree of generality: – G 2.1 Are there guidelines for the transformation separated from the example
of the transformation (if any)?– G 2.2 Does it detail the transformation process for all the constructs of the
input model?– G 2.3 Does the flattening technique support hierarchy?– G 2.4 Does the flattening technique support orthogonality?
A-MOST '15
Phase 3 bis: Quality Assessment
• Quality of the description of flattening in the publications• Based on Systematic Litterature Review protocol• Usability
– G 1.1 Is there a tool implementation? 50% (Only 5 / 15 could be found!)– G 1.2 Is there a small example? 90%– G 1.3 Is there a more significant case study (even if not fully detailed)? 27%– G 1.4 Are the input and output models described? 90%– G 1.5 Does the publication present the limitations of the transformation? 67%
• Degree of generality: – G 2.1 Are there guidelines for the transformation separated from the example
of the transformation (if any)? 80%– G 2.2 Does it detail the transformation process for all the constructs of the
input model? 67%– G 2.3 Does the flattening technique support hierarchy? 97%– G 2.4 Does the flattening technique support orthogonality? 70%
A-MOST '15
Phase 4: Mapping
A-MOST '15
93% are solution papers
flattening is a transformation used to bridge high-level models with existing lower level analysis tools and frameworks
Phase 4: Mapping
A-MOST '15
Model Checker input code, code dedicated to the synthesis of embedded systems, and Java code
Phase 4: Mapping
A-MOST '15
Consistency management, IP telephony, formal semantics to hierarchical state machines
Conclusion about Flattening
• Mostly used for model-based validation (testing, verification) and code generation
• Step towards a more general objective
• Impacts the quality of the description of flattening algorithms
– Precise constructs supported by the flattening transformation are not always provided
– Validity of the flattening transformation is barely addressed
A-MOST '15
CONCLUSION ABOUT SMS
… and some lessons learned
Anonymous reviewer
““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no
content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the
categories, etc.But there is zero information on algorithms for state
machine flattening, thus the paper provides me no added value.”
A-MOST '15
Anonymous reviewer
““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no
content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the
categories, etc.But there is zero information on algorithms for state
machine flattening, thus the paper provides me no added value.”
A-MOST '15
Anonymous reviewer
““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature
review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no content at all. There are lots and lots of words used to describe the methodology, the keywords used to
search, the categories, etc.But there is zero information on algorithms for state
machine flattening, thus the paper provides me no added value.”
A-MOST '15
Anonymous reviewer
““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no
content at all. There are lots and lots of words used to describe the methodology, the keywords used to search,
the categories, etc.But there is zero information on algorithms for state
machine flattening, thus the paper provides me no added value.”
A-MOST '15
Anonymous reviewer
““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no
content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the
categories, etc.But there is zero information on algorithms for state
machine flattening, thus the paper provides me no added value.”
A-MOST '15
Categorise published documents
… lessons learned
• SMS is a good starting point for a thesis– Systematic => protocol
• Define• Follow
– Learn fast reading• Lots of papers
– Learn literature search engines manipulation
• Quality assessment (step 3bis)– Learn what to do/not to do when writing a paper
A-MOST '15
THANK YOU FOR YOUR ATTENTION !
Models and tools available at https://projects.info.unamur.be/vibes/
