Analysing the concept of quality in

model-driven engineering literature:

a systematic review

Fáber D. Giraldo, Sergio España and Oscar Pastor

RCIS 2014

Marrakesh, Morocco, June 2014

Hi, I am Fáber

Giraldo

Agenda

Introduction & Motivation

Systematic review design

Results achieved

Discussion

Framework by Lindland, Sindre and Sølvberg [1994]

LANGUAGE

AUDIENCE

INTERPRETATION

DOMAINMODEL

PRAGMATICS

SYNTAX

SEMANTICS

completeness=77% completeness=85%

UC 7

PUBLISHING HOUSE

RETURNS SIGNED REPORT

DELIVERY NOTE

UC 13

SALESMAN ISSUES

PUBLISHING HOUSE

INVOICES

SALESMAN

PUBLISHING

HOUSE

UC 13. BASIC FLOW OF ACTIONS:

1. REVIEW DELIVERY NOTES

2. RETRIEVE PUB.HOUSE DATA

3. CALCULATE TOTAL AMOUNT

4. SEND INVOICE TO PUB.HOUSE

PHO 7

PUBLISHING HOUSE RETURNS SIGNED

REPORT DELIVERY NOTESALESMAN

PHO 13

SALESMAN ISSUES PUBLISHING HOUSE

INVOICESSALESMAN

SALESMANPUBLISHING

HOUSE

PUBLISHING

HOUSE

INVOICE

SIGNED D.N.

UC 7

PUBLISHING HOUSE

RETURNS SIGNED REPORT

DELIVERY NOTE

UC 13

SALESMAN ISSUES

PUBLISHING HOUSE

INVOICES

SALESMAN

PUBLISHING

HOUSE

UC 13. BASIC FLOW OF ACTIONS:

1. REVIEW DELIVERY NOTES

2. RETRIEVE PUB.HOUSE DATA

3. CALCULATE TOTAL AMOUNT

4. SEND INVOICE TO PUB.HOUSE

PHO 7

PUBLISHING HOUSE RETURNS SIGNED

REPORT DELIVERY NOTESALESMAN

PHO 13

SALESMAN ISSUES PUBLISHING HOUSE

INVOICESSALESMAN

SALESMANPUBLISHING

HOUSE

PUBLISHING

HOUSE

INVOICE

SIGNED D.N.

completeness=76% completeness=81%

N=35

MOF

ATL

METAMODEL

CA2UML

TRANSFORMATION

COMMUNICATION

ANALYSIS

METAMODEL

UML2 CLASS

DIAGRAM

METAMODEL

REQUIREMENTS

MODEL

UML

CLASS

DIAGRAM

Conforms to

Conforms to Conforms to

Based on Conforms toConforms to Based on

input

executed

output

Conforms to

Introduction & Motivation (I)

Many works on model quality

– Moody and ISO 9126

– SEQUAL (Krosgtie, Lindland, Solverg)

– Good practices in modelling (Mohagheghi)

– UML defects and guidelines

Most proposals are very theoretical (scarce validation).

Lack of conceptual consensus (what is model quality?).

It is unclear whether model-driven engineering principlesare considered.– Complementary modelling languages

– Model transformation

– Code generation

Introduction & Motivation (II)

Problematic phenomena:

– Existing MDE standards do not address quality

– No standard quality framework for MDE

Research goal:

– Review literature on model / modelling language quality

– Structure proposals

– Find out their applicability to MDE

– Discover trends

Agenda

Introduction & Motivation

Systematic review design

Results achieved

Discussion

Systematic review design (I)

Kitchenham’s guidelines for systematic review

Research questions:

– (RQ1) What does quality mean in the context of MDEliterature?

– (RQ2) What does it mean to say that an artifactconforms to the principles of MDE?

– (RQ3) Are current quality-related methods within MDEable to deal with a set of modeling languages?

Systematic review design (II)

Search

– Scientific databases (ACM Digital Lib., IEEExplore, Springer, Scopus, etc.)

– Indexing services (Google Scholar, DBLP )

– Conference Proceedings (CAISE, ER, RCIS, RE, etc.)

– Industrial repositories (OMG)

The process of reading and selecting was performed by a PhD student

under the supervision of two senior researchers

Results achieved (I)

Selection

– Inclusion criteria

– Exclusion criteria

First results Depuration

Systematic review design (III)

Data extraction

– Classification schemas to analyse retrieved studies include:

1. Kinds of artefacts whose quality is measured (e.g. models, modellinglanguages, sets of modelling languages)

2. Kinds of artefacts proposed to measure quality (e.g. methods and tools)

3. Degree of validation and operationalisation of 2.

Agenda

Introduction & Motivation

Systematic review design

Results achieved

Discussion

Results achieved (II)

Results achieved (III)

We identified 11 trends on the studies

– Grouping of studies depending on the theory on which they build.

0 0 0 0 0 0 0 0 0 0 0 0

3

4

5

2

11

0 0 0 0 0

1

0

1 1

0

1

0 0

1 1 1

0 0 0 0 0 0

1

3

0

2

3

4 4

2

1

0 00 0 0 0 0 0 0 0 0 0 0 0

1

0 0 0 00 0 0 0 0 0 0 0 0 0

1

2

0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0

1

3

1

0 00 0 0 0 0 0 0 0

1 1 1

0 0 0 0

1 1

0 0 0 0 0 0

1 1

3

4

3 3

1 1

0 0 00 0 0 0 0 0 0 0 0 0

1 1

2

0

1

0 00 0 0 0 0 0 0 0

1

2

1 1

0 0 0 0 00 0 0 0 0 0

1

0 0 0 0 0 0 0 0 0 0

0

1

2

3

4

5

6

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Num

ber

of

identifie

d stu

die

s

Year

Studies published by trends and years

MDWE SEQUAL 6C UML guid. MoSMe Tranform.

Empir. UML Underst. UML OPM2SysML Harel&Rumpe Structural

Results achieved (III)

We studied the influence on some authors on others

– We applied a sampling technique to identify influential studies

Agenda

Introduction & Motivation

Systematic review design

Results achieved

Discussion

Discussion (II)

(RQ1) What does quality mean in the context of MDEliterature?

– There are many studies on model/modelling language quality• Most focus on UML

• There is no agreement on the definition of quality (ill-defined)

– MDE community has paid little attention to the topic so far

– The definitions of quality in the trends do not converge

– Many quality evaluation methods and metrics are ad-hoc andspecific for assessing a method created by the same authors• To prove the benefits of their method (comparative experiments)

• To provide a suitable quality evaluation method

– Few quality evaluation methods are supported by tools• Arendt and Taentze complain about the difficulty

Discussion (II)

(RQ2)What does it mean to say that an artifact conformsto the principles of MDE?

– This remains unclear.

– The retrieved definitions about quality in models are a strongbasis to start the discussion

– However, most of the MDE core features are not considered indepth• suitability of languages to engineering goal

• suitability of a set of languages used in combination

• conformity to MDE principles

• management of abstraction levels

• model granularity

Discussion (II)

(RQ3) Are current quality-related methods within theMDE able to deal with a set of modeling languages?

– Complex information systems require multiple viewpointssupported by conceptual models (different modellinglanguages)

– Practically all studies deal with a single language

– Those who consider a set of languages do not provide asystematic, repeatable and generalisable method to evaluateany set of languages

– The evaluation of a set of modelling languages used incombination remains an open research question.

Discussion (IV)

Interesting future works

– Operationalise model quality proposals using technologicalframeworks such as EMF, MetaEdit...

– Compare model quality proposals when applied to specificMDE contexts• Laboratory research (controlled experiment)

• Field research (action-research, case study)

– Design a method for evaluating quality in MDE contexts(building on top of existing proposals)• Conceptual framework

• Methodological guidance

• Tool support

Ask me the

difficult

questions

fdgiraldo@pros.upv.es

Analysing the concept of quality in model-driven

engineering literature: a systematic review

Fáber D. Giraldo, Sergio España and Oscar Pastor

RCIS 2014

Marrakesh, Morocco, June 2014