How to define Quality Models for Measuring Software Quality

FP7-ICT-2011-8. Project #: (318082)

How to define Quality Models for Measuring Software Quality

FP7-ICT2011-8. Project #: (318082)

• A few words about speakers’ biographies and companies.• The Project• The Methodology• Support to the SW development life cycle• How does it work? • Applying the Method in Agile• Other uses of the Methodology

FP7-ICT2011-8. Project #: (318082)

The Project (1)

• An EU funded research project under FP7.• 8 organizations from Finland, Norway,

Germany, Luxembourg and Spain: Experts in SW Quality, Ontology modeling and implementation, SW Engineering methods, and SW Measurement.

FP7-ICT2011-8. Project #: (318082)

The Project (2)

• Objective: Create a solution to provide project managers, software engineers, testers and other actors with an objective measurement of quality for:

– Software Development processes– Software Products

FP7-ICT2011-8. Project #: (318082)

The Methodology (1)

• As part of the Project a Methodology has been created:– For gathering and exploiting data about the progress and

quality of software development projects and products.

– Covering the whole software development life cycle.

– Easy to apply in various types of IT Companies, SMEs or large enterprises.

– May by applied as stand-alone approach for SW QA as well as supported by quality management tools.

– Public: 1st and 2nd version are already shared online (visit link).

FP7-ICT2011-8. Project #: (318082)

The Methodology (2)

• An overview of the Methodology areas:

FP7-ICT2011-8. Project #: (318082)

The Methodology (3)

• A Model-based Methodology:– How to identify Quality Objectives.– How to plan a Measurement Program: definition

of Quality Indicators and Measurement.– How to select the metrics (by means of GQM).– How to collect data (auto/manual).– How to assess the validity of the measurement.– How to provide valuable feedback out of the

measurements.

FP7-ICT2011-8. Project #: (318082)

• (Acronym) solution is:

– Paradigm agnostic!– Agile or traditional LCs (V, waterfall, RUP…)– Adaptable to a volatile environment.

Support to SW development LC (1)

FP7-ICT2011-8. Project #: (318082)


Requirements

Implementation Unit Testing

Validation

Design Integration Testing

# Redundant reqs# Incomplete reqs# Format errors

% Blocking cr viol.% Critical cr violComplexity…

% Successful Uts% Coverage

# Redundant specs# Incomplete specs# Format errors

% Successful TCs% Coverage

# Misconceptions

% Successful TCs% Coverage

# Misconceptions

• What to measure?

FP7-ICT2011-8. Project #: (318082)


• What to measure?

• % Incomplete stories, i.e. Stories lacking story points, BV estimations, categorization or classification.

• # Scheduled vs. Solved issues: Ratio of solved/closed issues per sprint.• Technical debt: Indicator aggregating data about pending bugs, known issues, etc after

each sprint.• % of stories accepted per sprint.• Perception of the customer & Team enthusiasm by meand of surveys.• # Issues identified in retrospective meetings.

FP7-ICT2011-8. Project #: (318082)


Requirements, Stories, Issues

Implementation & Unit Testing

Validation, Acceptance or Integration Tests

Design

• Where to look:

Wikis, Confluence, JIRA, Greenhopper, Trac, Doors …

SpiraTest, TestRobot, HP QA Center, Selenium, JIRA …

Confluence, JIRA

SVN, SONARQUBE

Deployment Bamboo, Jenkins, Continuum…

FP7-ICT2011-8. Project #: (318082)

How does it work? (1)

• Create a Quality Model– It can be based on existing standards or models.– It can be an organization’s self-made model.– It can deal with process, products, or both.

Creation of a model consists of identifying expected values, weights and thresholds for:

• Quality Objectives, or aggregation of indicators to evaluate the achievement of an objective.

• Quality Indicators, or aggregation of metrics used to evaluate• Quality Metrics, or raw measurements obtained automatically or

manually

FP7-ICT2011-8. Project #: (318082)

How does it work? (2)

• Create Quality Objectives (QOs) or Select existing ones.

• Select or create Quality Indicators (QIs) by aggregating metrics.

CONTEXT AWARENESS & NEGOTIATION• QOs’ and QIs’ weights are adapted depending on the LC stage (e.g.

In the requirements stage a bigger relevance is given to completeness of specifications).

• Weights, Thresholds and Values are negotiated among the participants of the QA process

FP7-ICT2011-8. Project #: (318082)

The Five Methods (1)• Method 1: Quality Objectives Setting Method

Level Example of level Example of quality objectives

Product Mobile phone

High battery life, Good sound quality, Low

failure rate

ProcessWaterfall software development process

On time, On Budget

Sub-process Testing processHigh percentage of

automation, High code coverage

Features:

Find featureFast search results, low

memory usage

FP7-ICT2011-8. Project #: (318082)


FP7-ICT2011-8. Project #: (318082)


Step 1: Preparation and pre-assignment

I ndividual assignment 3-8 participants from different

roles ½ hours / participant

I ndividual/ pair assignment Few hours

Step 5: Post-workshop finalizing activities

Step 2: Brainstorming Step 3: Prioritization by voting Step 4: Objective elaboration

Workshop 3-8 participants from different

roles 4 hours / participant

FP7-ICT2011-8. Project #: (318082)


Name: Easy updateability

Description: Updating the software should be quick and easy, ideally possible without deep technical or product knowledge

Rationale: Direct cost savings related to updates. Reduced risk of errors during updates.

Related factors: Robustness of the software, configurability, quality of installer software

Votes 6

Quality indicator 1 Updating effort

Amount of average working hours consumed by making an update for a single customer installation.

Current level: 3 h

Target level: 15 min

Deadline: May 9, 2013

Easiness of collecting: We have it

FP7-ICT2011-8. Project #: (318082)

The Five Methods (6)• Method 2: Planning a measurement program

– Identifying quality indicator types:• Purpose of the Measurements• Measurability (numbers, human assessment, not measurable)• Automation of data collection• Target of the measurement• Frequency of the Feedback• Organizational Level of Information Use

– Planning the use of data

FP7-ICT2011-8. Project #: (318082)

The Five Methods (7)• Method 3: Defining the Metrics:

Goal/question/metric

GQM Model

FP7-ICT2011-8. Project #: (318082)



FP7-ICT2011-8. Project #: (318082)

The Five Methods (10)• Method 4: Data collection

– Successful measurement activities– Manual or automatic data collection

• Method 5: Reliability and validity of the measures– Assessing measurement validity– Evaluating proposed metrics

FP7-ICT2011-8. Project #: (318082)

Applying the Method in Agile• The 5 Methods can be applied:

– At the beginning of the project to define the Quality Objectives, Indicators and Metrics, Validity and Data collection.

– At the end of Iterations to review them.– At the Iteration planning to re-define as necessary

to adapt to changes.– During retrospectives sessions the need of

adapting to the proposed Q Model can be identified.

– Involving all the stakeholders.

FP7-ICT2011-8. Project #: (318082)

Other uses of the Methodology

• Support in the implementation and follow-up of certifications or standards (e.g. CMMi, TMMi, SPICE, ISO…) How? – Requirements of the certification model Quality

Objectives of the model– QIs and metrics of the QOs will indicate the

degree of fulfillment of the standard/certification model.

FP7-ICT2011-8. Project #: (318082)

Many thanks !!!

facebook.com/UQASAR

twitter.com/UQASAR

linkedin.com/groups/ UQASAR-Project-4682307

www.uqasar.eu

How to define Quality Models for Measuring Software Quality

Software

Transcript of How to define Quality Models for Measuring Software Quality