Measuring Software Product Quality: the ISO 25000 Series ...

Sponsored by the U.S. Department of Defense© 2004 by Carnegie Mellon University

Pittsburgh, PA 15213-3890

Measuring Software Product Quality:the ISO 25000 Series and CMMI

European SEPGJune 14, 2004

Dave Zubrow

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE 14 JUN 2004 2. REPORT TYPE

3. DATES COVERED 00-00-2004 to 00-00-2004

4. TITLE AND SUBTITLE Measuring Software Product Quality: the ISO 25000 Series and CMMI

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Carnegie Mellon University,Software Engineering Institute,Pittsburgh,PA,15213

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT Same as

Report (SAR)

18. NUMBEROF PAGES

29

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

© 2004 by Carnegie Mellon University

Carnegie MellonSoftware Engineering Institute

page 2

Objectives

Provide status on a new Software Product QualityMeasurement standard and its connection to CMMI

Provide ideas on how to get started with Software ProductQuality Measurement today



page 3

Outline

Background and Overview

Concepts and Models

Software Product Quality Measurement

Summary



page 4

Achieving Quality SoftwareRequires planning and intentional design

More than achieving the desired functionality

Must explicitly attend to both functional and non-functionalrequirements

Need to verify all requirements are being met throughoutthe life cycle



page 5

CMMI Definition for Quality Requirements

The phrase “quality and process-performanceobjectives” covers objectives and requirements forproduct quality, service quality, and processperformance. Process performance objectives includeproduct quality.



page 6

Requirements DevelopmentThis process area describes three types of requirements:• customer requirements (quality in use)• product requirements (external quality attributes)• product-component requirements (internal quality

attributes)

Taken together, these requirements address the needs ofrelevant stakeholders, including those pertinent to variousproduct life-cycle phases (e.g., acceptance testing criteria)and product attributes (e.g., safety, reliability,maintainability).

Requirements also address constraints caused by theselection of design solutions (e.g., integration ofcommercial off-the-shelf products).



page 7

Requirements Development Goals

SG 1 Develop Customer RequirementsStakeholder needs, expectations, constraints,and interfaces are collected and translated intocustomer requirements.

SG 2 Develop Product RequirementsCustomer requirements are refined andelaborated to develop product and product-component requirements.

SG 3 Analyze and Validate RequirementsThe requirements are analyzed and validated,and a definition of required functionality isdeveloped.



page 8

Process Management and PerformanceThe organization’s process needs and objectivescover aspects that include the following:• characteristics of the processes• process performance objectives, such as time to

market and product quality• process effectiveness

A quantitatively managed process is institutionalizedby doing the following:• controlling the process using statistical and other

quantitative techniques such that product quality,service quality, and process performance attributesare measurable and controlled throughout theproject (internal and external quality measures andcriteria)



page 9

Key Points in Relationship of CMMIand ISO 9126/25000 - 1

CMMI takes a total life cycle view and is inclusive in itsapproach to requirements development.

Requirements development explicitly seeks to havethe developer consider quality requirements.

Project and Process Management processes explicitlyconsider product quality as process performanceobjectives.

Neither the standard nor CMMI endorses a uni-dimensional view of quality.



page 10

Key Points in Relationship of CMMIand ISO 9126/25000 - 2Product Quality Requirements are transformed intodesigns and implemented via the Technical Solutionand Product Integration process areas.

The implementation of Product Quality Requirementsare monitored and confirmed via the ProjectManagement, Verification, and Validation processareas.

CMMI acknowledges the need for interaction andperhaps iteration among the related process areas tosatisfactorily identify, specify, and address ProductQuality Requirements.



page 11

Product QualityRequirements

Product QualityEvaluation

determines

Product QualityMeasurement

supports supports

CustomerUser

Development Organization

provide

AcquirerEvaluatorDeveloper

performguidance guidance

Relating Requirements, Evaluation,and Measurement



page 12

Outline


Concepts and Models


Summary



page 13

ISO/IEC 2501nQuality Model

Division

ISO/IEC 2501nQuality Model

Division

ISO/IEC 2504nQuality

EvaluationDivision


EvaluationDivision

ISO/IEC 2502nQuality Metrics

Division

ISO/IEC 2502nQuality Metrics

Division

ISO/IEC 2500nProduct Quality General Division

ISO/IEC 2500nProduct Quality General Division

Planning and Management

Planning and Management

General Overview and Guide to the SQuaRE

General Overview and Guide to the SQuaRE

SQuaRE: Architecture


RequirementDivision


RequirementDivision



page 14

Stakeholders’Needs in theirMinds

Stated,Implied orUnawareNeeds

Collected andIdentifiedStakeholders’(Business)Needs

Selected andSpecifiedNeeds & QIURequirements

FunctionalRequirements

ExternalQualityRequirements

FunctionalDesign &InternalQualityRequirements

NonFunctionalDesign &InternalQualityRequirements

So

licit & Id

entify

Select &

Sp

ecify

Needs and Requirements

Internal and External Quality Requirements may be stated in codingstandards, project quality goal statements, process descriptions (e.g., exitcriteria), test case descriptions, etc. They need not be explicitly identified asrequirements.



page 15

The Product Quality MeasurementReference Model



page 16

Quality In Use Model (ISO/IEC 9126)

Quality In UseQuality In Use EffectivenessEffectiveness

SafetySafety

ProductivityProductivity

SatisfactionSatisfaction



page 17

•Functionality

•Reliability

•Usability

•Efficiency

•Maintainability

•Portability

Subcharacteristics

Security

Replaceability

Testability

Resource utilization

Operability

Quality Characteristics

Suitability Accuracy Interoperability

Maturity Fault tolerance Recoverability

Understandability Learnability

Time behavior

Analyzability Changeability Stability

Adaptability Installability Co-existence

Attractiveness

Compliance

Compliance

Compliance

Compliance

Comp

Comp

Internal and External SoftwareQuality Model (ISO/IEC 9126)



page 18

Outline


Concepts and Models


Summary



page 19

Quality Model Elements andMeasurement Model Elements

Attribute

Subcharacteristic

Characteristic

MeasurementPrimitive

Quality Measure

Quality Measure

Conceptual Model Operationalization

represents

represents

represents

One or moreproduce

One or moreproduce

comprise

comprise



page 20

Relating the Quality and MeasurementModelsSoftware Product

Quality

QualityCharacteristics

QualityCharacteristics

QualityCharacteristicsQuality Sub

-characteristics

QualityAttribute

QualityAttributes

Quality EvaluationReport

AssessmentAnalysis Rating

Quality Measures

Function(Formula)

MeasurementPrimitives

MeasurementMethod

Quality Attribute



page 21

Relating the Quality Measurement Model to theISO Software Measurement Process (15939)

Quality EvaluationReport

AssessmentAnalysis Rating

Quality Measures

Function(Formula)

MeasurementPrimitives

MeasurementMethod

Quality Attribute

InformationProduct

Interpretation

Indicator

Analysis Model

Derived Measures

Measurement Method

Attribute

Measurement Function

Base Measures

Analysis2503025040

Definition&

Collection2502n



page 22

CMMI Measurement & AnalysisProcess Area Goals

Align Measurement and Analysis Activities

Provide Measurement Results

Institutionalize a Managed Process



page 23

Activities for Goal 1Align Measurement and Analysis Activities• Establish Measurement Objectives• Specify Measures• Specify Data Collection and Storage Procedures• Specify Analysis Procedures

Note: The first two practices directly address the need totranslate from the conceptual to the operational.



page 24

Activities for Goal 2Provide Measurement Results• Collect Measurement Data• Analyze Measurement Data• Store Data and Results• Communicate Results



page 25

CollectData

Analyze Data

INDICATOR TEMPLATE

ObjectiveQuestionsVisual Display

Interpretation

Evolution

Assumptions

X-referenceProbing Questions

Input(s)Data Elements

Responsibilityfor Reporting

Form(s)Algorithm

80

2040

60

100

Measurement Goal #_____:

Mapping ofM&APracticesto IndicatorTemplate

INDICATOR TEMPLATE


Interpretation

Evolution

Assumptions




Form(s)Algorithm

80

2040

60

100

Measurement Goal #_____:INDICATOR TEMPLATE


Interpretation

Evolution

Assumptions




Form(s)Algorithm

80

204060

100

Measurement Goal #_____:

EstablishMeasurement

Objectives

SpecifyData

CollectionProcedures

SpecifyMeasures

SpecifyAnalysis

Procedures Communicate

Results

StoreData &Results



page 26

Measuring External Quality to ManageSoftware DevelopmentQuality Characteristic/Subcharacteric: Efficiency/Time BehaviorOperational Measure: Response Time

Objective: Track satisfaction of user requirement for systemresponse time.

Questions: What is the system response time with respect tocommon transaction? What is the variability in response time?

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

Test 1

Test 3

Test 5

Test 7

Test 9

Test 1

1Tes

t 13

Test 1

5Tes

t 17

Test 1

9Tes

t 21

Test 2

3Tes

t 25

Test 2

7Tes

t 29

Res

po

nse

Tim

e

Obs Avg Target



page 27

Outline


Concepts and Models


Summary



page 28

Summary

Measurement links the specification of requirements toacceptance criteria

Quality is conceptual; measurement is operational.

GQ(I)M provides a means for moving from the conceptualto the operational.

The ISO 25000 series and the GQ(I)M Indicator Templatetogether can help with your implementation of CMMIRequirements Development, Verification, and Validation.



page 29

Contact Information

Dave Zubrow3118 SEI4500 Fifth AvePittsburgh, Pa 15213USA

+1-412-268-5243 (v)+1-412-268-5758 (f)

[email protected]

Measuring Software Product Quality: the ISO 25000 Series ...

Documents

Transcript of Measuring Software Product Quality: the ISO 25000 Series ...