read.pudn.comread.pudn.com/downloads454/doc/1911476/ISO_IEC 9126-2.pdf · ISO/IEC TR 9126-2:2003(E)...
Transcript of read.pudn.comread.pudn.com/downloads454/doc/1911476/ISO_IEC 9126-2.pdf · ISO/IEC TR 9126-2:2003(E)...
Reference numberISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003
TECHNICAL REPORT
ISO/IECTR
9126-2
First edition2003-07-01
Software engineering — Product quality — Part 2: External metrics
Génie du logiciel — Qualité des produits —
Partie 2: Métrologie externe
ISO/IEC TR 9126-2:2003(E)
PDF disclaimer This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2003 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body in the country of the requester.
ISO copyright office Case postale 56 • CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail [email protected] Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2003 – All rights reserved
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved iii
Contents Page
Foreword............................................................................................................................................................ vi
Introduction ...................................................................................................................................................... vii
1 Scope........................................................................................................................................................... 1
2 Conformance .............................................................................................................................................. 2
3 Normative references................................................................................................................................. 2
4 Terms and definitions ................................................................................................................................ 2
5 Abbreviated terms...................................................................................................................................... 2
6 Use of software quality metrics ................................................................................................................ 3
7 How to read and use the metrics tables .................................................................................................. 4
8 Metrics tables.............................................................................................................................................. 4 8.1 Functionality metrics............................................................................................................................... 5 8.1.1 Suitability metrics ........................................................................................................................ 5 8.1.2 Accuracy metrics ......................................................................................................................... 5 8.1.3 Interoperability metrics................................................................................................................ 5 8.1.4 Security metrics............................................................................................................................ 5 8.1.5 Functionality compliance metrics .............................................................................................. 6 8.2 Reliability metrics .................................................................................................................................. 14 8.2.1 Maturity metrics.......................................................................................................................... 14 8.2.2 Fault tolerance metrics .............................................................................................................. 14 8.2.3 Recoverability metrics ............................................................................................................... 14 8.2.4 Reliability compliance metrics.................................................................................................. 14 8.3 Usability metrics .................................................................................................................................... 25 8.3.1 Understandability metrics ......................................................................................................... 25 8.3.2 Learnability metrics ................................................................................................................... 25 8.3.3 Operability metrics..................................................................................................................... 26 8.3.4 Attractiveness metrics............................................................................................................... 26 8.3.5 Usability compliance metrics.................................................................................................... 26 8.4 Efficiency metrics .................................................................................................................................. 41 8.4.1 Time behaviour metrics ............................................................................................................. 41 8.4.2 Resource utilization metrics ..................................................................................................... 41 8.4.3 Efficiency compliance metrics.................................................................................................. 41 8.5 Maintainability metrics .......................................................................................................................... 52 8.5.1 Analysability metrics ................................................................................................................. 52 8.5.2 Changeability metrics ................................................................................................................ 52 8.5.3 Stability metrics.......................................................................................................................... 52 8.5.4 Testability metrics...................................................................................................................... 52 8.5.5 Maintainability compliance metrics.......................................................................................... 52 8.6 Portability metrics.................................................................................................................................. 60 8.6.1 Adaptability metrics ................................................................................................................... 60
ISO/IEC TR 9126-2:2003(E)
iv © ISO/IEC 2003 – All rights reserved
8.6.2 Installability metrics ...................................................................................................................60 8.6.3 Co-existence metrics..................................................................................................................60 8.6.4 Replaceability metrics................................................................................................................60 8.6.5 Portability compliance metrics..................................................................................................60
Annex A (informative) Considerations When Using Metrics........................................................................68 A.1 Interpretation of measures....................................................................................................................68 A.1.1 Potential differences between test and operational contexts of use....................................68 A.1.2 Issues affecting validity of results............................................................................................69 A.1.3 Balance of measurement resources .................................................................................................69 A.1.4 Correctness of specification..............................................................................................................69 A.2 Validation of Metrics ..............................................................................................................................69 A.2.1 Desirable Properties for Metrics ...............................................................................................69 A.2.2 Demonstrating the Validity of Metrics ......................................................................................70 A.3 Use of metrics for estimation (judgement) and prediction (forecast) ..............................................71 A.3.1 Quality characteristics prediction by current data .................................................................71 A.3.2 Current quality characteristics estimation on current facts ..................................................71 A.4 Detecting deviations and anomalies in quality problem prone components ..................................72 A.5 Displaying measurement results..........................................................................................................72
Annex B (informative) Use of Quality in Use, External & Internal Metrics (Framework Example) ...........73 B.1 Introduction ............................................................................................................................................73 B.2 Overview of Development and Quality Process .................................................................................73 B.3 Quality Approach Steps ........................................................................................................................74 B.3.1 General.........................................................................................................................................74 B.3.2 Step #1 Quality requirements identification ............................................................................74 B.3.3 Step #2 Specification of the evaluation....................................................................................75 B.3.4 Step #3 Design of the evaluation ..............................................................................................77 B.3.5 Step #4 Execution of the evaluation .........................................................................................77 B.3.6 Step #5 Feedback to the organization ......................................................................................77
Annex C (informative) Detailed explanation of metric scale types and measurement types ...................78 C.1 Metric Scale Types.................................................................................................................................78 C.2 Measurement Types...............................................................................................................................79 C.2.1 Size Measure Type......................................................................................................................79 C.2.2 Time measure type .....................................................................................................................82 C.2.2.0 General.........................................................................................................................................82 C.2.3 Count measure type ...................................................................................................................83
Annex D (informative) Term(s).........................................................................................................................85 D.1 Definitions...............................................................................................................................................85 D.1.1 Quality..........................................................................................................................................85 D.1.2 Software and user.......................................................................................................................85 D.1.3 Measurement...............................................................................................................................85
Table 8.1.1 Suitability metrics ...........................................................................................................................7 Table 8.1.2 Accuracy metrics ............................................................................................................................9 Table 8.1.3 Interoperability metrics ................................................................................................................10 Table 8.1.4 Security metrics ............................................................................................................................11
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved v
Table 8.1.5 Functionality compliance metrics............................................................................................... 13 Table 8.2.1 Maturity metrics ............................................................................................................................ 15 Table 8.2.2 Fault tolerance metrics ................................................................................................................ 19 Table 8.2.3 Recoverability metrics ................................................................................................................. 21 Table 8.2.4 Reliability compliance metrics .................................................................................................... 24 Table 8.3.1 Understandability metrics ........................................................................................................... 27 Table 8.3.2 Learnability metrics...................................................................................................................... 30 Table 8.3.3 Operability metrics a) Conforms with operational user expectations .................................. 32 Table 8.3.3 Operability metrics b) Controllable ......................................................................................... 33 Table 8.3.3 Operability metrics c) Suitable for the task operation........................................................... 34 Table 8.3.3 Operability metrics d) Self descriptive (Guiding)................................................................... 35 Table 8.3.3 Operability metrics e) Operational error tolerant (Human error free) .................................. 36 Table 8.3.3 Operability metrics f) Suitable for individualisation ............................................................. 37 Table 8.3.4 Attractiveness metrics ................................................................................................................. 39 Table 8.3.5 Usability compliance metrics ...................................................................................................... 40 Table 8.4.1 Time behaviour metrics a) Response time .............................................................................. 42 Table 8.4.1 Time behaviour metrics b) Throughput .................................................................................. 44 Table 8.4.1 Time behaviour metrics c) Turnaround time .......................................................................... 45 Table 8.4.2 Resource utilisation metrics a) I/O devices resource utilisation.......................................... 47 Table 8.4.2 Resource utilisation metrics b) Memory resource utilisation............................................... 48 Table 8.4.2 Resource utilisation metrics c) Transmission resource utilisation ..................................... 49 Table 8.4.3 Efficiency compliance metrics .................................................................................................... 51 Table 8.5.1 Analysability metrics.................................................................................................................... 53 Table 8.5.2 Changeability metrics .................................................................................................................. 55 Table 8.5.3 Stability metrics ............................................................................................................................ 57 Table 8.5.4 Testability metrics ........................................................................................................................ 58 Table 8.5.5 Maintainability compliance metrics............................................................................................ 59 Table 8.6.1 Adaptability metrics ..................................................................................................................... 61 Table 8.6.2 Installability metrics ..................................................................................................................... 63 Table 8.6.3 Co-existence metrics ................................................................................................................... 65 Table 8.6.4 Replaceability metrics.................................................................................................................. 66 Table 8.6.5 Portability compliance metrics ................................................................................................... 67 Table B.1 Quality Measurement Model ......................................................................................................... 73 Table B.2 User Needs Characteristics & Weights ........................................................................................ 74 Table B.3 Quality measurement tables ......................................................................................................... 75 Table B.4 Measurement plan........................................................................................................................... 77
ISO/IEC TR 9126-2:2003(E)
vi © ISO/IEC 2003 – All rights reserved
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
In exceptional circumstances, the joint technical committee may propose the publication of a Technical Report of one of the following types:
— type 1, when the required support cannot be obtained for the publication of an International Standard, despite repeated efforts;
— type 2, when the subject is still under technical development or where for any other reason there is the future but not immediate possibility of an agreement on an International Standard;
— type 3, when the joint technical committee has collected data of a different kind from that which is normally published as an International Standard (“state of the art”, for example).
Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to be reviewed until the data they provide are considered to be no longer valid or useful.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC TR 9126-2:2003, which is a Technical Report of type 2, was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software and system engineering.
This document is being issued in the Technical Report (type 2) series of publications (according to the Procedures for the technical work of ISO/IEC JTC 1) as a “prospective standard for provisional application” in the field of external metrics for quantitatively measuring external software because there is an urgent need for guidance on how standards in this field should be used to meet an identified need.
This document is not to be regarded as an “International Standard”. It is proposed for provisional application so that information and experience of its use in practice may be gathered. Comments on the content of this document should be sent to the ISO Central Secretariat.
A review of this Technical Report (type 2) will be carried out not later than three years after its publication with the options of: extension for another three years; conversion into an International Standard; or withdrawal.
ISO/IEC 9126 consists of the following parts, under the general title Software engineering — Product quality :
Part 1: Quality model
Part 2: External metrics
Part 3: Internal metrics
Part 4: Quality in use metrics
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved vii
Introduction
This Technical Report provides external metrics for measuring attributes of six external quality characteristics defined in ISO/IEC 9126-1. The metrics listed in this Technical Report are not intended to be an exhaustive set. Developers, evaluators, quality managers and acquirers may select metrics from this Technical Report for defining requirements, evaluating software products, measuring quality aspects and other purposes. They may also modify the metrics or use metrics which are not included here. This Technical Report is applicable to any kind of software product, although each of the metrics is not always applicable to every kind of software product.
ISO/IEC 9126-1 defines terms for the software quality characteristics and how these characteristics are decomposed into subcharacteristics. ISO/IEC 9126-1, however, does not describe how any of these subcharacteristics could be measured. ISO/IEC TR 9126-2 defines external metrics, ISO/IEC TR 9126-3 defines internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or the subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use.
This Technical Report is intended to be used together with ISO/IEC 9126-1. It is strongly recommended to read ISO/IEC 14598-1 and ISO/IEC 9126-1, prior to using this Technical Report, particularly if the reader is not familiar with the use of software metrics for product specification and evaluation.
Clauses 1 to 7 and Annexes A to D are common to ISO/IEC TR 9126-2, ISO/IEC TR 9126-3, and ISO/IEC 9126-4.
TECHNICAL REPORT ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 1
Software engineering — Product quality —
Part 2: External metrics
1 Scope
This Technical Report defines external metrics for quantitatively measuring external software quality in terms of characteristics and subcharacteristics defined in ISO/IEC 9126-1, and is intended to be used together with ISO/IEC 9126-1.
This Technical Report contains:
I. an explanation of how to apply software quality metrics
II. a basic set of metrics for each subcharacteristic
III. an example of how to apply metrics during the software product life cycle
This Technical Report does not assign ranges of values of these metrics to rated levels or to grades of compliance, because these values are defined for each software product or a part of the software product, by its nature, depending on such factors as category of the software, integrity level and users' needs. Some attributes may have a desirable range of values, which does not depend on specific user needs but depends on generic factors; for example, human cognitive factors.
This Technical Report can be applied to any kind of software for any application. Users of this Technical Report can select or modify and apply metrics and measures from this Technical Report or may define application-specific metrics for their individual application domain. For example, the specific measurement of quality characteristics such as safety or security may be found in International Standards or Technical Reports provided by IEC 65 and ISO/IEC JTC 1/SC 27.
Intended users of this Technical Report include:
— Acquirer (an individual or organization that acquires or procures a system, software product or software service from a supplier);
— Evaluator (an individual or organization that performs an evaluation. An evaluator may, for example, be a testing laboratory, the quality department of a software development organization, a government organization or a user);
— Developer (an individual or organization that performs development activities, including requirements analysis, design, and testing through acceptance during the software life cycle process);
— Maintainer (an individual or organization that performs maintenance activities);
— Supplier (an individual or organization that enters into a contract with the acquirer for the supply of a system, software product or software service under the terms of the contract) when validating software quality at qualification test;
— User (an individual or organization that uses the software product to perform a specific function) when evaluating quality of software product at acceptance test;
— Quality manager (an individual or organization that performs a systematic examination of the software product or software services) when evaluating software quality as part of quality assurance and quality control.
ISO/IEC TR 9126-2:2003(E)
2 © ISO/IEC 2003 – All rights reserved
2 Conformance
There are no conformance requirements in this Technical Report.
NOTE General conformance requirements for metrics are in ISO/IEC 9126-1 Quality model.
3 Normative references
ISO/IEC 9126-1:2001, Software engineering — Product quality — Part 1: Quality model
ISO/IEC TR 9126-31), Software engineering — Product quality — Part 3: Internal metrics
ISO/IEC 9126-41), Software engineering — Product quality — Part 4: Quality in use metrics
ISO/IEC 14598-1:1999, Information technology — Software product evaluation — Part 1: General overview
ISO/IEC 14598-2:2000, Software engineering — Product evaluation — Part 2: Planning and management
ISO/IEC 14598-3:2000, Software engineering — Product evaluation — Part 3: Process for developers
ISO/IEC 14598-4:1999, Software engineering — Product evaluation — Part 4: Process for acquirers
ISO/IEC 14598-5:1998, Information technology — Software product evaluation — Part 5: Process for evaluators
ISO/IEC 14598-6:2001, Software engineering — Product evaluation — Part 6: Documentation of evaluation modules
ISO/IEC 12207:1995, Information technology — Software life cycle processes
ISO/IEC 14143-1:1998, Information technology — Software measurement — Functional size measurement — Part 1: Definition of concepts
ISO/IEC 2382-20:1990, Information technology — Vocabulary — Part 20: System development
ISO 9241-10:1996, Ergonomic requirements for office work with visual display terminals (VDTs) — Part 10: Dialogue principles
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 14598-1:1999 and ISO/IEC 9126-1:2001 apply. They are also listed in Annex D.
5 Abbreviated terms
The following abbreviations are used in this Technical Report:
SQA — Software Quality Assurance (Group)
SLCP — Software Life Cycle Processes
1) To be published.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 3
6 Use of software quality metrics
These Technical Reports (ISO/IEC TR 9126-2 External metrics, ISO/IEC TR 9126-3 Internal metrics and ISO/IEC 9126-4 Quality in use metrics) provide a suggested set of software quality metrics (external, internal and quality in use metrics) to be used with the ISO/IEC 9126-1 Quality model. The user of these Technical Reports may modify the metrics defined, and/or may also use metrics not listed. When using a modified or a new metric not identified in these Technical Reports, the user should specify how the metrics relate to the ISO/IEC 9126-1 quality model or any other substitute quality model that is being used.
The user of these Technical Reports should select the quality characteristics and subcharacteristics to be evaluated, from ISO/IEC 9126-1; identify the appropriate direct and indirect measures, identify the relevant metrics and then interpret the measurement result in an objective manner. The user of these Technical Reports also may select product quality evaluation processes during the software life cycle from the ISO/IEC 14598 series of standards. These give methods for measurement, assessment and evaluation of software product quality. They are intended for use by developers, acquirers and independent evaluators, particularly those responsible for software product evaluation (see Figure 1).
software product effect of softwareproduct
quality in usemetrics
quality inuse
internalquality
internal metrics external metrics
externalquality
contexts ofusedepends on
influences influences
depends on
Figure 1 – Relationship between types of metrics
The internal metrics may be applied to a non-executable software product during its development stages (such as request for proposal, requirements definition, design specification or source code). Internal metrics provide the users with the ability to measure the quality of the intermediate deliverables and thereby predict the quality of the final product. This allows the user to identify quality issues and initiate corrective action as early as possible in the development life cycle.
The external metrics may be used to measure the quality of the software product by measuring the behaviour of the system of which it is a part. The external metrics can only be used during the testing stages of the life cycle process and during any operational stages. The measurement is performed when executing the software product in the system environment in which it is intended to operate.
The quality in use metrics measure whether a product meets the needs of specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in a specified context of use. This can be only achieved in a realistic system environment.
User quality needs can be specified as quality requirements by quality in use metrics, by external metrics, and sometimes by internal metrics. These requirements specified by metrics should be used as criteria when a product is evaluated.
It is recommended to use internal metrics having a relationship as strong as possible with the target external metrics so that they can be used to predict the values of external metrics. However, it is often difficult to design a rigorous theoretical model that provides a strong relationship between internal metrics and external metrics. Therefore, a hypothetical model that may contain ambiguity may be designed and the extent of the relationship may be modelled statistically during the use of metrics.
Recommendations and requirements related to validity and reliability are given in ISO/IEC 9126-1, Clause A.4. Additional detailed considerations when using metrics are given in Annex A of this Technical Report.
ISO/IEC TR 9126-2:2003(E)
4 © ISO/IEC 2003 – All rights reserved
7 How to read and use the metrics tables
The metrics listed in Clause 8 are categorized by the characteristics and subcharacteristics in ISO/IEC 9126-1. The following information is given for each metric in the table:
a) Metric name: Corresponding metrics in the internal metrics table and external metrics table have similar names.
b) Purpose of the metric: This is expressed as the question to be answered by the application of the metric.
c) Method of application: Provides an outline of the application.
d) Measurement, formula and data element computations: Provides the measurement formula and explains the meanings of the used data elements.
NOTE In some situations more than one formula is proposed for a metric.
e) Interpretation of measured value: Provides the range and preferred values.
f) Metric scale type: Type of scale used by the metric. Scale types used are; Nominal scale, Ordinal scale, Interval scale, Ratio scale and Absolute scale.
NOTE A more detailed explanation is given in Annex C.
g) Measure type: Types used are; Size type (e.g. Function size, Source size), Time type (e.g. Elapsed time, User time), Count type (e.g. Number of changes, Number of failures).
NOTE A more detailed explanation is given in Annex C.
h) Input to measurement: Source of data used in the measurement.
i) ISO/IEC 12207 SLCP Reference: Identifies software life cycle process(es) where the metric is applicable.
j) Target audience: Identifies the user(s) of the measurement results.
8 Metrics tables
The metrics listed in this clause are not intended to be an exhaustive set and may not have been validated. They are listed by software quality characteristics and subcharacteristics, in the order introduced in ISO/IEC 9126-1.
Metrics, which may be applicable, are not limited to these listed here. Additional specific metrics for particular purposes are provided in other related documents, such as functional size measurement or precise time efficiency measurement.
NOTE 1 It is recommended to refer a specific metric or measurement form from specific standards, technical reports or guidelines. Functional size measurement is defined in ISO/IEC 14143. An example of precise time efficiency measurement can be referred from ISO/IEC 14756.
Metrics should be validated before application in a specific environment (see Annex A).
NOTE 2 This list of metrics is not finalized, and may be revised in future versions of this Technical Report. Readers of this Technical Report are invited to provide feedback.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 5
8.1 Functionality metrics
An external functionality metric should be able to measure an attribute such as the functional behaviour of a system containing the software. The behaviour of the system may be observed from the following perspectives:
a) Differences between the actual executed results and the quality requirements specification;
NOTE 1 The quality requirements specification for functionality is usually described as the functional requirements specification.
b) FunctionaI inadequacy detected during real user operation which is not stated but is implied as a requirement in the specification.
NOTE 2 When implied operations or functions are detected, they should be reviewed, approved and stated in the specifications. Their extent to be fulfilled should be agreed.
8.1.1 Suitability metrics
An external suitability metric should be able to measure an attribute such as the occurrence of an unsatisfying function or the occurrence of an unsatisfying operation during testing and user operation of the system.
An unsatisfying function or operation may be:
a) Functions and operations that do not perform as specified in user manuals or requirement specification.
b) Functions and operations that do not provide a reasonable and acceptable outcome to achieve the intended specific objective of the user task.
8.1.2 Accuracy metrics
An external accuracy metric should be able to measure an attribute such as the frequency of users encountering the occurrence of inaccurate matters which includes:
a) Incorrect or imprecise result caused by inadequate data; for example, data with too few significant digits for accurate calculation;
b) Inconsistency between actual operation procedures and described ones in the operation manual;
c) Differences between the actual and reasonable expected results of tasks performed during operation.
8.1.3 Interoperability metrics
An external interoperability metric should be able to measure an attribute such as the number of functions or occurrences of less communicativeness involving data and commands, which are transferred easily between the software product and other systems, other software products, or equipment which are connected.
8.1.4 Security metrics
An external security metric should be able to measure an attribute such as the number of functions with, or occurrences of security problems, which are:
a) Failing to prevent leak of secure output information or data;
b) Failing to prevent loss of important data;
c) Failing to defend against illegal access or illegal operation.
ISO/IEC TR 9126-2:2003(E)
6 © ISO/IEC 2003 – All rights reserved
NOTE 1 It is recommended that penetration tests be performed to simulate attack, because such a security attack does not normally occur in the usual testing. Real security metrics may only be taken in “real life system environment”, that is “quality in use”.
NOTE 2 Security protection requirements vary widely from the case of a stand-alone-system to the case of a system connected to the Internet. The determination of the required functionality and the assurance of their effectiveness have been addressed extensively in related standards. The user of this standard should determine security functions using appropriate methods and standards in those cases where the impact of any damage caused is important or critical. In the other case the user may limit his scope to generally accepted “Information Technology (IT)” protection measures such as virus protection backup methods and access control.
8.1.5 Functionality compliance metrics
An external functionality compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, which are the software product failing to adhere to standards, conventions, contracts or other regulatory requirements.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 7
Tabl
e 8.
1.1
Suita
bilit
y m
etric
s
Exte
rnal
sui
tabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SL
CP
Ref
eren
ce
Targ
et
audi
ence
Func
tiona
l ad
equa
cy
How
ade
quat
e ar
e th
e ev
alua
ted
func
tions
?
Num
ber o
f fun
ctio
ns th
at
are
suita
ble
for p
erfo
rmin
g th
e sp
ecifi
ed ta
sks
com
parin
g to
the
num
ber
of fu
nctio
n ev
alua
ted.
X=1-
A/B
A=
Num
ber o
f fun
ctio
ns in
whi
ch p
robl
ems
are
dete
cted
in e
valu
atio
n
B= N
umbe
r of f
unct
ions
eva
luat
ed
0 <=
X <
= 1
Th
e cl
oser
to
1.0,
the
mor
e ad
equa
te.
Abso
lute
X= C
ount
/C
ount
A=
Cou
ntB=
Cou
nt
Req
uire
men
t sp
ecifi
catio
n (R
eq. S
pec.
) Ev
alua
tion
repo
rt
6.5
Valid
atio
n,
6.3
Qua
lity
Assu
ranc
e,5.
3 Q
ualif
icat
ion
test
ing
Dev
elop
er,
SQA
Func
tiona
l im
plem
enta
tion
com
plet
enes
s
How
com
plet
e is
the
impl
emen
tatio
n ac
cord
ing
to
requ
irem
ent
spec
ifica
tions
?
Do
func
tiona
l tes
ts (b
lack
bo
x te
st) o
f the
sys
tem
ac
cord
ing
to th
e re
quire
men
t spe
cific
atio
ns.
Cou
nt th
e nu
mbe
r of
mis
sing
func
tions
det
ecte
d in
eva
luat
ion
and
com
pare
w
ith th
e nu
mbe
r of f
unct
ion
desc
ribed
in th
e re
quire
men
t spe
cific
atio
ns.
X =
1 - A
/ B
A =
Num
ber o
f mis
sing
func
tions
det
ecte
d in
ev
alua
tion
B =
Num
ber o
f fun
ctio
ns d
escr
ibed
in
requ
irem
ent s
peci
ficat
ions
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Req
. spe
c.
Eval
uatio
n re
port
6.5
Valid
atio
n,
6.3
Qua
lity
Assu
ranc
e,5.
3 Q
ualif
icat
ion
test
ing
Dev
elop
er,
SQA
FOO
TNO
TES
1 In
put t
o th
e m
easu
rem
ent p
roce
ss is
the
upda
ted
requ
irem
ent s
peci
ficat
ion.
Any
cha
nges
iden
tifie
d du
ring
life
cycl
e m
ust b
e ap
plie
d to
the
requ
irem
ent s
peci
ficat
ions
bef
ore
usin
g in
m
easu
rem
ent p
roce
ss.
2 Th
is m
etric
is s
ugge
sted
as
expe
rimen
tal u
se.
NO
TE
Any
mis
sing
func
tion
cann
ot b
e ex
amin
ed b
y te
stin
g be
caus
e it
is n
ot im
plem
ente
d. F
or d
etec
ting
mis
sing
func
tions
, it i
s su
gges
ted
that
eac
h fu
nctio
n st
ated
in a
re
quire
men
t spe
cific
atio
n be
test
ed o
ne b
y on
e du
ring
func
tiona
l tes
ting.
Suc
h re
sults
bec
ome
inpu
t to
“Fun
ctio
nal i
mpl
emen
tatio
n co
mpl
eten
ess”
met
ric. F
or d
etec
ting
func
tions
whi
ch a
re im
plem
ente
d bu
t ina
dequ
ate,
it is
sug
gest
ed th
at e
ach
func
tion
be te
sted
for
mul
tiple
spe
cifie
d ta
sks.
Suc
h re
sults
bec
ome
inpu
t to
the
“Fun
ctio
nal
adeq
uacy
” met
ric. T
here
fore
, use
rs o
f met
rics
are
sugg
este
d to
use
bot
h th
ese
met
rics
durin
g fu
nctio
nal t
estin
g.
ISO/IEC TR 9126-2:2003(E)
8 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.1
(con
tinue
d)
Exte
rnal
sui
tabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SL
CP
Ref
eren
ce
Targ
et
audi
ence
Func
tiona
l im
plem
enta
tion
cove
rage
How
cor
rect
is th
e fu
nctio
nal
impl
emen
tatio
n?
Do
func
tiona
l tes
ts (b
lack
bo
x te
st) o
f the
sys
tem
ac
cord
ing
to th
e re
quire
men
t spe
cific
atio
ns.
Cou
nt th
e nu
mbe
r of
inco
rrect
ly im
plem
ente
d or
m
issi
ng fu
nctio
ns d
etec
ted
in e
valu
atio
n an
d co
mpa
re
with
the
tota
l num
ber o
f fu
nctio
ns d
escr
ibed
in th
e re
quire
men
t spe
cific
atio
ns
Cou
nt th
e nu
mbe
r of
func
tions
that
are
com
plet
e ve
rsus
the
ones
that
are
no
t.
X=1-
A /
B
A= N
umbe
r of i
ncor
rect
ly im
plem
ente
d or
m
issi
ng fu
nctio
ns d
etec
ted
in e
valu
atio
n B=
Num
ber o
f fun
ctio
ns d
escr
ibed
in
requ
irem
ent s
peci
ficat
ions
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Req
. spe
c.
Eval
uatio
n re
port
6.5
Valid
atio
n,
6.3
Qua
lity
Assu
ranc
e,5.
3 Q
ualif
icat
ion
test
ing
Dev
elop
er,
SQA
FOO
TNO
TES
1 In
put t
o th
e m
easu
rem
ent p
roce
ss is
the
upda
ted
requ
irem
ent s
peci
ficat
ion.
Any
cha
nges
iden
tifie
d du
ring
life
cycl
e m
ust b
e ap
plie
d to
the
requ
irem
ent s
peci
ficat
ions
bef
ore
usin
g in
m
easu
rem
ent p
roce
ss.
2 Th
is m
easu
re re
pres
ents
a b
inar
y ga
te c
heck
ing
of d
eter
min
ing
the
pres
ence
of a
feat
ure.
Func
tiona
l sp
ecifi
catio
n st
abili
ty
(vol
atili
ty)
How
sta
ble
is th
e
func
tiona
l sp
ecifi
catio
n af
ter
ente
ring
oper
atio
n?
Cou
nt th
e nu
mbe
r of
func
tions
des
crib
ed i
n fu
nctio
nal s
peci
ficat
ions
th
at h
ad to
be
chan
ged
afte
r the
sys
tem
is p
ut in
to
oper
atio
n an
d co
mpa
re
with
the
tota
l num
ber o
f fu
nctio
ns d
escr
ibed
in th
e re
quire
men
t spe
cific
atio
ns.
X =
1- A
/ B
A=
Num
ber o
f fun
ctio
ns c
hang
ed a
fter
ente
ring
oper
atio
n st
artin
g fro
m e
nter
ing
oper
atio
n B=
Num
ber o
f fun
ctio
ns d
escr
ibed
in
requ
irem
ent s
peci
ficat
ions
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/ C
ount
Req
. spe
c.
Eval
uatio
n re
port
6.8
Prob
lem
R
esol
utio
n5.
4 O
pera
tion
Mai
ntai
ner
SQA
FOO
TNO
TE
This
met
ric is
sug
gest
ed a
s ex
perim
enta
l use
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 9
Tabl
e 8.
1.2
Accu
racy
met
rics
Exte
rnal
acc
urac
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Accu
racy
to
expe
ctat
ion
Are
diffe
renc
es
betw
een
the
actu
al
and
reas
onab
le
expe
cted
re
sults
acc
epta
ble?
Do
inpu
t .vs
. out
put t
est
case
s an
d co
mpa
re th
e ou
tput
to re
ason
able
ex
pect
ed re
sults
. C
ount
the
num
ber o
f cas
es
enco
unte
red
by th
e us
ers
with
an
unac
cept
able
di
ffere
nce
from
reas
onab
le
expe
cted
resu
lts.
X=A
/ T
A= N
umbe
r of c
ases
enc
ount
ered
by
the
user
s w
ith a
diff
eren
ce a
gain
st to
reas
onab
le
expe
cted
resu
lts b
eyon
d al
low
able
T=
Ope
ratio
n tim
e
0<=X
Th
e cl
oser
to
0 is
the
bette
r.
Rat
io
A= C
ount
T= T
ime
X= C
ount
/ Ti
me
Req
. spe
c.
Use
r op
erat
ion
man
ual
Hea
ring
to
user
s Te
st re
port
6.5
Valid
atio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Use
r
FOO
TNO
TE
Rea
sona
ble
expe
cted
resu
lts m
ight
be
iden
tifie
d in
a re
quire
men
t spe
cific
atio
n, a
use
r ope
ratio
n m
anua
l, or
use
rs’ e
xpec
tatio
ns.
Com
puta
tiona
l Ac
cura
cy
How
ofte
n do
the
end
user
s en
coun
ter
inac
cura
te re
sults
?
Rec
ord
the
num
ber o
f in
accu
rate
com
puta
tions
ba
sed
on s
peci
ficat
ions
.
X=A
/ T
A= N
umbe
r of i
nacc
urat
e co
mpu
tatio
ns
enco
unte
red
by u
sers
T=
Ope
ratio
n tim
e
0<=X
Th
e cl
oser
to
0 is
the
bette
r.
Rat
io
A= C
ount
T= T
ime
X= C
ount
/ Ti
me
Req
. spe
c.
Test
repo
rt 6.
5 Va
lidat
ion
6.3
Qua
lity
Assu
ranc
e
Dev
elop
er
Use
r
Prec
isio
n H
ow o
ften
do th
e en
d us
ers
enco
unte
r re
sults
with
in
adeq
uate
pr
ecis
ion
?
Rec
ord
the
num
ber o
f re
sults
with
inad
equa
te
prec
isio
n.
X=A
/ T
A= N
umbe
r of
resu
lts e
ncou
nter
ed b
y th
e us
ers
with
leve
l of p
reci
sion
diff
eren
t fro
m
requ
ired
T= O
pera
tion
time
0<=X
Th
e cl
oser
to
0 is
the
bette
r.
Rat
io
A= C
ount
T= T
ime
X= C
ount
/ Ti
me
Req
. spe
c.
Test
repo
rt
6.5
Valid
atio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Use
r
NO
TE
D
ata
elem
ents
for
com
puta
tion
of e
xter
nal
met
rics
are
desi
gned
to
use
exte
rnal
ly a
cces
sibl
e in
form
atio
n, b
ecau
se i
t is
hel
pful
for
end
use
rs,
oper
ator
s,
mai
ntai
ners
or
acqu
irers
to u
se e
xter
nal m
etric
s. T
here
fore
, the
tim
e ba
sis
met
ric o
ften
appe
ars
in e
xter
nal m
etric
s an
d is
diff
eren
t fro
m in
tern
al o
nes.
ISO/IEC TR 9126-2:2003(E)
10 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.3
Inte
rope
rabi
lity
met
rics
Exte
rnal
inte
rope
rabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Dat
a ex
chan
geab
ility
(Dat
a fo
rmat
ba
sed)
How
cor
rect
ly h
ave
the
exch
ange
in
terfa
ce fu
nctio
ns fo
r sp
ecifi
ed d
ata
trans
fer b
een
impl
emen
ted?
Test
eac
h do
wns
tream
in
terfa
ce fu
nctio
n ou
tput
re
cord
form
at o
f the
sy
stem
acc
ordi
ng to
the
data
fiel
ds s
peci
ficat
ions
. C
ount
the
num
ber o
f dat
a fo
rmat
s th
at a
re a
ppro
ved
to b
e ex
chan
ged
with
oth
er
softw
are
or s
yste
m d
urin
g te
stin
g on
dat
a ex
chan
ges
in c
ompa
ring
with
the
tota
l nu
mbe
r.
X= A
/ B
A= N
umbe
r of d
ata
form
ats
whi
ch a
re
appr
oved
to b
e ex
chan
ged
succ
essf
ully
with
ot
her s
oftw
are
or s
yste
m d
urin
g te
stin
g on
da
ta e
xcha
nges
B=
Tot
al n
umbe
r of d
ata
form
ats
to b
e ex
chan
ged
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Req
. spe
c.
(Use
r man
ual)
Test
repo
rt
6.5
Valid
atio
n D
evel
oper
FOO
TNO
TE
It is
reco
mm
ende
d to
test
spe
cifie
d da
ta tr
ansa
ctio
n.
Dat
a ex
chan
geab
ility
(Use
r’s s
ucce
ss
atte
mpt
bas
ed)
How
ofte
n do
es th
e en
d us
er fa
il to
ex
chan
ge
data
bet
wee
n ta
rget
so
ftwar
e an
d ot
her
softw
are?
H
ow o
ften
are
the
data
tran
sfer
s be
twee
n ta
rget
so
ftwar
e an
d ot
her
softw
are
succ
essf
ul?
Can
use
r usu
ally
su
ccee
d in
ex
chan
ging
dat
a?
Cou
nt th
e nu
mbe
r of c
ases
th
at in
terfa
ce fu
nctio
ns
wer
e us
ed a
nd fa
iled.
a) X
= 1
- A
/ B
A= N
umbe
r of c
ases
in w
hich
use
r fai
led
to
exch
ange
dat
a w
ith o
ther
sof
twar
e or
sy
stem
s
B= N
umbe
r of c
ases
in w
hich
use
r at
tem
pted
to e
xcha
nge
data
b)
Y=
A / T
T=
Per
iod
of o
pera
tion
time
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
0<=Y
Th
e cl
oser
to
0, is
the
bette
r.
a)
Abso
lute
b)
Rat
io
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Y= C
ount
/ Ti
me
T= T
ime
Req
. spe
c.
(Use
r man
ual)
Test
repo
rt
5.4
Ope
ratio
n M
aint
aine
r
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 11
Tabl
e 8.
1.4
Secu
rity
met
rics
Exte
rnal
sec
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Acce
ss
audi
tabi
lity
How
com
plet
e is
the
audi
t tra
il co
ncer
ning
th
e us
er a
cces
s to
th
e sy
stem
and
dat
a? Ev
alua
te th
e am
ount
of
acce
sses
that
the
syst
em
reco
rded
in th
e ac
cess
hi
stor
y da
taba
se.
X= A
/ B
A= N
umbe
r of “
user
acc
esse
s to
the
syst
em
and
data
” rec
orde
d in
the
acce
ss h
isto
ry
data
base
B=
Num
ber o
f “ u
ser a
cces
ses
to th
e sy
stem
an
d da
ta” d
one
durin
g ev
alua
tion
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Test
spe
c.
Test
repo
rt
6.5
Valid
atio
n
Dev
elop
er
FOO
TNO
TES
1 Ac
cess
es to
dat
a m
ay b
e m
easu
red
only
with
test
ing
activ
ities
. 2
This
met
ric is
sug
gest
ed a
s an
exp
erim
enta
l use
. 3
It is
reco
mm
ende
d th
at p
enet
ratio
n te
sts
be p
erfo
rmed
to s
imul
ate
atta
cks,
bec
ause
suc
h se
curit
y at
tack
s do
not
nor
mal
ly o
ccur
in th
e us
ual t
estin
g. R
eal s
ecur
ity m
etric
s m
ay o
nly
be
take
n in
"rea
l life
sys
tem
env
ironm
ent",
that
is "q
ualit
y in
use
". 4
“Use
r acc
ess
to th
e sy
stem
and
dat
a” re
cord
may
incl
ude
“viru
s de
tect
ion
reco
rd“ f
or v
irus
prot
ectio
n. T
he a
im o
f the
con
cept
of c
ompu
ter v
irus
prot
ectio
n is
to c
reat
e su
itabl
e sa
fegu
ards
w
ith w
hich
the
occu
rrenc
e of
com
pute
r viru
ses
in s
yste
ms
can
be p
reve
nted
or d
etec
ted
as e
arly
as
poss
ible
.
Acce
ss
cont
rolla
bilit
y H
ow c
ontro
llabl
e is
ac
cess
to th
e
syst
em?
Cou
nt n
umbe
r of d
etec
ted
illega
l ope
ratio
ns w
ith
com
parin
g to
num
ber o
f ille
gal o
pera
tions
as
in th
e sp
ecifi
catio
n.
X= A
/ B
A= N
umbe
r of d
etec
ted
diffe
rent
type
s of
ille
gal o
pera
tions
B=
Num
ber o
f typ
es o
f ille
gal o
pera
tions
as
in
the
spec
ifica
tion
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Test
spe
c.
Test
repo
rt O
pera
tion
repo
rt
6.5
Valid
atio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
FOO
TNO
TES
1 If
it is
nec
essa
ry to
com
plem
ent d
etec
tion
of u
nexp
ecte
d ille
gal o
pera
tions
add
ition
al in
tens
ive
abno
rmal
ope
ratio
n te
stin
g sh
ould
be
cond
ucte
d.
2 It
is re
com
men
ded
that
pen
etra
tion
test
s be
per
form
ed to
sim
ulat
e at
tack
, bec
ause
suc
h se
curit
y at
tack
s do
not
nor
mal
ly o
ccur
in th
e us
ual t
estin
g. R
eal s
ecur
ity m
etric
s m
ay o
nly
be
take
n in
"rea
l life
sys
tem
env
ironm
ent",
that
is "q
ualit
y in
use
". 3
Func
tions
pre
vent
una
utho
rized
per
sons
from
cre
atin
g, d
elet
ing
or m
odify
ing
prog
ram
s or
info
rmat
ion.
The
refo
re, i
t is
sugg
este
d to
incl
ude
such
ille
gal o
pera
tion
type
s in
test
cas
es.
ISO/IEC TR 9126-2:2003(E)
12 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.4
(con
tinue
d)
Exte
rnal
sec
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Dat
a co
rrup
tion
prev
entio
n W
hat i
s th
e fre
quen
cy
of d
ata
corru
ptio
n ev
ents
?
Cou
nt th
e oc
curre
nces
of
maj
or a
nd m
inor
dat
a co
rrupt
ion
even
ts.
a) X
= 1
– A
/ N
A= N
umbe
r of t
imes
that
a m
ajor
dat
a co
rrupt
ion
even
t occ
urre
d N
= N
umbe
r of t
est c
ases
trie
d to
cau
se d
ata
corru
ptio
n ev
ent
b)
Y= 1
- B /
N
B= N
umbe
r of t
imes
that
a m
inor
dat
a co
rrupt
ion
even
t occ
urre
d c)
Z=
A /
T or
B /
T T=
per
iod
of o
pera
tion
time
(dur
ing
oper
atio
n te
stin
g)
0<=X
<= 1
Th
e clo
ser t
o 1.
0 is
the
bette
r. 0<
=Y<=
1
The
close
r to
1.0
is th
e be
tter.
0<=Z
Th
e clo
ser t
o 0,
is
the
bette
r.
a)
Abso
lute
b)
Abso
lute
c)
Rat
io
A= C
ount
B= C
ount
N
= C
ount
X=
Cou
nt/
Cou
nt
Y= C
ount
/C
ount
T=
Tim
e Z=
Cou
nt/
Tim
e
Test
spe
c.
Test
repo
rt O
pera
tion
repo
rt
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n
Mai
ntai
ner
Dev
elop
er
FOO
TNO
TES
1 In
tens
ive
abno
rmal
ope
ratio
n te
stin
g is
nee
ded
to o
btai
n m
inor
and
maj
or d
ata
corru
ptio
n ev
ents
. 2
It is
reco
mm
ende
d to
gra
de th
e im
pact
of d
ata
corru
ptio
n ev
ents
suc
h as
the
follo
win
g ex
ampl
es:
Maj
or (f
atal
) dat
a co
rrupt
ion
even
t: - r
epro
duct
ion
and
reco
very
impo
ssib
le;
- sec
ond
affe
ctio
n di
strib
utio
n to
o w
ide;
- i
mpo
rtanc
e of
dat
a its
elf.
Min
or d
ata
corru
ptio
n ev
ent:
- rep
rodu
ctio
n or
reco
very
pos
sibl
e an
d - n
o se
cond
affe
ctio
n di
strib
utio
n;
- im
porta
nce
of d
ata
itsel
f. 3
Dat
a el
emen
ts fo
r com
puta
tion
of e
xter
nal m
etric
s ar
e de
sign
ed to
use
ext
erna
lly a
cces
sibl
e in
form
atio
n, b
ecau
se it
is h
elpf
ul fo
r end
use
rs, o
pera
tors
, mai
ntai
ners
or a
cqui
rers
to u
se
exte
rnal
met
rics.
The
refo
re, c
ount
ing
even
ts a
nd ti
mes
use
d he
re a
re d
iffer
ent f
rom
cor
resp
ondi
ng in
tern
al m
etric
. 4
It is
reco
mm
ende
d th
at p
enet
ratio
n te
sts
be p
erfo
rmed
to s
imul
ate
atta
ck, b
ecau
se s
uch
secu
rity
atta
cks
do n
ot n
orm
ally
occ
ur in
the
usua
l tes
ting.
R
eal s
ecur
ity m
etric
s m
ay o
nly
be ta
ken
in "r
eal l
ife s
yste
m e
nviro
nmen
t", th
at is
"qua
lity
in u
se"
5 Th
is m
etric
is s
ugge
sted
as
an e
xper
imen
tal u
se.
6 D
ata
back
up is
one
of t
he e
ffect
ive
way
s to
pre
vent
dat
a co
rrupt
ion.
The
cre
atio
n of
bac
k up
ens
ures
that
nec
essa
ry d
ata
can
be re
stor
ed q
uick
ly in
the
even
t tha
t par
ts o
f the
ope
rativ
e da
ta a
re lo
st. H
owev
er, d
ata
back
up
is re
gard
ed a
s a
part
of th
e co
mpo
sitio
n of
the
relia
bilit
y m
etric
s in
this
repo
rt.
7 It
is s
ugge
sted
that
this
met
ric b
e us
ed e
xper
imen
tally
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 13
Tabl
e 8.
1.5
Func
tiona
lity
com
plia
nce
met
rics
Exte
rnal
func
tiona
lity
com
plia
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Func
tiona
l co
mpl
ianc
e H
ow c
ompl
iant
is th
e fu
nctio
nalit
y of
the
prod
uct t
o ap
plic
able
re
gula
tions
, st
anda
rds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber
of it
ems
requ
iring
co
mpl
ianc
e in
the
spec
ifica
tion.
D
esig
n te
st c
ases
in
acco
rdan
ce w
ith
com
plia
nce
item
s.
Con
duct
func
tiona
l tes
ting
for t
hese
test
cas
es.
Cou
nt th
e nu
mbe
r of
com
plia
nce
item
s th
at
have
bee
n sa
tisfie
d.
X =
1 -
A /
B A=
Num
ber o
f fun
ctio
nalit
y co
mpl
ianc
e ite
ms
spec
ified
that
hav
e no
t bee
n im
plem
ente
d du
ring
test
ing
B= T
otal
num
ber o
f fun
ctio
nalit
y co
mpl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e clo
ser t
o 1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser m
anua
l or
Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns Te
st
spec
ifica
tion
and
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TES
1 It
may
be
usef
ul to
col
lect
sev
eral
mea
sure
d va
lues
alo
ng ti
me,
to a
naly
se th
e tre
nd o
f inc
reas
ingl
y sa
tisfie
d co
mpl
ianc
e ite
ms
and
to d
eter
min
e w
heth
er th
ey a
re fu
lly s
atis
fied
or n
ot.
2 It
is s
ugge
sted
to c
ount
num
ber o
f fai
lure
s, b
ecau
se p
robl
em d
etec
tion
is a
n ob
ject
ive
of e
ffect
ive
test
ing
and
also
sui
tabl
e fo
r cou
ntin
g an
d re
cord
ing.
Inte
rfac
e
stan
dard
co
mpl
ianc
e
How
com
plia
nt a
re
the
inte
rface
s to
ap
plic
able
re
gula
tions
, st
anda
rds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of
inte
rface
s th
at m
eet
requ
ired
com
plia
nce
and
com
pare
with
the
num
ber
of in
terfa
ces
requ
iring
co
mpl
ianc
e as
in th
e sp
ecifi
catio
ns.
X= A
/ B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
inte
rface
s as
spe
cifie
d B=
Tot
al n
umbe
r of i
nter
face
s re
quiri
ng
com
plia
nce
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prod
uct
desc
riptio
n of
co
mpl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns Te
st
spec
ifica
tion
and
repo
rt
6.5
Valid
atio
n D
evel
oper
FOO
TNO
TE
All s
peci
fied
attri
bute
s of
a s
tand
ard
mus
t be
test
ed.
ISO/IEC TR 9126-2:2003(E)
14 © ISO/IEC 2003 – All rights reserved
8.2 Reliability metrics
An external reliability metric should be able to measure attributes related to the behaviours of the system of which the software is a part during execution testing to indicate the extent of reliability of the software in that system during operation. Systems and software are not distinguished from each other in most cases.
8.2.1 Maturity metrics
An external maturity metric should be able to measure such attributes as the software freedom of failures caused by faults existing in the software itself.
8.2.2 Fault tolerance metrics
An external fault tolerance metric should be related to the software capability of maintaining a specified performance level in cases of operation faults or infringement of its specified interface.
8.2.3 Recoverability metrics
An external recoverability metric should be able to measure such attributes as the software with system being able to re-establish its adequate level of performance and recover the data directly affected in the case of a failure.
8.2.4 Reliability compliance metrics
An external reliability compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, in which the software product fails to adhere to standards, conventions or regulations relating to reliability.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 15
Tabl
e 8.
2.1
Mat
urity
met
rics
Exte
rnal
mat
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Estim
ated
la
tent
faul
t de
nsity
How
man
y pr
oble
ms
still
exis
t tha
t may
em
erge
as
futu
re
faul
ts?
Cou
nt th
e nu
mbe
r of f
aults
de
tect
ed d
urin
g a
defin
ed
trial
per
iod
and
pred
ict
pote
ntia
l num
ber o
f fut
ure
faul
ts u
sing
a re
liabi
lity
grow
th e
stim
atio
n m
odel
.
X= {A
BS( A
1 -
A2 )
} / B
(X
: Est
imat
ed re
sidu
ary
late
nt fa
ult d
ensi
ty)
ABS(
)= A
bsol
ute
Valu
e
A1 =
tota
l num
ber o
f pre
dict
ed la
tent
faul
ts in
a
softw
are
prod
uct
A2 =
tota
l num
ber o
f act
ually
det
ecte
d fa
ults
B=
pro
duct
siz
e
0<=X
It
depe
nds
on
stag
e of
te
stin
g.
At th
e la
ter
stag
es,
smal
ler i
s be
tter.
Abso
lute
A1=
Cou
nt
A2=
Cou
nt
B=
Siz
e X=
Cou
nt/
Siz
e
Test
repo
rt O
pera
tion
repo
rt Pr
oble
m
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Test
er
SQA
Use
r
FOO
TNO
TES
1 W
hen
tota
l num
ber o
f act
ually
det
ecte
d fa
ults
bec
omes
larg
er th
an to
tal n
umbe
r of p
redi
cted
late
nt fa
ults
, it i
s re
com
men
ded
to p
redi
ct a
gain
and
est
imat
e m
ore
larg
er n
umbe
r. Es
timat
ed la
rger
num
bers
are
inte
nded
to p
redi
ct re
ason
able
late
nt fa
ilure
s, b
ut n
ot to
mak
e th
e pr
oduc
t loo
k be
tter.
2 It
is re
com
men
ded
to u
se s
ever
al re
liabi
lity
grow
th e
stim
atio
n m
odel
s an
d ch
oose
the
mos
t su
itabl
e on
e an
d re
peat
pre
dict
ion
with
mon
itorin
g de
tect
ed fa
ults
. 3
It m
ay b
e he
lpfu
l to
pred
ict u
pper
and
low
er n
umbe
r of l
aten
t fau
lts.
4 It
is n
eces
sary
to c
onve
rt th
is v
alue
(X) t
o th
e <0
,1>
inte
rval
if m
akin
g su
mm
aris
atio
n of
cha
ract
eris
tics.
Failu
re d
ensi
ty
agai
nst t
est
case
s
How
man
y fa
ilure
s w
ere
dete
cted
dur
ing
defin
ed tr
ial p
erio
d?
Cou
nt th
e nu
mbe
r of
dete
cted
failu
res
and
perfo
rmed
test
cas
es.
X= A
1 /
A2
A1 =
num
ber o
f det
ecte
d fa
ilure
s A2
= n
umbe
r of p
erfo
rmed
test
cas
es
0<=X
It
depe
nds
on
stag
e of
te
stin
g.
At th
e la
ter
stag
es,
smal
ler i
s be
tter.
Abso
lute
A1
= C
ount
A2
= C
ount
B
= S
ize
X=
Cou
nt/
Cou
nt
Test
repo
rt O
pera
tion
repo
rt Pr
oble
m
repo
rt
5.3
Inte
grat
ion
5.3
Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Test
er
SQA
FOO
TNO
TES
1 Th
e la
rger
is th
e be
tter,
in e
arly
sta
ge o
f tes
ting.
On
the
cont
rary
, the
sm
alle
r is
the
bette
r, in
late
r sta
ge o
f tes
ting
or o
pera
tion.
It is
reco
mm
ende
d to
mon
itor t
he tr
end
of th
is m
easu
re
alon
g w
ith th
e tim
e.
2 Th
is m
etric
dep
ends
on
adeq
uacy
of t
est c
ases
so
high
ly th
at th
ey s
houl
d be
des
igne
d to
incl
ude
appr
opria
te c
ases
: e.g
., no
rmal
, exc
eptio
nal a
nd a
bnor
mal
cas
es.
3 It
is n
eces
sary
to c
onve
rt th
is v
alue
(X) t
o th
e <0
,1>
inte
rval
if m
akin
g su
mm
aris
atio
n of
cha
ract
eris
tics.
ISO/IEC TR 9126-2:2003(E)
16 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.1
(con
tinue
d)
Exte
rnal
mat
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Failu
re
reso
lutio
n H
ow m
any
failu
re
cond
ition
s ar
e re
solv
ed?
Cou
nt th
e nu
mbe
r of
failu
res
that
did
not
re
occu
r dur
ing
defin
ed tr
ial
perio
d un
der t
he s
imila
r co
nditi
ons.
M
aint
ain
a pr
oble
m
reso
lutio
n re
port
desc
ribin
g st
atus
of a
ll th
e fa
ilure
s.
X=
A1 /
A2
A1
= n
umbe
r of r
esol
ved
failu
res
A2 =
tota
l num
ber o
f act
ually
det
ecte
d fa
ilure
s
0<=X
<= 1
Th
e cl
oser
to
1.0
is b
ette
r as
mor
e fa
ilure
s ar
e re
solv
ed.
a)
Abso
lute
A1=
Cou
nt
A2=
Cou
nt
A3 =
C
ount
X=
Cou
nt/
Cou
nt
Test
repo
rt O
pera
tion
(test
) re
port
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r SQ
A M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
to m
onito
r the
tren
d w
hen
usin
g th
is m
easu
re.
2 To
tal n
umbe
r of p
redi
cted
late
nt fa
ilure
s m
ight
be
estim
ated
usi
ng re
liabi
lity
grow
th m
odel
s ad
just
ed w
ith a
ctua
l his
toric
al d
ata
rela
ting
to s
imila
r sof
twar
e pr
oduc
t. In
suc
h a
case
, the
nu
mbe
r of a
ctua
l and
pre
dict
ed fa
ilure
s ca
n be
com
para
ble
and
the
num
ber o
f res
idua
l unr
esol
ved
failu
res
can
be m
easu
rabl
e.
Faul
t den
sity
H
ow m
any
faul
ts
wer
e de
tect
ed d
urin
g de
fined
tria
l per
iod?
Cou
nt th
e nu
mbe
r of
dete
cted
fau
lts a
nd
com
pute
den
sity
.
X= A
/ B
A
= nu
mbe
r of d
etec
ted
faul
ts
B =
prod
uct s
ize
0<=X
It
depe
nds
on
stag
e of
te
stin
g.
At th
e la
ter
stag
es,
smal
ler i
s be
tter.
Abso
lute
A=
C
ount
B
= S
ize
X=
Cou
nt/
Siz
e
Test
repo
rt O
pera
tion
repo
rt Pr
oble
m
repo
rt
5.3
Inte
grat
ion
5.3
Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Test
er
SQA
FOO
TNO
TES
1 Th
e la
rger
is th
e be
tter,
in e
arly
sta
ge o
f tes
ting.
On
the
cont
rary
, the
sm
alle
r is
the
bette
r, in
late
r sta
ge o
f tes
ting
or o
pera
tion.
It is
reco
mm
ende
d to
mon
itor t
he tr
end
of th
is m
easu
re
alon
g w
ith th
e tim
e.
2 Th
e nu
mbe
r of d
etec
ted
faul
ts d
ivid
ed b
y th
e nu
mbe
r of t
est c
ases
indi
cate
s ef
fect
iven
ess
of te
st c
ases
. 3
It is
nec
essa
ry to
con
vert
this
val
ue (X
) to
the
<0,1
> in
terv
al if
mak
ing
sum
mar
isat
ion
of c
hara
cter
istic
s.
4 W
hen
coun
ting
faul
ts, p
ay a
ttent
ion
to th
e fo
llow
ings
: - P
ossi
bilit
y of
dup
licat
ion,
bec
ause
mul
tiple
repo
rts m
ay c
onta
in th
e sa
me
faul
ts a
s ot
her r
epor
t; - P
ossi
bilit
y of
oth
ers
than
faul
ts, b
ecau
se u
sers
or t
este
rs m
ay n
ot fi
gure
out
whe
ther
thei
r pro
blem
s ar
e op
erat
ion
erro
r, en
viro
nmen
tal e
rror o
r sof
twar
e fa
ilure
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 17
Tabl
e 8.
2.1
(con
tinue
d)
Exte
rnal
mat
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Faul
t rem
oval
H
ow m
any
faul
ts
have
bee
n co
rrect
ed?
Cou
nt th
e nu
mbe
r of f
aults
re
mov
ed d
urin
g te
stin
g an
d co
mpa
re w
ith th
e to
tal
num
ber o
f fau
lts d
etec
ted
and
tota
l num
ber o
f fau
lts
pred
icte
d.
a) X
= A
1 /
A2
A1
= n
umbe
r of c
orre
cted
faul
ts
A2 =
tota
l num
ber o
f act
ually
det
ecte
d fa
ults
b)
Y=
A1
/ A
3
A3 =
tota
l num
ber o
f pre
dict
ed la
tent
faul
ts in
th
e so
ftwar
e pr
oduc
t
0<=X
<= 1
Th
e cl
oser
to
1.0
is b
ette
r as
few
er
faul
ts re
mai
n.
0<=Y
Th
e cl
oser
to
1.0
is b
ette
r as
few
er
faul
ts r
emai
n.
a)
Abso
lute
b)
Abso
lute
A1=
Cou
nt
A2=
Cou
nt
A3=
Cou
nt
X= C
ount
/ C
ount
Y=
Cou
nt/
Cou
nt
Test
repo
rt O
rgan
izat
ion
data
base
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
6.5
Valid
atio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
SQA
Mai
ntai
ner
FOO
TNO
TES
1 It
is re
com
men
ded
to m
onito
r the
tren
d du
ring
a de
fined
per
iod
of ti
me.
2
Tota
l num
ber o
f pre
dict
ed la
tent
faul
ts m
ay b
e es
timat
ed u
sing
relia
bilit
y gr
owth
mod
els
adju
sted
with
act
ual h
isto
rical
dat
a re
latin
g to
sim
ilar s
oftw
are
prod
uct.
3 It
is re
com
men
ded
to m
onito
r the
est
imat
ed fa
ults
reso
lutio
n ra
tio Y
, so
that
if Y
> 1
, inv
estig
ate
the
reas
on w
heth
er it
is b
ecau
se m
ore
faul
ts h
ave
been
det
ecte
d ea
rly o
r bec
ause
so
ftwar
e pr
oduc
t con
tain
s an
unu
sual
num
ber o
f fau
lts.
Oth
erw
ise,
whe
n Y
< 1,
inve
stig
ate
whe
ther
it is
bec
ause
ther
e ar
e le
ss th
an th
e us
ual n
umbe
r of d
efec
ts in
the
softw
are
prod
ucts
or b
ecau
se th
e te
stin
g w
as n
ot a
dequ
ate
to d
etec
t all
poss
ible
faul
ts.
4 It
is n
eces
sary
to c
onve
rt th
is v
alue
(Y) t
o th
e <0
,1>
inte
rval
if m
akin
g su
mm
aris
atio
n of
cha
ract
eris
tics.
5
Whe
n co
untin
g fa
ults
, pay
atte
ntio
n to
the
poss
ibilit
y of
dup
licat
ion,
bec
ause
mul
tiple
repo
rts m
ay c
onta
in th
e sa
me
faul
ts a
s ot
her r
epor
t.
Mea
n tim
e be
twee
n fa
ilure
s (M
TBF)
How
freq
uent
ly d
oes
the
softw
are
fail
in
oper
atio
n?
Cou
nt th
e nu
mbe
r of
failu
res
occu
rred
durin
g a
defin
ed p
erio
d of
ope
ratio
n an
d co
mpu
te th
e av
erag
e in
terv
al b
etw
een
the
failu
res.
a) X
= T
1 /
A
b)
Y =
T2
/ A
T1 =
ope
ratio
n tim
e
T2 =
sum
of t
ime
inte
rval
s be
twee
n co
nsec
utiv
e fa
ilure
occ
urre
nces
A
= to
tal n
umbe
r of a
ctua
lly d
etec
ted
failu
res
(Fai
lure
s oc
curre
d du
ring
obse
rved
op
erat
ion
time)
0<X,
Y Th
e lo
nger
is
the
bette
r as
long
er ti
me
can
be
expe
cted
be
twee
n fa
ilure
s.
a)
Rat
io
b)
Rat
io
A =
Cou
nt
T1 =
T
ime
T2 =
T
ime
X =T
ime
/ C
ount
Y
=Tim
e/ C
ount
Test
repo
rt O
pera
tion
(test
) rep
ort
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
test
ing
5.4
Ope
ratio
n
Mai
ntai
ner
Use
r
FOO
TNO
TES
1 Th
e fo
llow
ing
inve
stig
atio
n m
ay b
e he
lpfu
l: - d
istri
butio
n of
tim
e in
terv
al b
etw
een
failu
re o
ccur
renc
es;
- cha
nges
of m
ean
time
alon
g w
ith in
terv
al o
pera
tion
time
perio
d;
- dis
tribu
tion
indi
catin
g w
hich
func
tion
has
frequ
ent f
ailu
re o
ccur
renc
es a
nd o
pera
tion
beca
use
of fu
nctio
n an
d us
e de
pend
ency
. 2
Failu
re ra
te o
r haz
ard
rate
cal
cula
tion
may
be
alte
rnat
ivel
y us
ed.
3 It
is n
eces
sary
to c
onve
rt th
is v
alue
(X,Y
) to
the
<0,1
> in
terv
al if
mak
ing
sum
mar
isat
ion
of c
hara
cter
istic
s.
ISO/IEC TR 9126-2:2003(E)
18 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.1
(con
tinue
d)
Exte
rnal
mat
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Test
cov
erag
e(S
peci
fied
oper
atio
n sc
enar
io te
stin
g co
vera
ge)
How
muc
h of
re
quire
d te
st c
ases
ha
ve b
een
exec
uted
du
ring
test
ing?
Cou
nt th
e nu
mbe
r of t
est
case
s pe
rform
ed d
urin
g te
stin
g an
d co
mpa
re th
e nu
mbe
r of t
est c
ases
re
quire
d to
obt
ain
adeq
uate
test
cov
erag
e.
X= A
/ B
A= N
umbe
r of a
ctua
lly p
erfo
rmed
test
cas
es
repr
esen
ting
oper
atio
n sc
enar
io d
urin
g te
stin
g B=
Num
ber o
f tes
t cas
es to
be
perfo
rmed
to
cove
r req
uire
men
ts
0<=X
<=1
The
clos
er t
o 1.
0 is
the
bette
r tes
t co
vera
ge.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Req
. spe
c.,
Tes
t spe
c. o
r U
ser m
anua
lTe
st re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g
6.5
Valid
atio
n 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Test
er
SQA
FOO
TNO
TE
Test
cas
es m
ay b
e no
rmal
ised
by
softw
are
size
, tha
t is:
test
den
sity
cov
erag
e Y=
A /
C, w
here
C=
Size
of p
rodu
ct to
be
test
ed.
The
larg
er Y
is th
e be
tter.
Size
may
be
func
tiona
l siz
e th
at u
ser c
an m
easu
re.
Test
mat
urity
Is
the
prod
uct w
ell
test
ed?
CO
MM
ENT(
S) T
his
is to
pre
dict
the
succ
ess
rate
the
prod
uct w
ill ac
hiev
e in
futu
re te
stin
g.
Cou
nt th
e nu
mbe
r of
pass
ed te
st c
ases
whi
ch
have
bee
n ac
tual
ly
exec
uted
and
com
pare
it
to th
e to
tal n
umbe
r of t
est
case
s to
be
perfo
rmed
as
per r
equi
rem
ents
.
X= A
/ B
A= N
umbe
r of p
asse
d te
st c
ases
dur
ing
test
ing
or o
pera
tion
B= N
umbe
r of t
est c
ases
to b
e pe
rform
ed to
co
ver r
equi
rem
ents
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Req
. spe
c.,
Tes
t spe
c.,
or U
ser
man
ual
Test
repo
rt O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
3 Q
ualit
y As
sura
nce
Dev
elop
er
Test
er
SQA
FOO
TNO
TES
1 It
is re
com
men
ded
to p
erfo
rm s
tress
test
ing
usin
g liv
e hi
stor
ical
dat
a es
peci
ally
from
pea
k pe
riods
. It
is a
lso
reco
mm
ende
d to
ens
ure
that
the
follo
win
g te
st ty
pes
are
exec
uted
and
pas
sed
succ
essf
ully
: - U
ser o
pera
tion
scen
ario
; - P
eak
stre
ss;
- Ove
rload
ed d
ata
inpu
t. 2
Pass
ed te
st c
ases
may
be
norm
alis
ed b
y so
ftwar
e si
ze, t
hat i
s:
pass
ed te
st c
ase
dens
ity Y
= A
/ C, w
here
C
= Si
ze o
f pro
duct
to b
e te
sted
. Th
e la
rger
Y is
bet
ter.
Size
may
be
func
tiona
l siz
e th
at u
ser c
an m
easu
re.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 19
Tabl
e 8.
2.2
Faul
t tol
eran
ce m
etric
s
Exte
rnal
faul
t tol
eran
ce m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Bre
akdo
wn
avoi
danc
e H
ow o
ften
the
softw
are
prod
uct
caus
es th
e br
eak
dow
n of
the
tota
l pr
oduc
tion
envi
ronm
ent?
Cou
nt th
e nu
mbe
r of
brea
kdow
ns o
ccur
renc
e w
ith re
spec
t to
num
ber o
f fa
ilure
s.
If it
is u
nder
ope
ratio
n,
anal
yse
log
of u
ser
oper
atio
n hi
stor
y.
X= 1
- A
/ B
A= N
umbe
r of b
reak
dow
ns
B= N
umbe
r of f
ailu
res
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A =C
ount
B
=Cou
nt
X =C
ount
/ C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TES
1 Th
e br
eakd
own
mea
ns th
e ex
ecut
ion
of a
ny u
ser t
asks
is s
uspe
nded
unt
il sy
stem
is re
star
ted,
or i
ts c
ontro
l is
lost
unt
il sy
stem
is fo
rced
to b
e sh
ut d
own.
2
Whe
n no
ne o
r fe
w fa
ilure
s ar
e ob
serv
ed, t
ime
betw
een
brea
kdow
ns m
ay b
e m
ore
suita
ble.
Failu
re
avoi
danc
e H
ow m
any
faul
t pa
ttern
s w
ere
brou
ght
unde
r con
trol t
o av
oid
criti
cal a
nd s
erio
us
failu
res?
Cou
nt th
e nu
mbe
r of
avoi
ded
faul
t pat
tern
s an
d co
mpa
re it
to th
e nu
mbe
r of
faul
t pat
tern
s to
be
cons
ider
ed
X=A
/ B
A= N
umbe
r of a
void
ed c
ritic
al a
nd s
erio
us
failu
re o
ccur
renc
es a
gain
st te
st c
ases
of
faul
t pat
tern
B=
Num
ber o
f exe
cute
d te
st c
ases
of f
ault
patte
rn (a
lmos
t cau
sing
failu
re) d
urin
g te
stin
g
0<=X
<= 1
Th
e cl
oser
to
1.0
is b
ette
r, as
the
user
ca
n m
ore
ofte
n av
oid
criti
cal o
r se
rious
fa
ilure
.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
to c
ateg
oris
e fa
ilure
avo
idan
ce le
vels
whi
ch is
the
exte
nt o
f miti
gatin
g im
pact
of f
aults
, for
exa
mpl
e:
-Crit
ical
: ent
ire s
yste
m s
tops
/ or
ser
ious
dat
abas
e de
stru
ctio
n;
-Ser
ious
: im
porta
nt fu
nctio
ns b
ecom
e in
oper
able
and
no
alte
rnat
ive
way
of o
pera
ting
(wor
karo
und)
; -A
vera
ge: m
ost f
unct
ions
are
stil
l ava
ilabl
e, b
ut li
mite
d pe
rform
ance
occ
urs
with
lim
ited
or a
ltern
ate
oper
atio
n (w
orka
roun
d);
-Sm
all:
a fe
w fu
nctio
ns e
xper
ienc
e lim
ited
perfo
rman
ce w
ith li
mite
d o
pera
tion;
-N
one:
impa
ct d
oes
not r
each
end
use
r. 2
Failu
re a
void
ance
leve
ls m
ay b
e ba
sed
on a
risk
mat
rix c
ompo
sed
by s
ever
ity o
f con
sequ
ence
and
freq
uenc
y of
occ
urre
nce
prov
ided
by
ISO
/IEC
150
26 S
yste
m a
nd s
oftw
are
inte
grity
. 3
Faul
t pat
tern
exa
mpl
es
- out
of r
ange
dat
a - d
eadl
ock
Faul
t tre
e an
alys
is te
chni
que
may
be
used
to d
etec
t fau
lt pa
ttern
s.
4 Te
st c
ases
can
incl
ude
the
hum
an in
corre
ct o
pera
tion.
ISO/IEC TR 9126-2:2003(E)
20 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.2
(con
tinue
d)
Exte
rnal
faul
t tol
eran
ce m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Inco
rrec
t op
erat
ion
avoi
danc
e
How
man
y fu
nctio
ns
are
impl
emen
ted
with
in
corre
ct o
pera
tions
av
oida
nce
capa
bilit
y? C
ount
the
num
ber o
f tes
t ca
ses
of in
corre
ct
oper
atio
ns w
hich
wer
e av
oide
d to
cau
se c
ritic
al
and
serio
us fa
ilure
s an
d co
mpa
re it
to th
e nu
mbe
r of
exe
cute
d te
st c
ases
of
inco
rrect
ope
ratio
n pa
ttern
s to
be
cons
ider
ed.
X=A
/ B
A= N
umbe
r of a
void
ed c
ritica
l and
ser
ious
fa
ilure
s oc
curre
nces
B=
Num
ber o
f exe
cute
d te
st c
ases
of i
ncor
rect
op
erat
ion
patte
rns
(alm
ost c
ausin
g fa
ilure
) du
ring
test
ing
0<=X
<= 1
Th
e cl
oser
to
1.0
is b
ette
r, as
mor
e in
corre
ct u
ser
oper
atio
n is
av
oide
d.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TES
1 Al
so d
ata
dam
age
in a
dditi
on to
sys
tem
failu
re.
2 In
corre
ct o
pera
tion
patte
rns
- In
corre
ct d
ata
type
s as
par
amet
ers
- In
corre
ct s
eque
nce
of d
ata
inpu
t -
Inco
rrect
seq
uenc
e of
ope
ratio
n.
3 Fa
ult t
ree
anal
ysis
tech
niqu
e m
ay b
e us
ed to
det
ect i
ncor
rect
ope
ratio
n pa
ttern
s.
4 Th
is m
etric
may
be
used
exp
erim
enta
lly.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 21
Tabl
e 8.
2.3
Rec
over
abili
ty m
etric
s
Exte
rnal
reco
vera
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Avai
labi
lity
How
ava
ilabl
e is
the
syst
em fo
r use
dur
ing
the
spec
ified
per
iod
of ti
me?
Test
sys
tem
in a
pr
oduc
tion
like
envi
ronm
ent f
or a
spe
cifie
d pe
riod
of ti
me
perfo
rmin
g al
l use
r ope
ratio
ns.
Mea
sure
the
repa
ir tim
e pe
riod
each
tim
e th
e sy
stem
was
una
vaila
ble
durin
g th
e tri
al.
Com
pute
mea
n tim
e to
re
pair.
a)
X= {
To /
(To
+ Tr
) }
b)
Y= A
1 /
A2
To
= o
pera
tion
time
Tr =
tim
e to
repa
ir A1
= to
tal a
vaila
ble
case
s of
use
r’s
succ
essf
ul s
oftw
are
use
whe
n us
er a
ttem
pt
to u
se
A2=
tota
l num
ber o
f cas
es o
f use
r’s a
ttem
pt
to u
se th
e so
ftwar
e du
ring
obse
rvat
ion
time.
Th
is is
from
the
user
cal
labl
e fu
nctio
n op
erat
ion
view
.
0<=X
<=1
The
larg
er
and
clos
er to
1.
0 is
bet
ter,
as th
e us
er
can
use
the
softw
are
for
mor
e tim
e.
0<=Y
<=1
The
larg
er
and
clos
er to
1.
0 is
the
bette
r.
(a),(
b)
Abso
lute
To =
Tim
eTr
= T
ime
X= T
ime/
Ti
me
A1=
Cou
ntA2
= C
ount
Y= C
ount
/C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
It is
reco
mm
ende
d th
at th
is m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
Mea
n do
wn
time
Wha
t is
the
aver
age
time
the
syst
em s
tays
un
avai
labl
e w
hen
a fa
ilure
occ
urs
befo
re
grad
ual s
tart
up?
Mea
sure
the
dow
n tim
e ea
ch ti
me
the
syst
em is
un
avai
labl
e du
ring
a sp
ecifi
ed tr
ial p
erio
d an
d co
mpu
te th
e m
ean
time.
X= T
/ N
T=
Tot
al d
own
time
N
= N
umbe
r of o
bser
ved
brea
kdow
ns
The
wor
st c
ase
or d
istri
butio
n of
dow
n tim
e sh
ould
be
mea
sure
d.
0<X
The
smal
ler i
s th
e be
tter,
syst
em w
ill be
do
wn
for
shor
ter t
ime.
Rat
io
T= T
ime
N=
Cou
ntX=
Tim
e/C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
that
this
reco
vera
bilit
y m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
2 It
is n
eces
sary
to c
onve
rt th
is v
alue
(X) t
o th
e <0
,1>
inte
rval
if m
akin
g su
mm
aris
atio
n of
cha
ract
eris
tics.
ISO/IEC TR 9126-2:2003(E)
22 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.3
(con
tinue
d)
Exte
rnal
reco
vera
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Mea
n re
cove
ry
time
W
hat i
s th
e av
erag
e tim
e th
e sy
stem
take
s to
com
plet
e re
cove
ry
from
initi
al p
artia
l re
cove
ry?
Mea
sure
the
full
reco
very
tim
es fo
r eac
h of
the
time
the
syst
em w
as b
roug
ht
dow
n du
ring
the
spec
ified
tri
al p
erio
d an
d co
mpu
te
the
mea
n tim
e.
X= S
um(T
) / B
T=
Tim
e to
reco
very
dow
ned
softw
are
syst
em a
t eac
h op
portu
nity
N
= N
umbe
r of c
ases
whi
ch o
bser
ved
softw
are
syst
em e
nter
ed in
to re
cove
ry
0<X
The
smal
ler i
s th
e be
tter.
Rat
io
T= T
ime
N=
Cou
ntX=
Tim
e/C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
to m
easu
re th
e m
axim
um ti
me
of th
e w
orst
cas
e or
dis
tribu
tion
of re
cove
ry ti
me
for m
any
case
s.
2 It
is re
com
men
ded
that
this
reco
vera
bilit
y m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
3 It
is re
com
men
ded
to d
istin
guis
h th
e gr
ades
of r
ecov
ery
diffi
culty
, for
exa
mpl
e, re
cove
ry o
f des
troye
d da
taba
se is
mor
e di
fficu
lt th
an re
cove
ry o
f des
troye
d tra
nsac
tion.
4
It is
nec
essa
ry to
con
vert
this
val
ue (X
) to
the
<0,1
> in
terv
al if
mak
ing
sum
mar
isat
ion
of c
hara
cter
istic
s.
Res
tart
abili
ty
How
ofte
n th
e sy
stem
ca
n re
star
t pro
vidi
ng
serv
ice
to u
sers
w
ithin
a re
quire
d tim
e?
Cou
nt th
e nu
mbe
r of t
imes
th
e sy
stem
rest
arts
and
pr
ovid
es s
ervi
ce to
use
rs
with
in a
targ
et re
quire
d tim
e an
d co
mpa
re it
to th
e to
tal n
umbe
r of r
esta
rts,
whe
n th
e sy
stem
was
br
ough
t dow
n du
ring
the
spec
ified
tria
l per
iod.
X =
A / B
A=
Num
ber o
f res
tarts
whi
ch m
et to
requ
ired
time
durin
g te
stin
g or
use
r ope
ratio
n su
ppor
t B=
Tot
al n
umbe
r of r
esta
rts d
urin
g te
stin
g or
us
er o
pera
tion
supp
ort
0<=X
<=1
The
larg
er
and
clos
er to
1.
0 is
bet
ter,
as th
e us
er
can
rest
art
easi
ly.
Abso
lute
A =C
ount
B
=Cou
nt
X =C
ount
/ C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
to e
stim
ate
diffe
rent
tim
e to
rest
art t
o co
rresp
ond
to th
e se
verit
y le
vel o
f ino
pera
bilit
y, s
uch
as d
ata
base
des
truct
ion,
lost
mul
ti tra
nsac
tion,
lost
sin
gle
trans
actio
n, o
r te
mpo
rary
dat
a de
stru
ctio
n.
2 It
is re
com
men
ded
that
this
reco
vera
bilit
y m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
Res
tora
bilit
y H
ow c
apab
le is
the
prod
uct i
n re
stor
ing
itsel
f afte
r abn
orm
al
even
t or a
t req
uest
?
Cou
nt th
e nu
mbe
r of
succ
essf
ul re
stor
atio
ns
and
com
pare
it to
the
num
ber o
f tes
ted
rest
orat
ion
requ
ired
in th
e sp
ecifi
catio
ns.
Res
tora
tion
requ
irem
ent
exam
ples
: da
taba
se c
heck
poin
t, tra
nsac
tion
chec
kpoi
nt,
redo
func
tion,
und
o fu
nctio
n et
c.
X= A
/ B
A= N
umbe
r of r
esto
ratio
n ca
ses
succ
essf
ully
do
ne
B= N
umbe
r of r
esto
ratio
n ca
ses
test
ed a
s pe
r req
uire
men
ts
0<=X
<=1
The
larg
er
and
clos
er to
1.
0 is
bet
ter,
as h
e pr
oduc
t is
mor
e ca
pabl
e to
re
stor
e in
de
fined
ca
ses.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/ C
ount
Req
. spe
c.,
Test
spe
c. o
r U
ser m
anua
l Te
st re
port
Ope
ratio
n re
port
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TE
It is
reco
mm
ende
d th
at th
is m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 23
Tabl
e 8.
2.3
(con
tinue
d)
Exte
rnal
reco
vera
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Res
tore
ef
fect
iven
ess
How
effe
ctiv
e is
the
rest
orat
ion
capa
bilit
y? C
ount
the
num
ber o
f te
sted
rest
orat
ion
mee
ting
targ
et re
stor
atio
n tim
e an
d co
mpa
re it
to th
e nu
mbe
r of
rest
orat
ions
requ
ired
with
spe
cifie
d ta
rget
tim
e.
X= A
/ B
A= N
umbe
r of c
ases
suc
cess
fully
rest
ored
m
eetin
g th
e ta
rget
rest
ore
time
B= N
umbe
r of c
ases
per
form
ed
0<=X
<=1
The
larg
er
and
clos
er to
1.
0 is
the
bette
r, as
the
rest
orat
ion
proc
ess
in
prod
uct i
s m
ore
effe
ctiv
e.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/ C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n 6.
5 Va
lidat
ion
Use
r M
aint
aine
r
FOO
TNO
TE
It is
reco
mm
ende
d th
at th
is m
etric
incl
udes
onl
y th
e au
tom
atic
reco
very
pro
vide
d by
the
softw
are
and
excl
udes
the
mai
nten
ance
wor
k of
hum
an.
ISO/IEC TR 9126-2:2003(E)
24 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.4
Rel
iabi
lity
com
plia
nce
met
rics
Exte
rnal
relia
bilit
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Rel
iabi
lity
com
plia
nce
How
com
plia
nt is
the
relia
bilit
y of
the
prod
uct t
o ap
plic
able
re
gula
tions
, st
anda
rds
and
conv
entio
ns.
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber
of it
ems
requ
iring
co
mpl
ianc
e as
in th
e sp
ecifi
catio
n.
X =
1 -
A / B
A=
Num
ber o
f rel
iabi
lity
com
plia
nce
item
s sp
ecifi
ed th
at h
ave
not b
een
impl
emen
ted
durin
g te
stin
g B=
Tot
al n
umbe
r of r
elia
bilit
y co
mpl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser
man
ual o
r Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns Te
st
spec
ifica
tion
and
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TE
It m
ay b
e us
eful
to c
olle
ct s
ever
al m
easu
red
valu
es a
long
tim
e, to
ana
lyse
the
trend
of i
ncre
asin
gly
satis
fied
com
plia
nce
item
s an
d to
det
erm
ine
whe
ther
they
are
fully
sat
isfie
d or
not
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 25
8.3 Usability metrics
Usability metrics measure the extent to which the software can be understood, learned, operated, attractive and compliant with usability regulations and guidelines.
Many external usability metrics are tested by users attempting to use a function. The results will be influenced by the capabilities of the users and the host system characteristics. This does not invalidate the measurements, since the evaluated software is run under explicitly specified conditions by a sample of users who are representative of an identified user group. (For general-purpose products, representatives of a range of user groups may be used.) For reliable results a sample of at least eight users is necessary, although useful information can be obtained from smaller groups. Users should carry out the test without any hints or external assistance.
Metrics for understandability, learnability and operability have two types of method of application: user test or test of the product in use.
NOTE 1 User test
Users attempting to use a function test many external metrics. These measures can vary widely among different individuals. A sample of users who are representative of an identified user group should carry out the test without any hints or external assistance. (For general-purpose products, representatives of a range of user groups may be used.) For reliable results a sample of at least eight users is necessary, although useful information can be obtained from smaller groups.
It should be possible for the measures to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of known value. Results should report the mean value and the standard error of the mean.
Many of these metrics can be tested with early prototypes of the software. Which metrics are to be applied will depend on the relative importance of different aspects of usability, and the extent of subsequent quality in use testing.
NOTE 2 Test of the product in use
Rather than test specific functions, some external metrics observe the use of a function during more general use of the product to achieve a typical task as part of a test of the quality in use (ISO/IEC 9126-4). This has the advantage that fewer tests are required. The disadvantage is that some functions may only rarely be used during normal use.
It should be possible for the measures to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of known value. Results should report the mean value and the standard error of the mean.
8.3.1 Understandability metrics
Users should be able to select a software product, which is suitable for their intended use. An external understandability metric should be able to assess whether new users can understand:
• whether the software is suitable
• how it can be used for particular tasks.
8.3.2 Learnability metrics
An external learnability metric should be able to assess how long users take to learn how to use particular functions, and the effectiveness of help systems and documentation.
Learnability is strongly related to understandability, and understandability measurements can be indicators of the learnability potential of the software.
ISO/IEC TR 9126-2:2003(E)
26 © ISO/IEC 2003 – All rights reserved
8.3.3 Operability metrics
An external operability metric should be able to assess whether users can operate and control the software. Operability metrics can be categorized by the dialogue principles in ISO 9241-10:
• suitability of the software for the task
• self-descriptiveness of the software
• controllability of the software
• conformity of the software with user expectations
• error tolerance of the software
• suitability of the software for individualization
The choice of functions to test will be influenced by the expected frequency of use of functions, the criticality of the functions, and any anticipated usability problems.
8.3.4 Attractiveness metrics
An external attractiveness metric should be able to assess the appearance of the software, and will be influenced by factors such as screen design and colour. This is particularly important for consumer products.
8.3.5 Usability compliance metrics
An external usability compliance metric should be able to assess adherence to standards, conventions, style guides or regulations relating to usability.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 27
Tabl
e 8.
3.1
Und
erst
anda
bilit
y m
etric
s
Exte
rnal
und
erst
anda
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Com
plet
enes
s of
des
crip
tion
Wha
t pro
porti
on o
f fu
nctio
ns (o
r typ
es o
f fu
nctio
ns) i
s un
ders
tood
afte
r re
adin
g th
e pr
oduc
t de
scrip
tion?
Con
duct
use
r tes
t and
in
terv
iew
use
r with
qu
estio
nnai
res
or o
bser
ve
user
beh
avio
ur.
Cou
nt th
e nu
mbe
r of
func
tions
whi
ch a
re
adeq
uate
ly u
nder
stoo
d an
d co
mpa
re w
ith th
e to
tal
num
ber o
f fun
ctio
ns in
the
prod
uct.
X =
A / B
A
= N
umbe
r of f
unct
ions
(or t
ypes
of
func
tions
) und
erst
ood
B =
Tota
l num
ber o
f fun
ctio
ns (o
r typ
es o
f fu
nctio
ns)
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er p
oten
tial u
sers
und
erst
and
the
capa
bilit
y of
the
prod
uct a
fter r
eadi
ng th
e pr
oduc
t des
crip
tion.
D
emon
stra
tion
acce
ssib
ility
W
hat p
ropo
rtion
of
the
dem
onst
ratio
ns/
tuto
rials
can
the
user
ac
cess
?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. C
ount
the
num
ber o
f fu
nctio
ns th
at a
re
adeq
uate
ly d
emon
stra
ble
and
com
pare
with
the
tota
l nu
mbe
r of f
unct
ions
re
quiri
ng d
emon
stra
tion
capa
bilit
y.
X =
A / B
A=
Num
ber o
f dem
onst
ratio
ns /
tuto
rials
that
th
e us
er s
ucce
ssfu
lly a
cces
ses
B=
Num
ber o
f dem
onst
ratio
ns /
tuto
rials
av
aila
ble
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
can
find
the
dem
onst
ratio
ns a
nd/o
r tut
oria
ls.
Dem
onst
ratio
n ac
cess
ibili
ty in
us
e
Wha
t pro
porti
on o
f th
e de
mon
stra
tions
/ tu
toria
ls c
an th
e us
er
acce
ss w
hene
ver
user
act
ually
nee
ds to
do
dur
ing
oper
atio
n?
Obs
erve
the
beha
viou
r of
the
user
who
is tr
ying
to
see
dem
onst
ratio
n/tu
toria
l. O
bser
vatio
n m
ay e
mpl
oy
hum
an c
ogni
tive
actio
n m
onito
ring
appr
oach
with
vi
deo
cam
era.
X =
A / B
A=
Num
ber o
f cas
es in
whi
ch u
ser
succ
essf
ully
see
s de
mon
stra
tion
whe
n us
er
atte
mpt
s to
see
dem
onst
ratio
n
B= N
umbe
r of c
ases
in w
hich
use
r atte
mpt
s to
see
dem
onst
ratio
n du
ring
obse
rvat
ion
perio
d
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd (v
ideo
ta
pe a
nd
actio
n re
cord
)
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
can
find
the
dem
onst
ratio
ns a
nd/o
r tut
oria
ls w
hile
usi
ng th
e pr
oduc
t.
ISO/IEC TR 9126-2:2003(E)
28 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.1
(con
tinue
d)
Exte
rnal
und
erst
anda
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Dem
onst
ratio
n ef
fect
iven
ess
Wha
t pro
porti
on o
f fu
nctio
ns c
an th
e us
er o
pera
te
succ
essf
ully
afte
r a
dem
onst
ratio
n or
tu
toria
l?
Obs
erve
the
beha
viou
r of
the
user
who
is tr
ying
to
see
dem
onst
ratio
n/tu
toria
l. O
bser
vatio
n m
ay e
mpl
oy
hum
an c
ogni
tive
actio
n m
onito
ring
appr
oach
with
vi
deo
cam
era.
X =
A / B
A=
Num
ber o
f fun
ctio
ns o
pera
ted
succ
essf
ully
B=
Num
ber o
f dem
onst
ratio
ns/tu
toria
ls
acce
ssed
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
can
ope
rate
func
tions
suc
cess
fully
afte
r an
onlin
e de
mon
stra
tion
or tu
toria
l. Ev
iden
t fu
nctio
ns
Wha
t pro
porti
on o
f fu
nctio
ns (o
r typ
es o
f fu
nctio
n) c
an b
e id
entif
ied
by th
e us
er
base
d up
on s
tart
up
cond
ition
s?
Con
duct
use
r tes
t and
in
terv
iew
use
r with
qu
estio
nnai
res
or o
bser
ve
user
beh
avio
ur.
Cou
nt th
e nu
mbe
r of
func
tions
that
are
evi
dent
to
the
user
and
com
pare
w
ith th
e to
tal n
umbe
r of
func
tions
.
X =
A / B
A
= N
umbe
r of
func
tions
(or t
ypes
of
func
tions
) ide
ntifi
ed b
y th
e us
er
B =
Tota
l num
ber o
f act
ual f
unct
ions
(or
type
s of
func
tions
)
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
are
abl
e to
loca
te fu
nctio
ns b
y ex
plor
ing
the
inte
rface
(e.g
. by
insp
ectin
g th
e m
enus
). Fu
nctio
n un
ders
tand
-ab
ility
Wha
t pro
porti
on o
f th
e pr
oduc
t fun
ctio
ns
will
the
user
be
able
to
und
erst
and
corre
ctly
?
Con
duct
use
r tes
t and
in
terv
iew
use
r with
qu
estio
nnai
res.
C
ount
the
num
ber o
f use
r in
terfa
ce fu
nctio
ns w
here
pu
rpos
es a
re e
asily
un
ders
tood
by
the
user
an
d co
mpa
re w
ith th
e nu
mbe
r of f
unct
ions
av
aila
ble
for u
ser.
X= A
/ B
A= N
umbe
r of i
nter
face
func
tions
who
se
purp
ose
is c
orre
ctly
des
crib
ed b
y th
e us
er
B= N
umbe
r of f
unct
ions
ava
ilabl
e fro
m th
e in
terfa
ce
0 <=
X <
= 1
The
clos
er to
1.
0, th
e be
tter.
Abso
lute
A=
Cou
nt
B= C
ount
X= C
ount
/C
ount
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
are
abl
e to
und
erst
and
func
tions
by
expl
orin
g th
e in
terfa
ce (e
.g. b
y in
spec
ting
the
men
us).
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 29
Tabl
e 8.
3.1
(con
tinue
d)
Exte
rnal
und
erst
anda
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Und
erst
anda
ble
inpu
t and
ou
tput
Can
use
rs
unde
rsta
nd w
hat i
s re
quire
d as
inpu
t dat
a an
d w
hat i
s pr
ovid
ed
as o
utpu
t by
softw
are
syst
em?
Con
duct
use
r tes
t and
in
terv
iew
use
r with
qu
estio
nnai
res
or o
bser
ve
user
beh
avio
ur.
Cou
nt th
e nu
mbe
r of i
nput
an
d ou
tput
dat
a ite
ms
unde
rsto
od b
y th
e us
er
and
com
pare
with
the
tota
l nu
mbe
r of t
hem
ava
ilabl
e fo
r use
r.
X= A
/ B
A= N
umbe
r of i
nput
and
out
put d
ata
item
s w
hich
use
r suc
cess
fully
und
erst
ands
B=
Num
ber o
f inp
ut a
nd o
utpu
t dat
a ite
ms
av
aila
ble
from
the
inte
rface
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r M
aint
aine
r
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
can
und
erst
and
the
form
at in
whi
ch d
ata
shou
ld b
e in
put a
nd c
orre
ctly
iden
tify
the
mea
ning
of o
utpu
t dat
a.
ISO/IEC TR 9126-2:2003(E)
30 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.2
Lear
nabi
lity
met
rics
Exte
rnal
lear
nabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ease
of
func
tion
lear
ning
How
long
doe
s th
e us
er ta
ke to
lear
n to
us
e a
func
tion?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
.
T= M
ean
time
take
n to
lear
n to
use
a
func
tion
corre
ctly
0<T
The
shor
ter i
s th
e be
tter.
Rat
io
T= T
ime
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r M
aint
aine
r
FOO
TNO
TE
This
met
ric is
gen
eral
ly u
sed
as o
ne o
f exp
erie
nced
and
just
ified
. Ea
se o
f le
arni
ng to
pe
rfor
m a
task
in
use
How
long
doe
s th
e us
er ta
ke to
lear
n ho
w to
per
form
the
spec
ified
task
ef
ficie
ntly
?
Obs
erve
use
r beh
avio
ur
from
whe
n th
ey s
tart
to
lear
n un
til th
ey b
egin
to
oper
ate
effic
ient
ly.
T= S
um o
f use
r ope
ratio
n tim
e un
til u
ser
achi
eved
to p
erfo
rm th
e sp
ecifi
ed ta
sk w
ithin
a
shor
t tim
e
0<T
The
shor
ter i
s th
e be
tter.
Rat
io
T= T
ime
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r M
aint
aine
r
FOO
TNO
TES
1 It
is re
com
men
ded
to d
eter
min
e an
exp
ecte
d us
er’s
ope
ratin
g tim
e as
a s
hort
time.
Suc
h us
er’s
ope
ratin
g tim
e m
ay b
e th
e th
resh
old,
for e
xam
ple,
whi
ch is
70%
of t
ime
at th
e fir
st u
se a
s th
e fa
ir pr
opor
tion.
2
Effo
rt m
ay a
ltern
ativ
ely
repr
esen
t tim
e by
per
son-
hour
uni
t.
Effe
ctiv
enes
s of
the
user
do
cum
enta
tion
and/
or h
elp
syst
em
Wha
t pro
porti
on o
f ta
sks
can
be
com
plet
ed c
orre
ctly
af
ter u
sing
the
user
do
cum
enta
tion
and/
or
help
sys
tem
?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. C
ount
the
num
ber o
f tas
ks
succ
essf
ully
com
plet
ed
afte
r acc
essi
ng o
nlin
e he
lp
and/
or d
ocum
enta
tion
and
com
pare
with
the
tota
l nu
mbe
r of t
asks
test
ed.
X= A
/ B
A= N
umbe
r of t
asks
suc
cess
fully
com
plet
ed
afte
r acc
essi
ng o
nlin
e he
lp a
nd/o
r do
cum
enta
tion
B =
Tota
l of n
umbe
r of t
asks
test
ed
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
Thre
e m
etric
s ar
e po
ssib
le: c
ompl
eten
ess
of th
e do
cum
enta
tion,
com
plet
enes
s of
the
help
faci
lity,
or c
ompl
eten
ess
of th
e he
lp a
nd d
ocum
enta
tion
used
in c
ombi
natio
n.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 31
Tabl
e 8.
3.2
(con
tinue
d)
Exte
rnal
lear
nabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Effe
ctiv
enes
s of
use
r do
cum
enta
tion
and/
or h
elp
syst
ems
in u
se
Wha
t pr
opor
tion
of
func
tions
can
be
used
co
rrect
ly a
fter r
eadi
ng
the
docu
men
tatio
n or
us
ing
help
sys
tem
s?
Obs
erve
use
r beh
avio
ur.
Cou
nt th
e nu
mbe
r of
func
tions
use
d co
rrect
ly
afte
r rea
ding
the
docu
men
tatio
n or
usi
ng
help
sys
tem
san
d co
mpa
re
with
the
tota
l num
ber o
f fu
nctio
ns.
X =
A / B
A
= N
umbe
r of f
unct
ions
that
can
be
used
B
= To
tal o
f num
ber o
f fun
ctio
ns p
rovi
ded
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
This
met
ric is
gen
eral
ly u
sed
as o
ne o
f exp
erie
nced
and
just
ified
met
rics
rath
er th
an th
e ot
hers
. H
elp
acce
ssib
ility
W
hat p
ropo
rtion
of
the
help
topi
cs c
an
the
user
loca
te?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. C
ount
the
num
ber o
f tas
ks
for w
hich
cor
rect
onl
ine
help
is lo
cate
d an
d co
mpa
re w
ith th
e to
tal
num
ber o
f tas
ks te
sted
.
X =
A / B
A
= N
umbe
r of t
asks
for w
hich
cor
rect
onl
ine
help
is lo
cate
d
B =
Tota
l of n
umbe
r of t
asks
test
ed
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A=
Cou
ntB=
Cou
nt
X= C
ount
/C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
Hel
p fr
eque
ncy
How
freq
uent
ly d
oes
a us
er h
ave
to a
cces
s he
lp to
lear
n op
erat
ion
to c
ompl
ete
his/
her w
ork
task
?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. C
ount
the
num
ber o
f cas
es
that
a u
ser a
cces
ses
help
to
com
plet
e hi
s/he
r tas
k.
X =
A A
= N
umbe
r of a
cces
ses
to h
elp
until
a u
ser
com
plet
es h
is/h
er ta
sk.
0<=
X
The
clos
er to
0
is th
e be
tter.
Abso
lute
X=
Cou
nt
A =C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
ISO/IEC TR 9126-2:2003(E)
32 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
a) C
onfo
rms
with
ope
ratio
nal u
ser e
xpec
tatio
ns
Exte
rnal
Ope
rabi
lity
met
rics
a) C
onfo
rms
with
ope
ratio
nal u
ser e
xpec
tatio
ns
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
a) X
= 1
- A
/ B
A= N
umbe
r of m
essa
ges
or fu
nctio
ns w
hich
us
er fo
und
unac
cept
ably
inco
nsis
tent
with
th
e us
er’s
exp
ecta
tion
B=
Num
ber o
f mes
sage
s or
func
tions
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
a)
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
Ope
ratio
nal
cons
iste
ncy
in
use
How
con
sist
ent a
re
the
com
pone
nt o
f the
us
er in
terfa
ce?
Obs
erve
the
beha
viou
r of
the
user
and
ask
the
opin
ion.
b) Y
= N
/ U
OT
N=
Num
ber o
f ope
ratio
ns w
hich
use
r fou
nd
unac
cept
ably
inco
nsis
tent
with
the
user
’s
expe
ctat
ion
U
OT=
use
r ope
ratin
g tim
e (d
urin
g ob
serv
atio
n pe
riod)
0<=Y
Th
e sm
alle
r an
d cl
oser
to
0.0
is th
e be
tter.
b)
Rat
io
UO
T=
Tim
e N
= C
ount
Y= C
ount
/Ti
me
FOO
TNO
TES
1 U
ser’s
exp
erie
nce
of o
pera
tion
is u
sual
ly h
elpf
ul to
reco
gnis
e se
vera
l ope
ratio
n pa
ttern
s, w
hich
der
ive
user
’s e
xpec
tatio
n.
2 Bo
th o
f “in
put p
redi
ctab
ility”
and
“out
put p
redi
ctab
ility”
are
effe
ctiv
e fo
r ope
ratio
nal c
onsi
sten
cy.
3 Th
is m
etric
may
be
used
to m
easu
re “E
asy
to d
eriv
e op
erat
ion”
and
“Sm
ooth
Com
mun
icat
ion”
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 33
Tab
le 8
.3.3
Ope
rabi
lity
met
rics
b)
Con
trol
labl
e
Exte
rnal
Ope
rabi
lity
met
rics
b) C
ontr
olla
ble
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Erro
r cor
rect
ion
Can
use
r eas
ily
corre
ct e
rror o
n ta
sks?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
.
T= T
c -
Ts
Tc =
Tim
e of
com
plet
ing
corre
ctio
n of
sp
ecifi
ed ty
pe e
rrors
of p
erfo
rmed
task
Ts
= T
ime
of s
tarti
ng c
orre
ctio
n of
spe
cifie
d ty
pe e
rrors
of p
erfo
rmed
task
0<T
The
shor
ter i
s th
e be
tter.
Rat
io
Ts, T
c=
Tim
e T=
Tim
e
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
Use
r of t
his
met
ric is
sug
gest
ed to
spe
cify
type
s of
erro
rs fo
r tes
t cas
es b
y co
nsid
erin
g, fo
r exa
mpl
e, s
ever
ity (d
ispl
ayin
g er
ror o
r des
troyi
ng d
ata)
, typ
e of
inpu
t/out
put e
rror (
inpu
t tex
t erro
r, ou
tput
dat
a er
ror t
o da
taba
se o
r gra
phic
al e
rror o
n di
spla
y) o
r typ
e of
erro
r ope
ratio
nal s
ituat
ion
(inte
ract
ive
use
or e
mer
gent
ope
ratio
n).
Erro
r cor
rect
ion
in u
se
Can
use
r eas
ily
reco
ver h
is/h
er e
rror
or re
try ta
sks?
Obs
erve
the
beha
viou
r of
the
user
who
is o
pera
ting
softw
are.
a)
X= A
/ U
OT
A= n
umbe
r of t
imes
that
the
user
suc
ceed
s to
can
cel t
heir
erro
r ope
ratio
n U
OT=
use
r ope
ratin
g tim
e du
ring
obse
rvat
ion
perio
d C
OM
MEN
T(S)
W
hen
func
tion
is te
sted
one
by
one
, the
ratio
can
be
also
cal
cula
ted,
that
is
the
ratio
of n
umbe
r of f
unct
ions
whi
ch u
ser
succ
eeds
to c
ance
l his
/her
ope
ratio
n to
all
func
tions
.
0<=X
Th
e hi
gher
is
the
bette
r.
Rat
io
A= C
ount
U
OT
=
Tim
e X
= C
ount
/ T
ime
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
C
an u
ser e
asily
re
cove
r his
/her
inpu
t? O
bser
ve th
e be
havi
our o
f th
e us
er w
ho is
ope
ratin
g so
ftwar
e.
b)
X =
A / B
A=
Num
ber o
f scr
eens
or f
orm
s w
here
the
inpu
t dat
a w
ere
succ
essf
ully
mod
ified
or
chan
ged
befo
re b
eing
ela
bora
ted
B =
Num
ber o
f scr
eens
or f
orm
s w
here
use
r tri
ed to
mod
ify o
r to
chan
ge th
e in
put d
ata
durin
g ob
serv
ed u
ser o
pera
ting
time
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
, B=
Cou
ntX=
Cou
nt/
Cou
nt
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
ISO/IEC TR 9126-2:2003(E)
34 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
c)
Sui
tabl
e fo
r the
task
ope
ratio
n
Exte
rnal
Ope
rabi
lity
met
rics
c) S
uita
ble
for t
he ta
sk o
pera
tion
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Def
ault
valu
e av
aila
bilit
y in
us
e
Can
use
r eas
ily
sele
ct p
aram
eter
va
lues
for h
is/h
er
conv
enie
nt
oper
atio
n?
Obs
erve
the
beha
viou
r of
the
user
who
is o
pera
ting
softw
are.
C
ount
how
man
y tim
es
user
atte
mpt
s to
est
ablis
h or
to s
elec
t par
amet
er
valu
es a
nd fa
ils, (
beca
use
user
can
not
use
def
ault
valu
es p
rovi
ded
by th
e so
ftwar
e).
X =
1 - A
/ B
A= T
he n
umbe
r of t
imes
that
the
user
fail
to
esta
blis
h or
to s
elec
t par
amet
er v
alue
s in
a
shor
t per
iod
(bec
ause
use
r can
not
use
de
faul
t val
ues
prov
ided
by
the
softw
are)
B=
Tot
al n
umbe
r of t
imes
that
the
user
at
tem
pt to
est
ablis
h or
to s
elec
t par
amet
er
valu
es
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TES
1 It
is re
com
men
ded
to o
bser
ve a
nd re
cord
ope
rato
r’s b
ehav
iour
and
dec
ide
how
long
per
iod
is a
llow
able
to s
elec
t par
amet
er v
alue
s as
“sho
rt pe
riod”
. 2
Whe
n pa
ram
eter
set
ting
func
tion
is te
sted
by
each
func
tion,
the
ratio
of a
llow
able
func
tion
can
be a
lso
calc
ulat
ed.
3 It
is re
com
men
ded
to c
ondu
ct fu
nctio
nal t
est t
hat c
over
s pa
ram
eter
-set
ting
func
tions
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 35
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
d)
Sel
f des
crip
tive
(Gui
ding
)
Exte
rnal
Ope
rabi
lity
met
rics
d) S
elf d
escr
iptiv
e (G
uidi
ng)
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Mes
sage
un
ders
tand
-ab
ility
in u
se
Can
use
r eas
ily
unde
rsta
nd
mes
sage
s fro
m
softw
are
syst
em?
Is th
ere
any
m
essa
ge w
hich
ca
used
the
user
a
dela
y in
un
ders
tand
ing
befo
re
star
ting
the
next
ac
tion?
C
an u
ser e
asily
m
emor
ise
impo
rtant
m
essa
ge?
Obs
erve
use
r beh
avio
ur
who
is o
pera
ting
softw
are.
X
= A
/ UO
T A
= nu
mbe
r of t
imes
that
the
user
pau
ses
for
a lo
ng p
erio
d or
suc
cess
ivel
y an
d re
peat
edly
fa
ils a
t the
sam
e op
erat
ion,
bec
ause
of t
he
lack
of m
essa
ge c
ompr
ehen
sion
. U
OT
= us
er o
pera
ting
time
(obs
erva
tion
perio
d)
0<=X
Th
e sm
alle
r an
d cl
oser
to
0.0
is th
e be
tter.
Rat
io
A =C
ount
U
OT
=
Tim
e X
= C
ount
/ T
ime
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TES
1 Th
e ex
tent
of e
ase
of m
essa
ge c
ompr
ehen
sion
is re
pres
ente
d by
how
long
that
mes
sage
cau
sed
dela
y in
use
r un
ders
tand
ing
befo
re s
tarti
ng th
e ne
xt a
ctio
n.
Ther
efor
e, it
is re
com
men
ded
to o
bser
ve a
nd re
cord
ope
rato
r’s b
ehav
iour
and
dec
ide
wha
t len
gth
of p
ause
is c
onsi
dere
d a
“long
per
iod”
. 2
It is
reco
mm
ende
d to
inve
stig
ate
the
follo
win
g as
pos
sibl
e ca
uses
of t
he p
robl
ems
of u
ser’s
mes
sage
com
preh
ensi
on.
a)At
tent
iven
ess
: Atte
ntiv
enes
s im
plie
s th
at u
ser s
ucce
ssfu
lly re
cogn
ises
impo
rtant
mes
sage
s pr
esen
ting
info
rmat
ion
such
as
guid
ance
on
next
use
r act
ion,
nam
e of
dat
a ite
ms
to b
e lo
oked
at,
and
war
ning
of c
aref
ul o
pera
tion.
- D
oes
user
eve
r fai
l to
wat
ch w
hen
enco
unte
ring
impo
rtant
mes
sage
s?
- Can
use
r avo
id m
ista
kes
in o
pera
tion,
bec
ause
of
reco
gnis
ing
impo
rtant
mes
sage
s?
b) M
emor
abilit
y: M
emor
abilit
y im
plie
s th
at u
ser r
emem
ber i
mpo
rtant
mes
sage
s pr
esen
ting
info
rmat
ion
such
as
guid
ance
on
the
next
use
r act
ion,
nam
e of
dat
a ite
ms
to b
e lo
oked
at,
and
war
ning
of c
aref
ul o
pera
tion.
- C
an u
ser e
asily
rem
embe
r im
porta
nt m
essa
ges?
- I
s re
mem
berin
g im
porta
nt m
essa
ges
help
ful t
o th
e us
er?
- Is
it re
quire
d fo
r the
use
r to
rem
embe
r onl
y a
few
impo
rtant
mes
sage
s an
d no
t so
muc
h?
3 W
hen
mes
sage
s ar
e te
sted
one
by
one,
the
ratio
of c
ompr
ehen
ded
mes
sage
s to
the
tota
l can
be
also
cal
cula
ted.
4
Whe
n se
vera
l use
rs a
re o
bser
ved
who
are
par
ticip
ants
of o
pera
tiona
l tes
ting,
the
ratio
of u
sers
who
com
preh
ende
d m
essa
ges
to a
ll us
ers
can
be c
alcu
late
d.
Self-
expl
anat
ory
erro
r mes
sage
s
In w
hat p
ropo
rtion
of
erro
r con
ditio
ns d
oes
the
user
pro
pose
the
corre
ct re
cove
ry
actio
n?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. X=
A /
B A
=Num
ber o
f erro
r con
ditio
ns fo
r whi
ch th
e us
er p
ropo
ses
the
corre
ct re
cove
ry a
ctio
n B
=Num
ber o
f erro
r con
ditio
ns te
sted
0 <=
X <
= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
X =C
ount
/ C
ount
A
=Cou
nt
B =C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
This
met
ric is
gen
eral
ly u
sed
as o
ne o
f exp
erie
nced
and
just
ified
.
ISO/IEC TR 9126-2:2003(E)
36 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
e)
Ope
ratio
nal e
rror
tole
rant
(Hum
an e
rror
free
)
Exte
rnal
ope
rabi
lity
met
rics
e) O
pera
tiona
l err
or to
lera
nt (H
uman
err
or fr
ee)
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ope
ratio
nal
erro
r re
cove
rabi
lity
in u
se
Can
use
r eas
ily
reco
ver h
is/h
er w
orse
si
tuat
ion?
Obs
erve
the
beha
viou
r of
the
user
who
is o
pera
ting
softw
are.
X =
1 -
A /
B
A= N
umbe
r of u
nsuc
cess
fully
reco
vere
d si
tuat
ion
(afte
r a u
ser e
rror o
r cha
nge)
in
whi
ch u
ser w
as n
ot in
form
ed a
bout
a ri
sk b
y th
e sy
stem
B=
Num
ber o
f use
r erro
rs o
r cha
nges
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
, B=
Cou
ntX=
Cou
nt/
Cou
nt
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
The
form
ula
abov
e is
repr
esen
tativ
e of
the
wor
st c
ase.
Use
r of t
his
met
ric m
ay ta
ke a
ccou
nt o
f the
com
bina
tion
of 1
) the
num
ber o
f erro
rs w
here
the
user
is /
is n
ot w
arne
d by
the
softw
are
syst
em a
nd 2
) the
num
ber o
f occ
asio
ns w
here
the
user
suc
cess
fully
/ un
succ
essf
ully
reco
vers
the
situ
atio
n.
Tim
e be
twee
n hu
man
err
or
oper
atio
ns in
us
e
Can
use
r ope
rate
the
softw
are
long
eno
ugh
with
out h
uman
erro
r? O
bser
ve th
e be
havi
our o
f th
e us
er w
ho is
ope
ratin
g so
ftwar
e.
X =
T / N
(at t
ime
t dur
ing
[ t-T
, t] )
T
= op
erat
ion
time
perio
d du
ring
obse
rvat
ion
( or T
he s
um o
f ope
ratin
g tim
e be
twee
n us
er’s
hum
an e
rror o
pera
tions
) N
= nu
mbe
r of o
ccur
renc
es o
f use
r’s h
uman
er
ror o
pera
tion
0<X
The
high
er is
th
e be
tter.
Rat
io
T =
Tim
e N
= C
ount
X =
Ti
me
/ C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TES
1 H
uman
erro
r ope
ratio
n m
ay b
e de
tect
ed b
y co
untin
g be
low
use
r’s b
ehav
iour
: a)
Sim
ple
hum
an e
rror (
Slip
s): T
he n
umbe
r of t
imes
that
the
user
just
sim
ply
mak
es e
rrors
to in
put o
pera
tion;
b)
Inte
ntio
nal e
rror (
Mis
take
s): T
he n
umbe
r of t
imes
that
the
user
repe
ats
fail
an e
rror a
t the
sam
e op
erat
ion
with
mis
unde
rsta
ndin
g du
ring
obse
rvat
ion
perio
d;
c) O
pera
tion
hesi
tatio
n pa
use:
The
num
ber o
f tim
es th
at th
e us
er p
ause
s fo
r a lo
ng p
erio
d w
ith h
esita
tion
durin
g ob
serv
atio
n pe
riod.
U
ser o
f thi
s m
etric
is s
ugge
sted
to m
easu
re s
epar
atel
y fo
r eac
h ty
pe li
sted
abo
ve.
2 It
seem
s th
at a
n op
erat
ion
paus
e im
plie
s a
user
’s h
esita
tion
oper
atio
n.
It de
pend
s on
the
func
tion,
ope
ratio
n pr
oced
ure,
app
licat
ion
dom
ain,
and
use
r whe
ther
it is
con
side
red
a lo
ng p
erio
d or
not
for t
he u
ser t
o pa
use
the
oper
atio
n. T
here
fore
, the
eva
luat
or is
re
ques
ted
to ta
ke th
em in
to a
ccou
nt a
nd d
eter
min
e th
e re
ason
able
thre
shol
d tim
e. F
or a
n in
tera
ctiv
e op
erat
ion,
a "l
ong
perio
d" th
resh
old
rang
e of
1m
in. t
o 3
min
.
How
freq
uent
ly d
oes
the
user
suc
cess
fully
co
rrect
inpu
t erro
rs?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. a)
X=
A /
B
A= N
umbe
r of i
nput
erro
rs w
hich
the
user
su
cces
sful
ly c
orre
cts
B= N
umbe
r of a
ttem
pts
to c
orre
ct in
put
erro
rs
0<=X
<=1
The
clos
er to
1.
0 is
the
bette
r.
a)
Abso
lute
A= C
ount
B=
Cou
ntX=
Cou
nt/
Cou
nt
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
Und
oabi
lity
(Use
r err
or
corr
ectio
n)
How
freq
uent
ly d
oes
the
user
cor
rect
ly
undo
erro
rs?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. b)
Y=
A /
B
A= N
umbe
r of e
rror c
ondi
tions
whi
ch th
e us
er s
ucce
ssfu
lly c
orre
cts
B= T
otal
num
ber o
f erro
r con
ditio
ns te
sted
0 <=
Y <
= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
b)
Abso
lute
A= C
ount
B=
Cou
nt
Y= C
ount
/ C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
This
met
ric is
gen
eral
ly u
sed
as o
ne o
f exp
erie
nced
and
just
ified
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 37
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
f)
Sui
tabl
e fo
r ind
ivid
ualis
atio
n
Exte
rnal
ope
rabi
lity
met
rics
f) S
uita
ble
for i
ndiv
idua
lisat
ion
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Cus
tom
isab
ility
Can
use
r eas
ily
cust
omis
e op
erat
ion
proc
edur
es fo
r his
/her
co
nven
ienc
e?
Can
a u
ser,
who
in
stru
cts
end
user
s,
easi
ly s
et c
usto
mis
ed
oper
atio
n pr
oced
ure
tem
plat
es fo
r pr
even
ting
thei
r er
rors
? W
hat p
ropo
rtion
of
func
tions
can
be
cust
omis
ed?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. X=
A /
B
A= N
umbe
r of f
unct
ions
suc
cess
fully
cu
stom
ised
B=
Num
ber o
f atte
mpt
s to
cus
tom
ise
0 <=
X <
= 1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/ C
ount
Use
r man
ual
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TES
1 R
atio
of u
ser’s
failu
res
to c
usto
mis
e m
ay b
e m
easu
red.
Y
= 1
- (C
/ D
) C
= N
umbe
r of c
ases
in w
hich
a u
ser f
ails
to c
usto
mis
e op
erat
ion
D =
Tot
al n
umbe
r of c
ases
in w
hich
a u
ser a
ttem
pted
to c
usto
mis
e op
erat
ion
for h
is/h
er c
onve
nien
ce.
0<=Y
<= 1
, The
clo
ser t
o 1.
0 is
the
bette
r. 2
It is
reco
mm
ende
d to
rega
rd th
e fo
llow
ing
as v
aria
tions
of c
usto
mis
ing
oper
atio
ns:
- cho
se a
ltern
ativ
e op
erat
ion,
suc
h as
usi
ng m
enu
sele
ctio
n in
stea
d of
com
man
d in
put;
- com
bine
d us
er’s
ope
ratio
n pr
oced
ure,
suc
h as
reco
rdin
g an
d ed
iting
ope
ratio
n pr
oced
ures
; - s
et c
onst
rain
ed te
mpl
ate
oper
atio
n, s
uch
as p
rogr
amm
ing
proc
edur
es o
r mak
ing
a te
mpl
ate
for i
nput
gui
danc
e.
3 Th
is m
etric
is g
ener
ally
use
d as
one
of e
xper
ienc
ed a
nd ju
stifi
ed.
Ope
ratio
n pr
oced
ure
redu
ctio
n
Can
use
r eas
ily
redu
ce o
pera
tion
proc
edur
es fo
r his
/her
co
nven
ienc
e?
Cou
nt u
ser’s
stro
kes
for
spec
ified
ope
ratio
n an
d co
mpa
re th
em b
etw
een
befo
re a
nd a
fter
cust
omis
ing
oper
atio
n.
X =
1 -
A / B
A
= N
umbe
r of r
educ
ed o
pera
tion
proc
edur
es a
fter c
usto
mis
ing
oper
atio
n B
= N
umbe
r of o
pera
tion
proc
edur
es b
efor
e cu
stom
isin
g op
erat
ion
0<=X
< 1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TES
1 It
is re
com
men
ded
to ta
ke s
ampl
es fo
r eac
h di
ffere
nt u
ser t
ask
and
to d
istin
guis
h be
twee
n an
ope
rato
r who
is a
ski
lled
user
or a
beg
inne
r.
2 N
umbe
r of o
pera
tion
proc
edur
es m
ay b
e re
pres
ente
d by
cou
ntin
g op
erat
ion
stro
kes
such
as
clic
k, d
rug,
key
touc
h, s
cree
n to
uch,
etc
. 3
This
incl
udes
key
boar
d sh
ortc
uts.
ISO/IEC TR 9126-2:2003(E)
38 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.3
f) (
cont
inue
d)
Exte
rnal
ope
rabi
lity
met
rics
f) S
uita
ble
for i
ndiv
idua
lisat
ion
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Phys
ical
ac
cess
ibili
ty
Wha
t pro
porti
on o
f fu
nctio
ns c
an b
e ac
cess
ed b
y us
ers
with
phy
sica
l ha
ndic
aps?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. X=
A /
B
A= N
umbe
r of f
unct
ions
suc
cess
fully
ac
cess
ed
B= N
umbe
r of f
unct
ions
0 <=
X <
= 1
The
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Ope
ratio
n
(test
) rep
ort
Use
r m
onito
ring
reco
rd
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
Exam
ples
of p
hysi
cal i
nacc
essi
bilit
y ar
e in
abilit
y to
use
a m
ouse
and
blin
dnes
s.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 39
Tabl
e 8.
3.4
Attr
activ
enes
s m
etric
s
Exte
rnal
attr
activ
enes
s m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Attr
activ
e in
tera
ctio
n
How
attr
activ
e is
the
inte
rface
to th
e us
er?
Que
stio
nnai
re to
use
rs.
Que
stio
nnai
re to
ass
ess
the
attra
ctiv
enes
s of
the
inte
rface
to u
sers
, afte
r exp
erie
nce
of
usag
e
Dep
end
on it
s qu
estio
nnai
re
scor
ing
met
hod.
Abso
lute
Cou
nt
Que
stio
nnai
re
resu
lt 6.
5 Va
lidat
ion
5.3
Qua
lific
atio
n te
stin
g
5.4
Ope
ratio
n
Use
r H
uman
in
terfa
ce
desi
gner
Inte
rfac
e ap
pear
ance
cu
stom
isab
ility
Wha
t pro
porti
on o
f in
terfa
ce e
lem
ents
ca
n be
cus
tom
ised
in
appe
aran
ce to
the
user
’s s
atis
fact
ion?
Con
duct
use
r tes
t and
ob
serv
e us
er b
ehav
iour
. X=
A /
B A=
Num
ber o
f int
erfa
ce e
lem
ents
cu
stom
ised
in a
ppea
ranc
e to
use
r’s
satis
fact
ion
B= N
umbe
r of i
nter
face
ele
men
ts th
at th
e us
er w
ishe
s to
cus
tom
ise
0 <=
X <
= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
rs’
requ
ests
O
pera
tion
(te
st) r
epor
t
6.5
Valid
atio
n 5.
3 Q
ualif
icat
ion
test
ing
5.
4 O
pera
tion
Use
r H
uman
in
terfa
ce
desi
gner
FOO
TNO
TE
This
met
ric is
gen
eral
ly u
sed
as o
ne o
f exp
erie
nced
and
just
ified
.
ISO/IEC TR 9126-2:2003(E)
40 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.5
Usa
bilit
y co
mpl
ianc
e m
etric
s
Exte
rnal
usa
bilit
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Usa
bilit
y co
mpl
ianc
e
How
com
plet
ely
does
th
e so
ftwar
e ad
here
to
the
stan
dard
s,
conv
entio
ns, s
tyle
gu
ides
or r
egul
atio
ns
rela
ting
to u
sabi
lity?
Spec
ify re
quire
d co
mpl
ianc
e ite
ms
base
d on
sta
ndar
ds, c
onve
ntio
ns,
styl
e gu
ides
or r
egul
atio
ns
rela
ting
to u
sabi
lity.
D
esig
n te
st c
ases
in
acco
rdan
ce w
ith
com
plia
nce
item
s.
Con
duct
func
tiona
l tes
ting
for t
hese
test
cas
es.
X =
1 -
A / B
A=
Num
ber o
f usa
bilit
y co
mpl
ianc
e ite
ms
spec
ified
that
hav
e no
t bee
n im
plem
ente
d du
ring
test
ing
B= T
otal
num
ber o
f usa
bilit
y co
mpl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser m
anua
l or
Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
, st
yle
guid
es
or re
gula
tions
Test
sp
ecifi
catio
n an
d re
port
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TE
It m
ay b
e us
eful
to c
olle
ct s
ever
al m
easu
red
valu
es a
long
tim
e, to
ana
lyse
the
trend
of i
ncre
asin
gly
satis
fied
com
plia
nce
item
s an
d to
det
erm
ine
whe
ther
they
are
fully
sat
isfie
d or
not
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 41
8.4 Efficiency metrics
An external efficiency metric should be able to measure such attributes as the time consumption and resource utilisation behaviour of computer system including software during testing or operations.
It is recommended that the maximal and distribution time are investigated for many cases of testing or operations, because the measure is affected strongly and fluctuates depending on the conditions of use, such as load of processing data, frequency of use, number of connecting sites and so on. Therefore, efficiency metrics may include the ratio of measured actual value with error fluctuation to the designed value with allowed error fluctuation range, required by specification.
It is recommended to list and to investigate the role played by factors such as “CPU” and memory used by other software, network traffic, and scheduled background processes. Possible fluctuations and valid ranges for measured values should be established and compared to requirement specifications.
It is recommended that a task be identified and defined to be suitable for software application: for example, a transaction as a task for business application: a switching or data packet sending as a task for communication application; an event control as a task for control application; and an output of data produced by user callable function for common user application.
NOTE 1 Response time: Time needed to get the result from pressing a transmission key. This means that response time includes processing time and transmission time. Response time is applicable only for an interactive system. There is no significant difference when it is a standalone system. However, in the case of Internet system or other real time system, sometimes transmission time is much longer.
NOTE 2 Processing time: The elapsed time in a computer between receiving a message and sending the result. Sometimes it includes operating overhead time, other times it only means time used for an application program.
NOTE 3 Turn around time: Time needed to get the result from a request. In many cases one turn around time includes many responses. For example, in a case of banking cash dispenser, turn around time is a time from pressing initial key until you get money, meanwhile you must select type of transaction and wait for a message, input password and wait for the next message etc.
8.4.1 Time behaviour metrics
An external time behaviour metric should be able to measure such attributes as the time behaviour of computer system including software during testing or operations.
8.4.2 Resource utilization metrics
An external resource utilization metric should be able to measure such attributes as the utilized resources behaviour of computer system including software during testing or operating.
8.4.3 Efficiency compliance metrics
An external efficiency compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, which is the software product failing to adhere to standards, conventions or regulations relating to efficiency.
ISO/IEC TR 9126-2:2003(E)
42 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.1
Tim
e be
havi
our m
etric
s a
) Res
pons
e tim
e
Exte
rnal
tim
e be
havi
our m
etric
s
a) R
espo
nse
time
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Res
pons
e tim
e W
hat i
s th
e tim
e ta
ken
to c
ompl
ete
a sp
ecifi
ed ta
sk?
H
ow lo
ng d
oes
it ta
ke
befo
re th
e sy
stem
re
spon
se to
a
spec
ified
ope
ratio
n?
Star
t a s
peci
fied
task
. M
easu
re th
e tim
e it
take
s fo
r the
sam
ple
to c
ompl
ete
its o
pera
tion.
Ke
ep a
reco
rd o
f eac
h at
tem
pt.
T =
( tim
e of
gai
ning
the
resu
lt)
- (
time
of c
omm
and
entry
fini
shed
)
0 <
T Th
e so
oner
is
the
bette
r.
Rat
io
T= T
ime
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys
./Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
It is
reco
mm
ende
d to
take
acc
ount
of t
ime
band
wid
th a
nd to
use
sta
tistic
al a
naly
sis
with
mea
sure
s fo
r a lo
t of t
asks
(sam
ple
shot
s) a
nd n
ot fo
r onl
y on
e ta
sk.
Res
pons
e tim
e (M
ean
time
to
resp
onse
)
Wha
t is
the
aver
age
wai
t tim
e th
e us
er
expe
rienc
es a
fter
issu
ing
a re
ques
t unt
il th
e re
ques
t is
com
plet
ed w
ithin
a
spec
ified
sys
tem
load
in
term
s of
con
curre
nt
task
s an
d sy
stem
ut
ilisat
ion?
Exec
ute
a nu
mbe
r of
scen
ario
s of
con
curre
nt
task
s.
Mea
sure
the
time
it ta
kes
to c
ompl
ete
the
sele
cted
op
erat
ion(
s).
Keep
a re
cord
of e
ach
atte
mpt
and
com
pute
the
mea
n tim
e fo
r eac
h sc
enar
io.
X =
Tmea
n / T
Xmea
n Tm
ean
= ∑
(Ti)
/ N,
(for
i=1
to N
) TX
mea
n =
requ
ired
mea
n re
spon
se ti
me
Ti=
resp
onse
tim
e fo
r i-th
eva
luat
ion
(sho
t) N
= nu
mbe
r of e
valu
atio
ns (s
ampl
ed s
hots
)
0 <=
X
The
near
er to
1.
0 an
d le
ss
than
1.0
is th
e be
tter.
Abso
lute
Tmea
n=
Tim
e TX
mea
n=
Tim
e Ti
= Ti
me
N=
Cou
nt
X= T
ime/
Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
Req
uire
d m
ean
resp
onse
tim
e ca
n be
der
ived
from
spe
cific
atio
n of
requ
ired
real
-tim
e pr
oces
sing
, use
r exp
ecta
tion
of b
usin
ess
need
s or
obs
erva
tion
of u
ser r
eact
ion.
A u
ser c
ogni
tive
of th
e as
pect
s of
hum
an e
rgon
omic
s m
ight
be
cons
ider
ed.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 43
Tabl
e 8.
4.1
a) (
cont
inue
d)
Exte
rnal
tim
e be
havi
our m
etric
s
a) R
espo
nse
time
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Res
pons
e tim
e (W
orst
cas
e re
spon
se ti
me
ratio
)
Wha
t is
the
abso
lute
lim
it on
tim
e re
quire
d in
fulfi
lling
a fu
nctio
n?
In th
e w
orst
cas
e, c
an
user
stil
l get
resp
onse
w
ithin
the
spec
ified
tim
e lim
it?
In th
e w
orst
cas
e, c
an
user
stil
l get
repl
y fro
m th
e so
ftwar
e w
ithin
a ti
me
shor
t en
ough
to b
e to
lera
ble
for u
ser?
Cal
ibra
te th
e te
st.
Emul
ate
a co
nditio
n wh
ereb
y th
e sy
stem
reac
hes
a m
axim
um lo
ad s
ituat
ion.
R
un a
pplic
atio
n an
d m
onito
r re
sult(
s).
X= T
max
/ R
max
Tm
ax=
MA
X(Ti
) (fo
r i=
1 to
N)
Rm
ax =
requ
ired
max
imum
resp
onse
tim
e M
AX(T
i)= m
axim
um re
spon
se ti
me
amon
g ev
alua
tions
N
= nu
mbe
r of e
valu
atio
ns (s
ampl
ed s
hots
) Ti
= re
spon
se ti
me
for i
-th e
valu
atio
n (s
hot)
CO
MM
ENT(
S) D
istri
butio
n m
ay b
e ca
lcul
ated
as
illus
trate
d be
low
. St
atis
tical
max
imal
ratio
Y=
Tdev
/ R
max
T
dev
= Tm
ean
+ K
( D
EV )
Tde
v is
tim
e de
viat
ed fr
om m
ean
time
to th
e pa
rticu
lar t
ime:
e.g
. 2 o
r 3 ti
mes
of s
tand
ard
devi
atio
n.
K: c
oeffi
cien
t (2
or 3
) D
EV=S
QR
T{ ∑
( (Ti
-Tm
ean)
**2)
/ (N
-1)}
(for
i=1
to N
) Tm
ean
= ∑
(Ti)
/ N,
(for
i=1
to N
) TX
mea
n =
requ
ired
mea
n re
spon
se ti
me
0 <
X Th
e ne
arer
to
1 an
d le
ss
than
1 is
the
bette
r.
Abso
lute
Tmax
= Ti
me
Rm
ax=
Tim
e Ti
= Ti
me
N=
Cou
nt
X= T
ime/
Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
ISO/IEC TR 9126-2:2003(E)
44 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.1
Tim
e be
havi
our m
etric
s
b) T
hrou
ghpu
t
Exte
rnal
tim
e be
havi
our m
etric
s
b) T
hrou
ghpu
t M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Thro
ughp
ut
How
man
y ta
sks
can
be s
ucce
ssfu
lly
perfo
rmed
ove
r a
give
n pe
riod
of ti
me?
Cal
ibra
te e
ach
task
ac
cord
ing
to th
e in
tend
ed
prio
rity
give
n.
Sta
rt se
vera
l job
task
s.
Mea
sure
the
time
it ta
kes
for t
he m
easu
red
task
to
com
plet
e its
ope
ratio
n.
Kee
p a
reco
rd o
f eac
h at
tem
pt.
X =
A /
T A
= nu
mbe
r of c
ompl
eted
task
s T
= ob
serv
atio
n tim
e pe
riod
0 <
X Th
e la
rger
is
the
bette
r.
Rat
io
A= C
ount
T=
Tim
e X=
C
ount
/ Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
Thro
ughp
ut
(Mea
n am
ount
of
thro
ughp
ut)
Wha
t is
the
aver
age
num
ber o
f con
curre
nt
task
s th
e sy
stem
can
ha
ndle
ove
r a s
et u
nit
of ti
me?
Cal
ibra
te e
ach
task
ac
cord
ing
to in
tend
ed
prio
rity.
E
xecu
te a
num
ber o
f co
ncur
rent
task
s.
Mea
sure
the
time
it ta
kes
to c
ompl
ete
the
sele
cted
ta
sk in
the
give
n tra
ffic.
K
eep
a re
cord
of e
ach
atte
mpt
.
X =
Xmea
n / R
mea
n Xm
ean
= ∑
(Xi)/
N
Rm
ean
= re
quire
d m
ean
thro
ughp
ut
Xi =
Ai /
Ti
Ai =
num
ber o
f con
curre
nt ta
sks
obse
rved
ov
er s
et p
erio
d of
tim
e fo
r i-th
eva
luat
ion
Ti =
set
per
iod
of ti
me
for i
-th e
valu
atio
n N
= n
umbe
r of e
valu
atio
ns
0 <
X Th
e la
rger
is
the
bette
r.
Abso
lute
Xmea
n=
Cou
nt
Rm
ean=
C
ount
Ai
= C
ount
Ti
= Ti
me
Xi=
Cou
nt/
Tim
e N
= C
ount
X
= C
ount
/C
ount
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
Thro
ughp
ut
(Wor
st c
ase
thro
ughp
ut
ratio
)
Wha
t is
the
abso
lute
lim
it on
the
syst
em in
te
rms
of th
e nu
mbe
r an
d ha
ndlin
g of
co
ncur
rent
task
s as
th
roug
hput
?
Cal
ibra
te th
e te
st.
Emul
ate
the
cond
ition
w
here
by th
e sy
stem
re
ache
s a
situ
atio
n of
m
axim
um lo
ad. R
un jo
b ta
sks
conc
urre
ntly
and
m
onito
r res
ult(s
).
X =
Xmax
/ R
max
Xm
ax =
MA
X(Xi
) (fo
r i =
1 to
N)
Rm
ax =
requ
ired
max
imum
thro
ughp
ut.
MAX
(Xi)
= m
axim
um n
umbe
r of j
ob ta
sks
amon
g e
valu
atio
ns
Xi =
Ai /
Ti
Ai =
num
ber o
f con
curre
nt ta
sks
obse
rved
ov
er s
et p
erio
d of
tim
e fo
r i-th
eva
luat
ion
Ti =
set
per
iod
of ti
me
for i
-th e
valu
atio
n N
= nu
mbe
r of e
valu
atio
ns
0 <
X
The
larg
er is
th
e be
tter.
Abso
lute
Xmax
= C
ount
R
max
= C
ount
Ai
= C
ount
Ti
= Ti
me
Xi=
Cou
nt/
Tim
e N
= C
ount
Xd
ev=
Cou
nt
X =
Cou
nt/
Cou
nt
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
1 D
istri
butio
n m
ay b
e ca
lcul
ated
as
illust
rate
d be
low
. St
atis
tical
max
imal
ratio
Y=
Xdev
/ X
max
X
dev
= Xm
ean
+ K
( D
EV )
Xde
v is
tim
e de
viat
ed fr
om m
ean
time
to th
e pa
rticu
lar t
ime:
e.g
. 2 o
r 3 ti
mes
of s
tand
ard
devi
atio
n.
K: c
oeffi
cien
t (2
or 3
) D
EV=S
QR
T{ ∑
( (Xi
-Xm
ean)
**2)
/ (N
-1)}
(for i
=1 to
N)
Xmea
n =
∑(X
i)/N
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 45
Tabl
e 8.
4.1
Tim
e be
havi
our m
etric
s
c) T
urna
roun
d tim
e
Exte
rnal
tim
e be
havi
our m
etric
s
c) T
urna
roun
d tim
e M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Turn
arou
nd
time
Wha
t is
the
wai
t tim
e th
e us
er e
xper
ienc
es
afte
r iss
uing
an
inst
ruct
ion
to s
tart
a gr
oup
of re
late
d ta
sks
and
thei
r com
plet
ion?
Cal
ibra
te th
e te
st
acco
rdin
gly.
St
art t
he jo
b ta
sk. M
easu
re
the
time
it ta
kes
for t
he jo
b ta
sk to
com
plet
e its
op
erat
ion.
Ke
ep a
reco
rd o
f eac
h at
tem
pt.
T =
Tim
e be
twee
n us
er’s
fini
shin
g ge
tting
ou
tput
resu
lts a
nd u
ser’s
fini
shin
g re
ques
t
0 <
T Th
e sh
orte
r th
e be
tter.
Rat
io
T= T
ime
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
It is
reco
mm
ende
d to
take
acc
ount
of t
ime
band
wid
th a
nd to
use
sta
tistic
al a
naly
sis
with
mea
sure
s fo
r man
y ta
sks
(sam
ple
shot
s), n
ot o
nly
one
task
(sho
t).
Turn
arou
nd
time
(Mea
n tim
e fo
r tur
naro
und)
Wha
t is
the
aver
age
wai
t tim
e th
e us
er
expe
rienc
es a
fter
issu
ing
an in
stru
ctio
n to
sta
rt a
grou
p of
re
late
d ta
sks
and
thei
r com
plet
ion
with
in a
spe
cifie
d sy
stem
load
in te
rms
of c
oncu
rrent
task
s an
d sy
stem
ut
ilisat
ion?
Cal
ibra
te th
e te
st.
Em
ulat
e a
cond
ition
w
here
a lo
ad is
pla
ced
on
the
syst
em b
y ex
ecut
ing
a nu
mbe
r of c
oncu
rrent
ta
sks
(sam
pled
sho
ts).
M
easu
re th
e tim
e it
take
s to
com
plet
e th
e se
lect
ed
job
task
in
the
give
n tra
ffic.
K
eep
a re
cord
of e
ach
atte
mpt
.
X =
Tmea
n/TX
mea
n Tm
ean
= ∑
(Ti)/
N, (
for
i=1
to N
) TX
mea
n =
requ
ired
mea
n tu
rnar
ound
tim
e Ti
= tu
rnar
ound
tim
e fo
r i-th
eva
luat
ion
(sho
t) N
= n
umbe
r of e
valu
atio
ns (s
ampl
ed s
hots
)
0 <
X Th
e sh
orte
r is
the
bette
r.
Abso
lute
Tmea
n=
Tim
e TX
mea
n=
Tim
e Ti
= Ti
me
N=
Cou
nt
X= T
ime/
Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
ISO/IEC TR 9126-2:2003(E)
46 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.1
c) (
cont
inue
d)
Exte
rnal
tim
e be
havi
our m
etric
s
c) T
urna
roun
d tim
e M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Turn
arou
nd
time
(Wor
st
case
tu
rnar
ound
tim
e ra
tio)
Wha
t is
the
abso
lute
lim
it on
tim
e re
quire
d in
fulfi
lling
a jo
b ta
sk?
In th
e w
orst
cas
e,
how
long
doe
s it
take
fo
r sof
twar
e sy
stem
to
per
form
spe
cifie
d ta
sks?
Cal
ibra
te th
e te
st.
Emul
ate
a co
nditi
on w
here
by
the
syst
em re
ache
s m
axim
um lo
ad in
term
s of
ta
sks
perfo
rmed
. Run
the
sele
cted
job
task
and
m
onito
r res
ult(s
).
X= T
max
/ R
max
Tm
ax=
MA
X(Ti
) (fo
r i=
1 to
N)
Rm
ax =
requ
ired
max
imum
turn
arou
nd ti
me
MAX
(Ti)=
max
imum
turn
arou
nd ti
me
amon
g ev
alua
tions
N
= nu
mbe
r of e
valu
atio
ns (s
ampl
ed s
hots
) Ti
= tu
rnar
ound
tim
e fo
r i-th
eva
luat
ion
(sho
t)
0 <
X
The
near
er to
1.
0 an
d le
ss
than
1.0
is th
e be
tter.
Abso
lute
X= T
ime/
Ti
me
Tmax
=
Tim
e
Rm
ax=
Tim
e Ti
= Ti
me
N=
Cou
nt
Tdev
=
Tim
e
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
Dis
tribu
tion
may
be
calc
ulat
ed a
s illu
stra
ted
belo
w.
Stat
istic
al m
axim
al ra
tio Y
= Td
ev /
Rm
ax
Tde
v =
Tmea
n +
K (
DEV
) T
dev
is ti
me
devi
ated
from
mea
n tim
e to
the
parti
cula
r tim
e: e
.g. 2
or 3
tim
es o
f sta
ndar
d de
viat
ion.
K:
coe
ffici
ent (
2 or
3)
DEV
=SQ
RT{
∑( (
Ti-T
mea
n) **
2) /
(N-1
)} (fo
r i=1
to N
) Tm
ean
= ∑
(Ti)
/ N,
(for
i=1
to N
) TX
mea
n =
requ
ired
mea
n tu
rnar
ound
tim
e
Wai
ting
time
Wha
t pro
porti
on o
f th
e tim
e do
use
rs
spen
d w
aitin
g fo
r the
sy
stem
to re
spon
d?
Exec
ute
a nu
mbe
r of
scen
ario
s of
con
curre
nt
task
s.
Mea
sure
the
time
it ta
kes
to c
ompl
ete
the
sele
cted
op
erat
ion(
s).
Keep
a re
cord
of e
ach
atte
mpt
and
com
pute
the
mea
n tim
e fo
r eac
h sc
enar
io.
X =
Ta
/ Tb
Ta =
tota
l tim
e sp
ent w
aitin
g
Tb =
task
tim
e
0<=
X Th
e sm
alle
r th
e be
tter.
Abso
lute
Ta=
Tim
e Tb
= Ti
me
X= T
ime/
Tim
e
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Sys.
/Sw
. In
tegr
atio
n5.
3 Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 5.
5 M
aint
e-na
nce
Use
r D
evel
oper
M
aint
aine
r SQ
A
FOO
TNO
TE
If th
e ta
sks
can
be p
artia
lly c
ompl
eted
, the
Tas
k ef
ficie
ncy
met
ric s
houl
d be
use
d w
hen
mak
ing
com
paris
ons.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 47
Tabl
e 8.
4.2
Res
ourc
e ut
iliza
tion
met
rics
a)
I/O
dev
ices
reso
urce
util
izat
ion
Exte
rnal
reso
urce
util
isat
ion
met
rics
a)
I/O
dev
ices
reso
urce
util
isat
ion
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
I/O d
evic
es
utili
satio
n
Is th
e I/O
dev
ice
utilis
atio
n to
o hi
gh,
caus
ing
inef
ficie
ncie
s?
Exec
ute
conc
urre
ntly
a
larg
e nu
mbe
r of t
asks
, re
cord
I/O
dev
ice
utilis
atio
n, a
nd c
ompa
re
with
the
desi
gn o
bjec
tives
.
X =
A / B
A
= tim
e of
I/O
dev
ices
occ
upie
d B
= sp
ecifi
ed ti
me
whi
ch is
des
igne
d to
oc
cupy
I/O
dev
ices
0 <=
X <
= 1
The
less
than
an
d ne
arer
to
the
1.0
is th
e be
tter.
Abso
lute
A= T
ime
B= T
ime
X= T
ime/
Tim
e
Test
ing
repo
rt O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
SQA
I/O l
oadi
ng
limits
W
hat i
s th
e ab
solu
te
limit
on I/
O u
tilis
atio
n in
fulfi
lling
a fu
nctio
n? C
alib
rate
the
test
co
nditi
on. E
mul
ate
a co
nditi
on w
here
by th
e sy
stem
reac
hes
a si
tuat
ion
of m
axim
um lo
ad. R
un
appl
icat
ion
and
mon
itor
resu
lt(s)
.
X =
Amax
/ R
max
Am
ax =
MAX
(Ai),
(fo
r i =
1 to
N)
Rm
ax =
requ
ired
max
imum
I/O
mes
sage
s M
AX(A
i) =
Max
imum
num
ber o
f I/O
m
essa
ges
from
1st
to i-
th e
valu
atio
n N
= nu
mbe
r of e
valu
atio
ns.
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amax
=
Cou
nt
Rm
ax =
C
ount
Ai
= C
ount
N=
Cou
nt
X =
Cou
nt/
Cou
nt
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
I/O re
late
d er
rors
H
ow o
ften
does
the
user
enc
ount
er
prob
lem
s in
I/O
de
vice
rela
ted
oper
atio
ns?
Cal
ibra
te th
e te
st
cond
ition
s. E
mul
ate
a co
nditi
on w
here
by th
e sy
stem
reac
hes
a si
tuat
ion
of m
axim
um I/
O lo
ad. R
un
the
appl
icat
ion
and
reco
rd
num
ber o
f erro
rs d
ue to
I/O
failu
re a
nd w
arni
ngs.
X =
A / T
A
= nu
mbe
r of w
arni
ng m
essa
ges
or s
yste
m
failu
res
T =
Use
r ope
ratin
g tim
e du
ring
user
ob
serv
atio
n
0 <=
X
The
smal
ler i
s th
e be
tter.
Rat
io
A =
Cou
nt
T =
Tim
e X
= C
ount
/ Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r M
aint
aine
r SQ
A
Mea
n I/
O
fulfi
lmen
t rat
io
Wha
t is
the
aver
age
num
ber o
f I/O
rela
ted
erro
r mes
sage
s an
d fa
ilure
s ov
er a
sp
ecifi
ed le
ngth
of
time
and
spec
ified
ut
ilisat
ion?
Cal
ibra
te th
e te
st
cond
ition
. Em
ulat
e a
cond
ition
whe
reby
the
syst
em re
ache
s a
situ
atio
n of
max
imum
load
. Run
the
appl
icat
ion
and
reco
rd
num
ber o
f erro
rs d
ue to
I/O
failu
re a
nd w
arni
ngs.
X =
Amea
n / R
mea
n Am
ean
= ∑
(Ai)/
N
Rm
ean
= re
quire
d m
ean
num
ber o
f I/O
m
essa
ges
Ai =
num
ber o
f I/O
erro
r mes
sage
s fo
r i-th
ev
alua
tion
N =
num
ber o
f eva
luat
ions
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amea
n =
Cou
nt
Rm
ean
= C
ount
Ai
= C
ount
N=
Cou
nt
X =
Cou
nt/
Cou
nt
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
Use
r wai
ting
time
of I/
O
devi
ces
utili
satio
n
Wha
t is
the
impa
ct o
f I/O
dev
ice
utilis
atio
n on
the
user
wai
t tim
es?
Exec
ute
conc
urre
ntly
a
larg
e am
ount
of t
asks
and
m
easu
re th
e us
er w
ait
times
as
a re
sult
of I/
O
devi
ce o
pera
tion.
T =
Tim
e sp
ent t
o w
ait f
or fi
nish
of I
/O
devi
ces
oper
atio
n
0 <
T Th
e sh
orte
r is
the
bette
r.
Rat
io
T= T
ime
Te
stin
g re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A FO
OTN
OTE
It
is re
com
men
ded
that
the
max
imal
and
dis
tribu
ted
time
are
to b
e in
vest
igat
ed fo
r sev
eral
cas
es o
f tes
ting
or o
pera
ting,
bec
ause
the
mea
sure
s ar
e te
nd to
be
fluct
uate
d by
con
ditio
n of
use
.
ISO/IEC TR 9126-2:2003(E)
48 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.2
Res
ourc
e ut
iliza
tion
met
rics
b)
Mem
ory
reso
urce
util
izat
ion
Exte
rnal
reso
urce
util
isat
ion
met
rics
b)
Mem
ory
reso
urce
util
isat
ion
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Max
imum
m
emor
y ut
ilisa
tion
Wha
t is
the
abso
lute
lim
it on
mem
ory
requ
ired
in fu
lfillin
g a
func
tion?
Cal
ibra
te th
e te
st
cond
ition
. Em
ulat
e a
cond
ition
whe
reby
the
syst
em re
ache
s a
situ
atio
n of
max
imum
load
. Run
ap
plic
atio
n an
d m
onito
r re
sult(
s).
X =
Amax
/ R
max
Am
ax =
MAX
(Ai),
(fo
r i =
1 to
N)
Rm
ax =
requ
ired
max
imum
mem
ory
rela
ted
erro
r mes
sage
s M
AX(A
i) =
Max
imum
num
ber o
f mem
ory
rela
ted
erro
r mes
sage
s fro
m 1
st to
i-th
ev
alua
tion
N=
num
ber o
f eva
luat
ions
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amax
= C
ount
R
max
= C
ount
Ai
= C
ount
N
= C
ount
X
= C
ount
/ C
ount
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
Mea
n oc
curr
ence
of
mem
ory
err
or
Wha
t is
the
aver
age
num
ber o
f mem
ory
rela
ted
erro
r m
essa
ges
and
failu
res
over
a
spec
ified
leng
th o
f tim
e an
d a
spec
ified
lo
ad o
n th
e sy
stem
?
Cal
ibra
te th
e te
st
cond
ition
. Em
ulat
e a
cond
ition
whe
reby
the
syst
em re
ache
s a
situ
atio
n of
max
imum
load
. Run
the
appl
icat
ion
and
reco
rd
num
ber o
f er
rors
due
to
mem
ory
failu
re a
nd
war
ning
s.
X =
Amea
n / R
mea
n Am
ean
= ∑
(Ai)/
N
Rm
ean
= re
quire
d m
ean
num
ber o
f mem
ory
rela
ted
erro
r mes
sage
s Ai
= n
umbe
r of m
emor
y re
late
d er
ror
mes
sage
s fo
r i-th
eva
luat
ion
N =
num
ber o
f eva
luat
ions
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amea
n=
Cou
nt
Rm
ean=
C
ount
Ai
= C
ount
N
= C
ount
X
= C
ount
/ C
ount
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
Rat
io o
f m
emor
y er
ror/t
ime
How
man
y m
emor
y er
rors
wer
e ex
perie
nced
ove
r a
set p
erio
d of
tim
e an
d sp
ecifi
ed re
sour
ce
utilis
atio
n?
Cal
ibra
te th
e te
st
cond
ition
s.
Em
ulat
e a
cond
ition
w
here
by th
e sy
stem
re
ache
s a
situ
atio
n of
m
axim
um lo
ad.
Run
the
appl
icat
ion
and
reco
rd n
umbe
r of
erro
rs
due
to m
emor
y fa
ilure
and
w
arni
ngs.
X =
A / T
A =
num
ber o
f war
ning
mes
sage
s or
sys
tem
fa
ilure
s
T =
Use
r ope
ratin
g tim
e du
ring
user
ob
serv
atio
n
0 <=
X
The
smal
ler i
s th
e be
tter.
Rat
io
A =
Cou
nt
T =
Tim
e
X =
Cou
nt/
Tim
e
Test
ing
repo
rt
Ope
ratio
n re
port
show
ing
elap
se ti
me
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r M
aint
aine
r SQ
A
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 49
Tabl
e 8.
4.2
Res
ourc
e ut
iliza
tion
met
rics
c)
Tra
nsm
issi
on re
sour
ce u
tiliz
atio
n
Exte
rnal
reso
urce
util
isat
ion
met
rics
c)
Tra
nsm
issi
on re
sour
ce u
tilis
atio
n M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Max
imum
tr
ansm
issi
on
utili
satio
n
Wha
t is
the
abso
lute
lim
it of
tran
smis
sion
s re
quire
d to
fulfi
l a
func
tion?
Eval
uate
wha
t is
requ
ired
for t
he s
yste
m to
reac
h a
situ
atio
n of
max
imum
load
. Em
ulat
e th
is c
ondi
tion.
R
un a
pplic
atio
n an
d m
onito
r res
ult(s
).
X =
Amax
/ R
max
Am
ax =
MAX
(Ai),
(fo
r i =
1 to
N)
Rm
ax =
requ
ired
max
imum
num
ber o
f tra
nsm
issi
on re
late
d er
ror m
essa
ges
and
failu
res
MAX
(Ai)
= M
axim
um n
umbe
r of t
rans
mis
sion
re
late
d er
ror m
essa
ges
and
failu
res
from
1st
to
i-th
eva
luat
ion
N=
num
ber o
f eva
luat
ions
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amax
=
Cou
nt
Rm
ax =
C
ount
Ai
= C
ount
N=
Cou
nt
X =
Cou
nt/
Cou
nt
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
Med
ia d
evic
e ut
ilisa
tion
bala
ncin
g
Wha
t is
the
degr
ee o
f sy
nchr
onis
atio
n be
twee
n di
ffere
nt
med
ia o
ver a
set
pe
riod
of ti
me?
Cal
ibra
te th
e te
st
cond
ition
s. E
mul
ate
a co
nditi
on w
here
by th
e sy
stem
reac
hes
a si
tuat
ion
of m
axim
um tr
ansm
issi
on
load
. Run
the
appl
icat
ion
and
reco
rd th
e de
lay
in th
e pr
oces
sing
of d
iffer
ent
med
ia ty
pes.
X =
Sync
Tim
e/T
Sync
Tim
e =
Tim
e de
vote
d to
a c
ontin
uous
re
sour
ce
T =
requ
ired
time
perio
d du
ring
whi
ch
diss
imila
r med
ia a
re e
xpec
ted
to fi
nish
thei
r ta
sks
with
syn
chro
nisa
tion
The
smal
ler i
s th
e be
tter.
Rat
io
Sync
Tim
e =
Tim
e T
= Ti
me
X =
Tim
e/Ti
me
Test
ing
repo
rtO
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r M
aint
aine
r SQ
A
Mea
n
occu
rren
ce o
f tr
ansm
issi
on
erro
r
Wha
t is
the
aver
age
num
ber o
f tra
nsm
issi
on-re
late
d er
ror m
essa
ges
and
failu
res
over
a
spec
ified
leng
th o
f tim
e an
d sp
ecifi
ed
utilis
atio
n?
Cal
ibra
te th
e te
st
cond
ition
. Em
ulat
e a
cond
ition
whe
reby
the
syst
em re
ache
s a
situ
atio
n of
max
imum
load
. Run
the
appl
icat
ion
and
reco
rd
num
ber o
f erro
rs d
ue to
tra
nsm
issi
on fa
ilure
and
w
arni
ngs.
X =
Amea
n / R
mea
n Am
ean
= ∑
(Ai)/
N
Rm
ean
= re
quire
d m
ean
num
ber o
f tra
nsm
issi
on re
late
d er
ror m
essa
ges
and
failu
res
Ai =
Num
ber o
f tra
nsm
issi
on re
late
d er
ror
mes
sage
s an
d fa
ilure
s fo
r i-th
eva
luat
ion
N =
num
ber o
f eva
luat
ions
0<=
X
The
smal
ler i
s th
e be
tter.
Abso
lute
Amea
n=
Cou
nt
Rm
ean=
C
ount
Ai
= C
ount
N
= C
ount
X
= C
ount
/ C
ount
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r SQ
A
Mea
n of
tr
ansm
issi
on
erro
r per
tim
e
How
man
y tra
nsm
issi
on-re
late
d er
ror m
essa
ges
wer
e ex
perie
nced
ove
r a
set p
erio
d of
tim
e an
d sp
ecifi
ed re
sour
ce
utilis
atio
n?
Cal
ibra
te th
e te
st
cond
ition
s. E
mul
ate
a co
nditi
on w
here
by th
e sy
stem
reac
hes
a si
tuat
ion
of m
axim
um tr
ansm
issi
on
load
. Run
the
appl
icat
ion
and
reco
rd n
umbe
r of
erro
rs d
ue to
tran
smis
sion
fa
ilure
and
war
ning
s.
X =
A / T
A
= nu
mbe
r of w
arni
ng m
essa
ges
or s
yste
m
failu
res
T =
Use
r ope
ratin
g tim
e du
ring
user
ob
serv
atio
n
0 <=
X
The
smal
ler i
s th
e be
tter.
Rat
io
A =
Cou
nt
T =
Tim
e X
= C
ount
/ Ti
me
Test
ing
repo
rt O
pera
tion
repo
rt sh
owin
g el
apse
tim
e
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r M
aint
aine
r SQ
A
ISO/IEC TR 9126-2:2003(E)
50 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.2
c) (
cont
inue
d)
Exte
rnal
reso
urce
util
isat
ion
met
rics
c)
Tra
nsm
issi
on re
sour
ce u
tilis
atio
n M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Tran
smis
sion
ca
paci
ty
utili
satio
n
Is s
oftw
are
syst
em
capa
ble
of p
erfo
rmin
g ta
sks
with
in e
xpec
ted
trans
mis
sion
ca
paci
ty?
Exec
ute
conc
urre
ntly
sp
ecifi
ed ta
sks
with
m
ultip
le u
sers
, obs
erve
tra
nsm
issi
on c
apac
ity a
nd
com
pare
spe
cifie
d on
e.
X =
A / B
A
= tra
nsm
issi
on c
apac
ity
B =
spec
ified
tran
smis
sion
cap
acity
whi
ch is
de
sign
ed t
o be
use
d by
the
sof
twar
e du
ring
exec
utio
n
0 <=
X <
= 1
The
less
than
an
d ne
arer
to
the
1.0
is th
e be
tter.
Abso
lute
A= S
ize
B= S
ize
X= S
ize
/Si
ze
Test
ing
repo
rt O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
SQA
FOO
TNO
TE
It is
reco
mm
ende
d to
mea
sure
dyn
amic
ally
pea
ked
valu
e w
ith m
ultip
le u
sers
.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 51
Tabl
e 8.
4.3
Effic
ienc
y co
mpl
ianc
e m
etric
s
Effic
ienc
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Effic
ienc
y C
ompl
ianc
e H
ow c
ompl
iant
is th
e ef
ficie
ncy
of th
e pr
oduc
t to
appl
icab
le
regu
latio
ns,
stan
dard
s an
d co
nven
tions
?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber
of it
ems
requ
iring
co
mpl
ianc
e in
the
spec
ifica
tion.
X =
1 -
A / B
(X
: Rat
io o
f sat
isfie
d co
mpl
ianc
e ite
ms
rela
ting
to e
ffici
ency
) A=
Num
ber o
f effi
cien
cy c
ompl
ianc
e ite
ms
spec
ified
that
hav
e no
t bee
n im
plem
ente
d du
ring
test
ing
B= T
otal
num
ber o
f effi
cien
cy c
ompl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser m
anua
l or
Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns Te
st
spec
ifica
tion
and
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TE
It m
ay b
e us
eful
to c
olle
ct s
ever
al m
easu
red
valu
es a
long
tim
e, to
ana
lyse
the
trend
of i
ncre
asin
g sa
tisfie
d co
mpl
ianc
e ite
ms
and
to d
eter
min
e w
heth
er th
ey a
re fu
lly s
atis
fied
or n
ot.
ISO/IEC TR 9126-2:2003(E)
52 © ISO/IEC 2003 – All rights reserved
8.5 Maintainability metrics
An external maintainability metric should be able to measure such attributes as the behaviour of the maintainer, user, or system including the software, when the software is maintained or modified during testing or maintenance.
8.5.1 Analysability metrics
An external analysability metric should be able to measure such attributes as the maintainer’s or user’s effort or spent of resources when trying to diagnose deficiencies or causes of failures, or for identifying parts to be modified.
8.5.2 Changeability metrics
An external changeability metric should be able to measure such attributes as the maintainer’s or user’s effort by measuring the behaviour of the maintainer, user or system including the software when trying to implement a specified modification.
8.5.3 Stability metrics
An external stability metric should be able to measure attributes related to unexpected behaviour of the system including the software when the software is tested or operated after modification.
8.5.4 Testability metrics
An external testability metric should be able to measure such attributes as the maintainer’s or user’s effort by measuring the behaviour of the maintainer, user or system including software when trying to test the modified or non-modified software.
8.5.5 Maintainability compliance metrics
An external maintainability compliance metric should be able to measure an attribute such as the number of functions or occurrences of compliance problems, where the software product fails to adhere to required standards, conventions or regulations relating to maintainability.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 53
Tabl
e 8.
5.1
Anal
ysab
ility
met
rics
Exte
rnal
ana
lysa
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Audi
t tra
il ca
pabi
lity
Can
use
r ide
ntify
sp
ecifi
c op
erat
ion
whi
ch c
ause
d fa
ilure
? C
an m
aint
aine
r eas
ily
find
spe
cific
op
erat
ion
whi
ch
caus
ed fa
ilure
?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
tryi
ng
to re
solv
e fa
ilure
s.
X= A
/ B
A= N
umbe
r of d
ata
actu
ally
reco
rded
dur
ing
oper
atio
n B=
Num
ber o
f dat
a pl
anne
d to
be
reco
rded
en
ough
to m
onito
r sta
tus
of s
oftw
are
durin
g op
erat
ion
0<=X
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
Dia
gnos
tic
func
tion
supp
ort
How
cap
able
are
the
diag
nost
ic fu
nctio
ns
in s
uppo
rting
cau
sal
anal
ysis
? C
an u
ser i
dent
ify th
e sp
ecifi
c op
erat
ion
whi
ch c
ause
d fa
ilure
? (U
ser m
ay b
e ab
le to
av
oid
fallin
g in
to th
e sa
me
failu
re
occu
rrenc
e ag
ain
with
al
tern
ativ
e op
erat
ion.
) C
an m
aint
aine
r eas
ily
find
caus
e of
failu
re?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
tryi
ng
to re
solv
e fa
ilure
s us
ing
diag
nost
ics
func
tions
.
X= A
/ B
A=
Num
ber o
f fai
lure
s w
hich
mai
ntai
ner c
an
diag
nose
(usi
ng th
e di
agno
stic
s fu
nctio
n) to
un
ders
tand
the
caus
e-ef
fect
rela
tions
hip
B= T
otal
num
ber o
f reg
iste
red
failu
res
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
Failu
re a
naly
sis
capa
bilit
y C
an u
ser i
dent
ify
spec
ific
oper
atio
n w
hich
cau
sed
failu
re?
Can
mai
ntai
ner e
asily
fin
d ca
use
of fa
ilure
?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
tryi
ng
to re
solv
e fa
ilure
s.
X=1-
A /
B
A= N
umbe
r of f
ailu
res
of w
hich
cau
ses
are
still
not f
ound
B=
Tot
al n
umbe
r of r
egis
tere
d fa
ilure
s
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r O
pera
tor
ISO/IEC TR 9126-2:2003(E)
54 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
5.1
(con
tinue
d)
Exte
rnal
ana
lysa
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Failu
re a
naly
sis
effic
ienc
y
Can
use
r effi
cien
tly
anal
yse
caus
e of
fa
ilure
?
(Use
r som
etim
es
perfo
rms
mai
nten
ance
by
setti
ng p
aram
eter
s.)
Can
mai
ntai
ner
easi
ly fi
nd c
ause
of
failu
re?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
tryi
ng
to re
solv
e fa
ilure
s.
X= S
um(T
) / N
0<
=X
The
shor
ter i
s th
e be
tter.
Rat
io
T= T
ime
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 It
is re
com
men
ded
to m
easu
re m
axim
al ti
me
of th
e w
orst
cas
e an
d tim
e du
ratio
n (b
andw
idth
) to
repr
esen
t dev
iatio
n.
2 It
is re
com
men
ded
to e
xclu
de n
umbe
r of f
ailu
res
of w
hich
cau
ses
are
not y
et fo
und
whe
n m
easu
rem
ent i
s do
ne. H
owev
er, t
he ra
tio o
f suc
h ob
scur
e fa
ilure
s sh
ould
be
also
mea
sure
d an
d pr
esen
ted
toge
ther
. 3
From
the
indi
vidu
al u
ser’s
poi
nt o
f vie
w, t
ime
is o
f con
cern
, whi
le e
ffort
may
als
o be
of c
once
rn fr
om th
e m
aint
aine
r’s p
oint
of v
iew
. The
refo
re, p
erso
n-ho
urs
may
be
used
inst
ead
of ti
me.
Stat
us
mon
itorin
g ca
pabi
lity
Can
use
r ide
ntify
sp
ecifi
c op
erat
ion
w
hich
cau
sed
failu
re
by g
ettin
g m
onito
red
data
dur
ing
oper
atio
n?
Can
mai
ntai
ner e
asily
fin
d ca
use
of fa
ilure
by
get
ting
mon
itore
d da
ta d
urin
g op
erat
ion?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
tryi
ng
to g
et m
onito
red
data
re
cord
ing
stat
us o
f so
ftwar
e du
ring
oper
atio
n.
X= 1
- A
/ B
A= N
umbe
r of c
ases
whi
ch m
aint
aine
r (or
us
er) f
aile
d to
get
mon
itor d
ata
B=
Num
ber o
f cas
es w
hich
mai
ntai
ner
(or
user
) atte
mpt
ed to
get
mon
itor d
ata
reco
rdin
g st
atus
of s
oftw
are
durin
g op
erat
ion
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r D
evel
oper
M
aint
aine
r O
pera
tor
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 55
Tabl
e 8.
5.2
Cha
ngea
bilit
y m
etric
s
Exte
rnal
cha
ngea
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Cha
nge
cycl
e ef
ficie
ncy
Can
the
user
's
prob
lem
be
solv
ed to
hi
s sa
tisfa
ctio
n w
ithin
an
acc
epta
ble
time
scal
e?
Mon
itor i
nter
actio
n be
twee
n us
er a
nd s
uppl
ier.
Rec
ord
the
time
take
n fro
m th
e in
itial
use
r's
requ
est t
o th
e re
solu
tion
of
prob
lem
.
Aver
age
Tim
e : T
av =
Sum
(Tu)
/ N
Tu
= Tr
c - T
sn
Tsn=
Tim
e at
whi
ch u
ser f
inis
hed
to s
end
requ
est f
or m
aint
enan
ce to
sup
plie
r with
pr
oble
m re
port
Trc=
Tim
e at
whi
ch u
ser r
ecei
ved
the
revi
sed
vers
ion
rele
ase
(or s
tatu
s re
port)
N
= N
umbe
r of r
evis
ed v
ersi
ons
0<Ta
v Th
e sh
orte
r is
the
bette
r.,
exce
pt o
f th
e nu
mbe
r of
revi
sed
vers
ions
was
la
rge.
Rat
io
Tu=
Tim
e Tr
c,
Tsn
=
Tim
e N
= C
ount
Ta
v= T
ime
Prob
lem
re
solu
tion
re
port
Mai
nten
ance
re
port
O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Use
r M
aint
aine
r O
pera
tor
Cha
nge
impl
emen
tatio
n el
apse
tim
e
Can
the
mai
ntai
ner
easi
ly c
hang
e th
e so
ftwar
e to
reso
lve
the
failu
re p
robl
em?
Obs
erve
the
beha
viou
r of
the
user
and
mai
ntai
ner
whi
le tr
ying
to c
hang
e th
e so
ftwar
e.
Oth
erw
ise,
inve
stig
ate
prob
lem
reso
lutio
n re
port
or m
aint
enan
ce re
port.
Aver
age
Tim
e : T
av =
Sum
(Tm
) / N
Tm
=Tou
t - T
in
Tout
= Ti
me
at w
hich
the
caus
es o
f fai
lure
ar
e re
mov
ed w
ith c
hang
ing
the
softw
are
(or
stat
us is
repo
rted
back
to u
ser)
Tin=
Tim
e at
whi
ch th
e ca
uses
of f
ailu
res
are
foun
d ou
t N
= N
umbe
r of r
egis
tere
d an
d re
mov
ed
failu
res
0<Ta
v Th
e sh
orte
r is
the
bette
r, ex
cept
of t
he
num
ber o
f fa
ilure
s w
as
larg
e.
Rat
io
Tm=
Tim
e Ti
n,
Tout
=
Tim
e Ta
v= T
ime
Prob
lem
re
solu
tion
re
port
Mai
nten
ance
re
port
O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 It
is re
com
men
ded
to m
easu
re m
axim
al ti
me
of th
e w
orst
cas
e an
d tim
e ba
ndw
idth
to re
pres
ent d
evia
tion.
2
It is
reco
mm
ende
d to
exc
lude
the
failu
res
for w
hich
cau
ses
have
not
yet
bee
n fo
und
whe
n th
e m
easu
rem
ent i
s do
ne. H
owev
er, t
he ra
tio o
f suc
h ob
scur
e fa
ilure
s sh
ould
be
also
mea
sure
d an
d pr
esen
ted
toge
ther
. 3
From
the
indi
vidu
al u
ser’s
poi
nt o
f vie
w, t
ime
is o
f con
cern
, whi
le e
ffort
may
als
o be
of c
once
rn fr
om th
e m
aint
aine
r’s p
oint
of v
iew
. The
refo
re, p
erso
n-ho
urs
may
be
used
inst
ead
of ti
me.
ISO/IEC TR 9126-2:2003(E)
56 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
5.2
(con
tinue
d)
Exte
rnal
cha
ngea
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Mod
ifica
tion
com
plex
ity
Can
the
mai
ntai
ner
easi
ly c
hang
e th
e so
ftwar
e to
reso
lve
prob
lem
?
Obs
erve
beh
avio
ur o
f m
aint
aine
r who
is tr
ying
to
chan
ge th
e so
ftwar
e.
Oth
erw
ise,
inve
stig
ate
prob
lem
reso
lutio
n re
port
or m
aint
enan
ce re
port
and
prod
uct d
escr
iptio
n.
T =
Sum
(A /
B) /
N
A= W
ork
time
spen
t to
chan
ge
B= S
ize
of s
oftw
are
chan
ge
N=
Num
ber o
f cha
nges
0<T
The
shor
ter i
s th
e be
tter o
r th
e re
quire
d nu
mbe
r of
chan
ges
wer
e ex
cess
ive.
Rat
io
A= T
ime
B= S
ize
N=
Cou
nt
T= T
ime
Prob
lem
re
solu
tion
re
port
Mai
nten
ance
re
port
O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
A si
ze o
f sof
twar
e ch
ange
may
be
chan
ged
exec
utab
le s
tate
men
ts o
f pro
gram
cod
e, n
umbe
r of c
hang
ed it
ems
of re
quire
men
ts s
peci
ficat
ion,
or c
hang
ed p
ages
of d
ocum
ent e
tc.
Para
met
eris
ed
mod
ifiab
ility
C
an th
e us
er o
r the
m
aint
aine
r eas
ily
chan
ge p
aram
eter
to
chan
ge s
oftw
are
and
reso
lve
prob
lem
s?
Obs
erve
beh
avio
ur o
f the
us
er o
r the
mai
ntai
ner
whi
le tr
ying
to c
hang
e th
e so
ftwar
e.
Oth
erw
ise,
inve
stig
ate
prob
lem
reso
lutio
n re
port
or m
aint
enan
ce re
port.
X=1-
A /
B
A= N
umbe
r of c
ases
whi
ch m
aint
aine
r fai
ls
to c
hang
e so
ftwar
e by
usi
ng p
aram
eter
B=
Num
ber o
f cas
es w
hich
mai
ntai
ner
atte
mpt
s to
cha
nge
softw
are
by u
sing
pa
ram
eter
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
M
aint
enan
ce
repo
rt
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r U
ser
Softw
are
chan
ge c
ontr
ol
capa
bilit
y
Can
the
user
eas
ily
iden
tify
revi
sed
vers
ions
?
Can
the
mai
ntai
ner
easi
ly c
hang
e th
e so
ftwar
e to
reso
lve
prob
lem
s?
Obs
erve
the
beha
viou
r of
user
or m
aint
aine
r whi
le
tryin
g to
cha
nge
the
softw
are.
O
ther
wis
e, in
vest
igat
e pr
oble
m re
solu
tion
repo
rt or
mai
nten
ance
repo
rt.
X= A
/ B
A= N
umbe
r of c
hang
e lo
g da
ta a
ctua
lly
reco
rded
B=
Num
ber o
f cha
nge
log
data
pla
nned
to
be re
cord
ed e
noug
h to
trac
e so
ftwar
e ch
ange
s
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter o
r the
cl
oser
to0
the
few
er
chan
ges
have
ta
ken
plac
e.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Use
r man
ual
or
spec
ifica
tion
Prob
lem
re
solu
tion
re
port
Mai
nten
ance
re
port
O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 57
Tabl
e 8.
5.3
Stab
ility
met
rics
Exte
rnal
sta
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Cha
nge
su
cces
s ra
tio
Can
use
r ope
rate
so
ftwar
e sy
stem
w
ithou
t fai
lure
s af
ter
mai
nten
ance
? C
an m
aint
aine
r eas
ily
miti
gate
failu
res
caus
ed b
y m
aint
enan
ce s
ide
effe
cts?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
op
erat
ing
softw
are
syst
em
afte
r mai
nten
ance
. C
ount
failu
res
whi
ch u
ser
or m
aint
aine
r enc
ount
ered
du
ring
oper
atin
g so
ftwar
e be
fore
and
afte
r m
aint
enan
ce.
Oth
erw
ise,
inve
stig
ate
prob
lem
reso
lutio
n re
port,
op
erat
ion
repo
rt or
m
aint
enan
ce re
port.
X= N
a / T
a Y
= { (
Na
/ Ta)
/ (N
b / T
b) }
Na
= N
umbe
r of c
ases
whi
ch u
ser
enco
unte
rs fa
ilure
s du
ring
oper
atio
n af
ter
softw
are
was
cha
nged
N
b =
Num
ber o
f cas
es w
hich
use
r en
coun
ters
failu
res
durin
g op
erat
ion
befo
re
softw
are
is c
hang
ed
Ta =
Ope
ratio
n tim
e du
ring
spec
ified
ob
serv
atio
n pe
riod
afte
r sof
twar
e is
cha
nged
Tb
= O
pera
tion
time
durin
g sp
ecifi
ed
obse
rvat
ion
perio
d be
fore
sof
twar
e is
ch
ange
d
0<=X
,Y
The
smal
ler
and
clos
er to
0
is th
e be
tter.
Rat
io
Na,
Nb=
C
ount
Ta
,Tb=
Ti
me
X= C
ount
/Ti
me
Y=[(C
ount
/Ti
me)
/ (C
ount
/ Ti
me)
]
Prob
lem
re
solu
tion
re
port
Mai
nten
ance
re
port
O
pera
tion
repo
rt
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 X
and
Y im
ply
“ fre
quen
cy o
f enc
ount
erin
g fa
ilure
s af
ter c
hang
e” a
nd “f
luct
uate
d fre
quen
cy o
f enc
ount
erin
g fa
ilure
s be
fore
/afte
r cha
nge”
. 2
Use
r may
nee
d sp
ecifi
ed p
erio
d to
det
erm
ine
side
effe
cts
of s
oftw
are
chan
ges,
whe
n th
e re
visi
on-u
p of
sof
twar
e is
intro
duce
d fo
r res
olvi
ng p
robl
ems.
3
It is
reco
mm
end
to c
ompa
re th
is fr
eque
ncy
befo
re a
nd a
fter c
hang
e.
4 If
chan
ged
func
tion
is id
entif
ied,
it is
reco
mm
ende
d to
det
erm
ine
whe
ther
enc
ount
ered
failu
res
are
dete
cted
in th
e ch
ange
d fu
nctio
n its
elf o
r in
the
othe
r one
s. T
he e
xten
t of i
mpa
cts
may
be
rate
d fo
r eac
h fa
ilure
.
Mod
ifica
tion
impa
ct
loca
lisat
ion
(Em
ergi
ng
failu
re a
fter
chan
ge)
Can
use
r ope
rate
so
ftwar
e sy
stem
w
ithou
t fai
lure
s af
ter
mai
nten
ance
? C
an m
aint
aine
r eas
ily
miti
gate
failu
res
caus
ed b
y m
aint
enan
ce s
ide
effe
cts?
Cou
nt fa
ilure
s oc
curre
nces
af
ter c
hang
e, w
hich
are
m
utua
lly c
hain
ing
and
affe
cted
by
chan
ge.
X= A
/ N
A=
Num
ber o
f fai
lure
s em
erge
d af
ter f
ailu
re
is re
solv
ed b
y ch
ange
dur
ing
spec
ified
pe
riod
N=
Num
ber o
f res
olve
d fa
ilure
s
0<=X
Th
e sm
alle
r an
d cl
oser
to
0 is
the
bette
r.
Abso
lute
A= C
ount
N
= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
X im
plie
s “c
hain
ing
failu
re e
mer
ging
per
reso
lved
failu
re”.
It is
reco
mm
end
to g
ive
prec
ise
mea
sure
by
chec
king
whe
ther
cau
se o
f cur
rent
failu
re is
attr
ibut
ed to
cha
nge
for p
revi
ous
failu
re
reso
lutio
n, a
s po
ssib
le.
ISO/IEC TR 9126-2:2003(E)
58 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
5.4
Test
abili
ty m
etric
s
Exte
rnal
test
abili
ty m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Avai
labi
lity
of
built
-in te
st
func
tion
Can
use
r and
m
aint
aine
r eas
ily
perfo
rm o
pera
tiona
l te
stin
g w
ithou
t ad
ditio
nal t
est f
acilit
y pr
epar
atio
n?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
te
stin
g so
ftwar
e sy
stem
af
ter m
aint
enan
ce.
X= A
/ B
A= N
umbe
r of c
ases
in w
hich
mai
ntai
ner c
an
use
suita
bly
built
-in te
st fu
nctio
n B=
Num
ber o
f cas
es o
f tes
t opp
ortu
nitie
s
0 <=
X <
=1
The
larg
er
and
the
clos
er
to 1
.0 is
the
bette
r.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
Exam
ples
of b
uilt-
in te
st fu
nctio
ns in
clud
e si
mul
atio
n fu
nctio
n, p
re-c
heck
func
tion
for r
eady
to u
se, e
tc.
Re-
test
ef
ficie
ncy
Can
use
r and
m
aint
aine
r eas
ily
perfo
rm o
pera
tiona
l te
stin
g an
d de
term
ine
whe
ther
the
softw
are
is re
ady
for o
pera
tion
or n
ot?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
te
stin
g so
ftwar
e sy
stem
af
ter m
aint
enan
ce.
X= S
um(T
) / N
T=
Tim
e sp
ent t
o te
st to
mak
e su
re w
heth
er
repo
rted
failu
re w
as re
solv
ed o
r not
N
= N
umbe
r of r
esol
ved
failu
res
0<X
The
smal
ler i
s th
e be
tter.
Rat
io
T= T
ime
N=
Cou
ntX=
Tim
e /C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
X im
plie
s “a
vera
ge ti
me
(effo
rt) to
test
afte
r fai
lure
reso
lutio
n”. I
f fai
lure
s ar
e no
t res
olve
d or
fixe
d, e
xclu
de th
em a
nd s
epar
atel
y m
easu
re ra
tio o
f suc
h fa
ilure
s.
Test
re
star
tabi
lity
Can
use
r and
m
aint
aine
r eas
ily
perfo
rm o
pera
tiona
l te
stin
g w
ith c
heck
po
ints
afte
r m
aint
enan
ce?
Obs
erve
beh
avio
ur o
f use
r or
mai
ntai
ner w
ho is
te
stin
g so
ftwar
e sy
stem
af
ter m
aint
enan
ce.
X =
A / B
A
= N
umbe
r of c
ases
in w
hich
mai
ntai
ner
can
paus
e an
d re
star
t exe
cutin
g te
st ru
n at
de
sire
d po
ints
to c
heck
ste
p by
ste
p B=
Num
ber o
f cas
es o
f pau
se o
f exe
cutin
g te
st ru
n
0 <=
X <
=1
The
larg
er
and
the
clos
er
to 1
.0 is
the
bette
r.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/ C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 59
Tabl
e 8.
5.5
Mai
ntai
nabi
lity
com
plia
nce
met
rics
Exte
rnal
mai
ntai
nabi
lity
com
plia
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Mai
ntai
nabi
lity
com
plia
nce
How
com
plia
nt is
the
mai
ntai
nabi
lity
of th
e pr
oduc
t to
appl
icab
le
regu
latio
ns,
stan
dard
s an
d co
nven
tions
?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber
of it
ems
requ
iring
co
mpl
ianc
e in
the
spec
ifica
tion.
X =
1- A
/ B
A=
Num
ber o
f mai
ntai
nabi
lity
com
plia
nce
item
s sp
ecifi
ed th
at h
ave
not b
een
impl
emen
ted
durin
g te
stin
g B=
Tot
al n
umbe
r of m
aint
aina
bilit
y co
mpl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser m
anua
l or
Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns Te
st
spec
ifica
tion
and
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TE
It m
ay b
e us
eful
to c
olle
ct s
ever
al m
easu
red
valu
es a
long
tim
e, to
ana
lyse
the
trend
of i
ncre
asin
g sa
tisfie
d co
mpl
ianc
e ite
ms
and
to d
eter
min
e w
heth
er th
ey a
re fu
lly s
atis
fied.
ISO/IEC TR 9126-2:2003(E)
60 © ISO/IEC 2003 – All rights reserved
8.6 Portability metrics
An external portability metric should be able to measure such attributes as the behaviour of the operator or system during the porting activity.
8.6.1 Adaptability metrics
An external adaptability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to adapt software to different specified environments. When a user has to apply an adaptation procedure other than previously provided by software for a specific adaptation need, user’s effort required for adapting should be measured.
8.6.2 Installability metrics
An external installability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to install the software in a user specific environment.
8.6.3 Co-existence metrics
An external co-existence metric should be able to measure such attributes as the behaviour of the system or the user who is trying to use the software with other independent software in a common environment sharing common resources.
8.6.4 Replaceability metrics
An external replaceability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to use the software in place of other specified software in the environment of that software.
8.6.5 Portability compliance metrics
An external portability compliance metric should be able to measure such attributes as the number of functions with, or occurrences of compliance problems, where the software product fails to adhere to required standards, conventions or regulations relating to portability.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 61
Tabl
e 8.
6.1
Adap
tabi
lity
met
rics
Exte
rnal
ada
ptab
ility
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Adap
tabi
lity
of
data
str
uctu
res
Can
use
r or
mai
ntai
ner e
asily
ad
apt s
oftw
are
to
data
set
s in
new
en
viro
nmen
t?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
adap
t sof
twar
e to
op
erat
ion
envi
ronm
ent.
X =
A /
B
A =
The
num
ber o
f dat
a w
hich
are
ope
rabl
e an
d bu
t are
not
obs
erve
d du
e to
inco
mpl
ete
oper
atio
ns c
ause
d by
ada
ptat
ion
limita
tions
B=
The
num
ber o
f dat
a w
hich
are
exp
ecte
d to
be
oper
able
in th
e en
viro
nmen
t to
whi
ch
the
softw
are
is a
dapt
ed
0<=X
<=1
The
larg
er
and
clos
er to
1.
0 is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
Thes
e da
ta m
ainl
y in
clud
e ty
pes
of d
ata
such
as
data
file
s, d
ata
tupl
es o
r dat
abas
es to
be
adap
ted
to d
iffer
ent d
ata
volu
mes
, dat
a ite
ms
or d
ata
stru
ctur
es. A
and
B o
f the
form
ula
are
nece
ssar
y to
cou
nt th
e sa
me
type
s of
dat
a. S
uch
an a
dapt
atio
n m
ay b
e re
quire
d w
hen,
for e
xam
ple,
the
busi
ness
sco
pe is
ext
ende
d.
Har
dwar
e en
viro
nmen
tal
adap
tabi
lity
(ada
ptab
ility
to
hard
war
e de
vice
s an
d ne
twor
k fa
cilit
ies)
Can
use
r or
mai
ntai
ner e
asily
ad
apt s
oftw
are
to
envi
ronm
ent?
Is
sof
twar
e sy
stem
ca
pabl
e en
ough
to
adap
t its
elf t
o op
erat
ion
envi
ronm
ent?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
adap
t sof
twar
e to
op
erat
ion
envi
ronm
ent.
X= 1
- A
/ B
A= N
umbe
r of o
pera
tiona
l fun
ctio
ns o
f whi
ch
task
s w
ere
not c
ompl
eted
or n
ot e
noug
h re
sulte
d to
mee
t ade
quat
e le
vels
dur
ing
com
bine
d op
erat
ing
test
ing
with
en
viro
nmen
tal
hard
war
e B=
Tot
al n
umbe
r of f
unct
ions
whi
ch w
ere
test
ed
0<=X
<=1
The
larg
er is
th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
It is
reco
mm
ende
d to
con
duct
ove
rload
com
bina
tion
test
ing
with
har
dwar
e en
viro
nmen
tal c
onfig
urat
ions
whi
ch m
ay p
ossi
bly
be o
pera
tiona
lly c
ombi
ned
in a
var
iety
of u
ser o
pera
tiona
l en
viro
nmen
ts.
Org
anis
atio
nal
envi
ronm
ent
adap
tabi
lity
(Org
anis
atio
n ad
apta
bilit
y to
in
fras
truc
ture
of
org
anis
atio
n)
Can
use
r or
mai
ntai
ner e
asily
ad
apt s
oftw
are
to
envi
ronm
ent?
Is
sof
twar
e sy
stem
ca
pabl
e en
ough
to
adap
t its
elf t
o th
e op
erat
iona
l en
viro
nmen
t?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
adap
t sof
twar
e to
op
erat
ion
envi
ronm
ent.
X= 1
- A
/ B
A=
Num
ber o
f ope
rate
d fu
nctio
ns in
whi
ch
the
task
s w
ere
not c
ompl
eted
or n
ot e
noug
h re
sulte
d to
mee
t ade
quat
e le
vels
dur
ing
oper
atio
nal t
estin
g w
ith u
ser’s
bus
ines
s en
viro
nmen
t B=
Tot
al n
umbe
r of f
unct
ions
whi
ch w
ere
test
ed
0<=X
<=1
The
larg
er is
th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 It
is re
com
men
ded
to c
ondu
ct te
stin
g w
hich
take
s ac
coun
t of t
he v
arie
ties
of c
ombi
natio
ns o
f inf
rast
ruct
ure
com
pone
nts
of p
ossi
ble
user
’s b
usin
ess
envi
ronm
ents
. 2
“Org
anis
atio
nal e
nviro
nmen
t ada
ptab
ility”
is c
once
rned
with
the
envi
ronm
ent o
f the
bus
ines
s op
erat
ion
of th
e us
er’s
org
anis
atio
n. “S
yste
m s
oftw
are
envi
ronm
enta
l ada
ptab
ility”
is
conc
erne
d w
ith th
e en
viro
nmen
t of t
he te
chni
cal o
pera
tion
of s
yste
ms.
The
refo
re,
ther
e is
a c
lear
dis
tinct
ion.
ISO/IEC TR 9126-2:2003(E)
62 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.1
(con
tinue
d)
Exte
rnal
ada
ptab
ility
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Port
ing
user
fr
iend
lines
s C
an u
ser o
r m
aint
aine
r eas
ily
adap
t sof
twar
e to
en
viro
nmen
t?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
adap
t sof
twar
e to
op
erat
iona
l env
ironm
ent.
T= S
um o
f use
r ope
ratin
g tim
e sp
ent t
o co
mpl
ete
adap
tatio
n of
the
softw
are
to u
ser’s
en
viro
nmen
t, w
hen
user
atte
mpt
to in
stal
l or
chan
ge s
etup
0<T
The
shor
ter i
s th
e be
tter.
Rat
io
T=Ti
me
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
T im
plie
s “u
ser e
ffort
requ
ired
to a
dapt
to u
ser’s
env
ironm
ent”.
Per
son-
hour
may
be
used
inst
ead
of ti
me.
Syst
em
softw
are
envi
ronm
enta
l ad
apta
bilit
y (a
dapt
abilit
y to
O
S, n
etw
ork
softw
are
and
co-
oper
ated
ap
plic
atio
n so
ftwar
e)
Can
use
r or
mai
ntai
ner e
asily
ad
apt s
oftw
are
to
envi
ronm
ent?
Is
sof
twar
e sy
stem
ca
pabl
e en
ough
to
adap
t its
elf t
o op
erat
ion
envi
ronm
ent?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
adap
t sof
twar
e to
op
erat
ion
envi
ronm
ent.
X= 1
- A
/ B
A=
Num
ber o
f ope
ratio
nal f
unct
ions
of w
hich
ta
sks
wer
e no
t com
plet
ed o
r wer
e no
t en
ough
resu
lted
to m
eet a
dequ
ate
leve
l du
ring
com
bine
d op
erat
ing
test
ing
with
op
erat
ing
syst
em s
oftw
are
or c
oncu
rrent
ap
plic
atio
n so
ftwar
e
B= T
otal
num
ber o
f fun
ctio
ns w
hich
wer
e te
sted
0<=X
<=1
The
larg
er is
th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 It
is re
com
men
ded
to c
ondu
ct o
verlo
ad c
ombi
natio
n te
stin
g w
ith o
pera
ting
syst
em s
oftw
ares
or c
oncu
rrent
app
licat
ion
softw
ares
whi
ch a
re p
ossi
bly
com
bine
d op
erat
ed in
a v
arie
ty o
f use
r op
erat
iona
l env
ironm
ents
. 2
“Org
anis
atio
nal e
nviro
nmen
t ada
ptab
ility”
is c
once
rned
with
the
envi
ronm
ent f
or b
usin
ess
oper
atio
n of
use
r’s o
rgan
isat
ion.
“Sys
tem
sof
twar
e en
viro
nmen
tal a
dapt
abilit
y” is
con
cern
ed w
ith
the
envi
ronm
ent f
or te
chni
cal o
pera
tions
on
syst
ems.
The
refo
re, t
here
is a
cle
ar d
istin
ctio
n.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 63
Tabl
e 8.
6.2
Inst
alla
bilit
y m
etric
s
Exte
rnal
inst
alla
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ease
of
inst
alla
tion
Can
use
r or
mai
ntai
ner e
asily
in
stal
l sof
twar
e to
op
erat
ion
envi
ronm
ent?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to
inst
all s
oftw
are
to
oper
atio
n en
viro
nmen
t.
X =
A / B
A
= N
umbe
r of c
ases
whi
ch a
use
r su
ccee
ded
to in
cha
ngin
g th
e in
stal
l op
erat
ion
for h
is/h
er c
onve
nien
ce
B =
Tota
l num
ber o
f cas
es w
hich
a u
ser
atte
mpt
ed to
cha
nge
the
inst
all o
pera
tion
for
his/
her c
onve
nien
ce
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TES
1 Th
is m
etric
is s
ugge
sted
as
expe
rimen
tal u
se.
2 W
hen
time
basi
s m
etric
is re
quire
d, s
pent
tim
e fo
r ins
talla
tion
may
be
mea
sura
ble.
ISO/IEC TR 9126-2:2003(E)
64 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.2
(con
tinue
d)
Exte
rnal
inst
alla
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ease
of S
etup
R
etry
C
an u
ser o
r m
aint
aine
r eas
ily re
-try
set
-up
inst
alla
tion
of s
oftw
are?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is tr
ying
to re
-try
set
-up
inst
alla
tion
of
softw
are.
X =
1 -
A / B
A
= N
umbe
r of c
ases
in w
hich
use
r fai
ls in
re
-tryi
ng s
et-u
p du
ring
set-u
p op
erat
ion
B =
Tota
l num
ber o
f cas
es in
whi
ch u
ser
atte
mpt
to re
-try
setu
p du
ring
set-u
p op
erat
ion
0<=X
<= 1
Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B=
Cou
nt
X= C
ount
/C
ount
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
This
met
ric is
sug
gest
ed a
s ex
perim
enta
l use
.
FOO
TNO
TES
The
follo
win
g co
mpl
emen
tary
met
rics
may
be
used
. 1
Effo
rtles
s in
stal
latio
n U
ser’s
man
ual a
ctio
ns fo
r ins
talla
tion
X =
A A=
The
num
ber o
f use
r’s m
anua
l act
ions
nee
ded
for i
nsta
llatio
n 0<
X he
sm
alle
r is
the
bette
r. 2
Inst
alla
tion
ease
In
stal
latio
n su
ppor
ting
leve
l X =
A A
is ra
ted
with
, for
exa
mpl
e:
- Onl
y ex
ecut
ing
inst
alla
tion
prog
ram
whe
re n
othi
ng m
ore
is n
eede
d (e
xcel
lent
); - I
nstru
ctio
nal g
uide
for i
nsta
llatio
n (g
ood)
; - S
ourc
e co
de o
f pro
gram
nee
ds m
odifi
catio
n fo
r ins
talla
tion
(poo
r).
X= D
irect
Inte
rpre
tatio
n of
mea
sure
d va
lue
3 O
pera
tiona
l ins
talla
tion
effo
rt re
duct
ion
Use
r Ins
tall
Ope
ratio
n Pr
oced
ure
Red
uctio
n R
atio
X =
1-
A / B
A
= N
umbe
r of i
nsta
ll op
erat
ion
proc
edur
es w
hich
a u
ser h
ad to
do
afte
r pro
cedu
re re
duct
ion
B =
Num
ber o
f ins
tall
oper
atio
n pr
oced
ures
nor
mal
ly
0<=
X <=
1 Th
e cl
oser
to 1
.0 is
the
bette
r. 4
Ease
of u
ser’s
man
ual i
nsta
ll op
erat
ion
Easi
ness
leve
l of u
ser’s
man
ual i
nsta
ll op
erat
ion
X =
Scor
e of
eas
ines
s le
vel o
f use
r’s m
anua
l ope
ratio
n Ex
ampl
es o
f eas
ines
s le
vel a
re fo
llow
ing:
[v
ery
easy
] req
uirin
g on
ly u
ser’s
sta
rting
of i
nsta
ll or
set
-up
func
tions
and
then
obs
ervi
ng in
stal
latio
n;
[eas
y] re
quiri
ng o
nly
user
’s a
nsw
erin
g of
que
stio
n fro
m in
stal
l or s
et-u
p fu
nctio
ns;
[not
eas
y] re
quiri
ng u
ser’s
look
ing
up p
aram
eter
s fro
m ta
bles
or f
illing
-in b
oxes
; [c
ompl
icat
ed] r
equi
ring
user
’s s
earc
hing
par
amet
er fi
les,
look
ing
up p
aram
eter
s fro
m fi
les
to b
e ch
ange
d an
d w
ritin
g th
em.
X= D
irect
Inte
rpre
tatio
n of
mea
sure
d va
lue
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 65
Tabl
e 8.
6.3
Co-
exis
tenc
e m
etric
s
Exte
rnal
co-
exis
tenc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Avai
labl
e co
-ex
iste
nce
How
ofte
n us
er
enco
unte
rs a
ny
cons
train
ts o
r un
expe
cted
failu
res
whe
n op
erat
ing
conc
urre
ntly
with
ot
her s
oftw
are?
Use
eva
luat
ed s
oftw
are
conc
urre
ntly
with
oth
er
softw
are
whi
ch u
ser o
ften
uses
.
X =
A / T
A
= N
umbe
r of a
ny c
onst
rain
ts o
r un
expe
cted
failu
res
whi
ch u
ser e
ncou
nter
du
ring
oper
atin
g co
ncur
rent
ly w
ith o
ther
so
ftwar
e T
= Ti
me
dura
tion
of c
oncu
rrent
ly o
pera
ting
othe
r sof
twar
e
0<=X
Th
e cl
oser
to
0 is
the
bette
r.
Rat
io
A= C
ount
T=
Tim
e X=
Cou
nt/
Tim
e
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
SQA
Ope
rato
r
ISO/IEC TR 9126-2:2003(E)
66 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.4
Rep
lace
abili
ty m
etric
s
Exte
rnal
repl
acea
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Con
tinue
d us
e of
dat
a C
an u
ser o
r m
aint
aine
r eas
ily
cont
inue
to u
se th
e sa
me
data
afte
r re
plac
ing
this
so
ftwar
e to
pre
viou
s on
e?
Is s
oftw
are
syst
em
mig
ratio
n go
ing
on
succ
essf
ully
?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is re
plac
ing
softw
are
to p
revi
ous
one.
X =
A / B
A
= nu
mbe
r of d
ata
whi
ch a
re u
sed
in o
ther
so
ftwar
e to
be
repl
aced
and
are
con
firm
ed
that
they
are
abl
e to
be
cont
inuo
usly
use
d B
= nu
mbe
r of d
ata
whi
ch a
re u
sed
in o
ther
so
ftwar
e to
be
repl
aced
and
pla
nned
to b
e co
ntin
uous
ly re
usab
le
0<=
X <=
1 Th
e la
rger
is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
This
met
ric c
an b
e ap
plie
d to
bot
h ca
ses
of re
plac
ing
an e
ntire
ly d
iffer
ent s
oftw
are
and
a di
ffere
nt v
ersi
on o
f the
sam
e so
ftwar
e se
ries
to p
revi
ous
one.
Fu
nctio
n in
clus
iven
ess
Can
use
r or
mai
ntai
ner e
asily
co
ntin
ue to
use
si
mila
r fun
ctio
ns a
fter
repl
acin
g th
is
softw
are
to p
revi
ous
one?
Is
sof
twar
e sy
stem
m
igra
tion
goin
g on
su
cces
sful
ly?
Obs
erve
use
r’s o
r m
aint
aine
r’s b
ehav
iour
w
hen
user
is re
plac
ing
softw
are
to p
revi
ous
one.
X =
A / B
A
= nu
mbe
r of f
unct
ions
whi
ch p
rodu
ce
sim
ilar r
esul
ts a
s pr
evio
usly
pro
duce
d an
d w
here
cha
nges
hav
e no
t be
en re
quire
d B
= nu
mbe
r of t
este
d fu
nctio
ns w
hich
are
si
mila
r to
func
tions
pro
vide
d by
ano
ther
so
ftwar
e to
be
repl
aced
0<=
X <=
1 Th
e la
rger
is
the
bette
r.
Abso
lute
A= C
ount
B= C
ount
X=
Cou
nt/
Cou
nt
Prob
lem
re
solu
tion
re
port
Ope
ratio
n re
port
5.3
Qua
lific
atio
n te
stin
g 5.
4 O
pera
tion
5.5
Mai
nte-
nanc
e
Dev
elop
er
Mai
ntai
ner
Ope
rato
r
FOO
TNO
TE
This
met
ric c
an b
e ap
plie
d to
bot
h ca
ses
of re
plac
ing
an e
ntire
ly d
iffer
ent s
oftw
are
and
a di
ffere
nt v
ersi
on o
f the
sam
e so
ftwar
e se
ries
to p
revi
ous
one.
Use
r sup
port
func
tiona
l co
nsis
tenc
y
How
con
sist
ent a
re
the
new
com
pone
nts
with
exi
stin
g us
er
inte
rface
?
Obs
erve
the
beha
viou
r of
the
user
and
ask
the
opin
ion.
X= 1
- A1
/ A
2
A= N
umbe
r of n
ew fu
nctio
ns w
hich
use
r fo
und
unac
cept
ably
inco
nsis
tent
with
the
user
’s e
xpec
tatio
n
B= N
umbe
r of n
ew fu
nctio
ns
0<=X
La
rger
is
bette
r.
Abso
lute
A1
= C
ount
A2
= C
ount
X=
C
ount
/ C
ount
Test
repo
rt O
pera
tion
repo
rt
5.3
Inte
grat
ion
5.3
Q
ualif
icat
ion
test
ing
5.4
Ope
ratio
n 6.
3 Q
ualit
y As
sura
nce
Use
r U
ser
inte
rface
de
sign
er
Mai
ntai
ner
Dev
elop
er
Test
er
SQA
FOO
TNO
TES
1 Th
e ca
se th
at a
diff
eren
t sof
twar
e is
intro
duce
d to
repl
ace
for a
pre
viou
s so
ftwar
e, a
new
diff
eren
t sof
twar
e ca
n be
iden
tifie
d as
a c
urre
nt v
ersi
on.
2 In
cas
e th
at th
e pa
ttern
of i
nter
actio
n is
cha
nged
to im
prov
e us
er in
terfa
ce in
a n
ew v
ersi
on,,
it is
sug
gest
ed to
obs
erve
use
r’s b
ehav
iour
and
to c
ount
the
num
ber o
f cas
es w
hich
the
user
fa
ils to
acc
ess
func
tions
cau
sed
by u
nacc
epta
ble
conf
orm
ity a
gain
st u
ser’s
exp
ecta
tion
deriv
ed fr
om p
revi
ous
vers
ion.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 67
Tabl
e 8.
6.5
Port
abili
ty c
ompl
ianc
e m
etric
s
Exte
rnal
por
tabi
lity
com
plia
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Port
abili
ty
com
plia
nce
How
com
plia
nt is
the
porta
bilit
y of
the
prod
uct t
o ap
plic
able
re
gula
tions
, st
anda
rds
and
conv
entio
ns ?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber
of it
ems
requ
iring
co
mpl
ianc
e in
the
spec
ifica
tion.
X =
1- A
/ B
A=
Num
ber o
f por
tabi
lity
com
plia
nce
item
s sp
ecifi
ed th
at h
ave
not b
een
impl
emen
ted
durin
g te
stin
g B=
Tot
al n
umbe
r of p
orta
bilit
y co
mpl
ianc
e ite
ms
spec
ified
0<=
X <=
1 Th
e cl
oser
to
1.0
is th
e be
tter.
Abso
lute
A= C
ount
B= C
ount
X= C
ount
/C
ount
Prod
uct
desc
riptio
n (U
ser m
anua
l or
Sp
ecifi
catio
n)
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
nsTe
st
spec
ifica
tion
and
repo
rt
5.3
Qua
lific
atio
n te
stin
g 6.
5 Va
lidat
ion
Supp
lier
Use
r
FOO
TNO
TE
It m
ay b
e us
eful
to c
olle
ct s
ever
al m
easu
red
valu
es a
long
tim
e, a
naly
se th
e tre
nd o
f inc
reas
ing
satis
fied
com
plia
nce
item
s, a
nd d
eter
min
e w
heth
er th
ey a
re fu
lly s
atis
fied.
ISO/IEC TR 9126-2:2003(E)
68 © ISO/IEC 2003 – All rights reserved
Annex A (informative)
Considerations When Using Metrics
A.1 Interpretation of measures
A.1.1 Potential differences between test and operational contexts of use
When planning the use of metrics or interpreting measures it is important to have a clear understanding of the intended context of use of the software, and any potential differences between the test and operational contexts of use. For example, the “time required to learn operation” measure is often different between skilled operators and unskilled operators in similar software systems. Examples of potential differences are given below.
a) Differences between testing environment and the operational environment
Are there any significant differences between the testing environment and the operational execution in user environment?
The following are examples:
• testing with higher / comparable / lower performance of CPU of operational computer;
• testing with higher / comparable / lower performance of operational network and communication;
• testing with higher / comparable / lower performance of operational operating system;
• testing with higher / comparable / lower performance of operational user interface.
b) Differences between testing execution and actual operational execution
Are there any significant differences between the testing execution and operational execution in user environment?
The following are examples:
• coverage of functionality in test environment;
• test case sampling ratio;
• automated testing of real time transactions;
• stress loads;
• 24 hour 7 days a week (non stop) operation;
• appropriateness of data for testing of exceptions and errors;
• periodical processing;
• resource utilisation;
• levels of interruption;
• production pressures;
• distractions.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 69
c) User profile under observation
Are there any significant differences between test user profiles and operational user profiles?
The following are examples:
• mix of type of users;
• user skill levels;
• specialist users or average users;
• limited user group or public users.
A.1.2 Issues affecting validity of results
The following issues may affect the validity of the data that is collected.
(a) procedures for collecting evaluation results:
• automatically with tools or facilities / manually collected / questionnaires or interviews;
(b) source of evaluation results
• developers' self reports / reviewers’ report / evaluator’s report;
(c) results data validation
• developers' self check / inspection by independent evaluators.
A.1.3 Balance of measurement resources
Is the balance of measures used at each stage appropriate for the evaluation purpose?
It is important to balance the effort used to apply an appropriate range of metrics for internal, external and quality in use measures.
A.1.4 Correctness of specification
Are there significant differences between the software specification and the real operational needs?
Measurements taken during software product evaluation at different stages are compared against product specifications. Therefore, it is very important to ensure by verification and validation that the product specifications used for evaluation reflect the actual and real needs in operation.
A.2 Validation of Metrics
A.2.1 Desirable Properties for Metrics
To obtain valid results from a quality evaluation, the metrics should have the properties stated below. If a metric does not have these properties, the metric description should explain the associated constraint on its validity and, as far as possible, how that situation can be handled.
a) Reliability (of metric): Reliability is associated with random error. A metric is free of random error if random variations do not affect the results of the metric.
ISO/IEC TR 9126-2:2003(E)
70 © ISO/IEC 2003 – All rights reserved
b) Repeatability (of metric): repeated use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by the same evaluators, should produce the same results within appropriate tolerances. The appropriate tolerances should include such things as fatigue, and learning effect.
c) Reproducibility (of metric): use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by different evaluators, should produce the same results within appropriate tolerances.
NOTE 1 It is recommended to use statistical analysis to measure the variability of the results.
d) Availability (of metric): The metric should clearly indicate the conditions (e.g. presence of specific attributes) which constrain its usage.
e) Indicativeness (of metric): Capability of the metric to identify parts or items of the software which should be improved, given the measured results compared to the expected ones.
NOTE 2 The selected or proposed metric should provide documented evidence of the availability of the metric for use, unlike those requiring project inspection only.
f) Correctness (of measure): The metric should have the following properties: 1) Objectivity (of measure): the metric results and its data input should be factual: i.e., not influenced by
the feelings or the opinions of the evaluator, test users, etc. (except for satisfaction or attractiveness metrics where user feelings and opinions are being measured).
2) Impartiality (of measure): the measurement should not be biased towards any particular result.
3) Sufficient precision (of measure): Precision is determined by the design of the metric, and particularly by the choice of the material definition used as the basis for the metric. The metric user will describe the precision and the sensitivity of the metric.
g) Meaningfulness (of measure): the measurement should produce meaningful results about the software behaviour or quality characteristics.
The metric should also be cost effective: that is, more costly metrics should provide higher value results.
A.2.2 Demonstrating the Validity of Metrics
The users of metrics should identify the methods for demonstrating the validity of metrics, as shown below:
(a) Correlation
The variation in the quality characteristics values (the measures of principal metrics in operational use) explained by the variation in the metric values, is given by the square of the linear coefficient.
An evaluator can predict quality characteristics without measuring them directly by using correlated metrics.
(b) Tracking
If a metric M is directly related to a quality characteristics value Q (the measures of principal metrics in operational use), for a given product or process, then a change value Q(T1) to Q(T2), would be accompanied by a change metric value from M(T1) to M(T2), in the same direction (for example, if Q increases, M increases).
An evaluator can detect movement of quality characteristics along a time period without measuring directly by using those metrics which have tracking ability.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 71
(c) Consistency
If quality characteristics values (the measures of principal metrics in operational use) Q1, Q2,..., Qn, corresponding to products or processes 1, 2,..., n, have the relationship Q1 > Q2 > ...> Qn, then the corresponding metric values would have the relationship M1 > M2 > ...> Mn.
An evaluator can notice exceptional and error prone components of software by using those metrics which have consistency ability.
(d) Predictability
If a metric is used at time T1 to predict a quality characteristic value Q (the measures of principal metrics in operational use) at T2, prediction error, which is {(predicted Q(T2) - actual Q(T2) ) / actual Q(T2)}, would be within allowed prediction error range.
An evaluator can predict the movement of quality characteristics in the future by using these metrics, which measure predictability.
(e) Discriminative
A metric would be able to discriminate between high and low quality software.
An evaluator can categorize software components and rate quality characteristics values by using those metrics which have discriminative ability.
A.3 Use of metrics for estimation (judgement) and prediction (forecast)
Estimation and prediction of the quality characteristics of the software product at the earlier stages are two of the most rewarding uses of metrics.
A.3.1 Quality characteristics prediction by current data
(a) Prediction by regression analysis When predicting the future value (measure) of the same characteristic (attribute) by using the current value (data) of it (the attribute), a regression analysis is useful based on a set of data that is observed in a sufficient period of time.
For example, the value of MTBF (Mean Time Between Failures) that is obtained during the testing stage (activities) can be used to estimate the MTBF in operation stage.
(b) Prediction by correlation analysis When predicting the future value (measure) of a characteristic (attribute) by using the current measured values of a different attribute, a correlation analysis is useful using a validated function which shows the correlation.
For example, the complexity of modules during coding stage may be used to predict time or effort required for program modification and test during maintenance process.
A.3.2 Current quality characteristics estimation on current facts
(a) Estimation by correlation analysis When estimating the current values of an attribute which are directly unmeasurable, or if there is any other measure that has strong correlation with the target measure, a correlation analysis is useful.
ISO/IEC TR 9126-2:2003(E)
72 © ISO/IEC 2003 – All rights reserved
For example, because the number of remaining faults in a software product is not measurable, it may be estimated by using the number and trend of detected faults.
Those metrics which are used for predicting the attributes that are not directly measurable should be estimated as explained below:
• Using models for predicting the attribute;
• Using formula for predicting the attribute;
• Using basis of experience for predicting the attribute;
• Using justification for predicting the attribute.
Those metrics which are used for predicting the attributes that are not directly measurable may be validated as explained below:
• Identify measures of attributes which are to be predicted;
• Identify the metrics which will be used for prediction;
• Perform a statistical analysis based validation;
• Document the results;
• Repeat the above periodically.
A.4 Detecting deviations and anomalies in quality problem prone components
The following quality control tools may be used to analyse deviations and anomalies in software product components:
(a) process charts (functional modules of software)
(b) Pareto analysis and diagrams
(c) histograms and scatter diagrams
(d) run diagrams, correlation diagrams and stratification
(e) Ishikawa (Fishbone) diagrams
(f) statistical process control (functional modules of software)
(g) check sheets
The above tools can be used to identify quality issues from data obtained by applying the metrics.
A.5 Displaying measurement results
(a) Displaying quality characteristics evaluation results The following graphical presentations are useful to display quality evaluation results for each of the quality characteristic and subcharacteristic.
Radar chart; Bar chart numbered histogram, multi-variates chart, Importance Performance Matrix, etc.
(b) Displaying measures There are useful graphical presentations such as Pareto chart, trend charts, histograms, correlation charts, etc.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 73
Annex B (informative)
Use of Quality in Use, External & Internal Metrics (Framework Example)
B.1 Introduction
This framework example is a high level description of how the ISO/IEC 9126 Quality model and related metrics may be used during the software development and implementation to achieve a quality product that meets user’s specified requirements. The concepts shown in this example may be implemented in various forms of customization to suit the individual, organization or project. The example uses the key life cycle processes from ISO/IEC 12207 as a reference to the traditional software development life cycle and quality evaluation process steps from ISO/IEC 14598-3 as a reference to the traditional Software Product Quality evaluation process. The concepts can be mapped onto other models of software life cycles if the user so wishes as long as the underlying concepts are understood.
B.2 Overview of Development and Quality Process
Table B.1 depicts an example model that links the Software Development life cycle process activities (activity 1 to activity 8) to their key deliverables and the relevant reference models for measuring quality of the deliverables (i.e., Quality in Use, External Quality, or Internal Quality).
Row 1 describes the software development life cycle process activities. (This may be customized to suit individual needs). Row 2 describes whether an actual measure or a prediction is possible for the category of measures (i.e., Quality in Use, External Quality, or Internal Quality). Row 3 describes the key deliverable that may be measured for Quality and Row 4 describes the metrics that may be applied on each deliverable at each process activity.
Table B.1 Quality Measurement Model Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8 Phase Requirement
analysis (Software and systems)
Architectural design (Software and systems)
Software detailed design
Software coding and testing
Software integration and software qualification testing
System integration and system qualification testing
Software installation
Software acceptance support
9126 series model reference
Required user quality, Required internal quality, Required external quality
Predicted quality in use, Predicted external quality, Measured internal quality
Predicted quality in use,Predicted external quality, Measured internal quality
Predicted quality in use,Measured external quality, Predicted external quality, Measured internal quality
Predicted quality in use,Measured external quality, Predicted external quality, Measured internal quality
Predicted quality in use, Measured external quality, Measured internal quality
Predicted quality in use,Measured external quality, Measured internal quality
Measured quality in use,Measured external quality, Measured internal quality
Key deliverables of activity
User quality requirements (specified), External quality requirements (specified), Internal quality requirements (specified)
Architecture design of Software / system
Software detailed design
Software code, Test results
Software product, Test results
Integrated system, Test results
Installed system
Delivered software product
Metrics used to measure
Internal metrics(External metrics may be applied to validate specifications)
Internal metrics Internal metrics
Internal metrics External metrics
Internal metrics External metrics
Internal metrics External metrics
Internal metrics External metrics
Quality in use metrics Internal metrics External metrics
ISO/IEC TR 9126-2:2003(E)
74 © ISO/IEC 2003 – All rights reserved
B.3 Quality Approach Steps
B.3.1 General
Evaluation of the Quality during the development cycle is divided into the following steps. Step 1 has to be completed during the Requirement Analysis activity. Steps 2 to 5 have to be repeated during each process activity defined above.
B.3.2 Step #1 Quality requirements identification
For each of the Quality characteristics and subcharacteristics defined in the Quality model determine the User Needs weights using the two examples in Table B.2 for each category of the measurement. (Quality in Use, External and Internal Quality). Assigning relative weights will allow the evaluators to focus their efforts on the most important subcharacteristics.
Table B.2 User Needs Characteristics & Weights (a)
Quality in Use
CHARACTERISTIC WEIGHT (High/Medium/Low)
Effectiveness H Productivity H Safety L
Satisfaction M (b)
External & Internal Quality
CHARACTERISTIC SUBCHARACTERISTIC WEIGHT (High/Medium/Low)
Suitability H Accuracy H Interoperability L Security L
Functionality
Compliance M Maturity (hardware/software/data) L
Fault tolerance L Recoverability (data, process, technology) H
Reliability
Compliance H Understandability M Learnability L Operability H Attractiveness M
Usability
Compliance H Time behaviour H Resource utilization H
Efficiency
Compliance H Analyzability H Changeability M Stability L Testability M
Maintainability
Compliance H
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 75
Table B.2 b) (continued)
Adaptability H Installability L Co-existence H Replaceability M
Portability
Compliance H
NOTE Weights can be expressed in the High/Medium/Low manner or using the ordinal type scale in the range 1-9 (e.g.: 1-3 = low, 4-6 = medium, 7-9 = high).
B.3.3 Step #2 Specification of the evaluation
This step is applied during every development process activity.
For each of the Quality subcharacteristics defined in the Quality model identify the metrics to be applied and the required levels to achieve the User Needs set in Step 1 and record as shown in the example in Table B.3.
Basic input and directions for the content formulation can be obtained from the example in Table B.1 that explains what can be measured at this stage of the development cycle.
NOTE It is possible, that some of the rows of the tables would be empty during the specific activities of the development cycle, because it would not be possible to measure all of the subcharacteristics early in the development process.
Table B.3 Quality measurement tables (a)
Quality in Use Measurement Category
CHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Effectiveness Productivity Safety
Satisfaction
(b)
External Quality Measurement Category
CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Suitability Accuracy Interoperability Security
Functionality
Compliance Maturity (hardware/software/data)
Fault tolerance Recoverability (data, process, technology)
Reliability
Compliance
ISO/IEC TR 9126-2:2003(E)
76 © ISO/IEC 2003 – All rights reserved
Understandability Learnability Operability Attractiveness
Usability
Compliance Time behaviour Resource utilisation
Efficiency
Compliance Analyzability Changeability Stability Testability
Maintainability
Compliance Adaptability Instability Co-existence Replaceability
Portability
Compliance (c)
Internal Quality Measurement Category
CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Suitability Accuracy Interoperability Security
Functionality
Compliance Maturity (hardware/software/data)
Fault tolerance Recoverability (data, process, technology)
Reliability
Compliance Understandability Learnability Operability Attractiveness
Usability
Compliance Time behaviour Resource utilisation
Efficiency
Compliance Analyzability Changeability Stability Testability
Maintainability
Compliance Adaptability Instability Co-existence Replaceability
Portability
Compliance
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 77
B.3.4 Step #3 Design of the evaluation
This step is applied during every development process activity.
Develop a measurement plan (similar to example in Table B.4) containing the deliverables that are used as input to the measurement process and the metrics to be applied.
Table B.4 Measurement plan
SUBCHARACTERISTIC DELIVERABLES TO BE EVALUATED
INTERNAL METRICS TO BE
APPLIED
EXTERNAL METRICS TO BE
APPLIED
QUALITY IN USE METRICS TO BE
APPLIED 1. Suitability 1.
2. 3.
1. 2. 3.
1. 2. 3.
(Not Applicable)
2. Satisfaction 1. 2. 3.
(Not Applicable) (Not Applicable) 1. 2. 3.
3. 4. 5. 6.
B.3.5 Step #4 Execution of the evaluation
This step is applied during every development process activity.
Execute the evaluation plan and complete the column as shown in the examples in Table B.3. ISO/IEC 14598 series of standards should be used as a guidance for planning and executing the measurement process.
B.3.6 Step #5 Feedback to the organization
This step is applied during every development process activity.
Once all measurements have been completed map the results into Table B.1 and document conclusions in the form of a report. Also identify specific areas where quality improvements are required for the product to meet the user needs.
ISO/IEC TR 9126-2:2003(E)
78 © ISO/IEC 2003 – All rights reserved
Annex C (informative)
Detailed explanation of metric scale types and measurement types
C.1 Metric Scale Types
One of the following measurement metric scale types should be identified for each measure, when a user of metrics has the result of a measurement and uses the measure for calculation or comparison. The average, ratio or difference values may have no meaning for some measures. Metric scale types are: Nominal scale, Ordinal scale, Intervals scale, Ratio scale, and Absolute scale. A scale should always be defined as M'=F(M), where F is the admissible function. Also the description of each measurement scale type contains a description of the admissible function (if M is a metric then M'=F(M) is also a metric).
(a) Nominal Scale
M'=F(M) where F is any one-to-one mapping.
This includes classification, for example, software fault types (data, control, other). An average has a meaning only if it is calculated with frequency of the same type. A ratio has a meaning only when it is calculated with frequency of each mapped type. Therefore, the ratio and average may be used to represent a difference in frequency of only the same type between early and later cases or two similar cases. Otherwise, they may be used to mutually compare the frequency of each other type respectively.
Examples: Town transport line identification number, Compiler error message identification number
Meaningful statements are Numbers of different categories only.
(b) Ordinal Scale
M'=F(M) where F is any monotonic increasing mapping that is, M(x)>=M(y) implies M'(x)>=M'(y).
This includes ordering, for example, software failure by severity (negligible, marginal, critical, catastrophic). An average has a meaning only if it is calculated with frequency of the same mapped order. A ratio has a meaning only when it is calculated with the frequency of each mapped order. Therefore, the ratio and the average may be used to represent a difference in frequency of only the same order between early and later cases or two similar cases. Otherwise, they may be used to compare mutually the frequency of each order.
Examples: School exam.result (excellent, good, acceptable, not acceptable),
Meaningful statements: Each will depend on its position in the order, for example the median.
(c) Interval Scale
M'=aM+b (a>0)
This includes ordered rating scales where the difference between two measures has an empirical meaning. However the ratio of two measures in an interval scale may not have the same empirical meaning.
Examples: Temperature (Celsius, Fahrenheit, Kelvin), difference between the actual computation time and the time predicted
Meaningful statements: An arithmetic average and anything that depends on an order
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 79
(d) Ratio Scale
M'=aM (a>0)
This includes ordered rating scales, where the difference between two measures and also the proportion of two measures have the same empirical meaning. An average and a ratio have meaning respectively and they give actual meaning to the values.
Examples: Length, Weight, Time, Size, Count
Meaningful statements: Geometrical mean, Percentage
(e) Absolute Scale
M'=M they can be measured only in one way.
Any statement relating to measures is meaningful. For example the result of dividing one ratio scale type measure by another ratio scale type measure where the unit of measurement is the same is absolute. An absolute scale type measurement is in fact one without any unit.
Example: Number of lines of code with comments divided by the total lines of code
Meaningful statements: Everything
C.2 Measurement Types
C.2.0 General
In order to design a procedure for collecting data, interpreting fair meanings, and normalizing measures for comparison, a user of metrics should identify and take account of the measure type of measurement employed by a metric.
C.2.1 Size Measure Type
C.2.1.0 General
A measure of this type represents a particular size of software according to what it claims to measure within its definition.
NOTE Software may have many representations of size (like any entity can be measured in more than one dimension - mass, volume, surface area etc.).
Normalizing other measures with a size measure can give comparable values in terms of units of size. The size measures described below can be used for software quality measurement.
C.2.1.1 Functional Size Type
Functional size is an example of one type of size (one dimension) that software may have. Any one instance of software may have more than one functional size depending on, for example:
(a) the purpose for measuring the software size (It influences the scope of the software included in the measurement);
(b) the particular functional sizing method used (It will change the units and scale).
The definition of the concepts and process for applying a functional size measurement method (FSM Method) is provided by the standard ISO/IEC 14143-1.
ISO/IEC TR 9126-2:2003(E)
80 © ISO/IEC 2003 – All rights reserved
In order to use functional size for normalization it is necessary to ensure that the same functional sizing method is used and that the different software being compared have been measured for the same purpose and consequently have a comparable scope.
Although the following often claim that they represent functional sizes, it is not guaranteed they are equivalent to the functional size obtained from applying a FSM Method compliant with ISO/IEC 14143-1. However, they are widely used in software development:
1. number of spread sheets;
2. number of screens;
3. number of files or data sets which are processed;
4. number of itemized functional requirements described in user requirements specifications.
C.2.1.2 Program size type
In this clause, the term ‘programming’ represents the expressions that when executed result in actions, and the term ‘language’ represents the type of expression used.
1. Source program size
The programming language should be explained and it should be provided how the non executable statements, such as comment lines, are treated. The following measures are commonly used.
Non-comment source statements (NCSS) include executable statements and data declaration statements with logical source statements.
NOTE 1 New program size
A developer may use newly developed program size to represent development and maintenance work product size.
NOTE 2 Changed program size
A developer may use changed program size to represent size of software containing modified components.
NOTE 3 Computed program size
Example of computed program size formula is new lines of code + 0.2 x lines of code in modified components (NASA Goddard).
It may be necessary to distinguish a type of statements of source code into more detail as follows:
i. Statement Type
Logical Source Statement (LSS). The LSS measures the number of software instructions. The statements are irrespective of their relationship to lines and independent of the physical format in which they appear.
Physical Source Statement (PSS). The PSS measures the number of software source lines of code.
ii. Statement attribute
Executable statements;
Data declaration statements;
Compiler directive statements;
Comment source statements.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 81
iii. Origin
Modified source statements;
Added source statements;
Removed source statements;
♦ Newly Developed source statements: (= added source statements + modified source statements);
♦ Reused source statements: (= original - modified - removed source statements);
2. Program word count size
The measurement may be computed in the following manner using the Halstead's measure:
Program vocabulary = n1+n2; Observed program length = N1+N2, where:
• n1: Is the number of distinct operator words which are prepared and reserved by the program language in a program source code;
• n2: Is the number of distinct operand words which are defined by the programmer in a program source code;
• N1: Is the number of occurrences of distinct operators in a program source code;
• N2: Is the number of occurrences of distinct operands in a program source code.
3. Number of modules
The measurement is counting the number of independently executable objects such as modules of a program.
C.2.1.3 Utilized resource measure type
This type identifies resources utilized by the operation of the software being evaluated. Examples are:
(a) Amount of memory, for example, amount of disk or memory occupied temporally or permanently during the software execution;
(b) I/O load, for example, amount of traffic of communication data (meaningful for backup tools on a network);
(c) CPU load, for example, percentage of occupied CPU instruction sets per second (This measure type is meaningful for measuring CPU utilization and efficiency of process distribution in multi-thread software running on concurrent/parallel systems);
(d) Files and data records, for example, length in bytes of files or records;
(e) Documents, for example, number of document pages.
It may be important to take note of peak (maximal), minimum and average values, as well as periods of time and number of observations done.
C.2.1.4 Specified operating procedure step type
This type identifies static steps of procedures which are specified in a human-interface design specification or a user manual.
The measured value may differ depending on what kinds of description are used for measurement, such as a diagram or a text representing user operating procedures.
ISO/IEC TR 9126-2:2003(E)
82 © ISO/IEC 2003 – All rights reserved
C.2.2 Time measure type
C.2.2.0 General
The user of metrics of time measure type should record time periods, how many sites were examined and how many users took part in the measurements.
There are many ways in which time can be measured as a unit, as the following examples show.
(a) Real time unit
This is a physical time: i.e. second, minute, or hour. This unit is usually used for describing task processing time of real time software.
(b) Computer machinery time unit
This is computer processor's clock time: i.e. second, minute, or hour of CPU time.
(c) Official scheduled time unit
This includes working hours, calendar days, months or years.
(d) Component time unit
When there are multiple sites, component time identifies individual site and it is an accumulation of individual time of each site. This unit is usually used for describing component reliability, for example, component failure rate.
(e) System time unit
When there are multiple sites, system time does not identify individual sites but identifies all the sites running, as a whole in one system. This unit is usually used for describing system reliability, for example, system failure rate.
C.2.2.1 System operation time type
System operation time type provides a basis for measuring software availability. This is mainly used for reliability evaluation. It should be identified whether the software is under discontinuous operation or continuous operation. If the software operates discontinuously, it should be assured that the time measurement is done on the periods the software is active (this is obviously extended to continuous operation).
(a) Elapsed time
When the use of software is constant, for example in systems operating for the same length of time each week.
(b) Machine powered-on time
For real time, embedded or operating system software that is in full use the whole time the system is operational.
(c) Normalized machine time
As in “machine powered-on time”, but pooling data from several machines of different “powered-on-time” and applying a correction factor.
C.2.2.2 Execution time type
Execution time type is the time which is needed to execute software to complete a specified task. The distribution of several attempts should be analysed and mean, deviation or maximal values should be
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 83
computed. The execution under the specific conditions, particularly overloaded condition, should be examined. Execution time type is mainly used for efficiency evaluation.
C.2.2.3 User time type
User time type is measured upon time periods spent by individual users on completing tasks by using operations of the software. Some examples are:
(a) Session time
Measured between start and end of a session. Useful, as example, for drawing behaviour of users of a home banking system. For an interactive program where idling time is of no interest or where interactive usability problems only are to be studied.
(b) Task time
Time spent by an individual user to accomplish a task by using operations of the software on each attempt. The start and end points of the measurement should be well defined.
(c) User time
Time spent by an individual user using the software from time started at a point in time. (Approximately, it is how many hours or days user uses the software from beginning).
C.2.2.4 Effort type
Effort type is the productive time associated with a specific project task.
(a) Individual effort
This is the productive time which is needed for the individual person who is a developer, maintainer, or operator to work to complete a specified task. Individual effort assumes only a certain number of productive hours per day.
(b) Task effort
Task effort is an accumulated value of all the individual project personnel: developer, maintainer, operator, user or others who worked to complete a specified task.
C.2.2.5 Time interval of events type
This measure type is the time interval between one event and the next one during an observation period. The frequency of an observation time period may be used in place of this measure. This is typically used for describing the time between failures occurring successively.
C.2.3 Count measure type
If attributes of documents of the software product are counted, they are static count types. If events or human actions are counted, they are kinetic count types.
C.2.3.1 Number of detected fault type
The measurement counts the detected faults during reviewing, testing, correcting, operating or maintaining. Severity levels may be used to categorize them to take into account the impact of the fault.
C.2.3.2 Program structural complexity number type
The measurement counts the program structural complexity. Examples are the number of distinct paths or the McCabe's cyclomatic number.
ISO/IEC TR 9126-2:2003(E)
84 © ISO/IEC 2003 – All rights reserved
C.2.3.3 Number of detected inconsistency type
This measure counts the detected inconsistent items which are prepared for the investigation.
(a) Number of failed conforming items
Examples:
• Conformance to specified items of requirements specifications;
• Conformance to rule, regulation, or standard;
• Conformance to protocols, data formats, media formats, character codes.
(b) Number of failed instances of user expectation
The measurement is to count satisfied/unsatisfied list items, which describe gaps between user's reasonable expectation and software product performance.
The measurement uses questionnaires to be answered by testers, customers, operators, or end users on what deficiencies were discovered.
The following are examples:
• Function available or not;
• Function effectively operable or not;
• Function operable to user's specific intended use or not;
• Function is expected, needed or not needed.
C.2.3.4 Number of changes type
This type identifies software configuration items which are detected to have been changed. An example is the number of changed lines of source code.
C.2.3.5 Number of detected failures type
The measurement counts the detected number of failures during product development, testing, operating or maintenance. Severity levels may be used to categorize them to take into account the impact of the failure.
C.2.3.6 Number of attempts (trial) type
This measure counts the number of attempts at correcting the defect or fault. For example, during reviews, testing, and maintenance.
C.2.3.7 Stroke of human operating procedure type
This measure counts the number of strokes of user human action as kinetic steps of a procedure when a user is interactively operating the software. This measure quantifies the ergonomic usability as well as the effort to use. Therefore, this is used in usability measurement. Examples are number of strokes to perform a task, number of eye movements, etc.
C.2.3.8 Score type
This type identifies the score or the result of an arithmetic calculation. Score may include counting or calculation of weights checked on/off on checklists. Examples: Score of checklist; score of questionnaire; Delphi method; etc.
ISO/IEC TR 9126-2:2003(E)
© ISO/IEC 2003 – All rights reserved 85
Annex D (informative)
Term(s)
D.1 Definitions
Definitions are from ISO/IEC 14598-1 and ISO/IEC 9126-1 unless otherwise indicated.
D.1.1 Quality
External quality: The extent to which a product satisfies stated and implied needs when used under specified conditions.
Internal quality: The totality of attributes of a product that determine its ability to satisfy stated and implied needs when used under specified conditions.
NOTE 1 The term “attribute” is used (rather than the term “characteristic” used in 3.1.3) as the term “characteristic” is used in a more specific sense in ISO/IEC 9126 series.
Quality: The totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs.
NOTE 2 In a contractual environment, or in a regulated environment, such as the nuclear safety field, needs are specified, whereas in other environments, implied needs should be identified and defined.
Quality in use: The capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use.
NOTE 3 Quality in use is the user’s view of the quality of an environment containing software, and is measured from the results of using the software in the environment, rather than properties of the software itself.
NOTE 4 The definition of quality in use in ISO/IEC 14598-1 does not currently include the new characteristic of “safety”.
Quality model: The set of characteristics and the relationships between them, which provide the basis for specifying quality requirements and evaluating quality.
D.1.2 Software and user
Software: All or part of the programs, procedures, rules, and associated documentation of an information processing system. (ISO/IEC 2382-1:1993)
NOTE 1 Software is an intellectual creation that is independent of the medium on which it is recorded.
Software product: The set of computer programs, procedures, and possibly associated documentation and data designated for delivery to a user. [ISO/IEC 12207]
NOTE 2 Products include intermediate products, and products intended for users such as developers and maintainers.
User: An individual that uses the software product to perform a specific function.
NOTE 3 Users may include operators, recipients of the results of the software, or developers or maintainers of software.
D.1.3 Measurement
Attribute: A measurable physical or abstract property of an entity.
ISO/IEC TR 9126-2:2003(E)
86 © ISO/IEC 2003 – All rights reserved
Direct measure: A measure of an attribute that does not depend upon a measure of any other attribute.
External measure: An indirect measure of a product derived from measures of the behaviour of the system of which it is a part.
NOTE 1 The system includes any associated hardware, software (either custom software or off-the-shelf software) and users.
NOTE 2 The number of faults found during testing is an external measure of the number of faults in the program because the number of faults are counted during the operation of a computer system running the program to identify the faults in the code.
NOTE 3 External measures can be used to evaluate quality attributes closer to the ultimate objectives of the design.
Indicator: A measure that can be used to estimate or predict another measure.
NOTE 4 The measure may be of the same or a different characteristic.
NOTE 5 Indicators may be used both to estimate software quality attributes and to estimate attributes of the production process. They are indirect measures of the attributes.
Indirect measure: A measure of an attribute that is derived from measures of one or more other attributes.
NOTE 6 An external measure of an attribute of a computing system (such as the response time to user input) is an indirect measure of attributes of the software as the measure will be influenced by attributes of the computing environment as well as attributes of the software.
Internal measure: A measure derived from the product itself, either direct or indirect; it is not derived from measures of the behaviour of the system of which it is a part.
NOTE 7 Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself.
Measure (noun): The number or category assigned to an attribute of an entity by making a measurement.
Measure (verb): Make a measurement.
Measurement: The process of assigning a number or category to an entity to describe an attribute of that entity.
NOTE 8 “Category” is used to denote qualitative measures of attributes. For example, some important attributes of software products, e.g. the language of a source program (ADA, C, COBOL, etc.) are qualitative.
Metric: A measurement scale and the method used for measurement.
NOTE 9 Metrics can be internal or external.
Metrics includes methods for categorizing qualitative data.
ISO/IEC TR 9126-2:2003(E)
ICS 35.080 Price based on 86 pages
© ISO/IEC 2003 – All rights reserved