09-Software Reliability Improvement
Transcript of 09-Software Reliability Improvement
-
8/4/2019 09-Software Reliability Improvement
1/28
-
8/4/2019 09-Software Reliability Improvement
2/28
Software Reliability Improvement
In A Maintenance Environment
Overview of the technique
Product environment assumptions
Use of field data
Two critical measures
Test effectiveness & escaped bugs Improvement approach
Effect on reliability
AGENDA
-
8/4/2019 09-Software Reliability Improvement
3/28
Strategy
Use field defect data to improve theability of the pre-release testing to find
the types of bugs that show up in fieldusage of the software.
This type of improvement in the pre-release test effectiveness will lead tohigher reliability in the field (assumingthe bugs found in testing are fixed beforerelease).
The problem with software testing is thatexhaustive testing is not possible.
Test smarter, not harder
-
8/4/2019 09-Software Reliability Improvement
4/28
Product Environment There is a fielded software product
(stand-alone or embedded).
New versions are being issued on aregular, periodic basis.
The new versions are incrementalupgrades and enhancements to theprevious version (not next
generation).
Version 1.0 1.1 1.3 1.4 1.5 1.61.2
-
8/4/2019 09-Software Reliability Improvement
5/28
Assumptions
Each new version is anincremental change to the
previous version. The amount of change is
roughly the same in eachversion.
The proportion of newfeatures and bug fixes ineach release isapproximately constant.
Each new version sees asimilar usage profile anddegree of usage.
Carriedforward
unchanged
New features &
enhancements
Bug fixes
-
8/4/2019 09-Software Reliability Improvement
6/28
Two Critical Measures Needed
Test effectiveness Software reliability
Field data on defects can be used to quantifyboth of these and to direct improvement efforts.
-
8/4/2019 09-Software Reliability Improvement
7/28
Availability of Field Data
An environment where there is a
mechanism for reporting bugs from the
field, and it is done. Field service organization
Help desk
Bug tracking system
-
8/4/2019 09-Software Reliability Improvement
8/28
Looking At All Stages ofTesting in Concert Inspections (peer reviews)
Unit testing Integration testing
System Testing
Acceptance testing
Filter out bugs Filter out bugs Filter out bugs
-
8/4/2019 09-Software Reliability Improvement
9/28
Basic Method Look at bugs reported (from the field)
after the release of a new version.
Bugs reported from actual users.
In a data base (so you can do queries)
Bug reports must have a date/time stamp onthem.
Release of a
new version
Reported bugs Reported bugs
-
8/4/2019 09-Software Reliability Improvement
10/28
Test Effectiveness
The purpose of
testing is to find
bugs.
An effective test
process will do
that completely.
A measure of test
effectiveness:
escaped bugs.
-
8/4/2019 09-Software Reliability Improvement
11/28
Escaped Bugs
Version
Und
er
Develo
pment
-
8/4/2019 09-Software Reliability Improvement
12/28
The Surprise Factor
Problems in the software that are foundbefore release can be fixed or managed in
other ways (put in release notes, workarounds, etc.).
This method assumes that they will befixed.
Problems that are found for the firsttime after release frequently becomecrises because they are a surprise, and
the software is in production use whereproblems can cause down time, customerdis-satisfaction, etc.
-
8/4/2019 09-Software Reliability Improvement
13/28
Quantifying Test Effectiveness
Count the number of newly reporteddefects from the field after release of theversion.
Make sure they are not duplicates of
defects previously reported (prior torelease)
This is a zero defects type of metric
(down is better). Trend it from version to version.
-
8/4/2019 09-Software Reliability Improvement
14/28
Software Test Effectiveness Trend
Number of New Defects Reported After Release
of A Version
0
20
4060
80
100
120
Ver. 1.0 Ver. 1.1 Ver. 1.2 Ver. 1.3 Ver. 1.4 Ver.1.5 Ver. 1.6
Number
ofDefects
Severity 5 Severity 4 Severity 3 Severity 2 Severity 1
(The trend shown above is an undesirable trend)
-
8/4/2019 09-Software Reliability Improvement
15/28
Software Reliability - Definition Definition: The probability of failure-free operation of
the software for a specified period of time in a
specified environment. Key aspects:
Given time period
Specified set of operating conditions
Range of values: 0.000 to 1.000
Example: A software application has a reliability of
0.93 for 24 hours when used in a typical manner.
This means that the software would operate withoutfailure over a 24 hour period for 93 out of 100 of
those periods.
Source: Software Reliability, Musa et al, p.15
-
8/4/2019 09-Software Reliability Improvement
16/28
Assumptions Reliability Calculation
Released software is in use continuously. Each new version sees about the same
number of users and about the same
overall use profile.
Examples: 1) Web site, 2) A single system
in extended, continuous use
-
8/4/2019 09-Software Reliability Improvement
17/28
Software Reliability Data Gathering
Search the bug data base for bugsreported from field use.
Look at a time period immediately afterrelease of a new version.
Must judiciously select the time period.
Could also do an extended run in the lab.
Count bugs that cause a system failure. Must establish some criteria here May want to categorize them by severity
-
8/4/2019 09-Software Reliability Improvement
18/28
Field Data - Example
Spelling error1.65Sept.
10
121
Screen lay-out poor.1.64Sept. 7120
1.61.6
1.5
1.6
1.6
1.6
1.4
1.5
1.5
Version
Entry is lost.3Sept. 5119Report look-up causes crash.1Sept. 4118
Wording is poor.5Sept. 4117
Menu missing a selection.3Sept. 3116
Wrong data displayed.2Sept. 3115
Incorrect temperature calculated.2Sept. 3114
Menu tree problem.3Sept 1113
Screen lay-out4Aug. 31112
Screen lay-out5Aug. 15111
DescriptionSeverityDateNumber
Version 1.6 release date: Sept. 1
-
8/4/2019 09-Software Reliability Improvement
19/28
Software Reliability Calculation
Formula (Source: Software Reliability, Musa et al, P.91)
R= exp (-t t )where R= reliability
t= the number of failures/hour
t= the time period for which the
reliability is to be calculated
-
8/4/2019 09-Software Reliability Improvement
20/28
Software Reliability Calculation
5 failures in a 7 day (168 hour) period.
The reliability for periods of usage of 24 hours in
length is desired.
We have:t
=5/168 = 0.0298
t
= 24Therefore:
R= exp (-tt) = exp (-0.0298) (24) = 0.489
Example (using the data from the previous table)
What this tells us is that in 100 periods of time that are each
24 hours in length, this software will run failure-free (for all
users) in 48.9 of those 24 hour periods.
-
8/4/2019 09-Software Reliability Improvement
21/28
Alternate Reliability Metric If the software versions are not in continuous use, a
different reliability measure must be used.
Mean Time Between Failure (MTBF) Concept of production time and non-production
time
Count failures, as before, but must also determine
the number of hours that the software was inproduction when it incurred those failures.
MTBF = total number of production hours dividedby the total number of failures.
Works best when data from multiple installations isaggregated.
-
8/4/2019 09-Software Reliability Improvement
22/28
Improving Test Effectiveness For every escaped bug, do a root cause
analysis.
Determine why it was not caught by thetesting of the version.
Write a test case(s) to catch that and
similar bugs, and include the test case(s)in the regression test suite.
Net effect: Testing becomes more bullet-
proof , and fewer defects are released tothe field.
-
8/4/2019 09-Software Reliability Improvement
23/28
Plugging The Holes
-
8/4/2019 09-Software Reliability Improvement
24/28
Example
Problem Report: Clean recipes ran at the wrong
time.
Analysis: Happens when running cleans after nwafers.
New test cases for system testing phase:
1) No cleaning2) Clean after every wafer.
3) Clean after every 2 wafers.
4) Clean after every 24 wafers.
5) Clean after every 25 wafers.
-
8/4/2019 09-Software Reliability Improvement
25/28
Software Test Effectiveness
Trend - Good
Number of New Defects Reported After Release of AVersion
020
40
60
80
100
120
140
160
Ver. 1.0 Ver. 1.1 Ver. 1.2 Ver. 1.3 Ver. 1.4 Ver. 1.5 Ver. 1.6
NumberofDefects
Severity 5 Severity 4 Severity 3 Severity 2 Severity 1
-
8/4/2019 09-Software Reliability Improvement
26/28
Software Reliability Trend
Field Reliability of Released Versions
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.91
Ver. 1.0 Ver. 1.1 Ver. 1.2 Ver. 1.3 Ver. 1.4 Ver. 1.5 Ver.1.6
Reliabi
lity
-
8/4/2019 09-Software Reliability Improvement
27/28
Summary
Use field defect data to provide metrics onoverall test reliability and softwarereliability. Trend them over time.
For each escaped bug, write a test case(s)that will find that and similar bugs.Include this test case in the pre-release
testing of future versions. Testing becomes oriented toward the types
of problems that actually occur in the field.
This is testing smarter, not harder. Test effectiveness and field reliability will
go up.
-
8/4/2019 09-Software Reliability Improvement
28/28
Software Quality First
Jessee Ring Principal Consultant
40119 San Carlos Place
Fremont, CA 94539510-915-2353
www.sqa1st.com