Software Reliability Engineering

Software Reliability EngineeringDeFacto Model

Hans Sassenburg (SE-CURE AG)

Lucian Voinea (SolidSource BV)

1© SolidSource BV, SE-CURE AG, 2010.SRE - DeFacto Model

Contents

1. Introduction

2. Basic Model

3. Measures and Metrics

4. Extended Model – DeFacto

Appendix: Tool Examples


1. Introduction

• Software reliability is the probability of failure-freeoperation with respect to execution time and environment.

• Software reliability engineering (SRE) is the quantitativestudy of the operational behavior of software-basedsystems with respect to user requirements concerningreliability.

• SRE is increasingly adopted by many companies as itbecomes an important economic consideration(competitive advantage, dependencies, safety regulations),others still regard reliability a cost rather than a value.

• Creditable software reliability techniques are still in urgentneed.

• Hardware failures are repeatable and more predictable(wear-out) while software failures are largely unpredictable.


System-of-System Characteristics

• Continuous evolution and deployment– There is an increasing need to integrate new capabilities into a system

while it is operating.– New and different capabilities are deployed, and unused capabilities

will be dropped; the system evolves not in phases, but continuously.

• Heterogeneous, inconsistent, and changing elements– A system is not constructed from uniform parts: there are always some

misfits, especially as the system is extended and repaired.

• Erosion of the people/system boundary– People are not just users of a system; they are elements of the system,

affecting its overall emergent behavior.

• Normal failures– Software and hardware failures are the norm rather than the

exception.

SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 4

Errors, Defects and Failures

• Error

– Mistake made by a human action or tool during programming.

• Defect (Fault)

– Manifestation of an error during execution.

• Failure

– Incorrect software behaviour in operational context due to a defect.


Errors, Defects and Failures


Pre-release Post-release

Unit/integrationtest coverage

Errors (actual number unknown)

Manifested defects

Manifested failures

SizeComplexity

Testing

Systemtest coverage Time

Defect and failure rates drop after

releasing and continue to grow

depending on contexts of use

(operational profile), and number of users

Post-release situation


Errors (actual number unknown)

Manifested defects (actual number unknown)

Manifested failures

Released defects

Released manifested failures

Released errors

Featurecoverage

during operation

Theoretical maximum

Post-release

SRE Components

1. Error prevention

• to avoid, by construction, error occurrences

2. Defect removal

• to detect, by verificationand validation, the existence of errors and remove them

3. Failure forecasting

• to estimate, by statistical modeling, the presence of defects and occurrence of failures


feedback loops

2. Basic Model

• For software reliability prediction, you need a model ofthe relationship between the predicted variable andother measurable variables.

• Assumptions– We can accurately measure some property of software or

process.– A relationship exists between what we can measure and

what we want to know.– This relationship is understood, has been validated, and

can be expressed in terms of a formula or model.

• Few metrics have been demonstrated to be predictableor related to software reliability.


Problems with Existing Models

• Too many models, often overly complex

• Wrong assumptions

– Operation environment same as testing environment;

– Once a failure occurs, the fault causing it is removed;

– Fault removal will not introduce new faults;

– Pre-release defects can be used as predictor for post-release failures;

– …


Simplified Approach

• Distinguish in the reliability model three phasesfrom the product lifecycleError introduction

• Pre-release (especially during design and implementation)

Defect identification• Pre-release (especially during testing)

Failure manifestation• Pre and post-release

• Determine for each phase the main factors thatinfluence reliability


Basic Model with Reliability Factors

•Problem complexity

•Design complexity

•Code complexity

•Domain knowledge

•Project constraints

Error introduction

•Module and integration testing

•Model verification

Defect identification •System Testing

•Contexts of use

•Agility of demand

•Number of users

Failure manifestation

Drivers

Detection methods

Operational profile


3. Measures and Metrics

• Error introductionFactor: Design and code complexity unnecessarily high.Consequence: overlook achievable execution states during

programming = more errors.

• Defect identificationFactor: Code limitedly verifiedConsequence: limited number of execution states investigated.

• Failure manifestationFactor: Contexts of use not always considered/known.Consequence: clients use the product in ways that reveal

unconsidered execution states.


Error Introduction


Measures Metrics

Design complexity Fan-inFan-out

Code complexity SizeMcCabe cyclomatic complexityCode duplicationChange propagationHalstead Volume

Factor: Design and code complexity unnecessarily high

Defect Identification


Measures Metrics

Module testing completion

Code coverage during testing

Integration testing completion

Code and interface coverage during testing

Factor: Code limitedly verified

Failure Manifestation


Factor: Contexts of use not always considered/known

Measures Metrics

System test completion

Number of verified requirements withBoundary and Pairwise testing.

Usage context Number of verified requirements (withBoundary and Pairwise testing) from therequirements that are needed toimplement the functionality used 80% ofthe time.

Feedback Loops

• It can be argued that software parts with many errorsbeing found are likely to have more errors remaining,since the same factors that produced the error affectthe remainder of the code as well.

• This means that failure occurrences are notindependent, and in fact, will exhibit correlations thatdepend on the operational profile as well as on thesystem itself.

• It is therefore crucial discovering these correlationsacross multi-dimensional data to improve the errorprevention and defect removal processes.


4. Extended Model - DeFacto

• Basic Model– Distinguishes three phases with reliability factors.

• Extended Model - DeFacto– Assign measures to those reliability factors that

are relevant and derive appropriate metrics;

– Customize model to specific contexts byadding/removing factors and calibrating factorweights via correlations discovered using historicaldata.


DeFacto – Example Metrics

• Design complexityFan-in, Fan-out

• Code complexityMcCabe, Duplication

Error introduction

• Module + Integration testingCode coverage

Defect identification • System Testing

Requirement coverage with Boundary and Pairwise testing

Failure manifestation


Appendix: SolidFX = Fact eXtractor for C/C++

• The Fact eXtractor (SolidFX) is astandalone, non-intrusive solution forstatic analysis of industry-size projectswritten in the C and C++ programminglanguages.

• SolidFX uses proprietary technology toanalyze even the most complex C/C++code bases efficiently and robustly.

• SourceFX offers predefined analysisscenarios and metrics to measureC/C++ code quality, maintainability,modularity, detect potential bugs andextract design from source code – allfor coding faster, cleaner, better.


Automatically compute

McCabe complexity, Fan-in, Fan-out,…

Error introduction

measurement

www.solidsourceit.com

http://www.solidsourceit.com/

Appendix: SolidSDD = Software Duplication Detector

• The Source Code Duplication Detector(SolidSDD) is a standalone applicationfor detecting and managing sourcecode duplication (i.e., code clones) insoftware.

• It can be used to analyze large projectsand detect code that has been cloned(e.g., via cut-n-paste operations) duringdevelopment.

• The currently supported programminglanguages are C, C++, C# and Java. Inaddition to identifying the code clonefragments, SolidSDD offers an intuitivegraphical interface for assessing thecode duplication characteristics and thelocation of the duplicated fragments inthe code stack.

• SolidSDD is able to cope with variationsin the duplicated code; not only exactclones are detected but also duplicatedfragments that have been modified byrenaming variables or byinserting/removing code.


Automatically compute

code duplication

Error introduction

measurement



Appendix: SolidSX = Software eXplorer

• The Software Explorer (SolidSX) is astandalone Windows application thatgives insight in large software systems.SolidSX creates high-qualityvisualizations that simultaneously showthe structure, dependencies, metricson all types of source code elements(files, classes, methods, fields, etc.).

• By using hardware-acceleratedgraphics, SolidSX is able to display largeamounts of information in a clear andconcise manner and provides fast andeasy exploration through large sourcecodes.

• SolidSX uses plug-ins to importdependencies and metrics fromdifferent sources.


Discover correlations across

multidimensional data

Model calibration



Appendix: SolidSTA = Software Trend Analyzer

• The Software Trend Analyzer (SolidSTA)is a standalone, non-intrusive solutionfor monitoring and investigatingsoftware trends.

• SolidSTA uses a number of proprietaryand standard metric analyses to assessthe evolution of software qualityindicators for industry-size codeversioning repositories.

• The set of metrics can be extendedwith custom analyses via a plug-insystem with an open API.SolidSTA presents the analyses resultsin an intuitive way to enable users todiscover trend correlations and makefact-based informed decisions.

• Overviews of team activity or systemmetrics can be produced in minutes.

• No repository management expertise isrequired.


Compute change propagation and

analyze software history/trends.

Error introduction

measurementModel calibration




www.sw-benchmarking.org

• The Software Benchmarking Organization (SBO) wasfounded in 2009 by SE-CURE AG and SolidSource BV.

• The objective is to provide benchmarking services tothe software industry.

• The portfolio includes benchmarking studies,workshops, and supporting products.

• The portfolio is brought to the market through aninternational network with accredited partners.

Software Benchmarking Organization

http://www.sw-benchmarking.org/







Contact Information

• Dr. Lucian VoineaT +31 (40) 203 4290M +31 (6) 1436 3842E [email protected]

• Dr. Hans Sassenburg:T +41 (33) 733 4682M +41 (79) 231 6600E [email protected]


mailto:[email protected]






Software Reliability Engineering

Technology

Transcript of Software Reliability Engineering