SENG 637 Dependability Reliability & Dependability...

SENG 637SENG 637Dependability Reliability & Dependability Reliability & Dependability, Reliability & Dependability, Reliability & Testing of Software Testing of Software SystemsSystems

ClCl S ft D l tS ft D l tCleanroomCleanroom Software DevelopmentSoftware Development(Chapter 11)(Chapter 11)

Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （[email protected]）http://www enel ucalgary ca/People/far/Lectures/SENG637/

SENG635 (Winter 2007) [email protected] 1

http://www.enel.ucalgary.ca/People/far/Lectures/SENG637/

BackgroundBackgroundBackgroundBackground Chaos Report [Standish 1995]

Based on data representing 8 380 SE projects only 16 2% of projects Based on data representing 8,380 SE projects, only 16.2% of projects met the delivery date, the budget and with all of the specified features and functions. 31% of projects were cancelled before they were completed 52 7% were delivered with over-budget over-schedule orcompleted, 52.7% were delivered with over budget, over schedule or with fewer features and functions than specified.

Software Productivity Research [Chapman 2000]Software Productivity Research [Chapman 2000] %60 of the United State’s software work force is dedicated to fixing

software errors that could have been avoided. In addition, there are only 47 days in a calendar year dedicated to doing development oronly 47 days in a calendar year dedicated to doing development or enhancement of software applications. The rest is spent mainly on fixing the bugs.

[email protected] 2

The First Computer Bug!The First Computer Bug!The First Computer Bug!The First Computer Bug! On September 9th 1945,

Grace Murray Hopper wasGrace Murray Hopper was working on the Harvard University Mark II Aiken Relay Calculator when theRelay Calculator when the machine was experiencing problems.

An investigation showed An investigation showed that there was a moth trapped between the points of Relay #70 in Panel Fof Relay #70, in Panel F.

[email protected] 3

Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988.

http://www.history.navy.mil/photos/pers-us/uspers-h/g-hoppr.htm

Research QuestionResearch QuestionResearch QuestionResearch Question Is it possible to build software without any

bug in it?

Answer: May be. By using cleanroom software developmentsoftware development

[email protected] 4

Causes for Bugs in ProgramsCauses for Bugs in ProgramsCauses for Bugs in ProgramsCauses for Bugs in Programs The main causes for bugs in programs:

Design flaws Coding error

O h (i l di h l d ) Other (including human related error)

The first two can be eliminated by formal (e.g. box structure) design verification and

t t d d t C tifi tiautomated code generators. Certification testing will take care of the last.

[email protected] 5

Cleanroom SECleanroom SECleanroom SECleanroom SE Cleanroom software engineering (CSE) is an engineering

process for the development of high quality software withprocess for the development of high quality software with certified reliability with the emphasis on design with no defects and test based on software reliability engineeringconcepts.concepts.

CSE focuses on defect prevention instead of defect correction, and certification of reliability for the intended environment of use.

CSE yields software that is correct by mathematically sound design, and software that is certified by statistically valid testing.g

CSE represents a paradigm shift from traditional, craft-based SE practices to rigorous, engineering-based practices.

[email protected] 6

CSE: CharacteristicsCSE: CharacteristicsCSE: CharacteristicsCSE: Characteristics Objective:Objective: Achieve zero defects with certified

reliability Focus:Focus: Defect prevention rather than defect p

correction Process:Process: Incremental (short) developmentProcess:Process: Incremental (short) development

cycles; long product life

[email protected] 7

CSE: HistoryCSE: HistoryCSE: HistoryCSE: History 1983: Original idea of Cleanroom came from one of Dr. Harlan Mills’

published papers1987 P d b D Mill SE h d l Th 1987: Proposed by Dr. Mills as a SE methodology. The name “Cleanroom” was borrowed from the electronics industry

1988: Defense Advanced Research Projects Agency (DARPA) Software Technology for Adaptable Reliable Systems (STARS) focus on gy p y ( )Cleanroom

1991-1992: Prototyping of Cleanroom Process Guide 1992: A book of CSE published, foundation of CSE

1992 1993 A d Ai F D i f Cl 1992-1993: Army and Air Force Demonstration of Cleanroom Technology

1993-1994: Prototyping of Cleanroom tools 1995: Commercialization of a Cleanroom Certification Tool 1995: Commercialization of a Cleanroom Certification Tool 1995: Cleanroom and CMM Consistency Review …

[email protected] 8

ComparisonComparisonComparisonComparisonCraftCraft--Based SEBased SE Cleanroom SE

Sequential or chaos development

Informal design

Incremental development

Disciplined engineering Informal design

Unknown reliability

specification and design

Measured reliability

Individual development Peer reviewed engineering

Individual unit testing

Informal load or coverage testing

Team correctness verification

Statistical usage testing

[email protected] 9

Informal load or coverage testing Statistical usage testing

Cleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: Technologies Development practices are based on mathematical function

theorytheory Test practices are based on applied statistics.

Analysis and design models are based on incremental software model and created using box structure

t ti A b l t th t (representation. A box encapsulates the system (or some aspect of the system) at a specific level of abstraction.

Correctness verification is applied once the box structureCorrectness verification is applied once the box structure design is complete.

[email protected] 10

Cleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: Technologies Software is tested by defining a set of usage scenarios (i.e.,

i i l d ) d i i hoperations or operational modes), determining the probability of use for each scenario (i.e., operational profile), and then defining random tests that conform to theand then defining random tests that conform to the probabilities.

Error records are checked No corrective actions are taken Error records are checked. No corrective actions are taken. Only certification test is conducted to check whether errors (i.e., current failure intensity) meet the projected reliability ( , y) p j y(i.e., failure intensity objective) for the software component.


CSE: Processes /1CSE: Processes /1CSE: Processes /1CSE: Processes /1Cleanroom processes:1. Management process2. Specification process3. Development process4. Certification process


CSE: Management ProcessCSE: Management ProcessCSE: Management ProcessCSE: Management Process Project Planning

Cleanroom engineering guide Software development plan (incremental)

P j M Project Management Project record

Performance Improvement Performance improvement plan

Engineering Change Engineering change log


CSE: Specification Process /1CSE: Specification Process /1CSE: Specification Process /1CSE: Specification Process /1 Requirements AnalysisRequirements Analysis

Elicitation and analyzes of requirements Elicitation and analyzes of requirements Define requirements for the software product Obtain agreement with the customer on the requirements

Requirements are reconfirmed or clarified throughout the Requirements are reconfirmed or clarified throughout the incremental development and certification process.

Function SpecificationFunction SpecificationB th lt f R i t A l i Base on the result of Requirements Analysis

Specify the complete functional behavior of the software in all possible modes of use

Obtain agreement with the customer on the specified function as the basis for software development and certification


CSE: Specification Process /2CSE: Specification Process /2CSE: Specification Process /2CSE: Specification Process /2 Usage SpecificationUsage Specification

Identify and classify software users usage scenarios and Identify and classify software users, usage scenarios, and environments of use (operational modes)

Establish and analyze the probability distribution for ft d lsoftware usage models

Obtain agreement with the customer on the specified usage as the basis for software certification

Architecture SpecificationArchitecture Specification Define the conceptual model, the structural organization,

and the execution characteristics of the softwareand the execution characteristics of the software Architecture definition is a multi-level activity that spans

the life cycle


CSE: Specification Process /3CSE: Specification Process /3CSE: Specification Process /3CSE: Specification Process /3 Increment PlanningIncrement Planning

Allocate customer requirements defined in the Function Specification to a series of softwarei h i f h S f A hiincrements that satisfy the Software Architecture,

Define schedule and resource allocations for i t d l t d tifi tiincrement development and certification

Obtain agreement with the customer on the increment planincrement plan


CSE: Development ProcessCSE: Development ProcessCSE: Development ProcessCSE: Development ProcessIncrement 1Increment 1

RGRGBSSBSS FDFD CVCV CGCG CICI

TPTPSUTSUT CC

SESE RGRGBSSBSS FDFD CVCV CGCG CICI

SUTSUT CC

Increment 2Increment 2

SESE RGRGTPTP

SUTSUT CC

i i CG C d G iSE: System EngineeringRG: Requirement GatheringBSS: Box structure specificationFD: Formal DesignCV: Correctness Verification

CG: Code GenerationCI: Code InspectionSUT: Statistical Use TestingC: Certification TestTP: Test Planning


CV: Correctness Verification TP: Test Planning

Cleanroom Strategy /1Cleanroom Strategy /1Cleanroom Strategy /1Cleanroom Strategy /1 Requirement gathering (RG)Requirement gathering (RG)

A detailed description of customer level requirements for each increment.

Box structure specification (BSS)Box structure specification (BSS) Box structure specification (BSS)Box structure specification (BSS) Functional specification using box structure to separate

behavior, data and procedures.p

Formal design (FD)Formal design (FD) Specifications (black boxes) are refined to become

analogous to architectural (state boxes) and procedural (clear boxes) design.


Cleanroom Strategy /2Cleanroom Strategy /2Cleanroom Strategy /2Cleanroom Strategy /2 Correctness verification (CV)Correctness verification (CV)

A set of correctness verification activities on the design and moves later to code. First level verification is via application of a set of “correctness questions”.pp q

Code generation, inspection & verification (CG & Code generation, inspection & verification (CG & CI)CI) The box structure transformed to a programming language.

Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structureensure semantic conformance with the box structure.


Cleanroom Strategy /3Cleanroom Strategy /3Cleanroom Strategy /3Cleanroom Strategy /3 Statistical test planning (TP)Statistical test planning (TP)

Planning the test based on operational modes, operational profiles and reliability.

Statistical use testing (SUT)Statistical use testing (SUT) Statistical use testing (SUT)Statistical use testing (SUT) Creating test case, execute them and collecting error data.

Certification (C)Certification (C) Certification (C)Certification (C) Conducting certification test rather than reliability growth

to accept/reject developed software components (using reliability demonstration chart, etc).


Box Structure /1Box Structure /1Box Structure /1Box Structure /1

Box structures are used to move from an abstract specification to a detailed design providing implementation details


Box Structure /2Box Structure /2Box Structure /2Box Structure /2 Black boxBlack box

S ifi th b h i f t t f t Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response.

State boxState box Encapsulates state data and services (operations) Input to Encapsulates state data and services (operations). Input to

the state box and outputs are represented. Clear boxClear box

Transition function that are implied by the state box. It contains the procedural design of the state box.



S Rf: S* → R

S R

l k b

StateT

gBlack box

State

S Rg11

g12

g13

ccg1

f S* RS R

T clear box

f: S* → R

St t b

Black boxes (specifications)

State boxes (architectural designs)


State box Clear boxes (component designs)


CBCB1.1.11.1.1CBCB1.1.11.1.1

CBCB1.1.21.1.2CBCB1.1.21.1.2BBBB1.11.1

BBBB1.1.11.1.1

BBBBBBBB1.11.1

BBBB1.1.11.1.1

BBBB

SBSB1.1.11.1.1SBSB1.1.11.1.1

CBCB1.1.31.1.3CBCB1.1.31.1.3

BBBB11 BBBB1.21.2

BBBB1.1.21.1.2

BBBB1.1.31.1.3BBBB11 BBBB1.21.2

BBBB1.1.21.1.2

BBBB1.1.31.1.3

BBBBBBBBState boxState box

Clear boxClear box

BBBB1.n1.nBBBB1.n1.n

Black boxBlack box


Black Box Structure /4Black Box Structure /4Black Box Structure /4Black Box Structure /4 If having more than one black box or nested

black boxes verify the mapping

f f

g h g

Sequential split h

c


Parallel split



ff

hg c

g

Loop split

cg

Loop split

c


p pp p

Advantages of VerificationAdvantages of VerificationAdvantages of VerificationAdvantages of Verification Design verification has the following

advantages: Verification is reduced to a finite process Every step of design and every line of code can be

verified Near zero defect level is achieved Scalability is possible Better code than unit testing can be generated


CSE: Certification Process /1CSE: Certification Process /1CSE: Certification Process /1CSE: Certification Process /1 Usage modeling and test planningUsage modeling and test planning

A usage model represents a possible usage scenario of the software A usage model represents a possible usage scenario of the software Usage model is based on usage specification and is used for testing


CSE: Certification Process /2CSE: Certification Process /2CSE: Certification Process /2CSE: Certification Process /2 Statistical Testing and CertificationStatistical Testing and Certification

Testing is conducted in a formal statistical design under experimental control.

The software is demonstrated to perform correctly with The software is demonstrated to perform correctly with respect to its specification.

Statistically valid estimates of the properties addressed by the certification goals are derived for the software.

Management decisions on continuation of testing and certification of the software are based on statisticalcertification of the software are based on statistical estimates of software quality.


Cleanroom TestingCleanroom TestingCleanroom TestingCleanroom Testing Using statistical usage concept for testing. Determine a usage probability distribution via the

following steps:1) A l h ifi i id if f i li1) Analyze the specification to identify a set of stimuli

(direct and indirect input variables).2) Create usage scenarios (operational modes).2) Create usage scenarios (operational modes).3) Assign probability to use of each stimuli (operational

profile).4) Generate test cases for each stimuli according to the

usage probability distribution.


Certification TestCertification TestCertification TestCertification Test Cleanroom approach DOES NOT emphasize on

Unit or integration testing Bug fixing as a result of test and regression

C tifi ti d i l th f ll i Certification procedure involves the followings: Create usage scenarios Specify a usage profile Specify a usage profile Generate test cases from the profile Execute test cases and record failure data Compute reliability and certify the component or system

using reliability demo chart, etc.


Reliability Demo ChartReliability Demo ChartReliability Demo ChartReliability Demo Chart

An efficient way of h ki h h hchecking whether the

failure intensity objective (F) is met or not based on collectingnot based on collecting failure data at time points.

Vertical axis: failure Vertical axis: failure number

Horizontal axis:normalized failure datanormalized failure data, i.e.,

failure time F


ExampleExampleExampleExample Automated Teller Machine (ATM) Requirements:Requirements:

The customer has a PIN number and access-card to use the ATM

The customer can deposit, withdraw money from the account

Transaction involves no bank employee


Example: Usage ModelExample: Usage ModelExample: Usage ModelExample: Usage Model


Example: Black BoxesExample: Black BoxesExample: Black BoxesExample: Black Boxes Black boxesBlack boxes

Card Processor In: ValidCard(cardNum) Out: showMessage(message)

Boolean Cash Dispenser

In: enoughCashInMachine(amount)dispenseCash(amount)

Out: showMessage(message)dispense(amount)Boolean

Transaction Manager In: ValidCustomer(cardNum, pin)

AmountLimit(amount)EnoughCashInAccount(amount)

Out: showMessage(message) Boolean


Example: State BoxExample: State BoxExample: State BoxExample: State Box

/ i t d

Menu

/ insert card

[false][true]

Ch k

/ get pin [false]

Check MachineCash

[false]

[true]

Check

[true]

Cash

[true]

[false]

Check Account

[true] /get amount


Example: Clear Box Spec /1Example: Clear Box Spec /1Example: Clear Box Spec /1Example: Clear Box Spec /1

// Get customer PIN no

Menu

/ insert card//ValidCustomer(cardNum, pin)

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Bank returns false

Menu

/ insert card//// Show messageshowMessage(mesg);

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Bank returns true

Menu

/ insert card//// get amountgetAmount(amount);

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Bank returns false for daily limit

Menu

/ insert card// y// and/or balance// Show messageshowMessage(mesg);

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Bank returns true for daily limit

Menu

/ insert card// y// and balanceDispenser.enoughCashInAccount(amount)

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Dispenser returns false for

Menu

/ insert card// p// cash level// Show messageshowMessage(mesg);

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]



// Dispenser returns true for

Menu

/ insert card// p// cash amountDispenser.dispense(amount);

/ get pin

CheckMachine

[false][false]

[true]

CheckCheckMachine

Cash

[false]

CheckAccount

[true] /get amount

[true]


CSE: TeamCSE: TeamCSE: TeamCSE: Team Specification team:

Responsible for developing and maintaining the system specification Responsible for developing and maintaining the system specification

Development team: Responsible for developing and verifying the software The software is not executed during this process

Certification team: Responsible for developing a set of statistical tests to exercise the

f f d lsoftware after development Use reliability growth models to assess reliability


CSE: EvaluationCSE: EvaluationCSE: EvaluationCSE: Evaluation Basic features of Cleanroom development that

distinguishes it from other SE methodologies are: Formal specification (Box structure) Correctness verification Statistical certification test


Evaluation: Formal SpecEvaluation: Formal SpecEvaluation: Formal SpecEvaluation: Formal Spec Advantages:Advantages:

M th ti l d l i l f d ti f d fi i Mathematical and logical foundation for defining requirements accurately with precise notation.

Proactive versus reactive approach with regards to requirements validation.

Ambiguous, inconsistent and conflicting requirements are caught before the system test.g y

Box structure uses black, state, and clear box and it is a stepwise approach to refine requirements.Usage models define how the software is to be used by the Usage models define how the software is to be used by the customer.


Evaluation: Formal SpecEvaluation: Formal SpecEvaluation: Formal SpecEvaluation: Formal Spec Disadvantages:Disadvantages:

Requires extra skills and knowledge (e.g. mathematics). Requires substantial effort to fully express the system in

formal specificationformal specification. On average Cleanroom projects require 60-80% of the time used in

analysis and design Id l f f i i i i l d f di Ideal for safety or mission critical systems and not for ordinary commercial development.

Lacks good enough CASE tools supporting. Project specific

If time-to-market & conditions are issues, then might not be used


Evaluation: Incremental DevelEvaluation: Incremental DevelEvaluation: Incremental DevelEvaluation: Incremental Devel AdvantagesAdvantages

Quick and clean development in Cleanroom Engineering Quick and clean development in Cleanroom Engineering Continuous validation Provides measurable progress Manage higher risk requirements (i e prototype) Manage higher risk requirements (i.e. prototype). Tracking of requirements Stepwise building functionalities that satisfies stakeholders’

requirementsrequirements Allows for fast delivery on important parts Focus on planning and discipline at management level and technical

level Statistical testing make the project quality control in proper level Verifiable specifications


Evaluation: Incremental DevelEvaluation: Incremental DevelEvaluation: Incremental DevelEvaluation: Incremental Devel Disadvantages:Disadvantages:

Incomplete or conflicting requirements cannot be resolved at the beginning to determine increments

Risk analysis has not been incorporated explicitly Risk analysis has not been incorporated explicitly Need more care about configuration management Requires extra planning at both the management and q p g g

technical levels Stable requirements for each increment is needed, i.e.,

cannot adapt q ickl to “rapidl changing” req irementscannot adapt quickly to “rapidly changing” requirements


Evaluation: Certification TestEvaluation: Certification TestEvaluation: Certification TestEvaluation: Certification Test AdvantagesAdvantages

Determines a level of confidence that a software system conforms to a specification

Able to statistically evaluate and infer the quality of the software system to meet all requirements

Quantitative approach that is verifiable Quantitative data could be recorded and used later

f b h kifor benchmarking, etc.


Evaluation: Certification TestEvaluation: Certification TestEvaluation: Certification TestEvaluation: Certification Test DisadvantagesDisadvantages

Testing is derived from a usage model that must be exhaustive in order to select a subset for testing

Statistical testing and verification will be more reliable if it Statistical testing and verification will be more reliable if it is based on the some history data

It would be effective if it could be integrated with other testing methods

Testing is not suitable for bug-huntingH man resid al coding errors ma not be addressed Human residual coding errors may not be addressed


C St dC St d

Section 2Section 2

Case StudyCase Study

SENG635 (Winter 2007) [email protected] 55

Cleanroom SE: Case StudyCleanroom SE: Case StudyCleanroom SE: Case StudyCleanroom SE: Case Study Cleanroom software development relies on a

th ti ll d d l f d i tmathematically sound model of design to ensure that no defects are introduced into the softwaresoftware.

Cleanroom Software Specification and Design begins with an external view (black box) andbegins with an external view (black box), and is transformed into a state machine view (state box), and is fully developed into a procedurebox), and is fully developed into a procedure (clear box).


Box StructureBox StructureBox StructureBox Structure Box structures map system inputs and the

ti l hi t i ( i i t ) i tstimulus histories (previous inputs) into outputs.I Bl k B t t ffi i t t t Is Black-Box construct sufficient to represent this? e.g. Jackson modelN No

Box structure

inputs outputs

hi t


history

Cleanroom SE: Process /1Cleanroom SE: Process /1Cleanroom SE: Process /1Cleanroom SE: Process /11) Define the system requirements2) S if d lid t th bl k b2) Specify and validate the black box

Define the system boundary and specify all stimuli and responsesp

Specify the black box mapping rules Validate the black box with owners and users

) if d if h b3) Specify and verify the state box Specify the state data and initial state values Specify the state box transition function Specify the state box transition function Derive the black box behavior of the state box and

compare the derived black box for equivalence


Cleanroom SE: Process /2Cleanroom SE: Process /2Cleanroom SE: Process /2Cleanroom SE: Process /24) Design and verify the clear box

Design the clear box control structures and operations Embed uses of new and reused black boxes as necessary

D i h b b h i f h l b d Derive the state box behavior of the clear box and compare the derived state box to the original state box for equivalence

5) Repeat the process for new black boxes6) Convert to code)7) Certification test the code


Requirements AnalysisRequirements AnalysisRequirements AnalysisRequirements Analysis Example:

Build a simple calculator

Detailed definition of the calculator function and what it does must be given and verifiedand what it does must be given and verified with the customer Various formal methods can be used: graph Various formal methods can be used: graph

theory, automaton model, etc.


Black Box Structure /1Black Box Structure /1Black Box Structure /1Black Box Structure /1 Entry #1: first operand (xxx digits) Entry #2: calculation symbol (add, subtract) Entry #3: second operandEntry #3: second operand Entry #4: equal symbol

E it #1 l l ti lt Exit #1: calculation result

Box structure

inputs outputs

Box structure


p phistory

Black Box Structure /2Black Box Structure /2Black Box Structure /2Black Box Structure /2

calculator

1st operand null

calculator

1 operand nullnull

calculator

Calc symbol null

calculator

Calc symbol null1st operand

PushPush--down automaton modeldown automaton model


queue

Black Box Structure /3Black Box Structure /3Black Box Structure /3Black Box Structure /3

calculator

2nd operand null1st operand1st operandCalc sym

queue

calculator

Equal sym Calc results1st operandCalc sym


2nd operandqueue



f f

g h

Sequential split

g

cq ph

Parallel split




ff

hg c

g

Loop split

cg

Loop split

c


p pp p

State Box Structure /1State Box Structure /1State Box Structure /1State Box Structure /1 State transition diagram happy path

Error

non-numeric key pressed

Error


State Box Structure /2State Box Structure /2State Box Structure /2State Box Structure /2 Several state boxes can be generated

d di th bi ti f t bldepending on the combination of acceptable (unacceptable) inputs and historiesE l Example: If 1st operand is non-numeric and calc symbol are

typed the next state is error statetyped the next state is error state If 1st operand is numeric and any other key other

than calc symbol is typed the next state is error y ypstate

etc.


State Box Structure /3State Box Structure /3State Box Structure /3State Box Structure /3 State boxes should be generated for all

possible combinations of input(s) and history states.

The set of state boxes can easily grow beyond control!


Clear Box Structure /1Clear Box Structure /1Clear Box Structure /1Clear Box Structure /1


Code GenerationCode GenerationCode GenerationCode Generation Coding will be based on the clear boxes Use of automatic code generation tools is

encouraged to reduce the probability of human g p yerror


Cleanroom Testing /1Cleanroom Testing /1Cleanroom Testing /1Cleanroom Testing /1 Cleanroom testing teams must determine a

usage probability distribution for the software The operational profile can be used


Cleanroom Testing /2Cleanroom Testing /2Cleanroom Testing /2Cleanroom Testing /2 Suppose that the inputs to the calculator program are

Input percentage number

A1 1st operand (correct) % 22 0 – 21

A2 1st operand (incorrect) %3 22 – 24

B1 2nd operand (correct) % 22 25 – 46B1 2 operand (correct) % 22 25 46

B2 2nd operand (incorrect) %3 47 – 49

C1 Calculation symbol (correct) % 22 50 – 71

C2 Calculation symbol (incorrect) %3 72 – 74

D1 Equal symbol (correct) % 22 75 – 96D2 Equal symbol (incorrect) %3 97 – 99


D2 Equal symbol (incorrect) %3 97 99

Cleanroom Testing /3Cleanroom Testing /3Cleanroom Testing /3Cleanroom Testing /3 We must generate a sequence of usage test cases that conform

to the usage probability distribution.to the usage probability distribution. A series of random numbers are generated between 0 and 99

that corresponds to the probability of stimuli occurrence. F l th f ll i d b For example, the following random number sequences are generated: 14 – 95 – 26 – 44 : A1; D1; B1; B1 81 – 19 – 31 – 69 38 – 21 – 52 – 84

The testing team executes the test cases noted above (and Test case

g (others) and verifies software behavior against the specification for the system.


Cleanroom Testing /4Cleanroom Testing /4Cleanroom Testing /4Cleanroom Testing /4 For example for the test case

T1:T1: A1; D1; B1; B1A1; D1; B1; B1The inp t seq ence is:The input sequence is:

1. 1st operand2 E l b l2. Equal symbol3. 2nd operand4 2nd operand4. 2 operand

And the output should be: ErrorError


Cleanroom CertificationCleanroom CertificationCleanroom CertificationCleanroom Certification The verification and testing techniques lead to certification of

software components p Certification implies that the reliability can be specified for

each component. Each component would have a certified reliability under the p y

usage scenario and testing regime. This information is needed for future use of the components.

The certification approach involves five steps: pp p 1. Usage scenario is created. 2. A usage profile is specified. 3. Test cases are generated from the profile. 4. Tests are executed and failure data are recorded and analyzed. 5. Reliability is computed and certified.


CSE: Overall AdvantagesCSE: Overall AdvantagesCSE: Overall AdvantagesCSE: Overall Advantages Suitable for iterative and incremental software

d l tdevelopment. Uses formal specification that defines more

t l fli t d l taccurate, less conflict and complete requirements.C ti ifi ti f ft lit i Continuous verification of software quality is possible.S ft lit b tifi d i Software quality can be certified using software reliability engineering method.


CSE: Overall DisadvantagesCSE: Overall DisadvantagesCSE: Overall DisadvantagesCSE: Overall Disadvantages Cleanroom advocates the use of sequence-based

specifications These are better suited to problemsspecifications. These are better suited to problems with a high degree of logical interactions. Not suitable for black boxes used in numerical or highly computational applications.

Non-functional requirements (real-time, security constraints) and a significant portion of algorithmicconstraints) and a significant portion of algorithmic requirements are hard to be represented by Box structure.

After requirements changes, rework of the box-structure is a time-consuming process.


Statistical test data may be hard to be collected.

Conclusions Conclusions Conclusions Conclusions Cleanroom approach is a rigorous approach to

ft i i th t h h isoftware engineering that has emphasis on: Formal specification

M th ti l ifi ti f t f d i Mathematical verification of correctness of design Certification of software reliability

Cleanroom approach is yet to become a ti i ft d l tcommon practice in software development

industry because of emphasis on the above three points


three points

ReferencesReferencesReferencesReferences Linger, R. and Trammel, C. (1996). Cleanroom

Software Engineering Reference Model Version 1 0Software Engineering Reference Model Version 1.0. http://www.sei.cmu.edu/pub/documents/96.reports/pdf/tr022.96.pdfp

Wolack, C. (2001). Taking The Art Out of Software Development – An In-Depth Review of Cleanroom S ft E i iSoftware Engineering. http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm

Pressmen and Associates (2000). Cleanroom Engineering Resources.


http://www.rspa.com/spi/cleanroom.html

One Last Advice One Last Advice One Last Advice One Last Advice Want to impress your customers: use failure

intensity + reliability growth methodology!

Want to impress your boss (development): use failure density + zero time failurefailure density zero time failure methodology!

Want to impress yourself: use target failure i t it + li bilit d t ti h t!


intensity + reliability demonstration chart!

That is all folks!


SENG 637 Dependability Reliability & Dependability...

Documents

Transcript of SENG 637 Dependability Reliability & Dependability...