The Cleanroom Approach to Quality Software Development

(or ZeroDefict Software is really possible!)

References: The Cleanroom Approach to Quality Software

Development, Michael Dyer, Wiley 1992, IEEE Software paper by Linger in March 1994

issue

An analogy Compare sight typing with touch typing (In sight typing,

the learner looks at the keys.) Touch typing takes longer to learn but results in increased

productivity with higher quality of typing in the long run. Sight typing requires the operator to switch between the

text and keys and mistakes and lapses are more common. Compare this to:

The difference between Programming & Debugging and Specification & Refinement.

The latter takes longer to learn but the payoff is increased productivity and quality.

Key features of Cleanroom

Serious programming begins after formal specification, there is emphasis on more explicit design and verification from the specification.

There is no programmer testing. The Cleanroom approach combines mathematical

reasoning during specification and design refinement to code with statistical reasoning during test case generation and testing.

The aim is to produce zero defect software (or minimal defect software).

Cleanroom Successes - proven in use since the 1980s

The US 1980 Census System Controlled by 25kloc program which operated its entire

10 months in production with no failures observed.

The IBM Wheelwriter Typewiter Systems (1984) A 65 kloc program with millions of users and no failure

ever detected.

The US Space Shuttle Software Over 500 kloc while not completely zero defect, has

been zero defect in flight.

Improved Specification and testing Cleanroom is the first practical attempt

to place software development under statistical quality control, and to deliver software with a known and certified meantime to failure

(MTTF) The key practices are:

to use formal specification and verification methods to create software of sufficient quality to forego programmer testing (ie unit test/debug) of code and

to require statistical based testing for evaluating software reliability The pay off is that Cleanroom statistically based testing with

random sampling driven from input probability distributions has been shown to be highly effective at finding errors with high failure rates. (It is better at finding the errors that occur most often.)

Cleanroom Process Flow (overview)

Software design and

development

Process ErrorDiagnosis and

Correction

Software requirements specification

Incremental software delivery

Incremental statistical testing and regression testing

Software reliability measurement

Statistical Control

Action

Basis for FEEDBACKBasis for Level 5 Process ImprovementIncremental approach based on independent specifications allows parallel development if required

The Cleanroom Process Model (in more detail)

(stacked boxes indicate successive increments) Customer requirements

Specification

Functions Usage

Incremental Development Planning

Usage SpecificationFunctional Specification

Formal design correctness verification

Statistical test-case generation

Feedback of improvements

Statistical testing

Quality certification model

MTTF Estimates

TABLE 1.1 Cleanroom Component Techniques

Technology Cleanroom Focus Perspective

Baseline Capability Defined process Starting point Design and inspection Early quality visibility

Software Specification Software quality Focal point Formal description Software correctness Drive verification Usage/build data Customer acceptance Drive validation

Software Verification Software quality Software quality In construction Error prevention Correct designs In inspection Confirmed correctness Zero defect

No Developer Testing Software acceptance Software productivity

Statistical Testing Customer acceptance Requirements validation

MTTF Prediction Software reliability Certified MTTF

Statistical Process Process improvement Software warranty

FIGURE 1.2 Roadmap for Introducing Cleanroom Component Techniques

Correctness Verification

Formal Specifications

S/w ConfigurationManagement

No developer testing

Process Control

Continuousinspection

Statisticaltesting

MTTFmeasurement

BASELINE

PROCESS

Table 1.3 Trends in Software Quality

Total Defect Rate Postdelivery Rate

Traditional Development 50 to 60 5 to 10 Unstructured design Only testing for detection

Baseline Development 20 to 40 1 to 4 Structured programming Formal inspections

Advanced Development 0 to 20 0 to 1 Correctness verification Formal specification Statistical testing

[IBM results cited in Dyer.]

Trends in decreasing defect Rates based on improving development towards full use of Cleanroom methods

Finally more recently reported results from mid90s

IBM Cobol/SF Size: 85 Kloc of PL/l Testing Error Rate: 3.4

errors per kloc Productivity Rate: 740

Loc/person month

NASA satellite-control project Size: 40 kloc of FORTRAN Testing Error Rate: 4.5

errors per kloc Productivity Rate: 780

loc/person month

IBM 3090E tape drive Size: 86 kloc of C Testing Error Rate: 1.2

errors/kloc* (N.B. comparison with Unit

Testing)

Erisesson Telecom 0532 Operation System Size: 350 kloc Testing Error Rate: 1.0

error/kloc

[in Linger 94] (sample)

1. Requirements SpecificationFunction and Performance but with UsageProbabilities and Build Strategy

2. Software Design/ImplementationIncremental Software Development but withCorrectness verification not Unit Test

3. Independent Software TestIntegration and Test of Released Incrementsbut with Representative Statistical Usage Samples

4. Software AcceptanceDemonstrated Function and Performancebut with Certified Software MTTF

Summary of Cleanroom Impacts on the SLC