7

Cleanroom Software Engineering

Casey Ehlers

University of Wisconsin Platteville

Spring 2011

ehlersc@uwplatt.edu

Abstract

In a world where software is starting to appear in everything (from telephones to toasters), there is a growing necessity in the development of reliable and safe systems of code. Many theories, processes, and principles have been developed in order to strive towards safe and dependable software. This paper will showcase one of the more promising and new theories: the Cleanroom software process. The Cleanroom process applies statistical correctness verification to the formal method of incremental software design. Through usage testing each increment, the programmer is given well-defined statistics (through applying mathematics to the usage tests) as to how closely the software system is functioning compared to the desired outcome. This process leads to zero-failure software upon delivery and installation.

Introduction to Cleanroom Software Engineering

The term Cleanroom, when applied to software, can mean a couple different things. When defined vaguely, it can be a philosophy; the idea of striving towards reliable code. When defined more concretely, it refers to a process. This paper will highlight the latter. Just like more familiar processes (i.e. waterfall, spiral, or iterative), the Cleanroom process offers a well-defined way to develop and design systems of code. This process, which will be discussed more thoroughly in an upcoming section, relies on a traditional and familiar methodology of software design: incremental development. What makes Cleanroom separate from a standard incremental process is what is actually done during each increment. The focus of Cleanroom is to completely prevent errors from arising during the lifecycle of a piece of software. This differs from the norm, where errors are acceptable as an inescapable reality, and the focus should be on finding these errors. [1]

Error prevention through the Cleanroom process is done in a very innovative way. Each increment is verified for correctness and tested rigorously. These usage tests yield mathematical statistics to the customers and programmers which verify that the system is actually doing what it was designed to do. The more vigorous and well-defined the tests and increments are, the more accurate the mathematics can assess the current functionality of the system. [1] Ultimately, this process strives towards zero failures by the time the software is ready to be used. [2]

This paper looks to give a brief overview of the Cleanroom Software Process. The next section offers a quick summary of the history of the process. Then the development process will be highlighted. Sections on correctness verification and usage testing follow; which showcases what makes Cleanroom unique from the incremental process. Finally, benefits for using this software process will wrap up the report.

History of Cleanroom Software Engineering

The term Cleanroom and the theory behind it have been around for the past 30 years. Coined by Harland Mills in a 1980 paper, it was actually part of a different idea of Mills; the kernel. [1] Over the years, the term has slowly morphed into the process that it is identified as today. The actual word Cleanroom came from the hardware realm. Hardware Cleanroom is a very extensive and careful discipline used in the development of hardware. This engineering process sought after the prevention of defects, instead of dealing with defects as they show up. [2] Mills focused on the prevention of errors in software, which naturally explains the adaptation of the term Cleanroom.

Besides the similar names, software and hardware Cleanroom processes needed to be drastically different, Mills noted. Hardware failure was often a result of a physical defect in its engineering, whereas software failures were often blamed on poor logic and design. [1] This difference is what motivated Mills to develop the Cleanroom process into what it is today.

Cleanroom Software Development Process

Unlike more universal and conventional methods of software engineering, there isnt a uniform diagram that is used when describing the process. For this report, the diagram used in the Linger and Trammell report will be featured. [2] Since the incremental aspect of the developmental process is regularly used in industry, and more readily understood, this paper will only focus on the additions to the incremental process which transform it to a Cleanroom process. This includes how increments are functionally defined, how those increments go about being verified for correctness, and the results of this verification process.

Figure 1: The Cleanroom Process

What is Correctness?

Just so there isnt any ambiguity with the term correctness, a specific definition should be provided. When the term correctness is used in Software Engineering, traditionally it refers to whether or not the software system is actually doing what it is intended and designed to do. When engineers are designing a piece of software based on correctness, they should ask themselves simply, Does this do what I want? [3] When software isnt correct, a whole mess of problems could result. One of the big results of designing software that is not correct is bugs and defects. Since the aim of Cleanroom is to develop software that has significantly higher quality and predictable field reliability correctness is extremely important. [1]

Getting Started: Defining Increments

As soon as the requirements have been gathered and defined, it then becomes the engineers duty to design increments that can be used in the Cleanroom process. The increments are functionally defined, meaning that what an increment is going to do and how it is going to do it is specified. A software increment in this process is seen as a checkpoint or a deliverable. [4] This is because most of the increments that are developed in this process will traditionally be shown to the customer for feedback. Additionally, increments have to satisfy two conditions upon being completed. Firstly, they have to terminate properly. Secondly, they have to be equivalent to what is originally defined in the design of the increment. [1] Another thing to note with these increments is that they may be referred to as enumerations depending on who you are talking to. [4]

To elaborate on this definition, take the example of designing a calculator. Increments of a software system that acts as a calculator could be math functions of that calculator. For example, one increment could be strictly for the add function to the system. As engineers using the Cleanroom process, the increment would then be defined functionally and designed thoroughly before being implemented. One might use a state diagram to specify the add functions functionality. Although it isnt necessary, a state diagram could be very helpful for the Cleanroom process because it provides the engineers a rigorous and well-defined chart that the increment can be designed from. The more well-defined the increment is, the easier it will be to verify that they are correct during the next step of the engineering process. Besides state diagrams, engineers can also use mapping structures to develop their increments because mapping structures follow set mathematical rules for their relationships. These relationships can then be confirmed easily when it is time to verify that the increment or enumeration has been completed. [2]

Box Structures and Cleanroom

Another way to functionally define increments is through box mapping. Provided in the Trammel and Linger report there are three ways to map functions when defining increments. Black Box mapping takes the current input and the history of inputs (any prior input) and maps them to a desired response. [2] This type of mapping would be great for the calculator example because the history of what has been entered in the calculator matters in what the resulted output would be.

Another way to map functions is the state-box way. The state-box mapping structure is very similar to a state diagram because it takes an input and current state and maps it to a response and new state. Instead of needing a history, states are used. [2] The third way to map functions is a modification of the state-box way called clear-box. The clear-box takes the state-box structure and adds a procedure to it. [2]

Increments Continued

After functionally defining the adding increment for the calculator example, pseudo-code can be developed. The pseudo-code would contain necessary variables, inputs, and outputs for adding with a calculator. Each construct that should be used in coding the increment (i.e. for loops, while, switch) could be planned here so that when they are put into the increment, they can be verified that they work properly. A good example for what this looks like is provided at the start of the next section. Once the increments have been defined clearly, it is the engineers duty to produce working code that is as close to the design as possible. From there each increment can be verified for its correctness.

Correctness Verification

Correctness verification is the core idea behind Cleanroom engineering. As shown in Figure 1 above, upon delivering increments to the customer, the engineers of the system can also provide certification that the system does what it is designed to do via correctness verification. Since testing has been traditionally done when the code has been implemented, Cleanroom aims at finding those errors upon their introduction to the system. This shift in where errors are taken care of results in more reliable software that has been designed in a cost-effective manner. [1]

Verifying that that system is being designed correctly is the core idea behind the Cleanroom process. This process takes the specifications from the previous steps and affirms that the conditions of the specifications have been met. [2] An important thing to note is that even though there are countless ways a program may run, the control structures in that program (for example, a nested if statement) have a finite amount of outcomes. This is what is being looked at and verified to work. Ultimately, by looking at a finite number of outcomes for structures, instead of just overall paths a system can take during a lifecycle, defects are eliminated during the development of the code. [2]

Dyers book offers a great table that provides some examples of how certain control structures can be verified for correctness. Due to lack of space, lets highlight one simple example from the table. Consider the syntax structure of an if-then-else statement:

Figure 2: Construct Syntax Example

Using this general syntax of an if-else statement, it is easy to come up with a correctness proof for the 2 possible outcomes. Since the if-test could result in either being true or false, does the if- get executed when the if-statement is true and does the get executed when the if-statement is false? [1] By verifying that the if-tests in a system (along with any other major construct structure), software can be produced with minimum defects.

Besides constructs, inputs and methods can also be verified for correctness. In the example below, Dyer shows us what a correctness proof for a function would look like.

Figure 3: Design/Correctness Proof Example

This function is for a small program which takes in inputs as sides of a triangle and determines whether or not the triangle is scalene, isosceles, or equilateral. A small system like this gives a good example for what an increment would generally look like. Here, the engineer simply has to take what has been specified while designing the increment and show that it works properly. [1] Once the actual function has been produced, additional correctness proofs will have to be applied to any constructs used in the actual computation and production of the result. For example, since the specification mentions an EOF being used, there will at least be one while loop used to monitor the EOF. Verification would have to be done on this while loop to ensure that loop control and termination was working correctly.

Usage Testing

The same principles apply to testing an increment as they do for verifying an increment. Although Cleanroom aims at certifiably reliable software, it isnt cost efficient to test every path and every possible branch combination due to its almost infinite quantity. [2] However, testing techniques can be applied in junction with verification methods to produce software systems that are stable and reliable.

The first set of testing practices basically restricts path-based testing. Structured path testing tries to limit unnecessary paths by limiting the tests to a set amount of inputs. [1] Another way of suggested path testing looks at the boundaries and interiors of looping. For this type of testing, loop code would be tested with and without iteration. When the loop coded isnt iterated, it is considered the boundary case, and when the loop code is executed, it is considered the interior case. [1] Another possible way of usage testing during the Cleanroom process is called linear code sequence and jump testing. This is similar to stepping through a program with a debugger. Statements are broken down into sequences which then can be stepped through and tested for defects. [1] With applying the right testing methods, efficient and thorough testing can compliment Cleanroom correctness verification, and lead to very sound and safe software systems.

Analysis of Cleanroom Engineering

Looking at the method behind the Cleanroom process and seeing it implemented in industry, there are many substantial benefits to using this methodology. With very few minor and theoretical setbacks, the pros greatly outweigh the cons.

Benefits of Cleanroom

The biggest and most obvious benefit of the Cleanroom process is a drastic increase in reliability when it comes to the final delivered software. This benefit can be backed up statistically from industry examples. As Linger and Trammell report in their 2006 technical paper, customers of software that had been developed using the Cleanroom method had an increase in quality by 10 to 11 times. [2] This increase in quality yields greater profit margins and happier end users. Another report showed that 90 percent of defects were found before the actual testing phase (after incremental development) in software systems that were ranged from 1,000 to 50,000 lines of code. [3]

Even though the main goal for the Cleanroom process is to create zero-defect software, Cleanroom also strives towards cost effective and efficiently developed code. With such a well-defined and rigorous specification and verification process during the initial design and development of the software, less time is spent at the end fixing defects and errors that never made their way into the code. The earlier in the life-cycle those defects can be cleaned out of the code, the more productive the software team will be. NASA reports that on their projects that use the Cleanroom process they have seen a 4.6 times productivity increase, which yields to happy customers and less production costs. [2]

Are There Downsides of Cleanroom?

It is important to also consider the downsides of a software process before developing with it. Since no single methodology is perfect there are possible downsides of picking the Cleanroom process over more traditional methods. One initial critic of the Cleanroom method states that its process can be overwhelming and unnecessarily complex at times. A simple system or a system that wasnt life critical may not need such rigorous method of producing reliable code. [5] Additionally, the Cleanroom process requires formal training on all levels of software development, including managers and requirement engineers. Without everyone buying into the process, there will be issues with team cohesion and productivity. In the long run, if a software company is making the shift over to Cleanroom, it might be very beneficial, but trying to get a software team to switch over to a different method like Cleanroom can be harmful in the end. [5]

Summary and Conclusion

Overall, the Cleanroom process offers an innovative way to design quality software. By specifying increments based on functional requirements, defects are removed out of the system before the code is even finished. Through the incremental process, enumerations of code are specified and designed based on their functionality. Through different ways, the increment can then be verified to ensure that it is functioning properly. Instead of detecting errors, Cleanroom prevents them from even forming. For the right project with the right software team, this can be a valuable tool for producing code with high standards and great customer reviews.

References

[1] Michael Dyer. (1992) The Cleanroom Approach To Quality Software Development New York, NY: John Wiley & sons

[2] Richard C. Linger and Carmen J. Trimmell (1996) Cleanroom Software Engineering Reference Model

Version 1.0 Pittsburgh, PN

[3] Roger S. Pressmen (1992) Software Engineering: A Practitioners Approach (3rd Edition) New York, NY: McGraw-Hill Inc

[4] Shawn P. Garbett (2003) Cleanroom Software Engineering Retrieved from

http://drdobbs.com/architecture-and-design/184405405

[5] Chaelynne M. Wolak (2001) Taking the Art out of Software Development: An In Depth Review of Cleanroom Software Engineering Retrieved From http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm