1

Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

System Testing Though defect prevention techniques can be quite effective, and are to be encouraged, defects can and

will be introduced in a variety of different ways throughout its design and construction, regardless of the

process used (e.g., heavyweight/non-incremental or agile/incremental). Hence, software testing, which

makes use of the system/product while it is in operation or being operated on, is an essential part of all

defect elimination efforts. As discussed in the chapter on Software Quality Assurance, software testing

attempts to create trigger conditions (e.g., inputs) that will give rise to failures indicating the existence

of faults that can, hopefully, be isolated and eliminated. What differentiates system testing from unit

testing and integration testing is that the product is treated as if it were atomic (i.e., can’t be

decomposed into constituent parts).

This chapter begins with an overview of testing and a comparison of system testing and subsystem

testing (i.e. unit testing and integration testing as discussed in the chapter on Construction). It then

discusses different aspects of system testing in greater detail. Next, it considers system testing in

context. That is, it considers how system testing is used in heavyweight/non-incremental processes and

agile/incremental processes. It then considers different system testing tools and how design and

construction practices can facilitate system testing. Finally, it concludes with a discussion of various

organizational issues related to system testing.

Overview In the chapter on Software Quality Assurance Previously, testing was divided into four stages -- unit

testing, integration testing, alpha testing, and beta testing (the last two were collectively referred to as

system testing). To properly discuss system testing, we need a somewhat less abstract view.

The Testing Process Revisited As discussed in the chapter on Construction, testing often begins during the design process (with the

creation of unit tests) and certainly begins during construction (with the execution of unit tests and

integration tests). Unit testing happens in parallel. That is, each developer tests (to a greater or lesser

extent) the units that she/he is implementing.

After a sufficient number of units have been constructed and tested, they can be combined into larger

subsystems (sometimes called assemblages) and tested together. Conceptually, integration testing

begins after unit testing is completed but, in practice, there is a considerable amount of iteration and it

is often difficult to distinguish one from the other.

After integration testing has been completed, (an increment of) the product can be built from the “fully-

tested” subsystems. Despite the fact that the subsystems are “fully-tested”, there is no reason to expect

that the product will not fail. There are several reasons for this. First, the subsystems could still contain

faults. Second, there could be faults at the interfaces between the different subsystems. Third, because

the operation of the subsystems often depends on the state of the system, it may not have been

possible to “fully test” the subsystems. Hence, system testing is needed.

2

Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

The first stage of system testing is normally referred to as alpha testing. During this stage, employees of

the company that developed the product simulate the behavior of actual users. Alpha testing involves

both functional and non-functional tests, which often can be executed in parallel.

The second stage of system testing is normally referred to as beta testing, which involves actual users in

either a simulated environment or the actual environment. Like alpha testing, bet testing involves both

functional and non-functional tests, which can be executed in parallel.

This process is summarized in the following activity diagram.

One of the things that can be somewhat misleading about this diagram is the end node. While the

testing activity does, indeed, end at some point, the acceptance criteria used to determine when testing

is complete (and the negotiations that often take place) can be difficult to define. Particularly important

in this regard is the fact that one fault often hides/obscures other faults, which means that testing must

be conducted iteratively. This aspect of test planning will be ignored for now and considered in the

3

Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

chapter on Software Reliability. At this point, attention will be focused on the specification, description

and categorization of tests.

Differences between System and Subsystem Testing Subsystem testing (i.e., unit testing and integration testing) and system testing differ in two important

and related ways that are (at least partially) evident from this activity diagram. First, as mentioned

above, system testing involves (an increment of) the entire product whereas subsystem testing does

not. Second, system testing typically involves units/subsystems that were constructed by multiple

people. Unit tests, on the other hand, are typically created and executed by the same developer that

constructed the unit.

Combined, these two differences have important implications for the kinds of faults that give rise to the

failures that are detected during system testing. Subsystem testing is most likely to uncover algorithmic

faults, precision faults, and overload faults. System testing, on the other hand, might uncover these

kinds of faults but is also likely to uncover timing faults, standards faults, error checking faults, and data

handling faults.

Subsystem testing and system testing differ in several other important ways that are not evident from

this activity diagram. They differ in the kinds of people that are involved (not just the number), the

aspects of the design process that are used to create tests, the dependence of tests on the nature of the

product and the design and implementation paradigm used, and the relationships between the different

individuals that are involved.

First, subsystem testing is performed by people who have intimate knowledge of the engineering design

and implementation of the product, and, hence, can include both clear/white-box and black-box

techniques. System testing is performed by people who have an intimate knowledge of the product

design (i.e., the requirements) and, hence, only involves black-box techniques.

Second, subsystem tests are usually created from detailed requirements specifications and or detailed

user stories (i.e., sprintable stories or user stories about tasks). System tests, on the other hand, are

usually created from descriptions of interactions between users and the system (i.e., use cases or user

stories about feature). This has implications for the nature of the tests, the kinds of failures that they are

likely to trigger, and the state-dependent nature of the tests (i.e., the state of the system is likely to

change during a test).

Third, unit testing and integration testing depend, to some extent on the nature of the product (e.g., a

WWW service vs. a desktop application), whether the product is “Bought and Customized” or “Built”,

and the approach used to either customize or build the product (e.g., object-oriented vs. functional). By

the time system testing begins, however, these differences no longer matter (or, at least, matter much

less). Regardless of the way in which the product arose, it is going to be tested in essentially the same

way.

Finally, unit testing is more of an individual activity than system testing, which necessarily involves the

entire team. Hence, the problem of “finger pointing” is unlikely to arise in unit testing but may arise in

system testing. For example, in system testing opportunities arise for the software people blame the

hardware people, the GUI people to blame the database people, etc…. Obviously, this kind of “finger

pointing” can be very damaging, and steps must be taken to avoid it.

4

Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

Details of System Testing System testing can be considered at many different levels of abstraction. In the chapter on Quality

Assurance it was divided into two components, alpha testing and bet testing. In the activity diagram

above, it is divided into six different components. This section looks inside each of these six components

to provide additional detail. Despite this, the discussion that follows is still an abstraction and

considerable detail is omitted.

The differences and similarities between alpha testing, beta acceptance testing, and beta installation

testing are summarized in the following table and discussed more fully below.

Alpha Testing Beta Testing Acceptance Testing Installation Testing

Personnel Tester