System Testing - users.cs.jmu.edu · PDF file testing, integration testing, alpha testing, and...

Click here to load reader

  • date post

    19-Jul-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of System Testing - users.cs.jmu.edu · PDF file testing, integration testing, alpha testing, and...

  • 1

    Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

    System Testing Though defect prevention techniques can be quite effective, and are to be encouraged, defects can and

    will be introduced in a variety of different ways throughout its design and construction, regardless of the

    process used (e.g., heavyweight/non-incremental or agile/incremental). Hence, software testing, which

    makes use of the system/product while it is in operation or being operated on, is an essential part of all

    defect elimination efforts. As discussed in the chapter on Software Quality Assurance, software testing

    attempts to create trigger conditions (e.g., inputs) that will give rise to failures indicating the existence

    of faults that can, hopefully, be isolated and eliminated. What differentiates system testing from unit

    testing and integration testing is that the product is treated as if it were atomic (i.e., can’t be

    decomposed into constituent parts).

    This chapter begins with an overview of testing and a comparison of system testing and subsystem

    testing (i.e. unit testing and integration testing as discussed in the chapter on Construction). It then

    discusses different aspects of system testing in greater detail. Next, it considers system testing in

    context. That is, it considers how system testing is used in heavyweight/non-incremental processes and

    agile/incremental processes. It then considers different system testing tools and how design and

    construction practices can facilitate system testing. Finally, it concludes with a discussion of various

    organizational issues related to system testing.

    Overview In the chapter on Software Quality Assurance Previously, testing was divided into four stages -- unit

    testing, integration testing, alpha testing, and beta testing (the last two were collectively referred to as

    system testing). To properly discuss system testing, we need a somewhat less abstract view.

    The Testing Process Revisited As discussed in the chapter on Construction, testing often begins during the design process (with the

    creation of unit tests) and certainly begins during construction (with the execution of unit tests and

    integration tests). Unit testing happens in parallel. That is, each developer tests (to a greater or lesser

    extent) the units that she/he is implementing.

    After a sufficient number of units have been constructed and tested, they can be combined into larger

    subsystems (sometimes called assemblages) and tested together. Conceptually, integration testing

    begins after unit testing is completed but, in practice, there is a considerable amount of iteration and it

    is often difficult to distinguish one from the other.

    After integration testing has been completed, (an increment of) the product can be built from the “fully-

    tested” subsystems. Despite the fact that the subsystems are “fully-tested”, there is no reason to expect

    that the product will not fail. There are several reasons for this. First, the subsystems could still contain

    faults. Second, there could be faults at the interfaces between the different subsystems. Third, because

    the operation of the subsystems often depends on the state of the system, it may not have been

    possible to “fully test” the subsystems. Hence, system testing is needed.

  • 2

    Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

    The first stage of system testing is normally referred to as alpha testing. During this stage, employees of

    the company that developed the product simulate the behavior of actual users. Alpha testing involves

    both functional and non-functional tests, which often can be executed in parallel.

    The second stage of system testing is normally referred to as beta testing, which involves actual users in

    either a simulated environment or the actual environment. Like alpha testing, bet testing involves both

    functional and non-functional tests, which can be executed in parallel.

    This process is summarized in the following activity diagram.

    One of the things that can be somewhat misleading about this diagram is the end node. While the

    testing activity does, indeed, end at some point, the acceptance criteria used to determine when testing

    is complete (and the negotiations that often take place) can be difficult to define. Particularly important

    in this regard is the fact that one fault often hides/obscures other faults, which means that testing must

    be conducted iteratively. This aspect of test planning will be ignored for now and considered in the

  • 3

    Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

    chapter on Software Reliability. At this point, attention will be focused on the specification, description

    and categorization of tests.

    Differences between System and Subsystem Testing Subsystem testing (i.e., unit testing and integration testing) and system testing differ in two important

    and related ways that are (at least partially) evident from this activity diagram. First, as mentioned

    above, system testing involves (an increment of) the entire product whereas subsystem testing does

    not. Second, system testing typically involves units/subsystems that were constructed by multiple

    people. Unit tests, on the other hand, are typically created and executed by the same developer that

    constructed the unit.

    Combined, these two differences have important implications for the kinds of faults that give rise to the

    failures that are detected during system testing. Subsystem testing is most likely to uncover algorithmic

    faults, precision faults, and overload faults. System testing, on the other hand, might uncover these

    kinds of faults but is also likely to uncover timing faults, standards faults, error checking faults, and data

    handling faults.

    Subsystem testing and system testing differ in several other important ways that are not evident from

    this activity diagram. They differ in the kinds of people that are involved (not just the number), the

    aspects of the design process that are used to create tests, the dependence of tests on the nature of the

    product and the design and implementation paradigm used, and the relationships between the different

    individuals that are involved.

    First, subsystem testing is performed by people who have intimate knowledge of the engineering design

    and implementation of the product, and, hence, can include both clear/white-box and black-box

    techniques. System testing is performed by people who have an intimate knowledge of the product

    design (i.e., the requirements) and, hence, only involves black-box techniques.

    Second, subsystem tests are usually created from detailed requirements specifications and or detailed

    user stories (i.e., sprintable stories or user stories about tasks). System tests, on the other hand, are

    usually created from descriptions of interactions between users and the system (i.e., use cases or user

    stories about feature). This has implications for the nature of the tests, the kinds of failures that they are

    likely to trigger, and the state-dependent nature of the tests (i.e., the state of the system is likely to

    change during a test).

    Third, unit testing and integration testing depend, to some extent on the nature of the product (e.g., a

    WWW service vs. a desktop application), whether the product is “Bought and Customized” or “Built”,

    and the approach used to either customize or build the product (e.g., object-oriented vs. functional). By

    the time system testing begins, however, these differences no longer matter (or, at least, matter much

    less). Regardless of the way in which the product arose, it is going to be tested in essentially the same

    way.

    Finally, unit testing is more of an individual activity than system testing, which necessarily involves the

    entire team. Hence, the problem of “finger pointing” is unlikely to arise in unit testing but may arise in

    system testing. For example, in system testing opportunities arise for the software people blame the

    hardware people, the GUI people to blame the database people, etc…. Obviously, this kind of “finger

    pointing” can be very damaging, and steps must be taken to avoid it.

  • 4

    Fox and Bernstein, Software Engineering (Draft) – System Testing v1 Copyright © 2014

    Details of System Testing System testing can be considered at many different levels of abstraction. In the chapter on Quality

    Assurance it was divided into two components, alpha testing and bet testing. In the activity diagram

    above, it is divided into six different components. This section looks inside each of these six components

    to provide additional detail. Despite this, the discussion that follows is still an abstraction and

    considerable detail is omitted.

    The differences and similarities between alpha testing, beta acceptance testing, and beta installation

    testing are summarized in the following table and discussed more fully below.

    Alpha Testing Beta Testing Acceptance Testing Installation Testing

    Personnel Tester