Chapter 11: Testing

The dynamic verification of the behavior of a program on a finite set of test cases, suitable selected from the usually infinite execution domain, against the expected behavior

Testing Fundamentals

Testing is the one step in software engineering process that could be viewed as destructive rather than constructive. A successful test is one that breaks the software.

A successful test is one that uncovers an as yet undiscovered defect.

Testing can not show the absence of defects, it can only show that software defects are present.

For most software exhaustive testing is not possible.

Testing Unit testing

White-box Code walkthroughs and inspections

Integration testing Bottom-up Top-down Sandwich Big Bang

Functional testing (black-box) Performance testing Acceptance testing Installation testing

System testing

Functional testing Performance testing Acceptance testing Installation testing

System Testing Process

Functional testing: does the integrated system perform as promised by the requirements specification?

Performance testing: are the non-functional requirements met?

Functional Testing

Test cases derived from requirements specification document Black box testing Independent testers Test both valid and invalid input and the

success of the test is determined by the produced output

Equivalence partitioning Boundary values

Equivalence partitioning

Reduces the number of test cases to a necessary minimum and select the right test cases to cover all possible scenarios

Example: method accepts an int value for month (1..12) as a parameter

Performance Testing

Stress tests Volume tests Recovery tests Security tests Timing tests

Environmental tests Quality tests Maintenance tests Documentation tests Human factors

(usability) tests

Acceptance Tests

Enable the customers and users to determine if the built system meets their needs and expectations

Written, conducted, and evaluated by the customers

Types of Acceptance Tests

Pilot test: install on experimental basis

Alpha test: in-house test Beta test: customer pilot Parallel testing: new system operates

in parallel with old system

Installation test

Does the system run at the customer site(s)?

System Testing

Pop quiz

How does system testing differ from unit and integration testing?

How does unit testing differ from integration testing?

In a method, a variable did not get initialized properly.

Which type of testing would most likely expose this defect?

A. Unit testingB. Integration testingC. Functional testingD. Performance testingE. Acceptance testingF. Installation testing

A gas pump system is supposed to allow the user to choose whether or not a receipt is printed, but the print function has not been implemented.

Which type of testing is most likely to expose this defect?

A. Unit testingB. Integration testingC. Functional testingD. Performance testingE. Acceptance testingF. Installation testing

A configuration file used by the reporting subsystem is not placed in the correct directory in the customer's environment.

Which type of testing is most likely to expose this defect?

A. Unit testingB. Integration testingC. Function testingD. Performance testingE. Acceptance testingF. Installation testing

The customer is unhappy with the number of screens that must be traversed before getting to the parts list screen, a screen accessed frequently when using the system.

Which type of testing is most likely to expose this defect?

A. Unit testingB. Integration testingC. Function testingD. Performance testingE. Acceptance testingF. Installation testing

Short answer question

What is the difference between verification and validation testing?

Test Documentation Test plan: describes system and plan for

testing all functions and characteristics Test case specification: details each test

and defines criteria for evaluating each feature

Test incident report: results of each test Test report summary: lists all failures

from the tests that need to be investigated

Test Plan (part of 480 QA plan)

Define the subsystems to be tested (and not tested)

Describe the process to follow for unit, integration, system, and acceptance testing

List the test cases

Plan the test schedule

Test Documentation

Defect Tracking Form

Regression Testing

Identifies new faults that may have been introduced as current ones are being corrected

Verifies that a new version or release still performs the same functions in the same manner as an older version or release

Quality Assurance concerns

Software reliability: operating without failure under given condition for a given time interval

Software availability: operating successfully according to specification at a given point in time

Software maintainability: for a given condition of use, a maintenance activity can be carried out within stated time interval, procedures and resources

Testing Safety-Critical Systems Recognize that testing cannot remove all faults

or risks

Assume that every mistake users can make will be made Do not assume that low-probability, high-

impact events will not happen

Emphasize requirements definition, testing, code and specification reviews, and configuration control Cleanroom testing

Different Levels of Failure Severity

Catastrophic: causes death or system loss Critical: causes severe injury or major

system damage Marginal: causes minor injury or minor

system damage Minor: causes no injury or system damage

One-minute quiz

What is meant by regression testing?

Is regression testing used for verification or validation?