1

Testing

Course notes forCEN 4010

2

Outline

Introduction: terminology and philosophy

Factors that influence testing

Testing techniques

3

Why Do We Test?

We test to find bugs “Testing is the process of executing a program

with the intent of finding errors” A good test intends to find bugs

“A good test case is one that has a high probability of detecting an as-yet undiscovered error”

A test that finds no bug is a failure “A successful test case is one that detects an

as-yet undiscovered error”

4

Testing vs. Debugging

Testing != Debugging

While testing may reveal a symptom of an error, it may not uncover the exact cause of the error

Debugging is the process of locating the exact cause of an error, and removing that cause

5

Our Very Real Problem

We can never say for an arbitrary program: This software has no errors This software works only as intended This software is safe

Testing proves the presence, not the absence, of bugs

Absence of evidence is not evidence of absence

6

4 dimensions of the Testing

Outputs (i.e. Results)In

pu

ts

Valid InvalidV

alid

Inva

lid

Good Goal(includes

exceptions)

BUG!!!

Shouldalways

produceexceptions!

7

Where Does Testing Fit?

Requirements Analysis Design Implementation Test

Client's viewof theProblem andone or moreSolutions

Description of the PROBLEM:"WHAT is the business domain;"WHAT" are the entities involved?

Logical description of the SOLUTION:"HOW will we solve the problem:"HOW will we meet the environment, etc.constraints?

The computablerepresenation of thelogical solution.

Activity of verifyingand validating thesolution from differentperspectives.

8

When to test ?

Requirements Analysis Design Implementation Test

Code

Unit /Component

Test

Code Code

Unit /Component

Test

Unit /Component

Test

9

What other Tests Are There?

Analysis Design Implementation Test

IntegrationTest

SystemTest

AcceptanceTest

Testing of combinationsof components.Tested from theproducer's perspective.

Testing of totality ofcomponents.Tested from theproducer'sperspective.

Testing of totality ofcomponents.Tested from theclient's perspective.

10

Test activities: Analysis

Test the Models against the Problem specification “Play computer” through the models

Analysis models must accurately describe the problem to be solved, and the boundaries of the problem domain

11

Test activities: Design

Test the Models against the Solution domain. “Play computer” through the models Incorporating all required “design” classes

GUI frameworks Third-party software and class libraries Collection/Container classes Operating environment classes (wrappers) Interface classes to external resources. E.g.,

RDBMS, network, communications.

12

Test Metrics

Traditional Metrics: No. of bugs per 10,000 Source Lines of Code IEEE Standard 982.1 McCabe Complexity Measure Halstead Software Science Measures

…

13

Test Techniques: The “Best of all Possible Worlds” technique:

Run it...and make sure it doesn’t crash, then ship it!

For the rest of us: Desk Checking Inspections &Walkthroughs

Black-box/White-box testing Boundary testing Path testing Code coverage Error message testing Top-down & Bottom-up …

14

Desk Checking

Simple, labor-intensive Procedure

Developer has his/her design or code manually reviewed by another developer

“a second pair of eyes” Developer must justify decisions to

reviewer. Developer reviews/accepts/rejects

reviewer’s recommendations.

15

Inspections & Walkthroughs

Formal or informal as needed. Well-known technique. Procedure:

Group gather to review artifacts (docs to code). Group composed of “role” representatives (e.g.

“user”, “QA”, “Standards”, etc.). Rules of conduct apply to participants Responsibility for verification and validation of

artifact is removed from the developer.

16

Inspections

Author narrates, statement by statement, the logic of his program.

Reviewers listen to the author, raise questions and ask for clarification in their attempt to uncover errors.

Ironically, most of the errors are discovered by the author as he/she “teaches” the program to the group.

17

Walkthroughs Programmer does not narrate to the group. One of the group plays the role of “tester”. Tester comes to review armed with a small

set of simple test cases to apply to the program.

Group “play computer” by performing manual simulation of the system using the test data.

Data are important as a vehicle to stimulate discussion.

Most errors are uncovered by questioning of the author rather than “execution” of data.

18

Inspections & Walkthroughs When should I perform one?

Anytime, so long as the product under review is complete (i.e. tangible, understandable, and objective).

After each milestone with delivered artifacts (e.g. Requirements Spec., Analysis Models, Design Spec., pseudo-code or flowcharts prepared, program code written, etc.).

Yourdon’s milestones [YOUR79]: after the design artifacts are completed after the code is prepared but not compiled after the first compilation after the first “clean” compilation after the first test data set have been executed successfully after the programmer thinks all test cases have been executed

successfully

19

Inspections & Walkthroughs

Who should participate? Author/developer/presenter/producer Moderator/coordinator Scribe/secretary Standards Bearer Maintenance Expert User Representative GUI Expert

20

White-Box Testing

White-box testing is the testing of the underlying implementation of a piece of software (e.g., source code) without regard to the specification (external description) for that piece of software.

The goal of white-box testing of source code is to identify such items as (unintentional) infinite loops, paths through the code which should be

allowed, but which cannot be executed, and dead (unreachable) code.

21

Black-Box Testing

Black-box testing is the testing of a piece of software without regard to its underlying implementation.

Specifically, it dictates that test cases for a piece of software are to be generated based solely on an examination of the specification (external description) for that piece of software.

The goal of black-box testing is to demonstrate that the software being tested does not adhere to its external specification.

(Note that if there is no "external specification" it will be difficult to conduct black-box testing.)

22

Test Categories

Black Box Tests Boundary-value Analysis Equivalence Partitioning Cause-Effect Graphing Error Guessing Error Message Generation

from [MYER79]

White Box Tests Statement Coverage Decision/Condition

Coverage Multiple-Condition

Coverage

23

Black Box Tests Boundary-value Analysis

Uses test cases “generated on, and immediately around, the boundaries of the input an output for a given piece of software.”

Equivalence Partitioning Divide test sets into equivalence partitions: “collections of

items which can all be regarded as identical at a given level of abstraction. I.e., a set of data items which will all evoke the same general behavior from a given software module.”

Cause-Effect Graphing Generates test sets based on input combinations, specified

by a combinatorial logic network or Boolean relations, and the expected outputs from those input combinations.

24

Black Box Tests

Error Guessing An intuitive and empirical technique of

selecting test cases which elicit failures. A.k.a. “a gift”.

Error Message Generation Selecting test cases which will elicit the

error messages defined in the software under test.

25

White Box Tests

Statement Coverage a.k.a. “basis path testing” Use test cases which cause each executable

statement to be executed at least once.

Decision/Condition Coverage All statements are executed at least once, and

all binary decisions have a TRUE and FALSE outcome at least once, all exceptions are raised at least once, and all possible interrupts are forced to occur at least once.

26

White Box Tests

Multiple-Condition Coverage Use test cases such that all possible

combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.

27

Path Testing

tests all possible paths

the flowchart on the right has approximately 100 trillion possible paths

at 1ms/test, working 24 hrs/day, 365 days/year: it will take 3170 years

Loop

<=

20 ti

mes