Software Testing

• Testing types

• Testing strategy

• Testing principles

Testing Objectives

• Testing is a process of executing a program with the intent of finding an error

• A good test case is one that has a high probability of finding an as-yet undiscovered error

• A successful test is one that uncovers an as-yet undiscovered error

Reminders

• Testing is a process for finding semantic (logical) errors, but not syntactic (symbolical) errors, in implemented programs.

• Testing can reveal the presence of errors, NOT their absence.

• Can you tell the differences between testing, debugging and compilation debugging?

Testing Techniques

• White-box testing (WBT)– When knowing the internal workings of a

product– Some techniques

• Basis path testing

• Condition testing

• Data flow testing

Testing Techniques

• Black-box testing (BBT)– When knowing the specified function that a

product has been designed to perform

• Some techniques– Equivalence partitioning (Input domain testing)– Boundary value analysis– Comparison testing (back-to-back testing)

Basis Path Testing (WBT)

• Steps– Construct a flow chart based on a program– Compute (cyclomatic) complexity of the flow

chart, which also indicates the number of independent paths of the program

– Determine basis paths– Design test cases to go through these basis

paths

Cyclomatic Complexity

• The complexity of a program

• Three ways for calculating the complexity– The number of regions of the flow graph– (total edges) – (total nodes) + 2– Number of predicate edges + 1

Cyclomatic Complexity1

2

3

4

5 6

97 8

10

• 4 basis paths– 1-2-3-4-5-7-10– 1-2-3-4-5-8-10– 1-2-3-4-6-9-10– 1-2-10

(May have more than one option for definingbasis paths)

Condition Testing (WBT)

• Focus on testing each condition in the program

• Branch testing may be the simplest method for doing condition testing

• Steps– Define combinations of truth values for

variables in predicate statements– Design test cases to test all these combinations

Data Flow Testing (WBT)

• Select test paths of a program according to the locations of definitions and uses of variables in the program, then test these define-use chains.

• Steps– Determine define-use chains for each variables– Design test cases to pass through these define-

use chains (may have different testing criteria, ex. all-defines, all-uses, …)

Equivalence Testing (BBT)

• Divide input domain of a program into classes of data from which test cases can be derived

• Ex. If an input condition specifies a range or a value, one valid and two invalid equivalence classes are defined.

• Ex. Boolean: one valid and one invalid classes can be defined.

Boundary Value Analysis (BBT)

• Lead to a selection of test cases that exercise boundary values of the input domain

• Ex. A range v1 and v2, test cases should be designed with values v1 and v2, just above and just below v1 and v2, respectively.

Comparison Testing (BBT)

• Appropriate when multiple implementations of same specification have been produced

• Use same inputs to test whether they generate same outputs

• Verification: "Are we building the product right"

• The software should conform to its specification

• Validation: "Are we building the right product"

• The software should do what the user really requires

Verification & Validation (V&V)

• Is a whole life-cycle process - V & V must be applied at each stage in the software process.

• Has two principal objectives– The discovery of defects in a system– The assessment of whether or not the system is

usable in an operational situation.

The V & V Process

• Software inspections Concerned with analysis of the static system representation to discover problems (static verification)– May be supplement by tool-based document and code

analysis

• Software testing Concerned with exercising and observing product behaviour (dynamic verification)– The system is executed with test data and its operational

behaviour is observed

Static and Dynamic Verification

Inspections and Testing

• Inspections and testing are complementary and not opposing verification techniques

• Both should be used during the V & V process• Inspections can check conformance with a

specification but not conformance with the customer’s real requirements

• Inspections cannot check non-functional characteristics such as performance, usability, etc.

Software Inspections

• Involve people examining the source representation with the aim of discovering anomalies and defects

• Do not require execution of a system so may be used before implementation

• May be applied to any representation of the system (requirements, design, test data, etc.)

• Very effective technique for discovering errors

Program Inspections

• Formalised approach to document reviews

• Intended explicitly for defect DETECTION (not correction)

• Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialised variable) or non-compliance with standards

Inspection Pre-conditions• A precise specification must be available• Team members must be familiar with the

organisation standards• Syntactically correct code must be available• An error checklist should be prepared• Management must accept that inspection will

increase costs early in the software process• Management must not use inspections for staff

appraisal

Inspection Procedure• System overview presented to inspection team• Code and associated documents are

distributed to inspection team in advance• Inspection takes place and discovered errors

are noted• Modifications are made to repair discovered

errors• Re-inspection may or may not be required

Testing Principles

• All tests should be traceable to customers• Tests should be planned long begin testing begins• The Pareto principle applies to software testing• Testing should begin “in the small” and progress

toward to testing “in the large”• Exhaustive testing is impossible• To be more effective, testing should be conduced

by an independent third party

Software Testing Strategy

• Unit test -> Integration test -> Validation Test -> System Test

SystemEngineering

Requirements Design Code

Unit TestIntegration TestValidation TestSystem Test

Time

Unit Test

• Use white-box testing techniques• Test for each module, including interface, local

data structures, boundary conditions, independent paths, error handling paths, and so on.

Integration Test

• Test the correctness of integrations of modules.• Usually use an incremental integration strategy to

integrate and test modules.• Integration strategies

– Top-down integration

– Bottom-up integration

– Combination of both methods

Top-down testing

Level 2Level 2Level 2Level 2

Level 1 Level 1Testing

sequence

Level 2stubs

Level 3stubs

. . .

Bottom-up testing

Level NLevel NLevel NLevel NLevel N

Level N–1 Level N–1Level N–1

Testingsequence

Testdrivers

Testdrivers

Validation Testing

• Test whether the software functions in a manner that can be reasonably expected by the customer.

• The reasonable expectations are defined in the software requirements.

• Methods– Alpha testing-developer observes user’s operation and

records all errors from the operation– Beta testing-customer operates system in real

environment without developer, but reports all errors to developer after his operation

System Testing

• Focus on not only software, but also the integration of software, hardware and environment

• Different testing– Recovery testing– Security testing– Stress testing– Performance testing

Debugging Approaches

• Brute force-insert “write” statement to indicate the position of the system when an error occurs, and find the causes of the error before the “write” statement.

• Backtracking-begin at the site where a symptom has been uncovered, and trace backward in the source until the causes have been located.

• Cause elimination-list all the hypotheses about the causes of an error, check and eliminate these hypotheses one by one, until the real cause has been found.

PS.

• Real-time systems/applications are harder to be tested than other systems-time/temporal considerations (hard real-time systems/soft real-time systems)

• For testing object-oriented programs, please refer to Dr. Kung’s web page.