CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

34
CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing

Transcript of CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

Page 1: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

CS4723Software

Validation and Quality Assurance

Lecture 02Overview of Software Testing

Page 2: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

2

Approach to remove bugs

Testing Feed input to software and run it to see whether its

behavior is as expected

Limitations Impossible to cover all cases

Test oracles (what is expected)

Static checking Identify specific problems (e.g., memory leak) in the

software by scanning the code or all possible paths

Limitations Limited problem types

False positives

Page 3: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

3

Approach to remove bugs

Formal Proof Formally prove that the program implements the

specification

Limitations Difficult to have a formal specification The proof cost a lot of human efforts

Inspection Manually review the code to detect faults

Limitations: Hard to evaluate Sometime hard to get progress

Page 4: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

4

Answer is testing, why?

“50% of my employees are testers, and the rest spends 50% of their time testing”

---- Bill Gates, 1995

More reliable than inspection, relatively cheap Actually in the old days, when testing is expensive,

inspection was the major answer

You get what you pay (linear rewards) Compared to other 3 approaches

Inspection, static checking, formal proof

Page 5: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

5

Testing: Concepts

Test case

Test oracle

Test suite

Test script

Test driver

Test coverage

Page 6: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

6

Testing: Concepts

Test case An execution of the software with a given list of input

values

Include: Input values, sometimes fed in different steps Expected outputs

Test oracle The expected outputs of software by feeding in a list

of input values

A part of test cases

Hardest problem in auto-testing: test oracle problem

Page 7: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

7

Testing: Concepts: Example

Page 8: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

8

Testing: Concepts

Test suite A collection of test cases

Usually these test cases share similar pre-conditions and configuration

Usually can be run together in sequence

Different test suites for different purposes Smoke test, Certain platforms, Certain feature,

performance, …

Test Script A script to run a sequence of test cases or a test suite

automatically

Page 9: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

9

Testing: Concepts

Test Driver A software framework that can load a collection of test

cases or a test suite It can usually handle the configuration and comparison

between expected outputs and actual outputs

Test Coverage A measurement to evaluate how well the testing is done The measure can be based on multiple elements

Code Input combinations Specifications

Page 10: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

10

Granularity of Testing: V-model

Page 11: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

11

Granularity of testing

Unit / Module Testing Test of a single module

Integration Testing Test the interaction between modules

System Testing Test the system as a whole, by developers on test

cases

Acceptance Testing Validate the system against user requirements, by

customers with no formal test cases

Page 12: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

12

Stage of software Testing

Development-time testing Unit testing, Integration Testing

Before-release testing System testing, Acceptance Testing

User testing Actual usage -> field bugs & patches

Page 13: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

13

Types of testing by how they are designed

White box testing The tester knows everything about the

implementation They knows where the bugs are more probably be They can exercise paths in the code

Black box testing The tester are just like normal users They just try to cover input space and corner cases

Page 14: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

14

Black Box Testing: General Guidelines

Divide value range and cover each part Cover boundary values

Try to reach all error messages

Try to trigger potential exceptions

Feed invalid inputs

wrong formats, too long, too short, empty, … Try combinations of all above

Repeat same and use different inputs for many times if the input is a sequence

Page 15: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

15

Black Box Testing Techniques

Boundary value testing Boundary value analysis

Robustness testing

Worst case testing

Equivalence class testing

Decision table based testing

Page 16: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

16

Boundary Value Analysis

Errors tend to occur near the extreme values of an input variables

Boundary value analysis focuses on the boundary of the input space to identity test cases

Boundary value analysis selects input variable values at their

Minimum

Just above the minimum

A nominal value

Just below the maximum

Maximum

Page 17: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

17

Example of Boundary Value Analysis

Assume a program accepting two inputs y1 and y2, such that a < y1< b and c < y2 < d

a b

c

d

y2

y1

.. . .. . .

. .

Page 18: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

18

Single Fault Assumption for Boundary Value Analysis

Boundary value analysis is also augmented by

the single fault assumption principle

“Failures occur rarely as the result of the

simultaneous occurrence of two (or more) faults”

In this respect, boundary value analysis test

cases can be obtained by holding the values of

all but one variable at their nominal values, and

letting that variable assume its extreme values

Page 19: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

19

Generalization of Boundary Value Analysis

The basic boundary value analysis can be

generalized in two ways:

By the number of variables - (4n +1) test cases

for n variables

By the kinds of ranges of variables : map to

order Strings

Sequences

Complex Data Structures, e. g., trees

Page 20: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

20

Application Scenario of Boundary Value Analysis

Several independent variables that represent

bounded physical quantities

No consideration of the function of the

program, nor of the semantic meaning of the

variables

Good news: We can distinguish between

physical and other types of variables

Page 21: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

21

Robustness Testing

A simple extension of boundary value analysis

In addition to the five boundary value analysis values of variables, we add values slightly greater that the maximum (max+) and a value slightly less than the minimum (min-)

The main value of robustness testing is to force attention on exception handling

Page 22: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

UTSA CS377322

Example of Robustness Testing

a b

c

d

y2

y1

... . … . .

. . .

.

Page 23: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

23

Worst Case Testing

No single fault assumption: error happens when more than one variable has an extreme value

Considering n inputs in boundary analysis, we take the Cartesian product of the five values for 1, 2, 3, … n variables

We can have 5n test cases for n input variables The more interactions on inputs -> more on worse

case testing Input partitions: Length & Width vs. Length & price

Page 24: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

24

Example of Worst Case Testing

a b

c

d

y2

y1

.. . .... . .... . ..

.. . .... . ..

Page 25: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

25

Equivalence Class Testing

Divide the value range of an input to a number of subsets

Subsets are disjoint

The union of the subset if the value range

Values in one subset does not make difference for the software concerned

Water temp in a car: <32, 32 – 212, >=212

Normal colors vs. Metallic colors

Page 26: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

26

Example of Equivalence Class Testing

Page 27: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

27

Equivalence Class Testing

The use of equivalence class testing has two motivations:

Sense of complete testing The entire set is represented provides a form of

completeness Avoid redundancy

The disjointness assures a form of non-redundancy

Note Also check boundaries Combinations of inputs also follows the rule:

more interaction -> more combinations

Page 28: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

28

Equivalent Class for non-numeric inputs

Feature extraction For string and structure inputs Split the possible value set with a certain feature Example:

String passwd => {contains space}, {no space} It is possible to extract multiple features from one input Example:

String name => {capitalized first letter}, {not}

=> {contains space}, {not}

=> {length >10}, {2-10}, {1}, {0}

One test case may cover multiple features

Page 29: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

29

Decision Table

Make it easy to observe that all possible

conditions are accounted for

Decision tables can be used for:

Specifying complex program logic

Generating test cases with oracles

Page 30: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

30

Example of Decision Table

Conditions

Printer does not print Y Y Y Y N N N N

A red light is flashing Y Y N N Y Y N N

Printer is unrecognized Y N Y N Y N Y N

Actions

Check the power cable X

Check the printer-computer cable X X

Ensure printer software is installed X X X X

Check/replace ink X X X X

Check for paper jam X X

Printer Troubleshooting

Page 31: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

31

Decision Table Usage

The use of the decision-table model is

applicable when:

Specification is given or can be converted to a

decision table

The order in which the predicates are

evaluated does not affect the interpretation of

resulting action

Note:

Decision table needs not cover all

combinations

Page 32: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

32

White Box Testing: General Guidelines

Try to cover all branches Study the relationship between input value

and branch logic

Test more on complex modules Measure complexities of modules by code

size, number of branches and loops, number of calls and recursions

Page 33: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

33

White Box Testing: Techniques

More difficult than black box testing

Seldom done manually

Automatic support Symbolic execution

Complexity measurement and Defect prediction

Page 34: CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.

34

Review: Test overview

Test is the practical choice: the best affordable approach

Concepts: test case, test oracle, test suite, test driver, test script, test coverage

Granularity: unit, integration, system, acceptance

Type by design principle: black-box, white-box

Black-box-testing: boundary, equivalence, decision table

White-box-testing: branch coverage, complexity