CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.
-
Upload
esmond-chandler -
Category
Documents
-
view
226 -
download
5
Transcript of CS4723 Software Validation and Quality Assurance Lecture 02 Overview of Software Testing.
CS4723Software
Validation and Quality Assurance
Lecture 02Overview of Software Testing
2
Approach to remove bugs
Testing Feed input to software and run it to see whether its
behavior is as expected
Limitations Impossible to cover all cases
Test oracles (what is expected)
Static checking Identify specific problems (e.g., memory leak) in the
software by scanning the code or all possible paths
Limitations Limited problem types
False positives
3
Approach to remove bugs
Formal Proof Formally prove that the program implements the
specification
Limitations Difficult to have a formal specification The proof cost a lot of human efforts
Inspection Manually review the code to detect faults
Limitations: Hard to evaluate Sometime hard to get progress
4
Answer is testing, why?
“50% of my employees are testers, and the rest spends 50% of their time testing”
---- Bill Gates, 1995
More reliable than inspection, relatively cheap Actually in the old days, when testing is expensive,
inspection was the major answer
You get what you pay (linear rewards) Compared to other 3 approaches
Inspection, static checking, formal proof
5
Testing: Concepts
Test case
Test oracle
Test suite
Test script
Test driver
Test coverage
6
Testing: Concepts
Test case An execution of the software with a given list of input
values
Include: Input values, sometimes fed in different steps Expected outputs
Test oracle The expected outputs of software by feeding in a list
of input values
A part of test cases
Hardest problem in auto-testing: test oracle problem
7
Testing: Concepts: Example
8
Testing: Concepts
Test suite A collection of test cases
Usually these test cases share similar pre-conditions and configuration
Usually can be run together in sequence
Different test suites for different purposes Smoke test, Certain platforms, Certain feature,
performance, …
Test Script A script to run a sequence of test cases or a test suite
automatically
9
Testing: Concepts
Test Driver A software framework that can load a collection of test
cases or a test suite It can usually handle the configuration and comparison
between expected outputs and actual outputs
Test Coverage A measurement to evaluate how well the testing is done The measure can be based on multiple elements
Code Input combinations Specifications
10
Granularity of Testing: V-model
11
Granularity of testing
Unit / Module Testing Test of a single module
Integration Testing Test the interaction between modules
System Testing Test the system as a whole, by developers on test
cases
Acceptance Testing Validate the system against user requirements, by
customers with no formal test cases
12
Stage of software Testing
Development-time testing Unit testing, Integration Testing
Before-release testing System testing, Acceptance Testing
User testing Actual usage -> field bugs & patches
13
Types of testing by how they are designed
White box testing The tester knows everything about the
implementation They knows where the bugs are more probably be They can exercise paths in the code
Black box testing The tester are just like normal users They just try to cover input space and corner cases
14
Black Box Testing: General Guidelines
Divide value range and cover each part Cover boundary values
Try to reach all error messages
Try to trigger potential exceptions
Feed invalid inputs
wrong formats, too long, too short, empty, … Try combinations of all above
Repeat same and use different inputs for many times if the input is a sequence
15
Black Box Testing Techniques
Boundary value testing Boundary value analysis
Robustness testing
Worst case testing
Equivalence class testing
Decision table based testing
16
Boundary Value Analysis
Errors tend to occur near the extreme values of an input variables
Boundary value analysis focuses on the boundary of the input space to identity test cases
Boundary value analysis selects input variable values at their
Minimum
Just above the minimum
A nominal value
Just below the maximum
Maximum
17
Example of Boundary Value Analysis
Assume a program accepting two inputs y1 and y2, such that a < y1< b and c < y2 < d
a b
c
d
y2
y1
.. . .. . .
. .
18
Single Fault Assumption for Boundary Value Analysis
Boundary value analysis is also augmented by
the single fault assumption principle
“Failures occur rarely as the result of the
simultaneous occurrence of two (or more) faults”
In this respect, boundary value analysis test
cases can be obtained by holding the values of
all but one variable at their nominal values, and
letting that variable assume its extreme values
19
Generalization of Boundary Value Analysis
The basic boundary value analysis can be
generalized in two ways:
By the number of variables - (4n +1) test cases
for n variables
By the kinds of ranges of variables : map to
order Strings
Sequences
Complex Data Structures, e. g., trees
20
Application Scenario of Boundary Value Analysis
Several independent variables that represent
bounded physical quantities
No consideration of the function of the
program, nor of the semantic meaning of the
variables
Good news: We can distinguish between
physical and other types of variables
21
Robustness Testing
A simple extension of boundary value analysis
In addition to the five boundary value analysis values of variables, we add values slightly greater that the maximum (max+) and a value slightly less than the minimum (min-)
The main value of robustness testing is to force attention on exception handling
UTSA CS377322
Example of Robustness Testing
a b
c
d
y2
y1
... . … . .
. . .
.
23
Worst Case Testing
No single fault assumption: error happens when more than one variable has an extreme value
Considering n inputs in boundary analysis, we take the Cartesian product of the five values for 1, 2, 3, … n variables
We can have 5n test cases for n input variables The more interactions on inputs -> more on worse
case testing Input partitions: Length & Width vs. Length & price
24
Example of Worst Case Testing
a b
c
d
y2
y1
.. . .... . .... . ..
.. . .... . ..
25
Equivalence Class Testing
Divide the value range of an input to a number of subsets
Subsets are disjoint
The union of the subset if the value range
Values in one subset does not make difference for the software concerned
Water temp in a car: <32, 32 – 212, >=212
Normal colors vs. Metallic colors
26
Example of Equivalence Class Testing
27
Equivalence Class Testing
The use of equivalence class testing has two motivations:
Sense of complete testing The entire set is represented provides a form of
completeness Avoid redundancy
The disjointness assures a form of non-redundancy
Note Also check boundaries Combinations of inputs also follows the rule:
more interaction -> more combinations
28
Equivalent Class for non-numeric inputs
Feature extraction For string and structure inputs Split the possible value set with a certain feature Example:
String passwd => {contains space}, {no space} It is possible to extract multiple features from one input Example:
String name => {capitalized first letter}, {not}
=> {contains space}, {not}
=> {length >10}, {2-10}, {1}, {0}
One test case may cover multiple features
29
Decision Table
Make it easy to observe that all possible
conditions are accounted for
Decision tables can be used for:
Specifying complex program logic
Generating test cases with oracles
30
Example of Decision Table
Conditions
Printer does not print Y Y Y Y N N N N
A red light is flashing Y Y N N Y Y N N
Printer is unrecognized Y N Y N Y N Y N
Actions
Check the power cable X
Check the printer-computer cable X X
Ensure printer software is installed X X X X
Check/replace ink X X X X
Check for paper jam X X
Printer Troubleshooting
31
Decision Table Usage
The use of the decision-table model is
applicable when:
Specification is given or can be converted to a
decision table
The order in which the predicates are
evaluated does not affect the interpretation of
resulting action
Note:
Decision table needs not cover all
combinations
32
White Box Testing: General Guidelines
Try to cover all branches Study the relationship between input value
and branch logic
Test more on complex modules Measure complexities of modules by code
size, number of branches and loops, number of calls and recursions
33
White Box Testing: Techniques
More difficult than black box testing
Seldom done manually
Automatic support Symbolic execution
Complexity measurement and Defect prediction
34
Review: Test overview
Test is the practical choice: the best affordable approach
Concepts: test case, test oracle, test suite, test driver, test script, test coverage
Granularity: unit, integration, system, acceptance
Type by design principle: black-box, white-box
Black-box-testing: boundary, equivalence, decision table
White-box-testing: branch coverage, complexity