functional.ppt
-
Upload
softwarecentral -
Category
Documents
-
view
478 -
download
0
Transcript of functional.ppt
CS 406 Software Testing Fall 98Part II: Functional Testing
Aditya P. Mathur
Purdue University
Last update: July 19, 1998
Functional testing 2
Part II: Functional testing
Learning objectives-– What is functional testing?– How to perform functional testing?
• What are clues, test requirements, and test specifications?
• How to generate test inputs?
– What are equivalence partitioning, boundary value testing, domain testing, state testing, and decision table testing?
Functional testing 3
What is functional testing?
When test inputs are generated using program specifications, we say that we are doing functional testing.
Functional testing tests how well a program meets the functionality requirements.
Functional testing 4
The methodology
The derivation of test inputs is based on program specifications.
Clues are obtained from the specifications. Clues lead to test requirements. Test requirements lead to test specifications. Test specifications are then used to actually
execute the program under test.
Functional testing 5
Test methodology
Specifications
Clues
Test requirements
Test specifications
Test driver
Program
Oracle
Expected behavior
Actual behavior
Program output is correct
Program hasfailed; make a note and proceedwith testing orget into the debugmode.
or
Until specs.
Exhausted.
Functional testing 6
Specifications
Inputs and tasks:– Given inputs
– Perform tasks
0,,....,, 21 nIII n
0,,....,, 21 mTTT m
Functional testing 7
Specifications-continued
Input properties– Input
– must satisfy
Function f is a pre-condition on input
kI
),..,,..,( 21 nk IIIIf
kI
Functional testing 8
Specifications-continued
Two types of pre-conditions are considered:– Validated: those that are required to be
validated by the program under test and an error action is required to be performed if the condition is not true.
– Assumed: those that are assumed to be true and not checked by the program under test.
Functional testing 9
Specification: example
For the sort program:– Inputs are:
• N
• pointer to a sequence of length N
• pointer to an area in memory where the output sequence is to be placed.
Functional testing 10
Specification: example..continued
– Tasks to be performed:• Sort the sequence in ascending order
• Return the sorted sequence in an area provided.
• Return 1 if sorting is successful, -1 otherwise.
Functional testing 11
Preconditions for sort
Validated:– N>0– On failure return -1; sorting considered
unsuccessful.
Assumed:– The input sequence contains N integers.– The output area has space for at least N
integers.
Functional testing 12
Deriving pre-conditions
Pre-conditions result from properties of inputs.
Example: – alpha_sequence(name)
alpha_sequence is the string obtained from name by removing all characters other then A-Z, and a-z. Thus, if name is “A12C” then alpha_name is “AC”.
Functional testing 13
Deriving pre-conditions-continued
This leads to the following pre-condition:– Validated: the string alpha_sequence(name) is
shorter than name.– On failure: print “invalid name”.
This property could also lead to the pre-condition:– Assumed: the string alpha_ sequence(name) is
shorter than name.
Functional testing 14
Post-conditions
A post-condition specifies a property of the output of a program.
The general format of a post-condition is:– if condition then effect-1 {else effect-2}
Example:– For the sort program a post-condition is:
• if N>0 then {the output sequence has the same elements as in the input sequence and in ascending order.}
Functional testing 15
Post-condition-continued
– This could be stated more formally as:• if N>0 then
{
and each is a member of the input sequence and sort returns 1.
} else
{ the output sequence is undefined and sort returns -1.
}
NAAA ....21
Ni1Ai ,
Functional testing 16
Post-condition-continued
Another example:– if (A=B) and (B=C) then return “equilateral”;
Can you complete the above post-condition for a program that is required to classify a triangle given the length of three sides?
Convention: We will not nest if-then-else statements while specifying a post-condition.
Functional testing 17
Incompleteness of specifications
Specifications may be incomplete or ambiguous.
Example post-condition: if user places cursor on the name field then read a string
– This post-condition does not specify any limit on the length of the input string hence is incomplete.
Functional testing 18
Ambiguous specifications
It also does not make it clear as to– whether a string should be input only after the
user has placed the cursor on the name field and clicked the mouse or simply placed the cursor on the name field.
and hence is ambiguous.
Functional testing 19
Clues: summary
Clues are:– Pre-conditions– Post-conditions– Variables, e.g. A is a length implying thereby
that its value cannot be negative.– Operations, e.g. “search a list of names” or “find
the average of total scores”– Definitions, e.g. “filename(name) is a name is no
spaces.”
Functional testing 20
Clues-continued
Ideally variables, operations and definitions should be a part of at least one pre- or post-condition.
However, this may not be the case as specifications are not always written formally.
Hence look out for variables, operations, and definitions within a specification!
Functional testing 21
Test requirements
A test requirement is a description of how to test the program that is under test.
Here is a sample test requirement for a program that classifies a triangle given the length of three sides. – A, B, C are non-zero and positive.– One of A, B, C is negative; error condition.– One of A, B, C is zero; error condition.
Functional testing 22
Test requirements-derivation
Test requirements are derived from clues. For example, consider the following pre-
conditions (clues):• Assumed: A, B, and C are lengths
• Validated: A>0, B>0, C>0
These pre-conditions on A, B, and C lead to the test requirement given above.
Functional testing 23
Test requirements-derivation
Note that we have clumped pre-condition for each input variable into one condition. This is being done only for inconvenience.
It is recommended that pre-conditions be separated for each variable.
Functional testing 24
Test requirements-derivation
Note also that each validated pre-condition results in at least two requirements: one for the validated part and the other for the failure part.
In our example above we did not list all requirements. For example, we are content with testing “one of A, B, C is negative; error condition.”
Functional testing 25
Test requirements-derivation
Post-conditions also lead to test requirements.
For example, the partial post-condition:• if (A=B) and (B=C) then return “equilateral”
leads to the following test requirement:• A=B and B=C.
Functional testing 26
Compound validated pre-conditions
Compound pre-conditions are ones that use the and or or connectors.
Examples: validated compound pre-conditions:– Pre-condition: A and B– Pre-condition: user places the mouse over the
name field and clicks it.
Functional testing 27
Compound validated pre-conditions
The first of the above pre-conditions leads to four requirements:
• A true, B true (This is the validated part)
• A false, B true (This and the rest are failures)
• A true, B false
• A false, B false
You may work out the requirements for compound pre-condition with the or connector.
Functional testing 28
Compound validated pre-conditions
Compound validated pre-conditions could become quite complex.
Example: (A and (B or C)) Brute force method will lead to 8 test
requirements.
Functional testing 29
Compound validated pre-conditions
In general this will lead to too many test requirements.
We can prune them by leaving out those requirements that are unlikely to reveal a program error.
For example, consider the validated pre-condition: A or B.
Functional testing 30
Pruning test requirements
There are four possible test requirements:• A true, B true
• A false, B true
• A true, B false
• A false, B false
Consider a correct C implementation:if (!(A || B))
exit_with_error(“Error: A is %d, B is %d”, A, B);
else.. {/* the validated code comes here.*/}
Functional testing 31
Possible errors
Programmer forgets to check for one of the two cases resulting in the code:
• if (!A)
exit_with_error(“Error: A is %d, B is %d”, A, B);
or• if (!B)
exit_with_error(“Error: A is %d, B is %d”, A, B);
Functional testing 32
Possible errors-continued
Or use a wrong logical operator as in:if (!(A && B))
exit_with_error(“Error: A is %d, B is %d”, A, B);
Let us analyze how the four different tests will perform in each of the four implementations: one correct, and three incorrect ones.
Functional testing 33
Truth table: or condition
A B !(A || B) !(A&&B) !A !B
T F F T F T
F T F T T F
F F T T T T
T T F F F F
Inputs Correctimplementation
Incorrectimplementations
Notice this one: will it help find any of the three possible errors?
Functional testing 34
Truth table analysis
Case 1:– A test input with A=true and B=false will cause
the correct program to evaluate the condition to false.
– The two incorrect implementations, !(A&&B) and (!B) will evaluate the condition to true.
Functional testing 35
Truth table analysis-continued
– Both incorrect implementations will print the error message.
– The oracle will observe that the correct and the incorrect implementations behave differently.
– It will therefore announce failure for each incorrect implementation thereby pointing to an error.
End of Case 1.
Functional testing 36
Truth table analysis-continued
Case 2:– Test input A=false and B=true will reveal the
error in the two incorrect implementations, !(A&&B) and (!A).
Case 3:– Test input A=false and B=false might find a
fault in the then branch of the if condition.
Functional testing 37
Truth table analysis-continued
Case 4:– Test input A=true and B=true might find a fault
in the else branch of the if condition.
Thus, all four test inputs are likely to be useful.
Functional testing 38
Truth table analysis-continued
However, if we were to check for the correct implementation of the condition A or B, then only the first two inputs are necessary.
In this example, reducing the number of test specifications from 4 to 2 does not lead to any significant savings. When will the savings be significant?
Functional testing 39
Assumed pre-conditions
Each assumed pre-condition is likely to result in a test requirement.
Example:– Assumed: MODE is “on ground” or “flying”– This leads to two requirements:
• MODE is “on ground” , MODE is not “flying”
• MODE is not “on ground” , MODE is “flying”
Functional testing 40
Assumed pre-conditions
– These can be simplified to:• MODE is “on ground”
• MODE is “flying”
Functional testing 41
Test requirements checklist
Obtaining clues and deriving test requirements can become a tedious task.
To keep it from overwhelming us it is a good idea to make a checklist of clues.
This checklist is then transformed into a checklist of test requirements by going through each clue and deriving test requirements from it.
Functional testing 42
Test specifications
A test requirements indicates “how” to test a program. But it does not provide exact values of inputs.
A test requirement is used to derive test specification, which is the exact specification of values of input and environment variables.
Functional testing 43
Test specifications-continued
There may not be a one-to-one correspondence between test requirements and test specifications.
A test requirement checklist might contain 50 entries. These might result in only 22 test specifications.
The fewer the tests the better but only if these tests are of good quality!
Functional testing 44
Test specifications-continued
We will discuss test quality when discussing test assessment.
A test specification looks like this:– Test 2:
• global variable all_files is initially false.
• next_record is set to 1.
– Upon return expect:• all_files to be true
• next_record is last_record+1
Functional testing 45
Test specifications-continued
Notice the format of a test specification:– Each test is given a number which serves as its
identifier.
– There is a set of input values.
– There is a set of expected values upon return from execution. Any side effects on files or networks must also be specified here. In essence, all observable effects must be specified in the “Expect” part of a test specification.
Functional testing 46
Test specifications-continued
– Any side effects on files or networks must also be specified. In essence, all observable effects must be specified in the “Expect” part of a test specification.
– Similarly, values of all input variables, global or otherwise, must also be specified.
Functional testing 47
Test requirements to specifications
The test requirements checklist guides the process of deriving test specifications.
Initially all entries in the checklist are unmarked or set to 0.
Each time a test is generated from a requirement it is marked or the count incremented by 1.
Functional testing 48
Test requirements to specifications
Thus, at any point in time, one could assess the progress made towards the generation of test specifications.
One could also determine how many tests have been generated using any test requirement.
Functional testing 49
Test requirements to specifications
Once a test requirement has been marked or its count is more than 0 we say that it has been satisfied.
Some rules of thumb to use while designing tests:– Try to satisfy multiple requirements using only
one test.– Satisfy all test requirements.
Functional testing 50
Test requirements to specifications
– Avoid reuse of same values of a variable in different tests. Generating new tests by varying an existing one is likely to lead to tests that test the same part of the code as the previous one.
In testing, variety helps!
Though we try to combine several test requirements to generate one test case, this is not advisable when considering error conditions.
Functional testing 51
Test requirements to specifications
For example, consider the following:– speed_dial, an interval
• speed_dial<0 ,error
• speed_dial>120, error
– zones, an interval• zones <5, error
• zones>10, error
Functional testing 52
Test requirements to specifications
– One test specification obtained by combining the two requirements above is:
• speed_dial=-1
• zone=3
Now, if the code to handle these error conditions is:
Functional testing 53
Test requirements to specifications
if (speed_dial<0 || speed_dial>120)error_exit(“Incorrect speed_dial”);
if (zone<6 ||zone>10)error_exit(“Incorrect zone”);
– For our test, the program will exit before it reaches the second if statement. Thus, it will miss detecting the error in coding the test for zone.
error
Functional testing 54
Test requirements to specifications
Also, do not assume an error test to satisfy any other test requirement.
Example:– Consider the function:
• myfunction(int X, int Y);
– A test for the erroneous value of X might not test the code that uses Y.
Functional testing 55
Test requirements to specifications
Test specifications must not mention internal variables. Remember, a test specification aids in setting input variables to suitable values before the test begins. Values of internal variables are computed during program execution.
However, there are exceptions to the above rule. Can you think of one?
Functional testing 56
Equivalence partitioning
The input domain is usually too large for exhaustive testing.
It is therefore partitioned into a finite number of sub-domains for the selection of test inputs.
Each sub-domain is known as an equivalence class and serves as a source of at least one test input.
Functional testing 57
Equivalence partitioning
12
3
4
Input domain Input domain partitioned into four sub-domains.
Too manytest inputs.
Four test inputs, one selected from each sub-domain.
Functional testing 58
How to partition?
Inputs to a program provide clues to partitioning.
Example 1:– Suppose that program P takes an input X, X
being an integer.– For X<0 the program is required to perform
task T1 and for X>=0 task T2.
Functional testing 59
How to partition?-continued
– The input domain is prohibitively large because X can assume a large number of values.
– However, we expect P to behave the same way for all X<0.
– Similarly, we expect P to perform the same way for all values of X>=0.
– We therefore partition the input domain of P into two sub-domains.
Functional testing 60
Two sub-domains
X<0 X>=0
One test case:X=-3
Another test case:X=-15
All test inputs in the X<0 sub-domain are considered equivalent.The assumption is that if one test input in this sub-domain revealsan error in the program, so will the others.
This is true of the test inputs in the X>=0 sub-domain also.
Equivalence class
Equivalence class
Functional testing 61
Non-overlapping partitions
In the previous example, the two equivalence classes are non-overlapping. In other words the two sub-domains are disjoint.
When the sub-domains are disjoint, it is sufficient to pick one test input from each equivalence class to test the program.
Functional testing 62
Non-overlapping partitions
An equivalence class is considered covered when at least one test has been selected from it.
In partition testing our goal is to cover all equivalence classes.
Functional testing 63
Overlapping partitions
Example 2:– Suppose that program P takes three integers X,
Y and Z. It is known that:• X<Y
• Z>Y
Functional testing 64
Overlapping partitions
X<Y
X>=Y
Z>Y Z<=Y
X<Y, Z>YX=3, Y=4, Z=7
X<Y, Z<=YX=2, Y=3, Z=1
X>=Y, Z<=YX=15, Y=4, Z=1
X>=Y, Z>YX=15, Y=4, Z=7
Functional testing 65
Overlapping partition-test selection
In this example, we could select 4 test cases as:– X=4, Y=7, Z=1 satisfies X<Y– X=4, Y=2, Z=1 satisfies X>=Y– X=1, Y=7, Z=9 satisfies Z>Y– X=1, Y=7, Z=2 satisfies Z<=Y
Thus, we have one test case from each equivalence class.
Functional testing 66
Overlapping partition-test selection
However, we may also select only 2 test inputs and satisfy all four equivalence classes:– X=4, Y=7, Z=1 satisfies X<Y and Z<=Y– X=4, Y=2, Z=3 satisfies X>=Y and Z>Y
Thus, we have reduced the number of test cases from 4 to 2 while covering each equivalence class.
Functional testing 67
Partitioning using non-numeric data
In the previous two examples the inputs were integers. One can derive equivalence classes for other types of data also.
Example 3:– Suppose that program P takes one character X
and one string Y as inputs. P performs task T1 for all lower case characters and T2 for upper case characters. Also, it performs task T3 for the null string and T4 for all other strings.
Functional testing 68
Partitioning using non-numeric data
X: LC
X:UC
Y: null Y: not nullX: LC, Y: null
X: LC, Y: not null
X: UC, Y: not null
X: UC, Y: null LC: Lower case characterUC: Upper case characternull: null string.
Functional testing 69
Non-numeric data
Once again we have overlapping partitions. We can select only 2 test inputs to cover all
four equivalence classes. These are:– X: lower case, Y: null string– X: upper case, Y: not a null string
Functional testing 70
Guidelines for equivalence partitioning
Input condition specifies a range: create one for the valid case and two for the invalid cases.– e.g. for a<=X<=b the classes are
• a<=X<=b (valid case)
• X<a and X>b (the invalid cases)
Functional testing 71
Guidelines-continued
Input condition specifies a value: create one for the valid value and two for incorrect values (below and above the valid value). This may not be possible for certain data types, e.g. for boolean.
Input condition specifies a member of a set: create one for the valid value and one for the invalid (not in the set) value.
Functional testing 72
Sufficiency of partitions
In the previous examples we derived equivalence classes based on the conditions satisfied by input data.
Then we selected just enough tests to cover each partition.
Think of the advantages and disadvantages of this approach!
Functional testing 73
Boundary value analysis (BVA)
Another way to generate test cases is to look for boundary values.
Suppose a program takes an integer X as input.
In the absence of any information, we assume that X=0 is a boundary. Inputs to the program might lie on the boundary or on either side of the boundary.
Functional testing 74
BVA: continued
This gives us 3 test inputs:• X=0, X=-20, and X=14.
– Note that the values -20 and 14 are on either side of the boundary and are chosen arbitrarily.
Notice that using BVA we get 3 equivalence classes. One of these three classes contains only one value (X=0), the other two are large!
Functional testing 75
BVA: continued
Now suppose that a program takes two integers X and Y and that x1<=X<=x2 and y1<=Y<=y2.
x1 x2
y2
y1
1 2
34
5
6
7
8 9
1011
1213
14
Functional testing 76
BVA-continued
In this case the four sides of the rectangle represent the boundary.
The heuristic for test selection in this case is:– Select one test at each corner (1, 2, 3, 4).– Select one test just outside of each of the four
sides of the boundary (5, 6, 7, 8)
Functional testing 77
BVA-continued
– Select one test just inside of each of the four sides of the boundary (10, 11, 12, 13).
– Select one test case inside of the bounded region (9).
– Select one test case outside of the bounded region (14).
How many equivalence classes do we get?
Functional testing 78
BVA -continued
In the previous examples we considered only numeric data.
BVA can be done on any type of data. For example, suppose that a program takes a
string S and an integer X as inputs. The constraints on inputs are:– length(S)<=100 and a<=X<=b
Can you derive the test cases using BVA?
Functional testing 79
BVA applied to output variables
Just as we applied BVA to input data, we can apply it to output data.
Doing so gives us equivalence classes for the output domain.
We then try to find test inputs that will cover each output equivalence class.
Functional testing 80
BVA-continued
Example: each student to construct one!
Functional testing 81
Finite State Machines (FSMs)
A state machine is an abstract representation of actions taken by a program or anything else that functions!
It is specified as a quintuple:• A: a finite input alphabet
• Q: a finite set of states
• q0: initial state which is a member of Q.
Functional testing 82
FSMs-continued
• T: state transitions which is a mapping
Q x A--> Q
• F: A finite set of final states, F is a subset of Q.
– Example: Here is a finite state machine that recognizes integers ending with a carriage return character.
• A={0,1,2,3,4,5,6,7,8,9, CR}
• Q={q0,q1,q2}
• q0: initial state
Functional testing 83
FSMs-continued
• T: {((q0,d),q1),(q1,d),q1), (q1,CR),q2)}
• F: {q2}
A state diagram is an easier to understand specification of a state machine. For the above machine, the state diagram appears on the next page.
Functional testing 84
State diagram
q0 q1d
d
CRq2
Final state indicatedby concentric circles.
States indicated by circles.
State transitions indicatedby labeled arrows from one statethe another (which could be the same). Each label must be from the alphabet. It is also known asan event.
d: denotes a digit
Functional testing 85
State diagram-actions
q0 q1 q2d/set i to d
d/add 10*d to i
CR/output i
i is initialized to d when the machine moves from state q0 to q1.i is incremented by 10*d when the machine moves from q1 to q1.The current value of i is output when a CR is encountered.
Can you describe what this machine computes?Can you construct a regular expression that describes all strings recognized by this state machine?
x/y: x is an element ofthe alphabet and y is an action.
Functional testing 86
State machine: languages
Each state machine recognizes a language. The language recognized by a state machine
is the set S of all strings such that:– when any string s in S is input to the state
machine the machine goes through a sequence of transitions and ends up in the final state after having scanned all elements of s.
Functional testing 87
State diagram-errors
q0 q1 q2d/set I to d
d/add 10*d to I
CR/output I
q4 has been added to the set of states. It represents an error state. Notice that reset is a new member added to the alphabet.
CR/output error q4reset
Functional testing 88
State diagram-program
A state diagram can be transformed into a program using case analysis. Here is a C program fragment that embodies logic represented by the previous state diagram.
There is one function for each action. digit is assumed to be provided by the
lexical analyzer.
Functional testing 89
Program for “integer” state machine
case q0:
i=digit; /* perform action. */
state=q1; /* set next state. */
break; /* event digit is done. */
case q1:
i=i+10*digit; /* Add the next digit. */
state=q1;
break;/*…complete the program. */
/* state is global, with values q0, q1, q2. i is also global.*/
switch (state)
void event_digit()
{
Functional testing 90
Checking state diagrams
Unreachable state: One that cannot be reached from q0 using any sequence of transitions.
Dead state: One that cannot be left once it is reached.
Functional testing 91
Test requirements
Every state must be reached at least once, Obtain 100% state coverage.
Every transition must be exercised at least once.Obtain 100% transition coverage.
The textbook talks about duplicate transitions. No transitions are duplicate if the state machine definition we have given is used.
Functional testing 92
Example test requirements
For the “integer” state machine: – state machine transitions:
• event digit in state q0
• event CR in state q0
• event digit in state q1
• event CR in state q1
• event reset in state q4
Functional testing 93
More testing of state machines?
Yes, it is possible! When we learn about path coverage we will
discuss how more test requirements can be derived from a state diagram.
Functional testing 94
Test specifications
As before, test specifications are derived from test requirements.
In the absence of dead states, all states and transitions can be reached by one test.
It is advisable not to test the entire machine with one test case.
Develop test specifications for our “integer” state machine.
Functional testing 95
Decision tables
Requirements of certain programs are specified by decision tables.
Such tables can be used for deriving test requirements and specifications.
A decision table is useful when specifying complex decision logic
Functional testing 96
Decision tables
A decision table has two parts: – condition part– action part
The two together specify under what condition will an action be performed.
Functional testing 97
Decision table-nomenclature
• C: denotes a condition
• A: denotes an action
• Y: denotes true
• N:denotes false
• X: denotes action to be taken.
• Blank in condition: denotes “don’t care”
• Blank in action: denotes “do not take the action”
Functional testing 98
Bank example
Consider a bank software responsible for debiting from an account. The relevant conditions and actions are:
• C1: The account number is correct
• C2: The signature matches
• C3: There is enough money in the account
• A1: Give money
• A2: Give statement indicating insufficient funds
• A3: Call vigilance to check for fraud!
Functional testing 99
Decision tables
1 2 3 4 5C1 N N Y Y YC2 N N Y YC3 N Y NA1 XA2 X XA3 X
Functional testing 100
Example-continued
A1 is to be performed when C1, C2, and C3 are true.
A2 is to be performed when C1 is true and C2 and C3 are false or when C1 and C2 are true and C3 is false.
A3 is to be performed when C2 and C3 are false.
Functional testing 101
Default rules
Are all possible combinations of conditions covered?
No! Which ones are not covered? We need a default action for the uncovered
combinations. A default action could be an error report or a reset.
Functional testing 102
Example-test requirements
Each column is a rule and corresponds to at least one test requirement.
If there are n columns then there are at least n test requirements.
What is the maximum number of test requirements?
Functional testing 103
Example-test specifications
For each test requirement find a set of input values of variables such that the selected rule is satisfied.
When this test is input to the program the output must correspond to the action specified in the decision table.
Should the testing depend on the order in which the conditions are evaluated?
Functional testing 104
Summary
Specifications, pre-conditions, and post-conditions.
Clues, test requirements, and test specifications.
Clues from code. Test requirements catalog. Equivalence partitioning and boundary value
analysis.
Functional testing 105
Summary-continued
Finite state machine State diagram Events and actions Unreachable and dead states Test requirements and specifications for
state machines Decision tables, rules, actions