January 27, 2002 ECEN5033 University of Colorado -- Class Testing 1 Adequacy of a Class Test Suite...
-
date post
20-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of January 27, 2002 ECEN5033 University of Colorado -- Class Testing 1 Adequacy of a Class Test Suite...
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
1
Adequacy of a Class Test Suite Ideally – exhaustively test each class Practically – impossible or too hard Worth it to exhaustively test some classes with
high risk Measures of adequacy to increase confidence that
we have tested enough state-based coverage constraint-based coverage
Constraints are pre and post conditions and the class invariant
code-based coverage
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
2
State-based coverage How many of the transitions in a state transition diagram
are covered by the test suite? “Covered” = touched at least once May reveal each transition was covered but test values
do not adequately cover value ranges If test cases were generated correctly from a state
transition diagram with typical values and good boundary values, the test cases will achieve adequate state-based coverage
If test cases were generated from pre- and post conditions, then it is useful to check them against the state transition diagram to ensure each transition is covered.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
3
State-based coverage: object interaction Note how operations interact w.r.t. transitions
Current State
Input or event
Action Output Next State
State A Event 1 Output 1
State C
State B Event 2 Output 2
State C
State C Event 3 State D
Test cases for the transition from State C to State D may work if State C was reached from State A but not if State C was reached from State B. “State” is a function of history.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
4
State-based coverage: transition pairs Concerning problem on previous page:
Check that the test cases cover all pairs of transitions in the state transition diagram.
In previous table, create test cases to test: Transition from State A to State C and
transition from State C to State D Transition from State B to State C and
transition from State C to State D In other words, the event sequences:
Event 1 followed by Event 3Event 2 followed by Event 3
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
6
Hierarchical statechart for Elevator Control
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
8
Constraint-based coverage How many pairs of pre- and post conditions have
been covered? Using the technique described earlier for
generating test cases from pre and post conditions, if one test case is generated to satisfy each requirement, then the test suite meets this measure
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
9
Constraint-Based coverage: object interaction
For each operation opn that is not an accessor operation, identify the operations op1, op2, etc. for which
their preconditions are met when the post conditions for opn hold.
That is, post condition(opn) satisfies (>=) precondition(op1), precondition(op2), etc.
Then execute test cases for operation sequences opn-op1, opn-op2, etc.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
10
Class Diagram
Elevator Elevator Doors
Timer
Elevator Button
Up Floor Button
Floor
1 1 1 1
1
11
11
m
0…n
0…n
Travels-to
Travels-from
Opens-Closes Controls
Controls
Contains
velocity: ±m/sposition: meters
floor: integeron: Boolean
open: Booleanclosed: Booleanjammed: Boolean
time: seconds
floor: integerposition: meterson: Boolean
Down Floor Button
1
1
Contains
on: Boolean
Light
1on: Boolean
1
Turns-on
1
Turns-on 1Turns-on
1
1
Elevator Controller
1
0…n
ControlsRequests-elevator-of
Indicates-arrival-to
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
11
How do we complete the class test? States do not correspond 1-1 to methods or to
classes so the state transition diagram gives a different perspective
We want to adequately test the class Look at the class diagram and see which classes
send messages to it (“it” is the class under test) It could be difficult to create a driver to simulate
each class sending msgs to the class-under-test
If you only test each association in this application, it may be a weak level of testing for that class
Test each state transition, better yet, the pairs
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
12
Constraint coverage completes testing it
If you test class A calling every msg it can in class B, that’s more coverage but …
If you use existing classes to do that, it’s hard to get the right combination set up to make it happen
Alternative: Write a driver with sequences of messages to
access sequences of methods in the class you are testing.
Which sequences?
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
13
Which sequences??
Look at the post conditions of each method in the class you want to test
Look at preconditions of each Find sequences where
the Post(m1) >= pre(m2) the Post(m2) >= pre(m3) Send msgs to m1; m2; m3
Do this for all possible combinations
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
14
Completion check
See which associations were tested by comparing to the class diagram
See which state transitions were covered this way by comparing to the statechart
If some state transitions were missed or if some post/pre condition pairs or associations were missed, add a few tests
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
15
Where do the pre and post conditions come from?
Use cases?? The methods themselves? The class invariant?
Class TestingTesting Interactions, OATS
(Lecture 3 on Class Testing)
Software Engineering of
Standalone Programs
University of Colorado, Boulder
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
17
Overview Class Testing Testing Interactions between objects Testing Class Hierarchies
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
18
Testing Interactions Class Testing Testing Interactions between objects
Identifying & specifying object interactions Testing object interactions
Collection classes Collaborator classes Testing and Design Approach
Sampling COTS testing Patterns Testing exceptions
Testing Class Hierarchies
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
19
Testing Interactions with other classes An OO program is made of a collection of objects
that collaborate to solve a problem. Correct collaboration is necessary for program
correctness. In this section, we assume interactions are
sequential, not concurrent (that’s a subject for next semester)
Focus of interaction testing: Ensure that messaging occurs correctly between objects whose classes have already been tested separately.
Can happen embedded in application or in special environment using a test harness
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
20
What is an object interaction? Object interaction is:
request by a sender object to a receiver object to perform one of the receiver’s operations all of the processing performed by the receiver
to complete the request including a possible reply
Includes messages between an object and its components an object and other objects with which it is
associated
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
21
Object interaction impact During the processing of any single method invocation
on a receiving object Multiple object interactions can occur Want to consider impact of these on
the internal state of the receiving object those objects with which it has an association
Impact may be “no change” changes in attribute values in 1 or more objects
involved state changes in 1 or more objects including state changes of creation or deletion of
objects
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
22
Primitive and Non-Primitive Classes Interactions implied in a class spec where there
are references to other objects Primitive class can be instantiated and used
without needing to create any other instance Relatively few primitive classes in a program Since an OO program should model the objects in
a problem and all the relationships between them, non-primitive classes are common & essential
Non-primitive classes require the use of other objects in some or all of their operations
Identify the classes of those objects
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
23
Identifying Interactions Identify interactions by association relationships in the
class diagram regular associations aggregation composition
“Includes messages between an object and its components an object and other objects with which it is
associated “
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
24
Recognizing collaborators Collaborators may be addressed
directly, e.g. using a variable name by a pointer or a reference
dynamic type of the object may be different from the static type associated with the pointer
pointers and references are polymorphic, bound to an instance of any number of classes
Pre- and post conditions for operations in the public interface typically refer to states and/or specific attribute values of any collaborating objects
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
25
Collection and collaboration Collection class
maintains associations with instances of other classes but never actually interacts with those instances
that is, it never requests a service Collaborating class
Class with more extensive interactions Collection classes are less common
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
26
Testing Collection Classes What collection classes do
store references to these objects, representing one-to-many relationships between objects in a program
create instances of these objects delete instances of these objects
Specification refers to other objects does not refer to values computed based on
those objects
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
27
Identifying Collaborating Classes Non-primitive classes that are not collection
classes Use other objects in 1 or more operations as part of
their implementation When a post condition of an operation in a class’
interface refers to the state of an instance of an object and/or specifies that some attribute of that object is
used or modified,
that class is a collaborating class.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
28
Choosing testing “chunk” size Number of potential collaborations can grow too
large very quickly During class test: test composing object’s
interactions with its composed attributes As successive layers of aggregation are integrated,
test the interaction between an object and its associated objects
Defect visibility issue – If too large a “chunk” is chosen, intermediate results may be incorrect but not seen at the level of test-result verification
The more complex the objects, the fewer one should integrate prior to a round of testing
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
29
What factors increase object complexity? Number of parameters for each method Number of methods Number of state attributes in each object
Trying to test a chunk that is too complex can result in defects that are hidden from the tests – for example, leaving an object in an inappropriate state that is not expected and therefore not looked at in the test.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
30
Location of testing emphasis What are the implications of defensive and
contract design approaches when testing interactions?
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
31
Specifying interactions Remainder of slides assume
Operations defined by a class are specified by preconditions, post conditions, and class invariants
contract design approach
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
32
Contract vs. defensive approach & testing Contract More responsibility on
human designer Less class-level testing
due to fewer paths due to less error-checking code
MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?)
“illegal” for test cases to violate preconditions
Defensive More responsibility on
error-checking code More class-level testing
due to increased paths due to more code
Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met
Appropriate to violate preconditions to see if receiver catches that
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
33
Sampling Exhaustive testing: every possible test case
covering every combination of values Not reasonable much of the time
Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is
as good as we can do A sample is a subset of a population (e.g. all
possible test cases) that has been selected based on some probability distribution
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
34
Use profile (operational profile) Gave us a way to associate each use of the
system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the
probability of selection
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
35
Operational profile – stratified sample Stratified sample is a set of samples in which each
sample represents a specific subpopulation Example
Select test cases that exercise each component of the architecture
Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component.
Sample on each subset independent of the others Need a basis for stratifying
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
36
A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor
in the use cases Each actor uses some subset of the possible uses with
some frequency; rank by frequency Stratifying the test case samples by each actor provides a
way to increase the reliability of system Running selected tests
uses the system the way it will be used in typical situations;
finds defects likely to be found in typical use gives largest possible increase in reliability with least
effort
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
37
Test data generation for a range of values Generate test data first using the specification as
basis Select values based on boundary values
just within and just on the boundaries if contract just outside the boundaries also if defensive
Select values within intervals by sampling
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
38
Test data generation for a range of values, cont. Consider a random function over range 0.. N
such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over
many iterations Need to log actual values used in case of failure
so that the failed test case can be re-created. A randomly chosen value causing failure is
explicitly added to test suite and used to test the repaired software
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
39
Systematic sampling (+ boundary value testing)
Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling
over ranges We never omit boundary value test data
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
40
Issues about sampling for interaction testing Sampling issues
Different states can cause two objects from the same class to behave differently
A class family is a subset of a larger family Possibility of combinatorial explosion in the
number of test configurations
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
41
Impact of inheritance hierarchy on s.t.d.’s
Each member of the family may have different states that can cause two objects from the same class to behave differently.
In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there
will be the same number of states or more states in the derived class as in the base class
should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
42
Class families as subsets of others If a class family is a subset of a larger family, after
the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
43
Substitution Principle Only the following changes are allowed in defining
the behavior associated with a new subclass: Preconditions for each operation must be the
same or weaker – less constraining – than those of the superclass
Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass
Class invariant – must be the same or stronger; add more constraints
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
44
Orthogonal array testing applied to OO Orthogonal array testing is a sampling
technique to limit the explosion of test cases Define pair-wise combinations of a set of
interacting objects because most faults result from interactions due to two-way interactions
An orthogonal array is an array of values in which each column represents a factor.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
45
Orthogonal array testing applied to OO An orthogonal array is an array of values in
which each column represents a factor. A factor is
a variable in an experiment; in our case, a specific class family in the software system
Or states of that class family Each variable takes on a certain set of values
called levels (rows); in our case, specific classes or states of those classes
Each cell is an instance of the class, that is, a specific object or a particular state
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
46
OATS – orthogonal array testing system The factors (columns) are combined pair-wise
rather than representing all possible combinations of the levels for the factors.
Suppose we have three factors – A, B, C – and each has three levels
How many possible combinations of these values are there? 3 x 3 x 3
How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice?
factor=class family; level=class; cell=instance
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
47
How many pair-wise combinations? A B C 1 1 1 3 2 1 2 2 3 1 3 1 4 2 1 2 5 2 2 1 6 2 3 3 7 3 1 1 8 3 2 3 9 3 3 2
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
48
Mapping the terms of OATS to OO sw
array factor:
class family
factor:
class family state
level:
specific class
an instance of the specific class
A state of that instance
level:
specific class
an instance of the specific class
A state of that instance
level:
specific class
an instance of the specific class
A state of that instance
A state of that nstance
A state of that nstance
A state of that nstance
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
49
OATS – uses a balanced design Every level of a factor appears exactly the same
number of times as every other level of that factor Think of the rows of a table as test cases This is a systematic way of reducing the number
of test cases. If we decide to add more later, we know which
ones have not been run. Also logical – most errors are between pairs of
objects rather than among several objects
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
50
Look at elevator example (M&S, p. 230) Want to test interactions between elevator
controller, elevator, and wall button-panel Suppose:
Elevator can be basic, hi-speed, or express Elevator Controller can control basic, hi-speed,
or express Wall button-panel can be one-button panel or
two-button panel Elevator states: Moving to a floor, stopped, idle Elevator Controller states: idle, scheduling,
dispatching Wall button-panel states?
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
51
Steps from M & S, p. 230-232
1. Identify all factors (class families in this case)
2. Determine levels for each factor (states)
3. Locate a standard orthogonal array that fits the problem
1. We have 2 class families that each have 3 levels and three states
2. We have one class family that has _a_ levels and _b_ states
That’s how many possible test cases? 3 x 3 x 3 x 3 x a x b
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
52
Excerpt from Fig. 6.16, M&S, 21 x 37
1 2 3 4 5 6
1 1 1 1 1 1 1
2 1 1 2 2 2 2
3 1 1 3 3 3 3
4 1 2 1 1 2 2
5 1 2 2 2 3 3
6 1 2 3 3 1 1
7 1 3 1 2 1 3
8 1 3 2 3 2 1
9 1 3 3 1 3 2
10 2 1 1 3 3 2
11 2 1 2 1 1 3
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
53
Need a way to map Fig 6.16 to thisElevator Elevator
StateElev Ctrl
Elev Ctrl State
Wall Button Panl
Wall Button PnlStte
1
2
3
4
5
6
7
8
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
54
Mapping classes and states The class family with two levels (Wall Button
Panel) maps to Column 1 Its states map to Column 2 Column 3 can have a class family with three
levels; either elevator or elevator controller will do; Map Elevator Controller family to column 3 Map its states to column 4
What’s left? Map Elevator family to column 5 Map its states to column 6
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
55
Mapping levels – Wall Button Panel For Wall Button Panel
1 = 1-button panel 2 = 2-button panel
One-button Panel states (column 2) 1 = off 2 = on
For two-button panel (column 2) 1 = both off 2 = 1 requesting 3 = both requesting
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
56
Mapping levels – Elevator Controller Column 3 – elevator controller class family
1 = basic 2 = hi-speed 3 = express
Column 4 – elevator controller class states 1 = idle 2 = scheduling 3 = dispatching
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
57
Mapping levels -- Elevator Column 5 = elevator class family
1 = basic 2 = hi-speed 3 = express
Column 6 = elevator class states 1 = Moving to a floor 2 = stopped 3 = idle
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
58
Label Fig 6.16 with headers that “fit”
Wall-Button Panel
Wall-Button Pnl stte
Elev Ctrlr
Elev Ctrlr State
Elevator ElevatorState
1 1 1 1 1 1 1
2 1 1 2 2 2 2
3 1 1 3 3 3 3
4 1 2 1 1 2 2
5 1 2 2 2 3 3
6 1 2 3 3 1 1
7 1 3 1 2 1 3
8 1 3 2 3 2 1
9 1 3 3 1 3 2
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
59
When mapped to all of Fig 6.16 Each row becomes a test case Decode the level numbers for a row in the array
back to the individual lists for each factor In Fig 6.16, there are actually two more columns.
We can ignore them because we don’t need them.
Or we may decide we can include one more class family and its states in the future
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
60
What about mismatches? When there is a difference in the number of levels,
we can use a column that matches or exceeds the maximum
What about the Wall-Button Panel at level 1 where there is only one button? That only has two states but there can be a 3 in
column 2 associated with the 1 in the column 1 We interpret the 3 as if it were either 1 or 2.
What do you want to emphasize in the testing? Arbitrary assignment High frequency High risk
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
61
Other what-ifs? See M & S, p. 232
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
62
OATS adequacy criteria Ability to vary how completely the software under
test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base
classes from each hierarchy are tested Random – tester haphazardly selects cases from
several of the classes; not statistically random Representative – uniform sample ensures every
class is tested to some level Weighted representative – adds cases to the
representative approach based on relative importance or risk associated with the class