CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.
-
Upload
claud-mcdowell -
Category
Documents
-
view
218 -
download
0
Transcript of CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.
![Page 1: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/1.jpg)
CSE403 Software Engineering Autumn 2001More Testing
Gary Kimura
Lecture #10
October 22, 2001
![Page 2: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/2.jpg)
Today
• Continue on with testing
• Understand the distinction between white-box and black-box testing
• Understand high-level basics of black-box
• Understand coverage issues in white-box
![Page 3: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/3.jpg)
Discreteness
• It’s also important to remember that testing software is different from testing physical widgets– In general, physical widgets can be analyzed in terms of
continuous mathematics – Software is based on discrete mathematics
• Why does this matter? • In continuous math, a small change in an input
corresponds to a small change in the output – This allows safety factors to be built in
• In discrete math, a small change in an input can correspond to a huge change in the output
![Page 4: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/4.jpg)
Ken Butler
• “The only unit of change in a program is the whole program.”
• That is, if you change anything in a program, you should treat it as if you changed everything – Of course, this isn’t practical in all cases
– But it’s a darned good model that captures the discrete nature of software
![Page 5: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/5.jpg)
General observation I
• A goal of picking a test case is that it be characteristic of a class of other tests
• That is, one case builds confidence in how other cases will perform
![Page 6: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/6.jpg)
General observation II
• The overall objective is to cover as much of the behavior space as possible – It’s generally infinite
• In general, it’s useful to distinguish the notions of common vs. unusual cases for testing
![Page 7: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/7.jpg)
Black box testing
• Treat the unit (program, procedure, etc.) as a black box – You can hypothesize about the way it is built, but you can’t see it
• Depend on a specification, formal or informal, for determining whether it behaves properly
• How to pick cases that cover the space of behaviors for the unit? – Use heuristics
![Page 8: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/8.jpg)
Equivalence partitioning
• Based on input conditions – If input conditions are specified as a range, you have one valid
class (in the range) and two invalid classes (outside the range on each side)
– If specified as a set, then you can be valid (in the set) or invalid (outside the set)
– Etc.
![Page 9: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/9.jpg)
Boundary values
• Problems tend to arise on the boundaries of input domains than in the middle
• So, extending equivalence partitioning, make sure to pick added test cases that exercise inputs near the boundaries of valid and invalid ranges
![Page 10: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/10.jpg)
Others include
• Cause-effect graphing
• Data validation testing
• Syntax-direct testing
• …
![Page 11: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/11.jpg)
White box testing
• In this approach, the tester has access to the actual software
• They needn’t guess at the structure of the code, since they can see it
• In this approach, the focus often shifts from how the code behaves to what parts of the code are exercised
![Page 12: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/12.jpg)
White box coverage
• In black box, the tests are usually intended to cover the space of behavior
• In white box, the tests are usually intended to cover the space of parts of the program
![Page 13: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/13.jpg)
Statement coverage
• One approach is to cover all statements – Develop a test suite that exercises all of a program’s statements
• What’s a statement?
max = (x > y) ? x : b;
if x > y then max := xelse max :=yendif
![Page 14: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/14.jpg)
Weakness
• Coverage may miss some obvious issues
• In this example (due to Ghezzi et al.) a single test (any negative number for x) covers all statements – But it’s not satisfying with respect to input condition coverage, for
example
if x < 0 then
x := -x;
endif;
z := x;
![Page 15: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/15.jpg)
Edge coverage
• Another approach is to use a control flow graph (CFG) representation of a program – Essentially, a flowchart
• Then ensure that the suite covers all edges in the CFG
![Page 16: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/16.jpg)
Condition coverage
• Complex conditions can confound edge coverage
if (p != NULL) and (p->left < p->right) …
• Is this a single conditional statement in the CFG?
• How are short-circuit conditionals handled? – andthen, orelse
![Page 17: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/17.jpg)
Path coverage
• Edge coverage is in some sense very static
• Edges can be covered without covering paths (sequences of edges) – These better model the actual execution
![Page 18: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/18.jpg)
Example
![Page 19: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/19.jpg)
Path coverage and loops
• In general, we can’t bound the number of times a loop executes
• So there are an unbounded number of paths in general – We resort to heuristics like
those from black box testing to exercise these loops
![Page 20: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/20.jpg)
Usefulness?
• Even with a CFG and statement coverage as testing techniques there is still a lot to be desired– How does one pick the proper input that will actually exercise a
certain statement or path?
– What about complex interactions with other system events and components?
– Etc.
![Page 21: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/21.jpg)
Some more practical aspects
• Who tests the tests, especially a large complicated test? – For example, if your triangle test program generates random data,
who confirms the results? – Another example is testing trig functions.
• Testing the error cases can be a wider set of inputs. You have two problems – Making sure you have proper test coverage and – Making sure the results are correct.
• Fault injection is another way of testing systems.– For example, injecting I/O failures in a disk controller can test the
error cases for the disk driver and file system. – Another example is injecting memory allocation errors, to see how
programs behave when they run out of memory.
![Page 22: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/22.jpg)
Final note on testing
• It’s unsound
• It’s heuristic – Heuristic doesn’t mean undisciplined
• It’s extremely useful and important
• Good testing requires a special mindset – “I’m going to make that sucker fail!”
• Good coding requires a special mindset – “Nobody’s going to break my code!”
![Page 23: CSE403 Software Engineering Autumn 2001 More Testing Gary Kimura Lecture #10 October 22, 2001.](https://reader035.fdocuments.in/reader035/viewer/2022062321/56649ef95503460f94c0b3ca/html5/thumbnails/23.jpg)
Next time
• Wednesday– Group Project Work Day
• Friday– Bugs