Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing...
Transcript of Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing...
![Page 1: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/1.jpg)
Emina Torlak [email protected]
CSE 403: Software Engineering, Spring 2015courses.cs.washington.edu/courses/cse403/15sp/
Unit Testing
![Page 2: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/2.jpg)
Outline
2
• Software quality control
• Effective unit testing
• Coverage and regression testing
![Page 3: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/3.jpg)
introbasics of software quality control
![Page 4: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/4.jpg)
Errors and faults
4
Ariane 5: 37 sec after launch. Cost: $1 billion.
![Page 5: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/5.jpg)
Errors and faults
4
• Error: incorrect software behavior
• Example: Software controller for the Ariane 5 rocket crashed (and so did the rocket).
Ariane 5: 37 sec after launch. Cost: $1 billion.
![Page 6: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/6.jpg)
Errors and faults
4
• Error: incorrect software behavior
• Example: Software controller for the Ariane 5 rocket crashed (and so did the rocket).
• Fault: mechanical or algorithmic cause of error (bug)
• Example: Conversion from 64-bit floating point to 16-bit signed integer value caused an exception.
• Requirements specify desired behavior; if the system deviates from that, it has a fault.
Ariane 5: 37 sec after launch. Cost: $1 billion.
![Page 7: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/7.jpg)
Software quality control techniques
5
• Fault avoidance: prevents errors before the system is released.
• reviews, inspections, walkthroughs, development methodologies, testing, verification
• Fault tolerance: enables the system to recover from (some classes of) errors by itself.
• rollbacks, redundancy, mirroring
![Page 8: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/8.jpg)
Showing the presence and absence of bugs …
6
testing verification
![Page 9: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/9.jpg)
Showing the presence and absence of bugs …
6
testing verification
Detects the presence of bugs by running the code on a few carefully chosen inputs.
![Page 10: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/10.jpg)
Showing the presence and absence of bugs …
6
testing verification
Detects the presence of bugs by running the code on a few carefully chosen inputs.
Shows the absence of bugs on all possible inputs.
![Page 11: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/11.jpg)
Common kinds of testing
7
• Unit testing: tests the behavior of an individual module (method, class, interface)
• Black-box testing
• White-box testing
• System testing: tests the behavior of the system as a whole, with respect to scenarios and requirements
• Functional testing, integration testing
• Performance, load, stress testing
• Acceptance, usability, installation, beta testing
![Page 12: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/12.jpg)
uniteffective unit testing
![Page 13: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/13.jpg)
Two rules of unit testing
9
• Do it early and do it often
• Catch bugs quickly, before they have a chance to hide
• Automate the process if you can
• Be systematic
• If you thrash about arbitrarily, the bugs will hide in the corner until you're gone
![Page 14: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/14.jpg)
Four basic steps of a test
10
![Page 15: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/15.jpg)
Four basic steps of a test
10
1. Choose input data
• without looking at the implementation: black box
• with knowledge of the implementation: white box
![Page 16: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/16.jpg)
Four basic steps of a test
10
1. Choose input data
• without looking at the implementation: black box
• with knowledge of the implementation: white box
2. Define the expected outcome
![Page 17: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/17.jpg)
Four basic steps of a test
10
1. Choose input data
• without looking at the implementation: black box
• with knowledge of the implementation: white box
2. Define the expected outcome
3. Run on the input to get the actual outcome
![Page 18: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/18.jpg)
Four basic steps of a test
10
1. Choose input data
• without looking at the implementation: black box
• with knowledge of the implementation: white box
2. Define the expected outcome
3. Run on the input to get the actual outcome
4. Compare the actual and expected outcomes
![Page 19: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/19.jpg)
Four basic steps of a test
10
1. Choose input data
• without looking at the implementation: black box
• with knowledge of the implementation: white box
2. Define the expected outcome
3. Run on the input to get the actual outcome
4. Compare the actual and expected outcomes
This is hard! Need a set of test cases that is small enough to run quickly, yet large enough to cover [all] interesting program behaviors.
![Page 20: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/20.jpg)
Choosing inputs: two key ideas
11
![Page 21: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/21.jpg)
Choosing inputs: two key ideas
11
• Partition the input space
• Identify subdomains with the same behavior
• Pick one input from each subdomain
![Page 22: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/22.jpg)
Choosing inputs: two key ideas
11
• Partition the input space
• Identify subdomains with the same behavior
• Pick one input from each subdomain
• Boundary values
• Pick inputs at the edges of the subdomains.
• Effective at finding corner case bugs:
• off-by-one, overflow, aliasing, empty container
![Page 23: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/23.jpg)
Partitioning the input space
12
• Partition into
• a < b, a = b, a > b
• Pick an input from each class
• (1, 2), (0, 0), (2, 1)
// returns the maximum of a, b public static int max(int a, int b) { … }
![Page 24: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/24.jpg)
Partitioning the input space
12
• Partition into
• a < b, a = b, a > b
• Pick an input from each class
• (1, 2), (0, 0), (2, 1)
// returns the maximum of a, b public static int max(int a, int b) { … }
How would you partition the input space for • BigInteger multiplication?• Set intersection?
![Page 25: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/25.jpg)
Choosing boundary values
13
• Partition into
• a < 0, a > 0, a = 0 (boundary)
• Other boundary values
• Integer.MAX_VALUE
• Integer.MIN_VALUE
// returns|x| public static int abs(int a) { … }
![Page 26: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/26.jpg)
Choosing boundary values
13
• Partition into
• a < 0, a > 0, a = 0 (boundary)
• Other boundary values
• Integer.MAX_VALUE
• Integer.MIN_VALUE
// returns|x| public static int abs(int a) { … }
What are good boundary values for objects?
![Page 27: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/27.jpg)
Black box testing
14
• Explores alternate paths through the specification.
• Module under test is a black box: interface visible, internals hidden.
// If a >= b, returns a. Otherwise returns b. public static int max(int a, int b) { … }
• 3 paths, so 3 subdomains
• (1, 2) => 2
• (2, 1) => 2
• (0, 0) => 0
![Page 28: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/28.jpg)
Advantages of black box testing
15
• Process is not influenced by component being tested
• Assumptions embodied in code not propagated to test data.
• Robust with respect to changes in implementation
• Test data need not be changed when code is changed
• Allows for independent testers
• Testers need not be familiar with code
![Page 29: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/29.jpg)
Disadvantage of black box testing
16
• It will miss bugs in the implementation that are not covered by the specification
• Control-flow details
• Performance optimizations
• Alternate algorithms for different cases
![Page 30: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/30.jpg)
White box testing
17
• Explores alternate paths through the implementation
• Module under test is a clear box: internals visible.
boolean[] primeTable = new boolean[CACHE_SIZE];
boolean isPrime(int x) { if (x>CACHE_SIZE) { for (int i=2; i<x/2; i++) { if (x%i==0) return false;
} return true; } else { return primeTable[x]; }
}
• Important transition at around x = CACHE_SIZE
![Page 31: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/31.jpg)
(Dis)advantages of white box testing
18
![Page 32: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/32.jpg)
(Dis)advantages of white box testing
18
• Advantages
• Finds an important class of boundaries.
• Yields useful test cases.
• In isPrime example, need to check numbers on each side of CACHE_SIZE
• CACHE_SIZE-1, CACHE_SIZE, CACHE_SIZE+1
![Page 33: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/33.jpg)
(Dis)advantages of white box testing
18
• Advantages
• Finds an important class of boundaries.
• Yields useful test cases.
• In isPrime example, need to check numbers on each side of CACHE_SIZE
• CACHE_SIZE-1, CACHE_SIZE, CACHE_SIZE+1
• Disadvantages
• Tests may have the same bugs as implementation!
![Page 34: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/34.jpg)
Properties of good and bad unit tests
19
• Tests should be self-contained and not depend on each other implicitly or explicitly.
• "Smells" (bad things to avoid) in tests:
• Constrained test order
• Test A must run before Test B.
• Tests call each other
• Test A calls Test B.
• Mutable shared state
• Tests A/B both use a shared object.
![Page 35: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/35.jpg)
covercoverage and regression testing
![Page 36: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/36.jpg)
Measuring test suite quality with coverage
21
![Page 37: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/37.jpg)
Measuring test suite quality with coverage
21
• Various kinds of coverage
• Statement: is every statement run by some test case?
• Branch: is every direction of an if or while statement (true or false) taken by some test case?
• Path: is every path through the program taken by some test case?
![Page 38: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/38.jpg)
Measuring test suite quality with coverage
21
• Various kinds of coverage
• Statement: is every statement run by some test case?
• Branch: is every direction of an if or while statement (true or false) taken by some test case?
• Path: is every path through the program taken by some test case?
• Limitations of coverage
• Coverage is just a heuristic.
• 100% coverage may not be achievable.
• High-cost to approach the limit.
![Page 39: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/39.jpg)
Coverage measuring tools: EclEmma
22
![Page 40: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/40.jpg)
Regression testing
23
![Page 41: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/41.jpg)
Regression testing
23
• Whenever you find a bug
• Store the input that elicited that bug, plus the correct output
• Add these to the test suite
• Check that the test suite fails
• Fix the bug and verify the fix
![Page 42: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/42.jpg)
Regression testing
23
• Whenever you find a bug
• Store the input that elicited that bug, plus the correct output
• Add these to the test suite
• Check that the test suite fails
• Fix the bug and verify the fix
• Why is this a good idea?
• Ensures that your fix solves the problem.
• Helps to populate test suite with good tests.
• Protects against reversions that reintroduce bug:
• It happened at least once, and it might happen again
![Page 43: Unit Testing - courses.cs.washington.edu › courses › cse403 › 15sp › ...• Unit testing helps • convince others that a module works; • catch problems earlier. • Choose](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f1d38af08c662697357d25d/html5/thumbnails/43.jpg)
Summary
24
• Unit testing helps
• convince others that a module works;
• catch problems earlier.
• Choose test data to cover
• specification (black box testing)
• code (white box testing)
• Testing can’t generally prove the absence of bugs, but it can increase quality and confidence in the implementation.