EECE 310: Software Engineering

Testing / Validation

What will we learn ?

• Testing– Black-box testing– Glass-box testing– Integration testing

val·i·date 1. To declare or make legally valid. 2. To mark with an indication of official

sanction. 3. To establish the soundness of;

corroborate.

Can we do any of these with software?

Java’s LicenseREAD THE TERMS OF THIS AGREEMENT AND ANY PROVIDED SUPPLEMENTAL LICENSE TERMS (COLLECTIVELY "AGREEMENT") CAREFULLY BEFORE OPENING THE SOFTWARE MEDIA PACKAGE. BY OPENING THE SOFTWARE MEDIA PACKAGE, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCESSING THE SOFTWARE ELECTRONICALLY, INDICATE YOUR ACCEPTANCE OF THESE TERMS BY SELECTING THE "ACCEPT" BUTTON AT THE END OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL THESE TERMS, PROMPTLY RETURN THE UNUSED SOFTWARE TO YOUR PLACE OF PURCHASE FOR A REFUND OR, IF THE SOFTWARE IS ACCESSED ELECTRONICALLY, SELECT THE "DECLINE" BUTTON AT THE END OF THIS AGREEMENT.

READ THE TERMS OF THIS AGREEMENT AND ANY PROVIDED SUPPLEMENTAL LICENSE TERMS (COLLECTIVELY "AGREEMENT") CAREFULLY BEFORE OPENING THE SOFTWARE MEDIA PACKAGE. BY OPENING THE SOFTWARE MEDIA PACKAGE, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCESSING THE SOFTWARE ELECTRONICALLY, INDICATE YOUR ACCEPTANCE OF THESE TERMS BY SELECTING THE "ACCEPT" BUTTON AT THE END OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL THESE TERMS, PROMPTLY RETURN THE UNUSED SOFTWARE TO YOUR PLACE OF PURCHASE FOR A REFUND OR, IF THE SOFTWARE IS ACCESSED ELECTRONICALLY, SELECT THE "DECLINE" BUTTON AT THE END OF THIS AGREEMENT.

Java’s License

5. LIMITATION OF LIABILITY. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL SUN OR ITS LICENSORS BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO THE USE OF OR INABILITY TO USE SOFTWARE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. …

5. LIMITATION OF LIABILITY. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL SUN OR ITS LICENSORS BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO THE USE OF OR INABILITY TO USE SOFTWARE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. …

Java’s License2. RESTRICTIONS. … Unless enforcement is

prohibited by applicable law, you may not modify, decompile, or reverse engineer Software. You acknowledge that Software is not designed, licensed or intended for use in the design, construction, operation or maintenance of any nuclear facility. Sun disclaims any express or implied warranty of fitness for such uses.

2. RESTRICTIONS. … Unless enforcement is prohibited by applicable law, you may not modify, decompile, or reverse engineer Software. You acknowledge that Software is not designed, licensed or intended for use in the design, construction, operation or maintenance of any nuclear facility. Sun disclaims any express or implied warranty of fitness for such uses.

Software Validation

Process designed to increase our confidence that a program works as intended

For complex programs, guarantees are very unusual

This is why typical software licenses don’t make any claims about their program working

Increasing Confidence

TestingRun the program on set of inputs and check the results

VerificationArgue formally or informally that the program always

works as intended

Analysis / Code reviewsPoor programmer’s verification [arguable!]: examine

the source code to increase confidence that it works as intended

Code Reviews

• Inspect other’s code for bugs (manually)• Why code reviews alone are not enough ?– Ad-hoc techniques, no standard way of doing it– Not reproducible easily – too much reliance on

human factors– Errors can occur even when all standard coding

practices are followed (corner cases)– Often, things that are difficult for one person are

also difficult for another (Knight & Leveson study)

Formal Verification• Tests the correctness of a piece of code without

executing it– Requires detailed formal specifications of behavior– Can reveal the presence of hard-to-find bugs

Theorem-proving-Can prove correctness in all cases (inductive)-Requires significant expert intervention

Model-Checking-Proves correctness up to a finite depth bound-Can be completely automated, but not always scalable

Formal Verification

Testing and FishingUsing some successful tests to conclude that a program has no bugs, is like concluding there are no fish in the lake because you didn’t catch one!flickr cc: gsankary

EECE 310 - 12

Testing

Testing increases confidence that a program meets its specification

Testing can never be exhaustive for non-trivial programs

So pick a partition of the input space for testing

Systematic testing depends on partitioning1. Partition the set of possible behaviors of the system2. Choose representative samples from each partition3. Make sure we covered all partitions

How do you identify suitable partitions?That’s what testing is all about!!!

Methods: black box, glass box, ...

Goals of TestingRepeatable:If you find an error, you’ll want to repeat the test to show othersIf you correct an error, repeat the test to check you did fix it

Systematic:Select test sets that cover the range of behaviors of programSelect test sets that are representative of program’s real uses

Documented:Keep track of what tests were performed, and their resultsEstimate how much more testing needs to be performed

What will we learn ?

• Testing– Goals of Testing– Black-box testing– Glass-box testing

Naïve Testing

• Consider the specification of the sqrt procedure ?

• How will we test it ?• When are we done ?

float sqrt (float x, float epsilon) throws InvalidValueException

{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon y2 x+epsilon

float sqrt (float x, float epsilon) throws InvalidValueException

{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon y2 x+epsilon

Black-box Testing

• Generate test cases from the specification only (don’t look at code)

• Advantages:–Avoids making the same assumptions as the

programmer–Test data is independent of the procedure’s

implementation–Results can be interpreted without knowing

implementation details–Forces you to write a good specification

Test cases for black box testingPaths through the spec– E.g., choose test cases that cover each part of the

‘requires’, ‘modifies’ and ‘effects’ clauses

Boundary conditions– Choose test cases that are at or close to boundaries for ranges

of inputs– Test for aliasing errors (e.g., two parameters referring to the

same object)

Invalid input– The program should degrade gracefully, without experiencing

data loss– The program should throw an exception without propagating

the error further

Black Box Testing: Example

• Paths through the spec:– “x 0” means “x>0 or x = 0”, so test both “paths”– It is not always possible to choose tests to cover the effects clause…

• can’t choose test cases for “x-epsilon=y2” or “y2=x+epsilon”• if the algorithm always generates positive errors, can’t even generate y2 < x

• Boundary conditions:– As “x 0” choose:

• 1, 0, as values for x

– As “0.00001 < epsilon < 0.001” choose:• 0.000011, 0.00001, 0.0000099, 0.0011, 0.001, 0.00099, as values for epsilon

– Also, try very large & very small values for x

• Invalid input:

– Negative values for x and epsilon– Values for epsilon > 0.001, values for epsilon < 0.00001

float sqrt (float x, float epsilon) throws InvalidValueException{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon y2 x+epsilon

float sqrt (float x, float epsilon) throws InvalidValueException{ // EFFECTS: Throws InvalidValueException // if x < 0 or 0.00001 ≥ epsilon or epsilon ≥ 0.001. // Else returns y such that x-epsilon y2 x+epsilon

AliasingStatic void appendVector(Vector v1, Vector v2) throws NullPointerException {

// EFFECTS: if v1 or v2 is null, throws NullPointerException // else removes all elements of v2 and appends them// in reverse order to the end of v1

if (v1 == null) throw new NullPointerException(“Vector.appendVector”); while (v2.size ( ) > 0 ) {

v1.addElement( v2.lastElement( ) ); v2.removeElementAt(v2.size() – 1 ); }}

Black Box Testing: Abs Program

• Can you come up with black box test cases for the program shown below ?

int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x

int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x

Black Box Testing: Abs Program

• Can you come up with black box test cases for the program shown below ?– Will this error be detected by the test suite ?

int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x if (x < -5) return –x; else return x;}

int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x if (x < -5) return –x; else return x;}

What will we learn ?

• Testing– Goals of Testing– Black-box testing– Glass-box testing

Glass Box testing

• Examine the code and test all paths– Because black box testing can never guarantee we

exercised all the code• Path completeness:– A path is a sequence of statements in the code– A test set is path-complete if each path through

the code is exercised by at least one case in the test set• Not the same as saying each statement in the code is

reached!! (Why ?)

Glass Box Testing: Example

• There are four paths in the following code

• So we need at least 4 test-cases. Example:– x=3, y=2, z=1– x=3, y=2, z=4– x=2, y=3, z=2– x=2, y=3, z=4

• Are the above tests sufficient ?

int maxval(int x,int y,int z) {// EFFECTS: returns maximum// value of the three inputs if (x > y) {

if (x > z) return xelse return z }

else {if (y > z) return yelse return z } }

int maxval(int x,int y,int z) {// EFFECTS: returns maximum// value of the three inputs if (x > y) {

if (x > z) return xelse return z }

else {if (y > z) return yelse return z } }

x > yx > y

x > zx > z y > zy > z

Weaknesses of path completeness• Path completeness is insufficient

Consider the test case x=4, y=1, z=2. This is path complete. The program performs correctly on this test case but

the program is still wrong !!• Path completeness is usually infeasible

– How many paths are there through this program segment (for any a[i])?

– Loops are problematic. Try:• test 0, 1, 2, n-1, and n iterations, (n is the max number of

iterations possible)

for (j=0, i=0; i<100; i++)if a[i]=true then j=j+1

for (j=0, i=0; i<100; i++)if a[i]=true then j=j+1

int maxval (int x, y, z) {/* EFFECTS: returns maximum value

of the three inputs */ return z;}

int maxval (int x, y, z) {/* EFFECTS: returns maximum value

of the three inputs */ return z;}

How to approximate Path Completeness ?

1. Loops:– With variable amounts of iteration• 0, 1, 2 iterations• All possible ways to exit loop for each case

2. For each statement where an exception could be raised– Test case that attempts to cause the exception

3. Straight-line codeAll paths through the code

THESE ARE ALL HEURISTICS. THERE IS NO SILVER BULLET.

Path Completeness Approximation: Example

• Test NPException:– factorial(-1)

• Paths through the loop:– 0 iterations (n = 1)

• returns 1– 1 iteration (n = 2)

• returns 1 (Bug found !)• 2 iterations (n = 3)

• returns 2 (Bug found !)

The program has a bug the loop condition should be (i <= n) not (i < n)

int factorial( int n ) {int result = 1; int i;

if (n < 0) throw new NPException(“fact”);

for (i = 1; i < n; i++) { result = result * i;

} return result;}

Triangle Example• Assume that the triangle testing procedure was as

follows. Can you come up with white-box tests for it ?

String triangle (int x, int y, int z) { if ( (x + y <= z) || (x + z <=y) || (y + z <=x) )

return “Invalid”; if (x == y || x == z)

if (x==y && y==z) return “Equilateral”; else return “Isoceles”; return “Scalene”;}

Condition Completeness

• Path completeness is alone not enough in the triangle case as the same path can result from different combinations of conditionals

• Try to exercise all combinations of conditional statements in the code– Aim for condition completeness rather than path

completeness– Ensure that side-effects of conditionals are covered

Group Activity• static boolean isPalindrome (String s) throws

NullPointerException// EFFECTS: If s is null throws NullPointerException,// else returns true if s reads the same forward // and backward, else returns false.// E.g., “deed” and “a” are both palindromes.

Can you come up with black box test cases ?

Can you come up with glass box test cases ?

How much overlap is there among them ?

Black-box testing casesreason input expected

outputTesting all paths of the specificationTest for the null argument

null NullPointerException

Make it return true and false

“deed” TRUE“seed” FALSE

Testing boundary conditionsempty string “”one character string “d” TRUEInvalid inputsCan you think of any ?

Glass-box testing casesReason Input Expected output

Test for s.length( ) when s is null nullNullPointerExcepti

on

Not executing the loop “d” TRUE

Returning false in the first iteration “seed” FALSE

Returning true after the first iteration “zz” TRUE

Returning false in the second iteration “abca” FALSE

Returning true after the second iteration

“deed” TRUE

One more input of odd length “abc” FALSE

Putting all the test cases togetherReason Input Expected output

Test for s.length( ) when s is null & testing for null argument (bb)

nullNullPointerExcepti

on

Empty string (bb) “” TRUE

Not executing the loop & one character string (bb)

“d” TRUE

Returning true after the first iteration “zz” TRUE

One more input of odd length “abc” FALSE

Returning false in the first iteration & make it return false (bb)

“seed” FALSE

Returning true after the second iteration & make it return true (bb)

“deed” TRUE

Returning false in the second iteration “abca” FALSE

Summary so far: Testing

• Testing is vital for finding bugs in programs and writing robust code

• Testing can be black box or white box– Black box Testing : Looks only at specifications– Glass Box Testing: Looks at code, aim for complete

path completeness – though not possible always– Neither is sufficient to guarantee finding all errors

Unit testingeach unit is tested separately to check it meets its specification

Integration testingunits are tested together to check they work together. Two strategies:

Integration testing is hard:much harder to identify equivalence classesproblems of scaletends to reveal specification errors rather than integration errors

Integration Testing

Bottom upfor this

dependency graph, the test

order is:1) d2) e and r3) q4) p

p

q r

ed

Top downfor this structure chart the order is:

1) test a with stubs for b, c, and d

2) test a+b+c+d with stubs for e…k

3) test whole system

b

a

e f g

c

h i

d

j k

System-level testingOther types of testsfacility testing - does the system provide all the functions required?volume testing - can the system cope with large data volumes?stress testing - can the system cope with heavy loads?endurance testing - will the system continue to work for long periods?usability testing - can the users use the system easily?security testing - can the system withstand attacks?performance testing - how good is the response time?storage testing - are there any unexpected data storage issues?configuration testing - does the system work on all target hardware?installability testing - can we install the system successfully?reliability testing - how reliable is the system over time?recovery testing - how well does the system recover from failure?serviceability testing - how maintainable is the system?documentation testing - is the documentation accurate, usable, etc.operations testing - are the operators’ instructions right?regression testing - repeat all testing every time we modify the system!

Automated TestingIdeally, testing should be automated

tests can be repeated whenever the code is modified (“regression testing”)takes the tedium out of extensive testingmakes more extensive testing possible

Will need:test driver - automates the process of running a test set

sets up the environmentmakes a series of calls to the unit-under-test (UUT)saves results and checks they were rightgenerates a summary for the developer

test stub(s) - simulates part of the program called by the unit-under-testchecks whether the UUT set up the environment correctlychecks whether the UUT passed sensible input parameters to the stubpasses back some return values to the UUT (according to the test case)(stubs could be interactive - ask the user to supply return values)

Summary so far

Testing: a way to validate program correctnessPartition the testing space

-- using specification: black box testing-- using the code: glass box testing

A large numbers of other, non-functional tests to validate programsintegration testingother system tests (e.g., endurance, security, usability,

volume)

Automated testing

To do before next class

1.Read in the textbook:– Chapter sections 10.1, 10.2, – Chapter 4– Exercises 4.1 to 4.4 and 10.1