When to Test Less

Presented by Lan Guo

Introduction (1)

• Methods of software testing: functional, coverage, and user-oriented

• Phases of software testing: unit, integration, product, or regression

• White-box (coverage) testing uses the structure of the software to measure the quality of testing.

Introduction (2)

– Statement coverage– Decision coverage– Data flow coverage– Mutation testing

• Black-box testing aims at the specific features of the software.– Generate an operational profile, which is a list of

occurrence probabilities of each element in the input domain of the program

Introduction (3)

– The operational profile is dependent on the input from the customer.

• Difficulties in black-box testing– new software, new feature, no customer base– inadequate test set– coarse features, interacting features– multiple and unknown user groups

Introduction (4)

• There is a saturation effect associated with ALL testing methods.

Testing effort

Reliability

functional

decisionData flow

mutation

Introduction (5)

• Elaborate and expensive testing does not tend to find more faults than inexpensive manual or simple automatic testing schemes.

Theory of Testing (1)

• In black-box testing, the probability to find a fault in N random tests is

y = 1 - (1 - x)N

x: the probability of finding a fault for a random input• Black-box testing can be expensive. If x =

0.0001, 46,000 tests are needed to be 99% sure to reveal a fault.

Theory of Testing (2)

• Formal-methods testing is even more expensive compared to black-box testing.

• While-box testing only slightly reduces the number of tests required by black-box testing. In addition, the partition cost is very high.

Reality of Testing (1)

• Software-in-the-small projects produce software without rigorous testing.

• The software does not crash more often.

• A program’s shape is the shape of the pathways within the program. The more tangled the pathways, the more complex the program shape.

Reality of Testing (2)

• The simple shapes are common, which is demonstrated by du-pathways, control-flow diagrams and saturation effect.

• Saturation effect is consistent with programs containing either many portions with simple shapes--so the portions are easily reached-- or many portions that are so twisted in shape that we will never reach them.

Average Shape of Software

• Right shape: the program is either too complex to test or too simple to test elaborately.

• In either case, there is no point to conduct a lengthy and expensive test.

• Most programs are indeed the right shape for simple testing.

Proof: A mathematical model of building a pathway across a program from randomly selected inputs to some randomly selected state.

Assumption 1

• Programs are networks connecting system concepts. (NAYO)

• Optimizing compiler builds a network (control and data-flow graphs) from the code.– An execution trace of a procedural program shows what

statements are executed, which subroutines are care called, and in what order.

– A verification condition generator creates a proof tree (execution tree), including preconditions and post-conditions

Assumption 2

• A fault explanation tree is a tree whose leaves are inputs and whose root is a fault.

• If we arrive at a state which should not happen in this tree, it is a bug in our system.

Assumption 3

• Testing is a process of trying to generate a fault explanation tree from the program network.

• If we can’t generate such tree, we are confident that there is no fault in our program.

Assumption 4

• NAYO tree (explanation tree)

N: no edge

A: and node

Y: yes edge

O: or node

• No edge only connects or nodes

• Yes edge connects both and nodes and or nodes

Assumption 5

• Testing is the process of extracting NAYO trees (the explanations) from NAYO networks (the program).

Mathematics Equations (1)

• Given in inputs to a NAYO network with V nodes, the probability to find a fault is:

xo = in/V (A)

• The probability of reaching an and node with andp parents is the probability of reaching all its parents:

xand = xiandp (B)

Mathematics Equations (2)

• The probability of reaching an or node with orp parents is the probability of not missing all of its parents:

xor = 1 - (1 - xo)orp (C)• If the ratio of and nodes in a NAYO tree is andf,

then the ratio of or nodes in the same tree is orf = 1 - andf. The probability to reach some random node xj is:

xj = andf * xand + orf * xor (D)

Mathematics Equations (3)

• Rearrange Equation A to isolate the number of tests required to be 99% sure of finding a fault with probability xj is:

y = 0.99 = 1- ((1 - xj)N)

Therefore, N =log(1- 0.99)

log(1-xj)

Implementation

• We can approximately categorize the tree’s shape by the number of tests N required to reach that tree’s root to find a fault.

• Results from the Simulation RunsClassification Threshold Percent

Simple

Moderate

Hard

Overly complex

0 < N < 102

102 <=N<104

104<=N<106

N >= 106

36

19

25

20

Conclusion

• Black-box mathematics is blind to the internal structures of the program. If we include internal structure in the analysis, the result is more optimistic.

• The average case analysis of the shape of programs indicates that simple and RANDOM testing can be very valuable.

Related Research

• Simple and random testing can probe non-determinate systems.

• Limited testing is adequate for knowledge based systems.

Potential Future Research

• Extending static code analysis tools (optimizer and verification condition generator) to extract the parameters in the mathematical model from real-world programs, in order to monitor the change of program’s shape.

• Offering design guideline to avoid hard-shaped programs.

References (1)

• T. Menzies and B. Cukic, “When to Test Less”, IEEE Software, September/October 2000, pp. 107-112

• D. Hamlet and R. Taylor, “Partition Testing Does not Inspire Confidence”, IEEE Trans. Software Eng., Vol. 16, No. 12, Dec. 1990, pp. 1402-1411

• J. Horgan and A. Mathur, “Software Testing and Reliability”, The Handbook of Software Reliability Eng., M.R. Lyu, ed., McGraw-Hill, New York, 1996, pp. 531-565.

References (2)

• T. Menzies and B. Cukic, “Adequacy of limited testing for knowledge based systems”, International Journal on Artificial Intelligence Tools, Vol. 9, No. 1, 2000, pp. 153-172

• T. Menzies, B. Cukic, H. Singh, and J. Powell, “Testing Nondeterminate Systems”