Efficient and Effective Test Design

1

Justin Hunter Founder and CEO

September 17, 2009

Intent of today

2

William G. Hunter (my dad, a professor of applied statistics): Travelled to Nigeria for a year with our family to teach / share his expertise of Design of Experiments with chemical engineering students there because he thought it would help them do their jobs more efficiently and effectively.

Larry Ellison (Oracle CEO): Prioritizes making boatloads of money through software sales (... “not that there’s anything wrong with that”)

“This”(sharing effective methods)

Not “this”(convincing you to buy my tool and give me money)

I. Welcome

II. Introduction / Executive Summary -What challenges does combination testing solve?-How does combination testing work?-Where can it be used and what benefits does it deliver?-What are critical success factors and general lessons learned from 2 dozen projects?-Why isn’t this method of test design more well known and more widespread?

III. How the Proof of Concept Pilots Were Conducted -Structure of Pilots

-“2 testers in a room w/ Hexawise vs. 4 testers in a room w/out Hexawise”-Three hypotheses that were tested in each case study:

- No. 1 - Test design speed: savings of at least 30%- No. 2 - Test execution speed: savings of at least 25%- No. 3 - Cost of defect resolution: savings of at least 20%

IV. Blue Cross Blue Shield North Carolina Case Study - - What kind of testing projects were chosen? Why?- Findings for each of the three hypotheses - Were targeted savings achieved?- Ease of use / change management - How easy was it to learn? ...to implement?- Lessons learned - What went well? ... not so well? ... any surprises?- Turning the “speed v. quality” dial - How will efficiencies created be used?- Implications - Will you use Hexawise again? Why? Why not? Where?

Topics

3

Hexawise is a new test design tool. It is available through a SaaS (software as a service) model.

4

Sign up for a free trial. http://hexawise.com/users/new

(Only 4 required questions)

.What vs. How

5

What Should be Tested? How Should it be Tested?

• H/W Configurations

• S/W Configurations

• User Types

• Business Rules

• Sequences of Pathways

• Data

• Features

• Products Selected

• Primary Actions

• What Level of Detail

... or anything else that seems relevant

• What combinations should be

tested together?

• Test scripts in what order?

Treat these two topics separately for this presentation:.........................

........

What vs. How - Implications

6

What Should be Tested? How Should it be Tested?

• H/W Configurations

• S/W Configurations

• User Types

• Business Rules

• What Level of Detail

• Sequences of Pathways

• Data

• Features

• Products Selected

• Primary Actions

... or anything else that seems relevant

• What combinations should be

tested together?

• Test scripts in what order?

No change here:

This should continue to be SME-driven.

We have a better way:

Use Hexawise to determine what variables should be tested when (and in what combinations). Our scientifically-proven optimization methods create better tests, faster.

Challenges Addressed:

Too long to identify and document test cases - (manual process)

Inefficient test cases - (redundant testing and missing holes)

High defect resolution costs - (defects found late)

Order of Magnitude Benefits Delivered:

Assuming: 100 testers @ $100 / hour

1,500 hours / year per tester spent on test design and test execution

~ $5 million / year in benefits

Benefits Summary

7

Hexawise solves a constant problem: “There are way too many options to test everything. What should we test?”

6 browser choices

x 3 options x 2 options x 2 options x 2 options

x 4 options x 2 options x 3 options x 2 options

x 2 options = 13,824 possible tests...

...13,824 possible tests x 4 options x 4 options x 4 options

= 884,736 possible tests...

...884,736 possible tests x 5 optionsx 2 optionsx 2 optionsx 2 optionsx 2 optionsx 4 optionsx 2 optionsx 2 optionsx 2 optionsx 4 optionsx 2 optionsx 2 optionsx 2 options

72,477,573,120 possible tests

This single web page could be tested with

8

First, users input details of an application to be tested...

9

Next, users create tests that will cover interactions of every valid pair of values in as few tests as possible.

(1) Browser = “Opera” tested with (2) View = “Satellite?” Covered.(1) Mode of Transport = “Walk” tested with (2) Show Photos = “Y”? Covered.(1) Avoid Toll Roads = “Y” tested with (2) Show Traffic = “Y (Live)” ? Covered.

(1) Browser = IE6 tested with (2) Distance in = KM and (3) Zoom in = “Y” ? That is a 3-way interaction. It might not be covered in these 35 tests. See next page.

10

It also creates more thorough tests for all combinations involving 3 values, as below, or 4, 5 or even 6 values.

(1) Browser = IE6 tested with (2) Distance in = KM and (3) Zoom in = “Y” ? Covered. Any 3 valid values you can imagine? Yes, at least 1 of the 184 tests will cover all 3 together. If even higher quality is desired, all possible 4, 5, or 6-way interactions could be tested for.

11

It also creates mixed-strength solutions. This results in relatively more tests focused on selected priority areas.

(1) Browser = IE6 tested with (2) Distance in = KM and (3) Zoom in = “Y” ? Covered. Any 3 valid values you can imagine? Yes, at least 1 of the 184 tests will cover all 3 together. If even higher quality is desired, all possible 4, 5, or 6-way interactions could be tested for.

12

One of the advantages of this approach is that it creates coverage data...

% Coverage by Number of Tests100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

2 4 7 9 11 14 16 18 21 23 25 28

Every test plan has a finite number of valid combinations of parameter values (involving, in this case, 2 parameter values). The chart below shows, at each point in the test plan, what percentage of the total possible number of relevant combinations have been covered.

In this set of test cases, as in most, there is a significant decreasing marginal return.

13

... which is useful in determining “How much testing is enough?” / “When should we stop testing?”

0 25 50 75 100 125 150 175 200 225 250

If you found three defects in this test plan’s first 50 tests, you would find approximately

one more defect in the next 200 tests.

14

Standard Benefits

15

Test Execution

Time

Test DesignTime

Bug FixingCosts

Faster by at least 30%

Faster by at least 25%

Lower by at least 20%

Benefits Data

16

Source: “Combinatorial Testing” IEEE Computer, August, 2009. Dr. Rick Kuhn, Dr. Raghu Kacker, Dr. Jeff Lei, and Justin Hunter.

These benefits numbers are backed by empirical data. Small proof of concept pilots can be done w/in 2 weeks.

17

~85% of bugs can be identified by 2-way solutions during testing provided the tester inputs the defect-triggering features or data ranges or configuration options.

Source: “Combinatorial Testing” IEEE Computer, August, 2009. Dr. Rick Kuhn, Dr. Raghu Kacker, Dr. Jeff Lei, and Justin Hunter.

Benefits Data

I. Easy to Use (3 simple screens, made for SW testers)

II. Powerful (2-way to 6-way)

III. Flexible (mixed strength coverage)

IV. Insightful (“charting for everyone”)

V. Customizable (for both training and integration)

VI. Easy to Collaborate (e.g., notes with questions)

Hexawise Characteristics

18

Common Misperceptions

19

Change management is not always straightforward.

Resistance to new ideas and different ways of doing things is common.

Using pairwise and combinatorial test case generators is no exception; there are many objections raised by people hearing about it for the first time, including the ones on the next few slides...

SME Objections: Chess

20

The Experts

Kasparov and Karpov:

• Computers will never beat GrandMasters

• They lack the necessary:• Artistry• Instincts• Strategic insight

Reality

• No contest

• Not even close

SME Objections:Manufacturing

21

The Experts (in the 50’s, 60’s, and 70’s...)

Three phases:

• It won’t work

• It won’t work here

• “I thought of it first”

Reality (from the 80’s to current)

• The “Design of Experiments” methods that Hexawise employs are proven and widely practiced

• Toyota: “An engineer who does not understand Design of Experiments is not an engineer.”

Source: Dr. George Box, applied statistician

(Has helped dozens of manufacturing companies use Design of Experiments methods to handle

combinatorial explosions over a 30+ year career).

SME Objections: Software Testing

22

The Experts

Three phases:

• It won’t work

• It won’t work here

• “Oh wow.... It does

work here.”

Reality

• Test design times decrease by >30%

• Defects found per tester hour often more than double

• More defects are found earlier

• More defects are found overall

p. 95

20

Belief / Misconception

We’re “different” because we have a:

• Different industry or

• Different programming language or

• Different phase of testing or

• Different type of application or

• Different type of defect we’re

looking for

The Reality

None of these make a difference. Hexawise can (and does) deliver dramatic benefits whenever you can’t (or don’t want to) test every single possible use case.

Hexawise is being used successfully by hundreds of users in all kinds of different:

• Industries• Programming languages• Phases of testing• Types of applications

Objection: “It can’t test my ‘unique’ application.”

23

“Not all variables are = important”

24

Want to test some stuff “more”? 3 approaches:

1. (Blunt Approach) “IE7 is way more common than Opera and Safari”

“Clean”: add IE7 twice. IE7 tested twice as much but number of required tests might increase. “Cheat”: include “Opera or Safari” as a variable But now: is Safari tested at least once with “No. of stops in trip = 3”? Maybe.

II. (Subtle Approach) “Car directions are more common than public transit directions” or “Errors are more likely in XYZ new feature”

“Clean” solution: add each parameter in their order of importance For parameters with a small number of values: “mission accomplished” If Y/N: 60% of “Freebies” will automatically be “Y”, 40% will be “N” For parameters with a small number of values “no freebies” / nothing changes

III. (Math Whiz Approach) “Interactions between these [5] features are more likely to cause defects than with the rest of the variables”

See “mixed strength solutions” (slide 11).

?

Questions before case study

25

I. Welcome

II. Introduction / Executive Summary -What challenges does combination testing solve?- How does combination testing work?- Where can it be used and what benefits does it deliver?- What are critical success factors and general lessons learned from 2 dozen projects?- Why isn’t this method of test design more well known and more widespread?

III. How the Proof of Concept Pilots Were Conducted -Structure of Pilots

-“2 testers in a room w/ Hexawise vs. 4 testers in a room w/out Hexawise”-Three hypotheses that were tested in each case study:

- No. 1 - Test design speed: savings of at least 30%- No. 2 - Test execution speed: savings of at least 25%- No. 3 - Cost of defect resolution: savings of at least 20%

IV. Blue Cross Blue Shield North Carolina Case Study - - What kind of testing projects were chosen? Why?- Findings for each of the three hypotheses - Were targeted savings achieved?- Ease of use / change management - How easy was it to learn? ...to implement?- Lessons learned - What went well? ... not so well? ... any surprises?- Turning the “speed v. quality” dial - How will efficiencies created be used?- Implications - Will you use Hexawise again? Why? Why not? Where?

Blue Cross Blue Shield NC - Overview and Benefits

26

What are the benefits of Hexawise/combination testing...

...to the automated testing team?1. Effective test data generation - Using a scientific approach to generate

data is more efficient. The combinatorial data gathered from the Hexawise tool is far better than having a tester make an estimation based on past experience. Our experience has shown that by using the tool we can create test that have greater depth and breadth.

2. Having automatic generation of this data lends itself well to be adapted to test automation. The tool outputs CSV files which can be imported into an automation tool. Therefore fresh data can be at your disposal very quickly.

... to manual testers?1. While we did not use the tool for this purpose the above benefits would

apply equally.2. However, the fact that you can use the tool to reduce the amount of test

cases you run yet maintain a high amount of coverage is a benefit.

BCBSNC

27

What were the challenges with using Hexawise?

1. The integration of the data to the automated tests was not as easy as we first thought. We had to develop some Excel macros to convert the Hexawise data into data that we could plug directly in to our test automation (i.e. - We had to convert ranges (output from Hexawise ) into a single number within the range. Also, in one particular case, we had to change the format because of the complexity of the test we were doing. We could not just use the row/column format output from Hexawise. But these development tasks were one time tasks that we do not have to repeat each test cycle therefore our next cycle will benefit from the investment in time.

2. Sometimes understanding how to capture certain types of data input (or modeling) in the tool was challenging.

BCBSNC

28

What are lessons learned using Hexawise and/or other thoughts?

1. To ease the data integration an API would nice to have so that test automation can further exploit the benefits of Hexawise.

2. It behooves the user of Hexawise to clearly understand how the various inputs effect the SUT(System/Software under test). In our case as automation engineers, we had to go back to the SMEs (Subject Matter Experts) and consult them on the effects of the data we were about to hit there system with. This exercise proved to be eye opening to both us and them which resulted in better all around tests (increased coverage/efficiency) being designed.

3. Having some documented scenarios on inputting data would have been helpful. While most things are fairly obvious there are some techniques that need to be learned in order to get the best results from the tool.