HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.5613 Software Testing and Quality Assurance

Lecture 7, 1.10.2007

Combination Testing Strategies

Juha ItkonenSoberIT

2Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Typical testing task

3Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Testing combinations

Typical testing task includes testing features with many interacting optionsIn principle, all combinations of all input and output conditions and options should be testedIn practice, the number of combinations grows too bigSystematic techniques are needed to select a good subset of the combinations

Testing combinations of two or more input variables that affect the outcomeIntelligent and systematic way needed

4Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Combination Strategies

Combination testing strategies identify test combinations by combining values of different test object parameters based on some combinatorial strategy

Algorithms for selecting the combinations

At least 16 different strategies have been presentedM. Grindal, J. Offutt and S. F. Andler. “Combination testing strategies: a survey”, Journal of Software Testing, Verification and Reliability. 2005. Vol. 15:167-199.

5Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Example of combinatorial testing PowerPoint save options

5 checkboxesOne numeric fieldOne dropdown menu

5 optionsOne freetext fieldTwo radio buttons

Basic testing of these options without knowledge of their interrelations leads to many combinations:

2x2x2x2x2x4x5x3x2 = 384015 min per test case = 960h

6Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Combination strategies on different levels

Test casesCombinations of inputs and outputsCombinations of options Combinations of dataetc.

Test environmentsOperating systemsPlatforms (java, application server)DatabasesVersions of libraries or componentsetc.

7Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

From EP to combinations

Combination strategies complement other techniques

Equivalence partitioning or boundary analysis produces a large set of independent values for the input parametersThe next step towards more exhaustive testing is to cover the combinations of these individual values

8Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Coverage criteria for combination strategies

Each-used (1-wise) coverageEvery interesting value of each parameter is covered with some test case

Pair-wise coverageEvery possible pair of interesting values of any two parameters is covered

t-wise coverageEvery possible combination of interesting values of t parameters is covered

N-wise coverageAll possible combinations of interesting values of all N parameters are covered

9Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Variable strength coverage

Requires higher coverage among a subset of parameters and lower across all parametersFor example, for parameters A, B, C, and D

3-wise coverage among parameters B, C, and D2-wise coverage among all parameters

Can be used to reduce the total amount of testsBased on assumption of more tight dependencies and interaction between certain parameters

10Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Normal and error values in combinations

Normal (valid) and error (invalid) values should be treated differently

Error values typically lead to some error message or termination of execution

Only one error value of any parameter should be included in each test case

To avoid one error value masking anotherSingle error coverage

Normal coverageEach valid used coveragePair-wise valid coveraget-wise valid coverage

11Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Testing based on combinatorial strategies

All combinations strategyAll values once strategyBase Choice (default) strategyPair-wise Testing strategyDecision Tables

12Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Combinatorial testing example Testing a find function

A simple find dialogThree inputs

Text stringCase sensitivityDirection of the search

Assume following partitions (ignore illegal values for simplicity)

Text: uppercase / lowercase / mixedCase: match / do not matchDirection: normal / backwardsSearched text is found / is not found

13Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

All combinationsTest no. Text Match Case Direction Is found

1 U Y F Y

2 U Y F N

3 U Y B Y

4 U Y B N

5 U N F Y

6 U N F N

7 U N B Y

8 U N B N

9 L Y F Y

10 L Y F N

11 L Y B Y

12 L Y B N

13 L N F Y

14 L N F N

15 L N B Y

16 L N B N

17 M Y F Y

18 M Y F N

19 M Y B Y

20 M Y B N

21 M N F Y

22 M N F N

23 M N B Y

24 M N B N

All combinations for the three input fields plus the outcome of the test (last column)

3x2x2x2 = 24 combinations

14Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

All values once strategy

Text Match Case Direction Is found

1 L Y F N

2 U N F N

3 M N B Y

Simplest method for selecting tests is to test each option at least once

No combination coverageonly single-mode defects covered systematically

Some typical cases easily missedCan be used to test every feature once with minimal amount of test cases

Then select which ones have interactionsand create e.g. pair-wise combinations for those 3 combinations

15Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Base choice strategy

Set each variable in the default value (first row)and vary one variable at time in each test caseuntil all values of each variable are coveredGood approach as long as the parameters are independent

The most typical cases best coveredEasy to analyse what caused the failureNo systematic coverage for combinations

Test no. Text Match Case Direction Is found1 U Y F Y

2 L Y F Y

3 M Y F Y

4 U N F Y

5 U Y B Y

6 U Y F N

6 combinations

16Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Pair-wise strategy (a.k.a all-pairs testing) (Orthogonal array testing in some sources)

The tested combinations are selected so that the test cases cover all possible pairs of the values of the parametersBased on assumption that most of the defects in software systems are related to interaction between two variables (double-mode defects)

A certain values of some pair of variables causes the failure, not e.g. a certain combination of three or more variables

Pair-wise strategy reduces the number of tested combinations radically

10 variables with ten values each would require 10,000,000,000 test cases. Pair-wise strategy only requires 177 cases.

17Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Pair-wise testing exampleTest Text Match Case Direction Is Found

1 Upper Yes Forward Yes

2 Upper No Backwards No

3 Lower Yes Backwards Yes

4 Lower No Forward No

5 Mixed Yes Forward No

6 Mixed No Backwards Yes

All combinations of the values of each pair of parameters get testedTesting all pairs can catch defects caused by interaction of any two variables (i.e. double-mode defects)Note! Does not guarantee that typicalcases are covered

6 combinations

18Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Decision tables

Modelling complicated business rulesFocusing to interesting combinations

Business logic viewpointDifferent combinations of conditions lead to different expected outcome (actions)Does not have systematic coverage criteria

Rule 1 Rule 2 …

ConditionsCondition 1Condition 2

…

Actions

Action 1

…

19Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Decision table example: Discount rules

Young Pensioner Sale Sale with coupon

Sale with age discount

ConditionsAge < 15 > 65 15>=

age<=65

- < 15 or

> 65

Sale 0% 0% -50% -50% -50%Discount coupon

- - - Yes -

ActionsDiscount percent

10% 15% 50% 70% 55%

Bonus limit 10 € 25€ 100€ 100€ 50€

20Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

More information

More information on decision table testing and pairwise testing delivered as lecture notes Allpairs software for selecting pairwise test cases

Can be downloaded from course material pagesSimple command-line toolDocumentation is includedZip-file includes sample files for the above covered “find feature” example

allpairs find_feature_options.txt > find_feature_tests.txt