1

Simplified Modeling of Combinatorial Test Spaces

Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick

IBM Haifa Research Labs

2

Agenda

• Introduction

• Counters

• Properties

• Implementation

• Summary

3

Background – Restrictions

• Restrictions are a crucial part of real-life combinatorial models

• Effort and expertise required for capturing the restrictions is high

• Existing work concentrates on the ability to state complex restrictions– Typically as propositional formulas or

predicates– Can be captured using classification trees

4

The Problem

Restrictions are typically:

• Cumbersome to read and review

• Not very generalized (exclude specific conditions)

• Hard to maintain

5

Introduction

• An approach for handling some complex restrictions

• We introduce two new constructs to the CTD model:– Counter parameters– Value properties

• Real-life case studies

6

Counters

7

Counters – Example

• Consider testing an upgrade process for a virtualized server– Traverses the host and VMs– Upgrades various out-of-date components– Ten VMs, each has one of four OS versions:

RedHat 5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6– 2 versions are marked “preferred”– Triggered when at least half are out of date

8

Counters – Example – Cont.

• Parameters and values:

– VM1 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6

…

– VM10 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6

9

Counters – Example – Cont.

“Triggered when at least half are out of date”… ?!

10

Counters – Example – Cont.

• (Bad) option I: explicitly exclude all combinations in which less than half are out of date– All 653,312 of them !

11

Counters – Example – Cont.

• (Bad) option II: choose 5 VMs, and force them to be outdated– VM1 != “AIX6.1.6” && VM1 != “RedHat6.1” &&

VM2 != “AIX6.1.6” && VM2 != “RedHat6.1” &&…

VM5 != “AIX6.1.6” && VM5 != “RedHat6.1”

– 252 such (terrible) restrictions

12

Counters

• A counter is a parameter that counts values in other parameters

• Automatically evaluated in each test

• Can be used in restrictions and in coverage requirements

13

Counters – Back to the example

• Define counter “preferred” – Counts appearances of “RedHat6.1” and

“AIX6.1.6” in parameters VM1 … VM10

• One restriction that excludes:

preferred > 4

14

Auxiliary Variables

• The notion can be extended to auxiliary variables

• Parameters that do not define the test– but are rather a function of the defining

parameters

• Examples:– Sum of other numeric parameters– Count the number of duplicate values

15

(Touching on) Implementation

• Naïve implementation: Add the counter to the model, and restrictions that correctly map its values to the counted values

• Much better: Represent as functions on other parameters

• More details in the paper

16

Properties

17

Properties – Example

• Consider testing a Storage Area Network (SAN)– A server (one of four types – two x86-based,

two PowerPC-based)– An operating system (one of 17 versions –

two Windows-based, three AIX-based, one virtualized, others Linux-based)

– HBA (one of 11 types, of different manufacturers)

18

Properties – Example – Cont.

• Some OSs do not run on some processors (e.g., AIX on x86, Windows on PowerPC)

• Some HBAs do not have drivers for some OSs

• Some HBAs cannot connect to some processors

• A total of 299 pairs to be excluded

• Tedious and error-prone

19

Properties

• Properties may be defined for parameters

• A value to the property is assigned for each value of the parameter

• Can be used in restrictions and in coverage requirements

• Represent different abstractions of the parameters

20

Properties – Back to the example

• Define property “Architecture” for parameter “Server”– The two PowerPC servers will have value

PowerPC– The two x86 servers will have value x86

• Define property “OSFamily” for parameter “OS”, with values Windows, AIX, Linux

21

Properties – Back to the example – cont.• Define properties “X_Compatible”,

“P_Compatible”, “AIX_Compatible” and “Win_Compatible” for the HBA parameter– Give property values to the parameter values

according to their compatibility with x86, PowerPC, AIX and Windows, respectively

22

Properties – Back to the example – Cont.• The restrictions now disallow:

– Server.Architecture == “PowerPC” && OS.OSFamily == “Windows”

– Server.Architecture == “x86” && OS.OSFamily == “AIX”

– Server.Architecture == “PowerPC” && ! HBA.P_Compatible

– Server.Architecture == “x86” && ! HBA.X_Compatible

– OS.OSFamily == “AIX” && ! HBA.AIX_Compatible– OS.OSFamily == “Windows” && ! HBA.Win_Compatible

23

Simplified Modeling – Benefits

• More readable restrictions– Easier to review

• More general– Preserves time, reduces risk of errors

• More maintainable– e.g., a new Windows-based OS is introduced

24

Implementation

• Our symbolic BDD-based implementation is agnostic to test-space representation

• It merely requires one to symbolically represent the set of valid tests

• Simple to extend to support counters & properties

• More details in the paper

25

Summary

• Correctly capturing restrictions is a significant obstacle in combinatorial testing deployment

• We introduced two constructs that significantly simplify restrictions in some cases

• Examples based on real-life cases

26

Thank You For Listening