1
Simplified Modeling of Combinatorial Test Spaces
Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
IBM Haifa Research Labs
2
Agenda
• Introduction
• Counters
• Properties
• Implementation
• Summary
3
Background – Restrictions
• Restrictions are a crucial part of real-life combinatorial models
• Effort and expertise required for capturing the restrictions is high
• Existing work concentrates on the ability to state complex restrictions– Typically as propositional formulas or
predicates– Can be captured using classification trees
4
The Problem
Restrictions are typically:
• Cumbersome to read and review
• Not very generalized (exclude specific conditions)
• Hard to maintain
5
Introduction
• An approach for handling some complex restrictions
• We introduce two new constructs to the CTD model:– Counter parameters– Value properties
• Real-life case studies
6
Counters
7
Counters – Example
• Consider testing an upgrade process for a virtualized server– Traverses the host and VMs– Upgrades various out-of-date components– Ten VMs, each has one of four OS versions:
RedHat 5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6– 2 versions are marked “preferred”– Triggered when at least half are out of date
8
Counters – Example – Cont.
• Parameters and values:
– VM1 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6
…
– VM10 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6
9
Counters – Example – Cont.
“Triggered when at least half are out of date”… ?!
10
Counters – Example – Cont.
• (Bad) option I: explicitly exclude all combinations in which less than half are out of date– All 653,312 of them !
11
Counters – Example – Cont.
• (Bad) option II: choose 5 VMs, and force them to be outdated– VM1 != “AIX6.1.6” && VM1 != “RedHat6.1” &&
VM2 != “AIX6.1.6” && VM2 != “RedHat6.1” &&…
VM5 != “AIX6.1.6” && VM5 != “RedHat6.1”
– 252 such (terrible) restrictions
12
Counters
• A counter is a parameter that counts values in other parameters
• Automatically evaluated in each test
• Can be used in restrictions and in coverage requirements
13
Counters – Back to the example
• Define counter “preferred” – Counts appearances of “RedHat6.1” and
“AIX6.1.6” in parameters VM1 … VM10
• One restriction that excludes:
preferred > 4
14
Auxiliary Variables
• The notion can be extended to auxiliary variables
• Parameters that do not define the test– but are rather a function of the defining
parameters
• Examples:– Sum of other numeric parameters– Count the number of duplicate values
15
(Touching on) Implementation
• Naïve implementation: Add the counter to the model, and restrictions that correctly map its values to the counted values
• Much better: Represent as functions on other parameters
• More details in the paper
16
Properties
17
Properties – Example
• Consider testing a Storage Area Network (SAN)– A server (one of four types – two x86-based,
two PowerPC-based)– An operating system (one of 17 versions –
two Windows-based, three AIX-based, one virtualized, others Linux-based)
– HBA (one of 11 types, of different manufacturers)
18
Properties – Example – Cont.
• Some OSs do not run on some processors (e.g., AIX on x86, Windows on PowerPC)
• Some HBAs do not have drivers for some OSs
• Some HBAs cannot connect to some processors
• A total of 299 pairs to be excluded
• Tedious and error-prone
19
Properties
• Properties may be defined for parameters
• A value to the property is assigned for each value of the parameter
• Can be used in restrictions and in coverage requirements
• Represent different abstractions of the parameters
20
Properties – Back to the example
• Define property “Architecture” for parameter “Server”– The two PowerPC servers will have value
PowerPC– The two x86 servers will have value x86
• Define property “OSFamily” for parameter “OS”, with values Windows, AIX, Linux
21
Properties – Back to the example – cont.• Define properties “X_Compatible”,
“P_Compatible”, “AIX_Compatible” and “Win_Compatible” for the HBA parameter– Give property values to the parameter values
according to their compatibility with x86, PowerPC, AIX and Windows, respectively
22
Properties – Back to the example – Cont.• The restrictions now disallow:
– Server.Architecture == “PowerPC” && OS.OSFamily == “Windows”
– Server.Architecture == “x86” && OS.OSFamily == “AIX”
– Server.Architecture == “PowerPC” && ! HBA.P_Compatible
– Server.Architecture == “x86” && ! HBA.X_Compatible
– OS.OSFamily == “AIX” && ! HBA.AIX_Compatible– OS.OSFamily == “Windows” && ! HBA.Win_Compatible
23
Simplified Modeling – Benefits
• More readable restrictions– Easier to review
• More general– Preserves time, reduces risk of errors
• More maintainable– e.g., a new Windows-based OS is introduced
24
Implementation
• Our symbolic BDD-based implementation is agnostic to test-space representation
• It merely requires one to symbolically represent the set of valid tests
• Simple to extend to support counters & properties
• More details in the paper
25
Summary
• Correctly capturing restrictions is a significant obstacle in combinatorial testing deployment
• We introduced two constructs that significantly simplify restrictions in some cases
• Examples based on real-life cases
26
Thank You For Listening
Top Related