THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR...
Transcript of THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR...
THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR CONSTRUCTING
COVERING ARRAYS
by
Andreas Hundseid Ronneseth
A Thesis Presented in Partial Fulfillmentof the Requirements for the Degree
Master of Science
ARIZONA STATE UNIVERSITY
May 2006
THE BUILDING-BLOCK ALGORITHM: A NEW METHOD FOR CONSTRUCTING
COVERING ARRAYS
by
Andreas Hundseid Ronneseth
has been approved
April 2006
APPROVED:
, Chair
Supervisory Committee
ACCEPTED:
Department Chair
Dean, Division of Graduate Studies
ABSTRACT
Software systems consist of multiple interchangeable elements that interact with
each other. As the number of components increases, the size of a test suite that covers all
interactions grows exponentially. Test suites that cover all pairs of component interactions
have been proposed to mitigate this cost and still provide adequate fault detection. This
thesis proposes a new greedy test-generation method, called the Building-Block Algorithm
(BBA), that allows system testers to test a subset of the components in unit test and then
later incorporate the same set of test configurations in integration and system tests. It
uses smaller test suites to build larger test suites and often yields the smallest test suite
when compared to other greedy approaches. In one instance, 1124, a new lower upper
bound of 229 is reported. The results show that combining smaller test suites as building
blocks generated test suites faster than similar methods, thus reducing the cost of test suite
development.
iii
To my parents
iv
ACKNOWLEDGMENTS
First, I would like to thank Dr. Charles J. Colbourn for his excellent advice and
direction through this past year. His knowledge in this area is breathtaking and his insight
helped shape this thesis. With his schedule, I am amazed how he takes the time and care
to help each student, including myself. He is truly unique.
Dr. David C. Pheanis guided me through my undergraduate years, and taught
me computer engineering, personal finance, English grammar, and helped me obtain both
scholarships and my first job. I owe him my deep gratitude.
Also, I would like to thank Dr. James S. Collofello and Dr. Goran Konjevod for
serving on my supervisory committee. Dr. Renee C. Turban introduced me to this topic
during the course of my honors thesis and also provided the implementation of the Deter-
ministic Density Algorithm (DDA). Dr. Myra B. Cohen at University of Nebraska–Lincoln
shared her simulated annealing results. Thank you!
I would like to thank my family and especially my parents. They pushed me to come
to the United States in the first place, and I have not regretted that for even a second. Along
the way they have offered their love and advice from thousands of miles away. Without my
grandparents here in Arizona, I probably would have starved to death. I love you all.
Finally, my soulmate Melissa makes every day a little brighter!
v
TABLE OF CONTENTS
Page
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
CHAPTER 1 INTRODUCTION AND MOTIVATION . . . . . . . . . . . . . 1
CHAPTER 2 TECHNICAL BACKGROUND . . . . . . . . . . . . . . . . . 5
1. Combinatorial Models . . . . . . . . . . . . . . . . . . . . . . . 5
2. One-Row-at-the-Time Greedy Constructions . . . . . . . . . . . . . . 9
2.1. Automatic Efficient Test Generator . . . . . . . . . . . . . . . 9
2.2. Test Case Generator . . . . . . . . . . . . . . . . . . . . . . 10
2.3. Deterministic Density Algorithm . . . . . . . . . . . . . . . . . 12
3. IPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4. Combinatorial Constructions . . . . . . . . . . . . . . . . . . . . . 16
4.1. Orthogonal-Array Construction . . . . . . . . . . . . . . . . . 16
4.2. TConfig . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5. Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.1. Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . 20
5.2. Tabu Search . . . . . . . . . . . . . . . . . . . . . . . . . 20
6. Criteria for Comparing Algorithms . . . . . . . . . . . . . . . . . . 21
CHAPTER 3 THE BUILDING-BLOCK ALGORITHM AND DENSITY IMPLE-MENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1. An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2. The Building-Block Algorithm . . . . . . . . . . . . . . . . . . . . 27
3. Density Implementation . . . . . . . . . . . . . . . . . . . . . . . 29
vi
Page
3.1. Building-Block Density . . . . . . . . . . . . . . . . . . . . . 31
3.2. Building Blocks with Different Sizes . . . . . . . . . . . . . . . 33
3.3. Don’t-Care Positions . . . . . . . . . . . . . . . . . . . . . . 33
3.4. Implementation Summary . . . . . . . . . . . . . . . . . . . 34
4. Comparison of the Building-Block Algorithm and Other Methods . . . . 34
CHAPTER 4 COMPUTATIONAL RESULTS . . . . . . . . . . . . . . . . . 36
1. Different Building Blocks . . . . . . . . . . . . . . . . . . . . . . 37
1.1. Building Blocks from Bose’s Algorithm . . . . . . . . . . . . . . 37
1.2. Building Blocks from Heuristic Search . . . . . . . . . . . . . . 39
1.3. Building Blocks from DDA . . . . . . . . . . . . . . . . . . . 40
2. Comparison with Published Results . . . . . . . . . . . . . . . . . . 41
3. Execution Time . . . . . . . . . . . . . . . . . . . . . . . . . . 43
CHAPTER 5 FUTURE WORK AND CONCLUSION . . . . . . . . . . . . . 45
1. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
vii
LIST OF TABLES
Table Page
1. A modular ticket reservation system . . . . . . . . . . . . . . . . . 2
2. Test suite for ticket reservation system . . . . . . . . . . . . . . . . 2
3. Orthogonal array OA(4; 3, 2, 2) . . . . . . . . . . . . . . . . . . . . 6
4. Covering array CA(6; 2, 4, 2) . . . . . . . . . . . . . . . . . . . . . 7
5. Orthogonal Latin squares of order three . . . . . . . . . . . . . . . . 16
6. Construction of OA(9; 4, 3, 2) using Bose’s algorithm . . . . . . . . . . 18
7. Covering arrays A and B, both MCA(6; 2, 2, (2131)) . . . . . . . . . . 25
8. Partial covering array PMCA(6; 2, 4, (2232)) . . . . . . . . . . . . . . 26
9. Covering array MCA(N ; 2, 4, (2232)) . . . . . . . . . . . . . . . . . 26
10. BBA (using Bose’s algorithm) versus DDA . . . . . . . . . . . . . . 38
11. Mixed-level BBA (using Bose’s algorithm) versus DDA . . . . . . . . . 39
12. BBA (using tabu search) versus DDA . . . . . . . . . . . . . . . . . 40
13. BBA (using simulated annealing) versus DDA . . . . . . . . . . . . . 40
14. BBA (using DDA) versus DDA . . . . . . . . . . . . . . . . . . . 41
15. Comparison with Published Results . . . . . . . . . . . . . . . . . . 42
16. Building blocks for BBA for published results . . . . . . . . . . . . . 43
17. Execution time (BBA versus DDA) in seconds . . . . . . . . . . . . . 44
viii
LIST OF FIGURES
Figure Page
1. Concept figure of IPO growing the test suite horizontally and then vertically 15
2. Pseudo-code for the Building-Block Algorithm . . . . . . . . . . . . . 30
ix
CHAPTER 1
INTRODUCTION AND MOTIVATION
Software testing is an important part of the software development process. Software
testing is costly and time-consuming and therefore much research has gone into selecting the
appropriate test cases. As software is generally becoming more complex, it is also becoming
more modular in an attempt to reduce the complexity. A significant number of faults can
be attributed to unexpected interactions between software modules. Often, these faults are
caused by bad interactions between a relatively low number of modules. Kuhn and Reilly
studied 171 bugs for the Apache webserver and found that more than 70 % of the bugs
could be attributed to a bad interaction between only two modules. Interactions between
four or fewer modules caused 96 % of all the bugs in the study. Thus, we focus on creating
test sets that cover at least all t-way interactions where t is a small number, generally in
the range 2 ≤ t ≤ 6 [1]. Pairwise testing means t = 2, i.e., we test every pair of possible
interactions in a system.
A small example illustrates this approach. A software company is building a modular
software system for a travel agency built with four components. The software system has
to interact with three major airlines’ booking systems. A customer can book their flight
through the web, through an automated phone system, or by contacting an agent that uses
a specialized agent system. The travel agency issues the tickets through a web system,
through an e-ticket system, or by sending the customer a paper ticket. Finally, the travel
2
Table 1. A modular ticket reservation system
Airline Database Interface Issue Method
US Air MySQL Web Web TicketSouthwest Oracle Phone E-Ticket
KLM Postgre SQL Agent System Paper Ticket
Table 2. Test suite for ticket reservation system
Test No. Airline Database Interface Issue Method
1 US Air MySQL Web Web Ticket2 US Air Oracle Phone E-Ticket3 US Air Postgre SQL Agent System Paper Ticket4 Southwest MySQL Agent System E-Ticket5 Southwest Oracle Web Paper Ticket6 Southwest Postgre SQL Phone Web Ticket7 KLM MySQL Phone Paper Ticket8 KLM Oracle Agent System Web Ticket9 KLM Postgre SQL Web E-Ticket
agency uses three different database systems. Thus, we have a system of four components,
each with three different configuration possibilities as shown in Table 1.
If the software company wanted to exhaustively test this system, they would have to
test 34 = 81 different configurations. Instead we can employ pairwise testing and cover all
pairwise interactions in nine test cases as shown in Table 2. This offers substantial savings,
and for systems with more components and configurations, the savings become even more
apparent. Exhaustive testing for a system with 20 components each with three options
would require 320 or over three billion test cases. Pairwise testing on the same system
would require at most 15 test cases [2].
An interaction test can be modeled as a combinatorial object called a covering ar-
ray, which we define more formally later. Many methods have already been proposed to
3
construct these covering arrays. Several methods are direct constructions, such as TConfig
and Bose’s algorithm [3, 4]. These types of techniques do well on a limited set of inputs.
Heuristic-search methods, such as tabu search and simulated annealing have been used to
find some of the smallest known covering arrays [2, 5, 6]. Although the heuristic-search
methods can handle a wide array of inputs, this approach often take months of computa-
tion time. Greedy algorithms such as AETG, TCG, IPO, and DDA can construct a covering
array quickly, but with some loss of accuracy versus heuristic-search methods [7, 8, 9, 10].
All these methods are introduced in more detail later.
We present a new greedy method for constructing covering arrays called the Building-
Block Algorithm (BBA). Because this method uses smaller covering arrays to construct
larger covering arrays, it enables testers to reuse unit tests in integration or system tests.
An example illustrates the advantages of this attribute next.
Software systems are often built in subsystems and then integrated later in the
development cycle. For example, subsystem one has five components and subsystem two
has four components. The software tester tests subsystem one using automated pairwise
interaction testing. Subsystem two is tested the same way, although with a different set
of interaction tests. In integration testing, components in subsystem one interact with
individual components in subsystem two so the tester needs a test set that covers all pairwise
interactions between the nine components. Instead of starting over in integration testing,
BBA enables the tester to reuse all the automated tests from the unit test. The tester
also has to write a few more automated scripts to cover the additional interactions between
individual components in subsystem one and two. However, this is far less effort than
rewriting all the test scripts.
In this thesis we first introduce some combinatorial definitions in Chapter 2. Using
4
these definitions, we discuss some existing methods for constructing covering arrays and
criteria for evaluating different methods. Chapter 3 builds arguments that lead to the
development of the Building-Block Algorithm. An implementation of the algorithm is also
discussed in this chapter. The results in Chapter 4 show that BBA is generally as accurate or
more accurate and faster than existing greedy techniques. Finally, the last chapter discusses
future endeavors and draws some final conclusions.
CHAPTER 2
TECHNICAL BACKGROUND
The introduction of AETG in the mid-90s [7] sparked an increased interest in interac-
tion testing. The combinatorial, testing, and algorithm communities have all contributed to
the area. This chapter summarizes some of the researchers’ advances, especially in the last
decade. First, we introduce some combinatorial models, which help us describe the problems
in interaction testing. We then cover relevant algorithms for constructing test suites. The
algorithms fall into three main groups: Heuristic search, combinatorial constructions, and
greedy algorithms. The chapter ends with a discussion of criteria for comparing methods
of test suite generation.
For additional background information, we refer the reader to [11], which gives a
thorough review of research related to interaction testing and contains 140 references.
1. Combinatorial Models
An orthogonal array OAλ(N ; k, v, t) is an N × k array on v symbols such that every
N × t subarray contains all ordered subsets of size t from v symbols exactly λ times [12].
In other words, each tuple (or pair for t = 2) is covered exactly λ times. Orthogonal arrays
have the property that λ = Nvt . Table 3 is an orthogonal array OA(4; 3, 2, 2).
Mandl used orthogonal arrays for interaction testing of an Ada compiler [13].
Williams and Probert also did pair-wise testing using orthogonal arrays on a network in-
6
Table 3. Orthogonal array OA(4; 3, 2, 2)
1 a x1 b y2 a y2 b x
terface [3]. Orthogonal arrays provide the optimal test suites for certain values of v and
k when λ equals 1. However, for other values of v and k, orthogonal arrays only exist for
λ greater than 1. These orthogonal arrays may create unacceptably large test suites. The
literature has therefore largely moved away from orthogonal arrays for interaction testing
and instead concentrated on the less restrictive covering arrays, which require that every
tuples are covered at least once.
A covering array, CAλ(N ; t, k, v), is an N × k array with the exact same definition
as an orthogonal array, except it contains all ordered subsets from v symbols of size t at
least λ times. When λ equals 1, we omit λ and use the shorthand notation CAλ(N ; t, k, v).
In such a covering array we call t the strength, k the degree, and v the order [14]. The
covering array number, CAN(t, k, v) is the number of rows in an optimal covering array,
i.e., the minimum number of rows. If an orthogonal array exists for a set of parameters with
λ = 1 , it always yields CAN(t, k, v) since an orthogonal array is always a covering array
(but not vice versa). For example, an orthogonal array of size 9 exists for OA(9; 3, 2, 2),
thus CAN(2, 3, 3) equals 9. It also follows that the covering array number is always larger
or equal to vt (or λ × vt if λ is greater than 1).
Covering arrays as a model naturally map to the test suites. Each row of the covering
array represents a test case. The k columns represent the components or factors in the
system under test (SUT). The value in a column of a row represents the configuration or
7
Table 4. Covering array CA(6; 2, 4, 2)
Covering Array
Test f1 f2 f3 f4
1 a 1 x 32 b 2 y 33 a 2 x 44 b 1 y 45 a ⋆ y ⋆6 b ⋆ x ⋆
level for the factor for this test case. As shown in Table 4, we label the columns or factors
f1, f2, . . . , fk and each row 1, 2, . . . ,N .
The definition of a covering array does not require each column in each row to
contain a value. For example, say we have constructed the first five rows of covering array
CA(6; 2, 4, 2) in Table 4. At this point only one pair is uncovered overall, between factors
f1 and f3. The pair is {b, x}, so set column one in test case six to ’b’ and column three to
’x’. The covering array is now complete, and we can arbitrarily fix column two to ’1’ or
’2’ and column four to ’3’ or ’4’. Columns two and four of row six are don’t-care positions,
denoted by a ⋆. These positions can be fixed to any level of the particular factor without
changing the covering array’s property of covering every tuple at least once.
Although covering arrays give a natural mapping to test suites, covering arrays lack
the properties necessary to handle every testing situation. Specifically, covering arrays can
only handle systems where all the factors have the same number of levels. For example,
a system may have four factors with three levels and two factors with five levels, and we
should assume this is the more frequent case. Several authors have suggested that a mixed-
level covering array can handle this situation (see [15, 5, 16]). Cohen et al. [14] defined
good notation for these types of arrays and we use that notation in this thesis.
8
A mixed-level covering array, MCAλ(N ; t, k, (v1, v2, . . . , vk)), is an N × k array on
v symbols, where v =∑k
i=1vi, with the following properties:
1. Each column i (1 ≤ i ≤ k) contains only elements from set Si with |Si| = vi.
2. The rows of each N × t sub-array cover all t-tuples of values from the t columns at
least λ times.
We omit λ from the notation when λ equals 1, which is the normal occurrence.
In addition, the notation combines factors with an equal number of levels. For
example, ninety factors with ten levels can be combined to 1090. Thus, a multi-level cov-
ering array with ten factors, three with two levels and seven with four levels, can be writ-
ten as MCA(N ; t, 10, 2347). More formally, we can write a multi-level covering array as
MCA(N ; t, (wr1
1wr1
1. . . wrs
s )), where k =∑s
i=1ri and (wj : 1 ≤ j ≤ k) ⊆ {v1, v2, . . . , vk}
[10].
Finally, we introduce a combinatorial object called the partial mixed-level covering
array, PMCA(n/N ; t, k, (v1, v2, . . . , vk)), an n×k array on a maximum of v symbols, where
v =∑k
i=1vi, and the rows of each n×t subarray overall does not cover all t-tuples, although
some subarrays may. In other words, a partial mixed-level covering array (PMCA) is a set of
test rows that does not cover all the t-tuples yet. If you add N−n rows, the PMCA becomes
a complete mixed-level covering array. We generally do not know the value of N up front.
Analogously, a partial covering array, PCA(n/N ; t, k, v), shares the same definition as the
PCMA for a covering array. We use PMCAs in intermediate steps of the Building-Block
Algorithm (BBA) that we develop.
9
2. One-Row-at-the-Time Greedy Constructions
There are many greedy algorithms to construct covering arrays, and all of them
favor execution speed over accuracy. Three of the algorithms, the Automatic Efficient Test
Generator system (AETG) [7], Test Case Generator (TCG) [8], and Deterministic Density
Algorithm (DDA) [10], all build the covering arrays one row at the time. The general
approach is as follows:
1. Start with an empty test row, i.e., all of the factors are unfixed.
2. Select a factor to treat next.
3. Fix the factor you selected in step two to a level within the factor.
4. Repeat steps two and three until all the factors are fixed.
5. Add the complete test row to the test suite.
These steps are repeated N times until all the t-tuples are covered. Note that these algo-
rithms are strictly greedy; none of the algorithms undo a decision later.
This section gives a brief overview of AETG and TCG, and a more thorough overview
of DDA as it plays a larger role in the Building-Block Algorithm (BBA).
2.1. Automatic Efficient Test Generator. AETG is a random greedy algorithm
[7] that constructs multi-level covering arrays, MCA(N ; t, k, (v1, v2, . . . , vk)), one row at the
time. It generates M test-case candidates and then includes the test case that covers the
most new pairs in the test suite. If AETG has selected r test cases, next test case candidate,
r + 1 or T , is selected as follows:
1. Select a factor s and a value ν that appears in the most number of uncovered pairs.
Fix the level in the test case, i.e., Ts = fs,ν.
10
2. Let o2, o3, . . . , ok be a random order of the remaining k − 1 factors. The first factor
in the order is o1 = fs.
3. Say we have fixed a level for the first j ordered columns of T . Let fi be the factor and
fi,w be the level for the factor selected, for 1 ≤ i ≤ j and 1 ≤ w ≤ vi. For each level in
fj+1, count the number of uncovered pairs that appear with fj+1,w and the value at
Ti, for 1 ≤ i ≤ j. Fix column j +1 to level ν that appears with the maximum number
of uncovered pairs., i.e., Tj+1 = fj+1,ν. Each level of each factor is only considered
once per candidate test case. Also, any level is only compared to the previous j fixed
columns, o1, o2, . . . , oj , without considering what pairs may be covered ahead [7].
Although there is a large number of permutations of ordering the factors (k! to be
exact), AETG attempts to find at least one order that yields a higher number of newly
covered pairs than an average test row. The number of test cases were observed to grow
logarithmically in k for M = 50. The authors proved that a greedy algorithm could guar-
antee logarithmic growth if each new test case covered at least N/v2 new pairs, where N is
the remaining uncovered pairs and v is the number of levels in each factor. The algorithm
does not provide a guarantee that it will cover the maximum possible number of new pairs.
In fact, Colbourn et al. later proved that selecting a test case covering at least p pairs is
NP-complete [10]. We discuss this result in more detail later as we discuss DDA. The point
here is that AETG cannot guarantee logarithmic growth, only observe it.
2.2. Test Case Generator. Tung and Aldiwan introduced the TCG algorithm in
2000 [8], not long after the introduction of AETG. TCG’s main advantage over AETG is
that it is deterministic. The main steps of the algorithm are as follows:
1. The algorithm orders the factors decreasingly according to the of number levels for
11
each factor. Assign i to vi, where 1 ≤ i ≤ k such that v1 ≥ v2 ≥ · · · ≥ vk. Let M be
the maximum number of test-case candidates for each row, and set M = v1.
2. Generate as many as M test-case candidates. If the first j factors have been selected,
where 1 ≤ j ≤ k − 1, we select the (j + 1)th factor as follows: For each level in vj+1,
count the number of new uncovered pairs the level appear in with the previously j
selected levels. Select the level with the largest number of newly covered pairs. If
multiple levels cover the same number of pairs, add both partial arrays to the pool of
test-case candidates.
3. Select the test-case candidate that covers the most new pairs from the M candidates,
up to M candidates.
4. Repeat step two and three until all the pairs are covered.
The main difference between AETG and TCG is the factor ordering; AETG ran-
domly orders the factors (after fixing the very first factor) M times, where M is usually
fifty, and TCG deterministically orders the factors and then saves up to M ties, where M is
equal to the largest number of levels in a factor. However, as the two algorithms construct
each test row, they use a different approach when fixing a the factor involved in the most
number of uncovered pairs before fixing the factors involved in fewer uncovered pairs. This
heuristic merits some further discussion.
AETG handles this by first selecting a level for the factor with the most uncovered
pairs and then randomly ordering the rest of the factors a number of times, hoping to find a
test-case candidate that covers as close to the maximum number of pairs as possible. Note
that the first factor changes as test cases are added to the test suite since the factors are
covered unevenly.
12
TCG deals with this problem by initially defining an order that treats the factors
with the higher number of uncovered pairs first. However, this order does not change
during the course of the algorithm and thus ignoring the possible gain one would have if
the treatment of the factors were reordered after each test case.
Perhaps recognizing these heuristics, Cohen et al. [10] introduced the Deterministic
Density Algorithm (DDA) in 2004. This algorithm dynamically reorders the treatment of
factors as it adds new test rows.
2.3. Deterministic Density Algorithm. DDA constructs covering arrays one
row at the time, so the construction is similar to that of AETG and TCG. However, it adds
two more heuristics we call lookahead and dynamic factor ordering [10].
AETG and TCG selects a level based on the number of new pairs it would cover.
Specifically, when selecting a level for the ith factor, they only consider the interactions with
i−1 other factors. One could ask whether the interaction with the k−i other factors matter?
For example, if there are 100 factors, and we are currently selecting a level for the tenth
factor, the interaction with the last 90 is arguably more important than the interaction with
the first nine. DDA develops a heuristic that lets each greedy step look ahead and consider
the later interactions when selecting a level.
DDA also dynamically selects which factor to treat next. The factor is selected
based on the number of pairs it may cover with the treated factors and the number of
potential pairs it may cover with untreated factors. AETG calculates a similar heuristic for
the treatment of the first factor of each row (but uses a random order after that). TCG
also does this for the first few test rows, since it treats the factors with the most levels first,
but it never changes the order. This especially poses a problem for a covering array with
a constant number of levels. The pairs between the first factors get priority over the pairs
13
between the latter factors in the ordering. DDA consider the interactions with the later
factors when it selects which factor to treat next.
Let us develop the notion of density. Consider a covering array with k factors and
a factor i has vi number of levels. A local density between factors i and j is defined as
δi,j =ri,j
vivj, where ri,j is the number of uncovered pairs between the two factors. In plain
terms, the local density is the ratio of uncovered pairs versus total pairs between two factors.
Let the factor density for factor s be defined as
δs =∑
1≤i≤ki6=s
δi,s
At every step we treat the factor s with the highest density next.
To treat a specific factor we must select a level as the configuration for this compo-
nent in the test case. Let ρi,s,σ be the uncovered pairs between level σ in factor s and all
the levels in factor i times 1
vi. In other words, ρi,s,σ is the ratio of uncovered pairs between
a level and all levels in another factor and the initial number of pairs between these. Thus,
level density, ρs,σ, can be written as
ρs,σ =∑
1≤i≤ki6=s
ρi,s,σ
We use the level density as a heuristic to select a level when we treat a factor. The level
with the highest level density is added to the test case. Note that ρi,s,σ is equal to 1 if factor
i already has a selected level.
The densities as defined have some problems in practice, especially when they are
calculated for mixed covering arrays. Each pair between two factors, each with three levels,
contribute 1
9to the local density. A pair between two ten-level factors contribute 1
100to
the local density. The pairs in the first example are easier to cover than the pairs in the
14
second example, so we must adjust the density so each pair contribute an equal amount
to the local density. Let v2max be the largest number of levels for any factor. When we
calculate the local density for two factors, each with more than two levels, we replace the
denominator by vmax. When one of the factors have only one level or is already treated, we
use the denominator vmax. If both of the factors have only one level or are already treated,
we use 1 as the denominator.
The major steps of DDA then become:
1. Create a test row using one level from each of the k factors.
(a) Select factor s with the highest factor density δs.
(b) Select a level σ from fs with the highest level density ρs,σ and assign it to column
s of the current test row.
(c) Repeat steps (a) and (b) k times until there are no untreated factors.
2. Add the test row from step 1 to the test suite.
3. Repeat steps 1 and 2 until there are no uncovered pairs.
Unlike AETG and TCG, DDA does not create multiple test-case candidates per test
row in the final test suite. As the authors suggest, the time/accuracy tradeoff of keeping
several candidates merits further exploration [10].
3. IPO
Unlike the previous three greedy algorithms, IPO does not generate a test suite one
row at the time [9]. Instead, it first grows the partial test suite horizontally by adding one
factor at a time. In each step the test suite may also grow vertically if needed as shown in
Figure 1.
15
Figure 1. Concept figure of IPO growing the test suite horizontally and then vertically
IPO initially starts out by generating a pairwise test suite for the first two factors.
This test suite has size v1 × v2. Then it adds the third factor as a third column of the test
suite (see 3a in Figure 1), and selects levels for the third factor for each of the test rows
from the previous step. If there are still uncovered pairs between the first two factors and
the third factor, IPO vertically extends the test suite with additional rows to cover these
(see 3b in Figure 1). The horizontal and vertical growth continues until the algorithm has
included all the k factors, and the result is a complete covering array. The factors are also
added in a decreasing order of the number of levels, just like TCG, and IPO appears to have
some trouble with covering arrays where all the factors have the same number of levels.
The authors of IPO point out several advantages of IPO’s approach. Consider a
system with k factors, and the tester has developed a pairwise test suite for the system.
Later, the system grows, and the tester must add five factors. IPO can inherently extend the
current test suite to accommodate for these additional factors by using horizontal growth
and thus reuse the original test suite In another scenario, a test also has a complete test
16
Table 5. Orthogonal Latin squares of order three
0 1 21 2 02 0 1
0 2 11 0 22 1 0
(0,0) (1,2) (2,1)(1,1) (2,0) (0,2)(2,2) (0,1) (1,0)
suite. Later, the tester adds additional levels to some of the factors. IPO’s vertical growth
can also handle this situation.
As we show later, the Building-Block Algorithm has similar abilities to add both
factors and levels.
4. Combinatorial Constructions
Mathematicians have worked on different forms of combinatorial constructions
for centuries. Bose introduced a method for constructing orthogonal arrays of the size
OA(n2;n + 1, n, 2) [4]. More recently, Williams presented a more general algorithm called
TConfig that constructs covering arrays mainly from orthogonal arrays [3].
4.1. Orthogonal-Array Construction. A Latin Square is an n × n matrix on n
symbols such that each symbol appears exactly once in each row and each column. Two
Latin squares L1 = |aij | and L2 = |bij | of order n on n symbols 0, 1, . . . , n−1 are orthogonal if
every ordered pair occurs exactly once among the n2 pairs (aij , bij), where i = 0, 1, . . . , n−1
and j = 0, 1, . . . , n− 1. Table 5 shows two such Latin squares of order 3 and the nine pairs
these squares cover.
We can now start to see how Latin squares relate to orthogonal and covering arrays.
Each n2 ordered pair on the n symbols appear exactly once, which coincides with the exact
definition of an orthogonal array where every N × t subarray contains all the ordered pairs.
The example in Table 5 cover only two factors. We can assign row one through nine to each
17
of the elements by traversing from right to left and top to bottom over the elements in the
Latin square. The result would be orthogonal array OA(9; 2, 3, 2).
A set of mutually orthogonal Latin squares (MOLS) is a set of Latin squares where
every pair of squares are orthogonal. If we can find larger set of MOLS, we also can find
larger orthogonal arrays. A complete set of MOLS has n−1 Latin squares; in fact, no larger
set of MOLS exists. This paper uses one method that can construct OA(n2;n + 1, n, 2) if
n is a prime or a power of a prime. However, this would require a set of MOLS larger
than n − 1. Bose found he could add two additional squares that are mutually orthogonal
with all the other squares, but not Latin squares themselves. The technique uses properties
of Galois fields and calculates the orthogonal array directly using Galois-field arithmetic.
We simply present the algorithm here, but refer the reader to [4] for more details on this
construction.
Consider constructing an orthogonal array OA(v2; v + 1, v, 2), where v is a prime
power. The comma-separated columns of row (i, j), where 0 ≤ i < v and 0 ≤ j < v, may
be calculated using the following equations
(i, j) : {i, j, i + j, i + 2 · j, i + 3 · j, . . . , i + (v − 1) · j}
and each calculation is modulo v. We get N equals v2 as we loop over all values for i and
j, as shown in Table 6 for orthogonal array OA(9; 4, 3, 2). The last two columns of the
orthogonal array are Latin squares (as in Table 5) and form a complete set of MOLS. The
first two columns are mutually orthogonal with each other and the two other columns, but
they are not Latin squares.
If an orthogonal array exists with λ = 1, it always gives the lower bound for the
covering-array number, CAN(t, k, v), and provides a lower bound for other, seemingly
smaller, covering arrays. Consider an orthogonal array for k factors and v levels. You
18
Table 6. Construction of OA(9; 4, 3, 2) using Bose’s algorithm
(i, j) i j i + j i + 2 · j OrthogonalRow 0 ≤ i, j < 3 (mod 3) (mod 3) (mod 3) (mod 3) Array
1 (0, 0) 0 0 0 + 0 0 + 2 · 0 0 0 0 02 (0, 1) 0 1 0 + 1 0 + 2 · 1 0 1 1 23 (0, 2) 0 2 0 + 2 0 + 2 · 2 0 2 2 14 (1, 0) 1 0 1 + 0 1 + 2 · 0 1 0 1 15 (1, 1) 1 1 1 + 1 1 + 2 · 1 1 1 2 06 (1, 2) 1 2 1 + 2 1 + 2 · 2 1 2 0 27 (2, 0) 2 0 2 + 0 2 + 2 · 0 2 0 2 28 (2, 1) 2 1 2 + 1 2 + 2 · 1 2 1 0 19 (2, 2) 2 2 2 + 2 2 + 2 · 2 2 2 1 0
can remove up to k − 1 columns from the orthogonal array and still have a covering array
with 2 to k columns with a minimum number of rows. Similarly, you can remove levels from
k−2 factors s.t. at least two factors still have v levels, and still have a CA with the minimum
number of rows. The levels you remove are replaced by don’t-care positions. We call the
process of removing factors and levels a reduction of the orthogonal (or covering) array. For
example, OA(9; 4, 3, 2) in Table 6 can reduce to MCA(9; 2, 4, 2232) or CA(9; 2, 3, 3). We
use this in several results in Chapter 4 as building blocks for the building-block algorithm.
Further reductions, removing more than k − 2 factors or levels from more than k − 2 fac-
tors, result in a covering array where certain rows contain k − 1 or k don’t-care positions.
These rows can be removed, and thus we no longer have the same covering-array-number
guarantee, and a smaller covering array must exist.
4.2. TConfig. Williams and Probert use building blocks to construct covering ar-
rays in an algorithm they label TConfig [16]. The building blocks are orthogonal arrays and
reduced orthogonal arrays. In addition, TConfig utilizes some building blocks not based
19
on orthogonal arrays. These building blocks act as glue to cover the pairs not covered by
combining the orthogonal arrays.
Their basic idea is as follows: Place two identical orthogonal arrays side by side. If
the orthogonal array has k factors, this partial covering array has 2 · k factors. Each pair
within each orthogonal array is covered. Let i and j be indices into the columns of each
of the two orthogonal arrays. All pairs between columns i and j, where i 6= j, are covered
by the same argument as each of the pairs in the original array is covered. When i = j,
however, only pairs of the form (x, x) are covered, where 1 ≤ x ≤ v). There are are now
v2 − v uncovered pairs between the two columns, meaning at least v2 − v rows are required
to complete the covering array. TConfig can indeed complete the covering array in 2 ·v2 −v
rows total using a ”reduced array” in the last v2 − v rows.
TConfig has other generalizations that make it possible for the algorithm to handle
any covering array, mixed or fixed level. Our purpose is not to explain all these details, but
rather illustrate the use of building blocks to build larger covering arrays. The Building-
Block Algorithm uses building blocks similarly, but allows reordering of the rows within a
building block.
5. Heuristic Search
Heuristic-search techniques, also known as local-search techniques, have successfully
been applied to many combinatorial optimization problems, including the covering-array
problem. If you give these approaches enough run-time they usually find very small covering
arrays. This is an important criteria if each test case is expensive to run and time and
computation power is plentiful.
20
In this thesis we use both tabu-search and simulated-annealing results in the
Building-Block Algorithm.
5.1. Simulated Annealing. Stevens [6] used simulated annealing in his thesis to
construct smaller covering arrays. The algorithm starts out with a k×n matrix, where the
rows are test cases and the values in each column represent a level for a factor. Each test
row is filled with random configurations. This matrix will have an initial cost according to
the cost function; Stevens used the number of uncovered pairs as the cost. At each step,
the algorithm randomly selects a row and a column and change the value to a different
value within the levels of the corresponding factor. The proposed move is accepted with a
probability that decreases exponentially with time, i.e., the matrix cools until the probability
is zero. If the cost after a move is zero, the algorithm has found a covering array.
Since the algorithm cannot calculate the size of the resulting covering array (this is
NP-hard, see [10]), the algorithm has to guess an initial size. The size of the matrix stays
constant during one run. If the algorithm finds a covering array, the size is decreased by one
row and a new run is started. If a covering array cannot be found in a reasonable time, the
algorithm may have cooled too quickly or selected an initial size that was too optimistic.
5.2. Tabu Search. Tabu search also starts with an k×n matrix, just like simulated
annealing [2]. The matrix is initially filled with random values. The number of uncovered
tuples is the cost of the matrix at each step.
Nurmela uses a special heuristic neighborhood to propose moves [2]. First, randomly
select an uncovered tuple. Iterate over the n rows and find which rows only require changing
a single element to cover the tuple. These changes form the moves in the current neighbor-
hood. The algorithm tries each move and calculates the new cost after the proposed move.
21
The move with the highest reduction in cost is selected, unless the move is tabu. If several
moves has the same cost reduction, the algorithm selects one nontabu move at random.
A move is tabu if the element has changed in the last T moves. This prevents the
algorithm from looping and helps diversify the search. In his paper, Nurmela found that
relatively small values of T were sufficient, usually ten or fewer.
6. Criteria for Comparing Algorithms
As we have seen, many algorithms exist for creating covering arrays, and all are
designed with different goals in mind. Some algorithms are fast, others have more accuracy
(produce smaller covering arrays), and yet others offer more flexibility (seeding, uneven
strength coverage, avoids). This section defines these criteria and discusses some of the
algorithms in terms of these criteria.
A primary requirement, rather than criteria, is completeness of coverage [10], i.e.,
every pair (or tuple) must be covered. In most cases, we achieve this by creating a covering
array of strength two. A tester rarely have no additional constraints, though. Avoids are
tuples that are not necessarily needed in a test suite. For example, there may be certain
configurations that are possible, but that a company does not want to officially support.
Disallowed tuples are tuples that cannot appear in a test suite, e.g., a test configuration
may be physically impossible to run. If a tester has too many constraints, there may
be a problem with how the system is modeled, and the tester may want to go back and
reconsider the levels and factors. Another important criteria for an algorithm is the ability
to seed an algorithm with some initial test cases. The tester may have run these test cases
earlier, e.g., during requirements testing or as part of a regression test. AETG developed
two different input languages for constraints and discussed some possible implementation
22
approaches [7]. However, they failed to mention that some constraints make it impossible to
even complete one test case. This is an area that merits additional research in the future.
The combinatorial algorithms cannot handle constraints and seeds very well, since these
techniques rely on a certain internal structure of the tuples. Heuristic-search and greedy
algorithms’ approaches more easily lend themselves to handling constraints.
All the algorithms must make a classical tradeoff between efficient construction and
accuracy. For example, a heuristic-search algorithm may run for weeks to find a very
accurate test suite (i.e., with very few test cases). A greedy algorithm may find a test
suite in seconds, but the test suite may be significantly larger. Combinatorial constructions
are usually extremely fast and are very accurate in some cases. However, combinatorial
constructions often has constraints on the input, e.g., Bose’s algorithm requires v to be a
prime power. TConfig is more general, but often create overly large test suites for mixed-
level covering arrays, since it was designed with fixed-level covering arrays in mind. A tester
also has to make a similar tradeoff. If test execution is extremely expensive, the tester may
be willing to wait for weeks for a heuristic-search algorithm to find an accurate test suite.
However, if the tester has automated scripts that can quickly run on a software system, the
size of the test suite is of little importance.
Another desirable criteria is a worst-case guarantee on the size of the final test suite.
TConfig and DDA offer such a guarantee. Also, an algorithm’s average test-suite size should
be as low as possible. Finally, predictability may be an important criteria in some cases.
Results from heuristic-search or AETG may be impossible to reproduce and confirm later.
There are other, more complex, criteria that are still subject to additional research.
For example, an algorithm could create a covering array with uneven strength. Some factors
could be covered at strength two, others at strength three. One could take this a step further
23
and cover certain levels of a factor at a higher strength. A default value for a factor could
be covered at a higher strength if it is unlikely that a user will ever change this value.
From this section we can deduce that no algorithm that constructs covering arrays
will meet all testers’ requirements. A tester has to analyze the circumstances of the testing
environment and decide which algorithm handles the circumstances the best. Some of these
criteria may aid the tester in this decision.
CHAPTER 3
THE BUILDING-BLOCK ALGORITHM AND DENSITY
IMPLEMENTATION
In this chapter we introduce a new algorithm for constructing covering arrays, the
Building-Block Algorithm (BBA). A simple example first illustrates the approach. Next,
we define the algorithm formally and describe an implementation. Finally, we compare the
BBA approach to other algorithms for constructing covering arrays according to the criteria
discussed in Chapter 2.
1. An Example
Let us consider a small multi-level covering array such as MCA(N ; 2, 4, (2232)). To
construct this covering array, we break the problem into two covering arrays, A and B, of
the form MCA(6; 2, 2, (2131)) as shown in Table 7. By combining the two covering arrays
we can create a partial solution to MCA(N ; 2, 4, (2232)) with six rows. A complete covering
array, however, must have at least nine rows since there are nine pairs between factors f2
and f4. Thus, after combining the first six rows, we add extra rows at the end to cover the
remaining pairs.
What are the remaining pairs? There are two types of pairs to consider: The inner
pairs and the cross pairs. Covering arrays A and B cover the inner pairs, i.e., A covers
the pairs formed by f1 and f2 and B covers the pairs formed by f3 and f4. The remaining
pairs are cross pairs, i.e., the first symbol in a cross pair appear in f1 or f2 and the second
25
Table 7. Covering arrays A and B, both MCA(6; 2, 2, (2131))
A
Row f1 f2
1 0 a2 0 b3 0 c4 1 a5 1 b6 1 c
B
Row f1 f2
1 2 x2 2 y3 2 z4 3 x5 3 y6 3 z
symbol appear in f3 or f4. Unless A and B change, the inner pairs are guaranteed to be
covered in this construction and thus we must concentrate on covering the cross pairs.
How A and B are combined affects which cross pairs and how many cross pairs are
uncovered after the first six rows. First, consider simply combining row one in A with row
one in B, row two in A with row two in B, and so on until each row in A is combined with
the corresponding row in B. This operation preserves covering arrays A and B, so the inner
pairs are still covered in the first six rows as shown in Table 8. However, this partial covering
array leaves a very unsatisfactory set of uncovered cross pairs. In particular, between f2
and f4, only cross pairs (a, x), (b, y), and (c, z) are covered, leaving six uncovered cross
pairs between these two crucial factors. This approach forces a complete covering array to
twelve rows, a dismal result considering the problem. A second and better approach will
be discussed next.
The first approach did not consider covering the most important or hardest-to-cover
cross pairs first as it combined covering arrays A and B. If the first six rows cover six cross
pairs between factors two and four there would be only three pairs left to cover in this
factor pair. We achieve this by reordering the rows in A and B, such that any row in A can
combine with any row in B. Each row in A and each row in B appears exactly once, so the
26
Table 8. Partial covering array PMCA(6; 2, 4, (2232))
Row f1 f2 f3 f4
1 0 a 2 x2 0 b 2 y3 0 c 2 z4 1 a 3 x5 1 b 3 y6 1 c 3 z
Table 9. Covering array MCA(N ; 2, 4, (2232))
Covering Array
Row rA rB f1 f2 f3 f4
1 1 1 0 a 2 x2 5 2 1 b 2 y3 3 5 0 c 3 y4 4 6 1 a 3 z5 2 3 0 b 2 z6 6 4 1 c 3 x
7 - - ⋆ b 3 x8 - - ⋆ c 2 z9 - - ⋆ a ⋆ y
resulting partial covering array still has six rows and still covers the inner pairs in A and
B. We complete the covering array by adding three additional rows to cover the remaining
cross pairs as shown in Table 9. Columns rA and rB refer to the row numbers in covering
arrays A and B in Table 7.
By allowing reordering on the rows within A and B, the number of rows in the final
covering array is dramatically reduced. The Building-Block Algorithm’s fundamental idea
is to combine smaller covering arrays by reordering the rows and then append additional
rows to construct a complete covering array. The next section generalizes this idea more
formally.
27
2. The Building-Block Algorithm
The Building-Block Algorithm (BBA) consists of four major steps:
Step 1 Divide the set of k factors {f1, f2, . . . , fk} into ǫ factor groups
{e1, e2, . . . , eǫ}. Each factor is assigned exactly one factor group, such that
|e1|+ |e2|+ · · ·+ |eǫ| equals k and the set e1 ∪ e2 ∪ · · · ∪ eǫ equals {f1, f2, . . . , fk}.
Step 2 Construct a building block for each factor group ei. A building block
BB(ni; t, |ei|, ei) is a covering array with ni rows, strength t, with |ei| factors
specified by the factor group ei. All building blocks have the same strength as
the original covering array. A shorthand notation for a building block for a
factor group ei is bi.
Step 3 Construct a partial covering array, PMCA(η; t, k, (v1v2. . .vk)), by
combining the building blocks b1, b2, . . . , bǫ from step 2.
Step 4 Complete the covering array, MCA(N ; t, k, (v1v2. . .vk)), by adding
additional rows to the partial covering array in step 3.
In the example in section 1 we performed step one by dividing the factors
{f1, f2, f3, f4} into two factor groups, namely e1 = {f1, f2} and e2 = {f3, f4}. Next, we con-
structed the two building blocks, b1 = BB(6; 2, 2, {f1, f2}) and b2 = BB(6; 2, 2, {f3, f4}).
In step three we combined row one of A with row one of B, row two of A with row five of
B, and so on until each of the six rows were used once. In the last step we added test cases
(⋆, b, 3, x), (⋆, c, 2, z), and (⋆, a, ⋆, y) to complete the nine-row covering array.
At every step there are several decisions that must be made. The algorithm poses
these decisions; an implementation of the algorithm must answer how to make these deci-
sions.
28
In step one, an algorithm must decide ǫ and how to assign the k factors to the
ǫ factor groups. Some choices for ǫ appear undesirable, for example ǫ = k (one factor
per factor group), ǫ = 1 (all factors in one factor group), or any other ǫ that causes any
|ei| < t (a factor group has fewer than t factors, thus the building block will not cover any
inner tuples). The assignment of factors to factor groups greatly affect the size of each
building block in step two. For example, a good combinatorial construction or a previously
computed heuristic-search result may be available for step two if a factor group has certain
characteristics. Furthermore, depending on the implementation of step three, it may be
desirable that the size of the building blocks have approximately the same size.
The next step constructs building blocks b1 through bǫ. An implementation of the
BBA may freely select any method to construct these covering arrays. They may be con-
structed with accuracy as the main goal, or perhaps with a certain internal structure. One
could even construct different building blocks with different methods. In our implementa-
tion, smaller building blocks appear to have a positive effect on the overall size of the final
covering array.
The third step reorders the rows in each building block and combines them to create
a partial covering array. The most important decision here is how the rows are reordered
and combined so that the best mixture of cross tuples get covered. The building blocks
are rarely the same size, so the algorithm must decide, for building blocks bi and bj and
ni > nj , how to combine rows in bi with nonexisting rows in bj after nj rows have been
fixed. Also, there may be don’t-care positions in the building blocks; how and when will an
implementation fix these positions? They could be fixed in step three or they could be left
open and fixed before step four by going back through the rows.
29
Several challenges similar to those AETG [7], TCG [8], and DDA [10] face present
themselves here. First, the algorithm must determine in what order to consider the factor
groups when constructing a row. Next, the algorithm must select which test row within the
building block should be fixed. Other approaches could be considered; however, this paper
only considers this general greedy approach.
Finally, additional rows are appended in the fourth step to cover any uncovered
cross tuples not covered in the partial covering array from step three. Although the fourth
step should attempt to optimize the number of added test rows, it ultimately hinges on
the uncovered tuples left in step three. Any algorithm that does not expect a certain order
in the uncovered pairs is suitable. For example, TConfig may have problems efficiently
completing this step. AETG, DDA, TCG are greedy algorithms that would work, and
even heuristic-search approaches such as simulated annealing [6], tabu search [2], and hill
climbing [14] could complete the covering array.
All the steps of the algorithm is summarized in pseudo code in Figure 2.
3. Density Implementation
This section explains the details of one implementation of BBA. We discuss step
three in great detail, including how to reorder the rows in the building blocks, how to
handle don’t-care positions, and how to handle building blocks of different sizes. This
implementation does not discuss step one, but acknowledges that this is an area that merits
further study.
Both AETG and DDA naturally maps to BBA’s approach to constructing covering
arrays, especially in step three. A factor and a building block are similar, since the rows in
a building block can be viewed as the levels in a factor. Density seems like a natural choice
30
1 BBA(k, t, (f1, f2, . . . , fk))2
3 Declare MCA(N ; k, t, (v1v2. . .vk))4
5 Divide f1, f2, . . . , fk into ǫ factor groups e1, e2, . . . , ek
6
7 for i from 1 through ǫ8 Compute bi
9 endfor10
11 while (rows are unfixed)12 O = Select order to treat factor groups13 for j from 1 through ǫ14 Select row to fix in bOj
15 endfor16 Add newly created row to MCA17 endwhile18
19 Cover remaining uncovered tuples in MCA by adding additional test rows20 Return MCA
Figure 2. Pseudo-code for the Building-Block Algorithm
31
for several reasons: Density provides a logarithmic bound on the number of test rows [10].
Also, density is deterministic, thus giving it a slight edge when reproducibility is important.
Finally, DDA appears to have a slight advantage on execution time. The two algorithms
are essentially equal for other criteria such as accuracy, ability to find mixed-level covering
arrays, and allowing seeding.
This section therefore discusses how to implement the algorithm using density. In
this chapter we assume that existing algorithms construct the building blocks. The next
chapter presents results where DDA, combinatorial construction, and tabu search perform
step two. Throughout this paper we let DDA complete the covering array in step four.
3.1. Building-Block Density. Let us explore how we can select a heuristic based
on density for ordering the rows in the building blocks. We then use this heuristic to greedily
find and fix rows from each building block, constructing one row of the partial covering array
at the time. We can quickly identify that an exhaustive search over all the orders are not
feasible. There are∏ǫ
i=1ni! different combinations!
DDA [10] defines a local density between factors; see Section 2.3. We now define a
similar density for factor groups. Let wi be the sum of the levels in a factor group ei. The
number of cross pairs involving factor groups i and j is wjwj . For factor groups i and j, we
define the factor-group pair density to be Γi,j =Ri,j
wiwj, where Ri,j is the number of uncovered
pairs involving a value of a factor in factor group i and a value of a factor in factor group j.
In other words, the factor-group pair density is the fraction of cross pairs already covered
between the two factor groups. From this we can define a factor-group density for factor
group s to be
Γs =∑
1≤i≤ǫi6=s
Γi,s.
32
We fix the factor groups in descending factor-group density order, since a higher density
indicates that a factor group (and the factors within the factor group) is likely harder to
cover.
After selecting a factor group to fix we must select a row within the building block
for that factor group. Let Ψi,r,φ be 1
witimes the number of uncovered cross pairs involving
row φ in building block r and factor group i. The row density for row φ in building block
r is defined to be
Ψr,φ =∑
1≤i≤ǫi6=r
Ψi,r,φ
We select φ to maximize Ψr,φ and fix that building-block row in the appropriate row and
columns in the partial covering array before we select the next factor group to fix.
Colbourn et al. point out some concerns with their initial definition for density in
[10], and we have the same concerns here. The factor-group pair density is scaled by the
initial number of cross tuples between the two factor groups. If a pair of factor groups i
and j has wi = wj = 4, the scale factor is 16. Each cross pair covered reduces the density
by 1
16. A different pair of factor groups has a scale factor of 100. Each cross pair covered
between these two factor groups reduces the density by 1
100, a smaller amount per pair than
the first two factor groups. This runs counter to the notion that the first pair of factor
groups are easier to cover than the second pair. We address this by redefining the scaling
factor. Let wmax be the largest sum of the levels of any factor group. When calculating
the factor-group pair density for two unfixed factor groups, let the denominator be w2max.
When one factor group has already been fixed, let the denominator be wmax (and treat each
fixed factor as if it only had one level). Lastly, when both factor groups have already been
fixed, let the denominator be 1. We now correctly treat each pair equally from a density
standpoint and focus on the factor groups with a higher density of uncovered cross pairs.
33
3.2. Building Blocks with Different Sizes. Our implementation adds don’t-care
rows to smaller building blocks to make all the building blocks the same size. A don’t-care
row is a row with all don’t-care positions. First, find building-block r with the largest
number of rows and set nmax = nr. For each building block i, add nmax − ni don’t-care
rows. As the algorithm builds the partial array, fixing these positions covers more cross
pairs. Since all the inner pairs are already covered, these don’t-care positions simply cover
some of the inner pairs more than once, which still maintains the covering array property.
3.3. Don’t-Care Positions. Any implementation of the BBA should fix the don’t-
care positions in the building blocks. If these positions are not fixed, the size of the final
covering array is unnecessarily inflated. Our implementation fixes the don’t-cares after all
the rows of the building blocks have been combined, after step three but before step four.
The algorithm uses density (as DDA defines it) to fix these positions. After all the rows
are combined, there is a smaller chance that the new pairs that a don’t-care position covers
now, also are covered in later rows. Consider an algorithm that fixes the don’t-cares as the
building-block rows are combined. A certain level for a factor may appear attractive at the
time, so the algorithm fixes the position to this level. However, other rows remaining in the
building block cover these exact same cross pairs later, thus the earlier don’t-care position
was fixed without actually covering any pairs that would not be covered otherwise.
A problem with not fixing the don’t-care position is that we cannot calculate the row
and factor-group densities. The algorithm solves this problem by using density as defined
by DDA to fix the don’t-care positions in a temporary row and then calculates the row
densities. Before the algorithm adds the combined temporary row to the partial covering
array it “unfixes” the original don’t-care positions.
There is a caveat with this approach that is worth mentioning [17]. An unfixed
34
don’t-care position affects the densities later. For example, a factor-group density may be
higher because a don’t-care position was unfixed when it could otherwise have been fixed
and thus covered additional cross pairs. The algorithm may end up picking an undesirable
building-block row without taking into account that the don’t-care position earlier covers
some of the same cross pairs with a high probability when we later fix the position. This
area merits further exploration and is not treated here.
3.4. Implementation Summary. This section summarizes how the density im-
plementation performs each step of the algorithm. The user supplies step one in the argu-
ments. For example, the user may provide MCA(N ; t, k, (3446)), MCA(N ; t, k, (255366)),
and MCA(N ; t, k, (2632)). The algorithm treats these as building blocks and combines them
to create MCA(N ; t, k, (21136465366)). The building blocks are either supplied by the user
as seeds or the algorithm uses DDA1 to construct the building blocks. Before the third step
we add don’t-care rows to smaller building blocks until they all have a size of nmax. Next,
the algorithm iteratively selects a factor group and a row within the corresponding building
block and builds the partial covering array one row a the time. The next step fixes the
don’t-care positions in the partial covering array. Finally, DDA appends additional rows as
needed to cover the remaining cross pairs.
4. Comparison of the Building-Block Algorithm and Other Methods
The Building-Block Algorithm draws many ideas from existing algorithms in the
literature. In some ways, one can view the BBA as a natural evolution of these ideas. This
section discusses how these other algorithms are similar to the BBA and how they differ.
Williams et al. introduced TConfig [16], which uses building blocks to construct
covering arrays. As opposed to BBA, TConfig does not allow reordering of the rows in the
1Renee C. Turban at Arizona State University wrote the DDA implementation
35
building blocks. This inflexibility sometimes forces TConfig to create larger covering arrays.
Also, because the building blocks are specific combinatorial constructions, TConfig cannot
take advantage of previously computed building blocks. Overall, BBA and TConfig share
the idea of using building blocks, but the similarities end there.
IPO builds a covering array adding factors one at the time [9] . BBA could implement
a form of IPO by iteratively use one factor as a building block. However, this approach
would not cover any pairs in one of the building blocks since it only has one column. Both
IPO and BBA can easily add new factors to an existing covering array and both add new
rows to accomplish this. Since IPO handles the factors in a decreasing order of the number
of factors, IPO runs into trouble if the new factors have many levels compared to the factors
in the original covering array. BBA, on the other hand, handles the factor groups according
to density, so the order of the factors has less of an impact.
Finally, BBA draws many ideas from purely greedy algorithms such as AETG, TCG,
and DDA. All of these algorithms treat factors (or factor groups) in some order and then
fix a level for that factor. The covering array is built one row at the time, just like BBA.
All of them can take advantage of previously computed results. However, none of these
algorithms can take advantage of several previously computed results the way BBA does.
CHAPTER 4
COMPUTATIONAL RESULTS
In this chapter we present the computational results for BBA. There are three areas
of interest: the accuracy of BBA compared to IPO, AETG, TCG, TConfig, and especially
DDA; the execution speed of BBA compared to DDA; and a comparison of different invoca-
tions of the BBA using building blocks created by simulated annealing, tabu search, Bose’s
algorithm, and DDA. BBA uses DDA for step four to cover the remaining cross pairs in all
the results. Also, we only consider strength two throughout the chapter.
In order to save space and avoid redundant information, the tables use a shorthand
notation, where multiple building blocks represent a single, larger covering array. Let
β = (N, (v1v2 . . . vk)) be a single building block with k factors, N rows, strength two, and
vi levels for factor i, 1 ≤ i ≤ k. Note that β is also a covering array with N rows. BBA can
combine multiple βs together, i.e., β1, β2, . . . , βǫ, and collectively these form the covering
array β1β2 · · · βǫ. If several building blocks are the same, we use an exponent to signify a
multiple of a building block, e.g., β5 is a covering array where building block β is combined
five times. Thus, β = (9, 34) equals CA(9; 2, 4, 3) and β4 equals CA(N ; 2, 16, 3), where N
is the number of rows in the final covering array after all four building blocks are combined
and the uncovered cross pairs are covered using additional rows. The number of rows in the
covering array β is given by S(β) and the running time to construct β is given by T (β).
All the calculations were performed on a Pentium D 2.8 GHz processor, a dual-core
37
64-bit processor, with 2 GB of physical memory running Linux. Although the processor is
dual-core, only one core was used to perform the calculations.
1. Different Building Blocks
As we mentioned earlier, BBA can use covering-array results from other algorithms
as building blocks to build an even larger covering array. This section focuses on results
using building blocks from Bose’s algorithm, simulated annealing, tabu search, and DDA.
We compare all the computational results against results using DDA from [10], using
an implementation by Renee C. Turban at Arizona State University. Her implementation
contains several levels of tie breaking for both the factor density and level density. We
did not attempt to optimize the reported DDA results by trying different tie-breaking
techniques, but left all the tie-breaking settings at reasonable defaults. The tie-breaker for
factor density was to select the factor with the most pairs left, and if this was equal, to select
the first factor. The same tie-breakers were used for level density. Since our implementation
of BBA does not address tie-breaking, this appears to be a fair comparison.
1.1. Building Blocks from Bose’s Algorithm. Bose’s algorithm constructs very
small covering arrays (or orthogonal arrays) for a restricted set of inputs. Table 10 gives a
number of results using the direct construction and reduced orthogonal arrays (see Section
4.1) as building blocks. The top half of the table uses orthogonal arrays and the bottom
half uses reduced orthogonal arrays by inserting don’t-care positions.
The results in Table 10 indicate that BBA produces accurate results using good
building blocks compared to DDA. The first six results, which use orthogonal arrays as
building blocks, have an improvement of about 10 % or more over DDA. The last five BBA
results, which use reduced orthogonal arrays, have a smaller improvement over DDA than
38
Table 10. BBA (using Bose’s algorithm) versus DDA
Bose’s BBA using Bose’s DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)
1 (4, 23) 6 7 8 7 8 92 (9, 34) 15 15 18 17 19 213 (25, 56) 45 48 51 49 55 594 (49, 78) 91 99 104 97 111 1195 (121, 1112) 229 253 266 256 287 3116 (169, 1314) 325 350 370 367 412 446
7 (48, 68) 71 81 87 73 84 908 (49, 77) 91 98 103 94 105 1179 (120, 1010) 191 222 243 203 228 24910 (121, 119) 218 243 253 234 266 28711 (121, 118) 212 238 265 225 256 27912 (119, 99) 157 180 197 162 183 198
the first six results. These results suggest that accurate building blocks lead to smaller
covering arrays overall. The building block of size 119 for 99 in row ten of Table 10 does
not give a tight bound for this problem, and this fact carries over to the final covering-array
size for BBA. In this case, BBA constructs covering arrays comparable to that of DDA.
TConfig and BBA construct covering arrays with the same size for rows one through
four and row six for β2. BBA leaves v2−v pairs to cover between at least one pair of factors
after step three, just like TConfig, and the final size is thus 2 · v2 − v. Surprisingly, BBA
improved its solution for 1124 (in row five) by two rows over TConfig, from 231 to 229.
This improvement can be attributed to BBA’s step that allows reordering of the rows in
the building blocks. Colbourn reports a higher upper bound for 1124 [18], so we can report
that CAN(t, k, v) ≤ 229.
We see further evidence of these improvements in Table 11. In the first two rows of
this table, the building blocks for 68 and 77 are both reduced from the orthogonal array 78 of
size 49. TConfig would construct covering arrays with 91 rows for 7768 and 7868 (essentially
39
Table 11. Mixed-level BBA (using Bose’s algorithm) versus DDA
Building Blocksa BBA DDAb
Row Parameters β1 β2 β3 S(β1β2β3) S(β1β2β3)
1 7768 (48, 68) (49, 77) - 82 832 7868 (48, 68) (49, 78) - 79 863 101078 (49, 78) (120, 1010) - 139 1644 1112101078 (49, 78) (120, 1010) (121, 1112) 212 2325 11121020 (120, 1010) (120, 1010) (121, 1112) 240 2496 563423 (4, 23) (9, 34) (25, 56) 31 39
aThe first three rows use only two building blocksbInvoked directly with the parameters as the input
a reduced array from 716), but BBA improves these results to 82 and 79, respectively. Similar
improvements can be seen in Table 10, row seven, for inputs derived from the same source.
Table 11 also illustrates that BBA performs well with uneven building blocks.
1.2. Building Blocks from Heuristic Search. Heuristic-search algorithms gen-
erally produce covering arrays close to the upper bound when given enough execution time.
We obtained results from two heuristic-search algorithms, tabu search and simulated anneal-
ing, and used these results as building blocks for BBA. Overall, BBA’s results are slightly
better than DDA’s results.
BBA performs well in the first three rows of Table 12 against DDA, and slightly
better in the last two rows. Nurmela spent several CPU months calculating these building
blocks, and many give new best upper bounds. For example, in row five, Nurmela found a
covering array for 415317229 with 29 rows [2]. AETG found a covering array with 41 rows
using the same input. The tabu-search results for v = 3 are particularly good, and BBA
appears to use these good results to construct competitive covering arrays.
BBA continues to have slight edge over DDA using results from simulated annealing
as building blocks in Table 13. In rows one through four, simulated annealing found orthog-
onal arrays. Indeed, row three of Table 10 and row four of Table 13 are identical. Although
40
Table 12. BBA (using tabu search) versus DDA
Tabu BBAa DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)
1 (18, 330) 25 28 30 29 32 332 (20, 343) 28 30 32 31 34 343 (23, 383) 32 34 36 35 37 394 (21, 41339235) 30 34 36 32 35 375 (29, 415317229) 41 46 49 44 48 50
aThe building blocks originate from http://www.tcs.hut.fi/~kjnu/covarr.html (accessed 2/25/2006),a website mentioned in [2]
Table 13. BBA (using simulated annealing) versus DDA
SA BBAa DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)
1 (25, 53) 36 43 48 36 44 492 (25, 54) 40 48 52 42 49 533 (25, 55) 43 49 53 43 51 574 (25, 56) 45 48 51 49 55 595 (29, 57) 48 52 60 50 57 616 (36, 59) 52 60 65 55 61 677 (37, 510) 53 62 67 57 64 688 (40, 512) 57 65 70 59 67 719 (41, 513) 58 65 71 62 68 7310 (42, 514) 59 67 72 61 68 75
aMyra B. Cohen at University of Nebraska–Lincoln wrote the simulated annealing algorithm that pro-duced the covering arrays used as input in this table.
the building block for 56 have the same number of rows in each table, the actual order of
the rows is different. BBA still maintains the overall size of the final covering array. We
also combined a 56 building block from the simulated-annealing results and one 56 building
block from Bose’s algorithm and BBA still produced a 512 covering array with 45 rows.
1.3. Building Blocks from DDA. Finally, in Table 14 we used building blocks
constructed by DDA. A greedy algorithm such as DDA typically does not achieve the
same accuracy as Bose’s algorithm or heuristic search, so the building blocks in this table
are larger. Table 14 is divided in three sections, and each section corresponds to some of
41
Table 14. BBA (using DDA) versus DDA
DDA BBA DDA (combined)Row β S(β2) S(β3) S(β4) S(β2) S(β3) S(β4)
1 (9, 34) 15 15 18 17 19 212 (71, 77) 95 108 118 94 105 1173 (75, 78) 100 112 121 97 111 1194 (159, 1010) 211 237 258 203 228 2495 (127, 99) 166 189 204 162 183 198
6 (25, 330) 30 32 33 29 32 337 (28, 343) 32 34 36 31 34 348 (32, 383) 37 38 40 35 37 399 (28, 41339235) 34 38 41 32 35 3710 (35, 415317229) 45 50 52 44 48 50
11 (36, 56) 49 55 62 49 55 5912 (42, 58) 54 60 65 53 59 6413 (43, 510) 56 63 68 57 64 6814 (46, 511) 59 67 71 58 65 7015 (50, 513) 61 70 73 62 68 73
the same inputs as Bose’s algorithm, tabu search, and simulated annealing respectively.
Overall, BBA constructs covering arrays that are slightly larger than the covering arrays
DDA constructs. The only real exception is the first row, where DDA constructs the
orthogonal array for 34.
2. Comparison with Published Results
A common set of inputs for constructing covering arrays has emerged, and many
different methods have published results for at least some of these common inputs. Colbourn
et. al. published a table that summarized and compared results from AETG, TConfig, IPO,
TCG, and DDA [10]. We reproduced [10, Table 3] and added a column for BBA as shown
in Table 15.
BBA appears to be even with DDA and AETG and slightly better than IPO and
TConfig. TCG is also about even with BBA, although there are very few samples available
for comparison. IPO can easily reuse old test cases and add new factors to an existing
42
Table 15. Comparison with Published Results
Minimum Size of a Test Suite
Row Parameters BBA DDA AETG TCG IPO TConfig
1 513822 21 21 19 20
2 716151453823 45 43 45 45
3 514431125 28 27 30 30
4 6151463823 35 34 34 33
5 415317229 33 35 41 34 40
6 41339235 28 27 28 26 30
7 313 19 18 15 19 15
8 2100 14 15 10 15 14
9 440 40 43 49 40
10 4100 47 51 52 43
11 1020 191 201 180a 212 231
aor 198, see discussion in the text.
covering array. Since DDA can also perform the same operations, DDA appear to be a
better choice if these are important criteria for a specific application.
AETG delivers impressive results for 2100 and 1020. Other one-row-at-the-time
greedy algorithms such as DDA, TCG, and BBA construct dramatically larger covering
arrays for these inputs. We must differentiate between the commercially available (and
patented) AETG system and the greedy AETG algorithm described in [7]. Cohen et. al.
implemented AETG as described in [7] and they reported that their AETG implementation
constructed 1020 in 198 test rows [14]. The commercial product may include some simple
construction techniques, look-up tables of previously constructed covering arrays, and post
processing, none of which are included in the other greedy implementations. This must be
kept in mind when comparing the various greedy algorithms.
The composition of the factor groups in Table 15 affects the size of the final covering
array BBA constructs. Table 16 breaks down the input parameters into specific factor
groups. Some of the factor groups are repeated to match the factor groups to the input
43
Table 16. Building blocks for BBA for published results
Row Parameters β1 β2 β3 β4 β5
1 513822 (10, 5122) (9, 34) - - -
2 716151453823 (14, 7121) (12, 6121) (10, 5121) (16, 45) (9, 34)
3 514431125 (10, 5123) (16, 4421) (9, 34) (9, 3321)
4 6151463823 (16, 45) (12, 6121) (10, 5121) (9, 34) (8, 4121)
5 415317229 (16, 45) (9, 34) (9, 33) (4, 23) (4, 22)
6 41339235 (8, 4122) (9, 34) (9, 33) (4, 23) -
7 313 (9, 33) (9, 34) - - -
8 2100 (4, 23) (4, 22) - - -
9 440 (16, 45) - - - -
10 4100 (16, 45) - - - -
11 1020 (120, 1010) - - - -
parameters. For example, in row nine we repeat the factor group for 45 eight times to get
440. Most of the building blocks are orthogonal arrays or reduced orthogonal arrays. In
cases where there is not an easy mapping to an orthogonal array, we used DDA to create
the building block. These cases also illustrate that BBA can handle building blocks from
different sources, such as DDA and Bose’s algorithm.
3. Execution Time
Fast test-suite generation is one criteria we can use to evaluate a technique that
constructs covering arrays. All greedy methods are intended for speed, so TCG, IPO, and
DDA all are in the same range of timing, and AETG is slower due to the fact that it
generates M candidates per test row[10]. TConfig is faster since its method is almost a
direct construction. This section reports running times for BBA.
Table 17 compares BBA and DDA running times on the same computer system.
T (βx) means the time it took to construct covering array βx. It appears that BBA executes
faster, sometimes running five times faster than DDA. However, BBA is using precom-
44
Table 17. Execution time (BBA versus DDA) in seconds
BBA (s) DDA (s)β T (β2) T (β3) T (β2) T (β3)
(18, 330) 1.2 6.6 3.6 19(22, 368) 24 98 110 571
(169, 1314) 5.1 23 5.9 29(107, 815) 1.0 3.8 1.9 8.9
puted covering arrays as building blocks. For example, tabu search may have spent days
computing 815 with 107 test rows, and this time is not reflected in Table 17. BBA also
appear to have an advantage when the building blocks have fewer rows. DDA can construct
CA(295; 2, 14, 13), but in the table Bose’s algorithm constructed the same building block
with 169 rows. BBA spent 9.4 seconds constructing CA(N ; 2, 28, 13) using DDA’s building
block, almost doubling the comparable execution time in Table 17.
Although we cannot draw definitive conclusions from the Table 17, we can conclude
that BBA appears to have a reasonable execution time, especially when building blocks
near the upper bound is available.
CHAPTER 5
FUTURE WORK AND CONCLUSION
Although we have presented the Building-Block Algorithm (BBA) and our imple-
mentation as a fully functioning system, there are still many unanswered questions. This
chapter summarizes some of these questions and proposes possible new directions one could
explore. Finally, we draw some conclusions from our results and highlight some of the
advantages BBA has over other similar methods.
1. Future Work
As we mention in Section 3.4, our implementation does not select how to divide
the factors into factor groups. In some cases the user may have a specific set of factor
groups in mind, but as a general algorithm, this step should be performed automatically.
If this was done automatically, what should be the general goal? Should each building
block have approximately the same number of rows, and if so, how could one determine (or
approximate) that a specific division would reach that goal? If a few good building blocks
are available, one approach could divide the factor groups so they fit with the available
building blocks. This could have adverse effects on the remaining factor groups (if we
cannot find a perfect match). Further investigation is necessary to answer these questions.
BBA constructs competitive covering arrays when using building blocks constructed
by Bose’s algorithm and heuristic-search algorithms. BBA would be slow if it had to run
46
the heuristic-search methods in step two to get accurate building blocks, so a database of
the most accurate covering arrays available would benefit BBA. Each time BBA needs a
building block, it could first query a database before creating its own building block, thus
minimizing the time spent re-creating known results.
Our implementation of BBA did not include any tie-breakers for selecting factor
groups nor building-block rows. Tie breakers for factor groups may include random, first,
last, most uncovered pairs, etc. Ties between building-block rows could be broken using
several methods such as least-frequently used levels, most uncovered pairs between level and
single other factor, first, last, random, least-recently used levels. Some of these tie-breakers
would also require more calculations, so one must make a trade-off between execution time
and accuracy.
The definition of density in Section 3.1 may average over too many factors and
miss important information about cross pairs left between two factors. The local density
between two factors can still be high, even as the factor-group pair density is low compared
to other factor group pairs. This could lead to an effect similar to TConfig, where BBA
leaves v2 − v cross pairs between two factors after step three, thus forcing step four to add
v2 − v additional rows (see Section 1.1). We could either change the definition of density
or find a similar heuristic we could use as a tie breaker for building-block rows. Further
investigation should reveal how large of a problem this is, and from there one could make
a decision whether to change the definition of density or simply make it a tie breaker.
A larger question remains: Is density even the best way to match up building blocks?
We tried many different heuristics, mainly greedy strategies, but we found density to be the
most accurate out of these heuristics. However, other as yet unknown heuristics may prove
more efficient and accurate in the future. Heuristic-search techniques that move blocks of
47
configurations in a test row (instead of individual configurations) within the BBA framework
may be an interesting path to explore.
Although our current implementation for handling don’t-care positions appears rea-
sonable, we still have some concerns (see Section 3.3). We know that don’t-care positions
are important. For example, the result for β2 in row eleven of Table 15 increases from 191
to 206 if we do not fix the don’t-care positions at all. The impact of different methods
for handling don’t-care positions on the final covering-array size is largely unknown. If the
impact is rather small using other methods, we should prefer the fastest method.
DDA calculates step four in our implementation of BBA. We could use a heuristic-
search technique on these additional rows to improve the overall size of the covering array
without altering the rows fixed in step three. Depending on the user’s preference, the
heuristic search could be given a fixed time to improve the result in a post-processing step.
There is a limit to how many rows the post processing could optimize. Let Υ be the
maximum number of uncovered cross pairs between any two factor pairs after step three.
The post-processing step could not improve the result by more than N − (nmax + Υ) rows,
and in some cases even fewer rows.
Finally, the problem with avoids and disallowed pairs remains an issue for BBA (and
most other algorithms that construct covering arrays). This is largely an unsolved problem,
and we expect that extensive research will go into this area in the future.
2. Conclusion
The Building-Block Algorithm (BBA) is a general technique for constructing cover-
ing arrays one row at a time and is similar to techniques such as AETG, TCG, and DDA.
However BBA differs from these methods in that it combines several building blocks which
48
speed up the construction and potentially create more accurate covering arrays. TConfig
also uses building blocks, although in a stricter sense since TConfig does not reorder any
rows in its building blocks. The results show that BBA can take advantage of reordering
and thus constructs smaller covering arrays in many cases.
Furthermore, the results show that our implementation using density generally cre-
ates smaller covering arrays if the building blocks are more accurate. In a sense, BBA can
leverage the accuracy of its building blocks to create a smaller final test suite. This notion
should not necessarily be carried over to other possible future implementations since these
implementations may prefer other heuristics in their building blocks.
Finally, we can conclude that a test engineer can use BBA to leverage test cases from
part of a system to later test the complete system. IPO can grow a test suite by adding
factors, but it cannot take test suites from n subsystems and use those test suites in the
final test suite for the entire system. This attribute of BBA can save the test engineer time
and money, which are often scarce in software testing.
REFERENCES
[1] D. R. Kuhn and M. J. Reilly, “An investigation of the applicability of design of ex-periments to software testing,” in SEW ’02: Proceedings of the 27th Annual NASAGoddard Software Engineering Workshop (SEW-27’02). Washington, DC, USA: IEEEComputer Society, 2002, p. 91.
[2] K. J. Nurmela, “Upper bounds for covering arrays by tabu search,” Discrete AppliedMath., vol. 138, no. 1-2, pp. 143–152, 2004.
[3] A. W. Williams and R. L. Probert, “A practical strategy for testing pair-wise coverageof network interfaces,” in Proc. Seventh Intl. Symp. on Software Reliability Engineering,1996, pp. 246–254.
[4] R. C. Bose, “On the application of the properties of Galois fields to the constructionof hyper Graeco-Latin squares,” Sankhya, no. 3, pp. 323–338, 1938.
[5] J. Stardom, “Metaheuristics and the search for covering and packing arrays,” Master’sthesis, Simon Fraser University, 2001.
[6] B. Stevens, “Transversal covers and packings,” Ph.D. dissertation, University ofToronto, Toronto, 1998.
[7] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG system: Anapproach to testing based on combinatorial design,” IEEE Transactions on SoftwareEngineering, vol. 23, no. 7, pp. 437–444, 1997.
[8] Y. Tung and W. S. Aldiwan, “Automating test case generation for the new generationmission software system,” in IEEE Aerospace Conference, 2000, pp. 431–437.
[9] K. C. Tai and Y. Lei, “A test generation strategy for pairwise testing,” IEEE Trans-actions on Software Engineering, vol. 28, no. 1, pp. 109–111, 2002.
[10] C. J. Colbourn, M. B. Cohen, and R. C. Turban, “A deterministic density algorithmfor pairwise interaction coverage,” in Proceedings of the International Conference onSoftware Engineering (SE 2004), February 2004, pp. 345–352.
[11] C. J. Colbourn, “Combinatorial aspects of covering arrays,” Le Matematiche (Catania),no. 58, pp. 121–167, 2004.
[12] A. Hedayat, N. Sloane, and J. Stufken, Orthogonal Arrays. New York: Springer-Verlag,1999.
50
[13] R. Mandl, “Orthogonal latin squares: an application of experiment design to compilertesting,” Commun. ACM, vol. 28, no. 10, pp. 1054–1058, 1985.
[14] M. B. Cohen, P. B. Gibbons, W. B. Mugridge, and C. J. Colbourn, “Constructingtest suites for interaction testing,” in ICSE ’03: Proceedings of the 25th InternationalConference on Software Engineering. Washington, DC, USA: IEEE Computer Society,2003, pp. 38–48.
[15] M. Chateauneuf and D. Kreher, “On the state of strength-three covering arrays,”Journal of Combinatorial Designs, vol. 10, no. 4, pp. 217–238, 2002.
[16] A. W. Williams and R. L. Probert, “A measure for component interaction test cover-age,” in Proc. of the ACS/IEEE Intl. Conf. on Computer Systems and Applications.Washington, DC, USA: IEEE Computer Society, 2001, p. 304.
[17] C. J. Colbourn, Arizona State University, 2006, private communication.
[18] ——, “Strength two covering arrays: existence tables and projection,” Discrete Math-ematics, to appear (accepted December 2005).