Actual Test Coverage for Embedded Systems
Transcript of Actual Test Coverage for Embedded Systems
Actual Test Coverage for Embedded Systems
Mark TimmerUniversity of Twente
14th Dutch Testing DayNovember 27, 2008
Contents
1 Introduction
2 Evaluating actual coverage
3 Predicting actual coverage
4 Test suites
5 Example
6 Conclusions and future work
Actual Test Coverage for Embedded Systems Nov. 27, 2008 2 / 24
Introduction – coverage
Why coverage?
Testing is inherently incomplete
Testing does increase our confidence in the system
A notion of quality of a test suite is necessary
Coverage: ‘amount’ of specification / implementationexamined by a test suite
•
•
•
••
••
•
•
•
••
•
Coverage: 613 = 46%.
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 3 / 24
Introduction – coverage
Why coverage?
Testing is inherently incomplete
Testing does increase our confidence in the system
A notion of quality of a test suite is necessary
Coverage: ‘amount’ of specification / implementationexamined by a test suite
•
•
•
••
••
•
•
•
••
•
Coverage: 613 = 46%.
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 3 / 24
Introduction – coverage
Why coverage?
Testing is inherently incomplete
Testing does increase our confidence in the system
A notion of quality of a test suite is necessary
Coverage: ‘amount’ of specification / implementationexamined by a test suite
•
•
•
••
••
•
•
•
••
•
Coverage: 613 = 46%.
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 3 / 24
Introduction – coverage
Why coverage?
Testing is inherently incomplete
Testing does increase our confidence in the system
A notion of quality of a test suite is necessary
Coverage: ‘amount’ of specification / implementationexamined by a test suite
•
•
•
••
••
•
•
•
••
•
Coverage: 613 = 46%.
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 3 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Existing approaches to coverage
Early work on coverage
Statement coverage Path coverage
Limitations: - all faults are considered of equal severity- no probabilities- syntactic point of view
Recipe 1: vegetable soup
Chop an union
Slice carrots and mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 45 · 10 = 8
Recipe 2: vegetable soup
Chop an union
Slice a few carrots
Slice a mushroom
Boil one liter of water
Put everything in the water
Wait a while
Quality: 46 · 10 = 6.7
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 4 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct?
coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee!
10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct?
tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Semantic coverage
Starting point for my work: semantic coverage
Previous work by Brandan Briones, Brinksma and Stoelinga
System considered as black box
Semantic point of view
Fault weights
Labelled transition systems
s0s1 s2
20ct? 10ct?
coffee! tea!
δ
20ct? coffee! 10ct? tea!
Test cases
fail fail
x10ct?
δ xtea! coffee!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 5 / 24
Introduction – Fault weights and coverage measures
fail fail
fail pass fail
x10ct?
δ xtea! coffee!
20ct?
δ xcoffee! tea!
4 5
3 2
Absolute potential coverage
4 + 5 + 3 + 2 = 14
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 6 / 24
Introduction – Fault weights and coverage measures
fail fail
fail pass fail
x10ct?
δ xtea! coffee!
20ct?
δ xcoffee! tea!
4 5
3 2
Absolute potential coverage
4 + 5 + 3 + 2 = 14
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 6 / 24
Introduction – Fault weights and coverage measures
fail fail
fail pass fail
x10ct?
δ xtea! coffee!
20ct?
δ xcoffee! tea!
4 5
3 2
Absolute potential coverage
4 + 5 + 3 + 2 = 14
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 6 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Introduction - Limitations of potential coverage
pass fail fail pass
coffee! tea!
tea! xcoffee!xcoffee! xtea!xtea! coffee!
10 15
Absolute potential coverage
10 + 15 = 25
Some remarks
Not all faults can bedetected at once
Single executions cover onlysome faults
Executing more often couldincrease coverage
How many executions areneeded?
Necessary to includeprobabilities!
Actual Test Coverage for Embedded Systems Introduction Nov. 27, 2008 7 / 24
Overview of actual coverage
Actual coverage
1 Probabilistic execution model:
Branching probabilities (pbr)Conditional branching probabilities (pcbr)
2 Evaluating actual coverage:
Calculating the actual coverage of a given executionor sequence of executions
3 Predicting actual coverage:
Predicting the actual coverage a test case or testsuite yields.
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 8 / 24
Overview of actual coverage
Actual coverage
1 Probabilistic execution model:
Branching probabilities (pbr)Conditional branching probabilities (pcbr)
2 Evaluating actual coverage:
Calculating the actual coverage of a given executionor sequence of executions
3 Predicting actual coverage:
Predicting the actual coverage a test case or testsuite yields.
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 8 / 24
Overview of actual coverage
Actual coverage
1 Probabilistic execution model:
Branching probabilities (pbr)Conditional branching probabilities (pcbr)
2 Evaluating actual coverage:
Calculating the actual coverage of a given executionor sequence of executions
3 Predicting actual coverage:
Predicting the actual coverage a test case or testsuite yields.
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 8 / 24
Fault coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Fault coverage
A fault is covered by an execution if the execution gives usinformation about whether the fault is present or absent
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 9 / 24
Fault coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Fault coverage
A fault is covered by an execution if the execution gives usinformation about whether the fault is present or absent
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 9 / 24
Fault coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Fault coverage
A fault is covered by an execution if the execution gives usinformation about whether the fault is present or absent
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 9 / 24
Fault coverage
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10 15
Fault coverage
A fault is covered by an execution if the execution gives usinformation about whether the fault is present or absent
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 9 / 24
Fault coverage
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(0.6) tea! xcoffee! (0.4)
pass fail
coffee!
(0.99) tea! xcoffee! (0.01)
Conditional branching probabilities pcbr
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 10 / 24
Fault coverage
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(0.6) tea! xcoffee! (0.4)
pass fail
coffee!
(0.99) tea! xcoffee! (0.01)
Conditional branching probabilities pcbr
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 10 / 24
Fault coverage
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(0.6) tea! xcoffee! (0.4)
pass fail
coffee!
(0.99) tea! xcoffee! (0.01)
Conditional branching probabilities pcbr
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 10 / 24
Fault coverage
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(0.6) tea! xcoffee! (0.4)
pass fail
coffee!
(0.99) tea! xcoffee! (0.01)
Conditional branching probabilities pcbr
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 10 / 24
Fault coverage
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(1.0) tea! xcoffee! (0.0)
pass fail
coffee!
(0.6) tea! xcoffee! (0.4)
pass fail
coffee!
(0.99) tea! xcoffee! (0.01)
Conditional branching probabilities pcbr
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 10 / 24
Fault coverage
Fault coverage
1 If a fault is shown present, it is completely covered
2 If a fault is shown absent, it is partially covered.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Fault coverage coffee! coffee!
Fault coverage tea! tea!
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 11 / 24
Fault coverage
Fault coverage
1 If a fault is shown present, it is completely covered
2 If a fault is shown absent, it is partially covered.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Fault coverage coffee! coffee!
Fault coverage tea! tea!
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 11 / 24
Fault coverage
Fault coverage
1 If a fault is shown present, it is completely covered
2 If a fault is shown absent, it is partially covered.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Fault coverage coffee! coffee!10
Fault coverage tea! tea!0
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 11 / 24
Fault coverage
Fault coverage
1 If a fault is shown present, it is completely covered
2 If a fault is shown absent, it is partially covered.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Fault coverage coffee! coffee!0
Fault coverage tea! tea!15
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 11 / 24
Fault coverage
Fault coverage
1 If a fault is shown present, it is completely covered
2 If a fault is shown absent, it is partially covered.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Fault coverage coffee! coffee!4− 6.4− 7.8− 8.7
Fault coverage tea! tea!0
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 11 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) =
4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4
faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) =
4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8
faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0
faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) =
1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Evaluating actual coverage
Actual coverage
Actual coverage of an execution or sequence of executions:The sum of all fault coverages
fail
fail pass fail fail pass fail
xc!b! d!
a? a?
xb!d! c! xc!b! d!
7
(0.2)
4
(0.5)
6
(0.8)
9
(0.6)
2
(0.7)
faultCov(b! a? d!) = 4faultCov(b! a? c!) = 4.8faultCov(d! a? b!) = 0faultCov(d! a? d!) = 0faultCov(c!) = 1.4
absCov = 10.2
Actual Test Coverage for Embedded Systems Evaluating actual coverage Nov. 27, 2008 12 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P0.015 0.150
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P0.015 0.150
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P
10
0.015 0.150
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P
10
0.015 0.150
15
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P
10
0.015 0.150
154
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov
P
10
0.015 0.150
1546
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov P10 0.015
0.150
15 0.0054 0.7356 0.245
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov P ×10 0.015 0.15015 0.005 0.0754 0.735 2.9406 0.245 1.470
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Predicting actual coverage
Actual coverage distribution of a test case
The actual coverage distribution of a test case predicts its actualcoverage.
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
absCov P ×10 0.015 0.15015 0.005 0.0754 0.735 2.9406 0.245 1.470
E(absCov) = 4.635
+
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 13 / 24
Branching probabilities
Branching probabilities
The branching probabilities pbr describe how the implementation isexpected to behave
pass fail fail pass
coffee! tea!
tea! xcoffee! xtea! coffee!
10
(0.4)
15
(0.4)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 14 / 24
Branching probabilities
Branching probabilities
The branching probabilities pbr describe how the implementation isexpected to behave
pass fail fail pass
(0.75) coffee! tea! (0.25)
(1.0) tea! xcoffee! (0.0) x(0.0) tea! coffee! (1.0)
10
(0.4)
15
(0.4)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 14 / 24
Branching probabilities
Branching probabilities
The branching probabilities pbr describe how the implementation isexpected to behave
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 14 / 24
Sequences of executions
Suppose we perform three executions of
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
Possible observation: [blue, blue, red]
Actual coverage: 15 + 6.4 = 21.4
Many observations possible: O(|exec|n)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 15 / 24
Sequences of executions
Suppose we perform three executions of
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
Possible observation: [blue, blue, red]
Actual coverage: 15 + 6.4 = 21.4
Many observations possible: O(|exec|n)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 15 / 24
Sequences of executions
Suppose we perform three executions of
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
Possible observation: [blue, blue, red]
Actual coverage: 15 + 6.4 = 21.4
Many observations possible: O(|exec|n)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 15 / 24
Sequences of executions
Suppose we perform three executions of
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
Possible observation: [blue, blue, red]
Actual coverage: 15 + 6.4 = 21.4
Many observations possible: O(|exec|n)
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 15 / 24
Expected actual coverage for a sequence of executions
Theorem
E(actCovn) =∑
σa∈errt
f (σa) ·
((1− (1− pto(σa))n) · 1 +
n∑k=0
(n
k
)pto(σ)k(1− pto(σ))n−k ·
(1− pbr(a | σ))k ·
(1− (1− pcbr(a | σ)k)))
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 16 / 24
Expected actual coverage for a sequence of executions
Theorem
E(actCovn) =∑
σa∈errt
f (σa) ·
((1− (1− pto(σa))n) · 1 +
n∑k=0
(n
k
)pto(σ)k(1− pto(σ))n−k ·
(1− pbr(a | σ))k ·
(1− (1− pcbr(a | σ)k)))
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 16 / 24
Expected actual coverage for a sequence of executions
Theorem
E(actCovn) =∑
σa∈errt
f (σa) ·
((1− (1− pto(σa))n) · 1 +
n∑k=0
(n
k
)pto(σ)k(1− pto(σ))n−k ·
(1− pbr(a | σ))k ·
(1− (1− pcbr(a | σ)k)))
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 16 / 24
Asympotical behaviour of actual coverage
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
0 10 20 30 40 500
5
10
15
20
25
Number of executions
Exp
ecte
d ac
tual
cov
erag
e
Theorem
limn→∞
E(actual coveragen) = potential coverage
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 17 / 24
Asympotical behaviour of actual coverage
pass fail fail pass
(0.75) coffee! tea! (0.25)
(0.98) tea! xcoffee! (0.02) x(0.02) tea! coffee! (0.98)
10
(0.4)
15
(0.4)
0 10 20 30 40 500
5
10
15
20
25
Number of executions
Exp
ecte
d ac
tual
cov
erag
eTheorem
limn→∞
E(actual coveragen) = potential coverage
Actual Test Coverage for Embedded Systems Predicting actual coverage Nov. 27, 2008 17 / 24
Actual coverage for test suites
Very similar to actual coverage for test cases:sum all the fault coverages
Take into account in how many test cases an erroneous traceis contained
Again, an efficient formula for the expected actual coverageexists
Theorem
limn→∞
E(actual coveragen) = potential coverage
Actual Test Coverage for Embedded Systems Test suites Nov. 27, 2008 18 / 24
Example – chemical dispenser
s0s1 s2
s3 s4
s7
s5 s6
s8
x? y?
e! a!
w! b!
c! a!
w! a!
a! b!
δ
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 19 / 24
Example – chemical dispenser
s0s1 s2
s3 s4
s7
s5 s6
s8
x? y?
e! a!
w! b!
c! a!
w! a!
a! b!
δ
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 19 / 24
Example – chemical dispenser
s0s1 s2
s3 s4
s7
s5 s6
s8
x? y?
(0.76) e! a! (0.19)
(0.9375) w! b! (0.9375)
(0.475) c! a! (0.475)
(0.9375) w! a! (0.9375)
a!(0.9375)
b!(0.9375)
(0.9375)δ
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 19 / 24
Example – chemical dispenser
pass pass pass pass
xx? (1.0)
(0.76) e! a! (0.19)
xw! (0.9375)
xa! (0.9375)
xx? (1.0)
(0.76) e! a! (0.19)
xw! (0.9375) xb! (0.9375)
xa! (0.9375) xa! (0.9375)
xb! (0.9375)
xa! (0.9375)
xx? (1.0)
(0.76) e! a! (0.19)
xw! (0.9375) xb! (0.9375)
xa! (0.9375) xa! (0.9375)
(a) Test case t1 (pot.cov.: 1938)
pass pass pass pass
xy? (1.0)
(0.475) c! a! (0.475)
xw! (0.9375)
xb! (0.9375)
xy? (1.0)
(0.475) c! a! (0.475)
xw! (0.9375) xa! (0.9375)
xb! (0.9375) xb! (0.9375)
xa! (0.9375)
xb! (0.9375)
xy? (1.0)
(0.475) c! a! (0.475)
xw! (0.9375) xa! (0.9375)
xb! (0.9375) xb! (0.9375)
(b) Test case t2 (pot.cov.: 1938)
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 20 / 24
Prediction of actual coverage
E(A1t1) = 197.0 E(A1
t2) = 156.8
E(A5t1) = 729.1 E(A5
t2) = 639.8
E(A10t1 ) = 1076.9 E(A10
t2 ) = 1032.1
E(A50t1 ) = 1704.6 E(A50
t2 ) = 1848.0
E(A250t1 ) = 1917.7 E(A250
t2 ) = 1938.0
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 21 / 24
Prediction of actual coverage
E(A1t1) = 197.0 > E(A1
t2) = 156.8
E(A5t1) = 729.1 > E(A5
t2) = 639.8
E(A10t1 ) = 1076.9 > E(A10
t2 ) = 1032.1
E(A50t1 ) = 1704.6 < E(A50
t2 ) = 1848.0
E(A250t1 ) = 1917.7 < E(A250
t2 ) = 1938.0
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 21 / 24
Simulation and evaluation actual coverage
t1 t2
n E(Ant1
) Sim. std. E(Ant2
) Sim. std.
1 197.0 213.3 50.1 156.8 155.1 60.8
5 729.1 762.1 84.0 639.8 629.7 107.0
10 1076.9 1112.6 104.8 1032.1 1013.3 114.8
50 1704.6 1743.3 62.4 1848.0 1831.2 39.5
250 1917.7 1925.8 11.2 1938.0 1938.0 0.0
t1 t2
n E(Ant1
) Sim. std. E(Ant2
) Sim. std.
1 197.0 229.1 48.4 156.8 174.7 52.0
5 729.1 813.9 56.5 639.8 711.6 89.4
10 1076.9 1167.9 94.8 1032.1 1133.6 92.9
50 1704.6 1757.3 62.9 1848.0 1890.7 19.4
250 1917.7 1926.0 11.1 1938.0 1938.0 0.0
Actual Test Coverage for Embedded Systems Example Nov. 27, 2008 22 / 24
Conclusions and future work
Main results
New notion of coverage: actual coverage
Evaluating actual coverage of a given execution
Predicting actual coverage of a test case or test suite
For more details, see my Master’s Thesis(wwwhome.cs.utwente.nl/~timmer)
Directions for future work
Validation of the framework: tool support, case studies
Dependencies between errors
Accuracy of approximations
On-the-fly test derivation
Actual Test Coverage for Embedded Systems Conclusions and future work Nov. 27, 2008 23 / 24
Conclusions and future work
Main results
New notion of coverage: actual coverage
Evaluating actual coverage of a given execution
Predicting actual coverage of a test case or test suite
For more details, see my Master’s Thesis(wwwhome.cs.utwente.nl/~timmer)
Directions for future work
Validation of the framework: tool support, case studies
Dependencies between errors
Accuracy of approximations
On-the-fly test derivation
Actual Test Coverage for Embedded Systems Conclusions and future work Nov. 27, 2008 23 / 24
Conclusions and future work
Main results
New notion of coverage: actual coverage
Evaluating actual coverage of a given execution
Predicting actual coverage of a test case or test suite
For more details, see my Master’s Thesis(wwwhome.cs.utwente.nl/~timmer)
Directions for future work
Validation of the framework: tool support, case studies
Dependencies between errors
Accuracy of approximations
On-the-fly test derivation
Actual Test Coverage for Embedded Systems Conclusions and future work Nov. 27, 2008 23 / 24