Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry...
Transcript of Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry...
![Page 1: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/1.jpg)
Get Yourself Covered
– An Introduction to Test Coverage
Barry Savell
Associate Consultant
![Page 2: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/2.jpg)
What is test coverage?
• A map of where we’ve been
– What has been tested
– And what hasn’t
• Dynamic testing
– Actually executing code
• Testing level
– System testing
• Realistically can’t cover everything
– Component testing
![Page 3: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/3.jpg)
Why should you care about coverage?
• Risks – To safety
– To reputation
• Show S/W is fit for purpose
• Verify the components – Have the brakes been checked?
– What needs to be checked?
– What has been checked?
• All of the components – Even the simple parts can fail
– It only takes one O-ring …
![Page 4: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/4.jpg)
Types of coverage
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
• Various measures
• Differing thoroughness
• Example
– Simple code
– Logic flow diagram
![Page 5: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/5.jpg)
Types of coverage
• Function coverage
– Function called
– p1=0
– Statements may be missed
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
X
![Page 6: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/6.jpg)
Types of coverage
• Function coverage
• Statement coverage
– Each statement executed
• p1=1, p2=1
– May miss null branches
– Commonest coverage
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
X
![Page 7: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/7.jpg)
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
– Each branch executed
• p1=1, p2=1
– Including null branches
• p1=0, p2=1
– More test cases
– p2>0 is not tested
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
![Page 8: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/8.jpg)
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
• Condition coverage
– Individual Boolean sub-clauses
– Must evaluate True and False
• p1=1, p2=0
• p1=0, p2=1
– Not all branches
p1>
0
Result = p1 p2
>0
X
![Page 9: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/9.jpg)
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
• Condition Coverage
• Condition / Decision
coverage
– Condition and Branch
• p1=1, p2=0
• p1=0, p2=1
• p1=1, p2=1
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>
0
Result = p1 p2
>0
![Page 10: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/10.jpg)
Modified Condition / Decision Coverage
• Condition / decision coverage
– All conditions tested
– All branches tested
– Still not a great test
• MC/DC
– Modified Condition / Decision Coverage
– Show each condition independently
affects outcome
If (a or b) and c then
branch 1
Else
branch 2
Endif
Condition/decision coverage tests
a=true, b=true, c=true - branch 1
a=false, b=false, c=false - branch 2
a, b and c are boolean conditions (eg p1 > 0)
![Page 11: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/11.jpg)
Modified Condition / Decision Coverage
• Vary only condition a a=true, b=false, c=true -> branch 1
a=false, b=false, c=true - branch 2
• Vary only condition b a=false, b=false, c=true - branch 2
a=false, b=true, c=true - branch 1
• Vary only condition c a=false, b=true, c=true - branch 1
a=false, b=true, c=false - branch 2
• More tests – But some are duplicates
– Only four tests 1: a=true, b=false, c=true - branch 1
2: a=false, b=false, c=true - branch 2
3: a=false, b=true, c=true - branch 1
4: a=false, b=true, c=false - branch 2
If (a or b) and c then
branch 1
Else
branch 2
Endif
b
branch 1
c
branch 2
a
1,3 4
2
3,4
2,3,4
1
![Page 12: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/12.jpg)
Modified Condition / Decision Coverage
• Condition / decision coverage
– All conditions tested
– All branches tested
– Still not a great test
• MC/DC
– Modified Condition / Decision Coverage
– Show each condition independently
affects outcome
– More tests
– Safety related / critical systems
• Multiple condition coverage
– All combinations tested
– Many tests
If (a or b) and c then
branch 1
Else
branch 2
Endif
Condition/decision
a=true, b=true, c=true - branch 1
a=false, b=false, c=false - branch 2
MC/DC tests
a=true, b=false, c=true - branch 1
a=false, b=false, c=true - branch 2
a=false, b=true, c=true - branch 1
a=false, b=true, c=false - branch 2
Multiple condition coverage tests
a=false, b=false, c=false - branch 2
a=false, b=false, c=true - branch 2
a=false, b=true, c=false - branch 2
a=false, b=true, c=true - branch 1
a=true, b=false, c=false - branch 2
a=true, b=false, c=true - branch 1
a=true, b=true, c=false - branch 2
a=true, b=true, c=true - branch 1
![Page 13: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/13.jpg)
Other coverage types
• Loop coverage
• Entry / exit coverage
• Parameter value coverage
• State coverage (for FSMs)
• LCSAJ / JJ-path coverage
![Page 14: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/14.jpg)
How is coverage measured
• Modify code
– To report execution during tests
• Instrument your own code
– Manual
– Conditional compilation
• Use a test tool
– Does the modification for you
• Re-run tests
– On unmodified code
gcov
Cantata
LDMS
Parasoft
VectorCAST
IBM RTRT
Squish (Froglogic)
Test Tools:
![Page 15: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/15.jpg)
How much coverage do you need?
• Horses for courses
– Some things will need a lot
– Some not as much
• Perhaps
• Critical systems
– Safety related
– High integrity systems
• Business systems
– But are they really less critical?
![Page 16: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/16.jpg)
Set coverage standards
• Set coverage targets – Appropriate for your project
– For component testing
– And system testing
• What is 100% – Which coverage?
– 80% is 20% not tested - which 20%?
• A good default – At component testing
• 100% MC/DC
– At system testing • Lower percentage
• Entry/Exit coverage
![Page 17: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/17.jpg)
Preparation
• Design for testing
– Make your job easier
– Consider test issues in advance
• Design tests
– Plan tests
– To achieve coverage targets
![Page 18: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/18.jpg)
How to get 100%
• Measure coverage
– Using automated tool
• Analyse results
• Add tests
– If needed
• Automatic test addition
– Available in some tools
100%
![Page 19: Get Yourself Covered - T&VS · Get Yourself Covered – An Introduction to Test Coverage Barry Savell Associate Consultant . What is test coverage? • A map of where we’ve been](https://reader034.fdocuments.in/reader034/viewer/2022052003/6016a9aadb15830608171783/html5/thumbnails/19.jpg)
Get Yourself Covered - Summary
• Maps what has been tested
• Shows S/W fit for purpose
• Various types
– Modified Condition / Decision Coverage
• Involves instrumenting code
• Supported by tools
• Set coverage targets
– Appropriate to the system
Thanks