Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 1
6/11/2015
Better Software
Conference West
June 11, 2015
Las Vegas, Nevada
Mike Trites
Making Numbers Count
Metrics that Matter
About Me
Mike Trites
‣ Senior Test Consultant
‣ Over 10 years of experience in software testing, working
with several clients in multiple industry verticals
Certifications
‣ Black Box Software Testing Foundations – AST
‣ Certified Tester, Foundation Level – ISTQB
‣ Certified Tester, Advanced Level Test Manager – ISTQB
2 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 2
6/11/2015
Introduction
3 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
How is testing going?
“We’re on-track.” “95% of test cases so far
have passed.”
“We found 15 new defects yesterday.”
What I’m here for…
Illustrate the important role that metrics play in the software
testing process.
Explain how to give context when communicating metrics.
Describe how to use metrics to improve the testing process
for future projects.
Identify some common pitfalls and how to avoid them.
Identify a selection of metrics to:
‣ Measure progress during a test cycle.
‣ Determine when to stop testing.
‣ Measure the effectiveness of testing.
4 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 3
6/11/2015
What I’m NOT here for…
Provide a “one size fits all” solution for you or your
organization.
Why not?
‣ Metrics are situation / context specific.
‣ There is no silver bullet.
‣ There is no “right” answer.
‣ They will not solve all your problems.
5 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Agenda
Why Metrics are Important
Defining and Capturing Metrics
Reporting on Metrics
Identifying Trends
Misuse of Metrics
6 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 4
6/11/2015
Why Metrics are Important
8 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 5
6/11/2015
Why Metrics are Important
9 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Metrics play an important role in:
• Estimation
• Evaluating suspension criteria
• Evaluating exit criteria
• Status reporting
• Process improvement
Process Improvement
"Those who cannot remember the past are condemned to
repeat it.”
- George Santayana
What is “better”? How do we know if we’re better now than
we were?
Estimates:
‣ Comparing planned effort and actual effort
‣ Estimate variance and what factors affected it
10 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 6
6/11/2015
Metrics for Process Improvement
11 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Closure Period / Turn-around Time
• Measures how long it takes for a reported defect to be rectified
Defect Ratio
• Useful for assessing the effectiveness of regression or automation suites
Metrics for Process Improvement
12 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Defects by Root Cause
• Allows you to proactively avoid anticipated defects in the future
Closed Defects by Resolution
• Identifies instances of duplicate or invalid defects
Defect Removal Efficiency
• Identifies defects that are uncovered after the application is released
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 7
6/11/2015
Defining and Capturing Metrics
Defining Metrics
Every phase of the testing process can be measured.
Schedule / Budget
Test Coverage:
‣ Requirements
‣ Configurations
‣ Code
‣ Automation
Test Design
Test Execution
Defects
14 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 8
6/11/2015
Capturing Metrics
15 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Expensive commercial tools can be
useful, but are not always necessary.
• Examples:
‣ Test Case Management Tool
‣ Defect Management Tool
‣ Time Tracking System
There are less expensive tools that can
be used to achieve the same results:
• Examples:
‣ Bugzilla
‣ JIRA
‣ Excel
Reporting on Metrics
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 9
6/11/2015
Metrics Need Context
When metrics are used in isolation, their significance is left
open to interpretation.
Stakeholders can draw the wrong conclusions if no context
is provided and they are instead left to interpret the data on
their own.
Textual summary can provide context, but so can a good
chart.
17 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
What makes a good chart?
Accurate – Data is not purposely skewed or omitted.
Clear – Information is presented in a way that is
unambiguous and easy to interpret at a glance.
Concise – Chart is not cluttered with unneeded or
distracting data.
Includes any necessary:
‣ Titles
‣ Labels
‣ Data values
‣ Trend lines
‣ Annotations
18 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 10
6/11/2015
19 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Medium 68%
Low 18%
High 14%
Medium 68%
Low 18%
High 14%
Colour Conventions
Defects by Severity
More meaningful with
appropriate colours
0%10%20%30%40%50%60%70%80%90%
100%
Example: Test Progress
20 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0%10%20%30%40%50%60%70%80%90%
100%
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 11
6/11/2015
Example: Defects by Release
21 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
= 63 (49%)
= 56 (44%)
= 9 (7%)
Fixed / Planned
v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7
Dis
co
ve
red
v0.1 1 9 2
v0.2 7 10 15 1 1 3
v0.3 11 17 1 3
v0.4 1 31 10 3
v0.5 1 1
Example: Scorecard
22 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Execution
Progress
Pass
Rate
Blocked
TestsMust-Fix Other
Component 1 100 % 100 % 0 0 0
Component 2 100 % 78 % 0 0 4
Component 3 87 % 80 % 0 1 1
Component 4 95 % 100 % 0 0 4
Component 5 75 % 92 % 1 1 0
Component 6 98 % 79 % 2 1 4
Overall 93 % 88 % 3 3 13
Defects Test Execution
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 12
6/11/2015
http://dilbert.com/strips/comic/2007-05-16/
Dashboards
23 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Identifying Trends
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 13
6/11/2015
Trends
One measurement captures status at a point in time.
When a series of data points are examined, trends in that
data start to emerge.
Trends allow us to:
‣ Differentiate between systemic behaviour and temporary
anomalies.
‣ Make predictions about the future.
Accuracy increases with more historical data.
25 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
5
10
15
20
25
30
35
Defects Detected
Trends
26 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Day # of
Defects
1
2
3
4
5 15
6
7
8
9
10
?
?
?
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 14
6/11/2015
0
5
10
15
20
25
30
35
Defects Detected
0
5
10
15
20
25
30
35
Defects Detected
Trends
27 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Day # of
Defects
1
2 4
3
4
5 15
6
7
8
9
10
0
5
10
15
20
25
30
35
Defects Detected
0
5
10
15
20
25
30
35
Defects Detected
Trends
28 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Day # of
Defects
1
2 4
3
4
5 15
6
7
8
9
10 1
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 15
6/11/2015
0
5
10
15
20
25
30
35
Defects Detected
Trends
29 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Day # of
Defects
1
2 4
3
4
5 15
6 2
7
8 8
9
10 1
0
5
10
15
20
25
30
35
Defects Detected
Trends
30 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Day # of
Defects
1 3
2 4
3 10
4 7
5 15
6 2
7 3
8 8
9 2
10 1
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 16
6/11/2015
0
5
10
15
20
25
30
Example: Open vs. Closed Defects
31 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
5
10
15
20
25
30
0
2
4
6
8
10
12
14
16
18
0
2
4
6
8
10
12
14
16
18
Example: Open Defects
32 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 17
6/11/2015
Misuse of Metrics
Intentional Manipulation
“There are three kinds of lies: lies, damned lies, and
statistics.”
- Benjamin Disraeli
Metrics can be manipulated to make
any situation look good or bad.
34 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 18
6/11/2015
Misleading Graphs
An example of a good graph:
35 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
100
200
300
400
500
A B C D
430 455 448 465
Misleading Graphs
Inappropriate scale:
36 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
200
400
600
800
1000
A B C D
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 19
6/11/2015
Misleading Graphs
Scale does not start at 0:
37 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
420
425
430
435
440
445
450
455
460
465
A B C D
Tester 1 Tester 2
Tester 3
Tester 4
38 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misleading Graphs
Defects by Component
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 20
6/11/2015
11% 42%
5% 42%
Tester 1
Tester 2
Tester 3
Tester 4
Defects by Component
39 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misleading Graphs
Poor Choice of Format
40 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
12%
15%
14%
8% 11%
17%
13%
10% Tester 1
Tester 2
Tester 3
Tester 4
Tester 5
Tester 6
Tester 7
Tester 8
Defects by Component
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 21
6/11/2015
Better Choice of Format
41 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0%2%4%6%8%
10%12%14%16%18%
12%
15% 14%
8%
11%
16%
13%
10%
Defects by Component
Overloading
Often, managers are most
interested in defect statistics.
Does defect count measure:
‣ Tester effectiveness?
‣ Developer skill?
‣ Product quality?
‣ Effort remaining?
One metric does not tell the entire story.
42 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 22
6/11/2015
Performance Assessment
People optimize what we measure them against, at the
expense of what we don’t.
For example, if we assess testers based on their defect
counts, how much time will testers spend:
‣ Documenting their tests?
‣ Coaching other testers?
‣ Researching and polishing defect reports to make their
defect easier to understand, assess and replicate?
‣ Logging variants of the same failure to increase defect
count?
43 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Performance Assessment
44 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
100
200
300
400
500
Tester 1 Tester 2 Tester 3
Test Cases Executed
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 23
6/11/2015
Key Points to Remember
Key Points to Remember
Metrics can be captured for just about any testing activity.
Metrics play an important role in:
‣ Estimation
‣ Status reporting
‣ Evaluating suspension and exit criteria
‣ Process improvement
Metrics, like anything, should be planned in advance.
Reporting needs should be identified before setting up
tools.
To avoid misinterpretation, metrics need context.
46 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 24
6/11/2015
Key Points to Remember
Even when properly presented, too much information can
be overwhelming.
It is best to limit the number of metrics and reports that are
presented to stakeholders to only what is essential.
Capturing, analysing and reporting on metrics can be a
drain or waste of time if those metrics are not being used to
provide some value in return.
Metrics influence tester performance, but not always in the
way you might think.
Metrics should be used as a starting point for discussion.
One metric does not tell the entire story.
47 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Questions or Comments?
48 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 25
6/11/2015
Thanks for coming!
Contact Information:
©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
ca.linkedin.com/in/miketrites
@mike_trites
www.pqatesting.com
49
Top Related