Software Quality Metrics for Testers - StarWest 2013

Software Quality Metrics For Testers

Why and What to Measure

Meet Your Instructor

•  Phil Lew –  Telecommunications consultant and network designer –  Architect, Data warehousing product development –  Software product manager, BI product –  COO, large IT services company –  CEO, XBOSoft, software qa and testing services

•  Relevant specialties –  Software quality process improvement –  Software quality evaluation and measurement –  Software quality in use / UX design

STARWEST 2013 2

Session Spirit and Expectations

•  Interactive •  I won’t read the slides… •  Slides for you as a take-away •  Different than your handouts

– Always thinking of new examples and ideas, you can write me and I’ll send these to you

STARWEST 2013 3

Understand, Evaluate and Improve

•  If our end goal is improvement in what? •  To improve, we need to evaluate •  In order to evaluate, we must understand what

we are evaluating •  To do this… We need metrics

Can you think of other examples in our lives where this applies? Where do you use metrics

to evaluate and improve? STARWEST 2013 4

Metrics in real life

Food Eaten Weight Performance Race Results

STARWEST 2013 5

•  Calories •  Fat •  Carbohydrates •  Protein •  Time of day •  Vitamins •  …

•  Blood pressure •  Cholesterol •  Blood glucose •  Red cell count •  White cell count •  Hematocrit •  Hemoglobin •  Body fat % •  …

•  Placing •  …

•  Effort/Power •  Heart rate/Watts •  Speed •  Time

Intelligence Finesse

Context •  Training •  Sleep

Metrics - Benefits

•  Understand how QA, testing, and its processes and where the problems are

•  Evaluate the process and the product in the right context

•  Predict and control process and product qualities •  Reuse successful experiences

–  Feed back experience to current and future projects •  Monitor how something is performing •  Analyze the information provided to drive

improvements

STARWEST 2013 6

How can measurement help us (YOU)

•  Create a organizational memory – baselines of current practices-situation

•  Determine strengths and weaknesses of the current process and product –  What types of errors are most common?

•  Develop reasoning for adopting/refining techniques –  What techniques will minimize the problems?

•  Assess the impact of techniques –  Does more/less functional testing reduce defects?

•  Evaluate the quality of the process/product –  Are we applying inspections appropriately? –  What is the reliability of the product before/after

delivery? STARWEST 2013 7

All this is good

But in reality…

Why we need to measure ?

•  Our bosses want us to… •  They want someone to point fingers at •  They want to fire some people and save money •  They need to report to their managers

•  They want some basis on which to evaluate us and give us a raise!

•  We need to figure out a way to do beHer! •  We want to improve our work and improve soKware quality

STARWEST 2013 9

The Metric Conundrum

•  QA and Testing Language –  Defects –  Execution status –  Test cases –  Pass/fail rates –  DRE…

•  Business Language –  Cost effecRve –  ROI –  Cost of ownership –  Cost of poor quality –  ProducRvity –  Calls to help desk –  Customer saRsfacRon

–  Customer retenRon STARWEST 2013 10

In your organization…?

•  What measurements do you take in your organization and why?

•  Who uses them and for what?

STARWEST 2013 11

The Metric Reality

•  Measurement is like dinner. It takes 2-3 hours to make dinner, and 15 minutes to consume…

•  But… many metrics are never reviewed or analyzed (consumed)

•  WHY?

STARWEST 2013 12

The Metric Conundrum (cont.)

•  Test leads and test managers rarely have the right metrics to show or quantify value

•  Metric collection and reporting are a drag •  QA metrics usually focus only on test execution •  Test tools don’t have most of the metrics we

want •  Reports generated by QA are only rarely

reviewed •  Metrics are not connected to anything of value/

meaningful for ________. STARWEST 2013 13

Let’s solve this!

Outline

•  Choose the right metrics •  Design a measurement framework •  Report what is meaningful •  Use metrics not only to manage testing, but as a

Quality barometer •  Show value

STARWEST 2013 15

Guiding principles about metrics

•  Metrics can represent a snapshot •  Metrics can change over time •  Metrics can be simple – don’t have to be

complicated •  Metrics can assist us in making decisions today •  Metrics can assist us in predicting the future •  Some metrics are critical for some while useless

for others

STARWEST 2013 16

Metrics can represent a snapshot

•  Not only at a particular moment, like this week, but for a process

STARWEST 2013 17

Code quality

Internal Quality

Tested in a lab

External Quality

• What the end user sees

• What the end user experiences

In-‐Use Quality

At what stage in the process do you take measurements?

Metric Basics

•  What is a metric? •  What is a measurement? •  What is an indicator?

STARWEST 2013 18

Some Metrics – Let’s get started

•  How many of these do you know/use?

•  What others do you use?

•  What are they used for?

•  By who?

Schedule variance

Effort variance

Cost variance

Defect removal

effecRveness

ProducRvity Defect aging

CriRcal Defect rate

Test cases

executed

Test coverage

Pass/Fail Rate

ROI

AutomaRon coverage

STARWEST 2013 19

Metrics Exercise

Work in teams of 2-3 List the metrics you use, what you use them

for, who wants them, what decisions they make based on them

STARWEST 2013 20

Developing a Measurement Framework

Stakeholders

Measurement IdenRficaRon

Measurement Report Design

Measurement Framework

Data CollecRon Methods and Sources

STARWEST 2013 21

Stakeholders

•  Who are they? •  What are their objectives? •  What questions do they have? •  What answers can metrics provide?

STARWEST 2013 22

Possible Stakeholders

Stakeholder Objec0ve Ques0on Answer

CIO

CEO

VP – QA – Development

Director -‐ Development

Owner

Auditor

Test Lead

Partner

?

STARWEST 2013 23

Metric Q&A Worksheet Category Ques0on Metric Tool to Use Needed

Customer SaRsfacRon

Are we meeRng schedule?

Schedule Variance

Project Management Tool

None

SoKware TesRng Efficiency

SoKware Quality

What kinds of defects are most common?

Usability Are end users gedng lost, understand our UI

? ?

•  QuesRons are the fundamental building block •  Answering quesRons that lead to decisions à VALUE 24

Stakeholder Workshop (15 minutes)

•  Divide up into teams of 2-3 •  List out stakeholders •  Brainstorm stakeholder questions (2 per

stakeholder) •  Determine answers and metrics to that question •  Group your metrics into categories •  Present and discuss to the group

STARWEST 2013 25

Organizing our thoughts

•  Putting metrics into a framework •  Helps us to make sure we have all functions in

our organization and process covered •  Helps us to develop relationships between

different elements in our process

STARWEST 2013 26

Identify Metrics to Answer Questions

Customer and Management

Metrics

OperaRons

OpRmizaRon

Metric CalculaRon

Departmental Metrics

Project/Build Metrics

Planning

Quality

Status

Data CollecRon

Metric Identification

Req’ts. & Questions

Reporting

Efficiency and EffecRveness

Schedule and Budget

Defect types and quanRRes

Resource uRlizaRon

Snapshots and trends

STARWEST 2013 27

Measurement and Reporting •  Measure, build metrics AND develop reports that

answer the questions of your stakeholders

Planning

•  Requirements coverage % •  Test case prioriRes and complexity/atomicity

•  Requirements traceability •  Test ExecuRon prioriRzaRon •  Requirements aging •  Resource and budget variance

STARWEST 2013 28

Test Measurement and Reporting

•  Measure, build metrics AND develop reports that answer the questions of your stakeholders

Quality

•  Test pass/fail rate •  # open defects •  # priority 1 defects open •  # priority 1 defects found •  Re-‐opened defects •  Defect density •  Defects found in operaRon •  Defects found/test case/script •  Test execuRon progress

STARWEST 2013 29

•  Customer saRsfacRon •  Help desk calls



OperaRons •  Time spent gedng environment ready

•  Time to close defects – aging •  Effort per test case •  TesRng schedule variance •  Defect arrival and closure rate •  Test schedule variance

STARWEST 2013 30



OpRmizaRon •  AutomaRon script execuRon Rme

•  Regression Rme •  AutomaRon coverage •  Resource efficiency •  Resource effecRveness •  Test cycle Rme

STARWEST 2013 31



Status •  Test execuRon % complete •  Test case prioriRes and complexity/atomicity

•  Requirements status/trending •  Test ExecuRon EffecRveness •  Over Rme (trending) versus snapshot

STARWEST 2013 32

What’s Missing?

STARWEST 2013 33

Requirements

stability

Internal Quality

Defects AutomaRon

coverage

External Quality

Customer retenRon Help desk call rate

In-‐Use Quality

Can you think of where other metrics we have listed out before, and where they would fit?

Data and Reporting

OperaRons

OpRmizaRon

Metric CalculaRon

Customer and Management

Metrics

Departmental Metrics

Project/Build Metrics

Planning

Quality

Status

Automated Data

CollecRon

All for nothing unless you have supporting data

Config mgmt

Rqmt mgmt

Time tracking

Test mgmt

Project planning

Other?

STARWEST 2013 34

Exercise: Develop Your Own Framework

STARWEST 2013 35

Work backwards or forwards Work in small groups 2-3

Reporting

•  Work with stakeholders to report information so that they can make decisions

•  Metric is meaningless if not understood or misunderstood

•  Develop a report catalog – Metric library – Canned reports usually have limited

impact STARWEST 2013 36

Reporting Root Causes and Trends

•  The most important metrics you can report –  Root causes

•  Sources of defects •  Sources of mistakes •  Sources of delay or inefficiency •  Sources of improvement

–  Trending •  Showing metrics over time, pre-set intervals •  Showing change •  Make sure to differentiate Snapshot versus trend

STARWEST 2013 37

Reporting – Show Value

•  Answer Questions –  Are we delivering a quality product? –  Are our customers satisfied? –  Are we meeting timelines? –  Did we understand requirements correctly and timely? –  How much is QA costing? –  Are we meeting budget? –  How have we cut costs? –  How have we improved efficiency? –  What are the primary causes of delay? –  What are the causes of poor quality? –  Is quality improving? –  What is the status? – Current versus goal? –  Have we gotten better?

STARWEST 2013 38

Show Value – Connect the dots

–  Did we understand requirements correctly and timely?

–  What are the primary causes of delay?

–  What are the causes of poor quality?

STARWEST 2013 39

Requirements

stability

Internal Quality

Defects AutomaRon

coverage

External Quality

Customer retenRon Help desk call rate

In-‐Use Quality

QA and Testing Language

Business Language

–  Is quality improving? –  What is the status? –

Current versus goal? –  Are we delivering a

quality product? –  Are our customers

satisfied? –  Are we meeting

timelines?

–  How much is QA costing? –  Are we meeting budget

and schedule? –  How have we improved

efficiency? –  What are the primary

causes of delay? –  What are the causes of

poor quality?

Getting Started with Measurement

•  What questions-answers are important to your organization? – Develop a model/determine what to measure – Define improvement objectives

• Based on intuition/available data • Look at high payoff areas, problem areas

•  What data can you collect/Which technique can you use – Maybe some elements of the model drop out-

can’t be measured that easily STARWEST 2013 40

Set up an action plan

•  Define a metrics improvement plan – What are your goals?

•  Conduct stakeholder workshops – Involve all •  Build your measurement framework with phases

in the process and categories – Q&A catalog – Metric catalog – Report catalog

•  Set up the infrastructure and operationalize •  Start collecting and developing benchmarks

STARWEST 2013 41

What’s Next

•  Choose a project – start small and build up for early wins

•  Organize to support the change – Committed people – Recognize the new processes – Assign roles and resources

STARWEST 2013 42

Software Testing Metrics Take Aways

•  Quality is different for each organization –  No one right answer for what metrics –  Different metrics have different meanings for different

people – SET UP FRAMEWORK AND PLAN FOR YOUR Company

•  Management wants metrics –  Know what key questions need to be answered –  REPORTING - Metrics programs often fail due to poor

articulation •  Measurement and improvement is a long term effort, just

like a diet.

STARWEST 2013 43

Putting It All Together

Measurement Framework Improvement

Decisions

Stakeholders

STARWEST 2013 44

Questions and Answers

Thanks Please fill out an evaluation form and drop it in the collection basket

located at the back of the room.

www.xbosoft.com @xbosoft 408-350-0508

Philip Lew @philiplew [email protected]

Some resources: http://www.xbosoft.com/knowledge_center/

www.facebook.com/xbosoft

Software Quality Metrics for Testers - StarWest 2013

Technology

Transcript of Software Quality Metrics for Testers - StarWest 2013