Introduction to Bugs Measurement

by V. Novostavsky

Yes, it’s about metrics

TOC

• What we’re talking about• Why it's good, and when it’s evil• “Easy way”

Definition

• Distance function– from “anything” to numbers :)

Metrics in life: School

Metrics in life: Traders

Metrics in life: dou.ua

… in testing

– Usually we measure bugs• because there’re a lot of bugs

We can measure barely anything in testing, but that’s another story…

… in testing

– Number of open bugs– Bugs per module– Bugs reopened– Bugs posted in a week– Bugs rejected / accepted – A lot of other stuff…

… in testing

– Number of open bugs– Bugs per module– Bugs reopened– Bugs posted in a week– Bugs rejected / accepted – A lot of other stuff…

Why it's important

• tester's job is info support for the project

Why it's important

• tester's job is info support for the project• more and diverse info -> better conclusions ->

easier decisions

Why it's important

• tester's job is info support for the project• more and diverse info -> better conclusions ->

easier decisions• metrics are facts, operate with facts, not blah-

blah-blah

Why it's important

• tester's job is info support for the project• more and diverse info -> better conclusions ->

easier decisions• metrics are facts, operate with facts, not blah-

blah-blah• example: “big” manager

Why it's important

• tester's job is info support for the project• more and diverse info -> better conclusions ->

easier decisions• metrics are facts, operate with facts, not blah-

blah-blah• example: “big” manager

• NOTE: don't judge too quickly!

Example

500 open bugs this week… A lot of?

Example

Example

More bugs in a module… Does it mean it’s worse?

Example

Example

Less bugs this week… Is it good?

Example

Top issues

Top issues

• time waste!

Top issues

• time waste!

• wrong conclusions and politics!

Top issues

• time waste! make it easier to generate metrics

• wrong conclusions and politics!

Top issues

• time waste! make it easier to generate metrics

• wrong conclusions and politics!make it clear what they mean

“Easy way”

• Step 1: choose correct metrics for you

“Easy way”

• Step 1: choose correct metrics for you– Not so easy!

“Easy way”

• Step 2: ensure everybody gets what they mean

“Easy way”

• Step 3: automate them (+ use SQL)

“Easy way”

• Step 4: be prepared for change requests

“Easy way”

• Step 5: use often

“Easy way”

• Step 6: publish online

“Easy way”

• Step 6: publish online– Publish charts!

“Easy way”

1. choose correct metrics for you2. ensure everybody gets what they mean3. automate them4. be prepared for change requests5. use regularly6. publish online

Example: my metrics

• No classic test cases & project is not big– Bug metrics only

• Custom bugtracker– No access to DB

• 2 metrics - weekly– Open bugs: Trend in time– Open bugs: By modules

Example: tools

• MS XLS –> MS SQL

Example: tools

• MS XLS –> MS SQL

• Google Docs

Example: my trender

Example: my rollup

Thanks!

Q & A