The Problem withGood Enough Software

Bj RollisonTest Training Manager

Microsoft, Inc.

Overview

Discuss the intent and the reality of good enough software

Test engineering…where we are and where should we go

What is software quality?

Three properties of software quality Features

The cool stuff that makes customers want the software Attributes

Functionality, security, usability, reliability, maintainability Defects

The bugs that make users wish they bought something else

Defects = Quality There is no such thing as defective, high-quality

software

Good enough software

Utilitarian view of quality – potential positive consequences outweigh potential negative consequences

“…powerful alternative to the orthodox, expensive, and boring methodologies that aim at the best possible quality.”

“…secret to the success of Microsoft…”http://www.satisfice.com/articles/gooden2.pdf

Utilitarian view in practice

Good enough:Features exceeddefects

80/20 rule: Defects actuallyexceed features Defects

Features

Features

Development cycles in practice Hack-and-whack approach

Projects that rely on testing spend a lot of time and money on testing

Orthodox, boring engineered approach

Projects that rely on engineering practices use test to validate quality

http://www.sei.cmu.edu/tsp/

Plan?Code

1 defects / 10 LOCTest

Random, Subjective

PlanDetailed plan

ReviewedModeled

CodeInspectedReviewed

Unit tested

TestFocused

Microsoft’s success!!!

Complex products/ unwanted features Bugs, Security Patches & Service Packs Un-needed/unwanted upgrades Unpredictable, reactionary

The reality of good enough software Tends to focus on features & schedule

PM’s/Dev’s are rewarded for features not quality Heavy investment in testing to validate quality

Reliance on testing to find defects You can’t test in quality

Necessitates sustained engineering There’s never enough time to do it right the first

time, but there’s always enough time to fix it.

Features vs. Quality

“If Microsoft is not here in 10 years it won’t be because of a lack of features, it will be due to a lack of quality!” – Craig Zhou, Microsoft

Problems in the process Undefined features Feature creep Schedule driven releases

Testing Effectiveness

50%

Bugs

Testing Ideal

Testing Actual

Sustained Engineering Costs

Microsoft issued 60 patches for Windows 2000 in 2001

Security patches cost an enterprise customer about $300,000 to install

It costs Microsoft about $100,000 per patch Windows XP SP 1 – risk free, but why bother Maintenance has become similar to shipping

a product at Microsoft.

The changing role of test engineering

Hack – n - Whack Approach

ProgramMgt.

Development

Testing

Non

-det

aile

d sp

ecs

Write code

Integration testing

Unit testing

testingSyst

em

Engineering Approach

ProgramMgt.

Development

TestingDet

aile

d sp

ecs

Write code

Integration testing

Unit testing

System testing

Mod

el

Role of Software Test Engineer Validation

“Establishing confidence that a program…does what it is supposed to do.” – Hetzel, 1983

Falsification “Executing a program or system with the intent of finding

errors.” – Myers, 1979 Achieve testable designs

Spec reviews, modeling, writing hooks in code, etc. Provide information to developers

More than bugs, code reviews, inspections, metrics, etc. Provide information to management

“The penultimate objective of testing is to gather management information.” – Beizer, 1995

Questions