Post on 09-Jan-2017
Advanced Defect Management – The IBM Way
Sabarinath Venugopalan
Practice Leader, Nordic – IBM Test Services
© 2009 IBM Corporation
ERUC 2009
Agenda
Defect Reduction Methord
Test Planning and Optimization Workbench (TPOW)
Summary
Client Examples
Introduction to Advanced Defect Management
© 2009 IBM Corporation
ERUC 2009
Key Messages
The Next Generation of Testing is
emerging to meet the demands of the
intelligent enterprise
The key to the Next Generation of
Testing is informed actions, enabled by
predictive analytics
Advanced Defect Management is
powered by informed actions. It delivers
significant benefits to the major factors
governing the Next Generation of
Testing
© 2009 IBM Corporation
ERUC 2009
Today’s leading Enterprises are Intelligent. They are…
Anticipating
Precise
Aware
Questioning
Empowering
Gathers, senses, and uses
structured and unstructured
information from every
node, person, and sensor
within the environment
Predicts and prepares for the
future and doesn’t only react or
correct actions, but also steers
and evaluates trade-offs
LinkedConnects internal and
external functions front to
back across geographies in
a way that aligns to desired
business outcomes
Enables and extends
employees' memory,
insight and reach, as well
as the authority to decide
and act
Reserves the right to get
smarter by challenging its
status quo while creating
new opportunities
Uses only the most relevant
information to support timely
decisions/actions closer
to the point of impact and
consequence
Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009
© 2009 IBM Corporation
ERUC 2009
Old ways of decision making and management are breaking down. Organizations will need to adopt new ways of working to improve speed to insight and speed to impact
Traditional Approach New Approach
Instinct and intuition Fact driven
Corrective Directive
Efficient Optimized
Years, months, weeks Hours, minutes, seconds
Human insight Applied semantics
Decision support Action support
Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009
© 2009 IBM Corporation
ERUC 2009
The Next Generation of Testing will be driven by four factors. Advanced Defect Management substantially improves these factors.
Quality Improvement
DATA
INSIGHTS
Cost Reduction Risk Management
Schedule Acceleration
3
42
1
Analytical I Insightful I “Smarter” I Linked
Total View of Test
Managing Change will be paramount
© 2009 IBM Corporation
ERUC 2009
IBM’s Advanced Defect Management capabilities are based on two related analytical predictive capabilties. These are enabled by significant base of data collected over 25 years.
IBM® Defect Reduction Method™ is a patent pending, statistical,
objective defect classification and analysis process that helps
organizations find and fix defects as early as possible and prevent
future defects from being injected.
IBM® TPOW™ is an advanced, empirically risk-based planning tool
that delivers an optimized test strategy, supporting master test plan
and ongoing project planning. At the root of TPOW is the defect
prediction capabilities that that gives you insights about your
application projects.
© 2009 IBM Corporation
ERUC 2009
Advanced Defect Management drives dramatic reductions in cost and schedule while improving overall quality and lowering business risk
Cost Overview
(Total: $13,277,407)
Source: Test Planning and Optimization Workbench example
Cost Overview
(Total: $20,653,285)
Schedule: 89 days
Environment: 14 days
Schedule: 37 days
Environment: 5 hrs
$911,260
$10,240,335
$2,125,812
Defect fix cost
Business cost
Test cost
Defect fix cost $14,439,176
Test cost $1,579,838
Business cost $4,634,271
Cost ReductionDATA
INSIGHTS
35%+
Cost Reduction
© 2009 IBM Corporation
ERUC 2009
Car WashCar Wash
Car Wash
Car WashCar Wash
IBM is leveraging on Advanced Defect Management internally and in client engagements to substantially improve the four factors of Next Generation Testing.
TPOW Macro
Planning
(Reference
Project)
TPOW Micro
Planning
Test
Optimizers
Execution
Managed Testing Services
Test
Execution
Implement advanced Test
practices, processes and
optimizers with the full-life cycle.
Execute managed test services
leveraging globally deployed
resources executing test
according to the newly
implemented processes.
1000 – 5000
Function Points
Currenty in
development or
under major
enhancement
1B
2C
Execute detailed bottom-up test planning
leveraging the Macro Planning feature of the
Test Planning and Optimization Workbench.
3
Execute Pilot
TPOW Macro
Planning
(Client
Project Data)
1A2A
2B
2C
Conduct a Defect Analysis Starter which
executes IBM’s Defect Reduction Method to
gather quantitative data needed to customize a
client Reference Project
Execute IBM’s Test Planning and Optimization
Workbenchby starting with Macro Planning. IBM
assesses an existing application and, provides
recommended improvements.
1A
1B
Execute Test Optimizers recommended thru
Macro Planning. 2B
DAS
Execute IBM’s Test Planning and Optimization
Workbench Macro Planning using baseline
data gathered by the Defect Analysis Starter.
2A 3
Select target programs (s)
Applications
© 2009 IBM Corporation
ERUC 2009
Agenda
Defect Reduction Methord
Test Planning and Optimization Workbench (TPOW)
Summary
Client Examples
Introduction to Advanced Defect Management
© 2009 IBM Corporation
ERUC 2009
IBM® Defect Reduction Method™ (DRM) provides measurable insight into software quality and development processes
When does it start/end?
Used continually throughout the test life cycle
Defect classification can begin at any point that
defects have been recorded
Comprehensive analysis reports typically run at test
exit for each phase or for the release, but interim
reports should be generated periodically throughout
test execution to gain the most value from DRM
IBM® Defect Reduction Method™ is a patent
pending, statistical, objective defect classification and
analysis process that helps organizations find and fix
defects as early as possible and prevent future defects
from being injected.
DRM Classification DRM Analysis
►Systems engineering life cycle
Test Assess and Planning Test Execution and ReportingTesting project
startupTest Design Test Implementation and Close
Ongoing DRM activity
© 2009 IBM Corporation
ERUC 2009
The IBM® Defect Analysis Starter™ quickly identifies risks and helps improve the quality of system and application development. Application of DAS is typically completed in 3 weeks!
X X X
X X X
XX XX XX
XX XX XX
The IBM® Defect Analysis Starter™ provides business value by:
What is it?
A defect based analysis, using IBM®’s Defect Reduction Method™ (DRM), to
identify high priority cost reduction opportunities associated with software
test practices and defect prevention.
The IBM® Defect Analysis Starter™ utilizes:
An IBM approach that encompasses classification and analysis
Benchmarks based on defect data collected for nearly two decades,
mapped to industry, quality level, and test phase/activity
Metrics to objectively quantify and prioritize risk and opportunity, and to
measure quality
The IBM® Defect Analysis Starter™ includes:
Classification of client provided defects
Analysis and measurement of risks (General Test Effectiveness, Test
Design Effectiveness, System Stability and Completeness) and Quality
(Evaluation against Quality Benchmarks, Focus Area Identification, and
Artifact Value Assessment)
A Rough Order of Magnitude business case reflecting cost reduction
opportunity (Invalid Reduction, Production Defect Reduction, Earlier Defect
Removal, Cycle Time Reduction, and Prevention of Defect Injection)
Identifying and recommending specific actions to reduce risk and cost associated with developing and testing
systems or applications
Acknowledging and reinforcing client strengths
Guiding the client to utilize test practices that can increase their ROI while delivering improved quality
© 2009 IBM Corporation
ERUC 2009
Agenda
Defect Reduction Methord
Test Planning and Optimization Workbench (TPOW)
Summary
Client Examples
Introduction to Advanced Defect Management
© 2009 IBM Corporation
ERUC 2009
TPOW helps to answer challenging questions …
Am I doing too much/too little testing?
Am I getting my money’s worth out of testing?
Will an increased testing investment drive any further
improvement to quality?
How can I shift more of my investment to earlier project
phases/development?
What is the overall cost, schedule, and quality impact of
moving testing offshore?
How can I determine the quality of my application based
on my testing effort?
I just got a budget cut; what testing should I eliminate?
What impact will it have on application production quality?
How early in the life cycle can I develop an overall test
strategy, including effort, schedule, cost, and quality
predictions?
What if I….?
© 2009 IBM Corporation
ERUC 2009
TPOW works at multiple levels
Model initial cost
and benefits
Provides guidance in test
design and execution
Achieve the organization’s
quality, schedule and cost goals
Analytical
Project
Oriented
Micro
Planning
Macro
Planning
Ongoing
Project
Planning
Intelligence
Startup DeploymentSolution
closeSolution
outline
Macro
design
Micro
designBuildSolution
requirements
Industry and client benchmarking
21 3
4
© 2009 IBM Corporation
ERUC 2009
Example: Macro Planning - Create a Baseline
Baseline
Derive Function Points for High / Medium and Low
TPOW Baseline
Definition of Test Activities Definition of Triggers Distribution Allocation of Test Effort
– Per Test Activity
– Per Trigger
Definition of Global Resource Model
Benefit of using “Car Wash” Assume 1 Function Point for 10 KLOC
Macro
Planning
Macro
Planning
© 2009 IBM Corporation
ERUC 2009
Example: Macro Planning - Assess Process Initiatives
Baseline + DAS Initiative
Complete for medium case
Baseline + DAS
Assume 1 FP for 10 KLOC
Baseline
Outcome
Baseline Das initiative %
Test Cost 4,263,066.00$ 4,430,601.50$ -4%
Defect Fix cost 19,861,196.50$ 10,667,479.75$ 46%
Business cost 5,272,015.00$
Escaped defects 375.00 61 84%
Definition of Test Activities Definition of Triggers Distribution Allocation of Test Effort
– Per Test Activity
– Per Trigger
Definition of Global Resource Model
Benefit of using “Car Wash”
Macro
Planning
Macro
Planning
© 2009 IBM Corporation
ERUC 2009
Agenda
Defect Reduction Methord
Test Planning and Optmization Workbench (TPOW)
Summary
Client Examples
Introduction to Advanced Defect Management
© 2009 IBM Corporation
ERUC 2009
Reduction of Client PLM Incidents
PLM Incidents Trend
0
20
40
60
80
100
120
140
160
180
June
July
August
Septem
ber
October
Novem
ber
No.
of I
ncid
ents
Sev2
Sev3
Sev4
Total
Achieved through application of Defect Reduction Methods (DRM)
*Trough in August attributed to vacation
© 2009 IBM Corporation
ERUC 2009
Client Example: Selection of Accelerators based on TPOW Analysis
Scenario Escaped
defects 2)
Production
Defect rate
Cost to fix
Production
Defect 1) DKK
Potential
annual saving
Baseline 43.226 23% 562.000.000 0
TPOW 42.031 21% 546.000.000 16.000.000
+Carwash,+ DRM Phase1 +
Test Data Mgt phase 1&2
33.785 17% 439.000.000 123.000.000
+ DRM phase 2+3, + Test
data management phase 3
26.928 14% 350.000.000 212.000.000
TCOE 14.022 7% 182.000.000 380.000.000
© 2009 IBM Corporation
ERUC 2009
Team of testers spent 3 1/2 months developing
detailed estimates for a major development
project.
Since the project was in the design phase, they
were trying to develop estimates that would
support a fixed budget for testing.
The client wanted a guarantee that no more than
5% of the defects associated with the application
lifecycle would arise during production.
If weight was a measure of accuracy, the 10,378
person days of effort looked like a solid estimate.
The team asked for an independent estimate to
validate their numbers.
In just 1 1/2 days using advanced defect
methods, the assessment of the teams' estimate
yielded …
© 2009 IBM Corporation
ERUC 2009
Increase test effort to meet decreased defect rate ..
PD: Person Day
ST SIT PT UAT Total
XYZ 21.68%
2250
45.37%
4708
10.79%
1120
22.16%
2300
100%
10,378
Improvement
suggestion
26.85%
3812
51%
7242
6.41%
910
15.75%
2236
100%
14,200
DecreaseIncrease
Scenario Output
Projection Defect discovery and escape to production
Defect production rate = 368 / 1921 = 19%
Validation
and ImprovementValidation result
Given the quality objective of production defect rate 5%, the existing estimated test
effort (10,378PD) is not enough
Improvement suggestion1. Add the total test effort to 14,200PD
2. Adjust the test effort distribution on test activities (see table below)
© 2009 IBM Corporation
ERUC 2009
Agenda
Defect Reduction Methord
Test Planning and Optimization Workbench (TPOW)
Summary
Client Examples
Introduction to Advanced Defect Management
© 2009 IBM Corporation
ERUC 2009
Summary
Applying predictive analytics to reduce
defects and to move defects earlier in
the lifecycle delivers substantial gains
Taking a total view of testing generates
dramatic benefits
IBM’s capabilities in advanced defect
management are deemed unique by
technology analysts
© 2009 IBM Corporation
ERUC 2009
Contact:
Sabarinath Venugopalan
Practise Leader
IBM Test Services, Nordic
+46-70-793 2665
sabari@se.ibm.com