Net Squad Leader...Squad Leader Classic 1. INTRODUCTION TO THE GAME
Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.
-
Upload
lorraine-bond -
Category
Documents
-
view
219 -
download
2
Transcript of Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.
![Page 1: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/1.jpg)
Readiness Index – Is your application ready for Production?
Jeff TatelmanSQuAD October 2008
![Page 2: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/2.jpg)
Building a Metrics Program Typical Test Organizations Readiness for Change Metrics and Productivity
Production Readiness Introduction & Objective Four Key Metrics Reporting
Agenda
![Page 3: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/3.jpg)
Typical Test Organizations
No metrics collected No collection of requirements Limited formal reporting on
project status Some central test repositories
but not dash boarding results Minimal effort towards process
improvement
![Page 4: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/4.jpg)
Motivation Investment Skills Education Culture Support Staff Aids/Maturity Process Maturity
Dimensions Of Readiness
![Page 5: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/5.jpg)
Set Measurement Objectives Select Measures Develop Measurement Program How to Use the Results
Getting Started
![Page 6: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/6.jpg)
Determine Approach Which measures will be included as a priority Focus on a few to start with to get ‘quick wins’
Determine Scope One application? One project?
Determine Method Identify best method to track and analyze agreed metrics
Determine Timescale Full project lifecycle? Time boxed approach, eg. 1, 3 or 6 months
Determine Deliverables Report, audience, next steps
Set Measurement Objectives
![Page 7: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/7.jpg)
Metrics and Productivity
Strategic Is it to increase customer satisfaction? Is it to achieve CMM level 2? Is it to achieve industry standards?
Tactical Is it to improve vendor delivery? Is it to increase timeliness of system delivery?
Operational Is it to reduce defects found in testing? Is it to improve requirements definition?
![Page 8: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/8.jpg)
Measuring Unit Examples Strategic
Quality Customer satisfaction Timeliness (delivery) Personnel Industry benchmarking
Tactical Estimating and planning System quality System delivery cost/time Budget Productivity Cost of quality
Operational Schedule tracking Effort tracking Defects Problem resolution System availability
![Page 9: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/9.jpg)
Develop Measurement Program
Establish Specific Measurement Objectives Define a Critical Metrics Set Select Measures to Support the Metrics Set Put Collection Mechanisms into Place Determine Timing of Data Collection Establish Mechanism for Evaluating Results Communicate Results Establish Process for Future Planning Based on
Metrics
![Page 10: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/10.jpg)
How To Use The Results
Conduct root cause analysis of the data Identify areas which would have the biggest
impact and look to how to improve Implement procedures to continually
improve the processes Revisit original list and determine what to
address next
![Page 11: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/11.jpg)
Key Points to Remember
Don’t try to measure too much Understand the goals of your team before
you determine what to measure Determine which metrics support these goals Don’t let your metrics define the behavior of
the team Change or choose a set of relative metrics that can
not be manipulated Monitor and identify trends , define areas for
improvement
Allan Page Measurement that Matter, Better Software October 2005
![Page 12: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/12.jpg)
Production Readiness
Based On Mike Ennis’ Presentation at Star West 2006
![Page 13: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/13.jpg)
How do you know when a product is ready to ship?
Quality Metrics Customer Commitment Release Dates Are Preset Indefinite Testing Adequate Test Coverage Time & Resources Release Criteria
![Page 14: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/14.jpg)
Release Criteria No open critical or high defects Minimal number of medium & low
defects that have been approved by Senior Management
Product is able to run for 72 consecutive hours
No open installation or configuration issues
All pre-defined performance criteria has been met
![Page 15: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/15.jpg)
Determining What to Measure
Test Case Execution Percentage The percentage of tests that have been attempted during
the test cycle Test Case Success Percentage
The percentage of tests that have passed during the test cycle
Number of Unresolved Defects The number of open defects that are currently logged
against the product Defect Arrival Rate
The number of defects found in a given day, week or build
![Page 16: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/16.jpg)
Readiness Metrics Objective
Provide management an overall picture to assess if the project is ready to be placed in production.
Evaluate an application go / no-go production readiness status by Tracking test case execution and defect metrics
over time Calculating the Production Readiness Index based
on the above metrics
![Page 17: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/17.jpg)
Test Case Spreadsheet Example
![Page 18: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/18.jpg)
Defect Spreadsheet Example
![Page 19: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/19.jpg)
Setting Range For Each Metric
Test Case Execution Percentage 10 = 100% 9 = 90-99% 8 = 80-89% 7 = 70-79% … etc
Defect Arrival Rate This Week 10 = 0 9 = 1-2 8 = 3-4 … etc
![Page 20: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/20.jpg)
Creating and Analyzing Readiness Example
RATING RANGESDefects Arrival Rate 9 10 = 0, 9 = 1-2, 8 = 3-4, 7 = 4-5, 6 = 5-6, 5=6-7,
4=…
Total Open Defects 9 10 = 0-5, 9 = 5-10, 8 = 11-15, 7 = 16-20, 6 = 20-25 …
Test Success % 7 10 = 100%, 9 = 99-97%, 8 = 96-94%, 7 = 93-90%...
Test Completion % 6 10 = 100-95%, 9 = 94-90%, 8 = 89-85%, 7 = 84-80%..
TOTAL RATING 31 Green = 35-40, Yellow = 29-34 Red < 28
![Page 21: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/21.jpg)
0
2
4
6
8
10Test Completion
Test Success
Unresolved Defects
Defect Arrival Rate
Spider Chart Example
![Page 22: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/22.jpg)
Test Execution Ratings (green) - scale of 10, goal = 10Defect Ratings (red) - reverse scale of 10, goal = 0
8.9
7.7
3.1
1.1
0.0
2.0
4.0
6.0
8.0
10.0
Test Completion Test Success UnresolvedDefects
Defect ArrivalRate
Management Graph Example
![Page 23: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/23.jpg)
Readiness Index Example
Production Readiness Index - scale of 100
3220
30 34 38 3446 41
73
45 47 51
7281
94100
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Week / Day
![Page 24: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/24.jpg)
Managing the risks Understand the relationships between the metrics Learn to anticipate and minimize the risks before they
happen Always know the information behind the data
Are you ready to release? Redefine your Release Criteria using the individual/overall
rating scale Use colors for presentation & effectiveness Let the data speak for itself
Final Thoughts
![Page 25: Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008.](https://reader035.fdocuments.in/reader035/viewer/2022062715/56649d8d5503460f94a755ef/html5/thumbnails/25.jpg)
Usefulness Does it tell us what’s happening, wrong or need to know?
Trustworthiness Does not provide false indications
Timeliness Does it indicate the health of a system...advanced
warning? Simplicity
Too difficult to read, it will get abandoned Accessibility
Must be visible, easy to get to
Key Attributes Of a Good Measure