(Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort...
Transcript of (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort...
![Page 1: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/1.jpg)
(Software) Management by Numbers
© SE-CURE AG 1
SE-CURE AG (www.se-cure.ch) Dr. Hans SassenburgT +41 33 733 4682M +41 79 231 6600E [email protected]
SPIN – Oct. 2nd, 2009
![Page 2: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/2.jpg)
Question
© SE-CURE AG 2
Would you fly to Mars with the software of your organization?
![Page 3: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/3.jpg)
Contents
1. Software Industry Today
2. Empirical Information
3. Key Performance Indicators
4. Product Quality
5. Case Study Examples
6. Conclusions
3© SE-CURE AG
![Page 4: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/4.jpg)
Questions ...
• Did you ever go after the world„s next model/standard or hype?
– If yes, how (much) did it help you?
• Are you able today to tell what targeted, actual and historical values of your main KPIs are?
– If not, would you be interested to know?
© SE-CURE AG 4
![Page 5: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/5.jpg)
Software Industry Today
• The software industry still isn't doing a very good job at managing by the numbers
”Oftentimes, software projects are managed by just three metrics: schedule/effort and critical defects
found during testing”
• This is a flatland view for a multi-dimensional terrain problem: "flying a plane using only a watch and a fuel gauge“
• Other metrics must be in the equation as well, representing what will be built (product scope/quality) and how it is built (process)
© SE-CURE AG 5
![Page 6: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/6.jpg)
Biking (1/3)
© SE-CURE AG 6
My home
Summary:
32.67 km2 h 18 min
Question:
Am I fit?
![Page 7: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/7.jpg)
Mountain Biking (2/3)
© SE-CURE AG 7
My home
![Page 8: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/8.jpg)
Mountain Biking (3/3)
© SE-CURE AG 8
Elevation: 14-16%
Total ascent: 1„107 mCalories burnt: 1„697
![Page 9: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/9.jpg)
Problem Statement
”You can’t fix
what you refuse to measure”
• When quantitative information is missing:– Management is like an emperor without
clothes: decisions are not based on facts
– Improvement initiatives result in satisfying standards/models instead of tangibly improving measured capability
9© SE-CURE AG
![Page 10: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/10.jpg)
What is Improvement?
• It is NOT about
–Satisfying a model/standard
– Implementing the next hype
• It is about becoming
–More predictable (performance)
–Faster (efficiency)
–More effective (scope)
–Better (quality)
© SE-CURE AG 10
![Page 11: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/11.jpg)
Being Successful
• Knowing the (performance) capability of your organization through the collection and analysis of historical data
• Making credible commitments in terms of what will be delivered when against what cost
• Managing development once it starts; this requires earned value management, activity distribution management and effective scope and quality management
• Analyzing the impact of new initiatives by assessing how capability is affected in which areas; this prevents organizations from chasing hypes
© SE-CURE AG 11
![Page 12: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/12.jpg)
2. Empirical Information
• Empirical Models
– Putnam SLIM
– COCOMO II
• Empirical Laws
– Minimal development time!
• Benchmarking Data
– Guru: Capers Jones
• Observation
– Availability hardly known, seldom used ...© SE-CURE AG 12
![Page 13: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/13.jpg)
Empirical Models
© SE-CURE AG 13
Empirical Models(Putnam SLIM, COCOMO II)
+Benchmarking Data
features
schedule
effort
capability
characteristics(product, platform, personnel, project)
size
![Page 14: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/14.jpg)
Empirical Laws
© SE-CURE AG 14
effort
scheduleTnTs Te
-25% +100%
> 100%
-50%
Ee
Es
En
Im
po
ssib
le
Reg
ion
!
Legend:
Ts, Es : fastest scheduleTn, En : nominal schedule/effortTe, Ee : lowest effort
Impractical Region
![Page 15: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/15.jpg)
Schedule Acceleration
© SE-CURE AG 15
log (
effort
)
log (schedule)Tn‘
Optimal development time
Tn
Question:
What would even be a better change?
size ↓productivity ↑
staffing rate ↑
![Page 16: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/16.jpg)
Benchmark (Jones 2008)
© SE-CURE AG 16
![Page 17: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/17.jpg)
Cost Categories
© SE-CURE AG 17
Essential Non-Essential
= Quality Related
Value-Added Non-Value-Added
Core PreventionSupportAppraisal,
rework
![Page 18: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/18.jpg)
Non-Value-Added, Non-Essential
© SE-CURE AG 18
Quality Related Costs
Cost of Non-Conformance
Failure Costs
Internal External
Cost of Conformance
Prevention Costs
Appraisal Costs
![Page 19: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/19.jpg)
Benchmark - Process Efficiency (Jones/Sassenburg)
© SE-CURE AG 19
![Page 20: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/20.jpg)
© SE-CURE AG 20
Industr
y E
xam
ple
![Page 21: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/21.jpg)
3. Key Performance Indicators
© SE-CURE AG 21
”Getting too little or too much data is easy, identifying and extracting the relevant data and converting it to
meaningful information for everyone is the challenge”
![Page 22: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/22.jpg)
KPIs versus Metrics
• KPIs (= meaningful information) are meant to gauge progress toward or benchmark against vital, strategic objectives usually defined by upper management
• Metrics (= data) at a lower detail level are measured attributes of a process or a product
• The goal of KPIs is to foster greater visibility and faster reaction to opportunities and threats, hereby enabling informed decision-making
© SE-CURE AG 22
![Page 23: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/23.jpg)
KPI Selection Criteria
• Must support project management in analysing, planning and monitoring projects
• Must inform management where a project stands and in what direction it is heading
• Must support business units in measuring their capability improvement over time
• Must support organisations in comparing/benchmarking business units
© SE-CURE AG 23
![Page 24: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/24.jpg)
Questions to Answer
• Project Performance
– What is the prediction of the performance of the project?
• Process Efficiency
– How efficient is the development process?
• Product Scope
– How large and stable is the scope of the planned effort in terms of features and size?
• Product Quality
– What is the expected quality of the resulting product (reliability and maintainability)?
© SE-CURE AG 24
![Page 25: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/25.jpg)
Derived KPI Categories
• Project Performance (= how predictable?)
– Cost, schedule, staffing rate, productivity
• Process Efficiency (= how fast?)
– Effort distribution (Cost of Quality model)
• Product Scope (= how much?)
– Features, deferral rate, size, re-use
• Product quality (= how well?)
– Complexity, test coverage, removal efficiency, defect density
© SE-CURE AG 25
![Page 26: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/26.jpg)
Best Practice KPI Set
© SE-CURE AG 26
Project Performance Product Scope
Schedule Feature size
Budget Deferred
Staffing rate Size
Productivity Re-use
Process Efficiency Product Quality
Core Complexity
Support Test coverage
Prevention Defect density
Appraisal/rework Removal efficiency
![Page 27: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/27.jpg)
Typical Maturization Effects
© SE-CURE AG 27
Project Performance Product Scope
Schedule Feature size
Budget Deferred
Staffing rate Size
Productivity Re-use
Process Efficiency Product Quality
Core Complexity
Support Test coverage
Prevention Defect density
Appraisal/rework Removal efficiency
![Page 28: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/28.jpg)
Software Specific?
© SE-CURE AG 28
Project Performance Product Scope
Schedule no Feature size no
Budget no Deferred no
Staffing rate no Size no
Productivity no Re-use no
Process Efficiency Product Quality
Core no Complexity no
Support no Test coverage no
Prevention no Defect density no
Appraisal/rework no Removal efficiency no
![Page 29: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/29.jpg)
Key Points to Remember
1. Assessing real software performance capability requires a multi-dimensional view
2. Compliance to process models and/or standards is no guarantee for high performance capability, although it may help (pragmatism versus bureacracy)
3. The set of Key Performance Indicators allows measuring real performance capability improvements over time and benchmarking
© SE-CURE AG 29
![Page 30: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/30.jpg)
Internal View
How easy is the product to
understand, maintain and test?
1. Complexity 2. Test coverage3. Defect density4. Removal efficiency
4. Product Quality?
30© SE-CURE AG
How well does the product
work?
External View
influences
![Page 31: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/31.jpg)
How difficult is it to understand, maintain and test the product?
Complexity
31© SE-CURE AG
Understand Maintain Test
“Low coupling, high cohesion !”
Number of dependencies on other systems
Cyclomatic complexityFan-out
Number of places that require attention upon
modificationCyclomatic complexityChange propagation
Code duplication
Number of required tests and required test timeCyclomatic complexity
Complexity
ISO9126
![Page 32: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/32.jpg)
Cyclomatic Complexity
© SE-CURE AG 32
Number of Independent Test Paths = # edges - # nodes + 2or
Number of decision points + 1
sequence:1-2+2 = 1
if / then:3-3+2 = 2
while loop:3-3+2 = 2
do / until loop:3-3+2 = 2
........
???
![Page 33: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/33.jpg)
Any Preference?
© SE-CURE AG 33
Cyclomatic complexity?
![Page 34: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/34.jpg)
Empirical Values (SEI)
Cyclomatic Complexity
Risk Evaluation
Bad fix probability
1-10Low risk
testable code5%
11-20 Moderate risk 10%
21-50 High risk 30%
> 50Very high risk
untestable code> 40%
© SE-CURE AG 34
![Page 35: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/35.jpg)
Testing Productivity
© SE-CURE AG 35
100%
0%
effort/time
Low productivity
High productivity
Failure
s0%
effort/time
Low productivity
High productivity
Coverage rate Failure discovery rate
100%
Covera
ge
![Page 36: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/36.jpg)
Possible Coverage Strategy
• Objective: obtain high testing productivity
• Condition: reject complex code
• Steps
– Invoke at least <> functions in <>% of the source files (or classes)
– Invoke <>% of the functions
–Attain <>% condition/decision coverage in each function
© SE-CURE AG 36
![Page 37: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/37.jpg)
General Message
Low cyclomatic complexity
+
High test coverage
=
High quality code
© SE-CURE AG 37
![Page 38: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/38.jpg)
Defect Model
© SE-CURE AG 38
Defects removed
Incoming
defects(Residual)
Outgoing
defects
New defects(Injected)
Process Step:- remove some old defect- inject new defects- remove some new defects
![Page 39: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/39.jpg)
Example
Defects Req. Arch.Low Level
DesignCoding
UnitTesting
IntegrationTesting
System Testing
Residual 0 10 10 10 15 10 7
Injected 20 10 10 20 5 4 3
Removed 10 10 10 15 10 7 5
Remaining 10 10 10 15 10 7 5
Removal efficiency
50.0% 50.0% 50.0% 50.0% 50.0% 50.0% 50.0%
Total injected 20 30 40 60 65 69 72
Total removed 10 20 30 45 55 62 67
Cumulative efficieny
50.0% 66.7% 75.0% 75.0% 84.6% 89.9% 93.1%
© SE-CURE AG 39
Defect density is the number of remaining defects per size unit
![Page 40: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/40.jpg)
Measuring Product Quality
• Complexity
– Fact extractor systems (for fan-out, cyclomaticcomplexity)
– Archive data mining tools (for change propagation)
– Code clone detectors (for clone detection)
• Test coverage
– Use dedicated code coverage systems
– Choose a coverage measure beyond statement coverage
• Defect density and removal efficiency
– Use the defect reporting/tracking system
© SE-CURE AG 40
![Page 41: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/41.jpg)
Identifying Weak Areas
An aggregated number is good for monitoring, but to act you need to know where in the code measures are necessary
Software visualization can efficiently implement the link back to the product
Raw data tables can hardly give insight in the problem at hand
Look at the measurement data
Use insight boosting techniques
41© SE-CURE AG
![Page 42: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/42.jpg)
Usage: - show distribution of metrics (cyclomatic complexity, fan-out) on source code
Tree Maps
© SE-CURE AG 42
onlineQuote
employerPortal
phoenix
phoenix
acc
newBusiness
newBusinessInstallation
recon
reportServer
0 20Cyclomatic complexity
© S
olidSourc
e B
V
![Page 43: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/43.jpg)
Usage: - show distribution of dependencies (fan-out) and clones- correlate dependencies with metrics (cyclomatic complexity)
Hierarchical Edge Bundles
Id. 090601 - Slides
© SE-CURE AG 43
© S
olidSourc
e B
V
![Page 44: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/44.jpg)
Usage: - visualize trends and discover correlations in data tables
A table with 33.000 lines
Table Lens
© SE-CURE AG 44
© S
olidSourc
e B
V
![Page 45: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/45.jpg)
Modification request related update
Time
Trend Analysis
Id. 090601 - Slides
© SE-CURE AG 45
© S
olidSourc
e B
V
![Page 46: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/46.jpg)
5. Case Study Examples
• Context
–2 B2B manufacturers (Case A, case B)
–Product functionality highly determined by software
–Case A: applications are safety-critical
–Case B: security of information is crucial
• Current performance
–CMMI L2/3 (internal assessments)
© SE-CURE AG 46
![Page 47: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/47.jpg)
Issues
• No benchmarking data for deferral rate and re-use level
– Context specific and lack of data
– Finding: deferral rate unknown
• No benchmarking data for test coverage and complexity
– Market ...!
– Finding: test coverage unknown, complexity high
• Feature size (FP) = Product size (KLOC)?
© SE-CURE AG 47
![Page 48: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/48.jpg)
© SE-CURE AG 48
![Page 49: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/49.jpg)
© SE-CURE AG 49
![Page 50: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/50.jpg)
© SE-CURE AG 50
![Page 51: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/51.jpg)
© SE-CURE AG 51
![Page 52: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/52.jpg)
© SE-CURE AG 52
![Page 53: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/53.jpg)
Defect Analysis (Example)
© SE-CURE AG 53= 2.5*5 + 7.5*25 + 15*20 + 35*35 + 75*15
= (1*30 + 3*10 + 2*10)/(1 + 3 + 2)
Total number of injected defects = 5„000
= (5„000-375)/5„000 = 375/125
![Page 54: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/54.jpg)
© SE-CURE AG 54
![Page 55: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/55.jpg)
© SE-CURE AG 55
![Page 56: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/56.jpg)
Weakest Areas
1. Inefficient process– Sum of appraisal and internal failure costs is
high (> 45%)
– Post-release distribution would even be worse!
2. Low code quality– Code complexity is high (> 15)
These are the primary causes
© SE-CURE AG 56
![Page 57: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/57.jpg)
Measure 1
• Increase prevention costs (training, reviews, inspections)
• Expected effects:
– Will increase removal efficiency and lower defect density
– Will decrease appraisal costs (testing) and rework costs (defect removal)
– Will lead to faster schedule, lower effort and higher productivity
© SE-CURE AG 57
![Page 58: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/58.jpg)
Measure 2
• Improve code quality
• Expected effects:
– Will increase test coverage
– Will (again) increase removal efficiency and lower defect density
– Will (again) decrease appraisal costs (testing) and rework costs (defect removal)
– Will (gain) lead to faster schedule, lower effort and higher productivity
© SE-CURE AG 58
![Page 59: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/59.jpg)
6. Conclusions
• Process capability only is not necessarily a good measure for high performance
• This does not mean that process improvement is bad
• To make it worth the investment, one should improve along a set of best practice KPIs instead of meeting the model/standard requirements only
© SE-CURE AG 59
![Page 60: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/60.jpg)
KPIs as Improvement Drivers
• Once management starts actively using such KPIs, projects are forced to bring and keep not only their measurement process in place, but many other processes as well …
–Requirements Management
–Project Planning
–Project Monitoring and Control
–…
© SE-CURE AG 60
![Page 61: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/61.jpg)
Another Capability Model ...
© SE-CURE AG 61
Level 5
Quantitative Performance Optimization
Level 4
Quantitative Portfolio Management
Level 3
Standardized Measurements Across Projects
Level 2
Basic Measurements at Project Level
Level 1
No or Ad Hoc Measurements
![Page 62: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/62.jpg)
New Organization
• Software Benchmarking Organization
– International consortium
• Founding partners
– SE-CURE AG (www.se-cure.ch)
– SolidSource BV (www.solidsourceit.com)
• URL
– www.sw-benchmarking.org
© SE-CURE AG 62
![Page 63: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/63.jpg)
© SE-CURE AG 63
Announcement
• Benchmarking Study–Reference: best practice KPI set
• Markets–Finance, automotive industry
• Objectives
–Assess performance of participants
–Benchmark results with industry values (average, best-in-class)
![Page 64: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/64.jpg)
Reference
Dr. Ir. Ton Tinus, Senior Development Director(Philips Healthcare, BU CV):
“The benchmarking study executed by SE-CURE AG in April 2009 was very useful and has brought many new insights. In addition, benchmarking the results with industry values was a useful
exercise. The introduced best practice KPI set is considered very powerful and we have adopted
the entire set to monitor projects and improvements."
© SE-CURE AG 64
![Page 65: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/65.jpg)
Benchmarking Approach
• For each participant
– Make available project data for 1-3 projects
– SE-CURE AG and partners will analyse the data and benchmark the results
• Includes on-line visit with report as outcome
• All
– Overall (anonymized) results will be presented in 2010
– Optionally, results will be published at conferences and/or in refereed journals
• Interested?
– Contact me at [email protected]
– See also: www.sw-benchmarking.org
© SE-CURE AG 65
![Page 66: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/66.jpg)
Products and Services
See www.se-cure.ch/Products.html
See www.se-cure.ch/Services.html
© SE-CURE AG 66
![Page 67: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/67.jpg)
Project Reporting Cockpit
© SE-CURE AG 67
The Project Reporting Cockpit will enable you to define and monitor a set of KeyPerformance Indicators (KPIs) for your project. Historical, target and actual values aredisplayed in overviews and charts, from which information can be consumed at aglance.
![Page 68: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/68.jpg)
Project Support Toolkit
© SE-CURE AG 68
The Project Support Toolkit provides powerful instruments to project managers andmeasurements specialists to plan, monitor and control their projects on a moredetailed level, using the same set of KPIs as in the Project Reporting Cockpit.
![Page 69: (Software) Management by Numbers - wibas · (Software) Management by Numbers ... schedule/effort and critical defects found during testing ... Compliance to process models and/or](https://reader034.fdocuments.in/reader034/viewer/2022051802/5ae9598a7f8b9ad73f8bbc4f/html5/thumbnails/69.jpg)
1-Day Workshops
© SE-CURE AG 69