Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1
-
Upload
harold-van-heeringen -
Category
Documents
-
view
209 -
download
1
description
Transcript of Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1
![Page 1: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/1.jpg)
Outsourcing Test activities
How to prove cost reductionsHarold van Heeringen, Metrics Consultant
Sogeti
Wednesday May 28, Milan, Italy
![Page 2: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/2.jpg)
Outline
• Problem definition• Factors that influence Test
Performance• Model CASE• Results & Conclusions
![Page 3: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/3.jpg)
Problem definition
• More and more outsourcing test activities
• Contracts - x% of cost reductions for the client in 1,2,3 years
• So far: Impossible to prove!>Compare bills ?>What about volume increase/decrease ?>What about defects found… and defects not
found?
![Page 4: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/4.jpg)
Test activities
Functional design
Technical design
Coding + Unit test
Systems test
User Acceptance Test
Client
Client
Sogeti
Client
![Page 5: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/5.jpg)
Factors that influence Test Performance
• The most important factors:>Number of defects found (in relation to
number of defects present)Many defects found – did we test well?Few defects found – did we test badly?
>Number of hours spentWe have spent 1.000 hours testing this project. Is that
OK?
>Time interval availableBuild phase ended late, but end date is fixed. What
can we do in the time interval available?
![Page 6: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/6.jpg)
The relation between these factors
• Extra short duration...>How many extra
hours do we need?
• Extra long duration...>How many extra
defects will we find?>How many defects
are not found?
Impossib
le
Unpractical
Eff
ort
(h
our)
DurationD
efe
cts
rem
ain
ing
![Page 7: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/7.jpg)
When do we test well?
• More defects found than expected?• Less hours spent than expected?
• But how do we know the number of defects and hours expected??
• We need an estimation tool!
Defects found < expected
Defects found = expectated
Defects found > expected
hours spent < expected
? + ++hours spent = expected
- ± +hours spent > expected
-- - ?
![Page 8: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/8.jpg)
QSM SLIM Estimate
• Project Estimation> Hours, duration, fte, costs, etc
> But also: defects !!!
![Page 9: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/9.jpg)
Benchmark test performance
• Benchmark supplier performance against client performance – but how??>Every project is different >Duration is critical variable>Test hours and defects found are highly
dependent on the Build performance
• How can we know the client’s performance?
• Hours spent• Defects found• Defects not found
![Page 10: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/10.jpg)
Model proposed: 3 stages
>1: Baseline analysis – assess the client situation
• PI (productivity duration)• Cost per test hour• Defect Tuning factor
>2: Operational execution – Estimate, measure and benchmark performance
• Simulate project with actual duration, but with client baseline parameters
• Measure actuals• Calculate Cost reductions
>3: Evaluation – Evaluate whether the cost reduction targets have been met
![Page 11: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/11.jpg)
Model step 1: Baseline analysis
• Select 4 – 5 representative projects>Size the projects in COSMIC/FPA/slocs>Administrate results in SLIM Datamanager
• Results:>Detailed analysis per project>SLIM Estimate WBS>Average/median Productivity index>Defect tuning factor>Costs per test hour
![Page 12: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/12.jpg)
CASE: Baseline analysis
• Contract: outsourcing systems testing • Target: 8% Cost reductions 1st year
• Baselinedata:
• 1 test hour = 80 euro> activities?> Fixed costs?> Waiting hours?
Project Size in
CFP
Hours S-test
Defects S-test
Defects A-test /1st M production
A 310 554 12 8
B 512 937 27 5
C 212 508 22 12
D 224 732 52 18
E 487 878 35 8
![Page 13: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/13.jpg)
Model step 2: Operational execution
• Choice: Measure everything, or measure predefined set of projects1 – Measure size 2 – Estimate project with client parameter
set and actual duration3 – Compare client estimation with Sogeti
actual data4 – Calculate Test Performance5 – Calculate Cost Reduction
![Page 14: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/14.jpg)
CASE: Operational execution
Step 1 - 5 projects selected to measureStep 1 - Size measurement in COSMICStep 2 - Estimation in SLIM, using client baseline parameters and actual durationProject Q
Scenario Defects S-test FS
Defects A-test + 1M Prod. FAP
Defects Total FT
FS/FT
Client baseline
30 31 61 0,49
Scenario Test hours TU
Hour rate R
Costs C (TU*R)
Client baseline
2.045 80 163.600
![Page 15: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/15.jpg)
CASE: Operational execution
Step 3 – Compare Sogeti actuals against baseline scenarioProject Q
Scenario Test hours TU
Hour rate R
Costs C (TU*R)
Client baseline
2.045 80 163.600
Sogeti actuals
1.800 82 147.600
Scenario Defects S-test FS
Defects A-test + 1M Prod. FAP
Defects Total FT
FS/FT
Client baseline
30 31 61 0,49
Sogeti actuals
35 30 65 0,53
![Page 16: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/16.jpg)
CASE: Operational execution
Step 4 – Assess Test Performance projects> Outsourcing Effectiveness
Did we test better than the client would have done?
> Outsourcing EfficiencyDid we test more efficiently than the client would have done?
Outsourcing Efficiency OF
Baseline C – Actual C Baseline C
Outsourcing Effectiveness OV
(Actual FS/FT) – (Baseline FS/FT) Baseline (FS/FT)
*100%
*100%
![Page 17: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/17.jpg)
CASE: Operational execution
• Project Q>Outsourcing Effectiveness:
OV = (0.53-0.49)/0.49 ≈ 8,2 %
>Outsourcing Efficiency: OF = (163.600–147.600) / 147.600 ≈ 9,7%
Outsourcing effectiveness
Outsourcing efficiency
- 100 %
100 % 0 %
100 %
- 100 %
0 %
No Cost Reduction
Cost Reduction
![Page 18: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/18.jpg)
CASE: Operational execution
• Step 5 – Calculate Cost Reduction>Maybe client rates OV higher than OF or
vice versa>Contract – agree on weight factors
OV = Outsourcing EffectivenessOF = Outsourcing Efficiencyw1 = weight factor OVw2 = weight factor OF
Cost Reduction CR
(w1*OV) + (w2*OF) w1+w2
![Page 19: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/19.jpg)
Model step 3: Evaluation
• After contract period: Analyze results
Outsourcing effectiveness
Outsourcing efficiency
- 100 %
100 % 0 %
100 %
- 100 %
0 %
No Cost Reduction
Cost Reduction
8%
8%
![Page 20: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/20.jpg)
CASE: Evaluation
• Evaluation client X, 1st year
• Total CR: € 129.143 / € 1.383.570 ≈ 9,3%• Substantial cost reductions, even more than
was promised!!
Project OV OF C Baseline CR % CR €
Q 9,7% 8,2% € 163.600 8,95% € 14.642
R 12,3% 6,1% € 312.150 9,2% € 28.718
S 8,2% 3,0% € 122.650 5,6% € 6.868
T 6,4% 13,3% € 718.120 9,85% € 70.735
U 15,6% 8,8% € 67.050 12,2% € 8.180
Total € 1.383.570 € 129.143
![Page 21: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/21.jpg)
First results
• Clients reactions are very positive• Model should be evaluated in
practice and should be developed further
• Please help us further develop the model:>Metrieken.sogeti.nl>[email protected]
![Page 22: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/22.jpg)
Conclusions
• The model proposed is objective and can be used in all organizations and all projects
• The client can now see the real cost reductions they achieve by outsourcing
• The supplier can also use the model as a commercial instrument and is able to support claims of cost reduction based on experience
![Page 23: Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1](https://reader035.fdocuments.in/reader035/viewer/2022062704/555d5138d8b42a52368b49b4/html5/thumbnails/23.jpg)
metrieken.sogeti.nl