By : Ewa WasylkowskiSenior Consultant

Total Metrics

Benchmarking –Lessons You Don’t Want to

Learn the Hard Way

ACOSM 2003Sydney, Australia

Definitions“Benchmarking is the process of continuouslycomparing and measuring an organisation withbusiness leaders anywhere in the world to gaininformation which will help the organisation takeaction to improve its performance.”

American Productivity and Quality Centre Benchmarking Management Guide

Cambridge, Mass., 1993, Productivity Press

“Benchmarking is the process of determining ametric baseline for an organisational or functionalunit for the purpose of comparison”

Total Metrics – Metrics Glossary of Terms

Benchmarking Process Model

1. Determine BenchmarkPurpose

2.Determine BenchmarkMethod & Frequency

7. Filter, Map & ValidateData

12. Monitor Progress

PLAN

3. Select Benchmarker

4.Identify QuantitativeMeasures & QualitativeInformation

5. Identify Project Set

6. IdentifyBenchmarking Peers

8. Normalise AgainstPeer Data

9. Derive BenchmarkPerformance Indicators

10.Determine Gap

11. Benchmark Report -Recommendations

DO CHECK

13. Review Result

14 Resolve Disputes

15. RecalibrateBenchmarks

ACT

1. Determine BenchmarkPurpose

Benchmarking PurposesEstablish market position amongpeersDemonstrate competitiveness &continuous improvement in pricing &service levelsIdentify ‘Best Practice’Identify process improvementopportunitiesSet competitive range for metricsbaselineDecision making re outsourcing

Benchmarking MethodYear-on-Year– On-going annual reporting of

performance measures– Assesses trends - rate of improvement– Internal Benchmarking– Require 2-4 years data before suitable

base for comparison is availablePoint in Time– Assesses performance against peers– External Benchmarking– Must ensure benchmark period is

representative of past and/or futureperiods

IT Benchmarking AreaApplications DevelopmentMaintenance & SupportNetworkData CentreHelp DeskDesktop/LAN ServerCharge Back Practices

Benchmarking Lessons -Planning

There are no ISO (or other) standardsfor IT benchmarking.

Benchmarking is an imprecise tool as itis not possible to find directlycomparable:

OrganisationsContractsBenchmark Periods

The method and underlyingassumptions must be transparent andauditable.

Criteria for SelectingBenchmarker

Relevant Experience– Within Region eg Australia, Asia/Pacific– Within Industry Sector– With Organisations of similar size

Nature and Extent of BenchmarkDatabase for selected metrics– Attributes & Measures Collected– Segmentation by Industry– Australian/regional data– Projects by project type– Currency of data– Data integrity and validation– Comparable Peers !!!!!!

Criteria for SelectingBenchmarker (cont.)Methodology Proposed– Extent to which a customised

solution is possible (if required)– Sampling techniques (if proposed)– Sample findings, reports &

conclusionsLogistics– Required resource commitments– Ability to meet project timetable

Cost

Metrics/PerformanceIndicators

Applications Development– Productivity

• Hours per Function Point delivered• Function Points delivered per Full Time

Equivalent (FTE)– Cost Effectiveness

• $ Cost per Function Point delivered

Maintenance & Support– Productivity

• Function Points maintained per FTE– Cost Effectiveness

• $ Cost per 1000 FPs maintained.

Benchmarking Lessons– Planning cont.

Effective benchmarking begins atthe contract stage.Capabilities and costs ofbenchmarking organisations varywidely.It is naïve to expect a single resultthat factors in all the complexitiesof the systems development &business environment.

Identify Project SetBenchmark period determinescandidate projects.Sample set vs entire populationFor sample set:– Sampling techniques

• Systematic• Stratified• Cluster sampling

– Sample set sizeCategorise projects by type

A & D Project/Activity TypesDevelopment

New Custom Built

EnhancementCustom Built

Quarterly Release

Package Customisation

Pre-DevelopmentEstimates

Feasibility Studies

Business Cases

Cancelled, Withdrawn,On HoldPackages ‘Vanilla’Non Development

PM / Consulting only

DB Conversion & Tuning

Project Deployment/Impl.

Function PointGenerating

Non - FunctionPoint Generating

Productivity Metrics

Hrs per FP delivered

FPs delivered per FTE

$’s per FP delivered

Profile Metrics

% Effort Expended

% Cost Expended

(by category)

Lesson 7 !!!!

Only projects thatdeliver FP’s should be

included in A&DProductivity metrics

M & S Project/Activity TypesMaintenance & Support

Production Support

Software/Hardware Upgrades

Capacity Planning &Management

Corrective Maintenance

Perfective Maintenance

Decommissioning

Production Control &Scheduling

Application Security/AccessControl

Daily Operations Support

Support of ProductionEnvironment

Table Updates

Help Desk Support

Non - FunctionPoint Generating

M & S Metrics

Software Portfolio Size

FP’s maintained perFTE

$ Cost per 1000 FP’sMaintained

Identify Project Set

Start Benchmark Period

End Benchmark Period

Clearly “In” Projects

Effort & Cost Invoiced

RepresentativeBenchmark Year ?

Must establish whetherbenchmark period project set istypical of past and indicative offutureChanges in proportion of effortexpended may arise from– Planned changes in project type

mix– Altered demands of project types– Altered productivity & efficiency

Identify BenchmarkingPeers

Selecting benchmarking peers &Benchmarker are synonymous.– A valid benchmark must be

based upon comparable dataBe pro-active in selectingbenchmarking peers.– Profile your organisation and/or

contract.Establish representative set ofpeers

Benchmarking Lessons– Planning cont.

Inclusion/exclusion rules forprojects that span benchmarkperiod boundaries must be clearlydefined and equitable.

Data Filtering, Mapping & Validation

Measure STANDARDS MEASUREMENT METHOD

DATA VALIDATION

EFFORT

Project Chart of Accounts Project Phases Project Activities FTE Definition

Time Recording MethodResource Levels Uncollected Work EffortNormalised Effort Re-work Effort

Internal Validation Benchmark Validation ?

COST

Project Cost Model Direct Labour Cost only?

Actual vs Invoiced $ Spend Calculation

Financial Audit Benchmark Validation ?

FUNCTIONAL SIZE

FSM Standard Adjusted/Unadjusted FPs

Measured Result Default Counts Derived From PDR Approximated “Back-fired”

Internal Review External Audit Benchmark Validation ?

Data NormalisationFinancial Factors– Cost of Living– CPI Adjustments– Exchange Rates

Industry Sector and SizeOperating Environment– Differing Awards– Regulatory Constraints

Scope of ServicesTechnology PlatformsData Currency

Data Normalisationcont.

Service LevelsApplication Type & businessenvironmentProject MixSoftware Delivery Methods– Custom built– COTS

Regional MarketsOther contract unique

Benchmarking Lessons– Analysis

Benchmarking primarily involvesdata mapping not data collection.Data submitted must be uniformlyvalidated to ensure comparability.Effort & Cost data must becorrectly mapped to FPs delivered.The data normalisation methodmust be transparent andauditable.

Impact FactorsIdentify factors that impactbenchmark results which arebeyond the control of supplier:– Industry Sector– Technology Environment– Project Type Mix– Rework due to requirements

instability– Non-standard systems software– Other

Review ResultsReview by benchmarker beforeissuing draft report.Report presentationClient & Supplier ReviewUpdate based upon feedbackApprovalFinal Report

Resolve DisputesDetail dispute resolutionmechanism in contract.– Report review feedback– Internal dispute resolution

• Tie-breaker mechanism

– Arbitration - Independent person• Formal notice in writing• Benchmarker must co-operate• Benchmarker right of response

– Legal resolution

Benchmarking Lessons– Review

At outset agree process by which reportis issued, reviewed & approved.

In contract include provision forappointment of ‘Independent Person”.

A “dispute” may be simply the failureof two reasonable parties to reachagreement upon a possible range ofoutcomes.

Recalibrate BenchmarksBenchmarker to specifycompetitive rangeIdentification of “New Metrics”Adjustment of Contract Metricsto New MetricsRemoval of work from contractscope

Benchmarking Lessons– Act

Remember the primary purpose ofbenchmarking is process improvement.

Agree next benchmark period andbenchmarking process improvements.

Benchmarking is not a replacement forregular performance reporting.

Summary

Benchmarking should strengthenrather than divide the relationshipbetween client and supplier.

All parties should understand andacknowledge the limitations of thebenchmarking process.

The true benefits of benchmarking liein the discovery of processimprovement opportunities.

Thank Youand

Good Luck with yourBenchmarking !

Total Metrics Pty LtdSuite 1 / 667 Burke RoadCamberwell Victoria 3124

Phone (03) 9882 7611Fax (03) 9882 7633Email Ewa@Totalmetrics.com