1

Measuring the Future Measuring the Future MIT and Stanford Benchmark the Help DeskMIT and Stanford Benchmark the Help Desk

Enterprise 2006 Enterprise 2006 Educause Enterprise TechnologyEducause Enterprise Technology

May 25th 2006May 25th 2006

© Copyright William Clebsch, Greg Anderson, Chris Lundin, Oliver Thomas, 2006. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2

Deep BenchmarkingDeep Benchmarking

Project began summer 2002Project began summer 2002

Article published in EDUCAUSE Quarterly in Article published in EDUCAUSE Quarterly in Jan/Feb 2004Jan/Feb 2004

Details, charts, metrics, article available atDetails, charts, metrics, article available at

http://web.mit.edu/ist/about/benchmarkinghttp://web.mit.edu/ist/about/benchmarking

3

We had to go deep, then be strategic We had to go deep, then be strategic

Establish partnership, scope

General understanding• Processes, definitions

Site visitsRaw dataMetrics

Corrections

Sharing, teaching

Another tool for management, cultural change

Intense, deep,iterative

project work

4

• Define scope

• Set goals

• Form team

• Develop plan

• Teach team

• Launch

Prepare

• Research literature

• Visit sites

• Map processes

• Define data

• Collect data

• Define metrics

• Calculate metrics

Study &Define

• Draft a dashboard

• Populate with available data

• Test and revise

• Make easily repeatable

• Use in operations

Operationalize

• Use dashboard

• Publicize

• Replace existing reports

• Explore next area for bench-marking

Leverage

• Interpret at high-level

• Identify areas to explore

• Identify potential improvements

• Test initial changes

Interpret& Test

Continuously test and revise

Deep Benchmarking: 5 Iterative Phases

5

Metrics lead to action on many frontsMetrics lead to action on many fronts

OrganizationalOrganizational• Increased budget;

automated support

• Measure staff

• Stanford re-org: consolidation of all client-facing work

• Put HD staff on solid funding, not one-time $

ManagerialManagerial• Sharing metrics

with staff, teams

• Refining metrics – measure the right things

• Specific responses: e.g., create “swat” team for Oracle rollout

• Everyone tracks his/her time

CulturalCultural• Understanding

data; recognizing similarities, push for action

• Collaboration across schools

• Apply metrics to all projects

• Value people with technical and behavioral skills

6

We linked goals to specific metrics and We linked goals to specific metrics and then created a dashboardthen created a dashboard

Invest Appropriately• % of budget• Clients served/FTE

Be Cost Effective• Cost per case by topic • Total costs by topic• Cases by media, including

self-help

Be Responsive• Elapsed time per case

(days)• Call abandonment• Hold time• Time to answer• % of cases resolved

on 1st contact

Support Customer Needs with High Quality Service

• Annual customer survey• Spot-surveys on selected

transactions

Develop Effective, Mature Processes• # of contacts vs. # of days to resolve• Origin of Help Desk cases

Develop High Performing, Competent Teams

• Employee satisfctn. survey• Individual perf. metrics• Team performance metrics• Training $$ / FTE• % Help Desk certification• Case volume compared to

staff skills mixSupport Rollout of New Systems• Case volume by topic 3 months

before and after launch • Minutes per case

Cost per Ticket

$0

$20

$40

$60

$80

$100

$120

$140

$160

$180

$200

AccountsBackup

Business AppsBusiness Func

Cluster

ConnectivityCourseware

EmailHardware

OS Software

OtherPrinting

Desk SoftwareSecurity/Virus

Web

MIT Stanford

8

Total Annual Cost by Help Desk Topic

$0

$80,000

$160,000

$240,000

$320,000

$400,000

$480,000

AccountsBackup

Business AppsBusiness Func

Cluster

ConnectivityCourseware

EmailHardware

OS Software

OtherPrinting

Desk SoftwareSecurity/Virus

Web

Annual Cost

MIT

Stanford

9

Planning a deep benchmarking projectPlanning a deep benchmarking project

Choose a good partnerChoose a good partner Invest in site visitsInvest in site visits Plan for many iterationsPlan for many iterations Have a dedicated project managerHave a dedicated project manager Metrics must be actionable and tell a storyMetrics must be actionable and tell a story Allow time for others to internalizeAllow time for others to internalize Ensure management wants to use metrics in Ensure management wants to use metrics in

a meaningful waya meaningful way

10

Cultural Change:Cultural Change:Management by FactManagement by Fact

Use facts to dispel the myth of the anecdoteUse facts to dispel the myth of the anecdote

http://web.mit.edu/ist/about/benchmarkinghttp://web.mit.edu/ist/about/benchmarking

Questions? Questions?

11

Improving Help Desk ServicesImproving Help Desk ServicesThrough MeasurementThrough Measurement

12

Improving Help Desk Services Improving Help Desk Services through Measurementthrough Measurement

Measurement and metrics are not the goal; Measurement and metrics are not the goal; service improvement isservice improvement is

Where we started Where we started What we experienced:What we experienced:

– Individual PerformanceIndividual Performance– Team PerformanceTeam Performance

What we learned and how we’re moving What we learned and how we’re moving forwardforward

13

Measuring IndividualsMeasuring Individuals Few are measured as closely as the Help Desk staff: Few are measured as closely as the Help Desk staff:

dealing with staff concernsdealing with staff concerns The “Easy-To-Measure”The “Easy-To-Measure”

– Hours in Ready ModeHours in Ready Mode– Tickets ResolvedTickets Resolved– Calls-into-Tickets RatioCalls-into-Tickets Ratio– Direct customer feedbackDirect customer feedback

The “Hard-To-Measure”The “Hard-To-Measure”– Phone service qualityPhone service quality– Process improvement suggestionsProcess improvement suggestions– Getting to root cause and systemic fixesGetting to root cause and systemic fixes– Sharing knowledge in the groupSharing knowledge in the group– Contribution to project workContribution to project work

14

Daily Team PerformanceDaily Team PerformanceHDL1 Daily Performance Metrics

ConsultantTotal Time "Ready" mode (hh:mm:ss)

# Calls Answered

# Tickets Created

# Tickets Resolved

Help Desk Activity

Summary

Alicia 6:08:00 16 10 9 35

Fernando 5:48:00 12 8 5 25

Adel 5:51:00 16 7 10 33

Gaby Sick

Fred Sick

Rahel 6:48:00 20 9 8 37

Frank 5:23:00 13 11 14 38

Jamal Sick

Art 5:11:00 14 10 7 31

Average for Group 5:51:30 15 9 9 33

Totals 35:09:00 91 55 53

Consultant % Time "Ready" mode% Calls

Answered% Tickets Created

% Tickets Resolved

% of Phone Calls Turned into Cases

Alicia 17% 18% 18% 17% 63%

Fernando 17% 13% 15% 9% 67%

Adel 17% 18% 13% 19% 44%

Gaby Sick

Fred Sick

Rahel 19% 22% 16% 15% 45%

Frank 15% 14% 20% 26% 85%

Jamal Sick

Art 15% 15% 18% 13% 71%100% 100% 100% 100%

Thursday, May 13, 2010

15

Simplistic Measures Can Trigger Simplistic Measures Can Trigger Non-Productive Work (and not Non-Productive Work (and not

improve service)improve service) Focus on “time spent” can cause staff to Focus on “time spent” can cause staff to

inflate timeinflate time Focus on “tickets handled” can cause Focus on “tickets handled” can cause

“ticket stealing”“ticket stealing” Focus on Individual can damage TeamFocus on Individual can damage Team

– Aim for “Fair Share”Aim for “Fair Share”– Reward Fair Share contributions with extra Reward Fair Share contributions with extra

assignmentsassignments

16

Measuring TeamsMeasuring Teams The “Easy-To-Measure”The “Easy-To-Measure”

– Calls handled/Abandon RateCalls handled/Abandon Rate– First Contact ResolutionFirst Contact Resolution– Time to ResolutionTime to Resolution– ““Fair Share” Contributions Fair Share” Contributions

The “Hard-to-Measure”The “Hard-to-Measure”– Overall strength and quality of teamOverall strength and quality of team– Knowledge development within the teamKnowledge development within the team– Participation in service deploymentsParticipation in service deployments– Project and Knowledge Development workProject and Knowledge Development work

Teams: Compete or Support?Teams: Compete or Support?

17

Improving our Phone HandlingImproving our Phone Handling

Increased Increased hours in hours in “Ready Mode”“Ready Mode”

Gave feedback Gave feedback to staff on their to staff on their own “Ready own “Ready Time”Time”

Encouraged Encouraged web web submissions to submissions to the communitythe community

Abandon Rate (%)

0%

5%

10%

15%

20%

25%

Sep-02Nov-02Jan-03Mar-03May-03Jul-03Sep-03Nov-03Jan-04Mar-04May-04Jul-04Sep-04Nov-04Jan-05Mar-05May-05Jul-05Sep-05Nov-05Jan-06Mar-06May-06Jul-06

18

Encouraging Web SubmissionsEncouraging Web Submissions

HelpSU Submission Modes

35.0%

40.0%

45.0%

50.0%

55.0%

60.0%

65.0%

May-04Jul-04Sep-04Nov-04Jan-05Mar-05May-05Jul-05Sep-05Nov-05Jan-06Mar-06May-06Jul-06

Web/Auto Created Manually Created

Linear (Web/Auto Created) Linear (Manually Created)

19

De-emphasis Phone CallsDe-emphasis Phone Calls

Total 5-HELP Calls Received

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Sep-02Nov-02Jan-03Mar-03May-03Jul-03Sep-03Nov-03Jan-04Mar-04May-04Jul-04Sep-04Nov-04Jan-05Mar-05May-05Jul-05Sep-05Nov-05Jan-06Mar-06May-06Jul-06

20

Managing IT Services Managing IT Services By The NumbersBy The Numbers

Senior Management Support CriticalSenior Management Support Critical– Sharing “Daily Morning Report”Sharing “Daily Morning Report”– Sharing Weekly “5-Day Unresolved”Sharing Weekly “5-Day Unresolved”– Top 5/Bottom 5 GroupTop 5/Bottom 5 Group

» Sometimes group shame can go too farSometimes group shame can go too far

Accountability Embedded in Accountability Embedded in Director/Manager/Staff GoalsDirector/Manager/Staff Goals– HelpSU handling included in goals.HelpSU handling included in goals.

21

Help Desk “Morning Report”Help Desk “Morning Report”

Help Desk Morning ReportOverall Status:

On Target

May 15, 2006 Legend: On Target Minor Issues Issues

Today's Staffing Out In Yesterday's Phone Activity Unresolved HelpSU Cases Morning SnapshotGaby here for 4 hours today Answered 93

Abandoned 1 Night Service 0 ITSS ITSS ITSS HD

Equivalent FTE present . Total 0.5 8.5 Overflow to Voicemail 0 HDL1 HDL2 UNIXYesterday's Staffing Out In Requeued 0 Assigned 1 5 0

Fred, Gaby ,Jamal are out sick. Total 93 Work in Progress 15 10 1Alicia in at 10am Abandoned Rate 1% Pending 26 61 0 Avg Call (min) 5

Total 42 76 1

3 6

Yesterday's Planned EventsYesterday's Ticket

Activity

Web Cases Received 40 22 0

Cases Manually Created 55 9 0

Total 95 31 0

Yesterday's Ticket Status:

Assigned 0 1 0Work in Progress 9 4 0

Pending 18 9 08:00am Resolved 55 15 0

Unplanned Events:

See Call Analysis Graph tab

Today's Planned Events:

22

Unresolved Help RequestsUnresolved Help RequestsThe value of leadership and focusThe value of leadership and focus

Unresolved HelpSU Cases All ITSS Groups

After 5 Days

0

250

500

750

1,000

1,250

1,500

1,750

12/31/041/31/052/28/053/31/054/30/055/31/056/30/057/31/058/31/059/30/0510/31/0511/30/0512/31/051/31/062/28/063/31/064/30/065/31/066/30/06

5-Day Old Cases Linear (5-Day Old Cases)

23

Assuring Campus-Wide Service, Assuring Campus-Wide Service, Preserving Brand ReputationPreserving Brand Reputation

A single University-wide request tracking A single University-wide request tracking system (system (HelpSUHelpSU) has its pros and cons:) has its pros and cons:– Great economy of scale in our BMC/Remedy Great economy of scale in our BMC/Remedy

investment.investment.– Single customer interface raises user Single customer interface raises user

satisfactionsatisfaction– Behind the scenes, 250+ groups and 1,200+ Behind the scenes, 250+ groups and 1,200+

consultants makes quality control a challenge.consultants makes quality control a challenge.– Weekly work with all service providers in Weekly work with all service providers in

neededneeded

24

Med School Help RequestsMed School Help RequestsHelping our Partners FocusHelping our Partners Focus

Medical School Unresolved HelpSU Cases

More Than 5 Days Old

0

100

200

300

400

500

600

12/2/0512/16/0512/30/05

1/13/061/27/062/10/062/24/063/10/063/24/064/7/06

4/21/065/5/06

5/19/066/2/06

6/16/066/30/067/14/067/28/068/11/068/25/06

5-Day Old Unresolved MedIRT Groups Other Med Groups

Linear (5-Day Old Unresolved) Linear (Other Med Groups) Linear (MedIRT Groups)

25

Tools Keep On Serving: Tools Keep On Serving: Self-Service Password ResetsSelf-Service Password Resets

Password Self-Resets By Month

0

100

200

300

400

500

600

700

800

September

OctoberNovemberDecember

JanuaryFebruaryMarch

April May June July

August

Fy 2003/04 FY 2004/05 FY 2005/06

Linear (FY 2004/05) Linear (Fy 2003/04)

26

Our Current Improvement Our Current Improvement TargetsTargets

Improved phone service quality by auditing Improved phone service quality by auditing callscalls

Folding our ordering process (for all IT Folding our ordering process (for all IT services) into an expanded Service Deskservices) into an expanded Service Desk

Direct satisfaction surveys to a sampling of Direct satisfaction surveys to a sampling of our customers each weekour customers each week

Remote control support through WebExRemote control support through WebEx Investigating a knowledge base for both Investigating a knowledge base for both

support staff and customers. support staff and customers.

27

Absorbing the Benchmarking Project Absorbing the Benchmarking Project Into the Organization and CultureInto the Organization and Culture

Products, Methodology, and Theme

28

Absorbing the ProductsAbsorbing the Products

DashboardDashboard– Valuable as upward / outward reporting toolsValuable as upward / outward reporting tools– Continue to try to make dashboards work for Continue to try to make dashboards work for

operational managersoperational managers– Snapshot surveys provide critical component to Snapshot surveys provide critical component to

dashboarddashboard

29

Absorbing the Products - DashboardAbsorbing the Products - Dashboard

30

Absorbing the Products - DashboardAbsorbing the Products - Dashboard

31

Absorbing the Products - DashboardAbsorbing the Products - Dashboard

32

Absorbing the MethodologyAbsorbing the Methodology

Applied methodology to Network Security team Applied methodology to Network Security team with great effectwith great effect

Results:Results:– Significant process improvementSignificant process improvement

– Workload under controlWorkload under control

– Even stopped doing some things entirelyEven stopped doing some things entirely

– Getting customer feedback became positive experience!Getting customer feedback became positive experience!

33

Security Security Dashboard Dashboard

20042004

34

Security Security Dashboard Dashboard

20062006

35

Absorbing the MethodologyAbsorbing the Methodology

Okay, maybe that was a little too fast.

36

Absorbing the MethodologyAbsorbing the Methodology

37

Absorbing the ThemeAbsorbing the Theme

Using metrics to strengthen relationshipsUsing metrics to strengthen relationships Measuring successMeasuring success Getting staff to buy into itGetting staff to buy into it

“In God we trust; all others bring data.”- W. Edwards Deming

38

Absorbing the ThemeAbsorbing the Theme

Getting staff (and managers) to buy into itGetting staff (and managers) to buy into it– Must be real, direct value to them:Must be real, direct value to them:

» More resources?More resources?» Less effort?Less effort?» Better relationships?Better relationships?

– Give them control!Give them control!» Give them the tools to prove you wrong!Give them the tools to prove you wrong!» Challenge them to turn lagging indicators into leading onesChallenge them to turn lagging indicators into leading ones

– Get a Rob SmyserGet a Rob Smyser– Use metrics to support your story,Use metrics to support your story,

never instead of itnever instead of it

39

Cultural Change:Cultural Change:Management by FactManagement by Fact

Use facts to dispel the myth of the anecdoteUse facts to dispel the myth of the anecdote

http://web.mit.edu/ist/about/benchmarkinghttp://web.mit.edu/ist/about/benchmarking

Questions? Questions?