Performance Indicators in the Health Service

Outline Introduction Performance Management Data League Tables Issues

Performance Indicators in the Health Service

Paul Hewson1

11th April

1University of Plymouth, email [email protected]


1 Introduction

2 Performance Management

3 Data

4 League TablesAssessing uncertaintyCase MixMaking allowance for the size of an organisationFunnel PlotsMonitoring changes over timeMethods from Industrial Quality Control

5 Issues


Aims of Management SSUs

To assess the student’s ability to:

Define a health service management problem.

Demonstrate an understanding of the historical andcontemporary background of the problem. How did thingsevolve this way? What are the current issues that needaddressing and are driving change?

Define a strategy and research possible solutions.

Propose and justify a particular solution.

Present the problem, possible solutions, and proposed solutionverbally, including answering questions,

Use appropriate visual aids to support the verbal presentation,


Aims of Unit

Appreciation of methods used in the clinical design ofPerformance Indicators;

How to interpret performance indicators in the context ofrandom fluctuation;

How to make allowance for different case mix;

AND thinking about this in a real-world context.


In particular:

Performance Indicators are an increasingly importantmanagement tool in the health service, consider the very wellpublicised Healthcare Commission Indicators, QoF indicatorsin primary care, as well as recent publication of surgeonspecific mortality rates.

No longer tools imposed from without, in many casesscientific evidence and practitioner input has been used todesign a suitable set of measures.


So we are interested in:

Best practice in performance indicator design;

How routine clinical information is coded into databases thatultimately becomes a performance indicator;

how we assess uncertainty;

how we make valid comparison on units (surgeons, hospitals,areas) which may differ due to context or patient case mix;

How to satisfy the people paying our wages (or theirrepresentatives) that we are delivering continuousimprovement in patient care.


Aims of today

To plan the rest of the unit, contact, logistics, assessment

To consider an overview of the role and practice ofperformance management

To present some information on exciting(?) technical aspects

Financial aspects will be considered later. We might also wantto consider data coding in more detail later.

To discuss topics that may be suitable for assessment


Assessment

Assessment will consist of:

A presentation made to a small audience,

Audience may include fellow students,

Presentation will be in a semi-formal environment.

Any volunteers for video-recording?


Assessment

A 20-minute slot should be allowed for each student,approximately:

(Up to) 15 minutes for the presentation and to answerquestions;

5 minutes for feedback from the audience

In addition:

5 minutes should also be allowed in the programme for set upof each presentation.

The assessor will need 10 minutes to complete the assessmentform including written feedback.

colorredElectronic presentations will not necessarily score morethan non-electronic ones (OHPs).


Performance Management: One style among many

For the foreseeable future, performance management is here tostay. But two papers (there are plenty more) remind us that thereare other management styles:

Adab, P., A.M. Rouse, M.A. Mohammed and T. Marhsall(2002) “Performance league tables: the NHS deserves better”Brit.Med.J. 324:95-98

Davies, H. and J. Lampel (1998) “Trust in PerformanceIndicators” Quality in Health Care 7:159-162


Known Risks with Performance Management

Tunnel vision (ignoring non-measured aspects of a service);

Sub-optimisation (setting modest improvement goals);

Convergence (aiming to match the average);

Gaming (dealing with easiest clients / problems first);

Ossification (avoiding innovation);

Misrepresentation (see National Audit Office (2001)Inappropriate adjustments to NHS Waiting Lists London:National Audit Office).


Data Sources

“Public agencies are very keen on amassing statistics -they collect them, add them, raise them to the nth power,take the cube root and prepare wonderful diagrams. Butwhat you must never forget is that every one of thosefigures comes in the first instance from the villagewatchman, who just puts down what he damn pleases”

Sir Josiah Stamp 1880-1941 (Governor of the Bank of England)


Data Quality

You can find plenty of other examples. For today, consider thepaper by Speigelhalter et al. (2002)2. Consider in particular:

The number of different sources of data recording the sameevents;

The reason for collecting these different data sets;

How useful the data were from any of them.

2Spiegelhalter, D.J., P.Aylin, N.G.Best, S.J.W. Evans and G.D.Murray(2002) ‘ Commissioned analysis of surgical performance using routine data:lessons from the Bristol Enquiry” J.R.Statis.Soc.A 165:191-231


Data Validity: are you measuring what you want tomeasure

“Not everything that counts is counted, and noteverything that can be counted counts” Albert Einstein(approximate quote).

A couple of hospital based clinical examples where this has beenconsidered includes:

McGlynn, E. and S. Asch (1998) “Developing a clinicalperformance measure” American Journal of PreventativeMedicine 14:14-21

Dorsch M., R. Lawrence, R. Sapsford, J. Oldham, D.Greenwood, B. Jackson, C. Morrell, S. Ball, M. Robinson andA. Hall (2001) “A simple benchmark for evaluating quality ofcare of patients following acute myocardial infarction” Heart86:150-154


Designing PM systems: technical aspects

A semi-technical discussion has been prepared by a RoyalStatistical Society working partyhttp://www.rss.org.uk/main.asp?page=1222.One example considered are simple pass-fail indicators. Where arethese used?

http://www.rss.org.uk/main.asp?page=1222


Data Validity

∴ a large part of performance indicators surrounds defining themsensibly in the first place. Options for developing an indicatorinclude borrowing a definition from somewhere else:

Miles, H., E.Litton, A. Curran, L.Goldsworthy, P.Sharples andA. Henderson (2002) “The PATRIARCH study: Usingoutcome measures for league tables: Can a North Americanprediction of admission score be used in a United Kingdomchildren’s emergency department?” Emerg.Med.J. 19:536-538

as well as quite elaborate procedures for developing a clinicalconsensus as to what should be measured:

Normand, S-L.T., B. McNeil, L. Peterson and R. Palmer(1998) “Methodology matters - VIII. Eliciting expert opinionusing the Delphi technique: identifying performance indicatorsfor cardiovasular disease” International Journal for Quality inHealth Care 10:247-260


Several Famous Problems with league tables

Small changes in performance can lead to very large changesin rank;

Small organisations more affected than large ones(randomness);

There is no allowance for “case” mix or the context in whichthe organisation operates.

One study will be quoted (there are many which report similarresults) suggesting that between 1.6% and 2.3% of variation inmortality rate was due to institutional effects: see Merlo, J., P.-O.Ostegren, K. Broms, A. Bjork-Linne, and H. Liedholm (2001)“Survival after initial hospitalisation for heart failure: a multilevelanalysis of patients in Swedish acute care hospitals” J. Epidemiol.Community Health 55:323-329.


Assessing uncertainty

Uncertainty in League Tables

The following slide has been extracted from Marshall andSpiegelhalter (1998)3, a paper approaching citation classicstatus in the BMJ.

This first chart shows confidence intervals around the raw livebirth rate.

There are arguments that all Performance Indicators shouldcome with some assessment of the possible uncertainty. Whathappens with Healthcare Commission Indicators?

3Marshall, E. C. and D. J. Spiegelhalter (1998) “Reliability of league tablesof in vitro fertilisation clinics; retrospective analysis of live birth rates.” Br.Med. J.316:1701-1705


Assessing uncertainty

Uncertainty in League Tables


Case Mix

Case Mix

The following slide has also been extracted from Marshall andSpiegelhalter (1998).

They have now applied a statistical model which makes someadjustment for case mix.

Vertical lines indicate median, top quartile and lower quartilerankings. How many clinics are clearly very ”good” or very“bad”

Could this be used with individual surgeon indicators?


Case Mix

Allowing for case mix


Funnel Plots

Funnel Plots

Funnel plots are common in meta-analysis.

The following slide has been extracted from Spiegelhalter(2002) 4.

Two hospitals appear to have an unusually high readmissionrate following treatment for a stroke

What adjustment has been made for case mix?

4Spiegelhalter, D. J. (2002) “Funnel plots for institutional comparison(letters to the editor)” Qual.Saf. Health Care11:390-391


Funnel Plots

Funnel Plots


Monitoring changes over time


We return to Marshall and Spiegelhalter (1998)

Having adjusted for case mix, we also try to estimate whatchanges have happened over time, along with an associateduncertainty measure

What are the implications for press-releases heralding a 2.1% dropin crime, 0.3% drop in road accidents . . . (insert clinical example ofyour choosing)?



Changes over time


Methods from Industrial Quality Control

Quality Control Charts

Rather more has been done looking at longer runs of data

An overview of such charts in healthcare is given by Woodall,20065.

The basic idea is stop pompous statisticians taking your dataaway and creating over elaborate models which nobody elseunderstands

The hope is that when such charts are designed carefully,YOU assess whether anything funny is going on.

5Woodall, W.H., “The Use of Control Charts in Health-Care andPublic-Health Surveillance” Journal of Quality Technology38:89-104



Cusum Charts



Cusum Charts

(well, I had to put at least one piece of maths in somewhere)

CUSUM =T∑t=1

xt − x0

The aim of the CUSUM chart is to monitor performancerelative to a target x0.

Level lines are good, downward slopes are bad, crossing theV-mask is very bad, especially if you had plenty of warningthat this was going to happen.

Consider the following cusum plot from Chang and McLean(2006)6 for joint replacement wound blisters.

6Chang, W.R. and I.P McLean ”CUSUM: A tool for early feedback aboutperformance?” BMC Medical Research Methodology 6:8



Cusum Charts


Outstanding Issues

From HM Treasury7:

“Performance information is a cornerstone of ourcommitment to modernise government. It provides someof the tools needed to bolster improvements in publicsector performance . . . . . . Good quality information alsoenables people to participate in government and exertpressure for continuous improvement. In addition toempowering citizens, this information equips managersand staff within the public service to drive improvement.Performance information is thus a catalyst for innovation,enterprise and adaptation.”

7H.M. Treasury (2001) Choosing the right fabric: A framework forPerformance Information London: HM Treasury


Outstanding issues

So the Treasury believe in:

Driving continuous improvement; a management and apractitioner tool;

Empowering Citizens

But note:

The Treasury do a lot of driving by controlling finance!


Issues to consider

What do Performance indicators do for patient care?

What do Performance indicators do for clinical practice?

What is our public (Patients / Potential Patients / Localresidents) and how are they served by PerformanceInformation

Financing the NHS


Provider Help with Preparing SSU Assessments

1) I must not read, mark or correct any piece of SSU writtenwork (or draft) unless it is sent to me by the SSUadministration team for marking.

2) I must not listen to a verbal presentation in advance or correctslides prior to an assessed presentation.

3) I may answer any specific question posed to me (by students)regarding SSU assessment preparation.

4) I am encouraged to give general advice and guidance on howto write a good written assessment or how to deliver a goodpresentation (as appropriate) throughout your SSU provision.

5) Following the marking of the assessment I am free to discusswith the students any aspect of their assessment that I wishto.


Recap on assessment

The following aspects of your presentation will be explicitlyconsidered:

1) Knowledge & understanding of the management problem

2) Research of possible solutions

3) Justification of proposed action

4) Use of appropriate visual aids

5) Quality of report


Your task

Find an area of healthcare subject that is or could beperformance managed

Determine how to gather evidence on the clinical and“statistical” suitability of different ways of managingperformance

Performance Indicators in the Health Service

Education

Transcript of Performance Indicators in the Health Service