A ‘PROM’ising Future

Esther Kwong

Academic F2

Dept Primary Care and Public Health

The Relationship Between Patient Reported and Other Process

Outcomes at Trust Level

Project Supervisor: Dr Paul Aylin

Educational Supervisor: Dr Graham Easton

Dept Primary Care and Public Health

Contents

The Context –Setting the Scene for PROMs NHS

What are Outcomes

Measuring Patient Reported Outcomes

National PROMs Program Overview

Research Question

Methods

Results

Discussion & Conclusions

What have I learnt…

The Past

“Despite a century of developments in medical technology, and vast improvements in the ability of medical science to prevent, diagnose and treat disease and ill health, attempts to measure the outputs of health care in terms of their impact on patients’ health have not progressed beyond Florence Nightingale’s time.”

Getting the most out of PROMs Kings Fund 2010

The Context

Reasons for Healthcare:

Live Longer

Better Quality of Life

= Better Health Outcome

But health services traditionally focused on

one outcome

Mortality

Setting the Scene for PROMS

Darzi Review “High Quality for All”

NHS White Paper

Equity and Excellence: Liberating the NHS

Appraisal of new technologies –

PRO data incorporated in the evaluation of new technologies

Routine measurement of pre/post elective surgery PROMs

Since April 2009

DH Long Term Conditions PROMs Pilot

Since 2010

Patient Centred

If quality is to be at the heart of everything we do, it must be

understood from the perspective of patients.

Patients pay regard both to clinical outcomes and their experience of the service...

Lord Darzi The ultimate measure by which to judge the

quality of medical effort is whether it

helps patients, as they see it.

Donald Berwick

What Are Outcomes

Traditional Ways of Planning = Measuring in terms of OUTPUT

• Quantifying what is produced, implemented, provided, and developed in the health service

Increasing Focus = Measuring in terms of OUTCOME

• Quantifying extent of any health impact on patients

• Change in various dimensions ~physiological (e.g. functional status) or psychological (e.g. attitudes)

• Can be harnessed from different sources

Sources of Outcomes

Outcomes

Clinical

Clinician Reported

Patient Reported

Measuring Patient Reported Outcomes

Patient Reported Outcomes (PRO)• Health status as perceived by the patient

Patient Reported Outcomes Measures (PROMs)• Measurement tools to harness this information• Can be used in two points in time to record change in health status• Can be assessed against patient progress or health interventions

received• Various types available• Much dedicated research and analysis on validating questionnaire

types

Types of PROMS

EQ5D

For any condition

Different Disease states

Aggregation and comparison

Economic evaluations

Generic

Oxford Hip Score

Outperform on sensitivity

Centred on a particular aspect/ clinical detail

Focused – useful for informing

Condition Specific

National PROMs Program Overview

Since 1 April 2009

Providers required to collect and report PROMs

Four key NHS funded elective interventions• Unilateral hip replacements • Unilateral knee replacements• Groin hernia surgery• Varicose vein surgery

Expected to invite patients to complete a pre-operative PROMs questionnaire (Q1)

Post-operative questionnaires (Q2) are then sent to patients following their operation after a specified time period.

… the NHS will be the first health care system in the world to measure what it produces in terms of health, rather than in terms of the production of health care.Getting the most out of PROMs Kings Fund

… the NHS will be the first health care system in the world to measure what it produces in terms of health, rather than in terms of the production of health care.

Getting the most out of PROMs Kings Fund

PROMs Used for the National Program

• Multi-dimensional – five areas• Responses record three levels of severity• Scores are weighted and combined to give a

single index

EQ5D index score

• Self rating health related quality of life• Places self reported health state on a point in a

line• Line ranges from 0 to 100

EQ5D Visual Analogue Scale

• Validated tool specific for Total Hip Replacements• 12 questions to assess function and pain, 0-4

points• Given as a single summed score from 0 to 48

Oxford Hip Score

• Validated tool specific for Total Knee Replacements

• 12 questions to assess function and pain, 0-4 points

• Given as a single summed score from 0 to 48

Oxford Knee Score

Research Topic

Aim:

To explore the relationship between routinely collected patient reported and other process outcomes at trust level

Null Hypothesis:

There is no relationship between patient reported and other process outcomes at trust level

Methods:

Aggregate analysis conducted using STATA 11 on trust level data

Participation and Coverage 2010

Participation rate of 69.7%.

•245,488 eligible hospital episodes •171,080 pre-operative questionnaires returned

Return rate of 75.8%

•147, 974 post-operative questionnaires sent out• 112,163 returned

National PROMs Key Final Results 09-10 Overview

EQ-5D Index score87.2% of hip replacement respondents

77.6% of knee replacement respondents

Recorded an increase in general health following operation

Oxford Hip and Knee Score95.7% of hip replacement respondents

91.4% of knee replacement respondents

Recorded an improvement following operation

Data Sources

Hospital Episode Statistics (HES) PROMs data 2010 August 2011 Publication Used

Aggregate Trust Level Data

2010

Dr Foster Data

Orthopaedic Revision RatesOrthopaedic Readmission

CaseloadStaff to bed ratio

HSMR

National Joint Registry Data

Orthopaedic Procedures Caseload

HES Inpatient Data

Elective Surgery Waiting TimesEmergency Admission Caseload

Data Comparison

PROMs Outcomes 2010 Data- Hip and Knee Data

Case Adjusted Health Gain

(Q2-Q1)

• EQ5D Index • EQ5D Visual Analogue Scale• Oxford Hip/ Knee Score

Other Process Outcomes/ Hospital Indicators Compared

1. Hospital Standardised Mortality Ratios

2. Dr Foster Orthopaedic Revision Relative Rate

3. Dr Foster Orthopaedic Readmission Caseload

4. National Joint Registry Data Orthopaedic Procedure Caseload

5. Hospital Episode Statistics Elective Surgery Waiting times

6. Hospital Episode Statistics Emergency Admissions Caseload

7. Hospital Staff to Bed Ratios

Descriptive Results Hip

Outcome Case

Numbers

Number of Trusts

(Observations)Missing Trusts

MeanStandard Deviation

Inter-quartile Range

EQ5D Index 22,270 127 21 0.395 0.0364 0.4215

EQ5D VAS 21,653 128 20 7.53 2.23 3.362

Oxford Hip Score

24,682 131 17 19.3 1.28 20.1

05

10

15

Den

sity

.25 .3 .35 .4 .45Case Adjusted EQ5d HG

0.0

5.1

.15

.2D

en

sity

0 5 10 15EQ5D VAS Case Adjusted health gain

0.1

.2.3

.4D

en

sity

14 16 18 20 22Oxford Hip Score Case Adjusted health gain

Correlations Hip

Indicator EQ5DIndex

EQ5DVAS

OHS

Revision Relative Risk -0.05 0.0308 -0.1263Readmission

Caseload -0.0919 0.0847 -0.1431

SurgeryWait Times

(days) -0.0275 0.0777 0.0606Emergencies

Caseload -0.0225 -0.0143 -0.137Hip Procedures

Caseload 0.2042 0.1939 0.1567 Orthopaedic Procedures Caseload 0.1924 0.1881 0.1258

Nurse to Bed Ratio -0.0295 -0.0645 -0.066

Staff to bed Ratio -0.0402 -0.0801 -0.0437

HSMR 0.0573 0.0451 0.0334.2

5.3

.35

.4.4

5E

Q5D

Inde

x C

ase

Adj

uste

d H

ealth

Gai

n

0 500 1000 1500NJR no. Hip operation procedures

Case Adjusted EQ5d HG Fitted values

05

1015

EQ

5D V

AS

Cas

e A

djus

ted

Hea

lth G

ain

0 500 1000 1500NJR no. Hip operation procedures

EQ5D VAS Case Adjusted health gain Fitted values

1416

1820

22O

HS

Cas

e A

djus

ted

Hea

lth G

ain

0 500 1000 1500NJR no. Hip operation procedures

Oxford Hip Score Case Adjusted health gain Fitted values

Regression Hip Regression Model

Hip F Probability b Coefficient R2

ValueConfidence

Intervals

EQ5D index Hip Operation

Caseload0.0398 0.0000276 0.0334

1.31 x 10-5

539 x 10-5

EQ5D VASHip Operation

Caseload 0.0612 0.0015296 0.0275

-7.27 x 10-5

313 x 10-5

OHSHip Operation

Caseload0.0460 0.0009363

0.0305 1.69 X 10-5

186 x 10-5

EQ5D index Orthopaedic

Operation Caseload0.0453 0 .0000138 0.0317

0.0291 x 10-5

2.72 x 10-5

EQ5D VASOrthopaedic

Operation Caseload0.0640 0.000775 0.027

4.58 x 10-5

159 x 10-5

OHS OrthopaedicOperation Caseload 0.0742 0.000425 0.0245

4.33 x 10-5

90.2 x 10-5

EQ5D VASWaiting Time

0.0836 0.0378 0.0267-510 x 10-5

8030 x 10-5

OHSWaiting Time

0.0734 0.0223 0.0281-210 x 10-5

4670 x 10-5

Description Results Knee

Outcome Case Numbers

Number of Trusts

(Observations)

Missing Trusts

Mean Standard Deviation

Inter-quartile Range

EQ5D Index23,318 180 219 0.299 0.0369 0.45

EQ5D VAS 22,591 177 222 1.836 2.105 3.035

Oxford Knee Score 25,413 189 210 14.81 1.43 1.653

05

10

Den

sity

.2 .25 .3 .35 .4 .45 EQ5D case adjusted HG

0.0

5.1

.15

.2D

en

sity

-5 0 5 10EQ5D VAS Case Adjusted HG

0.1

.2.3

Den

sity

10 12 14 16 18 20Oxford Knee Score case adjusted HG

Correlations Knee

Indicator EQ5DIndex

EQ5DVAS

OKS

Revision Relative Risk

-0.09 -0.0795 -0.1459

Readmission Caseload

0.1524 0.0124 0.2188Surgery

Wait Times (days)

-0.0408 0.0388 0.1257

Emergencies Caseload -0.0447 -0.1586 -0.0283

Hip Procedures Caseload

0.2176 0.1014 0.181 Orthopaedic Procedures Caseload

0.2215 0.0389 0.1693

Nurse to Bed Ratio

-0.2407 -0.1899 -0.3408

Staff to bed Ratio-0.3242 -0.4102 -0.185

HSMR0.102 0.104 0.0625

.2.2

5.3

.35

.4.4

5E

Q5D

Inde

x C

ase

Adj

uste

d H

ealth

Gai

n

9 9.5 10 10.5 11 11.5HES Emergency Admissions Caseload

EQ5D case adjusted HG Fitted values

-50

510

EQ

5D V

AS

Cas

e A

djus

ted

Hea

lth G

ain

9 9.5 10 10.5 11 11.5HES Emergency Admissions Caseload

EQ5D VAS Case Adjusted HG Fitted values

10

12

14

16

18

20

OK

S C

ase

Ad

just

ed

He

alth

Gai

n

50 100 150 200 250Foster RR Knee Revision

Oxford Knee Score case adjusted HG Fitted values

Regression Knee

Regression model F Probability b Coefficient R2

ValueConfidence

Intervals

EQ5D index Emergency Admissions

Caseload0.0003 -0.0158 0.135 -0.024

-0.007

EQ5D VASEmergency Admissions

Caseload0.0341 -0.488 0.0372 -0.94

-0.037

OKSEmergency Admissions

Caseload0.0331 -0.350 0.0382 -0.77

-0.033

Discussion – Important Findings 1

Weak positive correlations between Hip and Orthopaedic Procedures Caseload and all Hip PROMs health gain • Suggests the more procedures a trust does the better its quality

of hip replacement procedure perceived by patient• This is an expected correlation direction

Weak negative correlations between Emergency Admission Caseload and all Knee PROMs health gain • Suggests the more emergency admissions a trust has the worse

the patient perceived outcome for a knee replacement procedure• unexpected correlation direction, warrants further exploration into

relational factors – such as trust specialisation and quality relationship

Discussion -Important Findings 2

Weak Positive Correlation Between EQ5D Visual Analogue Scale and Oxford Hip Score health gain for hip patients and Waiting Times for Elective Surgery

• Suggests the longer a patient waits for elective surgery in a trust the more health gain perceived from hip operation

• Unexpected correlation direction • Disease progression factors are adjusted for• May be explained by expectation management

‘Patient Satisfaction = Patient Experience - Patient Expectation’• Longer waiting times may decrease expectation affecting

perceived outcome • Lead time difference

Discussions Limitations

Recruitment Bias• LSHTM Report to Dept of Health on PROMs recorded correlation

coefficient of -0.38 between EQ5D score and every 20% increased recruitment, suggesting low recruitment rates can introduce bias

• The report recommended a target recruitment rate of 80%

Response Bias• Studies suggest non responders were younger in all PROMs, This is

particularly evidenced in orthopaedic PROMs

Patient Reported Outcome Measures (PROMs) in Elective Surgery Report to the Department of Health, London School of Hygiene and Tropical Medicine

Conclusions

Weak/ Lack of Correlations suggests Patient Reported Outcomes are capturing an added dimension of quality that traditional process outcome and clinical indicators were not measuring

Weak correlations findings at trust level maybe due to aggregation, this could eliminate clinical variation within and between hospital services as well as patient characteristics• Evidence from clinical governance concluded acute hospitals services were

‘A mix of good and bad’• Analysis of PROMs at clinical level and unadjusted data may provide further

explanations and strengthen correlations

Lack of evidence/ data available for statistical relationship significance for correlations • Further work building larger aggregate data set on PROMs• Analysis on new PROMs data, or analysis spanning two years of PROMs data

What I learnt from this Academic Rotation

Nature of Rotation

2 days a week in GP surgery clinical duties

3 days a week dept based research and teaching activities

Research

Literature search on PROMs

Insight in health services research

Handling aggregate data

Data management

Statistical analysis on STATA

Seminars and Journal Club

Experience of life as an academic!

TeachingFormal Teaching courses/ training

Clinical Methods teaching 3rd year Imperial Students

Problem Base Learning facilitator

GPWealth of clinical experiences

Primary care setting exposure

Consultation simulation training

References

Bevan G , Skeller M, 2011 Competition between hospital and clinical quality BMJ 2011; 342:d3589Berwick D, Hiatt H, Janeway P, Smith R. 1997 An ethical code for everybody in health care BMJ 1997;315:1633Black N, Browne J, Cairns J. 2006. Health care productivity. British Medical Journal 333: 312–313.Brooks R, Rabin R, de Charro F. 2003. The Measurement and Valuation of Health Status using EQ-5D: A European Perspective. Kluwer: Dordrecht.Browne J, Jamieson L, Lawsey, J, van der Meulen J, Black N, Cairns J, Lamping D, Smith, S, Copley L,Horrockes, J. 2007. Patient Reported Outcome Measures (PROMs) in Elective Surgery. Report to theDepartment of Health. Available from: www.lshtm.ac.uk/hsru/research/PROMs-Report-12-Dec-07.pdf.Burge P, Devlin N, Appleby J, Gallo F, Nason E, Ling T. 2006. Understanding patients’ choices at the point of referral. Technical report TR359-DOH, Cambridge: RAND Europe. Available from: www.rand.org/pubs/technical_reports/TR359/.Darzi L. 2008. High Quality Care for All. NHS Next Stage Review: Final Report, Department of Health, London.Available from: www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_085825.Dawson J, Rogers K, Doll H, Using Patient Reported Outcomes Routinely: An example in context of Shoulder surgery Open Epidemiology Journal 2010, 3,42-52 Department of Health. 2008. Guidance of the Routine Collection of Patient Reported Outcome Measures (PROMs).Department of Health document DH_081179[1].pdf.Devlin N, Appleby J. 2010. Getting the Most Out of PROMs: Putting Health Outcomes at the Heart of NHS Decision Making. Kings Fund/Office of Health Economics: London.Dolan P. 1997. Modelling valuations for EuroQol health states. Medical Care 35(11): 1095–1108.2010).Greenhalgh J, Long A, Flynn Rob, 2004 The use of patient reported outcome measures in routine clinical practice: Lack of theory or lack or impact. Social Science and Medicine 60 (2005) 833-843EuroQol Foundation. Springer: Rotterdam.Hospital Episode Statistics: Finalised Patient Reported Outcome Measures (PROMs) in England: April 2009 – March 2010 Isis Outcomes Patient Reported Outcome Measures from the University of Oxford, Orthopaedic Pros http://www.isis-innovation.com/outcomes/orthopaedic /

London School of Hygiene and Tropical Medicine Patient Reported Outcomes on Elective Surgery, Report To Department of Health Dec 2007, http://www.lshtm.ac.uk/php/hsrp/research/proms_report_12_dec_07.pdf NHS North West. 2010. Advancing quality. Available from: www.advancingqualitynw.nhs.ukNational Council on Ageing and Older People, 1998 health promotion strategy for older people in Ireland. Adding years to life and life to yearsNational Network of Libraries of Medicine Guide 3: Define Measurable Goals, Outputs and Outcomes http://nnlm.gov/outreach/community/goals.htmlOffice of Health Economics. 2008. NHS Outcomes, Performance and Productivity. Report of the Office of Health Economics Commission. OHE: London.Szende A, Oppe M, Devlin N. 2007. EQ-5D Valuation Sets: An Inventory, Comparative Review and Users’ Guide.

Acknowledgements

I want to thank all those in the department who have contributed their expertise and advice towards this project and towards my educational development

• Dr Paul Aylin, Dr Graham Easton, Dr Jenny Lebus, • Dr Michael Soljak, Dr Sonia Saxena• Dr Fiona Hamilton, Dr Matthew Harris, Dr Eszter Vamos• Elizabeth Cecil, Farzan Rahman, Dr Ghasem Yadegarfar