METHODS FOR THE ECONOMIC EVALUATION

OF CHANGES TO THE ORGANISATION AND

DELIVERY OF HEALTH SERVICES

Rachel Meacock

Centre for Health Economics, University of York

6th July 2017

Background

• Established methods exist for the economic evaluation of

new health technologies seeking NHS funding in England

• Treatments go through mandatory NICE appraisal process

• Changes to the organisation and delivery of health services,

including policy changes, (service interventions) are funded

from the same NHS budget, but not covered by this process

• Often rolled out without supportive evidence or evaluation

Background

• Inconsistent approaches - differing levels of scrutiny likely to

result in allocative inefficiency in the health system

• Resulted in a lack of methodological development and cost-

effectiveness evidence

• Whilst principle of assessing cost-effectiveness should apply

to all NHS spending, methods will need adapting in places to

enable evaluation of service interventions

Aim

To contribute to the development of methods for the economic

evaluation of service interventions

1. Method to quantify effects of service interventions in terms

of QALYs in absence of primary data collection on HRQoL

2. Demonstrate how survival analysis can be used to improve

treatment effect estimates associated with service

interventions (length of life component of QALY)

Part 1:

Quantifying the effects of service interventions in terms of

QALYs in the absence of primary data collection on HRQoL

Introduction

• Often estimate the impact of service changes on mortality

• Useful indicator of the impact of a programme, but tells us

nothing about the intervention’s value

• To assess cost-effectiveness, we need to estimate the

impact of a programme in terms of QALYs

• Problem = usually evaluate using administrative data which

does not contain information on HRQoL

Proposed method

• A QALY ‘tariff’ applied to mortality outcomes to estimate the

QALY gains associated with detected mortality reductions

• Discounted and quality-adjusted life expectancy (DANQALE)

tariff

• Stratified by single year of age (18 - 100) and sex

• Represents the average stream of remaining QALYs for an

individual i in each age-sex group a from general population

DANQALE

Two components of the QALY:

• Length of life component: Sex-specific life expectancy

estimates at each single year of age taken from 2008-10

ONS interim life tables

• QoL adjustment: Age-sex specific mean EQ-5D values from

2006 wave of the Health Survey for England

We calculate the DANQALE (𝑄𝑖𝑎) for each individual i in each

age-sex group a as:

𝑄𝑖𝑎 = 1 −𝑚𝑖

𝑘=𝑎

𝐿𝑎

𝑞𝑘 1 + 𝑟 − 𝑘−𝑎

• 𝑚𝑖 equals 1 if the individual dies within 30days and 0

otherwise

• k indexes ages from a to the life expectancy of an individual

currently aged a(𝐿𝑎)

• 𝑞𝑘 is HRQoL at age k

• r is the discount rate (3.5%)

Analysis

• Attach DANQALE to each individual in the data

• Perform analysis as normal, but on DANQALE variable

• Can then compare to costs of the programme, either at the

individual or total programme level

Mortality Discounted QALYs

(DANQALE)

Total QALYs

AQ -0.9** [-1.4, -0.4] 0.07** [0.04, 0.11] 5,227

Limitations

• Likely to over-estimate health gains enjoyed by additional

survivors as assumes those surviving past 30days experience:

– Average life expectancy of the general population

– Average QoL of the general population

BUT

• Only captures QALYs gained due to mortality reductions i.e.

deaths averted

• Does not capture pure QoL improvements

Potential extensions

• Might want to update QoL values – later waves of HSE, other

data sources (e.g. SF-6D from Understanding Society)

• Could use condition-specific QoL estimates from audits if

available

Reference

Meacock R, Kristensen S, Sutton M. (2014). The cost-

effectiveness of using financial incentives to improve provider

quality: a framework and application. Health Economics, 23, 1-

13.

Part 2:

Using survival analysis to improve estimates of life year

gains in policy evaluations

Introduction

• Focus on methodology for estimating the impact of service interventions on length of life

• Length of life is a key outcome in cost-effectiveness analysis:– Cost per life year gained

– Cost per QALY

• Evaluations attempting to take a lifetime horizon can use admin data to estimate changes in short-term mortality

• Convert these to projected life year gains using published estimates of life expectancy from the general population

Previous approach

• Estimate the impact of a programme in terms of 30 day

mortality (binary outcome)

• Estimated reductions in this mortality are then translated into

life years gained

– Patients dying within 30 days are attributed no survival days

(effectively assumed to die instantly)

– Patients surviving past 30 days are assigned the remaining

age/gender-specific life expectancy of the general population

Limitations to previous approach

• Length of life of patients affected by interventions is likely to

differ from the general population – may lead to incorrect

estimations of the impact on life years gained

• True impact of policies on survival may be more complex,

potentially impacting survival over the whole life course

• Such longer-term effects not captured by evaluations

focusing solely on mortality rates within e.g. 30 days

Proposed solution

• Even with minimal data of 1 financial year available in many

administrative data sets, possible to observe most patients

for longer than 30 days

• Prolonged follow-up often ignored in policy evaluations

• Survival analysis is commonly used in clinical trials to

extrapolate gains in life expectancy from observed trial data

• Utilises all available follow-up information on patients rather

than applying an arbitrary cut-off window

Aim

• Examine whether the additional information available within

admin data sets on survival beyond usual 30 days

considered, albeit censored, can be used to improve

accuracy of estimated life year gains

• Demonstrate the feasibility and materiality of using

parametric survival models commonly employed in clinical

trials analysis to extrapolate future survival in policy evals

Motivating example

• Previous CEA of Advancing Quality (AQ) P4P programme

• Examine pneumonia patients only

• Consider a typical situation – data on dates of admission and

death are available for 1 financial year pre and post AQ

• Parametric survival models to estimate the effect of AQ on

survival over lifetime horizon

• Compare to results obtained using previous method

Data

• Hospital Episode Statistics linked to ONS death records

• Patients admitted to hospital in England for pneumonia:

– 2007/08 (pre-AQ)

– 2009/10 (post-AQ)

• Data period: 1st April 2007 – 31st March 2011

• Risk-adjustment: primary & secondary diagnoses (ICD-10),

age, gender, financial quarter of admission, provider,

admitted from own home vs institution, emergency vs

transfer

Methods

1. Comparison of methods on a development cohort

• Cohort of patients admitted to any hospital in England prior

to introduction of AQ (2007/08)

• Compare 3 methods for estimating remaining life years using

data from 2007/08 only

• Compare to observed survival of this cohort now available up

to 31st March 2011

Purpose of development cohort

• Illustrate difference in magnitude of estimated remaining life

years of a patient population when:

– Additional information available on survival past 30 days is utilised

– Risk of death is taken from the population under investigation rather

than general population figures

• Exercise also used to select the most appropriate functional

form for the survival models later used to evaluate AQ

Method i

Simple application of published life expectancy tariffs

• Simplified version of DANQALE applied in original evaluation

(does not incorporate discounting or QoL)

• 30 day mortality assessed as a binary outcome

• Gender-specific general population life expectancy estimates

at each single year of age (18 – 100) attached to patients

surviving beyond 30 days to estimate remaining life

expectancy

Method i

Remaining life expectancy:

𝐿𝑖𝑔𝑎

= 𝑠𝑖30 ∙ 𝐿𝑔𝑎

𝐿𝑔𝑎 is the life expectancy of an individual of gender g who is

currently aged a

𝑠𝑖30 equals 1 if individual i survives more than 30 days and 0

otherwise

Method i

Implicitly assumes that individuals surviving beyond 30 days

after admission survive, on average, the life expectancy of the

general population

Will produce an inaccurate estimate of the actual life

expectancy:

a) Period of survival within 30 days is not incorporated

b) Assumes life expectancy of individuals surviving beyond 30 days

after admission will be equal to that of the general population of their

age and gender

Ignores information on observed survival available in data

Method ii

Short-term observed survival plus application of published life expectancy estimates

• Extend method i to utilise all information on mortality available within year of data (2007/08)

• Can follow patients for between 1 – 365 days depending on admission date

• For those that died during the period, number of days survived between admission and death are counted

• Life expectancy again applied to those surviving past the end of the financial year

Method ii

Improves on method i by:

• Eliminating problem a) period of survival within 30 days is not

incorporated

• Reducing, but not eliminating, inaccuracies due to problem

b) assuming life expectancy of those surviving beyond 30

days after admission will be equal to that of the general

population of their age and gender

Method iii

Extrapolation using survival models

• Improve on method used for extrapolation by estimating

parametric survival models on the observed year of data

• Predict lifetime survival based on mortality rates of the

population of interest

• Considered six standard parametric models (exponential,

Weibull, Gompertz, log-logistic, log-normal, generalised

gamma)

Method iii

Model fit assessed using:

• AIC

• Tests of whether restrictions on the parameters in the

generalised gamma model suggest it could be reduced to the

simpler models it nests

• Examination of residual plots

External validity of extrapolations assessed by comparing

proportion of the cohort predicted to be alive at annual intervals

to the observed survival now available to 31st March 2011

Method iii

• In our case, while standard parametric models were able to

fit the observed data well, the tails of these distributions did

not correctly represent the pattern of future mortality

• Hazard rates experienced by our patient cohort changed

over time – extremely high-risk period shortly after

emergency hospital admission not representative of lifetime

risk of those surviving past this period

Method iii

Solution = estimate survival in 2 separate models:

• Short-term survival during the first year estimated on the

observed 1 year of data

• Extrapolation of long-term survival (1 year + after admission)

based on a model estimated on data excluding first 30 days

following admission

Long-term models represent hazards experienced by our

patient cohort after the initial high-risk period following

emergency admission – still much higher than general

population but significantly lower than when first admitted

Allowed us to estimate the effect of covariates on survival in both the observed

and extrapolated period

Method iii

Improves on method i by:

• Again eliminating problem a) period of survival within 30

days is not incorporated

• Using information on mortality risk from the patient

population under investigation rather than general population

estimates

We compare results at each stage as assumptions are dropped

– illustrates materiality of these developments

Application

2. Application to the evaluation of AQ

• Stage 1 demonstrates the materiality of the difference

survival analysis makes to the estimated life years remaining

of our patient cohort

• Stage 2 illustrates how these models can be used in an

applied programme evaluation

Dichotomous difference-in-differences (DiD) design:

𝐿𝑖𝑗𝑡 = 𝑓(𝑎 + 𝑋′𝑏 + 𝑢𝑗 + 𝑣𝑡 + 𝛿𝐷𝑗1 ∙ 𝐷𝑡

2 + 휀𝑖𝑗𝑡)

• 𝐿𝑖𝑗𝑡 is the life expectancy of individual i treated in hospital j at

time t

• f(∙) is the link function

• X is the vector of case-mix covariates

• 𝑢𝑗 are provider fixed effects

• 𝑣𝑡 are time fixed effects

• 𝐷𝑗1 is a dummy = 1 for hospitals that become part of AQ

• 𝐷𝑡2 is a dummy = 1 in the periods after the introduction of AQ

• 휀𝑖𝑗𝑡 is an individual-specific error terms

• 𝛿 is the DiD term, which is our coefficient of interest

Application

First, consider situation where data on admissions and deaths

are available for 1 financial year pre and post AQ

• Pre AQ (2007/08)

• Post AQ (2009/10)

BUT, survival analysis can utilise additional follow-up on the

pre-intervention group collected during same period as initial

follow-up of the post-intervention group

• Examine how life expectancy estimates are affected when including

additional follow-up available (2008/09 – 2009/10) on pre-AQ group –

should improve accuracy of estimates

Application

Use average partial effects to calculate the effect of AQ on life

expectancy

Estimate life expectancy for individuals admitted to AQ

hospitals in the post-AQ period under 2 scenarios:

– DiD term set = 0 (absence of AQ)

– DiD term set = 1 (presence of AQ)

Compare results to those obtained using methods i and ii

(linear regression on gen pop life expectancy estimates)

RESULTS

Part 1: Development cohort

Annual mortality rates for females

Age General

population

Patients admitted for pneumonia

2007/08

years % % (n deaths)

20 0.02 6.12 (98)

30 0.04 4.71 (191)

40 0.10 11.33 (309)

50 0.24 17.53 (291)

60 0.56 27.23 (584)

70 1.46 42.78 (783)

80 4.52 59.63 (1,469)

90 14.60 77.50 (1,142)

100 39.19 89.90 (109)

• Highlights importance of using information on the risk of death from the patient

cohort under investigation rather than general population figures when

estimating remaining life years

• Using gen pop figures would underestimate the annual mortality rate by a factor

of between 2 (age > 100 years) and over 300 (age 20)

Exponential Weibull Gompertz Log-

normal

Log-

logistic

Generalised

gamma

Internal

validity:

AIC 326,943 288,141 291,563 283,531 285,139 283,386

External

validity:

Time point Predicted survival, % Observed

survival, %

31/03/08 56.76 60.02 60.02 60.10 59.78 60.21 61.05

31/03/09 36.02 46.51 52.87 49.08 47.63 48.96 49.73

31/03/10 25.93 39.26 52.37 43.92 41.77 43.67 43.86

31/03/11 20.04 34.30 52.24 40.49 37.92 40.14 39.31

Internal and external validity of different parametric survival functions

• Lowest AIC

• Wald test confirmed does not reduce to the log-normal

• Best performance on external validity – predicted proportion of cohort alive to

within 1% of observed survival at each of 4 annual time points available

Method Assessment period Extrapolation

method

Those alive at end

of assessment

period, n (%)

Estimated life years

remaining, mean

i Admission to 30 days Gen pop life

expectancy

82,208 (72.56%) 13.15

ii Admission to end of

financial year

Gen pop life

expectancy

69,158 (61.05%) 11.98

iii Admission to end of

financial year

Parametric survival

models

69,158 (61.05%) 9.19

Comparison of estimates of remaining life years for patients admitted

for pneumonia 2007/08 (n = 113,289)

• Taking account of additional information on survival past 30 days reduced the

estimate of average remaining life years by 9% (method ii)

• Using survival models to extrapolate future survival reduced original estimate by

30% (method iii)

RESULTS

Part 2: Application to the evaluation of AQ

Patients admitted in 2007/08 Patients admitted in 2009/10

North West Rest of England North West Rest of England

n 17,993 95,296 19,946 106,365

Age at admission 71.7 72.2 71.9 72.8

Female, % 49.8% 48.7% 50.3% 49.1%

Comorbidities, n 1.79 1.65 1.99 1.92

Unadjusted

mortality within 30

days

28.4% 27.3% 25.6% 26.0%

Dead by end of

the financial year

40.7% 38.6% 37.3% 37.3%

Descriptive statistics for patients admitted for pneumonia, by region

and time period

• Pre-AQ the unadjusted mortality rate was higher in the North West within 30

days of admission and persisted in the longer-term to end of financial year

• Unadjusted mortality rates decreased in both regions, with a greater reduction in

the North West – positive effect of AQ on mortality previously detected

Estimated effect of AQ on the remaining life expectancy of

patients admitted to hospitals in the North West in 2009/10

Method i Method ii Method iii

Source of life expectancy

estimates

Gen pop life

tables

Gen pop

life tables

Survival analysis using 1 financial year of

follow-up

Short-term model:

Entry time =

admission

Long-term model:

Entry time = 31 days

post admission

Estimation method OLS OLS Generalised gamma Generalised gamma

DiD coefficient

(robust t stat)

0.154

(2.39)

0.221

(3.04)

0.103

(2.64)

0.089

(1.71)

Observations 239,600 239,600 239,600 156,860

Deaths, n 63,845 91,272 91,272 26,785

Life expectancy for patients

admitted in North West

2009/1013.218 11.982 9.041

Counterfactual estimate, life

expectancy for patients in

North West in absence of AQ13.064 11.761 8.730

Effect of AQ on life

expectancy for those

admitted in North West0.154 0.221 0.311

Interpretation

• Lower absolute estimates of life expectancy both in the

presence and absence of AQ were expected from methods ii

and iii – additional deaths were observed and risk taken from

patient cohort under investigation

• Despite lower absolute estimates of life expectancy, estimate

of the effect of AQ increased – indicates that AQ impacted on

survival beyond 30 day post-admission window

• Generalised gamma parameterized in the AFT metric –

coefficients < 1 indicate time passes more slowly – failure

(death) expected to occur later as a result of AQ

Method iii

Source of life

expectancy estimates

Survival analysis using 1 financial

year of follow-up

Survival analysis using all available

follow-up ( to 31/03/10)

Short-term

model:

Entry time =

admission

Long-term model:

Entry time = 31

days post

admission

Short-term

model:

Entry time =

admission

Long-term model:

Entry time = 31

days post

admission

DiD coefficient

(robust t stat)

0.103

(2.64)

0.089

(1.71)

0.142

(3.62)

0.101

(2.26)

Observations 239,600 156,860 239,600 164,438

Deaths, n 91,272 26,785 110,747 45,290

Life expectancy for

patients admitted in

North West 2009/109.041 8.439

Counterfactual estimate,

life expectancy for

patients in North West in

absence of AQ

8.730 8.059

Effect of AQ on life

expectancy for those

admitted in North West0.311 0.380

Interpretation

Utilising additional follow-up data available on pre-AQ group:

• Increased precision of estimates

• Slightly decreased estimated remaining life expectancy for

the cohort both in the presence and absence of AQ

• Further increased estimated treatment effect of AQ

Discussion

Demonstrated:

– Feasibility of using parametric survival models to extrapolate future survival in policy evaluations

– Materiality of the impact this has on estimates of remaining life years of a patient cohort and a policy treatment effect

Detected impact of AQ beyond 30 day window usually assessed shows advantage of survival analysis – ability to capture effects of policies over the whole life course

In pre- and post- evaluation design, survival models can be developed on the pre-intervention population and predictive performance evaluated against observed follow-up available during post-intervention period – external validity

Future work

• For estimates of life years gained to be used in CEA, the

stream of remaining life years need to be adjusted for QoL

and discounted – quite simple extensions

• More sophisticated survival models

Reference

Meacock, Sutton, Kristensen, Harrison. (2017). Using survival

analysis to improve estimates of life year gains in policy

evaluations. Medical Decision Making, 37, 415-426.

Overall conclusions

• Development of methods and applications of economic

evaluation of service interventions has the potential to

improve allocative efficiency

• Still a LONG way to go, but (hopefully) offered some useful

approaches

• Both strands of health economics have made impressive

methodological progress in different aspects of evaluation –

could learn a lot from each other

THANK YOU

Contact details:

rachel.meacock@manchester.ac.uk

@RachelMeacock

Method i

Remaining life expectancy:

𝐿𝑖𝑔𝑎

= 𝑠𝑖30 ∙ 𝐿𝑔𝑎

𝐿𝑔𝑎 is the life expectancy of an individual of gender g who is

currently aged a

𝑠𝑖30 equals 1 if individual i survives more than 30 days and 0

otherwise

Method ii

𝐿𝑖𝑔𝑎

= 𝑠𝑖𝑡∗ ∙ 𝐿𝑔𝑎 + (1 - 𝑠𝑖

𝑡∗) ∙ (𝑡𝑖ϯ

- 𝑡𝑖0)

Where 𝑠𝑖𝑡∗ is a binary indicator equal to 1 if individual i survives

to the end of the observation period 𝑡∗

𝑡𝑖ϯ

is the date of death for individuals who die before the end of

the observation period

𝑡𝑖0 is the date of admission

Method iii

Following estimation of survival models, created additional rows

of data for each individual for each possible future year up to

age 100

Estimated the probability of surviving to that year, allowing for

the progression of time and increments in age – analogous to

estimating transition probabilities in a Markov model:

𝑚𝑖𝑡 (𝑎𝑖0, 𝑥𝑖) =

𝑠𝑖𝑡 (𝑎𝑖𝑡,𝑥𝑖)

𝑠𝑖,𝑡−1 (𝑎𝑖𝑡,𝑥𝑖)– 1

𝑚𝑖𝑡 is the probability that individual i will die by time t, given that

they have survived to time t-1

𝑠𝑖𝑡 is the probability that individual i will survive to time t, given

the values of their covariates x and their age 𝑎𝑖 at the time of

their admission

Then calculated the individual’s life expectancy using the sum of

the probability of surviving to the end of the first year and the

survival rates for each subsequent year, to a max age of 100:

𝐿𝑖

= 1 − 𝑚𝑖1 ∙ 𝑡∗ − 𝑡𝑖

0 +

𝑗= 𝑎𝑖0+1

𝐴

𝑠𝑖,𝑗−𝑎𝑖0 𝑎𝑖𝑗 , 𝑥𝑖 ∙ (1 −1

𝑚𝑖𝑗+1− 𝑎𝑖0

)

𝐿𝑖 is the life expectancy of individual i

𝑚𝑖1 is the probability that individual i will die by the end of the

first year

𝑡∗ - 𝑡𝑖0 is the length of time between the individual’s admission

date and the end of the first year

A is the maximum age (100 years)