Cost-effectiveness models to inform trial design:

Calculating the expected value of sample information

Alan Brennanand

J Chilcott, S Kharroubi, A O’Hagan

Overview • Principles of economic viability• 2 level Monte-Carlo algorithm & Mathematics• Calculating EVSI (Bayesian Updating) case studies

– Normal, Beta, Gamma distributions– Others – WinBUGS, and approximations.

• Illustrative and real example• Implications • Future Research

Example:- Economic viability of a proposed oil reservoir

• Some information suggesting there is oil• Could do further sample drilling to “size” the oil reservoir• Decision = “Go / No go” • Criterion = expected profit (net present value or NPV)

Is the sampling worthwhile? … that depends on …• Costs of collecting the data• Current uncertainty in reservoir size Expected gain from sampling = (P big reservoir*Big profits)+(P small reservoir*Big loss)–(Sample cost)

Analogies• Drug Development Project

– Go / No go decisions– Trial supports consideration of next decision (Phases to launch)– Criterion = Expected profit (NPV)– Correct decision

profit if good drug, avoided financial loss if not a good drug

• NICE / NCCHTA decision– Approval or not– Is additional research required before decision can be made– Criterion = Cost per QALY…. i.e. net health benefits– Correct decision better health (efficiently) if good drug,

avoided poor health investment if not a good drug

Principles• Strategy options with uncertainty about their performance• Decision to make• Sampling is worthwhile if

Expected gain from sampling - expected cost of sampling > 0

• Expected gain from sampling = Function (Probability of changing the decision|sample, .

amount of gain made / loss avoided)

• Applies to all decisions

Algortihm

2 Level EVSI - Research Design4, 5

0)Decision model, threshold, priors for uncertain parameters1) Simulate data collection: • sample parameter(s) of interest once ~ prior • decide on sample size (ni) (1st level)• sample a mean value for the simulated data | parameter of interest 2) combine prior + simulated data --> simulated posterior 3) now simulate 1000 times

parameters of interest ~ simulated posteriorunknown parameters ~ prior uncertainty (2nd level)

4) calculate best strategy = highest mean net benefit 5) Loop 1 to 4 say 1,000 times Calculate average net benefits 6) EVSI parameter set = (5) - (mean net benefit | current information)

Mathematics

2 Level EVSI - Mathematics 4, 5

Mathematical Formulation: EVSI for Parameters = the parameters for the model (uncertain currently).d = set of possible decisions or strategies.NB(d, ) = the net benefit for decision d, and parameters

Step 1: no further information (the value of the baseline decision) Given current information chose decision giving maximum expected net benefit. Expected net benefit (no further info) = (1)

i = the parameters of interest for partial EVPI -i = the other parameters (those not of interest, i.e. remaining uncertainty)

4 Brennan et al Poster SMDM 2002

5 Brennan et al Poster SMDM 2002

)NB(d,max Ed

2 Level EVSI - Mathematics 4, 5

Step 6: Sample Information on iExpected Net benefit, sample on i = (6)

Step 7: Expected Value of Sample Information on i(6) – (1) Partial EVSI =

(7)

This is a 2 level simulation due to 2 expectations 4 Brennan et al Poster

SMDM 20025 Brennan et al Poster

SMDM 2002

)NB(d,max|)NB(d,max EEEdidX i

idX iEE |)NB(d,max

Bayesian Updating

NormalBeta

Gamma

Normal Distribution

Normal Distribution 0= prior mean for the parameter

0= prior uncertainty in the mean (standard deviation) = precision of the prior mean

2pop = patient level uncertainty from a sample ( needed for Bayesian update

formula) = sample mean (further data collection from more patients /

clinical trial study entrants). = precision of the sample mean .

= sample variance 4 Brennan et al Poster

SMDM 20025 Brennan et al Poster

SMDM 2002

200 /1 I

X

2/1 sI

npop /22

Normal Distribution= implied posterior mean

(the Bayesian update of the mean following the sample information)

= implied posterior standard deviation

(the Bayesian update of the std dev following the sample information)

s

s

0

001

2022

0

221 /

/

nn

pop

pop

Normal Distribution - Implications• Implied posterior variance will always be smaller than the

prior variance because the denominator of the adjustment term is always larger than the numerator.

• If the sample size is very small then the adjustment term will almost be equal to 1 and posterior variance is almost identical to the prior variance.

• If the sample size is very large, the numerator of the adjustment term tends to zero, the denominator tends to the prior variance and so, posterior variance tends towards zero.

Normal DistributionNormal Bayesian Update (n=50)

0

50

100

150

200

250

300

0% 20% 40% 60% 80% 100%

Response Rate (T0)

Freq

uenc

y (1

000

sam

ples

)

Prior

Posterior aftersample 1

Posterior aftersample 2

Posterior aftersample 7

Beta Distribution

Beta / Binomial Distribution • e.g. % responders

• Suppose prior for % of responders is ~ Beta (a,b)

• If we obtain a further n cases, of which y are successful responders then

• Posterior ~ Beta (a+y,b+n-y)

Gamma Distribution

Gamma / Poisson Distribution e.g. no. of side effects a patient experiences in a year

• Suppose prior for mean number of side effects per person is ~ Gamma (a,b)

• If we obtain a further n samples, (y1, y2, … yn) from a Poisson distribution then

• Posterior for mean number of side effects per person ~ Gamma (a+ yi , b+n)

Bayesian Updating

Other Distributions

Other Distributions Beta Laplace

Inverted Beta LogisticCauchy 1 LognormalCauchy 2 NormalChi ParetoChi² 1 PowerChi² 2 RayleighErlang r-Distr.Expon. UniformFisher StudentGamma TriangularInverted Gamma WeibullGumbel

Bayesian Updating without a Formula • WinBUGS

• Put in prior distribution

• Put in data (e.g. sample of patients or parameter)

• Use MCMC to generate posterior (‘000s of iterations)

• Use posterior in model to generate new decision

• Loop round and put in a next data sample

• Other approximation methods (talk to Samer!)

Illustrative Model

First (Illustrative) Model • 2 treatments – T1 versus T0

• Criterion = Cost per QALY < £10,000

• Uncertainty in ……

• % responders to T1 and T0

• Utility gain of a responder

• Long term duration of response

• Other cost parameters

Illustrative Model d eNumber of Patients in the UK Uncertainty in Parameter MeansThreshold cost per QALY

Mean Model T0 T1 T0 T1Increment

(T1 over T0)Cost of drug 1 1 1,000£ 1,499£ 499£ % admissions 2% 2% 10% 11% 0%Days in Hospital 1.00 1.00 5.01 7.05 2.05

Cost per Day 200 200 589£ 589£ -£

% Responding 10% 10% 88% 70% -18%Utility change if respond 0.1000 0.0500 0.3119 0.2120 0.0998- Duration of response (years) 0.5 1.0 3.1 4.1 1.0

% Side effects 10% 5% 31% 24% -7%Change in utility if side effect 0.02 0.02 -0.06 -0.14 -0.08 Duration of side effect (years) 0.20 0.20 0.70 0.65 0.05-

Total Cost 1,308£ 1,937£ 629£

Total QALY 0.8394 0.5948 -0.2446

Cost per QALY 1,558£ 3,256£ -£2,570

Net Benefit of T1 versus T0 £7,086 £4,011 -£3,075

Sampled ValuesStandard Deviations

Illustrative Model a b cNumber of Patients in the UK 1,000 Threshold cost per QALY £10,000

Mean Model T0 T1Increment

(T1 over T0)Cost of drug 1,000£ 1,500£ 500£ % admissions 10% 8% -2%Days in Hospital 5.20 6.10 0.90

Cost per Day 400£ 400£ -£

% Responding 70% 80% 10%Utility change if respond 0.3000 0.3000 - Duration of response (years) 3.0 3.0 -

% Side effects 25% 20% -5%Change in utility if side effect -0.10 -0.10 0.00Duration of side effect (years) 0.50 0.50 -

Total Cost 1,208£ 1,695£ 487£

Total QALY 0.6175 0.7100 0.0925

Cost per QALY 1,956£ 2,388£ £5,267

Net Benefit of T1 versus T0 4,967£ 5,405£ £437.80

-£2,000

-£1,500

-£1,000

-£500

£0

£500

£1,000

£1,500

£2,000

-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Inc QALY

Inc

Cos

t

Illustrative Model d eNumber of Patients in the UK Uncertainty in Parameter MeansThreshold cost per QALY

Mean Model T0 T1 T0 T1Increment

(T1 over T0)Cost of drug 1 1 1,000£ 1,499£ 499£ % admissions 2% 2% 10% 11% 0%Days in Hospital 1.00 1.00 5.01 7.05 2.05

Cost per Day 200 200 589£ 589£ -£

% Responding 10% 10% 88% 70% -18%Utility change if respond 0.1000 0.0500 0.3119 0.2120 0.0998- Duration of response (years) 0.5 1.0 3.1 4.1 1.0

% Side effects 10% 5% 31% 24% -7%Change in utility if side effect 0.02 0.02 -0.06 -0.14 -0.08 Duration of side effect (years) 0.20 0.20 0.70 0.65 0.05-

Total Cost 1,308£ 1,937£ 629£

Total QALY 0.8394 0.5948 -0.2446

Cost per QALY 1,558£ 3,256£ -£2,570

Net Benefit of T1 versus T0 £7,086 £4,011 -£3,075

Sampled ValuesStandard Deviations

Illustrative Model a b cNumber of Patients in the UK 1,000 Threshold cost per QALY £10,000

Mean Model T0 T1Increment

(T1 over T0)Cost of drug 1,000£ 1,500£ 500£ % admissions 10% 8% -2%Days in Hospital 5.20 6.10 0.90

Cost per Day 400£ 400£ -£

% Responding 70% 80% 10%Utility change if respond 0.3000 0.3000 - Duration of response (years) 3.0 3.0 -

% Side effects 25% 20% -5%Change in utility if side effect -0.10 -0.10 0.00Duration of side effect (years) 0.50 0.50 -

Total Cost 1,208£ 1,695£ 487£

Total QALY 0.6175 0.7100 0.0925

Cost per QALY 1,956£ 2,388£ £5,267

Net Benefit of T1 versus T0 4,967£ 5,405£ £437.80

-£2,000

-£1,500

-£1,000

-£500

£0

£500

£1,000

£1,500

£2,000

-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Inc QALY

Inc

Cos

t

Illustrative Model Results • Baseline strategy = T1

• Cost per QALY = £5,267

• Overall EVPI = £1,351 per person

£1,319

£867 £878£716

£330

£-

£200

£400

£600

£800

£1,000

£1,200

£1,400

Valu

e of

Info

rmat

ion

All Six Durations Trial +Utility

UtilityStudy

Trial on %Response

EVSI (n=50)

EVSI for Parameter Subsets

EVI for % Responders to T0

0

50

100

150

200

250

0 50 100 150 200

Sample Size (n)

Valu

e of

Info

rmat

ion

EVSI (n)EVPI

Illustrative data collection cost = £100k fixed plus £500 marginal

Expected Net Benefit of Sampling

-200

-100

0

100

200

300

0 50 100 150 200

Sample Size (n)

Valu

e of

Info

rmat

ion EVSI

Cost ofSampling

ENBS

Expected Net Benefit of Sampling

Real Example

Second Example • Pharmaco-genetic Test to predict response

• Rheumatoid Arthritis

• Up to 20 strategies of sequenced treatments

• U.S. - 2 year costs and benefits perspective

• Criterion = Cost per additional year in response

• Range of thresholds ($10,000 to $30,000)

• Real uncertainty (modelled by Beta’s)

“Biologics”Anakinra ($12,697), Etanercept ($18,850), Infliximab ($24,112)*

Is Response Genetic?

91 patients, 150mg Anakinra, 24 week RCT1,2, gene = IL-1A +4845Positive response = reduction of at least 50% in swollen joints

1 Camp et al. American Human

Genetics Conf abstract 1088, 1999

2 Bresnihan

Arthritis & Rheumatism, 1998

0%

20%

40%

60%

80%

100%

Placebo Anakinra Gene+ve

Gene -ve

% a

chie

ving

"Sw

olle

n50"

0 .0 %

1 0 .0 %

2 0 .0 %

3 0 .0 %

4 0 .0 %

5 0 .0 %

6 0 .0 %

7 0 .0 %

8 0 .0 %

9 0 .0 %

1 0 0 .0 %

*Costs include monitoringAnakinra 100mgEtanercept 25mg eowInfliximab 3mg/kg 8 weekly 50% 50%100%

Respond ?Gene +ve? Yes …..

Yes AnakinraNo …..

PGtRespond?

Yes …..No Etanercept

No …..

Respond?Yes

AnakinraNo

Before 0 - 6 months

Before 0 - 6 months

A Pharmaco-Genetic Strategy

Strategy 1

Strategy 2

Partial EVSI: PGt Research onlyEVSI for PGt Research only

(for threshold = $20,000 per responder year gained)

$0.0$5.0

$10.0$15.0$20.0$25.0$30.0

10 20 50 100

200

500

1000

2000

5000

Perfec

t

Sample Size

EV

SI $

m

Caveat: Small No.of Simulations on 1st Level

Doing Fewer Calculations?

Properties of the EVSI curve • Fixed at zero if no sample is collected

• Bounded above by EVPI

• Monotonic

• Diminishing return

• Suggests perhaps exponential form?

• Tried with 2 examples – fitted curve is exponential function of the square root of n

EVI for % Responders to T0

0

50

100

150

200

250

0 50 100 150 200

Sample Size (n)

Valu

e of

Info

rmat

ion

EVSI (n)

EVPI

Exponential fit

EVSI = EVPI * [1- EXP( -0.3813 * sqrt(n) ]

Fitting an Exponential Curve to EVSI:Illustrative Model - % response to T0

Fitting an Exponential Curve to EVSI:

Pharmaco-genetic Test response Pharmaco-genetic Test Response

0

5

10

15

20

25

30

0 100 200 300 400 500

Proposed Sample Size (n)

Valu

e of

Info

rmat

ion

EVSI

Exponential Fit

EVPI

EVSI = EVPI * [1- EXP( -0.3118 * sqrt(n) ]

Unresolved Question • Does the following formula always provide a good fit?

• EVSI (n) = EVPI * [1 – exp -a*sqrt(n) ]

• The 2 examples are Normal and Beta

• Is it provable by theory?

Discussion Issues

Phase III trials Future Research Agenda

Discussion Issues – Phase III trials • Based on proving a clinical DELTA

• Implication is that if clinical DELTA is shown then adoption will follow i.e. it is a proxy for economic viability

• Often FDA requires placebo control (lower sample size), which implies DELTA versus competitors is unproven

• Could consider economic DELTA …….

Discussion Issues – Phase III trials Early “societal” economic models provide a tool for assessing:

1. What would be an economic DELTA?

2. Implied sample needed in efficacy trial for cost-effectiveness

3. What other information is needed to prove cost-effectiveness?

4. Will proposed clinical DELTA be enough for decision makers

Similar commercial economic models could link • proposed data collection with • probability of re-imbursement and hence with • expected profit (NPV)

Discussion Issues – Problems & Development Agenda

1. Technical - Bayesian Updating for other distributions

2. Partnership and case studies - to develop Bayesian tools for researchers who currently use frequentist only sample size calculation

3. Methods for complexity in Bayesian updating - e.g. the new trial will have slightly different patient group to the previous trial (meta-analysis and adjustment)

Conclusions• Can now do EVSI calculations from a societal

perspective using the 2 level Monte-Carlo algorithm

• Bayesian Updating works for case studies– Normal, Beta, Gamma distributions– Others need – WinBUGS, and/or approximations.

• Future Research Issues– Bayesian Technical – Collaborative Issues with Frequentist Sample Size

Thankyou