PRESENTING UNCERTAINTY AROUND COST-

EFFECTIVENESS ESTIMATES TO DECISION MAKERS: HOW

SURE ARE WE?Mohan Mohan BalaBala, PhD., PhD.

CentocorCentocor research & development, Incresearch & development, Inc..Josephine Josephine MauskopfMauskopf

RTI Health SolutionsRTI Health Solutions

Gary Gary ZarkinZarkinRTI InternationalRTI International

Overview of workshop

Present a brief review of the different Present a brief review of the different methods used to incorporate uncertainty methods used to incorporate uncertainty into economic analysis into economic analysis Examine issues with current approachesExamine issues with current approachesExplore alternate approaches to present Explore alternate approaches to present uncertainty to decision makersuncertainty to decision makers

Economic Analysis in Health Care

CostCost--effectiveness analysis often used by decision effectiveness analysis often used by decision makers to understand whether an intervention makers to understand whether an intervention provides good value for moneyprovides good value for moneyIncremental CostIncremental Cost--effectiveness Ratio (ICER) is effectiveness Ratio (ICER) is the metric commonly used to make decisions the metric commonly used to make decisions regarding costregarding cost--effectivenesseffectivenessNet Health Benefit (NHB) and Net benefit (NB) Net Health Benefit (NHB) and Net benefit (NB) are alternate metrics that have been proposedare alternate metrics that have been proposed

Definitions

ICERICER

NHBNHB

NB

C C –– CostCostQ Q –– QALYSQALYSV V –– Societal Societal

WTP WTP per per QALYQALY

∆∆CCICER =ICER =

∆∆QQ

NHB = NHB = ∆∆Q Q ––∆∆CC

VV

NB NB = V X NB = V X ∆∆Q Q –– ∆∆CC

CEA Decision Criterion

We will pay at most $V (often $50,000) for We will pay at most $V (often $50,000) for each additional QALYeach additional QALYSo we will choose A compared to B (A So we will choose A compared to B (A being more costly and more effective) ifbeing more costly and more effective) if

ICERICERABAB < V< VNHBNHBAB AB > 0> 0NBNBABAB > 0> 0

Estimating ICEREffectiveness (efficacy) estimates obtained Effectiveness (efficacy) estimates obtained from clinical trialsfrom clinical trials

These data commonly used as input to These data commonly used as input to models that provide QALY estimates models that provide QALY estimates May require extrapolation beyond trial May require extrapolation beyond trial periodperiod

Costs obtained from either clinical trials or Costs obtained from either clinical trials or secondary data sourcessecondary data sources

LongLong--term costs and cost offsets are term costs and cost offsets are modeledmodeled

Uncertainty around the cost and effectiveness Uncertainty around the cost and effectiveness estimates result in uncertainty around the ICERestimates result in uncertainty around the ICER

Sources of Uncertainty

Uncertainty around model inputs Uncertainty around model inputs (Parameter Uncertainty)(Parameter Uncertainty)

Utility associated with health statesUtility associated with health statesCost associated with health statesCost associated with health statesTransition probabilitiesTransition probabilities

Uncertainty around model structure (Model Uncertainty around model structure (Model Uncertainty)Uncertainty)

Not commonly accounted forNot commonly accounted for

Options for Incorporating Uncertainty

(Deterministic) Sensitivity Analysis(Deterministic) Sensitivity AnalysisVary 1 or 2 inputs at a timeVary 1 or 2 inputs at a timeScenario analysisScenario analysis

Confidence IntervalsConfidence IntervalsCostCost--effectiveness Acceptability Curveseffectiveness Acceptability CurvesStochastic league tablesStochastic league tables

Sensitivity Analysis - Tornado Diagram

Base Case

Efficacy (± 10%)

Drug Cost (± 10%)

Utility (± 10%)

Discount rate (0% - 10%)

ICER

Issues with One/Two-Way Sensitivity Analysis

No standards to determine which No standards to determine which parameter should be varied and how muchparameter should be varied and how muchSensitivity analysis involving more than Sensitivity analysis involving more than two parameters difficult to presenttwo parameters difficult to presentStill, these analyses are easy for Decision Still, these analyses are easy for Decision Makers to comprehend and can be Makers to comprehend and can be valuable in some casesvaluable in some cases

Scenario Analysis

Best or Worst case scenariosBest or Worst case scenariosAlternate realistic scenariosAlternate realistic scenarios

Accounts for correlations between input Accounts for correlations between input parameter valuesparameter valuesMay be easier for decision makers to May be easier for decision makers to understandunderstand

Confidence Interval

A 95% confidence interval for a parameter, based A 95% confidence interval for a parameter, based on sample data, has the property that 95% of the on sample data, has the property that 95% of the intervals will contain the true dataintervals will contain the true data

The interval has 95% probability of containing The interval has 95% probability of containing the true value of the parameterthe true value of the parameter

A number of different approaches have been A number of different approaches have been described to estimate the confidence interval described to estimate the confidence interval around the ICERaround the ICER

Many methods only described for the two Many methods only described for the two product caseproduct case

Confidence IntervalExpected Value

∆C

∆Q

Estimating the confidence interval around ICER

Utilizes the distribution of Utilizes the distribution of ∆∆C and C and ∆∆QQMethodsMethods

The box methodThe box methodTaylorTaylor’’s seriess seriesFiellerFieller’’s methods methodNonNon--parametric bootstrapparametric bootstrapParametric bootstrapParametric bootstrapJackknifeJackknife

A Comparison of the first four methods found A Comparison of the first four methods found FeillerFeiller’’ss method and nonmethod and non--parametric bootstrap to parametric bootstrap to perform the best perform the best [Polsky D. et al. Health Econ. 6:243]

Choice of method to estimate CI

Are the required assumptions satisfied?Are the required assumptions satisfied?Is the required data available?Is the required data available?

Cost-Effectiveness Acceptability Curves (CEAC)

Estimate probability that a given intervention will Estimate probability that a given intervention will meet the CEA decision criterion, and hence will meet the CEA decision criterion, and hence will be the optimal interventionbe the optimal interventionCEACsCEACs commonly derived using probabilistic commonly derived using probabilistic sensitivity analysis (PSA)sensitivity analysis (PSA)

Specify probability distributions for all Specify probability distributions for all uncertain input parametersuncertain input parametersPropagate uncertainty through the model using Propagate uncertainty through the model using simulationsimulationDerive probability that each intervention is Derive probability that each intervention is optimaloptimal

CEAC

Drug AProb(ICER>WTP)

WTP per QALY

CEAC – Example for two products

Drug A

Drug B

Prob

WTP per QALY

CEAC – Example for multiple products

0%10%20%30%40%50%60%70%80%90%

100%

$0 $20,000 $40,000 $60,000 $80,000 $100,000

WTP per QALY

Pro

babi

lity Drug ADrug B

Drug C

CEAC – Recognized Issues

Does not contain a decision ruleDoes not contain a decision ruleThe product with the highest probability of The product with the highest probability of being optimal may not be the one with the being optimal may not be the one with the largest expected net benefitlargest expected net benefit

Modification proposed to examine the Modification proposed to examine the probability that the product with the probability that the product with the largest expected net benefit is the optimal largest expected net benefit is the optimal productproduct

Claxton et al. Health Economics 14:339

Fenwick et al. Health Economics 10:779

Stochastic League Tables

Presents the probability that a specific Presents the probability that a specific intervention would be included in the intervention would be included in the optimal mix of interventions for various optimal mix of interventions for various levels of resource availabilitylevels of resource availability11

Analysis presented for multiple sets of Analysis presented for multiple sets of mutually exclusive alternativesmutually exclusive alternatives

1Hutubessy et al. Health Economics 10:473

Application of Uncertainty Analysis in Real World Decision

Making

Model

Drug Efficacy

Drug Cost

Other Model Parameters

ICER

Model

Drug Efficacy

Drug Cost

Other Model Parameters

ICER

Parameter Uncertainty

CI / Acceptance Curve

Communicating Results to Decision Makers

A lack of understanding of the calculations A lack of understanding of the calculations of the ICER can lead toof the ICER can lead to

Angry rejectionAngry rejectionUnquestioned acceptanceUnquestioned acceptance

It is critical to communicate the model It is critical to communicate the model structure, key inputs and assumptions firststructure, key inputs and assumptions firstParameter uncertainty and model Parameter uncertainty and model uncertainty can then be addresseduncertainty can then be addressed

Influence Diagram Corresponding to Model

Response to Drug

Disease progression rate from moderate to Severe

Time spent in Moderate and Severe disease states

Total QALYs

Total Cost

Need to communicate

What are the key inputs that drive the model What are the key inputs that drive the model results?results?What are the data sources for these key inputs?What are the data sources for these key inputs?

What is the uncertainty around these input What is the uncertainty around these input parameters?parameters?

How do these key inputs drive the final results?How do these key inputs drive the final results?E.g. How does the time spent in each health E.g. How does the time spent in each health state change with efficacy assumptionsstate change with efficacy assumptions

Tornado Diagram

Base Case

Efficacy (± 10%)

Drug Cost (± 10%)

Utility (± 10%)

Discount rate (0% - 10%)

ICER

Present Best vs Worst vs most-likely case scenarios

Based on the range of values for the key Based on the range of values for the key input parameters present, Best, Worst, and input parameters present, Best, Worst, and Most likely scenario analysisMost likely scenario analysis

Could be included in the tornado diagramCould be included in the tornado diagramNeed to provide justification for the choice Need to provide justification for the choice of the parameter values for these scenariosof the parameter values for these scenarios

Questions that the Decision Maker may ask

Which treatment should I chooseWhich treatment should I chooseCould use the model results for the most likely Could use the model results for the most likely scenario analysisscenario analysis

If I choose a particular treatmentIf I choose a particular treatmentWhat is the probability that I chose a treatment What is the probability that I chose a treatment that is not optimalthat is not optimal

Provided by the CEACProvided by the CEACWhat is the magnitude of the loss in welfare in What is the magnitude of the loss in welfare in scenarios where my choice is not the optimal scenarios where my choice is not the optimal one (Downside risk)one (Downside risk)

Issues with the CEAC

CEAC as currently used provides only the CEAC as currently used provides only the probability that a given product is the optimal probability that a given product is the optimal choice for a given value of Vchoice for a given value of VCEAC does not provide information for cases CEAC does not provide information for cases where it is not the optimal product what the loss of where it is not the optimal product what the loss of net benefit is compared to the optimal productnet benefit is compared to the optimal product

This information may be important to the This information may be important to the decision makerdecision maker

Example

Drug ADrug A Drug BDrug B

∆∆CostCost11 ∆∆QALYQALY11 ∆∆CostCost11 ∆∆QALYQALY11

ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11

11Compared to current standard of careCompared to current standard of care

• There is a 80% probability that drug A is optimal

• Which Drug should the decision maker choose

Example - ContinuedDrug ADrug A Drug BDrug B

∆∆CostCost11 ∆∆QALYQALY11 ∆∆CostCost11 ∆∆QALYQALY11

Scenario Scenario 1 (20%)1 (20%)

$20,000$20,000 --22 $15,000$15,000 11

Scenario Scenario 2 (80%)2 (80%)

$10,000$10,000 22 $15,000$15,000 11

ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11

11Compared to current standard of careCompared to current standard of care

Should the decision maker be provided this information

Proposed modification to CEAC

Plot the cumulative distribution of the difference Plot the cumulative distribution of the difference in NB between a given intervention X and the next in NB between a given intervention X and the next best interventionbest intervention

In cases where X is not the optimal intervention In cases where X is not the optimal intervention this will be the difference in NB between X and this will be the difference in NB between X and the optimal intervention, and will be negativethe optimal intervention, and will be negativeIn cases where X is the optimal intervention In cases where X is the optimal intervention this difference will be positivethis difference will be positive

So, the point where the curve intersects the So, the point where the curve intersects the YY--axis provides the probability that X is the axis provides the probability that X is the optimal intervention (the point on the optimal intervention (the point on the CEAC)CEAC)

Modified CEAC

P

Drug A

p1

$1,000

p2

-$1,000 0

1

∆NB vs next best option

Modified CEAC

P

Drug A

0

1

Drug B

∆NB vs next best option

How should the information be used to make a formulary decision

Should the model results be used to always Should the model results be used to always identify a single optimal intervention?identify a single optimal intervention?

If so, should this be based solely on expected If so, should this be based solely on expected valuevalue

Alternatively, should it be used to identify a set of Alternatively, should it be used to identify a set of interventions in cases where interventions cannot interventions in cases where interventions cannot be clearly differentiated from a costbe clearly differentiated from a cost--effectiveness effectiveness perspectiveperspective

Do these models capture all relevant Do these models capture all relevant information?information?How good is the benefit measure?How good is the benefit measure?

Use of CI-s in decision making: CE panel report

Test hypothesis regarding the sign and Test hypothesis regarding the sign and magnitude of the ICERmagnitude of the ICERAllow the decision maker to determine how Allow the decision maker to determine how much confidence they should place in the much confidence they should place in the ICERICERGuide decision about future researchGuide decision about future research

An alternative to CEAC?

A

NHB B

VChoose A

Need Better Information

Choose B

Conclusions

It is widely accepted that uncertainty around costIt is widely accepted that uncertainty around cost--effectiveness estimates should presented to effectiveness estimates should presented to decision makersdecision makersHowever, whether decision makers can understand However, whether decision makers can understand and incorporate this information into decision and incorporate this information into decision making is not well understoodmaking is not well understoodFurther examination on how we can most Further examination on how we can most effectively present this information is neededeffectively present this information is needed