Professor Nancy J. DevlinDirector of Research, OHEHTAiTokyo • May 2016

Structured decision making approaches to the inclusion of multiple criteria in HTA

Contents1. Multiple criteria in HTA decision making2. ISPOR Best Practice Taskforces on MCDA3. Experience from pilots of MCDA in HTA globally4. Why has MCDA not been implemented in HTA?5. Specific issues to be addressed re HTA6. Next steps7. Concluding remarks

1. Multiple criteria in HTA• Improving health is generally the primary goal of health care• But health care systems face multiple objectives that go beyond beyond

improvements in population health eg. not just how much health, but how it is distributed; other equity considerations; societal preferences for prioritising certain problems or patients (eg children).

• e.g. QALY maximisation is a pragmatic way of assessing value for money – but clearly does not capture everything that matters to society.

• HTA systems vary in the extent to which they are explicit and consistent in identifying what these other criteria are, and how they are reflected in their decision making processes.

• But HTA generally relies on deliberative processes to weigh up different kinds of evidence.

• The question isn’t ‘should multiple criteria be taken into account in HTA?’ – because they already are. • The question is: how best should multiple criteria taken into account?

Fully quantifiedPurely qualitative/deliberative

Criteria decided ex ante Core criteria – but with potential to make exceptions

Criteria decided on an ad hoc basis

Fixed criteria Criteria vary with explicit justification Criteria may vary

Fixed weights Trade-offs/weights normally the same – but may vary, and are always explicit

Trade- offs between criteria vary

Explicit and fully transparent process Process explicit and full transpareny Process not pre-agreed; basis for decisions not explicit or transparent

‘Structuring judgments’

The potential merits of structured decision making

Desirable features of HTA process: Consistency and replicability; Transparency (to taxpayers; stakeholders; signals to industry about what innovations are of value); Accountability .• Weighing up complex information is cognitively demanding

• Literature shows that individuals are subject to “cognitive bias”• Deliberative processes are influenced by group dynamics and factors including

chairing styles and dominant personalities• “The preferred options identified by MCDA are likely to out-perform the use of

intuitive judgement alone” (Devlin and Sussex, 2011)• Policy initiatives reflect the need to take into account multiple criteria simultaneously

and in a systematic way (e.g. VBA in the UK; GPS-Health in global context - Norheim et al, 2014)

• Increasingly a wide range of stakeholders, including patients and clinicians, have been involved in HTA (e.g. PACE in Scotland) - but unclear how stakeholders’ views influence final decisions

2. Growing interest in MCDA

But also some reservations about this set of techniques; and some misunderstandings about MCDA

Growing interest in use of MCDA in HTA globally

• NICE:• ‘Structured decision making’ included for the first time in the 2013 methods review• Explicit criteria for new highly specialised technologies process (currently under

consultation).

• Examples of one-off pilots in the literature :• Israeli Health Basket Committee (Golan and Hansen, 2012)• Colombia• Belgium• Italy (Radaelli et al., 2014)• Thailand (Youngkong et al., 2012)• Germany (IQWiG, 2013)

Israel’s Health Basket Committee pilot

• Bubble size = quality of evidence• Used to inform

a deliberative process

Source: adaptation of Golan and Hansen (2012)

MCDA in Lombardia, Italy

• For the implementation of new health technologies, the Lombardia region in Italy has introduced a system combining elements of the EUnetHTA Core model (for the assessment) and an MCDA approach (EVIDEM) as a decision-making aid

• The MCDA framework includes 9 broad dimensions and 20 criteria, including disease-, treatment-, financial- and social-related aspects

• This approach has been deemed successful and used for 26 technologies (Radaelli et al., 2014)

Thai pilot Performance matrix for the selection of interventions for assessment

Source: Youngkong et al., 2012

Thai pilot (2)• Assessment and appraisal of selected interventions based on• Value for money (incremental cost effectiveness ratio

against a threshold)• Budget impact

Source: Youngkong et al., 2012

IQWiG pilots• Aims were:• identifying patient-relevant outcomes in depression and hepatitis C• eliciting patient preferences on the selected outcomes using two approaches

(Analytic Hierarchy Process (AHP) and Discrete Choice Experiment (DCE))• enabling aggregation of outcome-specific efficiency frontiers based on

obtained weights

IQWiG pilots

• Both pilots concluded that MCDA approaches can be used to support the HTA process to incorporate patient preferences (Thokala et al., 2016)• Some challenges remain• E.g. “Which persons should be interviewed and how transferable are these

results to the entire patient population?” (IQWiG, 2015)

4. Why hasn’t MCDA been implemented in HTA?

• HTA organisations may have some discomfort with a requirement to be fully explicit about the basis for its decisions. • Fundamental misunderstandings about the use of MCDA to structure

deliberation: eg: the view that ‘If we use MCDA there wont be any point in having HTA committees – decisions would be made by machines and algorithms, and there wont be any role for judgement’• Conservatism with a small ‘c’ (the cost per QALY system is simple,

practical, and well accepted; moving away from it causes nervousness)• Important methodological challenges to work through specifically in

relation to use of MCDA in HTA.

5. Issues still to be addressed in the context of HTA

Criteria Weights Uncertainty Budget constraint

Established in advance; the same across all decisions? Or varying case-by-case?

Whose preferences should be used to establish the weights/trade-offs?

Uncertainty a separate criterion – or uncertainy handled via sensitivity analysis?

How to reflect cost? (eg separate criterion, as in EVIDEM? Or MCDA to create a composite measure of value, to be weighed up against cost?)

How to establish a legitimate set of criteria? (current practice in HTA? Stated objectives of the health care system? Consultation with the general public?)

Weights to reflect the views of those in HTA committees – or supplement these with quantitative evidence (stated preferences) of patients, the general public?

How is opportunity cost to be addressed in decision making involving multiple criteria? What are the implications for the cost effectiveness threshold?

6. Next steps• Further work required to establish appropriate methods and to

address methodological questions relating specifically to the development and use of MCDA in HTA• Plans to build on existing ISPOR taskforce, by a further taskforce

specifically to address the issues identified above.

7. Concluding remarks• Structured decision making approaches can offer considerable advantages in HTA by

providing a coherent/unifying framework for healthcare decision making • They do not aim to replace the judgement of HTA committees – but to help committees

ensure they exercise judgements in a consistent, explicit way.• What specific approaches are best will depend on the specific socio-political characteristics

of the health care system – ‘one size does not fit all’.• Consideration of cost and opportunity cost in a systematic way remains a methodological

challenge (not only from an MCDA perspective)• Do costs of implementation justify benefits of improved decision making? Formal

evaluation required.• Need to move beyond tendency of current pilots to focus on feasibility (‘can we do it?’) to

wider questions (‘do decision makers find it acceptable? What would ‘success’ look like?)

ReferencesDevlin and Sussex (2011). Incorporating Multiple Criteria in HTA. Methods and processes. OHE research https://www.ohe.org/publications/incorporating-multiple-criteria-hta-methods-and-processes

Golan and Hansen (2012). Which health technologies should be funded? A prioritization framework based explicitly on value for money. Israel Journal of Health Policy Research 1:44

Norheim O. F., Baltussen R., Johri M., et al., 2014. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Effectiveness and Resource Allocation 12:18

Radaelli et al., (2014). Implementation of EUnetHTA core Model® in Lombardia: the VTS framework. International Journal of Technology Assessment in Health Care 30:1; 105-12

Thokala et al, for the ISPOR MCDA Task Force (2016) MCDA for Health Care Decision Making – An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health (in press)

Walker, A., (2016). Challenges in using MCDA for reimbursement decisions on new medicines? Value in Health 19: 123-124

Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattanon Y (2012). Multicriteria decision analysis for including health interventions in the Universal Health Coverage Benefit package in Thailand. Value in Health 15; 961-970

IQWiG (2013). Analytic Hierarchy Process (AHP) – pilot project to elicit patient preferences in the indication “depression”. https://www.iqwig.de/download/Executive-summary-of-working-paper_Analytic-Hierarchy-Process-pilot-project.pdf