Post on 08-Sep-2018
Can We Reliably Benchmark HTA Organizations?
Michael Drummond Centre for Health Economics
University of York
Some Background
• In recent years, several reimbursement agencies have begun using HTA.
• Essentially, HTA has become ‘hard-wired’ into the reimbursement or coverage decision.
• Economic evaluation has an increasing prominence in these HTAs (for resource allocation decisions).
• It is important to establish some principles of ‘good practice’.
Methods (1)
• Developed 15 Key Principles for the Improved Conduct of HTA for Resource Allocation Decisions, based on the existing literature and our own views.
• Solicited comments from key opinion leaders in HTA.
• Published in IJTAHC 2008;24:244-58, along with editorial comments.
Methods (2) • Selected 14 HTA entities from 9 countries, to
reflect: - worldwide coverage; - traditional HTA agencies and reimbursement agencies; - established and newly-formed entities; - public and privately funded entities.
• Assessed whether the entities ‘supported’ and/or ‘implemented’ the various principles.
• Used our own expertise, supplemented by local experts for Australia, Brazil, Canada, Korea and Taiwan.
Methods(3)
• Based on the preliminary effort, several issues were raised
• A series of audit questions were developed which, it is argued, would enable us to benchmark HTA organizations in a more reliable manner
Results
• The Key Principles • The Assessments
• Additional considerations in
benchmarking HTA organizations • Audit questions for the Key principles
Key Principles for HTA
• Structure of HTA • Methods of HTA • Processes for Conduct of HTA • Use of HTA in Decision-making
Structure of HTA
• The goal and scope of the HTA should be explicit and relevant to its use.
• HTA should be an unbiased and transparent exercise.
• HTA should include all relevant technologies.
• A clear system for setting priorities for HTA should exist.
Methods of HTA • HTA should incorporate appropriate methods for
assessing costs and benefits. • A full societal perspective should be considered
when undertaking HTAs. • HTAs should explicitly characterize uncertainty
surrounding estimates. • HTAs should consider and address issues of
generalizability and transferability. • HTAs should consider a wide range of evidence and
outcomes.
Processes for Conduct of HTA
• Those conducting HTAs should actively engage all key stakeholder groups.
• Those undertaking HTAs should actively seek all available data.
• The implementation of HTA findings needs to be monitored.
Use of HTA in Decision-Making
• HTA should be timely. • HTA findings need to be communicated
appropriately to different decision-makers. • The link between HTA findings and
decision-making processes needs to be transparent and clearly defined.
Assessment Criteria
‘+’ Means that the organization ‘supported’ the Principle in written guidelines or other forms, regardless of whether it was followed.
‘++’ Means that the organization
‘implemented’ the Principle in published reports and that the decisions based on these reports demonstrate adoption of the Principle.
Key Principle
CMS (US)
Washington Medicaid/ DERP (US)
WellPoint
(US)
BCBS TEC
(US)
NICE (UK)
IQWiG
(Germany)
DAHTA@ DIMDI
(Germany)
Structure of HTA program
1 The goal and scope of the HTA
should be explicit and relevant to its use
++ ++ ++ ++ ++ ++ +
2 HTA should be an unbiased and transparent exercise
++ ++ ++ ++ ++ ++
3 HTA should include all relevant technologies
++ ++ ++ ++ ++
4 A clear system for setting priorities for HTA should exist
+ + ++ + ++
Methods of HTA
5 HTA should incorporate appropriate methods for assessing costs and benefits
+ ++ + ++
6 HTAs should consider a wide range of evidence and outcomes
++ + + ++ ++ ++ ++
7 A full societal perspective should be considered when undertaking HTAs
+
8 HTAs should explicitly characterize uncertainty surrounding estimates
++ ++ + +
9 HTAs should consider and address issues of generalizability and transferability
+ ++ +
Key Principle
CMS (US)
Washington Medicaid/ DERP (US)
WellPoint (US)
BCBS TEC (US)
NICE (UK)
IQWiG (Germany)
DAHTA@ DIMDI
(Germany)
Processes for conducting HTA
10 Those conducting HTAs should actively engage all key stakeholder groups
++ + ++ ++
11 Those undertaking HTAs should actively seek all available data
++ + ++ ++ ++ +
12 The implementation of HTA findings needs to be monitored
+ +
Use of HTA in decision making
13 HTA should be timely ++ ++ + + ++ +
14 HTA findings need to be communicated appropriately to different decision makers
++ ++ ++
15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined
+ + +
Notes: “+” signifies that the organization “supported” the principle in question in written guidelines or other form, regardless of whether they actually follow it. “++” means that the organization “implemented” the principle in published reports, and decisions based on these reports demonstrate adoption of the specific principle.
Key Principle
TLV (Sweden)
SBU (Sweden)
CADTH (Canada)
HIRA (Korea)
PBAC (Australia)
Anvisa (Brazil)
DHTA (Taiwan)
Structure of HTA program
1 The goal and scope of the HTA should be explicit and relevant to its use
+ + ++ ++ ++ + ++
2 HTA should be an unbiased and transparent exercise
++ ++ ++ + +
3 HTA should include all relevant technologies
++ + +
4 A clear system for setting priorities for HTA should exist
+ ++ ++ ++
Methods of HTA
5 HTA should incorporate appropriate methods for assessing costs and benefits
++ + ++ + ++ +
6 HTAs should consider a wide range of evidence and outcomes
++ ++ ++ + + ++
7 A full societal perspective should be considered when undertaking HTAs
++ ++ +
8 HTAs should explicitly characterize uncertainty surrounding estimates
++ + ++ +
9 HTAs should consider and address issues of generalizability and transferability
+ ++ + +
Key Principle
TLV (Sweden)
SBU (Sweden)
CADTH (Canada)
HIRA (Korea)
PBAC (Australia)
Anvisa (Brazil)
DHTA (Taiwan)
Processes for conducting HTA
10 Those conducting HTAs should actively engage all key stakeholder groups
++ + ++ +
11 Those undertaking HTAs should actively seek all available data
++ ++ + ++ + ++
12 The implementation of HTA findings needs to be monitored
++
Use of HTA in decision making
13 HTA should be timely ++ + + + + + ++
14 HTA findings need to be communicated appropriately to different decision makers
+ ++ ++ + +
15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined
++ + ++ ++
Summary of Findings
• Support of principles High: HTA should be unbiased and transparent Low : HTA should adopt a societal perspective
• Implementation of principles High: HTA should consider a wide range of evidence Low: Link between HTA and decision-making should be
clearly defined
Discussion Points • Only a convenience sample of HTA entities. • Different entities have different remits, which
affects the adherence to some principles. • The Key Principles merit more discussion: -some are controversial -some may be too idealistic -some are partially contradictory.
• Judgements of ‘implementation’ are impressionistic and we need a tighter, audit approach.
Additional Considerations in Benchmarking
• Understanding the varying statutory roles of HTA organizations
• Weighting the relative importance of the Key Principles
• Recognizing the varying stages of development of HTA organizations
• Developing unambiguous audit criteria
Understanding the Varying Statutory Role of HTA organizations
• The remit of an organization can limit its ability to
adhere to a principle • Some principles (eg. KP 3 HTAs Should Consider
All Relevant Technologies) relate to the whole jurisdiction, not an individual organization
• Therefore, need to distinguish between ‘maximum score’ and ‘maximum attainable score’
• Nevertheless, some elements of the remit need to be questioned within the jurisdiction (eg KP 7 HTAs Should Adopt a Societal Perspective)
Weighting the Relative Importance of the Key Principles
• Clearly, all key principles may not be of equal importance
• The relative weightings could vary by jurisdiction
• There are trade-offs between principles (eg between KP 10 on Stakeholder Involvement and KP 13 on Timeliness)
• Trade-offs should be discussed at the level of the individual jurisdiction
Recognizing the Varying Stages of Development of HTA organizations
• It takes time to formulate a view about the
various key principles • As HTA organizations undertake
studies/make decisions, case-law is developed
• Initially, the organization may not have the human resources to undertake all the tasks implied by the principles, even if it wanted to
Developing Unambiguous Audit Criteria
• Need audit criteria that can be defended
• The objective should be to develop a series of questions that can be answered ‘yes’ or ‘no’, or graded in an unambiguous manner (eg a Likert Scale)
• Paper published in the International Journal of Technology Assessment in Health Care 2012;28(2): 159-65
Audit Criteria for the 15 Key Principles
• The objective was to create, from the 15 Key Principles, a series of questions that can unambiguously be answered
• This preliminary list is based on the points we raise in the text of the original Key Principles paper
• It may be possible to supplement this list with other questions
Principle1.The goal and scope of the HTA should be explicit and relevant
to its use.
• Is the remit for the HTA organization
clearly defined? • Is a scoping document drawn up prior to
an HTA, containing information on the specific decision problem to be addressed, the alternatives to be considered, the relevant patient populations and the time horizon?
Principle 2. HTA should be an unbiased and transparent exercise
• Is the HTA organization independent of the body making the reimbursement or coverage decision?
• Are the recommendations of the HTA organization made by an independent expert advisory committee?
• Are the meetings of the committee held in public? • Are all supporting information and the basis of the
recommendations made publicly available? • Does the organization normally commission outside groups to
undertake the HTA? • Are the reports produced subjected to independent peer-
review prior to final determination? • Are the draft conclusions subject to review by stakeholders
and the public, with rationale underlying final determinations of contentious issues?
Principle 3. HTA should include all relevant technologies
• Are all types of technologies (e.g. drugs, devices, diagnostics, procedures, behavioural modification) considered?
• Within each category, are both new and existing technologies considered?
• In assessments of new technologies, are all relevant alternatives considered?
Principle 4. A clear system for setting priorities for HTA should exist
• Does a formal system for prioritizing and selecting topics exist?
• Is the priority-setting approach clear and transparent?
Principle 5. HTA should incorporate appropriate methods for assessing
costs and benefits • Does the HTA organization consider costs as well
as benefits and harms? • Does the HTA organization have published
methods guidelines for assessing the benefits, harms and costs of health technologies?
• Is a full systematic review of clinical evidence required as a basis for economic modeling?
• Does the team undertaking HTAs on behalf of the organization include individuals with skills in epidemiology/biostatistics, health services research and economics?
Principle 6. HTAs should consider a wide range of evidence and outcomes.
• Does the relevant clinical evidence include
observational and non-randomized studies, as well as RCTs?
• Does the HTA consider impacts on quality of life and other patient-reported outcomes, as well as clinical events?
• Does the HTA consider relevant sub-groups of the patient population (e.g. by baseline risk)?
Principle 7. A full societal perspective should be considered when
undertaking HTAs • Does the HTA consider only consider the impact on a
specific budget, for example for drugs? • Does the HTA consider all health care costs? • Can other costs be included as extra information? • Are productivity gains and losses (ie indirect costs and
benefits) considered when relevant? • Are costs for informal care included when relevant? • Are costs in added years of life included in the cost-
effectiveness ratio?
Principle 8. HTAs should explicitly characterize uncertainty surrounding
estimates • Does the HTA include a sensitivity analysis? • Are confidence intervals presented for key
estimates? • Have the key deficiencies in available data
been identified and discussed? • Is an agenda for key future research
proposed?
Principle 9. HTAs should consider and address issues of generalizability
and transferability
• Does the HTA organization have methods guidance for dealing with transferability issues when using data or analyses from other jurisdictions?
• Does the HTA organization consider the generalizability of the results of its studies to other patient populations, health care delivery systems or practice settings that are relevant for its jurisdiction?
Principle 10. Those conducting HTAs should actively engage all key stakeholder groups (eg professional
bodies, patient organizations, manufacturers). • Is the HTA organization formally required to engage stakeholders in
its activities? • Does the HTA organization involve stakeholders in the scoping of
HTAs? • Does the HTA organization have a mechanism for identifying the
relevant stakeholders • Does the organization encourage or require submissions of evidence
from stakeholders? • Does the organization allow stakeholders to comment on reports at
the draft stage? • Does the organization allow stakeholders to appeal against
recommendations/decisions? • Do the organization’s committees include stakeholder representation
(eg patient groups, technology manufacturers, clinical specialists)?
Principle 11. Those undertaking HTAs should actively seek all
available data • Does the systematic review of clinical
evidence include the grey literature and unpublished data?
• Does the HTA organization have processes for handling confidential data from manufacturers?
Principle 12. The implementation of HTA findings needs to be monitored
• Does the HTA organization develop an implementation plan for its HTAs?
• Does the HTA organization monitor the impact of its recommendations?
Principle 13. HTA should be timely.
• Does the HTA organization have a defined time period for conducting HTAs/producing recommendations?
• Does the HTA organization adhere to the agreed timelines?
• Does the organization have a mechanism to update its HTAs/recommendations within a given time period?
Principle14. HTA findings need to be communicated appropriately to
different decision-makers. • Does the HTA organization develop a
communications plan for its recommendations and decisions?
• Are separate versions of reports produced for different audiences (e.g. health professionals, decision-makers, general public)?
• Is the effectiveness of communication monitored and evaluated?
Principle 15. The link between HTA findings and decision-making
processes needs to be transparent and clearly defined
• Does the organization distinguish between the scientific assessment of the evidence and the appraisal decision?
• Does the organization have an explicit decision rule for acceptance/non-acceptance of health technologies?
• Does the organization have a transparent approach for weighing various considerations (e.g. cost-effectiveness, equity)?
• Does the agency recommend, or operate, conditional reimbursement/coverage-with-evidence-development schemes?
• Does the organization distinguish between identifiable sub-groups of the patient population when making decisions?
Conclusions • Benchmarking of HTA entities is currently
controversial, but it is feasible • New approaches need to be developed that
yield unambiguous assessments • Benchmarking could either be undertaken by
an independent organization, or by the HTA agencies themselves
• The main focus should be on self-improvement by the agencies themselves, rather than on external criticism