Rapid Response Health Technology Assessments Speed Versus Systematic Searching Hayley Fitzsimmons...
-
Upload
amber-morris -
Category
Documents
-
view
215 -
download
1
Transcript of Rapid Response Health Technology Assessments Speed Versus Systematic Searching Hayley Fitzsimmons...
Rapid Response Health Technology Assessments
Speed Versus Systematic Searching
Hayley Fitzsimmons
January 30, 2009 OLA Super Conference 2009
Agenda
• Background
• Evidence-based medicine movement
• Role of the Canadian Agency for Drugs and Technologies in Health (CADTH)
• CADTH’s rapid response HTIS service
• Speed versus systematic searching: Problems & issues
• Techniques used by the rapid response Health Technology Inquiry Service (HTIS) to ensure focused yet comprehensive searching
• Limits
• Database selection
• Filtering for study types (i.e. Systematic reviews or randomized controlled trials)
• Selective grey literature search
• Conclusion
Evidence-Based Medicine
To make decisions about:
1. what interventions to prescribe to a patient?
2. what interventions to fund with public money?
3. what diagnostic tool to purchase for your health care institution?
..base your decision upon the best evidence available.
Evidence-Based Medicine
“Over two million articles are published annually in the biomedical literature in
over 20 000 journals”
Ad Hoc Working Group for Critical Appraisal of the Medical Literature. Academia and clinic: a proposal for more informative abstracts of clinical articles. Ann Intern Med 1987;106:598-604.
Evidence-Based Medicine
Best evidence comes from:
• systematic review (SR)
• meta-analysis (MA)
• health technology assessment (HTA)
All three types of publications synthesize the relevant studies (usually but not exclusively RCTs) and apply strategies to help limit the bias which may occur in an individual study.
Evidence-Based Medicine
SR: A review of studies in which evidence has been systematically searched for, studied, assessed, and summarised according to predetermined criteria. (NLM)
MA: Systematic methods that use statistical techniques for combining results from different studies to obtain a quantitative estimate of the overall effect of a particular intervention or variable on a defined outcome. (NLM)
HTA: Technology assessment in health care is a multidisciplinary field of policy analysis. It studies the medical, social, ethical, and economic implications of development, diffusion, and use of health technology. (INAHTA)
Evidence-Based Medicine
Health Economic Reviews important because:
Systematic reviews, meta-analysis, Clinical trials and guidelines are concerned with
clinical benefits and harm and do not necessarily address the cost issue that most
decision makers need in order to make an informed decision on spending limited public
health care funds.
Hierarchy of Evidence
When no SRs, MAs, or HTAs available:
Evidence-Based Medicine
Where do clinical practice guidelines fit on the hierarchy of evidence ladder?
1. Old Style Guidelines or Consensus Statements
• GOBSAT (Good old boys sitting around a table)
• Not necessarily based upon clinical evidence
• Opinions of clinical experts
Bottom rung of hierarchy of evidence ladder “Case Studies, anecdote and personal opinion”
Evidence-Based Medicine
2. New Style Guidelines or Consensus Statements
• Guidelines produced today have been heavily influenced by the evidence-based medicine movement
• Guidelines are beginning to mimic systematic reviews
• Each guideline needs to be individually assessed as to the thoroughness in which it reviews the available evidence.
MAY fall on top rung of hierarchy of evidence ladder!
Evidence-Based Medicine Summary
“Health care providers, researchers, and policy makers are inundated with
unmanageable amounts of information; they need systematic reviews to
efficiently integrate existing information and provide data for rational decision
making.”
BMJ 1994;309(6954):597 (3 September)
Canadian Agency for Drugs and Technologies in Health
CADTH Mandate
The Canadian Agency for Drugs and Technologies in Health (CADTH) provides Canada’s federal, provincial and territorial health care decision makers with credible,
impartial advice and evidence-based information about the effectiveness and
efficiency of drugs and other health technologies
CADTH Technology Reports: Pros & Cons
PROS CONS
Comprehensive Expensive to produce
Extensive quality review both internal and external
Slow: takes an average of 10-15 months to complete
Reviews all aspects of a health technology including clinical effectiveness, cost-effectiveness, social, ethical, and economic issues
Not feasible to produce if there is not enough published evidence available (i.e. a new technology)
CADTH’s Rapid Review Health Technology Inquiry Service (HTIS)
February 2005:
In response to a growing demand from Canadian health-care administrators and decision makers to support quick decision making and tight timelines CADTH launched its’ rapid response, Health Technology Inquiry Service (HTIS)
Pros Cons
Fast Not as comprehensive
Able to handle new /rare technologies without much published evidence
Less rigorous quality review (depending on the report level)
Inexpensive Less detailed reports
Shorter reports Level one reports do not include a summary/synthesis of the results (bibliography only)
HTIS Report levels
.
Types of HTIS Requests
CADTH’s Information Services department
Our job:
• ensure researchers have all the available evidence on a topic
• failure to retrieve and analyze relevant materials can seriously compromise the quality of our reports
• time constraints of HTIS rapid reviews makes comprehensive retrieval challenging
Speed versus systematic searching: Problems & issuesHTIS Rapid Response Searches HTA Systematic Searches
Less time to search, especially for lower level reports which takes an average of two days
Takes a long time: Need to be comprehensive, pressure to retrieve everything of relevance
Cannot give research assistants/officers thousands of references for broad search requests, as they do not have the time to screen them
Normal to retrieve thousands of articles for research assistants/officers to screen
Very little or no time allotted for hand searching relevant studies (report dependent)
Time allotted for hand searching bibliographies of relevant studies
Due to tight timelines HTIS only searches major grey literature sites
HTA searches an extensive list of major and minor HTA sites as well as other relevant grey literature websites
Speed versus systematic searching: Problems & issues
Searching for rapid review HTIS program
Main problem
To find all the relevant evidence on subject in a timely manner, without
giving the researchers so much information that they cannot meet tight
HTIS rapid review timelines
Techniques used by HTIS Searchers to Provide Quick, Focused Results with Minimal Error
•Disclaimer & Methods Section
•Applying Limits
•Database Selection
•Filtering Searches for Study Type
•Selective/Focussed Grey Literature Search
Disclaimer
Section of HTIS Disclaimer on searching:
… HTIS responses are based on a limited literature search and are not comprehensive, systematic reviews. The intent is to provide a list of sources of the best evidence on the topic that CADTH could identify using all reasonable efforts within the time allowed…..
Methods Section
HTIS methods example: A limited literature search was conducted on key health technology assessment resources, including PubMed, the Cochrane Library (Issue 4, 2008), University of York Centre for Reviews and Dissemination (CRD) databases, ECRI, EuroScan, international health technology agencies, and a focused Internet search. Results include articles published between 2003 and November 2008, and are limited to English language publications only. Filters were applied to limit the retrieval to systematic reviews, health technology assessments, meta-analyses and guidelines.
Applying Limits
Normally the searches that we do are very broad for example a level 1 we recently did “chronic pain management for outpatients.”
We limited the search by:
• Guidelines
• 5 years
• English
Without limits we would retrieve thousands of articles and researchers would spend weeks weeding through all the literature and miss deadlines!
Applying Limits
Rapid review HTIS requests typically have the following limits:
- 5 years
- English language
- Human
- Study type (topic dependent)
CADTH Systematic review Technology Reports generally do not have date or language limits:
- Human limits
- Study type (occasional - topic dependent)
Applying Limits
• If there is too much information on the topic;
- if lower-evidence study types such as observational studies have been requested negotiate to see if they may be removed (especially if there are a lot of higher quality study types available)
- consider removing articles of the following document types:
- Comments
- Letters
- Case reports
- consider shortening the date range (negotiated with the requestor)
- discuss narrowing the topic with the requestor
Database Selection
Databases searched for HTIS reportsLevel 1 & 1.5:
Medline and/or PubMed (98% overlap) Cochrane Database of SRs (unless SRs not required) Centre for Reviews and Dissemination (CRD) – University of York Other major databases are only searched if little or nothing is found in
MedlineLevel 2:
Medline and/or PubMed (98% overlap) Cochrane Database of SRs (unless SRs not required) Centre for Reviews and Dissemination (CRD) – University of York If retrieval is small in Medline up to two other relevant databases
Level 3 & 4:
Medline and PubMed (98% overlap) Embase Cochrane Library Centre for Reviews and Dissemination (CRD) – University of York Two or more relevant databases
Filtering for Study Type
Searching for most CADTH reports is limited by study design, some of the most common include:
Systematic Reviews/HTAs/Meta-analysis
Randomized clinical trials
Controlled trials
Observational trials
Clinical practice guidelines
Economic studies
Limiting by Study Design using Filters
Filters are primarily used to limit searches to a certain study design for example the following PubMed filter to restrict searches to systematic reviews and/or HTAs.:
Systematic Reviews and HTAs Filter:
systematic[sb] OR (meta-analysis[pt] OR meta-analysis[mh] OR meta-analysis[tw] OR metanalysis[tw]) OR meta analy*[Title/Abstract] OR metaanaly*[Title/Abstract] OR met analy*[Title/Abstract] OR metanaly*[Title/Abstract] OR integrative research[Title/Abstract] OR integrative review*[Title/Abstract] OR integrative overview*[Title/Abstract] OR research integration*[Title/Abstract] OR research overview*[Title/Abstract] OR collaborative review*[Title/Abstract] OR collaborative overview*[Title/Abstract] OR systematic review*[Title/Abstract] OR health technology assessment*[tiab] OR "Technology Assessment, Biomedical"[mh] OR HTA*[tiab] OR "Cochrane Database Syst Rev"[Journal:__jrid21711] OR "health technology assessment winchester, england"[Journal]
Filtering for Study Type
No filter is infallible!
We tested/validated our SR/HTA filter with the help from the Hedges group at McMaster (PubMed filters are based upon Hedges filters) and found it was approximately 90% precise and had a 46.5% sensitivity score. This translates to:
• retrieving about 90% of the SRs/HTAs available in PubMed
• approximately 46% of the articles we retrieve will NOT be SRs or HTAs.
• the more precise the filter the less sensitive it is (more junk!)
Filters modified for HTIS use tend to be more precise but less accurate
Filtering for Study Type
There are three places to search when looking for study design, our filters attempt to cover all three possibilities:
Title/abstract fields
Publication type field
Subject heading field (e.g. MESH, or Emtree)
Example #1 study design found in all three fields:
Example #2 study design NOT found anywhere:
Finding Validated Filters
1. InterTASC Information Specialists' Sub-Group: Search filter resource http://www.york.ac.uk/inst/crd/intertasc/
Provides an extensive bibliography of published filters available by type (i.e. RCT), database (i.e. Embase) and search platform (i.e. PubMed, Ovid, etc.)
2. PubMed also has a few pre-loaded filters:http://www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml#reviews
3. Search of the literature
Observational Study Filters Listed in InterTASC
What is Grey Literature
•Conferences
• Theses
• Proposals
• Drafts
• Statistics
• Internal Reports
•Open access journals (un-indexed)
• Government documents (including HTAs)
•Press Releases
Grey Literature Matters Because…..
1. Health technology assessment (HTA) reports are peer reviewed and highly useful to health-care decision makers and are NOT usually indexed in PubMed !
Example: CADTH’s own Technology reports and overviews are not indexed in PubMed
2. “Unpublished trials contribute about 20% of the weight in individual meta-analysis."
Krishnan, R R. Evidence-based practice: how to read what you read. Psychopharmacology Bulletin 2004; 37(4): 88-91.
Grey literature Matters Because…..
• Another way of finding articles that were missed in the original database search due to:
bad or no indexing
inconsistent language
incomplete database search strategy
CADTH Grey Literature Checklist
“Grey Matters: A Practical Search Tool for Evidence-
based Medicine”http://cadth.ca/index.php/en/cadth/products
In order to use the tool:
save file
open file
“Open as read only?” will appear
click “No”
Grey Literature Searching for HTIS
PROBLEM:
Complete CADTH grey literature checklist used for Technology Assessments is almost 30 pages full of websites to search and can take up to a week to search comprehensively!
Solution:
Abridged version (7 pages - some of which is optional) of the complete CADTH checklist including Canadian and major evidence-based medicine sites is used to search lower level HTIS reports in order to save time while making sure most important grey literature is retrieved
Grey Literature Checklists used for HTIS
Level 1 & 1.5 and 2 reports
Abridged version of complete HTA Checklist including:
- Canadian and major international HTA websites
- Canadian and major international guideline sites
- Safety information from Canadian and a couple of other major regulatory agencies
- Focused Internet search
Grey Literature Checklists used for HTIS
Level 3 & 4:
The complete HTA Checklist but judgement is used as to which sites need to be searched depending on the question (i.e. a drug search will typically skip all sites dealing exclusively with devices and vice versa for a device search)
The sites chosen for level 3 and 4 reports normally include all HTA sites both Canadian and International as well as safety and, if required, economic sites
Conclusion
The techniques used by HTIS to limit search results to reasonable numbers;
• limits• study type filters• focused database and grey literature searches
allows us to capture most of the major evidence-based literature available in an efficient manner.
These same techniques can be useful for others requiring quick and efficient ways for searching evidence-based medicine.
HTIS Testimonial
“The information you provided is informing both our drug policy and funding people on discussions that are occurring for this entire class of drugs. I've been pleased with the report that was provided, and hope to see this service continued in the future … Very few health ministries in the country can possibly retain the 'in-house' capacity to do this type of review in this short a turnaround, so it is very good to know that CADTH had the foresight to develop this service.”
(Mr. Sean Delaney BSc, MPH, Manager, Province Wide Services, Alberta Health and Wellness, Edmonton AB)
Questions?
CADTH web site: www.cadth.ca
For More Information