The AHRQ Training Modules forthe Systematic Reviews Methods

Guide:An Introduction

Prepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

To review the goals and activities of the Evidence-based Practice Center (EPC) program in relation to the Effective Health Care (EHC) Program

To describe the roles and responsibilities of the different participants in the EHC Program in relation to the EPC program

To outline the purpose and components of the Training Modules for the Methods Guide for Comparative Effectiveness Reviews (CERs)

To define important terms and concepts used in the Training Modules for the Methods Guide for CERs

Learning Objectives

Section 1013 of the 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health care delivered through Medicare, Medicaid, and State Children’s Health Insurance programs.

AHRQ started the Effective Health Care Program in 2005 upon appropriation of funds.

In 2009, the American Recovery and Reinvestment Act (ARRA) awarded additional funding to AHRQ to:

broaden comparative effectiveness research,

build an infrastructure to support the expanded efforts, and

invest in innovative mechanisms to disseminate findings of research to health care decisionmakers.

What Is the Effective Health Care Program?

Conducts effectiveness and comparative effectiveness research to inform the health care decisions of patients, providers, and policymakers   Involves individual researchers, research centers, and

academic organizations in the United States and Canada  Comparative effectiveness research includes:

Reviews and syntheses of published and unpublished scientific evidence

New scientific evidence and analytic tools

Translates research findings into easily understood formats for patients, providers, and policymakers

What Does the Effective Health Care Program Do?

The conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in “real world” settings.

The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decisionmakers and responding to their expressed needs about which interventions are most effective for which patients under specific circumstances.

What Is Comparative Effectiveness Research?

Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.

Answer real questions Health care decisionmaking

Best evidence Internal and external validity

Comparative Real-world comparisons

Effectiveness Patient-centered outcomes of interest

The ABCs of Comparative Effectiveness Research

Relevant and timely Useful for decisionmakers Stakeholder input for topic nomination and development

Objective and scientifically rigorous Identification of methodological standards Careful consideration of potential conflicts of interest Ensure balanced approach to topic and research

Open for public participation and transparent Accountable Clear documentation of processes Stakeholder input for topic nomination, development, and

review of draft reports

Effective Health Care Program Principles

Slutsky J, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare. ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf.

Components of theEffective Health Care Program

Stakeholder Input and Involvement

1.Evidence Synthesis Research reviews summarize the scientific literature on comparative

effectiveness questions and highlight areas for future research. Comparative effectiveness and effectiveness reviews outline the effectiveness, benefits,

and harms of treatment options.

Technical briefs describe the state of the literature for new or emerging health care tests or treatments.

2.Evidence Generation New research reports analyze new data to answer comparative effectiveness

questions. Research on health care databases may be supported through the Effective Health Care

Program, specifically through the DEcIDE Network or the CERTs.

Clinical comparative effectiveness trials will be supported through extramural grants.

3.Dissemination and Translation Summary Guides translate findings from research reviews for patients,

providers, or policymakers.

How Does the Effective Health Care Program Answer Real-World Questions?

Evidence-based Practice Centers (EPCs) The John M. Eisenberg Center for Clinical Decisions and

Communications Science The Scientific Research Center (SRC) Partners for new research:

Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Network The Centers for Education and Research on Therapeutics (CERTS) Extramural grantees and other researchers

Stakeholders: Formal Stakeholder Group Key Informants Technical experts Public input

Task order officers at the Agency for Healthcare Research and Quality

Who Are the Key Playersin the Effective Health Care Program?

Each Player on the Team Has a Different Role

Patients, health care providers, and policymakers make better health care decisions

StakeholdersNominate topics, provide input on

questions, reviews, and

program

DEcIDE, CERTs, GranteesEvidence Generation

Evidence-based Practice Centers Evidence Synthesis and

Evidence Need Identification

Eisenberg Center Evidence Translation and Dissemination

Scientific Resource Centercoordinates topic selection/peer review/public input,

provides research support

AHRQ Task Order OfficersDevelop, coordinate, and monitor

The Game Plan

Evidence-based Practice Center researchers directly engage stakeholders for specific topics or reports Specific input through focus groups, key informant panels

Topic development and refinement Determining Research needs

Public input through posting and comment periods Draft Key Questions Draft research reviews

Effective Health Care (EHC) Program members engage stakeholders for the Program as a whole Open Web topic nominations Citizen’s forum (in development) Stakeholder groups provide input on EHC Program processes

Volunteer members for a 2-year term Representatives of various stakeholder groups

Engaging Stakeholders

Points of Stakeholder Engagementfor Systematic Reviews

Topic Generation

Topic Development

Topic Refinement

Research Review

Research Needs Development

Report Translation and Dissemination

Generate nominations

relevant to “real-world” health care

decisions

Clarify nomination and decisional

dilemma

Clarify and refine key questions to accurately reflect

the decisional issue

Provide clinical & methodological input to EPC’s

protocol

Provide input to develop and

prioritize research gaps

Test and provide input on summary

guide development

Evidence-based Practice Centers Eisenberg Center andAHRQ OCKT

Web-based and SRC Stakeholder

Engagement Team

Answer real-world health

questions

Ensure appropriate context is

considered

Ensure key questions accurately

reflect decisional dilemmas

Provide clinical &

methodological input

Craft meaningful

messages and disseminate

information to those who

need it

Develop and prioritize

research gaps

AHRQ Effective Health Care ProgramPoints of Stakeholder Engagement for Systematic Reviews

Pro

gra

mP

ha

seE

ng

ag

em

en

t P

urp

ose

Sta

keh

old

er

Inte

rest

AHRQ = Agency for Healthcare Research and Quality; OCKT = Office of Communications and Knowledge Transfer; SRC = Scientific Resource Center

Research reviews completed under the Effective Health Care and Generalist Programs are broad and balanced in scope: Useful to a broad range of stakeholders Scope determined by the Evidence-based Practice Center through topic development

Comparative effectiveness reviews Effectiveness reviews Technical briefs

Reviews requested by Federal partners may have specific purpose and scope as defined by the partner: Center for Medicare and Medicaid Services (CMS), through the Technology

Assessment Program U.S. Preventive Services Task Force (USPSTF) National Institutes of Health (NIH), through the Office of Medical Applications of

Research (OMAR) Other federal partners: AHRQ, Centers for Disease Control and Prevention (CDC),

Food and Drug Administration (FDA), and other NIH centers

Evidence-based Practice Centers Conduct Research Reviews for a Variety of Purposes

The Evidence-based Practice Center program was instituted in 1997. The following are the centers that were awarded contracts from the Agency for Healthcare Research and Quality in 2007. The contracts are competitive on a 5-year cycle. Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL Duke University, Durham, NC ECRI Institute, Plymouth Meeting, PA Johns Hopkins University, Baltimore, MD McMaster University, Hamilton, Ontario, Canada Minnesota Evidence-based Practice Center, Minneapolis, MN Oregon Evidence-based Practice Center, Portland, OR RAND Corporation, Santa Monica, CA RTI International-University of North Carolina, Research Triangle, NC Tufts Medical Center, Boston, MA University of Alberta, Edmonton, Alberta, Canada University of Connecticut, Storrs, CT University of Ottawa, Ottawa, Ontario, Canada Vanderbilt University Medical Center, Nashville, TN

The Current Evidence-based Practice Centers

The Scientific Resource Center (SRC) is currently* located at the Oregon Evidence-based Practice Center (EPC) and supports the activities of the Effective Health Care (EHC) Program by: Communicating with stakeholders and assisting EPC members to

ensure that EHC Program products meet the practical needs of providers, patients, and policymakers

Receiving and sorting topic nominations Facilitating and coordinating the decisionmaking process for

topic selection of research reviews Coordinating peer review and public input for research reviews Collaborating with EPC members to develop consistent and

rigorous scientific methods in conducting research reviews.

The Scientific Resource Center

*Awarded by competitive bid in 2007.

The John M. Eisenberg Center for Clinical Decisions and Communications Science is currently* located at Baylor College of Medicine in Houston, Texas.

The Center works with products developed by Evidence-based Practice Center members to translate research reviews into short, easy-to-read guides that can be used by consumers, clinicians, and policymakers.

Involvement of the Center begins at topic development and continues until the guides are disseminated, generally within 6 months of the final research review. Early involvement helps to: improve the usefulness of research products, and optimize fidelity of translation.

The John M. Eisenberg Center for Clinical Decisions and Communications Science

*Awarded by competitive bid in 2008.

Serve as the contracting officer technical representative for all Effective Health Care Program partners Specific task order officer listed for each task order Contact the task order officer for questions regarding:

Task order Collaboration or coordination with other partners in the EHC and Evidence-

based Practice Center Programs

Participate in calls and meetings for each research review step, from topic development through publication

Provide input on scientific, protocol, and process issues Help EPC researchers identify opportunities for

collaboration within the EHC Program to address common challenges and communicate best practices

Agency for Healthcare Researchand Quality Task Order Officers

In 2009, eight Evidence-based Practice Centers were awarded additional task orders to:

Develop topics for potential research review within a given area of concentration that corresponds to AHRQ's priority conditions or cross-cutting themes

Conduct research reviews within a given area of concentration

Identify important future research that is needed to fill gaps and help patients, physicians, and policymakers make health care decisions within a given area of concentration Develop methods for rigorous and transparent methods for determining

specific study design and prioritizing research needs Involve stakeholders, which include funders of new research and

investigators who conduct new research

American Recovery and Reinvestment Act–Funded Evidence-based Practice Center Activities

The eight Evidence-based Practice Centers (EPCs) that were awarded funding for 3 years through the American Recovery and Reinvestment Act (ARRA) in 2009 will focus on the following research areas:

Arthritis, functional limitations, and disability Minnesota EPC

Cancer and infectious disease Blue Cross and Blue Shield Association EPC

Cardiovascular and pulmonary disease Duke University EPC

Mental health RTI International-University of North Carolina EPC

Pregnancy, developmental delays, and autism Vanderbilt University EPC

Diabetes, obesity, and peptic ulcer disease Johns Hopkins University EPC

Health care delivery, prevention, and behavioral intervention Oregon EPC

Diagnostics and devices Tufts Medical Center EPC

Areas of Concentration for the ARRA-Funded Evidence-based Practice Centers

Evidence-based Practice Centers work with researchers, funders, and other stakeholders to determine important research needs:

The DEcIDE Network An AHRQ contract-funded network of research centers, primarily organized by

consortia around selected priority conditions Generates new comparative effectiveness evidence through analysis of existing

health care databases and patient registries, and new data collection

Centers for Education and Research on Therapeutics (CERTs) A network of research centers funded by AHRQ as cooperative agreements, each

addressing different themes, including diseases, populations, data systems, and methodologies

Generate, translate, and disseminate new evidence on benefits and harms of therapeutics

AHRQ extramural grantees AHRQ grant-funded researchers Conduct clinical comparative effectiveness studies

Partners for New Research

AHRQ = Agency for Healthcare Research and QualityDEcIDE = Developing Evidence to Inform Decisions about Effectiveness

Acumen, LLC, Burlingame, CA Brigham and Women's Hospital, Boston, MA Duke University, Durham, NC Harvard Pilgrim Health Care, Harvard Medical School, Boston, MA Johns Hopkins University, Baltimore, MD Outcome Sciences, Cambridge, MA RTI International, Research Triangle Park, NC University of Colorado at Denver and Health Sciences Center, Aurora, CO University of Illinois at Chicago, Chicago, IL University of Maryland at Baltimore, Baltimore, MD University of North Carolina at Chapel Hill, Chapel Hill, NC University of Pennsylvania School of Medicine, Philadelphia, PA Vanderbilt University Medical Center, Nashville, TN

The Current DEcIDE Network Centers*

*Awarded by competitive bid in 2005DEcIDE = Developing Evidence to Inform Decisions about Effectiveness

Clinical and economic issues in hospital settings — University of Chicago Consumer education and patient adherence — University of Texas M.D. Anderson

Cancer Center and Baylor College of Medicine Therapies for heart and blood vessel disorders — Duke University Medical Center Musculoskeletal disorders — University of Alabama Drug interactions, women's health — University of Arizona/The Critical Path

Institute Drug use, safety, and effectiveness in managed care — HMO Research Network Health information technology — Brigham and Women's Hospital Anti-infective therapeutics use and resistance — University of Pennsylvania

School of Medicine Therapeutic medical devices — Weill Medical College of Cornell University Mental health therapeutics — Rutgers, The State University of New Jersey Therapies for older adults and the effects of aging — University of Iowa Pediatric therapeutics — Cincinnati's Children's Hospital Medical Center Tools for optimizing prescribing — University of Illinois at Chicago Therapeutics for vulnerable populations — Vanderbilt University Medical Center

The Current CERTs Centers*

*Grants awarded by competitive bid in 2006CERTS = Centers for Education and Research on Therapeutics

The purpose of the training modules is: To provide practical training on the Methods Guide for Conducting

Comparative Effectiveness Reviews

The Methods Guide is: Intended to improve the transparency, consistency, and scientific rigor of the

Effective Health Care (EHC) Program A collaborative effort of AHRQ, the Scientific Resource Center (SRC), and the

Evidence-based Practice Centers (EPCs) A living document with revisions as new evidence and understanding grows

(initial draft posted in 2007) Posted on the EHC Program Web site

Other materials available to new EPC investigators or staff: The Methods Guide Stakeholder training (soon to be available to EPCs) Program materials, such as guides and templates (available to EPCs on

secure site) Learning Network (soon to be available to EPCs on secure site) Methods library and weekly e-mail methods article alert (contact task order

officer or SRC to be added to the listserv)

AHRQ Training Modules forthe Systematic Reviews Methods Guide

1. Topic Refinement2. Analytic Frameworks3. Study Eligibility Criteria4. Searching for Relevant Studies5. When To Select Observational Studies as Evidence 6. Data Extraction7. Assessing the Quality of Individual Studies8. Assessing Applicability9. Presentation of Findings10. Quantitative Synthesis (I)11. Quantitative Synthesis (II)12. Grading Strength of Evidence13. Reporting the Review

The Training Modules

Systematic Review Process Overview

Prepare Prepare TopicTopic

• Refine Refine topictopic

•Develop Develop analytic analytic frameworkframework

Search for and Search for and Select Studies for Select Studies for

InclusionInclusion

•Identify study Identify study eligibility criteriaeligibility criteria

•Search for Search for relevant studiesrelevant studies

•Select evidence Select evidence for inclusionfor inclusion

Extract Extract Data from Data from StudiesStudies

Analyze and Analyze and Synthesize Synthesize

StudiesStudies

•Assess the quality Assess the quality of individual studiesof individual studies

•Assess Assess applicabilityapplicability

•Present findingsPresent findings

•Synthesize Synthesize quantitative dataquantitative data

•Grade strength of Grade strength of evidenceevidence

Report Report Systematic Systematic

ReviewReview

Identify gaps and future research needsIdentify gaps and future research needs

Important Terms and Concepts (I)

Research Review or Evidence SynthesisResearch Review or Evidence Synthesis

Systematic Review Systematic Review

Comparative Effectiveness ReviewComparative Effectiveness Review Technical BriefTechnical Brief

Effectiveness ReviewEffectiveness Review

PICO ≈ PICO(TS) ≈ PICOTS ≈ PICOT ≈ PICOS Population Intervention Comparison Outcome Timing of outcome assessment Setting (sometimes study design)

Important Terms and Concepts (II)

www.effectivehealthcare.ahrq.gov Public Web site of the Effective Health Care (EHC) Program:

EHC Program published methods papers, including the Methods Guide EHC Program published research reviews and summary guides Opportunities for collaborations through listserv notifications Key question, draft report, and final report postings

https://www.kpchr.org/ehc/system/login.aspx Secure site available only to Evidence-based Practice Center (EPC) investigators Contact task order officer or Scientific Resource Center for access Enable “remember me” function to improve usability and access

Policy and process documents, including templates and topic development working documents

Learning Network — a forum for asking questions and sharing experiences and ideas

Contact information for EHC Program and partners http://mc.manuscriptcentral.com/ehc

Scholar One Manuscript Central — peer-review management system (contact task order officer or the Scientific Resource Center for access)

Upload draft and final documents through this site

Helpful Web Sites forEvidence-based Practice Center Investigators

Effective Health Care Program. Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, 2009. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318.

Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: U.S. Department of Health and Human Services; June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.

Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods Reference Guide for Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted August 2009. Available at: http://effectivehealthcare.ahrq.gov/healthInfo. cfm?infotype=rr&ProcessID=60.

References

Slutsky J, Atkins D, Chang S, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, Posted November 2008. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf.

Slutsky J, Atkins D, Chang S, et al. AHRQ series paper 1: comparing medical interventions: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:481-483.

References

This presentation was prepared by Stephanie Chang, M.D., M.P.H., and Elisabeth Kato, M.D., M.R.P., at the Agency for Healthcare Research and Quality (AHRQ) Center for Outcomes and Evidence.

Authors