Using the PCORI Methodology

Standards to Generate Robust,

Relevant, and Timely Evidence for

Patient-Centered Outcomes Research

Joe V. Selby, David Hickam, Brian Mittman, and

Sean Tunis

June 4, 2013

The audio and slide presentation will

be delivered directly to your

computer Speakers or headphones are required to hear the

audio portion of the webinar.

If you do not hear any audio now, check your

computer’s speaker settings and volume.

If you need an alternate method of accessing audio,

please submit a question through the Q&A pod.

Technical Assistance

Live technical assistance:

– Call Adobe Connect at (800) 422-3623

Refer to the ‘Technical Assistance’ box

in the bottom left corner for tips to

resolve common technical difficulties.

To submit a question:

1. Click in the Q&A box on the left side of your screen

2. Type your question into the dialog box and click the Send button

Questions may be submitted at

any time during the presentation

Welcome

Erin Holve, Ph.D., M.P.H.,

M.P.P.

– Senior Director of Research

& Education, AcademyHealth

A Collaborative Effort to

Improve Training in PCOR

PCORI Mini-Course Two morning panels

– Developing a Program of Comparative Effectiveness

Research at the Patient Centered Outcomes Research

Institute” (9:45-11:15am)

• Chair: Joe Selby, PCORI

• Faculty: Chad Boult, Rachael Fleurence, Romana Hasnain-

Wynia, and David Hickam, PCORI

– “PCORI Methods Pilot Projects: Learning from Engagement”

(11:30am-1:00pm)

• Chair: Sue Sheridan, PCORI

• Faculty: Katherne Bevans, Children’s Hospital of Philadelphia;

Renee Robinson, Southcentral Foundation; and Tiffany

Haynes, University of Arkansas for Medical Sciences

PCORI Mini-Course Afternoon workshop (June 25, 2:00-6:00pm ET)

– Heterogeneity of Treatment Effects

• Chair: Brian Mittman, PhD, Department of Veterans Affairs

• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins

University School of Medicine

– Adaptive and Bayesian Trial Designs

• Chair: Sally Morton, PhD, University of Pittsburgh

• Faculty: Jason Connor, PhD, Berry Consultants

– Data Registries

• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of

Maryland

• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome

Learning Objectives Provide an overview of the PCORI Methodology

Standards and the process used to develop the

standards;

Provide general guidance on using the standards

when responding to PCORI funding announcements;

and

Review plans for implementing the standards and

updating them in the future.

Webinar Faculty

Joe V. Selby, MD,

MPH

David Hickam, MD,

MPH Brian Mittman,

PhD Sean Tunis, MD,

MSc

Academy Health Webinar June 4, 2013

Joe V. Selby Executive Director, PCORI

Using the PCORI Methodology Standards to Generate Robust, Relevant, and Timely Evidence for Patient-Centered Outcomes Research

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

About PCORI

• An independent, non-profit organization authorized by Congress.

• Mission is to fund research that will provide patients, their caregivers and clinicians with the evidence-based information needed to make better-informed health care decisions.

• Committed to continuously seeking input from patients and a broad range of stakeholders to guide its work.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

What is Patient Centered Outcomes Research?

Patient-Centered Outcomes Research (PCOR) helps people and their caregivers communicate and make informed health care decisions, allowing their voices to be heard in assessing the value of health care options.

PCOR has the following characteristics:

• Actively engages patients and key stakeholders throughout the research process.

• Compares important clinical management options.

• Evaluates the outcomes that are most important to patients.

• Addresses implementation of the research finings in clinical care environments.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Approximate PCORI Funding

* 20% of this amount goes to AHRQ/HHS each year for dissemination and capacity-building

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Current Funding Opportunities and Events

http://www.pcori.org/funding-opportunities/funding-announcements/funding-center/

Announcement Letter of Intent Due

Assessment of Prevention, Diagnosis, and Treatment Options 06/15/13

Improving Healthcare Systems 06/15/13

Communication and Dissemination Research 06/15/13

Addressing Disparities 06/15/13

Improving Methods for Conducting Patient-Centered Outcomes Research

06/15/13

Improving Infrastructure for Conducting Patient Centered Outcomes Research: Phase One —Clinical Data Research Networks

06/19/13

Improving Infrastructure for Conducting Patient Centered Outcomes Research: Phase One —Patient-Powered Research Networks

06/19/13

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

David Hickam, MD, MPH, PCORI

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Developing the PCORI Methodology Standards

http://www.pcori.org/assets/PCORI-Authorizing-Legislation-032310.pdf

• Congressional Requirements- Patient Protection and Affordable Care Act, Subtitle D, Paragraph (6)(C)(i)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

PCORI Methodology Committee

http://www.pcori.org/about/governance-and-leadership/leadership/methodology-committee/

• Sherine Gabriel, MD, MSc (chair), Mayo Clinic

• Robin Newhouse, PhD, RN (vice chair), University of Maryland

• Naomi Aronson, PhD, BlueCross BlueShield

• Ethan Basch, MD, MSc, Memorial Sloan-Kettering Cancer Center

• Alfred O. Berg, MD, MPH, University of Washington

• David Flum, MD, MPH, University of Washington

• Steven Goodman, MD, MHS, PhD, Johns Hopkins School of Public Health

• Mark Helfand, MD, MS, MPH, Dept. of Veterans Affairs

• John Ioannidis, MD, DSc, Stanford University

• Michael S Lauer, MD, National Heart, Lung, and Blood Institute

• David O. Meltzer MD, PhD, University of Chicago

• Brian S. Mittman, PhD, Dept. of Veterans Affairs

• Sebastian Schneeweiss, MD, ScD, Harvard Medical School

• Jean R. Slutsky, Agency for Healthcare Research and Quality

• Mary Tinetti, MD, Yale University School of Medicine

• Clyde Yancy, MD, MSc, Northwestern University Feinberg School of Medicine

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

PCORI Methodology Standards

http://www.pcori.org/research-we-support/methodology/

PCORI Methodology

Committee Formed (2011)

Draft Methodology

Report Posted (July

2012)

Public Comment

Period (July-September

2012)

Proposed Revisions to

Methodology Standards

(November 2012)

Revised PCORI

Methodology Standards

(December 2012)

Final Methodology

Report (Spring 2013)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

11 Categories of Methodology Standards 47 Individual Standards

*Bold indicates topics to be discussed in detail at the June 25 in-person workshop at the AcademyHealth Annual Research Meeting

• Formulating Research Questions

• Patient-Centeredness

• Systematic Reviews

• General and Cross-cutting Methods for all PCOR

• Causal Inference

• Heterogeneity of Treatment Effects

• Preventing/Handling Missing Data

• Data Networks

• Adaptive and Bayesian Trial Designs

• Data Registries

• Studies of Diagnostic Tests

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Characteristics of the Methodology Standards

• Are minimal standards for performing comparative effectiveness research.

• Are intended to provide helpful guidance to researchers and those who use research results.

• Reflect generally accepted best practices.

• Provide guidance for both project protocols and reporting of results.

• Are used to assess the scientific rigor of funding applications.

• Context of research should drive use of the standards.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Standards that Apply to Most Projects

• Formulating research questions (6 standards)

• Patient-centeredness (4 standards)

• General and cross-cutting (6 standards)

• Missing data (5 standards)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Formulating Research Questions

Identify affected populations and health decision(s), and how study results will inform the health decision (RQ-3)

Identify and assess participant subgroups (RQ-4)

Select appropriate interventions and comparators and make explicit how each were selected (RQ-5)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Patient-Centeredness

Engage the population of interest and other relevant stakeholders

as appropriate and necessary in a given research context (PC-1)

Identify, select, recruit, and retain study participants representative of the population of interest (PC-2)

Use patient-reported outcomes when patients or people at risk of a condition are the best sources of information. (PC-3)

Support dissemination and implementation of study results (PC-4)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

General and Cross-cutting Methods

• Assess the suitability of the data source (GM-1)

• Document data analysis plans for major aims (GM-3)

• Document validated scales and tests

• Include characteristics of the scales and psychometric properties (GM-4)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Missing Data

• Use Validated Methods to Deal with Missing Data that Properly Account for Statistical Uncertainty Due to Missingness (MD-3)

• Record and Report All Reasons for Dropout and Missing Data, and Account for All Patients in Reports (MD-4)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Standards that Apply to Specific Data Sources and Study Designs

• Systematic reviews (1 standard)

• Causal inference (6 standards)

• Heterogeneity of treatment effects (4 standards)

• Data networks (2 standards)

• Adaptive trials (5 standards)

• Registries (3 standards)

• Diagnostic tests (5 standards)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Causal Inference

• Precisely Define the Timing of the Outcome Assessment Relative to the Initiation and Duration of Exposure (CI-3)

• Measure Confounders before Start of Exposure. Report data on confounders with study results (CI-4)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Diagnostic Tests

• Assess the Effect of Factors Known to Affect Diagnostic Performance and Outcomes (DT-3)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Brian Mittman, PhD, Department of Veterans Affairs

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

How to Use the Standards

• The standards represent “best practice” recommendations for improving the quality and value of PCOR and CER.

• The standards aim to achieve greater transparency, rigor and other improvements to strengthen the science and methods of PCOR and CER.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Implementing the standards: improving PCOR research practices • “Research practices” include decisions and actions by numerous

stakeholders (funders, reviewers, researchers, etc.)

• Changing these practices is challenging

– Researchers value independence and autonomy

– Researcher actions are guided by knowledge, experience, judgment

• Analogue: the challenge of improving research practices through standards is comparable to the challenge of improving clinical practices through evidence-based clinical practice guidelines.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Implementing research standards

• Professional practices are influenced by multiple factors operating at

multiple levels

• Implementation gaps must be thoroughly diagnosed to identify “root causes”

• Implementation campaigns must include multiple, coordinated components targeting the full spectrum of determinants of gaps

• Influences on research practices – and points of leverage to change practices – should be depicted in a causal, conceptual model

There are “no magic bullets”: examples of dramatic change in research practices (e.g., trial registration) are rare and exceptional.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Conceptual model of the research process

• Research question identification and prioritization (researchers, funding agencies)

• Study design and methods selection (researcher judgment guided by training, professional norms, published standards)

• Peer review (reviewer judgment guided by training, professional norms, published standards)

• Study governance and conduct (researcher judgment, possibly guided by reporting requirements, IRB review, DSMB review)

• Documentation and reporting (researcher judgment guided by reporting guidelines, journal requirements, peer review)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Dissemination and Future Directions

• Initial set of standards released in December 2012.

• PCORI Methodology Report will be released in Summer 2013.

• No change in the set of standards.

• Provides further guidance on interpretation and use.

• Responding to future PCORI Funding Announcements (PFAs).

• All future submissions must demonstrate adherence to/adoption of/consideration of the standards.

• PCORI will continue to develop tools and resources for researchers, reviewers and other stakeholders.

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Sean Tunis MD, MSc - CMTP

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Purpose of Standards: PMC draft report 7/12

“Departures from basic good research practices are partially responsible for the mismatch between the quality and relevance of the information research provides and the information patients need to make informed clinical decisions.”

“The initial range of topics was chosen to reflect areas in which the Committee believed that there were either substantial deficiencies or inconsistencies in how available methods were applied in practice, or for which there was specialized knowledge in how best to conduct research that had not been effectively disseminated.”

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Required by Law! (ACA)

“Methodological standards shall provide specific criteria for”:

• Internal validity (robust)

• Generalizability (relevant)

• Timeliness (timely)

• Feasibility

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Where to Focus

• Craft an important, relevant research question

• Informed by gaps in systematic reviews and meaningful input from patients and other decision makers

• Patient-centered care requires attention to subgroups and PROs

• Apply best practices in application of selected methods

• Provide full transparency

Clear, detailed protocol – explain deviations from standards

Follow existing reporting standards

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Asking the Right Question

“To produce information that is meaningful and useful to people when making specific health decisions, research proposals and protocols should describe: 1) the specific health decision the research is intended to inform; 2) the specific population for whom the health decision is pertinent; and 3) how study results will inform the health decision.”

• Requires dialogue with decision makers (more is better)

• Methods should serve research question; not vice versa

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Use of Patient Reported Outcomes

“Select outcomes based on input directly elicited from patient informants, people representative of the population of interest, either in previous studies or in the proposed research.”

• Use existing core outcomes measures sets when available

• COMET (Core Outcomes Measures in Effectiveness Trials)

• Effectiveness Guidance Documents (Green Park Collaborative)

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Feasibility: An Unresolved Issue

Standard for Studies of Dx Tests:

“Studies of clinical outcomes after diagnostic testing should use a prospective randomized study design when possible. If a non-randomized design is proposed, the reason for using an observational study (or modeling and simulation) should be addressed and efforts to minimize confounding documented.”

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE

Looking Forward

• Specifics of implementation of standards not yet clear

• Initial emphasis of methods standards on patients and caregivers as decision makers. Intent to expand scope to clinicians, payers/purchasers, policy makers, others.

• PCORI thinking continues to evolve based on experience and public feedback from researchers and other stakeholders

• How best to achieve optimal balance of internal validity, generalizability, timeliness and feasibility remains an important challenge

To submit a question:

1. Click in the Q&A box on the left side of your screen

2. Type your question into the dialog box and click the Send button

Submitting Questions

Expert Q&A

Erin Holve (facilitator)

Dave Hickam

Brian Mittman

Sean Tunis

Still time to sign up for the PCORI Mini-Course

• In-person events at AcademyHealth’s Annual Research Meeting in Baltimore, MD (June 25)

Two morning panels

• “Developing a Program of Comparative Effectiveness Research at the Patient Centered

Outcomes Research Institute” (9:45-11:15am)

• “PCORI Methods Pilot Projects: Learning from Engagement” (11:30am-1:00pm)

• Afternoon workshop (June 25, 2:00-6:00pm ET)

• Heterogeneity of Treatment Effects

• Chair: Brian Mittman, PhD, Department of Veterans Affairs

• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins University School of

Medicine

• Adaptive and Bayesian Trial Designs

• Chair: Sally Morton, PhD, University of Pittsburgh

• Faculty: Jason Connor, PhD, Berry Consultants

• Data Registries

• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of Maryland

• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome

• Click here to register: $50 for AcademyHealth members, $100 for nonmembers

Thank You

Please take a moment to fill out the

brief evaluation which will appear in your browser.