1

DrPH Seminar Session 1Use of Systematic Review in Public Health Policy

&Getting Started Defining SR Questions

Mei Chung, PhD, MPHResearch Assistant Professor

Nutrition/Infection Unit, Department of Public Health and Community Medicine, Tufts School of Medicine

2

Going Over the Syllabus

• Session 1 (9/11 5:30-8:30pm)– Homework due by 9/22 (Monday): posting

your homework on the class website (http://sites.tufts.edu/systematicreviews/home-page/) and comment on others’ posting

• Session 2 (9/22 4-7pm at Sackler 510-Computer Lab)– Bring your “Building a search strategy”

worksheet to the class– Homework due by 10/8 (Monday)

3

Going Over the Syllabus

• Session 3 (10/9 5:30-8:30pm)– Homework due by 10/20 (Monday)

• Session 4 (10/23 5:30-8:30pm)– Post your final presentation slides on the

class website by noon on 10/23

4

Outline of Session 1• Why systematic reviews are needed in public

health policy and practice• Very quick overview of SR methods• SR versus traditional narrative review

– Current debates on the scientific value of SR• How to formulate a SR research question• How to evaluate a SR (focusing only on SR

protocol/methods)

5

Evidence-based X

• Evidence-based public health is defined as the development, implementation, and evaluation of effective programs and policies in public health through application of principles of scientific reasoning, including systematic uses of data and information systems, and appropriate use of behavioral science theory and program planning models.

Source: Brownson, Ross C., Elizabeth A. Baker, Terry L. Leet, and Kathleen N. Gillespie, Editors. Evidence-Based Public Health. New York: Oxford University Press, 2003.

6

Evidence-based X

• Evidence-based dietetic practice is the use of systematically reviewed scientific evidence in making food and nutrition practice decisions by integrating best available evidence with professional expertise and client values to improve outcomes.

Source: Academy Scope of Dietetics Practice Framework https://www.andeal.org/evidence-analysis-process-overview

7

Organizations producing systematic reviews or using systematic review to inform evidence-based policy and practice guidelines

http://sites.tufts.edu/systematicreviews/mainpage/

8

Synthesizing Evidence• Narrative Reviews

• Systematic Reviews

• Meta-Analysis

• Decision Analysis

• Cost-effectiveness analysis

• Clinical practice guidelines

• Algorithms

9

What is a Systematic Review?(Sometimes called systematic evidence-based reviews or evidence review)

•Systematic review– a comprehensive summary of all available

evidence that meets predefined eligibility criteria to address a specific clinical question or range of questions

•Meta-analysis– commonly included in systematic reviews, a

statistical method that quantitatively combines the results from different studies

10

Basic Steps in a SR

Prepare topic

Search for studies

Screen studies Extract data

Analyze and synthesize data• Apply

qualitative and/or quantitative methods

Report findings

11

FORMULATE STUDY QUESTIONESTABLISH PROTOCOL

CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL

DATA EXTRACTION and QUALITY ASSESSMENT

ANALYSIS and INTERPRETATIONWEIGHTED AVERAGE

REGRESSIONSENSITIVITY ANALYSIS

LITERATURE SEARCH / RETRIEVAL

Steps of Performing a Systematic Review

Ask

Identify

Acquire

Appraise

Synthesize

12

Systematic review process flowchart

13

Systematic review and meta-analysis is a

retrospective exercise, suffering from all the

limitations of being an observational design.

14

For all research, and for systematic review in particular,

a clear research question is needed

An important clinical/public health question

might not be a meaningful research question

15

FORMULATE STUDY QUESTIONESTABLISH PROTOCOL

CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL

DATA EXTRACTION and QUALITY ASSESSMENT

ANALYSIS and INTERPRETATIONWEIGHTED AVERAGE

REGRESSIONSENSITIVITY ANALYSIS

LITERATURE SEARCH / RETRIEVAL

Steps of Performing a Systematic Review

Ask

Identify

Acquire

Appraise

Synthesize

16

Example - poorly formulated question:

Should dietary supplements be recommended to patients with hypertension?

17

PopulationSetting

Intervention OutcomesCondition of interest

Example: The Well - Formulated Question

Does omega-3 fatty acids decrease long-term CVD outcomes in older persons with mild to moderate hypertension?

Fish oilEPA+DHAALA

Fish intakeWalnut

Fatal and non-fatal strokesFatal and non-fatal Coronary Heart Disease (MI, sudden death)Cardiovascular events (above plus aneurysm, congestive heart failure, transient ischemic attacks)Total Mortality

> 1 year> 60 yrs oldoutpatients

Systolic 140-179Diastolic 90-109

The Cochrane Collaboration "How to Conduct a Cochrane Systematic Review" 1996

18

PICO approach to formulating answerable research question

Counsell, 1997

• P population• I intervention (or exposure)• C comparator• O outcomes• D study design

You’ll also see PICO, PICOS (study design), PICODD (+duration), PICOT (time), and others

19

PI(E)COS approach to formulating answerable research question

• What is the relevant population?• What is the intervention/exposure of interest?

• What is the appropriate comparison? • What are the important outcomes of interest?

• In what setting would the results be applicable?

You’ll also see PICOT (timing), PICOD (design/duration), and others

20

The PICO method to formulate research question on interventions

Population Interventions / Exposure

Comparator Outcomes

Primary prevention

Fish, fish oil, ALA

Placebo Overall mortality

Secondary prevention

Dosage No control Sudden death

Background intake

Active comparator

Revascular-ization

Duration Stroke

Blood pressure

21

PopulationsProblems with defining condition

• Varying definitions– lack of an adequate reference standard

(e.g., patients with anemia; patients with metabolic syndrome)

• Different levels of rigor– Loose vs. strict definitions (e.g. elderly vs.

adults men and women who are 60 years old or grater)

– Applicability/generalizability tradeoffs

22

Example – population of interest

• Primary Prevention - patients without prior history of cardiovascular disease– Country– Background diet

• Secondary Prevention – patients with prior history of cardiovascular disease

23

Example – Intervention / Exposure of Interest

What is an omega-3 fatty acid?

• EPA, DHA, (fish oil, fish)– Levels differ by type of fish– Levels (and/or effect) may differ by

preparation (broiled, fried fish sandwiches)• ALA (plant source: walnut, canola oil,

mustard seed, etc.)

24

Example – Outcomes of interest

• Hard outcomes (clinical events)– Overall mortality– Stroke– Myocardial infarction– Sudden death– Revascularization

• Soft [surrogate, intermediate] outcomes (biomarkers, measurements)– Coronary flow– Blood pressure– Lipid levels

• Intermediate– Diagnosis of hypertension

25

Analytic Framework• Series of specific questions can be

formulated into a model that analyzes all effects and interactions between intervention or exposure and outcomes

• Analytic framework can be used to clarify and generate questions (topics)

• Can highlight what aspects are known and unknown

• Can clarify what study designs may be best to address specific questions

25

26

USPSTF, Nelson et al., Ann Intern Med. 2009;151(10):727-737

26

27

Omega-3 FAs and CVD

27

28

FORMULATE STUDY QUESTIONESTABLISH PROTOCOL

CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL

DATA EXTRACTION and QUALITY ASSESSMENT

ANALYSIS and INTERPRETATIONWEIGHTED AVERAGE

REGRESSIONSENSITIVITY ANALYSIS

LITERATURE SEARCH / RETRIEVAL

Steps of Performing a Systematic Review

Ask

Identify

Acquire

Appraise

Synthesize

Will be covered in session 2

29

Goals of Data Extraction & Quality Assessment

• Data extraction– To collect key study characteristics and

results from published articles pertaining to the SR research question

– Important to use a standardized form, customized for the SR research question

• Quality / risk of bias assessment – To avoid “Garbage in, garbage out”– To assess the confidence in the validity of

study findings

30

Rationale for Quality Assessment

• Quality assessment of all studies included in the SRs is important:– estimate extent to which study’s design

and methods prevented systematic errors (biases)

– variation in quality may explain differences in results of SRs

– necessary even if there is little variability among studies (consistent trash is still trash)

31

Tools for Quality/Risk of Bias Assessment

• Many tools, but few “validated” tools:– Cochrane risk of bias assessments (

http://bmg.cochrane.org/assessing-risk-bias-included-studies): RCTs

– The Newcastle-Ottawa Scale (http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp): Observational studies

• No well-accepted nutrition/public health specific [content specific] quality assessment tools– Lichtenstein AH, Yetley EA, Lau J. Application of

systematic review methodology to the field of nutrition. J Nutr 2008;138:2207-2306

32

Linking Quality Assessment to Analysis

• as a threshold for inclusion and exclusion of studies in the review (generally not recommended)

• as a possible explanation for differences in results between studies

• as a variable in sensitivity analysis (test of robustness)

• as weights in statistical analysis of the results

33

FORMULATE STUDY QUESTIONESTABLISH PROTOCOL

CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL

DATA EXTRACTION and QUALITY ASSESSMENT

ANALYSIS and INTERPRETATIONWEIGHTED AVERAGE

REGRESSIONSENSITIVITY ANALYSIS

LITERATURE SEARCH / RETRIEVAL

Steps of Performing a Systematic Review

Ask

Identify

Acquire

Appraise

Synthesize

34

Qualitative & Quantitative Syntheses

• Qualitative synthesis – required – Summary tables (many different forms)

• Key study characteristics• Summary of study results

– Graphical presentation of study results (a plus)

• Quantitative synthesis (a.k.a meta-analysis) - optional– highly depending on types of results/data, and

reporting of the data– may not be appropriate – to pool or not to pool can

be a tricky decision

35

• Combine data from multiple studies to illustrate trends in the data

• May be focused on describing study characteristics, results, or both

• Can be designed to include characteristics of all included studies – Examples: funding sources, assessment method, country of study

• Can be designed for subsets of included studies– Examples: summary tables for randomized controlled trials,

prevalence studies, harms/side effects, outcomes for specific treatments

Summary Tables (I)

36

• Simplified entry (one row) for each study • Table columns may include, for example:

– PICOTS (may be listed in table title or headers)– Methodological quality– Applicability– Study size (weight)– Magnitude of effect

• A single study may be represented in multiple summary tables (e.g., different outcomes)

Summary Tables (II)

PICOTS = population, intervention, comparator, outcomes, timing, and setting

37

Example:Summary Table of Study Characteristics

A basic summary table is the “study characteristics” table.

The overall summary provides an overview of the state of the available studies in the literature.

Hartmann KE, et al. AHRQ Evidence Report/Technology Assessment No. 187. Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/ bladder/bladder.pdf.

38

Example:Summary Table of Study Characteristics (More

descriptive, most common)

39

Example: Summary Table for Cohort Studies

Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94.Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.

40

Example:Summary tables can be specialized for different types of outcomes

41

Summary Matrix

Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94.Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.

42

Example: Graphical

presentation of the study

results

43

An Example of an Assessment of Strength of Body of Evidence

High• High level of assurance with the validity of the results (based on

quality, applicability, effect size, consistency) for the key question• At least 2 high quality studies with long-term followup• No important disagreement across studies

Moderate• Good to moderate level of assurance with the validity of the

results• Fewer than 2 high quality studies• Little disagreement across studies in the results

Low• Low level of assurance with the validity of results• Based on studies of moderate to poor quality or limited

applicability

Insufficient• Little data or disagreement across or within studies

43

44

• Summary tables provide key information on study characteristics and study findings. Through table and graphical formats, respectively

• Properly constructed summary tables:– Effectively convey results– Provide an overview of the literature in a given field– Enable the reader to grasp results for subsets of the literature

Key Messages

45

Synonyms of Meta Analysis

• Quantitative overview/Synthesis• Pooling

– Less precise– Suggests that data from multiple sources are

simply lumped together• Combining

– Preferred by some– Suggests applying statistical procedures to data

46

Reading a Generic Forest Plot

Reference: Szajewska H. The role of meta-analysis in the evaluation of the effects of early nutrition on mental and motor development in children. Am J Clin Nutr. 2011 Dec;94(6 Suppl):1889S-1895S. Epub 2011 Apr 27. Review

47

Heterogeneity of Data~Diversity~

Clinical

• Are studies of similar treatments, populations, settings, design etc. such that an average effect would be clinically meaningful?

Methodological

• Are studies of similar design and conduct such that an average effect would be clinically meaningful?

Statistical

• Is the observed variability of effects greater than that expected by chance alone?

Are the characteristics and effects of studies sufficiently similar to estimate an average

effect?

48

Summary

49

RCT

Observational

Systematic review of RCTs

50

PRISMA Checklist

51

PRISMA Checklist

52

PRISMA Checklist

53

References• IOM (Institute of Medicine). 2011. Finding What Works in Health Care:

Standards for Systematic Reviews, Washington, DC: Natl. Acad. Press• IOM (Institute of Medicine). 2011. Clinical Practice Guidelines We Can Trust.

Washington, DC: The National Academies Press.• Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern

Med. 1997 Mar 1;126(5):376-80.