False dichotomies and health intervention research designs ... · Week Intervention group Control...

27
False dichotomies and health intervention research designs: Randomized trials are not always the answer Centre for Big Data Research in Health Seminar Series University of New South Wales Friday 3 rd November 2017 Professor Stephen Soumerai Harvard Medical School and Harvard Pilgrim Health Care Institute Department of Population Medicine

Transcript of False dichotomies and health intervention research designs ... · Week Intervention group Control...

Page 1: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

False dichotomies and health intervention

research designs: Randomized trials are not

always the answer

Centre for Big Data Research in

Health Seminar Series

University of New South Wales

Friday 3rd November 2017

Professor Stephen Soumerai

Harvard Medical School and

Harvard Pilgrim Health Care Institute

Department of Population Medicine

Page 2: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Source: Soumerai SB et al. J Gen Intern Med. 2017 Feb;32(2):204-209.

Page 3: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

“Information in administrative data

sets is spurious by default.”

John Ioannidis

Source: Ioannidis, JP. JAMA. 2013;309(13):1410-1.

Page 4: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Background

This statement prolongs the polarizing “all or

nothing” debate on “available data”

Administrative data are not always spurious

Page 5: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

RCTs: the “gold standard”

Usually infeasible in “natural

experiments”

Study endpoints can be manipulated

Patients, are often not generalizable

• May not be blind to treatment

Page 6: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

RCTs are not useful for most policy

interventions

Most national health policies, e.g., High

deductibles, Copays, Pay for Performance,

ACOs

Seat belt laws, Speed Limits

Banning medical technologies (e.g. drugs)

Opioid and antibiotic controls

Smoking regulations, etc.

Page 7: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

What do we mean by “False

Dichotomies”

RCTs vs “everything else”

Ignores Quasi-Experiments

Campbell & Stanley (1963)

• Revised in 1979 and 2002

• Three main categories

–RCTs

–Strong quasi-experiments

–Weak “pre-experiments”

Page 8: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Hierarchy of Strong and Weak Designs:

Capacity to Control for Biases

Strong Design: Often Trustworthy Effects

Intermediate Design: Sometimes

Trustworthy Effects

Weak Designs: Rarely Trustworthy Effects

(No Controls for Common Biases.)

Page 9: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Hierarchy of Strong and Weak Designs:

Capacity to Control for Biases

Strong Design: Often Trustworthy Effects

Multiple RCTs The “gold standard” of evidence,

incorporating systematic review of all

studies.

Single RCT A single, strong randomized

experiment, but sometimes not

generalizable

Interrupted time

series with control

series (CITS)

Baseline trends often allow visible

effects and control for biases. Two

controls.

Page 10: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Hierarchy of Strong and Weak Designs:

Capacity to Control for Biases

Intermediate design: Sometimes Trustworthy Effects

Single ITS Controls for trends, but no comparison.

Before and after

with comparison

group

Pre-post change using two single

observations. Comparability of baseline

unclear.

Weak Designs: Rarely Trustworthy Effects (No Controls)

Uncontrolled

pre-post

Single observations before and after

intervention, no baseline or control

group.

Cross-sectional

designs

Simple correlation, no baseline, no

measure of change.

Page 11: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

intervention intervention

Different Effects That Can Be

Observed in Time Series

before

after

before

after

intervention

beforeafter

intervention

before

after

Page 12: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Times series effects of drug benefit limits and cost sharing on the

average number of prescriptions per pt per month among

noninstitutionalized, chronically ill New Hampshire pts (n=860) and

other pts (n=8002). Soumerai, S et al. N Engl J Med. 1987;317(15)

Single ITS: Sometimes Trustworthy

Page 13: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Another ITS: Sometimes Trustworthy

Rates of antidepressant use and psychotropic drug poisoning per quarter before

and after the warnings among young adults (18-29) enrolled in 11 health plans in

nationwide Mental Health Research Network. Source: Lu CY et al. BMJ. 2014 Jun 18;348:g3596.

Page 14: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

ITS ~200 years ago: Puerperal fever monthly mortality rates at Vienna Maternity

Institution 1841-1849. Rates drop when implementing handwashing.

Source: Semmelweis I (1861). Die Aetiologie, der Begriff und die Prophylaxis des Kindbettfiebers. [The

etiology, concept, and prophylaxis of childbed fever]. Budapest and Vienna: Hartleben.

Page 15: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

A strong interrupted time-series design debunked IHI’s claim of

lives savedSource: AHRQ. Statistics on hospital stays. Accessed May 26, 2015.

Page 16: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Without baseline data the press

hyped the findings

AP headline 2008: “Campaign against

hospital mistakes says 122,000 lives

saved”

“A campaign to reduce lethal errors and

unnecessary deaths… has saved an estimated

122,300 lives in the last 18 months….”

“We in health care have never seen or

experienced anything like this,” said Dennis

O’Leary, president of JCAHO.”

Page 17: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Upper graph shows fatal and injurious crashes on Arizona interstate highways with the

increase to 65 MPH maximum speed limit. The lower graph indicates fatal and injurious

crashes on Arizona interstate highways with no change in the 55 MPH maximum speed

limit.

Source: Epperlein T. Arizona: Arizona Statistical Analysis Center; 1989.

ITS with Control Series

Page 18: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT
Page 19: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Objectives

Impacts of health system interventions

uncertain

Aim: Do the results of ITS differ from cluster

RCTs?

Page 20: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Results

ITS and RCTs were similar

• ITS with concurrent controls important

• Need to analyze baseline/follow-up trends

in cluster RCTs

Page 21: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Why RCTs should use controlled ITS

Ex: Dedicated chest pain unit, UK

Studied whether a chest pain unit

(CPU) would reduce hosp. admissions

14 hospitals randomized to establish a

chest pain unit, or not

90,000 visits with chest pain over 2 yrs

Source: Goodacre S et al. BMJ. 2007;335:659

Page 22: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Conventional Cluster (Diff-in-Diff)

RCT-perspective

0

50%

60%

70%

80%

10%

20%

30%

40%

Pro

po

rtio

n a

dm

itted

Before

Control CPU hospital

After

Reanalysis of data from Goodacre S et al. BMJ. 2007;335:659

Page 23: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Difference in admission rate between

intervention and control group

05

%1

0%

15%

20%

25%

DIffe

ren

ce in

adm

issio

n r

ate

(in

terv

en

tion

-con

tro

l)

0 5 10 15 20 25Months

Reanalysis of data from Goodacre S et al. BMJ. 2007;335:659

Page 24: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

02

04

06

0

Nu

mb

er

of clin

ical pro

ble

ms a

dd

ed

to m

edic

al re

co

rd

5 10 15 20 25 30 35 40 45 50Week

Intervention group

Control group

Wright et al

The average effect size

only tells us half the

story.

ITS of RCT to Increase Reporting of

Clinical Decision Problems in EHR

Reanalysis of data from Wright A et al. J Am Med Inform Assoc. 2012;19(4):555-561

Without ITS, the decay to

no effect is not observable.

Page 25: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

12

34

56

Fa

lls p

er

resi

de

nt-

yea

r

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17Month

Intervention group

Control group

Kerse 2004

ITS of RCT Data: Fall Prevention in LTC

Large Baseline and Follow-

up Difference Hidden in

Conventional RCT. Not

interpretable.

Reanalysis of data from Kerse, N et al. J Am Geriatr Soc 2004;52(4)524-31

Page 26: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Summary of ITS of RCT Studies

Interrupted time series analysis is valuable in

evaluation of health systems and policy

interventions

• When RCTs are not feasible

• In the analysis of data from cluster RCTs

Important information may be lost if cluster

RCTs do not consider changes over time

Page 27: False dichotomies and health intervention research designs ... · Week Intervention group Control group Wright et al The average effect size only tells us half the story. ITS of RCT

Conclusions

Research design is the first consideration in

addressing trustworthiness of research.

Medical and graduate schools should

emphasize weaknesses of uncontrolled or

cross-sectional designs and include stronger

research designs.

Well controlled studies can save lives, while

weak designs promote wasteful programs,

and jeopardize public health.