Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and...

14
Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER

Transcript of Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and...

Page 1: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Establishing Efficacy through Randomized Controlled

Clinical Trials

Ernst R. Berndt, Ph.D.

MIT and NBER

Page 2: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Goal of Today’s Presentation

• Help you to become critical, informed readers of reports/articles from randomized controlled trials (“RCTs”), and of associated research proposals

• Realize that “the devil is in the details”

Page 3: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

BackgroundBaseline health status,demographics,

Change in medical intervention, lifestyle, education,Health Status = f comorbidities, other concomitant(symptom & medications, history of non-

responsivenessdisease metrics) to TX, host of other factors

or

HS = f (HS0, MI, everything else)

Issue: How to measure HS / MI ?

Page 4: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Possibilities & Problems

• Retrospective data and use of multivariate statistical analysis

• Prospective “naturalistic” data and use of multivariate statistical analysis

• Prospective randomized controlled (double blinded?) trial

• Other?

Page 5: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Why Randomize?

• Members of the treatment (“TX”) and control groups tend to be comparable on all variables, known and unknown

• Provides basis for statistical analysis

Page 6: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Pitfalls to Randomization

• Timing (before vs. after patient enters trial)

• Keeping track of drop-outs (non-random?)– “intent to treat” analysis– “last observation carried forward”

• Blinding

• Randomization need not generate “median person”

Page 7: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

On External Validity

• What affects generalizability of study findings to population as a whole?– Exclusionary & inclusionary criteria

• How to assess generalizability?– Any rigorous test?

• Hawthorne effect present?

Page 8: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

P-Values and Significance Levels

Definitions: Observe Hypothesize

MI Does Make a Difference 0+ H+

MI Does Not Make a Difference 0- H-

P-Value: P (O+/H-) -- false-positive rate, often .05 or .01

-- aka Type 1 or alpha error

Beta: P (0-/H+) -- false negative probability; for given n,

lower alpha error rate => higher beta

Sensitivity: P (0+/H+) -- true-positive rate, 1-beta or “power”

Specificity: P (O-/H-) -- true-negative rate

Page 9: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Implications

• P-value does not directly tell us the probability that H+ is true

• Classical statisticians “reject the H-” but cannot “accept the H+” (but Bayesians can go further than classical statisticians …)

• For given beta, why do “head to head” RCTs require larger sample sizes than “placebo-controlled” trials?

• Difference between “statistical significance” and “practical importance”

Page 10: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Related P-Value Issues

• Subgroups – must they be specified ahead of time?– Data mining and data dredging– Bonferroni adjustments to overall distinct sub-

group P-values

• Stopping rules– Sequential design vs. chance result– FDA “confirmatory trials”

Page 11: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Internal Validity

• Does credible physiological theory suggest mechanism of action?

• Persuasive studies in animals?• Replicated in other human studies?

– FDA requirement for two RCTs

• Study undertaken by team with good credentials?

• Was publication in peer-reviewed journal?

Page 12: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Single vs. Multi-Site Trials

• MS accelerates patient recruitment

• Less homogeneity of patients and treatments in MS trials, but greater external validity

• MS trials typically simpler

• MS trials give evidence of reproducibility

Page 13: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Quality Control of RCTs

• Very detailed protocol, including endpoint metrics, a hallmark of a good study – done ex ante, not ex post– Use of “surrogate markers” / intermediate outcomes

• Accuracy of case report forms• Trial design isolates effect of MI?• What if randomization “fails” in TX vs. control group

on some dimensions?– Additional statistical analysis– Site effects?

• Keeping track of drop-outs, and reasons for drop-outs

Page 14: Establishing Efficacy through Randomized Controlled Clinical Trials Ernst R. Berndt, Ph.D. MIT and NBER.

Related Class Topics

• Efficacy vs. effectiveness• Use of clinical research organizations (CROs) in

outsourcing clinical trials• Regulations on disseminating efficacy findings

from RCTs• Use of retrospective medical claims data bases• Other outcomes (quality of life, costs, economic

benefits)