Will It Work?Efficacy Studies and Clinical Research

Glenn Pransky, M.D., M.Occ.H.

Director, Liberty Mutual Center for Disability Research

Associate Professor, UMass Medical School

How Do Doctors Decide?

• It ought to work - inductive reasoning

• Others vouch for it - abdication

• Demonstrated effect - deduction

Gastric Freezing

• Cooling to -5° C decreased secretions

• President of ACS tried -10° C; published case series of 20 pts: symptoms, X-ray healing

• 2500 gastric freezing machines sold 1962-66

• 15,000 treatments in US by 1969

Randomized Trial (Ruffin et al, NEJM, 1969; 281, p.16)

• Double - blinded intervention - treating MD and patient didn’t know

• Results:

– Recurrence in 30 (44%) of placebo (warm)

– Recurrence in 35 (51%) of treated patients

Efficacy

• For a given medical problem, efficacy is the probability that treatment significant improvement in outcome, under ideal conditions

• Effectiveness = efficacy in usual conditions of practice

Efficacy Studies

• Phase I - human dosage / toxicity

• Phase II - uncontrolled trial

• Phase III - randomized, controlled trial

RCT Strengths

• Show maximum effect vs. placebo

• Strongest design to bias from several potential sources

• Prove causation

• Compare treatments

RCT Weaknesses

• Infrequent side-effects missed

• Long latency effects often attenuated

• Community usage unknown

• problems with placebo definition

Home vs. Hospital Care for Suspect MI (Hill, Lancet, 1978, l:837)

• RCT, 6 week follow-up after acute ER visit

• Sickest patients excluded

• Results:

– HOME: 20% of 79 died

– HOSPITAL: 18% of 71 died

• Need 261 / grp. - detect 25% RR difference

• Need 45 / grp. - detect 50% RR difference

RCT Sample Size

• Type I error: false optimism; usually = 5% (p-value)

• Type II error: false pessimism; usually = 20%

• Power = prob. of finding A if A is true

• Required parameters:expected control outcome, expected intervention effect, rate of outcome / length follow-up

Preventive Intervention, Uncommon Adverse Outcome

• New treatment for HTN in pregnancy

• Goal: risk from 2% 1%

• Type 1 error 5%, type 2 error 20%

• 2511 individuals needed in each group!

Selection - Exclusions

• Old, young, demented, minorities, pregnant

• Liver or kidney disease - drug metabolism

• Noncompliant patients (run-in period)

Selection - Inclusions

• Volunteers - sicker, compliant

• Referral (selected) population

Treatment problems in efficacy studies

• Atypical = non-generalizable

• Randomized allocation with complex significant factor patterns– Propensity scoring as alternative

• Ethics of no rx. vs. comparison rx.

• Placebo effect

• Multifaceted intervention - investigator attributes to one element

Control

• Blinding - ideally pt., treaters, evaluators

– When is this not feasible?

• Surgical treatments - randomization, but patients & treaters know

Control

• Contamination type II error– sympathy, community care

• Cointervention type I error (unblinding)– ‘more is better,’ support of tertiary care

center vs usual care

Follow-up

• Adequate for stability of 1/ 2outcomes

• Multiple assessments

• Loss - sicker, toxicity, high study burden

• Assume the worst?

Surgery vs. Medical Rx for Bilateral Carotid Stenosis (Fields & Maslenikov, JAMA, 1970, 211: p1993)

• 79 surg, 79 med @ F/U 1 year later

• Surgery: 27% risk stroke/death (p=0.02)

• Of 16 LTFU, 15 were allocated surgery - all had early death or stroke

• Intention-to-treat analysis: risk 16%, p = 0.09

Outcomes

• Clinical / physiologic

• Function

• Employment

• Satisfaction, quality of life

• Side-effects

• Value of surrogate endpoints?

Analytic Methods

• Variable length follow-up Kaplan-Meier survival curve (for terminal event)

• Assumes constant probability

Problems

• Multiple comparisons

• Large trial: statistical but not practical significance

Typical Results

1 = follow-up too short (Type 2 error)

2 = follow-up too long; few subjects or overwhelming additional influences plateau usually = low numbers

0

50

100

t1 t2 t3 t4 t5

t

% a

live 1

2

Adherence Chola, Mortality in CDP (NEJM 303: 1038, 1980)

Risk reduction= .26-.16/.26= 38%

Convincing?

Post-MI secondary prevention trial with clofibrate.

CDP Study (2)

• Same results in placebo group

• Conclusion: compliance survival effect!

Are All Relevant Results Reported?

Measure Placebo Clofibrate

Avg. change serum chol. +1% -9%

Nonfatal MI/1000 7.2 5.8

Fatal MI/1000 1.7 1.6

Total deaths/1000 5.2 6.2

(Oliver et al WHO trial on primary prevention IHD using databrate Lancet 1980; 2, p. 379)

Generalizability

• Adequate description?

• Typical cohort, setting?

• Typical treatment; acceptable, cost, available

• Sideffects - effort to obtain information

Functional Restoration: Pitfalls in Evaluating Efficacy (Gatchel, Mayer, Hazard et al, Spine, 1991, 17:988)

• Staff Training

• Program duplication, same selection criteria & patients

• Consistent evaluation methods, close follow-up

• Report important details (drop-out, total costs)

• Understand key differences - jurisdiction, etc.

Placebo Effects, Other Issues in Pain Treatment (Turner, Deyo, et al., JAMA, 1994, 271:1609)

• Regression to mean apparent benefit regardless of treatment– Patients enroll @ worst point in cyclical course– Also may reflect meas. error, random variation

• Nonspecific Rx effect– Attention, concern; expectations of healing;

“facilitated” reporting

Placebo Effects

• “Change in illness attributable to symbolic import of treatment.”

• Up to 70% of responses to Rx’s initially thought to be efficacious eventually defined as placebo effects

• IMA procedure: 56% significant improvement, 42% NTG use – (Cobb, NEJM, 1959, 260, 1115-8: skin incision

placebo)

• Diskectomy - negative surgical exploration: 37% no sciatica, 42% no LBP

– (Spangfort, Acta Orth Scand, 1972, 142:1)

Factors Influencing Placebo Response

• Patient attitude to provider and treatment

• More compliance

• Provider attitude, unbliding

Placebo Effects in Pain Treatment

• Mimic expected dose-response relationship (pain medications)

• Mimic expected side-effects (drowsiness, nausea)

• Perception: large or injected = strong;

yellow = stimulant

• Suggestion worse condition (Nocebo effect)

Questions

1. Randomization?

2. Blinding?

3. Adequate data collection / reporting?

4. Statistical, clinical significance?

5. All participants included

6. Generalizable

7. Feasible?