Evidence Based MedicineEvidence Based Medicine

The Hierarchy of Evidence

Ora Paltiel, May 29, 2004

What is evidence? DefinitionWhat is evidence? Definition

That which-

1. Serves to prove or disprove something; that which is used for demonstrating the truth or falsity of something, support, proof

2. Serves as a ground for knowing something with certainty or for believing something with conviction

3. Is properly presented before a court as a means of establishing or disproving something alleged or presumed

Forms of evidenceWhy do we need to be convinced?

• Does A cause B?/ Is A associated with B?• PREVENTION- which preventive measures are

worthwhile?

• THERAPEUTICS• can the treatment work? (efficacy)

• does the treatment work? (effectiveness)• is it appropriate for my patient?

WHY are clinical trials needed?WHY are clinical trials needed?

• “Given the uncertain knowledge about disease course and the usual large variations in biologic measures, it is often difficult to say on the basis of uncontrolled clinical observation whether a new treatment has made a difference to outcome, and if it has, what the magnitude is.”

Ideal clinical trial is one that is randomized and double-blind

Consequences of not conducting appropriate clinical trials at the proper time can be serious and costly

• gastric freezing

• Fetal cell implants in Parkinsonism

• HRT

WHEN are clinical trials needed? WHEN are clinical trials needed?

• Before a therapy becomes standard

• After 10s 100s 1000s or millions have been exposed?

• Digitalis 200 years

• Estrogens 50 years

“The field of therapeutics is replete with examples of new modalities that were taken up with enthusiasm and proved worthless only after they had resulted in many years of futile cost and suffering”

A single treatment

Uncontrolled trial

Apply the treatment to a group of subjects and observe the results.

The problem

This approach makes it difficult to draw conclusions about a treatment’s effectiveness.

The unpredictable course of disease: the natural history of systemic lupus

erythematosus in a patient observed before the advent of immunosuppressive drugs

Can listening to classical music for six months lower cholesterol levels?

Dietary intake and biochemical measures that are affected by diet often change with season.

Example: cholesterol levels probably peak in the winter (holidays, diet) and are lower in the summer (climate, diet).

If the study started in January after a holiday season and ended in July after the greater availability of fresh produce added more fruits and salads to the diet – Wrong conclusion

Hawthorne Effect

Subjects enrolled in a study

• often increase their awareness of the problem being studied

• choose behaviors that they might not have considered otherwise.

Comparison group

When a treatment is described as effective, the question to keep in mind is “Compared to what?”

Placebo controlled trial is a study involving two treatments - the treatment under investigation and an inactive control

Historical controls

Advantages:1. Cheaper2. Faster3. New treatment not withheld from anyone

(“ethically easier”)Sources:1. Literature2. Previous hospital series3. Previous clinical trials

Meta-analysis of historical control studies

for cisplatinum based chemotherapy in

esophageal carcinoma.

Meta-analysis of randomised clinical trials

for cisplatinum based adjuvant/neoadjuvant

chemotherapy in esophageal carcinoma.

What about comparing a treatment group to a convenience sample - a sample chosen not through a formal sampling procedure, but because it is convenient?

Eg:Sunday, Tuesday, Thursday

Monday, Wednesday, Friday

BEWARE: Investigator bias

Subjects as their own controls

not uncommon to see attempts to drop controls from study in the name of “cost or convenience”.

A telltale phrase that should put you on alert is: “subjects were used as their own controls”.

Placebo effect

• Change in outcome based on having received something (belief in its efficacy)

• Called into question in recent meta-analysis (NEJM 2001;344:1954)

• Evidence of existence in trials of pain and chronic symptoms

Total effects of treatment are the sum of spontaneous improvement,

nonspecific responses and the effects of specific treatments

Mrs. XX, 56 year old lawyer

• Inferior MI one year ago• Stopped smoking since then• c/o Hot flashes, dryness, dyspareunia• No family history of breast cancer• Bone density below average for her age• Takes aspirin and B-blocker• Asks about HRT

HRT and secondary prevention of

CAD: Evidence until 1998- observational studies

1 Bush 1987. F/u of 2270 women LRC. RR 0.34 of CV death with HRT, excl. women with previous CV disease did not affect result

2 Sullivan 1990.Follow up 2268 women with coronary angios (1178 HRT users). HRT prolongs survival in women with CAD

HRT and secondary prevention of

CAD: Evidence until 1998- observational studies

3 Henderson 1991. 8881 women in retirement community. 20% lower mortality in estrogen users

4 O’Brien 1996. 204 women post-angioplasty/atherectomy. Estrogen use associated with decrease loss of lumen diameter, decreased restenosis in atherectomy not angioplasty

5 Newton 1997. 726 women retrospective cohort after 1st MI. F/U 2-13 years. RR for reinfarction 0.64 (95% CI 0.3-1.3).

6 Sullivan 1997. 1098 women post-CABG. Estrogen use predicted survival RR = 0.38

7 O’Keefe 1997. Retrospective. 337 women post elective PTCA 1982-94. RR 0.38 (0.19-0.79) for CV event (death, non-fatal MI stroke) for estrogen users

HRT and secondary prevention of

CAD: Evidence until 1998- observational studies

False Hopes

HERS study:Randomized trial of Estrogen Plus Progesterone for Secondary Prevention of Coronary Heart Disease in

Postmenopausal Women Hulley et al JAMA 1998;280:605

• randomized, placebo-controlled double blind trial

• 2763 women with CAD (MI, previous CABG or angioplasty, angio with >50% occlusion) in 20 centers

• Main Results: average follow-up of 4.1 yrs,

no difference in cardiovascular outcomes

HR 0.99 (95% CI 0.8-1.2)

Randomized Trial of Estrogens for Secondary Prevention of Coronary Heart Disease in

Postmenopausal Women JAMA. 1998;280:605

Discrepancy

Is selection bias the only answer?

Mrs XX is confused, are you?

What about primary prevention?What about primary prevention?

Estrogen and DementiaEstrogen and Dementia

• Women’s Health America Inc Issue 65, 2001

• estrogen helps brain cells stay healthy, women who take estrogen after menopause have an “unexpectedly low incidence of Alzheimer’s disease”

• women with Alzheimer’s disease taking estrogen suffer less severe symptoms and slower mental deterioration

Estrogen and dementia: News from Estrogen and dementia: News from RCT RCT

((Shumaker, JAMA 2003;289:2651)Shumaker, JAMA 2003;289:2651)

• 4532 women on WHIMS study free of dementia

• 61 developed probable dementia • 40 (66%) in estrogen arm and 21 (34%)

in placebo arm• HR 2.05 (1.21-3.48)• 6.7% showed significant cognitive

decline compared to 4.8% in placebo

Two questions

• Can I trust this evidence?

(is the witness reliable?)

• Can I apply the findings to my practice setting?