GIVING GRADED RECOMMENDATIONS –

CHALLENGES AND SOLUTIONS The New York Academy of Medicine

Teaching Evidence Assimilation for Collaborative Healthcare

New York, August 8, 2013

Yngve Falck-Ytter, MD, AGAF for the GRADE teamAssociate Professor, Case Western Reserve University, Case & VA

Medical CenterChief, Gastroenterology & Hepatology, VA Medical Center,

Cleveland

1

How did we make clinical decision? If the basis was not evidence – what

was it? Expert recommendations

2

Institute of Medicine March 2011 report:

“Clinical Practice Guidelines We Can Trust”1. Establishing transparency2. Management of conflict of interest3. Guideline development group

composition4. Evidence based on systematic reviews5. Method for rating strength of

recommendations6. Articulation of recommendations7. External review8. Updating

3

Quality of CPG based on IOM criteria

169 oncology CPGs evaluated (2005-2010)

60% published after 2007

Not a single CPG met all 8 IOM criteria

4Reams et al. Journal of Clinical Oncology 2013

Quality of CPG based on IOM criteria

5

2. C

OI

1. T

rans

pare

ncy

3. G

roup

com

posi

tion

4. B

ased

on

SR

5. R

atin

g re

cs

7. E

xt. r

evie

w

8. U

pdat

ing

6. W

ordi

ng

6

Before GRADE

Level of evidence

I

II

III

IV

V

Source of evidence

SR, RCTs

Cohort studies

Case-control studies

Case series

Expert opinion

A

Grades of recomend.

B

C

D

7

Before GRADE

Level of evidence

IaIbII

III

IV

V

Source of evidence

Meta-analysisRCTsCohort studies

Case-control studies

Case series

Expert opinion

A

Grades of recomend.

B

C

D

So what is quality of evidence?

Confidence in evidence Confidence in the evidence of benefits Confidence in the evidence of downsides Confidence in the evidence in the

balance

Recognizing that not all outcomes are equal

8

Importance of outcomes

9

Intermediate outcomesPositive hepatitis B core antibody

Amnestic response to re-challenge

Loss of protective surface antibody

Question (PICO)

Should health care worker receive booster vaccination vs. not?

Final health outcomesMortalityLiver cancerLiver cirrhosis

Chronic hepatitis B infection

Acute symptom. infection

I B II V III

A grading system needs to be outcome-centric

QualityQualityQuality

Old system

Outcome #1Outcome #2Outcome #3

GRADE

10

Grades of Recommendations Assessment, Development and Evaluation

11

70+ Organizations

12

2005 2006 2007 2008 2009 2010 2011

Where GRADE fits inPrioritize problems, establish panel

Find/appraise or prepare: Systematic reviewSearches, selection of studies, data collection and

analysis(Re-) Assess the relative importance of outcomes

Prepare evidence profile: Quality of evidence for each outcome and summary

of findingsGuidelines: Assess overall quality of evidenceDecide direction and strength of recommendation

Draft guidelineConsult with stakeholders and / or external peer

reviewerDisseminate guidelineImplement the guideline and evaluate

GRAD

E

13

GRADE expands quality of evidence determinants

Methodological limitations

Inconsistency of results

14

Risk of bias

Allocation concealment

Failure of blinding

Losses to follow-up

Incomplete reporting Indirectness

of evidence

Imprecision of results

Publication bias

15

GRADE: Quality of evidence

Although quality of evidence is a continuum, we suggest using 4 categories:

High Moderate Low Very low

For guidelines: The extent to which our confidence in an estimate of the treatment effect is adequate to support a particular recommendation.

Determinants of quality

RCTs start high Observational studies start low

16

Quality of evidence: beyond risk of biasDefinition: The extent to which our confidence in an

estimate of the treatment effect is adequate to support a particular recommendationMethodological

limitationsInconsistency

of resultsIndirectness of evidence

Imprecision of results

Publication bias

Risk of bias:

Allocation concealment

BlindingIntention-to-treatFollow-upStopped early

Sources of indirectness:

Indirect comparisons

PatientsInterventionsComparatorsOutcomes

17

18

Quality assessment criteria

Lower if…Quality of evidenceHigh

Moderate

Low

Very low

Study limitations(design and execution)

Inconsistency

Indirectness

Imprecision

Publication bias

Higher if…

What can raise the quality of evidence?

Study design

RCTs

Observational

studies

BMJ 2003;327:1459–61 19

2020

Question to the audience

A. High B. Moderate C. LowD. Very low

You review all colonoscopies for average risk colon cancer screening in your health system and document a percentage of patient who developed a perforation after the procedure (evidence of free air on imaging). No comparison group without colonoscopy available. Rate the quality of evidence for the outcome perforation:

21

22

Quality assessment criteria

Lower if…Quality of evidenceHigh

Moderate

Low

Very low

Study limitations(design and execution)

Inconsistency

Indirectness

Imprecision

Publication bias

Higher if…Study design

RCTs

Observational

studies

Large effect (e.g., RR 0.5)Very large effect (e.g., RR 0.2)

Evidence of dose-response gradient

All plausible confounding… …would reduce a demonstrated effect

…would suggest a spurious effect when results show no effect

23

GRADE evidence profile: HCC associated with HCV eradication

Morgan R, Baack B, Smith B, Yartel A, Pitasi M, Falck-Ytter Y. Ann Intern Med. 2013;158:329-337.

Summary of findings Anticipated absolute effects

Studies (N)

Issues Non-response

Viral eradication

Relative effects

HCCs with no Rx

HCCs after viral eradication

Quality of evidence

Outcome: Hepatocellular carcinoma (importance: critical for decision making)12 obs. studies (25,906)

No serious* risks of bias; large effect

990/16,3126.1%

145/9,1851.6%

RR 0.24

95% CI: 0.18 – o.31

All stages of fibrosis:17 HCCs per 1,000

14 fewer HCCs per 1,000[-12; -15]

due to large effect

* Most studies controlled for baseline liver disease severity (for example, presence of cirrhosis) and other important confounders, such as hepatitis B virus infection.

Advanced fibrosis:33 HCCs per 1,000

23 fewer HCCs per 1,000[-18; -26]

From evidence to recommendations

24

RCTObser-vational study

High level recommen-

dation

Lower level recommen-

dation

Old system

Quality of evidence

Balance between benefits, harms & burdens

Patients’ values &

preferences

GRADE

Values and preferences

Implicit value judgments in recommendations

Trade-offs: example prevention of VTE in surgery Thrombotic events

Deep vein thrombosis, pulmonary embolism Bleeding events

Gastrointestinal bleeds, operative site bleeds Inconvenience of injections

Variability in values and preferences

Case 77 y/o patient with atrial fibrillation,

mild CHF, HTN, DM and history of stroke (fully recovered)

Meds: warfarin, antihypertensives, statin, glyburide

Admitted with nausea/vomiting, then hematemesis; INR 2.5; 1 U blood transfused; EGD: no active bleed, possible Mallory Weiss

This is his second major bleed since he started warfarin one year ago

CHADS2 score

Acceptable additional bleeds? Study: Patients at high risk for atrial fibrillation

and high risk of stroke (h/o CHF/MI); internists and cardiologists

Warfarin decreases risk at cost of increased GI bleeds

Without treatment 100 patients will suffer: 12 strokes (six major, six minor), 3 serious GI bleeds

in 2 years

Warfarin would decrease strokes in 100 patients to 4 per 2 years (8 fewer strokes, 4 major, minor)

How many additional bleeds would you accept in 100 patients over a year, and still be willing to administer/take warfarin?

Slide courtesy of: G. Guyatt; Study: Devereaux et. al., 2001

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

MAXIMUM NUMBER OF ACCEPTABLE EXCESS BLEEDS

NU

MB

ER O

F PH

YSIC

IAN

S/PA

TIEN

TS

Physicians N=63

Patients N=61

PHYSICIAN AND PATIENT BLEEDING THRESHOLDS FOR WARFARIN

Slide courtesy: G. Guyatt; Study: Devereaux et. al., 2001

Strength of recommendation“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”

Understanding values & preferences necessary to trade-off benefits and downsides

Values and preferences should ideally be informed by systematic reviews, but evidence is often sparse

30

Example recommendation

ACCP AT9 recommendation:In patients undergoing major orthopedic surgery (e.g., total hip replacement), we suggest the use of LMWH in preference to the other agents.Patients who place a high value on avoiding bleeding complications and a low value on its inconvenience are likely to choose a compression device (IPCD) over the drug options.

31

4 determinants of the strength of recommendation

Factors that can weaken the strength of a recommendation

Explanation

Lower quality evidence The higher the quality of evidence, the more likely is a strong recommendation.

Uncertainty about the balance of benefits versus harms and burdens

The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely is a weak recommendation warranted.

Uncertainty or differences in patients’ values

The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.

Uncertainty about whether the net benefits are worth the costs

The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted. 32

Monthly cost of cancer drugs

33

11 of 12 cancer drugs approved by the FDA in 2012 cost more than $100,000 / year

Example: ipilimumab (metastatic melanoma)

    Estimated absolute effects at 2 years

 

Outcome Relative effect

(95% CI)

control(per

1000)

ipilimumab

(per 1000)

Difference(per 1000)

Certainty of the effect

Death  HR 0.68(0.55 to

0.85)850 725

125 fewer deaths

(49 to 202 fewer)

High

Serious immune-related

AEs

RR 7.37(4.42 to

12.3)39 289

249 more SAEs

(134 to 441 more)

High

Quality of life

         

Resource use

Ipilimumab: $120,000 (12 weeks induction (4 injections)); Dacarbazine: $400Increase in median survival: 3.6 moIncremental cost-effectiveness: ~$400,000.00 per life year; $ per QALY????

34

Developing recommendations

35

Implications of a strong recommendation Population: Most people in this situation

would want the recommended course of action and only a small proportion would not

Health care workers: Most people should receive the recommended course of action

Policy makers: The recommendation can be adapted as a policy in most situations

36

Implications of a conditional recommendation Population: The majority of people in this

situation would want the recommended course of action, but many would not

Health care workers: Be prepared to help people to make a decision that is consistent with their own values/decision aids and shared decision making

Policy makers: There is a need for substantial debate and involvement of stakeholders

37

Systematic review

Guideline development

PICO

OutcomeOutcomeOutcomeOutcome

Formulate

question

Rate

importance

Critical

ImportantCritical

Lessimportant

Create

evidence

profile with

GRADEpro

Summary of findings & estimate of effect for each outcome

Rate overall quality of

evidence across outcomes based

on lowest quality of critical outcomes

Panel

RCT start high, obs. data start

low1. Risk of bias2. Inconsisten

cy3. Indirectnes

s4. Imprecision5. Publication

bias

Grad

e

down

Grad

e

up

1. Large effect

2. Dose response

3. Confounders

Rate quality

of evidence

for each

outcomeSelect

outcomes

Very low

LowModerate

High

Formulate recommendations:

• For or against (direction)• Strong or weak (strength)

By considering: Quality of evidence Balance

benefits/harms Values and

preferencesRevise if necessary by considering:

Resource use (cost)

• “We recommend using…”• “We suggest using…”• “We recommend against using…”• “We suggest against using…”

Outcomes

across

studies

38

What GRADE isn’t

Not another “risk of bias” tool Not a quantitative system (no scoring

required) Not eliminate COI, but able to minimize Not “expensive”

Builds on well established principles of EBM Some degree of training is needed for any

system Proportionally adds minimal amount of

extra time to a systematic review

40

Summary Using GRADE enables organizations

to produce methodologically rigorous recommendations

It’s sensible, transparent, and systematic and fulfills requirements for use in performance measure production (e.g., NQF, PCPI)

International standardization facilitates direct comparisons across organizations and has the potential to reduce redundancy in efforts 41