Grading quality of evidence the GRADE approach
description
Transcript of Grading quality of evidence the GRADE approach
GRADING QUALITY OF EVIDENCETHE GRADE APPROACH
Society of General International Medicine
32nd Annual Meeting, May 14th 2009
Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPHEric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH
1
Session outline
Introductions, objectives (5 min)
Overview of the GRADE approach (25 min)
Applying the GRADE approach (45 min)
Wrap-up (10 min) Session evaluation (5
min)
Disclosure
Presenters are members of the GRADE working group and have received honoraria related to this work that were deposited into research accounts
No conflict of interest related to pharmaceutical industry
Objectives
Learning objectives
To enumerate GRADE categories for quality of evidence
To list the GRADE factors that affect the quality of evidence
To apply the GRADE approach to a specific body of evidence
To discuss the strengths and limitations of the GRADE approach
Overview of the GRADE approach
GRADE WORKING GROUP
Grades of Recommendation Assessment,
Development and Evaluation
CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008
“Extent to which confidence in estimate of effect adequate to support decision”
GRADE definition of Quality of Evidence
GRADE rating of outcomes
GRADE rates the quality of evidence for each outcome separately The type of evidence may be different
for different outcomes Different audiences are likely to have
varying perspective on the importance of outcomes
GRADE considers desirable and undesirable outcomes and rates their relative importance
9
Desirable outcomes lower mortality reduced hospital stay reduced duration of disease reduced resource expenditure
Undesirable outcomes adverse reactions the development of resistance costs of treatment
GRADE rating of outcomes
2
Critical for decision making
Important, but not critical for decision making
Of lowimportance
5
6
7
8
9
3
4
1
GRADE rating of outcomes
Ranking outcomes by their relative importance can help to focus attention on those outcomes that are considered most important
Outcome choice should be based on what is important, and not what was measured
12
GRADE rating of outcomes
GRADE uses a comprehensive and transparent conceptual framework for rating the quality of evidence
13
High:
Moderate:
Low:
Very low:
GRADE levels of Evidence
High: considerable confidence in estimate of effect
Moderate: further research likely to have
impact on confidence in estimate, may change estimate
Low: further research is very likely to impact on confidence, likely to change the estimate
Very low: any estimate of effect is very uncertain
GRADE levels of Evidence
Quality starts high for evidence from RCTs
Quality starts low for evidence from observational studies
5 factors lower the quality of evidence
3 factors can increase the quality of evidence
Determinants of quality
Factors that lower quality1. Study limitations (in design and
execution)2. Inconsistency3. Indirectness4. Reporting bias5. Imprecision
17
1. Study limitations (in design and execution)
Inappropriate randomization Lack of concealment Intention to treat principle violated Inadequate blinding Loss to follow-up Early stopping for benefit
Factors that lower quality
From Cates , CDSR 2008
CDSR 2008
Factors that lower quality
Overall judgment required
Factors that lower quality
2. Inconsistency Assess for inconsistency (Heterogeneity)
variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2
If inconsistency look for explanation patients, intervention, outcome, methods
If unexplained inconsistency downgrade quality
Factors that lower quality
Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.
2. Inconsistency
Heparin or vitamin K antagonists for survival in patients with cancer:
Factors that lower quality
Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories.
Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.
2. Inconsistency
Non-steroidal drug use and risk of pancreatic cancer:
Factors that lower quality
3. Indirectness of Evidence Differences in populations/patients
mild versus severe COPD
Differences in interventions all inhaled steroids, new vs. old
Differences in outcomes important vs. surrogate;
Factors that lower quality
Alendronate
Risedronate
Placebo
3. Indirectness of Evidence indirect comparisons
interested in A versus B have A versus C and B versus C
Factors that lower quality
4. Publication bias
Number of small studies
Faster and multiple publication of “positive” trials
Fewer and slower publication of “negative” trials
Factors that lower quality
Egger M, Smith DS. BMJ 1995;310:752-54 27
I.V. Mg in acute myocardial infarctionPublication bias
Meta-analysisYusuf S.Circulation 1993
ISIS-4Lancet 1995
Egger M, Cochrane Colloquium Lyon 2001 28
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Symmetrical:No publication bias
Egger M, Cochrane Colloquium Lyon 2001 29
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Asymmetrical:Publication bias?
Egger M, Smith DS. BMJ 1995;310:752-54 30
I.V. Mg in acute myocardial infarctionPublication bias
Meta-analysisYusuf S.Circulation 1993
ISIS-4Lancet 1995
Egger M, Smith DS. BMJ 1995;310:752-54 31
Meta-analysis confirmed by mega-trials
5. Imprecision small sample size
small number of events
wide confidence intervals uncertainty about magnitude of effect
how to decide if CI too wide? grade down one level? grade down two levels?
Factors that lower quality
Factors that raise quality1. Large magnitude of effect
2. Dose response relation
3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
33
1. Large magnitude of effect large (RRR 50%) can raise by one level very large (RRR 80%) can raise by two
levels common criteria
everyone used to do badly almost everyone does well
Examples oral anticoagulation for mechanical heart
valves insulin for diabetic ketoacidosis hip replacement for severe osteoarthritis
Factors that raise quality
2. Dose response relation
higher INR – increased bleeding
childhood lymphoblastic leukemia risk for CNS malignancies 15 years after
cranial irradiation no radiation: 1% (95% CI 0% to 2.1%) 12 Gy: 1.6% (95% CI 0% to 3.4%) 18 Gy: 3.3% (95% CI 0.9% to 5.6%)
Factors that raise quality
3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
Factors that raise quality
Example 1: higher death rates in private for-profit versus private not-for-profit hospitals
patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals
for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect)
Factors that raise quality
Example 2: hypoglycaemic drug phenformin causes lactic acidosis
The related agent metformin is under suspicion for the same toxicity.
Large observational studies have failed to demonstrate an association
Clinicians would be more alert to lactic acidosis in the presence of the agent
Factors that raise quality
Summary of GRADE framework for rating the quality of evidence
39
Quality of evidence
Study design Lower if Higher if
High Randomised trial Study quality: Serious limitations Very serious limitations I mportant inconsistency Directness: Some uncertainty Major uncertainty Sparse or imprecise data High probability of reporting bias
Strong association: Strong, no plausible confounders Very strong, no major threats to validity Evidence of a Dose response gradient All plausible confounders would have reduced the eff ect
Moderate
Low Observational study
Very low
Evidence Profiles and Summary of Findings (SoF) Tables summarize the rating of the quality of evidence across selected outcomes
41
42
43
Applying the GRADE approach
Exercise: parenteral anticoagulation for prolonging the survival of patients with cancer
Wrap-up
46
Advantages of GRADE
Developed by a widely representative group of international guideline developers
Clear separation between quality of evidence and strength of recommendations
Explicit evaluation of the importance of outcomes
Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings
47
Advantages of GRADE
Transparent process of moving from evidence to recommendations
Explicit acknowledgment of values and preferences
Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers
Useful for systematic reviews and health technology assessments, as well as guidelines
48
Disadvantages of GRADE
Involves a number of judgments that might affect its reliability
Requires expertise/training
49
Session evaluation
Thank you!