Gordon GuyattDepartment of Health Research MethodsEvidence and ImpactMcMaster University

¡ Co-chair of the GRADE working group

¡ Have written about dangers of conflict of interest – therefore, when it comes to intellectual conflict of interest, I have intellectual conflict of interest

¡ Current problems in health care

¡ What is a guideline?

¡ What makes a trustworthy guideline?§ Collect and summarize evidence§ Move from evidence to recommendations

¡ Slow dissemination of innovations

¡ Related problems especially U.S§ Over diagnosis, overtreatment§ Low value care, waste§ Runaway costs

¡ Trustworthy guidelines could help

Clinical guidelines - Not guidelines on how to do tests in the laboratory

¡ Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.

¡ Most guidelines deal with therapeutic issues

¡ Guidelines in diagnosis present special challenges

¡ GOBSAT or trustworthy

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Experts§ Concern about conflict of interest

¡ Methodologists§ Concern about conflict of interest

¡ Front-line clinicians

¡ Patients§ Concern about representativeness

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Types§ Financial, Intellectual, Professional

¡ Possible approaches§ Ignore - GOBSAT§ Declare § Forbid § Manage

▪ Balance▪ Selectively forbid

¡ 500 recommendations, 14 panels

¡ COI tension

¡ COI financial § Grant funding, speaker’s bureaus, § Consultancy, advisory board, ownership

¡ COI intellectual§ Authorship paper directly bearing recommendations

¡ Conflict, no participation make recommendation

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Partial thromboplastin time accurate test (more or less) § Pre-op PTT in major surgery improves outcomes

¡ D-dimer accurate test (more or less)¡ Suspected DVT, high probability improves outcomes¡ Suspected DVT, low probability improves outcomes

¡ ELISA for HIT accurate test (more or less)¡ ELISA any pt suspected HIT improves outcomes¡ ELISA in patients with intermediate or high probability 4T

score improves outcomes

¡ Very low prevalence§ False positive harm outweigh benefits

¡ Clinicians misinterpret test result¡ Clinicians just as good without the test¡ Test results don’t lead to change in

management¡ Clinicians don’t offer optimal management¡ Optimal management doesn’t improve

outcomes

¡ PICO format¡ Pt: Suspect heparin induced thrombocytopenia

¡ Intervention: 4T + ELISA § Low 4T heparin; Intermediate or high on ELISA result

¡ Comparator: 4T § heparin for low, argatroban for intermediate and high

¡ Outcomes:§ Thrombosis § Bleeding§ Mortality§ Resource use (cost)

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Systematic reviews of best evidence

¡ What makes a systematic review systematic?§ Explicit eligibility criteria§ Comprehensive search§ Explicit consideration of evidence limitations

▪ Risk of bias, Imprecision, Inconsistency

§ Duplicate judgements (eligibility, RoB)§ Appropriate analysis

Study designs RCTs

Studies inform mortality, morbidity, symptoms, quality of life, resource use

Randomised Trial orObservational Study

Accuracy Study

Target population

New test(s)Old test(s) New test(s) + Reference test

One

step

inference

Twostepinference

Managementdependingon results

Patient-important outcomes

TP + FP FN + TN TP FP FN TN

Assumptionsor indirect evidence aboutmanagementof patients correctly or incorrectly classified aspositive or negative with the newor old test(s)

Judgements about patient-important outcomeswith a new test and a reference test

Managementdependingon results

Managementdependingon results

Patient-important outcomes

TP + FP FN + TN

Managementdependingon results

Target population

Study designs Linked EvidenceLook for diagnostic test accuracy studies

And then draw inferences from other evidence

Randomised Trial orObservational Study

Accuracy Study

Target population

New test(s)Old test(s) New test(s) + Reference test

One

step

inference

Twostepinference

Managementdependingon results

Patient-important outcomes

TP + FP FN + TN TP FP FN TN

Assumptionsor indirect evidence aboutmanagementof patients correctly or incorrectly classified aspositive or negative with the newor old test(s)

Judgements about patient-important outcomeswith a new test and a reference test

Managementdependingon results

Managementdependingon results

Patient-important outcomes

TP + FP FN + TN

Managementdependingon results

Target population

Strategy4(4Tsscore)

PatientswithsuspectedHIT(n=1000)

Lowprob (n=558)

T(n=1)MB(n=0)

Totaladverseoutcomes:Thrombosis(n=40)Majorbleeds(n=30)

Argatroban Argatroban

Intermediateprob (n=360) Highprob (n=82)

HIT(n=1)

NotHIT(n=557)

HIT(n=50)

NotHIT(n=310)

HIT(n=52)

NotHIT(n=30)

Notx

T(n=18)MB(n=3)

T(n=9)MB(n=3)

T(n=3)MB(n=19)

T(n=9)MB(n=3)

T(n=0)MB(n=2)

Strategy5(4Tsscore+PolyspecificELISA):Testpatientswithanintermediate/high4Tsscore

andtreatifpositive

PatientswithsuspectedHIT(n=1000)

Lowprob (n=558)

T(n=1)MB(n=0)

Totaladverseoutcomes:Thrombosis(n=48versus40)Majorbleeds(n=12versus30)

Intermediateprob (n=360) Highprob (n=82)

HIT(n=1)

NotHIT(n=557)

HIT(n=52)

NotHIT(n=30)

Notx

T(n=18)MB(n=3)

HIT(n=50)

NotHIT(n=310)

ELISA+(n=48)

ELISA-(n=2)

ELISA+(n=41)

ELISA-(n=269)

T(n=18)MB(n=6)

ELISA+(n=50)

ELISA-(n=2)

ELISA+(n=4)

ELISA-(n=26)

T(n=11)MB(n=3)

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ Many available

¡ Cause of confusion, dismay

¡ GRADE (Grades of recommendation, assessment, development and evaluation)

¡ Guidance: BMJ 2004, 6 part series 2008

>100organizationshaveadoptedGRADE

AssessingQualityoftheEvidencebyOutcome

Table:'GRADE's'approach'to'rating'quality'of'evidence'(aka'confidence'in'effect'estimates)'For$each$outcome$based$on$a$systematic$review$and$across$outcomes$(lowest$quality$across$the$outcomes$critical$for$decision$making)

1.''Establish'initial'level'of'certainty'

' 2.''Consider'lowering'or'raising'

level'of'certainty'

' 3.''Final'level'of''

certainty'rating'

Study&design& Initial&certainty&&in&the&evidence&

' Reasons&for&considering&lowering&&or&raising&certainty&&

' Certainty&&in&the&evidence&&

across&those&considerations&''! 'Lower'if' ''" 'Higher'if*'

Randomized&trials! & High'certainty'

Risk'of'Bias!

Inconsistency!

Indirectness!

Imprecision'

Publication'bias'

Large'effect'

Dose'response'

All'plausible''confounding'&'bias'• would!reduce!a!demonstrated!effect!!

'''or'• would!suggest!a!spurious!effect!if!no!effect!was!observed!

High!⊕⊕⊕⊕'

! ' Moderate'⊕⊕⊕!'

Observational&studies! & Low'certainty'

Low'⊕⊕!!'

! ' Very'low'⊕!!!'

*upgrading criteria are usually applicable to observational studies only.

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

Whose values?: Patients

¡ Right panel ¡ Deal with conflict of interest

¡ Clearly articulated question¡ Gather and summarize the best evidence

¡ Rate quality of evidence

¡ Consideration of values and preferences

¡ Rate strength of clear recommendations

¡ strong recommendation§ benefits clearly outweigh risks/hassle/cost§ risk/hassle/cost clearly outweighs benefit

¡ What can downgrade strength?

¡ Low quality evidence

¡ Close balance between up and downsides

¡ Variability in patient preference§ strong, almost all same choice (> 90%)§ weak, choice varies appreciably

¡ Interaction with patient§ strong, just inform patient§ weak, ensure choice reflects values

¡ Use of decision aid§ strong, don’t bother; weak, use the aid

¡ Quality of care criterion§ strong, consider; weak, don’t consider

¡ Look to trustworthy guidelines – reject GOBSAT

¡ Panel: experts, methodologists, clinicians, patients

¡ Deal effectively with conflict of interest

¡ Gather and summarize the best evidence§ GRADE

¡ Rate strength of recommendations§ GRADE