Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal...
Transcript of Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal...
![Page 1: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/1.jpg)
Weighing the Evidence:Critical Appraisal Fundamentals
Lita Anglin, MSISCenter for Innovations in the Advancement of Care
Joey Nicholson, MLIS, MPH, CPHNYU Health Sciences Library
March 12, 2019
![Page 2: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/2.jpg)
Learning Objectives & COI Disclosure
In compliance with ACCME Standards for Commercial Support of CME activities, we as the speakers have no relevant financial relationships to disclose.
• What is critical appraisal?• How is it situated within the EBP process?• Common question types/study methodology• Practicing the framework
• Implicit, rapid, full• Critical appraisal tools
![Page 3: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/3.jpg)
What is critical appraisal?
![Page 4: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/4.jpg)
What is critical appraisal? Why would I use it?
A structured, systematic approach to evaluating information for validity, scientific rigor or quality, and usefulness to practice. The process helps clinicians to make clinical decisions based on evidence.
“Using critical appraisal skills and tools enables users of research evidence to reach their own judgements.”
CASP, Critical Appraisal Skills Program
![Page 5: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/5.jpg)
Reading to understand
Reading critically is an active way of reading – a deeper, more complex engagement with a text. It is analyzing, interpreting, and evaluating. It is looking at the text to see the patterns of argument, the validity of the argument, and the value of the work.
Jennifer Duncan. Reading Critically. The Writing Centre, University of Toronto Scarborough https://utsc.utoronto.ca/twc/
![Page 6: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/6.jpg)
Asking questions
• Divergent (background)
• Convergent (foreground/clinical)
![Page 7: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/7.jpg)
Three levels of critical appraisal
• Implicit critical appraisal• Rapid critical appraisal• Full critical appraisal
![Page 8: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/8.jpg)
![Page 9: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/9.jpg)
Try this!
![Page 10: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/10.jpg)
Part of the EBP Framework
![Page 11: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/11.jpg)
Evidence-based practice: step by step. American Journal of Nursing’s 12-part series from the Arizona State University College of Nursing and Health Innovation's Center for the Advancement of Evidence-Based Practice.
Melnyk BM, Fineout-Overholt E, Stillwell SB, Williamson KM
November 2009January 2010March 2010May 2010July 2010 – Critical Appraisal of the Evidence, Part ISeptember 2010 – Part IINovember 2010 – Part IIIJanuary 2011March 2011May 2011July 2011September 2011
![Page 12: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/12.jpg)
One Part of the EBP Framework0. Cultivate a spirit of inquiry.
1. Ask clinical questions in PICOT format.
2. Search for the best evidence.
3. Critically appraise the evidence.
4. Integrate the evidence with clinical expertise and patient preferences and values.
5. Evaluate the outcomes of the practice decisions/changes based on evidence.
6. Disseminate EBP results.American Journal of Nursing, Evidence-based Practice Step by Step series.
![Page 13: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/13.jpg)
13
4 explanations for a study finding
13
Truth
Bias
Chance
Confounding
![Page 14: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/14.jpg)
3 overarching questions to ask to get to the truth1. Are the results of the study valid and reliable?
i.e. Do they describe appropriate study methodology and show how they are accounting for bias, confounding, and chance? Are they measuring what they set out to measure?2. What are the results and how significant are they?
i.e. Do they report the size of effect of their outcome (odds ratio, relative risk, mean difference) and do they state how precise it is (confidence interval)?3. Do they apply to my situation/patient?
i.e. Is the population similar? Is the setting similar? Could similar results be expected in this setting?
![Page 15: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/15.jpg)
What is Evidence?
![Page 16: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/16.jpg)
The Metropolitan Museum of Art, New York. The Great Pyramid and The Great Sphinx, Francis Frith, 1858.
![Page 17: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/17.jpg)
Building the pyramid of evidence
Level VII: Expert opinion or consensus
Level VI: Qualitative or descriptive studies (including evidence implementation projects)
Level V: Systematic review of qualitative or descriptive studies
Level IV: Case-control or cohort studies
Level III: Controlled trials, not randomized
Level II: Randomized controlled trials
Level I: Systematic reviews or meta-analysis
![Page 18: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/18.jpg)
Many Levels of Evidence
![Page 19: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/19.jpg)
What makes the study designs different?
From the Centre for Evidence-Based Medicine: https://www.cebm.net/2014/04/study-designs/
![Page 20: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/20.jpg)
Systematic Reviews/Meta-Analyses – Level 1Best for: Synthesizing evidence for effects of an intervention across multiple studies – All Kinds of Questions
Advantages:• should include critical appraisal of studies examined;• pulls evidence into one place for easier decision making;• can help account for biases, confounding, and chance across lower level study designs.
Disadvantages:• highly susceptible to publication bias; • inconsistent reporting standards for results;• inconsistent adherence to strict methodology.
![Page 21: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/21.jpg)
Systematic Reviews/Meta-Analyses – Level 1
![Page 22: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/22.jpg)
Controlled Trials – Levels 2 and 3Best for: Studying the effect of an intervention on an outcome – Questions of Therapy
Advantages:• unbiased distribution of confounders;• blinding more likely;• randomization facilitates statistical analysis.
Disadvantages:• expensive: time and money;• volunteer bias;• ethically problematic at times.
![Page 23: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/23.jpg)
Controlled Trials – Levels 2 and 3
![Page 24: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/24.jpg)
Cohort Study – Level 4Best for: studying risk factors for developing a given outcome – Questions of Prognosis, Etiology (Cause), Therapy
Advantages:• ethically safe;• subjects can be matched;• can establish timing and directionality of events;• eligibility criteria and outcome assessments can be standardized;• administratively easier and cheaper than RCT.
Disadvantages:• controls may be difficult to identify;• exposure may be linked to a hidden confounder;• blinding is difficult;• randomization not present;• for rare disease, large sample sizes or long follow-up necessary.
![Page 25: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/25.jpg)
Cohort Study – Level 4
![Page 26: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/26.jpg)
Case-Control Study – Level 4Best for: studying rare disorders or circumstances – Questions of Prognosis, Etiology (Cause), Therapy
Advantages:• quick and cheap;• only feasible method for very rare disorders or those with long lag between exposure and outcome;• fewer subjects needed than cross-sectional studies.
Disadvantages:• reliance on recall or records to determine exposure status;• confounders;• selection of control groups is difficult;• potential bias: recall, selection.
![Page 27: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/27.jpg)
Case-Control Study – Level 4
![Page 28: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/28.jpg)
Qualitative and Meta-Syntheses – Levels 5 and 6Best for: Describing a population or problem and highlighting the variables around it to quantify the scope of the problem or explore behaviors/feelings about the problem
Advantages:• Cheap and simple;• Ethically safer.
Disadvantages:• Establishes association at most, not causality;• Highly susceptible to recall bias;• Confounders may be unaccounted for;• May not be representative.
![Page 29: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/29.jpg)
Qualitative and Meta-Syntheses – Levels 5 and 6
![Page 30: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/30.jpg)
The type of question drives study designClinical Question Type Most Likely Study Design You’ll Find
All Questions Systematic Reviews/Meta-Analyses > RCTs
Therapy Randomized Controlled Trials (RCTs)
Etiology (Cause) Cohort Studies
Diagnosis Prospective comparison to gold standard
Prevention Cohort Studies > Case-Control
Prognosis Cohort Studies > Case-Control > Case Series
Meaning Qualitative Studies
Quality Improvement RCTs
Cost Economic Evaluation Studies
![Page 31: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/31.jpg)
Are the results of the study valid and reliable?i.e. Do they describe appropriate study methodology and show how they are accounting for bias, confounding, and chance? Are they measuring what they set out to measure?
What are the results and how significant are they?i.e. Do they report the size of effect of their outcome (odds ratio, relative risk, mean difference) and do they state how precise it is (confidence interval)?
Do they apply to my situation/patient?i.e. Is the population similar? Is the setting similar? Could similar results be expected in this setting?
![Page 32: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/32.jpg)
Practicing critical appraisal
![Page 33: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/33.jpg)
Implicit critical appraisal - quality signals• Quality of the publication• Currency• Scope of the journal• Funding source• Study design• Sample size• Who is citing the work
Let’s try this!
![Page 34: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/34.jpg)
Rapid screening, CASP approachArticle Is the study valid? What are the results? Are the results useful?
Jones 2019
Smith, Thomas 2016
Geller and Young, 2017
Harrison et al 2019
![Page 35: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/35.jpg)
Rapid critical appraisal, Melnyk et al.
Evidence-Based Practice Step by Step: Critical Appraisal of the Evidence: Part I Fineout-Overholt E, Melnyk B; Stillwell,S; Williamson, K. AJN, American Journal of Nursing. 110(7):47-52, July 2010.
Article Level of Evidence How Well Conducted Usefulness to Practice
Jones 2019
Smith, Thomas 2016
Geller and Young, 2017
Harrison et al 2019
![Page 36: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/36.jpg)
Rapid critical appraisal of systematic reviews
Evidence-Based Practice, Step by Step: Critical Appraisal of the Evidence: Part II: Digging deeper—examining the "keeper" studies.
Fineout-Overholt, Ellen; Melnyk, Bernadette Mazurek; Stillwell, Susan B.; Williamson, Kathleen M.
AJN The American Journal of Nursing110(9):41-48, September 2010.
doi: 10.1097/01.NAJ.0000388264.49427.f9
![Page 37: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/37.jpg)
Critical appraisal, full
![Page 38: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/38.jpg)
Critical appraisal tools
![Page 39: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/39.jpg)
How can appraisal tools help?
Critical appraisal tools guide reviewers and improve the objectivity of the review process. They help us remember the task at hand, the criteria we are evaluating, and help us focus on very specific components and methods of the evidence. They ensure that the review process is (mostly) uniform among reviewers and that the process is documented and reproducible. Appraisal tools can help us approach evidence with less bias.
![Page 40: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/40.jpg)
What are some common tools and when are they used?
Checklists
CEBM, The Centre for Evidence-Based Medicine (Oxford)
CASP, Critical Appraisal Skills Programme (Oxford) (8 different lists)
JBI Critical Appraisal Tools (University of Adelaide/Joanna Briggs Institute)
BMJ Best Practice – Critical Appraisal Checklists
Tools for reviewers
Covidence, JBI SUMARI, Excel
![Page 41: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/41.jpg)
![Page 42: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/42.jpg)
JBI
![Page 43: Weighing the Evidence: Critical Appraisal …...Weighing the Evidence: Critical Appraisal Fundamentals Lita Anglin, MSIS Center for Innovations in the Advancement of Care Joey Nicholson,](https://reader033.fdocuments.in/reader033/viewer/2022042314/5f024dd77e708231d4039a4e/html5/thumbnails/43.jpg)
Thank you!