1

Critical Appraisal and the Systematic Review of the Literature

User’s Guide to the Medical LiteratureSources: JAMA :

Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice.Gastroenterology & Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

2

Learning Objectives

1. Describe the critical appraisal process and how it relates to a systematic review

2. Describe the purpose of a systematic review3. List the steps in the systematic review

process4. Critically appraise a selected article (the one

you found this week!)

3

What is critical appraisal?

Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice.Gastroenterology & Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

• A systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of the research findings. – …most important are an evaluation of the

appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design…

4

How to Read a Scientific Paper• …first read the title and abstract…– Review in your mind what you know about the topic…

• Allows you, as the reader, integrate the new information into your previous knowledge about the topic. That is, it is used as a part of the self-education process that any professional must continue throughout his/her career.

– If you are very familiar with the field, the Introduction can be skimmed or even skipped... the logical flow of most papers goes straight from the introduction to results…the paper should be read in that way as well…

by John W. Little and Roy Parker--University of Arizona: http://www.biochem.arizona.edu/classes/bioc568/papers.htm#reading

5

Top Ten Questions to Ask:

1. Is the study question relevant?2. Does the study add new knowledge?3. What type of research question is being

asked?4. Was the study design appropriate to the

research question?5. Was bias addressed?

Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice.Gastroenterology & Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

6

Questions, continued…

6. Was the study performed according to the original protocol?

7. Did the study test a stated hypothesis?8. Were the statistical analyses performed

correctly?9. Do the data justify the conclusions?10. Are there conflicts of interest?

Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice.Gastroenterology & Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

7

How is s critical appraisal accomplished?

8

Discuss your article• Database(s) searched• General topic area researched• Did you find a lot, too little, too much?• Article title• Type of research question/hypothesis• Was bias addressed?• Did the data justify the conclusions?• Other comments you want to share…

9

10

What is a systematic review?

• A systematic review is a review of scientific studies on a specific topic. It uses a formal process to:– Identify all relevant studies– Assess their quality (critical appraisal)– Summarize the evidence

Source: http://www.thecommunityguide.org/about/methods.html

11

Why do a systematic review?

• Systematic reviews help make sense of large bodies of scientific literature by applying the scientific process to:– Reduce bias in how conclusions are reached– Improve the power and precision of results – Summarize evidence about the effectiveness of

particular approaches for addressing a problem– Analyze generalisability of findings – Identify knowledge gaps and need for additional

researchSource: http://www.thecommunityguide.org/about/methods.html

12

What you will accomplish…• Clearly formulate/state the research question• Develop the research protocol• Conduct the literature search• Complete data extraction after developing a form• Appraise the quality of the selected studies• Analyze the data and results in narrative and

tabular formats• Interpret the results, including the strengths and

weaknesses of the studies

13

Systematic review methodology …• Were all relevant studies included/excluded?• Were selected articles appraised and data

extracted?• Was their sufficient detail?• Was the quality of the primary studies

assessed?• Did researchers assess appropriateness of

combing results?Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice.Gastroenterology & Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

14

The Systematic Review Process 1. Define the question/formulate the problem2. Conduct the literature search/locate and select the studies3. Apply inclusion and exclusion criteria/assess study quality4. Create data abstraction/find relevance and meaning in

your results5. Conduct analysis/draw conclusions

15

16

An example from JAMA via PubMed

17

18

Useful Resources• AHSL EBM Search Engine -

http://www.ahsl.arizona.edu/ebmsearch/index.cfm • BU Alumni Medical Library: Conducting a Systematic Review (under

Subject-Specific Tutorials)• Data Extraction Form Samples: www.cochrane.iwh.on.ca/pdfs/data

_extract_04.pdf and http://www.iss.it/binary/neph/cont/Data%20extraction%20form.1183648415.pdf

• Searching the Cochrane Database • The Centre for Evidence-Based Medicine

- Systematic Review of a Therapy Worksheet • Washington University St Louis Becker Library Evidence at Becker • What is a Systematic Review? (from the BMJ)• Wikipedia: http://en.wikipedia.org/wiki/Systematic_review

19

Take time now to answer these:

1. What type of research question was asked.?2. Did the study address the key potential

sources of bias? Explain.3. What is the stated hypothesis of the study?4. Did the data adequately justify the

conclusions? Discuss.

20

Did we accomplish the Learning Objectives

1. Describe the critical appraisal process and how it relates to a systematic review

2. Describe the purpose of a systematic review3. List the steps in the systematic review

process4. Critically appraise a selected article

21

Thanks….questions??

JDDoyle@email.arizona.edu

Office: 602-827-2062Mobile/Text: 602-689-5976