Journal of Clinical Epidemiology 63 (2010) 620e626

Summary-of-findings tables in Cochrane reviews improvedunderstanding and rapid retrieval of key information

Sarah E. Rosenbauma,*, Claire Glentonb, Andrew D. Oxmana

aNorwegian Knowledge Centre for the Health Services, Oslo, NorwaybDepartment of Global Health, Sintef, Oslo, Norway

Accepted 22 December 2009

Abstract

Objective: To measure the effects of a summary-of-findings (SoF) table on user satisfaction, understanding, and time spent finding keyresults in a Cochrane review.

Study Design and Setting: We randomized participants in an evidence-based practice workshop (randomized controlled trial [RCT] I)and a Cochrane Collaboration entities meeting (RCT II) to receive a Cochrane review with or without an SoF table. In RCT I, we measureduser satisfaction. In RCT II, we measured correct comprehension and time spent finding key results.

Results: RCT I: Participants with the SoF table (n 5 47) were more likely to ‘‘agree’’ or ‘‘strongly agree’’ that it was easy to find resultsfor important outcomes than (n 5 25) participants without the SoF tabled68% vs. 40% (P 5 0.021). RCT II: Participants with the SoFtable (n 5 18) were more likely to correctly answer two questions regarding results than (n 5 15) those without the SoF table: 93% vs.44% (P 5 0.003) and 87% vs. 11% (P ! 0.001). Participants with the SoF table spent an average of 90 seconds to find key informationcompared with 4 minutes for participants without the SoF table (P 5 0.002).

Conclusion: In two small trials, we found that inclusion of an SoF table in a review improved understanding and rapid retrieval of keyfindings compared with reviews with no SoF table. � 2010 Elsevier Inc. All rights reserved.

Keywords: Knowledge translation; Health numeracy; Risk communication; Systematic reviews; Information design; Randomized controlled trial

1. Background

Summaries of evidence for health professionals exist indifferent formats and for different purposes [1]. Structuredabstracts, sometimes the only part of a study or review thatreaders view or use [2,3], were originally developed to as-sist readers in retrieving, selecting, and critically appraisingrelevant literature [4e6]. More recently, other forms ofsummaries have surfaced, such as the ELPS (electroniclong, paper short), Short Cut, and Pico formats developedby the British Medical Journal [7,8], motivated by the needfor a better utilization of the respective advantages of papervs. the Web. Secondary journals, such as American Collegeof Physicians (ACP) Journal Club and other evidence-based journals [9e12], produce brief summaries of individ-ual studies and reviews, selected for their clinical relevanceand newsworthiness. These quality-assessed resources aimto limit the number of journals one needs to access to keep

* Corresponding author. Department of Communication, Norwegian

Knowledge Centre for the Health Services, Boks 7004, St Olavs plass,

Oslo 0130, Norway. Tel.: þ47-23255000; fax: þ47-23255010.

E-mail address: sarah.rosenbaum@kunnskapssenteret.no (S.E.Rosenbaum).

0895-4356/$ e see front matter � 2010 Elsevier Inc. All rights reserved.

doi: 10.1016/j.jclinepi.2009.12.014

abreast with new research and the amount of effort neededto spend on critical appraisal. Online services, such as Clin-ical Evidence [13] and UpToDate [14], package evidencesummaries together with general facts about the topic, rec-ommendations or links to guidelines, and other referencesto create a comprehensive one-stop reference.

The format, content, and representation of the data inthese summary types vary. Although some include tablesor figures, most tend to be dominated by text. Althoughthere is a growing amount of literature in the area of riskcommunication for consumers [15e18], we found few pub-lished studies specifically evaluating different forms of ab-stracts, synopses, or summaries with regard to their effecton clinicians’ understanding of the main messages or theirdecision making [19,20].

We sought to develop and evaluate a summary formatthat could be understood by users of systematic reviews,both health professionals and other relevant groups, andthat was feasible to implement in Cochrane reviews. Theproject was one of several initiatives within the CochraneCollaboration [21] in recent years to develop and evaluatesummaries of Cochrane reviews for different target groups.

621S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

What is new?

Key finding� Inclusion of a summary-of-findings (SoF) table in

a Cochrane review helped readers understand theresults more correctly and find key informationabout the main outcomes and quality of the evi-dence faster compared with a review with no table.

What this adds to what is known?� Evidence summaries and abstracts of systematic

reviews may be improved by adding SoF tables.

What are the implications? What should change now?� SoF tables will be included in Cochrane reviews.

Authors and publishers of other systematic reviewsand evidence summaries should consider includingSoF tables to facilitate more effective and efficientuptake of key information.

Our starting point was the GRADE summary-of-findings(SoF) table. Grading of Recommendations Assessment, De-velopment and Evaluation (GRADE) is a system for evaluat-ing quality of evidence, and it encourages authors to report onthe most important outcomes, including adverse effects[22,23]. An output from GRADE is a table presenting the re-sults (or lack of them) and quality-of-evidence scores on a sin-gle page in a standardized format. Based on extensivefeedback from stakeholders and testing with users, we devel-oped an SoF table for inclusion in Cochrane reviews. The de-velopment of the SoF table’s content, formatting, and datarepresentation is described in a separate article [24].

Here we report our evaluation of the SoF table. We mea-sured the effect of including an SoF table in a Cochrane re-view on user satisfaction, understanding, and time spentfinding key results.

2. Methods

We carried out two randomized controlled trials (RCTs)at different stages of the SoF table development. Theobjective of the first RCT was to assess users’ satisfactionwith the SoF table; the second RCT aimed to assess the ef-fect of the SoF table on users’ understanding of the reviewsand time spent to find answers.

2.1. Participants

The first trial took place during a plenary session atworkshop for newcomers to evidence-based practice. Par-ticipants were asked if they would help evaluate ways ofmaking reviews more accessible and were told that theydid not need to participate or return the questionnaires.

Seventy-two workshop participants and tutors completedthe questionnaires out of approximately 90 people present.These were largely health professionals, many of whomwere beginners in evidence-based health care and did nothave English as their first language.

The second RCT took place at a meeting for members ofContinental European Cochrane entities. The 33 participantswere staff members from Cochrane entities, including reviewgroup coordinators, trial search coordinators, and Centrestaff. Everybody present participated. Most did not have En-glish as their first language. Participants in both RCTs had atleast a basic understanding of what a systematic review was.

2.2. Comparisons

Participants were randomized into three groups:

� The review without an SoF table;� The review with an SoF table (placed after the

abstract) with limited formatting (as would be possi-ble in current Cochrane review software);� The review with an SoF table (placed after the

abstract) with full formatting (as might not be possi-ble in current Cochrane review software).

2.3. Randomization

We used block randomization with 25 blocks of three thatwas generated on http://www.randomization.com. The ques-tionnaires were numbered sequentially the day before andwere passed out to all of the participants at each meeting.

2.4. Outcome measurement

We measured outcomes through a multiple-choice ques-tionnaire that included questions about the participants andtheir degree of satisfaction with the accessibility of themain findings of the review. Participants first answeredthe questionnaire based on the version of the review theyhad received. Then, all participants were shown both for-matting versions of the SoF tables and were instructed toanswer a final set of questions measuring their preferencesand attitudes about the inclusion of SoF tables in reviews.

In the second trial, the questionnaire was modified to includequestions that measured actual understanding. We also askedparticipants to note how long it took them to find information.Structured discussions were carried out at the end of both trials.

2.5. Sample size

We used convenience samples for the two trials reportedhere. We initially planned to estimate the sample size fora larger trial based on an alpha of 0.05 for the overall per-ceived accessibility of the main findings of the review and80% power to detect a difference of 1 in the mean rating ofaccessibility, using data from these studies to estimate the ex-pected mean in the control groups and standard deviations,

Fig. 1. Flow diagram of randomized controlled trial I.

622 S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

without adjustment for clustering (within settings) or formultiple comparisons. Secondary analyses of the other ques-tions in the questionnaire were intended primarily to help ex-plain and interpret the results of the primary analysis. Afterthe first trial, we elected to focus the primary analysis ona more objective outcome measure: the proportions of partic-ipants answering correctly the questions about the risks ofsymptomless deep vein thrombosis (DVT) with and withoutcompression stockings for people at low risk. Assuming that50% of the participants would answer correctly without theSoF, an alpha of 0.05 and 80% power, we would need 58 par-ticipants per group to detect a 50% relative improvement.However, because of time and resource constraints and themagnitude of the effect observed in the second trial, we didnot recruit further participants.

3. Results

3.1. Randomized controlled trial Idassessing usersatisfaction with the SoF table

Of approximately 90 people present, 72 completed thequestionnaires (25 without an SoF table, 25 with a formattedSoF table, and 22 with an SoF table with limited formatting

Table 1

Information about participants in randomized controlled trial I

Total n 5 72 (51 health professionals, 28 researchers

[11 marked ‘both,’ 4 marked ’neither’]) Never (before now)

How often do you read reports of randomized trials? 6

How often do you read systematic reviews? 7

How often do you use the Cochrane Library? 23

V

How easy is it for you to understand reports of randomized trials? 2

How easy is it for you to understand reports of randomized trials? 1

that might be needed to meet limitations of the publishingsystem). See Fig. 1 for flow diagram. We have merged thedata for the limited and full formatting versions, as theresults showed no significant differences between thesetwo versions. For information about the participants, seeTable 1.

Participants reading reviews that included the SoF tablewere more likely to respond that:

� The results and quality of evidence were easy or veryeasy to find and to understand;� The main findings were easy to understand;� The main findings were accessible.

These differences ranged from 12% to 28% and were notstatistically significant, except for two measurements: theease to find results for important outcomes and the per-ceived accessibility of the quality of the evidence (Table 2).

Eighty-one percent agreed or strongly agreed that Co-chrane reviews should include SoF tables, 65% with theproposed format; 75% found the explanation sheet helpful.

3.1.1. Structured discussionAlthough most people felt that the results were accessi-

ble with or without the SoF tables, many people gave thewrong answer when they were asked to calculate the inter-vention group risk in a structured discussion after the trial.This discussion led us to the same finding we uncoveredearlier through user tests of the SoF table [24]: presentingresults of the intervention group risk framed as an absolutedifference (9 fewer per 1,000) caused comprehension prob-lems or uncertainty.

3.2. Randomized controlled trial IIdassessing theeffect of the SoF table on users’ correct understandingof reviews and time spent finding answers

After the first RCT, we made a major change based onthe finding that many participants had misunderstood thenumbers expressing absolute effect; we replaced this col-umn with a column for intervention group risk.

In the second RCT, using a revised version of the SoFtable, we tested correct understanding of the tables and timespent to find answers. Thirty-three completed question-naires were returned (18 without and 15 with an SoF table)(Fig. 2). Participants were staff members from Cochrane

Less than

once per year

1e4 Times

per year

5e10 Times

per year

More than 10

times per year

4 20 3 39

5 23 12 20

5 10 7 22

ery difficult Difficult Easy Very easy

29 34 5

34 32 3

Table 2

Perceived understanding and ease of use in RCT I

What was measured (agreement with statements, on

an anchored scale of 0e7, where 05strongly disagree, 65agree, and 75strongly agree)

% Replying ‘‘agree’’ or ‘‘strongly agree’’d%

of participants answering 6 or 7, combined (95% CI)

Without SoF

table (n525)

With SoF

table (both format

versions, n547)

P-Value: Pearson’s

chi-square

Review authors have indicated what they believe are the most important outcomes 80 (64e96) 81 (70e92) 0.93

It was easy to find this information (about which outcomes are important) 60 (41e79) 72 (60e85) 0.29

It was easy to find results for important outcomes 40 (21e59) 68 (55e81) 0.021

It was easy to find quality of the evidence for important outcomes 24 (7e41) 43 (28e57) 0.12

It was easy to understand the main findings 56 (37e75) 68 (55e81) 0.31

Main findings are presented in a such a way that they would

be helpful to me in making a decision

52 (32e72) 60 (46e74) 0.54

Overall perceived accessibility: scale 0e7, where 0

is ‘‘very inaccessible’’ and 7 is ‘‘very accessible’’

17 (2e32) 41 (27e56) 0.037

Abbreviations: RCT, randomized controlled trial; CI, confidence interval; SoF, summary of findings.

623S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

Entities who held editorial, administrative, technical, ormarketing positions within the Cochrane Collaboration.Six reported a health professional background, and 13 re-ported a background in research. For information aboutthe participants, see Table 3.

The results are shown in Tables 4 and 5. There werelarge differences in the proportion that answered correctlyquestions about the risk in the control group (44% vs.93%, P 5 0.003) and the risk in the intervention group(11% vs. 87%, P ! 0.001). There were also large differ-ences in actual time spent and the proportion that agreedor strongly agreed; hence, it was easy to find informationabout the quality of evidence for the main outcomes(24% vs. 73%, P 5 0.005, Pearson’s chi-square).

Most participants were positive about including the SoFtables and the format. Eighty-eight percent agreed orstrongly agreed that Cochrane reviews should have SoF ta-bles; 84% agreed with the format proposed, and 77% saidthat the explanations were helpful. Most (67%) preferredplacing SoF tables after the abstract.

Fig. 2. Flow diagram of randomized controlled trial II.

4. Discussion

Results from these evaluations showed that inclusion ofthe final version of an SoF table in a Cochrane reviewhelped readers understand the review correctly and madeit easier to find information about the quality of evidenceand the main outcomes, compared with a review with noSoF table.

This study has some limitations:

� In the first RCT, we tested people’s satisfaction withthe SoF tables rather than actual effect on comprehen-sion. However, we redesigned the protocol for thesecond RCT to measure correct comprehension. Forthis reason, the two RCTs are not identical and cannotbe directly compared.� The RCTs were small.

� Athough the second RCT included health profes-sionals and other users of systematic reviews, thisgroup was not necessarily representative of clinicians.� The formatting of the SoF table we tested may not be

possible to achieve in the Cochrane Library for tech-nical reasons. It is not clear how much of a differencethat will make.

4.1. User satisfaction

Participants for the most part perceived the review withthe SoF table as more accessible. But user testing con-ducted just before the first trial and structured discussionsconducted after this trial revealed that many had actuallymisunderstood the content in the SoF table [24]. User sat-isfaction, though important, because it relates to motiva-tional issues, can be a misleading indicator of the successof a piece of information if used alone. A systematic reviewlooking at the effect of different graph designs, for instance,found several studies where versions that users preferredperformed poorly when the same participants’ decision-making behavior was measured [25].

Table 3

Information about participants, RCT II

Total n 5 33 (6 health professionals, 13 researchers) Never (before now)

Less than

once per year

1e4 Times

per year

5e10 Times

per year

More than 10

times per year

How often do you read reports of randomized trials? 3 4 5 4 16

How often do you read systematic reviews? 3 1 8 5 15

How often do you use the Cochrane Library? 2 0 1 1 28

Very difficult Difficult Easy Very easy

How easy is it for you to understand reports of randomized trials? 1 17 10 1

How easy is it for you to understand reports of randomized trials? 1 18 11 1

624 S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

4.2. Time spent

Lack of time is an often-cited barrier to uptake of evi-dence by clinicians. A literature review from 2007 foundstudies reporting varying amounts of time needed to searchfor information, ranging from 53 minutes to medians of14.4 and 15 minutes [26]. One study found that primarycare physicians spent on average less than 2 minutes pursu-ing an answer to a clinical question [27]. If this is the case,then the findings from our study may be important; timespent to find key results dropped from an average of 4and 2.8 minutes to 1.5 and 1.3 minutes, respectively. Thiscould affect clinicians’ ability (or willingness) to accessthe evidence within time frames that are realistic in clinicalcontexts.

4.3. Comprehension issuesdpoor skills or poorinformation design?

Health professionals may be experts in their medicalfield, but many are nonexperts when it comes to biostatis-tics. Studies have consistently shown for more than 25years that health care providers display poor health numer-acy skills, such as basic computation, estimation, and statis-tical literacy [3,28e33]. High level of education is nota vaccination against low numeracy skills [34]. However,a health numeracy model by Ancker and Kaufman, basedon a theory of distributed cognition, emphasizes the inter-dependence of the skills of the care provider with the char-acteristics of the information [35]. An appropriately

Table 4

Correct understanding, RCT II

Questions asked

% Correct answ

Without SoF t

able (n518)

What is the risk of symptomless DVT after a long

flight for people at low risk who do not

wear compression stockings?

44 (21e67)

What would be the risk if they wore stockings? 11 (0e26)

How much confidence do the review authors have

in the estimated effect of stockings

on the risk of symptomless DVT?

67 (45e88)

What are the most important outcomes? 33 (9e57)dan

all 6 correctl

designed piece of information can compensate for lack ofskills in many ways (e.g., by easing cognitive load, makingcomputation simpler, filling in background knowledge theuser does not have, or rendering this background knowl-edge unnecessary).

This was illustrated clearly through both the develop-mental and evaluation phases of this project. First, we ob-served how small design details made a difference toparticipants’ experience of the SoF table in user testingand then demonstrated in an RCT that the inclusion of anSoF table in a systematic review improved the participants’correct understanding of the key results. The participants inthe second RCT were all involved in the work of the Co-chrane Collaboration and could, therefore, be expected topick up information from the original Cochrane reviewquickly. This makes the results of the (albeit small) studyeven more compelling.

4.4. Enabling more expert-like behavior

Medical information technologies should be designed tohelp the no-expert readers behave more like experts byenabling them to quickly find and focus their attention onthe parts of the information important for their tasks. Ourproject has indicated that adding an SoF table to a system-atic review may compensate for nonexpert levels of numer-acy in health professionals, helping them to access the mainresults in the review more quickly and comprehend themcorrectly.

er (95% CI)P-Value:

Pearson’s

chi-square

With SoF

table (n515)

93 (81e100) 0.003

87 (69e100) 0.001

87 (69e100) 0.18

swered

y

53% (28e79%)danswered

all 6 correctly

0.27

Table 5

Time spent finding answers, RCT II

Questions asked

Mean time spent

finding answer without

SoF table (n518) (min)

Mean time

spent finding answer

with SoF table (n515) (min)

P-Value:

Mann-Whitney

U-test

What is the risk of symptomless deep vein thrombosis (DVT)

after a long flight

for people at low risk who do not wear compression stockings?

4 1.5 0.002

What would be the risk if they wore stockings? 2.8 1.3 0.118

How much confidence do the review authors have

in the estimated effect of stockings on the risk

of symptomless DVT?

1.5 2.1 0.47

What are the most important outcomes? 1.9 2.0 0.88

Would you have needed more time to find this information? 64% would not need more time 79% would not need more time

625S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

5. Conclusion

The final version of the SoF table improved the under-standing and rapid retrieval of the key findings of the re-view compared with reviews with no SoF table. However,the trial we conducted was small, and generalizability ofthe results is uncertain. The Cochrane Handbook now rec-ommends that review authors include an SoF table in theirreview [36]. These SoF tables are a work in progress andwill continue to be developed and improved on. Work iscurrently in progress on producing and evaluating versionsof SoF tables for use in summaries targeted at consumersand policy makers. Future work includes development ofSoF tables for diagnostic accuracy reviews.

Acknowledgments

The authors thank Jan Ødegaard-Jensen for his greathelp with the statistical analysis and Arild Bjørndal for helpwith the manuscript.

Supplementary material

Supplementary material can be found, in the onlineversion, at doi: 10.1016/j.jclinepi.2009.12.014.

References

[1] Haynes B. Of studies, syntheses, synopses, summaries, and systems:

the ‘‘5S’’ evolution of information services for evidence-based

healthcare decisions. EBM Nursing 2007;10:6e7.

[2] PLoS Medicine Editors. The impact of open access upon public

health. PLoS Med 2006;3:e252.

[3] Novack L, Jotkowitz A, Knyazer B, Novack V. Evidence-based medi-

cine: assessment of knowledge of basic epidemiological and research

methods among medical doctors. Postgrad Med J 2006;82:817e22.

[4] Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More

informative abstracts revisited. Ann Intern Med 1990;113:69e76.

[5] Mulrow CD, Thacker SB, Pugh JA. A proposal for more informative

abstracts of review articles. Ann Intern Med 1988;108:613e5.

[6] Sollaci LB, Pereira MG. The introduction, methods, results, and dis-

cussion (IMRAD) structure: a fifty-year survey. J Med Libr Assoc

2004;92:364e7.

[7] Groves T, Godlee F. Innovations in publishing BMJ research. BMJ

2008;337:a3123.

[8] Delamothe T, Mullner M, Smith R. Pleasing both authors and

readers. Summary of electronic responses. BMJ 1999;319:580.

[9] ACP Journal Club. Clinical synopses. Quality-assessed, clinically

rated original studies and reviews from over 130 clinical journals.

Available at: http://www.acpjc.org. Accessed Oct 29, 2008.

[10] Evidence-based medicine for primary care and internal medicine. BMJ

Publishing Group Ltd; 2008. Available at. http://ebm.bmj.com/. Ac-

cessed Oct 29, 2008.

[11] Evidence-based mental health. BMJ Accessed Oct 29, 2008. Publish-

ing Group Ltd; 2008. Available at: http://ebmh.bmj.com/.

[12] Evidence-based nursing. BMJ Publishing Group Ltd; 2008. Available

at: http://ebn.bmj.com/ . Accessed Oct 29, 2008.

[13] Clinical evidence. BMJ Publishing Group Ltd; 2008. Available at: http://

clinicalevidence.bmj.com/ceweb/index.jsp . Accessed Oct 29, 2008.

[14] UpToDate. Available at: http://www.uptodate.com. Accessed Oct 29,

2008.

[15] O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H,

Rovner D, Holmes-Rovner M, et al. Decision aids for people facing

health treatment or screening decisions. Cochrane Database Syst Rev

2003;2:CD001431.

[16] Edwards AG, Evans R, Dundon J, Haigh S, Hood K, Elwyn GJ. Per-

sonalized risk communication for informed decision making about

taking screening tests. Cochrane Database Syst Rev 2006;4:

CD001865.

[17] Trevena LJ, Davey HM, Barratt A, Butow P, Caldwell P. A systematic

review on communicating with patients about evidence. J Eval Clin

Pract 2006;12:13e23.

[18] Lipkus IM. Numeric, verbal, and visual formats of conveying health

risks: suggested best practices and future recommendations. Med

Decis Making 2007;27:696e713.

[19] Barry HC, Ebell MH, Shaughnessy AF, Slawson DC, Nietzke F. Fam-

ily physicians’ use of medical abstracts to guide decision making:

style or substance? J Am Board Fam Pract 2001;14:437e42.

[20] Hartley J. Current findings from research on structured abstracts.

[literature review]. J Med Libr Assoc 2004;92:368e71.

[21] The Cochrane Collaboration. Available at: http://www.cochrane.org.

Accessed Oct 29, 2008

[22] Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y,

Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality

of evidence and strength of recommendations. BMJ 2008;336:924e6.

[23] GRADE Working Group. Grading quality of evidence and strength of

recommendations. BMJ 2004;328:1490.

[24] Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing

and stakeholder feedback contributed to the development of under-

standable and useful summary of findings tables for Cochrane

Reviews. J Clin Epidemiol 2010;63:607e19 [in this issue].

[25] Ancker JS, Senathirajah Y, Kukafka R, Starren JB. User testing and

stakeholder feedback contributed to the development of understandable

and useful summary of findings tables for Cochrane Reviews. J Am Med

Inform Assoc 2006;13:608e18.

626 S.E. Rosenbaum et al. / Journal of Clinical Epidemiology 63 (2010) 620e626

[26] Davies K. The information-seeking behaviour of doctors: a review of

the evidence. [review]. Health Info Libr J 2007;24:78e94.

[27] Ely JW, Osheroff JA, Ebell MH, Bergus GR, Levy BT,

Chambliss ML, et al. Analysis of questions asked by family doctors

regarding patient care. BMJ 1999;319:358e61.

[28] Estrada C, Barnes V, Collins C, Byrd JC. Health literacy and numer-

acy. JAMA 1999;282:527.

[29] Rao G. Physician numeracy: essential skills for practicing

evidence-based medicine. Fam Med 2008;40:354e8.

[30] Sheridan SL, Pignone M. Numeracy and the medical student’s ability

to interpret data. Eff Clin Pract 2002;5:35e40.

[31] Godwin M, Seguin R. Critical appraisal skills of family physicians in

Ontario, Canada. BMC Med Educ 2003;3:10.

[32] Young JM, Glasziou P, Ward JE. General practitioners’self ratings of skills

in evidence based medicine: validation study. BMJ 2002;324:950e1.

[33] Gigerenzer G, Gaissmaier EK-M, Schwartz LM, Woloshin S. Help-

ing doctors and patients make sense of health statistics. Assoc Psy-

chol Sci 2008;8:53e96.

[34] Lipkus IM, Samsa G, RimerBK.Generalperformanceon anumeracy scale

among highly educated samples. Med Decis Making 2001;21:37e44.

[35] Ancker JS, Kaufman D. Rethinking health numeracy: a multidisci-

plinary literature review. J Am Med Inform Assoc 2007;14:713e21.

[36] Higgins JPT, Green S, editors. Cochrane Handbook for Systematic

Reviews of Interventions. Version 5.0.2 [updated September 2009].

The Cochrane Collaboration; 2009. Available at: www.cochrane-

handbook.org.