8/12/2019 Methodology and Reports of Systematic Reviews and Meta-Analyses

1/3

Methodology and Reports of SystematicReviews and Meta-analysesA Comparison of Cochrane ReviewsWith Articles Published in Paper-Based JournalsAlejandro R. Jadad, MD, DPhil; Deborah J. Cook, MD, MSc; Alison Jones, BA; Terry P. Klassen, MD, MSc;

Peter Tugwell, MD, MSc; Michael Moher, MB; David Moher, MSc

Context.Review articles are important sources of information to help guidedecisions by clinicians, patients, and other decisionmakers. Ideally,reviews shouldinclude strategies to minimize bias and to maximize precision and be reported soexplicitly that any interested reader would be able to replicate them.

Objective.To compare the methodological and reporting aspects of system-atic reviews and meta-analyses published by the Cochrane Collaboration withthose published in paper-based journals indexed in MEDLINE.

Data Sources.The Cochrane Library, issue 2 of 1995, and a search ofMEDLINE restricted to 1995.

Study Selection.All 36 completed reviews published in the Cochrane Data-base of Systematic Reviews and a randomly selected sample of 39 meta-analysesor systematic reviews published in journals indexed by MEDLINE in 1995.

DataExtraction.Number of authors,trials, andpatients; trial sources;inclusionand exclusion criteria; language restrictions; primary outcome; trial quality assess-ment; heterogeneity testing;and effect estimates. Updatingby 1997wasevaluated.

Results.Reviews found in MEDLINE included more authors (median, 3 vs 2;P.001), moretrials (median,13.5vs 5; P.001), andmore patients(median, 1280vs 528; P.001) than Cochrane reviews. More Cochrane reviews, however, in-cluded a descriptionof the inclusion and exclusion criteria (35/36 vs 18/39; P.001)and assessed trial quality (36/36 vs 12/39; P.001). No Cochranereviews hadlan-guage restrictions (0/36 vs 7/39;P.01). There were no differences in sources oftrials, heterogeneity testing, or description of effect estimates. By June 1997, 18 of36 Cochrane reviews had been updated vs 1 of 39 reviews listed in MEDLINE.

Conclusions.Cochrane reviews appear to have greater methodological rigorand are more frequently updated than systematic reviews or meta-analyses pub-lished in paper-based journals.

JAMA.1998;280:278-280

THECOCHRANECollaboration(CC)isan international organization thataimstohelppeoplemake well-informeddecisionsabouthealthcareby preparing, maintain-

ing,andpromotingtheaccessibilityofsys-tematic reviews on the effects of healthcare interventions.1,2 The reviews pro-duced withinthe CC arepreparedfollow-ingstandardizedinstructionspublishedinthe Cochrane Handbook, which is widelyavailabletoreviewersinpaper,CD-ROM,or Internet format. These reviews are in-cluded in theCochrane Database of Sys-tematic Reviews, a quarterly electronic

publication.1,2 This electronic format al-lows reviewers to update or modify theCochrane reviews inresponseto newevi-dence or comments and criticisms fromreaders. To safeguard the rigor and rel-evance of the Cochrane reviews, the CChas several levels of peer review, includ-ingassessmentofprotocolsbyeditorsandreviewers and evaluation of the reviewsby methodology and content experts andby potential users.1,2

There are major differences between

the peer review process followed bymost paper-based journals and the sys-tem used by the CC. Few paper-based

journalsprovide authorswith explicitin-structions to report systematic reviewsand meta-analyses; their editors andpeer reviewers only provide input afterthe reviews have been completed. Oncethe reviews are published there is lim-ited opportunity for corrections of omis-

sions or mistakes and few incentives toproduce and publish updated versions.

It has been suggested recently thatCochrane reviews are less prone to biasthansystematicreviewsormeta-analysespublished in paper-based journals.3 Todate, however, we have been unable toidentify other studies assessing the im-pact of the CC on the methodology andreports of systematic reviews and meta-analyses.

Againstthisbackground,wedesigneda study to (1) compare the methodologi-cal and reporting aspects of Cochranereviews with those published in paper-

based journals and (2) evaluate the fre-quency with which systematic reviewsandmeta-analyses,published by theCCor in paper-based journals,are updated.

METHODS

Reviews were included in the study ifthey included only randomized con-trolled trials, and if they were publishedin the Cochrane Database of SystematicReviews or in paper-based journals in-dexed by MEDLINE. For the latter,they had to be coded by indexers asmeta-analyses, or described by the au-thors as systematic reviews, meta-analyses, integrative research reviews,overviews, quantitative syntheses, andpooling or combining studies.

For the purposes of this study, all 36Cochrane reviews published in the sec-ondissueof1995oftheCochrane Library(Update Software, Oxford, England)wereincluded.They werecomparedwith39 randomly selected paper-based sys-tematic reviews and meta-analyses,published in 32 different journals, identi-fied using a refined search strategy ofMEDLINE restricted to 1995(Table). To

From the Health Information Research Unit (Dr Ja-dad), Department of ClinicalEpidemiology and Biosta-tistics (Drs Jadad and Cook), and the Department ofMedicine (Dr Cook), McMaster University, Hamilton,Ontario; Thomas C. Chalmers Center for SystematicReviews, Childrens Hospital of Eastern Ontario Re-

search Institute, Ottawa, Ontario (Ms Jones, Drs Klas-senand Tugwell,and Mr D. Moher); theDepartments ofPediatrics (Mr D. Moher) andMedicine (DrTugwellandMr D. Moher), University of Ottawa, Ottawa, Ontario;and Department of Primary Health Care, University ofOxford, Oxford, England (Dr M. Moher).

Presented at the Third International Congress onPeer Review in Biomedical Publication, Prague, CzechRepublic, September 20, 1997.

Theviews expressedin the article do notnecessarilyrepresent the views of the UK National Health Service.

Reprints: Alejandro R. Jadad, MD, DPhil, HealthInformation Research Unit, Department of Clinical Epi-demiology and Biostatistics, McMaster University,1200 Main St W, Hamilton, Ontario L8N 3Z5, Canada(e-mail: jadada@fhs.mcmaster.ca).

278 JAMA, July 15, 1998Vol 280, No. 3 Comparison of Cohrane Reviews With Articles in JournalsJadad et al

1998 American Medical Association. All rights reserved.

wnloaded From: http://jama.jamanetwork.com/ on 01/31/2014

8/12/2019 Methodology and Reports of Systematic Reviews and Meta-Analyses

2/3

8/12/2019 Methodology and Reports of Systematic Reviews and Meta-Analyses

3/3

tem is expected to further improve thevalidity and relevance of the reviews.

ThisstudywasfundedbytheUKNationalHealthService Research and Development Programmegrant93/52/3.DrJadadwassupportedbyaNationalHealth Research Scholar Award from HealthCanada, Ottawa, Ontario, and Dr Cook is a careerscientist supported by the Ontario Ministry ofHealth, Toronto, Ontario.

A list of the articles included in this review is

available upon request.

References1. Bero L, Rennie D. The Cochrane Collaboration:preparing,maintaining,anddisseminatingreviewsoftheeffectsof healthcare.JAMA. 1995;274:1935-1938.2. Jadad AR, Haynes RB. The Cochrane Collabo-ration: advances and challenges in improving evi-dence-based decision making. Med Decis Making.1998;279:611-614.3. Egger M, Davey Smith G, Schneider M, MinderC. Bias in meta-analysis detected by a simple,graphical test.BMJ.1997;315:629-634.4. Jadad AR, McQuay HJ. Meta-analyses to evalu-

ate analgesic interventions: a systematic qualita-

tivereviewof theirmethodology. J ClinEpidemiol.1996;49:235-243.5. Assendelft WJJ, Koes BW, Knipschild PG,BouterLM. Therelationshipbetweenmethodologi-cal quality and conclusions in reviews of spinal ma-nipulation.JAMA.1995;274:1942-1948.6. Sacks HS, Reitman D, Pagano D, Kupelnick B.Meta-analysis: an update. Mt Sinai J Med.1996;3-4:216-224.7. Berlin JA, for the University of PennsylvaniaMeta-analysis BlindingStudy Group. Doesblindingof readers affect the results of meta-analyses? Lan-

cet.1997;350:185-186.

Discussion Sections in Reportsof Controlled Trials Publishedin General Medical JournalsIslands in Search of Continents?

Michael Clarke, DPhil; Iain Chalmers, MSc

SEVERALMAJOR health care journalshave already adopted the ConsolidatedStandards of Reporting Trials (CON-

SORT)recommendationstomakeiteasierfor readers to assess the quality of con-trolledtrials.1 Thisisthefirstjointattemptby biomedical journals to improve the

quality of reports of controlled trials, atopic of research for 4 decades.2 No othercategoryofbiomedicalreporthasreceived

suchsustained attention, and thisreflectsthe practical importance of controlled tri-als in guiding decisions in health care.

Previous research has highlighted de-ficiencies in descriptions of the materialsand methods used and the analysis andpresentation of results.2 Like most simi-lararticlesbeforeit,theCONSORTstate-ment concentrateson these 2 elements ofreports of controlled trials. By contrast,the quality of introduction and discussionsections in trial reports has received little

systematic scrutiny. The typical discus-sion section usually addresses a numberof dimensions, but, crucially, it is in thissection that readers will look for an an-swer to Bradford Hills bottom linequestion for any research article: Whatdoes it mean, anyway?3 This was recog-nizedin the CONSORT statement,whichincluded the recommendation that trial-ists should state general interpretationof the data in light of the totality (our em-phasis) of the available evidence.1

Other research has illustrated how se-lectivecitationofpreviousresearchinthediscussion sections of research articlescan be biased. Studies that have yieldedrelativelydramaticresultsare morelikelyto be cited in reports of subsequent simi-lar studies than previous studies yieldingunremarkable pointestimates of effects.4

In addition, authors from a particularcountry or specialty have been shown toselectively cite material generated fromwithin that country5,6 or specialty.7,8

Ideally,thediscussion sectionof the re-port of a new trial should involve the pre-sentation of an up-to-date systematic re-view,9 as was done, for example, in the1986 article on the First InternationalStudy of Infarct Survival (ISIS-1).10 Toshowhowcloselyreportsofcontrolledtri-als reflect this ideal more than a decadeafter ISIS-1, we assessed the discussionsections of allreportsof randomized trialspublished in May 1997 in 5 general medi-caljournals. This month waschosen to al-low as up-to-date an assessment as pos-sible and without prior knowledge ofthearticles to be published. We concen-trated on howwellthe discussion sections

From the Clinical Trial Service Unit, Oxford University(Dr Clarke), and the United Kingdom Cochrane Centre(Drs Clarke and Chalmers), Oxford, England.

Presented at the Third International Congress onPeer Review in Biomedical Publication, Prague, CzechRepublic, September 20, 1997.

Reprints: Michael Clarke, DPhil, ICRF Clinical TrialService Unit and Epidemiological Studies Unit, Rad-cliffe Infirmary, Oxford OX2 6HE, England (e-mail:mike.clarke@ctsu.ox.ac.uk).

Context.Several journals have adopted the Consolidated Standards ofReporting Trials (CONSORT) recommendations to make assessment of the qual-ity of randomized controlled trials (RCTs) easier. One of these recommendationsis that the trials results be discussed in light of the totality of the available evidence.

Objective.To assess the extent to which reports of RCTs published in 5 gen-eral medical journals have discussed new results in light of all available evidence.

Design.Assessment of the discussion sections in all 26 reports of RCTs pub-lished during May 1997 inAnnals of Internal Medicine,BMJ,JAMA,The Lancet,andThe New England Journal of Medicine.

Main Outcome Measure.The inclusion or mention of a systematic review inthe discussion section of each article.

Results.In only 2 articles were the RCTs results discussed in the context ofan updated systematic review of earlier trials. In a further4 articles, references weremade to relevant systematic reviews, but no attempts were made to integrate theresults of the new trials in updated versions of these reviews. One article was prob-ably the first published trial to address the question studied. The remaining 19 ar-ticles included no evidence that any systematic attempt had been made to set thereported trials results in the context of previous trials.

Conclusion.There is little evidence that journals have adequately imple-mented the CONSORT recommendation that results of an RCT be discussed inlight of the totality of the available evidence.

JAMA.1998;280:280-282

280 JAMA, July 15, 1998Vol 280, No. 3 Discussion Sections in Controlled Trial ReportsClarke & Chalmers

1998 American Medical Association. All rights reserved.

wnloaded From: http://jama.jamanetwork.com/ on 01/31/2014