Peer review: benefits, perceptions and alternatives

PRC Summary Papers 4

Peer review: benefits, perceptions and alternatives

Mark WareMark Ware Consulting

© 2008 Publishing Research Consortium

Published byPublishing Research Consortiumc/o The Publishers Association29B Montague StreetLondon WC1B 5BW

Printed in Great Britain

The Publishing Research Consortium (PRC) is a group representing publishers and societies supporting global research into scholarly communication, with the aim to provide unbiased data and objective analysis. Our objective is to support work that is scientific and pro-scholarship. Overall, we aim to promote an understanding of the role of publishing and its impact on research and teaching.

Outputs from work supported by the PRC are available from the website:www.publishingresearch.org.uk. Hard copies are available from The Publishers Association.

The founding partners are The Publishers Association, the Association of Learned and Professional Society Publishers, and the International Association of Scientific, Technical Medical Publishers. Corresponding partners include The Association of American University Presses and the Professional/Scholarly Publishing Division of the Association of American Publishers.

This summary report was commissioned by the Publishing Research Consortium (www.publishingresearch.org.uk) from Mark Ware Consulting, a publishing consultancy (www.markwareconsulting.com). Mark Ware Consulting was responsible for designing and managing the survey; the online hosting and data analysis was provided by Mike Monkman Media (http://mikemonkman.com/).

Contents

4 Executiveoverview

5 Introduction

6 Whatispeerreview?6 Typesofpeerreview7 Peerreviewdurations9 Thereviewer’sperspective10 Roleoftheeditor

12 Thebenefitsofpeerreview12 Improvementsinquality13 Peerreviewasafilter14 Viewsofsurveyrespondents

16 Critiquesofpeerreview17 Viewsofsurveyrespondents

18 Alternativeapproaches

18 Differenttypesofpeerreview19 Post-publicationreview19 Openpeerreview20 Reviewingauthors’data

20 Conclusions

20 Acknowledgements

©2008 Publishing Research Consortium

4


Peerreviewistheprocessofsubjectinganauthor’sscholarlymanuscripttothescrutinyofotherswhoareexpertsinthesamefield,priortopublicationinajour-nal.Thissummaryreportpresentssomefindingsfromanewinternationalsurveyofacademics,setwithinadescriptionofhowpeerreviewoperates,itsbenefits,somecritiquesofpeerreviewandthedevelopmentofalternativeapproaches.

Peerreviewiswidelysupportedbyacademics,whooverwhelmingly(93%)disagreedinoursurveythatpeerreviewisunnecessary.Thelargemajority(85%)agreedthatpeerreviewgreatlyhelpsscientificcommunicationandbelieved(83%)thatwithoutpeerreviewtherewouldbenocontrol.

Peerreviewimprovesthequalityofthepublishedpaper.Researchersover-whelmingly(90%)saidthatthemainareaofeffectivenessofpeerreviewwasinimprovingthequalityofthepublishedpaper,andasimilarpercentagesaidithadimprovedtheirownlastpublishedpaper,includingidentifyingscientificerrorsandmissingandinaccuratereferences.

Thereisadesireforimprovement.Whilethemajority(64%)ofacademicsdeclaredthemselvessatisfiedwiththecurrentsystemofpeerreviewusedbyjour-nals(andjust12%dissatisfied),theyweredividedonwhetherthecurrentsystemisthebestthatcanbeachieved.Therewasevidencethatpeerreviewistooslow(38%weredissatisfiedwithpeerreviewtimes)andthatreviewersareoverloaded(seeitem11below).

Double-blindreviewwaspreferred.Althoughthenormalexperienceofresearchersinmostfieldswasofsingle-blindreview,whenaskedwhichwastheirpreferredoption,therewasapreferencefordouble-blindreview,with56%select-ingthis,followedby25%forsingle-blind,13%foropenand5%forpost-publica-tionreview.Openpeerreviewwasanactivediscouragementformanyreviewers,with49%sayingthatdisclosingtheirnametotheauthorwouldmakethemlesslikelytoreview.

Double-blindreviewwasseenasthemosteffective.Double-blindreviewhadthemostrespondents(71%)whoperceivedittobeeffective,followed(indecliningorder)bysingle-blind(52%),post-publication(37%)andopenpeerreview(27%).

Double-blindreviewfacessomefundamentalobjections.Double-blindreviewwasprimarilysupportedbecauseofitsperceivedobjectivityandfairness.Manyrespondents,includingsomeofthosesupportingdouble-blindreview,didhow-everpointoutthatthereweregreatdifficultiesinoperatingitinpracticebecauseitwasfrequentlytooeasytoidentifyauthorsfromtheirreferences,typeofworkorotherinternalclues.

Post-publicationreviewwasseenasausefulsupplementtoformalpeerreview,ratherthanareplacementforit.Interestingly,thiswasdespiteaclearviewthatittendstoencourageinstantreactionsanddiscouragethoughtfulreview.

Limitedsupportforpaymentforreviewers.Respondentsweredividedonwhetherreviewersshouldbepaid,with35%infavourofand40%againstpay-ment.Amajority,however,supportedthepropositionthatpaymentwouldmakethecostofpublishingtooexpensive(52%for,18%against)andthelargemajorityofreviewers(91%)saidthattheyreviewedtoplaytheirpartasamemberoftheacademiccommunity.

Executive Overview

1

2

3

4

5

6

7

8

9

5


Mixedsupportforreviewofauthors’data.Amajorityofreviewers(63%)andeditors(68%)saidthatitisdesirableinprincipletoreviewauthors’data.Perhapssurprisingly,giventhatmanyreviewersreportbeingoverloaded(seebelow),amajorityofreviewers(albeitasmallone,51%)saidthattheywouldbepreparedtoreviewauthors’datathemselves,comparedtoonly19%whodisagreed.

Some90%ofauthorsinthesurveywerealsoreviewers.Theyreportedreview-inganaverageof8papersinthelast12months.Thelargemajorityofreviews(79%)wascarriedoutbyacoreofactivereviewers,whocompletedanaverageof14reviewsperyear,nearlytwicetheoverallfigure.Thisgroupreporteditwasoverloaded–doing14reviewsperyearcomparedtotheirpreferredmaximumof13–suggestingthereisaproblemwithreviewerworkloads.

Introduction

Thisreporttakesalookatpeerreview:whatitis,andhowitworksinpractice;thebenefitsofpeerreview;somecritiques;andsomealternativeapproaches.Itislargelybasedonanewinternationalsurveyof3040academics,lookingattheirbehaviourandattitudesandperceptionsofpeerreview.Thissummaryreportcontainsonlyasmallfractionofthedataavailableinthefullreport,1whichinterestedreaderscanfindonthePublishingResearchConsortiumwebsite.

10

11

1Ware, M. and Monkman, M. (2008) Peer review in scholarly journals: perspective of the scholarly community – an international study. Publishing Research Consortium. Available at www.publishingresearch.org.uk

�


What is peer review?

Peerreview,knownasrefereeinginsomeacademicfields,is(toquotetheun-peer-reviewedWikipedia)aprocessofsubjectinganauthor’sscholarlywork,researchorideastothescrutinyofotherswhoareexpertsinthesamefield.Inthisreportwewillconsideronlythepeerreviewofmanuscriptssubmittedtoacademicjournals(theothermainuseofpeerreviewisfortheawardofresearchgrants).

Editorialpeerreviewissaidtohavebegunintheearly18thcentury;forexample,theprefacetothefirstvolumeoftheRoyalSocietyofEdinburgh’sMedical Essays and Observations,publishedin1731,stated:‘Memoirssentbycorrespondencearedistrib-utedaccordingtothesubjectmattertothosememberswhoaremostversedinthesematters.Thereportoftheiridentityisnotknowntotheauthor.’2Duringthe19thandearly20thcentury,peerreviewdevelopedinafairlydisorganizedwayandmanyprominentjournaleditorsactedmorelikenewspapereditors,withlittleinterestinformalpeerreview.PeerreviewinthesystematizedandinstitutionalizedformweknowtodayhasdevelopedlargelysincetheSecondWorldWar,atleastpartlyasaresponsetothelargeincreaseinscientificresearchinthisperiod.

Injournalspeerreview,theauthor’smanuscriptisusuallysubjectedtosomeinitialcheckstoassessitssuitabilityforreview(forinstance,incompletemanuscriptsorworkthatwaspatentlypseudosciencewouldbedeclinedwithoutreview),afterwhichasmallnumberofreviewersareselected.Thetaskexpectedofthereviewersvariessomewhatfromjournaltojournal,butinessenceitisusuallytoassistthejournal’seditor(whomakesthefinaldecision)ondecidingwhetherornottoacceptthemanuscriptforpublication.Thereviewerwillcommentonthequalityoftheworkdone(forinstance,wastheexperimentaldesignappropriatetothequestionbeingstudied?)aswellasonitsoriginality(whatdoesitaddtowhatweknowalready?)anditsimportance(doesitmatter?).

Typesofpeerreview

Therearetwoapproachestopeerreviewincommonuseatpresent.Thenorminmostacademicdisciplines,knownassingle-blindreview,isfortheauthor’sidentitytobeknowntothereviewers,butforthereviewers’identitytobehiddenfromtheauthor.(ThisisthemethoddescribedabovebytheRoyalSocietyofEdinburghin1731.)Themainargumentfor‘blinding’thereviewers’identityisthatitallowsthemtocommentfreelywithoutfearofrepercussions.Conversely,single-blindreviewhasbeencriticisedforallowingallkindsofbiasandotherkindsofirresponsibilityonthepartofreviewerstoflourishbehindtheveilofsecrecy.(Weshalldiscussthecriticismsofpeerreviewinmoredetailbelow.)

Themainalternativeisknownasdouble-blindreview:inthisapproachtheidentitiesoftheauthorandreviewersarehiddenfromeachother.Becausethereviewerdoesnotknowtheauthorortheirinstitution,itisargued,theywillfocusonthecontentofthemanuscriptitself,unaffectedbyconsciousorunconsciousbias.

Anewerapproachtodealingwiththecriticismsofsingle-blindreviewisopenpeerreview:inthismodel,theauthor’sandreviewers’identitiesareknowntoeachother,andthereviewers’namesand(optionally)theirreportsarepublishedalongsidethepaper.Advocatesofopenreviewseeitasmuchfairerbecause,theyargue,somebodymakinganimportantjudgementontheworkofothersshouldnotdosoinsecret.Itisalsoarguedthatreviewerswillproducebetterworkandavoidoffhand,carelessorrudecommentswhentheiridentityisknown.

Morerecently,electronicpublishingtechnologyhasallowedavariantofopenreviewtobedeveloped,inwhichallreaders,notjustthereviewersselectedbytheeditor,areable

2From Rennie, D. (2003) Editorial peer review: its development and rationale. In F. Godlee, T. Jefferson (eds). Peer Review in Health Sciences. Second Edition. pp. 1-13. BMJ Books, London.

�


toreviewandcommentonthepaperandeventorateitonanumericalscalefollowingpublication.Thispost-publicationreviewcouldoccurwithorwithoutconventionalpre-publicationpeerreview.Thebenefitsareseentobethatittakesaccountofcom-mentsfromawiderrangeofpeople(‘thewisdomofcrowds’)andmakesthereviewamorelivingprocess.

Inoursurvey,wefoundthattheconventionalsingle-blindpeerreviewsystemwastheonemostcommonlyexperiencedbyauthors,with85%sayingtheyhadexperienceofitcomparedto45%fordouble-blind,23%foropenandjust8%forpost-publicationpeerreview(Figure1).Thisdoesvarybyacademicdiscipline:single-blindreviewwasthenorminlifesciences,physicalscienceandengineering,whiledouble-blindreviewwasmuchmorecommonforauthorsinhumanitiesandsocialsciences,andclinicalmedicalandnursingauthorshadexperienceofbothsystems.

Peerreviewdurations

Thepeerreviewprocessinevitablytakestime.Thesurveylookedatthisfromtheperspectiveofauthors,reviewersandeditors.

Authorsreportedthatthepeerreviewprocesstookanaverageof80days,withthelongesttimesinhumanitiesandsocialsciences.Theywereevenlysplitonwhetherornotthelengthoftimefromsubmissiontodecisionwassatisfactory,anditwasclearthattheauthorsexperiencingthelongestdelaysweretheleastsatisfied.Forreviewtimesof30daysorless,abouttwo-thirdsofrespondentsweresatisfiedwiththetime;thisdropssharplyat3–6monthsto19%,andto9%forreviewtimesinexcessof6months.

Editorsreportedaveragesubmission-to-acceptancetimesofroughly130days,splitroughlyequallybetweentheinitialpeerreviewstagetofirstdecision,andsubsequentrevisionstages.Nearlythree-quarters(72%)reportedtimesof6monthsorbelow.Timeswereshortestinmedicalandnursingjournals,andlongestinhumanitiesandsocialsciencesjournals.Mosteditorswerehappywithreviewingtimesontheirjournals,butasubstantialminority(aroundathird)wasunhappy.

Theoverwhelmingmajorityofeditors(98%)gavetheirreviewersadeadlineforresponding,withtheaveragedeadlinebeingabout34days,andwith63%ofeditorsgiving30daysorless.Deadlineswereshorterinmedicalandnursingresearch,andlongestinhumanitiesandsocialsciencesandphysicalsciencesandengineering.

Figure 1 Types of peer review experienced by authors and used by journal editors

Single-blind peer review

Double-blind peer review

Open peer review

Post-publication review

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Experienced by all respondents Used by editors’ journals

72%

85%

22%

45%

3%

23%

1%

8%

8


Free subscription

Acknowledgement in journal

Payment in kind

CME/CPD points

Name published

Report published anon.

Signed report published

Name disclosed to author

-60% -45% -30% -15% 0% 15% 30% 45% 60%

-7% 56%

-10% 44%

-12% 43%

-11% 20%

-38% 19%

-35% 16%

-47% 11%

Less likely More likely

-49% 10%

Figure 3 Factors affecting reviewers’ likelihood to review for a journal

Figure 2 Reasons for reviewing

To play your part as a member of the academic community

To enjoy being able to improve the paper

To enjoy seeing new work ahead of publication

To reciprocate the benefit when others review your papers

To enhance your reputation or further your career

Personal recognition/build a relationship with the editor

To increase the chance of a role in the journal’s editorial team

-50% -25% 0% 25% 50% 75% 100%

Disagree Agree

-3% 91%

-6% 78%

-11% 69%

-14% 67%

-28% 44%

-37% 30%

-50% 20%

9


Thereviewer’sperspective

Researchersreportedreviewingregularlyfor3.5journalsandforafurther4.2journalsoccasionally.Onaverage,reviewerssaidthattheyreviewedabout8papersintheprevious12months.Thisaveragefiguredisguisesthedistributionofreviewsamongreviewers.Weidentifiedthegroupofreviewerswhoreporteddoing6ormorereviewsinthelast12months(‘activereviewers’),andthisgroupmanagednearlytwiceasmanypapersastheaverage.Thismeantthatalthoughactivereviewersmadeupjust44%ofreviewersinoursurvey,theywereresponsiblefor79%ofreviews.

Reviewerssaythattheytookabout24daystocompletetheirlastreview,with85%reportingthattheytook30daysorless.Theyspentamedian5hours(mean9hours)perreview.ActivereviewersandthoseintheEnglish-speakingregionsreportedspendingconsiderablylesstimeperreviewthanlessfrequentreviewersandthosefromAsiaandtheRestofworld.

Weaskedreviewerstostatethemaximumnumberofreviewstheywerepreparedtoundertake.Theaveragefigureforallrespondentswas9reviews.Thiscomparestotheaverageof8reviewscompletedinthelast12months.Overall,therefore,therewouldappeartobeatleastsomeslackinthesystem.Thisapparentpositionofcomfortablecapacitybreaksdown,however,whenthedistributionofreviewsistakenintoaccount.Activereviewers(responsiblefor79%ofallreviews)proposedamaximumof13papers,comparedtotheiraverageof14reviewsdoneinthelast12months,suggestingthereisaproblemofrevieweroverloading.

Why reviewers reviewWewereinterestedtoexplorethereasonswhyreviewersreview,andwhatincentiveswereofferedandwhichwereeffective.

Ingeneral,respondentspreferredtooffermorealtruisticexplanationsforwhytheyreviewed(seeFigure2),withsubstantiallythemostpopularreasonbeing‘playingyourpartasamemberoftheacademiccommunity’.Self-interestedreasonssuchas‘toenhanceyourreputationorfurtheryourcareer’or‘toincreasethechanceofbeingofferedaroleinthejournal’seditorialteam’weremuchlessfrequentlyadvanced.Themostcommonrewardsforreviewingreportedbyeditorswerereviewerreceptionsatconferencesandwaiverofauthorcharges(e.g.publication,page,colour,offprintcharges)(both39%).Monetarypaymentwasrareatonly5%ofeditors,thoughmorecommonthancreditsforcontinuingprofessionaldevelopment(2%).Paymentwasmostcommoninhumanitiesandsocialsciencesjournals(9%).

Fromthereviewers’perspective,theincentivestheysaidweremostlikelytoencouragethemtoactforajournalwere(seeFigure3):

•afreesubscriptiontothejournal(56%saidthiswouldmakethemmorelikelyto reviewforthejournal)•acknowledgementinthejournal(44%)•paymentinkindbythejournal,forexamplewaiverofcolourorotherpublication charges,freeoffprints,etc.(43%).

Payment for reviewersReviewersweredividedonwhethertheyshouldbepaidforeachreviewtheycom-pleted:35%agreedthattheyshould,while40%disagreed.ThosefromtheAnglo-phoneregionswerethemostopposedtopayment,whereasresearchersfromAsiaandfromEuropewereonbalancejustinfavour(44%for,32%against).

Therewaslesssupportfortheideathatpaymentwouldreducetheobjectivityofpeerreview(28%for,43%against)butamajorityforthepropositionthatpaymentwouldmakethecostofpublishingtooexpensive(52%for,18%against).

Forthemostpart,respondents’viewsonthesequestionsappeartobepersonalmatters,independentoftheirfieldofresearch.Asalreadynoted,respondents

10


preferredtogivealtruisticreasonsforreviewing.Womenthroughoutthesurveytendedtoexpressmorealtruisticpositionsthanmen;theirresponseshereareconsistentwiththatposition,withwomenbeinglessinclinedthanmentothinkreviewersshouldbepaid,andmoreinclinedtoseethedownside.

Roleoftheeditor

Thefunctionoftheeditoristoselectthemostappropriatemanuscriptsforpubli-cationintheirjournalandtoensuretheyareeditedandpresentedinthebestwayforthejournal’sreadership.Theirpreciserolevariesconsiderablyfromjournaltojournal;forsomelargerscientificjournalsthereviewersmaybeselectedandmanagedbyaneditorialteamatthepublisher’sofficewiththeeditoronlybecominginvolvedoncethereviewerreportsarereceived(orinsomecases,onlyifthereisadisputebetweenreviewerstobeadjudicated),whileothereditorsaremuchmorehands-on,appointing,selectingandchasingupthereviewersthemselves.

Inthesurvey,wefoundthateditorssaidthatthenumberofpaperstheyhandled(i.e.thenumberonwhichtheymadeaccept/rejectdecisions)wasabout50peryear.Themajority(59%)handled25orfewerpapersbuttherewasasmallgroup(11%)ofmuchbusiereditorshandlingmorethan150.Editorsassignedabout2.3reviewersperpaper.Selectionofthereviewersbytheeditorthemselveswasonlythethirdmostpopularoption(reportedby28%ofeditors),wellbehindselectionbyamemberoftheeditor’steam(73%)andbyamemberofthepublisher’sstaff(43%).

Onlinemanuscriptsubmissionandtrackingsystemswereusedbyaboutthree-quartersofeditors.Theirusewasmorecommoninlifesciences(85%)andmarkedlylesscommoninhumanitiesandsocialsciences(51%).

Editorsreportedthattheaverageacceptanceratefortheirjournalswasabout50%,whichisconsistentwithotherstudies(Figure4).About20%ofsubmittedmanu-scriptsarerejectedpriortoreview(eitherbecauseofpoorquality(13%)orbeingoutofscope(8%))andanother30%arerejectedfollowingreview.Ofthe50%accepted,41%areacceptedsubjecttorevision.Acceptancerateswerelowerinhumanitiesandsocialsciences,andhigherinphysicalsciences/engineeringjournals.

TrishGroves(deputyeditorofthebmj)haswritten3thatanobviouswaytoimproveanyjournal’speerreviewsystemisto‘Tellauthorsandreviewerswhatyouwantfromthem.. . .Givereviewersclearbriefs,includingguidanceonwhattoincludeinthereview’.Inthelightofsuchcommonsenseadvice,itwassomewhatsurprisingtofindthat30%ofeditorsdidnotprovidereviewerswithachecklist.Theuseofchecklistswassomewhatlesscommoninhumanitiesandsocialsciencesjournals(45%notusing).Whereeditorsdidprovidechecklists,themostcommonquestionsinvolvedthestudymethodology(87%ofchecklists),relevance,importanceandpaperlength(Figure5).

Grovesalsowrote4‘Reviewershavealsotoldustheywantfeedbackontheirperfor-mancesothattheycanlearnandimprove.’Thisseemsanothercommonsensepositionbutonly28%ofeditorsinoursurveyreportedthattheygavefeedbacktoreviewersonthequalityoftheirreports.Themostcommonfeedbackgivenwasjustthepublicationoutcome.

3Groves, T. (200�) Quality and value: how can we get the best out of peer review? Nature (Nature Peer Review debate). doi:10.1038/nature04995

4ibid.

11


Figure 5Items used in reviewer checklists provided by editors

Methodology employed

Relevance

Importance

Paper length

Tables

Statistics

Originality

Experimental data

Illustrations

References

Language

Quality

Ethical issues

Other

87%

85%

78%

75%

60%

53%

53%

47%

42%

37%

35%

34%

26%

8%

0% 20% 40% 60% 80% 100%

Reject after review

30%

Reject prior

(poor quality)

13%

Reject prior

(out of scope)

8%

Accept, no revision

8%

Accept, with revision

41%

Figure 4 Ultimate fate of submitted manuscripts

12


The benefits of peer review

Whatarethebenefitsofpeerreview?Inoneview,therearebenefitsforallplayersinthesystem:editorsaresupportedintheirdecisionsbytheviewsofexperts;authorsbenefitfromtheassistanceofferedbyreviewersandfromthestatusconferredonthembypublicationinjournalswithhighpeerreviewstandards;readersbenefitbecauseofthefilteringthatpeerreviewprovidesandbythe‘sealofapproval’thatpeerreviewisthoughttoprovide;andevenreviewers(whodothebulkoftheworkfornodirectrecompense)benefittosomeextent(e.g.inseeingworkpriortopublication).

Lookingbeyondtheinterestsoftheparticularstakeholders,therearethreemainbenefitsadvocatedforpeerreview:

•improvementinthequalityofpublishedpapers;•filteringoftheoutputofpaperstothebenefitofreaders;•a‘sealofapproval’thatthepublishedworkmeetscertainstandards,inparticular forlayreaders.

Let’slookinmoredetailattheseproposedbenefits.

Improvementsinquality

Thereareanumberofwaysinwhichpeerreviewmightimprovethequalityofpublishedpapers,ofwhichthemostimportantare:

•theveryfactofaqualityhurdleorthreshold,whichwillmotivateauthorsto improvethequalityoftheirworkpriortosubmission;•thepeerreviewprocess,inwhichreviewers’commentsandcriticismsareaddressed bytheauthorbyrevisingthemanuscript.Testingofworkthroughthecriticismof peersisinabroadsenseattheheartofthescientificmethod.

Perhapssurprisingly,thereislittlescientificevidencetosupporttheuseofpeerreviewasamechanismtoensurequality(seebelow,underCritiques of peer review).Inoursurvey,however,thelargemajorityofauthors(around90%)wereclearthatpeerreviewhadimprovedtheirownlastpublishedpaperandasimilarproportionagreedwiththemoregeneralstatement‘peerreviewimprovesthequalityofthepublishedpaper’.

Respondentswhosaidthatpeerreviewhadimprovedtheirlastpaperwereaskedwhichaspectsofthepaperhadbeenimproved,andineachcasebyhowmuch(usinga1–5scale).TheresultsareshowninFigure6.

Some64%ofrespondentsreportedthatpeerreviewoftheirlastpublishedpaperhadidentifiedscientificerrors,demonstratingrealvaluebeingaddedbypeerreview,and78%saidithadidentifiedmissingorinaccuratereferences.

‘Madesuggestionsonpresentation’wasthemosthighlyratedaspect;94%ofthosewhosaidtheirpaperhadbeenimprovedreportedimprovementinthisarea,and55%ratedtheimprovementat4or5outof5.Thelanguageorreadabilitywasalsofrequentlycited(86%reportedsomeimprovementinthisarea).

Thosewithgoodaccesstothejournalsliteraturereportedlessimprovementinidentifyingmissingofinaccuratereferencesthanthosewithworseaccess.Thisiswhatwemightexpecttofindandillustratesonewayinwhichrestrictedaccesstoliteraturecanaffectresearchers.

Therewassomewhatlessimprovementreportedregardingtheidentificationofstatisticalerrorsthanforotherbenefits,although51%stillreportedsomeimprove-ment.Isthisbecauseauthorsarelesslikelytomakestatisticalerrorsthanother

13


kinds,orbecausereviewersarelesslikelytospotthem?Giventhatsomestudieshaveshownthatpublishedpapersarerifewithstatisticalerrors(e.g.EmilGarcia-BerthouandCharlesAlcarazfoundstatisticalinconsistenciesin38%ofpapersinNatureand25%intheBMJ5)thelatterseemsamorelikelyexplanation.

Peerreviewasafilter

Therearetwosensesinwhichpeerreviewandthejournalsysteminwhichitisembeddedcanfilterresearchoutputsforthebenefitofreaders.

First,peerreviewcouldbeseentofilteroutbadworkfromtheliterature,byreject-ingitforpublication.‘Badwork’herecouldmeanpoorlyconceivedorexecuted,orofminimaloriginalityorinterest,or‘bad’inthemoralsense,forinstanceinvolv-ingacademicfraudorplagiarism.Workthatdoesgetpublishedinapeer-reviewedjournalisseentohavemetsomequalitythresholdorgaineda‘sealofapproval’.Groupspromotingbetterpublicunderstandingofsciencewilloftenusepeerreviewinthisway;forinstance,theukgroupSenseAboutSciencepromotesunderstandingofpeerreview,whichitcallsthe‘essentialarbiterofscientificquality.’6

Thereare,however,atleasttwoproblemswiththisposition.Becausethepeerreviewstandardsofdifferentjournalsvary,itiswidelybelievedthatalmostanygenuineacademicmanuscript,howeverweak,canfindapeer-reviewedjournaltopublishitiftheauthorispersistentenough.Manuscriptsrejectedbyonejournalareroutinelysubmittedtoanother,probablyonewithalowerrejectionrate.Acceptancebyapeer-reviewedjournaldoesnotsayverymuchaboutthequalityororiginalityofapaperbutitmaystilldistinguishitfrompseudoscienceoregregiouslybadwork,andthisisthewayinwhichgroupslikeSenseAboutSciencebelieveitcanhelpthepublic.Theotherproblemisthatpeerreviewhasbeenshownnottobeparticularlyeffectiveasaqualitycontroltool,oratdetectingerrorsoroutrightfraud.(Theseproblemsarediscussedinmoredetailbelow,seeCritiques of peer review.)

Don’t know/Not applicable 1 (no improvement) 2 3 4 5 (substantial improvement)

Made suggestions

on presentation

The language or readability

Identified missing or

innacurate references

Identified scientific errors

Identified statistical errors

5% 12% 27% 37% 18% 1%

12% 20% 25% 27% 14% 2%

24% 19% 21% 21% 10% 3%

30% 16% 17% 18% 11% 6%

35% 15% 15% 13% 6% 14%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Figure �Improvements made by peer review to authors’ last published paper

5Garcia-Berthou, E. and Alcaraz, C. (2004) Incongruence between test statistics and P values in medical papers. BMC Medical Research Methodology 4: 13.

�See http://www.senseabout-science.org.uk/index.php/site/project/29/

14


Thesecondwayinwhichpeerreviewcanprovideafilterforreadersismuchmoreimportantforworkingacademics:itprovidesthebasisforthestratificationofjournalsbyperceivedquality(wherequalityisfrequentlytakentobeindicatedbytheimpactfactor,ameasureofhowoftenonaveragearticlesinthejournalinquestionarecited).Peerreviewthussupportsthesystemthatroutesthebetterpaperstothebetterjournalsandthisallowsacademicstofocustheirreadingonamanageablenumberofcorejournalsintheirfield.Publishersinparticularseethiskindoffilter-ingasoneofthemajorbenefitsofpeerreviewandthejournalssystem.

Respondentstooursurveyhavealotofconfidenceinthepeerreviewsystemtosupportthesefilteringfunctions(seeFigure7).Aswellasverystronglysupportingthenotionthatpeerreviewimprovesthequalityofpublishedpapers(asdiscussedabove),therewasalsostrongsupportfortheideathatitdeterminestheimportanceofthefindingsandtheoriginalityofthemanuscript.Therewassomewhatlesssupport(thoughstillanetmajority)forbelievingpeerreviewwaseffectiveatdetect-ingplagiarismandacademicfraud.

Viewsofsurveyrespondents

Overall satisfaction with peer reviewThemajority(64%)ofacademicsdeclaredthemselvessatisfiedwiththecurrentsystemofpeerreviewusedbyjournals,withjust12%sayingtheyweredissatisfied(Figure8).Therewasverylittlevariationamongstthesampleinthesefigures;forinstancetherewerenodifferencesbyage,genderorposition(seniority).

Respondents’attitudeswerealsotestedbyaskingfortheirdegreeofagreementordisagreementtowardsanumberofstatementsaboutpeerreview,asshowninFigure9.

Onthepositiveside,thelargemajority(85%)agreedwiththepropositionthatscientificcommunicationisgreatlyhelpedbypeerreview.Therewasasimilarlyhighlevelofsupport(83%)fortheideathatpeerreviewprovidescontrolinscientificcommunication.

Giventhegenerallylowlevelofoveralldissatisfactionwithpeerreview,though,itisperhapssurprisingthatastrongstatementlike‘peerreviewinjournalsneedsa

Figure �Views on the effectiveness of peer review in different areas

Disagree Agree

Improves quality

Determines importance

Determines originality

Picks best mss for journal

Detects plagiarism

Detects fraud

-30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

-3% 90%

-16% 60%

-17% 58%

-22% 49%

-24% 46%

-26% 43%

15


completeoverhaul’didnotreceivemoredisagreement–infactrespondentsweredivided,with35%disagreeingversus32%agreeing.Similarly,respondentsweredividedonwhetherthecurrentpeerreviewsystemisthebestwecanachieve,with32%agreeingversus36%disagreeing.

Therewas,however,virtuallynosupportfortheradicalpropositionthatpeerreviewwascompletelyunnecessary.

Onlyaminorityoverall(19%)agreedthatpeerreviewwasholdingbackscientificcommunication.Thosewithpoor/verypooraccesstothejournalsliteraturetendedtoagreemore(23%)thanthosewithexcellentaccess(16%).

Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied Don’t know

6% 59% 22% 10% 2% 1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Greatly helps scientific comm’n

No control without peer review

Needs complete overhaul

Current system is best achievable

Holds back scientific comm’n

Completely unnecessary

Disagree Agree

-100% -75% -50% -25% 0% 25% 50% 75% 100%

-5% 85%

-10% 83%

-35% 32%

-36% 32%

-63% 19%

-93% 3%

Figure 8 Overall satisfaction with the peer review system used by scholarly journals

Figure 9 Views on peer review

1�


The dissatisfied groupWhilethelargemajorityofrespondentsexpressedthemselvessatisfiedwiththepeerreviewsystemusedbyscholarlyjournals,aminority(12%)saidtheyweredissatisfiedorverydissatisfied.Itisinterestingtoaskwhatwecansayaboutthisgroup.

Intermsofdemographics,therearerelativelyfewdifferencesfromtheaverage.Therewerenosignificantdifferencesbyage,gender,typeoforganizationorposition(seniority).Byregion,theyweremoresomewhatlikelytobeintheAnglophoneregions,andlesslikelytobeinAsiaortheRestofworld.Lookingatfieldofresearch,theyweremostlikelytobeinhumanitiesandsocialsciences,andleastlikelyinphysicalsciences/engineering.

Intermsoftheirownexperienceofpeerreview,thisgroupreportedthatthepeerreviewoftheirlastpublishedpapertooksignificantlylongerthanaverage(about110comparedto80days),andtheyweremorelikelytobedissatisfiedwiththelengthoftimeinvolved.Thedissatisfiedgrouptendedtobesomewhatlesslikelytoreportthatpeerreviewhadimprovedtheirlastpublishedpaper,andlikelytogivelowerscorestotheimprovementstheydidreport.Wecannotfromthedatasayifthereisacausalrelationship;thatis,isthisgroupdissatisfiedwithpeerreviewbecausetheyhaveexperiencedlongertimesandlesspersonalbenefitontheirownpapers,ordoestheirdissatisfactionarisefromothercausesandthenleadthemtogivelesspositivescores?

Intermsofalternativeapproachestopeerreview,thisdissatisfiedgroupwasmorelikelytoagreethatopenandpost-publicationreviewwereeffective.Asasmallminority,however,theydidnotformthemainconstituencyforthesealternativeapproaches.

Critiques of peer review

Peerreviewisnotwithoutitscritics.Perhapsthestrongestcriticismisthatthereisalackofrealevidencethatpeer

reviewactuallyworks:forinstance,a2002studypublishedinthe Journal of the American Medical Association7concludedthat‘Editorialpeerreview,althoughwidelyused,islargelyuntestedanditseffectsareuncertain’.Similarly,theCochraneCollaboration(auk-basedinternationalhealthcareanalysisgroup)firstpublisheditsownreviewin2003,whichconcludedthattherewas‘littleempiricalevidencetosupporttheuseofeditorialpeerreviewasamechanismtoensurequalityofbiomedicalresearch,despiteitswidespreaduseandcosts’.Thelatestupdate(2007)oftheCochranereviewconfirmsthisconclusion8,thoughitisimportanttounder-standthatitissayingthattheevidencetosupportpeerreviewhasnotyetbeenproduced,notthatthereisevidencethatpeerreviewdoesnotwork.

Somehaveshownthatpeerreviewcanbeunreliable.Forinstanceonestudy9showedthatthechancesoftworeviewersagreeingaboutaparticularpaperwereonlyslightlybetterthanchance;inordertoproduceareliableresult,editorswouldneedtousesixreviewersforeachpaper.(Inpractice,theytypicallyusetwoorthree–theaveragereportedinthissurveywas2.3.)

Otherstudieshaveshownthatpeerreviewcanbenotverygoodatdetectingerrors.Godleeandcolleaguesatthebmjtookapaperabouttobepublished,insertedeightdeliberateerrors,andsentthepaperto420potentialreviewers:221(53%)responded.Theaveragenumberoferrorsspottedwastwo,nobodyspotted

�Jefferson, T., Alderson, P., Wager, E. and Davidoff, F. (2002) Effects of Editorial Peer Review: A Systematic Review. Journal of the American Medical Association 28�: 2�84-2�8�.

8Jefferson, T., Rudin, M., Brodney Folse, S. and Davidoff, F. (200�) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Sytematic Reviews 200�, Issue 2. Art. No.: MR00001�. DOI: 10.1002/14�51858.MR00001�.pub3

9Rothwell, P.M. and Martyn, C.N. (2000) Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 123: 19�4-19�9.

1�


morethanfive,and16%didn’tspotany10.Itisalsosaidthatpeerreview,particularlyinitssingle-blindform,offerstoo

muchscopeforbiasonthepartoftherevieweroreditor.Forinstance,paperspublishedinanissueoftheJournal of the American Medical Association11devotedtopeerreviewpresentedevidencefornationalitybias,languagebias,specialtybias,andperhapsevengenderbias,aswellastherecognisedbiastowardthepublicationofpositiveresults.

Oneresponsetotheproblemsofreviewerbiashasbeentomovetodouble-blindratherthansingle-blindreview.However,thesecrecyinvolvedin‘blinding’thereviewer’sidentityhasitselfbeencriticisedontwomaingrounds.Fromapragmaticviewpoint,moststudiesthathaveinvestigatedreviewerblindinghavefailedtomeasureimprovementsinthequalityofthereviewand,conversely,otherstudieshaveshownthatmakingthereviewer’sidentityknowntoauthorshadnoeffectonquality.12Thereisalsoastrongethicalargumentagainstsecrecy,namelythatitisseentobeunfairforsomebodymakinganimportantjudgementontheworkofotherstodosoinsecret.

Anotherargumentagainstdouble-blindingisthatitisverydifficultinpracticetodisguisetheidentityoftheauthorofanacademicmanuscriptfromaskilledreviewer;bydefinitionthereviewerisanexpertinthefieldwhowillfrequentlyknowthepreviousworkofauthorsinthefield.

Otherpragmaticcriticismsofpeerreviewincludethedelayitcausestopublica-tionandtheviewthatitdoesnotscaleefficientlywiththegrowthofscience.Thesurveyshowedsomebasisforeachofthese.Althoughtheaveragedelayreportedbyauthorsforpeerreviewwasonlyabout80days,39%reportedtimesofgreaterthan3months,and10%ofgreaterthan6months.Editorsreportedthattheaveragetimefromsubmissiontoacceptanceontheirjournalswasabout130days,with22%reportingtimesofmorethan6months.Therewasacorrelationbetweenthosereportinglongerreviewtimesandloweroverallsatisfactionwithpeerreview.Thesurveyalsoshowedthatthelargemajorityofreviewswereundertakenbyacoregroupofactivereviewerswhoappeartobeoverloaded.

Viewsofsurveyrespondents

Howdidsurveyrespondentsdealwiththesecriticisms?Inthemostpart,aswehavealreadyseen,respondentshadpositiveviewsaboutpeerreviewanditseffectivenessatimprovingthequalityofpublishedpapers.Theirviewsonalternativesystemsofpeerreview,whichhavebeenproposedatleastinpartasresponsestocriticismsofconventionalpeerreview,areexploredinthenextsection.

11Journal of the American Medical Association (1998) 280: issue 3.

10Godlee, F., Gale, C. R. and Martyn, C. N. (1998) Journal of the American Medical Association 280: 23�-240.

12E.g. Goldbeck-Wood, S. (1999) Evidence on peer review: scientific quality control or smokescreen? British Medical Journal 318: 44-45.

18


Alternative approaches

Differenttypesofpeerreview

Mostrespondentsinmostfieldsexperiencesingle-blindreviewasthenorm.Whenaskedwhichoptionstheythoughtwereeffective,however,respondentsexpressedaclearpreferencefordouble-blindreview,asshowninFigure10.Thelevelofsupportfortheeffectivenessofpost-publicationreviewissurprisinglyhigh.

Respondentsdidnothavepersonalexperienceofalltypesofreview.Thosewithexperienceofdouble-blindreviewweresubstantiallylesslikelytoratesingle-blindreviewaseffectivecomparedtoothers.Similarly,thosewhohadexperienceofopenpeerreviewandpost-publicationreviewasanauthorwereconsiderablymorelikelytoratethemaseffective.Itisnotable,though,thatalthough37%ofrespondentssaidthatpost-publicationreviewwaseffective,only8%hadhadexperienceofitasauthors–thissupportisthereforesomewhathypothetical.

Askedwhichofthefourpeerreviewtypeswastheirpreferredoption,therewasaclearpreferencefordouble-blindreview,with56%selectingthis,followedby25%forsingle-blind,13%foropenand5%forpost-publicationreview.Post-publicationreviewgetsmuchlesssupportherecomparedtotheperceptionsofitseffectiveness:thisisnotinconsistentbecauserespondentsclearlysawitasausefulsupplementtocurrentpeerreviewmethodsratherthanareplacementforthem.

Itwasclearfromtheverbatimcommentsthatthepreferencefordouble-blindreviewwaslargelyaresponsetothepotentialforbiasinsingle-blindreview:thereasonsgivenforthispreferencewereprimarilyitsobjectivityandfairness.

Single-blind

Double-blind

Open

Post-publication

0% 10% 20% 30% 40% 50% 60% 70% 80%

25%

52%

56%

71%

13%

27%

5%

37%

Thought effective Preferred option

Figure 10Types of peer review thought to be effective (multiple responses allowed), and respondents’ preferred choice (single response)

19


Post-publicationreview

Lookinginmoredetailatpost-publicationreview(Figure11),researcherssawitasausefulsupplementtoformalpeerreviewinquitelargenumbers(53%agreedcomparedto23%disagreeingwiththisstatement).Theyseethisusefulnessdespiteaclearperceptionthatittendstoencourageinstantreactionsanddiscouragethoughtfulreview.Thereislesssupportfortheideathatitcouldbealessgoodbutstillacceptablealternative(31%supportedversus43%opposed)andfairlystrongoppositiontotheideathatitcouldbeanequallypowerfulalternativetoformalpeerreview(57%opposedversus19%supported).Therewasevenstrongeroppositiontoreplacingpeerreviewwithpost-publicationratingsorusageorcitationstatisticstoidentifygoodpapers.

Openpeerreview

Supportforopenpeerreviewstartedtogrowduringthemid-1990s.TheBMJwasoneofthefirstmajorjournalstoadoptopenpeerreview,basingitsdecisionpartlyontheethicalcaseagainstsecrecyandpartlyontheevidencementionedabovethatblindingdidnotimprovereviewoutcomes.Openreview,however,remainsfarfrombeingthenorm.Themainargumentagainstitisthatreviewerswillbereluctanttocriticisetheworkofmoreseniorresearchersonwhomtheymaybedependentforcareeradvance-mentorgrantawards.During2006,thejournalNatureconductedatrialofopenpeerreview13;itwasnotasuccess–despiteinterestinthetrial,onlyasmallpropor-tionofauthorschosetoparticipate,andonlyafewcommentswerereceived,manyofwhichwerenotsubstantive.Feedbacksuggested‘thatthereisamarkedreluctanceamongresearcherstoofferopencomments’.

Inthesurvey,thenumbersofrespondentspreferringopenpeerreviewweresmallerthanforsingle-ordouble-blindpeerreview(about13%).Themainreasonsgivenforpreferringitwere:revieweraccountability,leadingtobetterreportsandlesslikelihoodofbias,andtheviewthatopenreviewmadereviewersmorecivil,madetheprocessmoreofadialoguewiththeauthorandgenerallyimprovedauthor/reviewercommunication.

Disagree Agree

Encourages instant reactions

Useful supplement to formal review

Authors less likely to submit

Readers fear offending authors

Would relieve load on reviewers

Acceptable (but weaker) alternative

An equally powerful alternative

-60% -45% -30% -15% 0% 15% 30% 45% 60%

-16% 56%

-23% 53%

-29% 41%

-31% 39%

-32% 38%

-43% 31%

-57% 19%

Figure 11Views on post-publication review

13Nature editors/publishers. (200�) Overview: Nature’s peer review trial. Nature doi:10.1038/nature05535.

20


Proponentsofopenpeerreviewwillalsohavetoovercomethefactthat47%ofreviewerssaidthatpublishingtheirsignedreportwouldmakethemlesslikelytoreviewforajournalandthatasimilarproportion,49%,wouldseedisclosureoftheirnametotheauthorasadisincentive(seeFigure3above).

Reviewingauthors’data

Asscienceutilizesmoreautomatedexperimentalequipmentandotherwisemovestowardsamoredata-centric‘e-science’model,theamountofdatathatsupports(andcouldpotentiallybelinkedto)theaveragescientificpaperincreases.Thequestionarisesastowhetherthisdatashoulditselfbesubjecttopeerreview.Thereareclearlyanumberofpracticalissues:doreviewershavethetimetodothis?Isthedatasufficientlystandardized,anddothesoftwaretoolsexisttohandleit?Areauthorsevenpreparedtosharetheirdatawithreviewers?

Amajorityofreviewers(63%)andeditors(68%)saidthatitwasdesirableinprin-cipletoreviewauthors’data.Perhapssurprisingly,amajorityofreviewers(albeitasmallone,51%)saidthattheywouldbepreparedtoreviewauthors’datathemselves,comparedtoonly19%whodisagreed.Thiswasdespite40%ofreviewers(and45%ofeditors)sayingthatitwasunrealistictoexpectpeerreviewerstoreviewauthors’data.

Conclusions

Thesurveythuspaintsapictureofacademicscommittedtopeerreview,withthevastmajoritybelievingthatithelpsscientificcommunicationandinparticularthatitimprovesthequalityofpublishedpapers.Theyarewillingtoplaytheirpartincarryingoutreview,thoughitisworryingthatthemostproductivereviewersappeartobeoverloaded.Manyoftheminfactsaytheyarewillingtogofurtherthanatpresentandtakeonresponsibilityforreviewingauthors’data.

Withinthispictureofoverallsatisfactionthereare,however,somesizeablepocketsofdiscontent.Thisdiscontentdoesnotalwaystranslateintosupportforalternativemethodsofpeerreview;forexamplesomeofthosemostpositiveaboutthebenefitsofpeerreviewwerealsothemostsupportiveofpost-publicationreview.

Acknowledgements

ThanksareduetoMikeMonkman(MikeMonkmanMedia,http://mikemonkman.com/)forhelpindesigningthesurvey,forprovidingtheonlinehostingfacilitiesandfortheanalysisofthesurveydata.ThanksarealsoduetoMayurAminandAdrianMulliganforhelpfuldiscussionsandsuggestionsregardingthequestionnairedesignandinterpretation,andtoLouiseHallforhelpwithsomedataanalysis.

TheworkwasfundedbythePublishingResearchConsortium.

Peer review: benefits, perceptions and alternatives

Documents

Transcript of Peer review: benefits, perceptions and alternatives