Post on 12-Feb-2017
PRC Summary Papers 4
Peer review: benefits, perceptions and alternatives
Mark WareMark Ware Consulting
© 2008 Publishing Research Consortium
Published byPublishing Research Consortiumc/o The Publishers Association29B Montague StreetLondon WC1B 5BW
Printed in Great Britain
The Publishing Research Consortium (PRC) is a group representing publishers and societies supporting global research into scholarly communication, with the aim to provide unbiased data and objective analysis. Our objective is to support work that is scientific and pro-scholarship. Overall, we aim to promote an understanding of the role of publishing and its impact on research and teaching.
Outputs from work supported by the PRC are available from the website:www.publishingresearch.org.uk. Hard copies are available from The Publishers Association.
The founding partners are The Publishers Association, the Association of Learned and Professional Society Publishers, and the International Association of Scientific, Technical Medical Publishers. Corresponding partners include The Association of American University Presses and the Professional/Scholarly Publishing Division of the Association of American Publishers.
This summary report was commissioned by the Publishing Research Consortium (www.publishingresearch.org.uk) from Mark Ware Consulting, a publishing consultancy (www.markwareconsulting.com). Mark Ware Consulting was responsible for designing and managing the survey; the online hosting and data analysis was provided by Mike Monkman Media (http://mikemonkman.com/).
Contents
4 Executiveoverview
5 Introduction
6 Whatispeerreview?6 Typesofpeerreview7 Peerreviewdurations9 Thereviewer’sperspective10 Roleoftheeditor
12 Thebenefitsofpeerreview12 Improvementsinquality13 Peerreviewasafilter14 Viewsofsurveyrespondents
16 Critiquesofpeerreview17 Viewsofsurveyrespondents
18 Alternativeapproaches
18 Differenttypesofpeerreview19 Post-publicationreview19 Openpeerreview20 Reviewingauthors’data
20 Conclusions
20 Acknowledgements
©2008 Publishing Research Consortium
4
©2008 Publishing Research Consortium
Peerreviewistheprocessofsubjectinganauthor’sscholarlymanuscripttothescrutinyofotherswhoareexpertsinthesamefield,priortopublicationinajour-nal.Thissummaryreportpresentssomefindingsfromanewinternationalsurveyofacademics,setwithinadescriptionofhowpeerreviewoperates,itsbenefits,somecritiquesofpeerreviewandthedevelopmentofalternativeapproaches.
Peerreviewiswidelysupportedbyacademics,whooverwhelmingly(93%)disagreedinoursurveythatpeerreviewisunnecessary.Thelargemajority(85%)agreedthatpeerreviewgreatlyhelpsscientificcommunicationandbelieved(83%)thatwithoutpeerreviewtherewouldbenocontrol.
Peerreviewimprovesthequalityofthepublishedpaper.Researchersover-whelmingly(90%)saidthatthemainareaofeffectivenessofpeerreviewwasinimprovingthequalityofthepublishedpaper,andasimilarpercentagesaidithadimprovedtheirownlastpublishedpaper,includingidentifyingscientificerrorsandmissingandinaccuratereferences.
Thereisadesireforimprovement.Whilethemajority(64%)ofacademicsdeclaredthemselvessatisfiedwiththecurrentsystemofpeerreviewusedbyjour-nals(andjust12%dissatisfied),theyweredividedonwhetherthecurrentsystemisthebestthatcanbeachieved.Therewasevidencethatpeerreviewistooslow(38%weredissatisfiedwithpeerreviewtimes)andthatreviewersareoverloaded(seeitem11below).
Double-blindreviewwaspreferred.Althoughthenormalexperienceofresearchersinmostfieldswasofsingle-blindreview,whenaskedwhichwastheirpreferredoption,therewasapreferencefordouble-blindreview,with56%select-ingthis,followedby25%forsingle-blind,13%foropenand5%forpost-publica-tionreview.Openpeerreviewwasanactivediscouragementformanyreviewers,with49%sayingthatdisclosingtheirnametotheauthorwouldmakethemlesslikelytoreview.
Double-blindreviewwasseenasthemosteffective.Double-blindreviewhadthemostrespondents(71%)whoperceivedittobeeffective,followed(indecliningorder)bysingle-blind(52%),post-publication(37%)andopenpeerreview(27%).
Double-blindreviewfacessomefundamentalobjections.Double-blindreviewwasprimarilysupportedbecauseofitsperceivedobjectivityandfairness.Manyrespondents,includingsomeofthosesupportingdouble-blindreview,didhow-everpointoutthatthereweregreatdifficultiesinoperatingitinpracticebecauseitwasfrequentlytooeasytoidentifyauthorsfromtheirreferences,typeofworkorotherinternalclues.
Post-publicationreviewwasseenasausefulsupplementtoformalpeerreview,ratherthanareplacementforit.Interestingly,thiswasdespiteaclearviewthatittendstoencourageinstantreactionsanddiscouragethoughtfulreview.
Limitedsupportforpaymentforreviewers.Respondentsweredividedonwhetherreviewersshouldbepaid,with35%infavourofand40%againstpay-ment.Amajority,however,supportedthepropositionthatpaymentwouldmakethecostofpublishingtooexpensive(52%for,18%against)andthelargemajorityofreviewers(91%)saidthattheyreviewedtoplaytheirpartasamemberoftheacademiccommunity.
Executive Overview
1
2
3
4
5
6
7
8
9
5
©2008 Publishing Research Consortium
Mixedsupportforreviewofauthors’data.Amajorityofreviewers(63%)andeditors(68%)saidthatitisdesirableinprincipletoreviewauthors’data.Perhapssurprisingly,giventhatmanyreviewersreportbeingoverloaded(seebelow),amajorityofreviewers(albeitasmallone,51%)saidthattheywouldbepreparedtoreviewauthors’datathemselves,comparedtoonly19%whodisagreed.
Some90%ofauthorsinthesurveywerealsoreviewers.Theyreportedreview-inganaverageof8papersinthelast12months.Thelargemajorityofreviews(79%)wascarriedoutbyacoreofactivereviewers,whocompletedanaverageof14reviewsperyear,nearlytwicetheoverallfigure.Thisgroupreporteditwasoverloaded–doing14reviewsperyearcomparedtotheirpreferredmaximumof13–suggestingthereisaproblemwithreviewerworkloads.
Introduction
Thisreporttakesalookatpeerreview:whatitis,andhowitworksinpractice;thebenefitsofpeerreview;somecritiques;andsomealternativeapproaches.Itislargelybasedonanewinternationalsurveyof3040academics,lookingattheirbehaviourandattitudesandperceptionsofpeerreview.Thissummaryreportcontainsonlyasmallfractionofthedataavailableinthefullreport,1whichinterestedreaderscanfindonthePublishingResearchConsortiumwebsite.
10
11
1Ware, M. and Monkman, M. (2008) Peer review in scholarly journals: perspective of the scholarly community – an international study. Publishing Research Consortium. Available at www.publishingresearch.org.uk
�
©2008 Publishing Research Consortium
What is peer review?
Peerreview,knownasrefereeinginsomeacademicfields,is(toquotetheun-peer-reviewedWikipedia)aprocessofsubjectinganauthor’sscholarlywork,researchorideastothescrutinyofotherswhoareexpertsinthesamefield.Inthisreportwewillconsideronlythepeerreviewofmanuscriptssubmittedtoacademicjournals(theothermainuseofpeerreviewisfortheawardofresearchgrants).
Editorialpeerreviewissaidtohavebegunintheearly18thcentury;forexample,theprefacetothefirstvolumeoftheRoyalSocietyofEdinburgh’sMedical Essays and Observations,publishedin1731,stated:‘Memoirssentbycorrespondencearedistrib-utedaccordingtothesubjectmattertothosememberswhoaremostversedinthesematters.Thereportoftheiridentityisnotknowntotheauthor.’2Duringthe19thandearly20thcentury,peerreviewdevelopedinafairlydisorganizedwayandmanyprominentjournaleditorsactedmorelikenewspapereditors,withlittleinterestinformalpeerreview.PeerreviewinthesystematizedandinstitutionalizedformweknowtodayhasdevelopedlargelysincetheSecondWorldWar,atleastpartlyasaresponsetothelargeincreaseinscientificresearchinthisperiod.
Injournalspeerreview,theauthor’smanuscriptisusuallysubjectedtosomeinitialcheckstoassessitssuitabilityforreview(forinstance,incompletemanuscriptsorworkthatwaspatentlypseudosciencewouldbedeclinedwithoutreview),afterwhichasmallnumberofreviewersareselected.Thetaskexpectedofthereviewersvariessomewhatfromjournaltojournal,butinessenceitisusuallytoassistthejournal’seditor(whomakesthefinaldecision)ondecidingwhetherornottoacceptthemanuscriptforpublication.Thereviewerwillcommentonthequalityoftheworkdone(forinstance,wastheexperimentaldesignappropriatetothequestionbeingstudied?)aswellasonitsoriginality(whatdoesitaddtowhatweknowalready?)anditsimportance(doesitmatter?).
Typesofpeerreview
Therearetwoapproachestopeerreviewincommonuseatpresent.Thenorminmostacademicdisciplines,knownassingle-blindreview,isfortheauthor’sidentitytobeknowntothereviewers,butforthereviewers’identitytobehiddenfromtheauthor.(ThisisthemethoddescribedabovebytheRoyalSocietyofEdinburghin1731.)Themainargumentfor‘blinding’thereviewers’identityisthatitallowsthemtocommentfreelywithoutfearofrepercussions.Conversely,single-blindreviewhasbeencriticisedforallowingallkindsofbiasandotherkindsofirresponsibilityonthepartofreviewerstoflourishbehindtheveilofsecrecy.(Weshalldiscussthecriticismsofpeerreviewinmoredetailbelow.)
Themainalternativeisknownasdouble-blindreview:inthisapproachtheidentitiesoftheauthorandreviewersarehiddenfromeachother.Becausethereviewerdoesnotknowtheauthorortheirinstitution,itisargued,theywillfocusonthecontentofthemanuscriptitself,unaffectedbyconsciousorunconsciousbias.
Anewerapproachtodealingwiththecriticismsofsingle-blindreviewisopenpeerreview:inthismodel,theauthor’sandreviewers’identitiesareknowntoeachother,andthereviewers’namesand(optionally)theirreportsarepublishedalongsidethepaper.Advocatesofopenreviewseeitasmuchfairerbecause,theyargue,somebodymakinganimportantjudgementontheworkofothersshouldnotdosoinsecret.Itisalsoarguedthatreviewerswillproducebetterworkandavoidoffhand,carelessorrudecommentswhentheiridentityisknown.
Morerecently,electronicpublishingtechnologyhasallowedavariantofopenreviewtobedeveloped,inwhichallreaders,notjustthereviewersselectedbytheeditor,areable
2From Rennie, D. (2003) Editorial peer review: its development and rationale. In F. Godlee, T. Jefferson (eds). Peer Review in Health Sciences. Second Edition. pp. 1-13. BMJ Books, London.
�
©2008 Publishing Research Consortium
toreviewandcommentonthepaperandeventorateitonanumericalscalefollowingpublication.Thispost-publicationreviewcouldoccurwithorwithoutconventionalpre-publicationpeerreview.Thebenefitsareseentobethatittakesaccountofcom-mentsfromawiderrangeofpeople(‘thewisdomofcrowds’)andmakesthereviewamorelivingprocess.
Inoursurvey,wefoundthattheconventionalsingle-blindpeerreviewsystemwastheonemostcommonlyexperiencedbyauthors,with85%sayingtheyhadexperienceofitcomparedto45%fordouble-blind,23%foropenandjust8%forpost-publicationpeerreview(Figure1).Thisdoesvarybyacademicdiscipline:single-blindreviewwasthenorminlifesciences,physicalscienceandengineering,whiledouble-blindreviewwasmuchmorecommonforauthorsinhumanitiesandsocialsciences,andclinicalmedicalandnursingauthorshadexperienceofbothsystems.
Peerreviewdurations
Thepeerreviewprocessinevitablytakestime.Thesurveylookedatthisfromtheperspectiveofauthors,reviewersandeditors.
Authorsreportedthatthepeerreviewprocesstookanaverageof80days,withthelongesttimesinhumanitiesandsocialsciences.Theywereevenlysplitonwhetherornotthelengthoftimefromsubmissiontodecisionwassatisfactory,anditwasclearthattheauthorsexperiencingthelongestdelaysweretheleastsatisfied.Forreviewtimesof30daysorless,abouttwo-thirdsofrespondentsweresatisfiedwiththetime;thisdropssharplyat3–6monthsto19%,andto9%forreviewtimesinexcessof6months.
Editorsreportedaveragesubmission-to-acceptancetimesofroughly130days,splitroughlyequallybetweentheinitialpeerreviewstagetofirstdecision,andsubsequentrevisionstages.Nearlythree-quarters(72%)reportedtimesof6monthsorbelow.Timeswereshortestinmedicalandnursingjournals,andlongestinhumanitiesandsocialsciencesjournals.Mosteditorswerehappywithreviewingtimesontheirjournals,butasubstantialminority(aroundathird)wasunhappy.
Theoverwhelmingmajorityofeditors(98%)gavetheirreviewersadeadlineforresponding,withtheaveragedeadlinebeingabout34days,andwith63%ofeditorsgiving30daysorless.Deadlineswereshorterinmedicalandnursingresearch,andlongestinhumanitiesandsocialsciencesandphysicalsciencesandengineering.
Figure 1 Types of peer review experienced by authors and used by journal editors
Single-blind peer review
Double-blind peer review
Open peer review
Post-publication review
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Experienced by all respondents Used by editors’ journals
72%
85%
22%
45%
3%
23%
1%
8%
8
©2008 Publishing Research Consortium
Free subscription
Acknowledgement in journal
Payment in kind
CME/CPD points
Name published
Report published anon.
Signed report published
Name disclosed to author
-60% -45% -30% -15% 0% 15% 30% 45% 60%
-7% 56%
-10% 44%
-12% 43%
-11% 20%
-38% 19%
-35% 16%
-47% 11%
Less likely More likely
-49% 10%
Figure 3 Factors affecting reviewers’ likelihood to review for a journal
Figure 2 Reasons for reviewing
To play your part as a member of the academic community
To enjoy being able to improve the paper
To enjoy seeing new work ahead of publication
To reciprocate the benefit when others review your papers
To enhance your reputation or further your career
Personal recognition/build a relationship with the editor
To increase the chance of a role in the journal’s editorial team
-50% -25% 0% 25% 50% 75% 100%
Disagree Agree
-3% 91%
-6% 78%
-11% 69%
-14% 67%
-28% 44%
-37% 30%
-50% 20%
9
©2008 Publishing Research Consortium
Thereviewer’sperspective
Researchersreportedreviewingregularlyfor3.5journalsandforafurther4.2journalsoccasionally.Onaverage,reviewerssaidthattheyreviewedabout8papersintheprevious12months.Thisaveragefiguredisguisesthedistributionofreviewsamongreviewers.Weidentifiedthegroupofreviewerswhoreporteddoing6ormorereviewsinthelast12months(‘activereviewers’),andthisgroupmanagednearlytwiceasmanypapersastheaverage.Thismeantthatalthoughactivereviewersmadeupjust44%ofreviewersinoursurvey,theywereresponsiblefor79%ofreviews.
Reviewerssaythattheytookabout24daystocompletetheirlastreview,with85%reportingthattheytook30daysorless.Theyspentamedian5hours(mean9hours)perreview.ActivereviewersandthoseintheEnglish-speakingregionsreportedspendingconsiderablylesstimeperreviewthanlessfrequentreviewersandthosefromAsiaandtheRestofworld.
Weaskedreviewerstostatethemaximumnumberofreviewstheywerepreparedtoundertake.Theaveragefigureforallrespondentswas9reviews.Thiscomparestotheaverageof8reviewscompletedinthelast12months.Overall,therefore,therewouldappeartobeatleastsomeslackinthesystem.Thisapparentpositionofcomfortablecapacitybreaksdown,however,whenthedistributionofreviewsistakenintoaccount.Activereviewers(responsiblefor79%ofallreviews)proposedamaximumof13papers,comparedtotheiraverageof14reviewsdoneinthelast12months,suggestingthereisaproblemofrevieweroverloading.
Why reviewers reviewWewereinterestedtoexplorethereasonswhyreviewersreview,andwhatincentiveswereofferedandwhichwereeffective.
Ingeneral,respondentspreferredtooffermorealtruisticexplanationsforwhytheyreviewed(seeFigure2),withsubstantiallythemostpopularreasonbeing‘playingyourpartasamemberoftheacademiccommunity’.Self-interestedreasonssuchas‘toenhanceyourreputationorfurtheryourcareer’or‘toincreasethechanceofbeingofferedaroleinthejournal’seditorialteam’weremuchlessfrequentlyadvanced.Themostcommonrewardsforreviewingreportedbyeditorswerereviewerreceptionsatconferencesandwaiverofauthorcharges(e.g.publication,page,colour,offprintcharges)(both39%).Monetarypaymentwasrareatonly5%ofeditors,thoughmorecommonthancreditsforcontinuingprofessionaldevelopment(2%).Paymentwasmostcommoninhumanitiesandsocialsciencesjournals(9%).
Fromthereviewers’perspective,theincentivestheysaidweremostlikelytoencouragethemtoactforajournalwere(seeFigure3):
•afreesubscriptiontothejournal(56%saidthiswouldmakethemmorelikelyto reviewforthejournal)•acknowledgementinthejournal(44%)•paymentinkindbythejournal,forexamplewaiverofcolourorotherpublication charges,freeoffprints,etc.(43%).
Payment for reviewersReviewersweredividedonwhethertheyshouldbepaidforeachreviewtheycom-pleted:35%agreedthattheyshould,while40%disagreed.ThosefromtheAnglo-phoneregionswerethemostopposedtopayment,whereasresearchersfromAsiaandfromEuropewereonbalancejustinfavour(44%for,32%against).
Therewaslesssupportfortheideathatpaymentwouldreducetheobjectivityofpeerreview(28%for,43%against)butamajorityforthepropositionthatpaymentwouldmakethecostofpublishingtooexpensive(52%for,18%against).
Forthemostpart,respondents’viewsonthesequestionsappeartobepersonalmatters,independentoftheirfieldofresearch.Asalreadynoted,respondents
10
©2008 Publishing Research Consortium
preferredtogivealtruisticreasonsforreviewing.Womenthroughoutthesurveytendedtoexpressmorealtruisticpositionsthanmen;theirresponseshereareconsistentwiththatposition,withwomenbeinglessinclinedthanmentothinkreviewersshouldbepaid,andmoreinclinedtoseethedownside.
Roleoftheeditor
Thefunctionoftheeditoristoselectthemostappropriatemanuscriptsforpubli-cationintheirjournalandtoensuretheyareeditedandpresentedinthebestwayforthejournal’sreadership.Theirpreciserolevariesconsiderablyfromjournaltojournal;forsomelargerscientificjournalsthereviewersmaybeselectedandmanagedbyaneditorialteamatthepublisher’sofficewiththeeditoronlybecominginvolvedoncethereviewerreportsarereceived(orinsomecases,onlyifthereisadisputebetweenreviewerstobeadjudicated),whileothereditorsaremuchmorehands-on,appointing,selectingandchasingupthereviewersthemselves.
Inthesurvey,wefoundthateditorssaidthatthenumberofpaperstheyhandled(i.e.thenumberonwhichtheymadeaccept/rejectdecisions)wasabout50peryear.Themajority(59%)handled25orfewerpapersbuttherewasasmallgroup(11%)ofmuchbusiereditorshandlingmorethan150.Editorsassignedabout2.3reviewersperpaper.Selectionofthereviewersbytheeditorthemselveswasonlythethirdmostpopularoption(reportedby28%ofeditors),wellbehindselectionbyamemberoftheeditor’steam(73%)andbyamemberofthepublisher’sstaff(43%).
Onlinemanuscriptsubmissionandtrackingsystemswereusedbyaboutthree-quartersofeditors.Theirusewasmorecommoninlifesciences(85%)andmarkedlylesscommoninhumanitiesandsocialsciences(51%).
Editorsreportedthattheaverageacceptanceratefortheirjournalswasabout50%,whichisconsistentwithotherstudies(Figure4).About20%ofsubmittedmanu-scriptsarerejectedpriortoreview(eitherbecauseofpoorquality(13%)orbeingoutofscope(8%))andanother30%arerejectedfollowingreview.Ofthe50%accepted,41%areacceptedsubjecttorevision.Acceptancerateswerelowerinhumanitiesandsocialsciences,andhigherinphysicalsciences/engineeringjournals.
TrishGroves(deputyeditorofthebmj)haswritten3thatanobviouswaytoimproveanyjournal’speerreviewsystemisto‘Tellauthorsandreviewerswhatyouwantfromthem.. . .Givereviewersclearbriefs,includingguidanceonwhattoincludeinthereview’.Inthelightofsuchcommonsenseadvice,itwassomewhatsurprisingtofindthat30%ofeditorsdidnotprovidereviewerswithachecklist.Theuseofchecklistswassomewhatlesscommoninhumanitiesandsocialsciencesjournals(45%notusing).Whereeditorsdidprovidechecklists,themostcommonquestionsinvolvedthestudymethodology(87%ofchecklists),relevance,importanceandpaperlength(Figure5).
Grovesalsowrote4‘Reviewershavealsotoldustheywantfeedbackontheirperfor-mancesothattheycanlearnandimprove.’Thisseemsanothercommonsensepositionbutonly28%ofeditorsinoursurveyreportedthattheygavefeedbacktoreviewersonthequalityoftheirreports.Themostcommonfeedbackgivenwasjustthepublicationoutcome.
3Groves, T. (200�) Quality and value: how can we get the best out of peer review? Nature (Nature Peer Review debate). doi:10.1038/nature04995
4ibid.
11
©2008 Publishing Research Consortium
Figure 5Items used in reviewer checklists provided by editors
Methodology employed
Relevance
Importance
Paper length
Tables
Statistics
Originality
Experimental data
Illustrations
References
Language
Quality
Ethical issues
Other
87%
85%
78%
75%
60%
53%
53%
47%
42%
37%
35%
34%
26%
8%
0% 20% 40% 60% 80% 100%
Reject after review
30%
Reject prior
(poor quality)
13%
Reject prior
(out of scope)
8%
Accept, no revision
8%
Accept, with revision
41%
Figure 4 Ultimate fate of submitted manuscripts
12
©2008 Publishing Research Consortium
The benefits of peer review
Whatarethebenefitsofpeerreview?Inoneview,therearebenefitsforallplayersinthesystem:editorsaresupportedintheirdecisionsbytheviewsofexperts;authorsbenefitfromtheassistanceofferedbyreviewersandfromthestatusconferredonthembypublicationinjournalswithhighpeerreviewstandards;readersbenefitbecauseofthefilteringthatpeerreviewprovidesandbythe‘sealofapproval’thatpeerreviewisthoughttoprovide;andevenreviewers(whodothebulkoftheworkfornodirectrecompense)benefittosomeextent(e.g.inseeingworkpriortopublication).
Lookingbeyondtheinterestsoftheparticularstakeholders,therearethreemainbenefitsadvocatedforpeerreview:
•improvementinthequalityofpublishedpapers;•filteringoftheoutputofpaperstothebenefitofreaders;•a‘sealofapproval’thatthepublishedworkmeetscertainstandards,inparticular forlayreaders.
Let’slookinmoredetailattheseproposedbenefits.
Improvementsinquality
Thereareanumberofwaysinwhichpeerreviewmightimprovethequalityofpublishedpapers,ofwhichthemostimportantare:
•theveryfactofaqualityhurdleorthreshold,whichwillmotivateauthorsto improvethequalityoftheirworkpriortosubmission;•thepeerreviewprocess,inwhichreviewers’commentsandcriticismsareaddressed bytheauthorbyrevisingthemanuscript.Testingofworkthroughthecriticismof peersisinabroadsenseattheheartofthescientificmethod.
Perhapssurprisingly,thereislittlescientificevidencetosupporttheuseofpeerreviewasamechanismtoensurequality(seebelow,underCritiques of peer review).Inoursurvey,however,thelargemajorityofauthors(around90%)wereclearthatpeerreviewhadimprovedtheirownlastpublishedpaperandasimilarproportionagreedwiththemoregeneralstatement‘peerreviewimprovesthequalityofthepublishedpaper’.
Respondentswhosaidthatpeerreviewhadimprovedtheirlastpaperwereaskedwhichaspectsofthepaperhadbeenimproved,andineachcasebyhowmuch(usinga1–5scale).TheresultsareshowninFigure6.
Some64%ofrespondentsreportedthatpeerreviewoftheirlastpublishedpaperhadidentifiedscientificerrors,demonstratingrealvaluebeingaddedbypeerreview,and78%saidithadidentifiedmissingorinaccuratereferences.
‘Madesuggestionsonpresentation’wasthemosthighlyratedaspect;94%ofthosewhosaidtheirpaperhadbeenimprovedreportedimprovementinthisarea,and55%ratedtheimprovementat4or5outof5.Thelanguageorreadabilitywasalsofrequentlycited(86%reportedsomeimprovementinthisarea).
Thosewithgoodaccesstothejournalsliteraturereportedlessimprovementinidentifyingmissingofinaccuratereferencesthanthosewithworseaccess.Thisiswhatwemightexpecttofindandillustratesonewayinwhichrestrictedaccesstoliteraturecanaffectresearchers.
Therewassomewhatlessimprovementreportedregardingtheidentificationofstatisticalerrorsthanforotherbenefits,although51%stillreportedsomeimprove-ment.Isthisbecauseauthorsarelesslikelytomakestatisticalerrorsthanother
13
©2008 Publishing Research Consortium
kinds,orbecausereviewersarelesslikelytospotthem?Giventhatsomestudieshaveshownthatpublishedpapersarerifewithstatisticalerrors(e.g.EmilGarcia-BerthouandCharlesAlcarazfoundstatisticalinconsistenciesin38%ofpapersinNatureand25%intheBMJ5)thelatterseemsamorelikelyexplanation.
Peerreviewasafilter
Therearetwosensesinwhichpeerreviewandthejournalsysteminwhichitisembeddedcanfilterresearchoutputsforthebenefitofreaders.
First,peerreviewcouldbeseentofilteroutbadworkfromtheliterature,byreject-ingitforpublication.‘Badwork’herecouldmeanpoorlyconceivedorexecuted,orofminimaloriginalityorinterest,or‘bad’inthemoralsense,forinstanceinvolv-ingacademicfraudorplagiarism.Workthatdoesgetpublishedinapeer-reviewedjournalisseentohavemetsomequalitythresholdorgaineda‘sealofapproval’.Groupspromotingbetterpublicunderstandingofsciencewilloftenusepeerreviewinthisway;forinstance,theukgroupSenseAboutSciencepromotesunderstandingofpeerreview,whichitcallsthe‘essentialarbiterofscientificquality.’6
Thereare,however,atleasttwoproblemswiththisposition.Becausethepeerreviewstandardsofdifferentjournalsvary,itiswidelybelievedthatalmostanygenuineacademicmanuscript,howeverweak,canfindapeer-reviewedjournaltopublishitiftheauthorispersistentenough.Manuscriptsrejectedbyonejournalareroutinelysubmittedtoanother,probablyonewithalowerrejectionrate.Acceptancebyapeer-reviewedjournaldoesnotsayverymuchaboutthequalityororiginalityofapaperbutitmaystilldistinguishitfrompseudoscienceoregregiouslybadwork,andthisisthewayinwhichgroupslikeSenseAboutSciencebelieveitcanhelpthepublic.Theotherproblemisthatpeerreviewhasbeenshownnottobeparticularlyeffectiveasaqualitycontroltool,oratdetectingerrorsoroutrightfraud.(Theseproblemsarediscussedinmoredetailbelow,seeCritiques of peer review.)
Don’t know/Not applicable 1 (no improvement) 2 3 4 5 (substantial improvement)
Made suggestions
on presentation
The language or readability
Identified missing or
innacurate references
Identified scientific errors
Identified statistical errors
5% 12% 27% 37% 18% 1%
12% 20% 25% 27% 14% 2%
24% 19% 21% 21% 10% 3%
30% 16% 17% 18% 11% 6%
35% 15% 15% 13% 6% 14%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Figure �Improvements made by peer review to authors’ last published paper
5Garcia-Berthou, E. and Alcaraz, C. (2004) Incongruence between test statistics and P values in medical papers. BMC Medical Research Methodology 4: 13.
�See http://www.senseabout-science.org.uk/index.php/site/project/29/
14
©2008 Publishing Research Consortium
Thesecondwayinwhichpeerreviewcanprovideafilterforreadersismuchmoreimportantforworkingacademics:itprovidesthebasisforthestratificationofjournalsbyperceivedquality(wherequalityisfrequentlytakentobeindicatedbytheimpactfactor,ameasureofhowoftenonaveragearticlesinthejournalinquestionarecited).Peerreviewthussupportsthesystemthatroutesthebetterpaperstothebetterjournalsandthisallowsacademicstofocustheirreadingonamanageablenumberofcorejournalsintheirfield.Publishersinparticularseethiskindoffilter-ingasoneofthemajorbenefitsofpeerreviewandthejournalssystem.
Respondentstooursurveyhavealotofconfidenceinthepeerreviewsystemtosupportthesefilteringfunctions(seeFigure7).Aswellasverystronglysupportingthenotionthatpeerreviewimprovesthequalityofpublishedpapers(asdiscussedabove),therewasalsostrongsupportfortheideathatitdeterminestheimportanceofthefindingsandtheoriginalityofthemanuscript.Therewassomewhatlesssupport(thoughstillanetmajority)forbelievingpeerreviewwaseffectiveatdetect-ingplagiarismandacademicfraud.
Viewsofsurveyrespondents
Overall satisfaction with peer reviewThemajority(64%)ofacademicsdeclaredthemselvessatisfiedwiththecurrentsystemofpeerreviewusedbyjournals,withjust12%sayingtheyweredissatisfied(Figure8).Therewasverylittlevariationamongstthesampleinthesefigures;forinstancetherewerenodifferencesbyage,genderorposition(seniority).
Respondents’attitudeswerealsotestedbyaskingfortheirdegreeofagreementordisagreementtowardsanumberofstatementsaboutpeerreview,asshowninFigure9.
Onthepositiveside,thelargemajority(85%)agreedwiththepropositionthatscientificcommunicationisgreatlyhelpedbypeerreview.Therewasasimilarlyhighlevelofsupport(83%)fortheideathatpeerreviewprovidescontrolinscientificcommunication.
Giventhegenerallylowlevelofoveralldissatisfactionwithpeerreview,though,itisperhapssurprisingthatastrongstatementlike‘peerreviewinjournalsneedsa
Figure �Views on the effectiveness of peer review in different areas
Disagree Agree
Improves quality
Determines importance
Determines originality
Picks best mss for journal
Detects plagiarism
Detects fraud
-30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
-3% 90%
-16% 60%
-17% 58%
-22% 49%
-24% 46%
-26% 43%
15
©2008 Publishing Research Consortium
completeoverhaul’didnotreceivemoredisagreement–infactrespondentsweredivided,with35%disagreeingversus32%agreeing.Similarly,respondentsweredividedonwhetherthecurrentpeerreviewsystemisthebestwecanachieve,with32%agreeingversus36%disagreeing.
Therewas,however,virtuallynosupportfortheradicalpropositionthatpeerreviewwascompletelyunnecessary.
Onlyaminorityoverall(19%)agreedthatpeerreviewwasholdingbackscientificcommunication.Thosewithpoor/verypooraccesstothejournalsliteraturetendedtoagreemore(23%)thanthosewithexcellentaccess(16%).
Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied Don’t know
6% 59% 22% 10% 2% 1%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Greatly helps scientific comm’n
No control without peer review
Needs complete overhaul
Current system is best achievable
Holds back scientific comm’n
Completely unnecessary
Disagree Agree
-100% -75% -50% -25% 0% 25% 50% 75% 100%
-5% 85%
-10% 83%
-35% 32%
-36% 32%
-63% 19%
-93% 3%
Figure 8 Overall satisfaction with the peer review system used by scholarly journals
Figure 9 Views on peer review
1�
©2008 Publishing Research Consortium
The dissatisfied groupWhilethelargemajorityofrespondentsexpressedthemselvessatisfiedwiththepeerreviewsystemusedbyscholarlyjournals,aminority(12%)saidtheyweredissatisfiedorverydissatisfied.Itisinterestingtoaskwhatwecansayaboutthisgroup.
Intermsofdemographics,therearerelativelyfewdifferencesfromtheaverage.Therewerenosignificantdifferencesbyage,gender,typeoforganizationorposition(seniority).Byregion,theyweremoresomewhatlikelytobeintheAnglophoneregions,andlesslikelytobeinAsiaortheRestofworld.Lookingatfieldofresearch,theyweremostlikelytobeinhumanitiesandsocialsciences,andleastlikelyinphysicalsciences/engineering.
Intermsoftheirownexperienceofpeerreview,thisgroupreportedthatthepeerreviewoftheirlastpublishedpapertooksignificantlylongerthanaverage(about110comparedto80days),andtheyweremorelikelytobedissatisfiedwiththelengthoftimeinvolved.Thedissatisfiedgrouptendedtobesomewhatlesslikelytoreportthatpeerreviewhadimprovedtheirlastpublishedpaper,andlikelytogivelowerscorestotheimprovementstheydidreport.Wecannotfromthedatasayifthereisacausalrelationship;thatis,isthisgroupdissatisfiedwithpeerreviewbecausetheyhaveexperiencedlongertimesandlesspersonalbenefitontheirownpapers,ordoestheirdissatisfactionarisefromothercausesandthenleadthemtogivelesspositivescores?
Intermsofalternativeapproachestopeerreview,thisdissatisfiedgroupwasmorelikelytoagreethatopenandpost-publicationreviewwereeffective.Asasmallminority,however,theydidnotformthemainconstituencyforthesealternativeapproaches.
Critiques of peer review
Peerreviewisnotwithoutitscritics.Perhapsthestrongestcriticismisthatthereisalackofrealevidencethatpeer
reviewactuallyworks:forinstance,a2002studypublishedinthe Journal of the American Medical Association7concludedthat‘Editorialpeerreview,althoughwidelyused,islargelyuntestedanditseffectsareuncertain’.Similarly,theCochraneCollaboration(auk-basedinternationalhealthcareanalysisgroup)firstpublisheditsownreviewin2003,whichconcludedthattherewas‘littleempiricalevidencetosupporttheuseofeditorialpeerreviewasamechanismtoensurequalityofbiomedicalresearch,despiteitswidespreaduseandcosts’.Thelatestupdate(2007)oftheCochranereviewconfirmsthisconclusion8,thoughitisimportanttounder-standthatitissayingthattheevidencetosupportpeerreviewhasnotyetbeenproduced,notthatthereisevidencethatpeerreviewdoesnotwork.
Somehaveshownthatpeerreviewcanbeunreliable.Forinstanceonestudy9showedthatthechancesoftworeviewersagreeingaboutaparticularpaperwereonlyslightlybetterthanchance;inordertoproduceareliableresult,editorswouldneedtousesixreviewersforeachpaper.(Inpractice,theytypicallyusetwoorthree–theaveragereportedinthissurveywas2.3.)
Otherstudieshaveshownthatpeerreviewcanbenotverygoodatdetectingerrors.Godleeandcolleaguesatthebmjtookapaperabouttobepublished,insertedeightdeliberateerrors,andsentthepaperto420potentialreviewers:221(53%)responded.Theaveragenumberoferrorsspottedwastwo,nobodyspotted
�Jefferson, T., Alderson, P., Wager, E. and Davidoff, F. (2002) Effects of Editorial Peer Review: A Systematic Review. Journal of the American Medical Association 28�: 2�84-2�8�.
8Jefferson, T., Rudin, M., Brodney Folse, S. and Davidoff, F. (200�) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Sytematic Reviews 200�, Issue 2. Art. No.: MR00001�. DOI: 10.1002/14�51858.MR00001�.pub3
9Rothwell, P.M. and Martyn, C.N. (2000) Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 123: 19�4-19�9.
1�
©2008 Publishing Research Consortium
morethanfive,and16%didn’tspotany10.Itisalsosaidthatpeerreview,particularlyinitssingle-blindform,offerstoo
muchscopeforbiasonthepartoftherevieweroreditor.Forinstance,paperspublishedinanissueoftheJournal of the American Medical Association11devotedtopeerreviewpresentedevidencefornationalitybias,languagebias,specialtybias,andperhapsevengenderbias,aswellastherecognisedbiastowardthepublicationofpositiveresults.
Oneresponsetotheproblemsofreviewerbiashasbeentomovetodouble-blindratherthansingle-blindreview.However,thesecrecyinvolvedin‘blinding’thereviewer’sidentityhasitselfbeencriticisedontwomaingrounds.Fromapragmaticviewpoint,moststudiesthathaveinvestigatedreviewerblindinghavefailedtomeasureimprovementsinthequalityofthereviewand,conversely,otherstudieshaveshownthatmakingthereviewer’sidentityknowntoauthorshadnoeffectonquality.12Thereisalsoastrongethicalargumentagainstsecrecy,namelythatitisseentobeunfairforsomebodymakinganimportantjudgementontheworkofotherstodosoinsecret.
Anotherargumentagainstdouble-blindingisthatitisverydifficultinpracticetodisguisetheidentityoftheauthorofanacademicmanuscriptfromaskilledreviewer;bydefinitionthereviewerisanexpertinthefieldwhowillfrequentlyknowthepreviousworkofauthorsinthefield.
Otherpragmaticcriticismsofpeerreviewincludethedelayitcausestopublica-tionandtheviewthatitdoesnotscaleefficientlywiththegrowthofscience.Thesurveyshowedsomebasisforeachofthese.Althoughtheaveragedelayreportedbyauthorsforpeerreviewwasonlyabout80days,39%reportedtimesofgreaterthan3months,and10%ofgreaterthan6months.Editorsreportedthattheaveragetimefromsubmissiontoacceptanceontheirjournalswasabout130days,with22%reportingtimesofmorethan6months.Therewasacorrelationbetweenthosereportinglongerreviewtimesandloweroverallsatisfactionwithpeerreview.Thesurveyalsoshowedthatthelargemajorityofreviewswereundertakenbyacoregroupofactivereviewerswhoappeartobeoverloaded.
Viewsofsurveyrespondents
Howdidsurveyrespondentsdealwiththesecriticisms?Inthemostpart,aswehavealreadyseen,respondentshadpositiveviewsaboutpeerreviewanditseffectivenessatimprovingthequalityofpublishedpapers.Theirviewsonalternativesystemsofpeerreview,whichhavebeenproposedatleastinpartasresponsestocriticismsofconventionalpeerreview,areexploredinthenextsection.
11Journal of the American Medical Association (1998) 280: issue 3.
10Godlee, F., Gale, C. R. and Martyn, C. N. (1998) Journal of the American Medical Association 280: 23�-240.
12E.g. Goldbeck-Wood, S. (1999) Evidence on peer review: scientific quality control or smokescreen? British Medical Journal 318: 44-45.
18
©2008 Publishing Research Consortium
Alternative approaches
Differenttypesofpeerreview
Mostrespondentsinmostfieldsexperiencesingle-blindreviewasthenorm.Whenaskedwhichoptionstheythoughtwereeffective,however,respondentsexpressedaclearpreferencefordouble-blindreview,asshowninFigure10.Thelevelofsupportfortheeffectivenessofpost-publicationreviewissurprisinglyhigh.
Respondentsdidnothavepersonalexperienceofalltypesofreview.Thosewithexperienceofdouble-blindreviewweresubstantiallylesslikelytoratesingle-blindreviewaseffectivecomparedtoothers.Similarly,thosewhohadexperienceofopenpeerreviewandpost-publicationreviewasanauthorwereconsiderablymorelikelytoratethemaseffective.Itisnotable,though,thatalthough37%ofrespondentssaidthatpost-publicationreviewwaseffective,only8%hadhadexperienceofitasauthors–thissupportisthereforesomewhathypothetical.
Askedwhichofthefourpeerreviewtypeswastheirpreferredoption,therewasaclearpreferencefordouble-blindreview,with56%selectingthis,followedby25%forsingle-blind,13%foropenand5%forpost-publicationreview.Post-publicationreviewgetsmuchlesssupportherecomparedtotheperceptionsofitseffectiveness:thisisnotinconsistentbecauserespondentsclearlysawitasausefulsupplementtocurrentpeerreviewmethodsratherthanareplacementforthem.
Itwasclearfromtheverbatimcommentsthatthepreferencefordouble-blindreviewwaslargelyaresponsetothepotentialforbiasinsingle-blindreview:thereasonsgivenforthispreferencewereprimarilyitsobjectivityandfairness.
Single-blind
Double-blind
Open
Post-publication
0% 10% 20% 30% 40% 50% 60% 70% 80%
25%
52%
56%
71%
13%
27%
5%
37%
Thought effective Preferred option
Figure 10Types of peer review thought to be effective (multiple responses allowed), and respondents’ preferred choice (single response)
19
©2008 Publishing Research Consortium
Post-publicationreview
Lookinginmoredetailatpost-publicationreview(Figure11),researcherssawitasausefulsupplementtoformalpeerreviewinquitelargenumbers(53%agreedcomparedto23%disagreeingwiththisstatement).Theyseethisusefulnessdespiteaclearperceptionthatittendstoencourageinstantreactionsanddiscouragethoughtfulreview.Thereislesssupportfortheideathatitcouldbealessgoodbutstillacceptablealternative(31%supportedversus43%opposed)andfairlystrongoppositiontotheideathatitcouldbeanequallypowerfulalternativetoformalpeerreview(57%opposedversus19%supported).Therewasevenstrongeroppositiontoreplacingpeerreviewwithpost-publicationratingsorusageorcitationstatisticstoidentifygoodpapers.
Openpeerreview
Supportforopenpeerreviewstartedtogrowduringthemid-1990s.TheBMJwasoneofthefirstmajorjournalstoadoptopenpeerreview,basingitsdecisionpartlyontheethicalcaseagainstsecrecyandpartlyontheevidencementionedabovethatblindingdidnotimprovereviewoutcomes.Openreview,however,remainsfarfrombeingthenorm.Themainargumentagainstitisthatreviewerswillbereluctanttocriticisetheworkofmoreseniorresearchersonwhomtheymaybedependentforcareeradvance-mentorgrantawards.During2006,thejournalNatureconductedatrialofopenpeerreview13;itwasnotasuccess–despiteinterestinthetrial,onlyasmallpropor-tionofauthorschosetoparticipate,andonlyafewcommentswerereceived,manyofwhichwerenotsubstantive.Feedbacksuggested‘thatthereisamarkedreluctanceamongresearcherstoofferopencomments’.
Inthesurvey,thenumbersofrespondentspreferringopenpeerreviewweresmallerthanforsingle-ordouble-blindpeerreview(about13%).Themainreasonsgivenforpreferringitwere:revieweraccountability,leadingtobetterreportsandlesslikelihoodofbias,andtheviewthatopenreviewmadereviewersmorecivil,madetheprocessmoreofadialoguewiththeauthorandgenerallyimprovedauthor/reviewercommunication.
Disagree Agree
Encourages instant reactions
Useful supplement to formal review
Authors less likely to submit
Readers fear offending authors
Would relieve load on reviewers
Acceptable (but weaker) alternative
An equally powerful alternative
-60% -45% -30% -15% 0% 15% 30% 45% 60%
-16% 56%
-23% 53%
-29% 41%
-31% 39%
-32% 38%
-43% 31%
-57% 19%
Figure 11Views on post-publication review
13Nature editors/publishers. (200�) Overview: Nature’s peer review trial. Nature doi:10.1038/nature05535.
20
©2008 Publishing Research Consortium
Proponentsofopenpeerreviewwillalsohavetoovercomethefactthat47%ofreviewerssaidthatpublishingtheirsignedreportwouldmakethemlesslikelytoreviewforajournalandthatasimilarproportion,49%,wouldseedisclosureoftheirnametotheauthorasadisincentive(seeFigure3above).
Reviewingauthors’data
Asscienceutilizesmoreautomatedexperimentalequipmentandotherwisemovestowardsamoredata-centric‘e-science’model,theamountofdatathatsupports(andcouldpotentiallybelinkedto)theaveragescientificpaperincreases.Thequestionarisesastowhetherthisdatashoulditselfbesubjecttopeerreview.Thereareclearlyanumberofpracticalissues:doreviewershavethetimetodothis?Isthedatasufficientlystandardized,anddothesoftwaretoolsexisttohandleit?Areauthorsevenpreparedtosharetheirdatawithreviewers?
Amajorityofreviewers(63%)andeditors(68%)saidthatitwasdesirableinprin-cipletoreviewauthors’data.Perhapssurprisingly,amajorityofreviewers(albeitasmallone,51%)saidthattheywouldbepreparedtoreviewauthors’datathemselves,comparedtoonly19%whodisagreed.Thiswasdespite40%ofreviewers(and45%ofeditors)sayingthatitwasunrealistictoexpectpeerreviewerstoreviewauthors’data.
Conclusions
Thesurveythuspaintsapictureofacademicscommittedtopeerreview,withthevastmajoritybelievingthatithelpsscientificcommunicationandinparticularthatitimprovesthequalityofpublishedpapers.Theyarewillingtoplaytheirpartincarryingoutreview,thoughitisworryingthatthemostproductivereviewersappeartobeoverloaded.Manyoftheminfactsaytheyarewillingtogofurtherthanatpresentandtakeonresponsibilityforreviewingauthors’data.
Withinthispictureofoverallsatisfactionthereare,however,somesizeablepocketsofdiscontent.Thisdiscontentdoesnotalwaystranslateintosupportforalternativemethodsofpeerreview;forexamplesomeofthosemostpositiveaboutthebenefitsofpeerreviewwerealsothemostsupportiveofpost-publicationreview.
Acknowledgements
ThanksareduetoMikeMonkman(MikeMonkmanMedia,http://mikemonkman.com/)forhelpindesigningthesurvey,forprovidingtheonlinehostingfacilitiesandfortheanalysisofthesurveydata.ThanksarealsoduetoMayurAminandAdrianMulliganforhelpfuldiscussionsandsuggestionsregardingthequestionnairedesignandinterpretation,andtoLouiseHallforhelpwithsomedataanalysis.
TheworkwasfundedbythePublishingResearchConsortium.