Post on 09-Jun-2015
description
The state of scientific publishing and a new model
Dwight KravitzChris Baker
Motivation: Improving Cost and Quality
Budget per Paper ($1000)N
IH B
ud
get
/ #
Pap
ers
Pu
blis
hed
Peer Review: Outline
• History and evolution
• Modern purpose
• Improving peer review
• Post-publication
History of Peer Review and Publishing
• Problems facing authors– Disseminating results– Claiming ownership
of results
• Publishing– Publishers had
distribution channels– Provides a record of
when results were published
History of Peer Review and Publishing
• Problems facing publishers– Establishing veracity
of results– Deciding which results
to publish given limited space
Henry Oldenburg, 1665Medical Essays and Observations, 1731
• Peer review– Stamp of scientific
approval– Prioritizes papers for
inclusion by publishers
55% Reject42% Revise3% Accept
55% Reject42% Revise3% Accept
Given Revision:98% Accept2% Reject
64% of papers are rejected at least
once
Direct Costs of the Current System
• Total Number of Reviews: 6.3 (2-15)• Total Days under Review: 122 (21-
321)• Total Hours for Revisions: 68 (5-
300)• Total Days to Publish: 221 (21-
533)• Publication Fees: $2000 • Thousands of dollars and hundreds of hours
per paper• Long delays impair every form of assessment• Variance hinders planning research programs
and careers
Peer Review: Outline
• History and evolution– Invention– Current form
• Modern purpose
• Improving peer review
• Post-publication
Original Purpose
Henry Oldenburg, 1665Medical Essays and Observations, 1731
• Reasons for publishing– Disseminating results– Claiming ownership
of results• Reasons for peer review– Establishing veracity
of results– Deciding which results
to publish given limited space
Modern Purpose• Reasons for
publishing– Disseminating results– Claiming ownership
of results• Reasons for peer review– Establishing veracity
of results– Deciding which results
to publish given limited space
Abstract: 9495Full Text: 8564PDF: 2549
Pre-reception
• Peer review allows authors to test the paper before sending it to the whole field– Catches fundamental errors– Highlights important overlooked
issues
Problems with Pre-reception
• Reviews are impure measures of quality
Problems with Pre-reception
• Reviews are impure measures of quality
• Review process is opaque to the wider field
Prioritization of the Literature
• Loop ranks papers based on which journal publishes them– Provides a way of deciding which papers to
read– Compensates for long publication lags
Problems with Prioritization
• Reviews not focused solely on quality• Inefficient (avg. lag is 221 days)• Creates detrimental short-term
incentives
Problems with Prioritization
http://pmretract.heroku.com
Budget per Paper ($1000)
Problems with Prioritization
http://pmretract.heroku.com
Budget per Paper ($1000)Retractions per 100k Papers
Problems with Prioritization
Fang & Casadevall, 2011
Problems with Prioritization
• Reviews not focused on solely on quality
• Inefficient (avg. lag is 221 days)• Creates detrimental short-term
incentives• Doesn’t work
Problems with Prioritization
Problems with Prioritization
Problems with Prioritization
Problems with Prioritization
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review
• Post-publication
Improving the current system• Make publishing more efficient and less
variable• Days to Publish: 221 (21-
533)• Years to First Cite: ~1.5 +
research
• Capture reviews in the prioritization• Total Number of Reviews: 6.3• Total Hours of Reviewing: ~31.5
• Focus reviews on science rather than publication
Decouple peer review from publishing
Proposed System
Decoupling peer review from publishing
• Allows reviewers to focus on scientific merit rather than publication
• Allows reviews to be captured for prioritization and rewards for reviewers
• Makes publishing simpler and more efficient
• Eliminates the short-term incentive to produce least publishable units
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review– Decoupling from publishing
• Post-publication
Format for papers
The need for post-publication review
Field Altering
Useful Datapoint
Why?
Shrug
Pro
port
ion
of
field
Field Altering
Useful Datapoint
Why?
Shrug
The need for post-publication review
We need more reviews to approximate these complex distributions.
Pro
port
ion
of
field Also enables
personalized prioritization of the literature.
Incentivizing post-publication review
Organizing the
literature
• The utility of modern peer review and publishing is prioritizing and organizing the literature.
Organizing the
literature
• The utility of modern peer review and publishing is prioritizing and organizing the literature.
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review– Decoupling from publishing
• Post-publication review??????