1

Peering into review

Innovation, credit & reproducibility

Ciaran Oneill&

Amye Kenall

2

www.biomedcentral.com/biome

3

Pitfalls of traditional peer review

Inconsistent

Bias

Favouritism

Abuse

Burden on researchers

Slow

4

Open peer review

(Medical journals)

5

“our goal is unapologetically ambitious:

to establish a new system of peer review to bolster productive scientific debate

and to provide scientists with useful guides to the literature”

Launch Editorial: Eugene Koonin, David Lipman, Laura Landweber

6

7

~ 50% reviewers disclose their name

~ 80% authors make the reports public

8

9

Decoupling peer review from the journal

10

Post-publication peer review

11

Community review

• Post-publication commenting• Open to authors already in PubMed

12

13

14

I thought these were peer reviewed?Problems in reproducibility

15

1. Ioannidis et al., (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41: 142. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8)

Out of 18 microarray papers, resultsfrom 10 could not be reproducedSloppy Science

16

#overlyhonestmethods

17

How to combat this? ( . . . from the journal side)

18

Dynamic Document Technology

19

20

Journal + database + Computational Tools

21

Reproducibility Starts with Peer

Review

22

• Repository of standardised and annotated multielectrode array data from mice and ferrets

• 366 recordings from 12 studies

• Authors submitted in knitr• Aided review process,

allowing reviewers to rerun analyses

• Authors reported it saved time—having a “natural record” of what you did

• Automatic updating of text you might overlook (figure legends, eg)

23

Some testimonials for KnitrAuthors (Wolfgang Huber)“I do all my projects in Knitr. Having the textual explanation, the associated code and the results all in one place really increases productivity, and helps explaining my analyses to colleagues, or even just to my future self.”

Reviewers (Christophe Pouzat) “It took me a couple of hours to get the data, the few custom developed routines, the “vignette” and to REPRODUCE EXACTLY the analysis presented in the manuscript. With few more hours, I was able to modify the authors’ code to change their Fig. 4. In addition to making the presented research trustworthy, the reproducible research paradigm definitely makes the reviewer’s job much more fun!

24

How to Scale?

25

Back to #overlyhonestmethods

26

Let’s delegate!

27

28

Why stop at publication?

More commenting?

Bring debate back to the journal?

DOIs for comments?

29

Questions?Amye Kenall

Journal Development Manager (Open Data), BioMed Central@AmyeKenall

amye.kenall@biomedcentral.com

Ciaran O’NeillAssociate Publisher, BioMed Central

@cjmoneill ciaran.o’neill@biomedcentral.com