Open Access and the Replicability Crisis

[email protected] of Psychology

@ceptional

The broadest problem in science:Our publishing system

http://www.slideshare.net/holcombea/

1

Academic knowledge is boxed in by expensive journals.

Scientist meets Publisher

http://www.youtube.com/watch?v=GMIY_4t-DR0

2

Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html

$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE

operating profit7% Woolworths12% BMW22% Coca-Cola23% Rio Tinto36% Apple34% Springer36% Elsevier42% Wiley

Thanks to Nick Scott-Samuel

3

$10,780 per article (not including charges for color figures)

$85 per page

$80 per page (introductory rate is even cheaper)

$1350 per article

JOURNAL / PUBLISHER COST ($USD)

$99 per life

4

“Open Access Hulk”

OA HULK WANTS TO KNOW WHO TO OCCUPY!

ELSEVIER!? ACS!? HARPERCOLLINS!?

YOU NAME IT, OA HULK WILL OCCUPY AND SMASH!

6

Monopoly + = = $Profit maximization

7

Monopoly + = = $Profit maximization

8

Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html

$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE

9

•Deposit your manuscripts in the university repository (http://ses.library.usyd.edu.au/

•Even with closed journals, you often have the right to deposit your final version (e.g. Word document before typeset by publisher)

•Funders, universities should mandate this.

•Publishers will adapt, as they have in physics.

GREEN ROAD

Stevan Harnad

I have the sense that researchers are under awful pressures to present two faces to the world, sometimes alternately, and sometimes simultaneously. On the one hand, they are supposed to be impartial, with no axe to grind. On the other hand, they are supposed to be, not only experts in experimental methodology, higher ed pedagogy, grant-writing, and the like, but also formulating strong a priori hypotheses that turn out to be robustly supported by evidence and, all other things being equal, happen to satisfy the interests of funding sources and journal editors. In other words, they are supposed to be excellent axe-grinders.

GOLD ROAD

The more agencies misunderstand the green road to open access, the more gold they lose

Article Processing Charge

$3,000

10

•Requirements from funders that publications be OA

•NIH (US) within 12 months

•Wellcome Trust (UK) within 6 months

•final grant payment withheld if you don’t comply

•NHMRC (Australia) within 12 months

•“publications arising from an NHMRC supported research project must be deposited into an open access institutional repository within a twelve month period from the date of publication.”

•ARC (Australia)

•You can use DP funds to pay open-access fees, but must be taken from the funds you were awarded to pay for other things.

•“Strongly encourages” open access, no teeth. Compliance rate very low.

Small steps

11

•Deposit your manuscripts in the university repository (http://ses.library.usyd.edu.au/

•Even with closed journals, you often have the right to deposit your final version (e.g. Word document before typeset by publisher)

•Funders, universities should mandate this.

•Publishers will adapt, as they have in physics.

GREEN ROAD

Stevan Harnad

I have the sense that researchers are under awful pressures to present two faces to the world, sometimes alternately, and sometimes simultaneously. On the one hand, they are supposed to be impartial, with no axe to grind. On the other hand, they are supposed to be, not only experts in experimental methodology, higher ed pedagogy, grant-writing, and the like, but also formulating strong a priori hypotheses that turn out to be robustly supported by evidence and, all other things being equal, happen to satisfy the interests of funding sources and journal editors. In other words, they are supposed to be excellent axe-grinders.

GOLD ROAD

The more agencies misunderstand the green road to open access, the more gold they lose

Article Processing Charge

$3,000

UK hybrid favours gold

researchers to choose gold OA payment over cost-free green OA wherever the former is offered. The result, of course, will be that all journals blithely offer hybrid gold OA

As a result, rather than reducing costs, Finch estimated that its proposal would require £38m/year of to pay APCs

12

Open Data: the next step

NHMRC: The next steps will be improving public and other researchers’ access to publicly funded data.

https://theconversation.edu.au/all-research-funded-by-nhmrc-to-be-accessible-free-of-charge-5486

13

The Replicability Crisis

Rule among early-stage venture capital firms that “at least 50% of published studies, even those in top-tier academic journals, can't be repeated with the same conclusions by an industrial lab” - Prinz, Schlange, & Asadullah. Nature Rev. Drug Discov. 10, 712 (2011)

Bayer HealthCare :only about 25% of published preclinical studies could be validated to the point at which projects could continue

Amgen Fiy-three papers were deemed ‘landmark’ studies (see ‘Reproducibility of research "ndings’)... scienti"c "ndings were con"rmed in only 6 (11%) cases

Why science is self-correcting

There's no point in scientific misconduct; it is always found.

Published on August 10, 2010 by Art Markman, Ph.D. in Ulterior Motives

Because scientists are always repeating each other's experiments, it is hard for a

fictitious result to hang on for very long.

14

The File-Drawer Problem

unpublished results

files

http://www.flickr.com/photos/nickperez/2569423078 t. magnum

15

The File-Drawer Problem•Difficult to publish non-replications and replications

•Most journals only publish papers that “make a novel contribution”

•Reviewers/editors tend to hold non-replicating manuscript to higher standard than original.

•Bem

•Little career incentive to publish a non-replication or a replication


unpublished results

files

16

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes ina scientific field, the less likely the research findings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

The File-Drawer Problem


“In summary, while we agree with Ioannidis that most

research findings are false...”

17

Barriers to publishing replications and failed-replications

• No glory in publishing a replication

• Few journals publish replications

• usually uphill battle even with those that do

• The wrath of the original researcher

18

File-drawer fixes

• Journals that don’t reject replications for being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

•◦ ◦

◦◦

19

• problems: incentives

http://psychfiledrawer.org/view_article_list.php

20


Pashler, Spellman,

Holcombe& Kang (2011)

DETAILS page: http://psychfiledrawer.org/replication.php?attempt=MTU%3D

21


http://psychfiledrawer.org/view_article_list.php

22

File-drawer fixes

• Journals that don’t reject replications for being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

◦ ◦

◦◦

✔

23

Registered Evidence in Psychological Science

1. Authors plan a replication study, exact not conceptual

2. They submit an introduction and methods section

3. It is sent to reviewers, including the targeted author

4. The editor decides whether to accept/reject, based on:

1. Reviewer comments regarding the proposed protocol

2. Importance of the study, judged by argument in the introduction, number of citations of original, reviewer comments

5. The Intro, Method and analysis plan, and reviewer comments are posted on the journal website

6. When the results come in, the authors write a conventional results and discussion section and that together with the raw data are posted, yielding the complete publication

1. some sort of minimal peer review needed for that. What exactly?

✔

J-REPS

Dan Simons

✔✔

24

Registered Evidence in Psychological Science

• Original author sort-of signed off on it, so can’t complain / hate the replication authors as much.

• Good way to start for a new PhD project, anyone planning to build on some already-published results

• Will post the raw data

• Will facilitate, publish meta-analyses when replications accrue

• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?

✔✔✔

25

Open Access and the Replicability Crisis

Documents

Transcript of Open Access and the Replicability Crisis