A replication crisis in the making: how we reward unreliable science
-
Upload
bjoern-brembs -
Category
Science
-
view
293 -
download
0
Transcript of A replication crisis in the making: how we reward unreliable science
![Page 1: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/1.jpg)
A replication crisis in the making: how we reward
unreliable science
Björn BrembsUniversität Regensburg - Neurogenetics
http://brembs.net - @brembs
![Page 2: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/2.jpg)
![Page 3: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/3.jpg)
![Page 4: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/4.jpg)
15th century: Adoration of the lamb
![Page 5: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/5.jpg)
21st century: Adoration of the Glam
![Page 6: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/6.jpg)
![Page 7: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/7.jpg)
![Page 8: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/8.jpg)
More and Better!
![Page 9: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/9.jpg)
Application Instructions
![Page 10: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/10.jpg)
Application Instructions
Publikationstätigkeit(vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in, Seniorautor/in, Impact-Punkte insgesamt und in den letzten 5 Jahren, darunter jeweils gesondert ausgewiesen als Erst- und Seniorautor/in, persönlicher Scientific Citations Index (SCI, h-Index nach Web of Science) über alle Arbeiten)
Publications:Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.
![Page 11: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/11.jpg)
Main Problems with the IF• Negotiable • Irreproducible • Mathematically
unsound
![Page 12: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/12.jpg)
![Page 13: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/13.jpg)
![Page 14: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/14.jpg)
![Page 15: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/15.jpg)
Negotiable
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
![Page 16: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/16.jpg)
![Page 17: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/17.jpg)
June, 2014 (19 months)
![Page 18: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/18.jpg)
Not Reproducible• Rockefeller University Press bought their data from Thomson Reuters• Up to 19% deviation from published records• Second dataset still not correct
Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091
![Page 19: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/19.jpg)
Not Mathematically Sound• Left-skewed distributions• Weak correlation of individual
article citation rate with journal IF
Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497
![Page 20: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/20.jpg)
Not Mathematically Sound
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
![Page 21: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/21.jpg)
‘QUALITY’
Is journal rank like astrology?
![Page 22: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/22.jpg)
Methodology I
Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273
![Page 23: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/23.jpg)
Methodology II
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
![Page 24: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/24.jpg)
Methodology III
Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77
![Page 25: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/25.jpg)
‘Quality’
Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847
![Page 26: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/26.jpg)
Excel Errors
DOI: 10.1186/s13059-016-1044-7 -omics studies
![Page 27: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/27.jpg)
p-Value Errors
Cog. Neurosci & PsychDOI: 10.1101/071530
![Page 28: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/28.jpg)
QUALITY?
“High-Impact” journals attract the most unreliable research
![Page 29: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/29.jpg)
![Page 30: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/30.jpg)
Statistical Power and Sample Size
Source: Daniel Lakens DOI: 10.1177/1745691614528520
![Page 31: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/31.jpg)
Statistical Power and Sample Size
Source: Daniel Lakens DOI: 10.1177/1745691614528520
![Page 32: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/32.jpg)
Statistical Power and Sample Size
Source: Daniel Lakens DOI: 10.1177/1745691614528520
![Page 33: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/33.jpg)
Statistical Power and Sample Size
![Page 34: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/34.jpg)
Productivity
Research questions:True:False:Significant:
20010010040
88444437
![Page 35: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/35.jpg)
Models
![Page 36: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/36.jpg)
Selecting for ‘productivity’
![Page 37: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/37.jpg)
Selecting for ‘novelty’
![Page 38: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/38.jpg)
PRODUCTIVITY?
“Publish-or-Perish” disadvantages meticulous scientists
![Page 39: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/39.jpg)
Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165
Irreproducibility
61%(n=100)
Open Science Collaboration
![Page 40: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/40.jpg)
INCENTIVES
Counting ‘Quality’ & Productivity => Selecting the sloppy scientists
![Page 41: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/41.jpg)
Scholarly Infrastructurean obscenely expensive anachronism
![Page 42: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/42.jpg)
PROBLEM I
Dysfunctional scholarly literature
![Page 43: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/43.jpg)
Antiquated Functionality• Limited access• Link-rot• No scientific impact analysis• Lousy peer-review • No global search• No functional hyperlinks• Useless data visualization• No submission standards• (Almost) no statistics• No content-mining• No effective way to sort,
filter and discover• No semantic enrichment• No networking feature• etc.
…it’s like the web in 1995!
![Page 44: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/44.jpg)
PROBLEM II
Scientific data in peril
![Page 45: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/45.jpg)
Small Data – Long Tail
Report on Integration of Data and Publications, ODE Report 2011http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications
![Page 46: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/46.jpg)
PROBLEM III
Non-existent software archives
![Page 47: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/47.jpg)
UNIVAC (A-2) 1953
![Page 48: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/48.jpg)
INFRASTRUCTURE
Antiquated and missing functionality
![Page 49: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/49.jpg)
“Publishers” parasitize public funds
Cost
s [th
ousa
nd U
S$/a
rticle
]
Legacy Modern(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)
(SciELOUbiquityScholasticaScienceOpenPeerJF1000ResearchFrontiersetc.)
![Page 50: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/50.jpg)
MONEY FOR NOTHING
Wasting billions on a parasitic industry
![Page 51: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/51.jpg)
SO MUCH FOR THAT
The disaster that is our scholarly infrastructure
![Page 52: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/52.jpg)
OPEN ACCESS ACTIVISM“Pretty please be open!”
![Page 53: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/53.jpg)
![Page 54: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/54.jpg)
![Page 55: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/55.jpg)
![Page 56: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/56.jpg)
![Page 57: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/57.jpg)
![Page 58: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/58.jpg)
![Page 59: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/59.jpg)
GitHub
![Page 60: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/60.jpg)
Scientific Code with Persistent Identifiers
![Page 61: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/61.jpg)
![Page 62: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/62.jpg)
![Page 63: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/63.jpg)
![Page 64: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/64.jpg)
![Page 65: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/65.jpg)
![Page 66: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/66.jpg)
openknowledgemaps.org
![Page 67: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/67.jpg)
575+ such solutions and counting…
![Page 68: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/68.jpg)
![Page 69: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/69.jpg)
Institutions are trying…
![Page 70: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/70.jpg)
The Department of Psychology embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open material, and pre-registrations. Candidates are asked to describe in what way they already pursued and plan to pursue these goals.
Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.
versus
![Page 71: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/71.jpg)
![Page 72: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/72.jpg)
APC-OA?
+70%
![Page 73: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/73.jpg)
APC-OA?
“The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.”
Emerald spokesperson
![Page 74: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/74.jpg)
WHAT NOW?
Save time and money (and make science open by default as an added benefit)
![Page 75: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/75.jpg)
![Page 76: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/76.jpg)
Potential for Innovation
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. doi:10.1038/495426a, Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)
Potential for innovation: 9.8b p.a.
Cost
s [th
ousa
nd U
S$/a
rticle
]
Legacy SciELO
![Page 77: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/77.jpg)
1. International Coordination
LEGAL
![Page 78: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/78.jpg)
![Page 79: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/79.jpg)
2. Cancel all subscriptions
![Page 80: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/80.jpg)
![Page 81: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/81.jpg)
3. Implement current technology
![Page 82: A replication crisis in the making: how we reward unreliable science](https://reader036.fdocuments.in/reader036/viewer/2022062503/58ecd1a01a28abd56b8b45f3/html5/thumbnails/82.jpg)
The square traversal process has been the foundation of scholarly
communication for nearly 400 years!