open access, p-hacking, registered reports, registered replication reports
-
Upload
alex-holcombe -
Category
Science
-
view
422 -
download
5
Transcript of open access, p-hacking, registered reports, registered replication reports
![Page 1: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/1.jpg)
Macquarie University 4 Dec 2015
A new era of large-scale replication and collaboration?
@ceptional
http://www.slideshare.net/holcombea/
• Open access
• P-hacking
• Preregistration, Registered Reports
• Open data, open methods
• Registered Replication Reports
![Page 2: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/2.jpg)
Macquarie University 4 Dec 2015 2
![Page 3: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/3.jpg)
Macquarie University 4 Dec 2015 3
Open access - Research funders will mandate it. Will we scholars shape the system of the future, or passively let publishers do it in their own interest?
• Green or Gold
• Who pays?
• Institution, granting agency, library (from subscription savings)
• Everything technically adequate should be published
• “contribution to the field” is subjective
• Competition for Nature on down wasteful, distorting
• Frontiers? PeerJ? Ubiquity Press?
![Page 4: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/4.jpg)
Macquarie University 4 Dec 2015 4
Our average base Article Processing Charge (APC) is only £300 (€375, $500) and this is the only thing we charge.
![Page 5: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/5.jpg)
5Slide courtesy Chris Chambers (Cardiff)
Science publishing has an incentive problem
![Page 6: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/6.jpg)
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put your money on ‘will replicate’?”
"What, you think I was born yesterday?”
"Fine," muttered the bookie.
#AcademicNoir
![Page 7: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/7.jpg)
Macquarie University 4 Dec 2015
P-hackingSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-
positive psychology undisclosed flexibility in data collection and
analysis allows presenting anything as significant. Psychological
Science
![Page 8: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/8.jpg)
Macquarie University 4 Dec 2015
False positive zones - researcher degrees of freedom high (p-hacking)
- victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant.
Where science is not self-correcting
The newer the domain, the more cutting-edge the technology, the less-defined the expected analyses, and the greater the opportunities for p-hacking
![Page 9: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/9.jpg)
Macquarie University 4 Dec 2015
courtesy Chris Chambers (Cardiff)
![Page 10: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/10.jpg)
Macquarie University 4 Dec 2015
Well, Mr. Hacker, your luck is about to run out. Down at the journals, they’re running a new game. They call it ‘preregistration’.
CC-BY Alex Holcombehttps://twitter.com/ceptional/status/663850868396044288
![Page 11: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/11.jpg)
Macquarie University 4 Dec 201511
courtesy Chris Chambers (Cardiff)
![Page 12: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/12.jpg)
Macquarie University 4 Dec 201512
courtesy Chris Chambers (Cardiff)
![Page 13: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/13.jpg)
Macquarie University 4 Dec 201513
https://osf.io/tvyxz/wiki/home/
![Page 14: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/14.jpg)
Macquarie University 4 Dec 201514
https://osf.io/tvyxz/wiki/home/
![Page 15: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/15.jpg)
Macquarie University 4 Dec 201515
https://osf.io/tvyxz/wiki/home/
100% P-hacking FreeHere, check our numbers.
Here’s how you can replicate our result.
The badges that are fixing science
![Page 16: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/16.jpg)
Macquarie University 4 Dec 2015
"This one's Psych Science. 1998. Put your money on ‘will replicate’?”
"What, you think I was born yesterday?”
"Fine," muttered the bookie.
![Page 17: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/17.jpg)
Macquarie University 4 Dec 2015 17
Failed to replicate my result?
Have you considered
the unknown moderators?
![Page 18: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/18.jpg)
Macquarie University 4 Dec 2015 18
Any one of us alone, he’ll say we messed up. But if we all run the replication together, we
can show the big guy we’re right.
![Page 19: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/19.jpg)
Macquarie University 4 Dec 2015
Meta-analysis forest plot
![Page 20: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/20.jpg)
Macquarie University 4 Dec 2015
Meta-analysis forest plot
Failed to replicate my result?
Have you considered the unknown moderators?
You messed up!
![Page 21: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/21.jpg)
Macquarie University 4 Dec 2015
![Page 22: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/22.jpg)
Macquarie University 4 Dec 2015
![Page 23: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/23.jpg)
Macquarie University 4 Dec 2015
![Page 24: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/24.jpg)
Macquarie University 4 Dec 2015
Registered studiesLarge-scale replicationOpen dataOpen protocol and code
A new era of large-scale replication
and collaboration?
![Page 25: open access, p-hacking, registered reports, registered replication reports](https://reader034.fdocuments.in/reader034/viewer/2022051404/58efb2de1a28ab39138b4577/html5/thumbnails/25.jpg)
Macquarie University 4 Dec 2015