Is Open Science a solution or a threat? - Urfist de...
Transcript of Is Open Science a solution or a threat? - Urfist de...
Office of Scholarly Communication
Is ‘Open Science’ a solution or a threat?
Open science, transparence et evaluation. Perspectives et enjeux pour les chercheurs.
https://sygefor.reseau-urfist.fr/#!/training/6701/7159/?from=true
Urfist de Bordeaux, France
4 April 2017
Dr Danny KingsleyHead of Scholarly CommunicationUniversity of Cambridge@dannykay68Slides -XXXXXX
Today’s talk
• The problem
– The problems caused by the way research is measured
• The solution?
– How Open Science can address these
• The reality
– Why it is difficult to implement Open Science ideas
• I will be live tweeting – so all links to papers will be tweeted as we go #XXXX
The coin in the realm of academia
An
ub
is3
Med
al: G
ust
av V
igel
and
(Sel
f-p
ho
togr
aph
ed)
GFD
L, C
C-B
Y-SA
-3.0
, via
Wik
imed
ia C
om
mo
ns
Steele, C., Butler, L. and Kingsley, D. “The Publishing Imperative: the pervasive influence of publication metrics” Learned Publishing, October 2006 Vol 19, Issue 4, pp. 277-290. 10.1087/095315106778690751/epdf
The only thing that counts in academia is publication of novel results in high impact journals
We are stuck
Image by Danny Kingsley
The insistence on the need to publish novel results in high impact journals is creating a multitude of problems with the scientific endeavour
The problems
• Problem 1: Reluctance to share data– (all disciplines)
• Problem 2: Hyperauthorship– (Physics)
• Problem 3: Reproducibility – (Psychology, Neuroscience, Pharmacology)
• Problem 4: Retraction– (Biological and Medical Sciences)
• Problem 5: Poor Science– (Sociology, economics, climate science also vulnerable)
• Problem 6: Attrition– (all disciplines)
Focus today
• Problem 1: Reluctance to share data– (all disciplines)
• Problem 2: Hyperauthorship– (Physics)
• Problem 3: Reproducibility – (Psychology, Neuroscience, Pharmacology)
• Problem 4: Retraction– (Biological and Medical Sciences)
• Problem 5: Poor Science– (Sociology, economics, climate science also vulnerable)
• Problem 6: Attrition– (all disciplines)
Problem 1: Data Excuse Bingo
Data Excuse Bingo created by @jenny_molloy
My data
contains
personal/se
nsitive
information
My data is
too
complicated
People may
misinterpret
my data
My data is
not very
interesting
Commercial
funder
doesn’t want
to share it
We might
want to use
it in another
paper
People will
contact me
to ask about
stuff
Data
Protection/
National
Security
It’s too big
People will
see that my
data is bad
I want to
patent my
discovery
It’s not a
priority and
I’m busy
I don’t know
how
I’m not sure
I own the
data
Someone
might steal/
plagiarise it
My funder
doesn’t
require it
Incompatible!
Data Excuse Bingo created by @jenny_molloy
My data
contains
personal/se
nsitive
information
My data is
too
complicated
People may
misinterpret
my data
My data is
not very
interesting
Commercial
funder
doesn’t want
to share it
We might
want to use
it in another
paper
People will
contact me
to ask about
stuff
Data
Protection/
National
Security
It’s too big
People will
see that my
data is bad
I want to
patent my
discovery
It’s not a
priority and
I’m busy
I don’t know
how
I’m not sure
I own the
data
Someone
might steal/
plagiarise it
My funder
doesn’t
require it
‘Someone might steal/plagiarise it’
‘A second concern held by some is that a new class of research person will emerge — people who had nothing to do with the design and execution of the study but use another group’s data for their own ends, possibly stealing from the research productivity planned by the data gatherers, or even use the data to try to disprove what the original investigators had posited. There is concern among some front-line researchers that the system will be taken over by what some researchers have characterized as “research parasites.”’
EDITORIAL ‘Data Sharing’, Dan L. Longo, M.D., and Jeffrey M. Drazen, M.D. N Engl J Med 2016; 374:276-277January 21, 2016 DOI: 10.1056/NEJMe1516564
Solution – reward data sharing
• REgistry of REsearch Data REpositorieshttp://www.re3data.org/
• Joint Declaration of Data Citation Principles https://www.force11.org/group/joint-declaration-data-citation-principles-final
Problem 3: Reproducibility
Scientists are very rarely rewarded for being right, they are rewarded for publishing in certain journals and for getting grants.
Image by Danny Kingsley
The nine circles of scientific hell (with apologies to Dante and xkcd)
Neuroskeptic Perspectives on Psychological Science
2012;7:643-644
Copyright © by Association for Psychological Science
Oh dear
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
“Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.”
Reproducibility project
Conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. • Replication effects = half the
magnitude of original effects (substantial decline)
• 97% of original studies had significant results
• 36% of replications had significant results
https://osf.io/ezcuj/
Crisis?
Nature, 533, 452–454 (26 May 2016) doi:10.1038/533452a http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
Interest at highest level
• Research Integrity Enquiry
– UK Government Science and Technology Committee - Submissions closed 10 March 2017
– https://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technology-committee/inquiries/parliament-2015/inquiry6/
CC
cre
dit
Jim
Tro
del
Time for a change
‘Richard Smith: Another step towards the post-journal world’ BMJ blog, 12 Jul, 16 http://blogs.bmj.com/bmj/2016/07/12/richard-smith-another-step-towards-the-post-journal-world/
Imag
e b
y D
ann
y K
ings
ley
Solution – Open Science
• We need to change the way we reward researchers by distributing the dissemination of outputs across the research lifecycle
• We will hear more about reproducibility and open science later today
• I will be talking now about the challenges of implementing Open Science in institutions
Resources if you want to know more
• The Case for Open Research – series of blogs July & August 2016– https://unlockingresearch.blog.lib.cam.ac.uk/?page_id=2#OpenResear
ch
• My talk about the open argument – “Reward, reproducibility and recognition in research - the case
for going Open” Eleventh Annual Munin Conference on Scholarly Publishing, 21 November 2016
– Slides: http://www.slideshare.net/DannyKingsley/reward-reproducibility-and-recognition-in-research-the-case-forgoing-open
– Video: http://septentrio.uit.no/index.php/SCS/article/view/4036
• Useful slides and list of references – "Fake Results": The Reproducibility Crisis in Research and Open
Science Solutions http://digitalcommons.uri.edu/lib_ts_presentations/48/
The challenges of implementing Open Science
It is difficult to get ANY change in research institutions
Imag
e b
y D
ann
y K
ings
ley
We need institutions to play along
• “Improving the quality of research requires change at the institutional level”
• Smaldino PE, McElreath R. 2016 The natural selection of bad science. R. Soc. open sci.3: 160384. http://dx.doi.org/10.1098/rsos.160384
• “Universities and research institutes should play a major role in supporting an open data culture”
• Science as an open enterprise The Royal Society Science Policy Centre report 02/12 Issued: June 2012 DES24782https://royalsociety.org/~/media/policy/projects/sape/2012-06-20-saoe.pdf
Resistance
• Generally institutions are reluctant to step up, partly because of the governance structure.
• The nature of research itself is changing profoundly. This includes extraordinary dependence on data, and complexity requiring intermediate steps of data visualisation. These eResearch techniques have been growing rapidly, and in a way that may not be understood or well led by senior administrators.– “Openness, integrity & supporting researchers”
Emeritus Professor Tom Cochrane https://unlockingresearch.blog.lib.cam.ac.uk/?p=307
Governance
• These are big changes that need to be pushed through the system.
• This is particularly complicated at Cambridge
https://www.governance.cam.ac.uk/governance/key-bodies/Pages/default.aspx
Change is S-L-O-W
Academics at the 800-year-old institution have a unique role in the running of their university and, along with owning their own intellectual property rights, members of the university's Regent House can lobby for a vote on all amendments and additions to the university's governing rules.The ancient system of governance has come under attack in the past for being too cumbersome, and ill-designed for the 21st century. The university has come under pressure from government to reform its system of governance and intellectual property rights.
“Dons clash with Cambridge over intellectual rights”, The Guardian, 2005https://www.theguardian.com/education/2005/nov/22/highereducation.businessofresearch
Esteem economy
• Academia is an unusual economy – no payment for publishing, instead esteem
• The people and institutions who have succeeded have done so within the current ‘economy’
• If the way research is rewarded changes, then the winners might not be winners any more
Ch
ris
Pott
er /
CC
BY
Academia is tribal
• ‘Invisible colleges’ relate to the community people have with their discipline.
• This stuff sounds scary! If people have not experienced things themselves they don’t believe it
A whole other tribal system
htt
p:/
/ww
w.c
am.a
c.u
k/fo
r-st
aff/
feat
ure
s/co
llege
s-an
d-u
niv
ersi
ty-a
-co
mp
lex-
rela
tio
nsh
ip
The people who sit on all the committees and make decisions are academics. While they hold these posts, they are still individuals whose research is based in a particular discipline.
Is it our ‘place’?
htt
p:/
/ww
w.k
eep
calm
-o-m
atic
.co
.uk/
p/k
eep
-ca
lm-a
nd
-kn
ow
-yo
ur-
pla
ce-3
/
What is Scholarly Communication?
• Association of College and Research Libraries (ACRL) 2003 definition:– "the system through which research and other
scholarly writings are created, evaluated for quality, disseminated to the scholarly community, and preserved for future use. The system includes both formal means of communication, such as publication in peer-reviewed journals, and informal channels, such as electronic listservs.”
• http://acrl.libguides.com/scholcomm/toolkit/
• Often Scholarly Communication services are run out of libraries
What is the role of the library?
• Discussion at RLUK2017 conference.
– Are librarians support staff or research partners?
– Should we be collaborating and partnering with the research community?
– Should we be leading the University over these issues?
• See: “Become part of the research process –observations from RLUK2017”
– https://unlockingresearch.blog.lib.cam.ac.uk/
Yes we should be driving this agenda
• Scholarly Communication takes a ‘meta’ view of the research ecosystem
• Disciplinary differences mean individual researchers come to the table with very specific perspectives
• They all think they are right
• Very few understand that things are different in other disciplines – and that these are as valid as their own
• Scholarly Communication is a research discipline of its own. This is not recognised by most academics!
And then there is the administration
You Tube Cambridge in Numbers https://www.youtube.com/watch?v=FwZsb2CkMsM
This is not easy
• “Academic administrators that I’ve talked to are genuinely confused about how to update legacy tenure and promotion systems for the digital era. This book is an attempt to help make sense of all this.”
– https://www.insidehighered.com/news/2016/10/06/qa-authors-book-scholarship-digital-era
So what are the problems?
• Lack of perceived need from the academic community for scholarly communication support and advice
• Questions about whether it is appropriate for libraries to be driving this agenda through the institution
• Institutions are set up to maintain the status quo• Researchers think they know all about how the
research ecosystem works. (They mostly don’t.) – See: “The value of embracing unknown unknowns’’
https://unlockingresearch.blog.lib.cam.ac.uk/?p=594
Start at the beginning not the end
08
04
09
rev
olv
ing
do
or-
1 b
y D
an4
th N
ich
ola
s C
C-B
Y 2
.0
• Making data and other non traditional research outputs available is difficult
• We need to train our research community in how to research openly– “Is Democracy the Right System? Collaborative Approaches to
Building an Engaged RDM Community” (2017) http://biorxiv.org/content/early/2017/01/28/103895
A lot of persuading!!
• Academics – don’t believe you
– don’t necessarily think they need you
• Institutions – not always supportive
– designed not to change
• Libraries– don’t think this is their role
– having something of a crisis of purpose as we move to an open world
Some institutions are standing up
Stan
d o
ut
fro
m t
he
cro
wd
by
Stev
en
Dep
olo
Flic
kr L
icen
sed
Un
der
CC
BY
2.0
Leading the way
• Indiana University-Purdue University Indianapolis (IUPUI) –– Have included open access as a value in promotion and
tenure guidelines (2016)http://crln.acrl.org/content/77/7/322.full
• University of Liege– “[The university] linked internal assessment to the
scientific output stored in {repository] ORBi. Those applying for promotion have no choice but to file all their publications in full text.” (2011) http://openaccess.eprints.org/index.php?/archives/853-The-Liege-ORBi-model-Mandatory-policy-without-rights-retention-but-linked-to-assessment-procedures.html
Research underway
• OOO Canada Research Network “Motivating Open Practices Through Faculty Review and Promotion - 25 October 2016– http://www.ooocanada.ca/motivating_open_practices_rpt
• NIH “Including Preprints and Interim Research Products in NIH Applications and Reports” – 6 October 2016– https://grants.nih.gov/grants/guide/notice-files/NOT-OD-
17-006.html
• Report: Next-generation metrics: Responsible metrics and evaluation for open science.– https://ec.europa.eu/research/openscience/pdf/report.pd
f
Open can mean success
• McGill University's Montreal Neurological Institute and Hospital, Canada– First academic institution to adopt an open
science approach
– Institute has received considerable donations in the wake of this decision
– $20 million in January 2017 to establish the Tanenbaum Open Science Institutehttps://www.mcgill.ca/newsroom/channels/news/mcgill-university-announces-transformative-20-million-donation-montreal-neurological-institute-and-264838
Lots of work to be done
Image by Danny Kingsley
Questions/Discussion
Thanks!
Dr Danny Kingsley
Head of Scholarly Communication
University of Cambridge
@dannykay68