Challenges for investigators –

generating reproducible research results

University of Minnesota, 3/2018

John P.A. Ioannidis, MD, DSc

C.F. Rehnborg Chair in Disease Prevention

Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics

Stanford University

Co-Director, Meta-Research Innovation Center at Stanford (METRICS)

Goodman, Fanelli, Ioannidis. Science Translational Medicine 2016

Different types of reproducibility

• Reproducibility of methods: the ability to understand or repeat as

exactly as possible the experimental and computational procedures.

• Reproducibility of results: the ability to produce corroborating

results in a new study, having followed the same experimental

methods.

• Reproducibility of inferences: the making of knowledge claims of

similar strength from a study replication.

Scientific discovery has become a boring nuisance: 96% of

the biomedical literature claims significant results (and the

vast majority of them say they are novel)

Chavalarias, Wallach, Li, Ioannidis, JAMA 2016

Typical recipe of research practices

• Solo, siloed investigator

• Small sample size studies

• Cherry-picking of one/best hypothesis

• Post-hoc

• P<0.05 is enough

• No registration

• No data sharing

• No replication

fMRI studies: only very small

studies discover a lot of foci

David et al, PLoS ONE 2013, Ioannidis et al Trends in Cognitive Sciences 2014

Szucs and Ioannidis, PLoS Biology 2017

Empirical studies on fields where

replication practices are common suggest

that most of the initially claimed

statistically significant effects in

observational associations are false

positives or substantially exaggerated

Candidate genes replicated through GWAS:

replication rate = 1.2%

Ioannidis, Tarone, McLaughlin, Epidemiology 2011

Prinz et al., Nature Reviews Drug Discovery 2011

Failed replication in preclinical

research

Replicated: only 6 of 53 landmark studies for

Amgen oncology drug target projects

• “The failure to win “the war on cancer”

has been blamed on many factors, … But

recently a new culprit has emerged: too

many basic scientific discoveries… are

wrong.”

Begley et al. Nature 2012

Begley and Ioannidis, Circulation Research 2015

Begley and Ioannidis, Circulation Research 2015

Potential solutions to improve

reproducibility

• Some solutions have already worked in specific fields and may need to be considered in other fields as well

• Other solutions are more speculative

• Empirical evidence as to their efficacy is needed

• Seemingly effective solutions may also have collateral damages

• Do no harm

Ioannidis, PLoS Medicine 2014

Large-scale collaboration and adoption of

replication culture

Replication – by whom?

• Same investigators

• Different investigators of the same school

• Different investigators of competing theories/hypotheses

• Combinations of the above

• Open to the wide public

Levels of registration

• Level 0: no registration

• Level 1: registration of dataset

• Level 2: registration of protocol

• Level 3: registration of analysis plan

• Level 4: registration of analysis plan and raw data

• Level 5: open live streaming

Sharing

data –

who,

when,

and how?

Doshi, Goodman,

Ioannidis, TiPS

2013

Science, December 2, 2016

Begley and Ioannidis, Circulation Research 2015

Modeling and modeling plus

experimentation

Grimes

et al,

JRSOS

2018

Re-engineering the reward system

Ioannidis and

Khoury, JAMA 2014

Understand and align interests of stakeholders

Concluding comments

• Reproducibility has different meanings across

different scientific disciplines but eventually all of

them aim at enhancing our trust in scientific findings

• There are many possible interventions that may

improve the efficiency of research practices and the

reproducibility of the scientific literature

• Empirical meta-research would be useful not only to

assess the prevalence of problems of reproducibility,

but also to assess the effectiveness and potential

harms of interventions that try to make research

more reproducible.

Daniele Fanelli

Steve Goodman

Shanil Ebrahim

Despina

Contopoulos-

Ioannidis

Georgia Salanti

Chirag Patel

Lars Hemkens

Ann Hsing

Lamberto Manzoli

Maria Elena Flacco

George Siontis

Denes Szucs

Kostas Siontis

Vangelis Evangelou

Kristin Sainani

Muin Khoury

Orestis Panagiotou

Florence Bourgeois

Special thanks

Joseph LauMalcolm MacLeodMarcus MunafoDavid AllisonJosh WallachFotini KarassaAthina TatsioniEvi NtzaniIoanna TzoulakiDemos KatritsisNikos PatsopoulosFainia KavvouraBrian NosekVictoria StoddenΕle ZegginiBelinda BurfordKostas TsilidisJodi Prochaska

Special thanks

Charitini StavropoulouEvropi Theodoratou Nikos PandisHuseyin NaciVanesa BellouΑntony DoufasLazaros BelbasisChris DoucouliagosStelios SerghiouAnna ChaimaniFotini ChatzinasiouStephania PapatheodorouFlorian NaudetTom HardwickePerrine JaniaudIoana-Alina CristeaShannon BrownleeVikas SainiMatthias EggerPatrick BossuytAndre UitterlindenDoug AltmanDeb ZarinKatherine Flegal

Special thanks

Shanthi Kapaggoda

Ewoud Schuit

Stefania Boccia

David Chavalarias

Jennifer Ware

Viswam Nair

Stephan Bruns

Dorothy Bishop

Tom Trikalinos

Kristina Sundquist

Johanna Int’hout

Kevin Boyack

Brett Thombs

Raj Manrai

Nazmus Saquib

Elizabeth Iorns

Abraham Verghese

Euan Ashley

Special thanks