January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retractions, Scientific...

Life after publication:

Reproducibility challenges

and the role of publishersVéronique Kiermer, Executive Editor

PLOS

January 2016

Retractions, reproducibility and the

integrity of the scientific record

Overview

• Retractions:

• Challenges and limitations from an editorial perspective

• Irreproducibility:

• Underlying causes

• What can editors and publishers do?

Retractions

• Retraction of a research article is a complete and permanent removal from the scientific record

• Although an article remains accessible to readers, it should no longer be cited.

• Reasons for retraction:

• Conclusions cannot be relied upon and are no longer supported (invalid results)

• Serious breach of research or publication ethics

Retraction trends

Van Noorden, Nature 478, 26-28 (2011)

In same period, volume of papers increased by 44%

Why are papers retracted?

Van Noorden, Nature 478, 26-28 (2011)

Why are retraction notices opaque?

Ideal retraction (and correction) notices:• Transparent about scientific facts and responsibility• Distinguish misconduct from honest errors• Result from careful and unbiased investigation• Are published without delay

In reality: • Cases are incredibly complex• It takes time and resources to investigate• Misconduct finding requires the determination of intent.• Journals do not have the means and the authority to

investigate misconduct cases• Journals rely on institutional investigations• Occasional legal issues

Journals and Universities: two perspectives on the same problem

Journals

• Focus on the integrity of the

scientific record

• Seek to rapidly correct the

scientific facts

• Can request original data,

explanations, experiments

Universities

• Focus on determining

responsibility and guilt

• Seek to take corrective

actions

• Can gain access to

records, lab notebooks,

sequester evidence

September 2009Elizabeth Wager, Virginia Barbour, Steven Yentis, Sabine Kleinert on behalf of COPE

Council

“The main purpose of retractions is to correct the literature and ensure its integrity rather than

to punish authors who misbehave.”

RETRACTION GUIDELINES

(For authorship disputes occurring before publication, see the relevant COPE flowcharts. http://publicationethics.org/files/u2/04A_Author_Add_Submitted.pdf and http://publicationethics.org/files/u2/04B_Author_Remove_Submitted.pdf)

Can authors dissociate themselves from a retracted publication?

If retraction is due to the actions of some, but not all, authors of a publication, the notice of retraction should mention this. However, most editors consider that authorship entails some degree of joint responsibility for the integrity of the reported research so it is not appropriate for authors to dissociate themselves from a retracted publication even if they were not directly culpable of any misconduct.

Are there grounds for legal proceedings if an author sues a journal for retracting, or refusing to retract, a publication?

Authors who disagree with a retraction (or whose request to retract a publication is refused) sometimes threaten journal editors with legal action. Concern over litigation can make editors reluctant to retract articles, especially in the face of opposition from authors.

Journals’ instructions for authors should explain the retraction procedure and describe the circumstances under which articles might be retracted. This information should be incorporated (e.g. by references) into any publishing agreements and brought to the authors’ attention. However, even if the publishing agreement or journal instructions do not set out specific conditions for retraction, authors usually would not have grounds for taking legal action against a journal over the act of retraction if it follows a suitable investigation and proper procedures.

However, legal advice may be helpful to determine appropriate wording for a notice of retraction or expression of concern to ensure that these are not defamatory or libellous. Nevertheless, retraction notices should always mention the reason(s) for retraction to distinguish honest error from misconduct.

Whenever possible, editors should negotiate with authors and attempt to agree a form of wording that is clear and informative to readers and acceptable to all parties. If authors consent to the wording of a retraction statement, this provides defence against a libel claim. However, prolonged negotiations about wording should not be allowed to delay the publication of a retraction unreasonably and editors should publish retractions even if consensus cannot be reached.

WWW.PUBLICATIONETHICS.ORG

Process and best practices

• Retractions have their own DOI.

• Permanent bi-directional linking & clear marking of the paper – syndicate to indexers.

• For corrections, clear indication if the paper was republished

CrossRef industry-wide initiative to provide a standard way for readers to locate the most up-to-date version of an article.

PLOS has adopted

Irreproducibility

PloS Medicine 2005doi: 10.1371/journal.pmed.0020124

Nature 2012doi:10.1038/483531a

Discussing irreproducibility

ü We are not talking about fraud.

ü We acknowledge that reasonable conclusions derived from legitimate observations can be disproved by subsequent knowledge and technology advancements.

ü We distinguish: replication ≠ generalization …and we draw conclusions accordingly.

ü We must talk about and reduce irreproducibility due poor reporting, experimental and analytical design, …

META-RESEARCH ARTICLE

Reproducible Research Practices andTransparency across the BiomedicalLiteratureShareen A. Iqbal1☯, Joshua D. Wallach2,3☯, Muin J. Khoury4,5, Sheri D. Schully4, John P.A. Ioannidis2,3,6,7*

1 Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgia, UnitedStates of America, 2 Department of Health Research and Policy, Stanford School of Medicine, Palo Alto,California, United States of America, 3 Meta-Research Innovation Center at Stanford, Stanford University,Stanford, California, United States of America, 4 Division of Cancer Control and Population Sciences,National Cancer Institute, National Institutes of Health, Bethesda, Maryland, United States of America,5 Office of Public Health Genomics, Centers for Disease Control and Prevention, Atlanta, Georgia, UnitedStates of America, 6 Stanford Prevention Research Center, Department of Medicine, Stanford University,Stanford, California, United States of America, 7 Department of Statistics, Stanford University School ofHumanities and Sciences, Stanford, California, United States of America

☯ These authors contributed equally to this work.* [email protected]

AbstractThere is a growing movement to encourage reproducibility and transparency practices inthe scientific community, including public access to raw data and protocols, the conduct ofreplication studies, systematic integration of evidence in systematic reviews, and the docu-mentation of funding and potential conflicts of interest. In this survey, we assessed the cur-rent status of reproducibility and transparency addressing these indicators in a randomsample of 441 biomedical journal articles published in 2000–2014. Only one study provideda full protocol and none made all raw data directly available. Replication studies were rare(n = 4), and only 16 studies had their data included in a subsequent systematic review ormeta-analysis. The majority of studies did not mention anything about funding or conflicts ofinterest. The percentage of articles with no statement of conflict decreased substantiallybetween 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles report-ing statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0%in 2014) increased. Articles published in journals in the clinical medicine category versusother fields were almost twice as likely to not include any information on funding and to haveprivate funding. This study provides baseline data to compare future progress in improvingthese indicators in the scientific literature.

Author SummaryThere is increasing interest in the scientific community about whether published researchis transparent and reproducible. Lack of replication and non-transparency decreases the

PLOS Biology | DOI:10.1371/journal.pbio.1002333 January 4, 2016 1 / 13

OPEN ACCESS

Citation: Iqbal SA, Wallach JD, Khoury MJ, SchullySD, Ioannidis JPA (2016) Reproducible ResearchPractices and Transparency across the BiomedicalLiterature. PLoS Biol 14(1): e1002333. doi:10.1371/journal.pbio.1002333

Academic Editor: David L Vaux, Walter and ElizaHall Institute of Medical Research, AUSTRALIA

Received: October 13, 2015

Accepted: November 19, 2015

Published: January 4, 2016

Copyright: This is an open access article, free of allcopyright, and may be freely reproduced, distributed,transmitted, modified, built upon, or otherwise usedby anyone for any lawful purpose. The work is madeavailable under the Creative Commons CC0 publicdomain dedication.

Data Availability Statement: All relevant data arewithin the paper and its Supporting Information files.All authors had full access to all of the data (includingstatistical reports and tables) in the study and cantake responsibility for the integrity of the data andaccuracy of the data analysis.

Funding: The authors received no specific fundingfor this work. The Meta-Research Innovation Centerat Stanford (METRICS) is supported by a grant fromthe Laura and John Arnold Foundation.

Competing Interests: The authors have declaredthat no competing interests exist.

PLOS Biology | DOI: 10.137/journal.pbio.1002333 | Jan 4, 2016

Random sample: 441 biomedical journal articles 2000-20141 study provided a full protocolNone made all raw data directly availableReplication studies were rare (n=4)The majority of studies did not mention anything about funding or conflict of interest

Reporting

Experimental bias

Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment

PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12







Macleod et al., PLOS Biology | DOI: 10.1371/journal.pbio.1002273 | Oct 12, 2015



Randomisation Blinded assessment Competing interest

Random sample (n=146) describing in vivo research:Limited reporting of measures to reduce the risk of bias

Analytical bias

PERSPECTIVE

The Extent and Consequences of P-Hacking inScienceMegan L. Head1*, Luke Holman1, Rob Lanfear1,2, Andrew T. Kahn1, Michael D. Jennions1

1 Division of Evolution, Ecology and Genetics, Research School of Biology, Australian National University,Acton, Canberra, Australia, 2 Department of Biological Sciences, Faculty of Science, Macquarie University,North Ryde, New SouthWales, Australia

* [email protected]

AbstractA focus on novel, confirmatory, and statistically significant results leads to substantial biasin the scientific literature. One type of bias, known as “p-hacking,” occurs when researcherscollect or select data or statistical analyses until nonsignificant results become significant.Here, we use text-mining to demonstrate that p-hacking is widespread throughout science.We then illustrate how one can test for p-hacking when performing a meta-analysis andshow that, while p-hacking is probably common, its effect seems to be weak relative to thereal effect sizes being measured. This result suggests that p-hacking probably does notdrastically alter scientific consensuses drawn from meta-analyses.

IntroductionThere is increasing concern that many published results are false positives [1,2] (but see [3]).Many argue that current scientific practices create strong incentives to publish statisticallysignificant (i.e., “positive”) results, and there is good evidence that journals, especially presti-gious ones with higher impact factors, disproportionately publish statistically significant results[4–10]. Employers and funders often count papers and weigh them by the journal’s impactfactor to assess a researcher’s performance [11]. In combination, these factors create incentivesfor researchers to selectively pursue and selectively attempt to publish statistically significantresearch findings.

There are two widely recognized types of researcher-driven publication bias: selection(also known as the “file drawer effect”, where studies with nonsignificant results have lowerpublication rates [7]) and inflation [12]. Inflation bias, also known as “p-hacking” or “selectivereporting,” is the misreporting of true effect sizes in published studies (Box 1). It occurs whenresearchers try out several statistical analyses and/or data eligibility specifications and thenselectively report those that produce significant results [12–15]. Common practices that lead top-hacking include: conducting analyses midway through experiments to decide whether tocontinue collecting data [15,16]; recording many response variables and deciding which to re-port postanalysis [16,17], deciding whether to include or drop outliers postanalyses [16], ex-cluding, combining, or splitting treatment groups postanalysis [2], including or excludingcovariates postanalysis [14], and stopping data exploration if an analysis yields a significantp-value [18,19].

PLOS Biology | DOI:10.1371/journal.pbio.1002106 March 13, 2015 1 / 15

OPEN ACCESS

Citation: Head ML, Holman L, Lanfear R, Kahn AT,Jennions MD (2015) The Extent and Consequencesof P-Hacking in Science. PLoS Biol 13(3): e1002106.doi:10.1371/journal.pbio.1002106

Published: March 13, 2015

Copyright: © 2015 Head et al. This is an openaccess article distributed under the terms of theCreative Commons Attribution License, which permitsunrestricted use, distribution, and reproduction in anymedium, provided the original author and source arecredited.

Funding: Funding for this research was provided byAustralian Research Council Grants awarded to MDJ,RL and LH. The funders had no role in study design,data collection and analysis, decision to publish, orpreparation of the manuscript.

Competing Interests: The authors have declaredthat no competing interests exist.

Abbreviations: NHST, Null hypothesissignificance testing.

PLOS Biology | DOI: 10.1371/journal.pbio.1002106 | Mar 13, 2015

Underlying issues

• experimental design• statistics literacy• data presentation

• data management• reagents validity

• pressure to publish• publication bias • replications and

refutations not pursued

training laboratory managementleadership & mentoringsize of laboratories

infrastructureoversight and compliancequality assurance

incentives for rigor, professionalism and good laboratory leadership

REMEDIES

researcherlab PI

university &research institution

university &research institution

scientific community

ISSUES

What can journals do?

Raiseawareness

& promoteresearch

Joint initiatives

Promote reporting standards

PLOS requires adherence to multiple reporting standards:- CONSORT, TREND, STARD, STROBE- PRISMA- MIAMESupports specialist standards

PLOS data policy

PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception.

When submitting a manuscript online, authors must provide a Data Availability Statement describing compliance with PLOS's policy. If the article is accepted for publication, the data availability statement will be published as part of the final article.

Since March 3, 2014

GuidanceGeneral guidelines: http://journals.plos.org/plosone/s/data-availabilityplos

EDITORIAL

PLOS Genetics Data Sharing Policy: In Pursuitof Functional UtilityGregory S. Barsh1,2*, Gregory M. Cooper2, Gregory P. Copenhaver3, Greg Gibson4, MarkI. McCarthy5,6,7, Hua Tang1, Scott M. Williams8¤

1 Department of Genetics, Stanford University School of Medicine, Stanford, California, United States ofAmerica, 2 HudsonAlpha Institute for Biotechnology, Huntsville, Alabama, United States of America,3 Department of Biology, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, UnitedStates of America, 4 Center for Integrative Genomics, School of Biology, Georgia Institute of Technology,Atlanta, Georgia, United States of America, 5 Wellcome Trust Centre for Human Genetics, University ofOxford, Oxford, United Kingdom, 6 Oxford Centre for Diabetes, Endocrinology & Metabolism, University ofOxford, Oxford, United Kingdom, 7 Oxford NIHR Biomedical Research Centre, Churchill Hospital, Oxford,United Kingdom, 8 Department of Genetics, Geisel School of Medicine, Dartmouth College, Hanover, NewHampshire, United States of America

¤ Current address: CaseWestern Reserve University School of Medicine, Cleveland, Ohio, United States ofAmerica* [email protected]

About a year ago, PLOS implemented a new process intended to further the overarching prin-ciple that data used in the work we publish should be accessible and reusable. The motivationgoes hand-in-hand with both our open access ethos and the scientific method itself: the validityof a conclusion depends on the ability to reproduce the underlying results.

In theory, PLOS’ new data policy, in which “all data underlying the findings described[must be] fully available,” is not so new; in practice, though, it may be perceived as burden-some, complicated, and/or inefficient. The policy and its purpose has been discussed exten-sively among PLOS Genetics Editorial Board members with regard to its potential impact oneditors, authors, the community, and research subjects.

The purpose of this editorial is 2-fold: to acknowledge and discuss aspects of the “data-shar-ing” process that are especially challenging, and to provide additional clarification and guid-ance in the context of a few scenarios that are especially relevant, all from the perspective ofworking scientists who read, evaluate, and contribute to research based on genetics and geno-mics. We also suggest a way forward that builds on what already works at PLOS: consultingwith multiple stakeholders whose interests intersect to develop consensus.

An issue that is often problematic for PLOS Genetics authors is the sheer volume of datagenerated by large-scale phenotyping and/or genotyping studies. Whether from a confocalmicroscope, a radiofrequency detector in an MRI, or a CCD in a DNA sequencing instrument,processing, filtering, and compression of digital data is inherent across many areas of modernbiology. It is often neither practical nor wise to archive and distribute the primary output ofdigital detectors; indeed, the question of what constitutes “raw data” is a moving target.

A second problematic issue arises from ethical concerns associated with human researchsubjects. The potential to identify research subjects based on genomic information has receivedconsiderable attention and has fostered the development of controlled access mechanisms, inwhich researchers must seek approval from data access committees. As with any new set of reg-ulations, there is a risk of creating more problems than are solved; from an editorial perspec-tive, we recognize that authors must commit significant resources to deposit data into

PLOSGenetics | DOI:10.1371/journal.pgen.1005716 December 14, 2015 1 / 5

OPEN ACCESS

Citation: Barsh GS, Cooper GM, Copenhaver GP,Gibson G, McCarthy MI, Tang H, et al. (2015) PLOSGenetics Data Sharing Policy: In Pursuit of FunctionalUtility. PLoS Genet 11(12): e1005716. doi:10.1371/journal.pgen.1005716

Published: December 14, 2015

Copyright: © 2015 Barsh et al. This is an openaccess article distributed under the terms of theCreative Commons Attribution License, which permitsunrestricted use, distribution, and reproduction in anymedium, provided the original author and source arecredited.

Funding: No specific funding was received for thisarticle.

Competing Interests: GSB and GPC are Editors-in-Chief of PLOS Genetics. GG, HT, and SMW areSection Editors for the Natural Variation Section forPLOS Genetics. GMC is an Associate Editor andMIM is a former Associate Editor for PLOS Genetics.

PLOS External Data Advisory GroupPLOS Data Lead: Emma GanleyAcademic chair: Phil Bourne

Discipline specific guidelines:

On January 7, 2016, a coalition of publishers signed an Open Letter committing to start requiring ORCID IDs in 2016.

1. Implementing best practices for ORCID collection and auto-update of ORCID records upon publication

2. Require ORCID IDs for corresponding authors and encourage for co-authors

Credit and accountability for authors

A simple taxonomy of research contributions developed under the leadership of CASRAI and NISO.

- Includes but not limited to traditional authorship roles

- Makes contributions machine-readable and portable

- Meant to inspire development: Mozilla badges, VIVO-ISF ontology, JATS integration, ORCID integration

Editorial and peer review evaluation

Editorial office:• Trial registration• Data deposition• Reporting guidelines• Ethical approval• Competing interests• Financial disclosures• Permissions• Plagiarism• Image integrity

Peer reviewers:• Methodology and

experimental design• Analysis• Statistics• Conclusions• Ethics

Limitations: - Science has become more cross-disciplinary- Confidential peer review can show biases- Paper should live on after publication- Still not enough incentives to publish all results

Peer review innovations

Missing pieces:

• Publication of all valid,

technically sound

research

• Transparent reporting

• Data, code and reagent

sharing

• Robust post-publication

discussions among

community

• Credit for all contributions

Thank you for listening!

[email protected]

January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retractions, Scientific...

Education

Transcript of January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retractions, Scientific...