Sense and Reproducibility: the problem of translating academic discovery to drug discovery Panelists•Ira Mellman (chair): VP of Research Oncology, Genentech; Prof of Biochemistry & Biophysics UCSF; former JCB Editor in Chief

•C. Glenn Begley: former VP of Global Oncology, Amgen

•Elizabeth Iorns: CEO, Science Exchange (Reproducibility Initiative)

The problem: biotech/pharma scientists have found it difficult to reproduce published work from academic groupsPrinz et al (2011) Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discovery 10:712 [Bayer]

Begley & Ellis (2012) Drug development: raise standards for preclinical cancer research. Nature 483:531 [Amgen]

Why was this discovered?

Industry and academia have different near-term goals:

Publication of interesting work that drives a field forward

vs Verification of published observations to justify

long-term, expensive drug discovery efforts

Questions to be addressed by the panel:• What is the nature of the reproducibility

problem?• Poor scientific/analytic quality?• Poor quality of the validation effort?• Generalizability vs bad science?

• How widespread is it?• Why has it occurred?• Problems are complex and difficult to

reproduce?• Corners are cut in the rush to publish?• Inaccurate data representation or analysis?

• What can we do about it?• Nothing?• Motivate higher standards?• Vigilantism?• Institutionalized data verification (Elizabeth

Iorns)• Journals set higher standards for

editing/data display?

The JCB experience:

• Since 2002, figures for all accepted manuscripts screened for inappropriate image manipulation (micrographs, gels)

• 10% of papers found to contain one or more examples

• 10% of these (1% overall) rejected after determination that manipulation fraudulently altered a key conclusion

• Frequencies have not changed in 10 years Issues:Desire to make data look “optimal”?Digital manipulation is easy to do?Cultural acceptance of digital manipulation

Sense and Reproducibility: the problem of translating academic discovery to drug discovery Panelists•Ira Mellman (chair): VP of Research Oncology, Genentech; Prof of Biochemistry & Biophysics UCSF; former JCB Editor in Chief

•C. Glenn Begley: former VP of Global Oncology, Amgen

•Elizabeth Iorns: CEO, Science Exchange (Reproducibility Initiative)

Discussion questions:• Is the reproducibility issue a new problem?

• Why is so much work apparently not reproducible?

• What should we do about it as a community?

• Will initiatives like Science Exchange have an impact?

• How can we guard against spurious claims?

• What is the role of journals and reviewers?

• What steps can we as individual scientists take to maximize the chances that our work can be reproduced?