What is Reproducibility?
The R* brouhaha(and how Research Objects can help)
Professor Carole GobleThe University of Manchester, UKSoftware Sustainability Institute, UKELIXIR-UK, FAIRDOM Association [email protected] International Workshop on Reproducible Open Science @ TPDL, 9 Sept 2016, Hannover, Germany
Acknowledgements• Dagstuhl Seminar 16041 , January 2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041• ATI Symposium Reproducibility, Sustainability and Preservation , April
2016– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/– https://osf.io/bcef5/files/
• C Titus Brown• Juliana Freire• David De Roure• Stian Soiland-Reyes• Barend Mons• Tim Clark• Daniel Garijo• Norman Morrison
“When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.”
Carroll, Through the Looking Glass
re-compute
replicatererun
repeat
re-examine
repurpose
recreate
reuse
restorereconstruct review
regeneraterevise
recycle
redo
robustness tolerance
verification compliance validation assurance
remix
Reproducibility of Reproducibility Research
Computational Science
http://tpeterka.github.io/maui-project/From: The Future of Scientific Workflows, Report of DOE Workshop 2015, http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd
1. Observational, experimental
2. Theoretical3. Simulation4. Data
intensive
BioSTIF
Computational Science
Scientific publications goals: (i) announce a result(ii) convince readers its correct.
Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension.
Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures.
Virtual Witnessing*
*Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985) Shapin and Schaffer.
Jill Mesirov
David Donoho
Datasets, Data collectionsStandard operating proceduresSoftware, algorithmsConfigurations, Tools and apps, services
Slide
share
Github
figsh
are
Commun
ityDB
Arxiv.o
rg
Pubm
ed
Docke
rim
age
Codes, code librariesWorkflows, scriptsSystem software Infrastructure Compilers, hardware
Systems of SystemsHeterogeneous hybrid patchwork of tools and service evolving over time
10 “Simple” Rules for Reproducible Computational
Research: RACE1. For Every Result, Keep Track of How It
Was Produced2. Avoid Manual Data Manipulation Steps3. Archive the Exact Versions of All
External Programs Used4. Version Control All Custom Scripts5. Record All Intermediate Results, When
Possible in Standardized Formats6. For Analyses That Include Randomness,
Note Underlying Random Seeds7. Always Store Raw Data behind Plots8. Generate Hierarchical Analysis Output,
Allowing Layers of Increasing Detail to Be Inspected
9. Connect Textual Statements to Underlying Results
10.Provide Public Access to Scripts, Runs, and Results
Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285
Record Everything
Automate Everything
Contain Everything
ExposeEverything
Preparation painindependent testing trials and
tribulations
[Norman Morrison]
replication hostility no funding, time, recognition, place to publishresource intensive access to the complete environment
Lab Analogy: Witnessing “Datascopes”
Input Data
Software
Output Data
ConfigParameters
Methodstechniques, algorithms, spec. of the steps, models
Materialsdatasets, parameters, algorithm seedsExperim
ent
Instrumentscodes, services, scripts, underlying libraries, workflows, , ref resourcesLaboratory
sw and hw infrastructure, systems software, integrative platformscomputational environment
Setup
“Micro” Reproducibility
“Macro” Reproducibility
Fixivity
Validate
Verify
Trust
Repeat, Replicate, Robust
[C Titus Brown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
Why the differences?
Reproduce, Trust
“an experiment is reproducible until another
laboratory tries to repeat it” Alexander Kohn
Repeatability:“Sameness”Same result1 Lab1 experiment
Reproducibility:“Similarity”Similar result> 1 Lab> 1 experiment
Validate
Verify
Method Reproducibility
the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.
Result Reproducibility (aka replicability)
obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible
What does research reproducibility mean? Steven N. Goodman, Daniele Fanelli, John P. A. Ioannidis Science Translational Medicine 8 (341), 341ps12. [doi: 10.1126/scitranslmed.aaf5027] http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf
ProductivityTrack differences
Validate
Verify
reviewers want additional workstatistician wants more runsanalysis needs to be repeatedpost-doc leaves, student arrivesnew/revised datasetsupdated/new versions of algorithms/codessample was contaminatedbetter kit - longer simulationsnew partners, new projects
Personal & Lab
Productivity
Public GoodReproducibili
ty
“Datascope” Lab Analogy
Methodstechniques, algorithms, spec. of the steps, models
Materialsdatasets, parameters, algorithm seedsExperim
ent
Instrumentscodes, services, scripts, underlying libraries, workflows, ref datasets
Laboratorysw and hw infrastructure, systems software, integrative platformscomputational environment
Setup
“Datascope” Lab Analogy
Methodstechniques, algorithms, spec. of the steps, models
Materialsdatasets, parameters, algorithm seedsExperim
ent
Instrumentscodes, services, scripts, underlying libraries, workflows, ref datasets
Laboratorysw and hw infrastructure, systems software, integrative platformscomputational environment
Setup Form
Function
“Datascope” Practicalities
Methodstechniques, algorithms, spec. of the steps, models
Materialsdatasets, parameters, algorithm seeds
Experim
ent
Instrumentscodes, services, scripts, underlying libraries, workflows, ref datasets
Laboratorysw and hw infrastructure, systems software, integrative platformscomputational environment
Setup
Living DependenciesScience, methods, datasetsquestions stay, answers change
breakage, labs decay, services and techniques come and go, new instruments, updated datasets, services, codes, hardware
One offs, streams,stochastics, sensitivities,scale, non-portable datablack boxes
supercomputer accessnon-portable softwarelicensing restrictionsunreliable resourcesblack boxescomplexity
T1 T2
evolving ref datasets,new simulation codes
EnvironmentArchived vs Active
Contained vs DistributedRegimented vs Free-for-
allWho owns the dependencies?
Dependencies -> Manage
Black boxes -> Expose
Dynamics -> FixityReliability
Replicate harder than Reproduce?
Repeating the experiment or the set up?
Container Conundrum Results will Vary
Replicability WindowAll experiments become less replicable over
timePrepare to repair
Levels of Computational Reproducibility
Coverage: how much of an experiment is reproducible
Orig
inal
Exp
erim
ent S
imila
r Exp
erim
ent Di
ffere
nt E
xper
imen
tPo
rtabi
lity
Depth: how much of an experiment is available
Binaries + Data
Source Code / Workflow+ Data
Binaries + Data + Dependencies
Source Code / Workflow+ Data + Dependencies
Virtual MachineBinaries + Data + Dependencies
Virtual MachineSource Code / Workflow+ Data + Dependencies
Figures + Data
[Freire, 2014]
Minimum: data and source code available under terms that permit inspection and execution.
Measuring Information Gain from Reproducibility
Research goal
Method/Alg.
Platform/Exec Env
Data Parameters
Input data
Actors
Information Gain Consistency
Robustness/S
ensitivity
Generality
Portability/Adoption 1
Portability/Adoption 2
Independent validation
Repurposability
Implementation/Code
No changeChangeDon’t care
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
http://www.dagstuhl.de/16041
How? Preserve by Reporting, Reproduce by Reading
Archived Record
Description Zoostandards, common metadata
How? Preserve by Maintaining, Repairing, ContainingReproduce by Running, Emulating, Reconstructing
Active Instrument Byte level Buildability Zoo
provenance
portability, preservation
robustness, versioning
access descriptionstandards
common APIslicensing, identifiers
standards,common metadata
change variation sensitivity
discrepancy handling
packaging, containers
FAIR RACE Reproducibility Dimensions
dependenciessteps
Research ObjectStandards-based metadata framework for logically and physically bundling resources with context,
http://researchobject.org
Bigger on the inside than the outsideexternal referencing
Manifest Constructi
on
Aggregates link things
togetherAnnotations
about things & their
relationships
Container
Research Object Standards-based metadata framework for logically and physically bundling resources with context, http://researchobject.org
Packaging content & links: Zip files, BagIt, Docker
images
Catalogues & Commons Platforms: FAIRDOM
Manifest Descripti
onDependencies
what else is needed
Versioning its evolution
Checklists what should be there
Provenance
where it came from
Identificationlocate things
regardless whereid
Systems Biology Commons• Link data,
models and SOPs
• Standards• Span resources• Snapshot +
DOIs• Bundle and
export• Logical bundles
Belhajjame et al (2015) Using a suite of ontologies for preserving workflow-centric research objects, J Web Semantics doi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
Workflow Research Objects exchange, portability and maintenance
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
Asthma Research e-Lab
Dataset building and releasing
Standardised packing of Systems Biology models
European Space Agency RO Library
Large dataset management for life science workflows
LHC ATLAS experiments
Notre Dame U Rostock
Encyclopedia of DNA Elements
PeptideAtlas
Words matter.
Reproducibility is not a end. Its a means to an end.Beware reproducibility
zealots.
50 Shades of Reproducibility.form vs function
A conundrum: big co-operative data-driven science makes
reproducibility desirable but also means
dependency and change are to be expected.
Lab analogy for computational
science
Bonus Slides
Top Related