Post on 14-Feb-2017
WhatisReproducibility?TheR*brouhaha(andhowResearchObjectscanhelp)
ProfessorCaroleGobleTheUniversityofManchester,UKSoftwareSustainabilityInstitute,UKELIXIR-UK,FAIRDOMAssociatione.V.carole.goble@manchester.ac.uk
SciReproWorkshop,TPDL,9Sept2016,Hannover,Germany
Acknowledgements• DagstuhlSeminar16041,January2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041
• ATISymposiumReproducibility,SustainabilityandPreservation,April2016– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/– https://osf.io/bcef5/files/
• CTitusBrown• JulianaFreire• DavidDeRoure• StianSoiland-Reyes• BarendMons• TimClark• DanielGarijo• NormanMorrison
“WhenIuseaword,"HumptyDumptysaidinratherascornfultone,"itmeansjustwhatIchooseittomean-neithermorenorless.”
Carroll,ThroughtheLookingGlass
re-compute
replicatererun
repeat
re-examine
repurpose
recreate
reuse
restore reconstruct review
regeneraterevise
recycle
redo
robustness tolerance
verifica(oncompliancevalida(onassurance
remix
ReproducibilityofReproducibilityResearch
ComputationalScience
http://tpeterka.github.io/maui-project/
From:TheFutureofScientificWorkflows,ReportofDOEWorkshop2015,http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd
1. Observational,experimental
2. Theoretical3. Simulation4. Dataintensive
BioSTIF
ComputationalScience
Scientificpublicationsgoals:(i)announcearesult(ii)convincereadersitscorrect.
Papersinexperimentalscienceshoulddescribetheresultsandprovideaclearenoughprotocoltoallowsuccessfulrepetitionandextension.
Papersincomputationalscienceshoulddescribetheresultsandprovidethecompletesoftwaredevelopmentenvironment,dataandsetofinstructionswhichgeneratedthefigures.
VirtualWitnessing*
*LeviathanandtheAir-Pump:Hobbes,Boyle,andtheExperimentalLife(1985)ShapinandSchaffer.
JillMesirov
DavidDonoho
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
10“Simple”RulesforReproducibleComputationalResearch:RACE
1. ForEveryResult,KeepTrackofHowItWasProduced
2. AvoidManualDataManipulationSteps3. ArchivetheExactVersionsofAllExternal
ProgramsUsed4. VersionControlAllCustomScripts5. RecordAllIntermediateResults,WhenPossiblein
StandardizedFormats6. ForAnalysesThatIncludeRandomness,Note
UnderlyingRandomSeeds7. AlwaysStoreRawDatabehindPlots8. GenerateHierarchicalAnalysisOutput,Allowing
LayersofIncreasingDetailtoBeInspected9. ConnectTextualStatementstoUnderlying
Results10. ProvidePublicAccesstoScripts,Runs,and
Results
SandveGK,NekrutenkoA,TaylorJ,HovigE(2013)TenSimpleRulesforReproducibleComputationalResearch.PLoSComputBiol9(10):e1003285.doi:10.1371/journal.pcbi.1003285
RecordEverythingAutomateEverythingContainEverythingExposeEverything
Preparationpainindependenttestingtrialsandtribulations
[NormanMorrison]
replicationhostilitynofunding,time,recognition,placetopublishresourceintensiveaccesstothecompleteenvironment
Reproducibility=HardWork
Data sets
Analyses
Open-Paper
Open-Review
DOI:10.1186/2047-217X-1-18 >11000 accesses
Open-Code
8 reviewers tested data in ftp server & named reports published
DOI:10.5524/100044
Open-PipelinesOpen-Workflows
DOI:10.5524/100038 Open-Data
78GB CC0 data
Code in sourceforge under GPLv3: http://soapdenovo2.sourceforge.net/
>5000 downloads
Enabled code to being picked apart by bloggers in wiki http://homolog.us/wiki/index.php?title=SOAPdenovo2
[Scott Edmunds]
WhatisreproducibilityItshardsoletsbesurewearedoingtherightthing,the
usefulthingandthetractablething
Weakerandstrongerformsofreproducibilitydependentonexpectations,methods,tasks,churnturnaround,howmuch
weactuallyknowandcanrestrict
LabAnalogy:Witnessing“Datascopes”
InputData
Software
OutputData
ConfigParameters
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,,refresources
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
“Micro”Reproducibility
“Macro”Reproducibility
TRUST
Validate
Verify
Repeat,Replicate,Robust
[CTitusBrown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
Whythedifferences?
Reproduce,Trust
“anexperimentisreproducibleuntilanotherlaboratorytriestorepeatit”
AlexanderKohnRepeatability:
“Sameness”Sameresult1Lab1experiment
Reproducibility:
“Similarity”Similarresult>1Lab>1experiment
Validate
Verify
MethodReproducibilitytheprovisionofenoughdetailaboutstudyproceduresanddatasothesameprocedurescould,intheoryorinactuality,beexactlyrepeated.
ResultReproducibility(akareplicability)obtainingthesameresultsfromtheconductofanindependentstudywhoseproceduresareascloselymatchedtotheoriginalexperimentaspossible
Whatdoesresearchreproducibilitymean?StevenN.Goodman,DanieleFanelli,JohnP.A.IoannidisScienceTranslationalMedicine8(341),341ps12.[doi:10.1126/scitranslmed.aaf5027]http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf
ProductivityTrackdifferences
Validate
Verify
reviewerswantadditionalworkstatisticianwantsmorerunsanalysisneedstoberepeatedpost-docleaves,studentarrivesnew/reviseddatasetsupdated/newversionsofalgorithms/codessamplewascontaminatedbetterkit-longersimulationsnewpartners,newprojects
PersonalProductivity
PublicGoodReproducibility
“Datascope”LabAnalogy
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
“Datascope”LabAnalogy
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
Form
Function
“Datascope”Practicalities
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
Change DependenciesScience,methods,datasetsquestionsstay,answerschange
breakage,labsdecay,servicesandtechniquescomeandgo,newinstruments,updateddatasets,services,codes,hardware
Oneoffs,streams,stochastics,sensitivities,scale,non-portabledatablackboxes
supercomputeraccessnon-portablesoftwarelicensingrestrictionsunreliableresourcesblackboxes
T1 T2
evolvingrefdatasets,newsimulationcodes
EnvironmentArchivedvsActive
ContainedvsDistributedMulti-owned
Dependencies->ManageBlackboxes->Expose
Change->FixReliability
ReplicateharderthanReproduce?Repeatingtheexperimentorthesetup?
ContainerConundrum ResultswillVary
ReplicabilityWindowAllexperimentsbecomelessreplicableovertime
Preparetorepair
LevelsofComputationalReproducibility
Coverage:howmuchofanexperimentisreproducible
Orig
inalExp
erim
ent
Simila
rExp
erim
ent
Differen
tExp
erim
ent
Portab
ility
Depth:howmuchofanexperimentisavailable
Binaries+Data
SourceCode/Workflow+Data
Binaries+Data+Dependencies
SourceCode/Workflow+Data+Dependencies
VirtualMachineBinaries+Data+Dependencies
VirtualMachineSourceCode/Workflow+Data+Dependencies
Figures+Data
[Freire,2014]
Minimum:dataandsourcecodeavailableundertermsthatpermitinspectionandexecution.
MeasuringInformationGainfromReproducibility
Researchgoal
Method/Alg.
Platform/ExecEnv
DataParameters
Inputdata
Actors
InformationGain
Implementation/Code
Nochange
Change
Don’tcare
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041
How? PreservebyReporting,ReproducebyReading
ArchivedRecord
DescriptionZoostandards,commonmetadata
How? PreservebyMaintaining,Repairing,ContainingReproducebyRunning,Emulating,Reconstructing
ActiveInstrument BytelevelBuildabilityZoo
provenance
portability,preservation
robustness,versioning
access descriptionstandards
commonAPIslicensing,identifiers
standards,commonmetadata
changevariationsensitivitydiscrepancyhandling
packaging,containers
FAIRRACEReproducibilityDimensions
dependencies
steps
ResearchObjectStandards-basedmetadataframeworkforlogicallyand
physicallybundlingresourceswithcontext,http://researchobject.org
Biggerontheinsidethantheoutside
externalreferencing
ManifestConstruction
Aggregateslinkthingstogether
Annotationsaboutthings&their
relationships
Container
ResearchObjectStandards-basedmetadataframeworkforlogicallyandphysicallybundlingresourceswithcontext,http://researchobject.org
Packagingcontent&links:Zipfiles,BagIt,Dockerimages
Catalogues&CommonsPlatforms:FAIRDOM
ManifestDescription
Dependencieswhatelseis
needed
Versioningitsevolution
Checklistswhatshouldbethere
Provenancewhereit
camefrom
Identificationlocatethings
regardlesswhere
id
SystemsBiologyCommonsfair-dom.orgfairdomhub.org
• Linkdata,modelsandSOPs
• Standards• Snapshot+DOIs• Spanresources
http://data.datacite.org/10.15490/seek.1.investigation.56
CitationG.Penkler;F.duToit;W.Adams;M.Rautenbach;D.C.Palm;D.D.vanNiekerk;J.L.Snoep;(2014):GlucosemetabolisminPlasmodiumfalciparumtrophozoites;FAIRDOMHub.http://dx.doi.org/10.15490/seek.1.investigation.56
SystemsBiologyCommons• Linkdata,models
andSOPs• Standards• Spanresources• Snapshot+DOIs• Bundleandexport• Logicalbundles
Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
WorkflowResearchObjectsexchange,portabilityandmaintenance
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
AsthmaResearche-Lab
Datasetbuildingandreleasing
StandardisedpackingofSystemsBiologymodels
EuropeanSpaceAgencyROLibrary
Largedatasetmanagementforlifescienceworkflows
LHCATLASexperiments
NotreDame URostock
EncyclopediaofDNAElements
PeptideAtlas
ROTypes:ManifestContentProfilesminimal,maximal,extensible
PID
Citation
Checklist
Version
Prov
enan
ce
Dependencies
JATS
Commonproperties
amongcontenttypes
Minimuminformationforonecontenttype
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
Wordsmatter.
Reproducibilityisnotaend.
Itsameanstoanend.Bewarereproducibilityzealots.
50ShadesofReproducibility.
formvsfunction
Aconundrum:bigco-operativedata-drivensciencemakesreproducibility
desirablebutalsomeansdependencyandchangearetobe
expected.
Labanalogyforcomputationalscience
Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
WorkflowResearchObjects
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.TitusBrown*]
RepeatableEnvironments
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.TitusBrown*]
MetadataObjects:ReproducibleReporting,Exchange
ChecklistProvenanceTracking
VersioningDependencies
container
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
Summary
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041
• VALIDATION
• DISCREPANCIES
• DEPENDENCIES&CHANGE
• MAINTENANCE!Tokeepitrunning
• INTEROPERABILITY
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
RewardPeerreviewCitationmechanismsTransitivecreditmapsCitingchangingresources
Cite….Original?MostRecent?AVersion?Dockerimage?Description?
BornReproducible
It’saspectrumWeakerforms
ResearchObjects,MetadataObjectsCitableReproduciblePackaging
InputData WorkflowDescription
Provenancetrace
VersionofCodes/Services
Outputs Configs
Executionenvironment
IntermediatesORCIDS URIs DOIs
T1 T2
evolvingreferencedatasets,newsimulationcodes
Dependencies
ZhaoetalWhyworkflowsbreak-UnderstandingandcombatingdecayinTavernaworkflows,8thIntlConfe-Science2012
Blackboxes
?JoppaetalSCIENCE3402013,MorinetalSCIENCE3362012
Rawdata
Secondarydata
Rawdata
Tertiarydata
Source
SecondarydatasetsSecondaryanalysis
MetaAnalysis
ModelDrivenScience–canIrerunmymodel?ModelSweeps,Whatarethesensitivities?
ReproducibilityRampancy
FAIRnessFindable
Accessible
Interoperable
ReusableIntelligible
Reproducible
Citable
Track&Countable
ReproducibilityRealities:ItsHard
Social
Technical
Methodological
recomputation.org
sciencecodemanifesto.org
“Datascope”Entropy->Preservation“Replication/ReproducibilityWindow”
Form
Function
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatforms,computationalenvironment
Analogy:TheLabdatascience|data-drivenscience
1.Why
2.How