What is Reproducibility? The R* brouhaha
Transcript of What is Reproducibility? The R* brouhaha
![Page 1: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/1.jpg)
WhatisReproducibility?TheR*brouhaha(andhowResearchObjectscanhelp)
ProfessorCaroleGobleTheUniversityofManchester,UKSoftwareSustainabilityInstitute,UKELIXIR-UK,[email protected]
SciReproWorkshop,TPDL,9Sept2016,Hannover,Germany
![Page 2: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/2.jpg)
Acknowledgements• DagstuhlSeminar16041,January2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041
• ATISymposiumReproducibility,SustainabilityandPreservation,April2016– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/– https://osf.io/bcef5/files/
• CTitusBrown• JulianaFreire• DavidDeRoure• StianSoiland-Reyes• BarendMons• TimClark• DanielGarijo• NormanMorrison
![Page 3: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/3.jpg)
“WhenIuseaword,"HumptyDumptysaidinratherascornfultone,"itmeansjustwhatIchooseittomean-neithermorenorless.”
Carroll,ThroughtheLookingGlass
re-compute
replicatererun
repeat
re-examine
repurpose
recreate
reuse
restore reconstruct review
regeneraterevise
recycle
redo
robustness tolerance
verifica(oncompliancevalida(onassurance
remix
![Page 4: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/4.jpg)
ReproducibilityofReproducibilityResearch
![Page 5: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/5.jpg)
ComputationalScience
http://tpeterka.github.io/maui-project/
From:TheFutureofScientificWorkflows,ReportofDOEWorkshop2015,http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd
1. Observational,experimental
2. Theoretical3. Simulation4. Dataintensive
![Page 6: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/6.jpg)
BioSTIF
ComputationalScience
![Page 7: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/7.jpg)
Scientificpublicationsgoals:(i)announcearesult(ii)convincereadersitscorrect.
Papersinexperimentalscienceshoulddescribetheresultsandprovideaclearenoughprotocoltoallowsuccessfulrepetitionandextension.
Papersincomputationalscienceshoulddescribetheresultsandprovidethecompletesoftwaredevelopmentenvironment,dataandsetofinstructionswhichgeneratedthefigures.
VirtualWitnessing*
*LeviathanandtheAir-Pump:Hobbes,Boyle,andtheExperimentalLife(1985)ShapinandSchaffer.
JillMesirov
DavidDonoho
![Page 8: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/8.jpg)
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
![Page 9: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/9.jpg)
10“Simple”RulesforReproducibleComputationalResearch:RACE
1. ForEveryResult,KeepTrackofHowItWasProduced
2. AvoidManualDataManipulationSteps3. ArchivetheExactVersionsofAllExternal
ProgramsUsed4. VersionControlAllCustomScripts5. RecordAllIntermediateResults,WhenPossiblein
StandardizedFormats6. ForAnalysesThatIncludeRandomness,Note
UnderlyingRandomSeeds7. AlwaysStoreRawDatabehindPlots8. GenerateHierarchicalAnalysisOutput,Allowing
LayersofIncreasingDetailtoBeInspected9. ConnectTextualStatementstoUnderlying
Results10. ProvidePublicAccesstoScripts,Runs,and
Results
SandveGK,NekrutenkoA,TaylorJ,HovigE(2013)TenSimpleRulesforReproducibleComputationalResearch.PLoSComputBiol9(10):e1003285.doi:10.1371/journal.pcbi.1003285
RecordEverythingAutomateEverythingContainEverythingExposeEverything
![Page 10: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/10.jpg)
Preparationpainindependenttestingtrialsandtribulations
[NormanMorrison]
replicationhostilitynofunding,time,recognition,placetopublishresourceintensiveaccesstothecompleteenvironment
![Page 11: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/11.jpg)
Reproducibility=HardWork
Data sets
Analyses
Open-Paper
Open-Review
DOI:10.1186/2047-217X-1-18 >11000 accesses
Open-Code
8 reviewers tested data in ftp server & named reports published
DOI:10.5524/100044
Open-PipelinesOpen-Workflows
DOI:10.5524/100038 Open-Data
78GB CC0 data
Code in sourceforge under GPLv3: http://soapdenovo2.sourceforge.net/
>5000 downloads
Enabled code to being picked apart by bloggers in wiki http://homolog.us/wiki/index.php?title=SOAPdenovo2
[Scott Edmunds]
![Page 12: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/12.jpg)
WhatisreproducibilityItshardsoletsbesurewearedoingtherightthing,the
usefulthingandthetractablething
Weakerandstrongerformsofreproducibilitydependentonexpectations,methods,tasks,churnturnaround,howmuch
weactuallyknowandcanrestrict
![Page 13: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/13.jpg)
LabAnalogy:Witnessing“Datascopes”
InputData
Software
OutputData
ConfigParameters
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,,refresources
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
![Page 14: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/14.jpg)
“Micro”Reproducibility
“Macro”Reproducibility
TRUST
Validate
Verify
![Page 15: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/15.jpg)
Repeat,Replicate,Robust
[CTitusBrown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
Whythedifferences?
Reproduce,Trust
![Page 16: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/16.jpg)
“anexperimentisreproducibleuntilanotherlaboratorytriestorepeatit”
AlexanderKohnRepeatability:
“Sameness”Sameresult1Lab1experiment
Reproducibility:
“Similarity”Similarresult>1Lab>1experiment
Validate
Verify
![Page 17: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/17.jpg)
MethodReproducibilitytheprovisionofenoughdetailaboutstudyproceduresanddatasothesameprocedurescould,intheoryorinactuality,beexactlyrepeated.
ResultReproducibility(akareplicability)obtainingthesameresultsfromtheconductofanindependentstudywhoseproceduresareascloselymatchedtotheoriginalexperimentaspossible
Whatdoesresearchreproducibilitymean?StevenN.Goodman,DanieleFanelli,JohnP.A.IoannidisScienceTranslationalMedicine8(341),341ps12.[doi:10.1126/scitranslmed.aaf5027]http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf
![Page 18: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/18.jpg)
ProductivityTrackdifferences
Validate
Verify
![Page 19: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/19.jpg)
reviewerswantadditionalworkstatisticianwantsmorerunsanalysisneedstoberepeatedpost-docleaves,studentarrivesnew/reviseddatasetsupdated/newversionsofalgorithms/codessamplewascontaminatedbetterkit-longersimulationsnewpartners,newprojects
PersonalProductivity
PublicGoodReproducibility
![Page 20: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/20.jpg)
“Datascope”LabAnalogy
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
![Page 21: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/21.jpg)
“Datascope”LabAnalogy
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
Form
Function
![Page 22: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/22.jpg)
“Datascope”Practicalities
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment
Change DependenciesScience,methods,datasetsquestionsstay,answerschange
breakage,labsdecay,servicesandtechniquescomeandgo,newinstruments,updateddatasets,services,codes,hardware
Oneoffs,streams,stochastics,sensitivities,scale,non-portabledatablackboxes
supercomputeraccessnon-portablesoftwarelicensingrestrictionsunreliableresourcesblackboxes
![Page 23: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/23.jpg)
T1 T2
evolvingrefdatasets,newsimulationcodes
EnvironmentArchivedvsActive
ContainedvsDistributedMulti-owned
Dependencies->ManageBlackboxes->Expose
Change->FixReliability
![Page 24: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/24.jpg)
ReplicateharderthanReproduce?Repeatingtheexperimentorthesetup?
ContainerConundrum ResultswillVary
ReplicabilityWindowAllexperimentsbecomelessreplicableovertime
Preparetorepair
![Page 25: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/25.jpg)
LevelsofComputationalReproducibility
Coverage:howmuchofanexperimentisreproducible
Orig
inalExp
erim
ent
Simila
rExp
erim
ent
Differen
tExp
erim
ent
Portab
ility
Depth:howmuchofanexperimentisavailable
Binaries+Data
SourceCode/Workflow+Data
Binaries+Data+Dependencies
SourceCode/Workflow+Data+Dependencies
VirtualMachineBinaries+Data+Dependencies
VirtualMachineSourceCode/Workflow+Data+Dependencies
Figures+Data
[Freire,2014]
Minimum:dataandsourcecodeavailableundertermsthatpermitinspectionandexecution.
![Page 26: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/26.jpg)
MeasuringInformationGainfromReproducibility
Researchgoal
Method/Alg.
Platform/ExecEnv
DataParameters
Inputdata
Actors
InformationGain
Implementation/Code
Nochange
Change
Don’tcare
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041
![Page 27: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/27.jpg)
How? PreservebyReporting,ReproducebyReading
ArchivedRecord
DescriptionZoostandards,commonmetadata
![Page 28: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/28.jpg)
How? PreservebyMaintaining,Repairing,ContainingReproducebyRunning,Emulating,Reconstructing
ActiveInstrument BytelevelBuildabilityZoo
![Page 29: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/29.jpg)
provenance
portability,preservation
robustness,versioning
access descriptionstandards
commonAPIslicensing,identifiers
standards,commonmetadata
changevariationsensitivitydiscrepancyhandling
packaging,containers
FAIRRACEReproducibilityDimensions
dependencies
steps
![Page 30: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/30.jpg)
ResearchObjectStandards-basedmetadataframeworkforlogicallyand
physicallybundlingresourceswithcontext,http://researchobject.org
Biggerontheinsidethantheoutside
externalreferencing
![Page 31: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/31.jpg)
ManifestConstruction
Aggregateslinkthingstogether
Annotationsaboutthings&their
relationships
Container
ResearchObjectStandards-basedmetadataframeworkforlogicallyandphysicallybundlingresourceswithcontext,http://researchobject.org
Packagingcontent&links:Zipfiles,BagIt,Dockerimages
Catalogues&CommonsPlatforms:FAIRDOM
ManifestDescription
Dependencieswhatelseis
needed
Versioningitsevolution
Checklistswhatshouldbethere
Provenancewhereit
camefrom
Identificationlocatethings
regardlesswhere
id
![Page 32: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/32.jpg)
SystemsBiologyCommonsfair-dom.orgfairdomhub.org
• Linkdata,modelsandSOPs
• Standards• Snapshot+DOIs• Spanresources
http://data.datacite.org/10.15490/seek.1.investigation.56
CitationG.Penkler;F.duToit;W.Adams;M.Rautenbach;D.C.Palm;D.D.vanNiekerk;J.L.Snoep;(2014):GlucosemetabolisminPlasmodiumfalciparumtrophozoites;FAIRDOMHub.http://dx.doi.org/10.15490/seek.1.investigation.56
![Page 33: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/33.jpg)
SystemsBiologyCommons• Linkdata,models
andSOPs• Standards• Spanresources• Snapshot+DOIs• Bundleandexport• Logicalbundles
![Page 34: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/34.jpg)
Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
WorkflowResearchObjectsexchange,portabilityandmaintenance
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
![Page 35: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/35.jpg)
AsthmaResearche-Lab
Datasetbuildingandreleasing
StandardisedpackingofSystemsBiologymodels
EuropeanSpaceAgencyROLibrary
Largedatasetmanagementforlifescienceworkflows
LHCATLASexperiments
NotreDame URostock
EncyclopediaofDNAElements
PeptideAtlas
![Page 36: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/36.jpg)
ROTypes:ManifestContentProfilesminimal,maximal,extensible
PID
Citation
Checklist
Version
Prov
enan
ce
Dependencies
JATS
Commonproperties
amongcontenttypes
Minimuminformationforonecontenttype
![Page 37: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/37.jpg)
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
![Page 38: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/38.jpg)
Wordsmatter.
Reproducibilityisnotaend.
Itsameanstoanend.Bewarereproducibilityzealots.
50ShadesofReproducibility.
formvsfunction
Aconundrum:bigco-operativedata-drivensciencemakesreproducibility
desirablebutalsomeansdependencyandchangearetobe
expected.
Labanalogyforcomputationalscience
![Page 39: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/39.jpg)
![Page 40: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/40.jpg)
![Page 41: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/41.jpg)
![Page 42: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/42.jpg)
Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
WorkflowResearchObjects
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.TitusBrown*]
![Page 43: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/43.jpg)
RepeatableEnvironments
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html
[C.TitusBrown*]
MetadataObjects:ReproducibleReporting,Exchange
ChecklistProvenanceTracking
VersioningDependencies
container
![Page 44: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/44.jpg)
Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services
Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware
![Page 45: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/45.jpg)
Summary
![Page 46: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/46.jpg)
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041
![Page 47: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/47.jpg)
• VALIDATION
• DISCREPANCIES
• DEPENDENCIES&CHANGE
• MAINTENANCE!Tokeepitrunning
• INTEROPERABILITY
![Page 48: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/48.jpg)
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
RewardPeerreviewCitationmechanismsTransitivecreditmapsCitingchangingresources
Cite….Original?MostRecent?AVersion?Dockerimage?Description?
![Page 49: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/49.jpg)
BornReproducible
It’saspectrumWeakerforms
![Page 50: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/50.jpg)
ResearchObjects,MetadataObjectsCitableReproduciblePackaging
InputData WorkflowDescription
Provenancetrace
VersionofCodes/Services
Outputs Configs
Executionenvironment
IntermediatesORCIDS URIs DOIs
![Page 51: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/51.jpg)
T1 T2
evolvingreferencedatasets,newsimulationcodes
![Page 52: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/52.jpg)
Dependencies
ZhaoetalWhyworkflowsbreak-UnderstandingandcombatingdecayinTavernaworkflows,8thIntlConfe-Science2012
Blackboxes
?JoppaetalSCIENCE3402013,MorinetalSCIENCE3362012
![Page 53: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/53.jpg)
Rawdata
Secondarydata
Rawdata
Tertiarydata
Source
SecondarydatasetsSecondaryanalysis
MetaAnalysis
![Page 54: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/54.jpg)
ModelDrivenScience–canIrerunmymodel?ModelSweeps,Whatarethesensitivities?
![Page 55: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/55.jpg)
ReproducibilityRampancy
![Page 56: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/56.jpg)
FAIRnessFindable
Accessible
Interoperable
ReusableIntelligible
Reproducible
Citable
Track&Countable
![Page 57: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/57.jpg)
ReproducibilityRealities:ItsHard
Social
Technical
Methodological
![Page 58: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/58.jpg)
recomputation.org
sciencecodemanifesto.org
![Page 59: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/59.jpg)
“Datascope”Entropy->Preservation“Replication/ReproducibilityWindow”
Form
Function
Methodstechniques,algorithms,spec.ofthesteps,models
Materialsdatasets,parameters,algorithmseeds
Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets
Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatforms,computationalenvironment
![Page 60: What is Reproducibility? The R* brouhaha](https://reader034.fdocuments.in/reader034/viewer/2022042605/58a2e5551a28aba27f8b8982/html5/thumbnails/60.jpg)
Analogy:TheLabdatascience|data-drivenscience
1.Why
2.How