Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing...
13
Open Microscopy Environment School of Life Sciences, University of Dundee Dundee, Scotland, UK openmicroscopy Permission to share 1 Josh Moore ( notjustmoore) GBI Virtual Workshop, 2021.01.29 “ cloud” Accessing cloud - hosted image data Day 3. Standard image file format for sharing big image data in the cloud
Transcript of Accessing “cloud” image data - Open Microscopy...921600 x 380928 pixels. Use cases: 1. Sharing...
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 1
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
“cloud”Accessing
cloud - hostedimage data
Day 3. Standard image file format for sharing big image data in the cloud
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 2
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Large data needs chunks & pyramids to be accessible.
Monolithic formats are difficult to access remotely.
Current formats for chunked access tends to be monolithic.
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 3
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Single TIFF file
437 GB
Faas et al. (idr0053, CC BY-NC-SA 3.0)J Cell Biol (2012)921600 x 380928 pixels
Use cases:
1. Sharing beyond current limits, e.g., downloading TBsfrom OMERO.
2. Remotely sharing withouta server.
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 4
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Remote access
…
21 chunks in the browser
Single TIFF file
437 GB
Remote copy
Local copy
Chunk size:
1024 x 1024(1 MB)
Another 21 chunks
Monolithic format Next-generation file format (NGFF)351 GB 88 GB 21 MB22 GB
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 5
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Maturity Expressivity Multi-resolution Performance
TIFF Ubiquitous 2D tiles TIFF spec Linear slowdown
HDF Well-supported 3D+ chunks No spec yet Slow writes *
Zarr/N5 Next-generation 3D+ chunks NGFF spec Highly parallel **
HDFMetadata
MetadataZarr
Suite of open file formats
Scientific providersCommercial providers
“Cloud” providers Remote
LocalO
bject“Cloud” à “Storage”
https://www.openio.io/blog/block-file-object-storage-evolution-computer-storage-systems
Filesystem Object storage
1 €/GB 0.01 €/GB
Gbps Tbps
10 µs 1 ms
I/O intensive Immutable
High frequency Versioned
… …
Software needsto be smarter.
Each object is a chunk.
FilesystemsNFS/SMBGPFSHDDSDD
NGFF
Proprietary File Formats (PFF)
Conversion
Analysis clients & libraries
Upload
Remote
LocalO
bject
On-the-flytranslation
On-the-flytranslation
Upload
Proprietary File Formats (PFF) Analysis clients & libraries
Upload
Remote
LocalO
bject
On-the-flytranslation
On-the-flytranslation
NGFF
Conversion
Upload
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 9
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Conversion
bioformats2raw raw2ometiff
OME-XMLTIFF
Chris Allan & GlencoeConverting Whole Slide Images to OME-TIFF: A New Workflow
glencoesoftware/bioformats2rawglenocesoftware/raw2ometiff
MetadataZarr/N5
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 10
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
Specifications:
• Multiscales
• Labels
• HCS Plates
Process:
• Discussions: https://image.sc
• Publication: https://ngff.openmicroscopy.org
• Samples: https://s3.embassy.ebi.ac.uk/idr/zarr
{“Object-per-chunk”
MetadataZarr/N5
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 11
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
PracticalDownload from S3Convert your dataShare your data on S3http://bit.ly/gbi-2021-ngff-1
Open Microscopy EnvironmentSchool of Life Sciences, University of DundeeDundee, Scotland, UK
openmicroscopyPermission to share 12
Josh Moore ( notjustmoore)GBI Virtual Workshop, 2021.01.29
A few last words on S3Endpoint Bucket Key (Access)
https://s3.embassy.ebi.ac.uk idr zarr/v0.1/zarr/9822151.zarr download
https://s3.embl.de bioim idr0062-tiffs/… download, upload
https://s3.eu-south-1.amazonaws.com … … …
https://s3.embl.de/bioim/s3-browser.html
Unique identifier for an object
idr0062
Z-stacks of TIFFs
idr0023
the authors,Thanks to the OME teams,
GehlenborgLab (HMS)
& our funders
Hériché et al. (idr0002, CC-BY 4.0)MBoC (2016)
Lamers et al. (idr0083, CC BY 4.0)Science (2020)
Blin et al. (idr0062, CC BY 4.0)PLOS Biology (2019)
Faas et al. (idr0053, CC BY-NC-SA 3.0)J Cell Biol (2012)
VivBDVTomancak Lab(MPI-CBG)
McDole et al. (idr0044, CC BY 4.0)Cell (2018)
N5Saalfeld Lab(Janelia)
https://www.openmicroscopy.org/teams
ZarrAlistair Miles(Oxford)
13
Acknowledgements
KLBKeller Lab(Janelia)
