Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.
description
Transcript of Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.
![Page 1: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/1.jpg)
Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats
Toby Cornish, M.D., Ph.D.
Department of Pathology
Johns Hopkins Medical Institutions
Baltimore, MD
![Page 2: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/2.jpg)
Overview• JPEG-2000 is a highly-efficient compression
standard well-suited to storage of whole slide images
• JP2 and Aperio SVS are two WSI file formats that employ JPEG2k compression
• Although the file formats are different, the data is essentially stored in the same format
• Few tools exist for manipulating these formats • It should be possible to open, manipulate, and
interconvert these formats using readily available, free and open libraries
![Page 3: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/3.jpg)
JPEG-2000
Compression
Codestreams
File Formats
JP2
Aperio SVS
Design
Access SVS Data
Convert SVS to JP2
Results
![Page 4: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/4.jpg)
File Formats
JP2
Aperio SVS
Design
Access SVS Data
Convert SVS to JP2
Results
JPEG-2000
Compression
Codestreams
![Page 5: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/5.jpg)
JPEG-2000
• A standard for wavelet-based image compression
• Joint Photographic Expert Group as a replacement for JPEG
• ISO/IEC 15444-1 and 15444-2 (parts 1 and 2)
• Part 1 defines a self-contained codestream (JPC) as well as a file format (JP2) that can wrap the codestream
![Page 6: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/6.jpg)
JPEG-2000, cont.
• JPIP (ISO/IEC 15444-9) is a related standard for streaming of large JPEG2k codestreams
• Several encoders/decoders both free and proprietary for JPEG2k– Vary widely in the implementation of the standard, but
most support JPEG2k part 1
• Mostly used for high capacity image storage in videogames, satellite imagery, and, more recently, medical imagery
![Page 7: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/7.jpg)
JPEG-2000, cont.
• Features:– Highly efficient compression
– Native tiling support for large images
– Native multiresolution decompression from a single codestream
– Support for an arbitrary number of image components (i.e. channels)
– JPEG2k codestreams are self-contained and can be embedded in other file formats
![Page 8: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/8.jpg)
File Formats
JP2
Aperio SVS
Design
Access SVS Data
Convert SVS to JP2
Results
JPEG-2000
Compression
Codestreams
![Page 9: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/9.jpg)
JPEG-2k codestreams
• The codestream are capable of indepently being decoded into an image
• All codestreams are tiled, but the simplest and most common form is the single tile codestream
• Single tile images are good for small images
• Multiple tile images better for handling large image data such as WSI
![Page 10: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/10.jpg)
JPEG2k codestream, single tile
1
One codestream – One tile
![Page 11: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/11.jpg)
JPEG2k codestream, single tile
SOC
QCD
COD
SIZ
Main header
Tile header 1
Packet 1
Packet n
SOT
COM
SOD
EOC
start of codestream
image and tile size
coding style defaults
quantization defaults
comments
start of tile
start of data
end of codestream
EOC
Tile data 1
JPEG2kcodestream
![Page 12: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/12.jpg)
JPEG2k codestream, multiple tiles
One codestream – Multiple tiles
11 12
13 14 15 16
1 2 3 4
5 6 7 8
9 10
![Page 13: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/13.jpg)
JPEG2k codestream, multiple tilesSOC
QCD
COD
SIZ
Packet 1
Packet n
SOT
COM
SOD
start of codestream
image and tile sizecoding style defaults
quantization defaultscomments
start of tilestart of data
EOC end of codestream
Packet 1
Packet n
SOTSOD
start of tilestart of data
Main header
Tile header 1
EOC
Tile data 1
JPEG2kcodestream
Tile header n
Tile data n
![Page 14: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/14.jpg)
Design
Access SVS Data
Convert SVS to JP2
Results
JPEG-2000
Compression
Codestreams
File Formats
JP2
Aperio SVS
![Page 15: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/15.jpg)
JP2
• Wrapped JPEG2k codestream
• Composed of mandatory and optional header boxes
• Extensible via UUIDs (Universal Unique Identifiers) and/or XML
![Page 16: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/16.jpg)
JP2 file format, cont.
JPEG2k signature box
File type box
JP2 header box
Contiguous codestream box
JPEG2k codestream
XML boxes
UIUD boxes
Main header
Tile header 1
EOC
Tile data 1
Tile header n
Tile data n
![Page 17: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/17.jpg)
Design
Access SVS Data
Convert SVS to JP2
Results
JPEG-2000
Compression
Codestreams
File Formats
JP2
Aperio SVS
![Page 18: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/18.jpg)
Aperio SVS file format• Developed by Aperio, Inc. (San Diego, CA)
• Essentially, a TIFF file with a tiled layout
• Tile data are JPEG2k codestreams
• Can consist of multiple images (pages) derived from slide scanning with one Image File Directory (IFD) per image
• Image metadata is stored in Tags
• The highest resolution data is always tiled
![Page 19: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/19.jpg)
SVS/TIFF structure
ImagePyramid
Thumbnail
Label
Macro
![Page 20: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/20.jpg)
SVS/TIFF structure, cont.Byte order
Version number
IFD offset
Tag entry count
Tag 1
Tag n
IFD 1
Image data
IFD 2
IFD 3
IFD 4
IFD 5
IFD 6
Tag entry count
Tag 1
Tag n
Image data
![Page 21: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/21.jpg)
SVS/TIFF Tiling
Multiple tiles – one codestream per tile
11 12
13 14 15 16
1 2 3 4
5 6 7 8
9 10
![Page 22: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/22.jpg)
SVS/TIFF tiling, cont.
IFD 1
Tile 1
Tile n
Tile 2
Tile 3
Tiff Tiles
JPEG2k codestream
Tag entry count
Tag 1
Tag n
Image data
Main header
Tile header 1
EOC
Tile data 1
![Page 23: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/23.jpg)
SVS/TIFF v JP2• Differences:
– JP2 contains one JPEG2k codestream, which contains multiple tiles
– SVS/TIFF contains multiple tiles, each of which contains one codestream with exactly one tile
• Similarities:– The tile data is essentially the exact same, but packaged
– Should permit direct copying of tile data between formats
![Page 24: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/24.jpg)
Design
Access SVS Data
Convert SVS to JP2
JPEG-2000
Compression
Codestreams
File Formats
JP2
Aperio SVS
Results
![Page 25: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/25.jpg)
Design
• Open source libraries (LibTiff, ImageMagick/Jasper)
• C programming language for portability
• First two goals:
– Read SVS/TIFF image data using custom software
– Convert SVS/TIFF files to JP2 without decompressing
![Page 26: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/26.jpg)
Read SVS/TIFF
Tile 1
Tile n
Tile 2
Tiff Tiles
Main headerTile header
EOC
Tile data 1
Main headerTile header
EOC
Tile data 2
Main headerTile header
EOC
Tile data n
IFD
SVSImage data
TIFF Header
Decode
Decode
Decode
![Page 27: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/27.jpg)
SVS/TIFF to JP2 conversion
Tile 1
Tile n
Tile 2
Tiff TilesIFD
SVSImage data
TIFF Header
Main headerTile header
EOC
Tile data 1
Main headerTile header
EOC
Tile data 2
Main headerTile header
EOC
Tile data n
Main headerTile header
EOC
Tile data 1
Tile header
Tile data 2
Tile header
Tile data 2
JPEG2kCodestream
JP2 Header
Mergedcodestream
![Page 28: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/28.jpg)
Issues• SVS format “quirks”
– Somewhat unusual component subsampling
• Limits the number of decoders that will read the data without complaining
– SVS tiles are transformed into YCrCb color space
• Requires a transform to RGB color space before use
• A compliant JP2 reader should do this, but not all do
![Page 29: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/29.jpg)
Results
JPEG-2000
Compression
Codestreams
File Formats
JP2
Aperio SVS
Design
Access SVS Data
Convert SVS to JP2
![Page 30: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/30.jpg)
Results
• Several tools were created, allowing:
• Direct access to SVS tile data– Manipulate, extract, and montage tiles– See LOCI Bioformats project
• Direct conversion of SVS files to JP2– Requires the SVS tile dimensions to be multiples of 64
– Embedded ICC profiles not propogated
![Page 31: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/31.jpg)
Future work• Implement software in Java
• Create a JP2 SVS/TIFF converter
• Investigate limitations: – Why must tile dimensions be multiples of 64?– Why do most decoders (notably the JAI decoders) fail to
decode JPEG2k codestreams from SVS?
• Include ICC profiles
• Develop a generic LibTIFF plugin for encoding/decoding JPEG2k image data
![Page 32: Tools for Manipulating JPEG-2000 Based Whole Slide Image Formats Toby Cornish, M.D., Ph.D.](https://reader031.fdocuments.in/reader031/viewer/2022020920/5681598b550346895dc6cfa3/html5/thumbnails/32.jpg)
AcknowledgementsJohns HopkinsKristen Lecksell
University of Wisconsin-Madison, LOCIMelissa LinkertCurtis Rueden
ReferencesTaubman and Marcellin, JPEG2000 Image Compression
Fundamentals, Standards and PracticeAperio Technical Docs (http://www.aperio.com/documents/)
LibrariesLibTIFF (http://www.remotesensing.org/libtiff/)ImageMagick (http://www.imagemagick.org/script/index.php)Jasper (http://www.ece.uvic.ca/~mdadams/jasper/)