Image Segmentation and Seg3D Ross Whitaker SCI Institute, School of Computing University of Utah.
Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software...
Transcript of Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software...
![Page 1: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/1.jpg)
Image Segmentation and Seg3DImage Segmentation and Seg3D
Ross WhitakerSCI Institute, School of Computing
University of Utah
modified byDana Brooks
Ross WhitakerSCI Institute, School of Computing
University of Utah
modified byDana Brooks
![Page 2: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/2.jpg)
OverviewOverview
Segmentation Intro• What is it
Strategies and state of the artSeg3D intro
Segmentation Intro• What is it
Strategies and state of the artSeg3D intro
![Page 3: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/3.jpg)
Segmentation: Why?Segmentation: Why?
Detection/recognition• Is there a lesion?
Quantifying object properties• How big is the tumor? Is is expanding
or shrinking? • Statistical analyses of sets of biological
volumes Building models
Detection/recognition• Is there a lesion?
Quantifying object properties• How big is the tumor? Is is expanding
or shrinking?• Statistical analyses of sets of biological
volumesBuilding models
![Page 4: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/4.jpg)
What is Segmentation?What is Segmentation?Different definitions/meanings
• Depends on context, person, etc.• Application• Type of output
- e.g. Lines vs pixelsDifferent tools for different
applications/needs • Tradeoffs between general and
specific • Tradeoffs between development and
processing effort
Different definitions/meanings• Depends on context, person, etc.• Application• Type of output
- e.g. Lines vs pixelsDifferent tools for different
applications/needs• Tradeoffs between general and
specific• Tradeoffs between development and
processing effort
![Page 5: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/5.jpg)
What is Segmentation?What is Segmentation?Isolating a specific region of interest
(“find the star” or “bluish thing”) Isolating a specific region of interest
(“find the star” or “bluish thing”)
“Delineation problem”
Binary
![Page 6: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/6.jpg)
What is Segmentation?What is Segmentation?
Partitioning images/volumes into meaningful pieces
Partitioning images/volumes into meaningful pieces
Labels“Partitioning problem”
![Page 7: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/7.jpg)
What is Segmentation?What is Segmentation?Assigning each pixel a type (tissue or
material) Assigning each pixel a type (tissue or
material)
“Classification problem”
FabricPaperGrass
![Page 8: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/8.jpg)
“Quick and easy” general-purpose seg tool
Time consuming
“Quick and easy” general-purpose seg tool
Time consuming
Delineation by Hand ContouringDelineation by Hand Contouring
![Page 9: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/9.jpg)
3D: slice-by-slice with cursor defining boundaUser variation (esp. slice to slice)
•3D feedback helps
Delineation by Hand Contouring
![Page 10: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/10.jpg)
More sophisticated: Deformable (Active Contour) Models
More sophisticated: Deformable (Active Contour) Models
Snakes (polyline)Level setsActive shape / appearance (Cootes &
Taylor) • Train models to learn certain shapes
Snakes (polyline)Level setsActive shape / appearance (Cootes &
Taylor)• Train models to learn certain shapes
Model is attracted to features
Model stays smooth
![Page 11: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/11.jpg)
Watershed SegmentationWatershed Segmentation
Watershed Regions
BoundaryFunction (e.g. grad. mag.)
Generalizes to any dimension or boundary measure
“Catchment Basin”
Drop of water
grad. descent
![Page 12: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/12.jpg)
Example: Image PartitioningExample: Image Partitioning
Jurrus et al., ISBI 2008
![Page 13: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/13.jpg)
Image PartitioningImage Partitioning
Cates et al., 2003
![Page 14: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/14.jpg)
Minimum Cut (Shi and Malik `00)
Minimum Cut (Shi and Malik `00)
Treat image as graph• Vertices -> pixels• Edges -> neighbors• Edge weights -> cost to cut• Must define neighbors to which a pixel
is connected
Treat image as graph• Vertices -> pixels• Edges -> neighbors• Edge weights -> cost to cut• Must define neighbors to which a pixel
is connected
Image
Neighborhoodconnections Weighted
Edges
Graph
![Page 15: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/15.jpg)
Pixel ClassificationPixel ClassificationT1, T2, PD
Feature Space Classification
Tasdizen et al.
![Page 16: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/16.jpg)
Registration of TemplatesRegistration of Templates
Align a known, segmented image to input data
Align a known, segmented image to input data
![Page 17: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/17.jpg)
So: What is The Best Way to Segment Images?
So: What is The Best Way to Segment Images?
Depends…• Kind of data: type of noise,
signal, etc. • What you are looking for: shape,
size, variability • Application specifics: how
accurate, how many • Expertise / patience: need to tune
parameters
Depends…• Kind of data: type of noise,
signal, etc.• What you are looking for: shape,
size, variability• Application specifics: how
accurate, how many• Expertise / patience: need to tune
parameters
![Page 18: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/18.jpg)
State of the art• Specific data and shapes
- Train a template or model (variability)- Deform to fit specific data
- e.g. active appearance/shape models- e.g. atlas-based statistical methods
• General data and shapes- So many methods- So may parameters-So few good ones in practice->hand contouring?
Where Do Things Stand Now?Where Do Things Stand Now?
![Page 19: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/19.jpg)
State of the Art Segmentation: Statistics and Learning
State of the Art Segmentation: Statistics and Learning
Intensities and image statistics• Grey-levels and neighborhoods
Positions and templates• Register templates with spatial
knowledge Shapes
• Learning statistics of contours and surfaces
• Nonlocal relationships• Differential geometry
Intensities and image statistics• Grey-levels and neighborhoods
Positions and templates• Register templates with spatial
knowledgeShapes
• Learning statistics of contours and surfaces
• Nonlocal relationships• Differential geometry
![Page 20: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/20.jpg)
Example: Head Segmentation MRI
Example: Head Segmentation MRI
Tissue classification• GM, WM, CSF• Skull stripping (nonbrain)• Prior based on statistical template
- Combine with registration- Priors on local configurations
Limbic system (subcortical structures) • Deformable shapes with priors
Tissue classification• GM, WM, CSF• Skull stripping (nonbrain)• Prior based on statistical template
- Combine with registration- Priors on local configurations
Limbic system (subcortical structures)• Deformable shapes with priors
![Page 21: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/21.jpg)
FreeSurferFreeSurfer
Fischl and Dale MGHFischl and Dale MGH
MRI WM Surfaces Partition
![Page 22: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/22.jpg)
EM-Segmenter, Slicer3EM-Segmenter, Slicer3Tissue classification
• Inhomogeneity correction• Gaussian mixture model
Simultaneous classification and template• Iterative• Probabilistic atlas/template
Tissue classification• Inhomogeneity correction• Gaussian mixture model
Simultaneous classification and template• Iterative• Probabilistic atlas/template
![Page 23: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/23.jpg)
Specific vs General MethodsSpecific vs General Methods
Specific• Automated• Moderately reliable (user QC)• Training/learning• Works for specific:
- anatomy- imaging modalities- applications
• Pathology?
Specific• Automated• Moderately reliable (user QC)• Training/learning• Works for specific:
- anatomy- imaging modalities- applications
• Pathology?
![Page 24: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/24.jpg)
General•User interaction
-Steering•Parameter tuning•GUIs•Assumptions about data•Last resort
-Hand contouring
Specific vs General Methods
![Page 25: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/25.jpg)
Region-Based vs Edge-Based Strategies
Region-Based vs Edge-Based Strategies
Region-based methods (connected) • Somehow locally
homogeneous • E.g. Flood fill
Edge-based methods• Regions are bounded by
features with sharp contrast
• E.g. Canny Edge Detector
Region-based methods (connected)• Somehow locally
homogeneous• E.g. Flood fill
Edge-based methods• Regions are bounded by
features with sharp contrast
• E.g. Canny Edge Detector
![Page 26: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/26.jpg)
Typical Edge/Region-Based Segmentation Pipeline
Typical Edge/Region-Based Segmentation Pipeline
Filtering•Blurring (low pass)•Nonlinear diffusion
Feature Extraction•Differential Geometry•Data Fusion
Segmentation•Automatic•User Assisted
Analysis
Image/Volu me
Data
![Page 27: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/27.jpg)
Example: LivewireExample: Livewire
Contour follows features• Shortest path between user-defined
landmarks • Need preprocessing and definition
of “features” Barrett, 1997
Contour follows features• Shortest path between user-defined
landmarks• Need preprocessing and definition
of “features”Barrett, 1997
![Page 28: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/28.jpg)
Seg3DSeg3DGoals
• End-user application• General purpose• User-assisted
Philosophy• Voxel/pixel-based• Layers and labels, 3D photoshop-ish• GUIs and user interaction for user-
assisted segmentation • 3D interaction to aid 2D views
Goals• End-user application• General purpose• User-assisted
Philosophy• Voxel/pixel-based• Layers and labels, 3D photoshop-ish• GUIs and user interaction for user-
assisted segmentation• 3D interaction to aid 2D views
![Page 29: Image Segmentation and Seg3D - Scientific Computing and ...Cross platform, WX widgets. Software design/user interface Software engineering • Wrapping ITK filters and image I/O •](https://reader035.fdocuments.in/reader035/viewer/2022081406/5f116daabc4da57bee0a1651/html5/thumbnails/29.jpg)
Seg3DSeg3DSoftware engineering
• Wrapping ITK filters and image I/O• Cross platform, WX widgets
Software design/user interface
Software engineering• Wrapping ITK filters and image I/O• Cross platform, WX widgets
Software design/user interfaceViews (reconfigurable)
Data/Parameters
Layers/image s