Nonrigid Image Registration Using Conditional Mutual Information Loeckx et al. IPMI 2007
3D Rigid/Nonrigid Registration
-
Upload
maisie-barnett -
Category
Documents
-
view
38 -
download
0
description
Transcript of 3D Rigid/Nonrigid Registration
3D Rigid/Nonrigid Registration
1) Known features, correspondences, transformation model – feature based
2) Specific motion type, unknown correspondences – feature based
3) Known transformation model, unknown correspondences – region based
4) Specific motion model – feature based
5) Unknown motion model, unknown correspondences – region based
Visual Motion
Jim Rehg
(G.Tech)
Motion (Displacement) of Environment
Imageplane dt
tdt
)()(
rv
SceneFlowMotion
Field
)(tr
)()( tt vPw
Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa).Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities
Examples of Visual Motion
Examples of Visual Motion
Examples of Visual Motion
Applications of Motion Analysis
Visual tracking
Structure recovery
Robot (vehicle) navigation
Applications of Motion Analysis
Visual tracking
Structure recovery
Robot (vehicle) navigation
Motion Segmentation
Where are the independently moving objects (and how many are there)?
Optical Flow
2-D velocity field describing the apparent motion in an image sequence
A vector at each pixel indicates its motion (between a pair of frames).
Ground truthHorn and Schunk
Optical Flow and Motion Field
In general the optical flow is an approximation to the motion field.
When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise.
We can always think of the optical flow as summarizing the temporal change in an image sequence.
Computing Optical Flow
Courtesy of Michael Black
Cost Function for Optical Flow
Ryx
SSD tyxItvyuxIvuE,
2)],,()1,,([),(
Courtesy of Michael Black
Lucas-Kanade Method
Brute-force minimization of SSD error can be inefficient and inaccurateMany redundant window evaluations
Answer is limited to discrete u, v pairs
Lucas-Kanade Method
Problems with brute-force minimization of SSD errorMany redundant window evaluations
Answer is limited to discrete u, v pairs
Related to Horn-Schunk optical flow equations
Several key innovationsEarly, successful use of patch-based model in low-level vision. Today
these models are used everywhere.
Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.
Optical Flow Estimation
Optical Flow Estimation
Optical Flow Constraint
Optimization
Optimization
Optimization
Quality of Image Patch
Eigenvalues of the matrix contain information about local image structureBoth eigenvalues (close to) zero: Uniform area
One eigenvalue (close to) zero: Edge
No eigenvalues (close to) zero: Corner
Contributions of Lucas-Kanade
Basic idea of patch or template is very old (goes back at least to Widrow)
But in practice patch models have worked much better than the alternatives:Point-wise differential equations with smoothnessEdge-based descriptions
Patchs provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.
Lets Talk Applications
Alain Pitiot, Ph.D.
Siemens Molecular Imaging - Advanced Applications
Medical Image Registration
(a short overview)
Summer School 2005
SOME APPLICATIONS
Medical Image Registration
Motivation
Advances in imaging technology novel modalitiessee beyond: inside (non-invasive),
during (dynamic processes),
at small scale (increased resolution)
Understanding and correlating structure & function- automated/aided diagnosis
- image guided surgery/radio-therapy
- treatment/surgery planning
- medical atlases
- longitudinal studies: disease progression, development
Definitions
Def. #1: put two images into spatial correspondencegoal: extract more/better information
Def. #2: maximize similarity between transformed source image & target image
CT (thorax) PET (thorax)
source image target image+
transformedtarget image
Anatomical Functional
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Homer Simpson(MRI, coronal section)
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Homer Simpson(rest position)
Homer Simpson(monkey position)
very similar shapes
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Homer SimpsonHomo sapiens sapiens brain
expect larger differences
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Homer Simpson(MRI)
Homer Simpson(labelled atlas)
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Registration basis- extrinsic
- intrinsic
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Registration basis- extrinsic
- intrinsic
stereotactic frame fast, explicit computation
prospective, often invasive, often rigid transf. only
Taxonomy
Nature of applicationSubject
- intrasubject
- intersubject
- atlas
Registration basis- extrinsic
- intrinsic
versatile, minimally invasive
no ground truth
PET scintillography
Taxonomy
Registration basisextrinsic
intrinsic- landmark based
- segmentation based
- voxel based
fast
accuracy limited by localization precision
CT PET
Taxonomy | Nature of Application
Registration basisextrinsic
intrinsic- landmark based
- segmentation based
- voxel based
segmented corpora callosa fast
accuracy limited by segmentation
combine with voxel based
Taxonomy | Nature of Application
Registration basisextrinsic
intrinsic- landmark based
- segmentation based
- voxel based
cryo. section myelin-stained histological section
most flexible approach
resource intensive
combine with previous techniques
Taxonomy | Nature of Application
Nature of input imagesModality
Combination:
- mono-modal: same modality for source and target
- multi-modal: different modality
Dimensionality- spatial: 2-D/2-D, 2-D/3-D, 3-D/3-D
- temporal
a few im
aging modalities
Taxonomy
Constraintsfusion maximize similarity between
transformed source & target
Transformation space- flexibility
rigid, affine, parameterized,
free-form- support
local, global
choose space that fitsanatomy and/or application
global local
rigid
affine para
meterized
fluid/elastic
Taxonomy
ConstraintsSimilarity measure
“intensities of matched images
verify criterion”
- to each hypothesis its measure:
conservation affine functional statistical
Taxonomy
conservation of intensity SSD
affine relationship correlation coefficient
functional relationship correlation ratio
statistical dependence mutual information
source target
ConstraintsSimilarity measure
“intensities of matched images
verify criterion”
- to each hypothesis its measure:
conservation affine functional statistical
Taxonomy
conservation of intensity SSD
affine relationship correlation coefficient
functional relationship correlation ratio
statistical dependence mutual information
source target
ConstraintsSimilarity measure
“intensities of matched images
verify criterion”
- to each hypothesis its measure:
conservation affine functional statistical
Taxonomy
conservation of intensity SSD
affine relationship correlation coefficient
functional relationship correlation ratio
statistical dependence mutual information
source target
ConstraintsSimilarity measure
“intensities of matched images
verify criterion”
- to each hypothesis its measure:
conservation affine functional statistical
Taxonomy
conservation of intensity SSD
affine relationship correlation coefficient
functional relationship correlation ratio
statistical dependence mutual information
source target
ConstraintsSimilarity measure
“intensities of matched images
verify criterion”
- to each hypothesis its measure:
conservation affine functional statistical
Taxonomy
conservation of intensity SSD
affine relationship correlation coefficient
functional relationship correlation ratio
statistical dependence mutual information
source target
OptimizationOften iterative
- deterministic
gradient descent- stochastic
simulated annealing
Taxonomy
OptimizationOften iterative
- deterministic
gradient descent- stochastic
simulated annealing
Taxonomy
OptimizationOften iterative
- deterministic
gradient descent- stochastic
simulated annealing
Taxonomy
OptimizationOften iterative
- deterministic
gradient descent- stochastic
simulated annealing
Heteroclite bag of tricks- progressive refinement
- multi-scale (multi-resolution)
Taxonomy
OptimizationOften iterative
- deterministic
gradient descent- stochastic
simulated annealing
bag of tricks- progressive refinement
- multi-scale (multi-resolution)
Taxonomy
Issues
ValidationNo ground truth in general case (ill-posed problem)
Precision, robustness, reliability, etc.
Semi-automated registrationIs fully-automated desirable ?
Which compromise between fully and semi ?
Specific Application
Image Guided Surgery
Conventional Surgery: Seeing surfaces
Provided by Nakajima, Atsumi et al.
Computer Assisted Surgery: seeing through surfaces
Goal: Assist Surgeons
Surgical Planning & SimulationMaximize Tumor Removal
Minimize Damage to Critical Structures
Intraoperative Visualizations via 3D Slicer
Pre-Operative Image Processing
Construct 3D ModelsSemi-Automated Segmentation
DTMRI Tract Tracing
Register all pre-operative data
Integrated Preoperative Data
F. Talos
Patient-specific models
•Gering_fmri
Segmentation of Neural Structures
Intraoperative Image Processing
Acquire one or more volumetric (interventional) MRI (iMRI) images
Determine non-rigid registration of Pre- and Intra-operative data
Construct Intraoperative Visualization
transmit image data and 3D models thru volumetric deformation
integrate with iMRI images and models
display with 3D Slicer
LCD screen in front of surgeon in iMRIcoordinate visualization with intraoperative instruments
3D Slicer: tool for
Visualization
Registration
Segmentation
Measurements
Realtime Integration
Provided by D. Gering
3D Slicer Demo...
More Examples
More examples of correspondence:Motion (tracking)
beating heart
We have to establish correspondence between specific pointson the object boundary from frame to frame
Template matching
• In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face templateimage
image
Flexible template matching
• In flexible template matching we estimate “position” of each rigid component of a template
3D Doctor
Multimodal registration
Warping
example