3D Rigid/Nonrigid Registration

70
3D Rigid/Nonrigid Registration 1) Known features, correspondences, transformation model – feature 2) Specific motion type, unknown correspondences – feature base 3) Known transformation model, unknown correspondences – region ba 4) Specific motion model – feature based 5) Unknown motion model, unknown correspondences – region based

description

3D Rigid/Nonrigid Registration. Known features, correspondences, transformation model – feature based Specific motion type, unknown correspondences – feature based 3) Known transformation model, unknown correspondences – region based 4) Specific motion model – feature based - PowerPoint PPT Presentation

Transcript of 3D Rigid/Nonrigid Registration

Page 1: 3D Rigid/Nonrigid  Registration

3D Rigid/Nonrigid Registration

1) Known features, correspondences, transformation model – feature based

2) Specific motion type, unknown correspondences – feature based

3) Known transformation model, unknown correspondences – region based

4) Specific motion model – feature based

5) Unknown motion model, unknown correspondences – region based

Page 2: 3D Rigid/Nonrigid  Registration

Visual Motion

Jim Rehg

(G.Tech)

Page 3: 3D Rigid/Nonrigid  Registration

Motion (Displacement) of Environment

Imageplane dt

tdt

)()(

rv

SceneFlowMotion

Field

)(tr

)()( tt vPw

Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa).Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

Page 4: 3D Rigid/Nonrigid  Registration

Examples of Visual Motion

Page 5: 3D Rigid/Nonrigid  Registration

Examples of Visual Motion

Page 6: 3D Rigid/Nonrigid  Registration

Examples of Visual Motion

Page 7: 3D Rigid/Nonrigid  Registration

Applications of Motion Analysis

Visual tracking

Structure recovery

Robot (vehicle) navigation

Page 8: 3D Rigid/Nonrigid  Registration

Applications of Motion Analysis

Visual tracking

Structure recovery

Robot (vehicle) navigation

Page 9: 3D Rigid/Nonrigid  Registration

Motion Segmentation

Where are the independently moving objects (and how many are there)?

Page 10: 3D Rigid/Nonrigid  Registration

Optical Flow

2-D velocity field describing the apparent motion in an image sequence

A vector at each pixel indicates its motion (between a pair of frames).

Ground truthHorn and Schunk

Page 11: 3D Rigid/Nonrigid  Registration

Optical Flow and Motion Field

In general the optical flow is an approximation to the motion field.

When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise.

We can always think of the optical flow as summarizing the temporal change in an image sequence.

Page 12: 3D Rigid/Nonrigid  Registration

Computing Optical Flow

Courtesy of Michael Black

Page 13: 3D Rigid/Nonrigid  Registration

Cost Function for Optical Flow

Ryx

SSD tyxItvyuxIvuE,

2)],,()1,,([),(

Courtesy of Michael Black

Page 14: 3D Rigid/Nonrigid  Registration

Lucas-Kanade Method

Brute-force minimization of SSD error can be inefficient and inaccurateMany redundant window evaluations

Answer is limited to discrete u, v pairs

Page 15: 3D Rigid/Nonrigid  Registration

Lucas-Kanade Method

Problems with brute-force minimization of SSD errorMany redundant window evaluations

Answer is limited to discrete u, v pairs

Related to Horn-Schunk optical flow equations

Several key innovationsEarly, successful use of patch-based model in low-level vision. Today

these models are used everywhere.

Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

Page 16: 3D Rigid/Nonrigid  Registration

Optical Flow Estimation

Page 17: 3D Rigid/Nonrigid  Registration

Optical Flow Estimation

Page 18: 3D Rigid/Nonrigid  Registration

Optical Flow Constraint

Page 19: 3D Rigid/Nonrigid  Registration

Optimization

Page 20: 3D Rigid/Nonrigid  Registration

Optimization

Page 21: 3D Rigid/Nonrigid  Registration

Optimization

Page 22: 3D Rigid/Nonrigid  Registration

Quality of Image Patch

Eigenvalues of the matrix contain information about local image structureBoth eigenvalues (close to) zero: Uniform area

One eigenvalue (close to) zero: Edge

No eigenvalues (close to) zero: Corner

Page 23: 3D Rigid/Nonrigid  Registration

Contributions of Lucas-Kanade

Basic idea of patch or template is very old (goes back at least to Widrow)

But in practice patch models have worked much better than the alternatives:Point-wise differential equations with smoothnessEdge-based descriptions

Patchs provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

Page 24: 3D Rigid/Nonrigid  Registration

Lets Talk Applications

Page 25: 3D Rigid/Nonrigid  Registration

Alain Pitiot, Ph.D.

Siemens Molecular Imaging - Advanced Applications

Medical Image Registration

(a short overview)

Summer School 2005

Page 26: 3D Rigid/Nonrigid  Registration

SOME APPLICATIONS

Medical Image Registration

Page 27: 3D Rigid/Nonrigid  Registration

Motivation

Advances in imaging technology novel modalitiessee beyond: inside (non-invasive),

during (dynamic processes),

at small scale (increased resolution)

Understanding and correlating structure & function- automated/aided diagnosis

- image guided surgery/radio-therapy

- treatment/surgery planning

- medical atlases

- longitudinal studies: disease progression, development

Page 28: 3D Rigid/Nonrigid  Registration

Definitions

Def. #1: put two images into spatial correspondencegoal: extract more/better information

Def. #2: maximize similarity between transformed source image & target image

CT (thorax) PET (thorax)

source image target image+

transformedtarget image

Anatomical Functional

Page 29: 3D Rigid/Nonrigid  Registration

Taxonomy

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Homer Simpson(MRI, coronal section)

Page 30: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Homer Simpson(rest position)

Homer Simpson(monkey position)

very similar shapes

Taxonomy

Page 31: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Homer SimpsonHomo sapiens sapiens brain

expect larger differences

Taxonomy

Page 32: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Homer Simpson(MRI)

Homer Simpson(labelled atlas)

Taxonomy

Page 33: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Registration basis- extrinsic

- intrinsic

Taxonomy

Page 34: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Registration basis- extrinsic

- intrinsic

stereotactic frame fast, explicit computation

prospective, often invasive, often rigid transf. only

Taxonomy

Page 35: 3D Rigid/Nonrigid  Registration

Nature of applicationSubject

- intrasubject

- intersubject

- atlas

Registration basis- extrinsic

- intrinsic

versatile, minimally invasive

no ground truth

PET scintillography

Taxonomy

Page 36: 3D Rigid/Nonrigid  Registration

Registration basisextrinsic

intrinsic- landmark based

- segmentation based

- voxel based

fast

accuracy limited by localization precision

CT PET

Taxonomy | Nature of Application

Page 37: 3D Rigid/Nonrigid  Registration

Registration basisextrinsic

intrinsic- landmark based

- segmentation based

- voxel based

segmented corpora callosa fast

accuracy limited by segmentation

combine with voxel based

Taxonomy | Nature of Application

Page 38: 3D Rigid/Nonrigid  Registration

Registration basisextrinsic

intrinsic- landmark based

- segmentation based

- voxel based

cryo. section myelin-stained histological section

most flexible approach

resource intensive

combine with previous techniques

Taxonomy | Nature of Application

Page 39: 3D Rigid/Nonrigid  Registration

Nature of input imagesModality

Combination:

- mono-modal: same modality for source and target

- multi-modal: different modality

Dimensionality- spatial: 2-D/2-D, 2-D/3-D, 3-D/3-D

- temporal

a few im

aging modalities

Taxonomy

Page 40: 3D Rigid/Nonrigid  Registration

Constraintsfusion maximize similarity between

transformed source & target

Transformation space- flexibility

rigid, affine, parameterized,

free-form- support

local, global

choose space that fitsanatomy and/or application

global local

rigid

affine para

meterized

fluid/elastic

Taxonomy

Page 41: 3D Rigid/Nonrigid  Registration

ConstraintsSimilarity measure

“intensities of matched images

verify criterion”

- to each hypothesis its measure:

conservation affine functional statistical

Taxonomy

conservation of intensity SSD

affine relationship correlation coefficient

functional relationship correlation ratio

statistical dependence mutual information

source target

Page 42: 3D Rigid/Nonrigid  Registration

ConstraintsSimilarity measure

“intensities of matched images

verify criterion”

- to each hypothesis its measure:

conservation affine functional statistical

Taxonomy

conservation of intensity SSD

affine relationship correlation coefficient

functional relationship correlation ratio

statistical dependence mutual information

source target

Page 43: 3D Rigid/Nonrigid  Registration

ConstraintsSimilarity measure

“intensities of matched images

verify criterion”

- to each hypothesis its measure:

conservation affine functional statistical

Taxonomy

conservation of intensity SSD

affine relationship correlation coefficient

functional relationship correlation ratio

statistical dependence mutual information

source target

Page 44: 3D Rigid/Nonrigid  Registration

ConstraintsSimilarity measure

“intensities of matched images

verify criterion”

- to each hypothesis its measure:

conservation affine functional statistical

Taxonomy

conservation of intensity SSD

affine relationship correlation coefficient

functional relationship correlation ratio

statistical dependence mutual information

source target

Page 45: 3D Rigid/Nonrigid  Registration

ConstraintsSimilarity measure

“intensities of matched images

verify criterion”

- to each hypothesis its measure:

conservation affine functional statistical

Taxonomy

conservation of intensity SSD

affine relationship correlation coefficient

functional relationship correlation ratio

statistical dependence mutual information

source target

Page 46: 3D Rigid/Nonrigid  Registration

OptimizationOften iterative

- deterministic

gradient descent- stochastic

simulated annealing

Taxonomy

Page 47: 3D Rigid/Nonrigid  Registration

OptimizationOften iterative

- deterministic

gradient descent- stochastic

simulated annealing

Taxonomy

Page 48: 3D Rigid/Nonrigid  Registration

OptimizationOften iterative

- deterministic

gradient descent- stochastic

simulated annealing

Taxonomy

Page 49: 3D Rigid/Nonrigid  Registration

OptimizationOften iterative

- deterministic

gradient descent- stochastic

simulated annealing

Heteroclite bag of tricks- progressive refinement

- multi-scale (multi-resolution)

Taxonomy

Page 50: 3D Rigid/Nonrigid  Registration

OptimizationOften iterative

- deterministic

gradient descent- stochastic

simulated annealing

bag of tricks- progressive refinement

- multi-scale (multi-resolution)

Taxonomy

Page 51: 3D Rigid/Nonrigid  Registration

Issues

ValidationNo ground truth in general case (ill-posed problem)

Precision, robustness, reliability, etc.

Semi-automated registrationIs fully-automated desirable ?

Which compromise between fully and semi ?

Page 52: 3D Rigid/Nonrigid  Registration

Specific Application

Page 53: 3D Rigid/Nonrigid  Registration

Image Guided Surgery

Page 54: 3D Rigid/Nonrigid  Registration

Conventional Surgery: Seeing surfaces

Provided by Nakajima, Atsumi et al.

Page 55: 3D Rigid/Nonrigid  Registration

Computer Assisted Surgery: seeing through surfaces

Page 56: 3D Rigid/Nonrigid  Registration

Goal: Assist Surgeons

Surgical Planning & SimulationMaximize Tumor Removal

Minimize Damage to Critical Structures

Intraoperative Visualizations via 3D Slicer

Page 57: 3D Rigid/Nonrigid  Registration

Pre-Operative Image Processing

Construct 3D ModelsSemi-Automated Segmentation

DTMRI Tract Tracing

Register all pre-operative data

Page 58: 3D Rigid/Nonrigid  Registration

Integrated Preoperative Data

F. Talos

Page 59: 3D Rigid/Nonrigid  Registration

Patient-specific models

•Gering_fmri

Page 60: 3D Rigid/Nonrigid  Registration

Segmentation of Neural Structures

Page 61: 3D Rigid/Nonrigid  Registration

Intraoperative Image Processing

Acquire one or more volumetric (interventional) MRI (iMRI) images

Determine non-rigid registration of Pre- and Intra-operative data

Page 62: 3D Rigid/Nonrigid  Registration

Construct Intraoperative Visualization

transmit image data and 3D models thru volumetric deformation

integrate with iMRI images and models

display with 3D Slicer

LCD screen in front of surgeon in iMRIcoordinate visualization with intraoperative instruments

Page 63: 3D Rigid/Nonrigid  Registration

3D Slicer: tool for

Visualization

Registration

Segmentation

Measurements

Realtime Integration

Provided by D. Gering

Page 64: 3D Rigid/Nonrigid  Registration

3D Slicer Demo...

Page 65: 3D Rigid/Nonrigid  Registration

More Examples

Page 66: 3D Rigid/Nonrigid  Registration

More examples of correspondence:Motion (tracking)

beating heart

We have to establish correspondence between specific pointson the object boundary from frame to frame

Page 67: 3D Rigid/Nonrigid  Registration

Template matching

• In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template:

- translation, rotation, scale,…

Face templateimage

image

Page 68: 3D Rigid/Nonrigid  Registration

Flexible template matching

• In flexible template matching we estimate “position” of each rigid component of a template

Page 69: 3D Rigid/Nonrigid  Registration

3D Doctor

Multimodal registration

Page 70: 3D Rigid/Nonrigid  Registration

Warping

example