Medical Image RegistrationMedical Image Registration
Yujun GuoYujun Guo
Dept.of CSDept.of CS
Kent State UniversityKent State University
OutlineOutline
Why registrationWhy registration
Registration basicsRegistration basics
Rigid registrationRigid registration
Non-rigid registrationNon-rigid registration
ApplicationsApplications
Modalities in Medical ImageModalities in Medical Image
Computed Tomography (CT), Magnetic Resonance (MR) imaging, Ultrasound, and X-ray give anatomic information.
Positron Emission Tomography (PET) and Single Photon Emission CT (SPECT) give functional information.
RegistrationRegistration
• Monomodality:Monomodality: A series of same modality A series of same modality images (CT/CT, MR/MR, images (CT/CT, MR/MR,
Mammogram pairs,…). Mammogram pairs,…). Images may be acquired weeks or months apart; Images may be acquired weeks or months apart;
taken from different viewpoints.taken from different viewpoints. Aligning images in order to detect subtle changes in Aligning images in order to detect subtle changes in
intensity or shapeintensity or shape
• Multimodality:Multimodality: Complementary anatomic and functional information Complementary anatomic and functional information
from multiple modalities can be obtained for the from multiple modalities can be obtained for the precise diagnosis and treatment.precise diagnosis and treatment.
Examples:PET and SPECT (low resolution, functional Examples:PET and SPECT (low resolution, functional information) need MR or CT (high resolution, information) need MR or CT (high resolution, anatomical information) to get structure anatomical information) to get structure information.information.
Registration Problem DefinitionRegistration Problem Definition
p = (825,856)
q = (912,632)
q = T(p;a)
Pixel location in first image Homologous pixel location in second image
Pixel location mapping function
Example Mapping FunctionExample Mapping Function
p = (825,856)
q = (912,632)
Pixel scaling and translation
Image RegistrationImage Registration
Define a transform T that will map Define a transform T that will map one image onto another image of the one image onto another image of the same object such that some image same object such that some image quality criterion is maximized.quality criterion is maximized.
A mapping between two images both A mapping between two images both spatially and with respect to intensityspatially and with respect to intensity
II22 = g (T(I = g (T(I11))))
Registration SchemeRegistration Scheme
ComponentsComponents
Feature SpaceFeature Space
Search Space or transformationSearch Space or transformation
Similarity MetricSimilarity Metric
Search StrategySearch Strategy
Feature SpaceFeature SpaceGeometric landmarks:Geometric landmarks:Points Points Edges Edges Contours Contours Surfaces, etc.Surfaces, etc.Intensities:Intensities:Raw pixel valuesRaw pixel values
23 35
24 56
Feature-based Feature-based Intensity-basedIntensity-based
Image transformationsTransformation
Inputimage
Output
image
w
y
x
w
y
x
876
543
210
'
'
'
mmm
mmm
mmm
100
543
210
mmm
mmm
affineM
Affine transformation
100
cossin
sincos
y
x
rigid t
t
M
Rigid transformationOriginal
shape
Rigid
Non-rigid
Similarity MetricSimilarity Metric
Absolute differenceAbsolute difference
SSD (Sum of Squared Difference)SSD (Sum of Squared Difference)
Correlation CoefficientCorrelation Coefficient
Mutual Information / Normalized Mutual Information / Normalized Mutual InformationMutual Information
Search StrategySearch Strategy
Powell’s direction set methodPowell’s direction set method
Downhill simplex methodDownhill simplex method
Dynamic programmingDynamic programming
Relaxation matchingRelaxation matching
Hierarchical techniquesHierarchical techniques
Multi-modality Brain image Multi-modality Brain image registrationregistration
Intensity-basedIntensity-based3D/3D Rigid transformation, DOF=6 3D/3D Rigid transformation, DOF=6 (3 translations, 3 rotations)(3 translations, 3 rotations)Maximization of Normalized Mutual Maximization of Normalized Mutual InformationInformationSimplex DownhillSimplex DownhillMulti-resolutionMulti-resolutionDataset: Vanderbilt UniversityDataset: Vanderbilt University
http://www.vuse.vanderbilt.edu/~image/registration/http://www.vuse.vanderbilt.edu/~image/registration/results.htmlresults.html
Mutual Information as Similarity Mutual Information as Similarity MeasureMeasure
Mutual informationMutual information is applied to measure the is applied to measure the statistic dependence between the image intensities of statistic dependence between the image intensities of corresponding voxels in both images, which is assumed to corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned.be maximal if the images are geometrically aligned.
a b BA
ABAB bPaP
baPbaPBAI
)()(
),(log),(),(
)|()(
)|()(
),()()(
ABHBH
BAHAH
BAHBHAH
Normalized Mutual InformationNormalized Mutual Information
Extension of Mutual InformationExtension of Mutual Information
Maes et. al.:Maes et. al.:
Studholme et. Al.:Studholme et. Al.:
Compensate for the sensitivity of MI to Compensate for the sensitivity of MI to changes in image overlapchanges in image overlap
)()(
)(2),(
),(),(),(
BHAH
AMIBANMI
BAMIBAHBANMI
),(
)()(),(
BAH
BHAHBANMI
Geometry TransformationGeometry Transformation Image Coordinate transform:
The features (dimension, voxel size, slice spacing, gantry tilt, orientation) of images, which are acquired from different modalities, are not the same.
From voxel units (column, row, slice spacing) to millimeter units with its origin in the center of the image volume.
11000
0
0
0
1
'
'
'
),,(222120
121110
020100
z
y
x
aaa
aaa
aaa
z
y
x
zyxT
Target Image & Template ImageTarget Image & Template Image
Target Image Grid
j
i
y
x
Target ImagePhysical Coordinates
y’
x’
Template ImagePhysical Coordinates
Template Image Grid
j
i
Space Transform
Images from the same patientImages from the same patient
Images provided as part of the project: “Retrospective Image Registration Evaluation”, NIH, Project No. 8R01EB002124-03, Principal Investigator, J. Michael Fitzpatrick, Vanderbilt University, Nashville,
TN.
256 x 256 pixels
MRI-T2
128 x 128 pixels
PET
Target Image ?
Template Image ?
InterpolationInterpolation
Nearest NeighborNearest Neighbor
Tri-linear InterpolationTri-linear InterpolationPartial-Volume Interpolation Partial-Volume Interpolation
Higher order partial-volume Higher order partial-volume interpolationinterpolation
Evaluating similarity measure for Evaluating similarity measure for each transformationeach transformation
y
Template Image
Transform
x
y
Target Image
x
OptimizationOptimization
Powell’s Direction Set methodPowell’s Direction Set method
Downhill Simplex methodDownhill Simplex method
Multi-resolutionMulti-resolution
Why Multi-resolutionWhy Multi-resolution Methods for detecting optimality can not guarantee that Methods for detecting optimality can not guarantee that a global optimal value will be found.a global optimal value will be found.
Time to evaluate the registration criterion is proportional Time to evaluate the registration criterion is proportional to the number of voxels.to the number of voxels.
The result at coarser level is used as the starting The result at coarser level is used as the starting point for the finer level.point for the finer level.
Currently multi-resolution approaches:Currently multi-resolution approaches:Sub-sampling Sub-sampling
AveragingAveraging
WaveletWavelet
Registration Result (I)Registration Result (I)
A typical superposition of CT-MR images.
Left : before registration Right: after registration.
Rigid transformation (II)Rigid transformation (II)
A typical superposition of MR-PET images.
Left : before registration Right: after registration.
MammographyMammographyBreast cancer is the second leading cause Breast cancer is the second leading cause of death among women in USA.of death among women in USA.Detected in its early stage, breast cancer Detected in its early stage, breast cancer is most treatable.is most treatable.MammographyMammography is the main tool for is the main tool for detection and diagnosis of breast detection and diagnosis of breast malignances.malignances.It reduces breast cancer mortality by 25% It reduces breast cancer mortality by 25% to 30% for women in the 50 to 70 age to 30% for women in the 50 to 70 age group group
Mammogram RegistrationMammogram Registration
Temporal/bilateral mammograms Temporal/bilateral mammograms varyvary– Breast compressionBreast compression– Breast positionBreast position– Imaging TechniqueImaging Technique– Change in BreastChange in Breast
Mammogram registration Mammogram registration techniquestechniques
Whole breast area vs. regionalWhole breast area vs. regional
Nipple locationNipple location
Control-point locationControl-point location
Rigid & non-rigid registrationRigid & non-rigid registration
Non-rigid Mammogram RegistrationNon-rigid Mammogram Registration
Intensity-basedIntensity-based
Elastic transformationElastic transformation
Multi-resolutionMulti-resolution
Demons algorithm (Thirion, 1996)Demons algorithm (Thirion, 1996)
DemonsDemons
Scene (Target)
Model (Template)
Transform
Demons (Cont.)Demons (Cont.)
Scene
Model
Transform
Forces
Demons (Cont.)Demons (Cont.)
Scene
Gradient
Intensity
Space
Desired Displacement
CurrentEstimation
DemonsDemons From Optical Flow From Optical Flow
Scene: f, Model: gScene: f, Model: g
Assumption: The intensity of a Assumption: The intensity of a moving object is constant with time moving object is constant with time
(1)
(2)
Description of the ApproachDescription of the Approach1.1. Select demon points.Select demon points.2.2. Compute the force Compute the force uu on the model on the model
at each of the selected demonsat each of the selected demons3.3. Determine a global transformation Determine a global transformation
based on the computed based on the computed uu and apply and apply it to the modelit to the model
4.4. If the model images is now If the model images is now registered to the scene image, stop. registered to the scene image, stop. Else, go to Step 2.Else, go to Step 2.
Registration ComponentsRegistration Components
Image IntensitiesImage Intensities
Non-rigid transformation, one Non-rigid transformation, one displacement vector for each pixeldisplacement vector for each pixel
Bilinear interpolationBilinear interpolation
Absolute difference as similarity Absolute difference as similarity metricmetric
Multi-resolutionMulti-resolution
Dataset: MIAS,DDSMDataset: MIAS,DDSM
Demons Results (I) Synthetic Images
Level=2
Level=3
Level=4
Level=5
Demons Result (II) MIASDemons Result (II) MIAS
Before registration
After rigid registration
Original images
After non-rigid registration
Ongoing registration topicsOngoing registration topics
Trade-off of computation and Trade-off of computation and accuracyaccuracy
Evaluation of registration resultsEvaluation of registration results
Visualization of registrationVisualization of registration
Applications: Change DetectionApplications: Change Detection
Images taken at different timesImages taken at different times
Following registration, the Following registration, the differences between the images may differences between the images may be indicative of changebe indicative of change
Deciding if the change is really there Deciding if the change is really there may be quite difficult may be quite difficult
Other ApplicationsOther ApplicationsMulti-subject registration to develop Multi-subject registration to develop organ variation atlases.organ variation atlases.– Used as the basis for detecting Used as the basis for detecting
abnormal variationsabnormal variations
Object recognition - alignment of Object recognition - alignment of object model instance and image of object model instance and image of unknown object (segmentation)unknown object (segmentation)
ReferencesReferencesMaes F,Collignon A, et al. “Multimodality image Maes F,Collignon A, et al. “Multimodality image registration by maximization of mutual registration by maximization of mutual information.” information.” IEEE Trans. Med. ImagingIEEE Trans. Med. Imaging. 1997, . 1997, V16,pp187-198V16,pp187-198
L.G.Brown, “A survey of image registration L.G.Brown, “A survey of image registration techniques,” techniques,” ACM Computing SurveysACM Computing Surveys, vol. 24, , vol. 24, no. 4, pp. 325–376, 1992.no. 4, pp. 325–376, 1992.
Jean-Philippe Thirion, “Non-Rigid Matching Using Jean-Philippe Thirion, “Non-Rigid Matching Using Demons,” IEEE Conference on Computer Vision Demons,” IEEE Conference on Computer Vision and Pattern Recognition,1996and Pattern Recognition,1996
Top Related