Feature based deformableregistration of neuroimages using

interest point and feature selection

Leonid Teverovskiy

Center for Automated Learning and DiscoveryCarnegie Mellon University

Description of the problem

Our task is to align given neuroimages so that their corresponding anatomical structures have the same coordinates

Description of the problem

Our task is to align given neuroimages so that their corresponding anatomical structures have the same coordinates

Existing approaches

• Landmark based registration. Deformation between images is calculated based on user defined correspondences between certain points, curves or surfaces on the neuroimages.- Not fully automatic.- Transformation of non-landmark points is interpolated from the transformation of landmark points.

Landmark based registration

Existing approaches

• Registration driven by a similarity measure. Deformation model is parameterized and then a numerical optimization procedure is used to find parameters that maximize some similarity measure. - Automatic, but prone to local maxima.- The more degrees of freedom deformation model has, the harder it is to find optimal parameters for it.

Affine registration driven by sum of square differences of the images

SSD: 0.0085

Sometimes it works

SSD: 0.0063

Affine registration driven by sum of square differences of the images

Sometimes it works

SSD: 0.0289

Affine registration driven by sum of square differences of the images

Sometimes it works

SSD: 0.0188

Affine registration driven by sum of square differences of the images

Sometimes it does not

Existing approaches

• Feature based registration. A feature vector is computed for each voxel. Correspondences between voxels in the reference image and voxels in the input image are estimated based on the simliarity of their feature vectors.- Best results among existing methods.- Existing systems have many hand tuned parameters, including components of feature vectors.

Feature based registration

Feature based registration

Our goals.

• Fully automatic method that selects which features to use depending on

- modality of the images;

- anatomical structures we care to register the most

• No restriction on the degrees of freedom of the deformation model

A few questions…Would it be a hard task to register these two images for a human?

Would it be a hard task to register these two images for a human?

Not a hard task

A few questions…

OK, then how about registering this image with a rotated copy of itself?

A few questions…

OK, then how about registering this image with a rotated copy of itself?

Looks much harder

We can get some idea about how difficult a registration task will be even without seeing the other image!

A few questions…

Not an easy task indeed

A few questions…

If there were some points that “stood out”, we could easily find what the rotation was…

A few questions…

If there were some points that “stood out”, we could easily find what the rotation was… provided we can determine correspondences correctly.

A few questions…

We can solve both problems

using the same mechanism

We are facing two different problems:

• How to find interesting points in the reference image automatically.

• How to find corresponding points in the input image.

Feature Extraction.

…

We compute various rotationally invariant features at different scales.

[Feature Vector] h(Pi|F)

Probability that given feature vector “belongs” to a certain pixel

in the reference image

A pixel in the input image Most likely correspondences in the reference image

If we knew h(Pi|F) we could do this :

We could find h(Pi|F) …

…if we knew what g(F|Pi) and q(Pi) were.

Probability of observing feature F at the pixel Pi Prior

Prior q(Pi)

• if we have a reason to believe that certain pixels in the reference image are more likely to correspond to the given pixel in the input image, we can express our beliefs through prior.

• we will use uniform prior for now.

We can estimate g(F|Pi)! • We have applied about 1560 affine transforms to the

reference image and computed features for each pixels in each of the resulting 1560 images.

• Thus we obtain 1560 feature vectors for each anatomical location in the reference image.

• We assume that components of feature vector are independent of each other and distributed according to a gaussian distribution.

• We find MLE of mean and variance for each gaussian • g(F|Pi) is a product of these gaussians.

[Feature Vector] h(Pi|F)

A pixel in the input image Blue dots represent correspondences with probability of 0.4999. Probability of other

correspondences is negligibly small

We are almost done; we need to have a way of distinguishing good correspondences …

[Feature Vector] h(Pi|F)

A pixel in the input image Blue dots represent correspondences with probability of 0.4999. Probability of other

correspondences is negligibly small

… from bad correspondences

Risk=∑h(Pi|F)L(Pi, Po)

• L(Pi, Po) – loss, which is a geometric distance between estimated corresponding pixel Pi and the correct corresponding pixel Po. • When Po is unknown, we use MAP estimate of Po instead.• Correspondences with low risk are “good” correspondences

Risk

Where are we now?

• We can find correspondences between pixels in the input image and pixels in the reference image using feature vectors computed on the pixels of the input image.

• And we can also determine interesting points by finding correspondences between pixels in the reference image and pixels in the … reference image!

Feature Selection

• Select feature subset for determining interesting points.

• Select feature subset for determining correspondences.

• Use sum of square differences (quality of registration) as a means of evaluating feature subsets.

Interesting point feature subset

Reference image

Input image

Correspondence feature subset

Interesting points

CorrespondencesRANSAC

h(Pi|F)

Affine Transform

TPS transform

Driving voxels

Registration quality

Birds eye view.

feedback

Feature pool

Experimental results.

Reference Image Input Image

Interesting points

Best correspondences

Driving voxels

Reference image Registered input image

Registration results

SSD: 0.0047

Difference image

More experiments

1. Select random set of interest points; select random subset of features to find correspondences

2. Select random set of interest points; use forward feature selection to find subset of features to be used for estimating correspondences

3. Select random subset of features to find interest points; select random subset of features to find correspondences

More experiments

4. Select random subset of features to find interesting points; use forward selection to choose subset of features used for determining the correspondences.

5. Select random subset of features to find interesting points; use forward selection to choose subset of features used for determining the correspondences. This time start from the subset used to find interesting points without one feature.

More experiments

6. Select random subset of features to find interesting points. Then employ forward selection for choosing subset of features to be used for determining the correspondences. Find a new set of interest points using this subset of features and iterate.

7. Select random subset of features to find interesting points. Then employ forward selection for choosing subset of features to be used for determining the correspondences. This time start from the subset used to find interesting points without one feature. Find a new set of interest points using this selected subset of features and iterate

More experiments.

For each feature selection strategy we run registration eight times, each time restarting at a random point.

Each run continues for 20 iterations.

Feature pool.22 features, all at the finest scale (for now).

4. First derivative (D1)9. Second derivative (D2) 14. Third derivative (D3)19. Fourth derivative (D4)24. Fifth derivative (D5)29. Gabor_0_3 (G1)34. Gabor_0_5 (G2)39. Gabor_2_7 (G3)44. Gabor_3_7 (G4)49. Gabor_4_9 (G5)54. Laplacian (L)

59. Harris (H)64. Intensity_1_mean (M1)69. Intensity_1_std(S1)74. Intensity_2_mean(M2)79. Intensity_2_std(S2)84. Intensity_4_mean(M3)89. Intensity_4_std(S3)94. Intensity_8_mean(M4)99. Intensity_8_std(S4)104. Intensity_16_mean(M5)109. Intensity_16_std(S5)

“Intensity_n_mean” is the mean of the pixel intensities inside a ring with inner radius log2(n) and outer radius n, centered at the given pixel.

“Intensity_n_std” is the standard deviation of the pixel intensities inside a ring with inner radius log2(n) and outer radius n, centered at the given pixel.

Feature selection significantly improves registration results. Here, a 100 interest points were used

Random IP Random edge IP IP selection

Registration error when 30 interesting points are used. Interesting points are selected 1) at random from all the image pixels, 2) at random from image pixels that lie on edges, 3) using interesting point selection;

Interest point selection has greater positive effect on the registration accuracy when number of interesting points is decreased.

Typical graph for the case when feature selection strategy number 7 is used. Brown line shows registration error if affine deformation is used, green line – when thin plate spline deformation is used.

Feature pool consists of 22 features. 8 features appear to be enough for good registration results.

Histogram of selected interesting point features when feature selection strategy number 7 is used

Histogram of selected correspondence features when feature selection strategy number 7 is used

Histogram of selected interesting point features when feature selection strategy number 6 is used

Histogram of selected correspondence features when feature selection strategy number 6 is used

Reference slice Input slice

Reference slice and input slice are midsagittal slices of neuroimages of different subjects. In addition, input slice was affinely transformed.

Registration results at each step of feature subset selection

Registration results at each step of feature subset selection

{M4} {M4, S1} {M4, S1, S5}

{M4, S1, S5, M5} {M4, S1, S5, M5, H} {M4, S1, S5, M5, H, G2}

SSD: 0.04719 SSD: 0.04043 SSD: 0.01954

SSD: 0.01644 SSD: 0.01777 SSD: 0.01625

Thank you