MIT AI Lab / LIDS

Laboatory for Information and Decision Systems &Artificial Intelligence LaboratoryMassachusetts Institute of Technology

A Unified Multiresolution Framework for

Automatic Target Recognition

Eric Grimson, Alan Willsky, Paul Viola, Jeremy S. De Bonet, and John Fisher

MIT AI Lab / LIDS

Outline

• Review Multiresolution Analysis Models– MAR (Multiresolution Auto-Regressive)

– MNP (Multi-scale Nonparametric)

• Applications of MNP Models– Synthesis and Super-Resolution

– Segmentation and Multi-Look Registration

– Classification/Recognition

• Continuing Efforts

MIT AI Lab / LIDS

V(x,y)={ }

coarse fine

Parent Vector

Multiresolution parent vector

NNM

NNNNNNN

NNMNNNNNNN

M

yxF

yxF

yxF

yxF

yxF

yxF

yxFyxFyxFyxV

2,

2,,

2,

2,

2,

2

,2

,2

,,2

,2

,2

,2

,,,,,,,,

10

11111

1110

1

01

00

0

MIT AI Lab / LIDS

Compare the Distribution of Parent Vectors

MIT AI Lab / LIDS

Formally...

...,,),(

2,

2,,

2,

2,, 21 mmmm

yxm

yxm

m WWyxV

• Parent Vector

),(

),(

),(

),(2

1

0

yxV

yxV

yxV

yxV

MIT AI Lab / LIDS

Freeman and Simoncelli

Steerable Pyramids

yxlWI ,,, yxlWI ,,,

MIT AI Lab / LIDS

Oriented Wavelet Pyramid

MIT AI Lab / LIDS

…for a SAR image

MIT AI Lab / LIDS

Capturing Structure (Texture Perspective)

MIT AI Lab / LIDS

Synthesis Results

MIT AI Lab / LIDS

Synthesis Results

MIT AI Lab / LIDS

MIT AI Lab / LIDS

Ergodic/Stationary

• A texture is assumed to be many samples of a single process – Each sample is almost certainly dependent on the other

samples

– But actual location of the samples does not matter

– (Space invariant process).

MIT AI Lab / LIDS

Heeger and Bergen

yxlyxll

yxlyxlyxl

WIP

WIPIP

,,,,,,,

,,,,,,,,,)(

yxlyxll

yxlyxlyxl

WIP

WIPIP

,,,,,,,

,,,,,,,,,)(

(.):

}]{[:

,:

,

,,,,

l

yxll

PModel

WIOObserve

WIGiven

(.):

}]{[:

,:

,

,,,,

l

yxll

PModel

WIOObserve

WIGiven

MIT AI Lab / LIDS

Heeger and Bergen Texture Synthesis Model

MIT AI Lab / LIDS

orig

inal

text

ure

patc

h

synt

hesi

zed

text

ure

patc

h

Sam

plin

g P

roce

dure

Analysis Synthesis

MIT AI Lab / LIDS

Not quite right...

MIT AI Lab / LIDS

Wavelet Representation of Edges

WaveletTransform

MIT AI Lab / LIDS

Pyramid Representation

MIT AI Lab / LIDS

Conditional Distributions

WaveletTransform

MIT AI Lab / LIDS

Probabilistic Model

)...,(),,(|),(

),(}{|),(

21 yxVyxVyxVP

yxVWIyxVP

lll

ll

)...,(),,(|),(

),(}{|),(

21 yxVyxVyxVP

yxVWIyxVP

lll

ll

...),,|(

),|(

)|()()()(

213

12

1

MMMM

MMM

MMM

VVVVP

VVVP

VVPVPWIPIP

...),,|(

),|(

)|()()()(

213

12

1

MMMM

MMM

MMM

VVVVP

VVVP

VVPVPWIPIP

yx

llll yxVyxVyxVPVP,

21 ...),(),,(|),( yx

llll yxVyxVyxVPVP,

21 ...),(),,(|),(

Markov

ConditionallyIndependent

SuccessiveConditioning

MIT AI Lab / LIDS

Estimating Conditional Distributions

...),(),,(

...),(),,(),,(

...),(),,(

...),(),,(),,(

...),(),,(|),(

21*

21*

21

21

21

yxVyxVP

yxVyxVyxVP

yxVyxVP

yxVyxVyxVP

yxVyxVyxVP

ll

lll

ll

lll

lll

...),(),,(

...),(),,(),,(

...),(),,(

...),(),,(),,(

...),(),,(|),(

21*

21*

21

21

21

yxVyxVP

yxVyxVyxVP

yxVyxVP

yxVyxVyxVP

yxVyxVyxVP

ll

lll

ll

lll

lll

i

ixxRxP )()(* i

ixxRxP )()(*• Non-parametrically

MIT AI Lab / LIDS

distribution Similarity

sample

Likelihood

distribution condition

exampleimage

synthesis

discrimination

registration

segmentation

denoising

super resolution

Outline

MIT AI Lab / LIDS

orig

inal

text

ure

patc

h

Sam

plin

g P

roce

dure

Analysis Synthesis

synt

hesi

zed

text

ure

patc

h

MIT AI Lab / LIDS

Multiresolution progression

MIT AI Lab / LIDS

MIT AI Lab / LIDS

Joint feature occurrence across resolution

MIT AI Lab / LIDS

Joint feature occurrence across resolution

MIT AI Lab / LIDS

MIT AI Lab / LIDS

MIT AI Lab / LIDS

Texture Synthesis Results

MIT AI Lab / LIDS

MIT AI Lab / LIDS

MIT AI Lab / LIDS

Registration pipeline

Tie-pointdetermination

Multiresolution texture match:flexible histograms

Multiresolutionalignmentsearch

Inputs are first equalized to remove imaging artifacts

Tie-point regions, which provide important matching information, are determined

A coarse-to-fine alignment search is used to bring the images into registration

Quality of registration is measured by comparing the flexible histogram texture match at the landmark regions.

MIT AI Lab / LIDS

• Distinctive regions provide significant constraint on the correct registration, while more recurrent areas provide little or no useful information.

Localized objects (such as structures or vehicles) match only few locations, thus providing strong constraints on registration.

Extended elements (e.g. roads or tree-lines) match a small area, providing a one dimensional constraint.

Common elements (e.g. grass or forest) match large portions of the image, and provide almost no useful information.

• Using the only the most distinctive regions as tie-points, reduces the computational requirements and increases the performance of most registration algorithms.

• By determining those regions which have low expected mutual information with other regions in the image,

• tie-points are found automatically

Tie-point determination

modelyxyx

;IEargmin

MIT AI Lab / LIDS

Using this metric, automatically determined tie-points correspond to the visual landmarks that a human observer would use.

Here, only vehicles provide distinct landmarks.

When present, roads and buildings provide useful landmarks as well.

Tie-point examples

MIT AI Lab / LIDS

Coarse Fine

At fine resolutions the registration objective function has many local

maxima, causing gradient based techniques to be highly sensitive to

initial “seeding” conditions.

At coarser resolutions there are fewer local maxima; however, the global maximum tends to be less accurate.

Coarse to fine alignment

MIT AI Lab / LIDS

Coarse-to-Fine Registration

In practice, actual data does tend conform to our qualitative assumptions.

Coarse resolutions lead to smooth, but inaccurate surfaces, while high resolutions are less smooth, but more accurate.

Coarse Fine

MIT AI Lab / LIDS

At each location in the tie-point region a parent vector is extracted. This vector consists of the multiresolution wavelet decomposition at that location. By measuring the frequency with which locations with similar parent structures occur, a flexible histogram is extracted.

V(x,y)={ }

coarse fine

Parent Vector

Measuring Visual Structure : Flexible Histogram I

MIT AI Lab / LIDS

Rtie-point

Rtest

parent structure

B (x,y)= 8

The registration objective function is the difference in visual structure between the tie-points and the corresponding regions, this is measured with the flexible histogram.

Measuring Visual Structure : Flexible Histogram II

MIT AI Lab / LIDS

A difference measure is acquired by comparing the histogram for the test region, measured with respect to the tie-point, to the the histogram for the tie-point measured with respect to itself.

Rlandmark

B(,x,y)= 8

2= (B-B’)2/B

B’(x,y)= 3

Rtie-point

Rtest

Measuring Visual Structure : Flexible Histogram III

MIT AI Lab / LIDS

Example Registration

MIT AI Lab / LIDS

Example Registration

MIT AI Lab / LIDS

Example Registration

MIT AI Lab / LIDS

Statistical target discrimination

distribution

image analysis

likelihood estimator

likelihood / similarity

When compared against a threshold value, this measure provides a discrimination function; comparison against the likelihoods of distributions from other model images, provides a classification mechanism.

IMODEL ITEST

MIT AI Lab / LIDS

Flexible histogram

IMODEL

IMODEL

parent vector

By measuring the frequency with which locations with similar parent vectors occur, a flexible histogram is extracted.

B (x,y)= 8

MIT AI Lab / LIDS

IMODEL

IMODEL

ITEST

Discrimination via histogram comparison

B(x,y)= 8

2 = (B-B’)2/B

The histogram for the image, measured with respect to the model, is compared to the the histogram for the model measured with respect to itself.

B’(x,y)= 3

MIT AI Lab / LIDS

qpD

yxIVyxIVR

yxIVyxIVR

yxB

yxByxB

yx

yx

yx

yx

||',',',,,

',',,,,

,

,',

,

','

','

,

22

otherwise

if

0

01 where

Flexible Histogram The frequency of locations in the image which have a parent structure, whose components are each within a threshold of the parent vector of some location in the model, is given by:

2S

A difference measure is calculated by taking chi-square difference between each such frequency count in the model and test image which approximates of the Kullbach-Liebler divergence.

Similarity can be measured by simply negating the distance.

','

1, ',',',,

1

yxyx DyxIVyxIVT

ZB

MIT AI Lab / LIDS

BMP2-C21 BTR70-C71 T72-132

Models

Models for target vehicles were generated from example images:

• generated from vehicles with different numbers from the target vehicles• only 10 examples, evenly distributed in heading angle• measured at a depression angle of 17 degrees (targets were at 15 degrees)

MIT AI Lab / LIDS

BMP2-9563 BMP2-9566

BTR70-C71

T72-812 T72-S7

Target vehicles

• Five target vehicles were used.

• Vehicles which differed from the target class were included as confusion targets.

• There were roughly 200 images in each class.

MIT AI Lab / LIDS

2S1 BRDM2 D7 T62

ZIL131 ZSU23

Confusion vehicles

Six additional confusion vehicles were used as well.

MIT AI Lab / LIDS

BMP2-C21 BTR70-C71 T72-132

Flexible Histograms

Template Matching

MIT AI Lab / LIDS

distribution | | condition

Prior beliefs about natural images

image analysis

Image + Noise

maximum likelihood sample

Blind Image Denoising