Multiresolution stereo Multiresolution stereo image matching using image matching using

complex waveletscomplex wavelets

Julian MagareyJulian Magarey

CRC for Sensor SignalCRC for Sensor Signal

and Information and Information ProcessingProcessing

Anthony Dick

Dept of Computer Science

University of Adelaide

Stereo Vision ProblemStereo Vision Problem

A Stereo PairA Stereo Pair

AIM: To recover 3D shape from stereo pair

Stereo MatchingStereo Matching

Find a point in each image which Find a point in each image which represents the same point in the scenerepresents the same point in the scene• corresponding pointscorresponding points

correspondingpoints

disparity

Feature Based MatchingFeature Based Matching

Detect and match distinctive featuresDetect and match distinctive features ProblemsProblems

• featureless areasfeatureless areas• occluded featuresoccluded features• same feature may appear differentsame feature may appear different

Multiresolution MatchingMultiresolution Matching

Match points at several levels of detailMatch points at several levels of detail

MATCH MATCHMATCH

CO

AR

SE

FIN

E

LeftImage

RightImage

Wavelet TransformWavelet TransformOriginal ImageResolution i,j

Level 1Res i/2, j/2

Level 2Res i/4, j/4

1A 1,1D 1,2D 1,3D 1,4D 1,5D 1,6D

2A 2,1D 2,2D 2,3D 2,4D 2,5D 2,6D

Multiresolution MatchingMultiresolution Matching

Now have multiresolution representationNow have multiresolution representation Level Level mm similarity distance measure: similarity distance measure:

6

1

2),(),()( )'()()',(n

mnmnm xDxDxxSD

where

x is a pixel in the level m representation of the left image

x’ is a pixel in the level m representation of the right image

Similarity distance surfaceSimilarity distance surface

Can extrapolate similarity surface about Can extrapolate similarity surface about x’ x’

00)(

2

1)',( fffffxxSD Tm

the surface minimum,

the location of the surface minimum,

a 2x2 curvature matrix,

are derived from

0f

where

6...1,)'(,)( ,, ixDxD mimi

SDf ,0

'x

Stereo Matching AlgorithmStereo Matching Algorithm

Now have basic matching algorithmNow have basic matching algorithm• perform wavelet transform on imagesperform wavelet transform on images• minimise minimise SD(x,x’)SD(x,x’) for all for all xx at top level at top level• use as starting point for finer level use as starting point for finer level

matchingmatching What if top level match is wrong?What if top level match is wrong? How do we interpolate matches to finer How do we interpolate matches to finer

level?level?

Coping with MismatchesCoping with Mismatches

Find a field of disparity vectors which minimisesFind a field of disparity vectors which minimises

uuu apsm EEE where

uapEis a directed measure of the difference between {u} and the unsmoothed disparity field

usmE is a measure of the uniformity of {u}

is a scalar controlling their relative influence

}{u

Regularisation FeaturesRegularisation Features

Based on Anandan [Based on Anandan [IJCV, 1989IJCV, 1989]] Use curvature matrix Use curvature matrix κκ to smooth more to smooth more

in directions of less certaintyin directions of less certainty

SDf ,0

Smooth more in this direction

Smooth less in this direction

Similarity surface contours

Coarse-to-fine Coarse-to-fine interpolationinterpolation

Robust disparity interpolationRobust disparity interpolation

= level m pixel = level m+1 pixel

D(a) = choice of {D(A), D(B), D(C), D(D)} whichminimises similarity distance

A B

C D

a

c

b

d

ResultsResults

Calibrated camera setupCalibrated camera setup Projective reconstructionProjective reconstruction Form textured VRML surfaceForm textured VRML surface

C1 C2

Left Camera Right Camera

3D Surface

ResultsResults

ResultsResults

ConclusionConclusion

Future workFuture work• lightinglighting• geometric constraint incorporationgeometric constraint incorporation• colour imagescolour images• camera self-calibrationcamera self-calibration• more than two imagesmore than two images

Already, results are promising!Already, results are promising!