StereoMatching

Segment-based Belief Propagation

Iolanthe II racing in Waitemata Harbour

Segmentation

• #1 on the Middlebury rankings• Klaus, Sormann and Karner, “Segment-based Stereo Matching Using

Belief Propagation and a Self-Adapting Dissimilarity Measure”, ICPR (3) 2006: 15-18

• Basic idea• Depth (disparity) changes should occur at region boundaries in

the image

so

• Segment the image and match the patches

Algorithm Pair of rectified images

Extract homogenous regions in reference image

Apply local window based matching

Extract set of disparity planes

Approximate optimaldisparity plane assignment

Disparity map

Step 1 - Segmentation

• Decompose reference imageRegions of homogenous colour | grey-scale

• Assume• Disparity values vary smoothly within these

regions• Note: Possibly sloping planes

• Fronto-planar assumption not required!

• Mean-shift colour segmentation used

• Gradient descent search for maxima in a density function over a high dimensional feature space

Feature space: spatial coordinates + associated attributes(including edge information)

Step 2 – Local matching

• Local matching in pixel domain• Window based correlation (SSD or SAD)• Gradient or non-parametric matching

algorithms possibleBetter tolerate gain and offset changes

• KSK use self-adapting combination score• C(x,y,d) = (1 - ) CSAD(x,y,d) + Cgrad(x,y,d)

where

• CSAD(x,y,d) = i,jN | IL(i,j) – IR(i+d,j) |

• Cgrad(x,y,d) = i,jN |x IL(i,j) - xIR(i+d,j) | + i,jN |y IL(i,j) - yIR(i+d,j) |

LR Differencein Intensity

LR Difference in Horizontal Gradient

LR Difference in Vertical Gradient

Step 2 – Local matching

• KSK use self-adapting combination score• C(x,y,d) = (1 - ) CSAD(x,y,d) + Cgrad(x,y,d)

where

• CSAD(x,y,d) = i,jN | IL(i,j) – IR(i+d,j) |

• Cgrad(x,y,d) = i,jN |x IL(i,j) - xIR(i+d,j) | + i,jN |y IL(i,j) - yIR(i+d,j) |

• N(x,y) is a 3 3 window• Nx(x,y) is a 3 2 window; Ny(x,y) is a 2 3 window• Weighting, chosen

• Winner-take-all – choose disparity with lowest cost

+• Maximising number of reliable correspondences

filtered out by LR crosscheck

LR Differencein Intensity

LR Difference in Horizontal Gradient

LR Difference in Vertical Gradient

Step 2 – Local matching

• Weighting, chosen• Winner-take-all – choose disparity with lowest cost

+• Maximising number of reliable correspondences

filtered out by LR crosscheck• Normalized dissimilarity measure used

• Reliable correspondences used to estimate signal-noise ratio (SNR)

• Because of normalization, a fixed truncation threshhold is set just above the noise level

Robust matching score

Step 3 – Disparity plane estimation

• Allow sloping planes• Abandon fronto-planar surface assumption!

• Disparity plane estimation

• Specify the plane with c1, c2, c3 :

• d = c1x + c2y + c3

Large number of possible planes Use only reliable disparities• Decompose fitting problem:

Decompose fitting problem: Fit • Horizontal and vertical slant separately• Find d/x ( d/y ) for each pixel• Distribution of derivatives d/x ( d/y ) is convolved with

Gaussian kernel to determine mean d/x

• Slant in the centre of a segment then determined by finding distribution of centre disparities for each reliable point as before

Step 3 – Disparity plane estimation

• Allow sloping planes• Abandon fronto-planar surface assumption!

• Estimate disparity planes

• Specify a plane with c1, c2, c3 :

• d = c1x + c2y + c3

Large number of possible planes Use only reliable disparities• Decompose fitting problem: Fit

• Horizontal and vertical slant separately• Find d/x ( d/y ) for each pixel• Distribution of derivatives d/x ( d/y ) is convolved

with Gaussian kernel to determine mean d/x

• Slant in the centre of a segment then determined from distribution of centre disparities for each reliable point as before

Step 4 – Disparity plane refinement

• Increase the accuracy of the disparity plane• Group regions that belong to the same

disparity plane• Calculate a matching cost• For the plane, P, fitted to segment, S:

• CSEG(S,P) = (x,y)S C(x,y,d)

• Disparity plane with the minimum matching cost is assigned to each segment

• Segments assigned to that disparity plane are grouped

• Repeat for all grouped segments

For each pixel in segment, S

Mismatch costfor disparity, d

Step 5 – Disparity plane assignment• Search for optimal segment disparity place

assignment• Formulate as an energy minimization problem

• Find a labelling, f, which matches a segment, s R, to a plane, f(s) in D• ‘Energy’ for labelling f is

• E(f) = Edata(f) + Esmooth(f)where

• Edata(f) = sR CSEG(s,f(s))and

• Esmooth(f) = ((si,sj)SN|f(si) f(sj) ) disc(si,sj) disc(si,sj) – discontinuity penalty – includes common

border lengths and colour similarity

SN set of adjacent segments

Step 5

• Use loopy belief propagation to find optimal labeling with minimum energy

• Results:• Matching – state-of-the-art!• Top ‘average’ rank on the Middlebury set• Sloping planes of ‘Venus’ are well handled • Still has problems with edges!!

• See results in fig 2 of the paper

• Pixelization not handled?

• Computation cost?• Not mentioned in paper!• Algorithm contains many repeated steps• Analysis of benefit of each one useful