Disparity Estimation Using A Color Segmentation V3

Disparity Estimation using aColor Segmentation

Xin Wang

Barcelona, 3rd Sep. 2009

Outline for the presentation

• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching

Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging

• Experiment Result and Analysis• Conclusions and Future Work• Questions

Task and Problem Identification

• Stereo Correspondence Problem Given an element in the left image, we search for the corresponding element

in the right image. Disparity is the displacement for pixel positions between two corresponding

points in these images. Epipolar Rectification allows simplifying the search problem from

2D to 1D.


• Challenges

Textureless Region : Matching in an untextured area becomes ambiguous, since there are many potential matching points of very similar intensity patterns.

Depth Discontinuity Region : Pixels whose neighbouring disparities differ by more than a given value, usually appear at the boundaries of different scene objects.

Occluded Region : Zones that are occluded in the matching image.


• Quality Measure : Reconstruction : To reconstruct the reference image in a back-ward way, is using reference image’s disparity map and the other view image for reconstruction.

Reconstruction Image PSNR (Peak Signal to Noise Ration) :

,where

L (ori) R (ori)

Disparity(Left)

Disparity(Right)


Occlusion mask is made from performing L-R check to groundtruth.

PSNR is Calculated in non-occluded region.

Algorithmic Overview

• Block Diagram of Algorithm

Initial Disparity Estimation

Color Segmentationto extract the label

Polynomial Model based DisparityRepresentation in Each Color

Segment (use label information)

Disparity ImprovementBy Region

Merging

Iterations

Final Disparity Map

Initial Disparity Estimation

• Window-based Local Matching : Matching Score : is a quantitative similarity measurement for two different

pixels. It is the criterion for the solution of correspondence problem.

Winner-take-all : disparity associated with the minimum cost value is

selected at each pixel.

Segmentation-based Approach

• Approach: Segmentation-based approaches divide one or sometimes both images into non-overlapping regions of homogeneous color.

• Assumptions: 1, Inside a segment of homogeneous colour the disparity values are expect

ed to follow some particular smooth disparity model (constant disparity, planar model, etc)

2, Disparity discontinuities are assumed to coincide with the boundaries

of those color regions.


• Polynomial Models for Each Segment Order 0 Polynomial Model :

Order 1 Polynomial Model :

Order 2 Polynomial Model :

(planar parallel to thecamera front plane)

(planar model)

(parabolic curved surface)


• Model Parameters Estimation

We use initial disparity as input data

Unknown Parameters: Order 0 : 1 parameter - > 1 point Order 1 : 3 parameters -> 3 points Order 2 : 6 parameters -> 6 points

Proposed Approaches for Model Fitting: Least Squares Random Sample and Consensus (RANSAC)


• Least Squares

Over-determined Problem : The number of points in each region is much higher than the number of equations needed to compute the analytical solution.

Best Fit for Observed Data : Sum of squared residuals has its least value. (but sensitive to outliers)

min( )


• RANSAC Non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability.

Assumption: A set of data is that the data consists of inliers and the data's contribution follows a certain set of model parameters, while the outliers do not fit the model.

Algorithm: Hypothesize-and-Test framework

HYPOTHESIZE: Minimal Sample Set (MSS) is randomly selected from initial input data.

TEST: Check the entire dataset with the estimated model from MSS. Data are consistent with the model are called consensus set (CS).


• RANSAC in Polynomial Model Fitting Polynomial Order 0 : Median filter Polynomial Order 1 and 2:

1. Select MSS : Randomly select 3 or 6 points from initial disparity points

2. Fit the Model by Solving Linear Equations System:

3. Select Inliers to Form the CS:

Distance calculation -> D = di – f (xi,yi,P) if below the distance threshold

4. Choose the best CS to Output Fitted Model: CS with most inliers


• Comparison between RANSAC & LSLS : Analytical solution, non-iterative solution. Less computation cost.

Very sensitive to outliers.

RANSAC: Robust to outliers, non-deterministic algorithm. Accuracy

will increase with increasing the iterations and strictness in thresholds.

More computation cost and the model may subject to

noise.

PSNR Comparison Test:

(Teddy 559 regions, Cones 553 regions, Venus 398 regions)

Disparity Refinement via Merging

• Problems : In some regions, due to unproper-segmentation,

occlusion, noise, lack of enough inliers, may result in

incorrect model estimation.

• Solution : Merge adjacent “good regions” to re-fit the model.

For lack of inliers region : enlarge the area , includes more inliers For occlusion area : get the support from neighbouring good region For small region : large distance separated points, more easy to fit a

correct model


• Merging Block Diagram :


• Region Similarity Measure Criteria Similarity in Model’s Parameters

Number of inliers

PSNR of Reconstructed Region

(Euclidean Distance Based)


• Implementation for similarity measure: Distance similarity in region’s parameters + PSNR criterion Distance similarity matrix will be calculated:

Due to distance similarity based on region’s parameters can not always be true, so I use PSNR criterion for actual merging step to secure merging process will always improve the disparity quality

Experiment Results

• Polynomial Model Fitting Compare with initial disparity, usually no improvement

Analysis : RANSAC works not well, due to initial color segmentation fails to meet the two assumptions, and some regions lack of enough inliers.

Parameters : Teddy (559 regions) , Cones (553 regions), Venus (398 regions)RANSAC iterations:200, inlier threshold:0.99 , distance threshold:0.5

Experiment Results

• Mixed Order Polynomial Model Fitting

Program runs on a 3GHz processor

Experiment Results

• Mixed Order Polynomial Model Fitting

Red: Order 0, Green: Order 1, Blue: Order 2,Black: No improvement

Experiment Results

• Polynomial order 2 Model Region Merging

Teddy (559 regions) , Cones (553 regions), Venus (398 regions)

Experiment Results

• Polynomial order 2 Teddy case

Experiment Results

• Polynomial order 1 model results:

Reconstruction using initial Disparity Reconstruction using merged disparity,160 rounds

Conclusions and Future Work

• Conclusions: LS is more sensitive to outliers, RANSAC is more robust Sometimes RANSAC will fail to estimate correct parameters, du

e to the segmented region’s size, noise etc. Small regions may not have enough inliers, region merging coul

d help Bad estimated regions could get support from neighbouring regi

ons in merging process

• Future work More accurate and computationally efficient region similarity me

asure should be further studied Try to do merging for all the possibilities for adjacent regions. Convert Matlab code to C++ version

Acknowledgement

Prof. Josep Ramon Morros i Rubio

Albert (Technical support)

All Professors and friends in TSC who

supported me before.

Questions

Thank you very much !

Questions?

Experiment Results

• Example of Region Merging Improvement (Teddy, order2)

A, improvement point

B, how label changes

C, disparity refinement

Experiment Results

• Drop point in the evolution curve (214 & 216region)

Experiment Results

• Order 1 Merging ->

Teddy

Experiment Results

• Poor Modeled Region in order2 Teddy

Disparity & Depth

• Disparity is proportional inversely to the depth

Experiment Results

• Local Matching How the local window size affect local matching

Too small window can not catch enough intensity variation to give the correct disparity in less-textured regions, too large window can not capture well small details

Teddy Image Cones Image Venus Image

Matching Score

• Matching Score: Matching measures such as SSD and SAD are strictly assuming the constant color con-straint. So if the computed area fails to meet this constraint, SSD and SAD measures may not be robust. While the other matching scores like gradient-based measure is more robust to changes in camera gain. So we could expect to combine the sum of absolute differences and a gradient based measure together to define the total matching score in order to increase the robustness.

• The weight w which balance the CSAD and CGRAD portions. It is determined by maxi-mizing the number of reliable corresponding pixel pairs that are filtered by performing cross-checking test.

Disparity Estimation Using A Color Segmentation V3

Documents

Transcript of Disparity Estimation Using A Color Segmentation V3