Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays
description
Transcript of Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays
Reconstruction with Depth and Color cameras for 3D
Autostereoscopic Consumer Displays
SAIT – INRIA collaborationPeriod: 15 July 2012 / 15 January 2013
Date: 3-4 December 2012
INRIA team
• Georgios Evangelidis, postdoc, 100%• Michel Amat, development engineer, 100%• Soraya Arias, senior development engineer, 20% • Jan Cech, posdoc, 20%• Radu Horaud, 10%
Past achievements
• A method and software for aligning TOF data with a stereoscopic camera pair
• Extension to the calibration of several TOF-stereo units
• 3D texture-based rendering of the TOF data using the color-image information
Publications
• One CVPR 2011 paper• A tutorial at ICIP 2011• One Springer Briefs book
just published• The two teams published
several other papers
Current achievements
• Finalization of the calibration & rectification methods/software
• TOF to stereo-pair mapping with filtering• TOF + texture in live mode• Disparity map initialization• Stereo correspondence based on seed-growing• Final high-resolution depth map with gap filling• A paper submitted to CVPR’13
Improved Calibration• New calibration board– mat sticker glued to a
rigid plane– plane attached to a tripod
• Refined Calibration algorithm– TOF-Stereo Calibration error: <1.5 pixel
• Improved Rectification– Rectification error:
<0.25 pixel
Given a calibrated TOF-Stereo system
• Each TOF point PT defines a correspondence between PL and PR
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
TOF-to-Left Mapping
• We use the left image as reference
TOF-to-Left Mapping is not perfect
Resolution mismatch
Left-to-Tof Occlusions
TOF-to-Left Mapping is not perfect
Left-to-Tof Occlusions: the depth decreases from left to right
Tof-to-Left Occlusions
TOF-to-Left Mapping is not perfect
Tof-to-Left Occlusions: the depth increases from left to right
Point Cloud filtering• We reject points in left-to-tof occluded area• We keep the minimum-depth points in case of
overlap (due to Tof-to-left occlusions)
Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point
cloud
Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point
cloud…• …and initialize the stereo disparity map It looks good,
but it’s noisy and non-accurate!
Seed-Growing Idea• Start from points with known disparities
(seeds) and propagate the disparity to neighboring points (video?)
• Main issues:– What are our seeds?– What is the visiting order of seeds?– How do I propagate the message?– How the stereo and depth data are fused within
this framework?
Depth-Color Fusion • Built on the seed-growing idea– A:Depth data, S: Stereo data, dN : neighbor of d– For each pixel (node), find its disparity value that
maximizes the posterior probability (MAP)
SS
AA
d N
dInput data
Pixel with unknown disparity
Range-search constraintPenalize the choice
wrt to depth information
Penalize the choice wrt to color information
Pixel with known disparityA represents the initial estimation of d (obtained by the previous interpolation)S represents the color matching cost that corresponds to d
Depth-Color Fusion• Bayes rule translates each posterior into a
likelihood
• If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme
SS
AA
d N
d
Input data
Pixel with unknown disparity
Pixel with known disparity
We are currently working on these
terms!
Because of the uniform distribution
Because of theBayes rule
Seed-Growing Idea (revisited)• For each pixel, an energy function is defined and we
look for its minimizer (disparity)• Main issues:– What are our seeds?
• the points from Tof-to-Left mapping after refinement – What is the visiting order of seeds?
• First visit reliable seeds (points with low energy value)– How do we propagate the message?
• Given the disparity of a seed, bound the disparity-range for its neighbor
– How the stereo and depth data are fused within this framework?• Described above
Examples
White areas: unreliable matches
Black areas: Occlusions
Examples (with gap filling)
Examples (with gap filling)
Examples (with gap filling)
Examples (with gap filling)
Paper Submission
• Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR 2013.
• An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.
Work during the remaining month
• Improve the accuracy of the matching by better exploiting the color/texture information
• Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second
• An updated version will be available at the end of the period (~15 January 2013)
Prospects for the next collaboration(1 February 2013 – 31 January 2014)
• Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas
• Depth disambiguation using TOF-TOF and TOF-stereo • Combine depth disambiguation with the seed-growing
algorithm• Perform full 3D realistic rendering with four TOF-
stereo units• Perform continuous 3D reconstruction with a moving
TOF-stereo unit