Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays SAIT – INRIA...
-
Upload
michael-gordon -
Category
Documents
-
view
217 -
download
1
Transcript of Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays SAIT – INRIA...
Reconstruction with Depth and Color cameras for 3D
Autostereoscopic Consumer Displays
SAIT – INRIA collaborationPeriod: 15 July 2012 / 15 January 2013
Date: 3-4 December 2012
INRIA team
• Georgios Evangelidis, postdoc, 100%• Michel Amat, development engineer, 100%• Soraya Arias, senior development engineer, 20% • Jan Cech, posdoc, 20%• Radu Horaud, 10%
Past achievements
• A method and software for aligning TOF data with a stereoscopic camera pair
• Extension to the calibration of several TOF-stereo units
• 3D texture-based rendering of the TOF data using the color-image information
Publications
• One CVPR 2011 paper• A tutorial at ICIP 2011• One Springer Briefs book
just published• The two teams published
several other papers
Current achievements
• Finalization of the calibration & rectification methods/software
• TOF to stereo-pair mapping with filtering• TOF + texture in live mode• Disparity map initialization• Stereo correspondence based on seed-growing• Final high-resolution depth map with gap filling• A paper submitted to CVPR’13
Improved Calibration• New calibration board– mat sticker glued to a
rigid plane– plane attached to a tripod
• Refined Calibration algorithm– TOF-Stereo Calibration error: <1.5 pixel
• Improved Rectification– Rectification error:
<0.25 pixel
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
Correspondences (samples) obtained by using the calibration parameters
• each correspondence comes from a TOF point• different color -> different depth
Left-to-Tof Occlusions
TOF-to-Left Mapping is not perfect
Left-to-Tof Occlusions: the depth decreases from left to right
Tof-to-Left Occlusions
TOF-to-Left Mapping is not perfect
Tof-to-Left Occlusions: the depth increases from left to right
Point Cloud filtering• We reject points in left-to-tof occluded area• We keep the minimum-depth points in case of
overlap (due to Tof-to-left occlusions)
Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point
cloud…• …and initialize the stereo disparity map It looks good,
but it’s noisy and non-accurate!
Seed-Growing Idea• Start from points with known disparities
(seeds) and propagate the disparity to neighboring points (video?)
• Main issues:– What are our seeds?– What is the visiting order of seeds?– How do I propagate the message?– How the stereo and depth data are fused within
this framework?
Depth-Color Fusion • Built on the seed-growing idea– A:Depth data, S: Stereo data, dN : neighbor of d– For each pixel (node), find its disparity value that
maximizes the posterior probability (MAP)
SS
AA
d N
dInput data
Pixel with unknown disparity
Range-search constraintPenalize the choice
wrt to depth information
Penalize the choice wrt to color information
Pixel with known disparityA represents the initial estimation of d (obtained by the previous interpolation)S represents the color matching cost that corresponds to d
Depth-Color Fusion• Bayes rule translates each posterior into a
likelihood
• If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme
SS
AA
d N
d
Input data
Pixel with unknown disparity
Pixel with known disparity
We are currently working on these
terms!
Because of the uniform distribution
Because of theBayes rule
Seed-Growing Idea (revisited)• For each pixel, an energy function is defined and we
look for its minimizer (disparity)• Main issues:– What are our seeds?
• the points from Tof-to-Left mapping after refinement – What is the visiting order of seeds?
• First visit reliable seeds (points with low energy value)– How do we propagate the message?
• Given the disparity of a seed, bound the disparity-range for its neighbor
– How the stereo and depth data are fused within this framework?• Described above
Paper Submission
• Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR 2013.
• An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.
Work during the remaining month
• Improve the accuracy of the matching by better exploiting the color/texture information
• Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second
• An updated version will be available at the end of the period (~15 January 2013)
Prospects for the next collaboration(1 February 2013 – 31 January 2014)
• Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas
• Depth disambiguation using TOF-TOF and TOF-stereo • Combine depth disambiguation with the seed-growing
algorithm• Perform full 3D realistic rendering with four TOF-
stereo units• Perform continuous 3D reconstruction with a moving
TOF-stereo unit