Francesc Moreno-NoguerComputer Vision Lab.Ecole Polytechnique Fédérale de Lausanne
Peter N. BelhumeurShree K. NayarColumbia University
SIGGRAPH 2007
Active Refocusing of Images and Videos
study
Abstract
Use an active illumination method for depth estimation from a single image
Acquired ImageComputed Depth
NearFar
Refocused (Near)Refocused (Far)Alternate Lighting
Outlines
Introduction Related Work Overview Projection Dot Defocus Analysis Dot Removal & Depth Estimation Realistic Refocusing Result Limits and Conclusions
Introduction of Refocusing
Challenges of Active Refocusing Dynamic scenes
Depth Estimation be done in a single frame
Active illumination Full resolution depth map Projection Dot removal
Partial Occlusions
xfy k captured
blur kernels at depth k
In-focus
RELATED WORK
Relative Work: Depth EstimationPassive Methods
Active Illumination Methods
Shape from shading Cannot handle depth
discontinuities
Coded Aperture [Levin et al. SIGGRAPH 07] Cam. H.W. modify Require Light Source
Pattern
Structured Light [Salvi et al. Pattern Recognition ,04] No pattern removal
Projector Temporal Defocus [Zhang & Nayar SIGGRAPH06]
Relative Work: Digital RefocusingRefocusing Given Depth
Light Field Photography
Synthesis Images: Ray Tracing [Cook SIGGRAPH84] Require complete 3D model
Real Images: Convolution [Photoshop; IrisFilter] Partial Occlusions Problem
Light Field Camera [Ng SIGGRAPH05] Cam. H.W. modify Resolution losses
Dappled Photography [Veeraraghavan SIGGRAPH07] Cam. H.W. modify Layer
Sparse Depth Map
Acquired Image
Dots Removed
Color Segmentation
Merged Segmentation
Dense Depth
Dots Depth Estimation by
Calibration
Dots Removal
Matting
Depth Map Completion using Segmentation
Depth Estimation
Depth Map
Realistic Refocusing
Dots Removed
Focal plane,Apertures,Window size of dots
PROJECTION DOT DEFOCUS ANALYSIS
System Design
Camera & Projector Coaxial have same Optical Axis
Projector
Blur Circle Diameter, D
fc uurfD
112
uf
Dr
u
fcv
v
fwDD c
w
w
with dot size w (in the projector plane)
Blur Circle Radiance, I
21I
f
p
uu
uf
Dr
u
fcv
2
1
2
I
vr
wu
u
uw
f
w
w
with dot size w (in the projector plane)
based on Image Irradiance Equation derived in [Horn 86]
Camera images of dot of 3*3 pixels projected onto different depths
Camera images of dot of 3*3 pixels projected onto different depths
DOT REMOVAL AND DEPTH ESTIMATION
Calibration Patches
Estimated
Sparse Depth MapDepth 1Depth 2 …
X =
Calibration Patches
Estimated
Sparse Depth MapDepth 1Depth 2 …
Calibration Patches
Estimated
Sparse Depth MapDepth 1Depth 2 …
Depth Estimation - ux
Non-textured Surface
Textured Surfaces (texture by itself introduces brightness variation)
riN
j ijcic ,, II
riN
jijc
iix uu ,Ivarminarg|
ici
ix uu ,Iminarg|
based on Unsupervised Learning Alg. [Figueiredo and Jain IEEE02]
DEPTH MAP COMPLETION USING SEGMENTATION
Depth Map Completion
Sparse Depth Map
Over-Segmentation
Mean-Shift[Comaniciu & Meer 02]
Iterative Merging
Depth Map Completion – Iterative Merging
Loop: Apply Greedy alg. to group segments Merge the two most
similar neighboring segments
Re-computes the features of the new merged segment
Iterative Merging
Similarity between Segments
Sim(i,j)=λC∙dist(Ci,Cj)+λD∙dist(Di,Dj)+λT∙dist(Ti,Tj)
Color C Depth D Texture T
Depth Map Completion – Refine the Depth Disc.
Noisy Depth Map
Matting Algorithm[Wang & Cohen
05]
REALISTIC REFOCUSING
Challenge of Refocusing Partial occlusions
Different parts of the lens may see different views at an object boundary
Create missing region by detecting discontinuities in depth map and extending the occluded surface using texture synthesis
Foreground/background transitions Pixels at depth discontinuities may
receive contributions from the fr. and bg. Blend fr./bg. images within the
boundary region
A)(RRR 1A GCFC
)A-(1RARR GCFC
Realistic Refocusing produces better results than existing approaches
Original
Realistic Refocusin
g
Canon + wide
aperture
Photoshop - blur
IrisFilter
Partial Occlusions
Refocusing with Alpha Maps
R RC Є F* += *R
C Є B
Background (B)
Boundary (C)
Foreground (F)
RC Є F
RC Є B
RESULT
Limitations
Due to Active Illumination
Translucent objects exhibit subsurface scattering
Blurred dots are too weak to detect Very dark Highly inclined surface
(> 70°)
Poor in outdoor with strong sunlight
ex: the ball and the table are assigned diff. depths due to errors on segmentation errors
Limitations
Due to sparse dots Sparsity of the depth estimation
Errors in the initial segmentation of the image
ex: incorrect depth due to segmentation err.
Conclusions
Contribution Future Work
An active illumination depth estimation with single Single Frame, Complete
Depth Map, Texture/Textureless scenes
Projected Light Patterns are Removed
High resolution refocusing of images and videos
Incorporate the method into digital cameras
Use intra-red source for projecting the dot patter to make the depth estimation more robust in the case of highly textured scenes
END
Top Related