Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin...
1
DENOISING OF VOLUMETRIC DEPTH CONFIDENCE FOR VIEW RENDERING Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin Kumar Rana, and Markus Flierl School of Electrical Engineering, KTH Royal Institute of Technology, Stockholm, Sweden 1 MOTIVATION Free-Viewpoint Television Digital Camera Virtual Camera Depth Map Estimation Stereo Matching Stereo Matching view ♯1 view ♯2 view ♯3 view ♯4 view ♯5 depth map ♯2 depth map ♯4 Inconsistency of Multiview Depth Imagery view ♯1 view ♯2 view ♯3 (a) Dancer. (b) Newspaper. 2 APPROACH Estimated Depth Maps depth map ♯1 depth map ♯2 depth map ♯n Volumetric Depth Confidence Perspective Projection Camera 1 Camera 2 Camera 3 A(+1) B(+2) C(+3) D(+2) E(+1) (0) Occluded Area -2 -2 -2 -2 -2 -2 -2 -2 -2 Transparent Medium Example: Confidence Assignment to Voxels • Each mapped depth pixel from a viewpoint con- tributes +1 as confidence for a voxel • Each occluded pixel from a viewpoint does not change the confidence for a voxel • Each ray traversing in a transparent voxel con- tributes -1 as confidence for a voxel 3 3D WAVELET DENOISING c Disney Why wavelet denoising? • It can create sparse coefficients and spreads out i.i.d. noise equally among all the coefficients • It can distinguish between signal and noise effi- ciently Wavelet Denoising Process Volume Decomposition Thresholding Volume Reconstruction Wavelet Coefficients Modified Coefficients Noisy Volume Discrete Wavelet Transform (DWT) Denoising Methods Inverse DWT Denoised Volume Adaptive Thresholding Using SURE Shrink Goal : Determine thresholds that remove noise efficiently • SURE (Stein’s Unbiased Risk Estimator) is used to estimate sub–band adaptive thresholds • For multivariate normal observations, SURE offers an unbiased estimate of the expected squared er- ror of the mean • Let {c i : i =1,...,l } be the noisy wavelet coeffi- cients in the j-th sub-band. SURE(θ ; c )= l − 2|{i : |c i | <θ }| + l i min(|c i |,θ ) 2 , where θ is a given threshold and |·| denotes the cardinality of a set • Set the SURE threshold θ S by θ S = arg min θ SURE(θ ; c ) 4 VIEW RENDERING Virtual View Hole Filling Denoised Volume Perspective Projection center depth map virtual center view left depth map right depth map left view right view 3D Warping -by checking depth consistency -by using 3 × 3 median filter 5 EXPERIMENTAL RESULTS Objective Results Rendered Views with Confidence Range [-3,3] Noisy Volume db1 Th.=0.01 db2 Th.=1.40 db3 Th.=1.20 db3 Adaptive db4 Th.=1.00 38.87 38.7 38.71 38.72 38.87 38.64 31.84 31.33 31.42 31.43 32.08 31.38 PSNR [dB] Dancer Newspaper Effect of Confidence Range on Newspaper Rendered Views MPEG Depth Maps Noisy Volume db3 Adaptive 31.63 31.84 32.08 32.04 32.33 32.32 PSNR [dB] [−3, +3] [−7, +7] Subjective Results Rendered views of Newspaper with Confidence Range [-7,+7] (a) Original view. (b) With MPEG depth maps. (c) With noisy depth volume. (d) With denoised volume. c Disney ... making important notes! • Adaptive thresholding in the 3D wavelet domain improves the quality compared to rendering with MPEG depth maps and noisy volumetric data, respectively • For synthetic scenes, the improvement is insignif- icant due to consistent description of the geom- etry • Volumetric depth confidence improves rendering results even without wavelet denoising when pro- jecting highest confidence voxels into the camera plane 6 CONCLUSIONS • Define volumetric depth confidence to handle in- consistent depth estimates • Use superposition principle to incorporate confi- dence information from multiple camera views • Denoise volumetric depth confidence by using adaptive 3D wavelet thresholding • Improve the visual quality of rendered views by using the denoised volumetric depth confidence
Transcript of Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin...
DENOISING OF VOLUMETRIC DEPTH CONFIDENCE FOR VIEW RENDERING
Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin Kumar Rana, and Markus Flierl
School of Electrical Engineering, KTH Royal Institute of Technology, Stockholm, Sweden
1 MOTIVATION
Free-Viewpoint Television
Digital Camera
Virtual Camera
Depth Map Estimation
Stereo
Matching
Stereo
Matching
view ♯1 view ♯2 view ♯3 view ♯4 view ♯5
depth map ♯2 depth map ♯4
Inconsistency of Multiview Depth Imageryview ♯1 view ♯2 view ♯3
(a) Dancer.
(b) Newspaper.
2 APPROACH
Estimated Depth Maps
depth map ♯1 depth map ♯2 depth map ♯n
Volumetric Depth Confidence
PerspectiveProjection
b
b
b
b
b
Camera 1 Camera 2 Camera 3
A(+1)
B(+2)
C(+3)
D(+2)
E(+1)
(0) Occluded Area
-2-2
-2
-2
-2
-2
-2
-2
-2
TransparentMedium
Example: Confidence Assignment to Voxels
•Each mapped depth pixel from a viewpoint con-tributes +1 as confidence for a voxel
•Each occluded pixel from a viewpoint does notchange the confidence for a voxel
•Each ray traversing in a transparent voxel con-tributes -1 as confidence for a voxel
3 3D WAVELET DENOISING
c© Disney
Why wavelet denoising?
• It can create sparse coefficients and spreads outi.i.d. noise equally among all the coefficients
• It can distinguish between signal and noise effi-ciently
Wavelet Denoising Process
VolumeDecomposition
Thresholding
VolumeReconstruction
Wavelet Coefficients
Modified Coefficients
Noisy Volume
DiscreteWaveletTransform(DWT)
DenoisingMethods
InverseDWT
Denoised Volume
Adaptive Thresholding Using SURE Shrink
Goal: Determine thresholds that remove noise efficiently
•SURE (Stein’s Unbiased Risk Estimator) is usedto estimate sub–band adaptive thresholds
•For multivariate normal observations, SURE offersan unbiased estimate of the expected squared er-ror of the mean
•Let {ci : i = 1, . . . , l} be the noisy wavelet coeffi-cients in the j-th sub-band.
SURE(θ; c) = l − 2|{i : |ci| < θ}| +l∑
i
min(|ci|, θ)2,
where θ is a given threshold and | · | denotes thecardinality of a set
•Set the SURE threshold θS by
θS = argminθ
SURE(θ; c)
4 VIEW RENDERING
Virtual
View
Hole
Filling
DenoisedVolume
Perspective Projection
center depth map
virtual center view
left
depth
map
right
depth
map
left view right view
3DWarping
-by checking depth consistency
-by using 3× 3 median filter
5 EXPERIMENTAL RESULTS
Objective Results
Rendered Views with Confidence Range [-3,3]
Noisy
Volume
db1
Th.=0.01
db2
Th.=1.40
db3
Th.=1.20
db3
Adaptive
db4
Th.=1.00
38.87 38.7 38.71 38.72 38.87 38.64
31.8431.33 31.42 31.43
32.08
31.38
PSNR
[dB]
Dancer
Newspaper
Effect of Confidence Range on NewspaperRendered Views
MPEG
Depth Maps
Noisy Volume db3 Adaptive
31.63
31.84
32.0832.04
32.33 32.32
PSNR
[dB]
[−3,+3]
[−7,+7]
Subjective Results
Rendered views of Newspaper with ConfidenceRange [-7,+7]
(a) Original view. (b) With MPEG depth maps.
(c) With noisy depth volume. (d) With denoised volume.
c© Disney
... making important notes!
•Adaptive thresholding in the 3D wavelet domainimproves the quality compared to rendering withMPEG depth maps and noisy volumetric data,respectively
•For synthetic scenes, the improvement is insignif-icant due to consistent description of the geom-etry
•Volumetric depth confidence improves renderingresults even without wavelet denoising when pro-jecting highest confidence voxels into the cameraplane
6 CONCLUSIONS
•Define volumetric depth confidence to handle in-consistent depth estimates
•Use superposition principle to incorporate confi-dence information from multiple camera views
•Denoise volumetric depth confidence by usingadaptive 3D wavelet thresholding
• Improve the visual quality of rendered views byusing the denoised volumetric depth confidence
