Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research...
-
date post
15-Jan-2016 -
Category
Documents
-
view
219 -
download
0
Transcript of Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research...
![Page 1: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/1.jpg)
Samuel W. Hasinoff Sing Bing Kang Richard Szeliski
Interactive Visual Media GroupMicrosoft Research{sbkang,szeliski}@microsoft.com
Dept. of Computer ScienceUniversity of [email protected]
Boundary Matting for View Synthesis
2nd Workshop on Image and Video Registration, July 2, 2004
![Page 2: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/2.jpg)
MotivationSuperior view synthesis & 3D editing from N-view stereo
Key approach: occlusion boundaries as 3D curves
• More suitable for view synthesis• Boundaries estimated to sub-pixel
Two major limitations – even with perfect stereo!• Resampling blur• Boundary artifacts
![Page 3: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/3.jpg)
B2B3
Matting problem: Unmix the foreground & background
Matting from Stereo
BFC )1(
Triangulation matting (Smith & Blinn, 1996)
• multiple backgrounds• fixed viewpoint & object
F
B1
Extension to stereo• Lambertian assumption
F
B3B1 B2
underdetermined
![Page 4: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/4.jpg)
Occlusion Boundaries in 3D Model boundaries as 3D splines (currently linear) Assumptions
boundaries are relatively sharp relatively large-scale objects no internal transparency
view 1 view 3view 2 (reference)
3D world
![Page 5: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/5.jpg)
Geometric View of Alpha
alpha partial pixel coverage on F side
simulate blurring by convolving with 2D Gaussian
otherwise,0
0)(,1)(
xdx
),0()(),( Gxx
j
j x)(
alpha depends only on projected 3D curve, x
integration over each pixel
F B
pixel j
![Page 6: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/6.jpg)
Related Work Natural image matting [Chuang et al., 2001]
based on color statistics
Intelligent scissors [Mortenson, 2000]
geometric view of alpha
- single image- user-assisted
![Page 7: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/7.jpg)
Related Work Bayesian Layer estimation [Wexler and Fitzgibbon, 2002]
matting from multiple images using triangulation + priors
- requires very high-quality stereo- alpha calculated at pixel level, only for reference - not suitable for view synthesis
![Page 8: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/8.jpg)
Boundary Matting Algorithm
3D world
view 1 view 3view 2 (reference)
find occlusion boundary in reference view backproject to 3D using stereo depth project to other views initial guess for Bi and F optimize matting
optimize
![Page 9: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/9.jpg)
Initial Boundaries From Stereo Find depth discontinuities Greedily segment longest four-connected curves
Spline control points evenly spaced along curve
Tweak - snap to strongest nearby edge
![Page 10: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/10.jpg)
Background Estimation
F
B1 B2
Use stereo to grab corresponding background-depth pixels from nearby views (if possible)
Color consistency check to avoid mixed pixels
B3
occluded
![Page 11: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/11.jpg)
Foreground Estimation
Invert matting equation, given 3D curve and B
Aggregate F estimates over all views
viewsviews
ii
iii FF
1
2
1
2 )(ˆˆ
BCF )1()(ˆ
BFC )1(
![Page 12: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/12.jpg)
Optimization
Objective: Minimize inconsistency with matting
over curve parameters, x, and foreground colors, F
Pixels with unknown B not included Non-linear least squares, using forward differencing
for Jacobian
views pixels
i j
jijijjiji BαFαCO1 1
2))(1()(),( xxFx
![Page 13: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/13.jpg)
Additional Penalty Terms Favor control points at strong edges
define potential field around each edgel
Discourage large motions (>2 pixels) helps avoid degenerate curves
![Page 14: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/14.jpg)
Naïve object insertion (no matting)
![Page 15: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/15.jpg)
Object insertion with Boundary Matting
![Page 16: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/16.jpg)
Naïve object insertion (no matting)
Object insertion with Boundary Matting
![Page 17: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/17.jpg)
Naïve object insertion (no matting)
Object insertion with Boundary Matting
boundaries calculated with subpixel accuracy
![Page 18: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/18.jpg)
Samsung commercial sequence
![Page 19: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/19.jpg)
Naïve object insertion (no matting)
Object insertion with Boundary Matting
![Page 20: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/20.jpg)
Boundary Matting Naïve method
![Page 21: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/21.jpg)
Boundary Matting Naïve method
![Page 22: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/22.jpg)
boundary mattingboundary matting (sigma = 13)boundary matting (sigma = 26)compositebackgroundno matting
Synthetic Noise
![Page 23: Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com Dept. of Computer.](https://reader035.fdocuments.in/reader035/viewer/2022081520/56649d565503460f94a34d24/html5/thumbnails/23.jpg)
Concluding Remarks
Boundary Matting better view synthesis refines stereo at occlusion boundaries subpixel boundary estimation
Future work incorporate color statistics extend to dynamic setting