Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi...
-
Upload
silas-francis -
Category
Documents
-
view
217 -
download
2
Transcript of Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi...
![Page 1: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/1.jpg)
Natural Video Matting using Camera Arrays
Neel S. Joshi Wojciech Matusik Shai Avidan
Mitsubishi ElectricResearch Laboratory
University of California,San Diego
![Page 2: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/2.jpg)
![Page 3: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/3.jpg)
![Page 4: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/4.jpg)
Related Work
• Studio Matting– Vlahos 71, Wallace 82, Smith and Blinn 96, Ben-Ezra 00,
Debevec et. al 02, McGuire et al. 06
• User Assisted Natural Matting– Ruzon and Tomasi 00, Chuang et al. 01, Chuang et al. 02,
Sun et al. 04, Li et al. 04, Rother et al 04, Li et al. 05, Wang et al. 05, Wang and Cohen 05, Levin et al. 06
• Automated Natural Matting– Wexler et al. 02, Zitnick et al. 04, McGuire et al. 05, Kolmogrov
et al. 05
– Our Method
![Page 5: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/5.jpg)
Contributions
• Extending the matting equation to use the variance of pixel measurements
• Using a camera array for alpha matting
• An automatic method for computing alpha mattes that is linear in the number of pixels
• A near real-time system for natural video matting
– 2 to 5 FPS
![Page 6: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/6.jpg)
Algorithm Overview
Foreground Object
Background
Camera Array
![Page 7: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/7.jpg)
Using a Camera Array
![Page 8: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/8.jpg)
Foreground and Background Parallax
![Page 9: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/9.jpg)
Aligning to a Foreground Plane
Foreground Plane
I0
In
I0 InIi
Ii
![Page 10: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/10.jpg)
Aligned Images
![Page 11: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/11.jpg)
Variance across cameras
Taking the Variance
Background
Background
Foreground
Foreground
Transparent(Mixture)
Transparent
I0 InIi
![Page 12: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/12.jpg)
Key Idea
alpha is a function of: variance across cameras images
![Page 13: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/13.jpg)
Traditional Alpha Matting [Smith and Blinn 96]
I = F + (1 - )B
I = observed image
= transparency
F = foreground
B = background
![Page 14: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/14.jpg)
Alpha Matting with Multiple Images
I = F + (1 - )B
I, F, and B are continuous random variables
is a scalar
![Page 15: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/15.jpg)
I = F + (1 - )B
Solving for alpha
2var(F) + (1 - )2var(B) - var(I) = 0
var(I) = var[F + (1 - )B]
![Page 16: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/16.jpg)
Solving for alpha and F
2var(F) + (1 - )2var(B) - var(I) = 0
)(mean1)(meanF BI
a2 + b + c = 0
a = var(F) + var(B) b = -2var(B) c = var(B) - var(I)
![Page 17: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/17.jpg)
Triangulation Matting: A Special Case of our Method
I1 I2
21
211BBII
[Smith and Blinn 96]
2
)var(2
21 III
0)var( F
2
)var(2
21 BBB
)var()var(
1BI
![Page 18: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/18.jpg)
F is a function of: [mean(I) ,mean(B)]
Variance and Mean Statistics
• We observe I
• F and B can’t be directly measured
is a function of: [var(I) ,var(F) ,var(B)]?
?
?
![Page 19: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/19.jpg)
Foreground = Low var(I)
Background = High var(I)
Threshold var(I) to get Trimap
![Page 20: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/20.jpg)
Propagating Variance and Mean
For each pixel in the unknown (gray) region:
– var(F) = var(Inf)
where Inf is the nearest pixel in the foreground
– var(B) = var(Inb), mean(B) = mean(Inb)
where Inb is the nearest pixel in the background
![Page 21: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/21.jpg)
Algorithm Steps
1. Calibrate camera array and film scene
2. Find the foreground depth
3. Compute var(I) and mean(I)
4. Threshold var(I) to get trimap
5. Estimate var(F), var(B) , mean(B)
6. Compute and F
![Page 22: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/22.jpg)
System and Implementation Details
• 8 640x480 (Bayer pattern) Basler Video Cameras
• 1 Desktop PC (P4 3.0 Ghz)
• Calibration:
– Color w/ Macbeth chart
– Bundle Adjustment
• Average Running Time:
2 to 5 FPS at 320 x 240
![Page 23: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/23.jpg)
500
2500
4500
![Page 24: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/24.jpg)
Our method
Using a known background
![Page 25: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/25.jpg)
![Page 26: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/26.jpg)
![Page 27: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/27.jpg)
![Page 28: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/28.jpg)
![Page 29: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/29.jpg)
500
2500
4500
![Page 30: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/30.jpg)
Our method
Using a known background
![Page 31: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/31.jpg)
![Page 32: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/32.jpg)
![Page 33: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/33.jpg)
![Page 34: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/34.jpg)
Limitations
• Self-occlusion: alpha is not view independent
Known Backgroun
d
Our Method
• Need background variation, i.e., var(B) should be large
X
![Page 35: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/35.jpg)
Conclusions
• Standard matting equation extended to work with variance and mean statistics
• Running time is linearly proportional to the number of pixels
• Automatic, near real-time alpha matting using a camera array
![Page 36: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/36.jpg)
Acknowledgments
• SIGGRAPH Reviewers
• Joe Marks, John Barnwell, Frédo Durand, Matthias Zwicker, Bennett Wilburn, Dan Morris, and Merrie Morris
• Our willing and patient “Actors”: Lavanya Sharan, Chien-i Tu, Bernd Bickel, Yuanzhen Li
• MERL (for my internship)
• NSF IGERT Grant (for additional funding)
![Page 37: Natural Video Matting using Camera Arrays Neel S. JoshiWojciech MatusikShai Avidan Mitsubishi Electric Research Laboratory University of California, San.](https://reader036.fdocuments.in/reader036/viewer/2022081519/56649d005503460f949d2d15/html5/thumbnails/37.jpg)
http://graphics.ucsd.edu/papers/camera_array_matting/