Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting...
Transcript of Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting...
![Page 1: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/1.jpg)
Prof. Feng Liu
Spring 2020
http://www.cs.pdx.edu/~fliu/courses/cs510/
05/07/2020
With slides by F. Durand, Y.Y. Chuang, R. Raskar, and C. Rother
![Page 2: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/2.jpg)
Last Time
Image segmentation
◼ Normalized cut and segmentation
2
![Page 3: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/3.jpg)
Today
Segmentation
◼ Interactive image segmentation
3Images from Rother et al. 2004
User
Input
Result
Magic Wand
(Photoshop)Intelligent Scissors
Mortensen and Barrett (1995)
GrabCutRother et al. 2004
![Page 4: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/4.jpg)
Start
Segmentation
◼ Interactive image segmentation
4Images from Sun et al. 2004
![Page 5: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/5.jpg)
Segmentation by Graph Cut
Interactive image segmentation using graph cut
Binary label: foreground vs. background
User labels some pixels
◼ usually sparser
Exploit
◼ Statistics of known Fg & Bg
◼ Smoothness of label
Turn into discrete graph optimization
◼ Graph cut (min cut / max flow)
F
B
F
F F
F B
B
B
5Slide credit: Y.Y. Chuang
![Page 6: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/6.jpg)
Energy function
Segmentation as Labeling
◼ one value per pixel, F or B
Energy(labeling) = data + smoothness
◼ Very general situation
◼ Will be minimized
Data: for each pixel
◼ Probability that this color belongs to F (resp. B)
Smoothness (aka regularization):
per neighboring pixel pair
◼ Penalty for having different label
◼ Penalty is down-weighted if the two
pixel colors are very different
◼ Similar in spirit to bilateral filter
One labeling
(ok, not best)
Data
Smoothness
6Slide credit: F. Durand
![Page 7: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/7.jpg)
Data term
A.k.a regional term
(because integrated over full region)
D(L)=i -log h[Li](Ci)
Where i is a pixel
Li is the label at i (F or B),
Ci is the pixel value
h[Li] is the histogram of the observed Fg
(resp Bg)
Note the minus sign
7Slide credit: F. Durand
![Page 8: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/8.jpg)
Hard constraints
The user has provided some labels
The quick and dirty way to include
constraints into optimization is to replace the
data term by a huge penalty if not respected.
D(L_i)=0 if respected
D(L_i)=K if not respected
◼ e.g. K= #pixels
8Slide credit: F. Durand
![Page 9: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/9.jpg)
Smoothness term
a.k.a boundary term, a.k.a. regularization
S(L)={j, i} in N B(Ci,Cj) (Li-Lj)
Where i,j are neighbors ◼ e.g. 8-neighborhood
(but I show 4 for simplicity)
(Li-Lj) is 0 if Li=Lj, 1 otherwise
B(Ci,Cj) is high when Ci and Cj are similar, low if there is a discontinuity between those two pixels◼ e.g. exp(-||Ci-Cj||
2/22)
◼ where can be a constant or the local variance
Note positive sign
9Slide credit: F. Durand
![Page 10: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/10.jpg)
Optimization
E(L)=D(L)+ S(L)
is a black-magic constant
Find the labeling that minimizes E
In this case, how many possibilities?
◼ 29 (512)
◼ We can try them all!
◼ What about megapixel images?
10Slide credit: F. Durand
![Page 11: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/11.jpg)
Labeling as a graph problem
Each pixel = node
Add two nodes F & B
Labeling: link each pixel to either F or B
F
B
Desired result
11Slide credit: F. Durand
![Page 12: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/12.jpg)
Data term
Put one edge between each pixel and F & B
Weight of edge = minus data term
◼ Don’t forget huge weight for hard constraints
◼ Careful with sign
B
F
12Slide credit: F. Durand
![Page 13: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/13.jpg)
Smoothness term
Add an edge between each neighbor pair
Weight = smoothness term
B
F
13Slide credit: F. Durand
![Page 14: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/14.jpg)
Min cut
Energy optimization equivalent to min cut
Cut: remove edges to disconnect F from B
Minimum: minimize sum of cut edge weight
B
F cut
14Slide credit: F. Durand
![Page 15: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/15.jpg)
Min cut <=> labeling
In order to be a cut:
◼ For each pixel, either the F or G edge has to be cut
In order to be minimal
◼ Only one edge label
per pixel can be cut
(otherwise could
be added)
B
Fcut
15Slide credit: F. Durand
![Page 16: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/16.jpg)
Computing a multiway cut
With 2 labels: classical min-cut problem◼ Solvable by standard flow algorithms
polynomial time in theory, nearly linear in practice
Code: C++ from OpenCV
◼ Matlab wrapper: http://www.wisdom.weizmann.ac.il/~bagon/matlab.html
◼ More than 2 terminals: NP-hard [Dahlhaus et al., STOC ‘92]
Code: http://vision.ucla.edu/~brian/gcmex.html
Efficient approximation algorithms exist◼ Within a factor of 2 of optimal
◼ Computes local minimum in a strong sense
even very large moves will not improve the energy
◼ Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.
16Slide credit: Y.Y. Chuang
![Page 17: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/17.jpg)
GrabCutInteractive Foreground Extraction
using Iterated Graph Cuts
Carsten Rother
Vladimir Kolmogorov
Andrew Blake
Microsoft Research Cambridge-UK
![Page 18: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/18.jpg)
Photomontage
GrabCut – Interactive Foreground Extraction 1
![Page 19: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/19.jpg)
Framework
Input: Image
Output: Segmentation
Parameters: Colour ,Coherence
Energy:
Optimization:
GrabCut – Interactive Foreground Extraction 4
![Page 20: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/20.jpg)
Graph Cuts
Boykov and Jolly (2001)
GrabCut – Interactive Foreground Extraction 5
Image
Min Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Foreground
(source)
Background
(sink)
![Page 21: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/21.jpg)
Iterated Graph Cut
User Initialisation
K-means for learning
colour distributions
Graph cuts to
infer the
segmentation
?
GrabCut – Interactive Foreground Extraction 6
![Page 22: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/22.jpg)
1 2 3 4
Iterated Graph Cuts
GrabCut – Interactive Foreground Extraction 7
Energy after each IterationResult
![Page 23: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/23.jpg)
Colour Model
Gaussian Mixture Model (typically 5-8 components)
Foreground &
Background
Background
Foreground
BackgroundG
GrabCut – Interactive Foreground Extraction 8
R
G
RIterated
graph cut
![Page 24: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/24.jpg)
Coherence Model
An object is a coherent set of pixels:
GrabCut – Interactive Foreground Extraction 9
Blake et al. (2004): Learn jointly
![Page 25: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/25.jpg)
Moderately straightforward
examples
… GrabCut completes automatically
GrabCut – Interactive Foreground Extraction 10
![Page 26: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/26.jpg)
Difficult Examples
Camouflage &
Low ContrastNo telepathyFine structure
Initial
Rectangle
Initial
Result
GrabCut – Interactive Foreground Extraction 11
![Page 27: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/27.jpg)
Evaluation – Labelled Database
Available online: http://research.microsoft.com/vision/cambridge/segmentation/
GrabCut – Interactive Foreground Extraction 12
![Page 28: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/28.jpg)
Comparison
GrabCutBoykov and Jolly (2001)
Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%
GrabCut – Interactive Foreground Extraction 13
User
Input
Result
![Page 29: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/29.jpg)
Summary
Magic Wand
(198?)
Intelligent Scissors
Mortensen and
Barrett (1995)
GrabCut
Rother et al.
(2004)
Graph Cuts
Boykov and
Jolly (2001)
LazySnapping
Li et al. (2004)
GrabCut – Interactive Foreground Extraction 22
![Page 30: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/30.jpg)
Interactive Digital Photomontage
Combining multiple photos
Find seams using graph cuts
Combine gradients and integrate
30
Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen, “Interactive Digital Photomontage”, SIGGRAPH 2004
Slide credit: Y.Y. Chuang
![Page 31: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/31.jpg)
31Slide credit: Y.Y. Chuang
![Page 32: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/32.jpg)
32Slide credit: Y.Y. Chuang
![Page 33: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/33.jpg)
33Slide credit: Y.Y. Chuang
![Page 34: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/34.jpg)
34Slide credit: Y.Y. Chuang
![Page 35: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/35.jpg)
35Slide credit: Y.Y. Chuang
![Page 36: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/36.jpg)
36Slide credit: Y.Y. Chuang
![Page 37: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/37.jpg)
photomontageset of originals
37Slide credit: Y.Y. Chuang
![Page 38: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/38.jpg)
Source images Brush strokes Computed labeling
Composite
38Slide credit: Y.Y. Chuang
![Page 39: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/39.jpg)
Brush strokes Computed labeling
39Slide credit: Y.Y. Chuang
![Page 40: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/40.jpg)
Student Paper Presentations
Presenter: Giampietro, James
◼ AutoCollage
C. Rother, L. Bordeaux, Y. Hamadi, A. Blake
SIGGRAPH 2006
Presenter: Green, Jordan
◼ Picture Collage
J. Wang, J. Sun, L. Quan, X. Tang, H. Shum
CVPR 2006
40
![Page 41: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture12.pdf · Next Time Matting Student paper presentations 05/12: Khan, Umairullah Video Tapestries with Continuous](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcfd281772a5845ef6b3d73/html5/thumbnails/41.jpg)
Next Time
Matting
Student paper presentations◼ 05/12: Khan, Umairullah
Video Tapestries with Continuous Temporal Zoom. C. Barnes, D.
Goldman, E. Shechtman, and A. Finkelstein
SIGGRAPH 2010
◼ 05/12: Kraiger, Keaton
Poisson image editing. P. Pérez, M. Gangnet, and A. Blake
SIGGRAPH 2003
◼ 05/14: Kwong, Marcus
Intelligent scissors for image composition. E. Mortensen and W. Barrett
SIGGRAPH 1995
◼ 05/14: Lajara, Armando
Learning to See in the Dark. C. Chen, Q. Chen, J. Xu and V. Koltun.
IEEE CVPR 2018
41