1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.
-
Upload
jeffrey-mckenzie -
Category
Documents
-
view
212 -
download
0
Transcript of 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.
![Page 1: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/1.jpg)
1
Roey Izkovsky
Yuval Kaminka
Matting
Helping Superman fly since 1978
![Page 2: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/2.jpg)
2
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 3: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/3.jpg)
3
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 4: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/4.jpg)
4
The matting problem - Motivation
Image and video editing
New backgroundComposite image
![Page 5: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/5.jpg)
5
The matting problem - Motivation
Image and video editing
Input image New image
![Page 6: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/6.jpg)
6
The matting problem
iiiii BFI )1(
– The separation of an image I intoI. Foreground object image F
II. Background image B
III.Alpha matte α – the opacity )10(
– Problem: extract F, B, α from image
hair fur
![Page 7: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/7.jpg)
7
Why is matting challenging?
• Under constrained problem:One equation, 3 unknowns
iiiii BFI )1(
We need to constrain the problem!
![Page 8: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/8.jpg)
8
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 9: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/9.jpg)
9
Previous work
Two types:
Known background Natural image
matting matting
![Page 10: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/10.jpg)
10
Known Background
• Blue screen Matting• Still under-constrained
– Solution: make more assumptions• “Foreground contains no blue”• Other foreground distribution assumption…
• Use two different backgrounds
• Main flaw: need to know the background…
iiiii BFI )1( Blue background Composite image
![Page 11: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/11.jpg)
11
Natural Image Matting
• The assumptions:– Smoothness of the alpha matte– GMM for the Background and Foreground
colors
• Initial estimate: trimap provided by the user
Input image Trimap
• Background
• Foreground
• Unknown
![Page 12: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/12.jpg)
12
Natural Image Matting
• The algorithms framework:– Estimate F, B distributions from close pixels– Find best α by some method
![Page 13: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/13.jpg)
13
Knockout
– Extrapolate F,B from close neighborhood
– Estimate α from calculated F, B values
![Page 14: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/14.jpg)
14
Bayesian
– Estimate F, B distributions in area
– Find best α matching distributions
CBFPBF
|,,maxarg,,
)(/)()()(,,maxarg,,
CPPBPFPBFCPBF
)()(,,maxarg,,
BPFPBFCPBF
![Page 15: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/15.jpg)
15
Bayesian
– P(F), P(B) from image samples
– P(C|F,B,α) using a distribution for C BFC )1(
)()(,,maxarg,,
BPFPBFCPBF
![Page 16: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/16.jpg)
16
Natural Image Matting
• Main flaw: Accurate trimap required• Tedious to provide manually
• Hard to extract automatically
In particular, not feasible to videos
Binary segmentation
Adding unknown region
Input imageTrimap
![Page 17: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/17.jpg)
17
Great.So let’s get started…
![Page 18: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/18.jpg)
18
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 19: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/19.jpg)
19
New Approach to Matting
Trimap reduces to scribbles
Two new methods– Iterative optimization approach
• Heuristic algorithmic optimization
– A closed form solution• Mathematical approach
Trimap Scribbles
![Page 20: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/20.jpg)
20
Iterative optimization approach
Jue Wang
Michael F. Cohen
![Page 21: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/21.jpg)
21
Iterative approach
![Page 22: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/22.jpg)
22
Iterative approach
• Score: fit to image data
+alpha matte smoothness
• Iteratively propagating estimated results.
![Page 23: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/23.jpg)
23
Iterative optimization - outline
• Initialize “work pixels” from scribbles
• Repeatedly:• Expand work pixels • Find best alpha matte
• Stop when finished
![Page 24: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/24.jpg)
24
Initialization
• Introducing:– ui - uncertainty variable
– Uc – work pixels
ui = 0
α = 0
Uc = {user scribbles}
ui = 0
α = 1
ui = 1
α = 0.5
)10( iu
![Page 25: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/25.jpg)
25
Optimization
Uc = {user scribbles + 15 pixel radius}
Our goal:
find α matte for Uc that minimizes the energy -
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
DataSmoothness
![Page 26: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/26.jpg)
26
Vd
Score for αp = α
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
N Possible values for F N Possible values for B
BF )1(
2))1(,( BFId p
Fw2
Fw3
Fw4
Fw1
Bw1
Bw2
Bw3
Bw4
Image color Ip
![Page 27: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/27.jpg)
27
Vd
• Fit measure of αp to Ip
• Score for αp = α :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
2
2
1 12 2
))1(,(exp
1
jip
N
i
N
j
Bj
Fi
BFIdww
N
Fi , Bj – possible values for F, B in the pixel
wFi, wB
j – corresponding weights
![Page 28: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/28.jpg)
28
Vd
F Samples
B Samples
2
2
1 12 2
))1(,(exp
1
jip
N
i
N
j
Bj
Fi
BFIdww
N
Fi , Bj – possible values for F, B in the pixel
wFi, wB
j – corresponding weights
)2/(
),(exp))(1(
2
2
r
ppspuw iii
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
p
α = 0.9
u = 0.2
α = 0.8
u = 0.3
α = 0.4
u = 0.5
α = 0.4
u = 0.4
α = 0.2
u = 0.3
α = 0.3
u = 0.3
α = 0.5
u = 1.0
What happens when there are
not enough F/B samples?
![Page 29: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/29.jpg)
29
Vd
• Score for αp = α :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
2
2
1 12 2
))1(,(exp
1)(
jipN
i
N
j
Bj
Fip
BFIdww
NL
• Discretize
• and normalize
},...,,{ 2521
j
jp
kpk
pd L
LV
)(
)(1)(
![Page 30: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/30.jpg)
30
Vs
• Matte smoothness :
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
)/)(exp(1),( 222121 ssV
![Page 31: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/31.jpg)
31
Iterative optimization – step 2
Uc = {user scribbles + 15 pixel radius}
Our goal: find α matte for Uc that minimizes the energy -
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
Uc Graph
Nodes = Pixels, Edges by 4-connectivity
![Page 32: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/32.jpg)
32
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
![Page 33: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/33.jpg)
33
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
αlog p
0-2
0.04-1.7
……
12.3
p q
mpq – message from p to q
t=0
y
0tpqm
)(),(min)(0 pqp
p
q kpd
kq
kps
k
kq
tpq VVm
Vector: p’s “opinion” for eachpossible α for q
![Page 34: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/34.jpg)
34
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
αlog p
0-1.6
0.04-1.2
……
12.2
p q
t=1
y
1tpqm
αlog p
01
0.040.7
……
1-2.1
0typm
BELIEF PROPAGATION
mpq – new message pq
myp – previous message yp
qrpr
kp
trp
kpd
kq
kps
k
kq
tpq
ppqp
p
q mVVm~
01 )()(),(min)(
![Page 35: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/35.jpg)
35
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=2,3,4…
y
tpqm
1typm
qrpr
kp
trp
kpd
kq
kps
k
kq
tpq
ppqp
p
q mVVm~
1 )()(),(min)(
![Page 36: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/36.jpg)
36
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=T (stopping time)
y
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.6
0.04-1.1
……
11.3
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.7
0.04-1.3
……
11
qr
qtrq
kqd kmV q
~
1 )()(
![Page 37: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/37.jpg)
37
Iterative optimization – step 2
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
GOAL: Minimize
BELIEF PROPAGATION
p q
t=T (stopping time)
y
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.4
0.04-1.5
……
11.3
αLog p
0-1.7
0.04-1.3
……
11
αLog p
0-1.7
0.04-1.3
……
11
Best state calculated for each node:
prp
trp
kpd
kp kmV p
p ~
1 )()(minarg
![Page 38: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/38.jpg)
38
Iterative optimization – step 3
),()(
~,
qp
qpUqp
sUp
pd
cc
VVV
Found α matte for Uc that minimizes the energy -
Update F, B and uncertainty:
**
1
))1((minarg,
*
2
,
**
Bj
Fi
jipBF
wwu
BFIBFji
![Page 39: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/39.jpg)
39
Iterative optimization - algorithm
•Initialize Uc, F, B, u and alpha matte from scribbles
•Repeatedly:•Expand Uc by another 15 pixel radius•Find best alpha matte (BP)•Update F,B,u for new matte
•Stop when total uncertainty is minimal
Initial matte Propagation of α matte Final matte
![Page 40: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/40.jpg)
40
Iterative optimization - Results
Input image Extracted matte
![Page 41: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/41.jpg)
41
Iterative optimization - Results
Input image
Extracted matte
Composite image
![Page 42: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/42.jpg)
42
Iterative optimization - Results
The ambiguity bunny
![Page 43: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/43.jpg)
43
Ambiguity bunny with trimap
Iterative optimization - Results
Scribbles result Trimap resultAmbiguity bunny with scribbles
![Page 44: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/44.jpg)
44
Iterative optimization - Summary
• Minimal user input• Applicable to video
• Sensitive to ambiguity in F, B• Uses simple color-model
• Performance:– 15-20 min. on a 640x480 image– Factor 50 reported by better implementation
![Page 45: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/45.jpg)
45
Fantastic.Let’s go on…
![Page 46: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/46.jpg)
46
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 47: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/47.jpg)
47
Closed form solution
Anat Levin
Dani Lischinski
Yair Weiss
![Page 48: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/48.jpg)
48
Closed form solution
• Assumption: local smoothness in F, B
cancel out unknowns from the matte equs.
• Solve for F,B and alphausing algebraic tricks.
![Page 49: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/49.jpg)
49
Closed form solution
Assumptions:– F,B locally smooth.
treat F,B as constant in a small window w
ww
ww
www
wiwiwiwii
BF
Bb
BFa
bIawiBFI
1
)1(
![Page 50: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/50.jpg)
50
Closed form solution
GOAL:
Minimize -
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
-Numerical stability
-Bias to smoother matte
wj
wiwi bIawi
ww
ww
www BF
Bb
BFa
1
![Page 51: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/51.jpg)
51
Closed form solution
• GOAL:– Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
211 )( bIa pp
299 )( bIa pp
29 ia
![Page 52: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/52.jpg)
52
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
3N Variables (N = image size)
We can rid a, b by algebraic manipulation
![Page 53: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/53.jpg)
53
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Theorem: for we have
),,(min)(,
baJJba
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
LJ T)(
Intuitively, L is some covariance matrix
kwjik k
kjki
kij
II
wL
),|(2
))((
||
1~
![Page 54: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/54.jpg)
54
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Proof: Rewrite in matrix form:
k ww
w
w kk
k
kb
a
I
I
baJ
2
||
1
2/1||
1
00
1
1
),,(
![Page 55: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/55.jpg)
55
Closed form solution
• Minimize:
Ij wiwwiwi
j
jjjabIabaJ 22)(),,(
Proof: Rewrite in matrix form:
0
1
1
2/1||
1
k
k
ww I
I
G
2
),,(
k
w
w
w
w k
k
k
k b
aGbaJ
0||
1
k
k
ww
By mean-least-squares, best a,b pair
for each window is:
kkkkkkw
Tww
Twww GGGba 1** )(),(
k
wwTww
Tww kkkkkk
GGGG2
1)(
),,(min)(,
baJJba
![Page 56: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/56.jpg)
56
Closed form solution
• Some more manipulation give the required result
LJ T)(
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
EXCITED?
GET YOUR I LOVE MATHT-SHIRT, NOW FOR ONLY $1999
k
wTww
Tww kkkkk
IGGGGJ2
1 ))(()(
![Page 57: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/57.jpg)
57
Closed form solution
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
21
21
)1(
)1(
wiwii
wiwii
BBB
FFF
Color line model:
OUT: F, B Constant within a window
IN: F, B are on some line
BGRc
wci
cwi bIawi
,,
R
GF1
F2
![Page 58: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/58.jpg)
58
Closed form solution
c
wci
cwi bIawi
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
![Page 59: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/59.jpg)
59
Closed form solution
c
wci
cwi bIawi
Ij wi c
cww
c
ci
cwi
j
jjjabIabaJ 22 )()(),,(
Now, as before, cost is:
And a,b can be cancelled out.
• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.
Use redundancy to allow a “color-line” model per window
![Page 60: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/60.jpg)
60
Closed form solution
Now problem reduced to finding best α for:
LJ T)(
k kwjik
kjki
kwkijij II
wL
),|(2
||
))((1
1||
1
L is Huge size NxN (N = # image pixels)
But Sparse…
![Page 61: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/61.jpg)
61
Closed form solution
• The algorithm:– Compute L– Solve for given the scribbles.
• Solving a sparse set of bilinear equationswith constraints (Lagrange multipliers)
– Find F, B given the matte• Adding smoothness assumptions on F, B
• Improvements:– Use larger environment in low cost by “pyramids”
LTminarg
![Page 62: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/62.jpg)
62
Closed form solution - Results
Input image Extracted matte
![Page 63: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/63.jpg)
63
Closed form solution - Results
Input image with scribbles
Problematic matte
![Page 64: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/64.jpg)
64
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
L is positive definite.
Eigenbasis: v1,…,vN
Eigenvalues: λ1 > λ2 > … > λN > 0
2211
211 ...)( NNNN
t
ii
LJ
v
![Page 65: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/65.jpg)
65
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
![Page 66: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/66.jpg)
66
Eigenvectors as guides
Small eigenvectors of L are
correlated with minimal matte
can guide user scribbles
Eigenvectors matching smallest eigenvalues
Guided scribbles Resulting matte
![Page 67: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/67.jpg)
67
Closed form solution - Summary
• Minimal user input• Provable optimality (under assumptions)• Assumes only smooth F,B (no color model)
• Applicable to video (as we speak…)
• Problematic with textures
• Performance:– 20-40 seconds for a 200x300 image– Expensive in memory
![Page 68: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/68.jpg)
68
Superb.Let’s sum up…
![Page 69: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/69.jpg)
69
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 70: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/70.jpg)
70
Comparison
Iterative approachPoisson Closed form solution
Input image
Matte ground truth
![Page 71: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/71.jpg)
71
Main improvements
Trimap based approaches
New approaches
User inputTrimapScribbles
Complex foreground
Poor results. Exact trimap required
Good results
VideoNot easily applicable.Applicable
![Page 72: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/72.jpg)
72
Comparison
Color ambiguity
Iterative approach Closed form Sensitive Sensitive
Solvable by adding more scribbles
![Page 73: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/73.jpg)
73
Comparison
Improving results…
Iterative approachBayesian Closed form solution
Ambiguity bunny
![Page 74: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/74.jpg)
74
Comparison
Optimality?
Iterative approach Closed form
Uses heuristics
to optimize
Provably optimal
But for the specific
(simplified) cost
![Page 75: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/75.jpg)
75
Comparison
Textures
Iterative approach Closed form
Assumes only
Alpha matte smooth
F,B must satisfy
color-line model
![Page 76: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/76.jpg)
76
Comparison
Rough edges
Iterative approach Closed form
Assumes
Alpha matte smooth
Can handle rough
edges
Input image with scribbles
matte results
![Page 77: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/77.jpg)
77
Comparison
Running time
Iterative approach Closed form
~20 sec. 20/40 seconds
Costly in memory
(For medium size image)
![Page 78: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/78.jpg)
78
Comparison
Tests
Iterative approach Closed form
No quantitative
results reported
Extensively tested
quantitative results
![Page 79: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/79.jpg)
79
Outline
• The matting problem• Previous work• New approaches:
– The iterative approach
Jue Wang, Michael F.Cohen
– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss
• Comparison and summary• Bonus?
![Page 80: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/80.jpg)
80
Environment Matting and Compositing
Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin
![Page 81: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/81.jpg)
81
Environment Matting
C = F + (1- )B + ~ Contribution of light from Environment
that travels through the object
R – reflectance imageT – Texture image
![Page 82: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/82.jpg)
82
Environment Matting?
Alpha Matte Environment Matte Photograph
![Page 83: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/83.jpg)
83
Environment Mattin
Alpha Matte Environment Matte Photograph
![Page 84: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/84.jpg)
84
Summary
• The matting problem• Old methods: require trimap• Two new methods from scribbles:
– Iterative optimization• Assume: matte smooth, F,B locally similar• Use heuristic optimization for alpha
– Close form solution• Assume: F, B locally smooth (color-line model)• Solve linear equations for alpha
![Page 85: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/85.jpg)
85
ANY LAST
![Page 86: 1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978.](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2d6a/html5/thumbnails/86.jpg)
86