K-Means, Expectation Maximization and...
Transcript of K-Means, Expectation Maximization and...
![Page 1: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/1.jpg)
K-Means, Expectation Maximization and
SegmentationD.A. Forsyth, CS543
![Page 2: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/2.jpg)
K-Means
• Choose a fixed number of clusters• Choose cluster centers and point-cluster allocations to
minimize error
• can’t do this by search• there are too many possible allocations.
• Algorithm• fix cluster centers; allocate points to closest cluster• fix allocation; compute best cluster centers• x could be any set of features for which we can compute a distance
(careful about scaling)
€
x j − µ i2
j∈elements of i'th cluster∑
i∈clusters
∑
![Page 3: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/3.jpg)
K-means clustering using intensity alone and color alone
Image Clusters on intensity Clusters on color
![Page 4: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/4.jpg)
K-means using color alone, 11 segments
Image Clusters on color
![Page 5: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/5.jpg)
K-means usingcolor alone,11 segments.
![Page 6: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/6.jpg)
K-means using colour andposition, 20 segments
![Page 7: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/7.jpg)
Mixture models and segmentation
• In k-means, we clustered pixels using hard assignments• each pixel goes to closest cluster center• but this may be a bad idea
• pixel may help estimate more than one cluster
• We will build a probabilistic mixture model
P (x|µ1, . . . , µk,!1, . . . ,!k,!) =!
i
!iP (x|µi,!)
![Page 8: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/8.jpg)
Mixture model
• Interpretation:• obtain pixel by
• choosing a mixture component • given that component, choosing pixel
• Natural to have each mixture component be a Gaussian
P (x|µ1, . . . , µk,!1, . . . ,!k,!) =!
i
!iP (x|µi,!)
![Page 9: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/9.jpg)
Mixture components
• Gaussians• are oriented “blobs” in the feature space• we will assume covariance is known, and work with mean• expression below
P (x|µi,!) ! exp"(x" µi)!!1(x" µi)
2
![Page 10: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/10.jpg)
Problem: Learning and IDLL
• We must estimate the mixture weights and means• Maximising likelihood is very hard
• in this form, sometimes known as incomplete data log-likelihood
L(!) =!
i
log P (xi|!) =!
i
log
"
#!
j
"jP (xi|µj ,!)
$
%
![Page 11: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/11.jpg)
Learning and CDLL
• Introduce hidden variables to get complete data log-likelihood• d_{ij} is 1 if data item i comes from blob j
• 0 otherwise• Learning would be easy if we knew which blob each data item came from
• weights: count what fraction came from which blob• means: average data items
• But we don’t
Lc(!, "ij) =!
ij
"ij log (#jP (xi|µj ,Σ))
![Page 12: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/12.jpg)
Working with CDLL
• Notice:• with an estimate of the parameters, can estimate blobs data came from
• this could be a soft estimate• we could plug this in, then reestimate the parameters
• Formal procedure:• start with estimate• form Q function, below (The E-step)
• maximise in parameters (The M-step)• Iterate
Q(!; !(n)) = EP (δ|x,θ(n)) (Lc(!, "))
![Page 13: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/13.jpg)
The E-step for Gaussian mixtures
• Notice that the expression for the Q function simplifies
EP (!|x,"(n)) (Lc(!, ")) = EP (!|x,"(n))
!
"#
ij
"ij [log #j + log P (xi|µj ,!)]
$
%
=
!
"#
ij
P ("ij = 1|x, !(n)) [log #j + log P (xi|µj ,!)]
$
%
![Page 14: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/14.jpg)
The E-step for Gaussian mixtures
• We rearrange using probability identities
P (!ij = 1|x, "(n)) =P (x, !ij = 1|"(n))
P (x|"(n))
=P (x|!ij = 1, "(n))P (!ij = 1|"(n))
P (x|"(n))
=P (x|!ij = 1, "(n))P (!ij = 1|"(n))
P (x|!ij = 1, "(n))P (!ij = 1|"(n)) + P (x, !ij = 0|"(n))
![Page 15: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/15.jpg)
The E step for Gaussian mixtures
• And substitute
P (xi|!ij = 1, "(n)) =1Z
exp!(xi ! µ(n)
j )!!1(xi ! µ(n)j )
2
P (xi, !ij = 0|"(n)) =!
k !=j
#k1Z
exp!(xi ! µ(n)
k )!"1(xi ! µ(n)k )
2
P (!ij = 1|"(n)) = #j
![Page 16: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/16.jpg)
• We must maximise
• in the mixture weights and in the means• we can drop log Z
The M step for Gaussian mixtures
!
ij
P (!ij = 1|x, "(n)) [log #j + log P (xi|µj ,!)] =
!
ij
P (!ij = 1|x, "(n))"log #j !
!(xi ! µj)!!1(xi ! µj)2
! log Z
#
![Page 17: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/17.jpg)
• differentiate, set to zero, etc.• regard the expectations as “soft counts”
• so mixture weights from soft counts as:
• and means from soft counts as:
Two ways
!j =!
i P ("ij = 1|xi, #(n))!i,j P ("ij = 1|xi, #(n))
µj =!
i xiP (!ij = 1|xi, "(n))!i,j P (!ij = 1|xi, "(n))
![Page 18: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/18.jpg)
Figure from “Color and Texture Based Image Segmentation Using EM and Its Application to Content Based Image Retrieval”,S.J. Belongie et al., Proc. Int. Conf. Computer Vision, 1998, c1998, IEEE
Segmentation with EM
![Page 19: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/19.jpg)
Affinity matrix
![Page 20: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/20.jpg)
Good split
![Page 21: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/21.jpg)
Measuring Affinity
Intensity
Texture
Distance
€
aff x, y( ) = exp − 12σ i
2
I x( )− I y( ) 2( )
€
aff x, y( ) = exp − 12σ d
2
x − y 2( )
€
aff x, y( ) = exp − 12σ t
2
c x( )− c y( ) 2( )
![Page 22: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/22.jpg)
Scale affects affinity
![Page 23: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/23.jpg)
• Simplest idea: we want a vector a giving the association between each element and a cluster
• We want elements within this cluster to, on the whole, have strong affinity with one another
• We could maximize
• But need the constraint • This is an eigenvalue problem - choose the eigenvector of A with largest
eigenvalue
Eigenvectors and cuts
€
aTAa
€
aTa = 1
![Page 24: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/24.jpg)
Example eigenvector
points
matrix
eigenvector
![Page 25: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/25.jpg)
More than two segments
• Two options• Recursively split each side to get a tree, continuing till the eigenvalues are
too small• Use the other eigenvectors
![Page 26: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/26.jpg)
Normalized cuts
• Current criterion evaluates within cluster similarity, but not across cluster difference
• Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference
• Write graph as V, one cluster as A and the other as B• Maximize
• i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph
€
assoc(A,A)assoc(A,V )
+
assoc(B,B)assoc(B,V )
![Page 27: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/27.jpg)
• Write a vector y whose elements are 1 if item is in A, -b if it’s in B
• Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones.
• Criterion becomes
• and we have a constraint• This is hard to do, because y’s values are quantized
Normalized cuts
€
minyyT D −W( )y
yTDy
€
yTD1 = 0
![Page 28: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/28.jpg)
• Instead, solve the generalized eigenvalue problem
• which gives
• Now look for a quantization threshold that maximises the criterion --- i.e all components of y above that threshold go to one, all below go to -b
Normalized cuts
€
maxy yT D −W( )y( ) subject to yTDy = 1( )
€
D −W( )y = λDy
![Page 29: K-Means, Expectation Maximization and Segmentationluthuli.cs.uiuc.edu/~daf/courses/cs543computervision/week 6/emseg.pdf · ... Expectation Maximization and Segmentation ... Color](https://reader031.fdocuments.in/reader031/viewer/2022022606/5b7bb89f7f8b9a004b8d3102/html5/thumbnails/29.jpg)
Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998