Chapter 14: SEGMENTATION BY CLUSTERING
-
Upload
hakeem-campbell -
Category
Documents
-
view
74 -
download
0
description
Transcript of Chapter 14: SEGMENTATION BY CLUSTERING
![Page 1: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/1.jpg)
Chapter 14:SEGMENTATION BY
CLUSTERING
1
![Page 2: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/2.jpg)
2
![Page 3: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/3.jpg)
Outline
• Introduction• Human Vision & Gestalt Properties• Applications
– Background Subtraction– Shot Boundary Detection
• Segmentation by Clustering– Simple Clustering Methods– K-means
• Segmentation by Graph-Theoretic Clustering– The Overall Approach– Affinity Measures– Normalized Cut
3
![Page 4: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/4.jpg)
Introduction
4
![Page 5: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/5.jpg)
General Ideas
• Tokens– Whatever we need to group (pixels, points, surface
elements, etc., etc.)
• Top down segmentation– Tokens belong together because they lie on the
same object
• Bottom up segmentation– Tokens belong together because they are locally
coherent
5
![Page 6: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/6.jpg)
• Why & What is segmentation– Emphasize the property– Make main idea interesting– There are too many pixels in a single image– We need some form of compact, summary
representation
6
![Page 7: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/7.jpg)
• What segmentation can do– Summarizing video– Finding machined parts– Finding people– Finding buildings in satellite
images– Searching a collection of images
7
![Page 8: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/8.jpg)
• Model Problems– Forming image segments
• Roughly coherent color and texture
– Finding body segments• Arms, torso, etc.
– Fitting lines to edge points• To fit a line to a set of points
8
![Page 9: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/9.jpg)
Segmentation as Clustering
• Partitioning– Decompose an image into regions– Decompose an image into extended blobs– Take a video sequence and decompose it into shots
• Grouping
9
![Page 10: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/10.jpg)
Human Vision & Gestalt Properties
10
![Page 11: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/11.jpg)
• Figure-ground discrimination– Grouping can be seen in terms of allocating some
elements to a figure, some to ground– Can be based on local bottom-up cues or high level
recognition
11
![Page 12: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/12.jpg)
• Figure-ground discrimination
12
![Page 13: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/13.jpg)
• A series of factors affect whether elements should be grouped together
13
![Page 14: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/14.jpg)
• A series of factors affect whether elements should be grouped together
14
![Page 15: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/15.jpg)
15
![Page 16: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/16.jpg)
16
![Page 17: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/17.jpg)
• Grouping by occlusions
17
![Page 18: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/18.jpg)
• Grouping by invisible completions
18
![Page 19: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/19.jpg)
Outline
• Introduction• Human Vision & Gestalt Properties• Applications
– Background Subtraction– Shot Boundary Detection
• Segmentation by Clustering– Simple Clustering Methods– K-means
• Segmentation by Graph-Theoretic Clustering– The Overall Approach– Affinity Measures– Normalized Cut
19
![Page 20: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/20.jpg)
ApplicationBackground Subtraction
• Anything that doesn’t look like a known background is interesting
20
![Page 21: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/21.jpg)
• Take a picture?• Not good…• The background changes slowly overtime…
21
![Page 22: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/22.jpg)
• Estimate the value of background pixels using a moving average.
• Ideally, the moving average should track the changes in the background.
22
![Page 23: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/23.jpg)
23
![Page 24: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/24.jpg)
Shot Boundary Detection
• Where substantial changes in a video are interesting.
24
![Page 25: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/25.jpg)
25
![Page 26: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/26.jpg)
• Computing distance– Frame differencing
– Histogram-based
– Block comparison
– Edge differencing
pixel-by-pixel
histogram-by-histogram
box-by-box
edge map-by-edge map
26
![Page 27: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/27.jpg)
Outline
• Introduction• Human Vision & Gestalt Properties• Applications
– Background Subtraction– Shot Boundary Detection
• Segmentation by Clustering– Simple Clustering Methods– K-means
• Segmentation by Graph-Theoretic Clustering– The Overall Approach– Affinity Measures– Normalized Cut
27
![Page 28: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/28.jpg)
Segmentation by Clustering Simple Clustering Methods
• Divisive clustering
28
![Page 29: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/29.jpg)
• Divisive clustering
29
p, q, r, s, t
r, s, t
p, q
qp r
s, t
s t
![Page 30: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/30.jpg)
• Agglomerative clustering
30
![Page 31: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/31.jpg)
• Agglomerative clustering
31
p, q, r, s, t
qp
p, q
r
r, s, t
s t
s, t
![Page 32: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/32.jpg)
• Q: what the distance of A:
32
( , ) ( , )s q q sd C C d C C
( , ) ( ( , ), ( , ), ( , ))q s i s j s i jd C C f d C C d C C d C C
![Page 33: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/33.jpg)
• Single-link clustering
33
( , ) min{ ( , ), ( , )}q s i s j sd C C d C C d C C
![Page 34: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/34.jpg)
• Complete-link clustering
34
( , ) max{ ( , ), ( , )}q s i s j sd C C d C C d C C
![Page 35: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/35.jpg)
• Group average clustering
( , ) { ( , ), ( , )}q s i s j sd C C avg d C C d C C
35
![Page 36: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/36.jpg)
K-means
36
![Page 37: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/37.jpg)
37
37
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 1
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 4
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 5
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 6
![Page 38: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/38.jpg)
38
![Page 39: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/39.jpg)
Outline
• Introduction• Human Vision & Gestalt Properties• Applications
– Background Subtraction– Shot Boundary Detection
• Segmentation by Clustering– Simple Clustering Methods– K-means
• Segmentation by Graph-Theoretic Clustering– The Overall Approach– Affinity Measures– Normalized Cut
39
![Page 40: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/40.jpg)
Segmentation by Graph-theoretic Clustering• Terminology for Graph
– Vertices– Edges
• Directed graph
• Undirected graph
• Weighted graph
• Self-loop
• Connected graph– Connected components
40
},{ EVG 4
2
3
![Page 41: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/41.jpg)
• Weighted graph represented by a square matrix
The Overall Approach
41
Undirected weighted graph Weighted matrix
![Page 42: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/42.jpg)
42
Set of well separated points Affinity matrix
![Page 43: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/43.jpg)
Affinity Measures
• Affinity by Distance
43
22
)()(exp),(
d
t yxyxyxaff
1.0d 2.0d 1d
![Page 44: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/44.jpg)
• Affinity by Intensity
• Affinity by Color
• Affinity by Texture
44
22
))()(())()((exp),(
I
t yIxIyIxIyxaff
2
2
2
))(),((exp),(
c
ycxcdistyxaff
22
))()(())()((exp),(
I
t yfxfyfxfyxaff
![Page 45: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/45.jpg)
Eigenvectors and segmentation
45
• Extracting a Single Good Cluster
{ association of element I with cluster n } ×{ affinity between I and j } ×
{ association of element j with cluster n}
nTn AwwA :Element affinities
nw :The vector of weight linking elements to the th clustern
![Page 46: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/46.jpg)
46
• Scaling by scales• Normalizes the weights by requiring that
• Differentiation and dropping
)1( nTnn
Tn wwAww
nw :eigenvector of A
nn wAw
1nTn ww
nw
Lagrangia
n
![Page 47: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/47.jpg)
47
Small2.0d
![Page 48: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/48.jpg)
• Extracting Weights for a Set of Clusters– quite tight– distinct.
48
The three largest eigenvalues of the affinity matrix for the dataset
![Page 49: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/49.jpg)
49
1.0d 2.0d 1d
![Page 50: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/50.jpg)
Outline
• Introduction• Human Vision & Gestalt Properties• Applications
– Background Subtraction– Shot Boundary Detection
• Segmentation by Clustering– Simple Clustering Methods– K-means
• Segmentation by Graph-Theoretic Clustering– The Overall Approach– Affinity Measures– Normalized Cut
50
![Page 51: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/51.jpg)
Normalized Cut
51
![Page 52: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/52.jpg)
• Association of set:
),(
),(
),(
),(),(
VBassoc
BAcut
VAassoc
BAcutBANcut
,( , ) ( , )
u A t Vassoc A V u t
A
A
B
52
• Normalized cut:
• Cut of set:
,( , ) ( , )
u A t Bcut A B u t
A
A
B
A
B
Minimize
![Page 53: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/53.jpg)
• Minimize Ncut– Transform Ncut equation to a matrical form.
53
Dyy
yADyxNcut
T
T
yx
)(min)(min
: affinity matrix
: , 0( )
: a vector which is giving the association
between each element and a cluster
1 ,node belong to one cluster:
,node belong to the other
j
i
D Dii ij Dij i j
x
iy y
b i
A
A
0 5 2 0 0 0 0 0 05 0 1 2 0 0 0 0 02 1 0 2 0 0 .1 0 00 2 2 0 .1 0 0 0 00 0 0 .1 0 3 0 1 00 0 0 0 3 0 1 4 70 0 .1 0 0 1 0 0 10 0 0 0 1 4 0 0 20 0 0 0 0 7 1 2 0
7 0 0 0 0 0 0 0 00 8 0 0 0 0 0 0 00 0 4. 0 0 0 0 0 00 0 0 4. 0 0 0 0 00 0 0 0 4. 0 0 0 00 0 0 0 0 15 0 0 00 0 0 0 0 0 5. 0 00 0 0 0 0 0 0 7 00 0 0 0 0 0 0 0 10
NP-Hard!
![Page 54: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/54.jpg)
Real Vector & Threshold
54
( )min ( ) min
T
x y T
y D yNcut x
y Dy
A
( )D y Dy A( ) 0D I A
0 5 2 0 0 0 0 0 0
5 0 1 2 0 0 0 0 0
2 1 0 2 0 0 .1 0 0
0 2 2 0 .1 0 0 0 0
0 0 0 .1 0 3 0 1 0
0 0 0 0 3 0 1 4 7
0 0 .1 0 0 1 0 0 1
0 0 0 0 1 4 0 0 2
0 0 0 0 0 7 1 2 0
7 0 0 0 0 0 0 0 0
0 8 0 0 0 0 0 0 0
0 0 4. 0 0 0 0 0 0
0 0 0 4. 0 0 0 0 0
0 0 0 0 4. 0 0 0 0
0 0 0 0 0 5 0 0 0
0 0 0 0 0 0 5. 0 0
0 0 0 0 0 0 0 7 0
0 0 0 0 0 0 0 0 1
• If y is allowed to take real values then the minimization can be done by solving the generalized eigenvalue system
• Note that ,so the first eigenvector is y=I with eigenvalue 0.
• The second smallest eigenvector is the real valued solution to this problem!
• Choosing a threshold to decide that nodes are assigned to one side or the other.
![Page 55: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/55.jpg)
Summarize
1. Define a similarity function between 2 nodes.
2. Compute affinity matrix (A) and degree matrix (D).
3. Solve
4. Use the eigenvector with the second smallest eigenvalue to bipartition the graph.
5. Recursively partition the segmented parts if necessary.
( )D y Dy A
55
![Page 56: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/56.jpg)
Example
56
![Page 57: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/57.jpg)
57
![Page 58: Chapter 14: SEGMENTATION BY CLUSTERING](https://reader036.fdocuments.in/reader036/viewer/2022081419/568132bb550346895d997af6/html5/thumbnails/58.jpg)
Thank you
58