Post on 11-Jul-2020
LECTURE 08 : SEGMENTATION
Venu Madhav GovinduDepartment of Electrical EngineeringIndian Institute of Science, Bengaluru
In this lecture we shall look at image segmentation
Segmentation
Image Segmentation
• Find groups of pixels (meaningful)
• What the basis of this grouping ?
• Context dependent
• Ambiguous with multiple interpretations
• General principles based on image primitives alone
• Consider a few well-known examples
• There are many more methods in the literature
Segmentation
Active Contours
• Snakes : Energy minimisation on 2-D spline (controls + imagedata terms)
• Eint =∫α(s)||fs(s)||2 + β(s)||fss(s)||2ds
• s is arc-length parametrisation
• f(s) = (x(s), y(s))
• α and β are first and second-order weighting functions
• This is internal energy alone
Segmentation
Active Contours
• Snakes : Minimise net sum of various energy terms
• Eimage = wlineEline + wedgeEedge + wtermEterm
• Line term : attract to ridges
• Edge term : strong gradients
• Term : line termination
• Most practical systems use edge terms only
Segmentation
Active Contours
• Eedge =∑
i−||∇I(f(i))||2
• Interactive applications use user-constraints
• Attract to anchor points
• Espring = ki||f(i)− d(i)||2
• Also repulsive forces can be used
• Snakes have tendency to shrink• Use external boundary as initialisation• Use expansionary force (move along normal)
Segmentation
Active Contours
• Solve sparse linear system (block diagonal due to difference eqns.)
• Alternate between
• linear system solution• linearisation of non-linear constraints such as edge energy
Segmentation
Dynamic Snakes
• If object is to be tracked, use previous estimate
• Can ‘predict’ based on model
• xi = Axi−1 + wi
• Kalman filtering
• Limitation : Gaussian error assumption is restrictive
• Need to remove it for clutter etc.
Segmentation
CONDENSATION
• Remove restrictive Gaussian error model
• Propagate general multi-modal distribution
• Particle Filtering
• Collection of weighted samples• Update according to linear model• General multiple samples according to updated model• Perturb according to diffusion
• Point samples represent and propagate conditional estimates ofmulti-modal density
Segmentation
CONditional propagATION or CONDENSATION
• Active Contours cannot easily change topology as curve evolves
• Alternative to this is level set
• Curve is defined by zero-crossing of characteristic function basedon image data
• Track curve by modifying the underlying embedding fuctioninstead of f(s)
Segmentation
Watershed Algorithm
• Single threshold does not work
• Works on gray-scale images
• Flood landscape at all local minima
• Label ridges wherever evolving components meet
• Also use steerable filters for local detail
• Suffers from over-segmentation
Segmentation
Mean Shift
• k-means and mixture of Gaussians - parametric model for pdf
• Mean shift implicitly models pdf
• Uses smooth continuous non-parametric model
• Efficiently finds peaks in high-dim data
• Peaks without explicitly computing complete function
• Modes are inverse of watershed
Segmentation
Mean Shift
• Estimate density function given samples
f(x) =∑i
k(x− xi) =∑i
k
(||x− xi||2
h2
)Parzen window method
Segmentation
Mean Shift
• Given f(x) can find local minima
• Too expensive to evaluate f(x) in h-d space
• Solution : find mode of pdf directly
• Start at a random point
∇f(x) =∑i
(xi − x)G(x− xi)
=∑i
(xi − x)g
(||x− xi||2
h2
)g(r) = −k
′(r)
Segmentation
⇒ ∇f(x) =
[∑i
G(x− xi)
]m(x)
where
m(x) =
∑i xiG(x− xi)∑iG(x− xi)
− x
m(x) is the mean-shift
yk+1 = yk +m(yk) =
∑i xiG(yk − xi)∑iG(yk − xi)
Mean Shift
Figure : Colour image segmentation using mean shift
• K(xj) = k(||xr||2h2r
)k(||xs||2h2s
)• Spatial location : xs = (x, y)
• Colour values : xr
• xj contains both spatial location and colour value
• M specifies size below which clusters are discarded
Normalized Cuts
Segmentation using Affinities
• Consider graph of similarity relations G = (V,E)
• In our case V can be image pixels
• E : edges between neighbouring pixels (say)
• A cut
• Deletes some edges• Separates vertices into two sets• Results in two segments
Normalized Cuts
cut(A,B) =∑
i∈A,j∈Bwij
Properties of a Cut
• Cut between two sets of vertices A and B
• A ∪B = V and A ∩B = ∅• Cost of cut is the sum of weights of deleted edges
• Can result in degenerate solutions (minimisation)
• Cut isolated vertex
Normalized Cuts
Modify to better reflect desired propertiesReplace
cut(A,B) =∑
i∈A,j∈Bwij
with
Ncut(A,B) =cut(A,B)
assoc(A, V )+
cut(A,B)
assoc(B, V )
Normalized Cuts
Ncut(A,B) =cut(A,B)
assoc(A, V )+
cut(A,B)
assoc(B, V )
where
assoc(A,A) =∑
i∈A,j∈Awij
assoc(A, V ) = assoc(A,A) + cut(A,B)
• assoc(A,A) : association within cluster A
• assoc(A, V ) : sum of all weights with vertices in A
• Normalization for relative sizes of A and B
Normalized Cuts
Ncut(A,B) =cut(A,B)
assoc(A, V )+
cut(A,B)
assoc(B, V )
Hardness of Cut
• Solving for optimal cut is NP-complete
• Can be solved by relaxing labels
• Define indicator x
• xi = +1 ⇐⇒ i ∈ A• xi = −1 ⇐⇒ i ∈ B
Normalized Cuts
• Define d = W1
• Define D = diag(d)
• Let y = ((1 + x)− b(1− x))/2 s.t. y.d = 0
• Normalized cuts problem can be redefined
miny
yT (D−W)y
yTDy
Normalized Cuts
miny
yT (D−W)y
yTDy
• Cost function solving by minimising Rayleigh quotient
• Equivalent to solving (D−W)y = λDy
• Equivalent to (I−N)z = λz
• N = D−12WD−
12 (Normalized affinity matrix)
• z = D12y
• Laplacian matrix L = I−N
• Spectral Clustering
Normalized Cuts
Issues and Details
• Multiple segmentation carried out recursively
• Can also use multiple eigenvectors to classify
• Weights can capture spatial and image similarity relationships
wij = exp
(−||Fi − Fj ||2
2σ2F
− ||xi − xj ||2
2σ2s
)
• Original method is slow
• Many variations and modifications to speed up
• Modern methods : Discrete optimization on MRF’s
• Neighbourhood need not be local
• L is sparse
Normalized Cuts