Image SegmentationImage Segmentation

A Graph Theoretic ApproachA Graph Theoretic Approach

Factors for Visual GroupingFactors for Visual Grouping

Similarity (gray level difference)Similarity (gray level difference)ProximityProximityContinuityContinuity

Reference: Reference: M. Wertheimer, “Laws of Organization in Perceptual Forms”, A Sourcebook of Gestalt Psychology, W.B. Ellis, ed., pp. 71-88, Harcourt, Brace, 1938.

What is the “correct” grouping?

Subjectivity in SegmentationSubjectivity in Segmentation Prior world knowledge neededPrior world knowledge needed Agglomerative and divisive techniques in grouping (or Region-based Agglomerative and divisive techniques in grouping (or Region-based

merge and split algorithms in image segmentation)merge and split algorithms in image segmentation) Local properties – easier to specify but poorer resultsLocal properties – easier to specify but poorer results

e.g. coherence of brightness, colour, texture, motione.g. coherence of brightness, colour, texture, motion Global properties – more difficult to specify but give better results Global properties – more difficult to specify but give better results

e.g. object symmetriese.g. object symmetries Image segmentation can be modeled as a graph partitioning and Image segmentation can be modeled as a graph partitioning and

optimization problemoptimization problem

PartitioningPartitioning

Divisive or top-down approachDivisive or top-down approach Inherently hierarchicalInherently hierarchicalWe must aim at returning a tree structure We must aim at returning a tree structure

(called the (called the dendogramdendogram) corresponding to a ) corresponding to a hierarchical partitioning scheme instead of hierarchical partitioning scheme instead of a single “flat” partitiona single “flat” partition

ChallengesChallenges

Picking an appropriate criterion to Picking an appropriate criterion to minimize which would result in a “good” minimize which would result in a “good” segmentationsegmentation

Finding an efficient way to achieve the Finding an efficient way to achieve the minimizationminimization

Modeling as a Graph Partitioning Modeling as a Graph Partitioning problemproblem

Set of points of the feature space represented as Set of points of the feature space represented as a weighted, undirected graph, G = (V, E)a weighted, undirected graph, G = (V, E)

The points of the feature space are the nodes of The points of the feature space are the nodes of the graph.the graph.

Edge between every pair of nodes.Edge between every pair of nodes. Weight on each edge, Weight on each edge, w(i, j)w(i, j), is a function of the , is a function of the

similarity between the nodes i and j.similarity between the nodes i and j. Partition the set of vertices into disjoint sets Partition the set of vertices into disjoint sets

where similarity within the sets is high and where similarity within the sets is high and across the sets is low.across the sets is low.

Weight Function for Brightness Weight Function for Brightness ImagesImages

Weight measure (reflects likelihood of two Weight measure (reflects likelihood of two pixels belonging to the same object)pixels belonging to the same object)

Representing Images as GraphsRepresenting Images as Graphs

Graph Weight Matrix, WGraph Weight Matrix, W

Segmentation and Graphs - Other Segmentation and Graphs - Other Common ApproachesCommon Approaches

Minimal Spanning TreeMinimal Spanning Tree Limited Neighbourhood SetLimited Neighbourhood Set

- Both approaches are computationally efficient Both approaches are computationally efficient but the criteria are based on local propertiesbut the criteria are based on local properties

- Perceptual grouping is about extracting global Perceptual grouping is about extracting global impressions of a scene; thus local criteria are impressions of a scene; thus local criteria are often inadequateoften inadequate

First attempt at global criterion First attempt at global criterion selectionselection

A graph can be partitioned into two disjoint A graph can be partitioned into two disjoint sets sets by simply removing the edges simply removing the edges connecting the two partsconnecting the two parts

The degree of dissimilarity between these two pieces can be computed as total weight of the edges that have been removed

More formally, it is called the ‘cut’

Graph CutGraph Cut

Optimization ProblemOptimization Problem Minimize the Minimize the cut cut valuevalue

No of such partitions is exponential (2^N) but the minimum cut can No of such partitions is exponential (2^N) but the minimum cut can be found efficientlybe found efficiently

Reference: Z. Wu and R. Leahy, “An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation”. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1101-1113, Nov. 1993.

Subject to the constraints:

Problems with min-cutProblems with min-cut

Minimum cut criteria favors cutting small sets of isolated nodes in the graph.

Solution – Normalized CutSolution – Normalized Cut

We must avoid unnatural bias for partitioning out small sets of points

Normalized Cut - computes the cut cost as a fraction of the total edge connections to all the nodes in the graph

where

Looking at it another way..Looking at it another way..

Our criteria can also aim to tighten Our criteria can also aim to tighten similarity within the groupssimilarity within the groups

Minimizing Ncut and maximizing Nassoc Minimizing Ncut and maximizing Nassoc are actually equivalentare actually equivalent

Matrix FormulationsMatrix Formulations

Let x be an indicator vector s.t. Let x be an indicator vector s.t.

xxii = 1, = 1, if i belongs to Aif i belongs to A

0,0, otherwiseotherwise

Assoc(A, A) = xAssoc(A, A) = xTTWxWxAssoc(A, V) = xAssoc(A, V) = xTTDxDxCut(A, V-A) = xCut(A, V-A) = xTT(D – W)x(D – W)x

Computational IssuesComputational Issues

Exact solution to minimizing normalized Exact solution to minimizing normalized cut is an NP-complete problemcut is an NP-complete problem

However, approximate discrete solutions However, approximate discrete solutions can be found efficientlycan be found efficiently

Normalized cut criterion can be computed Normalized cut criterion can be computed efficiently by solving a generalized efficiently by solving a generalized eigenvalue problemeigenvalue problem

AlgorithmAlgorithm

1. Construct the weighted graph 1. Construct the weighted graph representing the image. Summarize the representing the image. Summarize the information into matrices, W & D. Edge information into matrices, W & D. Edge weight is an exponential function of feature weight is an exponential function of feature similarity as well as distance measure.similarity as well as distance measure.

2. Solve for the eigenvectors with the 2. Solve for the eigenvectors with the smallest eigenvalues of:smallest eigenvalues of:

(D – W)x = LDx(D – W)x = LDx

Algorithm (contd.)Algorithm (contd.)

3. Partition the graph into two pieces using 3. Partition the graph into two pieces using the second smallest eigenvector. Signs tell the second smallest eigenvector. Signs tell us exactly how to partition the graph.us exactly how to partition the graph.

4. Recursively run the algorithm on the two 4. Recursively run the algorithm on the two partitioned parts. Recursion stops once partitioned parts. Recursion stops once the Ncut value exceeds a certain limit. the Ncut value exceeds a certain limit. This maximum allowed Ncut value controls This maximum allowed Ncut value controls the number of groups segmented.the number of groups segmented.

Computational Issues RevisitedComputational Issues Revisited Solving a standard eigenvalue problem for all

eigenvectors takes O(n^3) operations, where n is the number of nodes in the graph

This becomes impractical for image segmentation applications where n is the number of pixels in an image

For the problem at hand, the graphs are often only locally connected, only the top few eigenvectors are needed for graph partitioning, and the precision requirement for the eigenvectors is low, often only the right sign bit is required.

A Physical InterpretationA Physical Interpretation

Think of the weighted graph as a spring Think of the weighted graph as a spring mass systemmass system

Graph nodes physical massesGraph edges springs Graph edge weight spring stiffness Total incoming edge weights mass of

the node

A Physical Interpretation (contd..)A Physical Interpretation (contd..)

Imagine giving a hard shake to this spring-mass system, forcing the nodes to oscillate in the direction perpendicular to the image plane

Nodes that have stronger spring connections among them will likely oscillate together

Eventually, the group will “pop” off from the image plane The overall steady state behavior of the nodes can be

described by its fundamental mode of oscillation and it can be shown that the fundamental modes of oscillation of this spring mass system are exactly the generalized eigenvectors of the normalized cut.

Comparisons with other criteriaComparisons with other criteria Average Cut:Average Cut:

Analogously, Average Association can be defined as:Analogously, Average Association can be defined as:

Unlike in the case of Normalized Cut and Normalized Association, Unlike in the case of Normalized Cut and Normalized Association, Average Cut and Average Association do not have a simple Average Cut and Average Association do not have a simple relationship between themrelationship between them

Consequently, one cannot simultaneously minimize the disassociation across the partitions while maximizing the association within the groups

Normalized Cut produces better results in practiceNormalized Cut produces better results in practice

Comparisons with other criteria Comparisons with other criteria (contd..)(contd..)

Comparisons with other criteria Comparisons with other criteria (contd..)(contd..)

Average association has a bias for finding Average association has a bias for finding tight clusters – runs the risk of finding tight clusters – runs the risk of finding small, tight clusters in the data small, tight clusters in the data

Average cut does not look at within-group Average cut does not look at within-group similarity – problems when the dissimilarity similarity – problems when the dissimilarity between groups is not clearly definedbetween groups is not clearly defined

Consider random 1-D data points:Consider random 1-D data points:

Each data point is a node in the graph Each data point is a node in the graph and the weighted graph edge connecting two points is defined to be inversely proportional to the distance between two nodes

We will consider two different monotonically decreasing weight functions, w(i,j) = f(d(i,j)), defined on the distance function, d(i,j), with differents rate of fall-off.

Fast falling weight functionFast falling weight function

With this function, only close-by points are With this function, only close-by points are connected.connected.

Criterion used Second smallest eigenvector plot

InterpretationInterpretation

The cluster on the right has less within-group similarity compared with the cluster on the left.

In this case, average association fails to find the right partition.

Instead, it focuses on finding small clusters in each of the two main subgroups.

Slowly decreasing weight functionSlowly decreasing weight function

With this function, most points have non-With this function, most points have non-trivial connections with the resttrivial connections with the rest

Criterion used Second smallest eigenvector plot

InterpretationInterpretation

To find a cut of the graph, a number of edges with heavy weights have to be removed.

In this case, average cut has trouble deciding on where to cut.

ReferenceReference

J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.