1

Efficient Graph-Based Image SegmentationEfficient Graph-Based Image Segmentation

P. Felzenszwalb, D. HuttenlocherP. Felzenszwalb, D. Huttenlocher

2

OutlineOutline

Graph representationGraph representation Pairwise region comparison predicatePairwise region comparison predicate Segmentation algorithmSegmentation algorithm Proof of correctness and complexityProof of correctness and complexity Examples and resultsExamples and results

3

MotivationMotivation

Broad utility image segmentation with two propertiesBroad utility image segmentation with two properties Capture perceptually important features (groupings, Capture perceptually important features (groupings,

regions), which often reflect global aspects of the imageregions), which often reflect global aspects of the image Be highly efficient, running in time nearly linear in the Be highly efficient, running in time nearly linear in the

number of image pixelsnumber of image pixels

Graph-based method with greedy algorithm and adaptive Graph-based method with greedy algorithm and adaptive segmentation criterion (boundary evidence predicate)segmentation criterion (boundary evidence predicate)

4

MotivationMotivation

5

Graph-based segmentationGraph-based segmentation

Let be an undirected graph with vertices Let be an undirected graph with vertices is the set of elements to be segmentedis the set of elements to be segmented Edges correspond to pairs of neighboring verticesEdges correspond to pairs of neighboring vertices For each edge let be a weight of this edge, For each edge let be a weight of this edge,

which is a non-negative measure of dissimilarity between which is a non-negative measure of dissimilarity between neighboring elementsneighboring elements

A segmentation is a partition of into components such that A segmentation is a partition of into components such that each component (or region) corresponds to a connected each component (or region) corresponds to a connected component in a graph , where component in a graph , where

Goal is to have the elements in one component to be similar, Goal is to have the elements in one component to be similar, and elements in different components to be dissimilarand elements in different components to be dissimilar

6

Pairwise region comparison predicatePairwise region comparison predicate

Boundary evidence predicate Boundary evidence predicate Predicate compares the inter-component differences of Predicate compares the inter-component differences of

elements to the within component differenceselements to the within component differences Let the Let the internal difference internal difference of a component be a of a component be a

largest weight in the minimum spanning tree of the largest weight in the minimum spanning tree of the component, component,

Let the Let the difference betweendifference between two components be the two components be the minimum weight edge connecting the two componentsminimum weight edge connecting the two components

7

Pairwise region comparison predicatePairwise region comparison predicate

The region comparison predicate is defined as followsThe region comparison predicate is defined as follows

8

Segmentation algorithm and propertiesSegmentation algorithm and properties

A segmentation is A segmentation is too finetoo fine if there is some pair of regions if there is some pair of regions for which there is no evidence for a boundary for which there is no evidence for a boundary between thembetween them

Given two segmentations and of the same base set, we say Given two segmentations and of the same base set, we say that is that is refinementrefinement of when each component of is of when each component of is contained in (or equal to) some component ofcontained in (or equal to) some component of

is a is a proper refinementproper refinement of when of when A segmentation is A segmentation is too coarsetoo coarse when there exists a proper when there exists a proper

refinement of that is not too finerefinement of that is not too fine

For any (finite) graph there exists some For any (finite) graph there exists some segmentation that is neither too coarse nor too fine segmentation that is neither too coarse nor too fine

9

Kruskal's algorithmKruskal's algorithm

Kruskal's algorithm is a greedy algorithm in graph theory that Kruskal's algorithm is a greedy algorithm in graph theory that finds a minimum spanning tree for a connected weighted graphfinds a minimum spanning tree for a connected weighted graph

1.1. create a forest F (a set of trees), where each vertex in the create a forest F (a set of trees), where each vertex in the graph is a separate treegraph is a separate tree

2.2. create a set S containing all the edges in the graphcreate a set S containing all the edges in the graph3.3. while S is nonempty and F is not yet spanningwhile S is nonempty and F is not yet spanning

a)a) remove an edge with minimum weight from Sremove an edge with minimum weight from Sb)b) if that edge connects two different trees, then add it to if that edge connects two different trees, then add it to

the forest, combining two trees into a single treethe forest, combining two trees into a single treec)c) otherwise discard that edge.otherwise discard that edge.

10

Kruskal's algorithmKruskal's algorithm

11

Kruskal's algorithmKruskal's algorithm

12

Kruskal's algorithmKruskal's algorithm

13

Kruskal's algorithmKruskal's algorithm

14

Kruskal's algorithmKruskal's algorithm

15

Kruskal's algorithmKruskal's algorithm

16

Segmentation algorithmSegmentation algorithm

The input is a graph , with vertices and edges. The input is a graph , with vertices and edges. The output is a segmentation of into components The output is a segmentation of into components

1.1. Sort into , by non-decreasing edge weightSort into , by non-decreasing edge weight2.2. Start with a segmentation , where each vertex is in its own Start with a segmentation , where each vertex is in its own

componentcomponent3.3. Repeat step 4 forRepeat step 4 for4.4. Construct given as follows. Let and denote the Construct given as follows. Let and denote the

vertices connected by the -th edge in the ordering. If they are in vertices connected by the -th edge in the ordering. If they are in disjoint components of and is small compared to the disjoint components of and is small compared to the internal difference of both those components internal difference of both those components ( ), then merge the two components, ( ), then merge the two components, otherwise do nothing ( ).otherwise do nothing ( ).

5.5. ReturnReturn

17

Proof of correctness and complexityProof of correctness and complexity

The algorithm stops (loop in step 4 is finite)The algorithm stops (loop in step 4 is finite) The segmentation produced by the algorithm is not too fine The segmentation produced by the algorithm is not too fine

using the region comparison predicateusing the region comparison predicate The segmentation produced by the algorithm is not too coarse The segmentation produced by the algorithm is not too coarse

using the region comparison predicateusing the region comparison predicate

The segmentation produced by the algorithm does not depend The segmentation produced by the algorithm does not depend on which non-decreasing weight order of the edges is usedon which non-decreasing weight order of the edges is used

Time complexity of this greedy algorithm is Time complexity of this greedy algorithm is Steps 2-4 take Steps 2-4 take

18

Grid graphsGrid graphs

Let be an undirected graph, where each vertex Let be an undirected graph, where each vertex correspond to an image pixel of the original gray- correspond to an image pixel of the original gray-scale imagescale image

The edge set is constructed by connecting pairs of pixels that The edge set is constructed by connecting pairs of pixels that are neighbors in an 8-connected senseare neighbors in an 8-connected sense

In this case , so the time complexity is In this case , so the time complexity is for pixelsfor pixels

An edge weight function is based on absolute intensity An edge weight function is based on absolute intensity difference between neighborsdifference between neighbors

Color channels of color images are processed separately, and Color channels of color images are processed separately, and the intersection of these three sets is returnedthe intersection of these three sets is returned

19

Grid graphsGrid graphs

20

Grid graphsGrid graphs

21

Nearest neighbor graphsNearest neighbor graphs

Let be an undirected graph, where each vertex Let be an undirected graph, where each vertex correspond to a feature point (each pixel) of the original correspond to a feature point (each pixel) of the original gray-scale imagegray-scale image

The edge set is constructed by connecting pairs of feature The edge set is constructed by connecting pairs of feature points that are nearby in the feature spacepoints that are nearby in the feature space

There are several possibilities – a connection to a fixed There are several possibilities – a connection to a fixed number of nearest neighbors, a connection to all neighbors number of nearest neighbors, a connection to all neighbors in some fixed distance etc.in some fixed distance etc.

The weight of an edge is the (Euclidean) distance The weight of an edge is the (Euclidean) distance between two points in feature spacebetween two points in feature space

22

Nearest neighbor graphsNearest neighbor graphs

23

Nearest neighbor graphsNearest neighbor graphs

24

ResultsResults

25

ResultsResults

26

RemarksRemarks

Approach via nearest neighbor graph is closely related to Mean Approach via nearest neighbor graph is closely related to Mean Shift segmentation by Comaniciu and Meer (1999)Shift segmentation by Comaniciu and Meer (1999)

One can consider to use a quantile (eg. median) instead of One can consider to use a quantile (eg. median) instead of minimum weight in definition of difference between two minimum weight in definition of difference between two componentscomponents

However, this leads to NP-hard problemHowever, this leads to NP-hard problem