Perceptual Organization: Segmentation and Optical Flow
Slide 2
Inspiration from psychology The Gestalt school: Grouping is key
to visual perception The whole is greater than the sum of its parts
http://en.wikipedia.org/wiki/Gestalt_psychology subjective contours
occlusion familiar configuration
Motion and perceptual organization Even impoverished motion
data can evoke a strong percept G. Johansson, Visual Perception of
Biological Motion and a Model For Its Analysis", Perception and
Psychophysics 14, 201-211, 1973. YouTube video
Slide 6
Image segmentation
Slide 7
The goals of segmentation Obtain primitives for other tasks
Perceptual organization, recognition Graphics, image
manipulation
Slide 8
Goal 1: Primitives for other tasks Group together
similar-looking pixels for efficiency of further processing
Bottom-up process Unsupervised X. Ren and J. Malik. Learning a
classification model for segmentation. ICCV 2003.Learning a
classification model for segmentation. superpixels
Slide 9
Image parsing or semantic segmentation: Segments as primitives
for recognition J. Tighe and S. Lazebnik, ECCV 2010, IJCV 2013
Slide 10
Goal 2: Recognition Separate image into coherent objects
Bottom-up or top-down process? Supervised or unsupervised? Berkeley
segmentation database:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
image human segmentation
Slide 11
Goal 3: Image manipulation Interactive segmentation for
graphics
Slide 12
Approaches to segmentation Segmentation as clustering
Segmentation as graph partitioning Segmentation as labeling
Slide 13
Segmentation as clustering Source: K. Grauman
Slide 14
Image Intensity-based clustersColor-based clusters Segmentation
as clustering K-means clustering based on intensity or color is
essentially vector quantization of the image attributes Clusters
dont have to be spatially coherent
Slide 15
Segmentation as clustering Source: K. Grauman
Slide 16
Segmentation as clustering Clustering based on (r,g,b,x,y)
values enforces more spatial coherence
Slide 17
Segmentation as graph partitioning Node for every pixel Edge
between every pair of pixels (or every pair of sufficiently close
pixels) Each edge is weighted by the affinity or similarity of the
two nodes w ij i j Source: S. Seitz
Slide 18
Measuring affinity Represent each pixel by a feature vector x
and define an appropriate distance function small large Role
of
Slide 19
Segmentation as graph partitioning Break Graph into Segments
Delete links that cross between segments Easiest to break links
that have low affinity similar pixels should be in the same
segments dissimilar pixels should be in different segments ABC
Source: S. Seitz w ij i j
Slide 20
Graph cut Set of edges whose removal makes a graph disconnected
Cost of a cut: sum of weights of cut edges A graph cut gives us a
segmentation What is a good graph cut and how do we find one? A B
Source: S. Seitz
Slide 21
Minimum cut We can do segmentation by finding the minimum cut
in a graph Efficient algorithms exist for doing this Minimum cut
example
Slide 22
Minimum cut We can do segmentation by finding the minimum cut
in a graph Efficient algorithms exist for doing this Minimum cut
example
Slide 23
Normalized cut Drawback: minimum cut tends to cut off very
small, isolated components Ideal Cut Cuts with lesser weight than
the ideal cut * Slide from Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
Slide 24
Normalized cut To encourage larger segments, normalize the cut
by the total weight of edges incident to the segment The normalized
cut cost is: Intuition: big segments will have a large w(A,V), thus
decreasing ncut(A, B) Finding the globally optimal cut is
NP-complete, but a relaxed version can be solved using a
generalized eigenvalue problem w(A, B) = sum of weights of all
edges between A and B J. Shi and J. Malik. Normalized cuts and
image segmentation. PAMI 2000Normalized cuts and image
segmentation.
Slide 25
Normalized cut: Algorithm Let W be the affinity matrix of the
graph (n x n for n pixels) Let D be the diagonal matrix with
entries D(i, i) = j W(i, j) Solve generalized eigenvalue problem (D
W)y = Dy for the eigenvector with the second smallest eigenvalue
The ith entry of y can be viewed as a soft indicator of the
component membership of the ith pixel Use 0 or median value of the
entries of y to split the graph into two components To find more
than two components: Recursively bipartition the graph Run k-means
clustering on values of several eigenvectors
Slide 26
Example result
Slide 27
Challenge How to define affinities for segmenting highly
textured images?
Slide 28
Segmenting textured images Convolve image with a bank of
filters Find textons by clustering vectors of filter bank outputs
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture
Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and
Texture Analysis for Image Segmentation" Texton mapImage Filter
bank
Slide 29
Segmenting textured images Convolve image with a bank of
filters Find textons by clustering vectors of filter bank outputs
Represent pixels by texton histograms computed over neighborhoods
at some local scale Define affinities as similarities between local
texton histograms J. Malik, S. Belongie, T. Leung and J. Shi.
"Contour and Texture Analysis for Image Segmentation". IJCV
43(1),7-27,2001."Contour and Texture Analysis for Image
Segmentation"
Slide 30
Pitfall of texture features Possible solution: check for
intervening contours when computing affinities J. Malik, S.
Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for
Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture
Analysis for Image Segmentation"
Pro Generic framework, can be used with many different features
and affinity formulations Con High storage requirement and time
complexity: involves solving a generalized eigenvalue problem of
size n x n, where n is the number of pixels Normalized cuts: Pro
and con
Slide 34
Efficient graph-based segmentation Runs in time nearly linear
in the number of edges Easy to control coarseness of segmentations
Results can be unstable P. Felzenszwalb and D. Huttenlocher,
Efficient Graph-Based Image Segmentation, IJCV 2004Efficient
Graph-Based Image Segmentation
Slide 35
Segmentation as labeling Suppose we want to segment an image
into foreground and background Binary labeling problem Credit: N.
Snavely
Slide 36
Segmentation as labeling Suppose we want to segment an image
into foreground and background Binary labeling problem User
sketches out a few strokes on foreground and background How do we
label the rest of the pixels? Source: N. Snavely
Slide 37
Binary segmentation as energy minimization Define a labeling L
as an assignment of each pixel with a 0-1 label (background or
foreground) Find the labeling L that minimizes data term smoothness
term How similar is each labeled pixel to the foreground or
background? Encourage spatially coherent segments Source: N.
Snavely
Slide 38
: distance from pixel to background : distance from pixel to
foreground { computed by creating a color model from user- labeled
pixels Source: N. Snavely
Slide 39
Slide 40
Neighboring pixels should generally have the same labels Unless
the pixels have very different intensities : similarity in
intensity of p and q = 10.0 = 0.1 Source: N. Snavely
Slide 41
Binary segmentation as energy minimization For this problem, we
can efficiently find the global minimum using the max flow / min
cut algorithm Source: N. Snavely Y. Boykov and M.-P. Jolly,
Interactive Graph Cuts for Optimal Boundary and Region Segmentation
of Objects in N-D Images, ICCV 2001Interactive Graph Cuts for
Optimal Boundary and Region Segmentation of Objects in N-D
Images
Slide 42
Recall: Stereo as energy minimization I1I1 I2I2 D Energy
functions of this form can be minimized using graph cuts Y. Boykov,
O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via
Graph Cuts, PAMI 2001Fast Approximate Energy Minimization via Graph
Cuts W1(i )W1(i )W 2 (i+D(i )) D(i )D(i ) data term smoothness
term
Slide 43
GrabCut C. Rother, V. Kolmogorov, and A. Blake, GrabCut
Interactive Foreground Extraction using Iterated Graph Cuts,
SIGGRAPH 2004GrabCut Interactive Foreground Extraction using
Iterated Graph Cuts