Lecture 3: The Shape ContextWesley Snyder, Ph.D.

UWA, CSSE

NCSU, ECE

Lecture 3: The Shape Context – p. 1/21

Giant Quokka

Lecture 3: The Shape Context – p. 2/21

The Shape Context

• Need for invariance• Translation, rotation, scale

• How to achieve invariance• Definition of the Shape Context• Computing the Shape Context• The correspondence problem• Linear assignment problem• Implementation• Performance

Lecture 3: The Shape Context – p. 3/21

The Shape RecognitionProblem

• (For now) assume 2-D shapes• Allow arbitrary in-viewing plane

• rotation• translation• scale change

• Consider partial occlusion• Consider only boundary information

Lecture 3: The Shape Context – p. 4/21

InvarianceTranslation: Choose an object-centered point ofreference (center of gravity is good), and subtractthose coordinates from all the points on the bound-ary

Lecture 3: The Shape Context – p. 5/21

InvarianceScale: Options:• find distance from CG to boundary points

• Average, median, maximum• find distance between pairs of boundary

points• Average, median, maximum

Median of pairs seems to be most robust to out-liers

Lecture 3: The Shape Context – p. 6/21

InvarianceRotation: lots of ways. Let’s just do one: Make alldirections relative to the tangent line at each point.Which means, of course, that we have to find thetangent line.

Lecture 3: The Shape Context – p. 7/21

Finding the tangent to a curveLet pi = (xi, yi) be a point onthe boundary, and letpj denoteits neighbors along the boundary

We seek a line (shown here in green)that minimizes the sum of squaredperpendicular distances in a localneighborhood (shown in red) of thepoint of interest.

Lecture 3: The Shape Context – p. 8/21

Finding the Tangent to a Curveat a Point

Find the normal which minimizes the squared pro-jections of points onto that normal. This requiresthe points all be shifted to be relative to their centerof gravity. pj ← pj −m, wherem = 1

N

∑

j pj.

∑

j

(nTpj)2

Lecture 3: The Shape Context – p. 9/21

Finding the Tangent to a Curveat a Point

Unfortunately, this is minimized byn = 0, an in-teresting, but totally useless result. Fix ita by con-strainingn to be a unit vector. Use a Lagrangemultiplier:

∑

j

(nTpj)2 + λnTn = nT

(

∑

j

pjpTj

)

n + λnTn

aThere is a typo on this slide. Can you find it?

Lecture 3: The Shape Context – p. 10/21

DeterminenDifferentiate and set the result to zero (using thederivative of a quadratic form twice:

2∑

j

pjpTj n = 2λn

An eigenvalue problem!n is one of the eigenvec-tors of the scatter matrix. But which one? The onecorresponding to the larger eigenvalue (but whichone?). Find the vector from the center point (xi) tothe next point (xi+1),

δ =xi+1 − xi

||xi+1 − xi||

and project the eigenvector ontoδ. Lecture 3: The Shape Context – p. 11/21

Projecting onto δ

Project the principal eigenvector on this vectornamedδ. p = δ ·n. If p is negative, flip the sign ofthe eigenvector. Let this (possibly) new eigenvec-tor (which is the tangent vector) also be denotedn.

Lecture 3: The Shape Context – p. 12/21

Measure angles in the tangentframe

Shape Context requires we find the an-gle (in the tangent frame) of the vectorto each other point. The angle is justthe inner product of the tangent vector(which we just found) with the vectorto the other point.

θ = cos−1 n ·

(

pj

||pj||

)

Lecture 3: The Shape Context – p. 13/21

Shape Context for a PointFrom a point, P, measure distance and angle to allother points. Histogram it. That histogram is theshape context for that point.

No, of course it’s morecomplicated than that.The angle is relative tothe local tangent. Andthe measurements arelogs of distance, butthat’s the gist of it.

Lecture 3: The Shape Context – p. 14/21

Shape Context –entire shapeQuantize angle into 12 bins, log-distance into 5bins. Now, we have a histogram with 60 ele-ments, which is the shape context for a singlepoint. The difference between the SC of twopoints can be written

γPQ =1

2

K∑

k=i

(h(P, k)− h(Q, k))2

h(P, k) + h(Q, k)

Lecture 3: The Shape Context – p. 15/21

Shape Context –entire shapeAll these assignment costs may be calculated andentered into a matrix.

Γij = [γPQ] P ∈ Ci, Q ∈ Cj

(1)

The cost of matching shapes. For a scalar measureof this cost, find the functionf : Ci → Cj, a bijec-tion (1:1 and onto), minimizes the total cost. Thatis, to theP th element ofCi we must find exactlyone element ofCj to assign it to.

Lecture 3: The Shape Context – p. 16/21

The Linear AssignmentProblem

The linear assignment problem is a special caseof the transportation problem, which is a specialcase of the minimum cost flow problem, whichis a special case of a linear program. Simplexalgorithm– exponential cost in worst caseSpecial algorithms for the special cases

Lecture 3: The Shape Context – p. 17/21

LAP(2)Suppose we wish to assignn workers ton tasks,(or establish a correspondence betweenn pointson one boundary andn on another), and assumewe can compute the cost of assigning each workerto each task. Could describe those costs as a ma-trix:

a1 a2 a3 a4

b1 b2 b3 b4

c1 c2 c3 c4

d1 d2 d3 d4

For example,a3 is the cost of assigning workerato perform task 3.

Lecture 3: The Shape Context – p. 18/21

LAP(3)On each row, subtract the smallest member. Thatwill leave (at least) one zero. Rename each of theother elements using primes; for example:

0 a2′ 0 a4′

b1′ b2′ b3′ 0

0 c2′ c3′ c4′

d1′ 0 d3′ d4′

Lecture 3: The Shape Context – p. 19/21

LAP(4)Now, since it is possible to choose zeros in sucha way that each row and column has exactly onezero, we are done. The assignments are:

0 a2′ 0 a4′

b1′ b2′ b3′ 0

0 c2′ c3′ c4′

d1′ 0 d3′ d4′

Lecture 3: The Shape Context – p. 20/21

LAP(5)Of course, life isn’t always so nice. Sometimesyou can’t just assign like last slide.

0 a2′ 0 a4′

b1′ b2′ b3′ 0

0 c2′ c3′ c4′

d1′ 0 d3′ d4′

This is too hard. Let’s do an example:

Lecture 3: The Shape Context – p. 21/21