Feature Point Detection

Lecture 07 Computer Vision

• Dr George Stockman

Professor Emeritus, Michigan State University

• Dr Mubarak Shah

Professor, University of Central Florida

• Dr David A. Forsyth

Professor, University of Illinois at Urbana- Champaign

• Dr Kristen Grauman

Associate Professor, University of Texas at Austin

Material Citations

Suggested Readings

• Chapter 4

Richard Szeliski , “Computer Vision: Algorithms and Applications”, Springer; 2011.

• Excerpt chapter from Synthesis lecture draft: Visual Recognition, Kristen Grauman and Bastian Leibe.

Find a bottle:

4

Categories Instances Find these two objects

Can’t do

unless you do not

care about few errors…

Can nail it

Motivation

M. Brown and D. G. Lowe. Recognizing Panoramas. ICCV 2003

• Example: Build a Panorama

How do we build panorama?

• We need to match (align) images

How do we build panorama?

• We need to match (align) images

Matching with Features

• Detect feature points in both images

Matching with Features

• Detect feature points in both images • Find corresponding pairs

Matching with Features

• Detect feature points in both images • Find corresponding pairs • Use these pairs to align images

Local features: main components

• Detection: Identify the interest points • Description: Extract feature vector descriptor surrounding each interest point • Matching: Determine correspondence between descriptors in two views

(1) (1)

1 1[ ,........ ]dX x x

(2) (2)

2 1[ ,........ ]dX x x

Goal: interest operator repeatability

• Detect the same point independently in both images

No chance to find true matches!

We need a repeatable detector

Goal: descriptor distinctiveness

• For each point correctly recognize the corresponding one

We need a reliable and distinctive descriptor

Must provide some invariance to geometric and photometric differences between the two views

Matching with Features

• Some patches can be localized or matched with higher accuracy than others

Matching with Features - More problems

• How to find landmarks to match across two images?

• How to distinguish one landmark from another?

• How to match N landmarks from image 1 to M landmarks from image 2, assuming that image 2 has more and/or less than image 1 ?

Image 1 Image 2

Matching with Features - More problems

• The mapping between the two image spaces may be nonlinear or complicated

• There may be errors in the locations of detected point or area features

• There may be extra features in one image or the other

• There may be missed features in one image or the other

Where can we use it further?

• Automate object tracking

• Point matching for computing disparity

• Stereo calibration - Estimation of fundamental matrix

• Motion based segmentation

• Recognition

• 3D object reconstruction

• Robot navigation

• Image retrieval and indexing…………etc.

What is an interest point?

• Expressive texture - The point at which the direction of the boundary of object changes abruptly - Intersection point between two or more edge segments

Synthetic & Real Interest Points

• Corners are indicated in red

How to determine landmarks?

• By unique intensity/color structure of Neighborhood

• By geometry or topology of neighborhood

Image 1 Image 2

Vectors show how intensity neighborhoods moved: similar vectors are colored the same

Properties of Interest Point Detectors

• Detect all (or most) true interest points

• No false interest points

• Well localized

• Robust with respect to noise

• Efficient detection

Possible Approaches to Corner Detection

• Based on brightness of images - Usually image derivatives

• Based on boundary extraction

- First step edge detection

- Curvature analysis of edges

Harris Corner Detector

• Corner point can be recognized in a window

• Shifting a window in any direction should give a large change in intensity

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

Harris Corner Detector - Basic Idea

Flat region: No change in all directions

Edge: No change along the edge direction

Corner: Significant change in all directions

Correlation

Correlation

Cross correlation

Auto correlation

Correlation Vs SSD

SSD: Subtract and minimize Correlation: Multiply and maximize

The local image energy is approximately constant, so it can be neglected

Constant, so it can be neglected

• Change of intensity for the shift [u,v]:

Intensity Shifted

intensity

Window function

or Window function w(x,y) =

Gaussian 1 in window, 0 outside

Harris Detector: Mathematics

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Taylor Series

Representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point

• Change of intensity for the shift [u,v]:

Harris Detector: Mathematics

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

2

,

( , ) ( , ) ( ) ( , )x y

x y

E u v w x y I x y uI vI I x y

Window function

Shifted intensity

Intensity

Window function

Shifted intensity

Intensity

2

,

( , ) ( , ) x y

x y

E u v w x y uI vI

2

,

( , ) ( , )x

x y y

IE u v w x y u v

I

Taylor series

• Change of intensity for the shift [u,v]:

Harris Detector: Mathematics

,

( , ) ( , )x

x y

x y y

I uE u v w x y u v I I

I v

,

( , ) ( , )x

x y

x y y

I uE u v u v w x y I I

I v

( , ) ,u

E u v u v Mv

• For small shifts [u,v] we have a bilinear approximation:

• Where M is a 22 matrix computed from image derivatives:

Harris Detector: Mathematics

( , ) ,u

E u v u v Mv

2

2,

( , )x x y

x y x y y

I I IM w x y

I I I

E(u,v) is an equation of an Ellipse

M is covariance matrix

• Intensity change in shifting window: eigen value analysis

1, 2 – eigen values of M

direction of the slowest

change

direction of the fastest change

(max)-1/2

(min)-1/2

Ellipse E(u,v) = const

Harris Detector: Mathematics

( , ) ,u

E u v u v Mv

Eigen Vectors and Eigen Values

• The eigen vector, x, of a matrix A is a special vector, with the following property Where is called eigen value • To find eigen values of a matrix A first find the roots of: • Then solve the following linear system for each eigen value to find corresponding eigen vector

Ax x

det( ) 0A I

( ) 0A I x

Example

1 2 0

0 3 4

0 0 7

A

Eigen Values

1 2 37, 3, 1

1 2 3

1 1 1

4 , 2 , 0

4 0 0

X X X

Eigen Vectors

Eigen Values

1 2 3

det( ) 0

1 2 0 1 0 0

det 0 3 4 0 1 0 0

0 0 7 0 0 1

1 2 0

det 0 3 4 0

0 0 7

( 1 ) (3 )(7 ) 0 0

( 1 )(3 )(7 ) 0

1, 3, 7

A I

Eigen Vectors

1

2

3

1

2

3

1

( ) 0

1 2 0 0

0 3 4 0

0 0 7 0

0 2 0 0

0 4 4 0

0 0 8 0

1

0

0

A I x

x

x

x

x

x

x

X

2

2 3

3

1 2 3

0 2 0 0

0 4 4 0

0 0 8 0

1, 0, 0

x

x x

x

x x x

1 1

Eigen value

Eigen vector

• Intensity change in shifting window: eigen value analysis

1, 2 – eigen values of M

direction of the slowest

change

direction of the fastest change

(max)-1/2

(min)-1/2

Ellipse E(u,v) = const

Harris Detector: Mathematics

( , ) ,u

E u v u v Mv

1 and 2 are small; E is almost constant in all directions

Classification of image points using eigen values of M:

1

2

“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Harris Detector: Mathematics

Matrix - Determinant and Trace

Determinant

Trace

(k – empirical constant, k = 0.04-0.06)

Harris Detector: Mathematics

2

det traceR M k M

1 2

1 2

det

trace

M

M

• Measure of corner response in terms of and : 21

2

1 2 1 2R k

1

2

0

0M

• R depends only on eigen values of M

• R is large for a corner

• R is negative with large magnitude for an edge • |R| is small for a flat region

Harris Detector: Mathematics

1

2

“Corner”

R > 0

“Edge”

“Edge” R < 0

“Flat”

|R| small R < 0

• Algorithm - Brief:

– Find points with large corner response function R

(R > threshold)

– Take the points of local maxima of R

Harris Detector

Harris Detector: Workflow

• Compute corner response R

Harris Detector: Workflow

• Find points with large corner response: R>threshold

Harris Detector: Workflow

• Take only the points of local maxima of R

Harris Detector: Workflow

Harris Detector: Workflow

• If pixel value is greater than its neighbors then it is a local maxima

1 1R

Other Version of Harris Detectors

1 2

1 2

det( )

( )

MR

trace M

1R

Triggs

Szeliski (Harmonic mean)

Shi-Tomasi

Other Version of Harris Detectors

1

2

• Algorithm - Detailed:

– Compute horizontal and vertical derivatives of image Ix and Iy

– Compute three images corresponding to three terms in matrix M

– Convolve these three images with a larger Gaussian (window)

– Compute scalar cornerness value using one of the R

– Find local maxima above some threshold as detected interest points

Harris Detector

• Average intensity change in direction [u,v] can be expressed as a bilinear form:

• Describe a point in terms of eigen values of M: measure of corner response

• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive

( , ) ,u

E u v u v Mv

2

1 2 1 2R k

Harris Detector: Summary

• Rotation invariance

• Ellipse rotates but its shape (i.e. eigen values) remains the same

Corner response R is invariant to image rotation

Harris Detector: Some Properties

• Partial invariance to affine intensity change

- Only derivatives are used => invariance to intensity shift I I + b

- Intensity scale: I a I

R

x (image coordinate)

threshold

R

x (image coordinate)

Harris Detector: Some Properties

• But: non-invariant to image scale!

All points will be classified as edges Corner !

Harris Detector: Some Properties

• Quality of Harris detector for different scale changes

• Repeatability rate:

# correspondences # possible correspondences

C. Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Harris Detector: Some Properties

• Geometry – Rotation

– Similarity (rotation + uniform scale)

– Affine (scale dependent on direction)

valid for: orthographic camera, locally planar object

• Photometry – Affine intensity change (I a I + b)

Model of Image Change

• Harris Corner Detector

C. Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Rotation Invariant Detection

Assignment 02 – Will be uploaded !!!