Feature trackingClass 5

Read Section 4.1 of course noteshttp://www.cs.unc.edu/~marc/tutorial/node49.html

Read Shi and Tomasi’s paper on good features to track

http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf

Read Lowe’s paper on SIFT featureshttp://www.unc.edu/courses/2004fall/comp/290/089/papers/Lowe_ijcv03.pdf

Don’t forget: Assignment 1(due by next Tuesday before class)

• Find a camera• Calibration approach 1

• Build/use calibration grid (2 orthogonal planes)• Perform calibration using (a) DLT and (b) complete

gold standard algorithm (assume error only in images, model radial distortion, ok to click points by hand)

• Calibration approach 2• Build/use planar calibration pattern• Use Bouguet’s matlab calibration toolbox (≈Zhang’s

approach)http://www.vision.caltech.edu/bouguetj/calib_doc/

(or implement it yourself for extra points)

• Compare results of approach 1(a),1(b) and 2• Make short report of findings and be ready to

discuss in class

Single view metrology

• Allows to relate height of point to height of camera

Single view metrology

• Allows to transfer point from one plane to another

Feature trackingClass 5

Read Section 4.1 of course noteshttp://www.cs.unc.edu/~marc/tutorial/node49.html

Read Shi and Tomasi’s paper on good features to track

http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf

Read Lowe’s paper on SIFT featureshttp://www.unc.edu/courses/2004fall/comp/290/089/papers/Lowe_ijcv03.pdf

Feature matching vs. tracking

Extract features independently and then match by comparing descriptors

Extract features in first images and then try to find same feature back in next view

What is a good feature?

Image-to-image correspondences are key to passive triangulation-based 3D reconstruction

Compare intensities pixel-by-pixel

Comparing image regions

I(x,y) I´(x,y)

Sum of Square Differences

Dissimilarity measures

Compare intensities pixel-by-pixel

Comparing image regions

I(x,y) I´(x,y)

Zero-mean Normalized Cross Correlation

Similarity measures

Feature points

• Required properties:• Well-defined

(i.e. neigboring points should all be different)

• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)

Feature point extraction

homogeneous

edge

corner

Find points that differ as much as possible from all neighboring points

Feature point extraction

• Approximate SSD for small displacement Δ• Image difference, square difference for

pixel• SSD for window

Feature point extraction

homogeneous

edge

corner

Find points for which the following is maximum

i.e. maximize smallest eigenvalue of M

Harris corner detector

• Only use local maxima, subpixel accuracy through second order surface fitting

• Select strongest features over whole image and over each tile (e.g. 1000/image, 2/tile)

• Use small local window:• Maximize „cornerness“:

Simple matching• for each corner in image 1 find the corner

in image 2 that is most similar (using SSD or NCC) and vice-versa

• Only compare geometrically compatible points

• Keep mutual best matches

What transformations does this work for?

Feature matching: example

0.96 -0.40

-0.16

-0.39

0.19

-0.05

0.75 -0.47

0.51 0.72

-0.18

-0.39

0.73 0.15 -0.75

-0.27

0.49 0.16 0.79 0.21

0.08 0.50 -0.45

0.28 0.99

1 5

24

3

1 5

24

3

What transformations does this work for?

What level of transformation do we need?

Wide baseline matching

• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination

geometric transformations

photometric changes

Wide-baseline matching example

(Tuytelaars and Van Gool BMVC 2000)

Lowe’s SIFT features

Recover features with position, orientation and scale

(Lowe, ICCV99)

Position

• Look for strong responses of DOG filter (Difference-Of-Gaussian)

• Only consider local maxima

Scale

• Look for strong responses of DOG filter (Difference-Of-Gaussian) over scale space

• Only consider local maxima in both position and scale

• Fit quadratic around maxima for subpixel

Orientation

• Create histogram of local gradient directions computed at selected scale

• Assign canonical orientation at peak of smoothed histogram

• Each key specifies stable 2D coordinates (x, y, scale, orientation)

0 2

Minimum contrast and “cornerness”

SIFT descriptor• Thresholded image gradients are sampled

over 16x16 array of locations in scale space• Create array of orientation histograms• 8 orientations x 4x4 histogram array = 128

dimensions

Matas et al.’s maximally stable regions

• Look for extremal regions

http://cmp.felk.cvut.cz/~matas/papers/matas-bmvc02.pdf

Mikolaczyk and Schmid LoG Features

Feature tracking

• Identify features and track them over video• Small difference between frames• potential large difference overall

• Standard approach: KLT (Kanade-Lukas-Tomasi)

Good features to track

• Use same window in feature selection as for tracking itself

• Compute motion assuming it is small

Affine is also possible, but a bit harder (6x6 in stead of 2x2)

Example

Example

Synthetic example

Good features to keep tracking

Perform affine alignment between first and last frameStop tracking features with too large errors

Live demo

• OpenCV (try it out!)

LKdemo

Next class: triangulation and reconstruction

C1m1

L1

m2

L2

M

C2

Triangulation

- calibration

- correspondences