Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a...

26
Evaluation of the stability of SIFT keypoint correspondence across cameras or .. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces thursday may 11, 2005

Transcript of Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a...

Page 1: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

Evaluation of the stability of SIFT keypoint correspondence across cameras

or .. “can we put a ‘C’ in SIFT?”

max van kleek6.869: learning and interfacesthursday may 11, 2005

Page 2: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

ubiquitous computing: computers (and cameras) are everywhere!

Page 3: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

little sister: follow-me-around user modeling

object correspondence acrossvarying cameras/lighting/scenes is an important subgoal

Page 4: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

applications: buliding a personal life log for yourself, interest profiling, social network mining, health care

Page 5: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

identifying objects with local features: the SIFT transform

Page 6: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

Orientation histogram = SIFT feature vector

Page 7: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

recognizing objects

identified and orientedkeypoints used to votefor pose orientationsin a hough transform

pose-and-scale space

Page 8: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

keypoint correspondence

1. % keypoints detected2. stability of orientation histogram

object orientation (away from frontal parallel)

object deformation

lighting direction, intensity, shading

lens distortion, sharpness, ccd “quality”, noise, capture artifacts

Mikolajczyk, K., C. Schmidt “A Performance Evaluation of Local Descriptors”, CVPR ‘03

Lowe, D.G. “Distinctive Image Features fromScale-Invariant Keypoints”, ICJV 2004.

Me! well, sort of..

???

Page 9: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

cameras vary widely in sizes, configurations, capabilities, and prices

the experiment

Logitech QC Express

640x48016-bit colorYUV4:2:2AGC, Auto Exposuremanual focusUSB iface$15

Logitech QC Pro 3000

CCD by Phillips640x48016-bit colorYUV4:2:2, RGBAGC, Auto ExposureAuto WBmanual focusUSB iface$50

Sony EVI-D30Steerable NTSCcamera,DV capture card

720x480 luminance,less for colorRaw DV AGC, Auto ExposureAuto focus

~$300 + $200

Nikon Coolpix 990Digital still camera,

2048x1536RGBAuto Gain, Auto WB, Auto ExposureAuto focus

$1000 -> $500

Page 10: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

experiment setup:5 incandenscent lights12 ft between camera and subject

Page 11: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

acquiring image sets

background10 images stationary

2 front 2 face right 2 face left

for each camera:

= 16 images/cam * 4 cameras = 48

Page 12: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, disabled AE (locked to optimal settings)

Page 13: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

algorithm for keypoint correspondence

source image contrast stretch over whole set

contrast-stretchedbackground images

background model(mean)

-

find(p(img) < epsilon)

compute_sift_points

Page 14: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

dilate fg mask with a disc strel

intersectionfilter out bgkey points

keeping onlyrelevant keypointsby intersecting keypoints with foreground pts

Page 15: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

image A w/ sift keypoints

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

image B w/ sift keypoints

Page 16: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space

Page 17: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

orientation histograms for each keypoint in A

orientation histograms for each keypoint in B

match keypoints using nearest-neighbor in SIFT space

Page 18: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

1.1 -> 1.2 1.2 -> 1.1

sanity check: same (dv) camera, slightly different pose 1.1: 15 keypoints detected1.2: 11 keypoints detected

6 properly assigned8 in common

5 properly assigned8 in common

Page 19: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

4.1 -> 1.1 1.1 -> 4.1

nikon coolpix versus sony steerable 4.1: 18 keypoints detected1.1: 15 keypoints detected

5 properly assigned10 in common

5 properly assigned10 in common

Page 20: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

4.1 -> 3.1 3.1 -> 4.1

nikon coolpix versus qc pro 4.1: 18 keypoints detected3.1: 17 keypoints detected

2 properly assigned10 in common

5 properly assigned10 in common

Page 21: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

4.1 -> 2.1 4.1 -> 2.1

nikon coolpix versus qc express 4.1: 18 keypoints detected2.1: 22 keypoints detected

2 properly assigned10 in common

0 properly assigned10 in common

Page 22: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

other results:

qc pro vs qc exp (3.1 -> 2.1) 3.1->2.1 : 17 / 22 4 correct out of 6 in common2.1->3.1: 22 / 17 0 correct out of 6 in common

poor reproducibility with qcs?

qc exp test (is the qc exp just too noisy?)2.1->2.2 : 22/20 1 correct out of 8 in common2.2->2.1 : 20/22 2 correct out of 8 in common

yes.

qc pro reproducibility3.1->3.2: 17/24 6 correct out of 9 in common3.2->3.1 : 24/17 7 correct out of 9 in common

angle test using qc pro3.1->3.4 : 0 correct out of 0 in common

sensitive to out-of-plane rotation

Page 23: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

experiment setup:5 incandenscent lights3 ft between camera and robot

Page 24: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, (locked to optimal settings)

Page 25: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.
Page 26: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.

• parameters: – bins / histogram

– pixels / quadrant

– quadrants / keypoint

– gaussian dropoff covariance

keypoint splitting / multiple primary gradient directions

source keypoint merging

histogram blurring