Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a...
-
Upload
dennis-atkinson -
Category
Documents
-
view
217 -
download
0
Transcript of Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a...
![Page 1: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/1.jpg)
Evaluation of the stability of SIFT keypoint correspondence across cameras
or .. “can we put a ‘C’ in SIFT?”
max van kleek6.869: learning and interfacesthursday may 11, 2005
![Page 2: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/2.jpg)
ubiquitous computing: computers (and cameras) are everywhere!
![Page 3: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/3.jpg)
little sister: follow-me-around user modeling
object correspondence acrossvarying cameras/lighting/scenes is an important subgoal
![Page 4: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/4.jpg)
applications: buliding a personal life log for yourself, interest profiling, social network mining, health care
![Page 5: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/5.jpg)
identifying objects with local features: the SIFT transform
![Page 6: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/6.jpg)
Orientation histogram = SIFT feature vector
![Page 7: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/7.jpg)
recognizing objects
identified and orientedkeypoints used to votefor pose orientationsin a hough transform
pose-and-scale space
![Page 8: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/8.jpg)
keypoint correspondence
1. % keypoints detected2. stability of orientation histogram
object orientation (away from frontal parallel)
object deformation
lighting direction, intensity, shading
lens distortion, sharpness, ccd “quality”, noise, capture artifacts
Mikolajczyk, K., C. Schmidt “A Performance Evaluation of Local Descriptors”, CVPR ‘03
Lowe, D.G. “Distinctive Image Features fromScale-Invariant Keypoints”, ICJV 2004.
Me! well, sort of..
???
![Page 9: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/9.jpg)
cameras vary widely in sizes, configurations, capabilities, and prices
the experiment
Logitech QC Express
640x48016-bit colorYUV4:2:2AGC, Auto Exposuremanual focusUSB iface$15
Logitech QC Pro 3000
CCD by Phillips640x48016-bit colorYUV4:2:2, RGBAGC, Auto ExposureAuto WBmanual focusUSB iface$50
Sony EVI-D30Steerable NTSCcamera,DV capture card
720x480 luminance,less for colorRaw DV AGC, Auto ExposureAuto focus
~$300 + $200
Nikon Coolpix 990Digital still camera,
2048x1536RGBAuto Gain, Auto WB, Auto ExposureAuto focus
$1000 -> $500
![Page 10: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/10.jpg)
experiment setup:5 incandenscent lights12 ft between camera and subject
![Page 11: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/11.jpg)
acquiring image sets
background10 images stationary
2 front 2 face right 2 face left
for each camera:
= 16 images/cam * 4 cameras = 48
![Page 12: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/12.jpg)
320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, disabled AE (locked to optimal settings)
![Page 13: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/13.jpg)
algorithm for keypoint correspondence
source image contrast stretch over whole set
contrast-stretchedbackground images
background model(mean)
-
find(p(img) < epsilon)
compute_sift_points
![Page 14: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/14.jpg)
dilate fg mask with a disc strel
intersectionfilter out bgkey points
keeping onlyrelevant keypointsby intersecting keypoints with foreground pts
![Page 15: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/15.jpg)
image A w/ sift keypoints
orientation histograms for each keypoint in A
orientation histograms for each keypoint in B
image B w/ sift keypoints
![Page 16: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/16.jpg)
orientation histograms for each keypoint in A
orientation histograms for each keypoint in B
match keypoints using nearest-neighbor in SIFT space
![Page 17: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/17.jpg)
orientation histograms for each keypoint in A
orientation histograms for each keypoint in B
match keypoints using nearest-neighbor in SIFT space
![Page 18: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/18.jpg)
1.1 -> 1.2 1.2 -> 1.1
sanity check: same (dv) camera, slightly different pose 1.1: 15 keypoints detected1.2: 11 keypoints detected
6 properly assigned8 in common
5 properly assigned8 in common
![Page 19: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/19.jpg)
4.1 -> 1.1 1.1 -> 4.1
nikon coolpix versus sony steerable 4.1: 18 keypoints detected1.1: 15 keypoints detected
5 properly assigned10 in common
5 properly assigned10 in common
![Page 20: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/20.jpg)
4.1 -> 3.1 3.1 -> 4.1
nikon coolpix versus qc pro 4.1: 18 keypoints detected3.1: 17 keypoints detected
2 properly assigned10 in common
5 properly assigned10 in common
![Page 21: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/21.jpg)
4.1 -> 2.1 4.1 -> 2.1
nikon coolpix versus qc express 4.1: 18 keypoints detected2.1: 22 keypoints detected
2 properly assigned10 in common
0 properly assigned10 in common
![Page 22: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/22.jpg)
other results:
qc pro vs qc exp (3.1 -> 2.1) 3.1->2.1 : 17 / 22 4 correct out of 6 in common2.1->3.1: 22 / 17 0 correct out of 6 in common
poor reproducibility with qcs?
qc exp test (is the qc exp just too noisy?)2.1->2.2 : 22/20 1 correct out of 8 in common2.2->2.1 : 20/22 2 correct out of 8 in common
yes.
qc pro reproducibility3.1->3.2: 17/24 6 correct out of 9 in common3.2->3.1 : 24/17 7 correct out of 9 in common
angle test using qc pro3.1->3.4 : 0 correct out of 0 in common
sensitive to out-of-plane rotation
![Page 23: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/23.jpg)
experiment setup:5 incandenscent lights3 ft between camera and robot
![Page 24: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/24.jpg)
320x240 (or standard, and downsampled afterwards) RGB colorspace; jpeg quality 100;default camera settings except disabled AGC, (locked to optimal settings)
![Page 25: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/25.jpg)
![Page 26: Evaluation of the stability of SIFT keypoint correspondence across cameras or.. “can we put a ‘C’ in SIFT?” max van kleek 6.869: learning and interfaces.](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfef1a28abf838cba07a/html5/thumbnails/26.jpg)
• parameters: – bins / histogram
– pixels / quadrant
– quadrants / keypoint
– gaussian dropoff covariance
keypoint splitting / multiple primary gradient directions
source keypoint merging
histogram blurring