Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image...
Transcript of Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image...
![Page 1: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/1.jpg)
Image Features
Sanja Fidler CSC420: Intro to Image Understanding 1 / 1
![Page 2: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/2.jpg)
Image Features
Image features are useful descriptions of local or global image
properties designed (or learned!) to accomplish a certain task
You may want to choose di↵erent features for di↵erent tasks
Depending on the problem we need to typically answer threequestions:
Where to extract image features?
What to extract (what’s the content of the feature)?
How to use them for your task, e.g., how to match them?
Sanja Fidler CSC420: Intro to Image Understanding 2 / 1
![Page 3: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/3.jpg)
Image Features
Let’s watch a video clip
Sanja Fidler CSC420: Intro to Image Understanding 3 / 1
![Page 4: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/4.jpg)
Image Features
Where is the movie taking place?
Sanja Fidler CSC420: Intro to Image Understanding 4 / 1
![Page 5: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/5.jpg)
Image Features
Where is the movie taking place?
Sanja Fidler CSC420: Intro to Image Understanding 4 / 1
![Page 6: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/6.jpg)
Image Features
Where is the movie taking place?
Sanja Fidler CSC420: Intro to Image Understanding 4 / 1
![Page 7: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/7.jpg)
Image Features
Where is the movie taking place?
We matched in:
Distinctive locations:keypoints
Distinctive features:descriptors
Sanja Fidler CSC420: Intro to Image Understanding 4 / 1
![Page 8: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/8.jpg)
Image Features
Tracking: Where to did the scene/actors move?
Sanja Fidler CSC420: Intro to Image Understanding 5 / 1
![Page 9: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/9.jpg)
Image Features
Tracking: Where to did the scene/actors move?
We matched:
Quite distinctive locations
Quite distinctive features
Sanja Fidler CSC420: Intro to Image Understanding 5 / 1
![Page 10: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/10.jpg)
Image Features
A shot in a movie is a clip with a coherent camera (no sudden viewpoint
changes)
Sanja Fidler CSC420: Intro to Image Understanding 6 / 1
![Page 11: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/11.jpg)
Image Features
A shot in a movie is a clip with a coherent camera (no sudden viewpoint
changes)
We matched:
Globally – one descriptorfor full image
Descriptor can be simple,e.g. color
Sanja Fidler CSC420: Intro to Image Understanding 6 / 1
![Page 12: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/12.jpg)
Image Features
How could we tell which type of scene it is?
Sanja Fidler CSC420: Intro to Image Understanding 7 / 1
![Page 13: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/13.jpg)
Image Features
How could we tell which type of scene it is?
We matched:
Globally – one descriptorfor full image (?)
More complex descriptor:color, gradients, “deep”features (learned), etc
Sanja Fidler CSC420: Intro to Image Understanding 7 / 1
![Page 14: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/14.jpg)
Image Features
How would we solve this?
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 15: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/15.jpg)
Image Features
How would we solve this?
We matched:
One descriptor for fullpatch
Descriptor can be simple,e.g. color
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 16: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/16.jpg)
Image Features
How would we solve this?
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 17: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/17.jpg)
Image Features
How would we solve this?
We matched:
At each location
Compared pixel values
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 18: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/18.jpg)
Image Features
How would we solve this?
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 19: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/19.jpg)
Image Features
How would we solve this?
We matched:
Distinctive locations
Distinctive features
A�ne invariant
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 20: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/20.jpg)
Image Features
How would we solve this?
Sanja Fidler CSC420: Intro to Image Understanding 8 / 1
![Page 21: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/21.jpg)
Image Features
Detection: Where to extract image features?
“Interesting” locations (keypoints, interesting regions)
In each location (densely)
Description: What to extract?
What’s the spatial scope of the feature?
What’s the content of the feature?
Matching: How to match them?
Sanja Fidler CSC420: Intro to Image Understanding 9 / 1
![Page 22: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/22.jpg)
Image Features
Detection: Where to extract image features?
“Interesting” locations (keypoints) TODAY
In each location (densely)
Description: What to extract?
What’s the spatial scope of the feature?
What’s the content of the feature?
Matching: How to match them?
Sanja Fidler CSC420: Intro to Image Understanding 9 / 1
![Page 23: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/23.jpg)
Image Features:
Interest Point (Keypoint) Detection
Sanja Fidler CSC420: Intro to Image Understanding 10 / 1
![Page 24: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/24.jpg)
Application Example: Image Stitching
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 11 / 1
![Page 25: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/25.jpg)
Local Features
Detection: Identify the interest points.
Description: Extract feature vector descriptor around each interest point.
Matching: Determine correspondence between descriptors in two views.
[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 12 / 1
![Page 26: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/26.jpg)
Goal: Repeatability of the Interest Point Operator
Our goal is to detect (at least some of) the same points in both images
We have to be able to run the detection procedure independently per image
We need to generate enough points to increase our chances of detectingmatching points
We shouldn’t generate too many or our matching algorithm will be too slow
Figure: Too few keypoints ! little chance to find the true matches
[Source: K. Grauman, slide credit: R. Urtasun]Sanja Fidler CSC420: Intro to Image Understanding 13 / 1
![Page 27: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/27.jpg)
Goal: Distinctiveness of the Keypoints
We want to be able to reliably determine which point goes with which.
[Source: K. Grauman, slide credit: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 14 / 1
![Page 28: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/28.jpg)
What Points to Choose?
[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 15 / 1
![Page 29: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/29.jpg)
What Points to Choose?
Textureless patches are nearly impossible to localize.
[Adopted from: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 16 / 1
![Page 30: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/30.jpg)
What Points to Choose?
Textureless patches are nearly impossible to localize.
Patches with large contrast changes (gradients) are easier to localize.
[Adopted from: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 16 / 1
![Page 31: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/31.jpg)
What Points to Choose?
Textureless patches are nearly impossible to localize.Patches with large contrast changes (gradients) are easier to localize.
But straight line segments cannot be localized on lines segments with thesame orientation (aperture problem)
[Adopted from: R. Urtasun]Sanja Fidler CSC420: Intro to Image Understanding 16 / 1
![Page 32: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/32.jpg)
What Points to Choose?
Textureless patches are nearly impossible to localize.Patches with large contrast changes (gradients) are easier to localize.But straight line segments cannot be localized on lines segments with thesame orientation (aperture problem)
Gradients in at least two di↵erent orientations are easiest, e.g., corners!
[Adopted from: R. Urtasun]Sanja Fidler CSC420: Intro to Image Understanding 16 / 1
![Page 33: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/33.jpg)
What Points to Choose?
Textureless patches are nearly impossible to localize.Patches with large contrast changes (gradients) are easier to localize.
But straight line segments cannot be localized on lines segments with thesame orientation (aperture problem)
Gradients in at least two di↵erent orientations are easiest, e.g., corners!
[Adopted from: R. Urtasun]Sanja Fidler CSC420: Intro to Image Understanding 16 / 1
![Page 34: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/34.jpg)
Interest Points: Corners
How can we find corners in an image?
Sanja Fidler CSC420: Intro to Image Understanding 17 / 1
![Page 35: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/35.jpg)
Interest Points: Corners
We should easily recognize the point by looking through a small window.
Shifting a window in any direction should give a large change in intensity.
Figure: (left) flat region: no change in all directions, (center) edge: no changealong the edge direction, (right) corner: significant change in all directions
[Source: Alyosha Efros, Darya Frolova, Denis Simakov]
Sanja Fidler CSC420: Intro to Image Understanding 18 / 1
![Page 36: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/36.jpg)
Interest Points: Corners
Compare two image patches using (weighted) summed square di↵erence
Measures change in appearance of window w(x , y) for the shift
[Source: J. Hays]Sanja Fidler CSC420: Intro to Image Understanding 19 / 1
![Page 37: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/37.jpg)
Interest Points: Corners
Compare two image patches using (weighted) summed square di↵erence
Measures change in appearance of window w(x , y) for the shift
[Source: J. Hays]Sanja Fidler CSC420: Intro to Image Understanding 19 / 1
![Page 38: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/38.jpg)
Interest Points: Corners
Compare two image patches using (weighted) summed square di↵erence
Measures change in appearance of window w(x , y) for the shift
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 19 / 1
![Page 39: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/39.jpg)
Interest Points: Corners
Let’s look at EWSSD
We want to find out how this function behaves for small shifts
Remember our goal to detect corners:
Sanja Fidler CSC420: Intro to Image Understanding 20 / 1
![Page 40: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/40.jpg)
Interest Points: Corners
Using a simple first-order Taylor Series expansion:
I (x + u, y + v) ⇡ I (x , y) + u · @I@x
(x , y) + v · @I@y
(x , y)
And plugging it in our expression for EWSSD:
EWSSD(u, v) =X
x
X
y
w(x , y)⇣I (x + u, y + v)� I (x , y)
⌘2
⇡X
x
X
y
w(x , y)⇣I (x , y) + u · I
x
+ v · Iy
� I (x , y)⌘2
=X
x
X
y
w(x , y)⇣u
2
I
2
x
+ 2u · v · Ix
· Iy
+ v
2
I
2
y
⌘
=X
x
X
y
w(x , y) ·⇥u v
⇤ I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
� u
v
�
Sanja Fidler CSC420: Intro to Image Understanding 21 / 1
![Page 41: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/41.jpg)
Interest Points: Corners
Since (u, v) doesn’t depend on (x , y) we can rewriting it slightly:
EWSSD(u, v) =X
x
X
y
w(x , y)⇥u v
⇤ I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
� u
v
�
=⇥u v
⇤ ⇣X
x
X
y
w(x , y)
I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
�⌘
| {z }Let’s denotes this with M
u
v
�
=⇥u v
⇤M
u
v
�
M is a 2⇥ 2 second moment matrix computed from image gradients:
M =X
x
X
y
w(x , y)
I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
�
Sanja Fidler CSC420: Intro to Image Understanding 22 / 1
![Page 42: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/42.jpg)
How Do I Compute M?
Let’s say I have this image
Sanja Fidler CSC420: Intro to Image Understanding 23 / 1
![Page 43: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/43.jpg)
How Do I Compute M?
Let’s say I have this image
I need to compute a 2⇥ 2 second moment matrix in each image location
Sanja Fidler CSC420: Intro to Image Understanding 23 / 1
![Page 44: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/44.jpg)
How Do I Compute M?
Let’s say I have this image
I need to compute a 2⇥ 2 second moment matrix in each image location
In a particular location I need to compute M as a weighted average ofgradients in a window
Sanja Fidler CSC420: Intro to Image Understanding 23 / 1
![Page 45: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/45.jpg)
How Do I Compute M?
Let’s say I have this image
I need to compute a 2⇥ 2 second moment matrix in each image location
In a particular location I need to compute M as a weighted average ofgradients in a window
I can do this e�ciently by computing three matrices, I 2x
, I 2y
and I
x
· Iy
, andconvolving each one with a filter, e.g. a box or Gaussian filter
Sanja Fidler CSC420: Intro to Image Understanding 23 / 1
![Page 46: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/46.jpg)
Interest Points: Corners
We now have M computed in each image location
Our EWSSD is a quadratic function where M implies its shape
EWSSD(u, v) =⇥u v
⇤M
u
v
�
M =X
x
X
y
w(x , y)
I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
�
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 24 / 1
![Page 47: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/47.jpg)
Interest Points: Corners
Let’s take a horizontal “slice” of EWSSD(u, v):
⇥u v
⇤M
u
v
�= const
This is the equation of an ellipse
Figure: Di↵erent ellipses obtain by di↵erent horizontal “slices”
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 25 / 1
![Page 48: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/48.jpg)
Interest Points: Corners
Let’s take a horizontal “slice” of EWSSD(u, v):
⇥u v
⇤M
u
v
�= const
This is the equation of an ellipse
Figure: Di↵erent ellipses obtain by di↵erent horizontal “slices”
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 25 / 1
![Page 49: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/49.jpg)
Interest Points: Corners
Our matrix M is symmetric:
M =X
x
X
y
w(x , y)
I
2
x
I
x
· Iy
I
x
· Iy
I
2
y
�
And thus we can diagonalize it (in Matlab: [V,D] = eig(M)):
M = V
�1
00 �
2
�V
�1
Columns of V are major and minor axes of ellipse, ��1/2 are radius
[Source: J. Hays]Sanja Fidler CSC420: Intro to Image Understanding 26 / 1
![Page 50: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/50.jpg)
Interest Points: Corners
Columns of V are principal directions
�1
, �2
are principal curvatures
[Source: F. Flores-Mangas]Sanja Fidler CSC420: Intro to Image Understanding 27 / 1
![Page 51: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/51.jpg)
Interest Points: Corners
The eigenvalues of M (�1
, �2
) reveal the amount of intensity changein the two principal orthogonal gradient directions in the window
[Source: R. Szeliski, slide credit: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 28 / 1
![Page 52: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/52.jpg)
Interest Points: Corners
How do the ellipses look like for this image?
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 29 / 1
![Page 53: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/53.jpg)
Interest Points: Corners
How do the ellipses look like for this image?
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 29 / 1
![Page 54: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/54.jpg)
Interest Points: Corners
[Source: K. Grauman, slide credit: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 30 / 1
![Page 55: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/55.jpg)
Interest Points: Criteria to Find Corners
Harris and Stephens, ’88, is rotationally invariant and downweighs edge-likefeatures where �
1
� �0
R = det(M)� ↵ · trace(M)2 = �0
�1
� ↵(�0
+ �1
)2
Why go via det and trace and not use a criteria with �?
↵ a constant (0.04 to 0.06)
The corresponding detector is called Harris corner detector
Sanja Fidler CSC420: Intro to Image Understanding 31 / 1
![Page 56: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/56.jpg)
Interest Points: Criteria to Find Corners
Harris and Stephens, 88 is rotationally invariant and downweighs edge-likefeatures where �
1
� �0
R = det(M)� ↵trace(M)2 = �0
�1
� ↵(�0
+ �1
)2
Shi and Tomasi, 94 proposed the smallest eigenvalue of A, i.e., ��1/20
.
Triggs, 04 suggested�0
� ↵�1
also reduces the response at 1D edges, where aliasing errors sometimesinflate the smaller eigenvalue
Brown et al, 05 use the harmonic mean
det(A)trace(A)
=�0
�1
�0
+ �1
[Source R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 32 / 1
![Page 57: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/57.jpg)
Harris Corner detector
1 Compute gradients Ix
and I
y
2 Compute I
2
x
, I 2y
, Ix
· Iy
3 Average (Gaussian) ! gives M
4 Compute
R = det(M)�↵ trace(M)2 for each
image window (cornerness score)
5 Find points with large R (R >
threshold).
6 Take only points of local maxima,
i.e., perform non-maximum
suppression
Sanja Fidler CSC420: Intro to Image Understanding 33 / 1
![Page 58: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/58.jpg)
Example
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 34 / 1
![Page 59: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/59.jpg)
1) Compute Cornerness
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 35 / 1
![Page 60: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/60.jpg)
2) Find High Response
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 36 / 1
![Page 61: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/61.jpg)
3) Non-maxima Suppresion
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 37 / 1
![Page 62: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/62.jpg)
Results
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 38 / 1
![Page 63: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/63.jpg)
Another Example
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 39 / 1
![Page 64: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/64.jpg)
Cornerness
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 40 / 1
![Page 65: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/65.jpg)
Interest Points
[Source: K. Grauman]Sanja Fidler CSC420: Intro to Image Understanding 41 / 1
![Page 66: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/66.jpg)
Interest Points – Ideal Properties?
We want corner locations to be invariant to photometric transformationsand covariant to geometric transformations
Invariance : Image is transformed and corner locations do not changeCovariance : If we have two transformed versions of the same image, features
should be detected in corresponding locations
[Source: J. Hays]Sanja Fidler CSC420: Intro to Image Understanding 42 / 1
![Page 67: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/67.jpg)
Properties of Harris Corner Detector
Shift?
Harris corner detector is shift-covariant (our window functions shift)
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 43 / 1
![Page 68: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/68.jpg)
Properties of Harris Corner Detector
Rotation?
Second moment ellipse rotates but its shape (i.e. eigenvalues)remains the sameHarris corner detector is rotation-covariant
[Source: J. Hays]Sanja Fidler CSC420: Intro to Image Understanding 44 / 1
![Page 69: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/69.jpg)
Properties of Harris Corner Detector
Scale?
Corner location is not scale invariant/covariant!
[Source: J. Hays]
Sanja Fidler CSC420: Intro to Image Understanding 45 / 1
![Page 70: Image Features - Department of Computer Science ...fidler/slides/2019/CSC420/lecture6.pdf · Image Features Image features are useful descriptions of local or global image properties](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecad484c7ae396261573c97/html5/thumbnails/70.jpg)
Next Time
Can we also define keypoints that are shift, rotation and scaleinvariant/covariant?
What should be our description around keypoint?
Sanja Fidler CSC420: Intro to Image Understanding 46 / 1