Mo#on%Correspondence% - Welcome to the Department of Computer...
Transcript of Mo#on%Correspondence% - Welcome to the Department of Computer...
![Page 1: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/1.jpg)
Mo#on Correspondence
photoluver1@flickr
![Page 2: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/2.jpg)
Op#cal Flow
![Page 3: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/3.jpg)
![Page 4: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/4.jpg)
![Page 5: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/5.jpg)
Forward mo#on, 1-‐>2
![Page 6: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/6.jpg)
Backward mo#on: 1<-‐2
![Page 7: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/7.jpg)
Problem Defini#on
t t+1
1) Define regions of interests, or points of interests in the first image at ‘t’ 2) Search for correspondence in the second image at ‘t+1’
![Page 8: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/8.jpg)
![Page 9: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/9.jpg)
Challenges: Image appearance changes, even in the best cases!
![Page 10: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/10.jpg)
Mo#on simplified: G. ScoR
![Page 11: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/11.jpg)
Guy ScoR Ac#ng President of Zambia October 2014 to January 2015
![Page 12: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/12.jpg)
Approaches
Brightness Constancy Based Differen#al Technique, Lucas & Kanade (KLT)
Corner Feature Matching Discrete Matching Technique, ScoR & Longuet-‐Higgins
Takeo Kanade
![Page 13: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/13.jpg)
Differen#al Approach: KLT Tracker
• Detect corners features in first image • Use image patch as feature descrip#on
– Could be extended to color and texture descriptor • Use Lucas-‐Kanade algorithm to compute displacement of the pixels in the patch – Mo#on model could be transla#on (2 dof), affine (6 dof), or more general 3D models
• Subpixel accuracy • Do not need repeated detec#on
![Page 14: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/14.jpg)
Discrete Matching Approach
• Detect corners features in both images • Use image patch as feature descrip#on
– Could be extended to color, texture, SIFT/HOG descriptor
• Find correspondence as Permuta#on (C1, C2) – C1 (in image 1) is the best match to C2 (in image 2) – C2 (in image 2) is the best match to C1 (in image 1)
![Page 15: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/15.jpg)
Discrete Matching Approach
• Detect corners features in both images • Use image patch as feature descrip#on
– Could be extended to color, texture, SIFT/HOG descriptor
• Find correspondence as Permuta#on (C1, C2) – C1 (in image 1) is the best match to C2 (in image 2) – C2 (in image 2) is the best match to C1 (in image 1)
Need to seek Geometrical Valid matching
RANSAC Graph Matching
![Page 16: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/16.jpg)
Object Mo#on Only Compound Mo#on
• Slides from Andrew Cosand
G. L. ScoR, H. C. Longuet-‐Higgins
hRp://www.michaelbach.de/ot/mot_Ternus/index.html
![Page 17: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/17.jpg)
Ternus
![Page 18: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/18.jpg)
Ternus
![Page 19: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/19.jpg)
Ternus
![Page 20: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/20.jpg)
![Page 21: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/21.jpg)
ScoR & Longuet-‐Higgins
Define a distance metric between features Gij=e(-‐rij
2/2σ2)
Create matrix of rela#onships for all possible feature pairs
G11
Gij
![Page 22: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/22.jpg)
![Page 23: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/23.jpg)
Find a Permuta#on P btw points in image 1 to image 2, so that it ‘correlates best’, every point is happy.
t+1
maxP
X
i
X
j
PijGij = trace(PTG)
![Page 24: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/24.jpg)
Singular Value Decomposi#on
SVD factors a matrix into the product of two orthogonal matrices and a diagonal matrix of singular values (eigenvalues).
G = TDU, G is m-‐by-‐n, – T is orthogonal, m-‐by-‐m – D is diag(σ1, σ2, … σp), m-‐by-‐n, p=min{m,n} – U is orthogonal, n-‐by-‐n
![Page 25: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/25.jpg)
ScoR & Longuet-‐Higgins
Use Singular Value Decomposi#on on matrix. G = TDU
![Page 26: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/26.jpg)
ScoR & Longuet-‐Higgins Set diagonal elements of D to 1, ‘recover’ rela#onship matrix. P = TIU = TU
Elimina#ng singular matrix rescales data in feature space, essen#ally sphereing it.
![Page 27: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/27.jpg)
ScoR & Longuet-‐Higgins
Largest feature in row and column indicates mutual best match (correspondence)
![Page 28: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/28.jpg)
![Page 29: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/29.jpg)
![Page 30: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/30.jpg)
![Page 31: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/31.jpg)
Graph matching via SVD
maxP
X
i
X
j
PijGij = trace(PTG)
Goal is to:
The trick is to relax permuta#on P to an orthogonal matrix Q
1) Let F any orthogonal matrix, D a diagonal matrix
2) Transform G to D:
3) Transform solu#on of F by the same transforma#on:
trace(FT ·D) is max. at F = I
D = TT ·G · UT
trace(FTD) = trace(FTTTGUT ) = trace(UTFTTTG) = trace(QTG)
![Page 32: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/32.jpg)
Graph matching via SVD
maxP
X
i
X
j
PijGij = trace(PTG)
Goal is to:
trace(FTD) = trace(FTTTGUT ) = trace(UTFTTTG) = trace(QTG)
Q = T · F · UWhere orthogonal matrix
Since F = I P = T · I · U
![Page 33: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/33.jpg)
![Page 34: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/34.jpg)
![Page 35: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/35.jpg)
ScoR & Longuet-‐Higgins Geometrical Proper#es
• “In several of the examples we have described, and others too numerous to men#on, the circles were derived from the crosses by an affine transforma#on not involving rota#on, and in every case our algorithm succeeds in finding the feature correspondences created by this transforma#on.”
• “Because successive images in a sequence will oven be connected by transforma#ons that are affine or nearly so, this property is one to be welcomed, if not posi#vely required, in a sa#sfactory correspondence algorithm. The following argument is intended to explain why the algorithm performs so well in this respect.”
![Page 36: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/36.jpg)
• “if one set of points in a plane is mapped into another by a transla#on, an expansion or a shear deforma#on, then this 1:1 mapping minimizes the sum of the squares of the distances between corresponding points in the two”
Gij = dist(i, j);
si = Ai · ri + tAffine mo#on:
Discrete Graph World
Con#nuous Geometrical World
mimP
�
i
�
j
PijGij = trace(PT G)
![Page 37: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/37.jpg)
si = ri + t
Assume transla#on mo#on:
Show any other 1:1 mapping results in a greater value for the sum
where i' denotes the new partner of the point i.
�(ri � si�)2
Simple case: transla#on mo#on
![Page 38: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/38.jpg)
The simplest non-‐trivial case (1' = 2,2' = 3,3' =1). The aim is then to show that
![Page 39: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/39.jpg)
si = Ai · ri + t
For general Affine mo#on:
The aim is then to show that:
r · A · r +
A is symmetric and posi#ve definite
![Page 40: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/40.jpg)
Pilu’s Improvement
• Rogue features don’t correspond to anything, complica#ng the process.
• S&LH only deals with proximity and exclusivity.
• Similarity constraint can eliminate rogue features, which shouldn’t be similar to anything.
![Page 41: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/41.jpg)
Pilu’s Improvement
Modify rela#onship metric to include gray-‐level correla#on.
Gij = (e-‐(Cij – 1)2/2γ2) e(-‐rij2/2σ2)
Gij = ((Cij+1) /2) e(-‐rij2/2σ2)
– Adds similarity to feature space (kernel opera#on). – Rogue features can be eliminated because they are not similar to anything.
![Page 42: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/42.jpg)
![Page 43: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/43.jpg)
![Page 44: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/44.jpg)
References – M. Pilu, A direct method for stereo correspondence based on singular value decomposi#on
• variants – G. L. ScoR, H. C. Longuet-‐Higgins, An Algorithm for Associa#ng the Features of Two Images
![Page 45: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/45.jpg)
![Page 46: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/46.jpg)
![Page 47: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/47.jpg)
Forward mo#on, 1-‐>2
![Page 48: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/48.jpg)
Backward mo#on: 1<-‐2
![Page 49: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/49.jpg)
Problem Defini#on
t t+1
1) Define regions of interests, or points of interests in the first image at ‘t’ 2) Search for correspondence in the second image at ‘t+1’
![Page 50: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/50.jpg)
![Page 51: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/51.jpg)
Challenges: Image appearance changes, even in the best cases!
![Page 52: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/52.jpg)
Mo#on simplified: G. ScoR
![Page 53: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/53.jpg)
Approaches
Brightness Constancy Based Differen#al Technique, Lucas & Kanade (KLT)
Corner Feature Matching Discrete Matching Technique, ScoR & Longuet-‐Higgins
Takeo Kanade
![Page 54: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/54.jpg)
Differen#al Approach: KLT Tracker
• Detect corners features in first image • Use image patch as feature descrip#on
– Could be extended to color and texture descriptor • Use Lucas-‐Kanade algorithm to compute displacement of the pixels in the patch – Mo#on model could be transla#on (2 dof), affine (6 dof), or more general 3D models
• Subpixel accuracy • Do not need repeated detec#on
![Page 55: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/55.jpg)
Review:
![Page 56: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/56.jpg)
![Page 57: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/57.jpg)
![Page 58: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/58.jpg)
Op#cal Flow Assump#ons: Brightness Constancy
* Slide from Michael Black, CS143 2003
![Page 59: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/59.jpg)
Op#cal Flow Assump#ons:
* Slide from Michael Black, CS143 2003
![Page 60: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/60.jpg)
Op#cal Flow Assump#ons:
* Slide from Michael Black, CS143 2003
![Page 61: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/61.jpg)
Lucas-‐Kanade tracking Intensity constancy constraint:
![Page 62: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/62.jpg)
Define Sum of Squared Difference, SSD, error as:
![Page 63: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/63.jpg)
Three steps for solving this problem:
Solve for d, warp image, iterate with Newton Raphson.
![Page 64: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/64.jpg)
Step 1
![Page 65: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/65.jpg)
Differen#ate SSD with respect to d,
![Page 66: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/66.jpg)
Differen#ate SSD with respect to d,
![Page 67: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/67.jpg)
Differen#ate SSD with respect to d,
![Page 68: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/68.jpg)
Assume small mo#on, Taylor expansion of J(x+d) is
Step 2
![Page 69: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/69.jpg)
Assume small mo#on, Taylor expansion of J(x+d) is
![Page 70: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/70.jpg)
Combining previous equa#ons…
![Page 71: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/71.jpg)
Combining previous equa#ons…
together…
![Page 72: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/72.jpg)
Two unknown, two linear equa#ons
![Page 73: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/73.jpg)
![Page 74: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/74.jpg)
A: second moment matrix
![Page 75: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/75.jpg)
![Page 76: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/76.jpg)
Error vector b
![Page 77: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/77.jpg)
Error vector b
![Page 78: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/78.jpg)
![Page 79: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/79.jpg)
![Page 80: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/80.jpg)
What if A is not full rank? Recall we compute eigenvalue of A:
[v,d] = eig(A);
diag(d) contains the two eigenvalues, and we want
![Page 81: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/81.jpg)
Edge
– large gradients, all the same – large λ1, small λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 82: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/82.jpg)
Low texture region
– gradients have small magnitude – small λ1, small λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 83: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/83.jpg)
High textured region
– gradients are different, large magnitudes – large λ1, large λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 84: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/84.jpg)
Mo#on simplified: G. ScoR
![Page 85: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/85.jpg)
Iterna#on:
1) Update Ji+1(x) -‐> Ji(x+d) 2) Re-‐compute d, between Ji(x) and I(x)
![Page 86: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/86.jpg)
![Page 87: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/87.jpg)
![Page 88: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/88.jpg)
![Page 89: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/89.jpg)
![Page 90: Mo#on%Correspondence% - Welcome to the Department of Computer …cis.upenn.edu/~cis580/Spring2015/.../cis580-18-Correspondence-flow.… · Approaches% Brightness%Constancy%Based%](https://reader031.fdocuments.in/reader031/viewer/2022022608/5b88bc257f8b9a435b8e63c7/html5/thumbnails/90.jpg)