Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009,...

Computer Vision (EEE6503) Fall 2009, Yonsei Univ.

KanadeKanade--LucasLucas--TomasiTomasi (KLT)(KLT)Feature TrackerFeature Tracker

Computer Vision Lab.Computer Vision Lab.

Jae Kyu SuhrJae Kyu Suhr


IntroductionIntroduction


MotivationMotivation

Image sequence



Feature point detection



Feature point tracking



Tracking result


3D Reconstruction results3D Reconstruction results


Problem StatementProblem Statement



• I, J : 1st and 2nd grayscaled images

• I(x,y) : gray value of I at [x y]T

• Let u=[ux uy]T be a point on the 1st image I

• The goal is to find v on J, where I(u) and J(v) aresimilar. (v= u+d = [ux+dx uy+dy]T)

• d=[dx dy]T is the image velocity at uor the optical flow at u.



• Definition of the residual function ε(d).

• ωx and ωy integration window size parameterintegration window size (2ωx+1)×(2ωy+1).

(ux, uy)

wx

wy

I J

∑( - )2


Pyramid ImplementationPyramid Implementation


Why pyramid representation?Why pyramid representation?

• Standard KLT algorithm can deal with small pixel displacement.• Solution for this is a pyramidal implementation.• Let I0=I be the 0th level image• The pyramid representation is built recursively.

I0 → I1 → I2 → I3 → ILm … (Lm: 2~4)• IL-1 : the image at level L-1

IL : the image at level L


Pyramidal feature trackingPyramidal feature tracking

• A given point u in I, find its corresponding location v=u+d.

• Corresponding point of u(u0) on the pyramidal image IL is uL

• Simple overall pyramid tracking algorithm

dLm is computed at the pyramid level Lm

dLm-1 is computedwith an initial guess of dLm at Lm-1

This continues up to the level 0


• Let gL=[gLx gL

y]T be an initial guess at level L( gL is available from level Lm to level L+1 )

• Residual pixel displacement vector dL=[dLx dL

y]T

that minimizes the new image matching error function εL

• Window size (2ωx+1)×(2ωy+1) is constant for all pyramid

• gL is used to pre-translate the image patch in 2nd image J→ dL is small and therefore easy to compute using KLT algorithm




• Assume that dL is computed, new initial guess gL-1 at level L-1

• Initial guess for the deepest level Lm

• The final optical flow solution d


Standard KLT algorithmStandard KLT algorithm


Standard KLT optical flow computationStandard KLT optical flow computation

• Goal is to find dL that minimizes the matching function εL.Same operation is performed for all levels L, drop the superscript L and define the new images A and B

• Let us change the displacement vector and the point vector



• Goal is to find that minimizes the matching function

• To find the optimum

• Let us substitute by its 1st order Taylor expansion about the point

frame difference Image gradient



( G must be invertible).

Standard LK is valid only the pixel displacement is small.

(Because of the first order Taylor approximation)

Therefore, iterative version of LK is necessary.


Iterative KLT algorithmIterative KLT algorithm


Iterative KLT optical flow computationIterative KLT optical flow computation

• k : iteration index

•: initial guess from the previous iteration 1,2,…,k-1

• Bk : new translated image according to

• The goal is to compute that minimize



• One step LK optical flow computation is

• Ix, Iy are computed only once at the beginning of the iteration.

• G (2×2matrix) remains constant throughout the iteration loop.

• Only δIk needs to be recomputed at each iteration

• Once is computed, a new pixel displacement guess

• The iteration goes on until is smaller than a thresholdor reached at the maximum number of iteration.(5 are enough to reach convergence)



• At 1st iteration, the initial guess

• Assuming that K iterations are necessary to reach convergence,the final solution for the optical flow vector

• This overall procedure is repeated at all levels L-1, L-2, … , 0


Feature selectionFeature selection

• Goal is to find the location of v on J corresponding to u on I.How to detect u on I

• Find the point u whose G matrix is non-singularThe minimum eigenvalue of G must larger than a threshold.



1. Compute the G matrix and its minimum eigenvalue λm at every pixelin the image I.

2. Call λmax the maximum value of λm over the whole image.

3. Retain the image pixels that have a λm value larger than a threshold.

4. Retain the local maximum pixels (a pixel is kept if its λm value is larger than that of any other pixel in its 3×3 neighborhood).

5. Keep the subset of those pixels so that the minimum distancebetween any pair of pixels is larger than a given threshold distance.


Declaring a feature Declaring a feature ““LostLost””

• A feature point falls outside of the image.

• An image patch around the tracked point varies too much between image I and image J. (A cost function (SSD) is larger than a threshold).


Flow chartFlow chart

Input 1st image

Select feature points

Input 2nd image

Track feature points

Delete lost feature points

Save feature points

Save tracked points

Replace 1st image with 2nd image

Replace 2st image with next image

Select feature pointsto replace lost feature points

Save feature points


ReferencesReferences

• Bruce D. Lucas and Takeo Kanade. “An Iterative Image Registration Technique with an Application to Stereo Vision”. IJCAI, pages 674-679, 1981.

• J. Shi and C. Tomasi, "Good Features to Track," CVPR'94

• Jean-Yves Bouguet, “Pyramidal Implementation of the Lucas KanadeFeature Tracker Description of the algorithm”, Intel Corporation Microprocessor Research Labs.

• http://www.ces.clemson.edu/~stb/klt/ C++ codeKLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker

• OpenCV: CalcOpticalFlowLK, CalcOpticalFlowPyrLK

• http://vision.ucla.edu//MASKS/labs.html Matlab codeAn Invitation to 3D Vision


Thank youThank you

Q & AQ & A

Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009,...

Documents

Transcript of Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009,...