Optical Flow I - sci.utah.edugerig/CS6320-S2013/Materials/CS6320-CV-S201… · Optical Flow I Guido...
Transcript of Optical Flow I - sci.utah.edugerig/CS6320-S2013/Materials/CS6320-CV-S201… · Optical Flow I Guido...
Optical Flow I
Guido Gerig CS 6320, Spring 2013
(credits: Marc Pollefeys UNC Chapel Hill, Comp 256 / K.H. Shafique, UCSF, CAP5415 / S. Narasimhan, CMU / Bahadir
K. Gunturk, EE 7730 / Bradski&Thrun, Stanford CS223
Materials • Gary Bradski & Sebastian Thrun, Stanford CS223
http://robots.stanford.edu/cs223b/index.html • S. Narasimhan, CMU: http://www.cs.cmu.edu/afs/cs/academic/class/15385-
s06/lectures/ppts/lec-16.ppt • M. Pollefeys, ETH Zurich/UNC Chapel Hill:
http://www.cs.unc.edu/Research/vision/comp256/vision10.ppt • K.H. Shafique, UCSF: http://www.cs.ucf.edu/courses/cap6411/cap5415/
– Lecture 18 (March 25, 2003), Slides: PDF/ PPT • Jepson, Toronto:
http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf • Original paper Horn&Schunck 1981:
http://www.csd.uwo.ca/faculty/beau/CS9645/PAPERS/Horn-Schunck.pdf • MIT AI Memo Horn& Schunck 1980:
http://people.csail.mit.edu/bkph/AIM/AIM-572.pdf • Bahadir K. Gunturk, EE 7730 Image Analysis II • Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P.
Anandan, M. Black, K. Toyama
Tracking – Rigid Objects
What is Optical Flow (OF)?
}{),( iptI
1p
2p
3p
4p
)1( +tI
What is Optical Flow (OF)?
}{),( iptI
1p
2p
3p
4p
)1( +tI
What is Optical Flow (OF)?
Optical Flow
}{),( iptI
1p
2p
3p
4p
)1( +tI
What is Optical Flow (OF)?
Optical Flow
}{),( iptI
1p
2p
3p
4p
1vr2vr
3vr
4vr
}{ ivrVelocity vectors
)1( +tI
What is Optical Flow (OF)?
Optical Flow
}{),( iptI
1p
2p
3p
4p
1vr2vr
3vr
4vr
}{ ivrVelocity vectors
Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer
to 2D displacement of pixel patches on the image plane.
)1( +tI
What is Optical Flow (OF)?
Optical Flow
}{),( iptI
1p
2p
3p
4p
1vr2vr
3vr
4vr
}{ ivrVelocity vectors
Common assumption: The appearance of the image patches do not change (brightness constancy)
)1,(),( ++= tvpItpI iiir
Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer
to 2D displacement of pixel patches on the image plane.
)1( +tI
What is Optical Flow (OF)?
Optical Flow
}{),( iptI
1p
2p
3p
4p
1vr2vr
3vr
4vr
}{ ivrVelocity vectors
Common assumption: The appearance of the image patches do not change (brightness constancy)
)1,(),( ++= tvpItpI iiir
Note: more elaborate tracking models can be adopted if more frames are process all at once
Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer
to 2D displacement of pixel patches on the image plane.
Optical Flow
• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow
Optical Flow and Motion
• We are interested in finding the movement of scene objects from time-varying images (videos).
• Lots of uses – Motion detection – Track objects – Correct for camera jitter (stabilization) – Align images (mosaics) – 3D shape reconstruction – Special effects – Games: http://www.youtube.com/watch?v=JlLkkom6tWw
– User Interfaces: http://www.youtube.com/watch?v=Q3gT52sHDI4
– Video compression
13
Optical Flow: Where do pixels move to?
Related to: Optical flow
Where do pixels move?
Related to: Optical flow
Where do pixels move?
Tracking – Non-rigid Objects
(Comaniciu et al, Siemens)
Tracking – Non-rigid Objects
Optical Flow: Correspondence
Basic question: Which Pixel went where?
Optical Flow is NOT 3D motion field
http://of-eval.sourceforge.net/
Optical flow: Pixel motion field as observed in image.
Structure from Motion?
• Known: optical flow (instantaneous velocity)
• Motion of camera / object?
Optical Flow is NOT 3D motion field
http://en.wikipedia.org/wiki/File:Opticfloweg.png
22
Definition of optical flow
OPTICAL FLOW = apparent motion of brightness patterns
Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image
è
23
Optical Flow
• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow
Start with an Equation: Brightness Constancy
Point moves (small), but its brightness remains constant:
𝐼𝑡𝑡(𝑥, 𝑦) = 𝐼𝑡𝑡(𝑥 + 𝑢, 𝑦 + 𝑣)
𝐼 = 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 → 𝑑𝐼𝑑𝑐
= 0
𝐼𝑡 𝐼𝑡
Time: t Time: t + dt
Mathematical formulation
I (x(t),y(t),t) = brightness at (x,y) at time t
Optical flow constraint equation (chain rule):
0
=¶¶
+¶¶
+¶¶
=tI
dtdy
yI
dtdx
xI
dtdI
),,(),,( tyxItttdtdyyt
dtdxxI =+++ ddd
Brightness constancy assumption (shift of location but brightness stays same):
The aperture problem
0 =++ tyx IvIuI
, =dtdxu
dtdyv =
, ¶¶
=yII x ,
¶¶
=yII y t
IIt ¶¶
=
Horn and Schunck optical flow equation
The aperture problem
0 =++ tyx IvIuI
1 equation in 2 unknowns
, =dtdxu
dtdyv =
, ¶¶
=yII x ,
¶¶
=yII y t
IIt ¶¶
=
Horn and Schunck optical flow equation
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf ++=º
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf ++=º { 0)(
=¶
¶txf
Because no change in brightness with time
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf ++=º
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
{ 0)(
=¶
¶txf
Because no change in brightness with time
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf ++=º
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
Ix v It
{ 0)(
=¶
¶txf
Because no change in brightness with time
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Optical Flow: 1D Case Brightness Constancy Assumption:
)),(()),(()( dttdttxIttxItf ++=º
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
Ix v It
x
t
IIv -=Þ
{ 0)(
=¶
¶txf
Because no change in brightness with time
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
33
Tracking in the 1D case:
x
),( txI
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
34
Tracking in the 1D case:
x
),( txI )1,( +txI
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
35
Tracking in the 1D case:
x
),( txI )1,( +txI
p
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
36
vr ?
Tracking in the 1D case:
x
),( txI )1,( +txI
p
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
Spatial derivative
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
Spatial derivative
Temporal derivative tI
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
Spatial derivative
Temporal derivative tI
Tracking in the 1D case:
x
),( txI )1,( +txI
p
vr
xI
Spatial derivative
Temporal derivative tI
Tracking in the 1D case:
x
),( txI )1,( +txI
p
vr
xI
Spatial derivative
Temporal derivative tI
Tracking in the 1D case:
x
),( txI )1,( +txI
p
tx x
II¶¶
=px
t tII
=¶¶
=x
t
IIv -»
rAssumptions: • Brightness constancy • Small motion
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
tI
Iterating helps refining the velocity vector
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
tI
Temporal derivative at 2nd iteration
Iterating helps refining the velocity vector
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
tI
Temporal derivative at 2nd iteration
Iterating helps refining the velocity vector
Can keep the same estimate for spatial derivative
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
tI
Temporal derivative at 2nd iteration
Iterating helps refining the velocity vector
Can keep the same estimate for spatial derivative
x
tprevious I
Ivv -¬rr
Tracking in the 1D case:
x
),( txI )1,( +txI
p
xI
tI
Temporal derivative at 2nd iteration
Iterating helps refining the velocity vector
Can keep the same estimate for spatial derivative
x
tprevious I
Ivv -¬rr
Converges in about 5 iterations
From 1D to 2D tracking
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
1D:
From 1D to 2D tracking
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
1D:
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
+÷øö
çèæ
¶¶
¶¶
txtt tI
ty
yI
tx
xI
2D:
From 1D to 2D tracking
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
1D:
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
+÷øö
çèæ
¶¶
¶¶
txtt tI
ty
yI
tx
xI
2D:
0)(
=¶¶
+¶¶
+¶¶
txtt tIv
yIu
xI
From 1D to 2D tracking
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
txt tI
tx
xI
1D:
0)(
=¶¶
+÷øö
çèæ
¶¶
¶¶
+÷øö
çèæ
¶¶
¶¶
txtt tI
ty
yI
tx
xI
2D:
0)(
=¶¶
+¶¶
+¶¶
txtt tIv
yIu
xI
Shoot! One equation, two velocity (u,v) unknowns…
Optical Flow vs. Motion: Aperture Problem
Barber shop pole: http://www.youtube.com/watch?v=VmqQs613SbE
Optical Flow vs. Motion: Aperture Problem
Barber shop pole: http://www.youtube.com/watch?v=VmqQs613SbE
54
Optical Flow
• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
How does this show up visually? Known as the “Aperture Problem”
Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
Aperture Problem Exposed
Motion along just an edge is ambiguous Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
78
Aperture Problem in Real Life
79
Normal Flow
* Slide from Michael Black, CS143 2003
We get at most “Normal Flow” – with one point we can only detect movement perpendicular to the brightness gradient. Solution is to take a patch of pixels Around the pixel of interest.
Aperture Problem
Aperture Problem
82
Aperture Problem and Normal Flow
Computing True Flow
• Horn & Schunck • Schunck • Lukas and Kanade
Possible Solution: Neighbors
Two adjacent pixels which are part of the same rigid object: • we can calculate normal flows vn1 and vn2
• Two OF equations for 2 parameters of flow: �̅� = 𝑣𝑢
𝛻𝐼𝑡. �̅� − 𝐼𝑡𝑡 = 0 𝛻𝐼𝑡. �̅� − 𝐼𝑡𝑡 = 0
Considering Neighbor Pixels
Considering Neighbor Pixels
Jepson, Toronto: http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
Cluster center provides velocity vector common for all pixels in patch.
88
Optical Flow
• Brightness Constancy • The Aperture problem • Regularization: Horn & Schunck • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow
89
Horn & Schunck algorithm
è
90
Horn & Schunck algorithm
Additional smoothness constraint (usually motion field varies smoothly in the image → penalize departure from smoothness) :
,))()(( 2222 dxdyvvuue yxyxs +++= òòOF constraint equation term (formulate error in optical flow constraint) : ,)( 2 dxdyIvIuIe tyxc ++= òòminimize es+lec
è
91
Horn & Schunck algorithm
è
Variational calculus: Pair of second order differential equations that can be solved iteratively.
92
Horn & Schunck algorithm
è
93
Horn & Schunck
The Euler-Lagrange equations :
0
0
=¶¶
-¶¶
-
=¶¶
-¶¶
-
yx
yx
vvv
uuu
Fy
Fx
F
Fy
Fx
F
In our case ,
,)()()( 22222tyxyxyx IvIuIvvuuF ++++++= l
so the Euler-Lagrange equations are
,)(,)(
ytyx
xtyx
IIvIuIvIIvIuIu
++=D
++=D
l
l
2
2
2
2
yx ¶¶
+¶¶
=D is the Laplacian operator
è
94
Horn & Schunck
Remarks :
1. Coupled PDEs solved using iterative methods and finite differences
2. More than two frames allow a better estimation of It
3. Information spreads from corner-type patterns
è
,)(
,)(
ytyx
xtyx
IIvIuIvtv
IIvIuIutu
++-D=¶¶
++-D=¶¶
l
l
Discrete Optical Flow Algorithm Consider image pixel
• Departure from Smoothness Constraint:
•Error in Optical Flow constraint equation: • We seek the set that minimize: åå +=
i jijij cse )( l
])()(
)()[(41
2,1,
2,,1
2,1,
2,,1
jijijiji
jijijijiij
vvvv
uuuus
-+-
+-+-=
++
++
2)( tij
ijyij
ijxij
ij EvEuEc ++=}{&}{ ijij vu
NOTE: show up in more than one term
}{&}{ ijij vu
),( ji
Discrete Optical Flow Algorithm • Differentiating w.r.t and setting to zero:
• are averages of around pixel
e0)(2)(2 =+++-=
¶¶ kl
xkltkl
klykl
klxklkl
kl
EEvEuEuuue l
0)(2)(2 =+++-=¶¶ kl
ykltkl
klykl
klxklkl
kl
EEvEuEvvve l
klkl uv &
klkl uv & ),( lk),( vu
klxkl
yklx
klt
nkl
kly
nkl
klxn
klnkl E
EEEvEuE
uu])()[(1 22
1
++
++-=+
l
klykl
yklx
klt
nkl
kly
nkl
klxn
klnkl E
EEEvEuE
vv])()[(1 22
1
++
++-=+
l
Update Rule:
Horn-Schunck Algorithm : Discrete Case
• Derivatives (and error functionals) are approximated by difference operators
• Leads to an iterative solution:
yn
ijnij
xn
ijnij
Evv
Euu
a
a
-=
-=+
+
1
1
)(1 22yx
tn
ijyn
ijx
EEEvEuE
++
++=
la
neighbors of valuesof average theis ,vu
Intuition of the Iterative Scheme
u
v (Ex,Ey)
Constraint line
(u,v)
),( vu
The new value of (u,v) at a point is equal to the average of surrounding values minus an adjustment in the direction of the brightness gradient
Horn - Schunck Algorithm
Example
http://of-eval.sourceforge.net/
Results
Results
Optical Flow Result
10
Horn & Schunck, remarks
1. Errors at boundaries
2. Example of regularisation (selection principle for the solution of illposed problems)
è
Results of an enhanced system
Results http://www-student.informatik.uni-bonn.de/~gerdes/OpticalFlow/index.html
Results http://www.cs.utexas.edu/users/jmugan/GraphicsProject/OpticalFlow/
10
Optical Flow
• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow