Optical Flow I

Guido Gerig CS 6320, Spring 2013

(credits: Marc Pollefeys UNC Chapel Hill, Comp 256 / K.H. Shafique, UCSF, CAP5415 / S. Narasimhan, CMU / Bahadir

K. Gunturk, EE 7730 / Bradski&Thrun, Stanford CS223

Materials • Gary Bradski & Sebastian Thrun, Stanford CS223

http://robots.stanford.edu/cs223b/index.html • S. Narasimhan, CMU: http://www.cs.cmu.edu/afs/cs/academic/class/15385-

s06/lectures/ppts/lec-16.ppt • M. Pollefeys, ETH Zurich/UNC Chapel Hill:

http://www.cs.unc.edu/Research/vision/comp256/vision10.ppt • K.H. Shafique, UCSF: http://www.cs.ucf.edu/courses/cap6411/cap5415/

– Lecture 18 (March 25, 2003), Slides: PDF/ PPT • Jepson, Toronto:

http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf • Original paper Horn&Schunck 1981:

http://www.csd.uwo.ca/faculty/beau/CS9645/PAPERS/Horn-Schunck.pdf • MIT AI Memo Horn& Schunck 1980:

http://people.csail.mit.edu/bkph/AIM/AIM-572.pdf • Bahadir K. Gunturk, EE 7730 Image Analysis II • Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P.

Anandan, M. Black, K. Toyama

Tracking – Rigid Objects

What is Optical Flow (OF)?

}{),( iptI

1p

2p

3p

4p

)1( +tI

What is Optical Flow (OF)?

}{),( iptI

1p

2p

3p

4p

)1( +tI

What is Optical Flow (OF)?

Optical Flow

}{),( iptI

1p

2p

3p

4p

)1( +tI

What is Optical Flow (OF)?

Optical Flow

}{),( iptI

1p

2p

3p

4p

1vr2vr

3vr

4vr

}{ ivrVelocity vectors

)1( +tI

What is Optical Flow (OF)?

Optical Flow

}{),( iptI

1p

2p

3p

4p

1vr2vr

3vr

4vr

}{ ivrVelocity vectors

Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer

to 2D displacement of pixel patches on the image plane.

)1( +tI

What is Optical Flow (OF)?

Optical Flow

}{),( iptI

1p

2p

3p

4p

1vr2vr

3vr

4vr

}{ ivrVelocity vectors

Common assumption: The appearance of the image patches do not change (brightness constancy)

)1,(),( ++= tvpItpI iiir

Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer

to 2D displacement of pixel patches on the image plane.

)1( +tI

What is Optical Flow (OF)?

Optical Flow

}{),( iptI

1p

2p

3p

4p

1vr2vr

3vr

4vr

}{ ivrVelocity vectors

Common assumption: The appearance of the image patches do not change (brightness constancy)

)1,(),( ++= tvpItpI iiir

Note: more elaborate tracking models can be adopted if more frames are process all at once

Optical flow is the relation of the motion field: • the 2D projection of the physical movement of points relative to the observer

to 2D displacement of pixel patches on the image plane.

Optical Flow

• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow

Optical Flow and Motion

• We are interested in finding the movement of scene objects from time-varying images (videos).

• Lots of uses – Motion detection – Track objects – Correct for camera jitter (stabilization) – Align images (mosaics) – 3D shape reconstruction – Special effects – Games: http://www.youtube.com/watch?v=JlLkkom6tWw

– User Interfaces: http://www.youtube.com/watch?v=Q3gT52sHDI4

– Video compression

13

Optical Flow: Where do pixels move to?

Related to: Optical flow

Where do pixels move?

Related to: Optical flow

Where do pixels move?

Tracking – Non-rigid Objects

(Comaniciu et al, Siemens)

Tracking – Non-rigid Objects

Optical Flow: Correspondence

Basic question: Which Pixel went where?

Optical Flow is NOT 3D motion field

http://of-eval.sourceforge.net/

Optical flow: Pixel motion field as observed in image.

Structure from Motion?

• Known: optical flow (instantaneous velocity)

• Motion of camera / object?

Optical Flow is NOT 3D motion field

http://en.wikipedia.org/wiki/File:Opticfloweg.png

22

Definition of optical flow

OPTICAL FLOW = apparent motion of brightness patterns

Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image

è

23

Optical Flow

• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow

Start with an Equation: Brightness Constancy

Point moves (small), but its brightness remains constant:

𝐼𝑡𝑡(𝑥, 𝑦) = 𝐼𝑡𝑡(𝑥 + 𝑢, 𝑦 + 𝑣)

𝐼 = 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 → 𝑑𝐼𝑑𝑐

= 0

𝐼𝑡 𝐼𝑡

Time: t Time: t + dt

Mathematical formulation

I (x(t),y(t),t) = brightness at (x,y) at time t

Optical flow constraint equation (chain rule):

0

=¶¶

+¶¶

+¶¶

=tI

dtdy

yI

dtdx

xI

dtdI

),,(),,( tyxItttdtdyyt

dtdxxI =+++ ddd

Brightness constancy assumption (shift of location but brightness stays same):

The aperture problem

0 =++ tyx IvIuI

, =dtdxu

dtdyv =

, ¶¶

=yII x ,

¶¶

=yII y t

IIt ¶¶

=

Horn and Schunck optical flow equation

The aperture problem

0 =++ tyx IvIuI

1 equation in 2 unknowns

, =dtdxu

dtdyv =

, ¶¶

=yII x ,

¶¶

=yII y t

IIt ¶¶

=

Horn and Schunck optical flow equation

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=º

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=º { 0)(

=¶

¶txf

Because no change in brightness with time

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=º

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

{ 0)(

=¶

¶txf

Because no change in brightness with time

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=º

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

Ix v It

{ 0)(

=¶

¶txf

Because no change in brightness with time

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Optical Flow: 1D Case Brightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=º

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

Ix v It

x

t

IIv -=Þ

{ 0)(

=¶

¶txf

Because no change in brightness with time

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

33

Tracking in the 1D case:

x

),( txI

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

34

Tracking in the 1D case:

x

),( txI )1,( +txI

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

35

Tracking in the 1D case:

x

),( txI )1,( +txI

p

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

36

vr ?

Tracking in the 1D case:

x

),( txI )1,( +txI

p

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

Spatial derivative

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

Spatial derivative

Temporal derivative tI

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

Spatial derivative

Temporal derivative tI

Tracking in the 1D case:

x

),( txI )1,( +txI

p

vr

xI

Spatial derivative

Temporal derivative tI

Tracking in the 1D case:

x

),( txI )1,( +txI

p

vr

xI

Spatial derivative

Temporal derivative tI

Tracking in the 1D case:

x

),( txI )1,( +txI

p

tx x

II¶¶

=px

t tII

=¶¶

=x

t

IIv -»

rAssumptions: • Brightness constancy • Small motion

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

tI

Iterating helps refining the velocity vector

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

tI

Temporal derivative at 2nd iteration

Iterating helps refining the velocity vector

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

tI

Temporal derivative at 2nd iteration

Iterating helps refining the velocity vector

Can keep the same estimate for spatial derivative

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

tI

Temporal derivative at 2nd iteration

Iterating helps refining the velocity vector

Can keep the same estimate for spatial derivative

x

tprevious I

Ivv -¬rr

Tracking in the 1D case:

x

),( txI )1,( +txI

p

xI

tI

Temporal derivative at 2nd iteration

Iterating helps refining the velocity vector

Can keep the same estimate for spatial derivative

x

tprevious I

Ivv -¬rr

Converges in about 5 iterations

From 1D to 2D tracking

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

1D:

From 1D to 2D tracking

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

1D:

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

+÷øö

çèæ

¶¶

¶¶

txtt tI

ty

yI

tx

xI

2D:

From 1D to 2D tracking

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

1D:

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

+÷øö

çèæ

¶¶

¶¶

txtt tI

ty

yI

tx

xI

2D:

0)(

=¶¶

+¶¶

+¶¶

txtt tIv

yIu

xI

From 1D to 2D tracking

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

txt tI

tx

xI

1D:

0)(

=¶¶

+÷øö

çèæ

¶¶

¶¶

+÷øö

çèæ

¶¶

¶¶

txtt tI

ty

yI

tx

xI

2D:

0)(

=¶¶

+¶¶

+¶¶

txtt tIv

yIu

xI

Shoot! One equation, two velocity (u,v) unknowns…

Optical Flow vs. Motion: Aperture Problem

Barber shop pole: http://www.youtube.com/watch?v=VmqQs613SbE

Optical Flow vs. Motion: Aperture Problem

Barber shop pole: http://www.youtube.com/watch?v=VmqQs613SbE

54

Optical Flow

• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

How does this show up visually? Known as the “Aperture Problem”

Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

Aperture Problem Exposed

Motion along just an edge is ambiguous Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html

78

Aperture Problem in Real Life

79

Normal Flow

* Slide from Michael Black, CS143 2003

We get at most “Normal Flow” – with one point we can only detect movement perpendicular to the brightness gradient. Solution is to take a patch of pixels Around the pixel of interest.

Aperture Problem

Aperture Problem

82

Aperture Problem and Normal Flow

Computing True Flow

• Horn & Schunck • Schunck • Lukas and Kanade

Possible Solution: Neighbors

Two adjacent pixels which are part of the same rigid object: • we can calculate normal flows vn1 and vn2

• Two OF equations for 2 parameters of flow: �̅� = 𝑣𝑢

𝛻𝐼𝑡. �̅� − 𝐼𝑡𝑡 = 0 𝛻𝐼𝑡. �̅� − 𝐼𝑡𝑡 = 0

Considering Neighbor Pixels

Considering Neighbor Pixels

Jepson, Toronto: http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf

Cluster center provides velocity vector common for all pixels in patch.

88

Optical Flow

• Brightness Constancy • The Aperture problem • Regularization: Horn & Schunck • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow

89

Horn & Schunck algorithm

è

90

Horn & Schunck algorithm

Additional smoothness constraint (usually motion field varies smoothly in the image → penalize departure from smoothness) :

,))()(( 2222 dxdyvvuue yxyxs +++= òòOF constraint equation term (formulate error in optical flow constraint) : ,)( 2 dxdyIvIuIe tyxc ++= òòminimize es+lec

è

91

Horn & Schunck algorithm

è

Variational calculus: Pair of second order differential equations that can be solved iteratively.

92

Horn & Schunck algorithm

è

93

Horn & Schunck

The Euler-Lagrange equations :

0

0

=¶¶

-¶¶

-

=¶¶

-¶¶

-

yx

yx

vvv

uuu

Fy

Fx

F

Fy

Fx

F

In our case ,

,)()()( 22222tyxyxyx IvIuIvvuuF ++++++= l

so the Euler-Lagrange equations are

,)(,)(

ytyx

xtyx

IIvIuIvIIvIuIu

++=D

++=D

l

l

2

2

2

2

yx ¶¶

+¶¶

=D is the Laplacian operator

è

94

Horn & Schunck

Remarks :

1. Coupled PDEs solved using iterative methods and finite differences

2. More than two frames allow a better estimation of It

3. Information spreads from corner-type patterns

è

,)(

,)(

ytyx

xtyx

IIvIuIvtv

IIvIuIutu

++-D=¶¶

++-D=¶¶

l

l

Discrete Optical Flow Algorithm Consider image pixel

• Departure from Smoothness Constraint:

•Error in Optical Flow constraint equation: • We seek the set that minimize: åå +=

i jijij cse )( l

])()(

)()[(41

2,1,

2,,1

2,1,

2,,1

jijijiji

jijijijiij

vvvv

uuuus

-+-

+-+-=

++

++

2)( tij

ijyij

ijxij

ij EvEuEc ++=}{&}{ ijij vu

NOTE: show up in more than one term

}{&}{ ijij vu

),( ji

Discrete Optical Flow Algorithm • Differentiating w.r.t and setting to zero:

• are averages of around pixel

e0)(2)(2 =+++-=

¶¶ kl

xkltkl

klykl

klxklkl

kl

EEvEuEuuue l

0)(2)(2 =+++-=¶¶ kl

ykltkl

klykl

klxklkl

kl

EEvEuEvvve l

klkl uv &

klkl uv & ),( lk),( vu

klxkl

yklx

klt

nkl

kly

nkl

klxn

klnkl E

EEEvEuE

uu])()[(1 22

1

++

++-=+

l

klykl

yklx

klt

nkl

kly

nkl

klxn

klnkl E

EEEvEuE

vv])()[(1 22

1

++

++-=+

l

Update Rule:

Horn-Schunck Algorithm : Discrete Case

• Derivatives (and error functionals) are approximated by difference operators

• Leads to an iterative solution:

yn

ijnij

xn

ijnij

Evv

Euu

a

a

-=

-=+

+

1

1

)(1 22yx

tn

ijyn

ijx

EEEvEuE

++

++=

la

neighbors of valuesof average theis ,vu

Intuition of the Iterative Scheme

u

v (Ex,Ey)

Constraint line

(u,v)

),( vu

The new value of (u,v) at a point is equal to the average of surrounding values minus an adjustment in the direction of the brightness gradient

Horn - Schunck Algorithm

Example

http://of-eval.sourceforge.net/

Results

Results

Optical Flow Result

10

Horn & Schunck, remarks

1. Errors at boundaries

2. Example of regularisation (selection principle for the solution of illposed problems)

è

Results of an enhanced system

Results http://www-student.informatik.uni-bonn.de/~gerdes/OpticalFlow/index.html

Results http://www.cs.utexas.edu/users/jmugan/GraphicsProject/OpticalFlow/

10

Optical Flow

• Brightness Constancy • The Aperture problem • Regularization • Lucas-Kanade • Coarse-to-fine • Parametric motion models • Direct depth • SSD tracking • Robust flow • Bayesian flow