Lecture 20: Motion estimationahowens/eecs504/w20/slides/...Most slides from S. Lazebnik, which are...

1

Lecture 20: Motion estimation

Most slides from S. Lazebnik, which are based on other slides from S. Seitz, R. Szeliski, M. Pollefeys

2

Announcements• PS9 and PS10 deadlines extended to April 21

Optical flow

3 Source: S. Lazebnik

What moved where?

Motion is a powerful perceptual cue



5

G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik


6

G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik

Motion field


Estimating optical flow

8

• Common assumptions

• Brightness constancy: projection of the same point looks the

same in every frame

• Small motion: points do not move very far

• Spatial coherence: points move like their neighbors

I(x,y,t–1) I(x,y,t)

• Given two subsequent frames, estimate the motion field u(x,y) and v(x,y) between them

Source: S. Lazebnik

The brightness constancy constraint


Simple loss function [Lucas & Kanade 1981]. Find flow that minimizes:

L(u, v) =X

x,y

[I(x, y, t� 1)� I(x+ u(x, y), y + v(x, y), t)]2AAACOXicbVBLSwMxEM76tr6qHr0MFqHFWnZV0IsgelHwUMGq0K4lm6YazD5IJsWy9G958V94E7x4UMSrf8BsreBrYMj3fTPDZL4gkUKj6z44Q8Mjo2PjE5O5qemZ2bn8/MKpjo1ivMZiGavzgGouRcRrKFDy80RxGgaSnwXX+1n9rMOVFnF0gt2E+yG9jERbMIpWauarjZDiFaMyPeoVTRk6JdiBhjZhM70pQ7dXPyxmbxlwzSvBGlgKq2D6YsmmJZ0vgiX/Yr2ZL7gVtx/wF3gDUCCDqDbz941WzEzII2SSal333AT9lCoUTPJermE0Tyi7ppe8bmFEQ679tH95D1as0oJ2rGxGCH31+0RKQ627YWA7szv171om/lerG2xv+6mIEoM8Yp+L2kYCxpDZCC2hOEPZtYAyJexfgV1RRRlas3PWBO/3yX/B6XrF26i4x5uF3b2BHRNkiSyTIvHIFtklB6RKaoSRW/JInsmLc+c8Oa/O22frkDOYWSQ/wnn/ALA2pq8=

Source: S. Lazebnik

),()1,,( ),,(),( tyxyx vyuxItyxI ++=−

),(),(),,()1,,( yxvIyxuItyxItyxI yx ++≈−

Linearizing the right side using Taylor expansion:


10


0≈++ tyx IvIuIHence,

Brightness Constancy Equation:

Derivative in y direction

Source: S. Lazebnik


11

• How many equations and unknowns per pixel?

• One equation, two unknowns

• What does this constraint mean?

• The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown!

0=++ tyx IvIuI

0),( =+⋅∇ tIvuI

0)','( =⋅∇ vuI

edge

(u,v)

(u’,v’)

gradient

(u+u’,v+v’)

If (u, v) satisfies the equation,  so does (u+u’, v+v’) if

Source: S. Lazebnik

The aperture problem

12

Perceived motion

Source: S. Lazebnik

The aperture problem

13

Actual motion

Source: S. Lazebnik

The barber pole illusion

14http://en.wikipedia.org/wiki/Barberpole_illusion

Source: S. Lazebnik

http://en.wikipedia.org/wiki/Barberpole_illusion

The barber pole illusion

15http://en.wikipedia.org/wiki/Barberpole_illusion

Source: S. Lazebnik

http://en.wikipedia.org/wiki/Barberpole_illusion

• How to get more equations for a pixel?

• Spatial coherence constraint: assume the pixel’s

neighbors have the same (u,v)

• E.g., if we use a 5x5 window, that gives us 25 equations per pixel

Solving the aperture problem

16

0)(],[)( =+⋅∇ iti IvuI xx

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−=⎥⎦

⎤⎢⎣

⎡

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

)(

)()(

)()(

)()()()(

2

1

22

11

nt

t

t

nynx

yx

yx

I

II

vu

II

IIII

x

xx

xx

xxxx

!!!… … …

[Lucas & Kanade 1981] Source: S. Lazebnik

Lucas-Kanade flow

17

When is this system solvable?

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−=⎥⎦

⎤⎢⎣

⎡

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

)(

)()(

)()(

)()()()(

2

1

22

11

nt

t

t

nynx

yx

yx

I

II

vu

II

IIII

x

xx

xx

xxxx

!!!

[Lucas & Kanade 1981]

Least squares problem:

… … …Source: S. Lazebnik

Lucas-Kanade optical flow

18

(summations are over all pixels in the window)

• Solution given by

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−=⎥⎦

⎤⎢⎣

⎡

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

)(

)()(

)()(

)()()()(

2

1

22

11

nt

t

t

nynx

yx

yx

I

II

vu

II

IIII

x

xx

xx

xxxx

!!! 1122 ×××=nnbdA

bAA)dA TT =(

⎥⎥⎦

⎤

⎢⎢⎣

⎡−=⎥

⎦

⎤⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡

∑∑

∑∑∑∑

ty

tx

yyyx

yxxx

IIII

vu

IIIIIIII

M = ATA is the “second moment” matrix

[Lucas & Kanade 1981]

… … …Source: S. Lazebnik

Analyzing the second moment matrix

19 λ1

λ2

“Corner” λ1 and λ2 are large,   λ1 ~ λ2

λ1 and λ2 are small “Edge”  λ1 >> λ2

“Edge”  λ2 >> λ1

“Flat” region

• Estimation of optical flow is well-conditioned precisely for regions with high “cornerness”:

Eigenvalues

Source: S. Lazebnik

Conditions for solvability


Lucas-Kanade flow example

22

Input frames Output

Source: MATLAB Central File Exchange Source: S. Lazebnik

https://www.mathworks.com/matlabcentral/fileexchange/48744-lucas-kanade-tutorial-example-1

Fixing the errors in Lucas-Kanade

23

• The motion is large (larger than a pixel)

• Iterative refinement

• Multi-resolution (coarse-to-fine) estimation 

• Local ambiguity

• Smooth using graphical model refinement

Source: S. Lazebnik

Large motions

24 Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Initialize flow: u0(x, y) = v0(x, y) = 0 For each iteration i:

1. Linearize around current solution. Find an update for problem:   by solving linear least squares problem for each pixel:

2. Apply updates: and

Idea #1: iterative estimationGoal: minimize matching error

Iterative algorithm:

L(u, v) =X

x,y

x+NX

x0=x�N

y+NX

y0=y�N[I(x0, y0, t� 1)� I(x0 + u(x, y), y0 + v(x, y), t)]2

AAACcHicbZFda9swFIZlt1vb7CttbgrbmLYwmqxtsNtCd1MI3c0GY2SwJIXEDbKitKLyB9JRsDG+7v/rXX9Eb/oLJjsebMkOCL96n3Ns6bUfC67Ace4te239ydONza3as+cvXr6qb+8MVKQlZX0aiUhe+EQxwUPWBw6CXcSSkcAXbOjffCn4cM6k4lH4C9KYeQG5CvmMUwLGmtRvxwGBa0pE9j1v6QM8b+MzPFY6mGTJAU7zSu4ZN8GH+Ed+mSV43zyrprQgaUXSkoy+tZI9M2sWHLptwwrDIN0qXtkukNnN/+yg7V0eTepNp+OUhVeFW4kmqqo3qd+NpxHVAQuBCqLUyHVi8DIigVPB8tpYKxYTekOu2MjIkARMeVkZWI4/GmeKZ5E0KwRcun9PZCRQKg1801nEo5ZZYf6PjTTMPnsZD2MNLKSLD820wBDhIn085ZJREKkRhEpuzorpNZGEgvlHNROCu3zlVTE46rjHHefnSbN7XsWxiV6jD6iFXHSKuugr6qE+oujBalhvrLfWo71rv7PfL1ptq5ppoH/K/vQbOZqyVw==

where the window width/height is 2N+1

�u,�v = min�u,�v

L(ui +�u, vi +�v)AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=

argminAAAB9HicbVDLSgMxFL3js9ZX1aWbYBFclRkVdFl047KCfUA7lEyaaUPzGJNMsQz9DjcuFHHrx7jzb0zbWWjrgQuHc+5N7j1Rwpmxvv/trayurW9sFraK2zu7e/ulg8OGUakmtE4UV7oVYUM5k7RumeW0lWiKRcRpMxreTv3miGrDlHyw44SGAvclixnB1klhR0TqKcO6L5icdEtlv+LPgJZJkJMy5Kh1S1+dniKpoNISjo1pB35iQ/ecZYTTSbGTGppgMsR92nZUYkFNmM2WnqBTp/RQrLQradFM/T2RYWHMWESuU2A7MIveVPzPa6c2vg4zJpPUUknmH8UpR1ahaQKoxzQllo8dwUQztysiA6wxsS6nogshWDx5mTTOK8FFxb+/LFdv8jgKcAwncAYBXEEV7qAGdSDwCM/wCm/eyHvx3r2PeeuKl88cwR94nz9vZ5KG�u,�v = min

�u,�vL(ui +�u, vi +�v)

AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=

Ad = bAAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9nVgl6EqhePFewHtEvJZrNtbDZZkqxQlv4HLx4U8er/8ea/MW33oK0PBh7vzTAzL0g408Z1v53Cyura+kZxs7S1vbO7V94/aGmZKkKbRHKpOgHWlDNBm4YZTjuJojgOOG0Ho9up336iSjMpHsw4oX6MB4JFjGBjpdZ1iK5Q0C9X3Ko7A1omXk4qkKPRL3/1QknSmApDONa667mJ8TOsDCOcTkq9VNMEkxEe0K6lAsdU+9ns2gk6sUqIIqlsCYNm6u+JDMdaj+PAdsbYDPWiNxX/87qpiS79jIkkNVSQ+aIo5chINH0dhUxRYvjYEkwUs7ciMsQKE2MDKtkQvMWXl0nrrOqdV937WqV+k8dRhCM4hlPw4ALqcAcNaAKBR3iGV3hzpPPivDsf89aCk88cwh84nz85l448

ui+1 = ui +�u, vi+1 = vi +�vAAACHHicbVDLSsNAFJ34rPUVdelmsAhCpSRW0I1Q1IXLCvYBbQiT6bQdOpmEeQRK6Ie48VfcuFDEjQvBv3HaBqmtBy4czrmXe+8JYkalcpxva2l5ZXVtPbeR39za3tm19/brMtICkxqOWCSaAZKEUU5qiipGmrEgKAwYaQSDm7HfSIiQNOIPahgTL0Q9TrsUI2Uk3y5rP6VFdwSvoPYpLML2LWEKQX0Kk18nmXUS6NsFp+RMABeJm5ECyFD17c92J8I6JFxhhqRsuU6svBQJRTEjo3xbSxIjPEA90jKUo5BIL508N4LHRunAbiRMcQUn6uxEikIph2FgOkOk+nLeG4v/eS2tupdeSnmsFeF4uqirGVQRHCcFO1QQrNjQEIQFNbdC3EcCYWXyzJsQ3PmXF0n9rOSWS879eaFyncWRA4fgCJwAF1yACrgDVVADGDyCZ/AK3qwn68V6tz6mrUtWNnMA/sD6+gEQTp48


Idea #2: Multi-scale estimation

26 Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003

For each iteration i: 1. Linearize around current solution. Find an update for problem:   by solving linear least squares problem for each pixel:

2. Apply updates: and

Idea #2: Multi-scale estimation

�u,�v = min�u,�v

L(ui +�u, vi +�v)AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=

argminAAAB9HicbVDLSgMxFL3js9ZX1aWbYBFclRkVdFl047KCfUA7lEyaaUPzGJNMsQz9DjcuFHHrx7jzb0zbWWjrgQuHc+5N7j1Rwpmxvv/trayurW9sFraK2zu7e/ulg8OGUakmtE4UV7oVYUM5k7RumeW0lWiKRcRpMxreTv3miGrDlHyw44SGAvclixnB1klhR0TqKcO6L5icdEtlv+LPgJZJkJMy5Kh1S1+dniKpoNISjo1pB35iQ/ecZYTTSbGTGppgMsR92nZUYkFNmM2WnqBTp/RQrLQradFM/T2RYWHMWESuU2A7MIveVPzPa6c2vg4zJpPUUknmH8UpR1ahaQKoxzQllo8dwUQztysiA6wxsS6nogshWDx5mTTOK8FFxb+/LFdv8jgKcAwncAYBXEEV7qAGdSDwCM/wCm/eyHvx3r2PeeuKl88cwR94nz9vZ5KG�u,�v = min

�u,�vL(ui +�u, vi +�v)

AAACQ3icbVDNS8MwHE39nPOr6tFLcAgTZbQq6EUY6sGDhwnuA7dR0izbwtK0JGlhlP5vXvwHvPkPePGgiFfBdCs4Nx8EXt57P5LfcwNGpbKsF2NufmFxaTm3kl9dW9/YNLe2a9IPBSZV7DNfNFwkCaOcVBVVjDQCQZDnMlJ3B1epX4+IkNTn92oYkLaHepx2KUZKS4750LomTCEYHsGMRfACtjzKnXjWSrSDVB8jFt8mxdCh8BD+pqLJe3TgmAWrZI0AZ4mdkQLIUHHM51bHx6FHuMIMSdm0rUC1YyQUxYwk+VYoSYDwAPVIU1OOPCLb8aiDBO5rpQO7vtCHKzhSJydi5Ek59FydTFeQ014q/uc1Q9U9b8eUB6EiHI8f6oYMKh+mhcIOFQQrNtQEYUH1XyHuI4Gw0rXndQn29MqzpHZcsk9K1t1poXyZ1ZEDu2APFIENzkAZ3IAKqAIMHsEreAcfxpPxZnwaX+PonJHN7IA/ML5/APHGreo=

Ad = bAAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9nVgl6EqhePFewHtEvJZrNtbDZZkqxQlv4HLx4U8er/8ea/MW33oK0PBh7vzTAzL0g408Z1v53Cyura+kZxs7S1vbO7V94/aGmZKkKbRHKpOgHWlDNBm4YZTjuJojgOOG0Ho9up336iSjMpHsw4oX6MB4JFjGBjpdZ1iK5Q0C9X3Ko7A1omXk4qkKPRL3/1QknSmApDONa667mJ8TOsDCOcTkq9VNMEkxEe0K6lAsdU+9ns2gk6sUqIIqlsCYNm6u+JDMdaj+PAdsbYDPWiNxX/87qpiS79jIkkNVSQ+aIo5chINH0dhUxRYvjYEkwUs7ciMsQKE2MDKtkQvMWXl0nrrOqdV937WqV+k8dRhCM4hlPw4ALqcAcNaAKBR3iGV3hzpPPivDsf89aCk88cwh84nz85l448



Initialize flow: u0(x, y) = v0(x, y) = 0 For each scale s: Initialize flow from previous (coarser) scale

3. Extra trick for smoother flow: apply median filter to ui+1 and vi+1

2x scale

2x flow magnitude

Example

28

Flow visualization

Coarse-to-fine LK with median filtering

Coarse-to-fine LK Input two frames

Source: Ce Liu

Smoothness assumption

X

x,y

[I(x, y, t� 1)� I(u(x), v(y), t)]2 +X

p

X

p02N(u(p)� u(p0))2 + (v(p)� v(p0))2

AAACYHicbVFBT9swFHYCjNLBKOM2Lk+rJhKtVAkgbUe0XeAyMWkFpCZUjuuCheNE9ktHFOVPctthl/0SnDaTGOxJ9vv0vffZfp+TXAqDQfDLcVdW116tdza6rze33mz3dt5emKzQjI9YJjN9lVDDpVB8hAIlv8o1p2ki+WVy97WpX865NiJTP7DMeZzSGyVmglG01KT3MzJFOqnuB1DW4zOvyQPAg9CHAzjzCu/eH8DcK+2Ofnx9CB9hKcjrv2AfIqEgSineMiqrb3UNVpc3B9i07/sLlTdfUvOWmvT6wTBYBLwEYQv6pI3zSe8hmmasSLlCJqkx4zDIMa6oRsEkr7tRYXhO2R294WMLFU25iauFQTV8sMwUZpm2SyEs2KeKiqbGlGliO5s5zPNaQ/6vNi5w9jmuhMoL5IotL5oVEjCDxm2YCs0ZytICyrSwbwV2SzVlaP+ka00In4/8ElwcDsOjYfD9uH/ypbWjQ/bIe+KRkHwiJ+SUnJMRYeS3s+JsOlvOH7fjbrs7y1bXaTW75J9w3z0CWeOtTQ==

Goal: minimize matching error + smoothness [Horn and Schunck 1981]X

x,y

[I(x, y, t� 1)� I(u(x), v(y), t)]2 +X

p

X

p02N(u(p)� u(p0))2 + (v(p)� v(p0))2

AAACYHicbVFBT9swFHYCjNLBKOM2Lk+rJhKtVAkgbUe0XeAyMWkFpCZUjuuCheNE9ktHFOVPctthl/0SnDaTGOxJ9vv0vffZfp+TXAqDQfDLcVdW116tdza6rze33mz3dt5emKzQjI9YJjN9lVDDpVB8hAIlv8o1p2ki+WVy97WpX865NiJTP7DMeZzSGyVmglG01KT3MzJFOqnuB1DW4zOvyQPAg9CHAzjzCu/eH8DcK+2Ofnx9CB9hKcjrv2AfIqEgSineMiqrb3UNVpc3B9i07/sLlTdfUvOWmvT6wTBYBLwEYQv6pI3zSe8hmmasSLlCJqkx4zDIMa6oRsEkr7tRYXhO2R294WMLFU25iauFQTV8sMwUZpm2SyEs2KeKiqbGlGliO5s5zPNaQ/6vNi5w9jmuhMoL5IotL5oVEjCDxm2YCs0ZytICyrSwbwV2SzVlaP+ka00In4/8ElwcDsOjYfD9uH/ypbWjQ/bIe+KRkHwiJ+SUnJMRYeS3s+JsOlvOH7fjbrs7y1bXaTW75J9w3z0CWeOtTQ==

where p and p’ are neighboring pixels

Ed(u, v)AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahgpRdFfRYFMFjBfsh7VKy2bQNTbJLki2Upb/CiwdFvPpzvPlvTNs9aOuDgcd7M8zMC2LOtHHdbye3srq2vpHfLGxt7+zuFfcPGjpKFKF1EvFItQKsKWeS1g0znLZiRbEIOG0Gw9up3xxRpVkkH804pr7Afcl6jGBjpae7blhOztDotFssuRV3BrRMvIyUIEOtW/zqhBFJBJWGcKx123Nj46dYGUY4nRQ6iaYxJkPcp21LJRZU++ns4Ak6sUqIepGyJQ2aqb8nUiy0HovAdgpsBnrRm4r/ee3E9K79lMk4MVSS+aJewpGJ0PR7FDJFieFjSzBRzN6KyAArTIzNqGBD8BZfXiaN84p3UXEfLkvVmyyOPBzBMZTBgyuowj3UoA4EBDzDK7w5ynlx3p2PeWvOyWYO4Q+czx9F1Y9m

match cost Es(u, v)AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahgpRdFfRYFMFjBfsh7VKyabYNTbJLki2Upb/CiwdFvPpzvPlvTNs9aOuDgcd7M8zMC2LOtHHdbye3srq2vpHfLGxt7+zuFfcPGjpKFKF1EvFItQKsKWeS1g0znLZiRbEIOG0Gw9up3xxRpVkkH804pr7AfclCRrCx0tNdV5eTMzQ67RZLbsWdAS0TLyMlyFDrFr86vYgkgkpDONa67bmx8VOsDCOcTgqdRNMYkyHu07alEguq/XR28ASdWKWHwkjZkgbN1N8TKRZaj0VgOwU2A73oTcX/vHZiwms/ZTJODJVkvihMODIRmn6PekxRYvjYEkwUs7ciMsAKE2MzKtgQvMWXl0njvOJdVNyHy1L1JosjD0dwDGXw4AqqcA81qAMBAc/wCm+Ocl6cd+dj3ppzsplD+APn8wdc6491

smoothness

• This is a probabilistic graphical model! • Can solve using least squares (as in Lucas-Kanade) • Or using discrete optimization (e.g. belief propagation)

30

Flow visualization Coarse-to-fine LK

Input two frames Horn-Schunck

Smoothness assumption

Source: Ce Liu

Flow networks

[Sun et al., “PWC-Net”, 2018]

Traditional coarse-to-fine flow PWC-net

Simple application: slow motion• Flow used in lots of familiar places!

• E.g., video compression, denoising, action recognition, …  

• One application: use flow to estimate where pixel will be between frames 

• Synthesize intermediate frames

(x, y) at time t-1

(x + u, y + v) at time t

34

Next lecture: light and color

Lecture 20: Motion estimationahowens/eecs504/w20/slides/...Most slides from S. Lazebnik, which are...

Documents

Transcript of Lecture 20: Motion estimationahowens/eecs504/w20/slides/...Most slides from S. Lazebnik, which are...