Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical...
Transcript of Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical...
![Page 1: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/1.jpg)
Optical flow
16-385 Computer VisionSpring 2020, Lecture 24http://www.cs.cmu.edu/~16385/
![Page 2: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/2.jpg)
• Programming assignment 6 has been posted and is due on April 22th. - Any questions about the homework?- How many of you have looked at/started/finished homework 6?- Shorter than last year, but still requires significant time for training.
• Take-home quiz 10 has been posted and is due on April 19th.
• Prakhar has extra office hours today between 5-6 pm.
• Please see Piazza poll about an extra take-home quiz 12: https://piazza.com/class/k53x5h48my264d?cid=515
• Make-up lecture this Friday, noon-1:30 pm.
Course announcements
![Page 3: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/3.jpg)
• Continue convolutional neural networks.
• Quick intro to vision for video.
• Optical flow.
• Constant flow.
• Horn-Schunck flow.
Overview of today’s lecture
![Page 4: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Kris Kitani (16-385, Spring 2017).
![Page 5: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/5.jpg)
1. Image processing.
2. Geometry-based vision.
3. Physics-based vision.
4. Semantic vision.
5. Dealing with motion.
Course overview
Lectures 14 – 17
See also 16-823: Physics-based Methods in Vision
See also 15-463: Computational Photography
Lectures 7 – 13
See also 16-822: Geometry-based Methods in Vision
Lectures 1 – 6
See also 18-793: Image and Video Processing
We are starting this part now
Lectures 18 – 22
See also 16-824: Vision Learning and Recognition
![Page 6: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/6.jpg)
Computer vision for video
![Page 7: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/7.jpg)
Optical Flow
Constant Flow Horn-Schunck
![Page 8: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/8.jpg)
Optical flow used for feature tracking on a drone
![Page 9: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/9.jpg)
optical flow used for motion estimation in visual odometry
![Page 10: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/10.jpg)
Image Alignment
Lucas-Kanade
(Forward additive)
Baker-Matthews
(Inverse Compositional)
![Page 11: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/11.jpg)
Tracking in Video
KLT Mean shift
Kalman Filtering SLAM
![Page 12: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/12.jpg)
Optical flow
![Page 13: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/13.jpg)
Optical Flow
Problem Definition
Assumptions
Brightness constancy
Small motion
Given two consecutive image frames,
estimate the motion of each pixel
![Page 14: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/14.jpg)
Optical Flow(Problem definition)
Estimate the motion
(flow) between these two
consecutive images
How is this different from estimating a 2D transform?
![Page 15: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/15.jpg)
Key Assumptions(unique to optical flow)
Color Constancy(Brightness constancy for intensity images)
Small Motion(pixels only move a little bit)
Implication: allows for pixel to pixel comparison
(not image features)
Implication: linearization of the brightness
constancy constraint
![Page 16: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/16.jpg)
Approach
Look for nearby pixels with the same color(small motion) (color constancy)
![Page 17: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/17.jpg)
Brightness constancyScene point moving through image sequence
Assumption 1
![Page 18: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/18.jpg)
Brightness constancyScene point moving through image sequence
Assumption 1
![Page 19: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/19.jpg)
Brightness constancy
Assumption:Brightness of the point will remain the same
Scene point moving through image sequence
Assumption 1
![Page 20: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/20.jpg)
Brightness constancy
Assumption:Brightness of the point will remain the same
constant
Scene point moving through image sequence
Assumption 1
![Page 21: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/21.jpg)
Small motion
Assumption 2
![Page 22: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/22.jpg)
Small motion
Assumption 2
![Page 23: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/23.jpg)
Small motion
Optical flow (velocities): Displacement:
Assumption 2
![Page 24: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/24.jpg)
Small motion
Optical flow (velocities): Displacement:
… the brightness between two consecutive image frames
is the same
For a really small space-time step…
Assumption 2
![Page 25: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/25.jpg)
total derivative partial derivative
Equation is not obvious. Where does this come from?
These assumptions yield the …
Brightness Constancy Equation
![Page 26: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/26.jpg)
For small space-time step, brightness of a point is the same
![Page 27: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/27.jpg)
For small space-time step, brightness of a point is the same
Insight:
If the time step is really small,
we can linearize the intensity function
![Page 28: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/28.jpg)
Multivariable Taylor Series Expansion(First order approximation, two variables)
![Page 29: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/29.jpg)
Multivariable Taylor Series Expansion(First order approximation, two variables)
assuming small
motion
![Page 30: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/30.jpg)
assuming small
motion
cancel terms
Multivariable Taylor Series Expansion(First order approximation, two variables)
fixed point
partial derivative
![Page 31: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/31.jpg)
assuming small
motion
cancel terms
Multivariable Taylor Series Expansion(First order approximation, two variables)
![Page 32: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/32.jpg)
assuming small
motion
divide by
take limit
Multivariable Taylor Series Expansion(First order approximation, two variables)
![Page 33: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/33.jpg)
assuming small
motion
divide by
take limit
Multivariable Taylor Series Expansion(First order approximation, two variables)
![Page 34: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/34.jpg)
assuming small
motion
divide by
take limit
Brightness Constancy
Equation
Multivariable Taylor Series Expansion(First order approximation, two variables)
![Page 35: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/35.jpg)
shorthand notation
vector form
Brightness
Constancy Equation
(x-flow) (y-flow)
(1 x 2) (2 x 1)
![Page 36: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/36.jpg)
(putting the math aside for a second…)
What do the terms of the
brightness constancy equation represent?
![Page 37: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/37.jpg)
(putting the math aside for a second…)
What do the term of the
brightness constancy equation represent?
Image gradients(at a point p)
![Page 38: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/38.jpg)
(putting the math aside for a second…)
What do the term of the
brightness constancy equation represent?
flow velocities
Image gradients(at a point p)
![Page 39: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/39.jpg)
(putting the math aside for a second…)
What do the term of the
brightness constancy equation represent?
flow velocities
temporal gradient
How do you compute these terms?
Image gradients(at a point p)
![Page 40: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/40.jpg)
spatial derivative
How do you compute …
![Page 41: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/41.jpg)
spatial derivative
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
How do you compute …
![Page 42: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/42.jpg)
spatial derivative temporal derivative
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
How do you compute …
![Page 43: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/43.jpg)
spatial derivative temporal derivative
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
How do you compute …
frame differencing
![Page 44: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/44.jpg)
Frame differencing
1 1 1 1 1
1 1 1 1 1
1 10 10 10 10
1 10 10 10 10
1 10 10 10 10
1 10 10 10 10
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 10 10 10
1 1 10 10 10
1 1 10 10 10
0 0 0 0 0
0 0 0 0 0
0 9 9 9 9
0 9 0 0 0
0 9 0 0 0
0 9 0 0 0
(example of a forward difference)
- =
![Page 45: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/45.jpg)
Example:
1 1 1 1 1
1 1 1 1 1
1 10 10 10 10
1 10 10 10 10
1 10 10 10 10
1 10 10 10 10
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 10 10 10
1 1 10 10 10
1 1 10 10 10
y
x
y
x
- 0 0 0 -
- 0 0 0 -
- 9 0 0 -
- 9 0 0 -
- 9 0 0 -
- 9 0 0 -
y
x
- - - - -
0 0 0 0 0
0 9 9 9 9
- 0 0 0 0
0 0 0 0 0
- - - - -
y
x
0 0 0 0 0
0 0 0 0 0
0 9 9 9 9
0 9 0 0 0
0 9 0 0 0
0 9 0 0 0
y
x
3 x 3 patch
-1 0 1
-1
0
1
![Page 46: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/46.jpg)
spatial derivative optical flow
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
How do you compute …
temporal derivative
frame differencingHow do you compute this?
![Page 47: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/47.jpg)
spatial derivative optical flow
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
How do you compute …
temporal derivative
frame differencingWe need to solve for this!(this is the unknown in the optical
flow problem)
![Page 48: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/48.jpg)
spatial derivative optical flow
Forward difference
Sobel filter
Derivative-of-Gaussian filter
…
Solution lies on a line
Cannot be found uniquely
with a single constraint
How do you compute …
temporal derivative
frame differencing
![Page 49: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/49.jpg)
Solution lies on a straight line
The solution cannot be determined uniquely with a
single constraint (a single pixel)
many combinations of u and v will satisfy the equality
![Page 50: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/50.jpg)
known
unknown
We need at least ____ equations to solve for 2 unknowns.
![Page 51: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/51.jpg)
known
unknown
Where do we get more equations (constraints)?
![Page 52: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/52.jpg)
Horn-Schunck
Optical Flow (1981)
Lucas-Kanade
Optical Flow (1981)
‘constant’ flow(flow is constant for all pixels)
‘smooth’ flow(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method
(dense)
local method
(sparse)
small motion
![Page 53: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/53.jpg)
Constant flow
![Page 54: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/54.jpg)
Where do we get more equations (constraints)?
Assume that the surrounding patch (say 5x5) has
‘constant flow’
![Page 55: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/55.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
![Page 56: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/56.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
![Page 57: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/57.jpg)
Equivalent (see TQ10) to solving:
where the summation is over each pixel p in patch P
![Page 58: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/58.jpg)
where the summation is over each pixel p in patch P
Sometimes called ‘Lucas-Kanade Optical Flow’
(can be interpreted to be a special case of the LK method with a translational warp model)
Equivalent (see TQ10) to solving:
![Page 59: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/59.jpg)
When is this solvable?
![Page 60: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/60.jpg)
When is this solvable?
ATA should be invertible
ATA should not be too small
ATA should be well conditioned
l1and l
2should not be too small
l1/l
2should not be too large (l
1=larger eigenvalue)
![Page 61: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/61.jpg)
Where have you seen this before?
=
![Page 62: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/62.jpg)
Where have you seen this before?
Harris Corner Detector!
=
![Page 63: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/63.jpg)
Where have you seen this before?
Harris Corner Detector!
=
What are the implications?
![Page 64: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/64.jpg)
Implications
• Corners are when λ1, λ2 are big; this is also when
Lucas-Kanade optical flow works best
• Corners are regions with two different directions of
gradient (at least)
• Corners are good places to compute flow!
What happens when you have no ‘corners’?
![Page 65: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/65.jpg)
You want to compute optical flow.
What happens if the image patch contains only a line?
![Page 66: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/66.jpg)
Barber’s pole illusion
![Page 67: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/67.jpg)
Barber’s pole illusion
![Page 68: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/68.jpg)
Barber’s pole illusion
![Page 69: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/69.jpg)
Aperture Problem
In which direction is the line moving?
small visible image
patch
![Page 70: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/70.jpg)
Aperture Problem
In which direction is the line moving?
small visible image
patch
![Page 71: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/71.jpg)
Aperture Problem
![Page 72: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/72.jpg)
Aperture Problem
![Page 73: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/73.jpg)
Aperture Problem
![Page 74: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/74.jpg)
Aperture Problem
![Page 75: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/75.jpg)
Want patches with different gradients to
the avoid aperture problem
![Page 76: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/76.jpg)
Want patches with different gradients to
the avoid aperture problem
![Page 77: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/77.jpg)
1 1 1 1
2 2 2 2
3 3 3 3
4 4 4 4
5 5 5 5
- - - -
- 1 1 1
- 2 2 2
- 3 3 3
- 4 4 4
y
x
y
x
optical flow: (1,1)
H(x,y) = y I(x,y)
We recover the v of the optical flow but not the u.
This is the aperture problem.
Compute gradients Solution:
![Page 78: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/78.jpg)
Horn-Schunck optical flow
![Page 79: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/79.jpg)
Horn-Schunck
Optical Flow (1981)
Lucas-Kanade
Optical Flow (1981)
‘constant’ flow(flow is constant for all pixels)
‘smooth’ flow(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method
(dense)
local method
(sparse)
small motion
![Page 80: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/80.jpg)
Smoothness
most objects in the world are rigid or
deform elastically
moving together coherently
we expect optical flow fields to be smooth
![Page 81: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/81.jpg)
Key idea(of Horn-Schunck optical flow)
Enforce
brightness constancy
Enforce
smooth flow field
to compute optical flow
![Page 82: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/82.jpg)
Key idea(of Horn-Schunck optical flow)
Enforce
brightness constancy
Enforce
smooth flow field
to compute optical flow
![Page 83: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/83.jpg)
Enforce
brightness constancy
For every pixel,
![Page 84: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/84.jpg)
For every pixel,
lazy notation for
Enforce
brightness constancy
![Page 85: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/85.jpg)
Key idea(of Horn-Schunck optical flow)
Enforce
brightness constancy
Enforce
smooth flow field
to compute optical flow
![Page 86: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/86.jpg)
Enforce smooth flow field
u-component of flow
![Page 87: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/87.jpg)
Which flow field optimizes the objective?
?
![Page 88: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/88.jpg)
smallbig
Which flow field optimizes the objective?
![Page 89: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/89.jpg)
Key idea(of Horn-Schunck optical flow)
Enforce
brightness constancy
Enforce
smooth flow field
to compute optical flow
bringing it all together…
![Page 90: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/90.jpg)
Horn-Schunck optical flow
smoothness brightness constancy
weight
![Page 91: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/91.jpg)
HS optical flow objective function
Brightness constancy
Smoothness
![Page 92: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/92.jpg)
How do we solve this
minimization problem?
![Page 93: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/93.jpg)
Compute partial derivative, derive update equations(gradient decent!)
How do we solve this
minimization problem?
![Page 94: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/94.jpg)
Compute the partial derivatives of this huge sum!
smoothness term brightness constancy
![Page 95: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/95.jpg)
Compute the partial derivatives of this huge sum!
it’s not so bad…
how many u terms depend on k and l?
![Page 96: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/96.jpg)
Compute the partial derivatives of this huge sum!
it’s not so bad…
how many u terms depend on k and l?
ONE from brightness constancyFOUR from smoothness
![Page 97: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/97.jpg)
Compute the partial derivatives of this huge sum!
it’s not so bad…
how many u terms depend on k and l?
ONE from brightness constancyFOUR from smoothness
![Page 98: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/98.jpg)
Compute the partial derivatives of this huge sum!
(variable will appear four times in sum)
![Page 99: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/99.jpg)
Compute the partial derivatives of this huge sum!
short hand for
local average
(variable will appear four times in sum)
![Page 100: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/100.jpg)
Where are the extrema of E?
![Page 101: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/101.jpg)
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
![Page 102: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/102.jpg)
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
how do you solve this?this is a linear system
![Page 103: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/103.jpg)
ok, take a step back, why are we doing all this math?
![Page 104: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/104.jpg)
We are solving for the optical flow (u,v) given two
constraints
smoothness brightness constancy
We need the math to minimize this
(back to the math)
![Page 105: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/105.jpg)
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
how do you solve this?
Partial derivatives of Horn-Schunck objective function E:
![Page 106: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/106.jpg)
Recall
![Page 107: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/107.jpg)
Recall
(det A)
Same as the linear system:
(det A)
![Page 108: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/108.jpg)
Rearrange to get update equations:
new
value
old
average
![Page 109: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/109.jpg)
new
value
old
average
When lambda is small (lambda inverse is big)…
Recall:
![Page 110: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/110.jpg)
new
value
old
average
When lambda is small (lambda inverse is big)…
goes to
zero
goes to
zero
Recall:
![Page 111: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/111.jpg)
new
value
old
average
When lambda is small (lambda inverse is big)…
goes to
zero
goes to
zero
…we only care about smoothness.
Recall:
![Page 112: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/112.jpg)
ok, take a step back, why did we do all this math?
![Page 113: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/113.jpg)
We are solving for the optical flow (u,v) given two
constraints
smoothness brightness constancy
We needed the math to minimize this
(now to the algorithm)
![Page 114: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/114.jpg)
Horn-Schunck
Optical Flow Algorithm
1. Precompute image gradients
2. Precompute temporal gradients
3. Initialize flow field
4. While not converged
Compute flow field updates for each pixel:
Just 8 lines of code!
![Page 115: Optical flow - Carnegie Mellon School of Computer Science16385/lectures/lecture24.pdf · Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow](https://reader036.fdocuments.in/reader036/viewer/2022071218/6053e164e888837b39645d03/html5/thumbnails/115.jpg)
References
Basic reading:• Szeliski, Section 8.4.