Post on 21-Dec-2015
ComputerVision
Optical Flow
Marc PollefeysCOMP 256
Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama
ComputerVision
2
last week: polar rectification
ComputerVision
3
Last week: polar rectification
ComputerVision
4
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
ComputerVision
5
Hierarchical stereo matchingD
ow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
ComputerVision
6
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
ComputerVision
7
Example: reconstruct image from neighboring images
ComputerVision
8
Multi-view depth fusion
• Compute depth for every pixel of reference image– Triangulation– Use multiple views– Up- and down sequence– Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
ComputerVision
9
Real-time stereo on graphics hardware
• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate
results over support region• Trade-off between small and large support window
Yang and Pollefeys CVPR03
140M disparity hypothesis/sec on Radeon 9700pro140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hze.g. 512x512x20disparities at 30Hz
Shape of a kernel for summing up 6 levels
ComputerVision
10
Sample Re-Projections
near far
ComputerVision
11
(1x1)
(1x1+2x2)
(1x1+2x2 +4x4+8x8)
(1x1+2x2 +4x4+8x8 +16x16)
Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass
video
ComputerVision
12
Cool ideas
• Space-time stereo (varying illumination, not shape)
ComputerVision
13
More on stereo …
The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/
Recommended reading
D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).
ComputerVision
14
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
ComputerVision
15
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
16
Optical Flow:Where do pixels move to?
ComputerVision
17
Motion is a basic cue
Motion can be the only cue for segmentation
ComputerVision
19
Motion is a basic cue
Even impoverished motion data can elicit a strong percept
ComputerVision
20
Applications
• tracking• structure from motion• motion segmentation• stabilization• compression• Mosaicing• …
ComputerVision
21
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
22
Definition of optical flow
OPTICAL FLOW = apparent motion of brightness patterns
Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image
ComputerVision
23
Caution required !
Two examples :
1. Uniform, rotating sphere
O.F. = 0
2. No motion, but changing lighting
O.F. 0
ComputerVision
24
Caution required !
ComputerVision
25
Mathematical formulation
I (x,y,t) = brightness at (x,y) at time t
Optical flow constraint equation :
0
t
I
dt
dy
y
I
dt
dx
x
I
dt
dI
),,(),,( tyxItttdt
dyyt
dt
dxxI
Brightness constancy assumption:
ComputerVision
26
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
27
The aperture problem
0 tyx IvIuI
1 equation in 2 unknowns
dt
dxu
dt
dyv
y
II x
y
II y
t
II t
ComputerVision
28
The aperture problem
0
ComputerVision
29
ComputerVision
30
The aperture problem
ComputerVision
31
Remarks
ComputerVision
32
Apparently an aperture problem
ComputerVision
33
What is Optic Flow, anyway?
• Estimate of observed projected motion field
• Not always well defined!• Compare:
– Motion Field (or Scene Flow)projection of 3-D motion field
– Normal Flowobserved tangent motion
– Optic Flowapparent motion of the brightness pattern(hopefully equal to motion field)
• Consider Barber pole illusion
ComputerVision
34
ComputerVision
35
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
36
Horn & Schunck algorithm
Additional smoothness constraint :
,))()(( 2222 dxdyvvuue yxyxs besides OF constraint equation term
,)( 2 dxdyIvIuIe tyxc
minimize es+ec
ComputerVision
37
Horn & Schunck
The Euler-Lagrange equations :
0
0
yx
yx
vvv
uuu
Fy
Fx
F
Fy
Fx
F
In our case ,
,)()()( 22222tyxyxyx IvIuIvvuuF
so the Euler-Lagrange equations are
,)(
,)(
ytyx
xtyx
IIvIuIv
IIvIuIu
2
2
2
2
yx
is the Laplacian operator
ComputerVision
38
Horn & Schunck
Remarks :
1. Coupled PDEs solved using iterative methods and finite differences
2. More than two frames allow a better estimation of It
3. Information spreads from corner-type patterns
,)(
,)(
ytyx
xtyx
IIvIuIvt
v
IIvIuIut
u
ComputerVision
39
ComputerVision
40
Horn & Schunck, remarks
1. Errors at boundaries
2. Example of regularisation (selection principle for the solution of illposed problems)
ComputerVision
41
Results of an enhanced system
ComputerVision
42
Structure from motion with OF
ComputerVision
43
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
44
02),(
02),(
tyxy
tyxx
IvIuIIdv
vudE
IvIuIIdu
vudE
ComputerVision
45
ComputerVision
46
ComputerVision
47
ComputerVision
48
ComputerVision
49
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
50
ComputerVision
51
ComputerVision
52
ComputerVision
53
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
54
ComputerVision
55
ComputerVision
56
ComputerVision
57
ComputerVision
58
ComputerVision
59
ComputerVision
60
ComputerVision
61
ComputerVision
62
ComputerVision
63
ComputerVision
64
ComputerVision
65
ComputerVision
66
ComputerVision
67
ComputerVision
68
ComputerVision
69
ComputerVision
70
ComputerVision
71
ComputerVision
72
ComputerVision
73
ComputerVision
74
ComputerVision
75
ComputerVision
76
ComputerVision
77
ComputerVision
78
ComputerVision
79
ComputerVision
80
ComputerVision
81
ComputerVision
82
ComputerVision
83
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
84
ComputerVision
85
ComputerVision
86
ComputerVision
87
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
88
ComputerVision
89
ComputerVision
90
ComputerVision
91
ComputerVision
92
ComputerVision
93
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
94
ComputerVision
95
ComputerVision
96
ComputerVision
97
ComputerVision
98
ComputerVision
99
ComputerVision
100
Optical Flow
• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow
ComputerVision
101
ComputerVision
102
Rhombus Displays
http://www.cs.huji.ac.il/~yweiss/Rhombus/
ComputerVision
103
ComputerVision
104
ComputerVision
105
ComputerVision
106
ComputerVision
107
ComputerVision
108