Motion Detail Preserving Optical Flow Estimation
Li Xu1, Jiaya Jia1, Yasuyuki Matsushita2
1 The Chinese University of Hong Kong 2 Microsoft Research Asia
Conventional Optical Flow
• Middlebury Benchmark [Baker et al. 07]• Dominant Scheme: Coarse-to-Fine Warping
Large Displacement Optical Flow
• Region Matching [Brox et al. 09, 10]• Discrete Local Search [Steinbrucker et al. 09]
Both Large and Small Motion Exist
• Capture large motion• Preserve sub-pixel accuracy
Our Work
• Framework– Extended coarse-to-fine motion estimation for both
large and small displacement optical flow• Model– A new data term to selectively combine constraints
• Solver– Efficient numerical solver for discrete-continuous
optimization
Outline
• Framework– Extended coarse-to-fine motion estimation for both
large and small displacement optical flow• Model– A new data term to selectively combine constraints
• Solver– Efficient numerical solver for discrete-continuous
optimization
The Multi-scale Problem
The Multi-scale Problem
Ground truthGround truthGround truth
The Multi-scale Problem
Ground truthGround truthGround truth
Ground truth
…Estimate EstimateEstimate
Ground truthGround truth
The Multi-scale Problem
• Large discrepancy between initial values and optimal motion vectors
• Our solution – Improve flow initialization to reduce the reliance
on the initialization from coarser levels
Extended Flow Initialization
• Sparse feature matching for each level
Extended Flow Initialization
• Identify missing motion vectors
Extended Flow Initialization
• Identify missing motion vectors
Extended Flow Initialization
…
…
Extended Flow Initialization
Fuse
Outline
• Framework: extended initialization for coarse-to-fine motion estimation
• Model: selective data term • Efficient numerical solver
Data Constraints
• AverageI
xI
1 1(u, x) (u, x) (u, x)2 2DE D D
I 2 1(u, x) (x u) (x)I ID
I 2 1(u, x) (x u) (I I x)D • Gradient constancy
• Color constancy
I 2 1(u, x) (x u) (I I x)D
• Pixels moving out of shadow
Problems
pI 1 1p(u , ) 6.63D
• Color constancy is violated
I Ip1 1 p1 11 (u , ) (u , ) = 3.482
p pD D
• Average:
p1u : ground truth motion of p1
• Gradient constancy holdsp 1I 1 p(u , ) 0.32D
• Pixels undergoing rotational motion
Problems
• Color constancy holds
• Gradient constancy is violatedp2u : ground truth motion of p2
p 2I 2 p(u , ) 4.20D
• Average:
I Ip2 2 p2 21 (u , ) (u , ) = 2.242
p pD D
pI 2 2p(u , ) 0.29D
Our Proposal
• Selectively combine the constraints
where
I Ix
(u, ) (x) (u,x) (1 (x)) (u,x)DE D D 2(x) : {0,1}
I Ix
(u, ) (u,x(x) 1) ( ) (u,x)(x)DE D D
Comparisons
RubberWhale Urban22
2.5
3
3.5
4
4.5
5
colorgradientaverageours
AAE
Outline
• Framework: extended initialization for coarse to fine motion estimation
• Model: selective data term
• Efficient numerical solver
I Ix
(x) (u, x) (1 (x)) (u, x)D D
Energy Functions and Solver
• Total energy
• Probability of a particular state of the system
(u, )1(u, ) EP eZ
I Ix
(u, ) (x) (u, x) (1 (x)) (u, x) ( u, x)E D D S
(u, )1(u, ) EP eZ
(u, )1(u, ) EP eZ
Ix
I(x) (u, x) (1 (x)) (u, x)(u, ) ( u, x)E SD D
Mean Field Approximation
• Partition function
• Sum over all possible values of α
(u, )
{u} { 0,1}
EZ e
I Ix
( u,x) (u, x) ((x) (x)
{ 0,
1 ) (u, x)
{u 1}}
x
S D D
e e
(u, x)(u, x) II
x
1{ ( u,x) ln( )}
{u}
DDS e e
e
. . .
The effective potential Eeff (u) [Geiger & Girosi, 1989]
• Optimal condition (Euler-Lagrange equations)
• It decomposes to
II (u, x)(u, x)
x
1(u) ( u,x) ln( )DDeffE S e e
I I
I II I
(u,x) (u,x)
u I u I(u,x) (u,x)(u,x) (u,x)
u
(u, x) (u, x)
div( ( u,x)) 0
D D
D DD D
e eD De e e e
S
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S
I I
I II I
(u,x) (u,x)
(u,x) (u,u I u I
u
x)(u,x) (u,x)(u, x) (u, x)
div( ( u,x)) 0
D D
D DD D
e ee e e e
D D
S
( )x 1 ( )x
{
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {
Algorithm Skeleton
• For each level
• Extended Flow Initialization (QPBO)• Continuous Minimization (Iterative reweight)– Update– Compute flow field (Variable Splitting)
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {
Results
Averaging constraints Ours
Difference
Middlebury Dataset
EPE=0.74
Results from Different Steps
Coarse-to-fine
Extended coarse-to-fine
EPE=0.15 rank =1
EPE=0.24 rank =1
Large Displacement
Overlaid Input
Large Displacement
• Motion Estimates
Coarse-to-fine Our Result Warping Result
Comparison
• Motion Magnitude Maps
LDOP [Brox et al. 09 ] [Steinbrucker et al. 09] Ours
More Results
Overlaid Input
Conventional Coarse-to-fine Our Result
More Results
Overlaid Input
Coarse-to-fine Our Result
Conclusion
• Extended initialization (Framework)• Selective data term (Model)• Efficient numerical scheme (Solver)
• Limitations– Featureless motion details – Large occlusions
Thank you!
More Results
Overlaid Input
Coarse-to-fine Our Results
Top Related