Fast and Accurate Optical Flow Estimation

Primal-Dual Schemes andSecond Order Priors

Thomas Pock and Daniel CremersCVPR Group, University of Bonn

Collaborators: Christopher Zach, Markus Unger, Werner Trobin, and Horst Bischof

Variational Optical Flow – Short History

Horn and Schunck

1981

Black and Anadan, Cohen

1993

Aubert

2000 2004

Brox et al.

2006

Bruhn et al.

Outline• Model of Horn and Schunck• TV-L1 Model• Fast Numerical Scheme• Parallel Implementation• 2nd order Prior

The Model of Horn and Schunck [1]

Regularization Term Data Term (OFC)

+ Convex+ Easy to solve- Does not allow for sharp edges in the solution- Sensitive to outliers violating the OFC

[1] Horn and Schunck. Determinig Optical Flow. Artificial Intelligence, 1981

Can we do better?

• Replace quadratic functions by L1 – norms• Done by Cohen, Aubert, Brox, Bruhn, ...

+Allows for discontinuities in the flow field+Robust to some extent to outliers in the OFC+Still convex- Much harder to solve

How can we minimize this functional ?

• Compute Euler-Lagrange Equations

• Non-linear, non-smooth, ...

Standard Approach

• Replace L1 – norm by regularized variants (Charbonnier function)

• Example:• Small epsilon: Nearly degenerated• Large espilon: Smears edges

Our Approach(1)

• Introduce auxiliary variables and constraints

• Quadratic penalty

Our Approach(2)• What do we gain?• We solve a sequence of simpler problems

Algorithm[3]:1. For fixed (u´,v´), solve for(u,v) using Chambolle‘s algorithm[4]2. For fixed (u,v), solve for (u´,v´) using a 1D shrinkage formula3. Goto 1. until convergence

[2] Rudin, Osher and Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, 1992[3] Zach, Pock and Bischof. A Duality Based Algorithm for Realtime TV-L1 Optical Flow, DAGM 2007 [4] Chambolle. An Algorithm for Total Variation Minimization, 2004.

1D Problem

ROF Model [2]

Implementation

• Numerical scheme can be easily parallelized• We use state-of-the-art GPUs

Performance Evaluation

Image Size Frames per Second128x128 192256x256 108512x512 36

• TV-L1 Optical Flow Implemented in CUDA 2.0• Computed on Nvidia GeForce GTX 280• 25 Overall Iterations (5 Chambolle Iterations)

Results for TV-L1

Ground Truth:

Our Results:

Input Image:

2nd order Prior• TV regularization favors piecewise constant flow

fields (frontoparallel motion)• Extension to piecewise affine flow fields?• Approach of Cremers et al. [5]– Fixed number of regions

• Approach of Nir et al. [6]– Over-parametrized optical flow

• Our approach [7]– 2nd order derivatives to regularize flow field[5] Cremers and Soatto, Motion Competition: A Variational Framework for Piecwise Parametric Motion

Segmentation.[6] Nir, Bruckstein and Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007[7] Trobin, Pock, Cremers and Bischof, An Unbiased Second-Order prPior for High-AccuracyMotion Estimation, DAGM 2008

2nd-L1 Optical Flow

• 2nd order derivatives are not orthogonal• We use a transformation due to Danielsson [8]

• Optimization– Similar strategy to TV-L1

– 4th order PDE

[8] Danielsson and Lin, Efficient Detection of Second-Degree Variations in 2D and 3D Images, 2001.

Comparison

Ground truth TV-L1 2nd -L1

Results for 2nd-L1

Ground Truth:

Our Results:

Conclusion

• TV-L1 Optical Flow– Fast Numerical Scheme

• Parallel Implementation– Realtime Performance

• 2nd order prior – Piecewise affine motion

Recent Application: Tracking