Consistent Optical Flow forConsistent Optical Flow forConsistent Optical Flow for Consistent Optical Flow for Stereo VideoStereo Video

ICIP 2010

A it S ll t Ch i ti Li M MAnita Sellent, Christian Linz, Marcus MagnorTU Braunschweig, Germany

Monocular Video EditingMonocular Video EditingPropagating keyframe information based on

Monocular Video EditingMonocular Video Editing

Optical Flow

2Consistent Optical Flow for Stereo Video Anita Sellent

Monocular Video EditingMonocular Video EditingPropagating keyframe information based on

Monocular Video EditingMonocular Video Editing

Optical Flow

[Eisemann et al. 2009]

Inaccurate flow estimation requires manual

3Consistent Optical Flow for Stereo Video Anita Sellent

Inaccurate flow estimation requires manual interaction

Stereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingCorrespondence between temporally and spatially neighboring images

left right

4Consistent Optical Flow for Stereo Video Anita Sellent

Uncalibrated and non-synchronized cameras

Consistent Optical FlowConsistent Optical FlowConsistent Optical FlowConsistent Optical FlowOptical flow estimation between image pairs

Small errors accumulate quickly

time

5Consistent Optical Flow for Stereo Video Anita Sellent

left right

Key IdeaKey IdeaKey IdeaKey IdeaExploit redundant data in stereo videoCalculate flow on three images simultaneously

6Consistent Optical Flow for Stereo Video Anita Sellent

OutlineOutlineOutlineOutlineThe basic two image flowThe symmetry constraintThe loop constraintIterative flow estimationPropagation over a sequencep g qResults

7Consistent Optical Flow for Stereo Video Anita Sellent

BasicBasic TwoTwo Image FlowImage FlowBasic Basic TwoTwo Image FlowImage FlowBrightness constancy assumption

Smoothness term: total variation, Huber norm

0)()( 2,121 ≈+− wxIxISmoothness term: total variation, Huber norm

, G

2,1w∇ 2,15.0 wD ∇

Global energy

∫∫Ω

∇++− dxwwxIIw 2,12,121 )(min

2,1

λ

8Consistent Optical Flow for Stereo Video Anita Sellent

Splitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy Function[Zach et al. 2007][Werlberger et al 2009]

Auxiliary variable

∫2

[Werlberger et al. 2009]

2,1z

∫Ω

∇+−++− dxzzwwxIIzw 2,12,12,12,121,

2)(min2,12,1 θλ

Pointwise problemPointwise problem

2,12,12,1211)(min

21

zwwxIIw

−++−θ

λ

Image denoising problem2,1w θ

∫ ∇ d1i

9Consistent Optical Flow for Stereo Video Anita Sellent

∫Ω

∇+− dxzzwzw 2,12,12,1,

min2,12,1 θ

The Symmetry ConstraintThe Symmetry ConstraintThe Symmetry ConstraintThe Symmetry Constraint[Alvarez et al. 2007]

Symmetrie error:

0≈++= )()( 2,11,22,1 wxwxwSρ

2I :2,1w flow from to1I 2I

:1,2w flow from to2I 1I

1I

10Consistent Optical Flow for Stereo Video Anita Sellent

The Loop ConstraintThe Loop ConstraintThe Loop ConstraintThe Loop ConstraintLoop error:

0≈+++++= )()()( 3,22,11,32,13,22,1 wwxwwxwxwLρ

2I

I1I

3I

11Consistent Optical Flow for Stereo Video Anita Sellent

UpdateUpdate schemeschemeUpdate Update schemeschemeDetermine incremental update jidw ,

from brightness constancyEvaluate satisfaction of symmetry and loop

fconstraint for this estimate22LS ba eep ρρ −−=

jik

jik

ji dwpww ,,1

, +=+Update current flow fieldUse current flow estimation to determine

jiz ,

hjw ,

TV-optimized versionProceed with next flow field

12Consistent Optical Flow for Stereo Video Anita Sellent

Consistent UpdateConsistent UpdateConsistent UpdateConsistent UpdateUpdate flow field only when symmetry and loop constraint are satisfiedRely on smooth fill-in otherwiseBrightness constancy for occluded points is suppressed

13Consistent Optical Flow for Stereo Video Anita Sellent

Propagation in aPropagation in a SequenceSequencePropagation in a Propagation in a SequenceSequenceUse known flow fields to constrain the new flow fields

14Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

15Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

16Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

17Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

18Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

19Consistent Optical Flow for Stereo Video Anita Sellent

Propagation IIPropagation IIPropagation IIPropagation II… …

20Consistent Optical Flow for Stereo Video Anita Sellent

ResultsResultsResultsResultsTest sequences with ground-truth

[V d t l 2008][Vaudrey et al. 2008]

21Consistent Optical Flow for Stereo Video Anita Sellent

[Scharstein et al. 2007]

Results AEEResults AEEResults AEEResults AEE

AEE Wave enpeda Art Books DollsAEE Wave .enpeda Art Books Dolls

TV-L2 1.03 0.71 10.62 14.60 5.81

Symmetry 1.01 0.61 10.02 10.73 2.93

Loop 0.97 0.59 9.34 6.43 2.85

22Consistent Optical Flow for Stereo Video Anita Sellent

Results ConsistencyResults ConsistencyResults ConsistencyResults Consistency

Consistency Wave .enpeda Art Books DollspTV-L2 1.24 4.01 3.00 17.45 11.37

Symmetry 0.16 1.29 2.69 6.30 7.60

23Consistent Optical Flow for Stereo Video Anita Sellent

Loop 0.13 0.87 1.71 4.27 2.84

ConclusionConclusionConclusionConclusionEstimate flow fields between three images simultaneously Prevent error accumulationDisable brightness constancy for occluded points

Consistent and more accurate flow fields

24Consistent Optical Flow for Stereo Video Anita Sellent

Thank You for Your Attention!Thank You for Your Attention!

htt // hi t b d

25Consistent Optical Flow for Stereo Video Anita Sellent

https://graphics.tu-bs.de