Consistent Optical Flow forConsistent Optical Flow for Stereo Video · Consistent Optical Flow for...

Consistent Optical Flow forConsistent Optical Flow forConsistent Optical Flow for Consistent Optical Flow for Stereo VideoStereo Video

ICIP 2010

A it S ll t Ch i ti Li M MAnita Sellent, Christian Linz, Marcus MagnorTU Braunschweig, Germany

Monocular Video EditingMonocular Video EditingPropagating keyframe information based on

Monocular Video EditingMonocular Video Editing

Optical Flow

2Consistent Optical Flow for Stereo Video Anita Sellent

Monocular Video EditingMonocular Video EditingPropagating keyframe information based on

Monocular Video EditingMonocular Video Editing

Optical Flow

[Eisemann et al. 2009]

Inaccurate flow estimation requires manual


Inaccurate flow estimation requires manual interaction

Stereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingCorrespondence between temporally and spatially neighboring images

left right


Uncalibrated and non-synchronized cameras

Consistent Optical FlowConsistent Optical FlowConsistent Optical FlowConsistent Optical FlowOptical flow estimation between image pairs

Small errors accumulate quickly

time


left right

Key IdeaKey IdeaKey IdeaKey IdeaExploit redundant data in stereo videoCalculate flow on three images simultaneously


OutlineOutlineOutlineOutlineThe basic two image flowThe symmetry constraintThe loop constraintIterative flow estimationPropagation over a sequencep g qResults


BasicBasic TwoTwo Image FlowImage FlowBasic Basic TwoTwo Image FlowImage FlowBrightness constancy assumption

Smoothness term: total variation, Huber norm

0)()( 2,121 ≈+− wxIxISmoothness term: total variation, Huber norm

, G

2,1w∇ 2,15.0 wD ∇

Global energy

∫∫Ω

∇++− dxwwxIIw 2,12,121 )(min

2,1

λ


Splitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy Function[Zach et al. 2007][Werlberger et al 2009]

Auxiliary variable

∫2

[Werlberger et al. 2009]

2,1z

∫Ω

∇+−++− dxzzwwxIIzw 2,12,12,12,121,

2)(min2,12,1 θλ

Pointwise problemPointwise problem

2,12,12,1211)(min

21

zwwxIIw

−++−θ

λ

Image denoising problem2,1w θ

∫ ∇ d1i


∫Ω

∇+− dxzzwzw 2,12,12,1,

min2,12,1 θ

The Symmetry ConstraintThe Symmetry ConstraintThe Symmetry ConstraintThe Symmetry Constraint[Alvarez et al. 2007]

Symmetrie error:

0≈++= )()( 2,11,22,1 wxwxwSρ

2I :2,1w flow from to1I 2I

:1,2w flow from to2I 1I

1I


The Loop ConstraintThe Loop ConstraintThe Loop ConstraintThe Loop ConstraintLoop error:

0≈+++++= )()()( 3,22,11,32,13,22,1 wwxwwxwxwLρ

2I

I1I

3I


UpdateUpdate schemeschemeUpdate Update schemeschemeDetermine incremental update jidw ,

from brightness constancyEvaluate satisfaction of symmetry and loop

fconstraint for this estimate22LS ba eep ρρ −−=

jik

jik

ji dwpww ,,1

, +=+Update current flow fieldUse current flow estimation to determine

jiz ,

hjw ,

TV-optimized versionProceed with next flow field


Consistent UpdateConsistent UpdateConsistent UpdateConsistent UpdateUpdate flow field only when symmetry and loop constraint are satisfiedRely on smooth fill-in otherwiseBrightness constancy for occluded points is suppressed


Propagation in aPropagation in a SequenceSequencePropagation in a Propagation in a SequenceSequenceUse known flow fields to constrain the new flow fields


Propagation IIPropagation IIPropagation IIPropagation II… …


ResultsResultsResultsResultsTest sequences with ground-truth

[V d t l 2008][Vaudrey et al. 2008]


[Scharstein et al. 2007]

Results AEEResults AEEResults AEEResults AEE

AEE Wave enpeda Art Books DollsAEE Wave .enpeda Art Books Dolls

TV-L2 1.03 0.71 10.62 14.60 5.81

Symmetry 1.01 0.61 10.02 10.73 2.93

Loop 0.97 0.59 9.34 6.43 2.85


Results ConsistencyResults ConsistencyResults ConsistencyResults Consistency

Consistency Wave .enpeda Art Books DollspTV-L2 1.24 4.01 3.00 17.45 11.37

Symmetry 0.16 1.29 2.69 6.30 7.60


Loop 0.13 0.87 1.71 4.27 2.84

ConclusionConclusionConclusionConclusionEstimate flow fields between three images simultaneously Prevent error accumulationDisable brightness constancy for occluded points

Consistent and more accurate flow fields


Thank You for Your Attention!Thank You for Your Attention!

htt // hi t b d


https://graphics.tu-bs.de

Consistent Optical Flow forConsistent Optical Flow for Stereo Video · Consistent Optical Flow for...

Documents

Transcript of Consistent Optical Flow forConsistent Optical Flow for Stereo Video · Consistent Optical Flow for...