Video TexturesVideo Textures

Arno SchödlRichard Szeliski

David SalesinIrfan Essa

Microsoft Research, Georgia Tech

Still PhotosStill Photos

Video ClipsVideo Clips

Video TexturesVideo Textures

Problem StatementProblem Statement

video clip video texture

ApproachApproach

How do we find good transitions?

Finding Good Transitions Finding Good Transitions

Compute L2 distance Di, j between all frames

Similar frames make good transitions

frame ivs.

frame j

Markov Chain RepresentationMarkov Chain Representation

2 3 41

Similar frames make good transitions

Transition Costs Transition Costs

Transition from i to j if successor of i is similar to j

Cost function: Cij = Di+1, j

i

j

i+ 1

j-1

i j D i+ 1, j

Transition ProbabilitiesTransition Probabilities

Probability for transition Pij inversely related

to cost:

high low

2exp

ji

ji

CP

Preserving DynamicsPreserving Dynamics

Preserving Dynamics Preserving Dynamics

Preserving Dynamics Preserving Dynamics

Cost for transition ij

Cij = wk Di+k+1, j+kk = -N

N -1

i

j j+ 1

i+ 1 i+ 2

j-1j-2

i jD i, j - 1 D Di+ 1, j i+ 2, j+ 1

i-1

D i-1, j-2

Preserving Dynamics – Effect Preserving Dynamics – Effect

Cost for transition ij

Cij = wk Di+k+1, j+kk = -N

N -1

2 3 41

DeadendsDeadends

No good transition at the end of sequence

2 3 41

Future CostFuture Cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future CostFuture Cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future CostFuture Cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future CostFuture Cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

• Q-learning

Future CostFuture Cost

Future Cost – EffectFuture Cost – Effect

Finding Good LoopsFinding Good Loops

• Alternative to random transitions

• Precompute a good set of loops up front (using dynamic programming)

Visual DiscontinuitiesVisual Discontinuities

• Problem: Visible “Jumps”

Crossfading Crossfading

• Solution: Crossfade from one sequence to the other.

A i-2

A i-2

B j-2

15

…

…

3

1 2

2 1

3

4

4 4

4 4

4+ + +

A i-1

A i-1 / B j-2 A i-1 / B j-2 A i-1 / B j-2

B j-1

A i

B j

A i+1

B j+1

B j+1

Morphing Morphing

• Interpolation task:

A25 B2

5 C15+ +

Morphing Morphing

• Interpolation task:

• Compute correspondencebetween pixels of all frames

A25 B2

5 C15+ +

Morphing Morphing

• Interpolation task:

• Compute correspondence between pixels of all frames

• Interpolate pixel position andcolor in morphed frame

• based on [Shum 2000]

A25 B2

5 C15+ +

Results – Crossfading / MorphingResults – Crossfading / Morphing

Results – Crossfading / MorphingResults – Crossfading / Morphing

Jump Cut Crossfade Morph

CrossfadingCrossfading

Frequent Jump & CrossfadingFrequent Jump & Crossfading

Video PortraitVideo Portrait

Useful for web pages

Combine with IBR techniques

Video Portrait – 3DVideo Portrait – 3D

Region-based AnalysisRegion-based Analysis

• Divide video up into regions

• Generate a video texture for each region

Automatic Region AnalysisAutomatic Region Analysis

User selects target frame range

User-controlled Video TexturesUser-controlled Video Textures

slow variable fast

Lengthen / shorten video without affecting speed

Time WarpingTime Warping

shorter original longer

Summary Summary

• Video clips video textures

• define Markov process

• preserve dynamics

• avoid dead-ends

• disguise visual discontinuities

Summary Summary

• Extensions

• regions

• external constraints

• video-based animation

Discussion Discussion

• Some things are relatively easy

Discussion Discussion

• Some are hard