Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video...

Video SegmentationTal Kramer, Shai Bagon

Advanced Topics in Computer Vision Spring 2006

Video Segmentation

April 30th, 2006



Introduction

• Recognition and Segmentation• Min Cut Max Flow• Single Image Methods

– GrabCut– Lazy Snapping– …



Lazy Snapping

• Interactive User Interface



Lazy Snapping

• Energy minimization



Lazy Snapping

• Boundary overriding



Motivation

• Obvious Next Step• Video Cut & Paste• Video Manipulation and Editing



Introduction

• Frame by Frame– Time Consuming and Tedious

• Error With Simple Methods– Fast motions– Deforming silhouettes – Changing topologies



Introduction

• Two Papers– Video Object Cut

and Paste– Video Cutout



Video Object Cut and Paste

Yin Li, Jian Sun, Heung-Yeung Shum



Overview



Pre-segmentation

• Pre-Segmentation to All Frames



Key Frames

• Picking Key Frames.



Key Frames

• User Fore/Background Segmentation



3D Graph Cut Segmentation

• 3D Graph – G=(V,A)

• Labeling– Foreground = 1 – Background = 0

• Volume Between Successive Key Frames



3D Graph Construction

• 2 Kinds of Arcs:– AI – Intra

Frames (BLUE)

– AT – Inter Frame (RED)



3D Graph Construction

• Minimizing Equation:

• E1 – Global Color Models

• E2 – Penalizing Spatially

• E3 – Penalizing Temporally



Likelihood Energy

• GMMs Decide Label• In Key Frames:



GMM• Gaussian Mixture

Model

• Distance is Measured By:



Prior Energies

• E2, E3 Are the Same

• Distance of Adjacent Regions.

• β = (2 E (||cr – cs||2 ))-1



Prior Energies

• λ1 = 24

• λ2 = 12



3D Graph Segmentation



Errors

• Global Colors• Similarity to

Background• Thin Areas



Error Overriding

• Video tubes• Manual corrections



Video Tubes

• Local Color Models• Put Two Windows• Tracking Algorithm• Key Frames to

Solve

W1

WT



Fixing graph cut segmentation

• Minimizing:



Overriding Brush

• Fixing Boundary Manually



Manual error overriding



• Soften hard segmentation

Matting



Coherent Matting

• Boundary is not 0/1• Prevent Bolting Pixels• Smooth Paste



Coherent Matting



Example



Video CutOut

J. WANG, P. BHAT, A. COLBURN, M. AGRAWALA, M. COHEN. Interactive Video Cutout. ACM Trans. on Graphics

(Proc. of SIGGAPH2005), 2005



Video Cutout introduction

What’s new?• Different user interface• 3D graph formation• Refinement mechanism



System overview



3D Graph construction

• Hierarchical graph nodes:1. Frame by frame mean shift

segmentation2. Aggregating segments across

frames



• Pixel 26-neighborhood induce links• Lower level links induce higher level

link

3D Graph construction



• Stroking foreground and background over the 3D spatio-temporal volume

• Not segmenting any frame

User Interface



• Graph construction– User input propagates upward – Min cut uses yellow nodes

3D Min cut/Max flow



3D Min cut/Max flow

• Weights / Energy function– The energy function:

– Data term: color similarity to F/B model– Link term: cut likelihood

1,

, , , , , ,i i i i j i ji nghbrs i j

E D x c L x x c c



3D Min cut/Max flow

• Terms in energy function

Graph Cut Energy function

L LinkDF ForegroundDB Background

DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient



Data weight

• User input generates color model (GMM)

• Infinite weight preserves marked pixels

• Data weight = abiding to F/B color model






Data weight

White – high probability ForegroundBlack – Low probability Foreground






Data weight

White – high probability BackgroundBlack – Low probability Background






• Strong gradients segment border

• Link cost encourage cut at edges

Link weight Graph Cut Energy function





Link weight

White – low cut probabilityBlack – high cut probability






• Pixel span: (xo, yo, t)t>0

Data weight Graph Cut Energy function








• Local background model• Assuming camera is stabilized, video is

registered• Extracting “clean plate” • Weight per pixel span



• d(zi) = minimum color distance {“clean plate”, B marked pixel}.

• “Clean plate” cannot be always trusted

• Weight:




1 100

2 1N

NumFramese



Data weight

White – high probability BackgroundBlack – Low probability Background






• Link span: links between two adjacent pixel spans






• Strong edges exists within segment

• Small change over time• Local temporal link cost penalize

strong temporal gradient






Link weight

White – low cut probabilityBlack – high cut probability






• Energy function




3D Min cut/Max flow

1,


E D x c L x x c c

λ2

λ1

λ3



Iterative process

• The user refines the cut• Adds F/B strokes• Graph is re-computed

Nth iterationN+1th iteration



Post Processing



Post processing

• Binary cut obtained• Edges need refinement



• A pixel-level min cut around edges• Color model obtained form

boundary• Uniform edge cost = small

cut

Refinement



Matting

• Soften hard segmentation• Evaluate α Channel



Matting

• Refinement fixed boundary locally

• Global 3D mesh• α Channel along

mesh normals



Results



Performance



• Pros– Online 3D min cut – Spatio temporal smooth cut

• Cons– Does not handle shadows– Ignore motion blur (LPF to avoid

temporal aliasing)– Cannot separate translucent objects

Summary



ComparisonVideo object cut

and pasteVideo Cutout

Features

•Graph nodes

•UI

2D segmentation

Frame base interface

3D segmentation in 2 stages

spatial-temporal manipulation

Performance

•Preprocessing•Artist time•Post processing•Total

4-5 min25 min

?30 min

25 min10 sec per Min cut

30 min60 min



Questions?



• The total energy:

• Foreground and background terms:

• Background terms:

• Link terms:

Energy function

1,


E D x c L x x c c

;B Fi i

B F B F

D DD x B D x F

D D D D

2 , 2 ,1B i B L B GD x B D D

3 31i j G LL x x L L 3 0.3

1 100

2 1N

NumFramese

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video...

Documents

Transcript of Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video...