Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig...

39
Shape from Silhouettes I Guido Gerig CS 6320, Spring 2013 Credits: Marc Pollefeys, UNC Chapel Hill, some of the figures and slides are also adapted from J.S. Franco, J. Matusik’s presentations, and referenced papers)

Transcript of Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig...

Page 1: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Shape from Silhouettes I

Guido Gerig

CS 6320, Spring 2013

Credits: Marc Pollefeys, UNC Chapel Hill, some of the figures and slides are also adapted from J.S. Franco, J. Matusik’s

presentations, and referenced papers)

Page 2: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Shape from silhouettes

Automatic 3D Model Construction for Turn-Table Sequences,

A.W. Fitzgibbon, G. Cross, and A. Zisserman, SMILE 1998

Slides from

Lazebnik,

Matusik

Yerex

and others

Page 3: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Big Picture

• Multi-camera

environments

• Dynamic scene

• N cameras observe the scene and produce N

video streams

• What can we do with this data?

Outdoor data capturing with 9 video cameras behind the Ackland Museum,

UNC-Chapel Hill, 2006/8/24. Pictured by Jae Hak Kim.

Page 4: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Motivation: Movies

Sinha Sudipta, UNC PhD 2008

Page 5: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Motivation: 3D from Movies

Sinha Sudipta, UNC PhD 2008

Page 6: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Motivation: 3D from Movies: Replay from arbitrary viewpoints

Sinha Sudipta, UNC PhD 2008

Page 7: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

What can we do with this data?

• Reconstruct scene objects:

– shape from silhouettes

– photo-consistency

• Calibrate cameras

– recover epipolar geometry

• Fit specific models (articulated models)

Page 8: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Outline

• Silhouettes

– basic concepts

– extract silhouettes

– fundamentals about using silhouettes

– reconstruct shapes from silhouettes

– use uncertain silhouettes

– calibrate from silhouettes

• Perspectives and interesting ideas

Page 9: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Silhouettes of objects of interest

• Silhouettes are the regions where objects of

interest project in images

• Silhouettes can generally be obtained using

low level information (fast)

• They give information about the global shape

of scene objects

Page 10: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

How to extract silhouettes?

• Sometimes done manually (for offline

applications, ground truth and verifications)

• Region based-extraction (automatic)

– silhouette extraction is a 2-region image

segmentation problem, w/ specific

solutions:

• chroma keying (blue, green background)

• background subtraction (pre-observed static or

dynamic background)

(refer to segmentation course)

Page 11: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

How to extract silhouettes?

• Contour-based extraction

• focus on silhouette outline instead of region

itself

– snakes, active contours: fitting of a curve to high

gradients in image, local optimization

Yilmaz&Shah ACCV04

Page 12: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

How to extract silhouettes? (cont.)

• Background subtraction

• Simple thresholding

• Train an appearance model for each pixel, from a set of background images

• RGB 3D-Gaussian model

• HSV model

• GMM model

• Non-parametric model (histogram/kernel density function)

• Apply the pixel color to the model, then classify it to be foreground/background

• We will talk about this in more detail later

Page 13: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Why use a Visual Hull?

• Good shape representation

• Can be computed efficiently

• No photo-consistency required

• As bootstrap of many fancy

refinement …

- =

background

+

foreground

background foreground

Page 14: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Outline

• Silhouettes

– basic concepts

– extract silhouettes

– fundamentals about using silhouettes

– reconstruct shapes from silhouettes

– use uncertain silhouettes

– calibrate from silhouettes

• Perspectives and cool ideas

Page 15: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

What is shape from silhouette?

• The silhouette, or occluding contour of an object in an image contains some information about the 3D shape of the object.

• Given a single silhouette image of an object, we know that the 3D object lies inside the volume generated by back-projecting the silhouette area using the camera parameters.

Page 16: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

What is shape from silhouette?

• With multiple views of the same object, we can intersect the generalized cones generated by each image, to build a volume which is guaranteed to contain the object.

• The limiting smallest volume obtainable in this way is known as the visual hull of the object.

Page 17: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

One-view silhouette geometry

Page 18: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Multi-view silhouette geometry: the Visual Hull

• Maximal volume consistent

with silhouettes [Laurentini94] [Baumgart74]

• Can be seen as the intersection of viewing cones

Visual hull

Viewing cone

• Properties:

• Containment property: contains real scene objects

• Converges towards the shape of scene objects minus concavities as N increases

• Projective structure: simple management of visibility problems

Page 19: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Visual Hull: A 3D Example

Page 20: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Outline

• Silhouettes

– basic concepts

– extract silhouettes

– fundamentals about using silhouettes

– reconstruct shapes from silhouettes

– use uncertain silhouettes

– calibrate from silhouettes

• Perspectives and cool ideas

Page 21: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Algorithms

• Standard voxel based method Marching Intersections

• Exact polyhedral methods

• Image-based visual hulls

Page 22: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Voxel based

– First the object space is split up into a 3D grid of voxels.

– Each voxel is intersected with each silhouette volume.

– Only voxels that lie inside all silhouette volumes remain part of the final shape.

Page 23: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Visual hull as voxel grid

• Identify 3D region using voxel carving

– does a given voxel project inside all silhouettes?

• pros: simplicity

• cons: bad precision/computation time tradeoff

?

? ?

Page 24: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Classical voxel grid improvement: octrees

• Same principle, but refinement through space subdivision

[Szeliski TR 90’]

Page 25: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections Tarini et al., 2002

– The object space is again split up into a 3D grid.

– The grid used is made of 3 sets of rays, rather than voxels.

– Rays are aligned with the 3 axes, and store points of entry/exit into the volume

– Each silhouette cone can be converted to the marching intersections data structure.

– Then merging them is reduced to 1D intersections along each ray.

M. Tarini et al, Marching intersections, An

efficient Approach to Shape from Silhouette

Page 26: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections - example

Page 27: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections -Concept

•Given a curve

•Select reference grid

•Intersections between curve and horizontal and

vertical lines: MI

•Create look-up-table for each non-empty box

M. Tarini et al,

Marching

intersections, An

efficient Approach

to Shape from

Silhouette

Page 28: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections - Silhouettes

•Convert conoid structures to MI datastructure

•Intersection tested in 2D image: purely 2D

operation

•Intersection of conoids: AND operations on MI

datastructures

Page 29: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections - Silhouettes

Final step: Convert MI datastructure representing

all intersections to triangular mesh

Page 30: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Marching intersections - Silhouettes

Page 31: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Example: Student Project

• Compute visual hull with silhouette images from multiple calibrated cameras

• Compute Silhouette Image

• Volumetric visual hull computation

• Display the result

Page 32: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Algorithms

• Standard voxel based method

• Exact polyhedral methods

• Image-based visual hulls

Page 33: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Exact Polyhedral - example

Page 34: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Exact Polyhedral Methods Wojciech Matusik et al.

– First, silhouette images are converted to polygons. (convex or non-convex, with holes allowed)

– Each edge is back projected to form a 3d polygon.

– Then each polygon is projected onto each image, and intersected with each silhouette in 2D.

– The resulting polygons are assembled to form the polyhedral visual hull

Wojciech Matusik, An Efficient Visual Hull Computation Algorithm

Page 35: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Exact Polyhedral Methods

Wojciech Matusik, An Efficient Visual Hull Computation Algorithm

Page 36: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Wojciech Matusik et al.

Page 37: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

IBVH Results

• Approximately constant computation per pixel per camera

• Parallelizes

• Consistent with input silhouettes

Page 38: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

IBVH Results

Page 39: Shape from Silhouettes Igerig/CS6320-S2013/Materials/CS... · Shape from Silhouettes I Guido Gerig CS 6320, ... – shape from silhouettes ... • Train an appearance model for each

Image Based Visual Hulls

http://www.youtube.com/watch?v=

Lw9aFaHobao