From Images to Voxels Steve Seitz Carnegie Mellon University University of Washington seitz SIGGRAPH...

From Images to VoxelsFrom Images to Voxels

Steve SeitzSteve Seitz

Carnegie Mellon UniversityCarnegie Mellon University

University of WashingtonUniversity of Washingtonhttp://www.cs.cmu.edu/~seitzhttp://www.cs.cmu.edu/~seitz

SIGGRAPH 2000 Course onSIGGRAPH 2000 Course on3D Photography3D Photography

3D Reconstruction from Calibrated Images3D Reconstruction from Calibrated Images

Scene VolumeScene Volume

VV

Input ImagesInput Images

(Calibrated)(Calibrated)

Goal: Goal: Determine transparency, radiance of points in VDetermine transparency, radiance of points in V

Discrete Formulation: Voxel ColoringDiscrete Formulation: Voxel Coloring

Discretized Discretized


Input ImagesInput Images

(Calibrated)(Calibrated)

Goal: Goal: Assign RGBA values to voxels in VAssign RGBA values to voxels in Vphoto-consistentphoto-consistent with images with images

Complexity and ComputabilityComplexity and Computability

Discretized Discretized


N voxelsN voxels

C colorsC colors

33

All Scenes (CN3)Photo-Consistent

Scenes

TrueScene

IssuesIssues

Theoretical QuestionsTheoretical Questions• Identify class of Identify class of allall photo-consistent scenes photo-consistent scenes

Practical QuestionsPractical Questions• How do we compute photo-consistent models?How do we compute photo-consistent models?

1. C=2 (silhouettes)1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]Volume intersection [Martin 81, Szeliski 93]

2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case3. General Case• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]

Voxel Coloring SolutionsVoxel Coloring Solutions

Reconstruction from Silhouettes (C = 2)Reconstruction from Silhouettes (C = 2)

BinaryBinary ImagesImages

Approach: Approach: • BackprojectBackproject each silhouette each silhouette

• Intersect backprojected volumesIntersect backprojected volumes

Volume IntersectionVolume Intersection

Reconstruction Contains the True SceneReconstruction Contains the True Scene• But is generally not the same But is generally not the same • In the limit get In the limit get visual hullvisual hull

> Complement of all lines that don’t intersect SComplement of all lines that don’t intersect S

Voxel Algorithm for Volume IntersectionVoxel Algorithm for Volume Intersection

Color voxel black if on silhouette in every imageColor voxel black if on silhouette in every image• O(MNO(MN33), for M images, N), for M images, N33 voxels voxels

• Don’t have to search 2Don’t have to search 2NN33 possible scenes! possible scenes!

Properties of Volume IntersectionProperties of Volume Intersection

ProsPros• Easy to implement, fastEasy to implement, fast

• Accelerated via octrees [Szeliski 1993]Accelerated via octrees [Szeliski 1993]

ConsCons• No concavitiesNo concavities

• Reconstruction is not photo-consistentReconstruction is not photo-consistent

• Requires identification of silhouettesRequires identification of silhouettes

1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate3. Color if consistent3. Color if consistent

Voxel Coloring ApproachVoxel Coloring Approach

Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?

The Global Visibility ProblemThe Global Visibility Problem

Inverse VisibilityInverse Visibilityknown imagesknown images

Unknown SceneUnknown Scene

Which points are visible in which images?Which points are visible in which images?

Known SceneKnown Scene

Forward VisibilityForward Visibilityknown sceneknown scene

LayersLayers

Depth Ordering: visit occluders first!Depth Ordering: visit occluders first!

SceneScene

TraversalTraversal

Condition: Condition: depth order is depth order is view-independentview-independent

Panoramic Depth OrderingPanoramic Depth Ordering

• Cameras oriented in many different directionsCameras oriented in many different directions

• Planar depth ordering does not applyPlanar depth ordering does not apply

Panoramic Depth OrderingPanoramic Depth Ordering

Layers radiate outwards from camerasLayers radiate outwards from cameras

Panoramic LayeringPanoramic Layering

Layers radiate outwards from camerasLayers radiate outwards from cameras

Compatible Camera ConfigurationsCompatible Camera Configurations

Depth-Order ConstraintDepth-Order Constraint• Scene outside convex hull of camera centersScene outside convex hull of camera centers

Outward-LookingOutward-Lookingcameras inside scenecameras inside scene

Inward-LookingInward-Lookingcameras above scenecameras above scene

Calibrated Image AcquisitionCalibrated Image Acquisition

Calibrated TurntableCalibrated Turntable360° rotation (21 images)360° rotation (21 images)

Selected Dinosaur ImagesSelected Dinosaur Images

Selected Flower ImagesSelected Flower Images

Voxel Coloring Results (Video)Voxel Coloring Results (Video)

Dinosaur ReconstructionDinosaur Reconstruction72 K 72 K voxels coloredvoxels colored7.6 M 7.6 M voxels testedvoxels tested7 min. 7 min. to compute to compute on a 250MHz SGIon a 250MHz SGI

Flower ReconstructionFlower Reconstruction70 K 70 K voxels coloredvoxels colored7.6 M 7.6 M voxels testedvoxels tested7 min. 7 min. to compute to compute on a 250MHz SGIon a 250MHz SGI

Limitations of Depth OrderingLimitations of Depth Ordering

A view-independent depth order may not existA view-independent depth order may not exist

p q

Need more powerful general-case algorithmsNeed more powerful general-case algorithms• Unconstrained camera positionsUnconstrained camera positions

• Unconstrained scene geometry/topologyUnconstrained scene geometry/topology

Space Carving AlgorithmSpace Carving Algorithm


Image 1 Image N

…...

• Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene

• Repeat until convergenceRepeat until convergence

• Choose a voxel on the current surfaceChoose a voxel on the current surface

• Carve if not photo-consistentCarve if not photo-consistent

• Project to visible input imagesProject to visible input images

ConvergenceConvergence

Consistency PropertyConsistency Property• The resulting shape is photo-consistentThe resulting shape is photo-consistent

> all inconsistent points are removedall inconsistent points are removed

Convergence PropertyConvergence Property• Carving converges to a non-empty shapeCarving converges to a non-empty shape

> a point on the true scene is a point on the true scene is nevernever removed removed

V’

V

p

What is Computable?What is Computable?

The The Photo HullPhoto Hull is the UNION of all photo-consistent scenes in V is the UNION of all photo-consistent scenes in V

• It is a photo-consistent scene reconstructionIt is a photo-consistent scene reconstruction

• Tightest possible bound on the true sceneTightest possible bound on the true scene

• Computable via provable Computable via provable Space Carving AlgorithmSpace Carving Algorithm

True SceneTrue Scene

VV

Photo HullPhoto Hull

VV


The Basic Algorithm is UnwieldyThe Basic Algorithm is Unwieldy• Complex update procedureComplex update procedure

Alternative: Multi-Pass Plane SweepAlternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardwareEfficient, can use texture-mapping hardware

• Converges quickly in practiceConverges quickly in practice

• Easy to implementEasy to implement

Results Algorithm


• Step 1:Step 1: Initialize V to volume containing true scene Initialize V to volume containing true scene

• Step 2:Step 2: For every voxel on surface of V For every voxel on surface of V

> test test photo-consistencyphoto-consistency of voxel of voxel

> if voxel is inconsistent, carve itif voxel is inconsistent, carve it

• Step 3:Step 3: Repeat Step 2 until all voxels consistent Repeat Step 2 until all voxels consistent

Convergence: Convergence: • AlwaysAlways converges to a photo-consistent model (when converges to a photo-consistent model (when

all assumptions are met)all assumptions are met)

• Good results on difficult real-world scenes Good results on difficult real-world scenes

Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

• Consider cameras on only one side of planeConsider cameras on only one side of plane


True Scene Reconstruction

Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

• Consider cameras on only one side of planeConsider cameras on only one side of plane


Other ApproachesOther Approaches

Level-Set Methods Level-Set Methods [Faugeras & Keriven 1998][Faugeras & Keriven 1998]• Evolve implicit function by solving PDE’sEvolve implicit function by solving PDE’s

Transparency and Matting Transparency and Matting [Szeliski & Golland 1998][Szeliski & Golland 1998]• Compute voxels with alpha-channelCompute voxels with alpha-channel

Max Flow/Min Cut Max Flow/Min Cut [Roy & Cox 1998][Roy & Cox 1998]• Graph theoretic formulationGraph theoretic formulation

Mesh-Based Stereo Mesh-Based Stereo [Fua & Leclerc 95][Fua & Leclerc 95]• Mesh-based but similar consistency formulationMesh-based but similar consistency formulation

Virtualized Reality Virtualized Reality [Narayan, Rander, Kanade 1998][Narayan, Rander, Kanade 1998]• Perform stereo 3 images at a time, merge resultsPerform stereo 3 images at a time, merge results

Advantages of VoxelsAdvantages of Voxels• Non-parametricNon-parametric

> can model arbitrary geometrycan model arbitrary geometry

> can model arbitrary topologycan model arbitrary topology

• Good reconstruction algorithmsGood reconstruction algorithms

• Good rendering algorithms (splatting, LDI)Good rendering algorithms (splatting, LDI)

DisadvantagesDisadvantages• Expensive to process hi-res voxel gridsExpensive to process hi-res voxel grids

• Large number of parametersLarge number of parameters

> Simple scenes (e.g., planes) require lots of voxelsSimple scenes (e.g., planes) require lots of voxels

ConclusionsConclusions

Volume IntersectionVolume Intersection• Martin & Aggarwal, “Volumetric description of objects from multiple views”, Martin & Aggarwal, “Volumetric description of objects from multiple views”,

Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.

• Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.

Voxel Coloring and Space CarvingVoxel Coloring and Space Carving• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc.

Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.

• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, pp. 17-24.Vision (ICCV), 1998, pp. 17-24.

• Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. 307-314.307-314.

BibliographyBibliography

Related ReferencesRelated References• Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set

methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.pp. 336-344.

• Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517-524.Conf. on Computer Vision (ICCV), 1998, 517-524.

• Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499.Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499.

• Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35-56.image stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35-56.

• Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10.Stereo”, Proc. ICCV, 1998, pp. 3-10.

BibliographyBibliography

From Images to Voxels Steve Seitz Carnegie Mellon University University of Washington seitz SIGGRAPH...

Documents

Transcript of From Images to Voxels Steve Seitz Carnegie Mellon University University of Washington seitz SIGGRAPH...