Lecture 02 yasutaka furukawa - 3 d reconstruction with priors

ICVSS 20143D Reconstruction with Priors

Yasutaka Furukawa Washington University in St. Louis

What are priors?

Outline

• Why priors?

• Prior enforcement through MRF

• Prior enforcement through primitives

• Prior enforcement through shortest-path

Brief Introduction of Multi-View Stereo (MVS)

Camera pose 3d reconstruction

Input: Calibrated photographs

Output: 3D geometry

MVS PrincipleFeature matching

1995 1999 1998 1998

2004 2005 2005 2007Computer Vision: Algorithms and Applications [Richard Szeliski]

A decade ago... A Theory of Space by Space Carving

[ Kutulakos and Seitz, 1999 ]

Reconstructions w/ and w/o color

Volumetric graph cutsby Vogiatzis, Torr, and Cipolla

CVPR 2005

1. This is called Haniwa. 2. This is Roberto Cipolla’s Haniwa.

This will be a final quiz!

Results

[ CVPR 2007 ]

Visual Effects

Digital mapping

Apple Maps

Computer Vision in Industry

• Amazon

• Apple

• Facebook

• Google

• Microsoft

• Nokia

Computer Vision in Industry

• Amazon

• Apple

• Facebook

• Google

• Microsoft

• Nokia

Mountain View Seattle Zurich NYC LA …

Google Seattle - 3D Vision Team

• Bundling streetview data (by Sameer Agarwal)

• Photo Tours (by the team)

• Picasa face movie (by Ira Kemelmacher-Shlizerman and Rahul Garg)

• Lens Blur (by Carlos Hernandez)

Steve Seitz, Sameer Agarwal, Carlos Hernandez, David Gallup, Changchang Wu, Li Zhang

Face Movie[Ira Kemelmacher-Shlizerman, Eli Shechtman,

Rahul Garg, Steve Seitz]

Lens Blur[Carlos Hernandez]

Why am I here?

What if no texture?

Noise suppression

[reatedigitalmotion.com]

http://reatedigitalmotion.com

Compression

[bitewallpapers.com]

http://bitewallpapers.com

Why priors?

• Fails in some important cases

• Noise suppression

• Compression

Why priors?



• Compression

• Uncanny valley for 3D reconstruction

Uncanny valley

[www.cubo.cc/creepygirl]

http://www.cubo.cc/creepygirl

Uncanny valley for 3D reconstruction

More accurate, but looks worse

Less accurate, but looks better

People like everything about this picture

3D photography hates everythingabout this picture

Uncanny Valley for 3D Reconstruction

Ten Things You Should Know AboutLarge Scale 3D Reconstruction

(3DV 2013)

Martin Byröd Apple Maps

Why priors?



• Compression

• Uncanny valley for 3D reconstruction

Outline• Why priors?




Prior enforcement through MRF

Piecewise planar enforcement through MRF

Motivations

[Furukawa and Ponce, 2007]

Enforce Prior in MVS• Manhattan-world assumption

• Planarity

• Orthogonality

8.0

8.2

8.5

8.9

9.3

camera center image

screen

Depthmap Problem

8.0

8.2

8.5

8.9

9.3

camera center image

screen

Depthmap Problem

Discretized depth values{ 0.1, 0.2, …, 8.0, 8.1, … 9.8, 9.9}

Standard DepthmapMarkov Random Field (MRF)

A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]

A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]

Markov Random Field (MRF)


7 7 6 0 6 7

pq


0

pq


1

pq


2

pq


4

pq


0

pq



Graph-cuts (alpha-expansion)gives you very good solutions

Prefers front-parallel surfaces


We want piecewise planar

Piecewise planarity from MRF1. Advanced MRF and optimization

• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award

2. Integrate with top-down (primitive) approach

• Manhattan World Stereo [Furukawa et al., CVPR 2009]

• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]

• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]

• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]

Advanced MRF for Depthmap Estimation

Global Stereo Reconstruction under Second Order Smoothness Priors[Woodford et al., CVPR 2008] Best Paper Award

Reference image

Ground truth

Standard MRF

Standard MRF

p q

Standard MRF

p

q

Standard MRF

p q

Standard MRF

p q

MRF with a triple clique

p q r

Standard MRF

p q


p q r

0 = |3 + 5 - 2 x 4|

Standard MRF

p q


p

q

r

Optimization becomes a challenge

Standard MRF (submodular)

MRF with a triple clique (non-submodular)

• Graph-cuts(alpha-expansion)

• Belief propagation(TRW)

• QPBO • Belief propagation

(TRW?)

Optimization becomes a challenge

Standard MRF (submodular)

MRF with a triple clique (non-submodular)

• Graph-cuts(alpha-expansion)

• Belief propagation(TRW)

• QPBO • Belief propagation

(TRW?)

Fusion moves…

Experimental resultsReference image Neighboring image

Output depthmap

[Woodford et al.]

Comparative experimentReference image Ground truth

Standard MRF MRF with a triple clique

[Woodford et al.]

How to enforce piecewise planar1. Advanced MRF and optimization







Standard Depthmap

8.0

8.2

8.5

8.9

9.3

camera center image

screen

Planemap

[Furukawa and Ponce, 2007]

1. Extract Manhattandirections

2. Extract planes

PlanemapPossible planes a

b

c

d

e f g h1. Extract Manhattan

directions 2. Extract planes

PlanemapPossible plane ids a

b

c

d

e f g h

e

e

e

e

e

1. Extract Manhattandirections

2. Extract planes

Depthmap to PlanemapDepthmap ( )

Planemap ( )

Planemap MRF

Planemap MRF

p

Planemap MRF

p

7 6 0 6 7

Planemap MRF

p

10 3 10

Planemap MRF

0

pq

Planemap MRF

2

pq

Planemap MRF

3

pq

Planemap MRF

8

pq

Comparison

Standard method Manha9an Planemap

Kitchen -‐ 22 images

house -‐ 148 images

gallery -‐ 492 images

Reconstruction Results

20 seconds break

Relaxing Mahnattan

Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]

• Use sparse lines + sparse points to detect planes

• MRF + Graph-cuts

Relaxing Manhattan

[Sinha et al.]

Enable Curved Surfaces• Building reconstruction from a top down view

Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]

Enable Curved Surfaces[Zebedin et al.]

Piecewise Planar and Non-‐Planar Stereo for Urban Scene Reconstruction

David Gallup Jan-‐Michael Frahm Marc Pollefeys

University of North Carolina ETH Zurich

3D Reconstruction from Video

Street-‐Side Video Real-‐Time Stereo

106

Idea

3D ReconstructionVideo Frames

107

Enforce planarity where it looks like a building

First step: Planar and Non-‐Planar Stereo

108

Video

Real-‐Time Stereo Plane Detection Planemap (w/ non-‐planar))

Labels =

planes non-‐plane discard

Second step: Appearance Prior

109

Video Frames

Classification• divide image into regular grid • RGB, HSV, hue histogram, edge orientation histogram • 5000 human-‐labeled examples (5 images) • k-‐nearest-‐neighbor classifier

non-‐planar planar110

Hoiem et al. 2005, Xiao et al. 2009

AlgorithmVideo

Real-‐Time Stereo

Planar/Non-‐Planar Classification

Plane Detection

Planar and Non-‐Planar MRF

111

non-‐planar planar

Effect of Classifier

112

Without image prior With image prior

non-‐planar planar

Results

113

Video Frame Plane Detection

Planar Classification Reconstruction

non-‐planar

planar

Results

114

Summary: Shape priorsin Depthmap MVS

MRF with a triple term

Manhattan planemap Planemap

Planemap w/ curved surfaces Mix of planemap and depthmap

Outline

• Why priors?




Priors through primitives for large-scale indoor modeling

“Reconstructing the World’s Museums” [Xiao and Furukawa, 2012]

(Best Student Paper Award)

Technical Contributions

• Architectural shape priors through primitives

• Single consistent 3D model

Technical Contributions• Architectural shape priors through primitives

• Single consistent 3D model

Inverse Constructive Solid Geometry (CSG)



• Architectural shape priors through primitives!

• Single consistent 3D model (real merging)

+



• Architectural shape priors through primitives

• Single consistent 3D model (real merging)

+

+

-

+

+

-

CGAL

• From CSG to a mesh

Algorithm

Cut into Slices

gravityside view

Cut into Slices

point count

Cut into Slices

2D CSG Reconstruction

Bottom-up approach

point

2D line

2D rectangle

2D CSG


point à line


point à line

line à rectangle

From 4 line segments


point à line

line à rectanglerectangle à 2D CSG


Explain the data • Laser points • Free space

!


Repeat in each slice

Bottom-up approach

point

2D line

2D rectangle

2D CSG

Bottom-up approach

point

2D line

2D rectangle

point

2D line

2D rectangle

3D rectangle

3D CSG

2D CSG to 3D CSG

2D CSG

Rectangle primitive

2D CSG to 3D CSG

1. Generate primitives (cuboids)

2D CSG

Rectangle primitive

2D CSG to 3D CSG

3D CSG Reconstruction1. Generate primitives (cuboids)2. Build a 3D CSG

Algorithm on Run

Last Step

1. Remove Ceiling 2. Texture Mapping

Challenging Navigation

Frick Collection Gallery (New York City)

Goal

Start

Statistics

# of sub problems

Ground Ground+Aerial

Outdoor

Indoor

Ground vs. Aerial à Ground+Aerial

Google Streetview Google MapsGL

Furukawa et al.

Aerial

Google/Bing/NASA …

This paper This paper

So, museum recons. from

very high-end devices

“museums”

“restaurants”

“grocery stores”

{# of museums} <<{# of restaurants} + {# of grocery stores} +{# of clinic} + ...

Millions of small/medium-scale businesses

• Image-based for scalable deployment(especially for emerging markets)

• Compact model(for browser on low-end PCs)

Millions of small/medium-scale businesses

• Image-based for scalable deployment(especially for emerging markets)

• Compact mesh model(for browser on low-end PCs)

[ Cabral and Furukawa, 2014 ]

Outline

• Why priors?




2D outline reconstruction

Why priors?• Only reconstruct planar outline

• Texture mapping looks better

• Challenge is how to IGNORE clutter

Pipeline (standard)

1. Panorama images 2. MVS points

3. Point evidence 4. Free-space evidence

5. 2D room outline 6. 3D model

2D room outline reconstruction3. Point evidence 4. Free-space evidence

Outline should pass through this point

Threshold

2D room outline reconstruction4. Free-space evidence3. Point evidence

5. Shortest path formulation

Standard way

2D room outline reconstruction5. Shortest path formulation

2D room outline reconstruction5. Shortest path formulationModel complexity penalty

added to each edge

2D room outline reconstruction5. Shortest path formulation

2D room outline reconstruction

Standard

Ours

Features

• Regularize on {# of line segments}

• Piecewise planarity enforcement

• With Dynamic Programming,exactly control the number of vertices

13 vertices

Exact complexity control


15 vertices


17 vertices


19 vertices


23 vertices


27 vertices

Exact complexity control13 verts. 15 verts. 17 verts.

19 verts. 23 verts.

. . . . . .

MVS points Volumetric graph-cuts

Ours

Summary• Why priors?




Questions

A. Who owns the Haniwa in the volumetric graph-cuts paper?

1. Yasutaka Furukawa.

2. Roberto Cipolla.

3. British Museum.

B. What is Uncanny valley for 3D reconstruction?

1. As the number of authors in a paper goes up, the model quality goes down.

2. As the face realism goes up, the model looks creepier.

3. As the model accuracy goes up, the rendering quality goes down.

C. How can one enforce piecewise planarity with a standard MRF formulation easily?

1. Use super-pixel segmentation.

2. Use primitive detection.

3. Use object recognition.

Lunch Time!

Lecture 02 yasutaka furukawa - 3 d reconstruction with priors

Software

Transcript of Lecture 02 yasutaka furukawa - 3 d reconstruction with priors