Image-based Building Modeling - Princeton...

Image-based Building Modeling

Jianxiong Xiao

Department of Computer Science and Engineering

The Hong Kong University of Science and Technology

M.Phil. Thesis Defense

Jianxiong Xiao (CSE, HKUST) Image-based Building Modeling M.Phil. Thesis Defense 1 / 56

Introduction

Applications

Architecture and preservation, archaeology and anthropology:documenting and measuring older buildings and structures for conservationand preservation, generating 3D models for visualization and view studies,creating elevation drawings of existing structures and recti�ed photographsof façades from photo projects, producing photo-textured 3D models forrealistic walk-bys, surveying existing structures and objects etc.

Film, video, animation and multimedia web:build 3D models to use in animation and rendering programs, model objectsfor Computer Based Training, measure or model sets and locations, performperspective matching to synchronize a CG camera to a real photo, exportrealistic texture maps from original photographs, create life-likephoto-textured models with low polygon counts, etc.

Accident reconstruction:for diagrams and maps of the scene creation, for measurements of distance,crush, and placement, for ortho-photos generation of skid marks and othersurfaces, in order to make accurate measurements and maps quickly andeasily and create archives of crime scene evidence for analysis at a later date.


Introduction

Related Works

Paul Debevec, Camillo Taylor and Jitendra Malik (UC Berkeley)

Seth Teller et al. (MIT)

Next Generation 4-D Distributed Modeling and Visualization of

Battle�eld (ARO MURI)

I Avideh Zakhor et al. (UC Berkeley)I Ulrich Neumann and Suya You (USC)I ...

UrbanScape (DARPA)

I Marc Pollefeys et al. (UNC Chapel Hill)I David Nister et al. (University of Kentucky)

Ioannis Stamos et al. (CUNY)

...


Introduction

Presentation Coverage

J. Xiao and L. Quan. Multiple view semantic segmentation for street

view images. ICCV 2009. Sec. 3.3

J. Xiao, T. Fang, P. Zhao, M. Lhuillier, and L. Quan. Image-based

street-side city modeling. SIGGRAPH ASIA 2009. ACM ToG. Sec. 4

J. Xiao, T. Fang, P. Tan, P. Zhao, E. Ofek, and L. Quan. Image-based

facade modeling. SIGGRAPH ASIA 2008. ACM ToG.

P. Tan, T. Fang, J. Xiao, P. Zhao, and L. Quan. Single image tree

modeling. SIGGRAPH ASIA 2008. ACM ToG.

J. Xiao, J. Chen, D.-Y. Yeung, and L. Quan.Learning two-view stereo

matching. ECCV 2008.

J. Xiao, J. Chen, D.-Y. Yeung, and L. Quan. Structuring visual words

in 3D for arbitrary-view object localization. ECCV 2008.

J. Xiao, J. Wang, P. Tan, and L. Quan. Joint a�nity propagation for

multiple view segmentation. ICCV 2007.Jianxiong Xiao (CSE, HKUST) Image-based Building Modeling M.Phil. Thesis Defense 4 / 56

Introduction

Fast-forwording Preview


Introduction

Outline

1 Introduction

2 Multi-view Segmentation

Unary Likelihood

Multi-view Consistency

Geo-adaptive Training

3 Orthographic Representation

4 Structure Modeling

Joint Structure Analysis

Shape Regularization

Repetitive Pattern Rediscovery

Boundary Regularization

Texture Mapping

5 Experiment and Conclusion


Introduction

Data Capturing

buildingbuilding

street

car

camera

camera mounted on a car


Introduction

Point Matching Between Images


Introduction

Reconstruction: Motion and Structure


Introduction

Over-segmentation


Multi-view Segmentation

Outline

1 Introduction


Unary Likelihood









Texture Mapping



Multi-view Segmentation

Objective

building

ground

sky


Multi-view Segmentation Unary Likelihood

Outline

1 Introduction


Unary Likelihood









Texture Mapping




From Feature To Semantic Meaning

Features:

I 2D features: fAi and fPiI 3D features: fGi

Classi�er: one-vs-all AdaBoost

Pi

(li |fAi , fPi , fGi

)=

exp(H

(li , f

Ai , fPi , fGi

))∑l exp

(H

(l , fAi , fPi , fGi

)) . (1)

where H(l , fAi , fPi , fGi

)is the output of the AdaBoost classi�er for class l .



2D Features

Color and texture

I MedianI DeviationI SkewnessI Kurtosis statistics

Size

Shape

I The ratio of the region area to the perimeter squareI The moment of inertia about the center of the massI The ratio of the area to the bounding rectangle area

Pixel position



2D Features: Pixel Position

skygroundbuildingperson

vehicletreerecycle bin

occurence



3D Features


Multi-view Segmentation Multi-view Consistency

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Objective

building

ground

sky

building

ground

sky



Markov Random Field

The labeling problem is to assign a unique label, li , to each node, pi ∈ V.

The solution, L = {li}, can be obtained by minimizing a Gibbs energy

E (L) = ∑pi∈V

ψi (li )+ρ ∑eij∈E

ψij (li , lj) . (2)

GraphCut-based alpha expansion can be used to obtain a local optimized

label con�guration, L.



Potential Functions

E (L) = ∑pi∈V

ψi (li )+ρ ∑eij∈E

ψij (li , lj) . (3)

Unary potential:

ψi (li ) =− logPi(li |fAi , fPi , fGi

).

Binary potential: edges

I within the same image: segment on the boundariesI between di�erent images: consistency



Random Field Construction



Edges Within The Same Image

ψij (li , lj) = [li 6= lj ] ·g (i , j) , (4)

g (i , j) =1

ζ ‖ci −cj‖2 +1(5)

[li 6= lj ] allows us to capture the gradient information only along the

segmentation boundary.



Edges Between Di�erent Images

ψij (li , lj) = [li 6= lj ] ·λ |Tij |g (i , j) , (6)

Tij = {t = 〈x,(xi ,yi , i) ,(xj ,yj , j) , . . .〉} .


Multi-view Segmentation Geo-adaptive Training

Outline

1 Introduction


Unary Likelihood









Texture Mapping



Multi-view Segmentation Geo-adaptive Training

A�nity Clustering in the Label Pool

Sequence ASequence B

Sequence C


Orthographic Representation

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Overview

Near

Far



Patch Reconstruction

The normal and center of each patch are estimated from the set of 3D

points.

The local coordinate frame is aligned with the three principle

directions.

Standard deviations of all 3D points in three directions.

Normalized standard deviations.

Near

Far




Near

Far



Inverse Orthographic Composition



Robust Depth Accumulation



Robust Texture Accumulation


Structure Modeling Joint Structure Analysis

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Objective



Graph-based Bottom-up Merging

1 One pixel one vertex. 4-neighboring system to construct pairwise

graph.

2 Weight is de�ned on the color distance and normalized depth

di�erence.

3 Sort E by non-decreasing edge weight w .

4 Starting with an initial segmentation in which each vertex vi is in its

own component, the algorithm repeats for each edge eq = (vi ,vj) inorder for the following process:

I If vi and vj are in disjoint components Ci 6= Cj , and w (eq) is smallcompared with the internal di�erence of both those components,w (eq)≤MInt (Ci ,Cj),

I then the two components are merged.



Graph-based Bottom-up Merging

The minimum internal di�erence is de�ned as

MInt (C1,C2) = min(Int (C1)+ τ (C1) , Int (C2)+ τ (C2)) ,

where the internal di�erence of a component C is the largest weight in the

minimum spanning tree of the component

Int (C ) = maxe∈MST (C ,E)

w (e) .



Threshold Function

The di�erence in τ (C ) between two components must be greater than

their internal di�erence for an evidence of a boundary between them.

Favor a rectangular shape:τ (C ) is de�ned by the divergence ϑ (C )between the component C and a rectangle ϑ (C ) = |BC |/ |C |.For small components, Int (C ) is not a good estimate of the local

characteristics of the data. Therefore, we let the threshold function be

adaptive based on the component size,

τ (C ) =

(ρ

|C |

)ϑ(C)

.

τ is large for components that do not �t a rectangle, and two

components with large τ are more likely to be merged. A larger ρ

favors larger components, as we require stronger evidence of a

boundary for smaller components.


Structure Modeling Shape Regularization

Outline

1 Introduction


Unary Likelihood









Texture Mapping



Structure Modeling Shape Regularization

Rectangle Fitting


Structure Modeling Repetitive Pattern Rediscovery

Outline

1 Introduction


Unary Likelihood









Texture Mapping



Structure Modeling Repetitive Pattern Rediscovery

Objective


Structure Modeling Boundary Regularization

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Objective



1D Markov Random Field

012345

l = 3 l = 3 l = 3 l = 3 l = 1 l = 1 l = 1 l = 2 l = 2



Data Cost


Structure Modeling Texture Mapping

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Model Production



Texture Optimization


Experiment and Conclusion

Outline

1 Introduction


Unary Likelihood









Texture Mapping




Building Segmentation



Building Modeling

(a) (b) (c) (d) (e) (f) (a) (b) (c) (d) (e) (f)



Building Modeling



Summary

Improvements over Previous Works in Our Group:

Reconstruction: from 3D points to reliable orthographic depth map

Segmentation: from meaningless clustering to semantic classi�cation

Modeling: from interactive to automatic

SIGGRAPH Asia 2009 Reviewers:

�The paper addresses a di�cult and important problem�. �Although the

system still has limitations and the models show some artifacts, it is a clear

step ahead for the state of the art� and �represents a signi�cant progress

beyond the state of the art�.


Q&A

Q&A

Thank you very much.


Image-based Building Modeling - Princeton...

Documents

Transcript of Image-based Building Modeling - Princeton...