Procedural Modeling of Architectures towards 3D Reconstruction

Nikos ParagiosEcole Centrale Paris / INRIA Saclay Ile-de-France

Joint Work: P. Koutsourakis, L. Simon, O. Teboul & N. Komodakis

Outline• The language of Architecture

• Automatic generation of buildings

• Toward single image-based reconstruction

• Discrete MRFs, LP and Duality

04/11/0810/04/08

Part I:Shape Grammars

3D reconstruction• Emerging technology in information society– Post-production, games, navigation

• Current Methods: image based– Not scalable, computational expensive, low

level,

04/11/0810/04/08

The Geometry of Architecture• Very complex geometry…• …but deeply structured due to:

– Laws– Practical constraints– Economical constraints– Aesthetic motivation

• Each architectural style is constrained differently. But buildings of the same style obey the same rules.

04/11/0810/04/08

Simple example of structure

04/11/0810/04/08

Architecture as a large scale Lego• Architecture is made of repetitions of basic

elements, place holders or real elements.– Facades, floors, windows, balconies, doors…

• A specific style can be describe by :– The specification of the repetitions (geometry)– The nature of the elements (semantics)

04/11/0810/04/08

Shape Grammars• Procedural modeling is a set of techniques for

creating 3D models from a set of rules.• Traditional 3D modeling can be very time

consuming for large models. • Large scale architectural models are especially

suited for this kind of techniques.• There is a strong analogy with classical string

grammars.

04/11/0810/04/08

Basic Shapes and Scopes• The Basic Shapes are for Shape Grammars what

Symbols are for String Grammars• A Basic Shape is composed of

– A Semantic– A fixed geometry– Appearance attributes

• A Scope provides the orientation, scaling, position of a basic shape

04/11/0810/04/08

Shape

04/11/0810/04/08

• The shape is a hierarchical representation (a tree) of the building.

• Each node holds a basic shape with an associated scope.

Rules : Interacting with shapes• A rule can be described as:

precondition : LHS RHS

• Given a precondition on the context, a LHS shape is replaced by the RHS shape. From the hierarchical point of view some children are added to the node LHS, increasing the depth of the tree.

• Rules are built from operators

04/11/0810/04/08

The operators of the grammar

• Except the roof operators (which rely on computational geometry) the operators are fairly simple.– Transformation : rotation,translation, scaling– Split– Repeat– Mirror– Component Split– Roof operators : hipped, mansard

04/11/0810/04/08

Operators continued• Splits subdivide a scope along an axis• Component split decomposes a mesh into faces• Roof operators are based on Weighted Straight

Skeleton algorithm

04/11/0810/04/08

Step by step generation

04/11/0810/04/08

Generating : from details to large scale

04/11/0810/04/08

04/11/0810/04/08

04/11/0810/04/08

Demos• Video\ComplexBuilding.ogg• video\roofsAndTexture.ogg

04/11/0810/04/08

Grammar-based reconstruction

• A building is not described by a geometrical attributes anymore

• Building = sequence of rules

• Goal : optimize the sequence of rules that best explains a given facade image

04/11/0810/04/08

Model Quality Evaluation• Assuming that the camera has been weakly

calibrated, the elements of the resulting model can be reprojected on the image.

• The score of the model can be defined in terms of a distance between the observed regions and the known semantics.

04/11/0810/04/08

Part II:MRF optimization viathe primal-dual schema

The MRF optimization problem vertices G = set of objects

edges E = object relationshipsset L = discrete set of labels

Vp(xp) = cost of assigning label xp to vertex p(also called single node potential)

Vpq(xp,xq) = cost of assigning labels (xp,xq) to neighboring vertices (p,q) (also called pairwise potential)

Find labels that minimize the MRF energy (i.e.,

the sum of all potentials):

MRF hardness

MRF pairwise potential

MRF hardness

linear

exact global optimum

arbitrary

local optimum

metric

global optimum approximation

Move right in the horizontal axis,

But we want to be able to do that efficiently, i.e. fast

and remain low in the vertical axis (i.e., still be able to provide approximately optimal solutions)

The primal-dual schema Say we seek an optimal solution x* to the following

integer program (this is our primal problem):

(NP-hard problem)

To find an approximate solution, we first relax the integrality constraints to get a primal & a dual linear program:

primal LP: dual LP:

The primal-dual schema Goal: find integral-primal solution x, feasible dual solution y

such that their primal-dual costs are “close enough”, e.g.,

Tb y Tc x

primal cost of solution x

primal cost of solution x

dual cost of solution y

dual cost of solution y

*Tc x

cost of optimal integral solution x*

cost of optimal integral solution x*

*fT

T

c x

b y

**f

T

T

c x

c x

Then x is an f*-approximation to optimal solution x*

Then x is an f*-approximation to optimal solution x*

(only one label assigned per vertex)

enforce consistency between variables xp,a, xq,b and variable xpq,ab

The primal-dual schema for MRFs

Binaryvariables

xp,a=1 label a is assigned to node p

xpq,ab=1 labels a, b are assigned to nodes p, q

xp,a=1 label a is assigned to node p

xpq,ab=1 labels a, b are assigned to nodes p, q

The primal-dual schema for MRFs During the PD schema for MRFs, it turns out that:

each update of primal and dual

variables

each update of primal and dual

variables

solving max-flow in appropriately

constructed graph

solving max-flow in appropriately

constructed graph

Max-flow graph defined from current primal-dual pair (xk,yk) (xk,yk) defines connectivity of max-flow graph (xk,yk) defines capacities of max-flow graph

Max-flow graph is thus continuously updated

Resulting flows tell us how to update both: the dual variables, as well as the primal variables

for each iteration of primal-dual schema

primal-dual framework

primal-dual framework

Handles wide class of MRFsHandles wide class of MRFs

Approximatelyoptimal solutions

Approximatelyoptimal solutions

Theoretical guarantees AND tight certificates

per instance

Theoretical guarantees AND tight certificates

per instance

Significant speed-up

for static MRFs

Significant speed-up

for static MRFs

Significant speed-up

for dynamic MRFs

Significant speed-up

for dynamic MRFs

- New theorems- New insights into

existing techniques- New view on MRFs

- New theorems- New insights into

existing techniques- New view on MRFs

04/11/0810/04/08

Future Work• Efficient optimization framework, through neuro-

dynamic programming like methods• Image-based grammar learning• Augmentation of the expressiveness of the

grammar• Large scale representation of forests of buildings

04/11/0810/04/08

Questions ?

04/11/0810/04/08