1/50

Photo-Inspired Model-Driven 3D Object Modeling

Kai Xu1,2 Hanlin Zheng3 Hao (Richard) Zhang2 Daniel Cohen-Or4 Ligang Liu3 Yueshan Xiong1

1National Univ. of Defense Tech. 2Simon Fraser Univ.3Zhejiang Univ. 4Tel-Aviv Univ.

2/50

Photo-Inspired Model-Driven 3D Object Modeling

1National Univ. of Defense Tech. 2Simon Fraser Univ.3Zhejiang Univ. 4Tel-Aviv Univ.

Kai Xu1,2 Hanlin Zheng3 Hao (Richard) Zhang2 Daniel Cohen-Or4 Ligang Liu3 Yueshan Xiong1

3/50

3D content creation

Inspiration?

Inspiration a readily usable digital 3D model

4/50

Inspiration = real-world data

[Nan et al., 2010]

Realistic reconstruction

5/50

Creation of novel 3D shapes

Inspiration = design concept, mental picture, …

sketch

Creative inspiration

6/50

3D content creation is hard

2D-to-3D: an ill-posed problem: Shape from shading, sketch-based modeling, …

3D creation from scratch is hard: job for skilled artists

One of the most fundamental problems in graphics

Jim Kajiya’s Award Talk: Geometric modeling still hard!

7/50

Usable 3D content even harder

Models created are meant for subsequent use Editing, modification, generation of new models …

iWires [Gal et al. 2009]

8/50

Usable 3D content even harder

Creation of readily usable models Part information (segmentation) or characteristic curves (wires)

Structural relations between parts/wires

Correspondence among relevant models: co-segmentation, etc.

Component-wise controllers[Zheng et al. 2011]iWires [Gal et al. 2011]

Co-segmentation[Xu et al. 2010]

Hard shape analysis problems, esp. for man-made models

9/50

Key: model reuse

Reuse pre-existing 3D models

Particularly their pre-analyzed structures

Segmentation benchmarks[Chen et al. 2009, Kalogerakis et al. 2010]

Not only serve to evaluate, but also to create

10/50

Key: model reuse

Two primary modes of reuse: New creation via part re-composition

Modeling by example[Funkhouser et al. 2004]

Data-driven part suggestions[Chaudhuri et al. , 2010 & 2011]

Pre-existing structural information can be lost …

11/50

Key: model reuse

[Xu et al. 2010] [Kraevoy et al. 2009]

Varying part scalesAppearance-driven,

organic shapes

Two primary modes of reuse: New creation via part composition

New creation as a variation of existing models, e.g, a warp or deformation

12/50

Model-driven 3D content creation

Generate variations from a pre-analyzed candidate model set

13/50

Photo-inspired 3D modeling

Photographs: one of the richest source of modeling inspiration

On-line photographs, often only in single-views

14/50

Key features

Single photo coherent and structure-preserving 3D model

Photograph Retrieved candidate 3D model

3D creation

15/50

Creation readily usable

Subsequent model editing

16/50

Overview

17/50

Pre-analyzed candidate model set

Part correspondence [Xu et al. 2010]

Input model set Models in part correspondence

18/50

Pre-analyzed candidate model set

Component-wise controllers [Zheng et al. 2011] Controller primitives: cuboids and generalized cylinders

Interrelations: symmetry, proximity, etc.

19/50

Overview of our method

Step 1:Model-driven image-space object analysis

20/50

Model-driven image-space object analysis

Retrieval of representative model

Model-driven labeled segmentation

Graph cut segmentation

21/50

Overview of our method

Step 2:Candidate model retrieval

22/50

Candidate model retrieval

Query

Top 5 retrieved results

whole shape Light Field Descriptor

23/50

Candidate model retrieval

Query

Top 5 retrieved results

part-level Light Field Descriptor

Candidates may be randomly chosen --- modeling surprise

24/50

Overview of our method

The key step 3:Silhouette-driven deformation

25/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

Four sub-steps:

26/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

27/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

28/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

29/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

30/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

31/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

32/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

Before optimization After optimization

33/50

Silhouette-driven deformation

Silhouette correspondence

Initial controller reconstruction

Controller optimization

Underlying geometry

deformation

Before optimization After optimization Final geometry

34/50

Structure optimization at work

Initial controller reconstruction

Front-view

35/50

Structure optimization at work

Individual controller symmetry

Inter-controller symmetry

proximity constraintsInitial

configuration

iterative

Final configuration

36/50

Results

Candidate not always chosen as best so as to show the power of silhouette-driven warp

37/50

Tables

38/50

Lamps

39/50

The Google Chair Challenge

40/50

The Google Chair Challenge

41/50

The Google Chair Challenge

42/50

The Google Chair Challenge

?

43/50

The Google Chair Challenge

44/50

The Google Chair Challenge

45/50

Conclusion and limitations

Photo-inspired model-driven 3D content creation Utilizes two rich sources: photo inspirations and pre-analyzed 3D models

Structure-driven image analysis and silhouette-based deformation

Readily usable: variation less “intrusive” to retain pre-analyzed structures

Limitations: Variation does not create new structures, e.g., new connectivity or topology

Modeling at the coarse level, refined modeling to follow

Resemblance to photographed object is only through silhouette matching

Conflicts may occur between constraints to be enforced

46/50

Random candidate

Conflicting constraints

47/50

Future work

Photo-inspired model deformation only a start

Other inspirations for 3D content creation Sketch-inspired model variation

Interior feature curves

48/50

Future work

Photo-inspired model deformation only a start

Other inspirations for 3D content creation Sketch-inspired model variation

Interior feature curves

Bigger questions A common high-level structural representation, for individual or a set? −−−

low-level mesh reps seem like the wrong choice for modeling

Easy creation of new structures (topology) that well retain pre-analyzed structures −−− from geometry creation to structure creation

49/50

Acknowledgement

Anonymous reviewers

The authors of [Zheng et al. EG 2011]

Aiping Wang from NUDT

Grants NSERC (No. 611370)

Doctoral Program of Higher Education of China (No. 20104307110003)

the Israel Science Foundation

National Natural Science Foundation of China (61070071)

973 National Key Basic Research Foundation of China (No. 2009CB320801).

50/50

Thank you!

Project page: http://www.kevinkaixu.net/k/projects/photo-inspired.html