Structure Recovery by Part Assembly

Chao-Hui Shen1 Hongbo Fu2 Kang Chen1 Shi-Min Hu1

1Tsinghua University 2City University of Hong Kong

Background

• Consumer level scanning devices• Capture both RGB and depth• Reconstruction is challenging

– Low resolution– Noise– Missing data– …

Example-based Scan Completion

• Global-to-local and top-down [Kraevoy and Sheffer 2005; Pauly et al. 2005]

• Rely on the availability of suitable template model• However …

No suitable model!shape retrieval

Assembly-based 3D Modeling

• Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011]– Retrieve suitable parts to match user intent– Aim to support open-ended 3D modeling– Quite different goal from ours

• Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] – Result in database that grows exponentially– Significantly enlarge the existing database– But make storage and retrieval challenging

Our solution: Recover the Structure by Part Assembly• Structure recovery instead of geometry reconstruction• Do NOT prepare a large database• Retrieve and assemble suitable parts on the fly

Problem Setup

Input

Point cloud + Image (Single view)

Pre-segmented Repository Models (Parts + Labels)

……

Goal: Recover high-level structure

Assembly close to geometry

Output

……

Session: Acquiring and Synthesizing Indoor Scenes

An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012]

A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012]

Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012]

• Directly searching is computationally prohibitive

• Need a quick way to explore meaningful structures guided by:– Spatial layout of the parts in the repository models– Acquired data

Observations

Observations

• Complementary characteristics of point cloud & image

3D, more accurate cues for geometry & structure

Incomplete and noisy

Lack depth information

Capture the complete object

Algorithm Overview

Candidate Parts Selection Structure Composition Part Conjoining

……

Algorithm Overview

Candidate Parts Selection Structure Composition Part Conjoining

……

Candidate Parts Selection

• Goal: select a small set of candidates for each category • Achieved by retrieving parts that fit well some regions

Straightforward Solution

• Search for the best-fit parts over the entire domain– Disregards the semantics associated with each part and

the interaction between different parts

Unlikely to produce good results!X X X

X XX X X

Key Fact

• Man-made objects lie in a low dimensional space– Defined with respect to the relative sizes and positions of

shape parts [Ovsjanikov et al. 2011]

• Employ 3D repository model as a global context– Globally align the models with the input scan first

Search in a 3D offset window around the part

Part Matching Scheme

(part contour)

(2D field)

Geometric fidelity score

Geometric contribution score

3D 2Dedgemap

Total matching score

¿{𝑪𝒑 (𝒊 , 𝒋 )=𝟏 }

3D offset window

Candidate Parts

• Select top K parts with highest score for each category

Seat

Back

Arm

Front leg

……

……

……

……

……

…… …… …… …… …… ……

Algorithm Overview

Candidate Parts Selection Structure Composition Part Conjoining

……

Structure Composition

• Goal: compose the underlying structure by identifying a subset of candidate parts

Constraints for Promising Compositions

Geometric fidelity Proximity Overlap

having high score no isolated parts minimized intersection

Search and Evaluate

• Search for promising compositions under constraints

• Globally Evaluate the compositions

average geometry fidelity of parts total geometry fidelity

total geometry contribution

……optimal composition

Algorithm Overview

Candidate Parts Selection Structure Composition Part Conjoining

……

Part Conjoining

• Problem: the parts are loosely placed together• Goal: generate a consistent & complete model

Identification of Contact Points

• Refer to their parent models [Jain et al. 2012]

Matching of Contact Points

• Greedily match nearby contact points• Generate auxiliary contact points when necessary

auxiliary contact points

𝒑𝒎𝒊𝒌

𝒑𝒏𝒋𝒌

i j

identity scale

Global Optimization

transformed contact points

• Adjust the sizes {} and positions {} of parts• Make matched point as close as possible• Contact enforcement

• Shape preserving

• Global optimization

Results: Chairs

• 70 repository models, 11 part categories

Results: Tables

• 61 repository models, 4 part categories

Results: Bicycles

• 38 repository models, 9 part categories

Results: Airplanes

• 70 repository models, 6 part categories

Results: Creating New Structures

Results: Impact of Dataset

input data

Randomly picking some repository models

Summary

• A bottom-up structure recovery approach– Effectively reuse limited repository models– Automatically compose new structure– Handle single-view inputs by the Kinect system

• Future work– Multi-view inputs– Include style/functional constraints– Recover Indoor scenes

Thank you!

Project Page: http://cg.cs.tsinghua.edu.cn/StructureRecovery