CS Dept, City Univ.1 Low Latency Broadcast in Multi-Rate Wireless Mesh Networks LUO Hongbo.
Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua...
-
Upload
gwendolyn-kennedy -
Category
Documents
-
view
216 -
download
0
Transcript of Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua...
![Page 1: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/1.jpg)
Structure Recovery by Part Assembly
Chao-Hui Shen1 Hongbo Fu2 Kang Chen1 Shi-Min Hu1
1Tsinghua University 2City University of Hong Kong
![Page 2: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/2.jpg)
Background
• Consumer level scanning devices• Capture both RGB and depth• Reconstruction is challenging
– Low resolution– Noise– Missing data– …
![Page 3: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/3.jpg)
Example-based Scan Completion
• Global-to-local and top-down [Kraevoy and Sheffer 2005; Pauly et al. 2005]
• Rely on the availability of suitable template model• However …
No suitable model!shape retrieval
![Page 4: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/4.jpg)
Assembly-based 3D Modeling
• Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011]– Retrieve suitable parts to match user intent– Aim to support open-ended 3D modeling– Quite different goal from ours
• Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] – Result in database that grows exponentially– Significantly enlarge the existing database– But make storage and retrieval challenging
![Page 5: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/5.jpg)
Our solution: Recover the Structure by Part Assembly• Structure recovery instead of geometry reconstruction• Do NOT prepare a large database• Retrieve and assemble suitable parts on the fly
![Page 6: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/6.jpg)
Problem Setup
Input
Point cloud + Image (Single view)
Pre-segmented Repository Models (Parts + Labels)
……
Goal: Recover high-level structure
Assembly close to geometry
Output
……
Session: Acquiring and Synthesizing Indoor Scenes
An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012]
A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012]
Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012]
![Page 7: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/7.jpg)
• Directly searching is computationally prohibitive
• Need a quick way to explore meaningful structures guided by:– Spatial layout of the parts in the repository models– Acquired data
Observations
![Page 8: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/8.jpg)
Observations
• Complementary characteristics of point cloud & image
3D, more accurate cues for geometry & structure
Incomplete and noisy
Lack depth information
Capture the complete object
![Page 9: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/9.jpg)
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
![Page 10: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/10.jpg)
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
![Page 11: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/11.jpg)
Candidate Parts Selection
• Goal: select a small set of candidates for each category • Achieved by retrieving parts that fit well some regions
![Page 12: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/12.jpg)
Straightforward Solution
• Search for the best-fit parts over the entire domain– Disregards the semantics associated with each part and
the interaction between different parts
Unlikely to produce good results!X X X
X XX X X
![Page 13: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/13.jpg)
Key Fact
• Man-made objects lie in a low dimensional space– Defined with respect to the relative sizes and positions of
shape parts [Ovsjanikov et al. 2011]
• Employ 3D repository model as a global context– Globally align the models with the input scan first
Search in a 3D offset window around the part
![Page 14: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/14.jpg)
Part Matching Scheme
(part contour)
(2D field)
Geometric fidelity score
Geometric contribution score
3D 2Dedgemap
Total matching score
¿{𝑪𝒑 (𝒊 , 𝒋 )=𝟏 }
3D offset window
![Page 15: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/15.jpg)
Candidate Parts
• Select top K parts with highest score for each category
Seat
Back
Arm
Front leg
……
……
……
……
……
…… …… …… …… …… ……
![Page 16: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/16.jpg)
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
![Page 17: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/17.jpg)
Structure Composition
• Goal: compose the underlying structure by identifying a subset of candidate parts
![Page 18: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/18.jpg)
Constraints for Promising Compositions
Geometric fidelity Proximity Overlap
having high score no isolated parts minimized intersection
![Page 19: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/19.jpg)
Search and Evaluate
• Search for promising compositions under constraints
• Globally Evaluate the compositions
average geometry fidelity of parts total geometry fidelity
total geometry contribution
……optimal composition
![Page 20: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/20.jpg)
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
![Page 21: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/21.jpg)
Part Conjoining
• Problem: the parts are loosely placed together• Goal: generate a consistent & complete model
![Page 22: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/22.jpg)
Identification of Contact Points
• Refer to their parent models [Jain et al. 2012]
![Page 23: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/23.jpg)
Matching of Contact Points
• Greedily match nearby contact points• Generate auxiliary contact points when necessary
auxiliary contact points
![Page 24: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/24.jpg)
𝒑𝒎𝒊𝒌
𝒑𝒏𝒋𝒌
i j
identity scale
Global Optimization
transformed contact points
• Adjust the sizes {} and positions {} of parts• Make matched point as close as possible• Contact enforcement
• Shape preserving
• Global optimization
![Page 25: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/25.jpg)
Results: Chairs
• 70 repository models, 11 part categories
![Page 26: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/26.jpg)
Results: Tables
• 61 repository models, 4 part categories
![Page 27: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/27.jpg)
Results: Bicycles
• 38 repository models, 9 part categories
![Page 28: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/28.jpg)
Results: Airplanes
• 70 repository models, 6 part categories
![Page 29: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/29.jpg)
Results: Creating New Structures
![Page 30: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/30.jpg)
Results: Impact of Dataset
input data
Randomly picking some repository models
![Page 31: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/31.jpg)
Summary
• A bottom-up structure recovery approach– Effectively reuse limited repository models– Automatically compose new structure– Handle single-view inputs by the Kinect system
• Future work– Multi-view inputs– Include style/functional constraints– Recover Indoor scenes
![Page 32: Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong.](https://reader037.fdocuments.in/reader037/viewer/2022103022/56649d005503460f949d2f57/html5/thumbnails/32.jpg)
Thank you!
Project Page: http://cg.cs.tsinghua.edu.cn/StructureRecovery