CurveFusion: RGBD-based Reconstruction of 3D Thin...
Transcript of CurveFusion: RGBD-based Reconstruction of 3D Thin...
![Page 1: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/1.jpg)
Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3,
Christian Theobalt4, Wenping Wang1, Niloy J. Mitra2
1The University of Hong Kong 2University College London
3Adobe Research 4Max Planck Institute for Informatics
CurveFusion: RGBD-based Reconstruction of 3D Thin Structures
![Page 2: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/2.jpg)
CurveFusion: RGBD-based Reconstruction of 3D Thin Structures
To reconstruct a 3D tubular model from a sequence of raw RGBD images
![Page 3: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/3.jpg)
Examples of Thin Structures
![Page 4: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/4.jpg)
Related Works: Image-based methodsPatch-based Multi-view Stereo (PMVS) [Furukawa and Ponce 2010], [Tabb 2013], Line3D++ [Hofer et al. 2016], [Usumezbas et al. 2016], [Liu et al. 2017]
1) Correspondence problem;
2) Low reconstruction quality;
3) Sensitivity to camera pose estimation; E.g. [Liu et al. 2017]
4) Special image set up. E.g. 40 calibrated cameras used in [Tabb 2013]
Correspondence Problem Low Reconstruction Quality
![Page 5: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/5.jpg)
Related Works: Failure of Existing Fusion Methods
Bundle Fusion[3]Kinect Fusion[2]Volume Fusion[1]
[1] Curless et al. "A volumetric method for building complex models from rangeimages." SIGGRAPH 1996.[2] Newcombe et al. "KinectFusion: Real-time dense surface mapping andtracking." ISMAR 2011.[3] Dai et al. "Bundlefusion: Real-time globally consistent 3d reconstruction usingon-the-fly surface reintegration." TOG 2017.R
efer
ence
Inp
ut
![Page 6: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/6.jpg)
Our Reconstruction Result
![Page 7: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/7.jpg)
1) Noisy depth samples and missing thin structures
Key Challenges in Wire Reconstruction
2) Voxel grid cannot represent thin structures well
![Page 8: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/8.jpg)
curve skeleton
Extract skeletons from depth images
Curve Skeleton – A New Fusion Primitive
point cloud
![Page 9: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/9.jpg)
Scanning Video
![Page 10: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/10.jpg)
System Setup:
• Kinect V1 sensor
• ORB SLAM [1] for camera pose
System Setup and Assumptions
Assumptions:
• Sufficient features in background
• Wires of diameter 2 mm and up
• Non-black surface color
[1] Mur-Artal et al. "ORB-SLAM: a Versatile and Accurate Monocular SLAM System.“ IEEE Transactions on Robotics, 2015
![Page 11: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/11.jpg)
Separating point samples of thin structures from background
A full depth frame Depth samples of the thin structure
Key Issue #1: Data Segmentation
![Page 12: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/12.jpg)
1 frame 1000 frames
Group depth frames into bundles for data consolidation
L1 Skeletons
Aligned Point Clouds
Key Issue #2: Data Consolidation
4 frames 30 frames 240 frames
![Page 13: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/13.jpg)
1) Perform segmentation to obtain bundles of depth samples on the thin structure
Algorithm Pipeline of CurveFusion
2) Extract a partial skeleton from each bundle of depth samples3) Fuse all partial skeletons to form the complete skeleton of the whole wire model
![Page 14: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/14.jpg)
Segmentation Procedure
![Page 15: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/15.jpg)
Segmentation Procedure
![Page 16: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/16.jpg)
Bundle Point Cloud
L1 Skeletons
Aligned Skeletons
Extracting Curve Skeletons: L1 Skeleton[1]
[1] Hui Huang, Shihao Wu, Daniel Cohen-Or, Minglun Gong, Hao Zhang, Guiqing Li, and Baoquan Chen. 2013. L1-medial Skeleton of Point Cloud. In ACM SIGGRAPH. ACM, New York, NY, USA, Article 65, 8 pages.
![Page 17: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/17.jpg)
Bundle skeleton and fused partial skeleton
Red: Already Fused Partial SkeletonBlue: New Bundle SkeletonGreen: New Partial Skeleton
Fusing Bundle Skeletons Together
After alignment New partial skeletonAfter merge
![Page 18: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/18.jpg)
Fusing Bundle Skeletons Together
Red: Already Fused Partial SkeletonBlue: New Bundle Skeleton
before merge after mergecurve splitting
![Page 19: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/19.jpg)
Fusing Bundle Skeletons Together
![Page 20: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/20.jpg)
ResultsRGB Images Ground Truth Our Results
![Page 21: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/21.jpg)
ResultsRGB Images Ground Truth Our Results
![Page 22: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/22.jpg)
Scanning setup Reference image Aligned point cloud
Black objects cannot be scanned
Evaluation – Different Colors
![Page 23: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/23.jpg)
2mm1mm 3mm 4mm
Scanning setup References Aligned point cloud
Wires of diameter less than 2 mm cannot be captured by Kinect V1
Evaluation – Different Diameters
![Page 24: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/24.jpg)
Evaluation – Different Camera Pose EstimationsBundleFusion[1] KinectFusion[2]
Aligned point cloud
Our reconstruction Aligned point cloud
Our reconstruction
[1] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-Time GloballyConsistent 3D Reconstruction Using On the Fly Surface Reintegration. ACM TOG 36, 3, Article 24 (May 2017), 18 pages.[2] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton,Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In IEEE ISMAR (ISMAR ’11). IEEEComputer Society, Washington, DC, USA, 127–136.
![Page 25: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/25.jpg)
Evaluation – Different Sensors
![Page 26: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/26.jpg)
Our Result Liu et al. 2017Reference Image
Comparison with [Liu et al. 2017]
Lingjie Liu, Duygu Ceylan, Cheng Lin,WenpingWang, and Niloy J. Mitra. 2017. Imagebased Reconstruction of Wire Art. ACM SIGGRAPH 36, 4, Article 63 (July 2017), 11 pages.
![Page 27: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/27.jpg)
Reference Image Our Result
Tabb et al. 2013
Comparison with [Tabb et al. 2013]
A. Tabb. 2013. Shape from Silhouette Probability Maps: Reconstruction of Thin Objects in the Presence of Silhouette
Extraction and Calibration Error. In 2013 IEEE Conference on Computer Vision and Pattern Recognition.
![Page 28: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/28.jpg)
1. Cannot scan black wires or wires of diameter less than 2 mm
2. Cannot reconstruct dynamic wires
3. Wires are assumed to have a constant radius
4. Reconstruction is not real time.
Limitations
![Page 29: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/29.jpg)
1. Real time performance
2. Thin structures of varying radius or non-circular cross-section
Future Works
Reference Preliminary result
3. Hybrid structures
![Page 30: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/30.jpg)
Acknowledgements
• Thank Hui Huang, Amy Tabb and ZhengWang for their great help with thetesting and validation of our work.
• Thank Cheng Lin, Jiatao Gu, Maria Lam, Paul Guerrero for their help.
This work was partially funded by the ERC Starting Grant SmartGeometry (StG-2013-335373), the Research Grant Council of Hong Kong (GRF 17210718), a GoogleFaculty award, a UCL visiting student program, and gifts from Adobe.
![Page 31: CurveFusion: RGBD-based Reconstruction of 3D Thin Structuresgeometry.cs.ucl.ac.uk/.../curveFusion_ppt.pdf · Lingjie Liu1,2, Nenglun Chen1, Duygu Ceylan3, Christian Theobalt4, Wenping](https://reader036.fdocuments.in/reader036/viewer/2022071218/604fd0a89fe86818a45365d6/html5/thumbnails/31.jpg)
Thank You!