OBJECT-AWARE GUIDANCE FOR AUTONOMOUS SCENE …kevinkaixu.net/slides/nbo_sig18_talk.pdf ·...

OBJECT-AWARE

GUIDANCE FOR

AUTONOMOUS SCENE

RECONSTRUCTIONLigang Liu, Xi Xia, Han Sun, Qi Shen,

Juzhan Xu, Bin Chen, Hui Huang, Kai Xu

University of Science and Technology of China

Shenzhen University

National University of Defense Technology

Photography &

Recording Encouraged

Background

• Commodity RGB-D sensors

Microsoft Kinect PrimeSense Intel RealSense

Background

• RGB-D sensor allows real-time reconstruction

KinectFusion[Izadi et al. 2011]

Background

• Other real-time reconstruction methods

Voxel Hashing[Nießner et al. 2013]

ElasticFusion[Whelan et al. 2015]

Background

• Indoor scene reconstruction -> 3D object models

Background

• Human scanning is a laborious task [Kim et al. 2013]

consuming

Inaccurate

scanning

Background

• Modern robots are more and more reliable and controllable.

Unimation, 1958 Fetch, 2015

Motivation: Autoscanning with Robots

AutomaticNever feel

Stable and

precise

Existing Works: Single Objects

• High quality scanning and reconstruction of single object [Wu et al. 2014]

Existing Works: Unknown Scenes

• Two pass scene reconstruction and understanding.

• Can only use low-level information in first exploration pass.

exploration & reconstruction[Xu et al. 2017]

segmentation & recognition[Nan et al. 2012]

First Pass Second Pass

frontier-based exploration[Yamauchi et al. 1997]

Existing Works: Unknown Scenes

• Two pass scene reconstruction and understanding.

• Can only use low-level information in first exploration pass.

reconstruction & segmentation[Xu et al. 2015]

object recognition[Xu et al. 2016]

First Pass Second Pass

The Main Challenge

• How to automatically achieve scene reconstruction and understanding in one

Motivation

• Human explore unknown scenes object by object!

Our Solution

• Key idea: using recognized objects as a guidance map

We need to

Phase 1: The Next Best Object Problem

Which object should I

scan next?

Object of Interest (OOI)

Overview

Objectness

Objectness Based Segmentation

Objectness Based

Global Exploration Planning

Objectness Based

Local View Planning

Model-Driven Objectness

• Objectness should measure both similarity and completeness

Partial Matching

Database

Query Database Model

Partial Matching

Query Database Model

3DMatch [Zeng et al. 2016]

Partial Matching

Query Dataset Model

Model-Driven Objectness

Similarity CompletenessObjectness

Next Best Object

Distance Orientation Size

Objectness

Technical Challenge

• How to segment and recognize objects during reconstruction?

Missing data

Recognition and segmentation constitute a chicken-egg problem

Pre-segmentation

Indoor object Scanned Model

[Whelan et al. 2015] [Tateno et al. 2015]

Pre-segmented Components

Post-segmentation

• Couples segmentation and recognition in the same optimization

Post-segmentation

Post-segmentation Results

Pre-segmentation Post-segmentation

Database Construction

Two advantages:

• Decrease the difference between CAD model and scanned model

• Segmented components & component pairs can make retrieval easier

Phase 2: The Next Best View Problem

Which view of the OOI

should I scan next?

Next Best View

Evaluation

• Virtual scene dataset

SUNCG (66 scenes) ScanNet (38 scenes)

Comparison

• Comparing object recognition with PointNet++ [Qi et al. 2017]

Comparison

• Comparing Rand Index of segmentation

Comparison

• Comparing object coverage rate and quality against tensor field guided

autoscanning [Xu et al. 2017]

More Results

Conclusion

• Objectness based segmentation

• Pre-segmentation

• Post-segmentation

Key techniques:

• Objectness based reconstruction

• The next best object (NBO)

• The next best view (NBV)

Limitations

No similar models Cluttered scenes

Future Works

Combine image-based method Driverless car with LiDAR

Thank you for your attention！Data and code are available:

http://kevinkaixu.net/projects/nbo.html

Comparison

• Comparing object coverage rate and quality against tensor field guided

autoscanning [Xu et al. 2017]

Depth noise

Time Table

OBJECT-AWARE GUIDANCE FOR AUTONOMOUS SCENE …kevinkaixu.net/slides/nbo_sig18_talk.pdf ·...

Documents

Transcript of OBJECT-AWARE GUIDANCE FOR AUTONOMOUS SCENE …kevinkaixu.net/slides/nbo_sig18_talk.pdf ·...

Efﬁcient Scene Simulation for Robust Monte Carlo ...mfallon/publications/12_fallon_icra.pdfEfﬁcient Scene Simulation for Robust Monte Carlo Localization using an RGB-D Camera Maurice

SIGGRAPH 2017: Web-wide interactive 2017 Wrap...SIGGRAPH 2017 Web is our platform 3D + VR + AR Capable DOM Integrated - Declarative 3D Change scene content through DOM Render w/ WebGL

Accelerating Realism with the (NVIDIA Scene Graph)developer.download.nvidia.com/presentations/2009/...SIGGRAPH 2009 presentation Shading – Cg/CgFX Cg/CgFX developed for developers

Isometric SIGGRAPH 2004

CoReNet: Coherent 3D scene reconstruction from a single RGB …39.96.165.147/Seminar/YuHan_200915.pdf · 2020. 9. 15. · CoReNet: Coherent 3D scene reconstruction from a single RGB

Scene Coordinate Regression Forests for Camera Relocalization in RGB … · 2018-01-04 · Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images Jamie Shotton

SIGGRAPH 2006

3D indoor scene modeling from RGB-D data: a surveycg.cs.tsinghua.edu.cn/papers/CVMJ-2015-scene-moddeling.pdfComputational Visual Media DOI 10.1007/s41095-015-0029-x Vol. 1, No. 4,

SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite ...

SIGGRAPH 2017 Theater: SIGGRAPH in Japanese and Japan CG Showcase by Yoichi Ochiai

Efficient Scene Simulation for Robust Monte Carlo ... · Efficient Scene Simulation for Robust Monte Carlo Localization Using an RGB-D Camera Maurice Fallon, Hordur Johannsson and

Indoor Scene Recognition from RGB-D Images by Learning ...cvrc.ece.utexas.edu/Publications/ShaohuaWanICPR2014.pdf · Indoor Scene Recognition from RGB-D Images by Learning Scene Bases

Intrinsic Scene Properties from a Single RGB-D Image...Intrinsic Scene Properties from a Single RGB-D Image Jonathan T. Barron and Jitendra Malik UC Berkeley {barron, malik}@eecs.berkeley.eduAbstract

SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite · Figure 1: Comparison of RGB-D recognition benchmarks. Apart from 2D annotation, our benchmark provided high quality 3D annotation

Holistic3D Scene Parsing and Reconstruction from a Single ...openaccess.thecvf.com/content_ECCV_2018/papers/... · Holistic3D Scene Parsing and Reconstruction from a Single RGB Image

RGB-D Scene Recognition via Spatial-Related Multi-Modal ...crabwq.github.io/pdf/2019 RGB-D Scene Recognition... · Z. Xiong et al.: RGB-D Scene Recognition via Spatial-Related Multi-Modal

RGB-(D) Scene Labeling: Features and Algorithms

siggraph '82

Mobile crossplatformchallenges siggraph

SIGGRAPH 2010