CVPR2010: Advanced ITinCVPR in a Nutshell: part 7: Future Trend
Cvpr2010 open source vision software, intro and training part viii point cloud library - rusu -...
Transcript of Cvpr2010 open source vision software, intro and training part viii point cloud library - rusu -...
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Point Cloud Library3D Features. Registration. Surface Classification
Radu Bogdan RUSUJune 14, 2010
[Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
[Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (1/2)Point-based classification
Figure out ways to “classify the world”
PCL :: 3D FeaturesRadu Bogdan RUSU
[Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)Point-based classification
Two basic alternatives:1. I create a powerful, discriminative 3D feature descriptor;
I learn different classes of surface or object types;I use a machine learning classifier
(the solution efficiency depends on the method used−→ let your “neighbor” (the ML guy) solve it).
2. I use geometric reasoning techniques: segmentation, regiongrowing, robust estimators, octrees, ...
I fit linear (planes, lines) and non-linear (cylinders, spheres,etc) models to your data
PCL :: 3D FeaturesRadu Bogdan RUSU
[Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)Point-based classification
Two basic alternatives:1. I create a powerful, discriminative 3D feature descriptor;
I learn different classes of surface or object types;I use a machine learning classifier
(the solution efficiency depends on the method used−→ let your “neighbor” (the ML guy) solve it).2. I use geometric reasoning techniques: segmentation, region
growing, robust estimators, octrees, ...I fit linear (planes, lines) and non-linear (cylinders, spheres,
etc) models to your data
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Acquisition (1/1)How are Point Clouds acquired? Where do they come from?
Recap: Point Cloud acquisition sources:I laser/lidar sensors (2D/3D)I stereo camerasI time-of-flight (TOF) camerasI etc...
*PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi ,pj) > µ+ dthresh · σ}
I Distance Analysis:
I Before and after:
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi ,pj) > µ+ dthresh · σ}
I Before and after:
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
I Given a point cloud with x,y,z 3D point coordinates
I Select each point’s k -nearest neighbors, fit a local plane,and compute the plane normal
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
I Given a point cloud with x,y,z 3D point coordinates
I Select each point’s k -nearest neighbors, fit a local plane,and compute the plane normal
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (3/5)Surface Normal and Curvature Estimation
Cj =∑k
i=1 ξi · (pi − pj)T · (pi − pj), p = 1
k ·∑k
i=1 pi
ξi =
e− d2
iµ2 , pi outlier
1, pi inlier
σp = λ0λ0+λ1+λ2
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)Consistent Normal Orientation
Before
I Extended Gaussian ImageI Orientation consistent for:
1. registration2. feature estimation3. surface representation
I normals on theGaussian sphere
I should be in thesame half-space
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)Consistent Normal Orientation
Before After
(viewpoint − pi) · npi ≥ 0
or:
propagate consistencythrough an EMST
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (1/5)What scale to choose ?
bad scale (too small) good scale
Selecting the right scale (k -neighborhood) is problematic:
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (2/5)Solution: Compute the persistence of the feature over multiple scales
I Scales are independent, can be parallelizedI Where to threshold ?
I Computer Graphics: look for jumps in the feature curve
I Datasets in “robotics” have non-smooth surfaces. Solution:
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (3/5)Multiple radii analysis
L1-Manhattan L2-Euclidean Jeffries-Matusita
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (4/5)Multiple radii analysis
Bhattacharyya Chi-Square Kullback-Leibler
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (5/5)Example Result
RAW/intensity data Persistent Feature Histograms
*
Pf =n−1⋃i=1
[Pfi ∩ Pfi+1 ]
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)Basic Concepts :: PCL/Features/FPFHEstimation
I For every point pair〈(ps,ns); (pt ,nt)〉, letu = ns, v = (pt−ps)×u, w = u×v
f0 = 〈v ,nj〉f1 = 〈u, pj − pi〉/||pj − pi ||f2 = ||pj − pi ||f3 = atan(〈w ,nj〉, 〈u,nj〉)
ihist =∑x≤3
x=0
⌊fx ·d
fxmax −fxmin
⌋· dx
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)Basic Concepts :: PCL/Features/FPFHEstimation
f0 = 〈v ,nj〉f1 = 〈u, pj − pi〉/||pj − pi ||f2 = ||pj − pi ||f3 = atan(〈w ,nj〉, 〈u,nj〉)
ihist =∑x≤3
x=0
⌊fx ·d
fxmax −fxmin
⌋· dx
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)Basic Concepts :: PCL/Features/FPFHEstimation
I For every point pair〈(ps,ns); (pt ,nt)〉, letu = ns, v = (pt−ps)×u, w = u×v
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)Points lying on different geometric primitives
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (5/5)Complexity Analysis
Complexity is high: O(k2). Optimizations to the rescue!
Unordered OrderedPCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (1/5)Basic Concepts
Re-formulate: FPFH(p) = SPF (p) + 1k∑k
i=11ωk· SPF (pk )
Point Feature Histograms (PFH) Fast Point Feature Histograms (FPFH)
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (2/5)Theoretical formulation
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (3/5)Noise Analysis
Synthetically noiseless Synthetically noisified
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (4/5)Noise Analysis
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (5/5)Persistence Similarity
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (1/8)Point-based classification recap
Figure out ways to “classify the world”
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (2/8)Learning classes of surfaces
Concave vs Convex
13 classes total
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (3/8)Noise Analysis
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (4/8)Most Discriminating Features Selection
I How many training samples to generate ? Need tosimulate real scans.
I Solution: generate a lot, then select the mostdiscriminative ones
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (5/8)Classification results using different methods
SVM
KNN
KMeansTable 1. Classification results
Method used Noiseless NoisySVM RBF Sublinear kernel 95.26% 89.55%KNN Bhattacharyya (µ-dist) 87.11% 83.53%
K-Means (81D) Bhattacharyya 73.63% 70.74%
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (6/8)FPFH classification results
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (7/8)Classification results using PFH and SVM: 89.95%
*
*
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (8/8)Classification results using FPFH and CRF: 97.36%
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH ConceptsFrom local annotations (FPFH) to global (cluster) annotations (GFPFH)
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
FPFH ClassificationClassification results using FPFH and CRF: 98.27%
*
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH ClassificationClassification results using GFPFH and SVM: 95.13%
*PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (1/7)Multiple Scans :: PCL/Registration/IterativeClosestPoint*
points on similar surfaces
*
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (2/7)The classics
*
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (3/7)Outdoor Example: Persistence, Initial Alignment
*
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (4/7)Outdoor Example: Non-Linear Optimization
*PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (5/7)Outdoor Example: Non-Linear Optimization
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (6/7)Back to Indoor Environments
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (7/7)Registered 360◦ scans - 15 million points
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration [Conclusion]
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
PCL :: 3D FeaturesRadu Bogdan RUSU
Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration [Conclusion]
Questions?
http://www.ros.org/wiki/pclPCL :: 3D FeaturesRadu Bogdan RUSU