Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random...
Transcript of Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random...
![Page 1: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/1.jpg)
Direct Fitting of Gaussian Mixture Models
Leonid Keselman, Martial Hebert
Robotics InstituteCarnegie Mellon University
May 29, 2019
https://github.com/leonidk/direct_gmm1
![Page 2: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/2.jpg)
Representations of 3D data
2
Point Cloud Point Cloud+ Normals
Triangular Mesh
Nearest Neighbor Plane Fit Screened Poisson Surface Reconstruction
![Page 3: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/3.jpg)
Gaussian Mixture Models for 3D Shapes
3
GMM fit to object surface
Benefits• Closed-form expression• Can represent contiguous surfaces• Easy to build from noisy data• Sparse
![Page 4: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/4.jpg)
Gaussian Mixture Model (GMM)
4
! " = $%&'
()* +("; .*, Σ*) $
*)* = 1)* ≥ 0
Σ* is symmetric, positive-semidefinite
![Page 5: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/5.jpg)
Gaussian Mixtures as a shape representation
W. Tabib, C. O’Meadhra, N. MichaelIEEE R-AL (2018)
B. Eckart, K. Kim, A. Troccoli, A. Kelly, J. Kautz. CVPR (2016)
B. Eckart, K. Kim, J. Kautz. ECCV (2018)
5
Efficient Representation Mesh Registration Frame Registration
![Page 6: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/6.jpg)
Fitting a Gaussian Mixture Model
1. Obtain 3D Point Cloud2. Select Initial Parameters3. Iterate Expectation & Maximization
i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets parameters
6
![Page 7: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/7.jpg)
7
![Page 8: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/8.jpg)
The E-Step (Given GMM parameters)
8
!"# =1&#'" ((*#; ,", Σ")
&# =01'1 ((*#; ,1, Σ1)
Normalization constant for point j
Affiliation between point j & mixture i
![Page 9: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/9.jpg)
The M-Step (Given point-mixture weights)
9
!" = $%&'
($)&'
*+)%log /) 0(2%; 4), Σ))
8!"8/)
= 0
mixturespoints
8!"84)
= 0 8!"8Σ)
= 0
lower-bound loss
To get new parameters: takes derivatives, set equal to zero, and solve
4) =1;)$
%<)%2%/) =
;)=
Σ) =1;)$
%<)%(2%−4))(2%−4))?
<)% = +)%
;) =$%<)%
![Page 10: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/10.jpg)
10
![Page 11: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/11.jpg)
Geometric Objects in a Probability Distribution
11
Known curve in a given 2D probability distribution
![Page 12: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/12.jpg)
Geometric Objects in a Probability Distribution
12
Consider sampling N points from this curve
ℓ curve ≅()*+
,-(/))
![Page 13: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/13.jpg)
Geometric Objects in a Probability Distribution
13
Take a geometric mean to account for sample number
ℓ curve ≅ ()*+
,-(/))
+,
![Page 14: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/14.jpg)
Geometric Objects in a Probability Distribution
14
The curve will be the value in the limit
ℓ curve ≅ ()*+
,-(/))
+,
ℓ curve = lim,→6 ()*+
,-(/))
+,
= lim,→6exp log ()*+
,-(/))
+,
= lim,→6exp1< =
)*+
,log(-(/)))
![Page 15: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/15.jpg)
Geometric Objects in a Probability Distribution
15
ℓ curve ≅ ()*+
,-(/))
+,
ℓ curve = lim,→6 ()*+
,-(/))
+,
= lim,→6exp log ()*+
,-(/))
+,
= lim,→6exp1< =
)*+
,log(-(/))) = exp > log(-(/)) ?/
![Page 16: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/16.jpg)
Geometric Objects in a Probability Distribution
16
!= exp & log(+(,)) .,
1. If +(,) = 0 on curve, then L= 02. Invariant to reparameterization
![Page 17: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/17.jpg)
17
!" Area of each triangle#" Centroid of each triangle$", &", '" Triangle vertices
![Page 18: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/18.jpg)
The E-Step (Given GMM parameters)
18
!"# =1&#'" ((*#; ,", Σ")
&# =01'1 ((*#; ,1, Σ1)
Normalization constant for point j
Affiliation between point j & mixture i
2# Area of each triangle,# Centroid of each triangle3#, 4#, &# Triangle vertices
![Page 19: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/19.jpg)
The New E-Step (Given GMM parameters)
19
!"# =1&#'"(# )(+#; +", Σ")
&# =01'1(1)(+#; +1, Σ1)
Normalization constant for object j
Affiliation between object j & mixture i
(# Area of each triangle+# Centroid of each triangle2#, 3#, &# Triangle vertices
Taylor Approximation(2 terms)
![Page 20: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/20.jpg)
The M-Step (Given point-mixture weights)
20
!" =1%"&
'("')'
*" =%"+
Σ" =1%"&
'("'()'−!")()'−!")0
("' = 1"'
%" =&'("'
![Page 21: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/21.jpg)
The New M-Step (Given point-mixture weights)
21
!" =1%"&
'("')'
*" =%"+
Σ" =1%"&
'("' ()'−!")()'−!")0 + Σ'
("' = 2'3"'
%" =&'("'
2' Area of each triangle!' Centroid of each triangle4', 6', 7' Triangle vertices
Σ' =112 4'4'0 + 6'6'0 + 7'7'0 − 3 !'!'0
![Page 22: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/22.jpg)
What is Σ"?
22
![Page 23: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/23.jpg)
23
![Page 24: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/24.jpg)
ResultsDid all that math actually help us fit better/faster GMMs?
24
![Page 25: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/25.jpg)
25
Using different inputs
classic algorithm• Vertices of the mesh• Triangle centroids
our method• Approximate (E only)• Exact (E + M steps)
Measure the likelihood of a high-density point cloud (higher is better)
Evaluate across a wide range of mixtures(6 to 300)
![Page 26: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/26.jpg)
Full E+M method works in all cases
26
![Page 27: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/27.jpg)
Stable under even random initialization!
27
![Page 28: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/28.jpg)
ApplicationsAre these models actually more useful?
28
![Page 29: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/29.jpg)
Mesh Registration (P2D)
Method1. Apply a random rotation + translation to the point cloud2. Find transformation to maximize the likelihood of the points
• Perform P2D with GMMs fit toi. mesh verticesii. mesh triangles
29
Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)
![Page 30: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/30.jpg)
Mesh-based GMMs are more accurate
30
![Page 31: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/31.jpg)
Across multiple models
31
0
50
100
150
Armadillo Bunny Dragon Happy Lucy
Rotation Error(% of ICP)
points mesh
0
100
200
Armadillo Bunny Dragon Happy Lucy
Translation Error(% of ICP)
points mesh
![Page 32: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/32.jpg)
Frame Registration (D2D)
Method1. Use a sequence from an RGBD Sensor
• 2,500 frame TUM sequence from a Microsoft Kinect
2. Pairwise registration between t & t-1 frames• Optimize the D2D L2 distance• Build GMMs using square pixels as the geometric object
32
W. Tabib, C. O’Meadhra, N. Michael.“On-Manifold GMM Registration” IEEE R-AL (2018)
![Page 33: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/33.jpg)
Representing points using pixel squares
33
![Page 34: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/34.jpg)
D2D Registration Results
34
Compared to standard GMM• 2.4% improvement in RMSE• 22% faster D2D convergence
![Page 35: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/35.jpg)
Questions?
35
! = exp & log(+(,)) .,
![Page 36: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/36.jpg)
The End!
36
![Page 37: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/37.jpg)
Extra Slides
37
![Page 38: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/38.jpg)
How to fit a Gaussian Mixture Model?
1. Obtain any collection of objects2. Perform Expectation + Maximization
i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets new parameters
38
![Page 39: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/39.jpg)
Extension to arbitrary primitives
39Vasconcelos, Lippman. "Learning mixture hierarchies.” Advances in Neural Information Processing Systems (1999)
![Page 40: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/40.jpg)
Approximation
40
area-weighted geometric mean using the primitive’s centroids
![Page 41: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/41.jpg)
Product Integral Formulation
• Product integrals provide a resampling-invariant loss function• Given S samples, of M primitives, with N mixture components
• This can be evaluated in the limit of samples (with a geometric mean)
41
![Page 42: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/42.jpg)
42
![Page 43: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/43.jpg)
43
![Page 44: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/44.jpg)
44
![Page 45: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/45.jpg)
For GMMs we will use the lower bound
45
![Page 46: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/46.jpg)
P2D Registration Results
46
Model Rotation Error(% of ICP)
Translation Error(% of ICP)
points mesh points meshArmadillo 127 37 161 33Bunny 50 28 41 17Dragon 68 25 40 19Happy 101 27 85 27Lucy 95 23 122 35
![Page 47: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/47.jpg)
Mesh Registration with P2D
Method1. Apply a random rotation + translation to the point cloud2. Point-to-Distribution (P2D) registration of point cloud to GMM
• Perform tests with GMMs fit toi. mesh verticesii. mesh triangles
• Optimize the GMM likelihood with rigid body transformation (q & t)• BFGS Optimization using numerical gradients, starting from identity
47
Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)
![Page 48: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/48.jpg)
Acknowledgements
• Martial Hebert• Robotics Institute• Reviewers• Funding?
48
![Page 49: Direct Fitting of Gaussian Mixture ModelsMesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e9094763cfcd67412629470/html5/thumbnails/49.jpg)
Representing points using pixel squares
49