Neural Nets for 3D Surface Repair - GTC On Demand · 2017. 10. 27. · government-funded by...
Transcript of Neural Nets for 3D Surface Repair - GTC On Demand · 2017. 10. 27. · government-funded by...
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Deep 3D – Machine Learning for Reconstruction and Repair
of 3D SurfacesTalkID 23152
This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep learning techniques and an insightinto our approach for 3D surface repair.
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
• PhD Student at the Institute for Optical Systems at the HTWG Konstanz
• Main focus: Machine Learning for…• … Surface Reconstruction• … Defect Detection and Repair (Inpainting)• … Medical Imaging
Pascal Laube
government-funded by supervised by
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representation: The 2D case
Output
Grid in euclidean space Neural Network(in this case CNN)
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representation: In 3D?
Neural Network
Point Cloud
Mesh
Any manifold (NURBS, impl. surf., …)
?
?
?
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Voxels
[Vishakh Hegde et al., NIPS (2016)]
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Voxels
[Zhirong Wu et al., CVPR (2015)]
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Multi-View
[Hang Su et al., ICCV (2015)]
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Multi-View
[Liuhao Ge et al., CVPR (2016)]
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Graph Signal Processing
[M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]
• Graph Laplacian or Laplace Beltrami Operator as
∆𝑓 = −𝑑𝑖𝑣(𝛻𝑓)
• Laplacian Eigenfunctionsgeneralize to Fourier bases.
Convolution in the spectral domain is defined…
…but filters are base dependent.
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Representations: Graph Signal Processing
• Train filters in geodesic polar coordinates.
• Pool rotation angles
[J. Masci et al., ICCV (2015)]• Many other methods using
different kernels (heat diffusion, gauss…)
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Data Sets
• 127,915 CAD Models• 662 Object Categories• Different Subsets
• 51,300 Models• 270 Object Categories in 12.000 Model Subsets
Many smaller specialized Data Sets
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Problem: Defect on Surface with Detail- and Base-Geometry
Fraunhofer IPT
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Problem: Defect on Surface with Detail- and Base-Geometry
Werkzeugbau Siegfried Hofmann GmbH
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Problem: Defect on Surface with Detail- and Base-Geometry (3)
• High resolution meshes with > 1m vertices
• Base Geometry and Relief
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Our Approach
B-Spline Approx.
Approx. by Geometric Primitive
Multiresolut. Surfaces
SeperationBase Geo. – Detail Geo.
Surface with Defect Novelty Detection usingAutoencoders
Multiresolution Neural Netsfor Inpainting
Detail GeometryHeightmap
Base Geometry
1 2 3
or
or
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Novelty Detection using Autoencoders
• Defect unknown• Healthy state unknown
What do we know?
• Textures have to be ergodic:Statistical properties are constant for single sample and whole collection
2
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Novelty Detection using Autoencoders
Train Autoencoder on Ergodic Set of Textures
Autoencoder should be unableto sufficiently reconstruct Defects
2
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Novelty Detection using Autoencoders
Loss
Samples
2
Parallelizable to multiple GPUs
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Multiresolution Neural Nets for Inpainting: Texture Synthesis3
[L. Gatys et al., NIPS (2015)]
Activation Network Synth. Network
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
3
[L. Gatys et al., arxiv.org (2015)]
Multiresolution Neural Nets for Inpainting: Style Transfer
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Multiresolution Neural Nets for Inpainting: Example3
2048x2048
Defect
Closeup
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Multiresolution Neural Nets for Inpainting: Patches3
• Inpainting a Region with arbitrary size?• Inpaint Patch by Patch
Local Style Global Style
2048x2048
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Multiresolution Neural Nets for Inpainting: Results3
1. Start
2. Inpaint Patches:• Large Parent Weight
…
3. Inpaint Patches:• Apply Detail• Large Child Weight• Small Parent Weight
…
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Multiresolution Neural Nets for Inpainting: Results3
Result Result Closeup
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
3 Multiresolution Neural Nets for Inpainting: Results Heightmap
Parallelizable to multiple GPUs
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
3 Multiresolution Neural Nets for Inpainting: Results Surface
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Outlook
[J. Masci et al., ICCV (2015)]
[M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]
• Neural Nets in high dimensional irregular domains
• Michael M. Bronstein et al., “Geometric deep learning: going beyond Euclidean data” (2017)
• Michaël Defferrard, “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering” (2016)
government-funded by supervised by
GTC 2017, Munich, 11.10.2017
Thank You