2nd International Workshop on Point Cloud Processing Stuttgart...
Transcript of 2nd International Workshop on Point Cloud Processing Stuttgart...
Universität Stuttgart
ifpifp
Deriving Semanticsfrom Textured Meshes
2nd International Workshop on Point Cloud Processing
Stuttgart, December 04-05, 2019
ifpDominik Laupheimer
ifpifpUniversität Stuttgart
Textured Meshes
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
A mesh is a wired point cloud.
2019/12/05 2
ifpifpUniversität Stuttgart
• Reduced Memory Consumption
• Good Compression Behavior (Noise Reduction)
• Waterproof Surface Representation
Explicit Topology
Texture
• Unambiguous Normal Calculation
2019/12/05 3
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
• “Fusing” Data Representation
Imagery
Point Clouds
• Use-cases
Viewshed and Flood Analysis
City Models
Visualization + VR
2019/12/05 4
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
“Meshes are comprehensive maps for literally the whole world!”
Motivation: Why Meshes?
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
5
Snapshot of Google’s 3D representation (mesh).
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
6
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
?
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
7
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
Only Indoor Scenes
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
8
ifpifpUniversität Stuttgart
Data Acquisition
Ground Truth Generation
2019/12/05
LiDAR: 800 pts/m², footprint Ø < 3 cm
Photogrammetry: GSD @ 3.7 mm (nadir), 2.3 cm (oblique)
Cramer, M.; Haala, N.; Laupheimer, D.; Mandlburger, G. & Havel, P. , 2018:
Ultra-high precision UAV-based LiDAR and Dense Image Matching. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-1, 115-120.
DOI: 10.5194/isprs-archives-XLII-1-115-2018
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
9
ifpifpUniversität Stuttgart
Ground Truth Generation
2019/12/05
✋✎
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
10
LiDAR Point Cloud
Textured Mesh
Labeled Mesh
ifpifpUniversität Stuttgart
Labeled Ground Truth
2019/12/05
0. building mass/facade 1. roof 2. impervious surface 3. green space
4. mid and high vegetation 5. vehicle 6. chimney/antenna 7. clutter
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
11
0 1 2 3 4 5 6 7
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
12
ifpifpUniversität Stuttgart
Methodology
2019/12/05
• Ground Truth Generation
• Feature Calculation
Geometric & radiometric
Multi-scale contextual features
• Train Classifier
Multi-Branch 1D CNN*
RF
• Inference/Evaluation
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
13
* George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
ifpifpUniversität Stuttgart
Feature Calculation
Height above Ground
Density Horizontality
2019/12/05 14
Texture
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
Network Architecture
Feature-Based Multi-Branch 1D CNN
2019/12/05
George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
15
Scale 0
Scale 1
Scale 2
ifpifpUniversität Stuttgart
RF Prediction250 trees, depth: 25
Training Time: 6.6h
Accuracy: 79.01%
Inference Time: 45.83s
Results
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
16
Textured Mesh~300.000 faces
Ground TruthLabel Noise
1D CNN Prediction9.3 million parameters
Training Time: < 15min
Accuracy: 79.87%
Inference Time: 14.69s
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
17
Textured Mesh Prediction
(all features)
Prediction
(geometry only)
Prediction
(texture only)
Ground Truth
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
18
Textured Mesh Prediction Using
Geometric Features Only
Prediction Using
Geometric & Radiometric
Features
ifpifpUniversität Stuttgart 2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
19
Conclusion
• Alternative 3D Data Representation: Meshes
Data fusion
Good georeferencing/relative orientation of LiDAR and imagery necessary
• Ground Truth Generation
• Pipeline for Mesh Generation and Semantic Segmentation
Overall accuracy: ~80 %
Detection of buildings and mid/high vegetation works well
• Features
Geometry > Radiometry
Radiometry matters!
ifpifpUniversität Stuttgart
• Ground Truth Generation
Avoid label noise
Crowdsourcing
• Georeferencing
Hybrid georeferencingGlira, P.; Pfeifer, N.; Mandlburger, G., 2019:
Hybrid Orientation of Airborne LIDAR Point Clouds and Aerial Images.
ISPRS Annals of Photogrammetry, Remote Sensing and
Spatial Information Sciences, Volume IV-2/W5, 2019, pp.567-574.
DOI: 10.5194/isprs-annals-IV-2-W5-567-2019
• Features
Incorporate LiDAR features
Incorporate texture explicitly
Future Work
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
20