iQmulus/TerraMobilita benchmark on Urban Analysis
Transcript of iQmulus/TerraMobilita benchmark on Urban Analysis
![Page 1: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/1.jpg)
1 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
iQmulus/TerraMobilitabenchmark onUrban Analysis
Bruno Vallet, Mathieu Brédif, Béatriz Marcotegui, Andres Serna, Nicolas Paparoditis
![Page 2: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/2.jpg)
2 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Introduction
![Page 3: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/3.jpg)
08/07/20143 / 50 First iQmulus workshop : Urban Analysis Benchmark
Introduction
Mobile laser scanning (MLS) generates massive amount of data
Urban cores are obects of utmost interest :
Urban planning
Inventory and maintenance
Accessibility diagnostic
Need for tools to analyse MLS data acquired in urban cores
Need for a benchmark of existing tools
![Page 4: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/4.jpg)
08/07/20144 / 50 First iQmulus workshop : Urban Analysis Benchmark
Benchmark objectives
Trigger interest on MLS in scientific communities :
Computer vision
Photogrammetry/remote sensing
Geometry processing
Provide reliable and large scale ground truth for works on MLS
Define an ambitious goal for MLS based urban analysis
Provide an objective tool to compare the qualities of urban analysis algorithms
![Page 5: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/5.jpg)
08/07/20145 / 50 First iQmulus workshop : Urban Analysis Benchmark
Guidelines
Fully controlled annotation of the data. For each point :
object/segment id
class label
Very generic semantic tree to provide an ontology for urban scenes
Evaluation :
Multicriteria : not a ranking but an evaluation of the pros and cons of each benchmarked algorithm
Objective : no parameters/thresholds
![Page 6: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/6.jpg)
08/07/20146 / 50 First iQmulus workshop : Urban Analysis Benchmark
Outline
Dataset
Analysis problem statement
Ground truth production
Evaluation metrics
Participants & results
Conclusion
![Page 7: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/7.jpg)
7 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Dataset
![Page 8: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/8.jpg)
08/07/20148 / 50 First iQmulus workshop : Urban Analysis Benchmark
Data
Acquisition with Stereopolis MLS :
360° Riegl sensor, multiecho
Applanix georeferencing
Anisotropic resolution :
Across trajectory : Constant angular resolution (0.03°) => distance dependant geometric resolution
Along trajectory : Constant time resolution (10ms) => speed dependant geometric resolution
![Page 9: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/9.jpg)
08/07/20149 / 50 First iQmulus workshop : Urban Analysis Benchmark
Data attributes
Attributes :
X,Y,Z : coordinates of the echo in a geographical frame
X0,Y0,Z0 : coordinates of the laser center at the time this echo was acquired
Reflectance : backscattered intensity corrected for distance
num_echo : number of the echo in case of multiple returns
Time : time at which the point was acquired
Data provided in ply file format for easy and generic attribute handling.
![Page 10: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/10.jpg)
08/07/201410 / 50 First iQmulus workshop : Urban Analysis Benchmark
Data
![Page 11: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/11.jpg)
08/07/201411 / 50 First iQmulus workshop : Urban Analysis Benchmark
Area
10+1 zones in the center of Paris (6ème arrondissement)
Each zone has 30 (12) million points corresponding to 2 minutes of acquisition each and around 500m (depending on vehicle speed)
![Page 12: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/12.jpg)
12 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Analysis problem statement
![Page 13: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/13.jpg)
08/07/201413 / 50 First iQmulus workshop : Urban Analysis Benchmark
Scene analysis
We call scene analysis the combination of :
A segmentation of the scene in individual objects surfaces
A semantic labellisation (classification) of these objects
Participants are asked to provide a ply file, adding for each point :
A segment identifier id (defining the segmentation)
A class label class (defining the classification)
![Page 14: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/14.jpg)
08/07/201414 / 50 First iQmulus workshop : Urban Analysis Benchmark
Introduction : segmentation
![Page 15: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/15.jpg)
08/07/201415 / 50 First iQmulus workshop : Urban Analysis Benchmark
Introduction : classification
![Page 16: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/16.jpg)
08/07/201416 / 50 First iQmulus workshop : Urban Analysis Benchmark
Targeted Communities
Classification specialists :
Interested in classification ground truth
Not interested in object individualization
Growing interest in contextual classification
Segmentation specialists :
Growing interest in semantics to assist the segmentation
Detection specialists :
Detectors for specific object types
The semantic and geometric problems are connected
![Page 17: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/17.jpg)
08/07/201417 / 50 First iQmulus workshop : Urban Analysis Benchmark
Scene analysis : semantics
The semantic tree is very detailed :
Surface classes : Road
Curb
Sidewalk
Facade/building
Objects classes : Dynamic/static
Natural/man made
Punctual/linear/extended
Participants can go as deep as they wish in the semantics tree
Evaluation will be performed accordingly
![Page 18: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/18.jpg)
08/07/201418 / 50 First iQmulus workshop : Urban Analysis Benchmark
Scene analysis : semantics
![Page 19: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/19.jpg)
08/07/201419 / 50 First iQmulus workshop : Urban Analysis Benchmark
Scene analysis : semantics
![Page 20: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/20.jpg)
20 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Ground truth production
![Page 21: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/21.jpg)
08/07/201421 / 50 First iQmulus workshop : Urban Analysis Benchmark
Ground truth production tool
Requirements
Fast and easy navigation and annotation
Segmentation at point level
Interactivity/editability
We designed an inteface in sensor geometry :
Columns are points acquired consecutively
Consecutive columns correspond to points acquired at a time interval equal to the time for the laser beam to finish a 360° sweep
![Page 22: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/22.jpg)
08/07/201422 / 50 First iQmulus workshop : Urban Analysis Benchmark
Data: sensor space
![Page 23: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/23.jpg)
08/07/201423 / 50 First iQmulus workshop : Urban Analysis Benchmark
Segmentation
In 2D, segmentation is created and maintained by a partition graph
User is provided with graph editing tools :
Create a node (at a pixel corner) possibly on an existing edge
Create an edge (along pixel boundaries) : A straight line (Brezenham)
A minimal path for the cost :
Parameters = weights for Normal/Depth/Intensity difference term
User can interactively tune these parameters
Move an existing node (recomputes all adjacent edges)
![Page 24: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/24.jpg)
08/07/201424 / 50 First iQmulus workshop : Urban Analysis Benchmark
Other features
A segment can be split by any plane defined by :
Three points
Two points (vertical)
One point (vertical and orthogonal to beam direction
Plus an offset
Segments can be merged (necessary in case of occlusions)
Segments can be tagged by a label from the semantic tree
Zooming, Panning
Snapping
Import/Export point clouds with label/ids per points
Web based (javascript+webGL)
![Page 25: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/25.jpg)
08/07/201425 / 50 First iQmulus workshop : Urban Analysis Benchmark
Example
![Page 26: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/26.jpg)
08/07/201426 / 50 First iQmulus workshop : Urban Analysis Benchmark
Production details
Production of the learning dataset (12Mpts) with an alpha version of the tool
For the 10 zones of the benchmark :
10 participants
2 days production each
Around 60% of the 300 Mpts annotated
Easy production management thanks to the web based tool : Each participants gets a unique link alowing them to process a 30 Mpts
block
Their work is simply stored as a graph
Graphs are controlled and final ground truth ply files exported
![Page 27: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/27.jpg)
27 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Evaluation metrics
![Page 28: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/28.jpg)
08/07/201428 / 50 First iQmulus workshop : Urban Analysis Benchmark
Multicriteria evaluation
Evaluate the algorithm result:
As a classification algorithm: confusion matrix
As a detection algorithm : precision/recall for object classes
No notion of object for surface classes
![Page 29: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/29.jpg)
08/07/201429 / 50 First iQmulus workshop : Urban Analysis Benchmark
Precision/Recall
Need to answer the questions
Is a Ground truth (GT) object detected ?
Is an Algorithm result (AR) a good detection ?
Answer (and evaluation) requires to match objects from the GT to objects from the AR
This matching allows to define :
Precision = #(GT match AR)/#GT
Recall = #(AR match GT)/#AR
Thus precision/recall is defined on a subjective matching criterion
![Page 30: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/30.jpg)
08/07/201430 / 50 First iQmulus workshop : Urban Analysis Benchmark
Delocalisation
Ground truth Algorithm result
![Page 31: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/31.jpg)
08/07/201431 / 50 First iQmulus workshop : Urban Analysis Benchmark
Dilatation/Erosion
Dilatation Erosion
Ground truth Algorithm result
![Page 32: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/32.jpg)
08/07/201432 / 50 First iQmulus workshop : Urban Analysis Benchmark
Scission/fusion
Split (N to 1) Merge (1 to M)
Ground truth Algorithm result
![Page 33: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/33.jpg)
08/07/201433 / 50 First iQmulus workshop : Urban Analysis Benchmark
N to M associations
Ground truth Algorithm result
![Page 34: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/34.jpg)
08/07/201434 / 50 First iQmulus workshop : Urban Analysis Benchmark
Intersection/Union Ratio
Gives a « distance » between objets :
0 = no intersection
1 = perfect match
Matching often defined by a threshold on R
Above 0.5, no N to M matchings
But 0.5 is very strict
Precision/recall depends highly on this threshold
![Page 35: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/35.jpg)
08/07/201435 / 50 First iQmulus workshop : Urban Analysis Benchmark
Proposition
Give precision and recall as a function of this threshold :
No arbitrary (subjective) choice of a threshold
Compare algorithms by comparing curves
For thresholds below 0.5, also give the number of N to 1 and 1 to M pairings
![Page 36: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/36.jpg)
36 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Participants & results
![Page 37: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/37.jpg)
08/07/201437 / 50 First iQmulus workshop : Urban Analysis Benchmark
Participants
CMM - MINES ParisTech (Andres Serna, Beatriz Marcotegui):
Based on elevation images
Mathematical Morphology based image processing
Machine learning techniques
Does the full analysis (segmentation and classification)
Institute of Photogrammetry and Remote Sensing (IPF) – KIT (Martin Weinmann) :
Extract a variety of low-level geometric features
Supervised classification based on careful feature selection
Only classification evaluated
![Page 38: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/38.jpg)
08/07/201438 / 50 First iQmulus workshop : Urban Analysis Benchmark
Ground truth
![Page 39: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/39.jpg)
08/07/201439 / 50 First iQmulus workshop : Urban Analysis Benchmark
CMM result
![Page 40: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/40.jpg)
08/07/201440 / 50 First iQmulus workshop : Urban Analysis Benchmark
IPF result
![Page 41: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/41.jpg)
08/07/201441 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for CMM
Classification (one 30 Mpts zone) :
Surface/object : 92.6%
Building/ground surface : 98.3%
Curb/sidewalk/road : 98.4%
GT/AR Road&side Curb
road 71.8348 0.684865
sidewalk 25.7088 0.687476
curb 0.187855 0.896216
![Page 42: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/42.jpg)
08/07/201442 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for CMM
Classification (one 30 Mpts zone) :
Surface/object : 92.6%
Building/ground surface : 98.3%
Curb/sidewalk/road : 98.4%
GT/AR Road&side Curb
road 71.8348 0.684865
sidewalk 25.7088 0.687476
curb 0.187855 0.896216
![Page 43: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/43.jpg)
08/07/201443 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for CMM
Classification (one 30 Mpts zone) :
Surface/object : 92.6%
Building/ground surface : 98.3%
Curb/sidewalk/road : 98.4% but curb (2.3%) confused for sidewalk (0.7%) and road (0.7%) because of rasterization.
Static/mobile object : 91.8%
Pedestrian/2/4 wheelers : 99.3%
GT/AR pedestrian 2 wheelers 4+ wheelers
pedestrian 1.63193 0.00262888 0.123962
2 wheelers 0.388468 0.653378 0
4+ wheelers 0.112435 0.0281088 97.0591
![Page 44: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/44.jpg)
08/07/201444 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for CMM
Detection (one 30 Mpts zone) : All objects
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0
20
40
60
80
100
120
Precision
recall
0.1 0.2 0.3 0.4 0.5 0.6
0
2
4
6
8
10
12
14
16
1-n
m-1
![Page 45: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/45.jpg)
08/07/201445 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for CMM
Detection (one 30 Mpts zone) :
Static objects : Dynamic objects
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0
20
40
60
80
100
120
Precision
Recall
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0
20
40
60
80
100
120
Precision
recall
![Page 46: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/46.jpg)
08/07/201446 / 50 First iQmulus workshop : Urban Analysis Benchmark
Results for IPF
Classification (learning dataset only)
Surface/object : 87.8%
Ground/Building surface : 93.7%
Static/mobile object : 91.5%
Pedestrian/2/4 wheelers : 68.5%
GT/AR pedestrian 2 wheelers 4+ wheelers
pedestrian 4.06508 0.401652 0.0832806
2 wheelers 0.397657 8.72356 1.02395
4+ wheelers 10.4307 19.173 55.7012
![Page 47: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/47.jpg)
47 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Conclusion
![Page 48: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/48.jpg)
08/07/201448 / 50 First iQmulus workshop : Urban Analysis Benchmark
Conclusion
Very challenging benchmark :
Large dataset, requiring a large amount of work for ground truth production
Very detailed semantic tree
Difficult data: Vehicle stops (point accumulations)
Transversal roads (very different scanning geometry)
Objectivity : Manual production of the ground truth
Parameter free evaluation
![Page 49: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/49.jpg)
08/07/201449 / 50 First iQmulus workshop : Urban Analysis Benchmark
Perspectives
Perspectives :
Realeasing a larger part of the ground truth for learning
More targeted benchmarks (car type determination, static/mobile object determination, ...)
Benchmark will stay open for future participants
Having the participants provide an executable instead of a result : Comparison of timings
More validity to the benchmark results (no fine parameter tuning)
Vector evaluation for surface limits
Correcting the anisotropy in pointwise evaluation
![Page 50: iQmulus/TerraMobilita benchmark on Urban Analysis](https://reader031.fdocuments.in/reader031/viewer/2022012013/61589e02c860042abc0f4b68/html5/thumbnails/50.jpg)
50 / 50 08/07/2014 First iQmulus workshop : Urban Analysis Benchmark
Thank you for your attentionVisit us at
data.ign.fr/benchmarks/UrbanAnalysis