Post on 23-May-2018
3/13/2018
1
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
POINT CLOUD
ACQUISITION & STRUCTURING
Fabio REMONDINO
3D Optical Metrology (3DOM)
Bruno Kessler Foundation (FBK)
Trento, Italy
Email: remondino@fbk.eu
http://3dom.fbk.eu
1
with contributions from FBK-3DOM members:
Isabella Toschi, Fabio Menna, Emre Oezdemir,
Daniele Morabito, Elisa Farella, Erica Nocerino
Point Cloud Processing 2018
March 13th, 2018 – TU Delft, the Netherlands
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING2
hardware / sensors
dense image matching algorithms
CONTENTS
POINT CLOUD
ACQUISITION & STRUCTURING
classification / segmentation
mesh / polygonal model generation
3/13/2018
2
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING3
generation / updating of 3D city models
forest mapping / vegetation analytics
monitoring corridor infrastructures
volume computations (mining, landslides, etc.)
heritage documentation and valorization
Building Information Modeling (BIM)
flood modeling
change detection
tunnel inspection
monitoring coastal erosion
dike monitoring
etc.
WHERE DO WE USE POINT CLOUDS?
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
WHERE DO WE USE POINT CLOUDS?
3D mapping / quantification of snow / ice lost on the Marmolada glacier
Point cloud 2014
Point cloud 2014 vs 2009
3/13/2018
3
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
http://3dom.fbk.eu/repository/3Dpointclouds/celva/index.html
WHERE DO WE USE POINT CLOUDS?
3D mapping of WWI fortifications: underground tunnels and under-forest trenches,
classification of trench structures and gallery components (enrances, riflemen
emplacements, barracks, etc,
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
Brain anatomy with photogrammetric point clouds at 0.05 mm resolution to (i)
study the white matter for an exhaustive understanding of the brain diseases and (ii)
identify axons in the white matter responsible to “transport” information across the brain
http://3dom.fbk.eu/repository/brain/index.html
WHERE DO WE USE POINT CLOUDS?
[Nocerino, E., Menna, F., Remondino, F., Sarubbo, S., De Benedictis, A., Chioffi, F., Petralia, V., Barbareschi, M., Olivetti, E.,
Avesani, P., 2017: Application of photogrammetry to brain anatomy. ISPRS Int. Archives of PRS&SIS, XLII-2-W4, pp. 213-219]
3/13/2018
4
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
TECHNIQUES FOR POINT CLOUD GENERATION
0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km
10 Mil
1 Mil
100 000
10 000
1 000
100
10
1Ob
ject
/ S
cen
e C
om
ple
xity
[p
oin
ts/o
bje
ct]
Object / Scene Size
Close-range
photogrammetry
and
terrestrial laser scanners
Aerial
photogrammetry
and LiDAR
Satellite
3D Remote Sensing
Tactile / CMM
Hand
measurements
Topography GNSS
UAV
7
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING8
From 2000 there was a growing in popularity of LiDAR/TLS sensors for the production of
dense point clouds and photogrammetry could not efficiently deliver similar results
Many researchers shifted their research interests to LiDAR/TLS, resulting in a decline of new
advancements / developments of new photogrammetric methods
LiDAR/TLS became the dominant technology for 3D recording and modelling, replacing
photogrammetry in many application areas
The bottleneck of photogrammetry was represented by massive manual data processing,
high technical skills required, long processing time, etc.
Over the past 10 years, improvements in hardware and software (primarily pushed from the
Computer Vision community), have improved image-based tools and algorithms to the point
that nowadays photogrammetry and LiDAR/TLS can deliver comparable geometrical 3D
results
TECHNIQUES: IMAGING VS RANGING
3/13/2018
5
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING9
PHOTOGRAMMETRY LiDAR / Laser Scanning
Origin born ca 1850’s born ca 1960’s
Maturity
60’s-70’s (BBA); 90’s digital
sensors; 2000+ automated
methods / SfM / DIM
2000’s with first commercial TLS and
LiDAR/ALS solutions
Measurement
principleTriangulation
TOF (long-range) and triangulation
(short-range)
Spectrum /
RadiometryMultispectral (VIS-NIR)
Generally @ laser wavelength, rarely
multispectral
3D information To be derived direct
Scale absent (to be provided) Implicit (1:1)
Redundancy Multi-ray / Multi-view Single measure
Dependency Light, geometry, texture Distance, material, object reflectance
Statistics/ Quality
parametersFor each 3D
Generally one value for the entire
cloud
Point density 10-100 pts/sqm 1-50 pts/sqm
Precision/Accuracy XY most accurate than Z (depth) Z (depth) most accurate than XY
Target detection Top-most surface (DSM) Multiple targets per pulse (DTM/DSM)
TECHNIQUES: IMAGING VS RANGING
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
Photogrammetry ↔ passive method (passive sensors) ↔ image-based method
Scanning ↔ active method (active sensors) ↔ range-based method
Image data acquisition
Image pre-processing
Calibration and orientation
Dense 3D point cloud
generation
Surface generation, feature
extraction and texture mapping
Visualization, GIS products,
replicas, inspection, virtual
restoration, etc.
Range data acquisition
(dense 3D point cloud)
Editing and alignment
Surface generation, feature
extraction and texture mapping
Visualization, GIS products,
replicas, inspection, virtual
restoration, etc.1:8 1:10
TECHNIQUES: IMAGING VS RANGING
10
ACQUISITION
PROCESSING
STRUCTURING
VISUALIZATION
3/13/2018
6
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING11
POINT CLOUD ACQUISITION - SENSORS
PASSIVE HYBRID ACTIVE
AE
RIA
L
• Single frame cameras
• Multi-view cameras
(oblique)
• Single frame +
LiDAR
• Multi-view + LiDAR
• Traditional linear Airborne Laser
Scanning
• SPL/Geiger-mode Airborne
Laser Scanning
TE
RR
ES
TR
IAL
• DSLR cameras
• Panoramic cameras
• Mobile Mapping
systems
• Hand-held /
backpack system
• RGB-D sensors
• TOF laser scanner (long-range)
• Triangulation laser scanners
(short-range)
• Structured light systems (short-
range)
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING12
PASSIVE HYBRID ACTIVE
AE
RIA
L
• Single frame cameras
• Multi-view cameras
(oblique)
• Single frame +
LiDAR
• Multi-view + LiDAR
• Traditional linear Airborne Laser
Scanning
• SPL/Geiger-mode Airborne
Laser Scanning
TE
RR
ES
TR
IAL
• DSLR cameras
• Panoramic cameras
• Mobile Mapping
systems
• Hand-held /
backpack system
• RGB-D sensors
• TOF laser scanner (long-range)
• Triangulation laser scanners
(short-range)
• Structured light systems (short-
range)
Vexcel Ultracam Osprey
Midas Octoblique
POINT CLOUD ACQUISITION - SENSORS
3/13/2018
7
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING13
PASSIVE HYBRID ACTIVE
AE
RIA
L
• Single frame cameras
• Multi-view cameras
(oblique)
• Single frame +
LiDAR
• Multi-view + LiDAR
• Traditional linear Airborne Laser
Scanning
• SPL/Geiger-mode Airborne
Laser Scanning
TE
RR
ES
TR
IAL
• DSLR cameras
• Panoramic cameras
• Mobile Mapping
systems
• Hand-held /
backpack system
• RGB-D sensors
• TOF laser scanner (long-range)
• Triangulation laser scanners
(short-range)
• Structured light systems (short-
range)
Nadir
camera
Oblique
cameras
LiDAR
Leica CityMapper Leica CountryMapper
POINT CLOUD ACQUISITION - SENSORS
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING14
PASSIVE HYBRID ACTIVE
AE
RIA
L
• Single frame cameras
• Multi-view cameras
(oblique)
• Single frame +
LiDAR
• Multi-view + LiDAR
• Traditional linear Airborne Laser
Scanning
• SPL/Geiger-mode Airborne
Laser Scanning
TE
RR
ES
TR
IAL
• DSLR cameras
• Panoramic cameras
• Mobile Mapping
systems
• Hand-held /
backpack system
• RGB-D sensors
• TOF laser scanner (long-range)
• Triangulation laser scanners
(short-range)
• Structured light systems (short-
range)
Leica BackPack Pegasus Trimble Timms Siteco Road-Scanner Riegl VMX-2HA
POINT CLOUD ACQUISITION - SENSORS
3/13/2018
8
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
PASSIVE HYBRID ACTIVE
AE
RIA
L
• Single frame cameras
• Multi-view cameras
(oblique)
• Single frame +
LiDAR
• Multi-view + LiDAR
• Traditional linear Airborne Laser
Scanning
• SPL/Geiger-mode Airborne
Laser Scanning
TE
RR
ES
TR
IAL
• DSLR cameras
• Panoramic cameras
• Mobile Mapping
systems
• Hand-held /
backpack system
• RGB-D sensors
• TOF laser scanner (long-range)
• Triangulation laser scanners
(short-range)
• Structured light systems (short-
range)
• Laser output split in n x m array of laser
beamlets (SigmaSpace / Leica: 10 x 10;
Harris: 32 x 128)
• from 10 to 100 pts/sqm, up to 6 mil. pts/sec
• 5 times more effective than traditional
LiDAR
• Range of operation: 2000-4500 m AGL @
200 knot speed
POINT CLOUD ACQUISITION - SENSORS
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
DENSE IMAGE MATCHING (DIM)
http://www.mtzgeo.com/history.cfm
1950’s:
Analogue image matching
and stereoplotter
1960’s:
First digital cross-correlation
1980’s:
Least squares matching
&
Multi-photo matching
1990’s:
Digital stereo processing
systems
2000’s:
Close range photogrammetry,
convergent images, MVS, SGM
Today:
From feature matching to dense stereo
POINT CLOUD ACQUISITION – IMAGE ALGORITHMS
3/13/2018
9
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING17
Image rectification / epipolar
Select a matching criteria (what and how do I match?)
Use some assumptions (e.g. constant depth/disparity,
continuity constraint, etc.)
Apply local / global algorithms (aggregation) - iterative
updating
Apply optimization algorithms - energy (cost) formulation,
Markov Random Fields, graph algorithms, least squares,
etc.
Consider multi-view stereo (MVS)
Efficiency thru dynamic programming, GPU and FPGA
implementations
POINT CLOUD ACQUISITION – IMAGE ALGORITHMS
[Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014: State of the art in high
density image matching. The Photogrammetric Record, Vol. 29(146), pp. 144-166]
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING18
not only geometry data (3D coordinates), but also attributes, e.g.
GEOMETRY and ATTRIBUTES
PHOTOGRAMMETRY LiDAR
RGB intensity
reconstruction uncentainty returns
redundancy time
intersection angles strip
classes (post-processing) classes (post-processing)
normals (post-processing) normals
3/13/2018
10
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
Photogrammetry – attributes from bundle adjustment and dense image matching
GEOMETRY and ATTRIBUTES
intersection angles (20-90 deg) precision of 3D points (mean 0.06 mm)
redundancy of 3D points (3-65)
[Menna, F., Nocerino, E., Remondino, F., Dellepiane, M., Callieri, M. and Scopigno, R., 2016: 3D Digitization of an Heritage
Masterpiece - a Critical Analysis on Quality Assessment. ISPRS Int. Archives of PRS&SIS, Vol. XLI-B5, pp. 675-683]
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
POINT CLOUD STRUCTURING
Normally point clouds are unstructured 3D data (with few exceptions)
Structuring can be seen as
generation of an organized dataset (e.g. 2.5D grid DSM)
generation of structured 3D data (e.g. mesh model, 3D building models, etc.)
Bergamo (Italy) - AVT flight with Vexcel Osprey
and dense point cloud @ 10 cm resolution
Bergamo (Italy) - AVT flight with Vexcel Osprey
and 3D semantic modeling of building based on
primitive fitting
[Toschi, I., Ramos, M.M., Nocerino, E., Menna, F., Remondino, F., Moe, K., Poli, D., Legat, K., Fassi, F., 2017: Oblique
photogrammetry supporting 3D urban reconstruction of complex scenarios. ISPRS Int. Archives of PRS&SISI, Vol. XLII-1-
W1, pp. 519-526]
3/13/2018
11
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
POINT CLOUD STRUCTURING
Normally point clouds are unstructured 3D data (with few exceptions)
Structuring can be seen as
generation of an organized dataset (e.g. 2.5D grid DSM)
generation of structured 3D data (e.g. mesh model, 3D building models, etc.)
Trento (Italy) - AVT flight with Vexcel Osprey @ 10 cm GSD
Huge and unique mesh/polygonal model
3D modeling of each single building (LOD2)
with parametric shapes fitted on the DSM
vs
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
POINT CLOUD STRUCTURING
Normally point clouds are unstructured 3D data (with few exceptions)
Structuring can be seen as
generation of an organized dataset (e.g. 2.5D grid DSM)
generation of structured 3D data (e.g. mesh model, 3D building models, etc.)
generation of classified / segmented point clouds
Dortmund (ISPRS benchmark) – 3D dense point cloud and classification results using
deep learning (supervised) methods
3/13/2018
12
Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING
Technology is super rapidly democratizing and point clouds are
nowadays everywhere for many applications and at disposal of
many end-users
Point clouds are the real surveying product and have more added
value than derived products as they keep details and they are not
interpolated
From a business point of view, probably, the money maker is not
anymore the data acquisition part but the added value we can give
to point clouds for domain-specific applications
How do we enrich point clouds with attributes useful to end-users?
Which are the attribute that should be (always) linked to geometry?
How do we reliably and efficiently extract semantic information to
support domain-experts?
How do we store, visualize and transmit enriched point cloud data
to non-specialist users?
How do we provide analytics for multi-temporal point clouds?
CONCLUSIONS & OPEN ISSUES