Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing...
Transcript of Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing...
![Page 1: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/1.jpg)
1
ENPC Data Science Week
Deep Learning for Remote Sensing
Alexandre Boulch
![Page 2: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/2.jpg)
2ONERA
Research, Innovation, expertise and long-term vision for industry, French government and Europe
![Page 3: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/3.jpg)
3
Materials
Aerodynamics
Propulsion
Flight dynamics
Electromagnetism
Optics
Information Processing
![Page 4: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/4.jpg)
4
Materials
Aerodynamics
Propulsion
Flight dynamics
Electromagnetism
Optics
AutonomousUAVS
RemoteSensing
DataFusion
Image andVideo processing
Information Processing
![Page 5: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/5.jpg)
5Remote Sensing
Remote SensingObtaining information about objects without contact
Earth ObservationGathering information about Earth via Remote Sensors
● aerial● spatial● ground
![Page 6: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/6.jpg)
6Remote Sensing
Massive dataSatellites constellations covering the earth. e.g. Sentinel (ESA)
Applications examples● Urban area analysis
![Page 7: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/7.jpg)
7Remote Sensing
Massive dataSatellites constellations covering the earth. e.g. Sentinel (ESA)
Applications examples● Urban area analysis● Biomass estimation
![Page 8: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/8.jpg)
8Remote Sensing
Massive dataSatellites constellations covering the earth. e.g. Sentinel (ESA)
Applications examples● Urban area analysis● Biomass estimation● Oil spread detection● MNT estimation● Building deformation
from space
Need for robust, automatic and fast processing
![Page 9: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/9.jpg)
9Objective
Registered aerial or spatial images
Heterogeneous sources: - RGB - Hyperspectral - LIDAR - SAR ...
1 label per pixel
E.g. Tree, Building, Roads,...
SEMANTIC MAPINPUT DATA
MachineLearning
![Page 10: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/10.jpg)
10Classification problem
Labels:
Car Dog House Plane Toy Game Ball Cat Road Tree Lego Ski Food Pillow Sun Flower ...
Images + Labels
![Page 11: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/11.jpg)
11Classification problem
HOUSE
COMET PLANE
SKI
BALL
CAT
Labels:
Car Dog House Plane Toy Game Ball Cat Road Tree Lego Ski Food Pillow Sun Flower ...
Images + Labels Annotated images
![Page 12: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/12.jpg)
12Classification framework
Image Annotated image
Feature extraction
Human
Example
Same object, different colors
Classification
Image space : - High dimension (Image space) - No structure of the image - Not robust to changes
Direct classification is difficult
![Page 13: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/13.jpg)
13Classification framework
Image Annotated image
ClassificationFeature
extractionHuman
Example
Same object, different colors Similar after transformation
Gradient and threshold
![Page 14: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/14.jpg)
14Classification framework
Class 1
Class 2
Class 3
Image Annotated image
ClassificationFeature
extractionHuman
![Page 15: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/15.jpg)
15How to do it ?
Image Annotated image
ClassificationFeature
extractionHuman
Expert features
Emprical rules Emprical thresholds or SVM
Water is blue +
blue level > 200 green & red < 100
![Page 16: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/16.jpg)
16Designing features is difficult
Binary classification
Orange Not orange
![Page 17: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/17.jpg)
17
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
Designing features is difficult
![Page 18: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/18.jpg)
18Designing features is difficult
Round
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
![Page 19: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/19.jpg)
19Designing features is difficult
Round
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
![Page 20: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/20.jpg)
20Designing features is difficult
Round
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
Color: orange
![Page 21: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/21.jpg)
21Designing features is difficult
Round
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
Color: orange
No line
![Page 22: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/22.jpg)
22Designing features is difficult
Round
Binary classification
Orange Not orange
What is a good and generic description of oranges ?
Color: orange
No line
![Page 23: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/23.jpg)
23How to do it ?
Image Annotated image
ClassificationFeature
extractionHuman
Complex features
Complex features: - generic features - human design
SVM (support vector machines)
Bike
Histograms of Oriented Gradients (HOGs)
![Page 24: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/24.jpg)
24How to do it ?
Image Annotated image
ClassificationFeature
extractionHuman
Deep Neural Networks
Convolutions (First layers)
Fully connected layers
Input
Output
![Page 25: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/25.jpg)
25Deep neural networks
Input
Output
● Two step algorithm:FORWARD: predict the output given a inputBACKWARD: compute the gradient of the loss, update the parameters of the network (back propagation)
● A lot of parameters to optimize
● Efficient computing using GPUs
● State of the art
![Page 26: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/26.jpg)
26First convolution layer
AlexNet first convolution layer[Krizhevsky 2012]
Gabor filters[Daugman 1985]
![Page 27: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/27.jpg)
27Neural networks are not new
1980’s, Artificial Neural Networks
Fukushima, Neocognitron : A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position, Biological Cybernetics 36-4, 1980
Late 1980’s, Backpropagation algorithm applied to deep
neural networks
LeCun et al., Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation, 1, pp. 541–551, 1989
2000’s, GPU implementation, renewed interest in Deep
Learning
Ciresan et al., Deep Big Simple Neural Nets for Handwritten Digit Recognition, Neural
Computation, 22, pp. 3207–3220, 2010
Since 2010:
Highly multiclass object recognition (ImageNet challenges),
optical flow, segmentation. . .
![Page 28: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/28.jpg)
28What changed ?
Computing power
Data Internet
Deep Learning
![Page 29: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/29.jpg)
29What about Remote Sensing ?
More data availableSentinelLandsat…
Free dataESANSA / JPL
PlatformsPEPSGoogle Earth Engine
www.esa.int
![Page 30: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/30.jpg)
30Application examples
RADARTree Species recognition
OpticalUrban semantics
3D dataUrban semantics
![Page 31: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/31.jpg)
31Tree species recognition
Remningstorp (Sweden)
ObjectiveSemantic labeling of forest areas up to the species level
ChallengeDoing better than human eye.
![Page 32: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/32.jpg)
32Tree species recognition with SAR data
Radar penetrates vegetation, it reflects on object with size similar to wave length.
Radar data gives information on the structure of the forest.
![Page 33: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/33.jpg)
33Tree species recognition
Generated fom open data: Dept. of Forest Resource Management, Swedish University of Agricultural Sciences
7 classes:
Water No tree
Trees: Birch Oak Pine Spruce Misc.
![Page 34: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/34.jpg)
34Tree species recognition
HH HV VV channels RGB Color composition
SAR data
● Band P● Polarized● Vegetation
penetration
![Page 35: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/35.jpg)
35Tree species recognition
Ground Truth Histogramsand SVM
LeNetSmall neural
network
AlexNetLarge neural
network
![Page 36: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/36.jpg)
36Urban semantics
● Car park optimization● Road mapping● Urban extension
![Page 37: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/37.jpg)
37Using a classification framework
Classification Process
Inputs Multi source data
Color composition 3 channels
Superpixels or Regular grid
Set of small images (one per super pixel)
Semantic Map One label per superpixel
![Page 38: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/38.jpg)
38Using segmentation networks
Dense prediction
Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks
Nicolas Audebert, B. Le Saux, Sébastien Lefèvre
ACCV 2016
![Page 39: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/39.jpg)
39
3D data semantics usingsegmentation networks
Pick snap shots of thePoint cloud
Project back from 2d predictions to point cloud
First place on the Semantic 3D dataset
![Page 40: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/40.jpg)
40On going projects at ONERA
DeLTADeep learning for aerospatial
delta-onera.github.io
Big data forRemote Sensingw3.onera.fr/medusa
![Page 41: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/41.jpg)
41Intern and PhD positions
Internship opportunities
From October to January
PhD positions 2017
A joint geometrical and semantic approach to reconstructing digital model (ENPC / ONERA)
Remote sensing images registration using a deep framework (ONERA)
Deep learning for multi-temporal activity analysis in remote sensing (ONERA / ENST)
All opportunities on www.onera.fr
![Page 42: Deep Learning for Remote Sensing · ENPC Data Science Week Deep Learning for Remote Sensing Alexandre Boulch. ONERA 2 ... Data Fusion Image and Video processing Information Processing.](https://reader034.fdocuments.in/reader034/viewer/2022050516/5f9fe465c2981e6db310473b/html5/thumbnails/42.jpg)
42Conclusion
“We chose it because we deal with huge amounts of data. Besides, it sounds really cool.”
Larry Page - Google
www.esa.intRS in Information Processing team: Alexandre Boulch, Nicolas Audebert, Guillaume Brigot, Fabrice Janez, Elise Koeniguer, Bertrand Le Saux