Rohit Singh (@geonumist)Cédric Despierre Corporon (@ceddc)

Geospatial Deep Learningwith arcgis.learn

Session Overview

• What is AI, Machine Learning & Deep Learning• Machine Learning & Deep Learning

- What is Machine Learning?- How is it different from Deep Learning

• Deep Learning workflow in ArcGIS Pro• Geospatial Deep Learning with arcgis.learn• Types of models and their applications

- Training and deploying deep learning models- Scalable deep learning with Image Server

• Resources / Getting started

What is AI?

Neural Networks

TensorFlow

CNTK

Natural Language Processing

Cognitive Computing

GeoAI

Computer Vision

Dimensionality Reduction

Object Detection

Support Vector Machines

Object Tracking

Keras

PyTorch scikit-learn

fast.ai

Random Forest Machine Learning

Deep Learning

Artificial IntelligenceCaffe

Machine Learning

Deep Learning

Artificial Intelligence

Machine Learning

Deep Learning

Artificial Intelligence

Machine Learning

Deep Learning

Artificial Intelligence

CNTK TensorFlowPyTorch

Video game behavioral AI

Keras

ConvolutionalNeural Networks

fast.ai scikit-learn

Computer Vision

Natural Language Processing

Speech Recognition

Machine Learning

Labelled Data Sets: Features + Output

Call Drops

# Complains Subscribed Package

Call Rate Decline

Churned?

4 5 ABC 20% Yes

6 2 ABC 5% No

9 4 XYZ 12% Yes

Features Output (label)

Training Data(Historical)

For Learning..

New Data Prediction

Learning Models

Trained Model

Problem: Predicting Churn

7 5 KLM 8% Yes (75%)

Labelled Data Sets: Features + Output

Segment Type Proximity to Intersection

Time of Day Weather Accident

Highway 0.1 M Morning Raining Injury

Tunnel 0.3 M Evening Sunny Property

Inner 0.2 M Noon Foggy Injury

PipeAge Depth Temperature Pressure Break

20 3 m 95 F 20 P Yes

15 4 m 5 F 35 P Yes

6 3.5 m 2 F 17 P Yes

IS Indirect Fire

IS Explosive Attack

IS Vehicle Attack

Coalition Air Ops

ISIS Camp

0.2 M 2 M 3 M 0.7 M Yes

4 M 9 M 6 M 1 M No

1.2 M 5 M 5 M 0.5 M Yes

# Females 35 - 50 F&B Sales % Urban

Chic#

Competitors Sales

35,000 $400M 55% 15 $20M

14,000 $150M 15% 27 $5M

27,000 $210M 26% 9 $12M

Road Accidents Prediction Water Leakages Prediction

ISIS Location Prediction Retail Sales Prediction

ArcGIS has Machine Learning Tools

ArcGIS

Classification

Clustering

Prediction

Deep Learning Workflowin ArcGIS

Train DL Model

Detect Objects

Export Training Data For DL Model

Training Samples

ArcGIS Pro ArcGIS Proarcgis.learn

Classify Pixels

Collect Samples Export TrainingSamples

TrainPerform Inference

ArcGIS Enterprise ArcGIS Enterprise

ArcGIS Deep Learning Workflow

Data Labeling: Training Samples Manager

• Add Labels • Quickly Collect Samples• Save Samples to a Feature Class

Collect Samples

Export TrainingSamples

Train Perform Inference

Export Training Data for Deep Learning Tool

Collect Samples

Export TrainingSamples

Train Perform Inference

• Exports Samples to Training Images• Each Image has Labels• Supports various formats

Train Model

• arcgis.learn module in ArcGIS API for Python- No installation (ArcGIS Notebooks)- Easy to use - 3-5 lines of code- Models:

- Object Detection- Pixel Classification- Feature Classification

• External Deep Learning Frameworks- Tensorflow- PyTorch…

Collect Samples

Export TrainingSamples

Train Perform Inference

Consume Deep Learning ModelsPerform Inference

Model Definition

ArcGIS User

Inference results

Input Images

InferenceTools

• ArcGIS Image Analyst in Pro• ArcGIS Image Server on Enterprise

Inference Tools

• Classify Pixels Using Deep Learning

• Object Detection Using Deep Learning

Non Maximum Suppression

Collect Samples

Export TrainingSamples

Train Perform Inference

Classify Pixels Using Deep LearningRuns the model on an input raster to product a classified raster, each valid pixel has an assigned class label.

• Mini-batch support• Processor type: CPU or GPU• Parallel processing in ArcGIS Pro and distributed

raster analysis on Enterprise

Inference Tools

• ArcGIS Image Analyst in Pro• ArcGIS Image Server on Enterprise

Detect Objects Using Deep LearningRuns the model on an input raster to produce a feature classcontaining the objects it finds. • Batch processing

• Optional Non Maximum Suppression

• Processor type: CPU or GPU

• Parallel processing in Pro and distributed raster analysis

on Enterprise

Inference Tools

• ArcGIS Image Analyst in Pro • ArcGIS Image Server on Enterprise

arcgis.learn moduleDeep Learning using ArcGIS Python API

gis

geometry

network

schematics

features

realtime

widgets

mapping

env

geocoding

geoenrichment

geoprocessing

raster

geoanalytics

learn

arcgis.learn

The arcgis.learn module in ArcGIS API for Python enables GIS analysts and data scientists to easily adopt and apply deep learning in their workflows. It enables training state-of-the-art deep learning models with a simple, intuitive API.

Beyond Detections: End to End GeoAI Lifecycle enabled by ArcGIS Imagery AI Capabilities

ImageryAccess

Imagery Prep

Training Data Prep

Train & ConsumeModels

Run Inference at SCALE

Feedback Loop

TakeAction

Deploy Models to Production

Before After

• Installing External DL Frameworks

• Dozens of lines of Code

• HARD!

• No Installation (Notebooks)

• 3-5 lines• EASY

Not just “Training”!

Exporting Training Data arcgis.learn.export_training_data

Training DL Modelsarcgis.learn.SingleShotDetectorarcgis.learn.UnetClassifierarcgis.learn.FeatureClassifierarcgis.learn.PSPNetClassifierarcgis.learn.RetinaNetarcgis.learn.MaskRCNNarcgis.learn.EntityRecognizer

Preparing Data (Augmentation)arcgis.learn.prepare_data

Model Managementarcgis.learn.list_modelsarcgis.learn.Model

Model.installModel.uninstallModel.query_info

Inference APIsarcgis.learn.detect_objectsarcgis.learn.classify_pixels

Things you can do today with ArcGIS.LearnObject Detection using SSD, Pixel Classification using Unet, Feature Classification

Damaged Structures

Roads

Swimming Pools

Building Footprints

Oil Pads

Land Cover

Road CracksCars Palm Trees

Pipeline Encroachment

Computer Vision TasksApplied to GIS

Why use Deep Learning for Imagery

ImageNet Visual Recognition Challenge error rate

• Better than human accuracy at vision tasks

Semantic SegmentationAssign a label to each pixel

Cat

Ground

Sky

Turf/Grass

Building

Water

Pixel Classification

Applications:- Land Cover Classification- Pervious/Impervious mapping…

Models:- UNetClassifier- PSPNetClassifier

Image ClassificationAssign a label to a given image

Cat

Applications:- Damaged building classification- Clean or ‘green’ pools…

Undamaged Damaged

Feature ClassificationAssign a label to a given feature

Models (from torchvision):- Inception- ResNet- VGG…

Object DetectionFind objects and their location (bounding boxes)

Applications:- Detect trees, cars, airplanes, …

Models:- SingleShotDetector- RetinaNet

Instance SegmentationFind objects and their precise locations (masks or polygonal features)

Applications:- Building footprint extraction

Models:- MaskRCNN

Scalable Deep Learning

ArcGIS Enterprise for Scaling Deep Learning

• Leverage Raster Analytics to scale inference

• All desktop inferencing tools are accessible through enterprise

• Clients to invoke distributed inferencing – map viewer, Pro, notebooks

• Multi GPU support

• Requires the ArcGIS Image Server license

ExportTrainingDataforDeepLearning Uses a remote sensing image to convert labeled vector or raster data into deep learning training datasets. The output is a folder of image chips and a folder of metadata files.

DetectObjectsUsingDeepLearning Runs a trained deep learning model on an input raster to produce a feature class containing the objects it finds. The features can be bounding boxes or polygons around the objects found, or points at the centers of the objects.

ClassifyPixelsUsingDeepLearning Runs a trained deep learning model on an input raster to produce a classified raster with each valid pixel having a class label assigned.

QueryDeepLearningModelInfo Extracts the model specific settings from the model package item or model definition file.

InstallDeepLearningModel Installs the model package item from portal to the Raster Analysis Image Server.

UninstallDeepLearningModel Uninstalls the model package from portal to the Raster Analysis Image Server

ListDeepLearningModels Lists all the installed model packages on the Raster Analysis Image Server

Pro

+ Se

rver

Too

lsSe

rver

Onl

y To

ols

ArcGIS Enterprise Deep Learning Tools / Services

Image ServerImage Server

Image Server

ArcGIS Pro

Portal

Apps

DesktopAPIs

Web Apps, Maps, Dashboards

ArcGIS Server

ArcGIS Data Store

ArcGIS Services

Image Server

GDB

Content StoreDistributed Raster Data StoreImagery Storage

Notebook Server

Deep Learning Models on Portal

GPU per Server

ArcGIS Enterprise Deep Learning – System Architecture

60,000 buildingsArcGIS Pro: 1 GP100 GPU (16 GB): 4.5 hoursArcGIS Enterprise: 4 nodes RA server with 3 x P40 GPUs (24GB): 20 minutes

Building Detection – Large Scale ProcessingObject Detection using SSD, Pixel Classification using Unet, Feature Classification

Impervious Surface Classification

Coconut Tree Detection

Building Footprint Extraction

Damaged House Classification

Applications of Deep Learning to Imagery

Pixel Classification Object Detection Instance Segmentation Image Classification

End to End Deep Learning – Wide spectrum of deep learning models

Text / NLP

• arcgis.learn – model for extracting location and other entities- EntityExtractor model

Resources for Getting Started

arcgis.learn:• https://developers.arcgis.com/python• https://medium.com/geoai• https://github.com/Esri/arcgis-python-api

Practical deep learning for coders:• https://course.fast.ai/

Complete answersand select “Submit”

Scroll down to find the feedback section

Select the session you attended

Download the Esri Events app and find your event

Please Take Our Survey on the App