John Fry, Kenyon Huber-Wilker

Real-Time and Big Data Analysis for

Defense and Intelligence Workflows

things that move… things that “just happen”…

• airplanes

• vehicles

• animals

• storms

• crimes

• lightning

• accidents

• tweets

Real-Time GIS – Types of observations and data

stationary sensors…

• water gauges

• weather stations

• air quality sensors

• device temperature

• ships

• satellites

• trains

• people

What about storing, visualizing, and analyzing those features?

What about storing, visualizing, and analyzing those features?

Public Safetypolice fire

surveillance

Public Healthhospitals ambulances

Transitbuses taxis rail

trains crowds

Connected Carsautonomous driving traffic conditions holes

parking meters road conditions slippery areas

network improvements

Energy Usageelectricity gas smart meters

City Workerssanitation snow plows

Airportsflight status queues

plane location runway status

Buildingslighting hvac

occupancy counts

Internet of your Things

Weatherwarnings earthquakes

precipitation icy conditions

Peoplehealth monitoring

social activityTelecommunications

cell phone signals dropped calls

enabling geospatial insights with your IoT ArcGIS and IoT

Environmentnoise co2 nitrates

gases temperature humidity

atmospheric pressure radiationpesticides electromagnetic feedback

rain gauges water level gaugeswater quality air quality

Apps

DesktopAPIs

with real-time & big data capabilities

ArcGIS Enterprise

Ready to Use Real-Time ServicesArcGIS Living Atlas provides real-time content

https://www.esri.com/arcgis-blog/products/real-time/real-time/putting-the-live-in-living-atlas-of-the-world/

USA Weather Watches and Warnings

USA Short-Term Weather Warnings

USA Storm Reports

Current Weather and Wind Station Data

Active HurricanesRecent Hurricane

Satellite (MODIS) Thermal Hotspots and Fire Activity

National Weather Service Wind Forecast

National Weather Service Wind Gust Forecast

National Weather Service Wind Direction Forecast

National Weather Service Precipitation Forecast

National Weather Service Snowfall Forecast

National Weather Service Ice Forecast

National Weather Service Smoke ForecastRecent Weather Radar Imagery

And more…

• Weather and Disaster Live Feeds have been updated

• No subscription needed

• Shared as feature layers

• Auto updates in web map

John Fry

Ready to use live

services

Field Operations with ArcGIS

Field Operations

Workforce

Navigator

ArcGIS ProOperations Dashboard

Tracker

Explorer

Drone2Map

Survey123

Collector

UNDERSTAND

QuickCapture

COLLECT

WorkforceNavigator

Collector

Survey123Explorer

Tracker

G

QuickCapture

. . . Empowering Mobile Users

Tracker

Workforce

Collector

Navigator

Explorer

Survey123

QuickCapture

Integrated Mobile Apps for the Field

Demo – Tracker

and Operations

Dashboard

ArcGIS Enterprisewith real-time & big data capabilities

Apps

DesktopAPIs

analytics storage

visualization

ingestion

analytics

GeoAnalytics

Server

GeoAnalytics

Desktop

toolbox

Configurable inputs for any type of data❖ Ingest Data:

GeoEvent Server – Key Capability #1

Bringing real-time and big data capabilities to your ArcGIS Enterprise

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

Filter:

Blue / Orange

▪ Identify event records with specified attribute values

❖ Apply real-time analytics: Filtering

GeoEvent Server – Key Capability #2

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

▪ Identify event records with specified attribute values

▪ Determine spatial relationships with established geofences

❖ Apply real-time analytics: Filtering

GeoEvent Server – Key Capability #2

geofence

outside

inside

outside

geofence

disjoint crosses

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

✓ Field Calculator

✓ Field Mapper

✓ Buffer Creator

✓ GeoTagger

✓ Incident Detector

✓ Track Gap Detector

…

GeoEvent Server – Key Capability #2

▪ Configure and use over two dozen types of processors

❖ Apply real-time analytics: Processing

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

GeoEvent Server – Key Capability #2

▪ Configure and use over two dozen types of processors

❖ Apply real-time analytics: Processing

✓ Field Calculator

✓ Field Mapper

✓ Buffer Creator

✓ GeoTagger

✓ Incident Detector

✓ Track Gap Detector

…GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

▪ Configurable outputs enable processed events to bestored in a traditional relational geodatabase

▪ When data is received at a high volume, high velocity,or is accumulated over time, a spatiotemporal big datastore is an available noSQL option for enterprise storage

❖ Store data as feature records in a geodatabase

GeoEvent Server – Key Capability #3

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

▪ Relational geodatabase feature services supporttraditional RESTful web applications and APIs

▪ Big data map and feature services support on-the-flyfeature aggregation

❖ Visualize feature records

GeoEvent Server – Key Capability #4

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

▪ Relational geodatabase feature services supporttraditional RESTful web applications and APIs

▪ Big data map and feature services support on-the-flyfeature aggregation

▪ Stream Services

❖ Visualize feature records

DesktopApps and

APIs

ArcGISData Store

ArcGISEnterprise

GeoEventServer

GeoEvent Server – Key Capability #4

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Bringing real-time and big data capabilities to your ArcGIS Enterprise

▪ Configurable outputs provide the capability to format ane-mail or SMS text message

▪ How you choose to disseminate processed event recordsis entirely up to you

❖ Notify stakeholders about detected patterns of interest

GeoEvent Server – Key Capability #5

!

GeoEvent Server

server role

Spatiotemporal big

data store

Apps

Desktop

APIs

Consuming Real-Time Data

Stream Layer

Map LayerFeature Layer

• Stream layers in apps subscribe to stream services to immediately visualize observations

- does not require storage, low latency, no playback

• Map & Features layers in apps periodically poll to visualize most current observations

- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)

- history can be retrieved & queried for playback

ArcGIS

Enterprise

Stream Service

Send Features to a Stream Service subscribe (push)

polling (pull)

Map Service

Feature

ServiceGeoAnalytics

Server

Add a Feature to a BDS

Update a Feature in a BDS

spatiotemporal

big data store

Add or Update a Feature

EGDBGeoEvent

Server

Apps

Desktop

APIs

John Fry

Land Operations

Land Operations

• Use Case:

- My organization needs to understand the location of blue forces, red forces, and intelligence

collection platforms.

• Workflow:

- Geoevent:

- Input – Simulation file for military units

- Output – Update a feature service, stream service

- GeoEvent Service

- Ingest feed of military units and receive alerts when in a target area

- Ingest FMV asset feed and create frame and line and alert when

over a target position

- Operations Dashboard

- List widgets, Map Widgets

LandOps GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

LandOps GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

LandOps GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

LandOps GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

John Fry

Satellite Tracking

Satellite Tracking

• Use Case:

- My organization needs to understand the location of different platforms, what type they are, and

whether they pose an intelligence risk to my assets.

• Workflow:

- Geoevent:

- Input – https://n2yo.com/api/ satellite JSON feed

- Output – Update a feature service

- GeoEvent Service

- See location of what is above center of USA

- Request for Yaogan Series satellite across the globe

and alert whether they are approaching a US military bases

- Operations Dashboard

- List widgets, Bar Charts, Embedded Content, Indicator Widgets

John Fry

Satellite Tracking

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

Satellite GeoEvent Workflow – Key Components

Inputs OutputsFilters and Processors

Field Mapper

John Fry

Maritime Big Data

Maritime Big Data

• Use Case:

- My organization needs to ingest ship Automatic Identification System (AIS) position data

and store it

• Workflow:

- Geoevent:

- Input – Spire’s AIS - https://www.spire.com/en

- Output –

- Add a feature to spatiotemporal big data store

- GeoEvent Service

- Ingest SPIRE and add these positions to a spatiotemporal big data store

- WebAppBuilder:

- Query data

John Fry

Maritime Big Data

ArcGIS Enterprisewith real-time & big data capabilities

Apps

DesktopAPIs

analytics storage

visualization

ingestion

analytics

GeoAnalytics

Server

GeoAnalytics

Desktop

toolbox

GeoAnalytics parallelizes computing (Apache

Spark) to quickly analyze large amounts of

vector and tabular data

A collection of analysis tools to identify

patterns, relationships, anomalies and

incidents in large amounts of data across

space and time

What is GeoAnalytics?

GeoAnalytics is available through...

• ArcGIS Pro (2.4+) – GeoAnalytics Desktop Tool

-ArcGIS Pro Advanced license

• GeoAnalytics Server (10.5+) – use 1 or 3 machines to

distribute analysis

-Access tools in:

-Map Viewer

-ArcGIS Pro

-ArcGIS REST API

-ArcGIS API for Python

Summarize Data

Find Locations

Analyze Patterns

Use Proximity

Manage Data

GeoAnalytics Server Tools

Data Enrichment

Tool availability between clients may have minor differences (Map Viewer, Pro, REST API, Python API)

Summarize Data

Find Locations

Analyze Patterns

Use Proximity

Manage Data

Data Enrichment

GeoAnalytics Desktop Tools (ArcGIS Pro 2.4)

Summarize Data

Powerful Tool – Join Features

Spatial

• Intersects

• Equals• Near

• Contains• Within• Touches

• Crosses• Overlaps

Temporal

• Meets

• Met by• Overlaps

• Overlapped by• During• Contains

• Equals• Finishes

• Finished by• Starts• Started by

• Intersects• Near

Attribute

Features are matched

when the field values

in the join layer are

equal to field values in

the target layer.

Near spatial and temporal relationships require a distance/length of time to be specified. For more information see:

https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm

Joins in GeoAnalytics Server

When to use Desktop or Server for GeoAnalytics?

• Use GeoAnalytics Desktop when you want to:

- Process local data (from files, databases) faster than before on your own desktop machine

- Prototyping workflows you want to use with GeoAnalytics Server

• Use GeoAnalytics Server when you want to:

- Bring big data analysis to your entire organization

- Leverage the power of one or multiple server machines

- Connect to external big data storage and existing web layers

- Extend using custom analysis

John Fry

GeoAnalytics

Desktop toolbox

Kenyon Huber-Wilker

Cell Phone

Analysis

ArcGIS Enterprisewith real-time & big data capabilities

Apps

DesktopAPIs

analytics storage

visualization

ingestion

analytics

GeoAnalytics

Server

GeoAnalytics

Desktop

toolbox

Guides / Resources

Self-paced training and introductions to GeoEvent Server

Real-Time GIS: GeoEvent Server

• Step-by-Step Tutorials, free to download

• http://enterprise.arcgis.com/en/geoevent/latest/

get-started/geoevent-server-tutorials.htm

- Introduction

- Stream services

- Spatiotemporal Big Data Store

- Notifications

• Blogs and discussions on the forum

- http://links.esri.com/geoevent-forum

• Video recordings of technical workshops

- http://www.esri.com/videos

• GeoEvent Best Practices

Guides / Resources

GeoAnalytics

• Tutorial: Run a GeoAnalytics tool

• Spark-Powered Analysis with GeoAnalytics

Desktop Tools vs. GeoAnalytics Server

• GeoAnalytics Server Resource List

• What is GeoAnalytics Server?

• Detecting Incidents with Ozone Data

• Water Leaks blogs

• Data in GeoAnalytics Server

• Extend your Analysis with GeoAnalytics Server

and Spark (Run Python Script

Questions?

Print Your Certificate of Attendance

Print Stations Located in 150 Concourse Lobby

Tuesday12:30 pm – 6:30 pm

Expo

Hall B

5:15 pm – 6:30 pm

Expo Social

Hall B

Wednesday10:45 am – 5:15 pm

Expo

Hall B

6:30 pm – 9:30 pm

Networking Reception

Smithsonian National Museum

of Natural History

Download the Esri

Events app and find your event

Select the session

you attended

Scroll down to

“Survey”

Log in to access the

survey

Complete the survey

and select “Submit”

Please Share Your Feedback in the App