Real-Time and Big Data Analysis for Defense and Intelligence WorkflowsCurt HammillJohn FryTyson Quink
things that move… things that “just happen”…
• airplanes
• vehicles
• animals
• storms
• crimes
• lightning
• accidents
• tweets
Real-Time GIS – Types of observations and data
stationary sensors…
• water gauges
• weather stations
• air quality sensors
• device temperature
• ships
• satellites
• trains
• people
What about storing, visualizing, and analyzing those features?
What about storing, visualizing, and analyzing those features?
Public Safetypolice fire
surveillance
Public Healthhospitals ambulances
Transitbuses taxis rail
trains crowds
Connected Carsautonomous driving traffic conditions holes
parking meters road conditions slippery areas network improvements
Energy Usageelectricity gas smart meters
City Workerssanitation snow plows
Airportsflight status queues
plane location runway status
Buildingslighting hvac
occupancy counts
Internet of your Things
Weatherwarnings earthquakes
precipitation icy conditions
Peoplehealth monitoring
social activityTelecommunications
cell phone signals dropped calls
enabling geospatial insights with your IoT ArcGIS and IoT
Environmentnoise co2 nitrates
gases temperature humidityatmospheric pressure radiation
pesticides electromagnetic feedbackrain gauges water level gaugeswater quality air quality
Apps
DesktopAPIs
with real-time & big data capabilities
ArcGIS Enterprise
ArcGIS Enterprisewith real-time & big data capabilities
DesktopWeb Device
live & historicaggregates & features
map & feature servicestream service
live features
ArcGISEnterprise
GeoEventServer
analytics
GeoAnalyticsServer
spatiotemporalbig data store
storage analytics
visualization
ingestion
actuation
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
Platform Location Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
Sensors and Instruments Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
Weather Monitoring Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
National Security Data Feeds Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
Industrial, Utilities, Facility Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Serverserver role
spatiotemporal bigdata store
Configurable inputs for any type of data Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Filter:Blue / Orange
Identify event records with specified attribute values Apply real-time analytics: Filtering
GeoEvent Server – Key Capability #2
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Identify event records with specified attribute values Determine spatial relationships with established geofences
Apply real-time analytics: Filtering
GeoEvent Server – Key Capability #2
geofence
outside
inside
outside
geofence
disjoint crosses
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Field Calculator
Field Mapper
Buffer Creator
GeoTagger
Incident Detector
Track Gap Detector
…
GeoEvent Server – Key Capability #2
Configure and use over two dozen types of processors Apply real-time analytics: Processing
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Server – Key Capability #2
Configure and use over two dozen types of processors Apply real-time analytics: Processing
GeoEvent Serverserver role
spatiotemporal bigdata store
Field Calculator
Field Mapper
Buffer Creator
GeoTagger
Incident Detector
Track Gap Detector
…
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Configurable outputs enable processed events to bestored in a traditional relational geodatabase
When data is received at a high volume, high velocity,or is accumulated over time, a spatiotemporal big datastore is an available noSQL option for enterprise storage
Store data as feature records in a geodatabase
GeoEvent Server – Key Capability #3
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Relational geodatabase feature services supporttraditional RESTful web applications and APIs
Big data map and feature services support on-the-flyfeature aggregation
Visualize feature records
GeoEvent Server – Key Capability #4
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Relational geodatabase feature services supporttraditional RESTful web applications and APIs
Big data map and feature services support on-the-flyfeature aggregation
Stream Services
Visualize feature records
GeoEvent Server – Key Capability #4
GeoEvent Serverserver role
spatiotemporal bigdata store
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Configurable outputs (email, messages, push) How you choose to disseminate processed event records
is entirely up to you
Notify commanders and staff about detected patterns of interest
GeoEvent Server – Key Capability #5
!
GeoEvent Serverserver role
spatiotemporal bigdata store
Consuming Real-Time Data
Stream LayerMap Layer
Feature Layer
• Stream layers in apps subscribe to stream services to immediately visualize observations- does not require storage, low latency, no playback
• Map & Features layers in apps periodically poll to visualize most current observations- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)- history can be retrieved & queried for playback
ArcGISEnterprise
Stream Service
Send Features to a Stream Service subscribe (push)
polling (pull)
Map ServiceFeature ServiceGeoAnalytics
Server
Add a Feature to a BDSUpdate a Feature in a BDS
spatiotemporalbig data store
Add or Update a Feature
EGDBGeoEvent
ServerApps
Desktop
APIs
Copyright © Esri. All rights reserved.
Operations Dashboard for ArcGIS
• Monitor assets, events, or activities for at-a-glance decision making
• Configurable user experience
• Command center in the web
• Integrated into the enterprise
John FrySatellite Tracking
Satellite Tracking
• Use Case:- My organization needs to understand the location of different platforms, what type they are,
and whether they pose an intelligence risk to my assets.
• Workflow:- Geoevent:
- Input – https://n2yo.com/api/ satellite JSON feed- Output – Update a feature service- GeoEvent Service
- See location of what is above center of USA- Request for Yaogan Series satellite across the globeand alert whether they are approaching a US military bases
- Operations Dashboard- List widgets, Bar Charts, Embedded Content, Indicator Widgets
https://n2yo.com/api/
John FrySatellite Tracking
Satellite GeoEvent Workflow – Key ComponentsInput OutputField Mapper
Field Calculator -Height
Field Calculator -Geometry
Field Enricher
John FryAircraft Watchlist
Aircraft Watchlist Workflow
• Use Case:- My organization needs to understand the location of a
platform and enrich that location with watchlist information.
• Workflow:- Geoevent:
- Input – CSV and TCP input- Output – Update a feature service- GeoEvent Service
- Aircraft locations is enriched by filling out a geoform. A geofence around DC is used as an incident detector.
- Operations Dashboard- List widgets, Map Widget, Embedded Content
John FryAircraft Watchlist
Aircraft Workflow Key Components Input
Processors and filters
Field Enricher
Output
John Fry
Maritime Situational Awareness
Maritime Domain Awareness
• Use Case:- My organization needs to understand the location of vessels at sea and whether they are in
the area of a threat, or, pose a threat to operations.
• Workflow:- Geoevent:
- Input – Spire’s AIS- Output –
- Add a feature to Spatiotemporal big data store- Update an alert feature in hazard areas
- GeoEvent Service- Ingest SPIRE and add these positions to a data store to store many records, while also getting alerts when
ships enter NGA NAVAL Warning areas.
- Operations Dashboard- List widgets, Bar Charts, Indicator Widgets, Map Widgets
Maritime – GeoEvent Service
Input Output to BDS
Field Mapper
Filter Merchant
Vessels and Speed < 5
Filter Vessels in NAVAREA
Filter by Geofence not in EEZ
Calculate Field
Calculate Field
Field Mapper
Output alert feature service
What is GeoAnalytics Server?
• ArcGIS has a new way of processing spatiotemporal data that is designed for big data by leveraging distributed analytics and storage- Delivered as tools in ArcGIS GeoAnalytics Server
and ArcGIS Pro- Familiar workflows and UX in the portal Map Viewer
and ArcGIS Pro- Works with your existing GIS data and tabular data
– not just big data- Designed to perform both spatial and temporal
analysis
Aggregate PointsJoin FeaturesReconstruct TracksSummarize AttributesSummarize Within
Summarize Data
Find Locations
Analyze Patterns
Use Proximity
Manage DataFind Similar Locations
Calculate DensityCreate Space Time CubeFind Hot Spots
Create Buffers
Copy to Data Store
GeoAnalytics Server | Rich Collection of Analysis Tools
GeoAnalytics Server: analytic capabilitiesBatch analysis
Analyze Patterns
Use Proximity
Summarize Data
Calculate DensityFind Hot Spots
Create Space Time Cube
Join FeaturesAggregate PointsSummarize Within
Summarize AttributesReconstruct Tracks
Detect IncidentsFind Similar Locations
Geocode Locations
Create Buffers
Find Locations
“I want to…”
Manage Data Calculate FieldCopy To Data Store
GeoAnalyticsServer
summarize dataBatch analysis• Aggregate Points:
- “How does the spatial distribution of vehicle collisions change over time?”
- “What zip codes have the highest count of crimes incidents?”
- ”Where are there the most power outages?”
- ”What does my data look like?”
summarize dataBatch analysis• Aggregate Points:
- Input 1: Points, Input 2: Polygons (or generate bins)
- Output: Polygons
- Aggregates point data into either a square grid, a hexagonal grid, or user-supplied polygon features
- Users can choose to aggregate either spatially or spatiotemporally
- A count of points is returned for each grid cell (bins), in addition to optional attribute field statistics
summarize dataBatch analysis• Aggregate Points:
Methods of Aggregation
Spatial, into polygons:
Spatial, into bins:
summarize dataBatch analysis• Aggregate Points:
Methods of Aggregation
Spatiotemporal, into bins:
Spatiotemporal, into polygons:
Spatial, into polygons:
Spatial, into bins:
summarize dataBatch analysis• Join Features:
- “Which crime events occurred near sporting events, spatially & temporally?”
- “Which bodies of water intersect cities with populations greater than one million people?”
- “Which traffic jams occurred because of car accidents?”
summarize dataBatch analysis• Join Features:
- Input 1: Points, Polygons or Lines, Input 2: Points, Polygons or Lines
- Output: Points, Polygons or Lines
- Joins attributes from one layer to another based on spatial, temporal & attribute relationships, or any combination of those relationships.
summarize dataBatch analysis• Join Features:
Spatial
• Intersects• Equals• Near• Contains• Within• Touches• Crosses• Overlaps
Temporal
• Meets• Met by• Overlaps• Overlapped by• During• Contains• Equals• Finishes• Finished by• Starts• Started by• Intersects• Near
AttributeFeatures are matched when the field values in the join layer are equal to field values in the target layer.
Near spatial and temporal relationships require a distance/length of time to be specified. For more information see: https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm
Joins in GeoAnalytics Server
https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm
Tyson Quink
Maritime Historic Patterns
Tyson QuinkGlobal Analysis
ArcGIS Enterprisewith real-time & big data capabilities
DesktopWeb Device
live & historicaggregates & features
map & feature servicestream service
live features
ArcGISEnterprise
GeoEventServer
analytics
GeoAnalyticsServer
spatiotemporalbig data store
storage analytics
visualization
ingestion
actuation
Guides / Resources
Self-paced training and introductions to GeoEvent Server
Real-Time GIS: GeoEvent Server
• Step-by-Step Tutorials, free to download• http://enterprise.arcgis.com/en/geoevent/latest/
get-started/geoevent-server-tutorials.htm- Introduction- Stream services- Spatiotemporal Big Data Store- Notifications
• Blogs and discussions on the forum- http://links.esri.com/geoevent-forum
• Video recordings of technical workshops- http://www.esri.com/videos
• GeoEvent Best Practices
http://www.esri.com/videoshttp://proceedings.esri.com/library/userconf/proc17/tech-workshops/tw_356-376.pdf
GeoEvent Serverbest practices tutorial for multi-machine site deployment
• Available Now: http://links.esri.com/geoevent-multiplemachine
Print Your Certificate of AttendancePrint Stations Located at L Street Bridge
Tuesday Wednesday12:30 pm – 6:30 pm GIS Solutions Expo Hall D
5:15 pm – 6:30 pm GIS Solutions Expo SocialHall D
10:45 am – 5:15 pm GIS Solutions Expo Hall D
6:30 pm – 9:00 pm Networking ReceptionNational Building Museum
Please Take Our Survey on the AppDownload the Esri Events app and find your event
Select the session you attended
Scroll down to find the feedback section
Complete answersand select “Submit”
Real-Time and Big Data Analysis for Defense and Intelligence WorkflowsSlide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Operations Dashboard for ArcGISSatellite TrackingSatellite TrackingSatellite TrackingSatellite GeoEvent Workflow – Key Components�Aircraft WatchlistAircraft Watchlist WorkflowAircraft WatchlistAircraft Workflow Key Components �Maritime Situational AwarenessMaritime Domain AwarenessMaritime – GeoEvent ServiceWhat is GeoAnalytics Server?Slide Number 35Slide Number 36Slide Number 37Slide Number 38Slide Number 39Slide Number 40Slide Number 41Slide Number 42Slide Number 43Maritime Historic PatternsGlobal AnalysisSlide Number 47Guides / ResourcesSlide Number 49Slide Number 50Slide Number 51Slide Number 52
Top Related