Deploying Live Data with High Availability Servers -...
Transcript of Deploying Live Data with High Availability Servers -...
-
Deploying Live Data with High Availability Servers
Paul DoddDerrick Burke
-
Overview
Definitions
Server Deployment Options
Data Deployment Options
Data Update Options and Architecture
Looking at a Live Feed
-
http://www.esri.com/services/disaster-response
http://www.esri.com/services/disaster-response
-
Definitions Clearing the Air
-
Live Data, what do we mean?
Frequently Changing Content!
Real-Time Data- Changes moment-by-moment- Trickle or burst- No set intervals
Near Real-Time Data- Periodic or Batch changes- Bulk updates typical- Timed intervals
-
What Constitutes HA (High-Availability)?
Wikipedia- An implementation that ensures a prearranged level of operational
performance during a contractual period of time. (paraphrased)
Availability in Nines (scheduled and unscheduled downtime allowance per year)- Two.Five 9s (99.5%) = ~44 Hours- Three 9s (99.9%) = ~8 3/4 Hours- Four 9s (99.99%) = ~7/8 Hour or ~53 minutes
Performance and Monitoring- Predictable and Stable performance regardless of growth.- Up does not mean available Monitor your end-points
-
Design Factors for HA(as it applies to ArcGIS)
Target Audience- Limited Users (consistent) Static or Controlled Growth- Public (unlimited and varying demand) Auto Scaling
Application or Behavior- Read/Only Viewers or 3rd party source- Read/Write Editors, Geoprocessing (CPU intensive)
-
Design Factors for HA (cont)(as it applies to ArcGIS)
Data Requirements- Static or rarely updated Cache or Custom Extract- Random updates RDBMS- Frequently replaced or updated Stream, RDBMS, or Custom- Complexity How heavy is your data?
-
Architecting HA
Limit Points of Failure- CPU Multi-Site / Multi-Node- Storage Fault Tolerance with multiple I/O paths- Network Load Balanced with Failover
Cloud-Based or Internal or Hybrid- Greater flexibility, less infrastructure- Factor Physical vs. Virtual cost
-
Architecting HA (cont)
Plan for and test Backup / Failover strategies
Keep it simple!!!- An over-complicated design can open the door to failures!
-
Server Deployment Options
-
ArcGIS Server - HA Deployments
Multi-Machine Sites- Single-Cluster- Balance workload
Multi-Machine Clustered Sites- Divide services by function / purpose- Can be chatty- Increased administration
Simple / Dynamic Expansion Shared Config Store Scalable w/restrictions (shared resources)
-
ArcGIS Server - HA Deployments (cont)
Single-Machine HA Sites (active-active)- Very scalable (cloud)- Quick expansion (cloud)- Local Config Store / Data- High Volume- Public Access- External Network Load Balancer (NLB)
Controlled Administration- Structured Sites (staging / production)
Advanced solution
-
ArcGIS Server - HA Deployments (cont)
ArcGIS Online Dont forget!- Esri managed- Scaled- Load balanced- Minimal administration- Three 9s (99.9) for Hosted Feature Services
-
Data Deployment Options
-
ArcGIS Server - Data Deployment Options
Locally Managed- File Geodatabase
- Read-Only Services- Local to each Server
- Enterprise Geodatabase- Read-Write Services- Centralized RDBMS
- Supports Failover and Redundancy- Localized RDBMS on each Server
- Redundant, but can be difficult to maintain
Cloud Managed- Microsoft Azure?- Amazon RDS?- ArcGIS Online
- Hosted Feature Services- Map Services
-
Data Update Options and Architecture
-
Updating Live Data Formats
Real-Time Data- Sensors
- Vehicles- Manufacturing / Industrial- Field equipment- SCADA
- Internet Resources- Streaming (Twitter)- Wide Range of Formats- CSV, XML, KML, RSS, GeoRSS,
Near Real-Time Data- File based
- Shapefile- CSV, XML, KML, RSS, GeoRSS, - Raster / Spatial Analysis- NetCDF- BIN
- Bulk / Batch- Heavy Geoprocessing / Enrichment- Data Transformation- Data Aggregation (multiple sources)
-
Updating Live Data Options
GeoEvent Extension (GEE)- Selected Content
Aggregated Live Feed (ALF) methodology
-
Updating Live Data(what do I choose?)
SCADA System w/Alerts- GeoEvent Ext.
Raster Weather Data- ALF ArcPy w/Spatial Analyst
Vehicle Tracking- GeoEvent Ext. w/GeoFence
Shapefile Processing- ALF ArcPy
Source / Function GEE ALFSensors Networks X *Vehicle Tracking X *
Raster XWeather Data * X
Shapefile XFile GDB X
XML / KML X XCSV X X
JSON X XRSS / GeoRSS X X
BIN XNetCDF X
Internet Streams X *Geoprocessing X
GeoFence X ** = Depends on format and / or architecture
-
ArcGIS Server GeoEvent Extension
Data Bridge
Update / Output- Feature Service- Expose Stream Service- ArcGIS Online- Others
HA w/Clusters
-
Lets dive into GeoEvent Extension
Input Connector Read
Decision Filter
Compute Processor
Output Connector Write
-
ArcGIS Server GeoEvent Extension (cont)
GeoFence Alerting- Polygon(s)- Spatial Filter
Position Tracking- Inside / Outside- Enter / Exit
-
Aggregated Live Feed (ALF) Methodology
Python Solution GP w/ArcPy
Update / Output Read-Only File GDBs Feature Services OthersAnalysis
Flexible & Extensible
HA w/Redundancy
-
ALF Methodology NDFD Precipitation, Raster Processing
http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer
http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer
-
ALF Methodology Cloud / Distributed Feed
Internet
Log Files
WorkFileGDB
Feed Routine(Aggregator)
DeploymentLogic
Data
FileGDBArchive
Zip File
Feed AggregatorALFprocessor.py
Data DeploymentALFdeployer.py
Internet
WorkFolder
Download andDeployment
Logic
LiveFileGDB
ArcGIS Server
PublishedMap Service
AmazonS3
Common Storage(External or Internal)
-
ALF Feed Construction Whats required
Pre-requisites- Python- ArcPy- Others (.py, 7zip, degrib, )
Optional- IDE PyScripter, VS,
Components- ALFlib.py
- Function / Class Library
- ALFprocessor.py- Feed Processor +- .cfg (data processing logic)
- ALFdeployer.py- Archive Distribution
-
Lets dive into a ALF Live Feed
Flickr: Toolmantim
http://www.flickr.com/photos/toolmantim/6170448143/
-
Questions & Comments?
AGOL Community Page (samples, scripts) http://esriurl.com/LiveFeed
GeoEvent Extension Resources: http://server.arcgis.com/en/geoevent-extension/
Live Data Feeds : http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-
added/ http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds
http://esriurl.com/LiveFeedhttp://server.arcgis.com/en/geoevent-extension/http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-added/http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds
-
Rate This Sessionwww.esri.com/RateMyDevSummitSession
-
[email protected]@esri.com
@derrickburke
Thank You!
Slide Number 1Slide Number 2Slide Number 3OverviewSlide Number 5Slide Number 6Slide Number 7Definitions Clearing the AirLive Data, what do we mean?What Constitutes HA (High-Availability)?Design Factors for HA (as it applies to ArcGIS)Design Factors for HA (cont) (as it applies to ArcGIS)Architecting HAArchitecting HA (cont)Server Deployment OptionsArcGIS Server - HA DeploymentsArcGIS Server - HA Deployments (cont)ArcGIS Server - HA Deployments (cont)Data Deployment OptionsArcGIS Server - Data Deployment OptionsData Update Options and ArchitectureUpdating Live Data FormatsUpdating Live Data OptionsUpdating Live Data (what do I choose?)ArcGIS Server GeoEvent ExtensionLets dive into GeoEvent ExtensionArcGIS Server GeoEvent Extension (cont)Aggregated Live Feed (ALF) MethodologyALF Methodology NDFD Precipitation, Raster ProcessingALF Methodology Cloud / Distributed FeedALF Feed Construction Whats requiredLets dive into a ALF Live FeedQuestions & Comments?Rate This SessionSlide Number 36