Data Store Management Best PracticesMark CarlsonLaurence Clinton

Session Agenda

1. Overview2. Installation and Configuration3. Backing up the data store4. Restoring the data store5. Moving data store to a new machine6. Demo7. Upgrading8. Questions

Mark CarlsonOverview

+=ArcGIS

Enterprise

Portal for ArcGIS

ArcGIS Server

ArcGIS Data Store

(relational + tile cache)

+

ArcGIS Web Adaptor

+

+=ArcGIS

Enterprise

Portal for ArcGIS

ArcGIS Server

ArcGIS Data Store

(relational + tile cache)

+

ArcGIS Web Adaptor

+ +Spatiotemporal

Understanding Data Stores

Relational Tile Cache Spatiotemporal

Portal for ArcGIS

ArcGIS Server

ArcGIS Data Stores

ArcGIS Web Adaptor

ArcGIS Web Adaptor

Understanding Data Stores

Portal for ArcGIS

Hosting Server

ArcGIS Web Adaptor

ArcGIS Web Adaptor

Relational

Registered

Survey123for ArcGIS

Insights for ArcGIS

Workforcefor ArcGIS

GeoPlannerfor ArcGIS

Publish Hosted Features

Understanding Data Stores

RelationalPortal for ArcGIS

Hosting Server

ArcGIS Web Adaptor

ArcGIS Web Adaptor

Survey123for ArcGIS

Insights for ArcGIS

Workforcefor ArcGIS

GeoPlannerfor ArcGIS

Registered

Publish Hosted Features

Tile Cache

Scene Layers

SceneViewer

Understanding Data Stores

RelationalPortal for ArcGIS

Hosting Server

ArcGIS Web Adaptor

ArcGIS Web Adaptor

Survey123for ArcGIS

Insights for ArcGIS

Workforcefor ArcGIS

GeoPlannerfor ArcGIS

Registered

Publish Hosted Features

Tile Cache

SceneViewer

Spatiotemporal

Scene Layers

GeoAnalyticsServer

GeoEventServer

Tracker forArcGIS

GeoAnalytics ToolsArcGIS Pro, Map Viewer

Mark CarlsonInstallation and Configuration

Installation and Configuration | Considerations

• Install User- OS or Domain service account

• Estimate amount of disk space to account for installation, configuration, and data - Minimum requirements

- To install & configure- 13 GB of available disk space (with no data)

- You need to increase the available disk space based on your estimated data size

- Memory - The following minimum memory requirements assume you install each type of ArcGIS Data Store on its own machine

- Relational data store = 8 GB

- Tile cache data store = 8 GB

- Spatiotemporal big data store = 16 GB

- Highly recommended to be installed on dedicated machine- By default, consumes 50% of available memory

• Threshold- Data stores placed in read-only mode or stopped when machine disk space drops below threshold

ArcGIS Data Store Installation and Configuration Notes

A

• Communication used from ArcGIS Enterprise to the Data Store

- Relational : 9876 (TCP)

- Tile Cache : 29080 (HTTP), 29081 (HTTPS)

- Spatiotemporal (Big Data Store) : 9220, 9320 (TCP)

ArcGIS Data Store Installation and Configuration Notes

A

• Configure with ArcGIS Server - Web browser (Datastore Configuration Wizard)

- https://datastoremachine.domain.com:2443/arcgis/datastore

ArcGIS Data Store Installation and Configuration Notes

A

• Configure with ArcGIS Server | Option to configure additional Data Stores later - Web browser (Datastore Configuration Wizard)

- https://datastoremachine.domain.com:2443/arcgis/datastore

ArcGIS Data Store Installation and Configuration Notes

A

• Configure with ArcGIS Server | Spatiotemporal Data Store- Web browser (Datastore Configuration Wizard)

- https://datastoremachine.domain.com:2443/arcgis/datastore

ArcGIS Data Store Installation and Configuration Notes

A

• Configure with ArcGIS Server - Command line utility

- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores relational

- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores tilecache

- configuredatastore https://myserver.domain.com:6443/arcgis/admin siteadmin Pa$$w0rd c:\data\ --stores spatiotemporal

• Configure Backups- Command line utility

- ./configurebackuplocation.sh --location 'type=s3;location=s3bucket-name;name=awsbds;username=aws_access_key_id;password=aws_secret_access_key' --operation register --store spatiotemporal

ArcGIS Data Store Installation and Configuration Notes

A

• Command Line Utilities

- http://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-utility-reference.htm

- http://enterprise.arcgis.com/en/portal/latest/administer/linux/data-store-utility-reference.htm

- Example: describedatastore

ArcGIS Data Store Installation and Configuration Notes

A

- Example: describedatastore

ArcGIS Data Store Installation and Configuration Notes

A

• Standby Datastores- Applies to both Relational & Tile Cache

Primary

Network File Share for Data Store Backups

• Install & configure Data Store on Primary• Validate ports are open for standby

• Install & configure Data Store on Standby with same Domain Account used on the Primary

• Configure Standby with the same Hosting Server Site

• Verify the configuration summaryStandby

Ports

Replication

ArcGIS Data Store Installation and Configuration Notes

A

• Standby Datastores- Applies to both Relational & Tile Cache

Primary

Network File Share for Data Store Backups

• In the event the primary becomes unavailable, the standby machine becomes the primary machine

Standby

Replication

Primary

Sample ArcGIS Enterprise

A

Standby

Relational Data Store

Network File Share for Data Store Backups

Spatiotemporal Big Data Store

Portal for ArcGIS

Hosting ServerArcGIS Web Adaptor

ArcGIS Web Adaptor

AWS S3 BucketBackup Location

Primary

Tile Cache Data Store

Federated

Registered

Big Data File Share

GeoAnalyticsServer

GeoEventServer

Laurence ClintonBacking up the data store

• Implement good backup policy- Backups can be taken for all data store types at 10.5.x and above- Backups grow and consume disk space over time- Configure backup location to a shared drive

- Ensure ArcGIS Data Store account has permissions on shared drive- Advantages:

- Ensures restore of data store if machine goes down

- Ensures machine data store configured on does not run out of disk space

- Configure additional non-default back up locations also supported (file, S3, Azure Blob store)- Relational, spatiotemporal data store types

Usage: configurebackuplocation --location <backup-location> [configure-backup-location-options]

Supported configure-backup-location-options:

[--operation <change | register | unregister>]

[--store <relational | tileCache | spatiotemporal>]

[--prompt <yes|no>]

Usage: backupdatastore [backup-name] [--store <relational|tileCache|spatiotemporal>]

[--prompt <yes|no>]

Backing up the data store

• Backup schedule and retain days • Schedule

- How often backups are automatically generated

• Retain days- How long backups persist

• Default values - Retain days 7- Backup schedule every 4 days

Usage: updatebackupretaindays <num-of-days>

Usage: updatebackupschedule [--store relational| tileCache| spatiotemporal]

<--frequency num-of-days> [--starttime HH:MM:SS]

Backing up the data store

• Considerations for enabling / disabling point-in-time recovery for relational data store

- Can recover to specific date / time but less disk space required if disabled- Disabled by default at 10.5.1

- Point-in-time needed for WebGISDr tool- Changedbproperties: pitr enable | disable

Usage: changedbproperties --store <relational|tileCache|spatiotemporal> [configure-options]

Supported configure-options:

[--disk-threshold-readonly <disk-threshold-readonly>]

[--max-connections <max-connection-number>]

[--pitr <enable|disable>]

[--heap-size <heap-size-in-MB>]

[--rebalance <true|false>]

[--max-rebalance-off <time-in-minutes>]

[--allocation <true|false>]

[--prompt <yes|no>]

Backing up the data store

Laurence ClintonRestoring the data store

• Restoring options- Restoredatastore

Usage: restoredatastore[restore-options]Supported restore-options (Time need be in UTC):

[--store <relational|tileCache|spatiotemporal>][--target <most-recent|yyyy-mm-dd-hh:mm:ss>|source-backup-name][--source-loc <source-backup-loc>][--bound <true|false>][--data-dir <data_dir>][--server-url <server_url>][--server-admin <admin_user>][--server-password <admin_password>][--loaddata true|false][--prompt <yes|no>]

• Demo

Restoring the data store

Laurence Clinton

Moving the data store to a new machine

Moving ArcGIS Data Store to a different server

• There are times when organizations may need to move the ArcGIS Data Store content to a different server, such as a server hardware upgrade.

• Steps- Install ArcGIS Data Store on new server- Configure relational ArcGIS data store to ArcGIS Server site as standby Data Store- Once configured and data is fully replicated, promote the standby Data Store to primary- Remove the standby data store from the site (the original primary)- Uninstall Data Store software from original server; decommission.

A

Laurence ClintonDemo

Laurence ClintonUpgrading

• Stop service and copy content directory to safe location• Generate a backup / export• Ensure ArcGIS Data Store account has permissions on content directory• Server must be upgraded before ArcGIS Data Store

• Generating backupsbackupdatastore [backup-name][--store <relational|tileCache|spatiotemporal>] [--location <backup-location>] [--prompt <yes|no>]

Considerations before upgrading

New upgrade procedure

• Install software on all data store machines prior to configure- Uninstall / install upgrade no longer supported

• Relational, Tile cache, Spatiotemporal Big Data Store- Configure on any machine in the configuration

- Remaining machines are automatically upgraded

New upgrade experience

• Run describedatastore command• Validate data stores from ArcGIS Server Manager or Admin

- https://gisserver.domain.com:6443/arcigs/admin/

• Verify services still work

After Upgrading

Mark Carlson Laurence ClintonQuestions

Complete answersand select “Submit”

Scroll down to find the feedback section

Select the session you attended

Download the Esri Events app and find your event

Please Take Our Survey on the App