Automating ArcGIS

Deployments Using ChefCherry Lin (clin@esri.com)

Bill Major (bmajor@esri.com)

David Cordes (dcordes@esri.com)

Slides: http://bit.ly/2uiEG0p

Overview

• What, Why, When?

• Esri’s Chef Cookbooks

• Easy Cooking Demo

• Best Practices

• Case Study: Esri

What, Why, When

What is Automation and Why

• The linking of disparate systems and software in such a way that they

become self-acting or self-regulating.

• Reliable, repeatable and testable deployments and upgrades - DEV → TEST → PROD

- Moving between Data Centers

- Faster Disaster Recovery

• Quick configuration with disposable machines and rapid auto-scaling

• Automate and become more efficient at administration- Focus more on GIS tasks

- Prepare for larger scale and clouds

- Ready-to-use tools with minimum requirements to start.

SoftwareInstallation

Software Configuration

SoftwareOperation (HA, DR)

OrgAdministration

ContentManagement

Analytics

Operate

SoftwareInstallation

Software Configuration

SoftwareOperation (HA, DR)

GIS Infrastructure Automation

Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder

GIS Workflow Automation

Python API for ArcGIS, WebGISDR, Admin API,ArcPy, R-bridge for ArcGIS

OrgAdministration

ContentManagement

Analytics

SoftwareInstallation

Software Configuration

SoftwareOperation (HA, DR)

GIS Infrastructure Automation

Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder

ArcGIS Enterprise Software Setups

+=

ArcGIS

Enterprise

Portal

for ArcGIS

ArcGIS

Server

ArcGIS

Data Store

+

ArcGIS

Web Adaptor

+

ArcGIS Enterprise Software SetupsArcGIS Enterprise Software Setups

• No need to learn 3-party technology.

• Deploy exact the way you want

• Other Esri software

• Time consuming

• More manual steps

• Manual configurations following different

docs

• One time task

• For customized deployments

• Not recommended for highly available and

scalable deployments

ArcGIS Enterprise Builder

Portal

for ArcGIS

Hosting

Server

ArcGIS

Data Store

(relational + tile cache)

ArcGIS

Web Adaptor

ArcGIS

Web Adaptor

ArcGIS Enterprise Builder

• All-in-one deployment

• One GUI setup, 5 or 6 parameters

• Easy to use

• A lot less configurations

• One deployment pattern for one machine quick start

• Not for a system that needs quickly redeployed

• Not a DR deployment?

ArcGIS Enterprise Software SetupsIn the Cloud

• AWS

- Esri AMIs

- Esri CloudFormation Templates

- Python and Powershell scripts

•Microsoft Azure

- Esri Images

- Cloud Builder

•Highly recommend using our Images and tools

- Customization based on our Images and tools

- Details: AWS and Azure sessions

ArcGIS Enterprise Software SetupsEsri ArcGIS Chef Cookbooks

• IT standard automation framework

• Automate all that you can manually do

• Deploy exact the way you want. You have full control.

•Other Esri software

•Require small up-front investments

•Reliable, repeatable and testable

- Documented deployments

- High Availability and Disaster Recovery deployments

- Move between Data Centers

- Potential scale-ups

- Automate upgrades

ArcGIS Enterprise Software Setups

GIS Infrastructure Automation

Enterprise Builder

SoftwareSetups

ChefArcGIS Cookbooks

Quick start

Easy to use

Typical deployment pattern

Simple deployment

Limited control

Not for repeatable deployments

Reliable

Repeatable

Testable

Documented deployment

Full control

Esri’s Chef Cookbooks

What is Chef

• Chef

- One of the most popular IT automation platforms

- Supported on Windows and Linux

- Provide tools to take on configuration files

• Chef Components

- Cookbook

- Attribute

- Chef-Server vs. Client

• How it works

- Install ready-to-use cookbooks and define the configuration

- Run chef with the configuration file

• Recipe

- Attributes

- Action

• Property Settings: Attributes

• Run-list

ArcGIS Cookbook Recipe

Define You Own Attributes

Define You Own Run List

Steps to get started with a single machine deployment with Chef

• Check out Esri Github Chef Cookbooks website – v.3.1 just released

- https://github.com/Esri/arcgis-cookbook/wiki/Get-Started-With-ArcGIS-Cookbook

• Install the Chef Client (v. 12.5.1 or later)

• Copy ArcGIS Cookbooks to server and unzip to c:\chef

- https://github.com/Esri/arcgis-cookbook/releases/tag/v3.1.0

• Stage Setups and License Files

• Edit JSON configuration file to meet your own needs

- c:\chef\roles\webgis-windows.json

• Run chef-solo from command prompt ‘As Administrator’

- chef-solo -j C:\chef\roles\webgis-windows.json

Easy Cooking Demo

Supported ArcGIS Software

• ArcGIS Enterprise-Portal for ArcGIS

-ArcGIS Server

- Enabled Roles: Hosting, Image Server, GeoAnalytics, GeoEvent Server

-ArcGIS Data Store

- General, spatialtemporal

-ArcGIS Web Adaptor

• And other Esri Software:

- ArcGIS Pro

- ArcGIS Insight

- ArcGIS Desktop

- ArcGIS License Manager

Support ArcGIS Enterprise Deployment Patterns

Best Practices

ArcGIS Enterprise Software SetupsDistributed Deployment

• Running Chef on different machines in order from a workstation machine

- Different property files

- Runs in order as required by ArcGIS Enterprise

•Chef Server

- Central repository

- Assign roles to every machine

- Run the recipes/updates accordingly

- Setting up

-By yourself or

-Hosted Solution

ArcGIS Enterprise Software SetupsUpgrade to 10.5 or 10.5.1

• Required properties

-New Version #

-New Setups

-New License File

•Recommend to use Chef deploy before upgrade

-Easier to get the properties right

ArcGIS Enterprise Software SetupsWork with Other Automation Tools

• Esri ArcGIS Chef Cookbooks is ArcGIS deployment tool

• ArcGIS Infrastructure part of your automation

• Work with GIS workflow automation tools

-Python API for ArcGIS

-WebGISDR tool

• Other software deployment automation tools

• Whole system infrastructure automation

-High Availability and Disaster Recovery deployment

-Moving to different data centers

ArcGIS Enterprise Software SetupsChef Deployment Options

ArcGIS Enterprise Software SetupsDisconnected Environment Considerations

• Make sure you have proper Esri license files staged for use

• Software setups in a common location for access

• Setting up your own Chef Server

• Having Chef Client installation staged for installation

ArcGIS Enterprise Software SetupsTricks and Tips

• Consider creating a base image for some environment

- Have some processes done, e.g, setups

- Have some components baked in, e.g. base map data, security updates, etc.

•Chef Client version 12

•Set password in an environment variable

•“-l debug” for debug

•Attributes value of True or False, no “”. e.g. “configure_autostart”: true

•ArcGIS Cookbooks support enabling hosting server role and GeoAnalytics Server role.

•Support security configuration for ArcGIS Server.

ArcGIS Enterprise Software SetupsKey Points about ArcGIS Chef Automation

• Automate Your Manual Work

• Doesn’t require programming skills

• Need plan ahead

-Design

-Configuration

-Testing

•Repeatable

-Success in production environment

-Less downtime in production

-Faster recovery

Case Study : Esri

ArcGIS Enterprise : New Software Daily

Many software changes each day.

Daily testing prevents accumulation complexity.

Test Automation : Not just automated functional testing

Configuration automation

• Software installed and configured many times per build• Many platforms• Many software components• Many architectures• Limited resources

Hundreds of Scripts

• Every tester had their own configuration script

- Python- .Net - Java- Perl- Batch files- Etc

• Wasted effort• Hard to maintain

ArcGIS Cookbooks to the Rescue

Solution: Use our cookbooks

• Tested• Reliable• Maintained

Challenge #1: But I already have a script….

“Not invented here”

• Resistance to moving away from existing scripts.• Time eventually removed this barrier from most of our test

teams.

Challenge #2: But I don’t have time to learn a new technology...

Training

• In groups• One-on-one• Troubleshooting alias

Outcome: Successful adoption, focus on the functionality

Has taken time but we spend

• Less time maintaining and troubleshooting scripts.

• More time focused on functional testing.

Step 1: Prepare the image

Multi-Machine Configuration

Servertest-server.dev.geocloud.com

Portaltest-portal.dev.geocloud.com

Data Storetest-datastore.dev.geocloud.com

HostingRegistered

Tomcat + Web Adaptortest-datastore.dev.geocloud.com

Multi-Machine Architecture

Command

Machine

Baked an Image in Three Easy Steps

Install Chef

Copy Cookbook

System

Requirements

Step 2: Set the Parameters

Multiple Machine Automation

One Deployment - Many Machines - One Set of Parameters

Portal Server

Data Store Web Server

Parameters

Step 3: Create Role-Specific Runlists

Multiple Machine Automation

Run Lists - What is Run On Each Machine

Portal Server

Data Store Web Server

Parameters

Run Lists - What is Run On Each Machine

Portal Server

Data Store Web Server

Parameters

Run Lists - What is Run On Each Machine

Portal Server

Data Store Web Server

Parameters

Run Lists - What is Run On Each Machine

Portal Server

Data Store Web Server

Parameters

Run Lists - Federation - Back to Portal

Portal Server

Data Store Web Server

Parameters

Step 4: Create Shell Script (Batch File)

Multiple Machine Automation

Script Writing for Multiple Machines

Command Machine

● Scripts stored on command machine● Copied to test machines at deploy time● Run with ssh

Step 5: Run Deployment Script

Multiple Machine Automation

Run: Multi-Machine Configuration in 30 minutes

Q&A and Slides

Slides: http://bit.ly/2uiEG0p