Moving Data Analytics to the Cloud

"The views expressed in this presentation are those of the author(s) and do not necessarily reflect the official policy or position of the Air Force, the Department of Defense, or the U.S. Government."

2

• Why the Cloud• Obstacles to Cloud Migration• Our Cloud Data Analytics Experience• Use Case & Strategy Examples• Demonstration AWS Tools & Services• Cloud Migration Demonstration• Cloud Analytics Demonstration

Discussion Topics Cloud Migration Agenda

3

Reasons to Migrate/Why the Cloud

• Reliable backup/recovery planning and implementation• Improved control of development & test environments

• Faster transition of access• Standardize repository use & configuration management for version control• Much finer control of development & test environment costs• Promotes prototyping without long term capital investment

• Much simpler vertical and horizontal scaling• Storage optimization• IO optimization• Compute optimization

• Move from license-centric platforms to service-centric platforms (open source)• Architecting & provisioning unique or ‘fit for purpose” capabilities

4

So what do we do?

• Some easy wins for the cloud

• Program managed development and test environments• Web service/API integration

• Migration/export for data analytics & reporting

5

Obstacles to Migration• Fear of losing control of “my system”

• System is still there to be managed by you or your designee• Security Overload – FedRAMP, NIST, RMF etc.…

• Underlying system is similar and sometimes the same• Majority of your vendors are there• Forward leaning technical solutions face the same issues OnPrem• Cloud is typically more secure than OnPrem out of the box

• Acquisition Process• Treat as modernization or Tech Refresh• CCE migration team

• Fork Lift vs Refactor Freeze• Fork Lift usually faster• Refactor takes more native advantage of cloud architecture

• Complex legacy systems with large data sets• Many aspects of these systems don’t need to be complex or legacy• Data is extremely portable

6

Research Data Exchange (RDE)Project Background

• Platform to transportation data sharing system• Uses IOT devices to gather vehicle based traffic and route based metrics• Includes Real-Time Data Capture• Supports a wide range of development testing and demonstration scenarios for multi-modal

transportation applications• Uses AWS Infrastructure• Effectively manages compute costs• Promotes data use by simplifying data

capture and standardized exposure APIs

7

Data Analytics Use Case

• Have a legacy highly transactional system• Want to expand reporting and analysis capability• Both reporting and analysis capabilities are on demand• Have both canned/scheduled reports and ad hoc queries• Want flexibility in analysis data content• Don’t have the time/resources to move the existing large scale application• Don’t want capital investment associated with hardware and software buildout such as a

traditional Data Warehouse or Data Lake

SQL DBCurrent Data

StoresExisting

Application

Ad Hoc Storage

ExtractMigrate Process/

Analyze Expose

8

Choosing AWS as the cloud solution

• Demonstration AWS Tools & Services

• AWS Direct Connect• AWS EC2 Memory Optimized Instance• AWS DMS Data Migration Service• AWS Schema Conversion Tool• AWS Aurora & Athena• AWS Glue• AWS S3• AWS QuickSight

9

Use Cases for Migration or Hybrid• Example per purpose database

10

Cloud Migration Approach

11

Cloud Migration Demonstration

To AWS Data Migration Demonstration

https://console.aws.amazon.com/console/home?region=us-east-1#

12

Cloud Analytics Approach

13

Cloud Data Analytics Demonstration

To AWS Data Analytics Demonstration

https://console.aws.amazon.com/console/home?region=us-east-1#

14

Summary

• The Cloud offers a lot of advantages over on-prem.• There are several “easy win” use cases to get started.• The obstacles for cloud migration aren’t as daunting as it initially looks.• If you can’t move your current system, you can ‘fit for purpose’ instead of further tying

yourself to the legacy system.• Data Analytics & Reporting are a simple way to get started.• Tooling for Data Migration is excellent.• Tooling for Data Analytics and Reporting is simple to use and powerful.• IndraSoft is ready, willing and able to help you through the process!

15

Questions