HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD ... Web Services (AWS) Google Cloud Microsoft Azure Private...

© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Steve Bertoldi, Solutions Director, MarkLogic

HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION

SLIDE: 2


Agenda Cloud computing and on premise issues

Comparison of traditional vs cloud architecture

Review of use cloud use cases

SLIDE: 3


What is cloud computing? Self service provisioning

Elasticity and scale

Pay as you use services

Efficient and economic

Infrastructure as a Service (IaaS)

SLIDE: 4


Cloud Providers Public clouds

Amazon Web Services (AWS)

Google Cloud

Microsoft Azure

Private Clouds

GovCloud

SLIDE: 5


Issues with on premise infrastructures

Infrastructure related software bugs

Frequent patching causes down time

Patches require reboot forcing outages Systems unstable after patch

Hardware stability issues

Can be difficult to troubleshoot

Virtualized infrastructures as fixed assets

SLIDE: 6


Pain points realized by customers

Customers want a stable platform

Tired of infrastructure headaches

Bare metal management costs increasing

Procurement taking longer

Administration issues

Issues with some in-house directives

SLIDE: 7


Don’t need in-house systems

No need to be close to data center

Stop hugging servers!

The best of today will eventually be outdated

SLIDE: 8


The MarkLogic AWS solution MarkLogic is supported on AWS Linux

No other external dependencies

Higher network availability

Easier to procure and administer

Better disaster/backup recovery

Different availability zones

Simpler backups with Simple Storage Service (S3)

SLIDE: 9


Gains of MarkLogic and cloud implementations

Faster build and delivery times

Scale-on-demand and elastic capabilities

Easier tools and management

Internal cost reduction

Asset recovery

SLIDE: 10


TRADITIONAL VS AWS

SLIDE: 11


System builds – the old way

CPU specifications Cores, sockets, speed

Memory Dependent on CPU Size Capacity

Motherboard Chassis/capacity

Storage RAID?

Network Fabric/topology

Send to procurement

SLIDE: 12


Building the AWS way CPU/Memory/Networking

EC2

optimized instance

Amazon EBS Volume

Elastic Block Storage (EBS) GP2 PIOPS Magnetic

Disk/Storage

Elastic Compute Cloud (EC2) CPU/Memory Networking

Amazon Machine Image (AMI)

MarkLogic

Install MarkLogic

AWS cloud

SLIDE: 13


Migrating a typical MarkLogic Cluster

3 node MarkLogic cluster 2 CPUs, 8 cores each 32 GB RAM 3 TB of local storage

SLIDE: 14


Select a region

us-west-1 N. California

us-west-2 Oregon

us-east-1 N. Virginia

4 3

3

SLIDE: 15


Select the AMI Bootstrap image for the OS

Provides details about the instance

Copies image to disk

Amazon Machine Image (AMI)

SLIDE: 16


Choose an instance type Provides compute power

Choose based upon the system needs

c4.8xlarge

32 vCPUs

60 GiB RAM

Enhanced networking

EC2 instance small EC2 instance

large

SLIDE: 17


Define instance details Select availability zone (AZ)

Us-east-1b Us-east-1c

Us-east-1d Us-east-1e

availability zone

Do you need a Virtual Private Cloud (VPC)

Networking options/subnet

Many more options

SLIDE: 18


Add storage Need to have a root volume

Size to your needs

Data volume types

GP2

PIOPS

Magnetic

3TB GP2

Amazon EBS

SLIDE: 19


Setup Security Configure Security Group

Firewall rules to control traffic

Spans availability zones

SSH keys for private/public key authentication

Launch!

Us-east-1b Us-east-1c

security group

SSH:22 admin:8001 QC:8000

SSH keys

SLIDE: 20


Building out the cluster

Us-east-1b Us-east-1c Us-east-1d

First instance in the cluster

security group

Elastic Load Balancing

Repeat for remaining nodes Install MarkLogic

Elastic Load Balancer (ELB)

SLIDE: 21


Building MarkLogic clusters in AWS Unmanaged/manual install

Launching with a MarkLogic AMI directly

AWS Marketplace Bring your own license (BYOL)

Metered

MarkLogic Cloud Formation templates

Managed clusters

SLIDE: 22


USE CASES

SLIDE: 23


Use case: AWS for testing New system part of a larger infrastructure

Target SLAs defined

4 node cluster

Customer hardware in procurement – cannot change

Testing on internal virtualized systems

SLIDE: 24


Operations and problems

Transformation hub

Large amount of record loading

Not meeting SLAs

Bad performance on local VMs

SLIDE: 25


EC2 EC2

EC2

Us-east-1b

EC2 EC2

root

data

root

data

root

data

root

data

Setup Virtual Private Cloud

Testing infrastructure

Run times < SLA

Users connect through VPN 5 m4.4xlarge instances Build MarkLogic cluster Run loading tests

mlcp

SLIDE: 26


EC2 EC2

EC2

Us-east-1b

EC2 EC2

Transform job

trans


Transformation job Runs on a single node

Not meeting SLA Added elastic load balancer

Performance < SLA Routed traffic to all nodes

SLIDE: 27


Use case: Utilizing scaling features Search and lookup during the day

Millions of documents from legacy system

Transform batch job

Cannot meet SLAs

SLIDE: 28


EC2 EC2

Us-east-1d

EC2

Initial infrastructure

Us-east-1c Us-east-1b


APP

3 node architecture Medium sized instances

Loading occurs all day SLA in place

SLIDE: 29


EC2 EC2

Us-east-1d

EC2

Scaling infrastructure



APP

EC2 EC2 EC2

Us-east-1b

4 4 4 4 4 4

4 4 4 4 4 4 8 8 8 8 8 8 8 8 8 8 8 8

Us-east-1c Us-east-1d

Primary made smaller

Second cluster added 8 vCPU each

Load to secondary

Transform scale-up Loading disabled

Scale up instances 32 vCPU each

transform

Transform on all 3 nodes Met SLAs

SLIDE: 30


EC2 EC2

Us-east-1d

EC2

Scale down and switch



APP

EC2 EC2 EC2

Us-east-1b

4 4 4 4 4 4

4 4 4 4 4 4 8 8 8 8 8 8 8 8 8 8 8 8

Us-east-1c Us-east-1d

Transform complete 2 clusters exist

Scale 2nd cluster to 8 vCPUs Swap cluster at load balancer Clear primary

Attach loader to primary

SLIDE: 31


Use case: Elastic auto-scaling architecture

Document based system with searching

Provides alerting and highlighting

3 node primary system

Need to meet peak demands

SLIDE: 32


EC2

Master cluster

Master1 Us-east-1b


EC2

Master1 Us-east-1b

EC2

Master1 Us-east-1b

3 node cluster Separate availability zone

Used for highlighting C3.2xlarge

8vCPU 15GB RAM

SLIDE: 33


EC2

Foreign node

Master1 Us-east-1b


EC2

Master1 Us-east-1b

EC2

Master1 Us-east-1b

EC2

Foreign 1


replication Single node Uses database replication Search capability Scheduled auto-scaling

Base set for normal loads Increase for peak loads Reduce for weekend loads

SLIDE: 34


EC2

Scale up

Master1 Us-east-1b


EC2

Master1 Us-east-1b

EC2

Master1 Us-east-1b

EBS snapshot

EBS snapshot

EBS snapshot

AWS Cloud Formation Custom tools

EC2

Foreign 1

EC2

Foreign 2


replication

replication Snapshot master Create new foreign node Attach snapshots

Create volumes Setup replication

Attach foreign cluster Attach replicas

SLIDE: 35


EC2

Scale down

Master1 Us-east-1b


EC2

Master1 Us-east-1b

EC2

Master1 Us-east-1b AWS Cloud Formation

Custom tools

EC2

Foreign 1

EC2

Foreign 2


replication

replication Recover resources Disable replication

Remove replicas Detach foreign node

Cleanup instance Stop instance Remove instance Remove volumes

SLIDE: 36


WRAPPING UP

SLIDE: 37


Takeaways It works!

Produce results faster and cheaper

Flexibility, scalability and elasticity

Easy to test out several configurations

No restrictions from fixed infrastructures

New enhancements and growing feature sets

Cost declining

SLIDE: 38


Q&A

HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD ... Web Services (AWS) Google Cloud Microsoft Azure Private...

Documents

Transcript of HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD ... Web Services (AWS) Google Cloud Microsoft Azure Private...