DEPLOYMENT AND MANAGEMENT ON AWS:A DEEP DIVE ON OPTIONS AND TOOLSDanilo Poccia – Solutions Architect

AWS Elastic Beanstalk

AWS OpsWorks

AWS CloudFormation

Amazon EC2

CONVENIENCE CONTROL

HIGH-LEVEL DO IT YOURSELF

AWS Elastic Beanstalk

AWS OpsWorks

AWS CloudFormation

Amazon EC2

CONVENIENCE CONTROL

HIGH-LEVEL DO IT YOURSELF

AWS Elastic Beanstalk

AWS OpsWorks

AWS CloudFormation

Amazon EC2

CONVENIENCE CONTROL

HIGH-LEVEL DO IT YOURSELF

AWS ELASTIC BEANSTALK

QUICKLY DEPLOY AND MANAGE APPLICATIONS

Application

Application Version Environment

!

!Environment Configuration

Configuration Template

NEW FEATURES SINGLE INSTANCE

WORKER ROLE

.ebextensions/01install.config

packages: yum: libmemcached: [] ruby-devel: [] gcc: [] rubygems: chef: '0.10.2' apt: mysql-client: []

packages: msi: mysql: http://dev.mysql.com/.../mysql-connector-net-6.6.5.msi/.../

.ebextensions/02copy.config

files: "/home/ec2-user/myfile" : mode: "000777" owner: ec2-user group: ec2-user source: http://foo.bar/myfile "/home/ec2-user/myfile2" : mode: "000777" owner: ec2-user group: ec2-user content: | # this is my file # with content

.ebextensions/03run.config

commands: test: command: myscript.py cwd: /home/ec2-user env: myvarname: myvarvalue

.ebextensions/04run_after.config

container_commands: collectstatic: command: "django-admin.py collectstatic --noinput" 01syncdb: command: "django-admin.py syncdb --noinput" leader_only: true 02migrate: command: "django-admin.py migrate" leader_only: true 99customize: command: "scripts/customize.sh"

AWS Elastic Beanstalk Resource Types Reference

AWS Resource Resource Type Identifier

Amazon CloudWatch AWS::CloudWatch::Alarm

DynamoDB Table AWS::DynamoDB::Table

Amazon ElastiCache Cache Cluster AWS::ElastiCache::CacheCluster

Amazon ElastiCache Security Group AWS::ElastiCache::SecurityGroup

Amazon ElastiCache Security Group Ingress AWS::ElastiCache::SecurityGroupIngress

Amazon SNS Subscription AWS::SNS::Subscription

Amazon SNS Topic AWS::SNS::Topic

Amazon SQS Queue AWS::SQS::Queue

AWS Elastic Beanstalk Built-in Resources

Resource Name Description

AWSEBAutoScalingGroup The name of the Auto Scaling group that Elastic Beanstalk uses when it launches Amazon EC2 instances.

AWSEBAutoScalingLaunchConfiguration The name for the launch configuration settings that Elastic Beanstalk uses when it launches EC2 instances.

AWSEBEnvironmentName The name of the Elastic Beanstalk environment.

AWSEBLoadBalancer The name of the elastic load balancer used in the Elastic Beanstalk environment.

AWSEBRDSDatabase The name of the Amazon RDS database.

AWSEBSecurityGroup The name for the EC2 security group that Elastic Beanstalk uses when it launches EC2 instances.

Production Environment

Test Environment

master branch

test branch

Git Repository

git commit

git aws.push

git aws.push

Swap URLs

AWS OPSWORKS MODEL AND MANAGE

THE ENTIRE APPLICATION

STACKS

LAYERS

INSTANCES

APPS

LIFE CYCLE EVENTS

setup configure deploy undeploy shutdown

CONTINUOUS CONFIGURATIONne

w /

stop

ped

onlin

e

requ

este

d pe

nding

bo

oting

setup

configure

terminating shutting down

deploy

configure

AWS OPSWORKS SAMPLE USE CASE

GATHER AND SHIP LOGS TO AMAZON S3 AS SOON AS

THE CPU LOAD IS TOO HIGH

AWS OPSWORKS STORES 1-MINUTE METRICS

IN CLOUDWATCH

EVERY INSTANCE CREATES AN ALARM FOR HIGH CPU LOAD

CLOUDWATCH ALARM ACTION:

WRITE TO SNS TOPIC

SNS PUBLISHES TO QUEUE IN SQS

WATCHER INSTANCE

POLLSSQS FOR

NOTIFICATIONS

ON ALARM NOTIFICATION,CALL OPSWORKS API TO …

EXECUTE A SCRIPT ON THE

AFFECTED SERVER

LOGS ARE GATHERED AND

WRITTEN TO AMAZON S3

CHEF 11.10 + BERKSHELF

AWS CLOUDFORMATION INFRASTRUCTURE IS CODE

CLOUDFORMATION TEMPLATE

Stack

Stack Template

Stack TemplateConditions

Resources

References

Mappings

Outputs

Parameters

Conditions

Resources

References

Mappings

Outputs

Parameters

"Parameters" : {! "InstanceType" : {! "Type" : "String",! "Default" : "t1.micro",! "AllowedValues" : ["t1.micro", "m1.small", "m1.large"],! "Description" : "Enter t1.micro, m1.small, or m1.large. Default is t1.micro."! }!}

Conditions

Resources

References

Mappings

Outputs

Parameters

"Mappings" : {!"RegionMap" : {! "us-east-1" : {! "AMI" : "ami-76f0061f"! },! "us-west-1" : {! "AMI" : "ami-655a0a20"! },! "eu-west-1" : {! "AMI" : "ami-7fd4e10b"! },! "ap-southeast-1" : {! "AMI" : "ami-72621c20"! }! }!}

Conditions

Resources

References

Mappings

Outputs

Parameters "Parameters" : {! "EnvType" : {! "Description" : "Environment type.",! "Default" : "test",! "Type" : "String",! "AllowedValues" : ["prod", "test"]! }!},!!"Conditions" : {! "CreateProdInstance" : {"Fn::Equals" : [{"Ref" : "EnvType"}, "prod"]}!}!!"ProductionInstance" : {! "Type" : "AWS::EC2::Instance",! "Condition" : "CreateProdInstance",! "Properties" : {! "InstanceType" : "c1.xlarge",! "SecurityGroups" : [ { "Ref" : "ProdSecurityGroup" } ],! "KeyName" : { "Ref" : "ProdKeyName" },! "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}! }!}

Conditions

Resources

References

Outputs

Parameters

"Resources" :{! "MySimpleImage" : {! "Type" : "AWS::EC2::Image",! "Properties" : {! "ImageId" : "myLinuxBundle-2011-12-30",! }! }!}

Mappings

Conditions

References

Outputs

Parameters

"Parameters" : {! "MyURL" : {! "Type" : "String",! "Default" : "http://aws.amazon.com"! },!! ...!!"Outputs" : {! "URL" : {! "Value" : { "Ref" : "MyURL" }! }!}

Mappings

Resources

Conditions

Outputs

Parameters

"Outputs" : {! "URL" : {! "Value" : "http://aws.amazon.com/cloudformation"! }!}

Mappings

Resources

References

Conditions

Outputs

Mappings

References

Pseudo Parameters

Resource Properties

Parameters

Resources

Intrinsic Functions

Conditions

Outputs

Mappings

References

Pseudo Parameters

Resource Properties

Parameters

Resources

Resources : {! "MyVolume" : {! "Type" : "AWS::EC2::Volume",! "Properties" : {! "Size" : "4",! "SnapshotId" : "snap234",! "AvailabilityZone" : "us-east-1a"! }! }! }

Intrinsic Functions

Conditions

Outputs

Mappings

References

Pseudo Parameters

Resource Properties

Parameters

Resources

"Outputs" {! "MyStacksRegion" : { "Value" : { "Ref" : "AWS::Region" } }!}

Intrinsic Functions

Conditions

Outputs

Mappings

References

Pseudo Parameters

Resource Properties

Parameters

Resources "Outputs" : {! "URL" : {! "Value" : { "Fn::GetAtt" : [ "MyLoadBalancer", "DNSName" ] }! }!}

Intrinsic Functions

Name

Fn::Base64

Fn::FindInMap

Fn::GetAtt

Fn::GetAZs

Fn::Join

Ref

Stack TemplateConditions

Resources

References

Mappings

Outputs

Parameters Pseudo Parameters

Resource Properties

Intrinsic Functions

{    "Description"  :  "Create  RDS  with  username  and  password",    "Resources"  :  { !

       "MyDB"  :  {            "Type"  :  "AWS::RDS::DBInstance",            "Properties"  :  {                "AllocatedStorage"  :  "500",                "DBInstanceClass"  :  "db.m1.small",                "Engine"  :  "MySQL",                "EngineVersion"  :  "5.6",                "MasterUsername"  :  "MyName",                "MasterUserPassword"  :  "MyPassword"            }        }    } }

"AWS::CloudFormation::Init"  :  {    "config"  :  { !

       "packages"  :  {            "yum"  :  {                "mysql"                :  [],                "mysql-­‐server"  :  [],                "httpd"                :  [],                "php"                    :  [],                "php-­‐mysql"        :  []            }        },          "sources"  :  {            "/var/www/html"  :                  "https://s3.amazonaws.com/my-­‐builds/build-­‐v4.zip"        }    }

{      "Parameters"  :  {    "KeyName"  :  {            "Description"  :  "Name  of  an  existing  EC2                                              KeyPair  to  enable  SSH                                              access  to  the  instance",            "Type"  :  "String"        }    }, }

AMAZON EC2 API / SDK / CLI

aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text

for ip in $(!!

aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text!!

); do ssh -t ec2-user@$ip "sudo yum update"; done!!

for ip in $(!!

aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text!!

); do ssh -t ec2-user@$ip "cd /app ; git pull"; done!!

for ip in $(!!

aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text!!

); do ssh -t ec2-user@$ip "/app/update.sh"; done!!

AWS CLI http://aws.amazon.com/cli/!

https://github.com/aws/aws-cli!

+!./jq!

https://github.com/stedolan/jq

aws sns list-subscriptions |! jq -c '.Subscriptions[] |! select(.Protocol == "sqs")' |! while read s! do! if [ "$(aws sqs get-queue-url —queue-name! $(echo $s | jq -r .Endpoint |! sed ’s/^arn:aws:sqs:[^:]*:[^:]*://')! 2>&1 >/dev/null ;! echo $?)" != "0" ]! then! echo "Unsubscribing $s ..."! aws sns unsubscribe —subscription-arn! $(echo $s | jq -r .SubscriptionArn)! echo "Done."! fi;! done

AWS Elastic Beanstalk

AWS OpsWorks

AWS CloudFormation

Amazon EC2

CONVENIENCE CONTROL

HIGH-LEVEL DO IT YOURSELF

DEPLOYMENT AND MANAGEMENT ON AWS:A DEEP DIVE ON OPTIONS AND TOOLSDanilo Poccia – Solutions Architect