Moving to the Cloud

Nate Aune (Presented by Sally Kleinfeldt)

Jazkarta.comPlone Symposium East 2011

Penn State University

Agenda

• Benefits of cloud computing

• What services does Amazon provide?

• 5 minute launch of Plone on EC2

• Case study: Rhaptos hosting on EC2

• Questions?

What makes the Cloud so attractive?

Abstract resourcesFocus on your needs, not on hardware specs.

As your needs change, so should your resources.

On-demand provisioningAsk for what you need, exactly when you need it.

Get rid of it when you don’t need it.

Scalability in minutesScale out or in depending on usage needs.

Pay per consumptionNo contracts or long-term commitments.

Pay only for what you use.

Efficiency of ExpertsUtilize the skills, knowledge and resources of experts.

Most applications need:

• Compute

• Storage

• Messaging

• Payment

• Distribution

• Scale

• Analytics

Amazon delivers this

Characteristics of Truly Scalable Service

• Increasing resources results in a proportional increase in performance

• capable of handling heterogeneity, loosely coupled

• operationally efficient

• resilient

• becomes more cost effective when it grows

Flexible infrastructure

• Elastic Computing Cloud (EC2)

• Amazon Machine Images (AMI)

• Elastic IPs

• Elastic Block Storage (EBS)

• EBS bootable AMI

• Auto-scaling

• Elastic Load Balancing (ELB)

Other services

• Simple Storage Solution (S3)

• CloudFront (CDN)

• CloudWatch (monitoring)

Compared to Rackspace

• Rackspace is still catching up to AWS (just announced load balancing a few weeks ago)

• Rackspace is a bit easier to get started with

• sends you the root password

• machines already have a static IP

• the disk storage is persisted

• Prices are comparable to AWS

• Cheapest Rackspace VM is $14/mo

Private clouds

• Provision virtual machines on your own hardware

• Pros: you control your own machines.

• Cons: you have to manage your own machines

• Examples: Eucalyptus, OpenStack

http://open.eucalyptus.comhttp://openstack.org/

http://uec-images.ubuntu.com/

Text

mr.awsomeBuildout recipe to manage EC2 instances

Add an AWS part to buildout[buildout]

parts = aws

[aws]

recipe = zc.recipe.egg

eggs = mr.awsome

entry-points =

aws=mr.awsome:aws

assh=mr.awsome:aws_ssh

arguments = configpath="${buildout:directory}/etc"

Make a etc/aws.conf file[securitygroup:demo-server]description = Bristol Demo Serverconnections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0 tcp 8080 8080 0.0.0.0/0

[instance:demo-server]keypair = bristol-keypairsecuritygroups = demo-serverregion = us-east-1placement = us-east-1ainstance_type = t1.microimage = ami-480df921startup_script = startup-demo-serverfabfile = fabfile.py

Make a bash script: etc/startup-demo-server

#!/bin/bash

set -e -x

export DEBIAN_FRONTEND=noninteractive

wget http://www.enfoldsystems.com/pubkey.gpg

apt-key add ./pubkey.gpg

echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid universe" >> /etc/apt/sources.list

apt-get update && apt-get upgrade -y

apt-get install plone-default -y

/etc/init.d/plone-default

Set your access keys

#!/bin/bash

export AWS_ACCESS_KEY_ID="<your_access_key_id>"

export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"

Set these environment variables.

$ source setkeys.sh

$ env

File: setkeys.sh

Start the demo server$ ./bin/aws start demo-server

INFO: Instance 'demo-server' unavailable

INFO: Creating instance 'demo-server'

INFO: Instance created, waiting until it's available

....

INFO: Instance 'demo-server' available

INFO: Instance running.

INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com

INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com

Benefits of EC2• Seamless disaster recovery with repeatable

deployments using buildout and mr.awsome

• Easy launching of testing, staging and QA servers that are isolated from your production environment (stop when not used to avoid paying for them)

• Pay-for-what-you-use metered pricing to launch servers on-demand during peak periods

• Auto-scaling to launch new Zeo client machines and shut them down when no longer needed

•

Connexions & RhaptosA case study in deploying OSS to the cloud

s

Log In Contact Us Report a Bug

Search

Home Content Lenses About Us Help MyCNX

FEATURED CONTENT

New features

1 2 3

MY ACCOUNT

SPOTLIGHT

You are here: Home

Connexions is:a place to view and share educational material made ofsmall knowledge chunks called modules that can beorganized as courses, books, reports, etc. Anyone may viewor contribute:

authors create and collaborate

instructors rapidly build and share custom collections

learners find and explore content

More about us ...

Advanced Algebra IIThe world of second-yearalgebra comes alive in KennyFelder's Advanced Algebra II.Felder, a high school mathteacher in North Carolina,designed his course using anontraditional approach to a verytraditional subject. Centeredaround a Homework andActivities book, along withaccompanying Conceptual

Explanations and Teacher's Guide collections,Advanced Algebra II tackles topics by focusing oncomprehension rather than straight lecture delivery.

Advanced Algebra II was selected by California's FreeDigital Textbook Initiative to be included in a list offree textbooks available to California schools startingin Fall 2009.

Collaborative StatisticsCollaborative Statistics waswritten by two faculty members

FIND CONTENT

16547 reusable modules woven into 1007 collections.

or browse by ...

Language

Popularity

Title, author, etc.

CREATE CONTENT

Creating content in Connexions is as easy as 1, 2, 3:

Get an accountand log in to

your workspace.

Make a modulefrom scratch or

convert it from aWord doc.

Publish yourworks, sharing

them with the world.

Jump right in

Get an account

How to create a module inminutes

How to create a collectionwith existing modules

Guides and tutorials

New author guide

Connexions Tutorial andReference

Username

Password

Log in

Get an account

Forgot your password?

User feedback"I enjoy usingtheConnexionssite and thephilosophybehind itencouragesone to learnnew fields and

to network with others. Yoursite is my new homeschool onthe internet. It is helping meto grow as a writer ofacademic materials as I readwhat others have written so asto shape my own. I am alsoamazed at the range ofinstitutions where your writerscome from. Thanks for the useof these great facilities."

Go

ArtsBusinessHumanitiesMathematics and StatisticsScience and TechnologySocial Sciences

Search Content

Subject

Search

Login Preferences Help/Guide About Trac

Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search

Start Page Index History Last Change

Welcome to Connexions and Rhaptos Software DevelopmentConnexions ( cnx.org) is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, andjournals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality materialto be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site andEnterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecturehere.

Enterprise Rhaptos Connexions and Rhaptos Development

API's for Accessing Data Quality Assurance and Testing

Enterprise Rhaptos

Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime viathe World Wide Web. Rhaptos' unique features of permanent versions of all materials, a simple, semantic document language, and a powerfullensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a trulyreusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own privateauthoring platform and publishing repository.

Installation Instructions for Enterprise Rhaptos

API's for Accessing Connexions Data

Table of ContentsWelcome to Connexions and Rhaptos Software DevelopmentEnterprise RhaptosAPI's for Accessing Connexions Data

Accessing content, feeds, and statisticsDownloading, transforming and branding contentSoftware Architecture and Application Programming Interfaces

Connexions Rhaptos DevelopmentDeveloper Resources (Getting involved)Roadmap, Designs, DesiresStarting Points for Connexions and Rhaptos Software Development

Quality Assurance and TestingTest, Release, and Configuration DocsUsing the Connexions Rhaptos Trac Bug System

Release and Configuration Details for Connexions and RhaptosMore about TRAC

Virtualization of Rhaptos

Two primary reasons to move to the cloud:

1) Hurricane territory (disaster recovery)

2) Increase adoption by other universities

Cost was also a consideration but not a primary reason for carrying out the project.

Deploy to Amazon EC2 with Python scripts

• Define server profiles with OS dependencies and turn into an AMI (Amazon Machine Image)

• Install & configure application on EBS volume

• Take snapshot of volume

• Launch new AMIs and attach EBS volume created from saved snapshot.

Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009

Distributed Architecture

Internet

Webserver

Cache Sys

Load Balancer

s1

s3

s4 s5

s2

10

proxy

frontend1

frontend2

frontend3

backend

[macro:base-securitygroup]connections = tcp 22 22 0.0.0.0/0 tcp 80 80 0.0.0.0/0

[securitygroup:backend]<= macro:base-securitygroupdescription = Bristol backend machine running Zeo server

[securitygroup:frontend1]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 1connections = tcp 8080 8080 0.0.0.0/0

[securitygroup:frontend2]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 2connections = tcp 8080 8080 0.0.0.0/0

[securitygroup:proxy]<= macro:base-securitygroupdescription = Bristol proxy machine running Apache, Varnish, HAProxy

aws.conf - define security groups

[macro:base-instance]image = ami-480df921keypair = bristol-keypairregion = us-east-1placement = us-east-1cinstance_type = t1.microfabfile = fabfile.py

[instance:backend]<= macro:base-instancesecuritygroups = backendstartup_script = backend-startup.ship = xxx.xxx.xxx.xxx

[instance:frontend1]<= macro:base-instancesecuritygroups = frontend1 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx

[instance:frontend2]<= macro:base-instancesecuritygroups = frontend2 startup_script = frontend-startup.ship = xxx.xxx.xxx.xxx

[instance:proxy]<= macro:base-instancesecuritygroups = proxy startup_script = proxy-startup.ship = xxx.xxx.xxx.xxx

[buildout]parts = backend-script frontend-script proxy-script

[backend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/backend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = backend.cfg

[frontend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/frontend-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = frontend.cfg

[proxy-script]recipe = collective.recipe.templateinput = deployment/templates/proxy-startup-script.shoutput = deployment/proxy-startup.shsvn_url = http://svn.jazkarta.com/.../bristol-buildout/trunkbuildout_config = proxy.cfg

#!/bin/bashset -e -xexport DEBIAN_FRONTEND=noninteractiveapt-get update && apt-get upgrade -y

echo "----- Install system packages for Plone"apt-get -y install python2.6-dev subversion build-essential

echo "----- Create an unprivileged user zope with password disabled"adduser zope --disabled-password --gecos ""

echo "----- Check out the buildout"cd /home/zopeecho 'p\n' | svn co http://svn.jazkarta.com/.../.../trunk buildout

echo "----- Make the zope user the owner of this directory"chown -R zope:zope buildout

echo "----- Run the buildout as user zope"cd buildoutsudo -u zope python2.6 bootstrap.py -c backend.cfgsudo -u zope ./bin/buildout -v -c backend.cfg

echo "----- Start Zeo processes"sudo -u zope ./bin/supervisordsudo -u zope ./bin/supervisorctl status

echo "----- Make sure that Supervisor starts on reboot"ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisorchmod +x /etc/init.d/supervisorupdate-rc.d supervisor defaults

[buildout]backend-dns-name = ec2-174-129-192-63.compute-1.amazonaws.comfrontend1-dns-name = ec2-174-129-116-33.compute-1.amazonaws.comfrontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.com

Must define the FQDN of the Zeo clients and

Zeo servers

When these names are resolved *on* the server, the DNS server returns the private IP, which means you avoid paying

bandwidth charges for outgoing/incoming traffic.

$ ./bin/aws start backend$ ./bin/aws start frontend1$ ./bin/aws start frontend2$ ./bin/aws start proxy

Start up the servers with one command

Fault tolerantEliminating single points of failure.

Plone Conference 2009 – Federico C. Guizzardi – 21 Aprile 2009

Distributed Architecture

Internet

Webserver

Cache Sys

Load Balancer

s1

s3

s4 s5

s2

10

02/21/08

The following diagram summarizes this information.

Heartbeat monitors Zeo server and moves to backup in case of failure

DRDBDistributed storage system similar in principle in RAID,but which works over the network.

Improvements to the deployment process

• time to launch a new site was days

• brittle non-repeatable installation process

• delay in procuring hardware and installing OS dependencies

Before After

Improvements to the deployment process

• time to launch a new site was days

• brittle non-repeatable installation process

• delay in procuring hardware and installing OS dependencies

• time to launch a new site was minutes

• streamlined and automated installation process

• virtual machines are launched instantly and OS dependencies are already installed on the disk image.

Before After

FabricScripts to deploy to remote hosts.

http://www.fabfile.org

Fabricset(fab_user='plone',

fab_hosts=['localhost'],

local='/home/plone/prod/',

remote='/opt/zope/prod/',

site='budapest.buildout')

def deploy():

local('cd $(local)$(site)')

local('tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc')

run('cd $(remote)$(site); ./bin/supervisorctl shutdown')

run('rm -rf $(remote)$(site)')

put('$(site).tar.gz', '$(remote)$(site).tar.gz')

run('cd $(root)$(site) && tar zxf $(site).tar.gz')

start()

def start():

run('cd $(remote)$(site)')

run('./bin/supervisord')

Deploy

$ fab deploy

• Provides a local packaging of the source into a tarball

• Removes the old remotely deployed code

• Puts the source on the remote host

• Restarts the site

What does this one command do?

Benefits of Fabric

• Chain local-command, remote-command, and remote-copy commands

• Substitute variables, and the splitting of distinct operations (like in the case of the 'restart' command) so you run them separately, with a fab restart, for example.

Example Fabric file

• See Alex Clark’s Fabric file:

• http://github.com/aclark4life/Debian-Deploy-Plone/blob/master/fabfile.py

Boto

• Python library to script AWS services

• Programmatically:

• Launch new EC2 instances

• Attach EBS volumes

• Create S3 buckets

http://code.google.com/p/boto/

collective.hostoutBuildout-based recipe for deploying to remote host

http://plone.org/products/collective.hostout

[buildout]extends = buildout.cfg

parts += hostout prod

auto-checkout += collective.hostout

[sources]

collective.hostout = svn https://svn.plone.org/svn/collective/collective.hostout/trunk

[hostout]recipe = collective.hostout

user = root

#password = blahidentity-file = /Users/nateaune/id_rsa-gsg-keypair

pre-commands =

# ${buildout:directory}/bin/supervisorctl shutdown || echo 'Unable to shutdown'# post-commands =

# ${buildout:directory}/bin/supervisord

#effective-user = zope

[prod]

recipe = collective.hostout

extends = hostouthost = ec2-75-101-211-135.compute-1.amazonaws.com

buildout = buildout.cfg

path = /opt/zope/prod

Run the deploy commands

$ ./bin/buildout -NInstalling prod.Generated script '/bin/hostout'.

$ bin/hostout deployInvalid hostout hostouts are: prod

$ bin/hostout deploy prod...

Chef & Puppet• Configuration management tools to prepare

machines to behave according to their role.

• Unlike Fabric which “pushes” config to the server, with Chef/Puppet, the server “pulls” it’s configuration from a master node.

• Usually overkill for a simple deployment, but with a multi-server deployment can be useful, especially to keep machines up-to-date.

http://opscode.com/chef/http://www.puppetlabs.com/

Questions?

• http://rhaptos.org

• http://pypi.python.org/pypi/mr.awsome

• http://fabfile.org

• http://code.google.com/p/boto/

• http://pypi.python.org/pypi/collective.hostout