CLOUD COMPUTING IN PHP WITH THE AMAZON WEB SERVICES

Jeff Barr, Senior Web Services Evangelistjbarr@amazon.com@jeffbarr on Twitter

• Based in Seattle• Career path:

– Startups– Microsoft– Consultant to VCs and startups– Amazon Web Services

• Lead AWS Blogger• Author of

“Host Your Website in the Cloud”

INTRODUCTION

• Amazon / Cloud / AWS Intro• AWS Overview

– Programmable Infrastructure– S3, EC2, SDB, SQS

• Programming AWS in PHP– AWS SDK for PHP– Scalable Image Processing– Visualizing Infrastructure– Dynamic Infrastructure

• Wrapup / Q & A

SESSION OUTLINE

AMAZON’S THREE BUSINESSES

Consumer (Retail)Business

Tens of millions of active customer accounts

Seven countries: US, UK, Germany, Japan, France, Canada, China

SellerBusiness

Sell on Amazon websites

Use Amazon technology for your own retail website

Leverage Amazon’s massive fulfillment center network

Developers &IT Professionals

On-demand infrastructure for hosting web-scale solutions

Hundreds of thousands of registered customers

WHAT IS CLOUD COMPUTING?

First, think of your electricity service…

Power is available to you on-demand, you pay only for what you use…

…and you plug into a vast electrical grid managed by professionals to get you the lowest cost, most reliable power with much greater efficiency and safety than you could probably do on your own.

INTRODUCING AMAZON WEB SERVICES

AWS provides flexible, scalable, secure, and cost-effective IT infrastructure for businesses of all sizes around the world.

Compute power and storage is available to you on-demand, you pay only for the resources you use…

…running on scalable, reliable, and secure infrastructure operated by Amazon Web Services, based on the knowledge gleaned from over a decade of building efficient and dependable infrastructure for Amazon.com.

AMAZON WEB SERVICES

ComputeAmazon Elastic Compute Cloud (EC2)

- Elastic Load Balancing- Auto Scaling

StorageAmazon Simple Storage Service (S3)

- AWS Import/Export

Application

Content DeliveryAmazon

CloudFront

MessagingAmazon Simple Queue

Service (SQS)Amazon Simple Notification

Service (SNS)

PaymentsAmazon Flexible Payments Service

(FPS)

On-Demand Workforce

Amazon Mechanical Turk

Parallel Processing

Amazon Elastic MapReduce

MonitoringAmazon CloudWatch

DatabaseAmazon RDS

Amazon SimpleDBThird-Party Offerings

ManagementAWS Management

Console

ToolsAWS Toolkit for Eclipse

Java, PHP, Ruby, Python, .Net Developer

Centers

Isolated NetworkVirtual Private Cloud

Metering and Billing

Identity and Access Management

• All functionality accessed by APIs• Amazon and third-party libraries• Command-line tools• AWS Management Console• Third-party Tools

PROGRAMMABLE INFRASTRUCTURE

CODE TO LAUNCH AN EC2 INSTANCE

// Run an instance$EC2 = new AmazonEC2();

$Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small");

$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

• Scalable data storage in-the-cloud• Highly available and durable• Pay-as-you-go pricing:

– Storage: tiered $0.15/GB to $0.055/GB– Data Transfer Out: tiered $0.15/GB to $0.080/GB– Data Transfer In: Free until June 30, 2010– Requests: nominal charges

• Big and busy:– 152 billion objects– 150K requests/second

AMAZON SIMPLE STORAGE SERVICE (S3)

Prices and facts are current as of October 2010.

• Amazon EC2: on-demand compute power– Obtain and boot new server instances in minutes– Quickly scale capacity up or down

• Key features:– Support for Linux, Windows, and OpenSolaris– Supports all major web and application platforms– Deploy across Availability Zones for reliability– Elastic IPs provide greater flexibility– Persistent storage with Amazon Elastic Block Store– Monitoring (CloudWatch), Load Balancing, Auto-Scaling

AMAZON ELASTIC COMPUTE CLOUD (EC2)

EC2 PROGRAMMING INTERFACE (API)

• Images:– RegisterImage– DescribeImages– DeregisterImage– ModifyImageAttribute– DescribeImageAttribute– ResetImageAttribute

• Instances:– RunInstances– DescribeInstances– TerminateInstances– StopInstances– GetConsoleOutput– RebootInstances

• IP Addresses:– AllocateAddress– ReleaseAddress– AssociateAddress– DisassociateAddress– DescribeAddresses

• Keypairs:– CreateKeyPair– DescribeKeyPairs– DeleteKeyPair

• Security Groups:– CreateSecurityGroup– DescribeSecurityGroups– DeleteSecurityGroup– AuthorizeSecurityGroupIngress– RevokeSecurityGroupIngress

• Block Storage Volumes:– CreateVolume– DeleteVolume– DescribeVolumes– AttachVolume– DetachVolume– CreateSnapshot– DescribeSnapshots– DeleteSnapshot

• VPC:– CreateCustomerGateway– DeleteCustomerGateway– DescribeCustomerGateways– AssociateDhcpOptions– CreateDhcpOptions– DeleteDhcpOptions– DescribeDhcpOptions– CreateSubnet– DeleteSubnet– DescribeSubnets– CreateVpc– DeleteVpc– DescribeVpcs– CreateVpnConnection– DeleteVpnConnection– DescribeVpnConnections– AttachVpnGateway– CreateVpnGateway– DeleteVpnGateway– DescribeVpnGateways– DetachVpnGateway

AMAZON EC2 INSTANCE SPECS

Standard High-CPU High-Memory Cluster

Micro Small Large Extra Large

Medium

Extra Large

Extra Large

Double Extra Large

Quadruple Extra Large

Cluster Compute

Quadruple Extra Large

Bits 32 32 64 64 32 64 64 64 64 64

RAM 613 MB 1.7 GB 7.5 GB

15 GB 1.7 GB 7 GB 17.1 GB

34.2 GB 68.4 GB 23

Disk 0 160 GB 850 GB

1690 GB

350 GB 1690 GB 420 GB

850 GB 1690 GB 1690 GB

EC2 Compute Units

Burst to 2

1 4 8 5 20 6.5 13 26 33.5

Cores 1 1 2 4 2 8 2 4 8 8(Dual

Processor)

Firewall Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Linux Per Hour

$0.02 $0.085 $0.34 $0.68 $0.17 $0.68 $0.50 $1.20 $2.40 $1.60

Windows Per Hour

$0.03 $0.12 $0.48 $0.96 $0.29 $1.16 $0.62 $1.44 $2.88 N/A

Prices and facts are current as of October 2010.

• Simple, scalable storage solution for structured data– Provides core database functionality for data storage and

querying– No schema, no data modeling, no DBA– SQL queries

AMAZON SIMPLEDB

item description color material

123 Sweater Blue, Red

789 Shoes Black Leather

Store:PUT (item, 123), (description, Sweater), (color, Blue), (color, Red)

Query:SELECT * FROM Inventory WHERE material='Leather'

• Reliable, highly scalable, hosted queue for messaging• Build automated workflows for all applications• Coordinate multiple Amazon EC2 instances

AMAZON SIMPLE QUEUE SERVICE

Queue

Producer

Producer

Producer

Consumer

Consumer

GETTING STARTED WITH AWS

CreateDeveloper Account

Enter PaymentInformation

Sign Up for Desired Services

Retrieve Private and Public Keys

Build & Deploy Application

Monitor and Scale Application

MOVING RIGHT ALONG!

"Enough with the talk, show us some

code, Jeff!"

PROGRAMMING AWS WITH PHP

We’ll use the free AWS SDK for PHP libraries from http://aws.amazon.com/sdkforphp

SDK BASICS

Download and unpack Add directory to PHP’s include_path Add AWS keys to config.inc.php

define('AWS_KEY', 'J35NTGFCQOIUY3OMNSQQ');

define('AWS_SECRET_KEY', '99pizu2vVOK11rk9UAgWVj7PBGzWwertqJlgLV0c');

Include one file:

require_once('sdk.class.php');

SDK XML / PHP SIMPLEXML

• All methods return a SimpleXML Object:

[body] => SimpleXMLElement Object ( [Name] => sitepoint-aws-cloud-book [Prefix] => SimpleXMLElement Object () [Marker] => SimpleXMLElement Object () [MaxKeys] => 1000 [IsTruncated] => false [Contents] => Array ( [0] => SimpleXMLElement Object ( [Key] =>

images/2008_shiller_housing_projection.jpg

• Access as $Res->body->Contents[0]->Key

EXAMPLE 1 – SCALABLE PROCESSING PIPELINE

BUILDING A SCALABLE IMAGE PROCESSING PIPELINE

• Fetch a web page, store in Amazon S3• Parse page and extract links to images• Fetch first 16 images on page, store in Amazon S3• Render images as composite image

URL Queue

Fetch & Store Page

Parse Queue

Parse Page

Image Queue

Fetch Images

S3

Render Queue

Render Images &

PagesS3S3

SQS-BASED APPLICATION ARCHITECTURE

ARCHITECTURE ATTRIBUTES

• Simple– Each stage is easy to understand– AWS reduces low-level coding

• Scalable– Add more processes – Add more instances– Add more stages– Storage any amount of data

• Fault Tolerant– Messages remain in queues until processed – Messages reappear in queues if process dies

• Asynchronous– Each stage runs at its own speed– Build, test,run one stage at a time

DEFINE QUEUES

// Queuesdefine('URL_QUEUE', 'c_url');define('PARSE_QUEUE', 'c_parse');define('IMAGE_QUEUE', 'c_image');define('RENDER_QUEUE', 'c_render');define('FEED_QUEUE', 'c_feed');

CREATE QUEUES

// Create the SQS access object$SQS = new AmazonSQS();

for ($i = 1; $i < count($argv); $i++){ $Queue = $argv[$i];

$Res = $SQS->create_queue($Queue);

if ($Res->isOK()) { print("Created queue '${Queue}'\n"); }}

CREATE QUEUES & CHECK STATUS

$ ./create_queues.php c_url c_parse c_image c_render c_feed

$ ./crawl_queue_status.phpc_url c_parse c_image c_render----- ------- ------- --------0 0 0 0

LOCATE A QUEUE'S URL

// Create the SQS access object$SQS = new AmazonSQS();

// Locate the queue$QueueURL = FindQueueURL($SQS, URL_QUEUE);

POST MESSAGE TO QUEUE

// Create message $HistItem = array('Posted by ' . $argv[0] . ' at ' .

date('c'));

$Message = json_encode(array('Action' => 'FetchPage', 'Origin' => $argv[0], 'Data' => $argv[$i], 'History' => $HistItem)); // Post message $Res = $SQS->send_message($QueueURL, $Message);

if ($Res->isOK()) { print("Posted '${Message}' to QueueURL '${QueueURL}'\n"); }

RECEIVE MESSAGE FROM QUEUE

function PullMessage($SQS, $QueueURL){ while (true) { $Res = $SQS->receive_message($QueueURL); if ($Res->isOk()) { if (IsSet($Res->body->ReceiveMessageResult->Message)) { } else { sleep(1); } }}

RETURN RECEIVED MESSAGE

$Message = $Res->body->ReceiveMessageResult->Message; $MessageBody = $Message->Body; $MessageDetail = json_decode($MessageBody, true); $ReceiptHandle = $Message->ReceiptHandle;

return array('QueueURL' => $QueueURL, 'Timestamp' => date('c'), 'Message' => $Message, 'MessageBody' => $MessageBody, 'MessageDetail' => $MessageDetail, 'ReceiptHandle' => $ReceiptHandle);

SAMPLE PROCESSING STAGE

// Pull, process, postwhile (true){ // Pull the message from the queue $Message = PullMessage($SQS, $QueueURL);

if ($Message != null) { // Extract message detail $MessageDetail = $Message['MessageDetail']; $ReceiptHandle = $Message['ReceiptHandle']; $PageURL = $MessageDetail['Data']; … }}

DETAILED PROCESSING

// Fetch the pageprint("Processing URL '${PageURL}':\n");$HTML = file_get_contents($PageURL);print(" Retrieved " . strlen($HTML) . " bytes of HTML\n");

// Store the page in S3 $Key = 'page_' . md5($PageURL) . '.html'; if (UploadObject($S3, BOOK_BUCKET, $Key, $HTML,

S3_ACL_PUBLIC)) { // Get URL in S3 $S3URL = $S3->get_object_url(BOOK_BUCKET, $Key); print(" Uploaded page to S3 as '${Key}'\n");

DETAILED PROCESSING

// Form message to pass page along to parser $Origin = $MessageDetail['Origin']; $History = $MessageDetail['History']; $History[] = 'Fetched by ' . $argv[0] . ' at ' .

date('c');

$Message = json_encode(array('Action' => 'ParsePage', 'Origin' => $Origin, 'Data' => $S3URL, 'PageURL' => $PageURL, 'History' => $History));

// Pass the page along to the parser $Res = $SQS->send_message($QueueParse, $Message);

RENDER IMAGES

foreach ($ImageKeys as $ImageKey) { // Fetch the image print(" Fetch image '${ImageKey}'\n"); $Image = $S3->get_object(BOOK_BUCKET, $ImageKey);

// Convert it to GD format $ImageBits = ImageCreateFromString($Image->body);

// Copy it to proper spot in the destination print(" Render image at ${NextX}, ${NextY}\n"); ImageCopy($ImageOut, $ImageBits, $NextX, $NextY, 0, 0, ImageSx($ImageBits), ImageSy($ImageBits));

// Update position for next image $NextX += THUMB_SIZE + GAP_SIZE; if (($NextX + THUMB_SIZE) > $OutX) { $NextX = BORDER_LEFT; $NextY += THUMB_SIZE + GAP_SIZE; } }

EXAMPLE 2 – CREATE INFRASTRUCTURE GRAPH

PROCESSING FLOW

Get AWS Metadata

Build Object Model

Render Object Model as DOT

Graph

Render Dot Graph as PDF

GET AWS METADATA

// Create the service access objects$Service_EC2 = new AmazonEC2);$Service_S3 = new AmazonS3);$Service_SDB = new AmazonSDB);$Service_CF = new AmazonCloudFrontetKey);

// Fetch information about all of the EC2 objects$ResElasticIP = $Service_EC2->describe_addresses();$ResAvailabilityZones = $Service_EC2-

>describe_availability_zones();$ResInstances = $Service_EC2->describe_instances();$ResVolumes = $Service_EC2->describe_volumes();$ResSnapshots = $Service_EC2->describe_snapshots();

BUILD OBJECT MODEL

// EC2 Instancesforeach ($ResInstances->body->reservationSet->item as $ItemSet){ foreach ($ItemSet->instancesSet->item as $Item) { $InstanceId = (string) $Item->instanceId; $ImageId = (string) $Item->imageId; $State = (string) $Item->instanceState->name; $InstanceType = (string) $Item->instanceType; $AvailabilityZone = (string) $Item->placement->availabilityZone; $LaunchTime = (string) $Item->launchTime;

$Region->AddInstance(new Instance($AvailabilityZone, $InstanceId, $ImageId, $State, $InstanceType, $LaunchTime));

}}

SAMPLE OBJECT (EC2 INSTANCE)

class Instance{ var $State; var $ImageId; var $InstanceId; var $LaunchTime; var $InstanceType; var $AvailabilityZone;

function __construct(…) {} public function GetAvailabilityZone() {} private function GetLabel() {} function Render() {} function RenderEdges() {}}

RENDER OBJECT MODEL AS DOT GRAPH

function WriteNode($FP, $IndentLevel, $Node, $Label, $NodeType)

{ fwrite($FP, Indent($IndentLevel) . Quote($Node) . ' [' . 'label=' . Quote($Label) . ', ' . GetNodeStyle($NodeType) . "];\n");}

DOT TEXT

graph aws{ // Region subgraph "cluster_us-east-1" { label="Region us-east-1"; style=filled; fillcolor=yellow; color=black;

// Elastic IP Addresses subgraph "cluster_us-east-1_ips" { label="Elastic IP Addresses"; style=filled; fillcolor=darksalmon; color=black;

"75.101.154.199" [label="Elastic IP 75.101.154.199\nec2-75-101-154-199.compute-1.amazonaws.com", color=black, style=filled, fillcolor=wheat, shape=rect];

"us-east-1_phantom_ips" [label="", shape=point, style=invis]; }

RENDER DOT TEXT TO PDF

$ dot –Tpdf aws_meta.dot > ~jeff/public_html/aws_meta.pdf

FINISHED INFRASTRUCTURE GRAPH

EXAMPLE 3 – DYNAMICALLY INSTANTIATE INFRASTRUCTURE

EC2 Instance

10 GB EBS

Volume

100 GB EBS

Volume

Elastic IP

Address

RUN AN EC2 INSTANCE

// Create the EC2 access object$EC2 = new AmazonEC2(AWS_PUBLIC_KEY, AWS_SECRET_KEY);

// Run an instance$Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small");

$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

GET INSTANCE INFO

// Get the Id and Availability Zone of the instance$Instances = $Res->body->instancesSet;$InstanceId = $Instances->item->instanceId;$AvailabilityZone = $Instances->item->placement

->availabilityZone;

print("Launched instance ${InstanceId} " . "in availability zone ${AvailabilityZone}.\n");

ALLOCATE ELASTIC IP ADDRESS

// Allocate an Elastic IP address$Res = $EC2->allocate_address();if (!$Res->isOK()){ exit("Could not allocate public IP address.\n");}

// Get the allocated Elastic IP address$PublicIP = $Res->body->publicIp;print("Assigned IP address ${PublicIP}.\n");

ATTACH IP ADDRESS TO INSTANCE

// Associate the Elastic IP address with the instance$Res = $EC2->associate_address($InstanceId, $PublicIP);

if (!$Res->IsOK()){ exit("Could not associate IP address ${PublicIP} " . "with instance ${InstanceId}.\n");}

print("Associated IP address ${PublicIP} " . "with instance ${InstanceId}.\n");

CREATE 2 EBS VIRTUAL DISK VOLUMES

// Create two EBS volumes in the instance's availability zone

$Res1 = $EC2->create_volume(10, $AvailabilityZone);$Res2 = $EC2->create_volume(100, $AvailabilityZone);

if (!$Res1->isOK() || !$Res2->isOK()){ exit("Could not create EBS volumes.\n");}

// Get the volume Ids$VolumeId1 = $Res1->body->volumeId;$VolumeId2 = $Res2->body->volumeId;

ATTACH THE VOLUMES TO THE INSTANCE

$Res1 = $EC2->attach_volume($VolumeId1, $InstanceId, '/dev/sdf');

$Res2 = $EC2->attach_volume($VolumeId2, $InstanceId, '/dev/sdg');

if (!$Res1->isOK() || !$Res2->isOK()){ exit("Could not attach EBS volumes " . "${VolumeId1} and ${VolumeId2} " . "to instance ${InstanceId}.\n");}

Q & A

• Amazon Web Services: http://aws.amazon.com

• AWS Blog: http://aws.typepad.com

• jbarr@amazon.com

• twitter.com/jeffbarr

FOR MORE INFORMATION

Thanks!

jbarr@amazozn.com