Introduction to Amazon S3

Ashay Shirwadkar

AgendaWhat is Amazon S3?Storage ClassesNamespaceSecurityServer Side encryptionAccess ControlS3 APIs

But before that...

- Cloud computing, also on-demand computing, is a kind of Internet-based computing that provides shared processing resources.

- Resources being- Networks- Servers- Storage- Applications and services

What?

Why?- The term cloud is used as a metaphor for the Internet.

- So it means nothing. Just a nice word and now it's hot....

Types of storage

Client Server

Object

NFS/SMB/rsynciSCSI/AoE/Fiber ChannelREST APIs

What is amazon S3?Amazon S3 is acronym stands for Simple Storage Service .

S3 is web store, not a file system, it’s simple write once, read many (WORM) object store having eventual consistency.

“Write once” means that an object cannot be changed after it is written, and “read many” means that multiple copies of the object are made across different availability zones.

S3 is secure, durable & highly-scalable. It is accessed Via API’s (SOAP and REST)Server side encryptionData is stored with 99.999999999% durabilityStores data ranging from 1B to 5TB

A bedrock architectural component for many applicationsDropbox, Bitcasa, and Tahoe-LAFS-on-S3, among others, use S3 for online backup and synchronization

services.Tumblr, Spotify, and Pinterest host media on S3.

Consistancy

Durability

Load Balancers

Web Servers

Storage StorageIndexing Indexing

Web Servers

Load Balancers

RegionAvailability Zone Availability Zone

Cloud Storage Classes

StandardReduced

Redundancy Storage

Glacier

● Designed to provide high durability and high availability

● Designed to sustain concurrent loss of data in two availability zone.

● Objects you want to have high durability.

● E.g. Master copy of a movie media

● Designed to provide lesser redundancy with availability.

● Reduces cost by storing data at lower level of redundancy than in standard storage

● Objects you can afford to lose or can recreate.

● E.g. Different encoding of movie media.

● Suitable for archiving data, where data access is infrequent and retrieval time of several hours is acceptable.

● Uses very low cost amazon glacier service, but managed through s3.

● Objects you want to put in archive ( Rare Use).

● E.g. Digital archive of old movie media.

NamespacesThe S3 consists of Buckets and Objects. In a single bucket we can have multiple Objects.

Globally Uniquebucket name + object name (key) => Uniquely identify each object in a S3 cloud. Every object can be

addressed through bucket and key combination.

Buckets are similar to a directories

Object Name has to be unique within the bucketMax 1024 bytes UTF-8 Can have ‘path’ prefix

Namespaces

Amazon s3

johns-docshare userdocs src

drafts/rpt.doc style.css img/icon.ico swf/flash.swf user/foo.c user/bar.c

SecurityS3 provides regional service

Data never leaves region unless you move it

Server Side EncryptionAutomatic encryption of data at restStrong AES-256Enabled using simple PUT HeaderSelf managed i.e no need to manage key store

Server Side Encryption

ObjectBucket

Encrypted Data

Encryptedper-object key

per-object key

Key Management(monthly rotated)

Master Key

Encrypted Object

Access ControlS3 provides Policies, ACL’s and IAM (Identity and Access Management)

Use these to define rules for sharing objects or buckets

IAM Bucket Policies ACL’s● Fine Grained● Provide Role based

Access● Apply policies at role,

user and group level.

Allow

Actions:PutObject

Resource:arn:aws:s3:::mybucket/*

Bob John

AllowBob, John

Actions:PutObject

Resource:arn:aws:s3:::mybucket/*

My bucket

● Fine Grained● Apply Policies on bucket from

AWS console.● Incorporate user restrictions

without using IAM

● Coarse Grained● Apply access control at object

or bucket level.

AllowBob, John

Actions:Read

My bucket My Object

S3 APIAccessible through SOAP and REST API’s In S3, The operations can be divided into 3 categories- Operations on Service/s.

Get list of all buckets owned by the authenticated sender of the request.- Operations on Bucket/s.- Operations on Object/s.

User must have Access Key and Secret Access Key- Provide Temporary access to services - Keys can be generated through IAM.

String to sign- Every request has different string to sign.

S3 Authentication - Client side

GET /foo/bar.jpg HTTP/1.1Host: johnsmith.s3.amazonaws.comDate: Mon, 26 Mar 2007 19:37:58 +0000

Request

● Create Request

● Create HMAC-SHA1 Signature

GET\n\n\nMon, 26 Mar 2007 19:37:58 +0000\n/johnsmith/foo/bar.jpg

String to Sign StringToSign = HTTP-Verb + "\n" +Content-MD5 + "\n" +Content-Type + "\n" +Date + "\n" +Canonicalized Amz

Headers +CanonicalizedResource;

String to Sign Format

. . .String to Sign

Secret Access Key

HMAC calculation

and Base64

Encoding

Your Signature

● Send RequestGET /foo/bar.jpg HTTP/1.1Host: johnsmith.s3.amazonaws.comDate: Mon, 26 Mar 2007 19:37:58 +0000

Authentication: AWS Access Key:Signature

Request

S3 Authentication - Server side

● Retrieve Access Key

● Create HMAC-SHA1 Signature

. . .String to Sign

Secret Access Key

HMAC calculation

and Base64

Encoding

Calculated Signature

● Compare Two Signatures

GET /foo/bar.jpg HTTP/1.1Host: johnsmith.s3.amazonaws.comDate: Mon, 26 Mar 2007 19:37:58 +0000

Authentication: AWS Access Key:Signature

Request

Secret Access Key

Get Access key

Get SecretAccess key

Calculated Signature

Your Signature

Yes: Request is authenticatedNo: Request authentication fails

Operations on BucketsStandard Operations

Put Bucket - Creates bucket if does not exist.Get Bucket - List all the objects within the bucket.Delete Bucket - Deletes the bucket. All the objects within the buckets must be deleted.

Other operationsBucket lifecycle configuration - Set the lifecycle of objects within the bucketBucket policies - Set policies on bucketBucket location - Set the location of bucketBucket notification - Receive notifications when certain events happen in your bucketBucket logging - Enable logging for a bucketBucket request Payment - Returns the request payment configuration of a bucket.Bucket versioning - Enable versioning of objects within the bucket

Operations on ObjectsStandard Operations

Put Object - Creates Object.Post Object- POST is an alternate form of PUT that enables browser-based uploads.Get Object - Gets object along with its metadata.Head object - Gets only metadata.Delete Object - Deletes the Object.

Multipart UploadUpload a single object as a set of parts.Each part is a contiguous portion of the object's data.Upload for objects from 5 MB to 5 TB in size.

Other operationsObject ACL’s - set the ACL permissions for an object that already exists in a bucket. Object Copy - Creates a copy of an object

Multipart UploadInitiate Multipart Upload

Initiates a multipart upload and returns an upload ID.Provide this upload ID in each subsequent upload part requests.

POST /example-object?uploads HTTP/1.1Host: example-bucket.s3.amazonaws.comDate: Mon, 1 Nov 2010 20:34:56 GMTAuthorization: authorization string

RequestHTTP/1.1 200 OKx-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Mon, 1 Nov 2010 20:34:56 GMTContent-Length: 197Connection: keep-aliveServer: AmazonS3

<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Bucket>example-bucket</Bucket>

<Key>example-object</Key><UploadId>VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA</UploadId></InitiateMultipartUploadResult>

Responce

Multipart UploadUpload Part

Uploads a part in a multipart upload.

PUT /example-object?partNumber=1&uploadId=VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA HTTP/1.1Host: example-bucket.s3.amazonaws.comDate: Mon, 1 Nov 2010 20:34:56 GMTContent-Length: 10485760Content-MD5: pUNXr/BjKK5G2UKvaRRrOA==Authorization: authorization string

***part data omitted***

RequestHTTP/1.1 200 OKx-amz-id-2: Vvag1LuByRx9e6j5Onimru9pO4ZVKnJ2QRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Mon, 1 Nov 2010 20:34:56 GMTETag: "b54357faf0632cce46e942fa68356b38"Content-Length: 0Connection: keep-aliveServer: AmazonS3

Responce

Multipart UploadComplete Multipart Upload

Completes a multipart upload by assembling previously uploaded parts.

POST /example-object? uploadId=VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA HTTP/1.1Host: example-bucket.s3.amazonaws.comDate: Mon, 1 Nov 2010 20:34:56 GMTContent-Length: 391Authorization: authorization string

<CompleteMultipartUpload> <Part> <PartNumber>1</PartNumber> <ETag>"b54357faf0632cce46e942fa68356b38"</ETag> </Part> <Part> … </Part></CompleteMultipartUpload>

RequestHTTP/1.1 200 OKx-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==x-amz-request-id: 656c76696e6727732072657175657374Date: Mon, 1 Nov 2010 20:34:56 GMTConnection: closeServer: AmazonS3

<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

<Location>http://Example-Bucket.s3.amazonaws.com/Example-Object</Location> <Bucket>Example-Bucket</Bucket> <Key>Example-Object</Key> <ETag>"3858f62230ac3c915f300c664312c11f-9"</ETag></CompleteMultipartUploadResult>

Responce

Multipart UploadAbort Multipart Upload

Get Object

DELETE /example-object?uploadId=VXBsb2FkIElEIGZvciBlbHZpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZ HTTP/1.1Host: example-bucket.s3.amazonaws.comDate: Mon, 1 Nov 2010 20:34:56 GMTAuthorization: authorization string

Request

GET /example-object HTTP/1.1Host: example-bucket.s3.amazonaws.comDate: Mon, 1 Nov 2010 20:34:56 GMTAuthorization: authorization string

Request

API Operations-

API Operations contd.

DEMO

Create Bucket#!/bin/bash -xbucket=$1if [ -z "$1" ]then

echo "usage: ./bucket_put <bucket_name>"exit 1

firesource="/${bucket}/"dateValue=`date -R`stringToSign="PUT\n\n\n${dateValue}\n${resource}"s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X PUT \ -H "Host: ${bucket}.s3.amazonaws.com" \ -H "Date: ${dateValue}" \ -H "Authorization: AWS ${s3Key}:${signature}" \ http://${bucket}.s3.amazonaws.com/

< HTTP/1.1 200 OK< x-amz-id-2: jWt9BVmZkL1eU/i1gRoUrsB19/RHYwHGJZdst5ttGlLx7IvFPzHDSSNFluRyDRrCewG4xoFioJA=< x-amz-request-id: 9FC84EB054B018F9< Date: Wed, 12 Aug 2015 11:52:02 GMT< Location: /casoft< Content-Length: 0* Server AmazonS3 is not blacklisted< Server: AmazonS3<* Connection #0 to host calsoft.s3.amazonaws.com left intact

Request Responce

#!/bin/bash -xbucket=$1if [ -z "$1" ] then

echo "usage: ./bucket_list <bucket_name>"exit 1

firesource="/${bucket}/"dateValue=`date -R`s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X GET \ -H "Host: ${bucket}.s3.amazonaws.com" \ -H "Date: ${dateValue}" \ -H "Authorization: AWS ${s3Key}:${signature}" \ http://${bucket}.s3.amazonaws.com/

List Bucket< HTTP/1.1 200 OK< x-amz-id-2: nBbw0yclRZ4jPzPEECEKI0oCRMQrdIihEXbCVuUvYdXl75CGYH3/IcsPu/jxkCJb< x-amz-request-id: 90A0E3312B09453A< Date: Wed, 12 Aug 2015 11:55:48 GMT< x-amz-bucket-region: us-east-1< Content-Type: application/xml< Transfer-Encoding: chunked* Server AmazonS3 is not blacklisted< Server: AmazonS3<<?xml version="1.0" encoding="UTF-8"?>* Connection #0 to host calsoft.s3.amazonaws.com left intact<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>calsoft</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>

Request Responce

#!/bin/bash -xbucket=$1if [ -z "$1" ] then

echo "usage: ./bucket_delete <bucket_name>"exit 1

firesource="/${bucket}/"dateValue=`date -R`s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X DELETE \

-H "Host: ${bucket}.s3.amazonaws.com" \-H "Date: ${dateValue}" \-H "Authorization: AWS ${s3Key}:${signature}" \https://${bucket}.s3.amazonaws.com/

Delete Bucket< HTTP/1.1 204 No Content< x-amz-id-2: 4uuTflJqeUnewAYGmgghfiaBf/yfdja3DE7GmC9+e0QmBmE9T+2c/Ylt19jcndrM< x-amz-request-id: 96AEC8E1A534EC3E< Date: Wed, 12 Aug 2015 12:06:44 GMT* Server AmazonS3 is not blacklisted< Server: AmazonS3<* Connection #0 to host calsoft.s3.amazonaws.com left intact

Request Responce

#!/bin/bash -xbucket=$1file=$2if [[ (-z "$1") || (-z "$2") ]] then

echo "usage: ./object_upload <bucket_name> <object_name>"exit 1

firesource="/${bucket}/${file}"contentType="application/text"dateValue=`date -R`stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X PUT -T "${file}" \

-H "Host: ${bucket}.s3.amazonaws.com" \-H "Date: ${dateValue}" \-H "Content-Type: ${contentType}" \-H "Authorization: AWS ${s3Key}:${signature}" \https://${bucket}.s3.amazonaws.com/${file}

Create Object< HTTP/1.1 100 Continue* We are completely uploaded and fine< HTTP/1.1 200 OK< x-amz-id-2: OzR2U7CgsWwtHVbF8qcTiIpezFk5FVt9PxoFq9Px2QP8y7L0kOR2gQysfu9/EFNjUBdqIrzsE2o=< x-amz-request-id: 301904328CA5B6FF< Date: Wed, 12 Aug 2015 12:08:01 GMT< ETag: "78d5333e735ae15f5f19f2e76838b728"< Content-Length: 0* Server AmazonS3 is not blacklisted< Server: AmazonS3<* Connection #0 to host calsoft.s3.amazonaws.com left intact

Request Responce

#!/bin/bash -xbucket=$1file=$2if [[ (-z "$1") || (-z "$2") ]] then

echo "usage: ./object_download <bucket_name> <object_name>"

exit 1firesource="/${bucket}/${file}"dateValue=`date -R`stringToSign="GET\n\n\n${dateValue}\n${resource}"s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X GET \ -H "Host: ${bucket}.s3.amazonaws.com" \ -H "Date: ${dateValue}" \ -H "Authorization: AWS ${s3Key}:${signature}" \ https://${bucket}.s3.amazonaws.com/${file}

List Object< HTTP/1.1 200 OK< x-amz-id-2: 7yX1htWSKHVn+ssy32emeQoaF8WgFjRQuEio4PNzSyxjoJiPnmjcsmJvbJFZTTDx< x-amz-request-id: 1786B98B373C737E< Date: Wed, 12 Aug 2015 12:11:42 GMT< Last-Modified: Wed, 12 Aug 2015 12:08:01 GMT< ETag: "78d5333e735ae15f5f19f2e76838b728"< Accept-Ranges: bytes< Content-Type: application/text< Content-Length: 12* Server AmazonS3 is not blacklisted< Server: AmazonS3<Hello,World* Connection #0 to host calsoft.s3.amazonaws.com left intact

Request Responce

#!/bin/bash -xbucket=$1file=$2if [[ (-z "$1") || (-z "$2") ]] then

echo "usage: ./object_delete <bucket_name> <object_name>"

exit 1firesource="/${bucket}/${file}"dateValue=`date -R`stringToSign="DELETE\n\n\n${dateValue}\n${resource}"s3Key=’Your Access Key’s3Secret=’Your Secret Access Key’signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`curl -v -X DELETE \ -H "Host: ${bucket}.s3.amazonaws.com" \ -H "Date: ${dateValue}" \ -H "Authorization: AWS ${s3Key}:${signature}" \ https://${bucket}.s3.amazonaws.com/${file}

Delete Object< HTTP/1.1 204 No Content< x-amz-id-2: swMz88s6IV8i3dCwP6fSuklrubABX0O7XV1jBt7fUZtCP2x86IPozq+5Usy5wE7x< x-amz-request-id: 247E152DAB64A000< Date: Wed, 12 Aug 2015 12:14:58 GMT* Server AmazonS3 is not blacklisted< Server: AmazonS3<* Connection #0 to host calsoft.s3.amazonaws.com left intact

Request Responce

Q & A