CloudStack S3 configurationTech Preview

Sebastien GoasguenAugust 23rd

Introduction

• CloudStack provides an S3 compatible interface• In Apache CloudStack 4.0 (soon out), Cloudbridge

is now an integral part of the management server and not a separate server.

• This is not saying that CloudStack provides an S3 implementation. CloudStack supports object stores (e.g Swift, GlusterFS…) but is not itself an object store.

Steps to use S3 in CloudStack

• Specify the mount point where you want to store the objects

• Enable the service via global configuration settings

• Generate API keys for the user(s)• Register the user and associate a certificate • Use boto or other S3 clients

S3 mount point• S3 properties are set in /path/to/source/awsapi/conf/cloud-

bridge.properties or on the mgt server at $CATALINA_HOME/conf/cloud-bridge-properties

host=http://localhost:8080/awsapi

storage.root=/Users/john1/S3-Mount

storage.multipartDir=__multipart__uploads__

bucket.dns=false

serviceEndpoint=localhost:8080

Edit the storage.root to point to a file system mount point on the management server.

Enabling S3

• Via the GUI

• Via API call on integration API port 8096http://localhost:8096/client/api?command=updateConfiguration&name=enable.s3.api&value=true

Enabling S3

• Via an authenticated API call on port 8080 (e.g using a Python client)

apiurl = 'http://localhost:8080/client/api’

cloudstack = CloudStack.Client(apiurl,apikey,secretkey)

cloudstack.updateConfiguration \({‘name’:’enable.s3.api’,’value’:’true’})

Generate Keys

• Via the GUI

Generate Keys

• Via the API:http://localhost:8096/client/api?command=registerUserKeys&id=<id of the user>

Register the user

• Get the script from the source at /path/to/source/awsapi-setup/setup/cloudstack-aws-api-register

cloud-bridge-register --apikey=<User’s Cloudstack API key> --secretkey=<User’s CloudStack Secret key> --cert=</path/to/cert.pem> --url=http://<cloudstack-server-ip>:8080/awsapi

S3 Boto example 1/4• Import the boto S3 modules:

>>> from boto.s3.key import Key

>>> from boto.s3.connection import S3Connection

>>> from boto.s3.connection import OrdinaryCallingFormat

• Set your API keys, calling format and create the connection to the S3 endpoint:

>>> apikey='ChOw-pwdcCFy6fpeyv6kUaR0NnhzmG3tE7HLN2z3OB_s-ogF5HjZtN4rnzKnq2UjtnHeg_RjeDgdDAPyLA5gOw’>>>secretkey='IMY8R7CJQiSGFk4cHwfXXN3DUFXz07cCiU80eM3MCmfLs7kusgyOfm0g9qzXRXhoAPCOllGt637cWH-IRxXc3w’

>>> cf=OrdinaryCallingFormat()

>>> conn=S3Connection(aws_access_key_id=apikey,aws_secret_access_key=secretkey,is_secure=False,host='localhost',port=8080,calling_format=cf,path='/awsapi/rest/AmazonS3')

S3 boto example 2/4

• Note the path of the connection: /awsapi/rest/AmazonS3 , this is not consistent with the EC2 endpoint and will probably be fixed soon, it is also not consistent with the information in the configuration file. That’s why it’s a Tech Preview.

• Help welcome !!!

S3 Boto example 3/4

• Once you have the connection, start by creating a bucket, get a key and store a value for that key in the bucket.

>>> conn.create_bucket('test')

<Bucket: test>

>>> b=conn.get_bucket('test')

>>> k=Key(b)

>>> k.set_contents_from_string('This is a test')

>>> k.get_contents_as_string()

'This is a test'

S3 boto example 4/4

• Same thing with a file:

>>> conn.create_bucket('cloud')

<Bucket: cloud>

>>> b=conn.get_bucket('cloud')

>>> k=Key(b)

>>> k.set_contents_from_filename('/Users/runseb/Desktop/code/s3cs.py')

>>> k.get_contents_to_filename('/Users/runseb/Desktop/code/foobar’)

>>> conn.get_all_buckets()

[<Bucket: test>, <Bucket: cloud>]

Example of S3 Database tables• The cloudbridge database on the mgt server contains

information about the users registered• mysql> select * from usercredentials;• | ID | AccessKey | SecretKey | CertUniqueId |

| 1 | ChOw-pwdcCFy6fpeyv6kUaR0NnhzmG3tE7HLN2z3OB_s-ogF5HjZtN4rnzKnq2UjtnHeg_RjeDgdDAPyLA5gOw | IMY8R7CJQiSGFk4cHwfXXN3DUFXz07cCiU80eM3MCmfLs7kusgyOfm0g9qzXRXhoAPCOllGt637cWH-IRxXc3w | CN=AWS Limited-Assurance CA, OU=AWS, O=Amazon.com, C=US, serial=570614354026 |

• As well as the buckets (snippet cut)• mysql> select * from sbucket;• | ID | Name | OwnerCanonicalID | SHostID | CreateTime

| 1 | test | ChOw-pwdcCFy6fpeyv6kUaR0NnhzmG3tE7HLN2z23:42:21 | |

• | 2 | cloud | ChOw-pwdcCFy6fpeyv6kUaR0NnhzmG3tE7HLN2z3OB_s-ogF5HjZtN4rnzKnq2UjtnHeg_RjeDgdDAPyLA5gOw | 2 | 2012-08-23 23:42:29 | 0 |

Mount Point

• The mount point now contains a flat directory structure with two buckets, and in each bucket a file containing the value for that key

root@devcloud:/tmp/s3mount# ls -l

total 8

drwxr-xr-x 2 root root 4096 Aug 23 16:45 cloud

drwxr-xr-x 2 root root 4096 Aug 23 16:47 test

root@devcloud:/tmp/s3mount# cat test/2

This is a test

Conclusions

• This was all tested with DevCloud• Join the discussion on the future of the EC2/S3

compatibility of CloudStack

cloudstack-dev@incubator.apache.org#cloudstack on irc.freenode.net@CloudStack on Twitter