Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

Cloud Storage Simplified

Adrian Cole / Cloudsoft

Tuesday, November 1, 11

Adrian Cole (@jclouds)founded jclouds march 2009chief evangelist at Cloudsoft


Agenda

• intro• concepts• jclouds• code• awkward silence (or Q/A)


Blob Storage

4

Set<String> containers = namespacesInMyAccount;

Map<String, InputStream> keyValues = contentsOfContainer;


adrian@googlestorage

Love Letters

TronMovies

Goonies The Blob

ShrekThe One

putBlob

3d = trueurl = http://disney.go.com/tron

Blob Storage

5


6

Can I use BlobStores portably?


are they close enough?

7


who builds and runs your service?

8


9


tenancy

10


size and resumability

11


How many copies?

12


Consistency model?

13


Are you experienced? Is your product?

14


15


PUT /sushi.avi HTTP/1.1 Host: adriansmovies.s3.amazonaws.comContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: signature x-amz-meta-Chef: Kawasaki

PUT /<api version>/<account>/adriansmovies/sushi.avi HTTP/1.1 Host: storage.clouddrive.com Transfer-Encoding: chunkedX-Auth-Token: session-token X-Object-Meta-Chef: Kawasaki

PUT /adriansmovies/sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.netContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: SharedKey <app>:signaturex-ms-meta-Chef: Kawasaki

POST /namespace/adriansmovies/sushi.avi HTTP/1.1 Content-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTx-emc-uid: <uid> x-emc-signature: signature x-emc-meta: Chef=Kawasaki

CODE AND SIGN THE HTTP REQUEST

16


GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes= 734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1

POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_above Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...

PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1


17


POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...

PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8Content-Type: text/plainAuthorization: Basic GpjbG9=Kawasaki


18


19

GET https://api.gogrid.com/api/grid/server/add?v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&description=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1

POST http://serverManagementUrl/servers?format=json HTTP/1.1Accept: application/jsonContent-Type: application/json{"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}}

POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1Content-Type: application/x-www-form-urlencodedVersion=2010-06-15&Action=RunInstances&ImageId=ami-voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup.1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a

POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1

Accept: application/vnd.vmware.vcloud.vApp+xmlContent-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml

<InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd">

<VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" />

<InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8">

<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> </Item>









</VirtualHardwareSection> <NetworkConfigSection>

<NetworkConfig name="my-vapp"> <Features>

<FenceMode>allowInOut</FenceMode> </Features>

<NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig>

</NetworkConfigSection> </InstantiationParams>

</InstantiateVAppTemplateParams>

POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1Accept: Accept: application/jsonContent-Type: application/x-www-form-urlencodedlocation=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt-%2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3

POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1Accept: Accept: application/jsonContent-Type: application/json{"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}}

Deal with ErrorsDeal with ConcurrencyDeal with Complexity

do you want to


open source

feels like java (and clojure)

portability between clouds

deal with web complexity

unit testability

thread-safe and scalable

20


BlobStore LoadBalancer

Compute Table

Portable APIs

Embeddable

Provider-Specific Hooks

Over 30 Tested Providers!

21


jclouds locations

Location helps normalize placement across resource types All top-level resources have a locationLocation metadata is extensible

listAssignableLocations

IE

US

SG

22


jclouds modularity

APIs are software focused Providers are offering focusedAPI + location + defaults = Provider

jclouds-blobstore

s3

aws-s3

walrus

eucaluptus-partnercloud-s3

googlestorage

scality-ring

scaleup-storage

23


Alternatives to jclouds

Roll-your-own• RESTEasy, Jersey

Cloud-specific SDKs• typica, jets3t, Amazon SDK, vCloud SDK

Dasein Cloud API

24


BlobStore

25


java overview github jclouds/jclouds

// initcontext = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret);blobStore = context.getBlobStore();

// create containerblobStore.createContainerInLocation(null, “adriansmovies”);

// add blobblob = blobStore.blobBuilder("sushi.avi").payload(file).build();blobStore.putBlob(“adriansmovies”, blob);

26


clojure overview github jclouds/jclouds

(use 'org.jclouds.blobstore2)

(def *blobstore* (blobstore “azureblob” account key))

(create-container *blobstore* “movies”)(put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file))

27


Who’s coding BlobStore?

28


Filesystem github jclouds/jclouds

props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”);

blobStore = new BlobStoreContextFactory(). createContext("filesystem", props);

(blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”);

29


BlobRequestSigner github jclouds/jclouds

HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi");

(let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})])

30


What’s next?

• ACL support• OutputStream returns• Public/Private pathing

31


32

github jclouds-examples

@[email protected]

Questions?


https://github

https://github

mailto:[email protected]

mailto:[email protected]

Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

Technology

Transcript of Big Data & Cloud | Cloud Storage Simplified | Adrian Cole