Object storage systems: the underpinning of PRESENTATION ... · Object storage systems: the...

PRESENTATION TITLE GOES HEREObject storage systems: the underpinning of Cloud and Big Data initiatives

Mark O’Connell, EMC

Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

2


Abstract

Object storage systems: the underpinning of Cloud and Big Data initiatives

Object storage systems have risen to prominence in the storage industry and underlie both public and private cloud offerings. This talk will cover the needs of a cloud based storage system, why traditional approaches are insufficient to meet this challenge, the basic paradigms and architectures of object storage systems, how the architecture of an object storage system addresses the needs of cloud based storage, and challenges of using object storage systems vs traditional IT systems. Additionally this talk will compare and contrast the major object storage systems in use today, including Amazon’s S3, Microsoft’s Azure, EMC Atmos, the OpenStack Swift initiative, and others.

3


This talk will coverWhat is Cloud?

What are Object Storage systems?

How do Object Storage systems satisfy cloud use cases?

What is Big Data?

How do Big Data, Cloud, and Object Storage relate?

4


What is cloud?Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models.

http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf

5


5 Essential Cloud CharacteristicsOn-demand self-service.

Broad network access.

Resource pooling.

Rapid elasticity.

Measured service


6


What storage meets these criteria?On-demand self-service.

Broad network access.

Resource pooling.

Rapid elasticity.

Measured service


7


Object storage: Evolution of storage

8

Object storage


At first there was one – Block devicesCharacteristics

Disk, LUN, etcSimple, linear block addressesFixed size elements (512 byte block)Atomic access at the element level

AdvantagesSpeed and performanceAbility to satisfy many use cases

DisadvantagesDifficult to useFixed size LUNs

Caveats: may grow, thin LUNsFragmentation issues

If large LUNs are maintained for performance

9


Then there were two – FilesystemsCharacteristics

Built on blockSpace managementSharing – NFS, CIFSByte accessibleOS caching for performance

AdvantagesEasy to use, human understandable addressingStandard tools

DisadvantagesLimitations – path length, directory size, FS size, inodes, etc.Semantics heavyweight for some use casesStateful protocols, unsuitable for internet/mobile use

10


The Third Amigo – Object StorageCharacteristics

Advantages

Disadvantages

11


Defining CharacteristicsSingle, Flat namespace

Location independent addressing

“Constant time” read performance

Unlimited storage, autoconfiguring

Policies at object or bucket level (policies at user defined container level)

Per-object user definable metadata

12


Typical CharacteristicsScale out

Distributed

Eventually consistent

13


Common CharacteristicsREST/HTTP for internet/mobile access

Multi-tenancy

Self-service

Provisioning

Metering

14


Single, Flat NamespaceKey/Value store (Amazon S3, Azure, SWIFT)

May looks like a pathname, e.g. /foo/bar/filenameUnique within a bucketAbility to list based on common prefix and up to a delimiter

Otherwise no directory semanticsTypically used with a consistent hashing algorithm

Unique Identifier (Atmos, XAM)Opaque character stringMust typically be used with an application database to store the identifiersTypically globally unique

15


Location independent addressingLocation independent addressing

No relationship between a name and the location of the objectTwo objects with similar “names” are not necessarily colocated

Block systems: Reading N sequential blocks faster than reading N random blocks Flash based storage changes this effect

Filesystem: Accessing /foo/bar/file1 may speed up access to /foo/bar/file2 (caching effects)Filesystem: Reading from /foo/bar/file1 normally speeds up future reads of that same file

“Constant time” read performanceRead of each object completes in the same timeTypically achieved via a hash on object name

Hash internally translated to object location(s)

16


Unlimited storage and policiesUnlimited storage, autoconfiguring

Adding storage to the system automatically expands capacity available for all users“No limits” on total storage, total objects, etc

Policies at user defined container levelAmazon – bucket level policies, reduced redundancy at object levelAzure – storage account level policiesAtmos – per object policiesAmplidata – per namespace policy

17


Per-object user definable metadataWhat’s in a document beyond the content?

18


Per-object user definable metadataWhat’s in a document beyond the content?

19


Per-object user definable metadataApplications embed metadata in files in an application specific mechanism

What’s different in an object storage system?

Object systems provide standard ways to store metadataEnables easier processing of the metadataAllows different applications to manipulate or process the metadata

20


Object Storage: Typical Characteristics

Scale out

Distributed

21


Object Storage: Typical Characteristics

Eventually consistentWrite an object with all “1”sOverwrite the object with all “2”sRead from the object – should get “2”s (strong consistency)May get “1”s, will eventually get “2”sWill not get “5”s (a value that was never written)

Areas which generally show eventual consistencyQuorum readsNumber of object replicasAsynchronous replication with read allowedList after write eventually consistent

22


Common CharacteristicsREST/HTTP for internet/mobile access

Multi-tenancyMore than logical data segregationSeparation of users, administrators, etc

E.g. a per-filesystem administrator, a per-LUN administrator

Self-service

Provisioning

MeteringPay-as-you- go services, pay for pure consumption, not allocation

23


Object Storage Examples

Clouds powered by object storage

Underlying technologies

24


Summary: Storage EvolutionWith Block Storage:

Data is organized as an array of unrelated blocksHosts directly access blocks

With File/NAS Storage: Data is organized as an array of unrelated blocksOnboard file system places data on diskExternal systems directly access files within the onboard file system

With Object Storage: Application centric data storage, access, and management modelStores virtual containers that encapsulate data, data attributes, metadata, and Object IDs/keys

With Cloud Storage: Nearly limitless scalability designed to manage the explosion of unstructured data Capable of scaling across multiple physical locations, regardless of distanceAdvanced, policy driven data management based on attributes and metadataFlexible access methods that support traditional object storage models and new web based application architecturesSimple “set it and forget it” management with end user self service capabilities

25


Object storage satisfies cloud needsOn-demand self-service.

Unlimited storage, Multi-tenancy, Self provisioning

Broad network access.Scale out architecture, REST/HTTP

Resource pooling. Single flat namespace, unlimited storage, autoconfiguring, user defined policy scope, scale out, distributed

Rapid elasticity.Single flat namespace, autoconfiguring, location independent addressing, scale out

Measured service.Multi-tenancy, Metadata, Metering

26


What is Big Data?

27


Big Data DefinitionBig data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

Source: Gartner

28


Where does Big Data come from?Shopping habits

Web surfing

Social media and picture sharing sites

Car and appliance sensors

Emails

29


Leveraging Big DataMassive data collection

Collected across horizontal applications

What do online sites like Netflix, Pandora, Amazon, and others do?

Utilize data across customersMake predictions, recommendations

30


Big data needsBroad network access.

Scale out architecture, REST/HTTP

Resource pooling. Single flat namespace, unlimited storage, autoconfiguring, user defined policy scope, scale out, distributed

Rapid elasticity.Single flat namespace, autoconfiguring, location independent addressing, scale out

Commonality with cloudGood fit with object storage architectures

31


Object systems uniquely suit Big Data

Per-object metadata gives context to drive analyticsMedical records from patient XRecords from patients with asthma diagnosis

May add as agile response to real world situationsAsthmatic patients using experimental therapy Z

32


Summary

33


Object storage systemsObject storage systems build on block and file storage

Automate the management of these devices

Object storage systems are optimized for new usage patterns

Non-local access: Web, mobile, etcNon-stateful access: REST, HTTPScale out access across many independent streamsScalable access: Built for multiple independent use cases

Needs of cloud systems overlap with object systemsScale, Elasticity, Pooling of resource, Broad network access

Big data systems overlap with cloud and object

34


Attribution & Feedback

35

Please send any questions or comments regarding this SNIA Tutorial to [email protected]

The SNIA Education Committee thanks the following individuals for their contributions to this Tutorial.

Authorship History

Mark O’Connell March 2013

Additional ContributorsMichelle Scardino March 2013Joseph White March 2013Michael Fishman March 2013

Object storage systems: the underpinning of PRESENTATION ... · Object storage systems: the...

Documents

Transcript of Object storage systems: the underpinning of PRESENTATION ... · Object storage systems: the...