Post on 11-Feb-2022
PRESENTATION TITLE GOES HEREObject storage systems: the underpinning of Cloud and Big Data initiatives
Mark O’Connell, EMC
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
2
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Abstract
Object storage systems: the underpinning of Cloud and Big Data initiatives
Object storage systems have risen to prominence in the storage industry and underlie both public and private cloud offerings. This talk will cover the needs of a cloud based storage system, why traditional approaches are insufficient to meet this challenge, the basic paradigms and architectures of object storage systems, how the architecture of an object storage system addresses the needs of cloud based storage, and challenges of using object storage systems vs traditional IT systems. Additionally this talk will compare and contrast the major object storage systems in use today, including Amazon’s S3, Microsoft’s Azure, EMC Atmos, the OpenStack Swift initiative, and others.
3
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
This talk will coverWhat is Cloud?
What are Object Storage systems?
How do Object Storage systems satisfy cloud use cases?
What is Big Data?
How do Big Data, Cloud, and Object Storage relate?
4
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
What is cloud?Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models.
http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
5
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
5 Essential Cloud CharacteristicsOn-demand self-service.
Broad network access.
Resource pooling.
Rapid elasticity.
Measured service
http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
6
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
What storage meets these criteria?On-demand self-service.
Broad network access.
Resource pooling.
Rapid elasticity.
Measured service
http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
7
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object storage: Evolution of storage
8
Object storage
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
At first there was one – Block devicesCharacteristics
Disk, LUN, etcSimple, linear block addressesFixed size elements (512 byte block)Atomic access at the element level
AdvantagesSpeed and performanceAbility to satisfy many use cases
DisadvantagesDifficult to useFixed size LUNs
Caveats: may grow, thin LUNsFragmentation issues
If large LUNs are maintained for performance
9
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Then there were two – FilesystemsCharacteristics
Built on blockSpace managementSharing – NFS, CIFSByte accessibleOS caching for performance
AdvantagesEasy to use, human understandable addressingStandard tools
DisadvantagesLimitations – path length, directory size, FS size, inodes, etc.Semantics heavyweight for some use casesStateful protocols, unsuitable for internet/mobile use
10
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
The Third Amigo – Object StorageCharacteristics
Advantages
Disadvantages
11
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Defining CharacteristicsSingle, Flat namespace
Location independent addressing
“Constant time” read performance
Unlimited storage, autoconfiguring
Policies at object or bucket level (policies at user defined container level)
Per-object user definable metadata
12
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Typical CharacteristicsScale out
Distributed
Eventually consistent
13
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Common CharacteristicsREST/HTTP for internet/mobile access
Multi-tenancy
Self-service
Provisioning
Metering
14
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Single, Flat NamespaceKey/Value store (Amazon S3, Azure, SWIFT)
May looks like a pathname, e.g. /foo/bar/filenameUnique within a bucketAbility to list based on common prefix and up to a delimiter
Otherwise no directory semanticsTypically used with a consistent hashing algorithm
Unique Identifier (Atmos, XAM)Opaque character stringMust typically be used with an application database to store the identifiersTypically globally unique
15
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Location independent addressingLocation independent addressing
No relationship between a name and the location of the objectTwo objects with similar “names” are not necessarily colocated
Block systems: Reading N sequential blocks faster than reading N random blocks Flash based storage changes this effect
Filesystem: Accessing /foo/bar/file1 may speed up access to /foo/bar/file2 (caching effects)Filesystem: Reading from /foo/bar/file1 normally speeds up future reads of that same file
“Constant time” read performanceRead of each object completes in the same timeTypically achieved via a hash on object name
Hash internally translated to object location(s)
16
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Unlimited storage and policiesUnlimited storage, autoconfiguring
Adding storage to the system automatically expands capacity available for all users“No limits” on total storage, total objects, etc
Policies at user defined container levelAmazon – bucket level policies, reduced redundancy at object levelAzure – storage account level policiesAtmos – per object policiesAmplidata – per namespace policy
17
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Per-object user definable metadataWhat’s in a document beyond the content?
18
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Per-object user definable metadataWhat’s in a document beyond the content?
19
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Per-object user definable metadataApplications embed metadata in files in an application specific mechanism
What’s different in an object storage system?
Object systems provide standard ways to store metadataEnables easier processing of the metadataAllows different applications to manipulate or process the metadata
20
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object Storage: Typical Characteristics
Scale out
Distributed
21
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object Storage: Typical Characteristics
Eventually consistentWrite an object with all “1”sOverwrite the object with all “2”sRead from the object – should get “2”s (strong consistency)May get “1”s, will eventually get “2”sWill not get “5”s (a value that was never written)
Areas which generally show eventual consistencyQuorum readsNumber of object replicasAsynchronous replication with read allowedList after write eventually consistent
22
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Common CharacteristicsREST/HTTP for internet/mobile access
Multi-tenancyMore than logical data segregationSeparation of users, administrators, etc
E.g. a per-filesystem administrator, a per-LUN administrator
Self-service
Provisioning
MeteringPay-as-you- go services, pay for pure consumption, not allocation
23
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object Storage Examples
Clouds powered by object storage
Underlying technologies
24
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Summary: Storage EvolutionWith Block Storage:
Data is organized as an array of unrelated blocksHosts directly access blocks
With File/NAS Storage: Data is organized as an array of unrelated blocksOnboard file system places data on diskExternal systems directly access files within the onboard file system
With Object Storage: Application centric data storage, access, and management modelStores virtual containers that encapsulate data, data attributes, metadata, and Object IDs/keys
With Cloud Storage: Nearly limitless scalability designed to manage the explosion of unstructured data Capable of scaling across multiple physical locations, regardless of distanceAdvanced, policy driven data management based on attributes and metadataFlexible access methods that support traditional object storage models and new web based application architecturesSimple “set it and forget it” management with end user self service capabilities
25
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object storage satisfies cloud needsOn-demand self-service.
Unlimited storage, Multi-tenancy, Self provisioning
Broad network access.Scale out architecture, REST/HTTP
Resource pooling. Single flat namespace, unlimited storage, autoconfiguring, user defined policy scope, scale out, distributed
Rapid elasticity.Single flat namespace, autoconfiguring, location independent addressing, scale out
Measured service.Multi-tenancy, Metadata, Metering
26
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
What is Big Data?
27
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Big Data DefinitionBig data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
Source: Gartner
28
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Where does Big Data come from?Shopping habits
Web surfing
Social media and picture sharing sites
Car and appliance sensors
Emails
29
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Leveraging Big DataMassive data collection
Collected across horizontal applications
What do online sites like Netflix, Pandora, Amazon, and others do?
Utilize data across customersMake predictions, recommendations
30
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Big data needsBroad network access.
Scale out architecture, REST/HTTP
Resource pooling. Single flat namespace, unlimited storage, autoconfiguring, user defined policy scope, scale out, distributed
Rapid elasticity.Single flat namespace, autoconfiguring, location independent addressing, scale out
Commonality with cloudGood fit with object storage architectures
31
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object systems uniquely suit Big Data
Per-object metadata gives context to drive analyticsMedical records from patient XRecords from patients with asthma diagnosis
May add as agile response to real world situationsAsthmatic patients using experimental therapy Z
32
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Summary
33
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Object storage systemsObject storage systems build on block and file storage
Automate the management of these devices
Object storage systems are optimized for new usage patterns
Non-local access: Web, mobile, etcNon-stateful access: REST, HTTPScale out access across many independent streamsScalable access: Built for multiple independent use cases
Needs of cloud systems overlap with object systemsScale, Elasticity, Pooling of resource, Broad network access
Big data systems overlap with cloud and object
34
Object storage systems: the underpinning of Cloud and Big Data initiatives© 2013 Storage Networking Industry Association. All Rights Reserved.
Attribution & Feedback
35
Please send any questions or comments regarding this SNIA Tutorial to tracktutorials@snia.org
The SNIA Education Committee thanks the following individuals for their contributions to this Tutorial.
Authorship History
Mark O’Connell March 2013
Additional ContributorsMichelle Scardino March 2013Joseph White March 2013Michael Fishman March 2013