ECS/Cloud Object Storage - DevOps Day
-
Upload
bob-sokol -
Category
Technology
-
view
127 -
download
1
Transcript of ECS/Cloud Object Storage - DevOps Day
![Page 1: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/1.jpg)
1© Copyright 2015 EMC Corporation. All rights reserved.
OBJECT STORAGETECHNICAL DISCUSSION
![Page 2: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/2.jpg)
2© Copyright 2015 EMC Corporation. All rights reserved.
DATA GROWTH IS BREAKING TRADITIONAL STORAGESCALE-OUT FILE & OBJECT STORAGE CONTINUES TO GROW
• Overly complex– Multiple data protection schemes,
protocols, management tools
• Can’t economically scale– Inefficient, high overhead,
especially at geo-scale
• Not cloud-ready– Not architecturally suited and no
self-service
Source: IDC EMC Digital Universe Study 2014
![Page 3: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/3.jpg)
3© Copyright 2015 EMC Corporation. All rights reserved.
BLOCK FILE FILE BLOCK FILE
Today’s Storage Infrastructure
BLOCK
![Page 4: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/4.jpg)
4© Copyright 2015 EMC Corporation. All rights reserved.
WHY OBJECT?
![Page 5: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/5.jpg)
5© Copyright 2015 EMC Corporation. All rights reserved.
Object Storage Characteristics
Linear ScalabilityScales to billions of objects
No LockingNo lock on write or create operations
Geo-scaleGeo-replicated and distributed
Support for large filesObject sizes are in TBs
Web friendly Firewall friendly, http, REST accessibility
Metadata and extensibilityObjects can be extended to multiple policies (Immutability, retention, etc…)
![Page 6: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/6.jpg)
6© Copyright 2015 EMC Corporation. All rights reserved.
An Object platform offers … A flat namespace of millions of buckets
Buckets that scale to billions of objects
Geo distribution, protection, access
User meta data as a first-class entity
Snapshot consistency semantics
Multi-tenant access & metering
Multiple data access methods including via REST/HTTP (S3, Hadoop, CAS, Atmos & Swift)
OBJECT
OBJECT
OBJECT
OBJECT
OBJECT
OBJECT
![Page 7: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/7.jpg)
7© Copyright 2015 EMC Corporation. All rights reserved.
File and Object Storage ComparisonFile Object
Writing to a file requires exclusive lock Object supports multiple writes, no locking
Limit on number of files in a directory. Objects are limitless in size, 1 MB to TBs, Objects scale across multiple files
File meta-data is fixed by file system, no user meta-data
Objects support extensible meta-data
Large files hard to seek Objects can be viewed with no limitation
File create operations require directory to be in exclusivity lock
No locking required to create files
CIFS/NFS access not Web or firewall-friendly – relies on file/folder access control and session-based authentication
Easy, fine-grained authentication and access control (per object), HTTP, REST-based access
![Page 8: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/8.jpg)
8© Copyright 2015 EMC Corporation. All rights reserved.
Object Use Cases are Expanding
• Existing Use Cases: – Scalable content store for cloud-based
applications/services– Scalable content storage for vertical
applications– Tape rationalization/elimination
• Emerging Use Cases:– Storage for Big Data/Hadoop – NAS replacement/augmentation– Public IaaS alternative
• Migrate to alternative providers or “in-sourcing”
Applications, analytics and data growth drive Object
Source: 451 Research
![Page 9: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/9.jpg)
9© Copyright 2015 EMC Corporation. All rights reserved.
MODERN APPS ARE BREAKING TRADITIONAL STORAGENOT DESIGNED FOR CLOUD AND BIG DATA APPLICATIONS
• Architecture is too complex– Locking, replication, High Availability, geo-
distribution is complex
• Not Web or firewall friendly– Distributed (WAN) access is complex
• Storage silos impede development– Different hardware for every data type and
access protocol
![Page 10: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/10.jpg)
10© Copyright 2015 EMC Corporation. All rights reserved.
USE CASES
![Page 11: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/11.jpg)
11© Copyright 2015 EMC Corporation. All rights reserved.
GLOBAL CONTENT REPOSITORYON-PREMISE UNSTRUCTURED STORAGE PLATFORM
PROBLEM• Can’t cost-effectively manage or scale storage to
support explosive growth in unstructured content.• Traditional storage not suited for new Web, mobile
and cloud applications.• Difficult and costly to manage data lifecycle and
retention policies across archive silos and sites
VALUE• Reduce complexity and cost–one globally
accessible, geo-efficient archive that serves multiple applications and content types at lower cost than public cloud.
• Anywhere data access – All data globally accessible by Web, mobile and cloud apps.
• Enterprise-grade data protection – Efficient geo-protection and policy-based retention for basic compliance and governance.
https://accesspoint.yourcompany.com
U.K.L.A.
Memphis
Applications Tiering, Archiving, Backup
![Page 12: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/12.jpg)
12© Copyright 2015 EMC Corporation. All rights reserved.
MODERN APPLICATION PLATFORMEFFICIENT GEO-CAPABLE STORAGE & ANYWHERE ACCESS
https://accesspoint.yourcompany.com
U.K.L.A.
Memphis
PROBLEM• Traditional storage architecture not optimized for
multi-site, mobile access to content• Writing to multiple file systems and proprietary
APIs complicates development• Can’t access or process large data sets
VALUE• Anywhere access - Provides anywhere access to
geo-replicated content• Simpler, faster development - Supports
multiple industry standard APIs/protocols and anywhere access with strong consistency
• Unmatched access and efficiency - Geo-protection, active-active architecture optimizes both access and storage efficiency for Big Data – large and small files
![Page 13: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/13.jpg)
13© Copyright 2015 EMC Corporation. All rights reserved.
GEO-SCALE BIG DATA ANALYTICSEFFICIENT GEO-SCALE STORAGE & GLOBAL BIG DATA ANALYTICS
https://accesspoint.yourcompany.com
U.K.L.A.
New York
ANALYTICSPROBLEM• Large (and growing) data volumes lead to
exponential storage costs• Traditional Hadoop replication leads to
unmanageable DC footprint with data growth• Always have to move data to the analytics cluster
VALUE• Cost Efficient Storage• HDFS Archive – Bring state of the art patented
technology to provide highly dense storage for Hadoop
• Global Analytics –Bring analytics to geo-distributed data and archives
![Page 14: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/14.jpg)
14© Copyright 2015 EMC Corporation. All rights reserved.
PROBLEM• Unstructured data growth - Reclaim costly Tier 1 storage • Current solutions aren’t scalable or cost efficient• Instant access to cold-stored data is required • “No Public Cloud” policy - Data needs to be on-premises
VALUECosts less than public cloud - Provides on-premises security
U.K.L.A.
Memphis
LA
N/
WA
N
Video
UnstructuredData
Sensory Data
Images
COLD ARCHIVECOST EFFECTIVE LONG TERM RETENTION
![Page 15: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/15.jpg)
15© Copyright 2015 EMC Corporation. All rights reserved.
PROBLEM• Need cost effective solution to store hue amounts of
unstructured data generated by IOT and sensors• “No Public Cloud” policy - Data needs to be on-premises• Data collection via modern cloud applications requires
compatibility with APIS’s like S3 and OpenStack• Analytics workflow is slow, expensive and complicated using
Hadoop direct attach or public cloud storage
VALUE• Cost per GB is less than public clouds• Provide high availability with on-premises security• Compatible with S3, OpenStack, and other popular API’s• HDFS compatible and enables a streamlined Hadoop workflow
for “data in place” analytics
‘IOT’ CLOUD STORAGE PLATFORM‘INTERNET OF THINGS’ – SENSORY & TELEMETRY DATA COLLECTION
![Page 16: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/16.jpg)
16© Copyright 2015 EMC Corporation. All rights reserved.
EMC & OBJECT STORAGE
![Page 17: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/17.jpg)
17© Copyright 2015 EMC Corporation. All rights reserved.
Hyper Scale - Sales Out to Billions of Objects
“Public Cloud-Like” – Secure Access, Anytime, Anywhere
Comprehensive Multi-Tenant Management
Active/Active Geo-Distributed Architecture
Multiple Protocol Support – REST and HDFS Ready
Compelling Economics –Appliance or SW Only (DIY)
ECS HYPERSCALE CLOUD STORAGE
![Page 18: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/18.jpg)
18© Copyright 2015 EMC Corporation. All rights reserved.
CUSTOMERS CAN LEVERAGE COMMODITY PLATFORMS
SOFTWARE-DEFINED STORAGE
Software-DefinedStorage
CommodityPlatforms
![Page 19: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/19.jpg)
19© Copyright 2015 EMC Corporation. All rights reserved.
COMMODITY HARDWARE VALUE PROPOSITION
• Utilize standardized, open technologies and mass market components
• Individual components provide lower performance, reliability, etc.
• At sufficient scale, with the right software, the component pool provides superior characteristics
![Page 20: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/20.jpg)
20© Copyright 2015 EMC Corporation. All rights reserved.
ECS SOFTWAREEnterprise & SPsObject & HDFS DIY Commodity
ECS APPLIANCEEnterprise & SPsObject & HDFS
integrated appliance
ViPR DATA SERVICES
Enterprise & SPsObject & HDFS
File-based Arrays
CHOICE AND FLEXIBILITY
![Page 21: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/21.jpg)
21© Copyright 2015 EMC Corporation. All rights reserved.
EMC OBJECT ECOSYSTEM
Enterprise Information Archiving
Enterprise Content Management
Analytics
Cloud Gateways
Migration
Sync & Share
CLOUD BOOST
21© Copyright 2015 EMC Corporation. All rights reserved.
Analytics
Protocols CAS
![Page 22: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/22.jpg)
22© Copyright 2015 EMC Corporation. All rights reserved.
Learn | Try | Develop | Collaborate
Explore how-to videos, helpful guides, and training
Download ViPR FREE with no time limit – for non-production use
Access SDKs, FAQs, forums, technical documentation, sample apps, and more
Ask the experts, talk to peers, share ideas and experiences
www.emc.com/viprcommunity
JOIN THE ViPR COMMUNITY…
![Page 23: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/23.jpg)
![Page 24: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/24.jpg)
24© Copyright 2015 EMC Corporation. All rights reserved.
ECS TECHNICAL DETAIL• ECS STORAGE ENGINE • WRITE PATH, READ PATH, BOX CARTING• ECS GEO-CAPABILITIES
![Page 25: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/25.jpg)
25© Copyright 2015 EMC Corporation. All rights reserved.
ECS ARCHITECTURE OVERVIEW
Object
Storage Engine
HDFS NFS*• Multi-head access - ability to access same data concurrently
through multiple access protocols.
• Provides High Availability and Scalability.• Manages transactions and persistent data.• Protects data against failures, corruption and disasters
ECS Appliance
Commodity
EMC and 3rd party file arrays
![Page 26: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/26.jpg)
26© Copyright 2015 EMC Corporation. All rights reserved.
COMPREHENSIVE DATA ACCESSCOMPATIBILITY WITH COMMON INDUSTRY API’S
• Simultaneous access to underlying data through multiple interfaces– Object, HDFS, File (future)
• HDFS compatible with Cloudera, Hortonworks, Pivotal etc.
• Support for S3, Swift, Atmos and Centera CAS APIs object
• Extensions to APIs– Byte-Range updates, Atomic appends, Rich
ACLs etc.
ATMOS
![Page 27: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/27.jpg)
27© Copyright 2015 EMC Corporation. All rights reserved.
DESIGN PRINCIPLE: LAYERED ARCHITECTURE
Limitless Scale: Each layer is independently scalable, highly available, and has no single point of failure.
Scale-Out Architecture: Scale by adding more nodes, no special nodes or roles
Global Namespace: Any node has full system view off data and meta-data
Persistence Layer
Storage Engine
JBODs
OBJECTHDFS
OBJECTHDFS
OBJECTHDFS
![Page 28: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/28.jpg)
28© Copyright 2015 EMC Corporation. All rights reserved.
TRANSACTION FLOW (WRITE)
Node 1
1. Create Object request (Name, data, metadata)
Node 1 Node 4 Node 5
2. Write of data and metadata in chunk
All three copies written in parallel. Write successful only if all copies ack
Node 2
5. Back to client
3. Index update (name, location) to the owner
partition
4. Journal write
![Page 29: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/29.jpg)
29© Copyright 2015 EMC Corporation. All rights reserved.
ERASURE CODING
Data is written into chunks - 3 copies
Erasure coding begins as the chunks are shipped
Once EC completes, the data becomes fully protected and the 3 copies are deleted
A
A
AA
![Page 30: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/30.jpg)
30© Copyright 2015 EMC Corporation. All rights reserved.
GARBAGE COLLECTION
In Append-only systems updates/deletes cause files to have blocks of data that are unused
This is done at the level of chunk
Unused chunks reclaimed by a background task
![Page 31: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/31.jpg)
31© Copyright 2015 EMC Corporation. All rights reserved.
Node 1
1. Read Object request
3. Read data
Node 2
4. Send data back
2. Get Location
TRANSACTION FLOW (READ)
![Page 32: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/32.jpg)
32© Copyright 2015 EMC Corporation. All rights reserved.
BOX CARTING: CHUNK WRITE
Node
Buffered Writer
Acks(PARALELL SYNC WRITE)
![Page 33: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/33.jpg)
33© Copyright 2015 EMC Corporation. All rights reserved.
DATA WRITTEN IN APPEND ONLY CHUNKS
• Data is written in an append-only pattern.
• No data is overwritten or modified.
• No locking required for I/O.
• No cache invalidation required.
• Journaling, snapshot and versioning natively built-in
• ECS stores all types of data and index in “chunks”
• Chunks are– Logical containers of
contiguous space (128MB)
– Written in an append-only pattern
• All data protection operations are done on chunks
![Page 34: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/34.jpg)
34© Copyright 2015 EMC Corporation. All rights reserved.
ECS STORAGE ENGINE: KEY BENEFITS
• All nodes can process write requests for the same object simultaneously, and write to different sets of disks.
• Throughput takes advantage of all spindles and NICs in cluster.
• Payload from multiple small objects are aggregated in memory and written in a single disk write
• Efficient storage for both small and large data
![Page 35: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/35.jpg)
35© Copyright 2015 EMC Corporation. All rights reserved.
Unstructured configurations– Object & HDFS
Available in multiple capacities within a rack
Clustering across racks scales to 100s of PBs
HARDWARE CONFIGURATIONS
![Page 36: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/36.jpg)
36© Copyright 2015 EMC Corporation. All rights reserved.
ANALYTICS
![Page 37: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/37.jpg)
37© Copyright 2015 EMC Corporation. All rights reserved.
SAMPLE HADOOP WORKFLOW
HDFS
Analytical Models(Hive, HAWQ)
Data Visualizations(Tableau)
Variety of Data Sources
Data Cleansing
Ingest
Data Scientists
Store
Analyze
Surface
![Page 38: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/38.jpg)
38© Copyright 2015 EMC Corporation. All rights reserved.
HADOOP PROCESSING MODELSHARED STORAGE MODEL
VNX VMAX Isilon CommodityVNX Isilon 3rd Party Commodity
• Enables common Data Lake for LOB application storage and analytics
• Scale compute & storage independently
• Multiple distributions, clusters connect to the same data
ECS
![Page 39: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/39.jpg)
39© Copyright 2015 EMC Corporation. All rights reserved.
CHALLENGES WITH HDFS
• HDFS not Enterprise-Grade– Requires three full copies of data, no erasure
coding– No Geo-distribution, Limited DR, Multi-tenancy– Inefficient for handling small files
• High Availability Still In Progress– No Active-Active Failover even with the
secondary NN
• DAS architecture not suitable for some customers– Lack of Enterprise Data Governance Features
![Page 40: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/40.jpg)
40© Copyright 2015 EMC Corporation. All rights reserved.
HDFS DATA SERVICE OVERVIEW• Addresses limitations of off-the-
shelf HDFS
• Brings HDFS to existing storage hardware
• Enables HDFS/Object/File scenarios
• Flexible software model allows for future co-location of compute and storage
![Page 41: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/41.jpg)
41© Copyright 2015 EMC Corporation. All rights reserved.
HDFS DATA SERVICE OVERVIEW• API head
– Custom client/server protocol optimized for high scale
– Uses the same unstructured storage engine as ECS/ViPR Object data service
• Client library over the HDFS API– Provides a viprfs:// drop-in replacement for HDFS 2.0– Can be seamlessly added to existing Hadoop
distributions
• Implemented as a Hadoop Compatible Filesystem (HCFS)– Supports HDFS 2.0 and 2.2
![Page 42: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/42.jpg)
42© Copyright 2015 EMC Corporation. All rights reserved.
HDFS ARCHITECTURE
RM / AsM
Commodity Compute & Storage
Node Manager
Data Store
M a p R e d u c e Ta s k
MapReduce Task
Client
Node Manager
Data Store
M a p R e d u c e Ta s k
MapReduce Task
Node Manager
Data Store
M a p R e d u c e Ta s k
MapReduce Task
NAME NODE
SECNAME NODE
![Page 43: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/43.jpg)
43© Copyright 2015 EMC Corporation. All rights reserved.
ViPR/ECS HDFS ARCHITECTURE
Client
Node ManagerM a p R e d u c e Ta s k
MapReduce Task
ViPR/ECS Client
ViPR/ECS Client
Node ManagerM a p R e d u c e Ta s k
MapReduce Task
ViPR/ECS Client
ViPR/ECS Client
Node ManagerM a p R e d u c e Ta s k
MapReduce Task
ViPR/ECS Client
ViPR/ECS Client
NAME NODERM/AsM
SECNAME NODE
![Page 44: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/44.jpg)
44© Copyright 2015 EMC Corporation. All rights reserved.
Customer’s Hadoop Compute Cluster
HDFS – ECS APPLIANCE DEPLOYMENT
ViPR Controller VMViPR Controller
VMViPR Controller VM
Data Read/Write
Object/HDFS
…
Object/HDFS
![Page 45: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/45.jpg)
45© Copyright 2015 EMC Corporation. All rights reserved.
HDFS DATA SERVICE/ECS ARCHITECTURE
ECS Storage Engine
HDFSAPI
Head
S3API
Head
Customer’s Hadoop Compute Cluster
ViPR Data Service Node
Data Read/Write via ViPR HDFS
![Page 46: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/46.jpg)
46© Copyright 2015 EMC Corporation. All rights reserved.
HDFS DATA SERVICE VALUE PROPOSTION
• High Availability Built-In, No SPOF
• Avoids multiple copies of data
• Erasure Coding Support
• Geo-Distributed Across Sites
• Multi-tenancy, Metering, Chargeback
• Allows Byte-Range Updates Through S3 Interface
• ViPR Controller aids in Management & Monitoring
![Page 47: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/47.jpg)
47© Copyright 2015 EMC Corporation. All rights reserved.
UPCOMING ECS INTEGRATIONS
![Page 48: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/48.jpg)
48© Copyright 2015 EMC Corporation. All rights reserved.
ISILON CLOUDPOOLSSMART TIERING TO OBJECT STORES
Key Features
Benefits
Stub to Cloud of choice
Extending SmartPools workflow to CloudPools
Ability to send encrypted data to the cloud
Compression for efficient transport
Simple policy based management
Combine file & object store benefits
Use stubs to optimize local storage space, with offsite archive protection
Seamless placement and availability of data per policy
One Accessible namespace
SmartPools -> CloudPools
Clients
SMB | NFS | REST| HDFS | SWIFT
OneFS
Service Provider
PublicCloud
ECS
![Page 49: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/49.jpg)
49© Copyright 2015 EMC Corporation. All rights reserved.
EMC CLOUDBOOST LONG TERM RETENTION TO THE CLOUD
LAN LAN/WANCloudBoostappliance
DesktopsLaptops
Files NAS/NDMP
VMware &Hyper-V
Databases
Email Applications
DB
ROBO
Protected by NetWorker, Avamar, NetBackup
Key Features
Benefits
Long-term retention to ECS for NetWorker, Avamar, NetBackup
Inline variable de-duplication and compression
Data encrypted in-flight and at rest
Cloud choice: private/public clouds
Appliance cache for ROBO
Capacity of up to 6PB logical per appliance
Central management via a cloud portal
Lower storage cost per TB
Efficiency: Reduced network and storage consumption
Lower risk, operational overhead than tape.
Airtight security
![Page 50: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/50.jpg)
50© Copyright 2015 EMC Corporation. All rights reserved.
WRITE PATH, READ PATH, BOX CARTING
![Page 51: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/51.jpg)
51© Copyright 2015 EMC Corporation. All rights reserved.
TRANSACTION FLOW (WRITE)
Node 1
1. Create Object request (Name, data, metadata)
Node 1 Node 4 Node 5
2. Write of data and metadata in chunk.
All three copies written in parallel. Write successful only if all copies ack.
Node 2
5. Back to client
3. Index update (name, location) to
the owner partition.
4. Journal write
![Page 52: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/52.jpg)
52© Copyright 2015 EMC Corporation. All rights reserved.
ERASURE CODING
Data is written into chunks 3 copies
Once a chunk fills to 128 MB, erasure coding starts
Once it is completed and data is protected the 3 copies are deleted.
A
A
AA
![Page 53: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/53.jpg)
53© Copyright 2015 EMC Corporation. All rights reserved.
GARBAGE COLLECTION
In Append-only systems updates/deletes cause files to have blocks of data that are unused.
This is done at the level of chunk
Unused chunks reclaimed by a background task
![Page 54: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/54.jpg)
54© Copyright 2015 EMC Corporation. All rights reserved.
Node 1
1. Read Object request
3. Read data
Node 2
4. Send data back
2. Get Location
TRANSACTION FLOW (READ)
![Page 55: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/55.jpg)
55© Copyright 2015 EMC Corporation. All rights reserved.
BOX CARTING: CHUNK WRITE
Node
Buffered Writer
Acks(PARALELL SYNC WRITE)
![Page 56: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/56.jpg)
56© Copyright 2015 EMC Corporation. All rights reserved.
• ECS GEO-STORAGE OVERVIEW• DATA PROTECTION• GLOBAL DATA ACCESS
ECS GEO-CAPABILITES
![Page 57: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/57.jpg)
57© Copyright 2015 EMC Corporation. All rights reserved.
ECS GEO-STORAGE OVERVIEW
• Data Protection– Protection against data center failure– Seamless failover and recovery
• Global Data Access– Global namespace– Ability to read/write data from any site
• Optimized Storage– Low storage overhead– WAN Optimization
• Applicable to all unstructured Storage Engine based heads– Object, HDFS– File when available
![Page 58: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/58.jpg)
58© Copyright 2015 EMC Corporation. All rights reserved.
DATA PROTECTION
![Page 59: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/59.jpg)
59© Copyright 2015 EMC Corporation. All rights reserved.
INDUSTRY SOLUTION: MIRROR COPY
• Mirrored copy in a backup site
• Benefit: Achieves Local reconstruction on hardware failure
• Shortcoming: Storage overhead -> 2.66xPrimary Secondary
![Page 60: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/60.jpg)
60© Copyright 2015 EMC Corporation. All rights reserved.
INDUSTRY SOLUTION: DISTRIBUTED ERASURE CODING
• Distributing fragments across sites
• Benefit: Achieves low Storage Overhead ~ 1.6x
• Shortcoming: Disk/Node failure requires fragments to be fetched over the WAN.
Site 1 Site 2
Site 3 Site 4
![Page 61: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/61.jpg)
61© Copyright 2015 EMC Corporation. All rights reserved.
ECS MODEL: BEST OF BOTH WORLDS
• Achieves low Storage Overhead ~ 1.8x
• Local hardware failure recovery requires no WAN traffic.
• Handles local hardware and full data center failures– Disk, Node, Rack, Data Center are
failure domains
Site 1 Site 2
Site 3 Site 4
![Page 62: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/62.jpg)
62© Copyright 2015 EMC Corporation. All rights reserved.
GLOBAL DATA ACCESS
![Page 63: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/63.jpg)
63© Copyright 2015 EMC Corporation. All rights reserved.
INDUSTRY SOLUTION: SEGREGATED NAMESPACE
• Customers are asked to pick a location for each bucket.
• Shortcoming: sites are vertical silos, unaware of each other’s namespaces.
Site 1 Site 2
app app
Bucket BBucket A
![Page 64: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/64.jpg)
64© Copyright 2015 EMC Corporation. All rights reserved.
INDUSTRY SOLUTION: MULTI-ACCESS WITH EVENTUAL CONSISTENCY
• Global Namespace with read only replicas.
• Replicas have eventual consistency
• Shortcomings: Difficult to write applications against eventual consistency models
Site 1 Site 2
app
Bucket A
app
read only
![Page 65: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/65.jpg)
65© Copyright 2015 EMC Corporation. All rights reserved.
ECS: MULTI-ACCESS WITH STRONG CONSISTENCY
• Global Namespace: buckets stretches across sites
• Global Access: Any data can be read and written to any site
• Strongly consistent: Always returning latest version without requiring synchronous write.
Site 1 Site 2
Bucket A
app app app
![Page 66: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/66.jpg)
66© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZED STORAGE
• Low Storage Overhead: ~1.8x replication over head across 4 sites
• WAN Optimization: All node and disk failures are repaired within the site, without any WAN traffic.
![Page 67: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/67.jpg)
67© Copyright 2015 EMC Corporation. All rights reserved.
STORAGE OVERHEAD
# of Data Centers Overhead1 1.33 x
2 2.67 x
3 2.00 x
4 1.77 x
5 1.67 x
6 1.60 x
7 1.55 x
8 1.52 x
![Page 68: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/68.jpg)
68© Copyright 2015 EMC Corporation. All rights reserved.
ECS GEO KEY DIFFERENTIATORS
• Tolerates one site disaster along with up to 2 node failures in all the rest of the sites.
• Component failures are recovered using fragments from local site without WAN traffic
• Geo-efficient (~1.8 copies across 4 sites) without WAN read/write penalties
![Page 69: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/69.jpg)
69© Copyright 2015 EMC Corporation. All rights reserved.
ECS APPLIANCE
![Page 70: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/70.jpg)
70© Copyright 2015 EMC Corporation. All rights reserved.
Not Contradictory!
Components are Commodity
x86 Servers
Ethernet Networking
SATA Disk Drives
Innovation in how they’re put together to enable reliability, availability, and serviceability!
COMMODITY INNOVATION
![Page 71: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/71.jpg)
71© Copyright 2015 EMC Corporation. All rights reserved.
ECS APPLIANCE CHARACTERISTICS
• Use COTS Components– Economies of scale
• Density Optimized– Up to 72TB Raw / Rack Unit– Saves Power/GB, Real Estate costs, etc.
• Labor Optimized– Manage the cluster, not the devices– Maximize Serviceability
• Protection Efficiency– Geo-efficient storage
![Page 72: ECS/Cloud Object Storage - DevOps Day](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d37207bb61eb024e8b4684/html5/thumbnails/72.jpg)