Storage Solutions in the AWS Cloud - Amazon S3 · Cloud Storage AWS offers multiple cloud-based...
Transcript of Storage Solutions in the AWS Cloud - Amazon S3 · Cloud Storage AWS offers multiple cloud-based...
On-Premise Storage Options
SAN – network-attached block devices: LUNs
DAS – local block devices (disks)
NAS – network attached file systems, filers
Differ in, Performance, Durability, and Cost
Often used to form a Storage Hierarchy
Traditional Storage
Cloud Storage
AWS offers multiple cloud-based storage options.
• Amazon S3 (Simple Storage Service)
• Amazon EBS (Elastic Block Storage)
• Amazon EC2 Local Instance Store (Ephemeral volumes)
Each has a unique combination of performance, durability, cost, and interface
Often used to form a Storage Hierarchy
Other Storage options?
Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well:
Cloudfront
Amazon RDS
SimpleDB
SQS
Relational Databases on EC2
Storage by the Numbers
S3 EBS Local
Scale Unlimited Storage
5TB objects
100 name spaces (buckets)
1TB per volume
~44TB per server
147 – 1690GB
1-5 volumes
Durability 99.999999999% (standard)
99.99%
(RRS)
.1-.5% AFR Ephemeral design
requires application-
level redundancy
Latency High Low Very Low
Management Zero Maintenance Snapshots, RAID,
Consistency
Manual Redundancy
Locality Region-Centric AZ-Specific In-machine
Interface REST or SOAP
Web Services
Block Device Block Device
S3 (Simple Storage Service)
Pay for exactly what you use
Highly Durable
Limitlessly scalable
Natively On-Line
S3 Best Practices
Replicate Metadata
Parallel IO for max speed
Multipart Upload
Sorted Writes
Resource-level IAM permissions
Amazon EBS
Pay for what you provision
POSIX compliant network attached block device
Native Redundancy and Write Cache; optimized for random writes and reads.
Application hosting
Data Processing
Database
Large Data
EBS Use Cases and Users
Envoy Media Group Case Study Storage for multimedia content delivery on Amazon Web Services (AWS)
Gluster - 640 W. California Ave. Suite 200, Sunnyvale, CA 94086 - 1.800.805.5215 - www.gluster.com
Introduction
Established in 2006, Envoy Media Group is an innovative marketing firm
that offers its clients highly-targeted media campaigns across multiple
media channels including television, radio, the Internet, direct video and
with a unique
and powerful advantage. By capitalizing on their multiplemarketing
channels, Envoy caters to selected businesses offering a focused plan and
implementation of concentrated campaigns both online and offine.
Envoy moved their IT infrastructure from a traditional co-location facility to Amazon Web Services to gain greater flexibility, take
advantage of -as-you- e their media assets were always available, easy
to share, and promptly delivered, Envoy selected Gluster in concert with Amazon EC2, EBS volumes, and S3.
Challenges Envoy currently hosts their entire web presence on Amazon Web Services (AWS)
which has helped them meet the needs of their growing business and customer
base. One of their biggest challenges was ensuring the IT infrastructure was able
to respond to the highly dynamic nature of demand such as when campaigns
launch and then grow exponentially. From a data storage standpoint, they also
needed a solution that integrated easily with AWS and provided high availability.
Storing the data locally on EC2 instances is not suffcient since that data is not
persistent and would be lost if a server reboots. Amazon EBS volumes provide persistent storage, but a volume can only be attached
to one instance at a time a problem if your business requires hundreds of web servers to be provisioned in a few minutes. Envoy
needed to ensure their data was easily shared.
our server infrastructure, but in terms of addressing our storage requirements Gluster was
the only solution on the market that allowed us to tie the storage pieces id Michael
Taggart, chief technology officer at Envoy Media Gr -to-use solution that could integrate with AWS and
deliver the scalability and cost-effectiveness we were looking for. solution was able to deliver these
benefits
Solution Envoy deployed an integrated solution using Amazon EC2, EBS,
and S3 for backup. Gluster provides the network fle system for
the web servers and computational servers running in the EC2
instances. Media files are stored on EBS volumes which are
mounted via four Gluster storage nodes and can be accessed and
shared across as many EC2 web server instances as necessary.
The Gluster storage servers are organized in pairs, with data
replicated between each pair for high availability and the ability
to survive server failure. Additionally, data is backed up to S3.
-to-use interface and fexibility made the installation
Gluster is POSIX compliant we did not need to make any changes to our existing storage problem
with software and are not letting the hardware dictate how we operate or design
We needed an easy-to-use solution
that could integrate with AWS and
deliver the scalability and cost-
effectiveness we were looking for.
storage solution
was able to deliver these benefits
EC2 Instance Store or Ephemeral Volumes
Free with your EC2 Instance
Zero Network Overhead; local, direct attached resource.
High performance for sequential reads and writes
Temporary; goes away when your instance does
Better Together: Cross-system patterns
EBS <-> S3: Snapshots
S3 <-> SimpleDB: meta-data store
S3 <-> Local Storage: Caching
Two types of EC2 AMIs and Instances:
Instance store (“S3-backed”)Original EC2, boot from ephemeral storage
Can Start and Terminate only
All data is ephemeral (unless separate EBS volume attached)
EBS root (“boot from EBS”)Newer, boot from EBS volume
Can Start, Stop, Create Image, and Terminate
Ephemeral storage still available, but not exposed by defaultUse instance Block Device Mapping (command line and API only)
Additional EBS volumes can be attached to either type
Short Detour: S3 vs EBS AMIs
So, which should I choose?
Might be all of them!
S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources
EBS: Applications that require POSIX, Databases, Data Warehouses
Ephemeral: Cache/Swap/Temp, Applications that require minimum latency
Test!
It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.
AWS Import/Export
Faster than Internet transfer and more cost effective than upgrading your connectivity
Use cases: data migration, offsite backup, direct data interchange, disaster recovery
WAN Optimization
Asperatech, Riverbed, Tsunami-UDP, etc
How do I get it to the Cloud?
Read the originals!
Storage Options in the AWS Cloud:
https://media.amazonwebservices.com/AWS_Storage_Options.pdf
Storage Use Cases in the AWS Cloud:
https://media.amazonwebservices.com/AWS_Storage_Use_Cases.pdf