Scale Your App for the Holidays with Amazon DynamoDB
-
Upload
amazon-web-services -
Category
Documents
-
view
108 -
download
1
description
Transcript of Scale Your App for the Holidays with Amazon DynamoDB
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Scale Your App for the Holidayswith DynamoDB
build high scale applications in daysfast | scalable | cost-effective
David Pearson, Business Development Manager
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
What Is NoSQL?
CAP theorem Not Only SQL
non-relationalunstructured
distributed
horizontally scalableschema-free
eventually consistent
BASE
key-value
open-source
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
scale-up =bigger servers
scalability
performanceRDBMS
versatile
change application to use… read slaves read caches data shards
scale-out = more servers
infrastructure scaling+
application scaling
big data challenge:the cost of scalability
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
NoSQL
specialized scale-out without application changes
scalability
performanceRDBMS
versatile
infrastructure scaling=
application scaling
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
NoSQL
General Characteristics
scalability
complexity
RDBMS
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
NoSQL @ Amazon
a story in three parts…
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
RDBMS used for “all kinds of access patterns”
NoSQL @ Amazonkey-value access
complex queries
transactions
analyticspart one - early days…
data (re-)partitioning
bigger hardware is tempting
trade-off with consistency
versatileeasy to learneasy to query
scalability
availability
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
replicated DHT with consistency management
NoSQL @ Amazon
part two - dynamo…
• Consistent hashing • Optimistic replication• “Sloppy quorum”• Anti-entropy mechanisms• Object versioning
specialized technology
limited query capabilities
simpler consistency
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
replicated DHT with consistency management
NoSQL @ Amazon
part two - dynamo…
higher availability
incremental scalability
predictable performance
eventual consistency
no need to re-architect applications
just add hardware!
simpler query model
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
replicated DHT with consistency management
NoSQL @ Amazon
part two - dynamo…
higher availability
incremental scalability
predictable performance
no strong consistency
scaling effort required
operational complexity
much better, but…
still required developers to be administrators…
install, patch, upgrade, balance clusters
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
NoSQL @ Amazon
part three - Amazon DynamoDB…
Fast & Predictable Performance
Seamless Scalability
Zero Administration
nosql database service
“Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” – Don MacAskill, CEO
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
• No compromise on durability or availability for performance• Express scale needs in simpler terms, not servers and disks• Scale will be our problem and not our customers • Extremely easy to use with no administration• Provide consistently low latencies
NoSQL @ Amazon
Design Philosophy
provisioned throughput
continuous replication across multiple AZ’s
backed by SSDs
consistent, disk-only writes (not memory)
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
November Traffic at Amazon.com
76% = wasted hardware
Capacity needed before DynamoDB
Actual traffic
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Capacity needed before DynamoDB
Actual traffic
November Traffic at Amazon.com
Capacity we can provision with DynamoDB
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Case Study
Stores user objects in cloud
Queries and object searches are served by DynamoDB
“We were excited by how fast we were able to put DynamoDB into production and how muchdeveloper time we saved. In addition, DynamoDB lets us scale up and down easily by simplyreserving increased throughput capacity when we need it and dialing it back when we don’t”
- Russel Dicker, Amazon CloudDrive
metadata in DynamoDB
objects in S3
list all the objects in my drive
find all my music albums
small objects move faster!
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Goal = enable customers to see Cloud Drivephotos on their Kindle
Need = low latency access to the metadata, at any scale
Result = lower total cost of ownership (tco)lower admin effort required to scale
Cloud Drive Photos
hardware costs + operational costs + opportunity cost of feature development
future tco
#customers X#photos
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
image thumbnailsalbum cover art
links to S3
DynamoDB access pattern
list all images for photos uploaded by a customer
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Two decisions + three clicks = ready for use
Getting started with DynamoDB…
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Two decisions + three clicks = ready for use
primary keys +
provisioned level of throughput (easy to change)
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
• Reserve throughput for each table• Set at table creation, increased and decreased via an API call
$0.01 per hour for every 10 units of Write Capacity$0.01 per hour for every 50 units of Read Capacity$1.00 per GB-month of Storage
Provisioned Throughput
... for items up to 1KB in size
=10 writesper second…
=50 strongly consistentreads per second… free tier…
100MB storage + 5 writes/sec + 10 reads/sec each month
effectively assigns each tableits own set of servers
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Items are indexed by primary keysingle hash keys and composite keys
key-value access =extreme performance
hash + range
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
deviceid = 21EC2020-3AEA-1069-A2DD-08002B30309D total = 25.00
deviceid = 74ED9134-3FEC-9902-E8BA-19733F49779C total = 50.00
table = collection of items
hash key
item (max size 64k) =collection of attributes
attribute (key value pair)string, number, binary
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
userid = 100 date = 2012-10-24-09-00-10 total = 25.00
userid = 100 date = 2012-10-24-09-00-11 total = 50.00
hash key range key
mapping 1:M relations
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
DynamoDBAPIs CreateTable
UpdateTable
DeleteTable
DescribeTable
ListTables
PutItem
GetItem
UpdateItem
DeleteItem
BatchGetItem
BatchWriteItem
Query
Scan
manage tables
query specific items OR scan the full table
read and write items
bulk select or update
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Available for hash-and-range primary key tables
Retrieve all items by hash key
Range key conditions: ==, <, >, >=, <=, begins with, between
Counts. Top and bottom n values. Paged responses
Query Patterns
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Designing a Photo Store
Getting started with DynamoDB…
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Photo Store
use case usage patternsdata design
enable users to upload and view photos from any device
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Photo Store
use case usage patternsdata design
upload photos
view photos by time
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Photo Store
use case usage patternsdata design
table photos
hash userid
range timestamp +
photoid
attributes location,
resolution,
comments, tags, s3link
data model is optimizedfor retrieval performance
tags enable search integration
s3link points to raw files
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Photo Store
use case usage patternsdata design
table photos
hash userid
range timestamp +
photoid
attributes location,
resolution,
comments, tags, s3link
Potential Enhancements…• Albums• Search integration (CloudSearch)• Global caching and delivery of
media from S3 (CloudFront)
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Enhancing relational DB performance… and cost
Extreme scalability…
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Infrastructure Scaling
Read Scaling #1
Read Scaling #2 (hot keys)
Write Scaling
Scaling Relational DB’s
bigger hardware (scaling up)
read replicas (slaves)
read caching (memcached)
data sharding
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Infrastructure Scaling
Read Scaling #1
Read Scaling #2 (hot keys)
Write Scaling
Scaling Relational DB’s
application changes required
bigger hardware (scaling up)
read replicas (slaves)
read caching (memcached)
data sharding
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Scaling Relational DB’s with DynamoDB
Identify Design the newImplement code
candidates DynamoDB table changes in app
move simple, high-scale workloads
Tables with high transaction volume (esp writes)
Primary key-only is preferred (one non-PK index ok)
No dependencies (FK’s, triggers, procedures)
hopefully, forthe last time
maximize access performance
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
DynamoDB Scalability @ HalfBrick Studios
• Fruit Ninja Frenzy (facebook)
• Moved game data into DynamoDB
• Grew from 1 million to 8 million active monthly users in two weeks
8M
1M
“it’s really tough to quickly scale a normal database
system to handle that kind of rapid increase in load”
- Glen Arrowsmith, Systems Architect
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
DynamoDB Scalability
“Creating a table that can serve 100,000 writes/second is no more
work than creating a table that can serve 10 writes/second”
Werner Vogels, Amazon CTO
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Super Bowl promotion
Millions of interactions over a relatively short period
Built the app in 3 days – from design to production-ready
Weatherbug app – lightning detection & alerting for 6M phones
Extending to 40M users/month
Developed and tested in weeks
“1/20th of the cost of the traditional DB approach”
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
DynamoDBFast
Scalable
Cost-Effective
• minimal development time and effort
• consistently low latency
• effortless scaling to meet workload demand
• Reduces DB costs and increases reliability
• Free tier reduces initial development costs
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
• Building Applications with DynamoDB (Matt Wood) google search = youtube "building applications with dynamodb"http://www.youtube.com/watch?v=4jZthAFKAE4http://www.slideshare.net/AmazonWebServices/building-applications-with-dynamodb
• From the Super Bowl to WeatherBug (Werner Vogels)http://www.allthingsdistributed.com/2012/06/amazon-dynamodb-growth.html
Recommended Resources
© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
Questions?
aws.amazon.com/dynamodb