February 2016 Webinar Series - Introduction to DynamoDB
-
Upload
amazon-web-services -
Category
Technology
-
view
1.828 -
download
3
Transcript of February 2016 Webinar Series - Introduction to DynamoDB
![Page 1: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rick Houlihan, Principal Solutions ArchitectFebruary 2016
Introduction to DynamoDBhttp://bit.ly/NoSQLDesignPatterns
![Page 2: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/2.jpg)
What to expect from the session
Brief history of data processing DynamoDB Internals
- Tables, API, data types, indexes
- Scaling and data modeling
Design patterns and best practices Event driven applications and DDB Streams
![Page 3: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/3.jpg)
What is a Database?
“A place to put stuff my app needs.”– Average Developer
![Page 4: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/4.jpg)
Timeline of Database TechnologyDa
ta P
ress
ure
![Page 5: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/5.jpg)
Data Volume Since 2010
Dat
a Vo
lum
eHistorical Current
90% of stored data generated in last 2 years
1 Terabyte of data in 2010 equals 6.5 Petabytes today
Linear correlation between data pressure and technical innovation
No reason these trends will not continue over time
![Page 6: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/6.jpg)
Technology Adoption and the Hype Curve
![Page 7: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/7.jpg)
Why NoSQL?
Optimized for storage Optimized for compute
Normalized/relational Denormalized/hierarchical
Ad hoc queries Instantiated views
Scale vertically Scale horizontally
Good for OLAP Built for OLTP at scale
SQL NoSQL
![Page 8: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/8.jpg)
It’s all about aggregations…
Document Management Process ControlSocial Network
Data TreesIT Monitoring
![Page 9: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/9.jpg)
SQL vs. NoSQL Access Pattern
![Page 10: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/10.jpg)
Amazon DynamoDB
Document or Key-Value Scales to Any WorkloadFully Managed NoSQL
Access Control Event Driven ProgrammingFast and Consistent
![Page 11: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/11.jpg)
TableTable
Items
Attributes
PartitionKey
Sort Key
MandatoryKey-value access patternDetermines data distribution
OptionalModel 1:N relationshipsEnables rich query capabilities
All items for key==, <, >, >=, <=“begins with”“between”“contains”“in”sorted resultscountstop/bottom N values
![Page 12: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/12.jpg)
00 55 A954 FFAA00 FF
Partition Keys
Id = 1Name = Jim
Hash (1) = 7B
Id = 2Name = AndyDept = Eng
Hash (2) = 48
Id = 3Name = KimDept = Ops
Hash (3) = CD
Key Space
Partition Key uniquely identifies an itemPartition Key is used for building an unordered hash indexAllows table to be partitioned for scale
![Page 13: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/13.jpg)
Partition 3
Partition:Sort Key uses two attributes together to uniquely identify an ItemWithin unordered hash index, data is arranged by the sort keyNo limit on the number of items (∞) per partition keyExcept if you have local secondary indexes
Partition:Sort Key
00:0 FF:∞
Hash (2) = 48
Customer# = 2Order# = 10Item = Pen
Customer# = 2Order# = 11Item = Shoes
Customer# = 1Order# = 10Item = Toy
Customer# = 1Order# = 11Item = Boots
Hash (1) = 7B
Customer# = 3Order# = 10Item = Book
Customer# = 3Order# = 11Item = Paper
Hash (3) = CD
55 A9:∞54:∞ AAPartition 1 Partition 2
![Page 14: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/14.jpg)
Partitions are three-way replicated
Id = 2Name = AndyDept = Engg
Id = 3Name = KimDept = Ops
Id = 1Name = Jim
Id = 2Name = AndyDept = Engg
Id = 3Name = KimDept = Ops
Id = 1Name = Jim
Id = 2Name = AndyDept = Engg
Id = 3Name = KimDept = Ops
Id = 1Name = Jim
Replica 1
Replica 2
Replica 3
Partition 1 Partition 2 Partition N
![Page 15: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/15.jpg)
Indexes
![Page 16: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/16.jpg)
Local secondary index (LSI)
Alternate sort key attributeIndex is local to a partition key
A1(partition)
A3(sort)
A2(item key)
A1(partition)
A2(sort) A3 A4 A5
LSIs A1(partition)
A4(sort)
A2(item key)
A3(projected)
Table
KEYS_ONLY
INCLUDE A3
A1(partition)
A5(sort)
A2(item key)
A3(projected)
A4(projected) ALL
10 GB max per partition key, i.e. LSIs limit the # of range keys!
![Page 17: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/17.jpg)
Global secondary index (GSI)
Alternate partition and/or sort keyIndex is across all partition keysUse composite sort keys for compound indexes
A1(partition) A2 A3 A4 A5
A5(partition)
A4(sort)
A1(item key)
A3(projected) INCLUDE A3
A4(partition)
A5(sort)
A1(item key)
A2(projected)
A3(projected) ALL
A2(partition)
A1(itemkey) KEYS_ONLY
GSIs
Table RCUs/WCUs provisioned separately for GSIs
Online indexing
![Page 18: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/18.jpg)
How do GSI updates work?
Table
Primary tablePrimary
tablePrimary tablePrimary
tableGlobal
Secondary Index
Client1. Update request
2. Asynchronous update (in progress)
2. Update response
If GSIs don’t have enough write capacity, table writes will be throttled!
![Page 19: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/19.jpg)
LSI or GSI?
LSI can be modeled as a GSIIf data size in an item collection > 10 GB, use GSIIf eventual consistency is okay for your scenario, use GSI!
![Page 20: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/20.jpg)
Scaling
![Page 21: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/21.jpg)
Scaling
Throughput Provision any amount of throughput to a table
Size Add any number of items to a table
- Max item size is 400 KB- LSIs limit the number of range keys due to 10 GB limit
Scaling is achieved through partitioning
![Page 22: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/22.jpg)
Throughput
Provisioned at the table level Write capacity units (WCUs) are measured in 1 KB per second Read capacity units (RCUs) are measured in 4 KB per second
- RCUs measure strictly consistent reads- Eventually consistent reads cost 1/2 of consistent reads
Read and write throughput limits are independent
WCURCU
![Page 23: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/23.jpg)
Partitioning Math
In the future, these details might change…
Number of Partitions
By Capacity (Total RCU / 3000) + (Total WCU / 1000)
By Size Total Size / 10 GB
Total Partitions CEILING(MAX (Capacity, Size))
![Page 24: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/24.jpg)
Partitioning Example
Table size = 8 GB, RCUs = 5000, WCUs = 500
RCUs per partition = 5000/3 = 1666.67WCUs per partition = 500/3 = 166.67Data/partition = 10/3 = 3.33 GB
RCUs and WCUs are uniformly spread across partitions
Number of Partitions
By Capacity (5000 / 3000) + (500 / 1000) = 2.17
By Size 8 / 10 = 0.8
Total Partitions CEILING(MAX (2.17, 0.8)) = 3
![Page 25: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/25.jpg)
What causes throttling?
If sustained throughput goes beyond provisioned throughput per partition
Non-uniform workloads Hot keys/hot partitions Very large bursts
Mixing hot data with cold data Use a table per time period
From the example before: Table created with 5000 RCUs, 500 WCUs RCUs per partition = 1666.67 WCUs per partition = 166.67 If sustained throughput > (1666 RCUs or 166 WCUs) per key or partition, DynamoDB
may throttle requests- Solution: Increase provisioned throughput
![Page 26: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/26.jpg)
What bad NoSQL looks like…
Par
titio
n
Time
Heat
![Page 27: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/27.jpg)
Getting the most out of DynamoDB throughput
“To get the most out of DynamoDB throughput, create tables where the hash key element has a large number of distinct values, and values are requested fairly uniformly, as randomly as possible.”
—DynamoDB Developer Guide
Space: access is evenly spread over the key-space
Time: requests arrive evenly spaced in time
![Page 28: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/28.jpg)
Much better picture…
![Page 29: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/29.jpg)
Data Modeling
![Page 30: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/30.jpg)
1:1 relationships or key-values
Use a table or GSI with an alternate partition keyUse GetItem or BatchGetItem API
Example: Given an SSN or license number, get attributes
Users TablePartition key AttributesSSN = 123-45-6789 Email = [email protected], License = TDL25478134SSN = 987-65-4321 Email = [email protected], License = TDL78309234
Users-License-GSIPartition key AttributesLicense = TDL78309234 Email = [email protected], SSN = 987-65-4321License = TDL25478134 Email = [email protected], SSN = 123-45-6789
![Page 31: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/31.jpg)
1:N relationships or parent-children
Use a table or GSI with partition and sort keyUse Query API
Example:Given a device, find all readings between epoch X, Y
Device-measurementsPartition Key Sort key AttributesDeviceId = 1 epoch = 5513A97C Temperature = 30, pressure = 90DeviceId = 1 epoch = 5513A9DB Temperature = 30, pressure = 90
![Page 32: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/32.jpg)
N:M relationships
Use a table and GSI with partition and sort key elements switchedUse Query API
Example: Given a user, find all games. Or given a game, find all users.
User-Games-TablePartition Key Sort keyUserId = bob GameId = Game1UserId = fred GameId = Game2UserId = bob GameId = Game3
Game-Users-GSIPartition Key Sort keyGameId = Game1 UserId = bobGameId = Game2 UserId = fredGameId = Game3 UserId = bob
![Page 33: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/33.jpg)
Hierarchical DataTiered relational data structures
![Page 34: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/34.jpg)
How OLTP Apps Use Data
Mostly Hierarchical Structures
Entity Driven Workflows Data Spread Across Tables Requires Complex Queries
![Page 35: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/35.jpg)
Hierarchical Structures as Items…Use composite sort key to define a HierarchyHighly selective result sets with sort queriesIndex anything, scales to any size
![Page 36: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/36.jpg)
… or as Documents (JSON)
JSON data types (M, L, BOOL, NULL)Indexing via Streams/LambdaFully Atomic Updates
![Page 37: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/37.jpg)
Scenarios and Best Practices
Messaging Online VotingTime Series Data
GamingProduct Catalog
![Page 38: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/38.jpg)
Event LoggingStoring time series data
![Page 39: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/39.jpg)
Time Series TablesEvents_table_2015_April
Event_id(Partition)
Timestamp(Sort)
Attribute1 …. Attribute N
Events_table_2015_MarchEvent_id(Partition)
Timestamp(Sort)
Attribute1 …. Attribute N
Events_table_2015_FeburaryEvent_id
(Partition)Timestamp
(Sort)Attribute1 …. Attribute N
Events_table_2015_JanuaryEvent_id
(Partition)Timestamp
(Sort)Attribute1 …. Attribute N
RCUs = 1000WCUs = 1
RCUs = 10000WCUs = 10000
RCUs = 100WCUs = 1
RCUs = 10WCUs = 1
Current table
Older tables
Hot
dat
a
Don’t mix hot and cold data; archive cold data to Amazon S3
Older tables
![Page 40: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/40.jpg)
Use a table per time period
Pre-create daily, weekly, monthly tables
Provision required throughput for current table
Writes go to the current table
Turn off (or reduce) throughput for older tables
Dealing with time series data
![Page 41: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/41.jpg)
Product CatalogPopular items (read)
![Page 42: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/42.jpg)
Partition 12000 RCUs
Partition K2000 RCUs
Partition M2000 RCUs
Partition 502000 RCU
Scaling Bottlenecks
Product A Product B
Shoppers
70,000/sec
ProductCatalog Table
SELECT Id, Description, ...FROM ProductCatalogWHERE Id="POPULAR_PRODUCT"
![Page 43: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/43.jpg)
![Page 44: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/44.jpg)
Partition 1 Partition 2
ProductCatalog Table
DynamoDB
User User
SELECT Id, Description, ...FROM ProductCatalogWHERE Id="POPULAR_PRODUCT"
Cache Popular Items
![Page 45: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/45.jpg)
![Page 46: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/46.jpg)
Messaging AppLarge itemsFilters vs. indexesM:N Modeling—inbox and outbox
![Page 47: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/47.jpg)
MessagesTable
Messages App
David
SELECT *FROM MessagesWHERE Recipient='David'LIMIT 50ORDER BY Date DESC
Inbox
SELECT *FROM MessagesWHERE Sender ='David'LIMIT 50ORDER BY Date DESC
Outbox
![Page 48: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/48.jpg)
Recipient Date Sender Message
David 2014-10-02 Bob …
… 48 more messages for David …
David 2014-10-03 Alice …
Alice 2014-09-28 Bob …
Alice 2014-10-01 Carol …
Large and small attributes mixed
(Many more messages)
Messages Table
50 items × 256 KB each
Partition key Sort key
Large message bodiesAttachments
SELECT *FROM MessagesWHERE Recipient='David'LIMIT 50ORDER BY Date DESC
Inbox David
![Page 49: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/49.jpg)
Computing inbox query cost
Items evaluated by query
Average item size
Conversion ratio
Eventually consistent reads
50 * 256KB * (1 RCU / 4KB) * (1 / 2) = 1600 RCU
![Page 50: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/50.jpg)
Separate the bulk data
Uniformly distributes large item readsQuery Inbox-GSI: 1 RCU (50 sequential items at 128 bytes)BatchGetItem Messages: 1600 RCU (50 separate items at 256 KB)
David
Recipient Date Sender Subject MsgId
David 2014-10-02 Bob Hi!… afed
David 2014-10-03 Alice RE: The… 3kf8
Alice 2014-09-28 Bob FW: Ok… 9d2b
Alice 2014-10-01 Carol Hi!... ct7r
Inbox-GSI
MsgId Body
9d2b …
3kf8 …
ct7r …
afed …
Messages Table
![Page 51: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/51.jpg)
Inbox GSI
Define which attributes to copy into the index
![Page 52: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/52.jpg)
Outbox Sender
Outbox GSI
SELECT *FROM MessagesWHERE Sender ='David'LIMIT 50ORDER BY Date DESC
![Page 53: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/53.jpg)
Inbox Outbox
Messaging app
MessagesTable
InboxGlobal secondary
index
OutboxGlobal secondary
index
David
![Page 54: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/54.jpg)
Distribute large items
Reduce one-to-many item sizes
Configure secondary index projections
Use GSIs to model M:N relationship between sender and recipient
Important when: Querying many large items at once InboxMessagesOutbox
![Page 55: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/55.jpg)
Multiplayer Online GamingQuery filters vs.composite key indexes
![Page 56: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/56.jpg)
Partition Key
GameId Date Host Opponent Status
d9bl3 2014-10-02 David Alice DONE
72f49 2014-09-30 Alice Bob PENDING
o2pnb 2014-10-08 Bob Carol IN_PROGRESS
b932s 2014-10-03 Carol Bob PENDING
ef9ca 2014-10-03 David Bob IN_PROGRESS
Games Table
Hierarchical Data Structures
![Page 57: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/57.jpg)
Query for incoming game requests
DynamoDB indexes provide partition and sortWhat about queries for two equalities and a sort?
SELECT * FROM GameWHERE Opponent='Bob‘AND Status=‘PENDING'ORDER BY Date DESC
(hash)
(range)(?)
![Page 58: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/58.jpg)
Approach 1: Query filter
Partition Key
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
Secondary Index
BobSort Key
![Page 59: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/59.jpg)
Approach 1: Query filter
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
Secondary Index
BobSELECT * FROM GameWHERE Opponent='Bob' ORDER BY Date DESCFILTER ON Status='PENDING'
(filte
red
out)
![Page 60: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/60.jpg)
Needle in a haystack
Bob
![Page 61: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/61.jpg)
Use query filter
Send back less data “on the wire”
Simplify application code
Simple SQL-like expressions AND, OR, NOT, ()
Important when: Your index isn’t entirely selective
![Page 62: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/62.jpg)
Approach 2: Composite key
StatusDateDONE_2014-10-02
IN_PROGRESS_2014-10-08
IN_PROGRESS_2014-10-03
PENDING_2014-09-30
PENDING_2014-10-03
StatusDONE
IN_PROGRESS
IN_PROGRESS
PENDING
PENDING
Date2014-10-02
2014-10-08
2014-10-03
2014-10-03
2014-09-30
+ =
![Page 63: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/63.jpg)
Approach 2: Composite key
Partition Key
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Secondary Index
Sort Key
![Page 64: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/64.jpg)
Approach 2: Composite key
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Secondary Index
BobSELECT * FROM GameWHERE Opponent='Bob' AND StatusDate BEGINS_WITH 'PENDING'
![Page 65: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/65.jpg)
Needle in a sorted haystack
Bob
![Page 66: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/66.jpg)
Game-scores-table
Sparse indexes
Id (Partition) User Game Score Date Award
1 Bob G1 1300 2012-12-23
2 Bob G1 1450 2012-12-23
3 Jay G1 1600 2012-12-24
4 Mary G1 2000 2012-10-24 Champ
5 Ryan G2 123 2012-03-10
6 Jones G2 345 2012-03-20
Scan sparse GSIs
Award-GSI
Award(Partition) ID User Score
Champ 4 Mary 2000
![Page 67: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/67.jpg)
Replace filter with indexes
Concatenate attributes to form useful
secondary index keys
Take advantage of sparse indexes
Important when: You want to optimize a query as much as possible
Status + Date
![Page 68: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/68.jpg)
Real-Time VotingWrite-heavy items
![Page 69: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/69.jpg)
Partition 11000 WCUs
Partition K1000 WCUs
Partition M1000 WCUs
Partition N1000 WCUs
Votes Table
Candidate A Candidate B
Scaling bottlenecks
50,000/sec
70,000/sec
Voters
Provision 200,000 WCUs
![Page 70: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/70.jpg)
Write sharding
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4Candidate A_7 Candidate B_8
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
![Page 71: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/71.jpg)
Write sharding
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4Candidate A_7 Candidate B_8
UpdateItem: “CandidateA_” + rand(0, 10)ADD 1 to Votes
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
![Page 72: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/72.jpg)
Votes Table
Shard aggregation
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_5
Candidate A_6 Candidate A_8
Candidate A_7 Candidate B_8
Periodic Process
Candidate ATotal: 2.5M
1. Sum2. Store Voter
![Page 73: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/73.jpg)
Shard write-heavy partition keys
Trade off read cost for write scalability
Consider throughput per partition key
Important when: Your write workload is not horizontally scalable
![Page 74: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/74.jpg)
DynamoDB Streams
![Page 75: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/75.jpg)
Stream of updates to a table
Asynchronous
Exactly once
Strictly ordered Per item
Highly durable Scale with table
24-hour lifetime
Sub-second latency
DynamoDB Streams
![Page 76: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/76.jpg)
View Type Destination
Old image—before update Name = John, Destination = Mars
New image—after update Name = John, Destination = Pluto
Old and new images Name = John, Destination = MarsName = John, Destination = Pluto
Keys only Name = John
View types
UpdateItem (Name = John, Destination = Pluto)
![Page 77: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/77.jpg)
Stream
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5
Table
Shard 1
Shard 2
Shard 3
Shard 4
KCLWorker
KCLWorker
KCLWorker
KCLWorker
Amazon Kinesis Client Library Application
DynamoDBClient
Application Updates
DynamoDB Streams and Amazon Kinesis Client Library
![Page 78: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/78.jpg)
DynamoDB StreamsUS East (N. Virginia)
Open Source Cross-Region
Replication Library
Asia Pacific (Sydney)
Cross-region replication
EU (Ireland) Replica
![Page 79: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/79.jpg)
DynamoDB Streams and AWS Lambda
![Page 80: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/80.jpg)
Triggers
Lambda function Notify change
Derivative Tables CloudSearch
ElastiCache
![Page 81: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/81.jpg)
Collect and de-dupe data in DynamoDB
Aggregate data in-memory and flush periodically
Important when: Performing real-time aggregation and analytics
Analytics with DynamoDB Streams
![Page 82: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/82.jpg)
Architecture
![Page 83: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/83.jpg)
Reference Architecture
![Page 84: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/84.jpg)
Elastic Event Driven Applications
![Page 85: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/85.jpg)
Thank you!
Free Access to All DynamoDB Self-paced Labs: bit.ly/DynamoDBLabsNormally 10 USD per lab. Promotion expires March 31st, 2016.
bit.ly/NoSQLDesignPatterns
![Page 86: February 2016 Webinar Series - Introduction to DynamoDB](https://reader038.fdocuments.in/reader038/viewer/2022103010/58ec98dc1a28aba33a8b45a9/html5/thumbnails/86.jpg)
AWS Summit – Chicago: An exciting, free cloud conference designed to educate and inform new customers about the AWS platform, best practices and new cloud services.
Details• April 18-19, 2016 • Chicago, Illinois• @ McCormick Place
Featuring• New product launches• 50+ sessions, labs, and bootcamps• Executive and partner networking
Register Now• Go to aws.amazon.com/summits• Click on The AWS Summit - Chicago … then register.• Come and see what AWS and the cloud can do for you.
Chicago – April 18-19