Post on 17-Aug-2020
Moving to DynamoDB: Practical Lessons Learned
Ryan ParkInfrastructure Engineer
@StanfordRyan
Build Better APIs, Togetherwith Runscope API Monitoring & Testing
API TRAFFIC DEBUGGING
Quickly diagnose integration headaches.
Verify that your APIs are returning the right data.
DATA VALIDATION
UPTIME MONITORING
Be the first to know when an API is down.
Solve API problems as a team.
COLLABORATION
50 MM
100 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
50 MM
100 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
Monthly API Test Runs
100 MM
200 MM
300 MM
400 MM
500 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
100 MM
200 MM
300 MM
400 MM
500 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
Cumulative Test Runs(database rows)
What now?
Some results are huge
PROBLEM
SOLUTION
DynamoDB (record size < 10 KB)
Amazon S3 (record size > 10KB)
+
Hash Key: Result ID Global Secondary Index: Test ID
REVERSE INDEX IN DYNAMODB
s3_index
Selecting a Partition Key and Sort Key
Partition Key: Test ID
TEST RESULTS: THE OBVIOUS CHOICE
Sort Key: Result ID
HOW MANY PARTITIONS WILL WE HAVE?
EXAMPLE: SMALL, BUSY TABLE
Table Size 6 GB
Read Capacity Units 15,000
Write Capacity Units 2,000
Partitions:
EXAMPLE: LARGE TABLE
Table Size 380 GB
Read Capacity Units 2,100
Write Capacity Units 400
Partitions:
EXAMPLE: BULK IMPORT
Table Size 0 GB
Read Capacity Units 30
Write Capacity Units 10,000
Partitions:
EXAMPLE: BULK EXPORT & BACKUP
Table Size 4 GB
Read Capacity Units 30,600
Write Capacity Units 200
Partitions:
🦄 Partition
🐱 Partition 🐝 Partition
data hits 10 GB or
provisioned capacity hits limits
🐙 Partition 🐒 Partition
data hits 10 GB or
provisioned capacity hits limits
🐱 Partition 🐝 Partition
🐝 Partition
data hits 10 GB or
provisioned capacity hits limits
🐔 Partition 🐳 Partition
🐙 Partition 🐒 Partition
🐙 Partition 🐒 Partition 🐔 Partition 🐳 Partition
HOW MANY PARTITIONS?
Usually a power of 2.
TABLE PARTITIONS
test_results
Partition Reads Writes Hash Keys
( 200 100 α, δ, η, κ…
🕵 200 100 β, ε, θ, λ…
* 200 100 γ, ζ, ι, μ…
TABLE PARTITIONS
test_results
(α
δ
η
κ
🕵β
ε
θ
λ
*γ
ζ
ι
μ
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
TABLE PARTITIONS
test_results
(α 1 α 2
δ 1 δ 2
🕵β 1 β 2
ε 1 ε 2
*γ 1 γ 2
ζ 1 ζ 2
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
TABLE PARTITIONS
test_results
(α 1 α 2 α 3 α 4
δ 1 δ 2
🕵 *γ 1 γ 2
ζ 1 ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7
β 1
ε 1
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
200 RCU, 100 WCU
TABLE PARTITIONS
test_results
Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4
Test 5, Result 1 Test 5, Result 2
Test 4, Result 1
Test 7, Result 1
Test 3, Result 1 Test 3, Result 2
Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
200 RCU, 100 WCU
Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4
Test 5, Result 1 Test 5, Result 2
Test 4, Result 1
Test 7, Result 1
Test 3, Result 1 Test 3, Result 2
Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8
ProvisionedThroughputExceededException
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
200 RCU, 100 WCU
Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4
Test 5, Result 1 Test 5, Result 2
Test 4, Result 1
Test 7, Result 1
Test 3, Result 1 Test 3, Result 2
Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8
ProvisionedThroughputExceededException
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
200 RCU, 100 WCU
Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4
Test 5, Result 1 Test 5, Result 2
Test 4, Result 1
Test 7, Result 1
Test 3, Result 1 Test 3, Result 2
Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8200 RCU, 600 WCU 200 RCU, 600 WCU 200 RCU, 600 WCU
SHORT TERM FIX: OVER-PROVISION ALL PARTITIONS
test_results
200 RCU, 100 WCU
Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4
Test 5, Result 1 Test 5, Result 2
Test 4, Result 1
Test 7, Result 1
Test 3, Result 1 Test 3, Result 2
Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8
SHORT TERM FIX: OVER-PROVISION ALL PARTITIONS
test_results
200 RCU, 600 WCU 200 RCU, 600 WCU 200 RCU, 600 WCU
Result 58 (Test 7)
Result 55 (Test 4)
LONG TERM FIX: REPLACE THE HASH KEY
Result 41 (Test 7)
Result 8 (Test 3)
Result 3 (Test 9)
Result 37 (Test 7)
Result 73 (Test 7)
Result 87 (Test 6)
Result 27 (Test 3)
Result 63 (Test 7)
Result 85 (Test 2)
Result 78 (Test 7)
test_results_v2
200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU
DEBUGGING EXCEPTIONS
AWS APIs return useful debugging data:
"ConsumedCapacity": { "CapacityUnits": 3.0, "TableName": "test_results", "Table": { "CapacityUnits": 3.0 } }
DEBUGGING EXCEPTIONS
AWS APIs return useful debugging data:
Content-Type: application/x-amz-json-1.0 Date: Wed, 27 Jan 2016 02:31:20 GMT X-Amzn-Requestid: S0JI33OC2OPW73KHD028LJWY9W5...
DEBUGGING EXCEPTIONS
Libraries hide the details.
DEBUGGING EXCEPTIONS
RUNSCOPE TRAFFIC INSPECTOR?
“Quickly diagnose integration headaches.”
USING BOTOCORE HOOKS
100 MM
200 MM
300 MM
400 MM
500 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
100 MM
200 MM
300 MM
400 MM
500 MM
Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov
Cumulative Test Runs(database rows)
Everything is going to be
200 OK