Scaling with mongo db - SF Mongo User Group 7-19-2011
36
Scaling with MongoDB Jared Rosoff (jsr @10gen.com ) - @forjared
-
Upload
jared-rosoff -
Category
Technology
-
view
3.237 -
download
0
description
My talk from SF Mongo User Group on 7-19-2011
Transcript of Scaling with mongo db - SF Mongo User Group 7-19-2011
How do we do it today?
• We use a relational database but … – We don’t use joins – We don’t use transactions – We add read-only slaves– We added a caching layer– We de-normalized our data– We implemented custom sharding– We buy bigger servers
HOW’S THAT WORKING OUT FOR YOU?
Costs go up
Launch
+30
Days
+90
Days
+6
Months
+1 Year
Productivity goes downProject Start
De-normalize data model
Stop using joins
Custom caching layer
Custom sharding
By engineers, for engineers
The landscape
Depth of functionality
Sca
labi
lity
& P
erfo
rma
nce
MemcachedKey / Value
RDBMS
Scaling your app
1. Use documents 2. Indexes make me happy3. Knowing your working set4. Disks are the bottleneck5. Replication makes reading fun6. Sharding for profit
SCALING YOUR DATA MODEL
Documents
{ author : "roger", date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Spirited Away", tags : [ "Tezuka", "Manga" ], comments : [ { author : "Fred", date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)", text : "Best Movie Ever” } ]}
Disk Seeks & Data Locality
Seek = 5+ ms Read = really really fast
Disk Seeks & Data Locality
Post
AuthorCommen
t
Disk Seeks & Data LocalityPost
Author
Comment
Comment
Comment
Comment
Comment
OPTIMIZED INDEXES
Table scans
1 2 3 4 5 6 7
Looked at 7 objects
Find where x equals 7
Tree Lookup
7
6
5
4
3
2
1
Looked at 3 objects
Find where x equals 7
Random Index
Entire index must fit in RAM
Right AlignedOnly small portion in
RAM
WORKING SET SIZE
Disk
Working Set
RAM
Active Documents
+ Used Indexes
Disk
Page Fault
RAM
App
1
23
4
5
1. App requests document
2. Document not in memory
3. Evict a page from memory
4. Read block from disk
5. Return document from memory
Figuring out working Set> db.foo.stats() {
"ns" : "test.foo","count" : 1338330,"size" : 46915928,"avgObjSize" : 35.05557523181876,"storageSize" : 86092032,"numExtents" : 12,"nindexes" : 2,"lastExtentSize" : 20872960,"paddingFactor" : 1,"flags" : 0,"totalIndexSize" : 99860480,"indexSizes" : {
"_id_" : 55877632,"x_1" : 43982848
},"ok" : 1
}
Size of data
Size on disk (and in
memory!)
Size of all indexes
Average document
size
Size of each index
DISK CONFIGURATIONS
Single Disk
~200 seeks / second
RAID0
~200 seeks / second ~200 seeks / second ~200 seeks / second
RAID10
~400 seeks / second ~400 seeks / second ~400 seeks / second
REPLICATION
Replica Sets
Primary
Secondary
Secondary
Read / Write
Read
Read
Replica SetsPrimary
Secondary
Secondary
Read / Write
Read
Read
Secondary
SecondaryRead
Read
SHARDING
Shard 10..10
Shard 210..20
Shard 320..30
Shard 430..40
Primary
Secondary
Secondary
Primary
Secondary
Secondary
Primary
Secondary
Secondary
Primary
Secondary
Secondary
MongoS MongoS
400GB Index?
Shard 10..10
Shard 210..20
Shard 320..30
Shard 430..40
400GB Index?
100GBIndex!
100GBIndex!
100GBIndex!
100GBIndex!
SUMMARY
Summary
1. Use documents to your advantage!2. Optimize your indexes 3. Understand your working set4. Use a sane disk configuratino5. Use replicas to scale reads 6. Use sharding to scale writes &
working RAM
