Post on 05-Dec-2014
description
MongoDB Memory
ManagementDemystified
Alon Horev
MongoDB World 2014
Meta
Alon Horev
@alonhorev
Why should you care?
MongoDB
Memory Mapped Files
Page Cache
Storage
MongoDB
Memory Mapped Files
Page Cache
Storage
Storage
RAM SSD
HDD
volatil
epersiste
nt
Throughput in MB
Price per GB5.5$ 0.5$ 0.05$
6400 650 1-160
Hardware Configuration
MongoDB
Memory Mapped Files
Page Cache
Storage
Page Cache
Process
User Space Kernel Space
Process
read(fd, *buffer, count)
Page CacheSystem
call
Page Cache – Read
Example File
Page 1
Page 2
Page 3
File descriptorAt 2,000
End at 10,000
Page in
cache?
offset+count pages
Read from disk and store in cache
Read from cache and copy to
*buffer
No
Yes
Disk
Page Cache
Process
write(fd, *buffer, count)
Systemcall
Page Cache – Write
Update pageAnd mark as
dirty
After X secondsflush to disk
Page Reclamation
LRU – Least Recently Used
$ free -g total used free cachedMem: 64 61 3 55-/+ buffers/cache: 5 58Swap: 16 0 16
Free
MongoDB
Memory Mapped Files
Page Cache
Storage
Memory Mapped Files
Process File
2000
1000
4000
5000
mmapProcess
B File
Process A
MongoDB
Memory Mapped Files
Page Cache
Storage
MongoDBMaps everything: documents, indexes, journal
Running top:
ChallengesNo control over what is saved in memory
Warm-up
Expensive queries
Mitigation PlanProtect MongoDB with an API
Enforce index usage
Pass a query timeout (from 2.6)
Example of a simple API
def find_samples(start_time, end_time):return samples.find({‘time’: {‘$gte’:
start_time, ‘$lt’: end_time}})
ChallengesLack of Inter-process prioritization
Mitigation: isolate mongo
Estimate required memory
How big is the working set?
Working SetContains:
Documents
Indexes
Padding (!)
Doc 1 Doc 2 Doc 3
0 4k
Padding
Working Set AnalysisPlanning
Monitoring
Planning
db.samples.stats()
dataSize
indexSizes
ColdWarmHot
Month
Last 2 weeks
1 week
1 week
MonitoringOnline
top, iostat
db.currentOp(), mongostat, mongomem
Offline
Profiling collection
MMS/Graphite
MongomemTop collections:
local.oplog.rs 11218 / 49865 MB (22.496883%) [25 extents]
samples.quarter 3661 / 219714 MB (1.666450%) [128 extents]
samples.hour 1629 / 10921 MB (14.924107%) [26 extents]
Total resident pages: 16508 / 280500 MB (5.885%)
MongomemProcedure:
Stop the database
Clear the page cache:
echo 1 > /proc/sys/vm/drop_caches
Start the database
Run queries that should return fast
Run mongomem!
What to monitor?Thrashing
Page faults
Disk utilization
Symptoms
Queued queries
High locking ratios
iostat
$ iostat –xm 1 /dev/sdaDevice: r/s w/s rMB/s wMB/s %utilsda 570.00 0.00 31.28 0.00 100.00
mongostatUses db.serverStatus()
Metrics per second:
Page faults
Queued reads (qr)
Offline monitoringMMS/Graphite
Mandatory!
OptimizationSmaller = faster!
Less memory
Higher disk throughput
Schema
Shorten keys
firstName -> first -> f
Size vs. count
Optimizing indicesUnused indices
Sparse
Indices should fit in memory
A
Index on name:
Older Newer
Index on creation_time:
Z
SummaryHow it works
Challenges
Monitor
Optimize