Scaling MongoDB for real time analytics
-
Upload
david-tollmyr -
Category
Technology
-
view
1.319 -
download
0
description
Transcript of Scaling MongoDB for real time analytics
SHORTCUTSAROUND THE
MISTAKES WE’VEMADE SCALING
MONGODB
David Tollmyr, Platform lead
@effataslideshare.net/tollmyr
What we doWe want to make digital advertising an amazing user experience.There is more to metrics that clicks.
Ads
Data
Assembling sessionsexposure
pingping
ping ping
ping
event
event
ping
session➔ ➔
Information
Crunching
session
session
session
session
sessionsession
session session
session
session
session
session
session
➔ ➔ 42
Metrics
Reports
What we doTrack ads, make pretty reports.
That doesn’t sound so hardWe don’t know when sessions endThere’s a lot of dataIt’s all done in (close to) real time
Numbers200 Gb logs100 million data pointsper day~300 metrics per data point= 6000 updates / s at peak
How we use(d) MongoDB“Virtual memory” to offload data while we wait for sessions to finishShort time storage (<48 hours) for batch jobs, replays and manual analysisMetrics storage
Why we use MongoDBSchemalessness makes things so much easier, the data we collect changes as we come up with new ideasSharding makes it possible to scale writesSecondary indexes and rich query language are great features (for the metrics store)It’s just… nice
Btw.We use JRuby, it’s awesome
STANDING ON THE SHOULDERS
OF GIANTS WITH JRUBY
slideshare.net/iconara
A story in 9 iterations
secondary indexes and updates1st iteration
One document per session, update as new data comes alongOutcome: 1000% write lock
#1Everything is aboutworking around the
GLOBALWRITELOCK
MongoDB 1.8.1
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)
MongoDB 2.0.0
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)
using scans for two step assembling2nd iteration
Instead of updating, save each fragment, then scan over _id to assemble sessions
using scans for two step assembling2nd iteration
Outcome: not as much lock, but still not great performance. We also realised we couldn’t remove data fast enough
#2Everything is aboutworking around the
GLOBALWRITELOCK
#3Give a lot of
thought to your
PRIMARYKEY
partitioning3rd iteration
Partitioning the data by writing to a new collection every hourOutcome: complicated, fragmented database
#4Make sure you can
REMOVE OLD DATA
sharding4th iteration
To get around the global write lock and get higher write performance we moved to a sharded cluster.Outcome: higher write performance, lots of problems, lots of ops time spent debugging
#5Everything is aboutworking around the
GLOBALWRITELOCK
#6SHARDINGIS NOT A
SILVER BULLETand it’s complex, if you can, avoid it
#7IT WILL FAIL
design for it
moving things to separate clusters5th iteration
We saw very different loads on the shards and realised we had databases with very different usage patterns, some that made autosharding not work. We moved these off the cluster.Outcome: a more balanced and stable cluster
#8Everything is aboutworking around the
GLOBALWRITELOCK
#9ONE DATABASE
with one usage pattern
PER CLUSTER
#10MONITOR
EVERYTHINGlook at your health
graphs daily
monster machines6th iteration
We got new problems removing data and needed some room to breathe and think Solution: upgraded the servers to High-Memory Quadruple Extra Large (with cheese).
♥I
#11Don’t try to scale up
SCALE OUT
#12When you’re out of ideas
CALL THE EXPERTS
partitioning (again) and pre-chunking7th iteration
We rewrote the database layer to write to a new database each day, and we created all chunks in advance. We also decreased the size of our documents by a lot.Outcome: no more problems removing data.
#13Smaller objects means a smaller database, and a smaller database means
LESS RAM NEEDED
#14Give a lot of
thought to your
PRIMARYKEY
#15Everything is aboutworking around the
GLOBALWRITELOCK
realize when you have the wrong tool8th iteration
Transient data might not need all the bells and whistles.
Outcome: Redis gave us 100x performance in the assembling step
#16When all you have is a
HAMMEReverything looks like a
NAIL
rinse and repeat9th iteration
We now have the same scaling issues later in the chain.
Outcome: Upcoming rewrite to make writes/updated more effectiveRedis was actually slower
#17Everything is aboutworking around the
GLOBALWRITELOCK
Thank you
@effataslideshare.net/tollmyr
engineering.burtcorp.comburtcorp.com
richmetrics.com
Since we got time…
EC2Tips
You have three copies of your data, do you really need EBS?Instance store disks are included in the price and they have predictable performance.m1.xlarge comes with 1.7 TB of storage.
Avoid bulk insertsTips
Very dangerous if there’s a possibility of duplicate key errors
It’s not fixed in 2.0 even though the driver has a flag for it.
Safe modeTips
Run every Nth insert in safe modeThis will give you warnings when bad things happen; like failovers