The importance of indexes in mongo db
14
The Importance of Indexes in MongoDB How we increased the loading speed Profile Gliphs and Insights James Toyer, Lead Software Engineer at Glipho
description
Transcript of The importance of indexes in mongo db
The Importance of Indexes in MongoDB
How we increased the loading speed Profile Gliphs and Insights
James Toyer, Lead Software Engineer at Glipho
What is Glipho?
Social network for text based content
Aims to better engage writers and readers
Original content only
Not an aggregator
Automatically share to Facebook, LinkedIn and Twitter
Insights Page
Load up the gliphs a writer has
Iterate through, and sum, actions for each gliph
Load and sum actions for the writers profile
This can be over 100 calls to the database We know it’s inefficient
but it does the job for now
Insights Document Structure
Timestamp – when the action took place
EntityId – identifier of the original entity
ActionType – the type of the entity (probably should be entity type)
Action – the actual action that took place
Helpful Error Page…it’s all gone wrong!
Troubleshooting
CPU spiking? NO
Memory high? NO
Disk IO high? NO
Are there any actual regular hits happening? NO
Do you know anything? NO
Crack out the code performance tools…
Pre-Index performance
• 3 passes on each filter page
• Average time for each page to load = 3.9 seconds
• “ListAll” method calls the database
• “ListAll” is iterated over for each gliph in the database and the profile (in this case ~10 times)
• Average time in “ListAll” 256ms
More Troubleshooting
Is the code doing obviously stupid things? NO
Has Linq screwed you over again? NO
Do you trust the driver? PROBABLY
Check the database ~ 400,000 documents (now ~690,000)
No indexes
Know your query
GetMongoQuery code Output
Index analysis
Without action field Query structure
Query time before index: 334ms
Index
Query time after index: >1ms
With action field Query structure
Query time before index: 409ms
Index
Query time after index: >1ms
Post-Index performance
Pre-index performance Page load time:
3.9s
“ListAll” method execution time: 256ms
Post-index performance Page load time:
72ms
“ListAll” method execution time: >0.2ms
Page load time deceased by 98%
“ListAll” method execution time decreased by 99%
Gliph listings for Writers
Problems: Slow loading
Sometimes erroring out
Reasons: Indexes were no longer
accurate
Code had changed
Solution: New indexes
Remove old indexes
What did I learn?
Know exactly what queries are being run
Don’t do a “best guess” on an index. Test them out
Don’t “forget” to add indexes
Ensure your indexes evolve as your queries do
