1. #MongoDBDaysMythbusting: UnderstandingHow We Measure thePerformance of MongoDBAlvin RichardsSenior Director of Performance Engineering, MongoDB

2. Before we start We are going to look a lot at C++ kernel code Java benchmarks JavaScript tests And lots of charts And its going to be awesome! 3. Measuring "Performance"https://www.youtube.com/watch?v=7wm-pZp_mi0 4. Benchmarking Some common traps Performance measurement & diagnosis What's next 5. Part OneSome Common Traps 6. "We all live in a house on fire, no fire departmentto call; no way out, just the upstairs window tolook out of while the fire burns the house downwith us trapped, locked in it."The Milk Train Doesn't Stop Here AnymoreTennessee Williams 7. #1 Time taken to Insert xDocumentslong startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 8. #1 Time taken to Insert xDocumentslong startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 9. #1 Time taken to Insert xDocumentslong startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 10. #1 Time taken to Insert xDocumentslong startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 11. #1 Time taken to Insert xDocumentslong startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 12. So that looks ok, right?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis(); 13. What are else you measuring?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis();Object creation and GCmanagement? 14. What are else you measuring?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis();Object creation and GCmanagement?Thread contention onnextInt()? 15. What are else you measuring?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis();Object creation and GCmanagement?Thread contention onnextInt()?Time to synthesize data? 16. What are else you measuring?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis();Object creation and GCmanagement?Thread contention onnextInt()?Time to synthesize data?Thread contention onaddAndGet()? 17. What are else you measuring?long startTime = System.currentTimeMillis();for (int roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;BasicDBObject doc = new BasicDBObject();doc.put("_id",id);doc.put("k",rand.nextInt(numMaxInserts)+1);String cVal = ""doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i]=doc;}coll.insert(aDocs);numInserts += documentsPerInsert;globalInserts.addAndGet(documentsPerInsert);}long endTime = System.currentTimeMillis();Object creation and GCmanagement?Thread contention onnextInt()?Time to synthesize data?Thread contention onaddAndGet()?Clock resolution? 18. Solution: Pre-Create the objects// Pre Create the Object outside the LoopBasicDBObject[] aDocs = new BasicDBObject[documentsPerInsert];for (int i=0; i < documentsPerInsert; i++) {BasicDBObject doc = new BasicDBObject();String cVal = "";doc.put("c",cVal);String padVal = "";doc.put("pad",padVal);aDocs[i] = doc;}Pre-create non varyingdata outside the timingloopAlternative Pre-create the data in a file; load from file 19. Solution: Remove contention// Use ThreadLocalRandom generator or an instance of java.util.Random per threadjava.util.concurrent.ThreadLocalRandom rand;for (long roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;doc = aDocs[i];doc.put("_id",id);doc.put("k", nextInt(rand, numMaxInserts)+1);}coll.insert(aDocs);numInserts += documentsPerInsert;}// Maintain count outside the loopglobalInserts.addAndGet(documentsPerInsert * roundNum);Remove contentionnextInt() by makingThread local 20. Solution: Remove contention// Use ThreadLocalRandom generator or an instance of java.util.Random per threadjava.util.concurrent.ThreadLocalRandom rand;Remove contentionnextInt() by makingThread localfor (long roundNum = 0; roundNum < numRounds; roundNum++) {for (int i = 0; i < documentsPerInsert; i++) {id++;doc = aDocs[i];doc.put("_id",id);doc.put("k", nextInt(rand, numMaxInserts)+1);}coll.insert(aDocs);numInserts += documentsPerInsert;}// Maintain count outside the loopglobalInserts.addAndGet(documentsPerInsert * roundNum);Remove contention onaddAndGet() 21. Solution: Timer resolutionlong startTime = System.currentTimeMillis();long endTime = System.currentTimeMillis();long startTime = System.nanoTime();long endTime = System.nanoTime() - startTime;"granularity of the valuedepends on theunderlying operatingsystem and may belarger""resolution is at least asgood as that ofcurrentTimeMillis()"Source http://docs.oracle.com/javase/7/docs/api/java/lang/System.html 22. General Principal #1Know what you aremeasuring 23. #2 Response time to return allresultsBasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis(); 24. #2 Response time to return allresultsBasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis(); 25. #2 Response time to return allresultsBasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis(); 26. #2 Response time to return allresultsBasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis(); 27. So that looks ok, right?BasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis(); 28. What are else you measuring?BasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Each doc is is 4080 byteson disk with powerOf2Sizes 29. What are else you measuring?BasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Each doc is is 4080 byteson disk with powerOf2SizesUnrestricted predicate? 30. What are else you measuring?BasicDBObject doc = new BasicDBObject();doc.put("v", str); // str is a 2k stringfor (int i=0; i < 1000; i++) {doc.put("_id",i); coll.insert(doc);}BasicDBObject predicate = new BasicDBObject();long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate);DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Each doc is is 4080 byteson disk with powerOf2SizesUnrestricted predicate?Measuring Time to parse &execute query Time to retrieve alldocumentBut also Cost of shipping ~4MBdata through networkstack 31. Solution: Limit the projectionBasicDBObject predicate = new BasicDBObject();predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20));BasicDBObject projection = new BasicDBObject();projection.put("_id", 1);long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate, projection );DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Return fixed range 32. Solution: Limit the projectionBasicDBObject predicate = new BasicDBObject();predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20));BasicDBObject projection = new BasicDBObject();projection.put("_id", 1);long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate, projection );DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Return fixed rangeOnly project _id 33. Solution: Limit the projectionBasicDBObject predicate = new BasicDBObject();predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20));BasicDBObject projection = new BasicDBObject();projection.put("_id", 1);long startTime = System.currentTimeMillis();DBCursor cur = coll.find(predicate, projection );DBObject foundObj;while (cur.hasNext()) {foundObj = cur.next();}long endTime = System.currentTimeMillis();Return fixed rangeOnly project _idOnly 46k transferredthrough network stack 34. General Principal #2Measure only what youneed to measure 35. Part TwoPerformancemeasurement &diagnosis 36. "Every experiment destroys some of theknowledge of the system which was obtained byprevious experiments."The Physical Principles of the Quantum Theory (1930)Werner Heisenberg 37. Broad categories Micro Benchmarks Workloads 38. Micro benchmarks: mongo-perf 39. mongo-perf: goals Measure commands Configure Single mongod, ReplSet size (1 -> n), Sharding Single vs. Multiple DB O/S Characterize Throughput by thread count Compare 40. What do you get?Better 41. What do you get?Measuredimprovementbetween rc0 andrc2Better 42. Benchmark source codetests.push( { name: "Commands.CountsIntIDRange",pre: function( collection ) {collection.drop();for ( var i = 0; i < 1000; i++ ) {collection.insert( { _id : i } );}collection.getDB().getLastError();},ops: [{ op: "command",ns : "testdb",command : { count : "mycollection",query : { _id : { "$gt" : 10, "$lt" : 100 } } } }] } ); 43. Benchmark source codetests.push( { name: "Commands.CountsIntIDRange",pre: function( collection ) {collection.drop();for ( var i = 0; i < 1000; i++ ) {collection.insert( { _id : i } );}collection.getDB().getLastError();},ops: [{ op: "command",ns : "testdb",command : { count : "mycollection",query : { _id : { "$gt" : 10, "$lt" : 100 } } } }] } ); 44. Benchmark source codetests.push( { name: "Commands.CountsIntIDRange",pre: function( collection ) {collection.drop();for ( var i = 0; i < 1000; i++ ) {collection.insert( { _id : i } );}collection.getDB().getLastError();},ops: [{ op: "command",ns : "testdb",command : { count : "mycollection",query : { _id : { "$gt" : 10, "$lt" : 100 } } } }] } ); 45. Benchmark source codetests.push( { name: "Commands.CountsIntIDRange",pre: function( collection ) {collection.drop();for ( var i = 0; i < 1000; i++ ) {collection.insert( { _id : i } );}collection.getDB().getLastError();},ops: [{ op: "command",ns : "testdb",command : { count : "mycollection",query : { _id : { "$gt" : 10, "$lt" : 100 } } } }] } ); 46. Code Change 47. Workloads "public" workloads YCSB Sysbench "real world" simulations Inbox fan in/out Message Stores Content Management 48. Example: Bulk Load Performance16m DocumentsBetter55% degradation2.6.0-rc1 vs 2.4.10 49. Ouch where's the tree in thewoods? 2.4.10 -> 2.6.0 4495 git commits 50. git-bisect Bisect between good/bad hashes git-bisect nominates a new githash Build against githash Re-run test Confirm if this githash is good/bad Rinse and repeat 51. Code Change - Bad Githash 52. Code Change - Fix 53. Bulk Load Performance - FixBetter11% improvement2.6.1 vs 2.4.10 54. The problem with measurement Observability What can you observe on the system? Effect What effects does the observation cause? 55. mtools 56. mtools MongoDB log file analysis Filter logs for operations, events Response time, lock durations Plot https://github.com/rueckstiess/mtools 57. Response Times > 100msBulk Insert 2.6.0-rc0Ops/SecTime 58. Response Times > 100msBulk Insert 2.6.0-rc0 vs. 2.6.0-rc2Floor raised 59. Code Change Yielding Policy 60. Code Change 61. Response TimesBulk Insert 2.6.0 vs 2.6.1Ceiling similar, lower floorresulting in 40%improvement in throughput 62. Secondary effects of Yield policy changeWrite lock time reducedOrder of magnitude reductionof write lock duration 63. Unexpected side effects ofmeasurement?> db.serverStatus()Yes will cause a read lock to be acquired> db.serverStatus({recordStats:0})No lock is not acquired> mongostatYes - until SERVER-14008 resolved, uses db.serverStatus() 64. CPU sampling Get an impression of Call Graphs CPU time spent on node and called nodes 65. Setup & building with google-profiler> sudo apt-get install google-perftools> sudo apt-get install libunwind7-dev> scons --use-cpu-profiler mongod 66. Start the profiling> mongodb dbpath Note: Do not use fork> mongo> use admin> db.runCommand({_cpuProfilerStart: {profileFilename: 'foo.prof'}})Execute some commands that you want to profile> db.runCommand({_cpuProfilerStop: 1}) 67. Sample start vs. end of workload 68. Sample start vs. end of workload 69. Code change 70. Public Benchmarks Not all forks arethe same YCSB https://github.com/achille/YCSB sysbench-mongodb https://github.com/mdcallag/sysbench-mongodb 71. Part ThreeAnd next? 72. "The future sucks. Change it.""I'm way cool Beavis, but I cannot change thefuture."Beavis & Butthead 73. What we are working on mongo-perf UI refactor Adding more micro benchmarks (geo, sharding) Workloads Adding external benchmarks Creating benchmarks for common use cases Inbox fan in/out Analytical dashboards Stream / Feeds Customers, Partners & Community 74. Here's how you can help change thefuture! Got a great workload? Great benchmark? Want to donate it? alvin@mongodb.com 75. Don't be that benchmark#1 Know what you are measuring#2 Measure only what you need tomeasure 76. #MongoDBDaysThank YouAlvin Richardsalvin@mongodb.com / @jonnyeightSenior Director of Performance Engineering, MongoDB