MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation
-
Upload
ronwarshawsky -
Category
Technology
-
view
5.977 -
download
0
description
Transcript of MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation
![Page 1: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/1.jpg)
Enteros, Inc.
MongoDB Performance Tuning
Load Testing
NoSQL Now! 2013 Conference San Jose
[email protected] Ron Warshawsky
August 23, 2013
![Page 2: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/2.jpg)
2
MongoDB Overview
Agenda
Overview performance tuning
Overview load testing
![Page 3: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/3.jpg)
3
MongoDB Areas Of Performance Tuning
OS tuning Storage tuning
Database tuning Load Testing
![Page 4: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/4.jpg)
4
MongoDB Areas Of Performance Tuning
• OS Tuning • Follow ulimits per mongo manual
• http://docs.mongodb.org/manual/reference/ulimit/
• Follow production notes per mongo manual • http://docs.mongodb.org/manual/administration/production-notes/ • No atime on fs • No hugepages • Use NTT to sync time between the nodes
• Try “tuned”
• Best when used in staging/load testing environment under
realistic load
• CPU, Network and Storage tuning
![Page 5: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/5.jpg)
5
MongoDB Areas Of Performance Tuning
• Storage Tuning • RAID 10
• Ext4 or XFS
• Delay sharding with better I/O
• SSD + FlashCache – shutterfly tested
• https://github.com/facebook/flashcache/
• Provisioned IOPS on AWS
![Page 6: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/6.jpg)
6
MongoDB Areas Of Performance Tuning
• Storage Tuning • Ext4 + external journal on SSD – good I/O results
• Make sure to test • journal_async_commit is used • http://www.raid6.com.au/posts/fs_ext4_external_journal/
![Page 7: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/7.jpg)
7
MongoDB Areas Of Performance Tuning
Softlayer study/ benchmarking “MongoDB Performance Analysis: Bare Metal v. Virtual” http://blog.softlayer.com/2012/mongodb-performance-analysis-bare-metal-v-
virtual/
Highlights:
• When a working data set is smaller than available memory, query performance increases.
• The number of clients performing queries has an impact on query performance because more data is being actively cached at a rapid rate.
• The addition of a separate Journal Mount volume significantly improves performance. Because the Small (SM) engineered server does not include a secondary mount for Journals, whenever MongoDB began to journal, the disk I/O associated with journaling was disruptive to the query and update operations performed on the Data Mount.
• The best deployments in terms of operations per second, stability and control were the configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These configurations are available in both Medium and Large offerings, and I’d highly recommend them.
![Page 8: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/8.jpg)
8
MongoDB Areas Of Performance Tuning
OS Performance monitoring and analysis tools:
• A) Linux utilities: iostat, vmstat, mpstat, sar, free -tm
• B) Open source monitoring / performance analysis systems: Nagios, Cacti
• http://tag1consulting.com/blog/mongodb-cacti-graphs • https://github.com/mzupan/nagios-plugin-mongodb C) Oracle OS Watcher – Free tool, performance collection and chart
generator, easy to operate, very reliable, covers CPU, processes, top processes, I/O, network, etc
OSWatcher Black Box Analyzer User Guide (Doc ID 461053.1)
![Page 9: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/9.jpg)
9
MongoDB Areas Of Performance Tuning
Database Tuning
• Database configuration • Preallocate space (noprealloc = false)
• journalCommitInterval – range of 2 to 300, default 100. Can be
adjusted to for better performance
• Option to disable services: 1. HTTP Interface using nohttpinterface = true. 2. Scripting Engine using noscripting = true (server-side java
script). 3. REST service using rest = false.
![Page 10: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/10.jpg)
10
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools • Mongostat – based on “eBay” “freecon” utility.
• Use “–discover” option for members of replicaSet / shared
cluster
• Displays critical performance metrics: • Flushes/Faults/locked db/idx miss/qr/qw
• Collect 24x7 and save for analysis
![Page 11: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/11.jpg)
11
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools • Mongotop
• Per collection I/O time tracking
• Per database lock tracking (--lock option)
![Page 12: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/12.jpg)
12
MongoDB Areas Of Performance Tuning
Database network traffic analysis
• Database network monitoring tools • Mongosniff – sniffing MongoDb requests
• Can capture invalid BSON requests from a network (--objcheck option)
• Wireshark – support for MongoDB protocol (not BSON), great options for network performance and latency analysis
![Page 13: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/13.jpg)
13
MongoDB Areas Of Performance Tuning
Database Tuning
• Collections / Fragmentation
Collections in a MongoDB database can become fragmented. This can be a particularly serious problem if data usage patterns are relatively unstructured. In the long run, this will result in databases taking up more space on disk and in RAM to hold the same amount of data, it will make many database operations noticeably slower, and it can reduce overall query capacity significantly.
compact collection: db.runCommand ( { compact: '<collection>', paddingBytes: 100 } ) Automate compaction: http://blog.parse.com/2013/03/26/always-be-compacting/ ./mongo_compact.rb -d userdata1,userdata2,userdata3
![Page 14: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/14.jpg)
14
MongoDB Areas Of Performance Tuning
Database Tuning
Profiler
db.setProfilingLevel(level, slowms) db.setProfilingLevel(1) – collects slow requests Output: system.profile.op – operation system.profile.ns – namespace system.profile.ntoreturn – documents returned system.profile.nscanned – documents scanned … http://docs.mongodb.org/manual/reference/database-profiler/ Profiler deep dive https://speakerdeck.com/kgorman/mongodb-profiler-deep-dive
![Page 15: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/15.jpg)
15
MongoDB Areas Of Performance Tuning
Database Tuning
• Indexes
• Explain db.collection.find({…}).explain() Watch for scans
• Indexes + Profiler = DEX - suggest what indexesare needed
https://github.com/jwilder/mongodb-tools http://architects.dzone.com/articles/mongodb-performance-tuning-
dex
![Page 16: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/16.jpg)
16
MongoDB Areas Of Performance Tuning
Database Monitoring and performance analysis UI
a) MongoDB monitoring service (MMS)
b) Open Source monitoring software http://docs.mongodb.org/manual/administration/monitoring/
c) Enteros High Load Capture for MongoDB
![Page 17: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/17.jpg)
17
MongoDB Areas Of Performance Tuning
• Replication / Replica set tuning Watch for replication lag Problem: a) Weak secondary b) Write bursts c) Index build d) Secondary locked for backup e) Secondary offline Measure lag: db.printSlaveReplicationInfo() db.printReplicationInfo() Get time difference MMS provides replication lag chart and alerts
![Page 18: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/18.jpg)
18
MongoDB Areas Of Performance Tuning
• Sharding Tuning • Shard size
• Migration of chunks
• Failed migration of chunks
![Page 19: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/19.jpg)
19
MongoDB Load Testing
Load testing
1) Why load testing?
• Validate upgrades
• Validate fixes
• Validate hardware and platform changes (to / from cloud, etc)
• Validate multiples of production load
![Page 20: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/20.jpg)
20
MongoDB Load Testing
2) Options for Load testing
A) Benchmark Load Testing • Mongo-Perf - Create standard TPCC-like load on MongoDB • https://github.com/mongodb/mongo-perf
B) Disk I/O load testing
• Mongoperf
• Iozone http://www.iozone.org/
![Page 21: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/21.jpg)
21
MongoDB Load Testing
C) Real traffic Load Testing
Parse PCAP files and generate production-like load http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-
protocol/
Create custom tool
Enteros Load2Test (pre-GA)
![Page 22: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/22.jpg)
Understand business requirements and it’s impact on production system design
Design for performance, scalability and availability
Implement monitoring and performance analysis across layers
Implement reliable production-like load testing
MongoDB Summary
![Page 23: MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation](https://reader034.fdocuments.in/reader034/viewer/2022052301/554f7d42b4c905d25b8b489e/html5/thumbnails/23.jpg)
Enteros, Inc http://www.enteros.com/
Enteros is an innovative software company specializing in Performance Management and Load Testing Software for Production Databases - RDBMS and NOSQL/Big Data Enteros solutions enable IT professionals to identify and remediate performance problems in business-critical databases with unprecedented speed, accuracy and scope. Ron Warshawsky; [email protected] 408-207-8408
MongoDB Summary