High Performance, Scalable MongoDB

in a Bare Metal Cloud

Harold Hannon, Sr. Software Architect

100k servers

24k customers

23million domains

Reach 13 data centers16 network POPs20Gb fiber interconnects

The 3 V’s

VelocityVariety

Volume

Physical Deployment

Public Cloud

The Dream

The Reality

Deployment Complexity

Bare Metal Cloud

Bare Metal• Build to your specs• Robust, quickly scaled environment• Management of all aspects of environment• Great for Big Data Solutions like MongoDB

Cloud Subscription• Preconfigured• Performance Tuned• Bare Metal Single Tenant• Complex Environment Configurations

Pre-configurations• Set SSD Read Ahead Defaults to 16 Blocks – SSD drives have

excellent seek times allowing for shrinking the Read Ahead to 16 blocks. Spinning disks might require slight buffering so these have been set to 32 blocks.

• noatime – Adding the noatime option eliminates the need for the system to make writes to the file system for files which are simply being read — or in other words: Faster file access and less disk wear.

• Turn NUMA Off in BIOS – Linux, NUMA and MongoDB tend not to work well together. If you are running MongoDB on NUMA hardware, we recommend turning it off (running with an interleave memory policy). If you don’t, problems will manifest in strange ways like massive slow downs for periods of time or high system CPU time.

• Set ulimit – We have set the ulimit to 64000 for open files and 32000 for user processes to prevent failures due to a loss of available file handles or user processes.

Use ext4 – We have selected ext4 over ext3. We found ext3 to be very slow in allocating files (or removing them). Additionally, access within large files is poor with ext3.

Deployment SerenityShard 1

mongosmongodconfig

mongosmongod

mongosmongod

Shard 2

mongosmongodconfig

mongosmongod

mongosmongod

Shard 3

mongosmongodconfig

mongosmongod

mongosmongod

The Proof is in the Pudding

Tests Performed• Small MongoDB Cloud Subscription vs Shared

Virtual Instance• Medium MongoDB Cloud Subscription vs

Shared Virtual Instance• SSD and 15K SAS

• Large MongoDB Cloud Subscription vs Shared Virtual Instance• SSD and 15K SAS

Small TestSmall (SM) Cloud Subscription MongoDB ServerSingle 4-core Intel 1270 CPU64-bit CentOS8GB RAM2 x 500GB SATAII – RAID11Gb Network

Virtual Provider Instance4 Virtual Compute Units64-bit CentOS7.5GB RAM2 x 500GB Network Storage – RAID11Gb Network

Small Test

Tests PerformedSmall Data Set (8GB of .5mb documents)200 iterations of 6:1 query-to-update operationsConcurrent client connections exponentially increased from 1 to 32Test duration spanned 48 hours

Small TestAverage Read Operations per Second by Concurrent Client

Small TestPeak Read Operations per Second by Concurrent Client

Small TestAverage Write Operations per Second by Concurrent Client

Small TestPeak Write Operations per Second by Concurrent Client

Medium TestMedium (MD) Cloud Subscription MongoDB ServerDual 6-core Intel 5670 CPUs64-bit CentOS36GB RAM2 x 64GB SSD – RAID1 (Journal Mount)4 x 300GB 15K SAS – RAID10 (Data Mount)1Gb Network – Bonded

Virtual Provider Instance26 Virtual Compute Units64-bit CentOS30GB RAM2 x 64GB Network Storage – RAID1 (Journal Mount)4 x 300GB Network Storage – RAID10 (Data Mount)1Gb Network

Medium Test

Tests PerformedSmall Data Set (32GB of .5mb documents)200 iterations of 6:1 query-to-update operationsConcurrent client connections exponentially increased from 1 to 128Test duration spanned 48 hours

Medium Test 15k SASAverage Read Operations per Second by Concurrent Client

Medium Test 15k SASPeak Read Operations per Second by Concurrent Client

Medium Test 15k SASAverage Write Operations per Second by Concurrent Client

Medium Test 15k SASPeak Write Operations per Second by Concurrent Client

Medium Test SSDAverage Read Operations per Second by Concurrent Client

Medium Test SSDPeak Read Operations per Second by Concurrent Client

Medium Test SSDAverage Write Operations per Second by Concurrent Client

Medium Test SSDPeak Write Operations per Second by Concurrent Client

Large TestLarge (LG) Cloud Subscription MongoDB ServerDual 8-core Intel E5-2620 CPUs64-bit CentOS128GB RAM2 x 64GB SSD – RAID1 (Journal Mount)6 x 600GB 15K SAS – RAID10 (Data Mount)1Gb Network – Bonded

Virtual Provider Instance26 Virtual Compute Units64-bit CentOS64GB RAM (Maximum available on this provider)2 x 64GB Network Storage – RAID1 (Journal Mount)6 x 600GB Network Storage – RAID10 (Data Mount)1Gb Network

Large Test

Tests PerformedSmall Data Set (64GB of .5mb documents)200 iterations of 6:1 query-to-update operationsConcurrent client connections exponentially increased from 1 to 128Test duration spanned 48 hours

Large Test 15k SASAverage Read Operations per Second by Concurrent Client

Large Test 15k SASPeak Read Operations per Second by Concurrent Client

Large Test 15k SASAverage Write Operations per Second by Concurrent Client

Large Test 15k SASPeak Write Operations per Second by Concurrent Client

Large Test SSDAverage Read Operations per Second by Concurrent Client

Large Test SSDPeak Read Operations per Second by Concurrent Client

Large Test SSDAverage Write Operations per Second by Concurrent Client

Large Test SSDPeak Write Operations per Second by Concurrent Client

RSD

Cloud Provider MongoDB Cloud Subscription

Small 6-36% 1-9%

Medium 8-43% 1-8%

Large 8-93% 1-9%

Avoid Storage I/O Altogether

If you must VirtualizeGo with Local Disks

Journal and Data on Separate Mounts

Watch Sync Delay

More information:

www.softlayer.com