Download - Webinar: Capacity Planning

Server Engineer

Shaun Verch

Capacity Planning:

Deploying MongoDB

Capacity Planning

• Why is it important?

• What is it?

• When is it important?

• How is it actually done?

• What are the consequences of not planning?

Why does it matter?

What is Capacity Planning?

Requirements

Resources

• Availability• Throughput• Responsiveness

Requirements

• Availability• Throughput• Responsivenes

s

Requirements to Hardware

Resource Usage

• Storage

– IOPS

– Size

– Data & Loading Patterns

• Memory

– Working Set

• CPU

– Speed

– Cores

• Network

– Latency

– Throughput

Storage

• Active

• Archival

• Loading Patterns

• Integration (BI/DW)

Storage

• Active

• Archival

• Loading Patterns

• Integration (BI/DW)

Example IOPS

Example IOPS

7,200 rpm SATA ~ 75-100 IOPS

15,000 rpm SAS ~ 175-210 IOPS

Amazon

EBS/Provisioned

~ 100 IOPS "up to" 2,000

IOPS

Amazon SSD 9,000 – 120,000 IOPS

Storage Capability

Intel X25-E (SLC) ~ 5,000 IOPS

Fusion IO ~ 135,000 IOPS

Violin Memory 6000 ~ 1,000,000 IOPS

Example IOPS

7,200 rpm SATA ~ 75-100 IOPS

15,000 rpm SAS ~ 175-210 IOPS

Amazon

EBS/Provisioned

~ 100 IOPS "up to" 2,000

IOPS


Storage Capability

Intel X25-E (SLC) ~ 5,000 IOPS

Fusion IO ~ 135,000 IOPS

Violin Memory 6000 ~ 1,000,000 IOPS

Cost of IOPS

7,200 rpm SATA ~ 75-100 IOPS

15,000 rpm SAS ~ 175-210 IOPS

Amazon

EBS/Provisioned

~ 100 IOPS "up to" 2,000

IOPS


Storage Costs

Memory

• Working Set– Active Data in Memory

– Measured Over Periods

Memory

• Work:

–Sorting

–Aggregation

–Connections

SORTS

Connections

Aggregations

Memory & Storage

><?

Working Set

Number of distinct pages

accessed per unit of time

Working Set

Number of distinct pages

accessed per second

Working Set

4 distinct pages per second

Working Set


Worst case 4 disk accesses

Working Set


Working Set


Worst case disk access on every op

Memory & Storage

MOPs

PFs

CPU

• Non-indexed Data

• Sorting

• Aggregation

– Map/Reduce

– Framework

• Data

– Fields

– Nesting

– Arrays/Embedded-Docs

Network

• Latency

– WriteConcern

– ReadPreference

– Batching

• Throughput

– Update/Write Patterns

– Reads/Queries

What is failure?

• We have failed at Capacity Planning when our

resources don’t meet our requirements

• Because our requirements can have many

dimensions, we may exceed our requirements in

one characteristic but not meet them in another

• This means that we can spend many $$$ and still

fail!

What about Legacy Hardware?

• Let’s hope whatever worked for this legacy

technology also works for MongoDB

• Same principles of Capacity Planning still apply

• Before it's too late!

• When?

Capacity Planning: When

Start Launch Version 2

Capacity Planning is Measurement

Measuring early gives you a comparison point for when you need to do it again

Velocity of Change

• Limitations -> takes time

– Data Movement

– Allocation/Provisioning (servers/mem/disk)

• Improvement

– Limit Size of Change (if you can)

– Increase Frequency

– MEASURE its effect

– Practice

Repeat (continuously)

• Repeat Testing

• Repeat Evaluations

• Repeat Deployment

Monitoring

Monitoring Storage

Memory

CPU

Network

Application Metrics

Tools

• MMS (MongoDB Monitoring Service)

• MongoDB: mongotop, mongostat

• Linux: iostat, vmstat, sar, etc

• Windows: Perfmon

Measure realistic loads (generated by Load testing)

Models

• Load/Users

– Response Time/TTFB

• System Performance

– Peak Usage

– Min Usage

Starter Questions

• What is the working set?

– How does that equate to memory

– How much disk access will that require

• How efficient are the queries?

• What is the rate of data change?

• How big are the highs and lows?

Deployment Types

All of these use the same resources:

• Single Instance

• Multiple Instances (Replica Set)

• Cluster (Sharding)

• Data Centers

Questions?

Server Engineer, MongoDB

Shaun Verch

Thank You