Measuring Storage Performance in the Cloud - USENIX · Measuring Storage Performance in the Cloud...

27th Large Installation System Administration Conference

November 3–8, 2013 • Washington, D.C.

Measuring Storage Performance in the Cloud

Jeff Darcy

GlusterFS & Red Hat



Overview

Performance

Cloud

Storage

Cloud Storage Performance 2




Performance



Types of Performance

Bandwidth Throughput Latency Latency Variation

Network Gb/second PPS Milliseconds (average)

99th percentile

Storage GB/second IOPS Milliseconds (average)

99th percentile

Often improves with thread count

Often deteriorates with thread count




Tail at Scale = Fail

• If a request hits ten systems, with 10x latency 1% of the time, average latency doubles


wasted time



Aggregating Data

Global start/end = 200MB/s = WRONG

Additive = 183MB/s = WRONG




Aggregating Data

Stonewalling = 243MB/s = REALLY WRONG




Aggregating Data

• If your data is “X per second” then measure per second

0

2000

4000

6000

8000

10000

12000

14000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

client9

client8

client7

client6

client5

client4

client3

client2

client1

client0

Staircase

Stragglers




Performance Storage




Storage Performance Factors

Small requests vs.

large requests

Data vs.

metadata

Cached/buffered vs.

synchronous

Read vs.

write




Data Testing: iozone • Can test block and file storage

• Data only

– Sequential/random, O_SYNC/O_DIRECT, AIO, …

– Could be better w.r.t. spatial distribution

– Limited support for cluster testing

• “Stonewalling” by default

• Zillions of command-line options

• fio is very similar Cloud Storage Performance 11



iozone options -c include close -e include fsync

-o use O_SYNC -O report ops/second

-r record size -s file size

-l thread count -C show child stats

-i 0 sequential write

-i 1 sequential read

-I 9 random pwrite

+ 67 more




Sample iozone output

Children see throughput for 4 rewriters = 49124.63 ops/sec Parent sees throughput for 4 rewriters = 47810.32 ops/sec Min throughput per process = 10874.55 ops/sec Max throughput per process = 13628.46 ops/sec Avg throughput per process = 12281.16 ops/sec Min xfer = 831.00 ops Child[0] xfer count = 1024.00 ops, Throughput = 13628.46 ops/sec Child[1] xfer count = 950.00 ops, Throughput = 12620.25 ops/sec Child[2] xfer count = 920.00 ops, Throughput = 12001.36ops/sec Child[3] xfer count = 831.00 ops, Throughput = 10874.55 ops/sec




Metadata Testing

• Filebench

– Workload Model Language

– both data and synchronization

• Dbench

– trace replay

• Somebody really needs to do better!




Object Testing

• COSBench

– Object read, write, delete

– Java + XML(ish) + Windows-style .ini files

– Parallel and even distributed

– still…

• Somebody needs to do much better!




Performance

Cloud

Storage




Noisy Neighbors

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

IOPS




Performance Distributions

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

1 4 7

10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97

100

Amazon

Rackspace

Host Virtual




Performance Ratios

0

50

100

150

200

250

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Ne

two

rk (

MB

/s)

Disk (4KB writes/s)

Chart Title

Amazon

Rackspace




Other Problems

• “Clunker” instances

– Netflix: kill and start a new one

• Network inconsistency

– same host vs. same switch vs. ???

• Cheating

– ignore O_SYNC, non-battery-backed cache




Conclusions

• Massive variability is what makes this hard

– test many types, many instances, many times

– reduce other variables (e.g. workloads)

– automate, automate, automate

• Think in terms of probabilities instead of averages

• Use mathematical models or simulation to determine appropriate “insurance level”




Modeling Example

• Goal: 99% probability of 100K IOPS

• Same data as above

Provider Ideal From Model Ratio

Amazon 7 13 1.86

Rackspace 14 28 2.00

Host Virtual 8 11 1.38

http://hekafs.org/index.php/2013/05/performance-variation-in-the-cloud/




Thank You!

[email protected]

mailto:[email protected]

Measuring Storage Performance in the Cloud - USENIX · Measuring Storage Performance in the Cloud...

Documents

Transcript of Measuring Storage Performance in the Cloud - USENIX · Measuring Storage Performance in the Cloud...