Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation
-
Upload
james-tong -
Category
Technology
-
view
5.172 -
download
1
description
Transcript of Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation
![Page 1: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/1.jpg)
Beyond The Numbers
Baron Schwartz
![Page 3: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/3.jpg)
Who Am I?
● Maatkit● Innotop● Aspersa● JavaScript Libraries
● Percona Toolkit● Monitoring Plugins● Online Tools
![Page 4: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/4.jpg)
● Consulting
● Support● Remote DBA
● Engineering● Conferences &
Training
● Percona Server
● Percona XtraBackup● Percona XtraDB
Cluster
● Percona Toolkit
● Many More
![Page 5: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/5.jpg)
Today's Agenda
● Benchmarks● Aggregation and Distributions● Performance, Capacity & Utilization● Rules of Thumb● Queueing Theory and Scalability
![Page 6: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/6.jpg)
Benchmarks
![Page 7: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/7.jpg)
What's Missing?
● Distribution● Time Series● Response Times● Parameters● Goals● System Specs
![Page 8: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/8.jpg)
What's Misleading?
● Logarithmic X-Axis● Interpolation
![Page 9: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/9.jpg)
What's Good?
● Y-Axis Reaches 0● No Fake-Smoothing
![Page 10: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/10.jpg)
Behind a Single Dot
![Page 11: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/11.jpg)
Look At All That Data...
![Page 12: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/12.jpg)
What's With The Grid Lines?!?!?
![Page 13: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/13.jpg)
Better Benchmarks
What does an ideal benchmark report look like?
![Page 14: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/14.jpg)
Clear Benchmark Goals
● Validating hardware configuration● Comparing two systems● Checking for regressions● Capacity planning● Reproducing bad behavior to solve it● Stress-testing to find bottlenecks
![Page 15: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/15.jpg)
Hardware and Software
● Specs for CPU, disk, memory, network● Software versions (OS, SUT, benchmark)● Filesystem, RAID controller● Disk queue scheduler
![Page 16: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/16.jpg)
Presenting Results
● Ideally, make raw results available● Include metrics from OS (CPU, RAM, IO,
network)● Generate some plots to summarize
● This is where the rubber meets the road!
![Page 17: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/17.jpg)
Better Aggregate Measures
● Average● Percentiles
● 95th● 99th
● Maximum● Observation Duration
● Question: how bad can 95th percentile be?
![Page 18: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/18.jpg)
More Aggregate Measures
● Median (50th Percentile)● Standard Deviation● Index of Dispersion
![Page 19: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/19.jpg)
Better...
![Page 20: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/20.jpg)
Better Still...
![Page 21: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/21.jpg)
Keep It Coming...
![Page 22: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/22.jpg)
Throughput AND Response Time
![Page 23: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/23.jpg)
Performance
● What is Performance?● Two Metrics
● Response Time (time per task)● Throughput (tasks per time)
● They're not reciprocals● More on this later
![Page 24: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/24.jpg)
What Performance Isn't
● CPU Usage● Load Average● Other metrics of resource consumption
![Page 25: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/25.jpg)
Performance
● I often focus on response time● It represents user experience● Throughput indicates capacity rather than
performance
● For benchmarking, throughput is primary
![Page 26: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/26.jpg)
Utilization
● The portion of time during which the resource is busy● i.e. there is at least one thing in progress
![Page 27: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/27.jpg)
Utilization is Confusing
● Be very careful with tools that report utilization
● From the Linux iostat man page:● “%util: Percentage of CPU time during which
I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.”
● Can you parse that? Is it true?
![Page 28: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/28.jpg)
Capacity
● What is Capacity?
![Page 29: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/29.jpg)
Capacity
![Page 30: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/30.jpg)
Capacity – My Definition
Capacity is the maximum throughput
... at achievable concurrency
... with acceptable performance
... as defined by response time
... meeting specified constraints
... over specified observation intervals.
![Page 31: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/31.jpg)
Capacity Example
● What is capacity of the system at a concurrency of 32 with 10-second 95th-percentile response time not to exceed 2ms over a 60-minute duration?
● To determine this, we need goal-seeking benchmark software● Most benchmark software can't do this
![Page 32: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/32.jpg)
Benchmarks, etc Recap
● Most benchmarks reveal very little● Benchmark reports reveal even less● It's good to go beyond the surface
![Page 33: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/33.jpg)
Amdahl's Law
● “The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program.” - Wikipedia
● It's basically a law of diminishing returns.
![Page 34: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/34.jpg)
Should I Defragment My Disk?
● Method 1: Google “defragment”● Method 2: Try it and see● Method 3: Measure if the disk is a
bottleneck
![Page 35: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/35.jpg)
Spolsky -vs- Millsap
![Page 36: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/36.jpg)
Spolsky -vs- Millsap
![Page 37: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/37.jpg)
Amdahl's Law
● Don't try to optimize little things.
![Page 38: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/38.jpg)
Little's Law
● N = XR● That is,
● Concurrency = Throughput * Response Time
● This holds regardless of queueing, arrival rate distribution, response time distribution, etc.
![Page 39: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/39.jpg)
Little's Law Example
● If disk IOs average 4ms...● And there are 280 IOs per second...● Then the disk's average concurrency is:
● N = 280 * .004● N = 1.12
● Do you believe this?● When might it not be true?
![Page 40: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/40.jpg)
Little's Law Example #2
● If disk utilization is 98%● And there are 280 IOs per second● What do we know?
![Page 41: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/41.jpg)
Utilization Law
● U = SX● Also independent of distributions, etc...
● That is,● Utilization = Service Time * Throughput
● Utilization = 98% and Throughput = 280● S = U/X● Service Time = .98 / 280 = .0035
![Page 42: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/42.jpg)
Queueing Theory
● How can we predict the amount of queueing in a system?
● How can we predict its response times?● How can we predict capacity?
![Page 43: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/43.jpg)
Erlang Queueing
● Erlang's formulas model the probability of queueing for a given arrival rate, service time, and number of servers.
● A “server” is anything capable of serving a request.● CPUs● Disks
![Page 44: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/44.jpg)
CPU -vs- Disk Queueing
● Scenario: 4-CPU, 4-disk (RAID0) server● Thought experiment:
● How do processes queue for CPU?● How do I/O requests queue on disks?
![Page 45: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/45.jpg)
Notation
● Typically see something like M/M/1● Each letter is a placeholder in A/S/n
● A = Arrival distribution● S = Service-time distribution● n = Number of servers
● A and S can be one of:● Markov● Deterministic● General
![Page 46: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/46.jpg)
CPUs -vs- Disks
● CPUs: M/M/4
● Disks: 4 x {M/M/1}
![Page 47: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/47.jpg)
M/M/1 Queueing
cmg.org
![Page 48: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/48.jpg)
M/M/n Queueing
cmg.org
![Page 49: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/49.jpg)
Erlang C Function
● M/M/n queueing is modeled by Erlang C● See http://en.wikipedia.org/wiki/Erlang_(unit)
![Page 50: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/50.jpg)
What's Wrong With Erlang C?
● You must validate your arrival times.● You must validate your service times.● The equation is hard to work with.● In practice, it's hard to use Erlang C.
![Page 51: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/51.jpg)
Scalability
● Queueing causes non-linear scaling.● But first, let's talk about linearity.
![Page 52: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/52.jpg)
System Scalability
Concurrency
Thr
ough
put
Why?
![Page 53: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/53.jpg)
Universal Scalability Law
Concurrency
Thr
ough
put
Linear
Amdahl
USL
![Page 54: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/54.jpg)
Amdahl Scalability
![Page 55: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/55.jpg)
USL Scalability
![Page 56: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/56.jpg)
USL Scalability Modeling
![Page 57: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/57.jpg)
USL Performance Modeling
![Page 58: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/58.jpg)
Scalability Limitations
● Locks● Synchronization points● Shared resources● Duplicated data to be kept in sync● Weakest-link problems
![Page 59: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/59.jpg)
RAID10 On EBS
● Which is faster?● RAID 10 over 10 EBS volumes● RAID 10 over 20 EBS volumes
● Hint: http://goo.gl/Xm92Y● Also, http://goo.gl/fAEIL
![Page 60: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/60.jpg)
Debunking “Linear”
● Ask to see the actual numbers.● They shouldn't be rounded off suspiciously.● They must be truly linear.● They must intersect the point (0, 0).
![Page 61: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/61.jpg)
Debunking, Example #1
![Page 62: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/62.jpg)
Is it Linear?
![Page 63: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/63.jpg)
It's Not Linear
![Page 64: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/64.jpg)
Resources
● Naomi Robbins' Blog● http://blogs.forbes.com/naomirobbins/
● Percona White Papers● http://www.percona.com/
● Neil J. Gunther● Guerrilla Capacity Planning
● http://www.contextneeded.com/
![Page 65: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/65.jpg)
Questions?
![Page 66: Benchmarks, performance, scalability, and capacity what s behind the numbers_ presentation](https://reader034.fdocuments.in/reader034/viewer/2022052619/555e0ae8d8b42a9e188b4a5c/html5/thumbnails/66.jpg)
[email protected]@xaprb