Performance & Benchmarking

What Matters?

• Which airplane has best performance:

CPU Time

• Time to run a program:

GHz Myth

• GHz measures cycles / second• Faster is better, but only if architecture

otherwise constant

MIPS

• MIPS = Millions of Instructions Per Second– Inverse of these two rates:

– Can't compare different architectures• Especially RISC/CISC

FLOPS

• FLOPS = Floating Point Operations Per Second– Same issue as MIPS…• What exactly is a FLOP?• How many does a program take?

Synthetic Benchmarks

• Standard "program" run on different machines• Early programs:– Whetstone – floating point ops– Dhrystone – integer ops

• Issues:– Small size… don't test memory– Compiler optimizations targeted at benchmarks

Benchmark Suites

• Collection of large/real world programs tested

What do you care about?

• Some parts of suite focus on– Integer– Floating Point– Memory

• Generally don't test IO

SPEC

• Standard Performance Evaluation Corporation• https://

www.spec.org/cgi-bin/osgresults?conf=cpu2006

• Industry consortium, licenses test suite

Geekbench

• Consumer focused, cross platform

http://browser.primatelabs.com/geekbench3/

TPC

• Transaction Processing Council benchmarks– Business transaction oriented simulation• High volume, short duration• Communication• Disk IO

Application Testing

• If one application is make or break, benchmark with it:

Benchmark Mathematics

• Comparisons: System A is x% faster than B

Benchmark Mathematics

• Car A travels 10 miles in 3 minutes, Car B 10 miles in 4 minutes. How much faster is A?

Average Performance

• Average performance across multiple programs may be important

Weighted Average

• If we know jobs have distinctly different usages, need weighted mean:

Arithmetic Mean Issues

• Skewed by outliers:

Arithmetic Mean Issues

• Normalized task performance easier to talk about– "C takes 10 times longer than A to run program V"

Arithmetic Mean Issues

• Speedup calculations depend on normalization target:

Average Performance

• Geometric mean:– Multiply n value, take the nth root

Average Performance

• Geometric mean:– Multiply n value, take the nth root

Or

𝐺=(50 ∙200 ∙250∙ 400 ∙5000 )15

Geometric Mean

• Geometric Mean:– Less dependent on outliers

Normalized

• Which machine we normalize against doesn't matter…

Harmonic Mean

• Machine does equal reads & updates on DB– Can process 10,000 reads/second– Can process 2,000 writes/second

• What is the average query rate?

Harmonic Mean

• Machine does equal reads & updates on DB– Can process 10,000 reads/second– Can process 2,000 writes/second

• What is the average query rate?– NOT (10,000 + 2,000) / 2 = 6,000

A Car

• Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

A Car

• Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

𝑎𝑣𝑔 𝑠𝑝𝑒𝑒𝑑=𝑡𝑜𝑡𝑎𝑙𝑚𝑖𝑙𝑒𝑠𝑡𝑜𝑡𝑎𝑙 𝑡𝑖𝑚𝑒

A Car

• Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

𝑎𝑣𝑔 𝑠𝑝𝑒𝑒𝑑=𝑡𝑜𝑡𝑎𝑙𝑚𝑖𝑙𝑒𝑠𝑡𝑜𝑡𝑎𝑙𝑡𝑖𝑚𝑒

=30+301+0.5

=6 01.5

=40

Harmonic Mean

• Harmonic mean:– Averages rates

𝐻=𝑛

1𝑥1

+1𝑥2

+1𝑥3

+⋯+1

𝑥𝑛−1

+1𝑥𝑛

Harmonic Mean

• Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

𝐻=2

160

+130

=236 0

=2120

=40

Harmonic Means

• Machine does equal reads & updates on DB– Can process 10,000 reads/second– Can process 2,000 writes/second

𝐻=2

110000

+12000

=2

0 .0006=3337

𝑞𝑢𝑒𝑟𝑟𝑖𝑒𝑠𝑠𝑒𝑐

Types

• Each mean says different things:

• Others:– Weighted geometric

• SPEC uses for categories

– Weighted harmonic