Beyond Averages

36
BEYOND AVERAGES Dan Kuebrich / appneta.com

description

When raw data becomes overwhelming, we turn to abstraction to understand our world. In our systems, the data is always overwhelming. Solutions like summary statistics have come to our rescue, and they are good—up to a point. In order to truly understand our systems, we need to know when and how to sidestep those abstractions,to get deep, detailed performance insight. In this brief diatribe inspired by John Rauser’s 2011 Velocity keynote “Look at Your Data”, I’ll explore techniques for visualizing the underlying structure of performance data and how this empowers drilling down to populations and individual samples in the data set. Video: http://www.youtube.com/watch?v=InyHBnd_chw

Transcript of Beyond Averages

Page 1: Beyond Averages

BEYOND AVERAGESDan Kuebrich / appneta.com

Page 2: Beyond Averages
Page 3: Beyond Averages

A few of my favorite abstractions

•Abstraction lets us trade information for actionability

•Min, max, average, quantiles, stdev

•That’s a great trade!• ... right?

Page 4: Beyond Averages

Averages: average at best

Page 5: Beyond Averages

Averages: average at best

Page 6: Beyond Averages

Averages: average at best

Page 7: Beyond Averages

Averages: average at best

Page 8: Beyond Averages

Percentiles: 1 of 100 slices

95%

Page 9: Beyond Averages

Percentiles: 2 of 100 slices

95%

10%

Page 10: Beyond Averages

Percentiles: 2 of 100 slices

95%

10%

Page 11: Beyond Averages

Percentiles: 2 of 100 slices

95%

10%

Page 12: Beyond Averages

Percentiles: 2 of 100 slices

95%

10%

Page 13: Beyond Averages

Computers are hard

• Rarely do we have a single normal distribution underlying the data

• Different users, different requests, different resources, different instances, different times

Page 14: Beyond Averages

Is there a place between Averageland and “A Beautiful Mind”?

http://now-here-this.timeout.com/2012/10/07/crazy-walls-of-clues-from-tv-film-reviewed-by-carrie-from-homeland/

Page 15: Beyond Averages

HistogramsFr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 16: Beyond Averages

Populations revisited

95%

10%

Page 17: Beyond Averages

HistogramsFr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 18: Beyond Averages

Populations re-revisited

95%

10%?

Page 19: Beyond Averages

3d Histograms?Fr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 20: Beyond Averages

3d Histograms?Fr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Time

Page 21: Beyond Averages

HeatmapsFr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 22: Beyond Averages

HeatmapsFr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 23: Beyond Averages

HeatmapsFr

eque

ncy

(eg.

# o

f cal

ls)

Value(eg. latency)

Page 24: Beyond Averages

HeatmapsVa

lue

(eg.

late

ncy)

Time

Page 25: Beyond Averages

OK, but what about the real world?

http://www.justincarmony.com/blog/2012/06/05/customizing-graphite-charts-for-clearer-results/

Page 26: Beyond Averages

Mystery #1

Page 27: Beyond Averages

Mystery #1

Page 28: Beyond Averages

Mystery #1

Page 29: Beyond Averages

Mystery #1

Page 30: Beyond Averages

Mystery #1

Page 31: Beyond Averages

Mystery #2

Page 32: Beyond Averages

Mystery #2

bottom 98%

Page 33: Beyond Averages

Mystery #2

all of it

Page 34: Beyond Averages

Mystery #3

Page 35: Beyond Averages

Mystery #3: UNSOLVED