Views and opinions expressed in this presentation are solely of its authors and do not necessarily represent those of Alphabet, Inc or its subsidiaries, including Google, Inc.

Select images and formulae are provided with permission from Google, Inc

Percentile-Based ApproachTo Forecasting Workload

Growth

Alex Gilgur, Douglas Browning, Stephen Gunn,Xiaojun Di, Wei Chen, and Rajesh Krishnaswamy

IT Capacity and Performance 41st International Conference by

the Computer Measurement Group (CMG'15)

San Antonio, TX November 5, 2015Session 525

How often do you see such patterns?

How do you predict them?

“What's in a name? That which we call a rose / By any other name /Would smell as sweet”

● Useful Workload Measures:○ Outlier Boundary ○ Mean + “Z” Standard Deviations (e.g. “3 sigma”, “6 sigma”, etc.)○ 95th percentile○ 90th percentile○ 75th percentile○ Simple Average (Mean)○ Median○ 25th percentile

● Define Workload via Little's Law:○ Number of units of work in the system

■ Number of packets in flight■ Number of queries in queue

○ W = X * T

● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:

■ “Z-sigma”■ Quantile Regression

● p95● Outlier Boundaries

○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases

What We Will Talk About

“It may be normal, darling; but I'd rather be natural.”Truman Capote, Breakfast at Tiffany’s

Usual Assumptions:

● Residuals are Normally distributed● Mean and StDev of residuals are constant

“Double, Double, Toil and Trouble”

Solutions

And the Winner Is...

… Predicting 95th Percentile Directly !

Why not Split the Workload into Two Servers?

● Sometimes app1 and app2 have to go to the same server for processing.

● To size the server (VM/Network Link/Storage LUN/… ), we only want to forecast the upper bound.

● Sometimes it’s hard to get additional capacity: ○ budget, justification, approvals, etc.○ Cloud helps, but...

Percentile-based Modeling is often the only solution

QuantReg for the 2 types of workload

fits very nicely in both cases

Should we Size Hardware for 95%-ile of Workload?

5% of the time SLA (99.9%? 99.999%?)

will be violated

IQR vs. 5th and 95th Percentiles

IQR excludes “true” outliers

IQR (Tukey’s method) 5th and 95th Percentiles

Shouldn’t we Size Resources for Non-Outliers Instead?

John Tukey’s IQR method

● Why?○ Normality does not matter○ 5th & 95th percentiles in this scenario

would have “outlawed” a good part of data points that are NOT outliers

● Why Not?○ Multi-Modal distributions

We size for SLA, as long as traffic stays within outlier boundaries

Tukey’s Boundaries in Trended Data

Tukey’s Boundaries in Trended Data

Tukey’s Boundaries in Trended Data

● don’t predict based on all data:○ find natural groupings (GMM, DBSCAN, ...) ○ then fit the model○ use the higher cluster to guarantee QoS

Tukey’s Method● robust boundaries● distribution-agnostic● can be used to guarantee high QoS

○ Unimodal Distribution ○ Multi-Modal Distribution

Long Story Short

● Forecast Percentiles

● Find Natural Groupings

● Size For Outlier Boundaries

● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:

■ “Z-sigma”■ Quantile Regression

● p95● Outlier Boundaries

○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases

What We Will Talk About Next

Knees, Hyperbolae, and Sensitivity

Capacity:

Workload:

In a closed (constrained) system, sensitivity of throughput to latency increases with the throughput

In Human Terms

● As throughput increases, latency can only increase.

● As latency increases, throughput in a constrained queueing system can only decrease.

● As we increase throughput in a constrained system near its saturation point, its upper percentiles must grow at a slower pace than lower percentiles.

In Mathematical Terms

Quantile Compression Theorem:

IF raw demand on a constrained system X′ is

moderated via a monotonically increasing damped function X = f(X′),

THEN, as the system is approaching saturation,

smaller percentiles of moderated demand X grow on average faster than higher percentiles.

This is only a presentation; for mathematical proof, please see the paper.

Long Story Short

“It’s just there...”-Miles Davis

Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.

In Practical Terms: are We Constrained?

Percentile trajectories diverge; we are NOT constrained here.

In Practical Terms: are We Constrained?

p5 and p95 trajectories converge; we ARE GETTING constrained here.

Percentile trajectories are almost all parallel; we are almost NOT constrained here.

In Practical Terms: are We Constrained?

“It’s always the quiet ones”

In Practical Terms: are We Constrained?Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`

p95 trajectory is growing slower than p50; we ARE constrained here.

In Practical Terms: are We Constrained?Predictions made:p75` > p95` > p50` > p97.5`

Line Predicted by p95

Line Predicted by p50

Line Predicted by p75

Line Predicted by p97.5

Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`

p95 trajectory is growing slower than p50; we ARE constrained here.

In Practical Terms: are We Constrained?Predictions made:p75` > p95` > p50` > p97.5`

Line Predicted by p95

Line Predicted by p50

Line Predicted by p75

Line Predicted by p97.5

Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`

Observed Growth Rates:p75` > p95` > p50` > p97.5`

p95 trajectory is growing slower than p50; we ARE constrained here.

In Statistical Terms: When Resource is Unconstrained

Unbounded Resource Throughput: Unimodal; Asymmetric; Skew is Constant

Bounded (Constrained) Resource Throughput: may become Bimodal; Skew may vary

In Statistical Terms: When Resource is Constrained

Can we Measure Asymmetry?

Not All Distributions are Easy to Deal With

What if...

...Mean and

Variance are

undefined

?

Not All Distributions are Easy to Deal With

What if...

...Mean and

Variance are

undefined

?

Percentiles win!

What if...

...Mean and

Variance are

undefined

?

Long Story Short

When resource is constrained:

1. Distribution changes:a. becomes left-skewedb. becomes bimodal

2. Skew is very important

3. Percentile-based Skew is the preferable statistic

Some More Examples that you May Have Seen Before

Growth Was Constrained

Unconstrained Growth

Right-Skewed (long right tail)

Controlled Growth

Left-Skewed (long left tail)

What We Will Talk About Next

● Workload Forecasting○ When Linear Regression fails○ Workload statistics:■ “Z-sigma”■ Quantile Regression● p95● Outlier Boundaries

○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases

“None that I know will be, much that I fear may chance”

● Regression: ○ Business Metrics○ Little's Law○ Time-related Covariates

● Time Series Analysis (Forecasting):○ EWMA○ ARIMA

Is it right to Size Resources Using Upper Percentiles of Bounded Data?

Forecasting demand using bounded data leads to undersizing the resource

Doing so is the path to the dark side.

Resource Constraint =>

Quantile Compression =>

Underforecasting the load =>

Undersizing the resource

Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.

Can we infer unbounded lines from bounded data?

TimeStamp

Ske

w1. Find Skew for Unbounded Data

2. Forecast Upper and Lower Percentiles to the Time Horizon of Interest

3. Infer Unbounded Upper Percentiles (Skew = const)

4. If (unbounded = forecasted) => system is still unbounded

5. If (unbounded > forecasted & forecast > history) => system will be constrained

Throughput Forecasting Algorithm

Get U(t)StartIdentify the most

appropriate trend type

Done

Predict Trajectories for the LB

(p25) and Median

(LB’, M’) = Prediction for Low

Bounds and Median

Save the forecast

For each timestamp

Build hourly

boxplots

data

Thro

ughput

Thro

ughput

Identify the most

appropriate trend type

Thro

ughput LINT

hro

ughput

LOG

Thro

ughput

EXPTh

rou

gh

pu

t

QUAD

Th

roughput

PWRTh

roughput

R2 = 0.45

R2 = 0.34

R2 = 0.47

R2 = 0.38

R2 = 0.46

Trend Type Selection

● we know the variance is huge● we are selecting TREND TYPE● we are NOT selecting MODEL

Now we can use T-test

A few words about R2

Now we can use T-test

LINThro

ughput R2 = 0.45

QUAD

Thro

ughput

R2 = 0.46

EXPThro

ughput

R2 = 0.47

LOGThro

ughput

R2 = 0.34

PWRThro

ughput

R2 = 0.38

A few words about R2

Trend Type SelectionT

hro

ughput

LIN

Thro

ughput

R2 = 0.45

● we know the variance is huge● we are selecting TREND TYPE● we are NOT selecting MODEL

MODELS: “LIN”, “PWR”, “EXP”, “LOG”,“QUAD”

Identify the most

appropriate trend type

Long Story Short

Forecasting Algorithm:

1. Compute the Skew

2. Identify the Trend Type

3. Forecast p25 and p50

4. Apply Skew to Compute Upper Percentiles

5. Compute Outlier Boundaries

● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:

■ “Z-sigma”■ Quantile Regression

● p95● Outlier Boundaries

○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases

What We Will Talk About Next

Th

rou

gh

put

Th

rou

gh

put

Use Cases: Unbounded: How Far to the Threshold?

Threshold

1000

250

Non-Outliers above threshold

Pr {traffic > threshold} > 5%

Another interesting scenario: Forecasting Resource Congestion Zone

Forecasting ResourceCongestion Zone

By predicting collision points for different percentiles,we can get a general idea of a Resource Congestion Zone

HAL9000: I've just picked up a fault

in the AE35 unit. It's going to go

100% failure in 72 hours.

Use Cases: Unbounded: How Much to Add?

(unbounded = forecasted) => system is still unbounded

Use Cases: Bounded (Congested): How Much to Add?

(unbounded > forecasted) => system was, and will be, constrained

Use Cases: Bounded (Congested): How Much to Add?

(unbounded > forecasted) => system may have been, and will be, constrained

Long Story Short

● Feedback Loop & Quantile Compression:

○ “It’s just there”:■ explicitly, via the protocol.■ implicitly, in the saturation dynamics.

● Do not assume anything!

○ Especially about shapes of distributions.

● Do not forecast p95!

○ Forecast Outlier Boundaries instead.○ Mean and Variance are overrated!

● Do Size Hardware for the would-have-been-

unbounded Forecasts

Alex Gilgur

agilgur@google.com / alexgilgur@gmail.com

+1 (408) 475-7582 / +1 (408) 828-2115

Appendix

“Big 7” of Linearizable Equations

odds log

Is it Right to Size Resources Using Upper Percentiles?

Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.

Is it Right to Size Resources Using Upper Percentiles?

ForecastingMethods:

● EWMA

● ARIMA

● Regression

EWMA models are very specific and computationally fast, but they have to be told trend

(linear or exponential) and seasonality (additive or multiplicative).

ARIMA model will implicitly account for trends, seasonality, and stationarity of the data.

Autocorrelation of ARIMA residuals provide all the periodicities that have been missed.

For stationary data, use ARIMA

For non-stationary data, use EWMA

EWMA and ARIMA overlap

When to use Regression:

● data are monotonic.

● seasonality is NOT statistically significant.

● EWMA and ARIMA fail.

When to use Quantile Regression:

● Upper and Lower bounds behave differently.

● Outliers are possible.

For each data set, we can run a model competition, computing forecast model quality based

on a weighted sum of model goodness of fit, model suitability for forecasting, data stationarity

and data variability, and selecting the model that works best for each data set.

EWMA

ARIMA

Quantile Regression