CMG15 Session 525
64
Views and opinions expressed in this presentation are solely of its authors and do not necessarily represent those of Alphabet, Inc or its subsidiaries, including Google, Inc. Select images and formulae are provided with permission from Google, Inc Percentile-Based Approach To Forecasting Workload Growth Alex Gilgur, Douglas Browning, Stephen Gunn, Xiaojun Di, Wei Chen, and Rajesh Krishnaswamy IT Capacity and Performance 41 st International Conference by the Computer Measurement Group (CMG'15) San Antonio, TX November 5, 2015 Session 525
-
Upload
alex-gilgur -
Category
Engineering
-
view
62 -
download
1
Transcript of CMG15 Session 525
Views and opinions expressed in this presentation are solely of its authors and do not necessarily represent those of Alphabet, Inc or its subsidiaries, including Google, Inc.
Select images and formulae are provided with permission from Google, Inc
Percentile-Based ApproachTo Forecasting Workload
Growth
Alex Gilgur, Douglas Browning, Stephen Gunn,Xiaojun Di, Wei Chen, and Rajesh Krishnaswamy
IT Capacity and Performance 41st International Conference by
the Computer Measurement Group (CMG'15)
San Antonio, TX November 5, 2015Session 525
How often do you see such patterns?
How do you predict them?
“What's in a name? That which we call a rose / By any other name /Would smell as sweet”
● Useful Workload Measures:○ Outlier Boundary ○ Mean + “Z” Standard Deviations (e.g. “3 sigma”, “6 sigma”, etc.)○ 95th percentile○ 90th percentile○ 75th percentile○ Simple Average (Mean)○ Median○ 25th percentile
● Define Workload via Little's Law:○ Number of units of work in the system
■ Number of packets in flight■ Number of queries in queue
○ W = X * T
● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:
■ “Z-sigma”■ Quantile Regression
● p95● Outlier Boundaries
○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases
What We Will Talk About
“It may be normal, darling; but I'd rather be natural.”Truman Capote, Breakfast at Tiffany’s
Usual Assumptions:
● Residuals are Normally distributed● Mean and StDev of residuals are constant
“Double, Double, Toil and Trouble”
Solutions
And the Winner Is...
… Predicting 95th Percentile Directly !
Why not Split the Workload into Two Servers?
● Sometimes app1 and app2 have to go to the same server for processing.
● To size the server (VM/Network Link/Storage LUN/… ), we only want to forecast the upper bound.
● Sometimes it’s hard to get additional capacity: ○ budget, justification, approvals, etc.○ Cloud helps, but...
Percentile-based Modeling is often the only solution
QuantReg for the 2 types of workload
fits very nicely in both cases
Should we Size Hardware for 95%-ile of Workload?
5% of the time SLA (99.9%? 99.999%?)
will be violated
IQR vs. 5th and 95th Percentiles
IQR excludes “true” outliers
IQR (Tukey’s method) 5th and 95th Percentiles
Shouldn’t we Size Resources for Non-Outliers Instead?
John Tukey’s IQR method
● Why?○ Normality does not matter○ 5th & 95th percentiles in this scenario
would have “outlawed” a good part of data points that are NOT outliers
● Why Not?○ Multi-Modal distributions
We size for SLA, as long as traffic stays within outlier boundaries
Tukey’s Boundaries in Trended Data
Tukey’s Boundaries in Trended Data
Tukey’s Boundaries in Trended Data
● don’t predict based on all data:○ find natural groupings (GMM, DBSCAN, ...) ○ then fit the model○ use the higher cluster to guarantee QoS
Tukey’s Method● robust boundaries● distribution-agnostic● can be used to guarantee high QoS
○ Unimodal Distribution ○ Multi-Modal Distribution
Long Story Short
● Forecast Percentiles
● Find Natural Groupings
● Size For Outlier Boundaries
● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:
■ “Z-sigma”■ Quantile Regression
● p95● Outlier Boundaries
○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases
What We Will Talk About Next
Knees, Hyperbolae, and Sensitivity
Capacity:
Workload:
In a closed (constrained) system, sensitivity of throughput to latency increases with the throughput
In Human Terms
● As throughput increases, latency can only increase.
● As latency increases, throughput in a constrained queueing system can only decrease.
● As we increase throughput in a constrained system near its saturation point, its upper percentiles must grow at a slower pace than lower percentiles.
In Mathematical Terms
Quantile Compression Theorem:
IF raw demand on a constrained system X′ is
moderated via a monotonically increasing damped function X = f(X′),
THEN, as the system is approaching saturation,
smaller percentiles of moderated demand X grow on average faster than higher percentiles.
This is only a presentation; for mathematical proof, please see the paper.
Long Story Short
“It’s just there...”-Miles Davis
Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.
In Practical Terms: are We Constrained?
Percentile trajectories diverge; we are NOT constrained here.
In Practical Terms: are We Constrained?
p5 and p95 trajectories converge; we ARE GETTING constrained here.
Percentile trajectories are almost all parallel; we are almost NOT constrained here.
In Practical Terms: are We Constrained?
“It’s always the quiet ones”
In Practical Terms: are We Constrained?Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`
p95 trajectory is growing slower than p50; we ARE constrained here.
In Practical Terms: are We Constrained?Predictions made:p75` > p95` > p50` > p97.5`
Line Predicted by p95
Line Predicted by p50
Line Predicted by p75
Line Predicted by p97.5
Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`
p95 trajectory is growing slower than p50; we ARE constrained here.
In Practical Terms: are We Constrained?Predictions made:p75` > p95` > p50` > p97.5`
Line Predicted by p95
Line Predicted by p50
Line Predicted by p75
Line Predicted by p97.5
Unconstrained Growth Rates:P97.5` > p95` > p75` > p50`
Observed Growth Rates:p75` > p95` > p50` > p97.5`
p95 trajectory is growing slower than p50; we ARE constrained here.
In Statistical Terms: When Resource is Unconstrained
Unbounded Resource Throughput: Unimodal; Asymmetric; Skew is Constant
Bounded (Constrained) Resource Throughput: may become Bimodal; Skew may vary
In Statistical Terms: When Resource is Constrained
Can we Measure Asymmetry?
Not All Distributions are Easy to Deal With
What if...
...Mean and
Variance are
undefined
?
Not All Distributions are Easy to Deal With
What if...
...Mean and
Variance are
undefined
?
Percentiles win!
What if...
...Mean and
Variance are
undefined
?
Long Story Short
When resource is constrained:
1. Distribution changes:a. becomes left-skewedb. becomes bimodal
2. Skew is very important
3. Percentile-based Skew is the preferable statistic
Some More Examples that you May Have Seen Before
Growth Was Constrained
Unconstrained Growth
Right-Skewed (long right tail)
Controlled Growth
Left-Skewed (long left tail)
What We Will Talk About Next
● Workload Forecasting○ When Linear Regression fails○ Workload statistics:■ “Z-sigma”■ Quantile Regression● p95● Outlier Boundaries
○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases
“None that I know will be, much that I fear may chance”
● Regression: ○ Business Metrics○ Little's Law○ Time-related Covariates
● Time Series Analysis (Forecasting):○ EWMA○ ARIMA
Is it right to Size Resources Using Upper Percentiles of Bounded Data?
Forecasting demand using bounded data leads to undersizing the resource
Doing so is the path to the dark side.
Resource Constraint =>
Quantile Compression =>
Underforecasting the load =>
Undersizing the resource
Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.
Can we infer unbounded lines from bounded data?
TimeStamp
Ske
w1. Find Skew for Unbounded Data
2. Forecast Upper and Lower Percentiles to the Time Horizon of Interest
3. Infer Unbounded Upper Percentiles (Skew = const)
4. If (unbounded = forecasted) => system is still unbounded
5. If (unbounded > forecasted & forecast > history) => system will be constrained
Throughput Forecasting Algorithm
Get U(t)StartIdentify the most
appropriate trend type
Done
Predict Trajectories for the LB
(p25) and Median
(LB’, M’) = Prediction for Low
Bounds and Median
Save the forecast
For each timestamp
Build hourly
boxplots
data
Thro
ughput
Thro
ughput
Identify the most
appropriate trend type
Thro
ughput LINT
hro
ughput
LOG
Thro
ughput
EXPTh
rou
gh
pu
t
QUAD
Th
roughput
PWRTh
roughput
R2 = 0.45
R2 = 0.34
R2 = 0.47
R2 = 0.38
R2 = 0.46
Trend Type Selection
● we know the variance is huge● we are selecting TREND TYPE● we are NOT selecting MODEL
Now we can use T-test
A few words about R2
Now we can use T-test
LINThro
ughput R2 = 0.45
QUAD
Thro
ughput
R2 = 0.46
EXPThro
ughput
R2 = 0.47
LOGThro
ughput
R2 = 0.34
PWRThro
ughput
R2 = 0.38
A few words about R2
Trend Type SelectionT
hro
ughput
LIN
Thro
ughput
R2 = 0.45
● we know the variance is huge● we are selecting TREND TYPE● we are NOT selecting MODEL
MODELS: “LIN”, “PWR”, “EXP”, “LOG”,“QUAD”
Identify the most
appropriate trend type
Long Story Short
Forecasting Algorithm:
1. Compute the Skew
2. Identify the Trend Type
3. Forecast p25 and p50
4. Apply Skew to Compute Upper Percentiles
5. Compute Outlier Boundaries
● Workload Forecasting○ When Classical Methods Fail○ Workload statistics:
■ “Z-sigma”■ Quantile Regression
● p95● Outlier Boundaries
○ Knees, Hyperbolae, and Sensitivity○ Quantile Compression○ Predicting the Workload○ Use Cases
What We Will Talk About Next
Th
rou
gh
put
Th
rou
gh
put
Use Cases: Unbounded: How Far to the Threshold?
Threshold
1000
250
Non-Outliers above threshold
Pr {traffic > threshold} > 5%
Another interesting scenario: Forecasting Resource Congestion Zone
Forecasting ResourceCongestion Zone
By predicting collision points for different percentiles,we can get a general idea of a Resource Congestion Zone
HAL9000: I've just picked up a fault
in the AE35 unit. It's going to go
100% failure in 72 hours.
Use Cases: Unbounded: How Much to Add?
(unbounded = forecasted) => system is still unbounded
Use Cases: Bounded (Congested): How Much to Add?
(unbounded > forecasted) => system was, and will be, constrained
Use Cases: Bounded (Congested): How Much to Add?
(unbounded > forecasted) => system may have been, and will be, constrained
Long Story Short
● Feedback Loop & Quantile Compression:
○ “It’s just there”:■ explicitly, via the protocol.■ implicitly, in the saturation dynamics.
● Do not assume anything!
○ Especially about shapes of distributions.
● Do not forecast p95!
○ Forecast Outlier Boundaries instead.○ Mean and Variance are overrated!
● Do Size Hardware for the would-have-been-
unbounded Forecasts
Appendix
“Big 7” of Linearizable Equations
odds log
Is it Right to Size Resources Using Upper Percentiles?
Quantile Compression:As the system is approaching saturation, smaller percentiles of moderated demand X grow on average faster than higher percentiles.
Is it Right to Size Resources Using Upper Percentiles?
ForecastingMethods:
● EWMA
● ARIMA
● Regression
EWMA models are very specific and computationally fast, but they have to be told trend
(linear or exponential) and seasonality (additive or multiplicative).
ARIMA model will implicitly account for trends, seasonality, and stationarity of the data.
Autocorrelation of ARIMA residuals provide all the periodicities that have been missed.
For stationary data, use ARIMA
For non-stationary data, use EWMA
EWMA and ARIMA overlap
When to use Regression:
● data are monotonic.
● seasonality is NOT statistically significant.
● EWMA and ARIMA fail.
When to use Quantile Regression:
● Upper and Lower bounds behave differently.
● Outliers are possible.
For each data set, we can run a model competition, computing forecast model quality based
on a weighted sum of model goodness of fit, model suitability for forecasting, data stationarity
and data variability, and selecting the model that works best for each data set.
EWMA
ARIMA
Quantile Regression
