About me
● Principal SRE @Acquia (Cloud Data Team)
● Joined in December 2011
● Location: Lisbon, Portugal
● Co-authored Seeking SRE w/ Machine Learning for SRE (O’Reilly)
● Founder and Lead of the Portuguese Drupal Association
● Fun Facts:
○ Presented in DevOps events including DrupalCons.
○ Dedicated father of 2 kids and still manages to study and write.
○ First Linux installation: Slackware in 1994.
○ Former theatre actor.
AgendaThe problem
What is Capacity
Why do Capacity Planning
Relation to Site Reliability Engineering
Budget & Capacity Planning
Load Testing
Performance Tuning vs. Capacity Planning
What to measure
How to measure
How to track capacity
Forecasting
First Easy Steps
Conclusions
The ProblemSite Launch & User Expectations
Falcon Heavy launch, Spacex
Typical Drupal Site Launch
What about Capacity Planning??
- Disable devel
- Configure cron
- Check The Uplo
ad Sizes & Exe
cution Time
- Check Recipien
t Email Addres
ses
- Set The File P
ermissions
- Pro-tect Your
Root Account
- Check Per-mis-
sions
- Turn Off Error
Reporting
- Han-dle 404 Er
rors Gracefull
y
- Check Robots.t
xt
- Com-bine Patha
uto With Globa
l Redirect
- Cre-ate A Main
-te-nance Page
- Con-fig-ure Ca
ching
- Css And Javasc
ript Optimisat
ion
- Check Unpub-li
shed Con-tent
Is Not Visible
- Con-fig-ure St
atistics
- Monitor the Si
te
-** Plan
for Failure *
*
User Expectations
Drupal click screenshot
● The end goal of capacity planning is a smooth and speedy experience for the users
● Varies depending on what type of application is and what portion of the application they interact with
No silver bullet
● Plenty of capacity but a slow website or unavailable
● Capacity is only one part of making the end-user experience fast
● We want to measure and track to make forecasts
● Intolerable amount of latency should raise a flag
What is Capacity
resources required to run your services in the context you have chosen to run them
Carbon Fiber Tank, SpaceX
Capacity in Site Reliability Engineering (SRE)
● Capacity: The maximum amount of output a product deployment is capable of completing in a given period of time
● Capacity planning: Process that determines the resources needed, like people, instances, CPU, memory, time and more, for the company to meet changing demands for its services
● In the Drupal World we focus mostly on serving WEB capacity
Resource management
The Art of Capacity PlanningArun Kejariwal, John Allspaw"O'Reilly Media, Inc."
● Ensure proper resources are available to handle load
● Define procurement and an approval process
● Justify capital needs
● Manage resources after deployment
Why do Capacity Planning
Kroger grocery store, Lexington Kentucky, 1947, by Brett Streutket
Quick and Dirty Math
● Only spend as much as you actually need
● Be ahead of sharp growth
● Avoid emergencies
Stay Fast and Reliable
Site Reliability Engineering
Rocket Laboratory, 1952NASA/William A. Bowles
Ben Treynor - Google
...an SRE team is responsible for the availability, latency,
performance, efficiency, change management, monitoring,
emergency response, and capacity planning of their service(s)...
“
“
Demand Forecasting and Capacity Planning
● Ensuring that there is sufficient capacity and redundancy
● Serve projected future demand with the required availability
● Ensure the required capacity is in place by the time it is needed
● Take both organic and inorganic growth into account
http
s://u
nspl
ash.
com
/pho
tos/
mex
eVP
lTB
6k
How SRE advocates for Capacity Planning
● Perform regular load testing
● Incorporate SLOs on Capacity
● Capacity is critical to availability, therefore the SRE team leads capacity planning initiatives and provisioning
http
s://u
nspl
ash.
com
/pho
tos/
DX
9X0g
0Cg8
8
Budget & Capacity Planning
Vintage Grow Your Money by Chris Potter, ccPixs.com
Keeping the costs low
● Meet with Finance, Engineering and Product
● Gather Systems and Application metrics
● Use that data to justify the investment
Three forces that impact Capacity Planning
Product
FinanceEngineering
Plan
Load Testing“Hope is not a strategy”
St. Margrethen - Load Test by Kecko
Load testing a Drupal stack
● How to load test? “Hit it until it breaks”
● Include the points of failure in the calculations
● Determining backend limits can be tricky
● Use those resource ceilings as a basis while predicting future growth
https://docs.acquia.com/acquia-cloud/arch/
A Few Load testing Toolssimulate
● Loadrunner
○ http://bit.ly/microfocus-loadrunner
● Iago
○ https://github.com/twitter/iago
● JMeter
○ http://jmeter.apache.org/
collect● Prometheus
○ http://www.prometheus.io/
● Signalfx
○ http://www.signalfx.com/
● Cacti
○ http://cacti.net
● Ganglia
○ http://ganglia.info
● Nagios
○ http://nagios.org/
http
s://w
ww
.goc
omic
s.co
m/c
alvi
nand
hobb
es/1
986/
11/2
6
Performance Tuning vs. Capacity planning
(different goals)
Top Speedby Alexander Nie
What to measuredefining the metrics
End-of-lifeby Dennis van Zuijlekom
Divide & Conquer
● Splitting nodes
● Understand capacity demands of each node
● Measure more distinctly
● How requests or queries per second affect resources
Identifying the key resources to measure
● Disk space (MB)
● Disk throughput (IOPS)
● CPU performance (FLOPS)
● RAM memory (MB)
● Network bandwidth (Mbps)
● Network IP pool (Netmask)
● Others
How to measure
Living Computer Museum, Seattle
http
://w
ww
.bre
ndan
greg
g.co
m/P
erf/l
inux
_per
f_to
ols_
full.
png
| Tools to measure on Linux servers |
Collecting resources on web servers
TODO: CODE
● Example script that sends metrics to statsd
● Low footprint using /proc, df and ps
● For a constant reliable monitoring service use collectd: https://collectd.org or Telegraf: https://www.influxdata.com/time-series-platform/telegraf/
How to track Capacity
Store and display time-series
● Signalfx
● Cacti
● Ganglia
● Graphite
● Signalfx
● Datadog
● Ruxit
● LogicMonitor
● Sematext
● CoScale
● Riemann
● Prometheus
● Sensu
● Idera
● Bijk
● X-Pack
● vRealize Hyperic HQ
A couple of load testing tips
load testing Tutorials:https://www.tutorialspoint.com/jmeter
https://www.blazemeter.com/load-testing
docker app for grafana: https://github.com/kamon-io/docker-grafana-graphite
Forecasting (predicting trends)
Numbers And Finance by SeniorLiving.org
Predict the future?
● Use Context & Math
● Make educated guesses
● Long-term view is generally steady
● Generate estimates to sustain growth
● Use an adjustable process
● Forecast guides autoscaling policies
Ceilings and Historical data
● Daily storage consumption example
● Metric: total available disk space
● Cumulative total provides an historical perspective
● We can predict future needs
● Storage will probably be exhausted in the ceiling to where the line is headed
Curve fitting
● Curve fitting
● Creative & Scientific
● Stay ahead of growth
● Use time-series data
● Forecast by constructing new data points beyond the known
● Reconciliation of what we know and the best fit equation
● Consider context before math
y = mx+b
Forecasting Peak-Driven Resource Usage
● Track how the peaks change over time
● Extrapolate from that data to predict future needs
● Identify the server resource ceilings
● Find a relation between resources and application-level work
● Decide if we should scale vertically or horizontally
● and perform proactive autoscalling
● Fityk is an Open Source Software for nonlinear fitting of analytical functions to data.
● Incorporate cfityk scripts into automated curve fitting, like:
cfityk ricardo-disk.fit
@0 < ricardo-disk.csv
guess Quadratic
fit
info formula
quit
Returns the formula: 4888.18 + 363.063 * x + 8.91132 + -1.55119*x + 0.0660771*x^2
Homepage: https://fityk.nieto.pl/
cfityk ricardo-disk.fit
@0 < ricardo-disk.csv
guess Quadratic
fit
info formula
quit
Automating Forecasts with fityk & cfityk
Small demo: https://youtube.com/watch?v=EZnyq1Hr_7I
Forecasting with Machine Learning
Seeking SREConversations About Running Production Systems at ScalePublisher: O'Reilly Media
● Most popular method for curve-fitting in fityk is Levenberg-Marquardt
● ML is also an option for forecasting (book I co-authored)
● Code examples and guideshttps://github.com/ricardoamaro/MachineLearning4SRE
Start with Easy Steps
Get Started
1. Select a process owner.2. Identify the resources to be measured.3. Measure these resources.4. Compare to maximum capacity.5. Collect workload forecasts.6. Use forecasts for IT resource requirements.7. Map requirements onto existing utilizations.8. Predict when the system will be out of capacity.9. Update forecasts and utilizations.
Set a Goal!
● Two Classes:
○ Load: usually expressed in arrival rate or peak rate of requests hitting the serviceeg. target for 10.000 authenticated concurrent Drupal users
○ Performance: usually expressed in the form of Service Level Objectiveseg. 99th percentile of all requests should return in less 500ms
Be proactive ( plan & document ahead)
Picasso drawing with Paloma and Claude at Villa la Galloise, 1953.By Edward Quinn, EdwardQuinn.com.
Capacity Planning Dashboard
● Support your conclusions with metrics in a dashboard
● Both manual scaling and auto scaling decision should be based on real data
● When to scale?○ date and time (be alerted if needed)
● How to scale?○ vertical, horizontal or diagonal scaling
(Example) Drupal Cluster Dashboard
type value
limit/ node
ceiling units
limit (total)
current (peak)
peak %
Estimated days left
Varnishcache
28 1024 req/sec 2048 600 29% 830
Web 31 80 busy calls 160 145 90% 12
Database 15 60 connections 120 96 80% 36
Storage 14 30 TB 30 14 46% 21
Conclusions
Drive the system to the appropriate level of risk for the lowest cost.
Questions?
The 1 Year and 1 hour Capacity Plan in the Drupal World
Join us forcontribution opportunities
Mentored Contribution
First TimeContributor Workshop
GeneralContribution
#DrupalContributions
What did you think?
https://events.drupal.org/node/22330
https://www.surveymonkey.com/r/DrupalConSeattle
Top Related