Control Theory for SRE - USENIX · 10/4/2019 Control Theory for SRE...
15
10/4/2019 Control Theory for SRE pidtalk.kube.gauntletwizard.net/presentation.slide#1 1/15 Control Theory for SRE Control Theory for SRE Using PID Controllers to scale your services smoothly Using PID Controllers to scale your services smoothly 2 October 2019 2 October 2019 Thomas Hahn Thomas Hahn Site Reliablity Engineer, TCB Technologies. Site Reliablity Engineer, TCB Technologies. Mark Hahn Mark Hahn Devops Practice Director, Ciber Global LLC. Devops Practice Director, Ciber Global LLC.
Transcript of Control Theory for SRE - USENIX · 10/4/2019 Control Theory for SRE...
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 1/15
Control Theory for SREControl Theory for SREUsing PID Controllers to scale your services smoothlyUsing PID Controllers to scale your services smoothly2 October 20192 October 2019
Thomas HahnThomas HahnSite Reliablity Engineer, TCB Technologies.Site Reliablity Engineer, TCB Technologies.
Mark HahnMark HahnDevops Practice Director, Ciber Global LLC.Devops Practice Director, Ciber Global LLC.
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 2/15
Control Theory For SREControl Theory For SRE
22
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 3/15
History of Control TheoryHistory of Control Theory
What is a PID controller? Why use one?What is a PID controller? Why use one?
Steam Engines and Centrifugal GovernorsSteam Engines and Centrifugal Governors
Cruise Control in your CarCruise Control in your Car
Ship SteeringShip Steering 33
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 4/15
Centrifugal Governor - Ealiest application.Centrifugal Governor - Ealiest application.
A A Centrifugal GovernorCentrifugal Governor (https://en.wikipedia.org/wiki/Centrifugal_governor) (https://en.wikipedia.org/wiki/Centrifugal_governor) is one of the earliest examples of a PIDis one of the earliest examples of a PIDcontroller.controller.
44
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 5/15
"The possibility of obtaining more accurate steering by automatic means"The possibility of obtaining more accurate steering by automatic meansthan can be accomplished by manual control with its inherent limitationthan can be accomplished by manual control with its inherent limitationdue to the low sensitiveness of the human eye in dectecting slow angualrdue to the low sensitiveness of the human eye in dectecting slow angualrmotions, fatigue, etc., becoms of greater importancw ithe increase in sizemotions, fatigue, etc., becoms of greater importancw ithe increase in sizeof ships and cost of fuel"of ships and cost of fuel"
N. Minorsky, Directional Stability of Automatically Steered Bodies, May 1922N. Minorsky, Directional Stability of Automatically Steered Bodies, May 1922 55
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 6/15
Math of a PID controllerMath of a PID controller
Error at time Error at time tt: : e(t)e(t) = = SPSP - - PVPV
Diagram from Diagram from https://en.wikipedia.org/wiki/PID_controller#Mathematical_formhttps://en.wikipedia.org/wiki/PID_controller#Mathematical_form (https://en.wikipedia.org/wiki/PID_controller#Mathematical_form) (https://en.wikipedia.org/wiki/PID_controller#Mathematical_form) 66
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 7/15
PID Math in PrometheusPID Math in Prometheus
- record: control_terms_kp - record: control_terms_kp expr: avg(control_kp) * pid2d_error expr: avg(control_kp) * pid2d_error - record: control_terms_ki - record: control_terms_ki expr: avg(control_ki) * sum_over_time(pid2d_error[10m]) expr: avg(control_ki) * sum_over_time(pid2d_error[10m]) - record: control_terms_kd - record: control_terms_kd # The d term is extrapolated out to the same window as the Ki term # The d term is extrapolated out to the same window as the Ki term expr: avg(control_kd) * idelta(pid2d_error[1m]) * count_over_time(pid2d_error[10m]) expr: avg(control_kd) * idelta(pid2d_error[1m]) * count_over_time(pid2d_error[10m])
77
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 8/15
PID Math in PrometheusPID Math in Prometheus
- record: pid2d_location - record: pid2d_location expr: (pid2d_location + pid2d_velocity) or vector(0) expr: (pid2d_location + pid2d_velocity) or vector(0) - record: pid2d_error - record: pid2d_error expr: avg(control_setpoint) - pid2d_location expr: avg(control_setpoint) - pid2d_location
- record: pid2d_deltav - record: pid2d_deltav expr: clamp_min(clamp_max((control_terms_kp + control_terms_ki + control_terms_kd), 100), -100) expr: clamp_min(clamp_max((control_terms_kp + control_terms_ki + control_terms_kd), 100), -100) - record: pid2d_velocity - record: pid2d_velocity expr: (pid2d_velocity + pid2d_deltav) or vector(0) expr: (pid2d_velocity + pid2d_deltav) or vector(0)
88
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 9/15
Introducing the technologiesIntroducing the technologies
PrometheusPrometheus
GrafanaGrafana
ControllerController
KubernetesKubernetes 99
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 10/15
Demo - A simple PID Controller.Demo - A simple PID Controller.
1010
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 11/15
Tuning a PID controllerTuning a PID controller
Hand tuningHand tuning
Ziegler-Nichols Method https://en.wikipedia.org/wiki/Ziegler%E2%80%93Nichols_methodZiegler-Nichols Method https://en.wikipedia.org/wiki/Ziegler%E2%80%93Nichols_method(https://en.wikipedia.org/wiki/Ziegler%E2%80%93Nichols_method)(https://en.wikipedia.org/wiki/Ziegler%E2%80%93Nichols_method)
DampingDamping
Other methodsOther methods 1111
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 12/15
About this talkAbout this talk
The source for this presentation can be found at The source for this presentation can be found at gitlab.com/gauntletwizard/pid-controller-talkgitlab.com/gauntletwizard/pid-controller-talk (https://gitlab.com/gauntletwizard/pid-controller-talk) (https://gitlab.com/gauntletwizard/pid-controller-talk)
The presentation itself is available online atThe presentation itself is available online atpidtalk.kube.gauntletwizard.net/presentation.slidepidtalk.kube.gauntletwizard.net/presentation.slide (http://pidtalk.kube.gauntletwizard.net/presentation.slide) (http://pidtalk.kube.gauntletwizard.net/presentation.slide)
Areas for further work: Areas for further work: - Add more explanation to this respository - Add more explanation to this respository - Build a Kuberenets Model- Build a Kuberenets Model 1212
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 13/15
References:References:
S. Bennett, "A brief history of automatic control," in IEEE Control Systems Magazine, vol.S. Bennett, "A brief history of automatic control," in IEEE Control Systems Magazine, vol.16, no. 3, pp. 17-25, June 1996. doi: 10.1109/37.50639416, no. 3, pp. 17-25, June 1996. doi: 10.1109/37.506394 (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=506394) (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=506394)
N. Minorsky, "DIRECTIONAL STABILITY OF AUTOMATICALLY STEERED BODIES," inN. Minorsky, "DIRECTIONAL STABILITY OF AUTOMATICALLY STEERED BODIES," inAmerican Society of Naval Engineers, doi: 10.1111/j.1559-3584.1922.tb04958.xAmerican Society of Naval Engineers, doi: 10.1111/j.1559-3584.1922.tb04958.x(https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1559-3584.1922.tb04958.x)(https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1559-3584.1922.tb04958.x)
1313
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 14/15
Thank youThank you
Thomas HahnThomas HahnSite Reliablity Engineer, TCB Technologies.Site Reliablity Engineer, TCB [email protected]@tcbtech.com (mailto:[email protected]) (mailto:[email protected])
@gauntletwizard@gauntletwizard (http://twitter.com/gauntletwizard) (http://twitter.com/gauntletwizard)
Mark HahnMark HahnDevops Practice Director, Ciber Global LLC.Devops Practice Director, Ciber Global [email protected], [email protected]@ciber.com, [email protected] (mailto:[email protected],%[email protected]) (mailto:[email protected],%[email protected])
https://www.linkedin.com/in/markphahnhttps://www.linkedin.com/in/markphahn (https://www.linkedin.com/in/markphahn) (https://www.linkedin.com/in/markphahn)
10/4/2019 Control Theory for SRE
pidtalk.kube.gauntletwizard.net/presentation.slide#1 15/15
