SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
CPU Utilization Control in Distributed Real-Time Systems
Chenyang LuDepartment of Computer Science and Engineering
2SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Why CPU Utilization Control?Overload protection
CPU over-utilization system crashNightmare for mission-critical applications and
“always-on” E-businesses
Meet deadlinesCPU utilization < schedulable utilization bound
3SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
End-to-End Task Modelin Distributed Real-Time Systems
Periodic task Ti = a chain of subtasks {Tij} located on different processors Subtasks run at a same rate
Task rate can be adjusted within a range Higher rate higher utility
Remote Invocation
Subtask
T1
T2
T3
T11 T12 T13
P1P2 P3
4SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Bi: Utilization set point of processor Pi (1 ≤ i ≤ n)
ui(k): Utilization of Pi in kth sampling period
rj(k): Rate of task Tj (1 ≤ j ≤ m) in kth sampling period
subject to rate constraints:
Rmin,j rj(k) Rmax,j (1 ≤ j ≤ m)
Problem Formulation
n
iii
njkrkuB
j 1
2
}1)|({))((min
5SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Challenge: Uncertainties Execution times?
Unknown sensor data or user input
Request arrival rate? Aperiodic events Bursty service requests
Disturbance? Denial of Service Attacks
Control-theoretic approaches to adaptive softwareRobust performance in face of workload uncertainty
6SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Single-Processor Solution:
Feedback Control Real-Time Scheduling
Adaptation based on single-input-single-output control
Monitor
Processor
OS
Application
Sensor Inputs
Set pointUs = 69%
Task RatesR1: [1, 5] HzR2: [10, 20] Hz
Middleware
ActuatorController
u(k)
{r(k+1)}
FCS
C. Lu, X. Wang, and C. Gill, Feedback Control Real-Time Scheduling in ORB Middleware, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'03), May 2003.
7SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
What’s New in Distributed Systems? Need to control utilization of multiple processors Utilization of different processors are coupled with each
other due to end-to-end tasksReplicating FCS on all processors does not work!
Constraints on task rates
T1
T2
T3
T11 T12 T13
P1P2 P3
8SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
EUCON: Multi-Input-Multi-Output Control
)(
)(1
kr
kr
m
ModelPredictiveController
1 min,1 max,1
min, max,
,
n m m
B R R
B R R
)(
)(1
ku
ku
n
Distributed System
(m tasks, n processors)
UtilizationMonitor
RateModulator RM
UM UM
RM
Feedback Loop
Precedence Constraints
Subtask
ControlInput
MeasuredOutput
C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): 550-561, June 2005.
9SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Control Theoretic Methodology
1. Derive a dynamic model of the controlled system
2. Design a controller
3. Analyze stability
10SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Dynamic Model: One Processor
Si: set of subtasks on Pi
cjl: estimated execution time of Til running on Pi
may not be correct gi: utilization gain of Pi
unknown ratio between actual and estimated change in utilization
models uncertainty in execution times
ijl ST
jjliii krcgkuku )1()1()(
11SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Dynamic Model: Multiple Processors
G: diagonal matrix of utilization gains F: subtask allocation matrix
models the coupling among processors fij = cjl task Tj has a subtask Tjl on processor Pi
fij = 0 if Tj has no subtask on Pi
u(k) = u(k-1) + GFr(k-1)
T1
T2
T11
P1P2
T21
T22
T3
T31
3122
2111
0
0
cc
ccF
12SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Model Predictive Control Advanced control technique for coupled MIMO control
problems with actuator constraints. Minimize a cost function over an interval in the future.
Compute an input trajectory that minimizes cost subject to actuator constraints.
Predict cost based on a system model and feedback.
Combines optimization, model-based prediction, and feedback.
13SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Model Predictive Controller
At a sampling instant Compute inputs in future sampling periods
r(k), r(k+1), ... r(k+M-1)
to minimize a cost function Cost is predicted using
(1) feedback u(k-1)
(2) approximate dynamic model Apply r(k) to the system
At the next sampling instant Shift time window and re-compute r(k+1), r(k+2), ...
r(k+M) based on feedback
14SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Model Predictive Controller in EUCON
Least Squares Solver
)(
)(1
ku
ku
n
)(
)(1
kr
kr
m
Model Predictive Controller
SystemModel
CostFunction
ReferenceTrajectory
nB
B
1 Rate
Constraints
Least Squares Solver
)(
)(1
ku
ku
n
)(
)(1
kr
kr
m
Model Predictive Controller
SystemModel
CostFunction
ReferenceTrajectory
nB
B
1 Rate
Constraints
Difference with reference trajectory
Desired trajectory for u(k) to converge to B
Constrained optimization solver
)1(
)1(1
kr
kr
m
15SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Stability Analysis Stability: system converges to equilibrium point from any
initial condition
Equilibrium point = utilization set points B
If stable, utilization of all processors converge to their set points whenever feasible
Derive stability condition tolerable range of G
tolerable variation of execution times
Stability analysis establishes analytical guarantees on utilization despite uncertainty
16SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Simulation: Stable System
0
0.2
0.4
0.6
0.8
1
0 50 100 150 200 250 300
Time (sampling period)
CP
U u
tiliz
atio
n
P1 P2 Set Point
execution time factor = 0.5(actual execution times = ½ of estimates)
17SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Simulation: Unstable System
execution time factor = 7(actual execution times = 7 times estimates)
0
0.2
0.4
0.6
0.8
1
0 100 200 300
Tim e (sam pling period)
CP
U u
tiliz
atio
n
P1 P2 Set Point
18SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Stability Stability: system converges to desired utilizations from any initial
condition
Derive stability condition tolerable range of execution times
Analytical assurance on utilizations despite uncertainty
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
0 2 4 6 8 10
CP
U u
iliz
ati
on
execution-time factor
Courier, 24Courier, 24Courier, 24Courier, 24Courier, 24Courier, 24Courier, 26Courier, 30Courier, 30Courier, 24Courier, 24Courier, 30Courier, 30
DeviationAverageSet point
Overestimation of execution times prevents oscillation
Predicted bound for stability
actual execution time / estimation
19SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
FC-ORB Middleware
Feedback lane
Remote request lanes
Priority Manager
RateModulator
ModelPredictiveController
Remote request lanes
Utilization Monitor
)(
)(
)(
3
2
1
ku
ku
kuMeasuredOutput
)(
)(
2
1
kr
krControlInput
Priority Manager
RateModulator
Utilization Monitor
Priority Manager
RateModulator
Utilization Monitor
X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.
20SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Workload Uncertainty
time-varying execution times
disturbance from periodic tasks
21SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Processor Failure
1. Norbert fails.2. move its tasks to other processors.3. reconfigure controller4. control utilization by adjusting
task rates
22SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Summary: Model Predictive Control
Robust utilization control for distributed systems Handles coupling among processors Enforce constraints on task rates Analyze tolerable range of execution times
23SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
References Centralized control: EUCON
C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): 550-561, June 2005.
Decentralized control: DEUCON X. Wang, D. Jia, C. Lu and X. Koutsoukos,
DEUCON: Decentralized End-to-End Utilization Control for Distributed Real-Time Systems, IEEE Transactions on Parallel and Distributed Systems, 18(7): 996-1009, July 2007.
Middleware: FC-ORB X. Wang, C. Lu and X. Koutsoukos,
Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.
Controllability and feasibility X. Wang, Y. Chen, C. Lu and X. Koutsoukos,
On Controllability and Feasibility of Utilization Control in Distributed Real-Time Systems, Euromicro Conference on Real-Time Systems (ECRTS'07), July 2007.
Project page: http://www.cse.wustl.edu/~lu/aqc.htm
Top Related