Fast Decentralized Power Capping for Server Clusters

Reza AzimiSchool of Engineering

Brown UniversityProvidence, RI

reza_azimi@brown.edu

Masoud BadieiSEAS

Harvard UniversityCambridge, MA

mbadieik@seas.harvard.edu

Xin ZhanSchool of Engineering

Brown UniversityProvidence, RI

xin_zhan@brown.edu

Na LiSEAS

Harvard UniversityCambridge, MA

nali@seas.harvard.edu

Sherief RedaSchool of Engineering

Brown UniversityProvidence, RI

sherief_reda@brown.edu

Abstract—Power capping is a mechanism to ensure that thepower consumption of clusters does not exceed the provisionedresources. A fast power capping method allows for a safeover-subscription of the rated power distribution devices,provides equipment protection, and enables large clusters toparticipate in demand-response programs. However, currentmethods have a slow response time with a large actuationlatency when applied across a large number of servers asthey rely on hierarchical management systems. We proposea fast decentralized power capping (DPC) technique thatreduces the actuation latency by localizing power managementat each server. The DPC method is based on a maximumthroughput optimization formulation that takes into accountthe workloads priorities as well as the capacity of circuitbreakers. Therefore, DPC significantly improves the clusterperformance compared to alternative heuristics. We implementthe proposed decentralized power management scheme on areal computing cluster. Compared to state-of-the-art hierarchi-cal methods, DPC reduces the actuation latency by 72% up to86% depending on the cluster size. In addition, DPC improvesthe system throughput performance by 16%, while using only0.02% of the available network bandwidth. We describe how tominimize the overhead of each local DPC agent to a negligibleamount. We also quantify the traffic and fault resilience of ourdecentralized power capping approach.

I. INTRODUCTION

Over the last decade, the architecture of datacenterhas evolved significantly to accommodate emerging onlineresource-intensive workloads such as software as a serviceand social networking. Modern architectures facilitate powermanagement and reduce power provisioning costs by cap-ping the peak power capacity of the datacenter. Fast powercapping is a desirable feature for three main reasons:

• Power Over-subscription: To increase the efficiencyof datacenters, many servers are normally hosted by anelectric circuit than its rated power permits [1]. Thispower over-subscription is justified by the fact that thenameplate ratings on servers are higher than the servers’actual power utilization. Moreover, rarely all serverswork at their peak powers simultaneously. In the casethat the power consumption of subscribed servers peakat the same time and exceed the circuit capacity, powermust be capped quickly to avoid tripping the circuitbreakers.

• Equipment Protection: Power capping can be usedas a safety measure to reduce power consumption ofservers when supporting equipment fails. For instance,the breakdown of a datacenter’s computer room airconditioning (CRAC) system may result in a suddentemperature increase of IT devices [2]. In this scenario,power capping can help maintain the baseline temper-ature inside the datacenter.

• Demand-Response Participation: Dynamic powercapping regulates the total power consumption underdynamic time-varying power caps. This is an importantfeature for short-term trading where energy transactionsare cleared simultaneously to match electricity supplywith demand in real time [3]. Even in a more staticday-ahead energy market, fluctuations in diurnal patternof submitted queries may necessitate a fast responsefrom power management tools to ensure an optimizedperformance for the cluster.

To facilitate the enforcement of individual server powercaps, current generation of servers are equipped with powermanagement mechanisms that can throttle the power usagevia RAPL [4] or dynamic voltage frequency scaling (DVFS)[5]. Based on these two mechanisms, different power cap-ping schemes have been developed to coordinate the powercaps dynamically across multiple servers and applications[6], [1], [7]. To coordinate power caps across servers, currenttechniques require data aggregation from all computing unitsthat include workload, application priority, and thermal foot-print information among many others. Facebook’s Dynamois an example of such hierarchical structure [6].

We observe that hierarchical power capping techniquessuch as Dynamo have a slow response time with dead timecaused by its actuation latency, which makes it inadequatefor tracing dynamical power caps at a fast time scale [1].Moreover, due to employing heuristic methods for reducingthe power of servers, power capping techniques can resultin a significant performance degradation. Our main insightis that decentralized power capping allows for localizingpower capping computation at each server which improvesthe speed and minimizes the performance degradation dueto power capping. We propose a fully decentralized powercapping (DPC) framework, where each server has a DPC

agent that locally computes its power usage such that (i) theaggregated throughput of the entire cluster is maximized,(ii) workload priorities are taken into account, and (iii) thepower usage of the cluster is capped at a certain threshold.Specifically, in our proposed framework, each DPC agentfirst transmits messages to neighboring DPC agents usingthe cluster’s network. It then updates its power consumptionin the form of a state space model, where the states are localpower consumption and power cap violation estimation,and the inputs are estimates of neighboring agents’ statestransmitted over the cluster’s network.

The DPC scheme expands and improves upon the dis-tributed power management framework we developed earlier[8]. In the proposed DPC scheme, we exploit the workloads’priorities to mitigate the performance degradation that mayresult from a server power capping. Moreover, the newDPC framework incorporates the capacity of multiple circuitbreakers into the power capping decision making process.We also focus on the implementation and practical aspects ofthe DPC method whereas in our earlier work, the algorithmperformance analysis was limited to simulations. In thiswork, we also analyze the network topology and proposea new more effective penalty function for our algorithm.

In this paper, we implement and evaluate our proposedDPC framework on an experimental computing cluster of16 Xeon-based servers. For comparison, we also implementand test three other classes of power management methods,namely a uniform power allocation, Facebook’s Dynamoalgorithm [6], and a centralized power capping method[9]. To investigate the performance of DPC framework, weevaluate a number of metrics, including (i) attained systemperformance for workloads with priority, (ii) response tovarying workload utilization, (iii) convergence rates and net-work traffic as a function of DPC communication topology,(iv) dead time to actuation, and (v) fault resilience.

The rest of this paper is organized as follows. In SectionII, we present a decentralized power capping algorithmand provide the underlying architecture. In Section III, weprovide a comprehensive set of experimental results andcompare our distributed framework with existing methodsincluding Facebook’s Dynamo algorithm. In Section IV,we review related works on power capping techniques. InSection V we discuss our results and conclude this paper.

II. PROPOSED DPC FRAMEWORK

Before we describe a mathematical formulation for theproposed power capping technique, we provide a generaloverview of the DPC framework.

The decentralized power capping has a structure as shownin Figure 1. In this structure, users submit jobs with differentpriorities to the job scheduler (SLURM)[10]. The schedulerin turn allocates jobs to the servers based on their priorities.Each server is equipped with a DPC agent with threecomponents; 1) Workload monitor (WM), 2) DPC, and 3)power controller (PC). DPC agent of server i receives thecurrent workloads’ characteristics including a throughput

PC WM

server 1 DPC

slurmd workload

𝑈1, 𝑤1 𝑝1

PC WM

server 3 DPC

slurmd workload

PC WM

server 2 DPC

slurmd workload

𝑈2, 𝑤2

PC WM

server 4 DPC

slurmd workload

𝑈3, 𝑤3 𝑈4, 𝑤4

𝑝2

𝑝3 𝑝4

users submit jobs

SLURM

Figure 1: Structure of DPC algorithm. Jobs get submitted toSLURM and workload monitor (WM) get the workload informationfrom SLURM daemon (slurmd). DPC gets the workload informa-tion from WM and actuate the power cap using the power controller(PC).

function Ui(·) and the workload’s priority information wi

from the workload monitor. It then solves the optimizationproblem formulated in Subsection II-A. The solution of theoptimization provides the local power cap of server i, pi,for its power controller module to apply.

Note that our decentralized power capping framework isiterative. At each iteration, each DPC agent computes localdecision variables and communicates their local informationwith their neighbors. We take into account the physicalpower limits of the Circuit Breakers (CBs) to avoid circuittripping. Therefore, the DPC power capping frameworkallows form power over-subscription. In other words, theplanned peak power demand can be higher than what issupplied, which improves the efficiency of cluster.

A. Problem Formulation

The DPC algorithm is based on a weighted sum through-put maximization problem subject to (i) a power cap R0 onthe cluster power consumption, (ii) a power cap Rk, k =1, 2, · · · , r for each circuit breaker, and (iii) the powerconstraint of each server. More specifically, we consider aheterogeneous system where the i-th active server in thecluster of n-nodes N = {1, 2, · · · , n} has a utility functionUi(pi), where Pmin

i ≤ pi ≤ Pmaxi , where Pmin

i and Pmaxi

represent the minimum and maximum power consumptionof the i-th server. Following the method of [11], we alsoconsider a set of weight factors wi ∈ R+, i = 1, 2, · · · , nthat determine the workload priority, i.e., a large weightcorresponds to a high priority workload. Note that thethroughput function Ui(·) and the weight wi > 0 of eachserver are not fixed. Rather, they change depending onthe type of the workload being processed by each server.However, in the design of DPC, we formulate the sumthroughput maximization problem with a set of fixed utilityfunctions for servers. Upon change in the workload ofeach server, DPC re-adjusts the optimal power consumption

Update actions

p2

Update states Engage

local controller

Actuation

Exchange messages

e12 e21 p1 e2

e1

Optimization

p1

p2 e12, p1 e21, p2

Figure 2: The main steps of DPC algorithm.

for the new workload configuration. Also for all types ofpractical workloads under consideration, the utility functionsof all servers are concave, as verified by our results inSection III.

We also consider a set of r circuit breakers {CBk}rk=1

that form a cover of the cluster of n servers, i.e., each of nservers is mapped to one or more of the circuit breakers. Tosimplify our formulation, we define CB0 := N as the setof all servers in the cluster.

To develop the DPC algorithm, we formulate the follow-ing sum throughput optimization problem:

maxp1,p2,··· ,pn

n∑i=1

wiUi(pi) (1a)

subject to :∑

i∈CBk

pi ≤ Rk, k = 0, 1, · · · , r, (1b)

Pmini ≤ pi ≤ Pmax

i , i = 1, 2, · · · , n. (1c)

B. DPC Algorithmic Construction

Herein, we outline the DPC algorithm. We defer the moretechnical detail of our derivations to Appendix A.

The main objective of DPC is to provide a decentralizedprocedure for solving the optimization problem in Eqs. (1a)-(1c). However, the challenge in designing such a procedureis that the power usages of servers are coupled through thepower capping constraints in Eq. (1b). We decouple the op-timization problem (1a)-(1c) by augmenting the local utilityUi(pi) of each server i ∈ N with a penalty function definedon the estimation terms. Maximizing the augmented utilityat each server guarantees that the power cap constraints inEq. (1b) are satisfied.

For each power capping constraint k = 0, 1, . . . , r inEq. (1b) in DPC algorithm, we define an estimate (belief)variable eki for each server i. Specifically, eki (t) ≤ 0 meansthat the ith server’s estimate from the kth power cappingconstraint in Eq. (1b) is satisfied, whereas eki (t) ≥ 0indicates a constraint violation proportional to the magnitudeof eki (t). Servers communicate these estimation terms withtheir neighboring agents to obtain accurate values of theconstraint violation. The main steps of DPC algorithms aredepicted in Figure 2. Each server maintains a vector of statevariables

(pi(t),

{eki (t)

}rk=0

)where pi(t) is the power usage

and eki (t) is the estimation term for the kth power cappingconstraint. Our algorithm is iterative and at each iteration tand for each server i ∈ N , DPC has the following steps (seeFig. 2): REPEAT STEPS I-III

i) Update actions: compute a vector of actions,(p̂i(t),

{eki→j(t)

}k∈{0,1,··· ,r}j∈N(i)

), where p̂i(t) is the

change of power usage and eki→j(t) is the messagepassed from agent i to its neighbor j; Here N(i) ⊂ Nis the set of all neighbors of the agent i. To computethe actions, we apply a gradient ascent method on thelocal augmented utility function which is a combinationof local utility function Ui(pi) and a penalty functiondefined on the estimation errors eki , k ∈ {0, 1, · · · , r}.

ii) Exchange messages: pass the messages eki→j(t) andekj→i(t) to (resp. from) neighbors j ∈ N(i).

iii) Update states: update the state variables(pi(t),

{eki (t)

}rk=0

)based on the action vector

computed in the previous step and the receivedmessages.

v) Engage local controller: actuate the power cap to theserver.

We have formalized the above description in the formof the pseudo-code in Algorithm 1. See Appendix A fordetails of our derivations. To initialize the algorithm, we setthe power usage pi(0) = Pmin

i . Moreover, when i ∈ CBk

for k = 0, 1, · · · , r the initial estimate terms are given by

eki (0) =1

|CBk|

Rk −∑

j∈CBk

Pminj

, (2)

and otherwise,

eki (0) = 0. (3)

The algorithm we proposed requires choosing free param-eters including the step size ε and µ. These parameters mustbe selected based on the cluster size and convergence speed.For example, while choosing a small value for the step sizeε guarantees that the solution of DPC is sufficiently close tothe optimal solution of Eqs. (1a)-(1b), it results in a slowerconvergence rate and thus a longer runtime for DPC. For allthe experiments in this paper, we set ε = 4 and µ = .01.

C. DPC Implementation Choices

We conclude this section by discussing a few aspects ofDPC implementation on the cluster.

Choice of Network: DPC is a decentralized method andthus naturally utilizes datacenter’s network infrastructureto establish connection between DPC agents. We analyzethe impact of communication topology on DPC resource

Algorithm 1 DPC: DECENTRALIZED POWER CAPPING

1: Initialize µ > 0 and a constant step size ε > 0. Choosepi(0) = Pmin

i and eki (0) as in Eq. (2).AT TIME ITERATION t SERVER i:

2: Compute the action p̂i(t) according to,

p̂i(t) = ε∂Ui(pi(t))

∂p̂i(t)+ εµ

∑k∈M(i)

max{0, eki (t)};

Let M(i) ⊆ {0} ∪ {1 · · · r} be the subset of CBs thatthe server i is subscribed to.

3: if p̂i(t) + pi(t) ≤ Pmini then p̂i(t) = Pmin

i − pi(t)4: if p̂i(t) + pi(t) ≥ Pmax

i then p̂i(t) = Pmaxi − pi(t)

5: Compute the action eki→j(t) according to,

eki→j(t) = µε(eki (t− 1)− ekj (t− 1)

)6: Send eki→j(t) and receive ekj→i(t) to (resp. from) all

neighbors j ∈ N(i).7: Update the states pki (t+1), eki (t+1) for all i ∈ N , andk ∈ {0, 1, · · · , r} according to

pi(t+ 1) = pi(t) + p̂i(t),

eki (t+ 1) = eki (t) + p̂i(t)1{i∈CBk}

+∑

j∈N(i)

(ekj→i(t)− eki→j(t)),

where 1{i∈CBk} = 1 if i ∈ CBk and 1{i∈CBk} = 0otherwise.

8: Output: the power consumption pi for all i ∈ N .

utilization and convergence. Specifically, we consider Watts-Strogatz model [12] to generate connectivity networks withsmall-world properties [12]. Small world networks are char-acterized by the property that they are clustered locally andhas a small separation globally. That is, most servers can bereached with only few hops.

Watts-Strogatz model generates small-world networkswith two structural features, namely clustering and averagepath length. These features are captured by two parameters:the mean degree k and a parameter β that interpolatesbetween a lattice (β = 0) and a random graph (β = 1).

In this model, we fix β = 0 to obtain regular graphsand select various mean degrees k. Figure 3 illustrates thegenerated graphs from this model with N = 16 servers. Fork = 2, we obtain the ring network, where each DPC agentis connected with two neighbors. For k = 16 we obtain acomplete graph, where each agent is connected to all otheragents in the given cluster.

For each underlying graph, we execute the decentralizedDPC algorithm on our cluster where the total power capis R0 = 2.6kW and we assume to have two CBs, eachsupplying power to 8 of our 16 server cluster whereR1 = R2 = 1.6kW. We measure various performancemetrics such as 1) the network bandwidth (BW) utilization

(a) (b) (c) (d)

Figure 3: Generated graphs for DPCs agent where each vertex is aDPC agent and each edge indicates two agents that are neighbors.Graphs are generated from Watts-Strogatz model where each withβ = 0 and mean degree (a) k = 4 (b) k = 8 (c) k = 12 and (d)k = 16.

Average # iter node BW cluster BW communication computationdegree k (kB/s) (kB/s) (ms) (ms)

16 547 1880.8 27644 331.0 1.914 550 1200.3 18021 314.0 2.212 571 1064.8 15990 295.1 2.410 599 934.7 14038 274.6 2.4

8 636 790.2 11875 233.0 2.36 796 739.5 11109 247.9 3.14 1108 685.6 10287 253.6 3.52 3234 993.0 14917 634.6 10.6

Table I: The effect of changing topologies on DPC.

per node, 2) the total cluster BW, 3) communication timeper node, and 4) the computation time per node. Theresults of these measurements are summarized in Table I.From the table we observe that the number of iterationsrequired for DPC to converge decreases as k increases. Thisobservation is intuitive as for large k, the total number ofedges increases and thus the consensus among servers canbe reached faster. This in turn reduces the number of DPC’siterations required to converge to the optimal solution.However, increasing the edges increases the number ofmessages and network utilization at each iteration of thealgorithm. As k increases, the network utilization decreasesuntil the turning point k = 4 where the trend reverses.Based on this data, we use a network topology with k = 4in all the following experiments for DPC.

Fault Tolerance: Server failure occurs frequently in largescale clusters. In the case of failure of a DPC agent, theagent’s socket1 will be closed and neighbors can send aquery to the failed server and restart its DPC agent. If theDPC agent stops responding due to a more severe issue,two scenarios may occur: 1) the socket gets closed, 2) thecommunication time between agents exceeds a specifiedtimeout threshold. In either case, the neighbors take thisoccurrence as a failure. After agents identify a failurein their neighborhood, they take the failed server out ofthe neighborhood list. Moreover, each agent continues theoptimization process on a different connectivity graph thatexcludes the failed server. In the network generated byWatts-Strogatz model with β = 0, the connectivity of thenetwork is maintained as long as the number of failed agentsin the vicinity of a agent is smaller than the average degreek. After the failed server is fixed, its corresponding DPCagent communicates with its neighbors so that it will be

1The communication between agents is established using the standardLinux socket interface.

included again to their neighborhood list.In Subsection II-B, we explained how DPC algorithm

adopts if the total power cap changes. There are twoscenarios under which a discrepancy between the totalpower cap and the algorithm beliefs of total power capcan occur, namely (i) when there is a change in the powercap, or (ii) when there is a server failure. In practice,this discrepancy can be injected as an error term to anyrandomly chosen server. The error propagates to all serversdue to the consensus step in the decentralized algorithm.Note since the server is chosen arbitrarily, if it is notresponsive, another random server can be chosen.

Computation/Communication Overhead: Anydecentralized power management method consumes bothcommunication and computational resources to operate.Therefore, we propose a ‘sleeping’ mechanism for eachDPC agent to minimize the impact of DPC on the cluster’sperformance. Specifically, each DPC agent can execute asmany as five thousand iterations of DPC per second. Afterconvergence, DPC’s speed can be reduced by sleepingin between iterations. This in turn reduces both networkutilization and CPU cycles required for optimization. Inour DPC implementation, the sleeping cycle is enforcedby updating DPC at the rate of ten iterations per second.This rate is sufficient to effectively capture the changes inworkload characteristics and the power cap. To minimizecache contentions and the overhead of scheduling, wealways fix the core affinity of the DPC agent to one fixedcore.

Comparison with other power capping schemes: Wecompare DPC to other existing power capping techniques.In particular, we implement three other methods:

Dynamo: Dynamo is the power management system usedby Facebook datacenters [6]. It uses the same hierarchyas the power distribution network, where the lowest levelof controllers, called leaf controllers, are associated with agroup of servers. A leaf controller uses a heuristic high-bucket-first method to determine the amount of power thatmust be reduced from each server within the same prioritygroup. In this framework, priorities of workloads are deter-mined based on the performance degradation that they incurunder power capping.

Compared to the DPC algorithm, Facebook’s Dynamo hasa larger latency for two reasons:

1) DPC agents continually read the power consumptionand locally estimate power cap violations. Therefore,DPC agents respond to variations in the workloadand power caps almost instantly. Notice that the timerequired to reach the consensus in our decentralizedframework is much smaller than the timescale of varia-tions in workload and power caps. In contrast, Dynamoscheme has a large latency due to its hierarchicaldesign. In particular, the leaf controllers in Dynamo

scheme compute the power caps and broadcast them tolocal agents of servers to actuate. However, to obtainstable power readings from servers and to computethe power caps, the leaf controllers must measure thepower consumption only after they reach their steadystates. As a result, if there is a sudden change in systemvariables after power consumption are measured, thesechanges are not taken into account in the power capsuntil the next power reading cycle.

2) By localizing the power cap computations at eachserver, the computation and actuation of power capcan be overlapped concurrently. In contrast, Dynamorequires stable power readings from servers, and thuspower cap computation and actuation are carried inseparate phases.

We also note that DPC improves the system throughputperformance compared to heuristic methods such as Dynamobecause DPC incorporates workload characteristics and theirpriorities to maximize the system throughput under specifiedpower caps. Due to the fact that the power cap of eachserver is determined based on an optimization problem, theDPC framework reduces the negative impact of the powercapping on the system throughput performance. In compari-son, heuristic power capping methods, such as Dynamo, mayadversely affect the system performance.

Uniform Power Allocation: In this scheme, the total powerbudget is evenly divided among active servers, regardless oftheir workloads’ priorities.

Centralized Method: In this method, the optimizationproblem in Eqs. (1a)-(1b) is solved in a centralized manneron a single server [9]. In particular, the centralized coordi-nator aggregates servers’ local information and solves theoptimization problem in Eqs. (1a)-(1b) using off-the-shelfsoftware packages such as CVX solver [13]. The centralizedcoordinator then broadcasts the local power consumption toservers. While a centralized method ensures the maximumthroughput, it scales poorly for large clusters of thousandsservers and cannot be employed in practice.

III. EVALUATION

A. Experimental SetupIn this section, we report the experimental setup of DPC

on our cluster. Our capping software is available online. 2

Infrastructure: The experimental cluster consists of16 Dell PowerEdge C1100 servers, where each server hastwo Xeon quad-core processors, 40 GB of memory, and a10 Gbe network controller. All servers have Ubuntu 12.4and are connected with top-of-the-rack Mellanox SX1012switch. Performance counter values are collected from allservers using the perfmon2. We use SLURM as our clusterjob scheduler [10]. Servers at full loads can consume about220 Watts. We instrument each server with a power meter

2The DPC implementation codes can be found in our research lab githubrepository. https://github.com/scale-lab/DPC.

that reads the server’s power consumption at 10 Hz.

Workloads: We use two types of workloads. For batchprocessing applications, we use the HPC Challenge (HPCC)and NAS parallel benchmark (NPB) suites to select ourHPC workloads [14], [15]. In particular, in our experimentswe focus on a mix of known CPU-bound workloads, e.g.,hpl from HPCC and ep from NPB, and memory-boundworkloads, e.g., mg from NPB and RA from HPCC, asthey represent two ends of the workload spectrum. We useclass size C for the workloads selected from NPB whichapproximately takes a minute to complete in the absenceof power capping the servers. For workloads from HPCC,we select a matrix size with the same runtime. For latency-sensitive transactional workloads, we use MediaWiki 1.22.6[16] which is a 14 GB copy of the English version ofWikipedia on a MySQL database. We use two clients togenerate loads using Siege 3.1 [17].

Performance metric: We use retired jobs per hour as thethroughput metric. To quantify the total throughput of ourcluster of 16 servers, we calculate the total number of jobsretired per hour during the experiment. For total power, wesum the measured power of all 16 servers. For the webserving, we evaluate at the tail latency (99th percentile) asour performance metric.

Power controller (PC): To enforce the local power target ateach servers, we implement a software feedback controllersimilar to Pack & Cap [18], where the controller adjusts thenumber of active cores according to the difference betweenthe power target and the current power consumption. Toavoid an oscillatory behavior around the power target, weconsider a buffer area of two percent below the power targetin which the controller remains idle. When the differencebetween the power target and current power consumption ispositive and more than two percent, the controller increasesthe number of active cores. Similarly, a negative differenceresults in a decrease in the number of active cores. In ourexperiments, we engage the controller every 200 ms.

For a fair comparison between different methods, all theexperiments and reported results in the paper, includingthose of Dynamo, are based on our core-affinity powercontroller. Nevertheless, both the RAPL controller in theDynamo scheme [6] and the core-affinity power controllerwe use in this paper take approximately two seconds tostabilize. Therefore, we followe the three seconds samplingrule that is recommended in [6] for our implementation ofDynamo’s leaf controllers.

Workload monitor (WM): The main task of WM is todetermine the workload priorities and utility function forDPC algorithm to solve optimization in Eqs. (1a)-(1c). WMalso monitors different resource utilization, performancecounters, and workloads information. All the information

rate 1 2 3 6 12 20 30 60(per minute)

overhead 0.1 0.1 0.4 0.6 0.7 0.9 1.7 2.0(%)

Table II: The effect of update rate of DPC on the overhead.

gets logged with the time-stamp for further analysis.

Overhead: As described earlier, any decentralized powermanagement framework must be computationally inexpen-sive to minimize the performance degradation. To measurethe overhead of DPC algorithm, we execute it withoutenforcing the power caps. We then calculate the overheadby comparing the results with the case that the cluster doesnot use any optimization algorithm and thus no resources isallocated to DPC.

The overhead of running the DPC agent on each nodeis determined by how often DPC needs to be run at fullspeed. DPC runs at full speed when power caps are neededto be re-calculated. Re-calculating the power caps are onlyrequired when the total power cap or the configuration ofworkloads changes. After re-calculations, DPC agents runin the background to monitor changes but use sleepingcycles to reduce overhead. To quantify the overhead ofDPC algorithm, we varied the rate at which DPC needs tocalculate power caps from once per minute to every secondand recorded the system throughput for three differentruns. Table II shows the average overhead of DPC as thefunction of re-calculation rate. In our experiments, thetypical re-calculation rate is 1-2 per minute, which leads tonegligible overhead.

We considered the following three main experiments.1) Prioritized Workloads: In the first set of experiments

we compare DPC and Dynamo focusing only on theworkloads’ priorities. We show the advantage of DPCover Facebook’s Dynamo.

2) Utility Maximization: In the second set of experimentswe consider the case when the throughput utility func-tions are known, and we show the performance im-provement over heuristics that can be attained throughsolving the utility maximization problem.

3) Scalability and Fault Tolerance: In the third setof experiments, we evaluate the advantage of DPCcompared to the centralized method and Dynamo interms of scalability and fault-tolerance.

B. Prioritized Workloads

To demonstrate the performance gain achieved by minimiz-ing the latency in DPC, we consider a scenario where twotypes of workloads are served in the cluster, namely a batchof HPC jobs running on half of the servers, and the otherhalf are web servers. We take the latter as the cluster’s pri-mary workload. Because web queries are latency-sensitive,a coarse power capping on this type of services can resultin violation of service level agreement (SLA). Therefore,

500 510 520 530 540 550 560 570

time(s)

406080

100120140

rps

a. primary workload

500 510 520 530 540 550 560 570

time(s)

2.42.62.8

pow

er (

kW) b. total power consumption

DPC Dynamo total cap

500 510 520 530 540 550 560 570

time(s)

1.2

1.4

1.6

pow

er (

kW)

c. primary and secondary workload power consumptionDPC secondary

Dynamo secondary

DPC primary

Dynamo primary

Figure 4: Detailed comparison between DPC and Dynamo for aminute of experiment. Panel (a): load on the web servers, Panel(b): total power consumption of the cluster, and Panel (c): powerconsumption of each sub-cluster running the primary (web servers)and secondary (batch jobs) workload for each method.

to meet the total power cap and the provisioned resourceconstraints, the power consumption of secondary workloadmust be capped. Accordingly, the objective is to satisfythe power constraints and maximize the throughput of thesecondary workload.

To attain this objective, we prioritize the web queriesin both Dynamo and DPC to avoid power capping. Wegenerate approximately 40 to 120 queries per second to theweb servers while a batch of HPC jobs are processed onthe rest of the clusters as the secondary workload. In theimplementation of Dynamo, we assume that all servers andtwo CBs are assigned to a single leaf controller.

The experiment is implemented for an hour duration,where we fix the total power cap to 2.7kW and assume all16 servers are protected by two CBs with power capacityof R1 = R2 = 1.6kW . Figure 4.(a) shows the load on theweb servers as the primary workload. From Figure 4.(b), weobserve that both methods successfully cap the total powerconsumption. Moreover, from Figure 4.(c), we observe thevariations in the primary workload’s power consumptiondue to changes in the number of submitted queries. LocalDPC agents are able to estimate cap violations in a fastdecentralized way, and thus they provide a faster reactiontime to the changes in the power cap profile of the primaryworkload.

DPC agents are running on all of the 16 servers and poweris divided between the primary and secondary workloads.The primary workload always has the maximum number ofactive cores due to a higher priority. As explained earlier,power controller (PC) constantly monitors the power target

500 520 540 560

time(s)

0

20

40

60

80

activ

e co

res

a. active cores breakdown

DPC secondary

Dynamo secondary

DPC primary

Dynamo primary

1000 2000 3000

time(s)

103

104

105

netw

ork

(B/s

)

b. network utilization

DPC Dynamo

Figure 5: Panel (a): DPC and Dynamo’s active number of coresfor each type of workload in a minute of experiment. Panel (b):Network utilization of the Dynamo’s leaf controller and averageserver for DPC through the experiment.

and power consumption and sets Pmax accordingly. Whenthe primary workload power consumption decreases, PCupdates Pmax on each node in the DPC algorithm. Theaffected nodes in DPC algorithm updates their power capsand using the messages communicated between the agents,power passes from the primary to the secondary workload.When the primary workload’s power consumption increases,again PC updates Pmax and because of higher priorities,power passes from the secondary to the primary workload.

Due to different structural design, DPC and Dynamo havedifferent response times. As mentioned earlier, Dynamomust wait for the local power controllers to stabilize tocompute the power caps, and actuation delay of the con-trollers determine how fast Dynamo can sample. In contrast,DPC agents estimate power cap violations independently ina decentralized way. In addition, DPC overlaps power capcalculation with the actuation because both are done locally.This fast reaction time in turn results in a more efficientpower allocation to the secondary workload.

Figure 5.(a) shows the active number of cores for eachworkload. In both DPC and Dynamo methods, all theavailable cores are allocated on the eight servers that areprocessing the primary workload. However, due to a higherpower target, DPC allocates more cores to the secondaryworkload. During an hour of experiment, DPC provides16% improvement in the secondary job’s throughputcompared to Dynamo. The response-time tail latency (99thpercentile) of the primary workload is unaffected in bothmethods since both methods allocate maximum number ofactive cores to avoid any performance degradation to thelatency sensitive workload; see Figure 5.(a). From Figure5.(b) we observe that the network utilization of Dynamo’sleaf controller and the average network utilization of allnodes for DPC through the experiment. Although DPC hasa higher network utilization compared to Dynamo, at itspeak network utilization, DPC consumes only 0.02% of theavailable bandwidth of a 10Gb Ethernet network controller.Thus the DPC network overhead is negligible.

140 160 180 200 220

power(w)

0

2

4

6

8

norm

aliz

ed th

roug

hput

hplmgepRA

Figure 6: Modeled (lines) and observed (markers) normalizedthroughput as the function of power consumption.

C. Utility Function Experiments

In this section, we consider the case where the utilityfunctions are known. We compare solutions like DPC andthe centralized to other heuristic methods such as uniformand Dynamo. We consider the behavior of these methods intwo cases of a dynamic power caps and a dynamic load.

Utility functions determine the relationship between thethroughput and power consumption using empirical data.There are many studies on characterizing this relationship[19], [20], [9]. We adapt a quadratic form to characterizethe relationship between throughput and power consumption.We assume that all the workloads and their correspondingutility functions are known a prior. The workload moni-tor (WM) receives the current workload information fromSLURM daemon (slurmd) running on each node and selectsthe correct utility function from a bank of known quadraticfunctions. Slurmd is the daemon of SLURM which mon-itors current jobs on the server and accepts, launches, andterminates running jobs upon request.

Figure 6 shows the normalized throughput function ofworkloads selected from the HPCC and NPB benchmarks,where we fix the server power cap at various values andmeasure the average throughput for each workload. We alsoobserve from Figure 6 that for all practical workloads inthis paper, the throughput is a concave function of the serverpower. Throughout this experiment we assume DPC and cen-tralized method only uses the throughput/power relationshipas utility function. As Figure 6 shows, the throughput ofCPU-bound workloads (hpl and ep) are affected more bypower capping. To use the only knob that Dynamo offersto do workload-aware power capping, we assumed higherpriority for CPU-bound workloads. Again for Dynamo weassume that all servers and two CBs are assigned to a singleleaf controller.

1. Dynamic Caps: We now consider a scenario where thetotal power cap must be reduced from 2.8 to 2.5 kW dueto a failure in the cooling unit. Throughout the experiment,our 16-node cluster is fully utilized with a mix of differentworkloads. We again assume all the 16 servers are protectedby two CBs where each can handle 1.6 kW. Figure 7.(a)

200 400 600 800 1000 1200 1400 1600 1800time(s)

2.5

3

pow

er (

kW) a. total power consumption

DPC Dynamo centralized uniform total cap

200 400 600 800 1000 1200 1400 1600 1800

time(s)

0

5

jobs

b. running jobs

mghpl

epRA

200 400 600 800 1000 1200 1400 1600 1800time(s)

100

105

1010

netw

ork

(B/s

) c. network utilizationDPC Dynamo centralized uniform

Figure 7: Comparison between DPC, Dynamo, centralized, and auniform power capping under a dynamic power cap.

shows how each method reacts to changes in the totalpower cap and the mix of workload as shown in Figure7.(b). While all the power capping methods can successfullycap the total power under the given power cap, DPC andthe centralized methods consistently provide 8% higher jobthroughput over uniform. Further, Dynamo can only improvethe jobs throughput by 5.2% compared to the uniformpower capping. The corresponding network utilization isdepicted in Figure 7.(c) with a logarithm scale in bytesper second. The centralized approach, Dynamo, and uniformpower capping methods make a negligible use of the networkinfrastructure. DPC engages in full capacity only when thepower cap or the mixture of workloads change which can beobserved as the spikes in Figure 7.(c). We also observe thatDPC slows down after convergence, which in turn reducesthe communication rate of DPC and minimizes the overhead.Although DPC has the highest network utilization among allthe considered methods due to its decentralized design, ituses only 0.1% of the available network bandwidth of eachserver’s 10Gbe network at its peak. Hence, from a practicalpoint of view, the network overhead in DPC is negligible.

2. Dynamic Load : In this experiment, we evaluate eachmethod under dynamic workloads. In this experiment, thetotal power cap is set to 2.8 kW and all the 16 serversare protected by two CBs each of 1.6 kW capacity. Thefirst batch of workloads is submitted to the cluster, wherefive jobs are running on the cluster at the beginning ofthe experiment. We choose the workloads to be a mix ofmemory and CPU-bounds applications. In the beginning ofthis experiment, the jobs occupy five servers in the clusterand the remaining servers are idle. Approximately nineminutes in the experiment, the second batch of jobs (a mixof memory and CPU-bounds applications) are submitted tothe cluster such that all the servers are fully utilized. Wedemonstrate the total power consumption and correspondingjob status in Figure 8. When only a few workloads arerunning on the cluster, a large amount of power is available

200 400 600 800 1000

time(s)

2

2.5

3

pow

er (

kW) a. total power consumption

DPCDynamocentralized

uniformtotal capno capping

200 400 600 800 1000

time(s)

0

10

jobs

b. total jobs running

Figure 8: Power and number of jobs running on the cluster in thedynamic-load experiment.

560

610

660

710

uniform Dynamo DPC centralized

job

th

rou

ghp

ut

(j

ob

s/h

ou

r)

dynamic-cap-exp

dynamic-load-exp

Figure 9: job throughput of the two experiments.

to be allocated to utilized servers. However, when more jobsare submitted under a restrictive power cap, the power capof each server needs to be computed to maximize the clusterthroughput performance. The centralized and decentralizedmethods find the optimal power caps for each server basedon the workload characteristics. Therefore, these two meth-ods provide a better job throughput compared to Dynamoand uniform power capping. More precisely, the centralizedand DPC outperform the uniform power capping by 7%. Incomparison, Dynamo only provides 4% improvement overuniform power capping. Similar to the previous experiment,DPC has the most network utilization which at its peak isabout 0.1% of the available bandwidth for each server.

Figure 9 shows the jobs throughput performance in thetwo experiments with a dynamic power cap and dynamicworkloads. We observe that DPC outperforms the uniformpower allocation and Dynamo schemes by 7% and by3%, respectively. To obtain these results, experiments arerepeated three times, which we observed 0.2% standarddeviation between three trials.

D. Scalability and Fault Tolerance1. Scalability: In this section, we compare DPC with thecentralized and Dynamo methods in terms of the scalibil-ity. The actuation latency consists of three parts, (i) thecomputation time, (ii) the communication time, and (iii)the controller actuation time. In the following experiments,we are interested in the computation and communicationtime, i.e., the time it takes for a power capping method tocompute the power caps and set it as the power target for

the power controller. We exclude the actuation time of thepower controller in the reported results as it adds the sameamount of delay to all methods we consider. Based on thedata from measurements on the real-world 16-server cluster,we compute the latency of a cluster with 4096 servers.

To estimate the computation and communication timeof DPC for large number of nodes, we first used Matlabsimulations to compute the number of iterations required forDPC convergence in large clusters. The computation time ofeach iteration is measured from our cluster. We use the utilityfunctions of the workloads from our cluster. To consider therandomness effect, we average the DPC convergence for10 different trials. Communication time of each iterationis the time needed to send and receive messages fromneighbors and we measure it again from our cluster. Then,we estimate the computation and communication time ofDPC by multiplying the computation and communicationtime of each iteration and the number of iterations.

The computation time of the centralized method is theruntime of the CVX solver which we measure from oursystem. To measure the communication time of the central-ized method, we measure the time needed for sending andreceiving messages from a centralized coordinator, where thelocal power cap of each server is computed, to the servers.As Dynamo reported [6], Dynamo’s leaf controller that canhandle up to a thousand nodes has the pulling cycle of3 seconds. For more than a thousands nodes, upper-levelcontrollers are needed which have pulling cycle of 9 seconds.

The latency of different power capping methods are shownin Figure 10 in logarithm scale. The latency of centralizedmethod grows cubically, as centralized method uses CVXquadratic programming solver that suffers from computa-tional complexity of approximately O(n3) [21]. Dynamo’slatency is the function of a hierarchy depth and DPC’slatency is the function convergence iteration and as ourresults shows it grows linearly.

Dynamo has smaller actuation time compared to thecentralized method; however, it lacks performance efficiencyas we see throughout experiments. DPC is both the fastestsolution and delivers the optimal performance. Centralizedmethod solves the optimization like DPC but it cannot beused in practice because of the large actuation latency.

1E+00

1E+01

1E+02

1E+03

1E+04

1E+05

1E+06

256 512 1024 2048 4096

reac

tio

n t

ime

(ms)

cluster size (#servers)

DPC Dynamo centralized

Figure 10: The power capping reaction time of each method.

DPC also has a negligible network overhead for largerclusters. The messages between each pair of servers ateach iteration of DPC has the same length. Because thenumber of neighbors is fixed, network utilization is onlya function of number of convergence iterations. Again weused Matlab to compute the number of iterations requiredfor DPC convergence in larger clusters. To calculate thenetwork utilization for larger clusters, we measured networkutilization per iteration from our cluster and multiply itby the number of iterations needed for larger cluster toconverge. Although DPC utilize the network more thanthe centralized method and Dynamo, it only occupies uptoless than 1% of the available network bandwidth. Thus thenetwork overhead is negligible.

2. Fault Tolerance: Another advantage of the decentral-ized method is that it does not have a single point of failureby its nature. In this paper, we focus on server failures anddo not consider the correlated failures due to network switchfailures and power outages. A failed node is assumed to betaken out of the cluster and consumes no power from thepower cap throughout the experiment.

We choose k = 4 for the topology of DPC, where eachnode is connected to 4 other nodes. In this case, a nodewill still remain connected, even if 3 of its neighbors fail.In our DPC agent implementation, when one node fails,its neighbors will notice the failed node due to lack ofresponse and remove the node out of their neighbors list. Theoptimization will run with the active nodes since they arestill connected. The difference between the total power targetand actual power that the cluster is consuming is injectedas an error to one of the node estimation as explained inSubsection II-C. One of the nice features of DPC is that theerror estimation can be injected to any of the active nodes.This avoids a single point of failure and reinforces our fullydecentralized claim for DPC.

In this experiment, the total power cap was fixed at 2.8kW and three neighbor servers fail approximately at 30th,40th and 50th seconds of the experiment. Our power monitorinfrastructure monitors the total power consumption of thecluster and power cap. In case of an error, it pings serversand injects the error to one of the working servers for DPC.Figure 11.(a) shows the power consumption of DPC methodsduring the nodes failure experiment. Figure 11.(b) shows theservers network utilization and it shows DPC re-calculatesthe power caps after each server failure to use the powerbudget of the failed server for other running servers. Aftereach failure, power consumption of the cluster undershootsbecause the failed servers are taking out of the cluster. DPCcompensates by allocating the failed server’s budget to otheractive servers. After the third failure, all remaining thirteenservers are uncapped since the total power cap is above whatis consumed by the cluster.

IV. RELATED WORK

The problem of coordinated power capping in large scalecomputing clusters and its effect on different classes of

20 30 40 50 60 70

time(s)

2.6

2.8

3

pow

er (

kW)

a. total power consumption

DPC total cap

20 30 40 50 60 70

time(s)

104

106

108

netw

ork

(B/s

) b. network utilization

Figure 11: Total power consumption of the cluster and the averagenetwork utilization in the case of servers failure.

workloads has been studied extensively [6], [3], [5], [1], [7].Among different types of workloads, however, power cap-ping on online data intensive services (OLDI) has receiveda great deal of attention [22], [5]. This is due to the fact thatOLDI services exhibit a wide dynamic load range and thuswould benefit significantly from ‘energy proportionality’wherein servers consume power in proportion to their loads.

An important issue in power capping techniques is toselect an appropriate power cap in order to maximize thenumber of hosted servers in a datacenter [23]. A commonpractice is to ensure that the peak power never exceedsthe branch circuits capacity as it causes tripping in circuitbreaker (CB). However, this approach is overly conserva-tive. By carefully analyzing the tripping characteristics ofa typical CB, the system performance can be optimizedaggressively through power over-subscription without therisk of tripping CB. It must be mentioned nevertheless thatover-subscribing power at the rack level is unsafe, as notedby Fan. et al. [24], given that large Internet services arecapable of driving hundreds of servers to high-activity levelssimultaneously.

The main challenge in implementing coordinated powercapping is to control the latency. A detailed examinationof latency in hierarchical models [1] shows that even asmall actuation latency, i.e., the latency in control knobs,can cause instability in hierarchical, feedback-loop powercapping models. The main difficulty is due to the factthat when feedback loops operate on multiple layers in thehierarchy, stability conditions demand that the lower layercontrol loop must converge before an upper layer loop canbegin the next iteration. Despite this observation, recently ahierarchical structure for power capping, dubbed as Dynamo,has been proposed [6] and implemented on Facebook’sdatacenter which we compared our work against thoroughly.

V. CONCLUSIONS

Power capping techniques provide a better efficiency indatacenters by limiting the aggregate power of servers tothe branch circuit capacity. As a result, power capping isdeemed as an important part of modern datacenters. In this

paper, we proposed DPC, a fast power capping methodthat maximizes the throughput of computer clusters in afully decentralized manner. In contrast to most existingpower capping solutions that rely on simplistic heuristics,we used distributed optimization techniques that allow eachserver to compute its power cap locally. We showed thatDPC provides a faster power capping method compared toFacebook’s Dynamo heuristic. It also improves the systemthroughput by 16% compared to Dynamo while using only0.02% of the available network bandwidth. The powercapping framework we proposed also takes into accountworkloads priority by augmenting the throughput functionsof each workload with a multiplicative factor. Additionally,we showed that when the utility function at each server isknown, DPC provides the optimal performance in terms ofjobs per hour metric.

ACKNOWLEDGMENT

The research of Reza Azimi, Xin Zhan, and Sherief Reda ispartially supported by NSF grants 1305148 and 1438958. Theresearch of Masoud Badiei and Na Li is supported by IBM PhDFellowship and NSF CAREER grant 1553407.

REFERENCES

[1] A. A. Bhattacharya, D. Culler, A. Kansal, S. Govindan, andS. Sankar, “The need for speed and stability in data centerpower capping,” in Sustainable Computing: Informatics andSystems, vol. 3, no. 3. Elsevier, 2013, pp. 183–193.

[2] R. Azimi, X. Zhan, and S. Reda, “Thermal-aware layout plan-ning for heterogeneous datacenters,” in ACM Proceedings ofthe 2014 International Symposium on Low Power Electronicsand Design, 2014, pp. 245–250.

[3] H. Chen, C. Hankendi, M. C. Caramanis, and A. K. Coskun,“Dynamic server power capping for enabling data centerparticipation in power markets,” in IEEE Proceedings of theInternational Conference on Computer-Aided Design, 2013,pp. 122–129.

[4] T. Kidd, “Power management states: P-states, c-states, andpackage c-states,” Intel Development Resources, 2014.

[5] D. Lo, L. Cheng, R. Govindaraju, L. A. Barroso, andC. Kozyrakis, “Towards energy proportionality for large-scalelatency-critical workloads,” in IEEE SIGARCH ComputerArchitecture News, vol. 42, no. 3, 2014, pp. 301–312.

[6] Q. Wu, Q. Deng, L. Ganesh, C.-H. Hsu, Y. Jin, S. Kumar,B. Li, J. Meza, and Y. J. Song, “Dynamo: facebook’s datacenter-wide power management system,” in IEEE Interna-tional Symposium on Computer Architecture (ISCA), 2016,pp. 469–480.

[7] H. Zhang and H. Hoffmann, “Maximizing performance undera power cap: A comparison of hardware, software, and hybridtechniques,” in ACM ASPLOS, 2016, pp. 545–559.

[8] M. Badiei, X. Zhan, R. Azimi, S. Reda, and N. Li, “Diba:Distributed power budget allocation for large-scale computingclusters,” in IEEE International Symposium on Cluster, Cloudand Grid Computing (CCGrid), 2016, pp. 70–79.

[9] X. Zhan and S. Reda, “Power budgeting techniques for datacenters,” in IEEE Transactions on Computers, vol. 64, no. 8,2015, pp. 2267–2278.

[10] A. B. Yoo, M. A. Jette, and M. Grondona, “Slurm: Simplelinux utility for resource management,” in Workshop on JobScheduling Strategies for Parallel Processing. Springer,2003, pp. 44–60.

[11] O. Sarood, A. Langer, A. Gupta, and L. Kale, “Maximizingthroughput of overprovisioned hpc data centers under a strictpower budget,” in IEEE Proceedings of the InternationalConference for High Performance Computing, Networking,Storage and Analysis, 2014, pp. 807–818.

[12] D. J. Watts and S. H. Strogatz, “Collective dynamics of’small-world’ networks,” in nature, vol. 393, no. 6684. Na-ture Publishing Group, 1998, pp. 440–442.

[13] M. Grant, S. Boyd, and Y. Ye, “CVX: Matlab software fordisciplined convex programming,” 2008.

[14] D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning,R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson,T. A. Lasinski, R. S. Schreiber et al., “The NAS parallelbenchmarks,” in International Journal of High PerformanceComputing Applications, vol. 5, no. 3. SAGE Publications,1991, pp. 63–73.

[15] P. R. Luszczek, D. H. Bailey, J. J. Dongarra, J. Kepner,R. F. Lucas, R. Rabenseifner, and D. Takahashi, “The HPCchallenge (HPCC) benchmark suite,” in conference on Super-computing. Citeseer, 2006, p. 213.

[16] MediaWiki, “Mediawiki — mediawiki, the free wikiengine,” 2015. [Online]. Available: https://www.mediawiki.org/w/index.php?title=MediaWiki&oldid=1446376

[17] J. Fulmer, “Siege http regression testing and benchmarkingutility,” URL http://www. joedog. org/JoeDog/Siege.

[18] R. Cochran, C. Hankendi, A. K. Coskun, and S. Reda, “Pack& cap: adaptive DVFS and thread packing under power caps,”in ACM International Symposium on Microarchitecture, 2011,pp. 175–185.

[19] A. Gandhi, M. Harchol-Balter, R. Das, and C. Lefurgy, “Opti-mal power allocation in server farms,” in ACM SIGMETRICSPerformance Evaluation Review, vol. 37, no. 1, 2009, pp.157–168.

[20] B. Rountree, D. K. Lowenthal, M. Schulz, and B. R.De Supinski, “Practical performance prediction under dy-namic voltage frequency scaling,” in IEEE InternationalGreen Computing Conference and Workshops (IGCC), 2011,pp. 1–8.

[21] M. Andersen, J. Dahl, Z. Liu, and L. Vandenberghe, “Interior-point methods for large-scale cone programming,” in Opti-mization for machine learning. MIT, 2011, pp. 55–83.

[22] D. Meisner, C. M. Sadler, L. A. Barroso, W.-D. Weber, andT. F. Wenisch, “Power management of online data-intensiveservices,” in IEEE International Symposium on ComputerArchitecture (ISCA),, 2011, pp. 319–330.

[23] X. Fu, X. Wang, and C. Lefurgy, “How much power over-subscription is safe and allowed in data centers,” in ACMInternational Conference on Autonomic Computing, 2011, pp.21–30.

[24] X. Fan, W.-D. Weber, and L. A. Barroso, “Power provisioningfor a warehouse-sized computer,” in ACM SIGARCH Com-puter Architecture News, vol. 35, no. 2, 2007, pp. 13–23.

[25] L. Xiao, S. Boyd, and S.-J. Kim, “Distributed average consen-sus with least-mean-square deviation,” in Journal of Paralleland Distributed Computing, vol. 67, no. 1. Elsevier, 2007,pp. 33–46.

APPENDIX

In this appendix, we provide more detail for the deriva-tions in Subsection II-B. We consider a network betweenservers which we abstract by the graph G = (N,E), whereE ⊆ N × N denotes the set of edges, i.e., servers i and jare connected to each other iff (i, j) ∈ E. Let N(i) ⊆ Nbe the set of all servers that are connected to server i onthe network, i.e., j ∈ N(i) iff (i, j) ∈ E. In a generalform, we define the augmented utility function of each serveri = 1, 2, · · · , N as follows:

U ′i(pi(t), {ekj (t)}j∈N(i)∪{i}

):= Ui(pi(t))− µVi

({ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i}

), (4)

where Vi : R(r+1)(|N(i)|+1) → R≥0 is a penalty function thattakes the estimates ekj (t) of all constraints and all neighborsN(i) as well as server i as the argument, and outputs anon-negative real value. Here, µ ≥ 0 is a tunable parameterthat determines the significance of the penalty function. Theutility function in Eq. (4) consists of two parts. The firstpart Ui(pi(t)) is associated with the throughput at server i,while the second part Vi

({ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i}

)is included

to reduce the error terms eki (t) to zero. By choosing aproper penalty function Vi(·) and the parameter µ, we candefine a local procedure for each server i ∈ N such thatpi(t) converges to the optimal solution of the formulatedoptimization problem in Eqs. (1a)-(1c).

Implicit in the structure of the penalty function Vi in Eq.(4) is some form of communication among servers. In par-ticular, servers exchange information about their estimateseki (t) to create consensus. Let {eki→j , e

kj→i}j∈N(i) denotes

such messages sent and received from the i-th server to itsneighbors j ∈ N(i). Based on the received messages, thebelief of the i-th server is updated in the form of a state-space model, wherein (pi(t), {eki (t)}k∈{0,1,··· ,r}) are states.Further, for all k = 0, 1, · · · , r and i = 1, 2, · · · , n we have

pi(t+ 1) = pi(t) + p̂i(t), (5a)

eki (t+ 1) = eki (t) + p̂i(t)1{i∈CBk} (5b)

+∑

j∈N(i)

(ekj→i(t)− eki→j(t)),

where 1{·} is an indicator function that takes the value of oneif i ∈ CBk and zero otherwise. We note that p̂i(t) in the firstequation and p̂i(t)1{i∈CBk} +

∑j∈N(i)(e

ki→j(t)− ekj→i(t))

in the second equation are the inputs of the linear state-space system in Eqs. (5a)-(5b). In this formulation, p̂i(t−1)can also be sought as the amount of change in the powerusage limits of each server i at iteration t. We also notethat in the machinery characterized in Eqs. (5a)-(5b), themessages {eki→j(t), e

kj→i(t)}j∈N(i)(t) propagate the belief

eki (t) of server i to its neighbors j ∈ N(i).By putting Eq. (4) and Eqs. (5a)-(5b) together, we arrive

at the following local convex optimization problem at the

i-th server,

maxp̂i(t),{eki→j(t)}

k∈{0,1,··· ,r}j∈N(i)

U ′i

(pi(t), {ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i}

)(6a)

subject to : (5a)− (5b)

Pmini ≤ pi(t) ≤ Pmax

i , ∀i ∈ N. (6b)

It now remains to determine the form of the penaltyfunction Vi in Eq. (4) as well as the values of power changep̂i(t) and messages {ei→j(t)}k∈{0,1,··· ,r}j∈N(i) at each algorithmiteration t ∈ [T ]. By our construction in Eq. (4), we observethat an admissible penalty function must satisfy

Vi

({ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i}

)= 0,

if for all j ∈ N(i) ∪ {i} and k ∈ {0, 1, · · · , r} we haveekj (t) ≤ 0. That is, when the beliefs of all servers inN(i)∪{i} indicates that the power capping constraints in Eq.(1b) are satisfied, the penalty function must vanish. Althoughsuch a choice of penalty function is not unique, we considerin this paper an admissible function given by

Vi

({ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i}

):=

1

2

r∑k=0

∑j∈N(i)∪{i}

[max{0, ekj (t)}

]2.

To determine the values of power change p̂i(t) andmessages {eki→j(t)}

k∈{0,1,··· ,r}j∈N(i) , we use a projected gradi-

ent ascent direction for Ui(pi(t), {ekj (t)}k∈{0,1,··· ,r}j∈N(i)∪{i} ). Let

M(i) ⊆ {0, 1, · · · , r} be the subset of CBs that the serveri is subscribed to. We compute:

p̃i(t) = ε ·∂U ′i(pi(t), {ekj (t)}

k∈{0,1,··· ,r}j∈N(i)∪{i} )

∂p̂i

∣∣∣∣p̂i=0

= εdUi(pi(t))

dp̂i(t)− εµ

∑k∈M(i)

max{0, eki (t)}, (7)

where ε > 0 is a time-independent step size in the gradientascent method. The resulting action is obtained by theprojection p̂ti = PrP(pi(t))[p̃i(t)] onto the set P(pi(t)) :={p̃ ∈ R≥0 : pi(t) + p̃ ∈ [Pmin

i , Pmaxi ]}. This projection

step guarantees that the new updated power consumptionprofile given by pi(t+1) = pi(t)+ p̂i(t) respects the powerrestriction of the server, i.e., Pmin

i ≤ pi(t + 1) ≤ Pmaxi .

Simplifying this projection results in the thresholding step(Step 4) of Algorithm 1.

Similarly, for all j ∈ N(i) we choose

eki→j(t) = µε(eki (t)− ekj (t)

). (8)

Note that based on the structure of the messages in Eq. (8),we obtain the following state-space update for the estimatesfrom Eq. (5b),

eki (t) =(1− 2µε)eki (t) + 2µε∑

j∈N(i)

ekj (t) + p̂i(t)1{i∈CBk},

which is a standard step for achieving consensus in dis-tributed averaging algorithms, e.g. see [25].