VirtualPower: Coordinated Power Management inVirtualized Enterprise Systems

Ripal NathujiCERCS Research Center

School of Electrical and Computer EngineeringGeorgia Institute of Technology

Atlanta, GA 30032rnathuji@ece.gatech.edu

Karsten SchwanCERCS Research Center

College of ComputingGeorgia Institute of Technology

Atlanta, GA 30032schwan@cc.gatech.edu

ABSTRACTPower management has become increasingly necessary inlarge-scale datacenters to address costs and limitations incooling or power delivery. This paper explores how to inte-grate power management mechanisms and policies with thevirtualization technologies being actively deployed in theseenvironments. The goals of the proposed VirtualPower ap-proach to online power management are (i) to support theisolated and independent operation assumed by guest vir-tual machines (VMs) running on virtualized platforms and(ii) to make it possible to control and globally coordinatethe effects of the diverse power management policies ap-plied by these VMs to virtualized resources. To attain thesegoals, VirtualPower extends to guest VMs ‘soft’ versions ofthe hardware power states for which their policies are de-signed. The resulting technical challenge is to appropri-ately map VM-level updates made to soft power states toactual changes in the states or in the allocation of underlyingvirtualized hardware. An implementation of VirtualPowerManagement (VPM) for the Xen hypervisor addresses thischallenge by provision of multiple system-level abstractionsincluding VPM states, channels, mechanisms, and rules. Ex-perimental evaluations on modern multicore platforms high-light resulting improvements in online power managementcapabilities, including minimization of power consumptionwith little or no performance penalties and the ability tothrottle power consumption while still meeting applicationrequirements. Finally, coordination of online methods forserver consolidation with VPM management techniques inheterogeneous server systems is shown to provide up to 34%improvements in power consumption.

Categories and Subject DescriptorsC.0 [Computer Systems Organization]: General—Sys-tem architectures; D.4.7 [Operating Systems]: Organiza-tion and Design; K.6.4 [Management of Computing and

Information Systems]: System Management

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SOSP’07, October 14–17, 2007, Stevenson, Washington, USA.Copyright 2007 ACM 978-1-59593-591-5/07/0010 ...$5.00.

General TermsDesign, Experimentation, Management

KeywordsPower management, Virtualization

1. INTRODUCTIONThe necessity of power management in computing sys-

tems has become increasingly evident in both mobile andenterprise environments. Limitations in battery capacitiesand demands for longer device lifetimes have motivated re-search for mobile and embedded systems [12, 34, 37, 38].For server systems, online management is important for twosignificant reasons. First, there are power delivery and cool-ing limitations in datacenter environments that result frommeeting the ever increasing performance and scalability de-mands of enterprise applications with a combination of tech-nological advances like increased densities, small form factorblade servers, and software methods for server consolidationthrough virtualization. Here, the nearly 60 Amps per rackcurrently provisioned in datacenters could become a bot-tleneck for high density configurations, even if the relatedcooling issues can be solved [33]. Online methods for min-imizing power consumption and, when necessary, enforcinglimits on rack power draws can help deal with such bottle-necks. A second motivation for online power managementis the cost of power and cooling capabilities in datacenters.A datacenter consisting of 30,000 square feet and consum-ing 10MW, for instance, requires an accompanying coolingsystem that costs from $2-$5 million [25]. The yearly costof running this cooling infrastructure can reach up to $4-$8million [33].

To manage the power properties of server systems, re-searchers have considered issues that include managing ther-mal events [25] and utilizing hardware management supportlike processor frequency scaling for power budgeting [10,33]. An aspect of future datacenter environments not ad-dressed by previous work, however, is the proliferation ofvirtualization techniques into the enterprise domain. Specif-ically, the growing capabilities of hardware support for vir-tualization [28] and of software solutions like Xen [4] orVMware [36] have made it easier to flexibly use datacen-ter resources for applications [3]. The resulting interplay ofpower management and virtualization may be formulatedmore precisely as follows:

1. ‘Soft’ and ‘Hard’ power scaling – Virtualization creates

new possibilities for scaling the allocation of physicalresources to guest virtual machines (VMs), both us-ing ‘soft’ techniques that exploit a hypervisor’s abilityto limit hardware usage by guests, and ‘hard’ tech-niques that leverage underlying hardware support suchas processor frequency scaling. To be effective, virtu-alization layer power management must exploit both ofthese methods.

2. Independence and coordination – Guest VMs are de-signed to execute their own, independent OS- and/orapplication-specific methods for power management.For example, the Linux operating system allows fordynamic voltage and frequency scaling (DVFS) of theprocessor, where governing policies can be loaded intothe kernel or executed in userspace daemons. Applica-tion specific policies that can address real-time work-loads [30, 31] or meet application requirements withminimum power consumption [11] can then be inte-grated. From these facts, it is clear that a necessaryelement of power management is the need to coordinatebetween VM-level solutions and global goals that con-cern platform-, rack-, and then datacenter-level powerconsumption.

3. Flexibility in management – Modern datacenters thatutilize virtualization often house multiple generationsof equipment with different attributes and manage-ment capabilities [26], and deploy a variety of applica-tions that have distinct requirements expressed by Ser-vice Level Agreements (SLAs). These traits dictate theuse of diverse dynamic management policies. There-fore, to effectively address virtualized environments itis important to give administrators the flexibility toprovide their own power management policies.

The VirtualPower approach to power management pre-sented in this paper can exploit both hardware power scalingand software-based methods for controlling the power con-sumption of underlying platforms. Its power managementactions take into account guest VMs’ independent powermanagement policies, and can coordinate across multiplesuch policies to attain desired global objectives. Effectivefor both fully and para-virtualized guests, VirtualPower’sabstractions and methods may be summarized as follows.First, to permit guest VMs to run their own, independentpower management methods, VirtualPower exports a richset of virtualized (i.e., ‘soft’) power states, termed Virtu-alPower Management states – VPM states. Guest VM-levelpower management policies, then, observe and act uponthese states when carrying out their management actions.Second, coordination is based on the fact that independentVM-level management policies change power states via priv-ileged actions that, for example, modify model-specific reg-isters (MSRs) in hardware. VirtualPower can leverage thefact that these events are trapped by the hypervisor to cre-ate an abstraction of VPM channels. These channels de-liver guest VM power management actions as a set of ‘soft’state updates that can be used for application-aware man-agement at the virtualization layer. Third, VirtualPower en-ables flexibility by encoding the actual power managementactions carried out by the infrastructure as VPM rules, pro-vided by hardware vendors and/or system administrators.VPM rules treat the ‘soft’ VPM state changes conveyed by

VPM channels as ‘hints’ for assigning actual shadow VPMstates for guest VM resources, a control relationship we termstate-based guidance. Finally, VPM rules are based on a richset of underlying VPM mechanisms that provide a uniformbasis for implementing management methods across hetero-geneous hardware platforms.

The implementation of VirtualPower with the Xen hyper-visor significantly extends the infrastructure’s power man-agement capabilities. On this basis, this paper’s contribu-tions include: (1) an experimental study of power manage-ment and its implications in virtualized systems, (2) the useof VPM channels and states to obtain application-specificpower/performance tradeoffs for representative enterprise ap-plications, (3) the definition and evaluation of multiple man-agement actuators in the form of our VPM mechanisms, and(4) the evaluation of VirtualPower with sets of rules thatjointly implement a tiered policy-driven approach to onlinepower management. Experimental evaluations are based onmicro-benchmarks, the RUBiS enterprise application, andtransactional workloads. Using VPM rules that implementrepresentative power management policies, measured resultsshow improvements of up to 31% in the active power con-sumption of underlying computing platforms. They alsodemonstrate VirtualPower’s abilities to perform QoS-awarepower throttling and to exploit the consolidation capabilitiesinherent in modern multicore platforms. Moreover, whenusing the tiered VPM rule components for managing VMsacross heterogeneous platforms, we reduce power consump-tion up to 17% by exploiting power management heterogene-ity and up to 34% by performing runtime consolidation ontomore power-efficient hardware. Finally, VirtualPower en-ables future work on management policies and mechanismsthat can extend across multiple devices and/or separatelymanageable platform components like memory, can take intoaccount additional properties like thermal events [15], andcan address the multiple machines, racks, and enclosuresfound in modern datacenters.

The remainder of this paper is organized as follows. Sec-tion 2 reviews related work. This is followed by a discussionin Section 3 on how virtualization impacts power manage-ment and the resulting technical challenges. Section 4 out-lines the components of the VirtualPower architecture anddescribes the manner in which they address these challenges.After describing experimental methodology in Section 5, weevaluate the viability of soft scaling as a power managementmechanism in Section 6, and the performance of coordinatedpolicy-based management with VirtualPower in Section 7.Conclusions and future work are reviewed in Section 8.

2. RELATED WORKManaging power and thermal issues has been addressed

extensively for single platforms, particularly their proces-sor components. Brooks and Martinosi propose mechanismsto enforce thermal thresholds on the processor [5]. Formemory-intensive workloads, dynamic voltage and frequencyscaling (DVFS) during memory-bound phases of executionhas been shown to provide power savings with minimal per-formance impact. Solutions based on this premise includehardware-based methods [23] and OS-level techniques thatset processor modes based on predicted application behav-ior [18]. Power budgeting of SMP systems with a perfor-mance loss minimization objective has also been implementedvia CPU throttling [20]. VirtualPower leverages the invest-

ments in application-specific power management representedby these approaches by encouraging their use, rather thanreplacing them, in virtualized systems.

In high performance settings, energy savings can be ob-tained for parallel programs during their communication pe-riods using hardware frequency scaling capabilities in clus-ters [24]. For server systems, Chase et al. discuss how toreduce power consumption in datacenters by turning serverson and off based on demand [6]. Identifying the need fordisk power management in datacenters, support for energy-efficient disk arrays has been extended [39]. Incorporatingtemperature awareness into workload placement in datacen-ters was proposed by Moore et al. [25], along with emula-tion environments for investigating the thermal implicationsof power management [15]. Researchers have also investi-gated using such cluster-level reconfiguration in conjunctionwith DVFS [8], or by using spare servers [32] to improve thethermal and power properties of enterprise systems. Othermethods have enforced power budgets by allocating power ina non-uniform manner across nodes [10], at the granularityof blade enclosures [33], or even amongst virtual machinesusing energy accounting capabilities [35]. The experimentalevaluations performed in this paper show that global man-agement policies like those above are easily realized withappropriate VPM rules based on VirtualPower’s rich set ofpower control mechanisms.

In datacenter environments, an important attribute of on-line power management is its ability to deal with hardwareheterogeneity. This is evident from prior work on man-agement extensions for heterogeneous multicore architec-tures [21] and in cluster environments, the latter proposinga scheduling approach for power control of processors withvarying fixed frequencies and voltages [13]. In enterprise sys-tems, intelligent methods for request distribution have lever-aged hardware heterogeneity to curtail power usage [16].The ability to exploit heterogeneity in platform hardwareand underlying power management capabilities for datacen-ters has also been investigated [26]. Experiments presentedin this paper demonstrate the importance of support for het-erogeneity in the VirtualPower infrastructure and providefurther evidence of the utility of heterogeneity awareness inpower management techniques.

3. MANAGING VIRTUALIZED SYSTEMSThis section discusses the impact of virtualization on the

power management landscape of enterprise systems. Of par-ticular interest are barriers to utilizing guest VM- or appli-cation specific policies for enabling workload-aware manage-ment in these environments. We identify such hurdles anddescribe how VirtualPower is designed to overcome them.

3.1 Scalable Enterprise EnvironmentsDesired benefits from virtualization include improved fault

isolation and independence for guest VMs [4], performanceisolation [19], and the ease of seamless migration of VMsacross different physical machines [7]. Jointly, these permitusage models in which VMs freely and dynamically sharepools of platform and datacenter resources, as also consid-ered in recent research on autonomic management [2, 14]and as indicative of next generation enterprise managementenvironments. The goal, of course, is the delivery of flexi-ble and scalable enterprise infrastructures, as illustrated inFigure 1.

Figure 1: Scalable Enterprise Infrastructure.

Power management directly threatens the independenceand performance isolation properties of virtualization tech-nologies. First, how can guest VMs realize the goals of theirapplication-specific power management policies when thereis a disassociation between the virtual resources they aremanaging and the physical resources actually being used?Here, a specific issue is that guest machines may have in-consistent views of the management capabilities of the un-derlying resources they are trying to manage, particularlyin the presence of VM migration. Second, in datacentersettings, inconsistencies are aggravated by the need to fre-quently update platforms, to replace them to deal with fail-ures, or to add new platforms for capacity increases. Anatural outcome of such changes is increased heterogene-ity, with respect to platform power properties, performancecharacteristics, and/or management capabilities. Isolationand independence, however, demand that VMs’ power man-agement policies not be altered to deal with such changes.The VirtualPower approach maintains the isolation and in-dependence properties of virtualization systems by exportingto VM-level management policies ‘soft’ VPM states, whereinguest machines have a consistent view of hardware manage-ment capabilities regardless of the underlying set of physicalresources. The implementation of the approach, then, per-forms state-based guidance to map changes to these softstates to changes in underlying hardware according to asso-ciated VPM rules, as described further in Section 4.

3.2 Leveraging Guest VM PoliciesThe increasingly apparent need for power management

has led to the implementation of system- and application-level policies that carefully balance hardware performancestates to attain application-specific notions of quality-of-service (QoS). An example of such a policy is the onde-mand governor integrated into the Linux kernel. This pol-icy scales the frequency of the processor based upon theCPU utilization observed by the operating system and iseffective for workloads with variations in processor utiliza-tion, such as web servers. Other management policies in-clude those specific to real-time systems, where processors’DVFS capabilities are used to reduce power consumptionwhile still maintaining application deadlines [30, 31]. SinceVM-level management policies can meet application-specificQoS constraints, it is highly desirable to use them to man-age the power/performance tradeoffs of modern computingplatforms. Giving these policies direct access to hardwarepower management facilities, however, negates the notionof virtualized hardware resources. Moreover, direct access

would threaten desired performance isolation properties, asevidenced on thermally sensitive multicore chips where a VMincreasing its frequency may heat up its core in a mannerthat compromises another core’s ability to run at a speedsufficient for meetings its VM’s desired QoS. Worse, suchactions might be malicious, as with a VM compromised bya power virus.

VirtualPower responds to these challenges with an ap-proach that takes advantage of the power management poli-cies built into guest VMs, but interprets them as ‘hints’rather than executable commands. These hints are acquiredby exploiting the ACPI [17] interface for power managementprovided by modern platforms. When guest VMs attemptto perform privileged operations on this interface, currenthypervisors ignore them, but VirtualPower intercepts themand maps them to VPM channels. These channels, in turn,provide them as useful hints to VirtualPower’s VPM rules.VPM rules may then use the ‘soft’ state requests that makeup these hints as inputs to localized power managementand/or to global policies that coordinate across multipleguest VMs and multiple machines. The combination of VPMstates and channels provides for coordinated power manage-ment while maintaining VM independence.

3.3 Limitations of Hardware ManagementAn issue with utilizing hardware support for power man-

agement in virtualized systems is that resources within amanagement domain may be shared by multiple guest VMs.For example, memory DIMMs that are accessible via thesame bus and/or share the same voltage plane may con-tain memory that is allocated to many guests. This meansthat this component cannot be managed unless all guest ma-chines desire reduced memory bandwidth via bus scaling, orare currently not utilizing the DIMMs such that they can bepowered down. Similar limitations exist for disks that con-tain multiple partitions, each of which may be allocated todifferent guests. Any one partition can be power-managedby placing the disk into a low power state only if all parti-tions can be placed into that state.

Figure 2: DVFS Limitations on a Dual Core Chip.

An approach to overcoming limitations for power man-aging shared resources is to use time domain multiplexing,which sets components to the hardware states that reflectthe power management criteria of the currently executingVM. Unfortunately, this approach has two significant draw-backs. First, it can only be used if the transition time ofresources between management states is much smaller thanthe scheduling time granularity used by the hypervisor. Sec-ond, even if this were the case, this approach is not promis-ing in the context of multicore platforms. Here, the intent toconcurrently execute a large number of guest VMs on avail-

able cores significantly reduces the ability to utilize timedomain multiplexing for reasons that include the following.Consider the use of DVFS to manage the processor com-ponent of multicore platforms. Since the dynamic powerconsumption of a core is proportional to the product of fre-quency and voltage squared, in order to obtain significantenergy or power savings, voltage scaling must be performedin conjunction with frequency scaling. In multicore pack-ages, however, even when the frequency of cores can bescaled independently, there is only one voltage rail into thepackage from the motherboard. Therefore, the voltage isconstrained by the highest frequency of any core in the pack-age. As shown in Figure 2, this restriction can significantlylimit the system-level benefits of hardware power manage-ment, where power is only reduced by 4% when scaling fre-quency on only one processor, but is reduced by 19% whenboth are scaled at the same time. VirtualPower overcomeshardware limitations derived from resource sharing and lim-ited opportunities for voltage scaling by providing mecha-nisms for power management that go beyond mere relianceon hardware operating modes. These mechanisms includeconsolidation techniques and runtime changes to hypervisor-level scheduling. The outcome is a rich set of VPM mecha-nisms for VPM rules that exploit both ‘soft’ and hard powerscaling techniques.

4. VIRTUALPOWER ARCHITECTUREIn keeping with common practice, VirtualPower does not

require modifications to guest operating systems. Further,while implemented with the Xen [4] virtualization frame-work, the VirtualPower architecture and abstractions aredesigned for the wide range of virtualization solutions cur-rently deployed or under development. Therefore, most ofits functionality resides as a controller in the driver domain,termed Domain Zero (Dom0) in current systems, with onlysmall changes required to the underlying virtual machinemonitor or hypervisor. Specifically, VirtualPower’s monitor-ing functionality is integrated into the hypervisor, whereasits higher level mechanisms and policies (i.e., VPM mecha-nisms and rules) reside in Dom0. As a result, policies haveeasy access to CPU and device power states, and they cantake advantage of Dom0’s control plane functionality for cre-ating guest virtual machines, migrating them, . . . etc.

Figure 3: VirtualPower Management Architecture.

Figure 3 illustrates the VirtualPower architecture. Eachphysical platform runs a hypervisor and associated Dom0.Guest VMs perform power management based upon the‘soft’ VPM states exported to them. VPM channels captureguest VM power management requests, via updates to these‘soft’ states, in the hypervisor. This information is thenpassed to power management software components, com-

posed of sets of VPM rules running in Dom0, which finally,use VPM mechanisms to actually carry out managementdecisions. As described further in Section 7, our currentVPM rules implement a tiered policy approach: local PM-Lpolicies perform actions corresponding to resources on lo-cal platforms, and a higher-level PM-G policy componentuses rules responsible for coordinating global decisions (e.g.,VM migration). We next describe in more detail the VPMstates, channels, and mechanism components that permitVPM rules to perform coordinated power management.

4.1 VPM StatesIt is important to leverage the substantial investments and

knowledge about online power management embedded inguest operating systems. Technically, this implies the needto provide to the policies run by guest VMs what appears tobe a richly manageable set of underlying resources, even ifthe hardware platform does not actually provide the differ-ent power states being exported. In heterogeneous hardwareenvironments, for instance, a guest machine with sophisti-cated power management policies may initially be deployedon hardware that does not support power management andlater be migrated to manageable components. VirtualPoweris designed to permit guest VM policies to run in both sys-tems, by exporting to them ‘soft’ VPM states and then us-ing appropriate VPM rules and VPM mechanisms to real-ize them. These VPM states need not reflect the actualoperating points (i.e., Px states [17]) supported by under-lying hardware, but instead, they consist of a set of per-formance states made available by VirtualPower for use byVMs’ application-specific management policies. This is im-plemented by simply defining these ‘soft’ states in the rel-evant portions of the ACPI tables created for guest VMs.In addition, for management flexibility, VM-level changes to‘soft’ VPM states are disassociated from the assignment ofshadow VPM states to actual resources by VPM rules.

In exporting VPM states, a limitation is imposed by thefact that the ACPI interface specification defines a maxi-mum of sixteen Px states for a device, thereby limiting thenumber of possible soft states available for each manage-able component. We do not consider this an issue because,in general, most devices, such as processors, only supporta small number of performance states. Therefore, even inheterogeneous environments where one must define a set ofVPM states that allows VMs to provide useful hints acrossa range of machines, there is sufficient ability to define arich set of VPM states that express the performance scalingcapabilities of multiple physical platforms.

4.2 VPM ChannelsVPM channels are the abstraction via which guest VMs

provide input on power management to VPM rules, as il-lustrated in Figure 3. Since hardware power managementis privileged, whenever guests attempt to execute such op-erations, they are trapped by the hypervisor, whereuponVirtualPower re-packages them as VPM events made avail-able to the VPM rules in Dom0. In particular, the informa-tion encapsulated by a VPM event includes a timestamp,the ‘soft’ state set by the guest VM, and information re-garding the actual performance state being provided to theguest when the request was made. The implementation ofVPM event information retrieval by VPM rules is an asyn-chronous one, where VPM event information is temporarily

stored in the hypervisor using a cyclic buffer from whichVPM rules may retrieve it. The timing and frequency ofretrievals depend upon whether the control algorithm beingused requires polling or event-based retrieval techniques.

The implementation of VPM channels uses hypercalls andXen event channels. VirtualPower defines its own hyper-call interface which can be used to perform a variety ofactions for VPM rules. The interface defines a VPM_POLL

operation that allows VPM rules in Dom0 to query for VPMevents corresponding to a particular guest VM. VPM rulescan use this operation to periodically retrieve updates aboutthe state changes desired by guest VMs. In addition, Vir-tualPower provides a Xen event channel that may be usedif VPM rules desire immediate notification of desired statechanges by guest VMs. This is useful when timely responsesto VM behavior are necessary for balancing power manage-ment with performance requirements.

4.3 VPM MechanismsVPM mechanisms are the base level support for online

power management by VPM rules. They provide uniformways of dealing with the diversity of underlying platformpower management options. The mechanisms supportedby VirtualPower include hardware scaling, soft scaling, andconsolidation.

Hardware Scaling.Hardware scaling capabilities vary across different plat-

form and device architectures. Moreover, whether or notsuch scaling is possible or effective depends on VM-levelresource sharing (e.g., VMs running across multiple coresin multicore platforms). The VPM mechanism supportinghardware scaling is straightforward, permitting VPM rulesto set the hardware states to be used during the execution ofa particular guest VM via a VPM_SET_PSTATE operation onthe VirtualPower hypercall interface. Of course, the actualrules that decide upon such state changes can be non-trivial.For example, policy rules must determine whether or nota set of hardware performance states for VMs can createconflicts due to resource sharing. In the particular case ofprocessor DVFS management, such conflicts are handled byhardware that ensures that the voltage provided to the chipis sufficient for the core running at the highest frequency.

Soft Scaling.Hardware scaling is not always possible, or it may provide

only small power benefits, as shown in our earlier exampleof utilizing DVFS on multicore chips. In response, we intro-duce the notion of ‘soft’ resource scaling, which uses resourcescheduling to emulate the performance loss of the hardwarescaling action that would otherwise be performed. For pro-cessor management, this entails modifying the hypervisor’sscheduling attributes for a VM to emulate the VM’s desiredperformance mode. For instance, if a VM has requestedscaling a core to half of its previous performance state, thenwith soft scaling, the hypervisor scheduler may reduce theguest’s maximum time slice in a period by half. The nec-essary adjustments to scheduling parameters are performedusing a VPM_SET_SOFT operation with the VirtualPower hy-percall interface.

An obvious issue with soft scaling is that the performancedegradation attained in this fashion may not accurately rep-resent what the application would have observed from actual

hardware scaling. For example, a guest VM may request areduced processor frequency, because it expects to observelittle or no performance degradation for its current memory-bound application workload. If soft scaling proportionallychanges CPU allocation in response to such requests, the ap-plication would experience notable and unexpected levels ofperformance loss. Our implementation of soft scaling, there-fore, is carried out by VPM rules that operate with feedbackloops based upon state-based guidance in which such degra-dation is observed, indirectly, by VM requests for increasedperformance states. This results in policies that dynami-cally tune to what extent, if any, soft scaling is applied to aphysical resource.

Soft scaling can result in substantial power benefits, fortwo reasons. First, there is the idle power management ofresources. Components such as processors are achieving in-creasingly substantive savings when idle compared to ac-tive, even when active power management cannot provideimprovements. Thus, soft scaling is most effective when itresults in processors being idle for some time (i.e., no VMscurrently running on any of their threads). Second, addi-tional savings may be derived from appropriately managingsoft scaling across multiple resources, by coordinating theirconcurrent use [27] and/or by use of consolidation.

Consolidation.When soft scaling multiple VMs for multiple resources,

such as cores on a multicore chip, it becomes possible toshare resources, perhaps to totally idle one core while fullyusing another one. An obvious first benefit from such re-source consolidation is the substantial power improvementsderived from placing offloaded resources into their idle orsuspend states. An interesting second benefit is derivedfrom resource heterogeneity. In particular, in datacenterenvironments, there likely exist physical resources that aremore power efficient for certain workloads or that are moreamenable to online management. By combining soft scal-ing with VM re-mapping or migration (i.e., consolidation),therefore, it is possible to map multiple ‘compatible’ in-stances of virtual resources to appropriately efficient phys-ical resources. This dynamic migration capability is imple-mented using existing control interfaces in Dom0.

5. EVALUATION METHODOLOGY

5.1 Experimental SetupExperimental evaluations of the VirtualPower manage-

ment infrastructure use standard multicore server hardware.Our testbed consists of multiple dual core Pentium 4 ma-chines, which are identical in terms of their hardware ca-pabilities and components. These machines have processorsbased upon the Intel Netburst microarchitecture, 3GB ofmemory, gigabit network cards, and 80GB hard drives. Interms of manageability, they support two physical operatingmodes for their processor cores: 3.2GHz and 2.8GHz.

Power data is obtained using an Extech 380801 power an-alyzer, which allows for out of band measurements via alaptop, thereby avoiding undesirable measurement effects onthe system under test. Power is measured ‘at the wall’, inorder to include the AC power consumption of the entiresystem. We obtain power traces at the maximum samplingrate of 2Hz using the Extech device, then use these tracesto calculate offline the power consumption characteristics

Figure 4: Power Measurement Setup.

seen during experiments. Figure 4 summarizes our powermeasurement configuration.

5.2 Guest Applications and PoliciesExperiments are conducted with representative enterprise

workloads, using the RUBiS tiered web service applicationand using transactional loads. The distributed componentsof these applications are instantiated in their own virtualmachines, with each VM mapped to a single virtual proces-sor. The different workload characteristics of these codes,along with the varying VM-level policies which drive theirmanagement, present interesting challenges to VirtualPower’smanagement policies, as explained next.

Transactional Workloads.Many backend services deployed in enterprise systems and

datacenters are of a transactional nature. For example, aspart of an ongoing collaboration with our corporate partnerDelta Air Lines, we have obtained workload traces from abackend subsystem responsible for tracking operational rev-enue earned from the services offered by the company [1].The backend operates by continually receiving, queuing, pro-cessing, and then, emitting revenue-relevant events, withcurrent incoming event rates at the level of thousands perhour. Two interesting elements of these event traces are(1) that event queuing delays often outweigh actual eventprocessing times and (2) that arrival rates vary significantlyover the course of a day. For example, a large batch ar-rival of events submitted every day at 4 AM EST createsqueue lengths as high as 4000. From (1), it is clear that forthese types of applications, changes in the execution times ofevents due to power management may have negligible effecton overall processing time, due to queuing delays. From (2),it is evident that there exist possibilities for management toexploit changes in application behavior across larger timescales.

For transactional applications like these, the goal of back-end servers is to process requests at rates that are suffi-ciently high to meet certain application-specific SLAs. Forthe actual traces received from our corporate partner, theseSLAs are dictated by revenue reporting requirements. Moregenerally, different transactional applications will have dif-ferent performance requirements, and in addition, such re-quirements may vary over time (e.g., due to changes in re-porting requirements or when dynamic pricing methods areused). In all such cases, the principal performance metricfor a transactional service is whether it can run its trans-actions at some currently desired minimum processing rate.For power management, this means that management poli-cies can exploit the slack available between minimum de-sired rate and the current rate being observed. Accordingly,

we define a simple VM-level power management policy thatkeeps track of ‘performance slack’ while also maintaining adesired transaction execution rate. When slack is large, thepolicy requests reduced processor frequencies, and if slackbecomes negative, it scales up the underlying processor’sperformance state. Experiments show that this simple con-trol policy proves to effectively enforce a defined application-specific transaction processing rate. For purposes of experi-mentation, we use the well-known SPEC CPU2000 suite tocreate code modules that carry out computationally expen-sive transactions. These code modules are driven by vary-ing transaction rates and rate requirements. Future stepsin this work will consider transactional computations usingthe Nutch open source search engine [29].

Tiered Web Service Workloads (RUBiS).RUBiS is a well-known tiered web service benchmark, im-

plementing some of the basic functionality of an auctionwebsite, including selling, browsing, and bidding. We usea PHP-based two node configuration of RUBiS where thereis a web server front end (Apache) connected to a databasebackend (MySQL). Load is provided using a third client ma-chine. All of these nodes are connected via a gigabit Ether-net switch. Web service applications typically exhibit varia-tions in processor utilization based on current request arrivalrates from clients and on current request behaviors. Thischaracteristic can be exploited for power management viareactive policies that attempt to scale the processor to an ap-propriate performance state based on current load statistics.Since VM-level policies like Linux’s ondemand governor aredesigned to exploit this fact, this is the application-specificpolicy used in our experiments with the RUBiS VMs.

6. VIABILITY OF SOFT SCALINGAs described in Section 4.1, VPM states act to disassociate

the management actions of virtual machine policies from themanageability support of underlying physical resources. OurPentium 4 based processors support two operating frequen-cies of 3.2GHz and 2.8GHz. With the use of VPM mecha-nisms, however, we can perform power management actionsat the virtualization layer beyond just these two hardwareperformance states, resulting in the export of five different‘soft’ Px states (frequencies) to guest VMs. In particular,guest VMs see frequency capabilities of 3.2GHz, 2.8 GHz,2.0 GHz, 1.6GHz, and 800MHz. When guest VMs attemptto set these states, it is up to VPM rules to map these re-quests to actual changes (or not) of the shadow states theymaintain. Changes applied to shadow states trigger changesin the Xen scheduler’s scheduling attributes and/or in un-derlying processor power states.

Experimental measurements of soft scaling presented inthis paper use the Xen hypervisor’s earliest deadline first(SEDF) scheduler, which allows for dynamic tuning of theamount of processing time allotted to a VM’s virtual CPU(vcpu). In addition, based on the SEDF parameters, thescheduler can be set to be work-conserving, or to limit CPUtime in a non-work-conserving fashion. We take advantageof its non-work-conserving mode in order to use the sched-uler as a control actuator via the VPM soft scaling mecha-nism. For example, consider the case of a VM executing withthe physical processor at 3.2GHz and using work-conservingscheduling parameters. Based on VM power managementrequest information, VPM rules wish to set the shadow VPM

state of the VM to 1.6GHz to reduce the performance of theapplication by 50%. One method for doing this is to use theVirtualPower hypercall interface to schedule the vcpu withhalf its original slice size and with the non-work-conservingsetting. Another method is to use hardware scaling in con-junction with soft scaling. In this example, a VPM rule canspecify a reduced physical frequency f l and simultaneouslysoft scale the resource to 1

2∗

3.2GHzfl

of its original amount.

(a) Performance Error (b) Scaling Power Effects

Figure 5: Soft Scaling Characteristics.

In Figures 5(a) and 5(b), we compare VPM rules thatuse 3.2GHz versus 2.8GHz as physical frequencies for real-izing the different VPM states less than 3.2GHz. In thesemeasurements, rules define VPM states so as to attain linearperformance reductions based on frequency for computation-bound workloads. In other words, the 1.6GHz VPM stateshould provide half the performance of the 3.2GHz state.Figure 5(a) shows how actual measured performance degra-dations are within a few percent of desired goals, therebydemonstrating the viability of the approach. Errors arelarger for the 2.8GHz physical frequency, but the implemen-tation is designed to forego potential (slight) improvementsin power to ensure ‘negative’ errors, that is, to attain perfor-mance slightly better than would be expected from a strictlylinear degradation. In Figure 5(b), we observe the powertradeoffs between using only soft scaling versus using softand hard scaling simultaneously, for a single instance of thecomputation-bound transactional application. From this fig-ure, the power benefits of soft scaling with either physicalfrequency are apparent, with increased power reductions ofup to 48W when the processor is running at 3.2GHz. Sum-marizing, these experiments establish (1) the viability of softscaling and perhaps more importantly, (2) the importanceof simultaneously using soft and hard scaling to create therich set of VPM states desired by guest VMs and their ap-plications.

Table 1: VPM State Definitions.Hardware Soft (Scheduler)

VPM State CPU Work-conserving Slice

3.2GHz 3.2GHz Yes -2.8GHz 2.8GHz Yes -

3.2GHz No 88ms2.0GHz 2.8GHz No 71ms

3.2GHz No 63ms1.6GHz 2.8GHz No 57ms

3.2GHz No 50ms800MHz 2.8GHz No 29ms

3.2GHz No 25ms

0

1

2

3

4

0 10 20 30 40 50

Per

form

ance

Sta

te (

Px)

Time (s)

WS

0

1

2

3

4

0 10 20 30 40 50

Per

form

ance

Sta

te (

Px)

Time (s)

DB

(a) No Active Power Management

0

1

2

3

4

0 10 20 30 40 50

Per

form

ance

Sta

te (

Px)

Time (s)

WS

0

1

2

3

4

0 10 20 30 40 50

Per

form

ance

Sta

te (

Px)

Time (s)

DB

(b) Active Power Management

Figure 6: VPM Channel Feedback for State-Based Guidance with VirtualPower.

Table 1 lists the parameters used by VPM rules to de-fine VPM states. Each state can be implemented using onlysoft scaling or with a combined hardware and software scal-ing approach. Experimentally, we find that the ‘slice’ def-initions for the 3.2GHz mode and for the hardware-scaledversion of the 2.8GHz mode are irrelevant because of theiruse of work-conserving scheduling. For the remaining states,the slices correspond to periods of 100ms. A tradeoff whenusing both soft and hard scaling is that reduced hardwarefrequencies imply larger allocation slices for VMs, as foundin Table 1. As seen later in our evaluations, the interest-ing and somewhat counterintuitive outcome of this fact isthat for minimizing power consumption across multiple ma-chines, it is sometimes necessary to implement VPM stateswith higher hardware frequencies so that multiple vcpus canbe mapped to a single machine (i.e., to permit consolidationfor exploiting low idle/sleep power or power-efficient hard-ware). For example, if two VMs are being set to the 1.6GHzVPM state, using soft scaling alone at a hardware frequencyof 3.2GHz, they each require 50% of a CPU allowing forconsolidation, whereas if hardware scaling is used, the re-quirement is 57% per VM. Since VirtualPower allows VPMrules to define VPM states based upon hard and soft scal-ing mechanisms as they see fit, such dynamic, coordinatedmanagement is easily realized with our approach.

7. POLICY BASED COORDINATIONThe VirtualPower infrastructure can be used to imple-

ment complex management policies. We demonstrate thisfact with hierarchical power management policies [10]. Theseare comprised of local policies (PM-L) running on each phys-ical system and driven by local guest VM and platform pa-rameters, coordinated by global policies (PM-G) with knowl-edge about rack- or blade-level characteristics and require-ments. Specifically, PM-L policies obtain VMs’ desired powerstates via VPM channels and use state-based guidance todetermine suitable local shadow VPM states. The higherlevel PM-G policies use the ‘consolidation’ VPM mecha-nism based on information about the shadow VPM statesassigned to VMs by PM-L policies and the platforms onwhich they run (e.g., rack or blade enclosures). One objec-tive of these experiments is to evaluate VirtualPower’s abil-ity to run diverse rules with different and varying goals whileexercising the multiple VPM mechanisms presented to theserules. Goals sought by the VPM rules policies demonstratedinclude power minimization and power throttling/limiting,the latter designed to enforce limits on blade- or rack-levelpower draws, perhaps triggered by thermal events. Anotherobjective is to demonstrate, experimentally, the importance

of coordination: (1) across the different levels of abstractionat which VM-level vs. virtualization layer policies run, and(2) across policies running on different platforms.

7.1 PM-L Policies: Platform ManagementThe VirtualPower approach establishes an indirect feed-

back mechanism that permits power management at the vir-tualization level to take into account the desires of VM spe-cific management policies using state-based guidance. Theexistence of this desirable feedback loop is established withtraces of power management requests from the web server(WS) and database (DB) VMs for our RUBiS workload inFigure 6. Here, we denote VMs’ requests in terms of de-sired Px states, where P0 is 3.2GHz, P1 is 2.8GHz, P2 is2.0GHz,. . . etc. Figure 6(a) shows the requests made by thetwo VMs without active VirtualPower management. Thatis, VPM rules are disabled and do not perform any hardor soft resource scaling. When active power managementis enabled and PM-L policies are run, a dramatic changeis seen in the request patterns of both VMs in Figure 6(b).This demonstrates the interplay between active managementin the virtualization layer with the internal policies run byguest VMs. In other words, the Figure visualizes perfor-mance feedback data communicated from VMs to the virtu-alization layer power management. We next evaluate howthis indirect feedback loop can be used for effective, coordi-nated platform- and system-level power management.

A simple virtualization-level policy is one that attempts tominimize power consumption while maintaining the desiredperformance of an application, the latter expressed via theVM’s power management requests. We term this policy PM-Lmin. An alternative useful local policy is one that throttlespower consumption for certain periods of time, perhaps dueto thermal events or transient power delivery issues [5, 6].PM-Lthrottle is a power throttling policy that attempts tomaintain average power consumption at some desired level.Finally, to address changes in request behavior experiencedfor the transactional loads explained earlier, a third usefullocal policy, termed PM-Lplan, is one that trades off poten-tial short term reductions in power usage with longer termgains enabled by planning based on monitoring historicaltrends in power management request behavior. We nextexperiment with these policies.

Minimizing Power in the Presence of QoS Constraints.

A common objective for power management policies isto minimize execution time power consumption while main-taining the quality of service (QoS) required by applications.

As outlined in Section 4, VirtualPower attains this goal byobserving and reacting to guest VMs’ management requests.Using combined hard and soft scaling to export a set of fivevirtual frequency states to guest VMs, the experimental re-sults shown here demonstrate that the PM-Lmin VPM rulesuccessfully assigns suitable shadow VPM states to meet aVM’s QoS goals while also reducing its power consumption.Power management behavior with this policy is depicted inFigure 7, where the straightforward ‘mapping’ version of thePM-Lmin policy reacts to a change request by a VM by sim-ply making the appropriate change to some correspondingshadow state.

120

130

140

150

160

170

180

190

200

210

0 100 200 300 400 500 600 700 800 900

Pow

er (

W)

Time (s)

Figure 7: Power Trace of Transactional Application.

The power trace in Figure 7 measured for the transac-tional application is obtained from an experiment in whichthe VPM state for the application is initialized to its maxi-mum (or 3.2GHz). The VM-level policy determines whetheror not a scaling request should be made between transactionexecutions. These decision points can be seen in the figurewhere the power consumption drops for a very small pe-riod of time. We observe that over time, the VM issuesseveral frequency reduction requests since there is slack be-tween its performance and the necessary computation powerto maintain its required processing rate. When such slackis depleted, the system scales the computational resourcesback up to an appropriate performance level, in response tosubsequent VM-issued management requests. Via this feed-back loop, the PM-L policy is able to provide desired levelsof performance for the transactional VM across all experi-ments run in this series.

(a) Single Transaction VM (b) Two Transaction VMs

Figure 8: Transactional Application Power Results.

An analysis of the power reduction capabilities of the‘mapping’ PM-Lmin policy used with transactional appli-cations is shown in Figure 8. The effectiveness of this pol-icy is measured with varying rate requirements, where for

presentation purposes, the processing rate values are nor-malized by the rate that requires the highest (i.e., 3.2GHz)power state on our Pentium 4 machine. Figure 8(a) de-picts the measured average power consumption of a singleinstance of the transactional VM. Also shown are compar-ative results attained with a non-VirtualPower-enabled sys-tem, where the guest VM’s performance state is always setto the maximum. These measurements establish that ourapproach can capture and then effectively use VM-specificpower/performance tradeoffs via its VPM channels and mech-anisms. In fact, with VirtualPower, we see power benefitsof up to 31% for transactional VMs that require reducedperformance (i.e., they execute transactions with low QoSrequirements). The diminishing returns seen with decreaseddesired rate/throughput are because the soft scaling VPMmechanism reduces active power consumption linearly, butis limited by the idle power consumption of the platform.

Figure 8(b) depicts measurements with two backend trans-actional VMs that both run on the same dual core platform.The purpose is to understand whether and to what extentsimple policies like ‘mapping’ PM-Lmin can reduce powerconsumption in the presence of multiple VMs with possiblydifferent individual performance requirements. With VM-level performance requirements expressed as ‘rate mixes’ inthe figure, it is apparent that with performance slack, themapping PM-L policy again successfully reduces executiontime power consumption. Moreover, the policy recognizesand successfully exploits the differences in VM-level perfor-mance needs.

(a) Performance (b) Power Consumption

Figure 9: RUBiS Experimental Results.

Realistic web service applications have behaviors that aremore complex than those of the transactional codes we haveevaluated so far. To experiment with these, we use thetiered RUBiS application, running its web server (WS) anddatabase (DB) virtual machines on separate physical ma-chines, with a PM-L policy executing in each Dom0. Againusing the ‘mapping’ PM-Lmin policy, we measure the result-ing performance, in terms of requests served per second, andpower consumption for various client loads in Figure 9 (eachload class normalized by comparative measurement of the“policy off” case). Interestingly, while this policy is able tolimit total performance impact to within 5% across all loads,its power savings aren’t appreciably better than the perfor-mance degradation being experienced. This is because theondemand governor used by the RUBiS VMs is extremelyreactive, as shown in Figure 6. A simple ‘mapping’ PM-Lpolicy that carries out each of these requests can be inef-ficient, because it is unable to capitalize on periods of lowutilization. A better policy for applications like these is anaveraging “Avg” PM-Lmin policy, which smooths the VPM

state request pattern observed from the RUBiS VMs. Ourimplementation of this policy polls for VM power manage-ment requests received on VPM channels, once per secondfrom Dom0. If a VM has made any requests, an averagingfunction is used along with the current VPM state of the VMto determine a new suitable state. Otherwise, if the shadowVPM state of the VM does not match the last received re-quest, the VPM state is shifted towards the last requestby one performance state. Figure 9 shows that this policyprovides similar performance characteristics to the mappingapproach, but reduces power consumption further, by up to4%. Overall, we conclude that our VirtualPower infrastruc-ture allows power minimization policies to effectively reducepower within application specific performance requirementsfor all of our experimental workloads.

Power Throttling with VPM Mechanisms.In modern datacenters, there is a need for policies that

can limit the average power consumption of a system, per-haps in response to some temporary event like a reductionin cooling or power delivery capabilities, or in order to ad-here to a global power provisioning strategy [9]. Ongoingindustry research is developing system support for powerconsumption monitoring that can be used as feedback forsuch policies [22, 33]. Unfortunately, current platforms likeour dual core Pentium 4 systems have limited performancestates for this type of management. Soft scaling correctsthis deficiency, and it makes it possible to construct policieslike PM-Lthrottle, which we describe and evaluate next.

160 180 200 220 240

0 1

2 3

4 0 1 2 3 4

140 160 180 200 220 240

Power (W)

VM1 Px stateVM0 Px state

Power (W)

Figure 10: Transactional Workload Management.

Consider a single dual core platform executing two trans-actional VMs. Each of these VMs can be assigned a staticPx state independently, by setting its shadow VPM statein the hypervisor. Figure 10 provides the associated powerconsumption trends. An interesting observation from thisfigure is that there is a significant power reduction whenboth VMs are set to VPM states of P1 or less, comparedto either of them running at P0. Particularly, the powerconsumption of both VMs at P1 is 8% less than with oneVM at P0 and the other at P4. This is because for ourplatform, voltage scaling can only be performed on the dualcore package if both cores are scaled down. A more generalobservation from this figure is that soft scaling substantiallyimproves power limiting capabilities: hardware scaling aloneprovides up to 20% throttling capability, while soft scalingextends to up to 40%. These results again highlight the needto utilize both hard and soft scaling in conjunction for effec-tive management.

Next, we again consider the RUBiS setup with two ma-chines in Figure 11. If the power of both machines is limitedsimultaneously such as in a rack or enclosure setting, from

105 120 135 150 165

0 1

2 3

4 0 1 2 3 4

80 100 120 140 160 180

Fetches/sec

DB Px stateWS Px state

Fetches/sec

(a) Performance

255 270 285 300 315

0 1

2 3

4 0 1 2 3 4

250 270 290 310

Power (W)

DB Px stateWS Px state

Power (W)

(b) Power Consumption

Figure 11: RUBiS Management.

the figure, we see that with RUBiS, power savings scale withperformance degradation with the first few VPM states (P0-P2). Within this range, an up to 9% performance degra-dation can be traded for a 10% power reduction. If PM-Lthrottle needs to limit power beyond this, then performancedrops by as much as 45% for an up to 20% reduction inpower. This nonlinearity exists because for distributed ap-plications like RUBiS, their end-to-end performance metricscan be significantly affected by local performance degrada-tions. These experiments demonstrate, for a single applica-tion, the effects of coordinated, multi-machine power man-agement.

Figure 12: Non-uniform VPM State Allocation.

Our final measurements in this set of experiments go be-yond coordinating across multiple machines to also demon-strate the importance of coordination across multiple VMpolicies, specifically, in order to attain some global goal,such as limiting a server’s total power consumption to about180W. Consider two guests VM0 and VM1 with differentpower/performance tradeoffs, expressed by the commandsthey issue to their respective VPM channels. A “fair”, ap-plication unaware approach to throttling the machine woulddivide the power budget uniformly and execute both VMsat the P2 VPM state. On the other hand, if VM0 needs toprocess transactions at a normalized rate of 0.8 while VM1requires only a rate of 0.25, then when executing at P2, VM0will try to upscale its frequency, since its performance is toolow to maintain its desired transaction rate. At the sametime, VM1 will attempt to down-scale its processor. Withthe workload-awareness enabled by VirtualPower, a coordi-nation policy can exploit information about VM desires todetermine that the first VM can run at P1, while the sec-ond can run at P4. Although both assignments observe the180W power limit, Figure 12 shows how the non-uniformVirtualPower allocation provides adequate performance forboth VMs, while a uniform allocation overprovisions VM1and causes VM0 to underperform.

VirtualPower Enabled Power Planning.As a final example of a PM-L policy that can benefit from

the VirtualPower system, we briefly consider how powermanagement requests observed from VMs can be used tocreate power management plans based on predicted behav-ior. For example, an observation about the traces from DeltaAir Line’s enterprise subsystem is that a large batch of re-quests arrives for processing at the same time every day. Bytracking power management requests associated with theseevents, a PM-Lplan policy can make more appropriate de-cisions than other policies. To highlight how this type ofpolicy is enabled by the VPM architecture, we next describea PM-Lplan that attempts to throttle power consumption to180W on a machine over some time period.

120

130

140

150

160

170

180

0 200 400 600 800 1000 1200

Pow

er (

W)

Time (s)

No PlanPlan

Figure 13: Power Tradeoffs of a Planning Policy.

Consider the ‘mapping’ PM-Lmin policy explained earlier.This policy is ‘aggressive’ in that each VM is treated inde-pendently and scaled down as much as possible. A globalplanning policy, however, may wish to limit scaledown inorder to complete all existing transactions prior to the largebatch arrival, particularly when multiple VMs are involved.Figure 13 depicts measurements of an illustrative examplein which a large batch job arrives after 700 seconds. The“no plan” approach illustrates what happens if the minimalVPM state corresponding to each VM’s currently desiredQoS is assigned to each machine. With this approach, ini-tial power consumption is low, just under 140W. In orderto provide adequate performance when the second VM re-ceives requests, power exceeds the desired throttling point.Therefore, the policy would have to reduce the VPM stateof one or both VMs, thereby violating their performancerequirements. On the other hand, using past VM requestbehavior, a planning policy can predict the future high de-mand experienced by the second VM, and preemptively pro-vide additional performance to the first VM, so that it maycomplete its transactions. In Figure 13, the “plan” resultsshow that the throttling point is maintained using this ap-proach. This illustration underlines that infrastructures likeVirtualPower must be able to run diverse and rich policies,including those that require past observation and/or use pre-dictive techniques.

7.2 PM-G Policies: Global CoordinationCoordination across multiple local policies has already

been recognized as an important element of any manage-ment infrastructure. A specifically interesting case for powermanagement is that global policies can exploit the ever-increasing efficiency in idle, standby, and sleep power soughtand attained by hardware vendors, by using VirtualPower’s

consolidation mechanism. It is a well-known concept, ofcourse, that consolidation can be used to minimize the num-ber of currently active resources so that standby and sleepstates can be used. The new capability provided by Virtu-alPower is that aggressive soft scheduling based on observedVM-level management requests can be used to substantiallyimprove consolidation-based power management. In addi-tion, we show that a PM-G policy that combines consolida-tion with migration can perform power efficient allocationsin heterogeneous environments.

Figure 14: Consolidation with Sleep States.

Soft Scale-Enabled Consolidation.To understand the benefits of consolidation, particularly

in multicore machines, consider Figure 14. The figure pro-vides power measurements of two Pentium 4 machines whenthere are two transactional VMs, each with the same desiredprocessing rate. Both machines run the mapping PM-Lmin

policy described earlier. The figure compares the measuredpower consumption of the two machines when the VMs areon separate physical machines vs. being consolidated ontoone multicore machine vs. the projected power consump-tion when there is consolidation and when the idle platformis shut down or placed into a low power sleep/standby state(which we assume consumes close to zero power). We seefrom the figure that at high processing rates (i.e., high VPMstates), there is an up to 10% power improvement derivedfrom consolidation. This is due to the power characteristicsof the multicore chips where when one core is being highlyutilized, from a global power perspective, it is better to usethe second core on the same chip. At lower request rates,however, we see that the only significant power reductionstems from the act of placing the unused resources, in thiscase a machine, into a low power/off state.

A VirtualPower-enabled system can aggressively exploitthe power characteristics of multicore chips. Specifically,by monitoring lower level PM-L policies, a PM-G policy candetermine whether and when soft scaled VMs should be con-solidated. In our implementation, PM-L policies spawn com-munication threads that listen for information requests fromPM-G policies. Upon a request, PM-L policies respond witha list of guest domains currently executing and the shadowVPM states used to run them. The PM-G policy can thenact based upon its knowledge about the use of shadow VPMstates on some set of machines. Given the definition of ourVPM states, of particular interest are VMs that run at statesP3 and P4 for long durations. An illustrative example isone in which four transactional VMs with normalized de-sired processing rates of 0.4 are deployed onto two physical

machines. Table 2 provides the power consumption of thesemachines under various configurations.

Table 2: Managing Two Machines with Four VMs.

Configuration Power (W)

Two VMs per machine (no PM) 490WTwo VMs per machine (PM-Lmin) 318WConsolidated with 1 idle machine 362WConsolidated with 1 sleep/off machine 242W

The scenario illustrates several interesting facts. First,when consolidating all four VMs onto one machine, thesemachines are set to run at P3, or the 1.6GHz VPM state.The implementation of this state reduces the sizes of VMslices by setting the physical frequency of the two cores to3.2GHz (rather than the lower 2.8GHz). This implementa-tion of the VPM state consumes more power, but it alsoallows for improved power consumption from a global per-spective (i.e., due to consolidation and the associated effi-cient use of sleep power). Indeed, we see from Table 2 thatpower consumption increases under consolidation, comparedto two active machines managed by PM-Lmin policies, untilthe idle machine is placed into a sleep mode which provides a24% reduction in power consumption across both machines.

Exploiting Heterogeneity in Power Management.Upgrade cycles and cost considerations are causing in-

creased levels of platform heterogeneity in datacenter envi-ronments. Even within a platform and processor generation,there may be differences between components. In particu-lar, due to fabrication issues, it is becoming necessary to binprocessors into different groups based on their frequency andmanagement capabilities. Therefore, our next experimentalscenario considers the heterogeneity case in which there aresome machines that support hardware scaling, while othersdo not. In our testbed, this simply involves treating someof our Pentium machines as non-scalable. Figure 15 showshow this fact can affect the power consumption of one of ourtransactional VMs. With varying processing rates, we seefrom the figure that the benefits of hardware scaling supportcan be significant, up to 8%, especially for application VMsthat can be executed at reduced VPM states.

Figure 15: Power Management Heterogeneity.

VM consolidation and migration are important methodsfor dealing with PM heterogeneity. To illustrate, considerthree transactional VMs with normalized performance re-quirements of 0.4 running on a single machine. On a plat-form with hardware scaling and a PM-Lmin policy, these

workloads consume 180W, but they consume 218W on aplatform without hardware management support. This con-stitutes a 17% reduction in power. It is clear therefore, thatin the context of VirtualPower-enabled consolidation, PM-Gpolicies can and should be designed to intelligently leveragehardware manageable components whenever permitted byVMs’ performance requirements.

Resource Heterogeneity Exploitation with VPM States.

The final results shown in this paper concern aggressiveconsolidation to exploit heterogeneity in the power and per-formance tradeoffs of datacenter platforms. In particular,we include in our experimental setup a dual core platformusing Intel’s new Core microarchitecture, and we enable liveVM migration as one of the VPM mechanisms offered toVPM rules. In this setup and for the transactional VMsused in our work, migration time is measured to be, on av-erage, 8.5 seconds when the machine is executing two VMs,and 4.4 seconds when there is only one VM.

350

400

450

500

550

600

0 50 100 150 200 250 300P

ower

(W

)

Time (s)

Figure 16: Consolidation with Heterogeneity.

The experiment deploys four transactional VMs with nor-malized rate requirements of 1.0, two on each dual core Pen-tium 4 machine. All machines run a PM-Lmin policy, andthere is a PM-G policy whose goal is to maximize usageof the Core microarchitecture-based system since it is quitepower efficient and also provides significant performance. Infact, it consumes around 140W when running both cores ascompared to the 240W used by the Pentium based machines.Moreover, it can execute transactions at roughly 2.4 timesthe rate of the older machines.

The PM-G policy begins by offloading two VMs from oneof the P4 platforms and onto the Core microarchitecture-based platform. Due to the improved performance expe-rienced by the VMs after migration, they request reducedperformance states. The ability to obtain these hints trans-parently across these heterogeneous platforms highlights thebenefits of our VPM state abstraction for VM policies. Overtime, the PM-G policy detects that the PM-Lmin policyrunning on the Core microarchitecture machine has reducedallocations to the existing VMs, freeing up room for addi-tional consolidation. As shown by the experiment’s powertrace in Figure 16, this causes a chain of migrations. Here,the Pentium machines are not powered down or placed intosleep states, but instead, continue to consume idle power.Nonetheless, we observe power savings of almost 200W, or34%, by allowing for extensive use of the more power-efficientresources when all four VMs are finally migrated. Thisexperimental scenario highlights the ability of the Virtu-

alPower infrastructure to support complex policy strategiesthat leverage heterogeneity, as well as the ability of its VPMchannels and VPM states to provide seamless coordinationacross systems and migration activities.

8. CONCLUSIONS AND FUTURE WORKPower management has become an important problem

in enterprise systems, where costs as well as limitationsin cooling and power delivery have made it necessary toextend active power management support into these envi-ronments. At the same time, end users are deploying newmethods for system virtualization, to benefit from their iso-lation properties and/or from the increased levels of flexibil-ity derived from server consolidation. Our research has ex-tended virtualization solutions to support rich and effectivepolicies for active power management. The VirtualPowerinfrastructure presented in this work advocates two basicideas: (1) to present guest virtual machines with what ap-pears to be a rich set of ‘soft’ power states accessible totheir application-specific policies, termed VPM states, andthen (2) to use the state changes requested by VMs as in-puts to virtualization-level management policies. These in-clude local policies architected to efficiently use specific plat-forms and their power management capabilities, coupledwith global policies that consider goals derived from entireapplications running across multiple machines and/or de-rived from global constraints, such as blade- or rack-levellimitations on maximum power consumption. Extensive ex-perimentation with multiple processors, including the newpower-efficient Intel Core microarchitecture, demonstrate thebenefits of active power management with VirtualPower aswell as the suitability of VirtualPower’s abstractions of VPMstates, channels, mechanisms, and rules. They also demon-strate that with the state-based guidance approach to ac-tive power management realized in VirtualPower, it is possi-ble to capture and respond to the application-specific powermanagement desires and policies implemented in guest vir-tual machines without any need for application specificityat the virtualization level. Experimental evaluations alsoshow substantial benefits derived from VirtualPower’s use(i.e., with improvements in power consumption of up to 34%without appreciable losses in performance).

Our future work will extend the VirtualPower infrastruc-ture across multiple dimensions. One of these is to includeabstractions and mechanism support for VM-specific powerthrottling and power allocation. The idea is to ensure ‘fair’power allocations across different VMs. An outcome of suchfairness would be the ability to prevent power viruses fromdamaging machine performance. Another set of extensionswill consider the potential performance and thus, power de-pendencies in applications like RUBiS, where often, it isonly certain application components in certain VMs thatcritically affect application performance. Runtime methodsfor identifying these via VirtualPower’s state-based guidanceapproach would permit downscaling of all but the most im-portant VMs for these applications.

9. ACKNOWLEDGMENTSWe thank the anonymous reviewers and our shepherd,

Mendel Rosenblum, for their feedback on improving this pa-per. This work has been supported by an NSF ITR awardand by Intel Corporation via an Intel Foundation Student

Fellowship. Extensive exposure to modern multicore plat-forms and virtualization technologies at Intel Corporationhave been invaluable to our research, including discussionswith Intel personnel, most notably Eugene Gorbatov andRob Knauerhase. We also acknowledge additional inputsand support from Dilma Da Silva and Freeman Rawson atIBM. Finally, we would like to thank Intel Corporation fortheir generous donation of resources and platforms used toperform this work. Ripal Nathuji was awarded an SOSPstudent travel scholarship, supported by Infosys, to presentthis paper at the conference.

10. REFERENCES[1] S. Agarwala, F. Alegre, K. Schwan, and

J. Mehalingham. E2eprof: Automated end-to-endperformance management for enterprise systems. InProceedings of the IEEE/IFIP InternationalConference on Dependable Systems and Networks(DSN), June 2007.

[2] J. Almeida, V. Almeida, D. Ardagna, C. Francalanci,and M. Trubian. Resource management in theautonomic service-oriented architecture. InProceedings of the IEEE International Conference onAutonomic Computing (ICAC), June 2006.

[3] Amazon Elastic Compute Cloud.http://aws.amazon.com/ec2.

[4] P. Barham, B. Dragovic, K. Fraser, S. Hand,T. Harris, A. Ho, R. Neugebauer, I. Pratt, andA. Warfield. Xen and the art of virtualization. InProceedings of the ACM Symposium on OperatingSystems Principles (SOSP), 2003.

[5] D. Brooks and M. Martonosi. Dynamic thermalmanagement for high-performance microprocessors. InProceedings of the 7th International Symposium onHigh-Performance Computer Architecture (HPCA),January 2001.

[6] J. Chase, D. Anderson, P. Thakar, A. Vahdat, andR. Doyle. Managing energy and server resources inhosting centers. In Proceedings of the 18th Symposiumon Operating Systems Principles (SOSP), 2001.

[7] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul,C. Limpach, I. Pratt, and A. Warfield. Live migrationof virtual machines. In Proceedings of the 2ndACM/USENIX Symposium on Networked SystemsDesign and Implementation (NSDI), May 2005.

[8] E. N. Elnozahy, M. Kistler, and R. Rajamony.Energy-efficient server clusters. In Proceedings of theWorkshop on Power-Aware Computing Systems,February 2002.

[9] X. Fan, W.-D. Weber, and L. Barroso. Powerprovisioning for a warehouse-sized computer. InProceedings of the International Symposium onComputer Architecture (ISCA), June 2007.

[10] M. Femal and V. Freeh. Boosting data centerperformance through non-uniform power allocation. InProceedings of the IEEE International Conference onAutonomic Computing (ICAC), 2005.

[11] K. Flautner and T. Mudge. Vertigo: Automaticperformance-setting for linux. In Proceedings of theSymposium on Operating Systems Design andImplementation (OSDI), 2002.

[12] J. Flinn and M. Satyanarayanan. Energy-aware

adaptation for mobile applications. In Proceedings ofthe Symposium on Operating Systems Principles(SOSP), December 1999.

[13] S. Ghiasi, T. Keller, and F. Rawson. Scheduling forheterogeneous processors in server systems. InProceedings of the International Conference onComputing Frontiers, 2005.

[14] S. Graupner, R. Konig, V. Machiraju, J. Pruyne,A. Sahai, and A. V. Moorsel. Impact of virtualizationon management systems. Technical report,Hewlett-Packard Labs, 2003.

[15] T. Heath, A. P. Centeno, P. George, L. Ramos,Y. Jaluria, and R. Bianchini. Mercury and freon:Temperature emulation and management in serversystems. In Proceedings of the InternationalConference on Architectural Support for ProgrammingLanguages and Operating Systems (ASPLOS),October 2006.

[16] T. Heath, B. Diniz, E. V. Carrera, W. Meira Jr., andR. Bianchini. Energy conservation in heterogeneousserver clusters. In Proceedings of the 10th Symposiumon Principles and Practice of Parallel Programming(PPoPP), 2005.

[17] Hewlett-Packard, Intel, Microsoft, Phoenix, andToshiba. Advanced configuration and power interfacespecification. http://www.acpi.info, September 2004.

[18] C. Isci, G. Contreras, and M. Martonosi. Live, runtimephase monitoring and prediction on real systems withapplication to dynamic power management. InProceedings of the 39th International Symposium onMicroarchitecture (MICRO-39), December 2006.

[19] Y. Koh, R. Knauerhase, P. Brett, M. Bowman,Z. Wen, and C. Pu. An analysis of performanceinterference effects in virtual environments. InProceedings of the IEEE International Symposium onPerformance Analysis of Systems and Software(ISPASS), 2007.

[20] R. Kotla, S. Ghiasi, T. Keller, and F. Rawson.Scheduling processor voltage and frequency in serverand cluster systems. In Proceedings of the Workshopon High-Performance, Power-Aware Computing(HP-HPAC), 2005.

[21] R. Kumar, D. Tullsen, P. Ranganathan, N. Jouppi,and K. Farkas. Single-isa heterogeneous multi-corearchitectures for multithreaded workload performance.In Proceedings of the International Symposium onComputer Architecture (ISCA), June 2004.

[22] C. Lefurgy, X. Wang, and M. Ware. Server-level powercontrol. In Proceedings of the IEEE InternationalConference on Autonomic Computing (ICAC), June2007.

[23] H. Li, C. Cher, T. Vijaykumar, and K. Roy. Vsv:L2-miss-driven variable supply-voltage scaling for lowpower. In Proceedings of the IEEE InternationalSymposium on Microarchitecture (MICRO-36), 2003.

[24] M. Lim, V. Freeh, and D. Lowenthal. Adaptive,transparent frequency and voltage scaling ofcommunication phases in mpi programs. InIEEE/ACM Supercomputing, November 2006.

[25] J. Moore, J. Chase, P. Ranganathan, and R. Sharma.Making scheduling cool: Temperature-aware workloadplacement in data centers. In Proceedings of the

USENIX Annual Technical Conference, June 2005.

[26] R. Nathuji, C. Isci, and E. Gorbatov. Exploitingplatform heterogeneity for power efficient data centers.In Proceedings of the IEEE International Conferenceon Autonomic Computing (ICAC), June 2007.

[27] R. Nathuji and K. Schwan. Reducing system levelpower consumption for mobile and embeddedplatforms. In Proceedings of the InternationalConference on Architecture of Computing Systems(ARCS), March 2005.

[28] G. Neiger, A. Santoni, F. Leung, D. Rodgers, andR. Uhlig. Intel virtualization technology: Hardwaresupport for efficient processor virtualization. In IntelTechnology Journal(http://www.intel.com/technology/itj/2006/v10i3/),August 2006.

[29] Nutch. http://lucene.apache.org/nutch.

[30] P. Pillai and K. Shin. Real-time dynamic voltagescaling for low-power embedded operating systems. InProceedings of the 18th ACM Symposium on OperatingSystems Principles (SOSP), October 2001.

[31] C. Poellabauer, L. Singleton, and K. Schwan.Feedback-based dynamic frequency scaling formemory-bound real-time applications. In Proceedingsof the 11th Real-Time and Embedded Technology andApplications Symposium (RTAS), March 2005.

[32] K. Rajamani and C. Lefurgy. On evaluatingrequest-distribution schemes for saving energy inserver clusters. In Proceedings of the IEEEInternational Symposium on Performance Analysis ofSystems and Software (ISPASS), March 2003.

[33] P. Ranganathan, P. Leech, D. Irwin, and J. Chase.Ensemble-level power management for dense bladeservers. In Proceedings of the International Symposiumon Computer Architecture (ISCA), 2006.

[34] B. Seshasayee, R. Nathuji, and K. Schwan.Energy-aware mobile service overlays: Cooperativedynamic power management in distributed mobilesystems. In Proceedings of the IEEE InternationalConference on Autonomic Computing (ICAC), June2007.

[35] J. Stoess, C. Lang, and F. Bellosa. Energymanagement for hypervisor-based virtual machines. InProceedings of the USENIX Annual TechnicalConference, June 2007.

[36] J. Sugerman, G. Venkitachalam, and B.-H. Lim.Virtualizing i/o devices on vmware workstation’shosted virtual machine monitor. In Proceedings of theUSENIX Annual Technical Conference, 2001.

[37] W. Yuan and K. Nahrstedt. Energy-efficient softreal-time cpu scheduling for mobile multimediasystems. In Proceedings of the ACM Symposium onOperating Systems Principles (SOSP), 2003.

[38] H. Zeng, C. Ellis, A. Lebeck, and A. Vahdat.Currentcy: A unifying abstraction for expressingenergy management policies. In Proceedings of theUSENIX Annual Technical Conference, June 2003.

[39] Q. Zhu, Z. Chen, L. Tan, Y. Zhou, K. Keeton, andJ. Wilkes. Hibernator: Helping disk arrays sleepthrough the winter. In Proceedings of the 20th ACMSymposium on Operating Systems Principles (SOSP),October 2005.