Optimizing Energy-Efficiency in High-AvailableScientific Cloud Environments

Altino M. SampaioCIICESI, Escola Superior de Tecnologia e Gestao de

Felgueiras, Instituto Politecnico do Porto

Felgueiras, Portugal

Email: ams@estgf.ipp.pt

Jorge G. BarbosaLIACC, Departamento de Engenharia Informatica

Faculdade de Engenharia, Universidade do Porto

Porto, Portugal

Email: jbarbosa@fe.up.pt

Abstract—Virtualization technologies empower constructionof flexible computing environments, promising an opportunityfor energy and resource cost optimization, while enhancingsystem availability and achieving high performance. A crucialrequirement for effective consolidation is to be able to efficientlyutilize system resources for high-availability computing, andenergy-efficiency optimization, so as to reduce operational costsand carbon footprints to the environment. In this work, wepropose a consolidation technique to improve the performanceof energy- and reliability-aware scheduling algorithms. For that,we carefully tune an energy optimization mechanism, whichdetects energy optimizing opportunities, and executes power- andfailure-aware decision making algorithms to readjust virtual-to-physical mappings. We conduct simulations injecting syntheticjobs which characteristics follow the last version of the GoogleCloud tracelogs. The results indicate that our strategy improveswork per Joule ratio in about 9.7%, as well working-efficiencyin almost 15.6%, maintaining similar levels of completion jobs.

Keywords—cloud computing; virtualization; fault-tolerance;energy-efficiency.

I. INTRODUCTION

Cloud computing has recently gained popularity as a re-source platform for on-demand, high-availability, and high-scalability access to resources. It represents a specializeddistributed computing paradigm, providing better use of dis-tributed resources, while offering dynamic flexible infrastruc-tures and quality of service (QoS). From the clouds’ users pointof view, it gives the illusion of infinite computing resources,available on-demand [1], [2]. Virtualization [3] is an essentialtechnology in cloud computing, because it renders flexibleand scalable system services, creating a powerful computingenvironment. In fact, with virtual machine (VM) technologiesresources can be dynamically allocated, expanded, shrunk ormoved, as demand varies. Moreover, with the developmentof virtualization tools like KVM, Xen, VMware, etc., newcapabilities are introduced, such as VM live migration, VMcheckpoint/restart, and VM pause/unpause, while maintainingthe performance overhead for applications within acceptablerange [3], [4].

Nowadays, cloud computing is progressively being adoptedin different scenarios, such as business applications, socialnetworking, scientific computation and data analysis exper-iments, etc. [5], [6]. Scientific computing usually involvesthe selection, construction and maintenance of large systems.The use of cloud computing in this scenario is advantageous,

since users do not have to concern about managing andmaintaining their own hardware and software infrastructure, aswell with service placement and availability issues. Scientificapplications and experiments are usually composed of a certainnumber of tasks, with various computation and data needs.By using cloud based technologies, scientists can have easyaccess to large distributed infrastructures, with the abilityto scale up and down the computing resources accordingto the applications needs. Virtualized cloud resources enableperformance isolation, on-demand creation and customizationof execution environments, from which scientists can benefit.From the scientists point of view, cloud computing providesmany resources as they need, when they need them and for aslong as they need them.

Despite all these advantages delivered by cloud com-puting paradigm, remains, though, big challenges related toautonomous management of virtualized infrastructures, such asenergy efficiency, and preserve of users’ service level agree-ments (SLA) in face of infrastructure failures [7]. In fact, cur-rent clusters infrastructures supporting cloud computing con-sume huge amounts of electrical energy, contributing to highoperational costs and carbon footprints to the environment. Inthis context, it is of paramount importance to adopt measuresto ensure efficient processing and utilization of the computinginfrastructure, so as to reduce energy consumption. From thepoint of view of cloud providers, dynamic consolidation ofVMs represents a very useful technique to increase profitmargin by reducing energy costs. It consists in consolidatingVMs in the minimum number of physical machines (PMs),in order to reduce the overall power consumption. However,reducing the power consumption is challenging since it canlead to SLA violations. On the other hand, as networkedcomputing infrastructures continue to grow in scale and incomplexity, mean time between failures (MTBF) contract[8]. Failures occurrences negatively impact operational costsas well, due to its influence on systems performance, andconsequent effect in SLAs.

In this paper, we study the impact in energy-efficiency andcompletion rate of users’ jobs, derived from the application ofa sliding-window condition detection mechanism that detectsenergy optimizing opportunities and interacts with schedulingalgorithms to efficiently manage a cloud infrastructure, con-cerning energy. It works by dynamically consolidating sets ofindependent tasks, which compose a scientific job, in whichthe resources in the computing infrastructure are subject of

2013 IEEE Third International Conference on Cloud and Green Computing

978-0-7695-5114-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CGC.2013.20

76

2013 IEEE Third International Conference on Cloud and Green Computing

978-0-7695-5114-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CGC.2013.20

76

2013 IEEE Third International Conference on Cloud and Green Computing

978-0-7695-5114-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CGC.2013.20

76

failure. We consider the problem of dynamically map VMs toPMs, in a power- and failure-aware manner. For that, thesevirtual-to-physical resources mapping decisions consider boththe performance status, power-efficiency, and reliability levelsof compute nodes. The goal is to extend the work presentedin [9], which constructs reliable and high-available computingenvironments, by carefully tuning the energy optimizationmechanism, in order to improve energy-efficiency of the cloudinfrastructure, while maximizing the rate of completed jobs.

The remainder of this paper is organized as follows. SectionII introduces and discusses related work based on dynamicplacement and provisioning of VMs. Section III presents indetail the proposed energy optimizing mechanism for dynamicscheduling of VMs, in an energy- and failure-aware manner.Section IV describes the simulation set up, presents the simu-lation results, and discusses the performance of the proposedalgorithms. Section V concludes the paper.

II. RELATED WORK

This paper provides a different approach to dynamicscheduling of virtual machines in clusters, by considering bothoptimization of energy-efficiency and physical nodes relia-bility, so as to provide energy and high-available computingimproved environments. There has been important work re-searching very close issues. Concerning resource consolidationand optimization of power consumption, Entropy [10] aims todynamically reduce the number of active nodes, while main-taining performance in homogeneous clusters. It uses Choco[11] library to solve the constraint satisfaction optimizationproblem and takes migration overhead into account. vGreen[12] exploits the relationship between a VM’s characteristics,such as instructions per cycle, memory accesses, etc., and itsperformance and power consumption. Authors contribute witha novel hierarchical metric that capture power and performanceprofiles on the VMs. PsciMapper [13] attempts to consoli-date workloads with dissimilar requirements, and reduce totalpower consumption and resource requirements simultaneously,with lower execution slowdown. Workloads are consolidated tothe given set of servers using the Nelder-Mead [14] algorithm.Laszewski et al. [15] focus on reducing power consumptionin high performance cluster computing. Authors minimize theCPU consumption by dynamically scaling down processorfrequencies, which minimizes the processor supply voltage,without significantly increase the overall VMs execution time.Xu and Fortes focused on initial [16] and dynamic [17]mapping of VMs to physical resources to simultaneouslyminimize total resource wastage, power consumption, andthermal dissipation costs. They proposed an improved geneticalgorithm with fuzzy logic, and multi-objective utility function,to find the best solution. Authors also defined threshold valuesand observation windows sizes for condition detection andstabilization of computing infrastructure. PADD scheme [18]proposes to minimize energy consumption while satisfyingSLAs requirements. It works by applying safety capacitiesmargins and dynamically migrating VMs onto fewer PMsduring periods of low utilization, and expanding them ontomore nodes if usage rises. SLA violations can occur, sincePADD scheme is best-effort based.

Various works addressed the issue of computing environ-ment availability and reliability, targeting SLA fulfillment.

For example, Feller et al. built Snooze [19] to dynamicallyconsolidate workloads in heterogeneous resources. The con-solidation takes into account VMs migration and CPU usage,and applies replication technique to achieve fault-tolerance.Loveland et al. [20] combines virtualization technology toprovide high-availability (HA) configurations based on re-dundancy (such as active/active and active/passive), whileminimizing costs. Nagarajan et al. [21] addressed the firstcomprehensive study of proactive fault tolerance (FT) usingVM migration mechanism. Authors exploit Xen to migrate aMPI task from a health-deteriorating node to a healthy one.Their solution integrates intelligent performance monitoringinterface for health inquiries with Ganglia, which determinesnode targets based on load averages to where migration shouldoccur. Each node in the cluster has an approximate view of theentire cluster. It combines proactive and reactive mechanismsto decrease the cost of reactive fault tolerance by loweringthe checkpoint frequency. In case of node failure withoutprior health deterioration symptoms, the system automaticallyreverts to reactive fault tolerance approach, by restartingfrom the last checkpoint. Song [22] investigates and proposesfailure-aware node selection strategies for the construction andreconfiguration of VMs to enhance system availability andachieving high performance. The approach leverages proactivefailure management techniques, based on VMs migrations,and considers both the performance and reliability status ofcompute nodes in making selection decisions. Author proposesOptimistic Best-Fit (OBFIT) and Pessimistic Best-Fit (PBFIT)algorithms to determine the best qualified nodes to allocateVMs which will run user jobs. Experiments showed that thehigher rate of successfully completed jobs was achieved byusing OBFIT and PBFIT strategies. Results evinced that theapproach enabled an increase of 17.6% in job completion ratecompared with that achieved in the current LANL HPC cluster.

All these works address specific issues, such as minimiza-tion of power consumption, SLA fulfillment, etc. However,our work is distinct in tasks consolidation in virtualizedenvironments, since it aims to improve energy-efficiency, andcompletion rate of users’ jobs through proactive fault-tolerancetechnique. For that, we take into account failure predictions inthe computing infrastructure, and the power consumption atconstant sampling intervals, to dynamically adapt the comput-ing infrastructure to produce the same work with less energy.

III. ENERGY- AND RELIABILITY-AWARE SCHEDULING

This section provides the formal description for a energy-efficient and failure-aware enabled cloud architecture, whichdynamically maps VMs to PMs, so as to improve completionrate of users’ jobs. This work extends [9], [17], [22] by intro-ducing a consolidation mechanism to improve the performanceof the schedules produced by those algorithms.

A. System Overview

We consider a typical private cloud computing resource M ,that is composed of h physical machines, M = {m1, ...,mh}.Physical hosts are homogeneous having the same CPU capac-ity C, memory capacity R, network bandwidth N , equal accessto a shared storage space S for storing VMs disk images,and a predicted time in future for occurrence of failure Fi,which can vary among PMs, mi = {C,R,N, S, Fi}. Failure

777777

events are defined as the occurrence of any anomaly, causedby hardware or software faults, and unstable behaviour thatprecludes computing infrastructure components to work.

In a typical usage scenario, cloud users submit their jobs,and the cloud manager component reserves the necessaryresources from the cloud infrastructure to run that jobs. Eachjob j = (Tj , dj) is composed by a set of n independentCPU-intensive tasks tq ∈ Tj , q ∈ {1, . . . , n}, and a deadlinedj . Once tasks are independent, the job deadline is also thedeadline of its longest task. To each job, the cloud managerruns the dynamic resource allocation algorithm that createsand manages the VMs that will execute the tasks. Each VMwill run on top of one PM at a time. The set of VMsconstitute the user’s virtual cluster execution environment. Ajob is scheduled only if a VM can be assigned to each one ofits tasks. A VM encapsulates the task execution environmentand is the unit of migration in the system. In our model,multiple distinct VMs can be mapped to a single PM. Eachtask has a completion deadline and the VM running it will becollected when the deadline is finished. The cloud manageris responsible to collect the VMs. To each job is given adeadline to complete, which equals the deadline of its longesttask. Jobs deadlines start counting as soon cloud users submitthem. However, cloud computing environment acts as best-effort, meaning that resources could not be available at thattime, delaying the start of jobs execution. We leverage stopand copy migration mechanism to assist the management ofcomputing environment. It provides faster migration processesthan live migration does, which is paramount in the case offailure prediction inaccuracies. Such feature is present in mostof today’s virtualization technologies, such as Xen, allowingus to move VMs from one PM to another with minimal serviceinterruption.

The cloud manager continuously fetches virtual and phys-ical machines status, such as nodes’ reliability, resourcescontention, power consumption, tasks execution progress, etc.Based in collected information, cloud manager makes deci-sions about the opportunity to improve power-efficiency orthe need to migrate VMs to tolerate PMs’ failures. Thisinformation flow, and relative control blocks, is illustratedin Figure 1. Prediction of the occurrence time of PMs nextfailure can be determined using the tool presented in [23].Its authors argue that the predictor tool can accurately predictfailure occurrences with an average accuracy of 76.5%.

Fig. 1. Information flow for autonomous dynamic management of virtualclusters.

If a PM fails unpredictably or fails before its runningVMs conclude the migration process, then those VMs willbe respawned in spared PMs. Improving of energy-efficiencywill be discussed in more detail in subsection III-D. It isthe responsibility of cloud manager to attain the performancerequired by users.

B. Problem Formulation

In this subsection we present the mathematical basis of theoptimization problem to solve energy- and reliability-awaredynamic mappings of VMs to PMs, meeting performanceconstraints.

The power consumed by active physical nodes is mainlydictated by the CPU resource [24]. Hence, it is important toefficiently manage the CPU usage in order to reduce the overallenergy consumed by the cloud infrastructure. Based on [17],the power consumption P of a PMi can be estimated based onthe linear power model Pi = p1+p2×CPU%, where CPU%is the percentage of CPU utilization for a given time interval,that is measured for PMi at runtime. The p1 and p2 factorsare, respectively, the power consumption when PM is in idlemode, and the additional portion of power consumption due toCPU utilization, which is typically proportional to the overallsystem load. Equation (1) expresses the power-efficiency for aPM i, at a specific sample time, and denotes how much workis being done by the consumed power.

EPi =CPUi%

p1 + p2× CPUi%× (p1 + p2) (1)

The power-efficiency varies in the interval [0, 1], andreaches its maximum value when CPU usage is 100%. Byefficiently consolidating VMs, idle PMs can be switched tosleep mode, which can improve power consumption if weconsider that an idle PM consumes from 60% to 70% of itstotal power [24].

The optimization problem consists in finding a schedule toall tasks of the submitted jobs so that the energy consumed isminimized. Every task t in all submitted jobs (J) can run in anymachine m ∈M . A schedule is defined by a function schedJ :J → M that assigns to each task t ∈ J a set of machinesMt ⊂ M . Due to node failures, each task may execute ina single machine or may be migrated more than once. Thefailure prediction mechanism allows to specify the off timesof each machine and a schedule for a task may be a set ofmachines where the task executes in one at a time.

The objective function is the maximization of the averagepower-efficiency to execute all jobs, defined by equation 2, forall active physical nodes u, at all sample times f .

EP =

∑fs=1

∑u

i=1EPi

u

f, ∀u ≤ h (2)

The optimization problem is subjected to the followingconditions given by equations (3)–(6) that represent the ca-pacity and reliability constraints. Equation (3) denotes thata task tq can be scheduled in physical node i if its re-sources demand ri(tq), plus the resources required by all tasks

787878

{ri(t1), ..., ri(tq−1)} that run on the same physical node i, donot exceed that node’s capacity Ci.

ri(tq) +

q−1∑k=1

ri(tk) ≤ Ci (3)

Equation (4) indicates that a task can be scheduled ormigrated to a non-reliable physical node i, which is predictedto fail at instant δi, if it provides the minimum resourcesrequired by task tq so that the remaining work, after nodei failure, can be completed by its deadline without exceedingtask maximum required resources, max r(tq).

ri(tq)× δi +max r(tq)× (dtq − δi −mei) ≥Wtq ,

if(δi ≤Wtq

max r(tq)∧ ri(tq) ∈ [min r(tq), max r(tq)])

(4)

where mei is the migration overhead from the current emachine to the next i physical node. The minimum resources,min r(tq), are defined for task tq as the ratio of its workload tothe deadline. Likewise, we also assume that a task tq only mayuse a maximum amount of resources, max r(tq), to executeat maximum speed, which value is defined by the user.

Equation (5) means that if we run a task tq on node iwith failure time δi greater than its deadline dtq , then we canallocate the strictly necessary amount of resources to completethe task within its deadline, without migration.

ri(tq)× dtq ≥Wtq ,

if(δi >Wtq

min r(tq)∧ ri(tq) ∈ [min r(tq), max r(tq)])

(5)

The deadline constraint for task tq is given by equation 6,where FT is the finish time of the task and dtq is the deadlineof tq .

FT (tq) ≤ dtq (6)

The deadline constraint may lead to an empty schedule,depending on the resources available, their reliability, and onthe gap between each task execution time and its deadline. Thealgorithms considered in this paper are best-effort so that theytry to finish as many jobs as possible by their deadlines andby minimizing the energy required to execute them.

C. Scheduling Algorithms

Due to the NP-Complete nature of the scheduling problem,the algorithms that have been proposed are heuristic based andcomply with both objectives in two steps. First, they selectthe PMs that optimize most the overall power-efficiency andthen, prioritize tasks by the difference between task’s deadlineand estimated execution time, in order to maximize the jobscompletion rate.

In this paper we propose a PM consolidation mechanismin order to improve the scheduling algorithms proposed to

optimize the power efficiency. The algorithms considered are[9], [22]: (i) common best-fit (CBFIT); (ii) OBFIT; (iii)minimum time task execution (MTTE); and (iv) relaxed timetask execution (RTTE).

The OBFIT algorithm weights and selects, from a set ofPMs that will not fail before task’s deadline, the PM that hasboth the minimum required capacity and minimum reliabilityto run a VM. In turn, CBFIT strategy selects, from all availablePMs, the PM that has the minimum necessary capacity torun a VM, in order to optimize power consumption. Thus,CBFIT does not take into account nodes’ reliability. Equation(7) represents the reliability weight for a given PMi. The PMreturning the lowest RPMi value is the most reliable.

RPMi =1

2δi−dtq−mei(7)

The MTTE and RTTE strategies start by creating a list ofsorted tasks, in ascending order, according to the differencebetween relative tasks deadlines and their minimum executiontime. Then, the algorithms pick up the first task from thelist, and determine the set of suitable machines through theequations (3) to (5). Then, selects the PM that improves moreits power-efficiency (1). If there is more than one PM with thesame EPi value then the PM with higher reliability is selected(lower RPMi value). In cases where there is no physicalnodes providing the minimum required resources to run ajob by its deadline, that job will wait until cloud computingenvironment releases the necessary resources. The MTTE andRTTE algorithms differ in the sense that the former tries alwaysto reserve the maximum amount of resources needed to executethe task, while the latter only reserves the minimum requiredresources to complete the task within its deadline.

D. PM Consolidation mechanism

The PM consolidation mechanism is proposed here toreduce the consumed energy resulted from the schedules pro-duced by the algorithms referred above, but keeping the samerate of completed user jobs. The consolidation is performedby rescheduling VMs when the usage rate of a PM is belowa given threshold. This work aims to define the conditionsthat should be observed in order to perform a consolidationoperation.

VM migration is already considered in the formulationproblem presented in this section, and occurs if the taskslocated in a PM that is about to fail, do not finish beforethe predicted fail. However, Clouds are intrinsically a verydynamic environments, because resources are constantly beingassigned and released as VMs are created and destroyed. Ad-ditionally, submitted tasks have high heterogeneous resourcesrequirements. This behaviour represents an opportunity tomigrate running VMs during their lifetime to consolidate PMsand optimize the Cloud energy-efficiency.

The consolidation mechanism is implemented by a sliding-window condition detection mechanism. This mechanism al-lows us to tune the window length containing a certain numberof samples for analysis, the threshold value specifying theminimum CPU% utilization below which a consolidation eventis considered, and the required number of those detected events

797979

within the sliding-window to trigger an action. The CPU%utilization samples are taken at 1 minute intervals. In thispaper, we henceforth represent by τ the CPU% utilizationthreshold value, and by γ the number of events detected withinthe sliding-window.

In Figure 2 we present the flow control which is imple-mented by the cloud manager. The sensors data representvirtual and physical machines status, such as nodes’ reliability,resources contention, power consumption and tasks executionprogress. For an active PM (i.e. any PM running VMs), when-ever γ samples, evincing CPU% utilization falling below τ , aredetected within the window analysis, a condition detection istriggered. In such cases, the scheduling algorithm is executedto readjust the VMs to PMs mapping, in order to improvethe overall power efficiency. The optimization of the currentVMs to PMs mapping is carried out in two steps: (i) selectionof VMs that can be migrated; (ii) selection of PMs capableof improving the power efficiency. However, VMs migrationsso as to provide fault-tolerance have priority compared tomigrations targeting power consumption improvement. Thenew placement is achieved using stop and copy migration ofVMs. Additionally, we do not define any maximum CPU%utilization threshold value since we assume that running tasksdo not meet resources scarcity during execution, as cloudmanager only schedules tasks if there exists the minimumnecessary amount of required resources to complete the jobsby their deadlines.

Fig. 2. Flow control implemented in the cloud manager, for failure- andpower-aware reconfiguration of virtual-to-physical mappings.

The energy efficiency improving mechanism took intoconsideration stabilization issues. In fact, during optimizationof the current VMs allocation, additional, and often unsuitable,number of migration processes can occur, which can createcloud infrastructure instability and ultimately lead to lowerrate of completion jobs. This phenomenon occurred becausean excessive number of VMs migrations led to additional delayin completion time of VMs’ migration process, which, in somecases, preclude migrations to complete before nodes’ failures.In order to mitigate such degradation in the algorithms’ perfor-mance, we applied a condition imposing that the next sliding-window samples are ignored for a PM to which all executingVMs have migrated at less than γ samples.

IV. EVALUATION AND RESULTS

In this section we investigate how to tune the sliding-window condition detection parameters in order to achieve

superior energy efficiency. We start by describing the sim-ulation scenario, and then we analyse the trade-off betweenthe completion rate of users’ jobs and energy consumed. Theimpact in the number of VM migrations is also assessed.

A. Performance Metrics

In order to evaluate the performance evidenced by thealgorithms, with and without the consolidation mechanism, weconsider three metrics: (i) completion rate of users’ jobs (8);(ii) the ratio of useful MFlop processed to the energy consumed(9); and (iii) working-efficiency (10).

The completion rate of users’ jobs, RJ (8), is calculatedas the ratio of completed jobs JC with respect to the numberof submitted jobs JS .

RJ =JCJS

(8)

The energy efficiency metric EM presented in (9) evincesthe amount of energy consumed, in Joule, to produce usefulwork. For useful work we mean accounting MFlop from suc-cessfully completed jobs only, JC . It is calculated by dividingthe sum of all tasks’ workloads, of successfully completedjobs, by the overall energy consumption. In turn, the energyis calculated by multiplying the computing infrastructure’saverage power consumption (i.e. for all active physical nodesu, at all sample times f ), with the number of sample timesf , times 60 (since the samples are taken at each minute). Wehenceforth represent this metric as MFlop / Joule.

EM =

∑j(θj ×

∑nt=1 Wt)∑f

s=1

∑u

i=1Pi

u

��f×��f × 60

,

θj =

{1, if job j completed

0, otherwise

(9)

Equation (10) expresses the working-efficiency, EW , andrepresents the quantity of useful work done (i.e. completionrate of users’ jobs) by the consumed power. It is determinedby multiplying RJ , the completion rate of users’ jobs, and theaverage power-efficiency, based on (1), for all active physicalnodes u, at all sample times f .

EW =

∑fs=1

∑u

i=1EPi

u

f×RJ , ∀u ≤ h (10)

Equations (9) and (10) express the amount of usefulwork done at different perspectives. The first, quantifies theamount of useful MFlop by the consumed energy; and thelatter measures the quantity of useful work done, that is thecompletion rate of users’ jobs by the consumed power. Thebest algorithm should be the one that maximizes both metrics,that is processing more MFlop and concluding more jobs, forless energy consumed.

808080

B. Workloads Characteristics

In our simulations, we have created a set of 3614 syntheticjobs, which characteristics are based in the last version ofGoogle Cloud tracelogs [25], [26] and [28]. Basically, thejobs inter-arrival time, tasks length, tasks CPU capacity re-quirements, and number of tasks per job, are modelled usinga Lognormal distribution. Each task deadline is rounded upto 10% more of its minimum necessary execution time. Ajob deadline equals the deadline of its longest task. In [29]is referred that a Weibull distribution, with shape parameterof 0.8, describes well the MTBF of physical nodes and entiresystem. The same authors refer that for the mean time to repair(MTTR), a Lognormal distribution should apply. We have usedthis distributions in our simulations to model failures.

C. Simulation Setup

We have simulated a cloud computing infrastructure com-posed of 50 homogeneous physical nodes. The CPU capacityof physical nodes was considered to be 800 Million FloatingPoint Operations Per Second (MFlops) units, which values canbe evaluated using Linpack [30]. The power consumption offully loaded physical nodes was considered to be 250 W1. Thep1 and p2 parameters, for equation (1), where set to 70% and30% of full power consumption, respectively. VMs migrationoverhead was set to 12 seconds, as measured in [22]. Theaverage MTBF and MTTR where defined to 100 minutes and20 minutes, respectively. We also considered a failure predictortool with average prediction accuracy of 75%, which is aboutthe value referred in [22], [23]. The size of the sliding-windowcondition detection mechanism was set to 5 minutes.

D. Results Analysis

In this subsection we present and discuss the results ob-tained from simulations. First, we try different combinationvalues for CPU% utilization threshold τ , and number ofevents detected γ, to find out which conjunction optimizesthe trade-off between optimization of the energy consumptionand completion rate of users’ jobs.

We first start by evolving the sliding-window parametersτ ∈ {55, 60, 65, 70, 75, 80, 85}%, and γ ∈ {1, 2, 3}. In Figure3 is plotted the amount of useful MFlop executed related tothe overall energy consumed. The combination of τ = 55%with γ = 3 enables RTTE algorithm to achieve the highestratio of useful work done to the overall energy consumption.We next conjugate these results with the ones obtained forworking-efficiency, applying the same combination of sliding-window parameters. The results are illustrated in Figure 4. Inthis case, working-efficiency best result is obtained for τ =55% and γ = 3, which means it is synchronized with bestvalue for MFlop / Joule metric. Moreover, the best value forworking-efficiency is provided again by the RTTE strategy,which reflects a good usage of consumed energy to producework.

Considering the above results, we fixed the energy optimiz-ing sliding-window parameters in τ = 55% and γ = 3, andwe conducted simulations, with and without energy optimizing

1About the average value from ”Fourth Quarter 2012 SPECpower ssj2008Results”, available at http://www.spec.org/power ssj2008/results/res2012q4/.

mechanism, in order to compare the performance improvementin energy consumption for tested algorithms. In Figure 5 ispresented a comparison between the results obtained with andwithout the application of our energy improving mechanism,for MFlop / Joule metric. As we can observe from thegraphic, there is a generalized improvement provided by allalgorithms. In particular, RTTE strategy performs better thanall the other algorithms for both situations, as we expected.With the application of the energy optimizing mechanism,RTTE algorithm improves MFlop / Joule metric in about 9.7%,compared to the result obtained without dynamic optimization.

In the specific case of working-efficiency, the results plot-ted in Figure 6 show the superior performance evidencedby the RTTE algorithm for both cases, with and withoutenergy optimization applied. In fact, and comparing both cases,RTTE algorithm denotes an improvement of almost 15.6% inworking-efficiency. On the other hand, we notice a switching ofbest scheduling performer between OBFIT and CBFIT fromFigure 5 to Figure 6. As already mentioned, MFlop / Joulemeasures the amount of useful MFlop by the consumed energy,while the working-efficiency tries to measure the quantityof useful work (i.e. completion rate of jobs) done by theconsumed power. This means that OBFIT can process superioramount of useful MFlop, than CBFIT does, and using energybetter. However, if we make a coarse-grained analysis aboutthe work produced efficiently (i.e. completion rate of jobs, andnot useful MFlop) by the energy consumed, we conclude thatCBFIT algorithm uses more efficiently the consumed energythan OBFIT does.

Although there is a significant increase in the number ofmigrations incurred, the use of the optimization mechanismdoes not produce significant impact in the completion rate ofusers’ jobs for all the 4 algorithms considered, as shown inFigure 7. CBFIT performs worse in both situations than allthe other algorithms because it does not consider the reliabilityof physical nodes when scheduling tasks, unlike all the otheralgorithms. This explains its high number of migrations andrespawns of VMs incurred for CBFIT algorithm, which valuesare illustrated in Figure 8. A great number of migrationsimposes non-negligible negative impact in the completion rateof jobs.

Fig. 3. Ratio between useful MFlop and energy consumed for diverse energyoptimization γ and τ parameters.

818181

Fig. 4. Working-efficiency for diverse energy optimization γ and τ param-eters.

Fig. 5. Comparison of ratio between useful MFlop and energy consumed,with and without energy optimization mechanism.

On the other hand, OBFIT and MTTE achieves the samelevel of completed jobs with and without the consolidationmechanism. This means that with the mechanism proposedin this paper, the schedules produced by these algorithms aremore energy-efficient without compromising completion jobrate. In the specific case of RTTE algorithm, shown to producethe best performance in figures 3, 4, 5 and 6, there is anegligible degradation of less than 0.48% in the completionrate of users’ jobs, with the application of energy optimizationmechanism. Therefore, we can conclude that the consolidationmechanism improves the performance of the energy- andfailure-aware state of the art scheduling algorithms for cloudinfrastructures.

V. CONCLUSIONS

In this paper, we have studied the application of a mech-anism that detects opportunities to optimize energy consump-tion. The aim is to manage energy- and failure-aware virtualclusters. For that, we have implemented a sliding-windowcondition detection mechanism, and we have analysed the bestcombination of its configurable parameters to improve bothMFlop / Joule and working-efficiency metrics. It works by dy-

Fig. 6. Comparison of working-efficiency, with and without energy optimiza-tion mechanism.

Fig. 7. Comparison of completion rate of users’ jobs, with and withoutenergy optimization mechanism.

namically detecting energy inefficiencies and interacting withstate of the art scheduling algorithms in order to consolidaterunning VMs during their lifetime. We have created a seto synthetic jobs, which characteristics followed the GoogleCloud tracelogs. The results showed that the overall energyconsumed by the infrastructure to produce useful work isoptimized. In particular, we have observed that the energyoptimizing mechanism provided the best energy-efficiency fora minimum CPU% utilization threshold τ = 55%, combinedwith a number of events detected γ = 3, in a 5 minutessliding-window. Considering this configuration, all algorithmsimproved their performance in terms of energy consumption.For the RTTE algorithm, the improvement on MFlop / Joule isabout 9.7%, on working-efficiency is almost 15.6%, comparedto the results obtained without dynamic optimization, andwith a very small negative impact of less than 0.48% in thecompletion rate of users’ jobs.

REFERENCES

[1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski,et al., ”Above the Clouds: A Berkeley View of Cloud Computing,” Dept.Electrical Eng. and Comput. Sciences, University of California, Berkeley,Tech. Rep. UCB/EECS, vol. 28, Feb. 2009.

828282

Fig. 8. Comparison in the number of migrations and respawns, with andwithout energy optimization mechanism.

[2] M. Armbrust, I. Stoica, M. Zaharia, A. Fox, R. Griffith, A. D. Joseph,et al., ”A view of cloud computing,” Communications of the ACM, vol.53, no. 4, 2010, pp. 50-58.

[3] G. Vallee, T. Naughton, C. Engelmann, H. Ong, and S. L. Scott,”System-Level Virtualization for High Performance Computing,” Proc.16th Euromicro Conference on Parallel, Distributed and Network-BasedProcessing, Feb. 2008, pp. 636-643, doi: 10.1109/PDP.2008.85.

[4] K. Ye, X. Jiang, S. Chen, D. Huang, and B. Wang, ”Analyzing andModeling the Performance in Xen-Based Virtual Cluster Environment,”Proc. 12th IEEE International Conference on High Performance Com-puting and Communications (HPCC), Sep. 2010, pp. 273-280, doi:10.1109/HPCC.2010.79.

[5] C. Vecchiola, S. Pandey, and R. Buyya, ”High-Performance CloudComputing: A View of Scientific Applications. Computing,” Proc. 10thInternational Symposium on Pervasive Systems, Algorithms, and Net-works (ISPAN), Dec. 2009, pp. 4-16, doi: 10.1109/I-SPAN.2009.150.

[6] M. Keller, D. Meister, A. Brinkmann, C. Terboven, and C. Bischof,”eScience Cloud Infrastructure,” Proc. 37th EUROMICRO Conferenceon Software Engineering and Advanced Applications (SEAA), Aug.2011, pp. 188-195, doi: 10.1109/SEAA.2011.38.

[7] L. Schubert, ”The Future of Cloud Computing Opportunities for Euro-pean Cloud Computing Beyond 2010,” 2010.

[8] Z. Liu, S. Cho, ”Characterizing Machines and Workloads on a GoogleCluster,” Proc. International Conference on Parallel Processing Work-shops (ICPPW), Sept. 2012, pp. 397-403, doi: 10.1109/ICPPW.2012.57.

[9] A. M. Sampaio, J. G. Barbosa, ”Dynamic Power- and Failure-AwareCloud Resources Allocation for Sets of Independent Tasks,” Proc. IEEEInternational Conference on Cloud Engineering (IC2E), March 2013, pp.1-10, doi: 10.1109/IC2E.2013.16.

[10] F. Hermenier, X. Lorca, J. Menaud, G. Muller, and J. Lawall, ”En-tropy: a Consolidation Manager for Clusters,” Proc. 2009 ACM SIG-PLAN/SIGOPS International Conference on Virtual Execution Environ-ments (VEE), Mar. 2009, pp. 41-50, doi: 10.1145/1508293.1508300.

[11] N. Jussien, G. Rochart, and X. Lorca, ”The CHOCO constraint pro-gramming solver,” Workshop on OpenSource Software for Integer andConstraint Programming OSSICP08, 2008.

[12] G. Dhiman, G. Marchetti, and T. Rosing, ”vGreen: A System forEnergy Efficient Computing in Virtualized Environments,” Proc. 14thACM/IEEE International Symposium on Low Power Electronics and De-sign (ISLPED), Aug. 2009, pp. 243-248, doi: 10.1145/1594233.1594292.

[13] Q. Zhu, J. Zhu, and G. Agrawal, ”Power-aware Consolidationof Scientific Workflows in Virtualized Environments,” Proc. 2010ACM/IEEE International Conference for High Performance Computing,Networking, Storage and Analysis (SC), Nov. 2010, pp. 1-12, doi:10.1109/SC.2010.43.

[14] B. J. A. Nelder, and R. Meadf, ”A simplex method for function

minimization,” The Computer Journal, vol. 7, 1965, pp.308-313, doi:10.1093/comjnl/7.4.308.

[15] G. von Laszewski, L. Wang, A. J. Younge, and X. He, ”Power-aware scheduling of virtual machines in DVFS-enabled clusters,” Proc.IEEE International Conference on Cluster Computing and Workshops(CLUSTER), Aug. 2009, pp. 1-10, doi: 10.1109/CLUSTR.2009.5289182.

[16] J. Xu, and J. A. B. Fortes, ”Multi-Objective Virtual Machine Placementin Virtualized Data Center Environments,” Proc. 2010 IEEE/ACM Int’lConference on Green Computing and Communications & Int’l Confer-ence on Cyber, Physical and Social Computing (CPSCom), Dec. 2010,pp. 179-188, doi: 10.1109/GreenCom-CPSCom.2010.137.

[17] J. Xu, and J. Fortes, ”A Multi-objective Approach to Virtual MachineManagement in Datacenters,” Proc. 8th ACM International Confer-ence on Autonomic Computing (ICAC), Jun. 2011, pp. 225-234, doi:10.1145/1998582.1998636.

[18] M. Y. Lim, F. Rawson, T. Bletsch, and V. W. Freeh, ”PADD: PowerAware Domain Distribution,” Proc. 29th IEEE International Conferenceon Distributed Computing Systems (ICDCS), Jun. 2009, pp. 239-247,doi: 10.1109/ICDCS.2009.47.

[19] E. Feller, L. Rilling, C. Morin, R. Lottiaux, and D. Leprince ”Snooze:A Scalable, Fault-Tolerant and Distributed Consolidation Manager forLarge-Scale Clusters,” Proc. 2010 IEEE/ACM Int’l Conference on GreenComputing and Communications & Int’l Conference on Cyber, Physicaland Social Computing, 2010, pp. 125-132, doi: 10.1109/GreenCom-CPSCom.2010.62.

[20] S. Loveland, E. M. Dow, F. LeFevre, D. Beyer, and P. F. Chan,”Leveraging virtualization to optimize high-availability system con-figurations,” IBM Systems Journal, vol. 47, 2008, pp. 591-604, doi:10.1147/SJ.2008.5386515.

[21] A. B. Nagarajan, F. Mueller, C. Engelmann, and S. L. Scott, ”ProactiveFault Tolerance for HPC with Xen Virtualization,” Proc. 21st AnnualInternational Conference on Supercomputing (ICS), Jun. 2007, pp. 23-32, doi: 10.1145/1274971.1274978.

[22] S. Fu, ”Failure-aware resource management for high-availability com-puting clusters with distributed virtual machines,” Journal of Paralleland Distributed Computing, vol. 70, April 2010, pp. 384-393, doi:10.1016/j.jpdc.2010.01.002.

[23] S. Fu, C. Z. Xu, ”Exploring event correlation for failure prediction incoalitions of clusters,” Proc. 2007 ACM/IEEE conference on Supercom-puting (SC), Nov. 2007, pp. 41:1-41:12, doi: 10.1145/1362622.1362678.

[24] A. Berl, E. Gelenbe, M. Di Girolamo, G. Giuliani, H. De Meer,M. Q. Dang, and K. Pentikousis, ”Energy-Efficient Cloud Computing,”The Computer Journal, vol. 53, no. 7, Aug. 2009, pp. 1045-1051, doi:10.1093/comjnl/bxp080.

[25] Z. Liu, S. Cho, ”Characterizing Machines and Workloads on a GoogleCluster,” Proc. International Conference on Parallel Processing Work-shops (ICPPW), Sept. 2012, pp. 397-403, doi: 10.1109/ICPPW.2012.57.

[26] C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, M. A. Kozuch,”Towards understanding heterogeneous clouds at scale: Google traceanalysis,” Intel Science and Technology Center for Cloud Com-puting, Tech. Rep., 2012, Available: http://www.pdl.cs.cmu.edu/PDL-FTP/CloudComputing/ISTC-CC-TR-12-101.pdf

[27] C. Walck, ”Hand-book on Statistical Distributions for experimentalists,”Particle Physics Group, Fysikum University of Stockholm, Stockholm,Internal Report SUFPFY/9601, Sept. 2007.

[28] I. S. Moreno, P. Garraghan, P. Townend, J. Xu, ”An Approach forCharacterizing Workloads in Google Cloud to Derive Realistic ResourceUtilization Models,” Proc. IEEE International Symposium on Service-Oriented System Engineering (SOSE), March 2013, pp. 49-60, doi:10.1109/SOSE.2013.24.

[29] B. Schroeder, G. A. Gibson, ”A Large-Scale Study of Failures in High-Performance Computing Systems,” IEEE Transactions on Dependableand Secure Computing, vol. 7, no. 4, Oct.-Dec 2010, pp. 337-350, doi:10.1109/TDSC.2009.4.

[30] Linpack, 2012. http://www.netlib.org/linpack/

838383