Achieving Thermal-Resiliency for MulticoreHard-Real-Time Systems

Pradeep M. Hettiarachchi1, Nathan Fisher1, and Le Yi Wang2

1Department of Computer Science, Wayne State University2Department of Electrical and Computer Engineering , Wayne State University

Abstract—Multicore processor based system designs are in-creasingly utilized as the processing platform for complexhard-real-time and embedded applications. These real-time systemsneed to operate under various physical and design constraints.Much research has focused on thermal-aware real-time systemsdesigns. However, no results exist to investigate the resource allo-cation and the system degradation under external thermal con-straints in a predictable manner. This paper proposes a control-theoretic framework to ensure hard-real-time deadlines onamultiprocessor platform in a dynamic thermal environment. Weuse real-time performance modes to permit the system to adapt tochanging conditions. Also, we show how the system designer canuse our framework to allocate asymmetric processing resourcesupon a multicore CPU and still maintain thermal constraints.We develop analysis for determining what modes the system cansupport for a given external thermal condition. Our system designextends the derivation of thermal-resiliency (originally proposedfor uniprocessor systems) to multicore systems and determinesthe limitations of external thermal stress that any hard-real-time performance mode can withstand. Simulations and physicaltestbed results show that our algorithm predicts how a systemwill gracefully and predictably degrade under external thermalstress.

I. I NTRODUCTION

Designing a multicore thermal-aware real-time system iscomplex; processor cores with different thermal characteris-tics are closely packed in a CPU die; therefore, each coremight need a separate thermal model, and thermal interactionbetween cores also needs to be considered. If a proposedsystem operates in a dynamic thermal-varying environmentwith changing temperatures, then the thermal-controller designneeds to ensure the timing constraints of the real-time tasksunder various environmental conditions. Modern embeddedsystems are extremely complex, and, therefore, identificationof some of the system limitations through testing are lim-ited. Furthermore, the system designer is often compelled toconsider the trade-offs between performance and the physicalconstraints of the system (e.g., temperature) due to optimizedform-factor and the thermal-packaging cost. For a smooth,robust design-phase and a reliable final product, the systemde-signer must be equipped with appropriate design frameworksthat helps the designer to evaluate the implication of the trade-offs. In this paper, we propose a method for a system designerto analyze the trade-offs between the real-time performanceand environmental constraints of a hard real-time system.

Many modern processors possess different thermal man-

agement (DTM) capabilities. However, DTM features mustbe properly utilized in hard real-time system designs, other-wise, these complex systems might compromise their timingconstraints. Therefore, a framework that considers all externalsystem capabilities such as DTM and real-time requirementsisessential. Furthermore, a useful framework should also permitthe system designer to predict the behavior of the system,even if the operating environment is not a priori known orpredictable.

There are already available frameworks for permitting atrade-off between a real-time level of service and resourcerequirements. For example, Ghosh et al. [10] proposed aframework for mapping the level of service and resourcerequirements for dynamic environmental conditions. Thesetechniques might guarantee the real-time deadlines for apreviously-determined level of resources, but they do not ad-dress the scenario where requirements can dynamically change(resulting in a real-time mode change). Specifically, theycannot guarantee hard real-time deadlines when a system maydynamically switch between real-time modes. Furthermore,none of these previously-proposed techniques can be used toobtain a precise, formal quantification of the thermal stressthat the muticore system can withstand.

In this paper, we extend the concept ofreal-time thermal-resiliency to multicore platforms. Real-time thermal-resiliencyis a system-design metric that quantifies the maximum externaloperating temperature that the system can withstand for aspecified real-time performance mode. To define thermal-resiliency, let us assume theC’th core of a multicoresystem with q different hard-real-time performance modesMC,0,MC,1, . . . ,MC,q where modes are ordered in increasinglevels of real-time performance withMC,q guaranteeing thehighest level that can accommodate most processor intensivereal-time task, andMC,0 the lowest, that can accommodatereal-time task with the least processing requirement. For thissystem, the real-time thermal-resiliency for thei’th mode ofthe C’th processorMC,i, denoted asΛ(MC,i, Tref), representsthe predicted maximum external operating temperature forwhich the system will continue to operate at performancemodeMC,i or higher and maintain a CPU reference temper-ature ofTref.

Towards this goal, we design a framework to calculate andverify the real-time guarantees of a multicore system in the

presence of unpredictable dynamic environmental conditions.Thus, we are proposing a thermal-stress analysis mechanismfor multicore real-time systems. Using our proposed method,the system designer can specify, a priori, a precise quan-tification of the hard-real-time performance degradation ofmulticore system due to external thermal events. Further,our thermal analysis framework explains how the modes de-grade gracefully and predictably under external thermal stress.Therefore, using our framework, the system designer canpredict the thermal-resiliency of a performance mode. Finally,our framework addresses the issue of allocating asymmetricmode requests from different cores in a multicore CPU.

To summarize, this paper gives the following contributions:

• We propose a controller for a multicore system withrudimentary power-control mechanisms (e.g., cores canbe active or inactive). Our objective is to ensure thatthe maximum core temperature tracks a given referencetemperatureTref. We show that our controller is ableto maintain system stability and controllability. Further-more, using the system specification, we can obtain aclosed-form quantification of the thermal-resiliency foreach operating mode for each core of the system.

• The proposed mechanism is also helpful for analyzingmulticore systems under certain capacity constraints. Forexample, a system designer will often statically pin amission-critical hard-real-time task to a specific core andexecute the remaining, less-critical real-time tasks on theremaining cores. Under this situation, the designer needsto know the the potential capabilities that the systemmight support under various external thermal conditions;we provide formal support to analyze this situation.

• We empirically evaluate the efficacy of our control al-gorithm and associated analysis upon a multicore CPU.We show (through simulations and testbed runs) that ourmodel and characterization of thermal-resiliency closelypredicts the system’s thermal behavior even in a dynamicoperating environment.

§Organization. This paper presents a methodology for de-signing and analyzing thermal-resilient multicore hard-real-time systems. Section II presents a brief review of previouswork on thermal-aware (real-time and non-real-time) computersystems. Section III presents the hardware, real-time, andthermal models used throughout the paper. Section IV detailsthe design of our thermal-resilient controller. Section V derivesthermal-resiliency functionΛ for our control system. SectionVI describes the results of our simulations and implementationupon testbed hardware. Section VII concludes the paper.Although our methodology provides formal system guaranteeswhich require formal derivations and proofs, due to spacelimitations, all formal proofs and derivations are given intheappendix of an extended version of this paper [12].

II. RELATED WORK

In this section, we overview the prominent work andprevious research on control-theoretic-based thermal-aware,

real-time thermal-aware system design, and multicore real-time systems at high-level. The pioneering work by Brooksand Martonosi [3] emphasized the importance of dynamicthermal management schemes and proposed policies andmechanisms for implementing dynamic thermal managementfor a CPU. They developed Wattch, a reference/simulation toolfor architectural-level power modeling. Similarly, theHotSpot,an architectural-level thermal-modeling tool developed bySkadron et al. [21] is also widely used to analysis thermalprofile of micro-architectural level.

Ghosh et al. [10] proposed a framework for mapping,the level of service and resource requirements for dynamicenvironmental conditions. They presented an integrated QoSoptimization, which is performed using Q-RAM [18]. Lu etal. [14] proposed adaptive utilization based multi-processorsreal-time design for constrained MIMO systems. Fu et al. [7]proposed a control based solution for simultaneous thermaland timeliness guarantees for distributed real-time embeddedsystems running in unpredictable environments. Fu et al. [8]proposed a control-theoretic framework for multicore proces-sor real-time systems that adjust the system utilization throughDVFS to meet the thermal goals. In contrast to the work by Fuet al. [7] which relies upon mapping techniques to adapt to thevarying environment conditions and provides only soft-real-time guarantees to adopt the varying environmental conditions,we provide the system designer real-time and performanceguarantees.

Yao et al. [22] discussed an online adaptive mechanism,based on the utilization control, for multiprocessor real-timesettings with online system identification and LQ controllersfor a system with multiple constraints. Fu et al. [6] suggesteda solution that integrates core-level feedback control withprocessor-level optimization to minimize dynamic and leak-age power consumption of a multi-core real-time embeddedsystem. This research may offer accurate control solutionssuitable for soft real-time systems that needs to adjust theutilization by means of task rate adjustments at runtime.However, their methods have limited applicability for systemsthat need hard real-time and multi-mode capabilities. Seo etal. [19] studied energy-efficient multicore real-time schedulingusing processor-wide DVFS; however, their results do notprovide any hard real-time or performance guarantee. Further,each of these prior results do not provide a mechanism tospecify the graceful degradation of the system’s operatingmodes in an unfavorable environment.

While no previously-proposed techniques exist for obtain-ing a formal quantification on a muticore system, we haverecently explored the idea of thermal-resiliency for single-core systems [11]. However, our previous work does notexplain or predict the real-time design and analysis frameworkfor multicore settings, specifically when the heterogeneousexecution (processing) distribution is required by differentcores.

2

III. M ODELS

A. System Hardware Model and Testbed

We develop our power model to represent a wide rangeof embedded processors with a minimal amount of powermanagement capabilities. Therefore, we assume a multicoreprocessor system withactive and inactive power modes.We denote the instantaneous CPU power ofC’th core asPC

cpu(t), (C ∈ {1 . . .m}) and assume it dissipates thermalpower at a constant ratePact andPinc in the active and inactivemodes, respectively. Also, we assume that processor consumeseact andeinc amount of energy to activate/deactivate from inac-tive/active modes. Further, during this interval, the processoris inactive while in the low-power state and unavailable forpayload task execution. If the processor is minimally activeinstead of unavailable during inactive interval, the system willbehave better than the analysis and our results will continueto be valid.

Our control system for the active/inactive processors willenforce strict periodic mode changes. For this purpose, weemploy thermal-aware periodic resource [1] model, whichis an extension of the well-known periodic resource modelproposed by Shin and Lee [20] for compositional real-timesystems. In the thermal-aware periodic resource model, theprocessing resource is characterized with a two-tuple(Π,ΘC).The parameterΠ is called theresource period and ΘC iscalled theresource capacity of theC’th core. We will assumethat Π(> 0) is subject to the system tick granularity. Theinterpretation is that processor will be active forΘ amount oftime at the beginning of each successiveΠ-length intervals.Within each processor allocation, an arbitrary uniprocessorscheduling algorithm (e.g.,EDF or RM) may be employedto schedule the underlying task system. See Figure 1 for anillustration of the thermal-aware periodic resource.

B. System Software Model

We consider a system withm number of CPU coresand each core has specific number of possible performancemodes. Also, the task migration at runtime is not permissibleand the tasks are statically partitioned within the availableprocessors. We assume each performance modeMC,i of theC’th core is characterized by asporadic task system1 [15]with ni tasks and the resource capacityΘC,i, where ΘC,i

represents the minimum resource capacity required fori’thmode. That is,MC,i =

({

τC,i1 , τ

C,i2 , . . . , τC,ini

}

,ΘC,i)

where

each τC,ij ∈ MC,i is a sporadic task characterized by a

three-tuple(eC,ij , dC,ij , p

C,ij ) andΘC,i is the minimum capacity

required to meet the deadlines of the tasks ofMC,i. (Note thatwe are abusing notation by allowingMC,i to represent theset of tasks and the two-tuple of the mode’s task system andrequired resource capacity.) In this three-tuple representationfor a task,eC,ij is the worst-case execution requirement, dC,ij

1Note, we will be assuming the sporadic task model throughoutourobjectives, but the results could be extended to other task models withoutmuch change.

Θ1(i)

Π

Mode Change (Core #1)

Θ1(i) Θ

1(i) Θ1(j)

Mode Change

Θ1(j) Θ

1(j)

. . .

ΘC(i)

Π

Mode Change (Core #C)

ΘC(i) Θ

C(i) ΘC(j)

Mode Change

ΘC(j) Θ

C(j)

. . .

Θ2(i)

Π

Mode Change (Core #2)

Θ2(i) Θ

2(i) Θ2(j)

Mode Change

Θ2(j) Θ

2(j)

. . .

...

...

Fig. 1: The sampling and mode change in our thermal control system.The blocks indicate time periods during with the processor is activeunder the thermal-aware periodic resource model. Sporadictasks arescheduled within the activation blocks .

is therelative deadline, andpC,ij is theminimum inter-arrivalseparation parameter. A sporadic taskτC,ij may produce aninfinite sequence of jobs, where each job has an executionrequirement ofeC,ij time units and must completedC,ij timeunits after its arrival. The first job ofτC,ij may arrive at anytime after system-start time; however, successive jobs ofτ

C,ij

must arrive at leastpC,ij time units apart. For this paper, weassume that the resource periodΠ is identical in all modes.For modeMC,i, a resource capacity ofΘC,i is providedevery resource period. Figure 1 illustrates the processing-timeallocation in two different modes.

We will assume that there is a partial ordering of real-time performance modes based on their “computational re-quirements” to meet all of a mode’s deadlines. The relationMC,i � MC,j indicates thatMC,i is more computationallyintensive thanMC,j. For notational convenience, we willassume that modeMC,0 represents the mode where with notasks andΘC,0 equal to zero. Furthermore, for this paper,we assume that the modes are well-ordered and have beenindexed in increasing order of computational requirements;i.e., MC,0 � MC,1 � MC,2 � . . . � MC,q. While thereare many possible ways to define the� relation, the onlyordering required from the perspective of our thermal controlis that MC,i � MC,j , if and only if, ΘC,i ≤ ΘC,j; i.e., toreduce the temperature of the system, we need to decreasethe processing-time allocation. Further, we do not requireanyspecific relationship with modes on different processors. Forexample, for anyi, there is no required relationship betweenMCp,i and MCq,i (for Cp, Cq ∈ {1, . . . ,m}). Our systemallows jobs of the new mode to be released, as soon as legallyallowable, while jobs of the old mode are still active.

The scheduling of real-time performance modeMC,i uponthe thermal-aware periodic resource may be done by any real-time scheduling algorithm (e.g., earliest-deadline-firstor rate-monotonic [13]). However,ΘC,i must be sufficiently large forthe scheduling algorithm to correctly schedule all jobs of thetask set ofMC,i (i.e., {τC,i1 , τ

C,i2 , . . . , τC,ini

}) and (potentially)any jobs from the previous mode that have not completedby the mode change. To obtain a proper resource allocation,

3

Tenv

P1cpu

Core#1

Core#2Core#3

Core#4

P3cpu

P2cpu

P4cpu

I11I44

I33 I22

I12

I23

I34

I41

Tcpu11

Tcpu22

Tcpu33

Tcpu44

Tcpu12

Tcpuij = Tcpujj − Tcpuii ∀i, j ∈ {1 . . . 4}

Tcpuii, ∀i, j ∈ {1 . . . 4} (i’th CPU Temperature W.R.T. Environment)

P lcpu

Pdcpu

Fig. 2: The basic equivalent electrical circuit of the thermal modelof the CPU and its working environment. (For simplicity, thefigureshows the structure with 4 adjacent cores.) The arrow direction showsthe current (A) direction of the equivalent electrical circuit.

ΘC,i, for each mode, we use our previously-proposed hard-real-time schedulability [5] test (for EDF scheduling underhardware/software mode changes in the periodic resourcemodel) to search for a safe value ofΘC,i for each mode toensure that deadlines are alwaysmet.

C. Power/Thermal Model

The thermal architecture (model) of a multicore processorshould contain a component to represent each processor coreas well as inter-core thermal effects. To make the analysispossible, we use the duality principle in electrical and thermalcircuits to describe the dynamics of the power dissipatingsource using electrical resistance/capacitance (RC) circuits.Figure 2 shows the basic equivalent circuit of a multicoreCPU and its surrounding environment. We assume that thetotal dissipated power ofC’th CPU core,PC

cpu is equal to the

sum of the power due to dynamic currentPdcpu

Cand the power

due to leakage current ofC’th core,P lcpu

C. Also, we assume

that the leakage and the dynamic current parts within differentcores are approximated with single component.

Let V Ccpu(t) andV C

env(t) represent the equivalent voltages fortemperatures of theC’th core of the CPU and environment(room) respectively. LetT C

cpu be the instantaneous relativetemperature of theC’th core of the CPU with respect to theimmediate environment (e.g., CPU casing),T C

env be the relativetemperature of the immediate environment.

Let Penv(t) represents, the environment power dissipation.Let Rd

cpuij, Rlcpuij, Renv, Cd

cpuij, Clcpuij, andCenv represent the dy-

namic and leakage thermal-resistance, environment resistance,CPU dynamic and leakage capacitance, and environment ca-pacitance betweeni’th and j’th cores,i, j ∈ {1 . . .m}.

We can derive the following,

Tcpu(t) = ATcpu(t) + BPcpu(t), (1)

z−1+

+C

G

K

H+

-

∫

+

- γI

ve(k) x(k)y(k)

Tref(k)− Tenv(k)

u(k)

f

MAX(Y )

Fig. 3: The thermal control design with state feedback and integralactuator

where, Tcpu(t) =(

T Ccpu(t)

)

m×1, Tcpu(t) =

(

T Ccpu(t)

)

m×1,

Pcpu(t) =(

PCcpu(t)

)

m×1, andA = (A)m×m,B = (B)m×m

are state-space parameters. The detailed derivations definitionsof these matrices are given in the Appendix of the extendedversion of this paper [12].

IV. CONTROLLER DESIGN

First, we design a thermal controller assuming that an idealsystem with continuous power modes. In Section IV-A, wewill extend the controller design to a processor with onlyactive/inactive power modes.

We need our controller to accurately follow the referencetemperature,Tref. Also, to eliminate steady-state tracking error,we design our system as a servo with an integrator as shownin the Figure 3.

Define an additional error vectorve(t) in continuous timeas,

ve(t)def=

∫ t

0

(Tref − Tcpu(t)− Tenv(t))dt

ve(t)def= Tref − Tcpu(t)− Tenv(t) (2)

= Tref −max(

[

Tcpu(t)]

)

− Tenv(t).

Then, the system input is calculated with a gainKo andintegral constantγI in the following equation,

Pcpu(t) = −Ko

[

Tcpu(t)]

+ γIve(t), (3)

where,Ko = (Kij)m×m. We employ well-established, stan-

dard techniques from optimal control theory to deriveKo

and prove stability. Due to their standard nature, details arepresented in an appendix of an extended version of thispaper [12].

A. Continuous Power Emulation with Active/Inactive PowerModes

In this section, we explain how the resource capacity,ΘC

is manipulated to produce the control input valuePCcpu. Any

modern CPU has a discrete set of operating frequencies. Thepower dissipation (consumption) for each operating frequencyis not a variable and it is fixed.2 However, the control designwe proposed requires a continuous input,Pcpu(t) for its proper

2Under experimental conditions and due to the variation of the workload,we may find slight variation on the power consumption value for a specificoperating frequency for different workload executions; however, for ourexperimental setting, the variation is not large.

4

functionality. Since the CPU power cannot be varied contin-uously, the controller designed in the previous section cannotbe directly applied to the setting of discrete active/inactivepower modes. Therefore, the design of the continuous power-modes controller described in the previous section needs tobemodified to accommodate the active/inactive power mode set-ting by applying pulse-width modulation (PWM) techniques.As we explained earlier in the Section III, the active/inactivepower modes will be modeled via the thermal-aware periodicresource model with parametersΠ and ΘC. This could beachievable via choosing the appropriate values ofΠ andΘC.The Π value is a design parameter which may be chosen atcontroller design-time and will be assumed fixed throughoutcontroller execution. The only constraint that our frameworkplaces on the chosen value ofΠ is that it must evenly dividethe sampling interval lengthTs (i.e., Ts = κΠ for someκ ∈ N

+).LetΘC(k) denote the value of the resource capacity over the

k’th sampling period onC’th core. For determining theΘC(k)value, we use a method based on theprinciple of equivalentareas (PEA) for converting any arbitrary input signal (PC

cpu(k))into an equivalent PWM signal [9] and assume thezero-order hold (ZOH), the input signal is held constant over thesampling period. Specifically, for thek’th sampling interval,the inputPC

cpu(k) is held over theTs-length interval, resultingin a total energy dissipation ofTs · P

Ccpu(k) over the interval.

To get the equivalent area (i.e., energy) as the (ideal) systemwith continuous power modes, we must setΘC(k) such thatthe periodic modulations between the power modes ofPact

andPinc dissipate the equivalent amount of energy over theTs-length interval. Figure 4 illustrates the area equivalencebetween the continuous and PWM controllers.

kth Sample (k + 1)th Sample (k + 2)th Sampletime

P 1(k) P 1(k + 1)P 1(k + 2)

Θ1(k)

Π

Θ1(k) Θ1(k) Θ1(k + 1) Θ1(k + 1) Θ1(k + 1)

kth Sample(k + 1)th Sample (k + 2)th Sample

timeΠ

. . .

. . .

kth Sample (k + 1)th Sample (k + 2)th Sampletime

P C(k)P C(k + 1)

P C(k + 2)

ΘC(k)

Π

ΘC(k) ΘC(k) ΘC(k + 1) ΘC(k + 1) ΘC(k + 1)

kth Sample(k + 1)th Sample (k + 2)th Sample

timeΠ

. . .

. . .

...

...

Fig. 4: The simplified power and modulation relationship

More formally, we may derive the following relationshipbetweenPC

cpu(k) andΘC(k),

κΠPC

cpu(k) = κ(

eact+

∫ ΘC(k)

0

Pactdt

+ einc +

∫ Π

ΘC(k)

Pincdt)

⇒ PC

cpu(k) =

(

Pact− Pinc

Π

)

ΘC(k)

+ Pinc +1

Π(eact+ einc), (4)

whereΘC(k) the resource capacity of theC’th CPU for i’thmode andPC

cpu(k) denotes the relevant power ofC’th CPU,C ∈ {1. . . .m}. Note that, the above Equation 4 gives us away to calculate the total power consumption of any real-timetask by using the time it takes to complete the task and theΘvalue.Algorithm 1 Control Algorithm

Require: Reference TemperatureTref; Feedback GainK; In-tegral ConstantγI ; PWM PeriodΠ; Number of PWMperiods in a sampling periodκ.

1: while At beginning of sampling period[tℓ, tℓ+1) : tℓ ≡κℓΠ do

2: SampleTcpu(tℓ) + Tenv(tℓ).3: ve(tℓ) = Tref − Tenv(tℓ)−max(Tcpu(tℓ))

4: Tot ve(tℓ) = Tot ve(tℓ−1) + γIκΠ

(

ve(tℓ)+ve(tℓ−1))

25: for C = 1 to m do6: PC

cpu(tℓ) =(

Tot ve(tℓ)γA −(

[KC,r]Tcpu(tℓ))

)

; r ∈

{1 . . .m}

7: ΘC(tℓ) = min

(

Π×(PC

cpu(tℓ)−Pinc)−(eact+einc)

Pact−Pinc,Π

)

8: i = max{i ∈ Zq+1 | ΘC,i ≤ ΘC(tℓ)}9: Update real-time performance mode toMC,i.

10: Set PWM to operate at period ofΠ and width ofΘC(tℓ).

11: end for12: end while

The PWM controller pseudocode is presented in Algo-rithm 1. The controller proposed here consists of two in-tegrated operations: the thermal controller and the PWMmodulator. The first step is to obtain the sample CPU tem-perature (Line 2 of Algorithm 1). Then, we select maximumtemperature from theTcpu vector and calculate the error forthe integrator (Line 3).

Next, we calculate the control signal for each input of thesystem as follows: we calculate the target state feedback gainfor each input and subtract it from the integrator error wecalculated in the previous step (Line 6). Next, we calculatethe equivalentΘC from the property of Equation (4) (Line 7)3.Finally, the appropriate mode is selected (Line 8), the modechange is performed (Line 9), and the pulse-width modulatoris invoked for the nextκ Π-length intervals (Line 10). Werepeat these steps (Line 6 to 10) for each CPU core (which iscorresponding to the number of control inputs, according toour model).

It is important to note thatΘC(tℓ) calculated in Line 7 for aC’th Core does not have to be equal theΘC,i for the selectedmode; we must only select the highest mode withΘC,i ≤ΘC(tℓ). (If ΘC(tℓ) is larger, we are only giving the mode moreprocessing than it requires.) It should also be observed that alloperations, except for finding the appropriate mode, may bedone inO(m) time. Finding the highest real-time performancemode that may execute can be done inO(m lg q) time (via

3We assumeeact andeinc are negligibly small and omit in the simulation.

5

binary search) whereq is the number of real-time performancemodes.

V. THERMAL-RESILIENCY CALCULATION

In this section, we explain how to derive the real-timethermal-resiliencyΛ(MC,i, Tref) for a given real-time perfor-mance modeMC,i and reference temperatureTref. Assume thesystem is in the steady-state. Therefore, the error value fromthe integrator output become zero. Then we get the following:

Tref = max(Tcpu(k)) + Tenv(k). (5)

Further, at the steady state, we do not observe any temper-ature increment from the CPU. Assume that the system hasreached the steady-state by the(k − 1)’th sampling period.Therefore,T C

cpu(k) = T Ccpu(k − 1), Tenv(k) = Tenv(k − 1),

and ΘC(k) = ΘC(k − 1). We can calculate theΘC(k) atsteady state. Further, our extended version of this paper ( [12])shows that we can calculate the CPU temperature,T C

cpu(k) fora givenΘC(k) values. Since we are interested in obtainingΛ(MC,i, Tref), we may fixTref andΘC(k) = ΘC,i.

Now now briefly outline how to obtain a solution forΛ(MC,i, Tref).4 We can calculate the steady-state CPU tem-perature using feedback gain equation of the controller. Aswe know that the steady-stateΘC, C ∈ {1 . . .m} for a givenexternal temperature is fixed, we calculate theΘC for variousmodes and will derive the thermal-resiliency. Therefore, fromour schedulability analysis, we calculate the(Θi)m×1 vector,for a given mode set and the we can derive the power of thesystem from Equation 4. Our extended version of this papershows that the feedback value can be calculated asKo and

(

Pcpu(t)C

)

m×1= −Ko

(

T C

cpu(t))

m×1+ γIve(t),

and therefore,

⇒(

T C

cpu(t))

m×1= −Ko

−1(

(

Pact− Pinc

Π

)

(

ΘC(k))

m×1

+ Pinc +1

Π(eact+ einc)) +Ko

−1γIve(t),

(6)

is obtained and it can be easily solvable. We use the Equation5to calculate the thermal-resiliency.

⇒[

Tenv(t)]

= Tref(t) +Ko−1(

(

Pact− Pinc

Π

)

(

Θi(C))

m×1

− Pinc +1

Π(eact+ einc))−Ko

−1γIve(t),

(7)

4The approach may be generalized when there is bounded steady-state error.However, the approach will be similar, and we omit the details due to space.

and therefore,

Λ(M1,i, Tref)...

Λ(Mm,i, Tref)

= Tref +Ko

−1(

(

Pact− Pinc

Π

)

Θ1,i(k)...

Θm,i(k)

− Pinc +1

Π(eact+ einc))−Ko

−1γIve(t).

(8)

TheΘC,i(k) represents the minimum capacity ofi’th modeon C’th core. We will see later, from our simulations and thetestbench runs, that the thermal-resiliency of different cores onthe same CPU does not vary too much; this is self explainable:due to the closely-coupled thermal architecture of the CPU,there is no provision on one CPU to exhibit substantiallyhigher temperate values from rest of the cores. Therefore,we cannot necessarily expect the thermal-resiliency vary fromcore to core.

Further, assume that we assign a predetermined capacityvalue to one of the cores,ΘCa ∀Ca ∈ {1 . . .m}, 0 < ΘCa < Πand calculate theΘC values for the rest of the cores. Assumethe final capacity vector as

(

ΘCf)

m×1. As explained earlier,

we can calculate theTcpu values using(

ΘCf)

m×1, and the

the final resiliency value vector also can be calculated. In thiscase, we generate the resiliency value for the system while afixed capacity was assigned (by design) to a specific core.

VI. VALIDATION

Our evaluation of the proposed method is carried out intwo steps. We first generate actual system parameters fromthe testbed and use them in our simulations. Furthermore,we implement our algorithm in the experimental testbed andcompare the simulation results with testbed observations.

A. Simulations

We use simulations to demonstrate the validity of ourproposed framework. In our simulations, we consider a systemwith 8 CPU cores. In our simulation, first, we calculatethe temperature vector of cores,Tcpu and select its peaktemperature value. Then, we calculate the control input ofthe system according to our algorithm. The following taskparameters are used in our simulations:

• Each sporadic taskτC,ij = (eC,ij , dC,ij , p

C,ij ) has a period

pC,ij uniformly drawn from the interval[5, 15]. The ex-

ecution time requirementeC,ij set to the task utilizationtimespC,ij , where task utilization is calculated using theUUnifast algorithm [2]. For each task,dC,ij equalspC,ij .The tasks are scheduled by EDF in each core.We use a partitioned scheduling; tasks are assigned intoprocessor core statically, while task migration withinprocessor cores is not an option during runtime.

• The total number of tasks is eight for each core; each taskτCj has three different real-time performance modes whereτC,2j = (eC,2j , d

C,2j , p

C,2j ); τC,1j = (.2eC,1j , d

C,1j , p

C,1j ); and

τC,0j means that task is not selected. From set of all

6

possible combinations of tasks, we have selected fifteenmodes with utilizations ranging from zero to one. In orderto accuratelydistinguish hotspotson the processor, thesame tasks/modesare present on all cores.

We refer to the controller described in Algorithm 1 asMulticore Temperature Regulated Optimized Capacity (M-TROC). We do not compare the simulation performance ofour proposed method with any other algorithm due to the lackof our knowledge on any other research that considers theadaptive thermal-aware scheduling on multicore hard real-timesystems.

TABLE I: Testbed Parameters

Parameter Variable ValueCPU Active Power Pact(Θmax) 65 WCPU Idle Power Pinc(Θmin) 20 WServer Period Π 20 msSampling Time Ts 200 msIntegral Gain γI 0.1

The power and control parameters are given in Table I.For G, H , Optimal Feedback,Ko, and Q and R matrix(in performance Index) refer to the extended version of thispaper [12]. We primarily use the testbed to generate the systemparameters with the help of System Identification (SI). We usethe SI tools provided by Matlab to derive the system state-space parameters. We use the Predictive Error Method (PEM)in SI toolbox, as it is better for the MIMO system-parametergeneration process. Also we use the system parameters gener-ated from our testbed as the simulation parameters. We observea matching of our testbed readings and the simulation. Moredetails on this process are contained in the extended versionof the paper [12].

B. Testbed Details

To prove the theoretical results through experiments, wehave built a hardware testbed using an Intel i7-950 multi-core processor with modified Linux kernel (2.6.33.7.2-rt30PREEMPT RT). Our testbed consists of8 processor cores,4 of them are physical and each physical core consists of2 hyper threads5. We use model specific registers (MSR) tomeasure the testbed processor core temperatures and Phid-gets 4-port temperature sensor board to measure the envi-ronment temperature. A loadable kernel module was devel-oped to activate and vary the frequency modulation level ofthe processor dynamically. We control the clock-frequencymodulation ratio and select the higher and the lowest fre-quency modulation indices to emulate the low and the higherpower levels; this was done by dynamically updating theIA32 CLOCK MODULATION MSR to 12.5% and 87.5%modulation ratios for the active and inactive power modeemulation.

We develop a Linux native posix thread libraries (NTPL)based multi-threaded application. Our application consists of

5We consider this system as8 core processor because each processorcore has their own individual MSR and allow us to measure the individualtemperature values.

Temp Driver(MSR)

Thread Activator

τ1 . . . τ8τ2

nanosleep Timer

Scheduler Simulator(Real-Time Loop)

EDF Mode

ControllerFM Drv

Phidget Drv(MSR)

. . . .

Core#8

Frequency Mod

Thread Activator

τ1 . . .τ8

τ2

nanosleepTimer

Scheduler Simulator(Real-Time Loop)

EDF Mode

Core#1

LTPLTP

τidlτidl

ΘC

ΘC

Θ1,iτ1,ij Θ1Θ1,i

M1,i

Θ8,iτ8,ij Θ8,i Θ8

M8,i

Fig. 5: The block diagram of the testbed implementation

a parallel scheduler simulator (PSS) and a thread activator.Our PSS has multiple instances of a scheduler simulator code;each scheduler simulator only manages a specific processorcore. Each scheduler simulator is charged with simulating onlythe tasks assign to its core and can access and controls thesetask threads. We call these task threads as local thread pool(LTP). These task threads in LTP are only allowed to run ina specified processor. Figure 5 shows essential components ofthe implementation of our testbed.

The real-time loop runs as a high-priority (higher prioritythan the threaded IRQ handlers) thread. Now we explain asingle iteration of the real-time. First, the scheduler simulatorinvokes the optimal controller. Then, the optimal controllerreads all the processor core temperatures, environment tem-perature, and calculates the optimal

(

Θi)

m×1vector for the

next period. The calculatedΘC, C ∈ {1 . . .m}, values areapplied to each scheduler simulator to select and activatetheir corresponding job threads from LTP. First, the schedulersimulator selects the appropriate mode corresponding to theΘC value; then, schedule simulator selects the jobs (threads)based on EDF from its LTP (to sufficiently fill theΘ) anddispatches them into a thread activator contiguously. Duringthis the entireΘ period, the CPU core is set to the highestpower level. If no job (thread) is available to dispatch duringΘ time interval (according to EDF), the idle task (thread) isselected. During theΠ − Θ period, the idle job (thread) isexecuted, and the CPU core is set to the low power level.

The scheduler simulator emulates the schedule tick func-tionality of the Linux kernel in a higher level granularity(abstraction). Similar to the Linux kernel scheduler tick,thescheduler simulator, with the help of thread activator, sleeps(usesnanosleep() in Linux) until it wakes up in the schedul-ing boundaries. Our thread activator wakes up in unequal tick

7

0 1000 2000 3000 4000 5000 60000

5

10

15

Time (x 200ms)

Mo

de

#

System Mode Variation with Tenv

0 1000 2000 3000 4000 5000 60000

0.2

0.4

0.6

0.8

1

Time (x 200 ms)

Uti

lizat

ion

System Utilization Variation with Tenv

Max Offered USelected U

Mode

0 1000 2000 3000 4000 5000 600020

40

60

80

100

Time (x 200 ms)

Tem

per

atu

re °

C

CPU Thermal Behavior with Tenv

Tcpu1

+ Tenv

Tcpu2

+ Tenv

Tcpu3

+ Tenv

Tcpu4

+ Tenv

Tcpu5

+ Tenv

Tcpu6

+ Tenv

Tcpu7

+ Tenv

Tcpu8

+ Tenv

Tenv

0 1000 2000 3000 4000 5000 600082

82.5

83

83.5

Fig. 6: The controller is capable of maintainingTref over considerableTenv temperature variation. This simulation shows theTref = 83◦Cstays fixed and theTenv varies over22 to 32 ◦C linearly. Theutilization and the modes are shown for a single core and foundto be the same for all other cores. (The utilization values are also thesame for all the processors for last two decimal places).

intervals to schedule jobs, raises the appropriate thread (fromLTP) which should have the priority, and goes back to thesleep. This process repeats and the amount of time allocatedtoeach job depends on EDF and the total time depends on theΘC

given by the optimal controller. Also, the scheduler simulator(with the help of a thread activator) in each core completes theabove process in parallel. Whenκ×Π is passed, the schedulersimulator invokes the optimal controller to calculate theΘC

value again.

C. Results

In Figure 6, the M-TROC thermal response, the utilization,and the instantaneous modes have been shown for a givenfixed reference temperature,Tref (equals to83◦C) and linearlyvarying (increasing and decreasing) environment temperature,Tenv, from 20◦C to 32◦C. The behavior of controller inthis environment is stable and with a minimum effort thecontroller achieves its goal. (Note the enlarged windows ofthe third graph). Our control algorithm selects the maximumcore temperature from all the CPU cores to regulate to thereference temperature,Tref. However, the temperature graphof the simulation (and the testbed results as well) shows thatthe temperature values of all the CPU cores try to convergeto a (nearly) same temperature value. As we mentioned early,the normal operation of a CPU does not allow to formulatesubstantially larger hotspots and the core temperatures tend toconverge.

Figure 7 shows the behavior of M-TROC whenTenv isdynamically changed over time. The difference of this graphover the Figure 6 is that theTenv varies stepwise. Further noticethat, we have assigned a fixed mode to the core#1. Observe

0 1000 2000 3000 4000 5000 60000

5

10

15

Time (x 200 ms)

Mo

de

System Mode Variation with Tenv

with Core #1 Mode Fixed

0 1000 2000 3000 4000 5000 60000

0.2

0.4

0.6

0.8

1

Time (x 200 ms)

Uti

lizat

ion

System Utilization Variation with Tenv

when Core #1 Utilization is Fixed

Fixed Mode (Core #1)Compromized Mode (with Core #1 Fixef)Regular Mode

Max Offered U Selected U Fixed U (Core #1) Regular U

0 1000 2000 3000 4000 5000 600020

40

60

80

100

Time (x 200 ms)

Tem

per

atu

re °

C

CPU Thermal Behavior with Tenv

Core #1 #2 #3 #4 #5 #6 #7 #8 Tenv

Fig. 7: The mode variation of cores atTref = 83◦C with respect tothe environment temperature. We have fixed a mode on core #1 andrest of the cores were controlled to seek the best possible mode. Theutilization and the modes are shown for a single core and found tobe the same for all other cores. (The utilization values are also thesame for all the processor for last two decimal places).

that controller is able to track the reference temperatureTref

despite the sharp changes inTenv. For both controllers, theutilization appropriately tracks the changes in environmenttemperature. When the environment temperature increases,controller decreases the system utilization and increase theutilization again when the environment temperature drops.

Regarding the real-time performance, figures displayingdeadline miss ratios have been omitted as no deadline misswas experienced for controller in all the simulations. Thisis because we select the possible mode with availableΘ sothat the deadline missing is guaranteed not to occur. We havedeveloped techniques to obtain the minimum safe resourcecapacity in the presence of mode changes and used here incalculating the anticipated mode at each mode change instance.Also, M-TROC guarantees that no deadlines are ever misseddue to verification using a multi-modal schedulability test[5]as described in Section III-B.

D. Experiments upon Hardware Testbed

To further confirm the validity of the theoretical results, wehave run a task system with eight tasks, each with three modesin each CPU core (identical to the simulation setting), on ourhardware testbed. Regarding our real-time tasks, we assigneach task to perform heavy numerical calculations while exe-cuting on the system. Our hardware testbed behaves similar tothe simulations of the previous subsection. Figure 11 presentsthe testbed runs for a fixed environment and varying referencetemperature,Tref settings. As suggested from the simulations,we clearly see the ability of our testbed to quickly convergeto a stable state. Figure 10 shows how the testbed behaveswhen theTref is varied comparatively slowly. Also, note thatthe control signal shown below shows that it reaches a steadystate, which is corresponding to theΘ of a corresponding CPU

8

8082

8486

8890

92

2224

2628

3032

0

5

10

15

Tref

C°

Mode Variation over Tenv

and Tref

Tenv

C°

Mo

de

( @

Π=2

0)

Regular Mode

Fig. 8: Thermal-resiliency for the system for all the possible Tenv andTref values. This is generated from simulation results.

8082

8486

8890

92

2224

2628

3032

0

5

10

15

Tref

C°

System Performance, when Mode is Fixed on a Selected Core

Tenv

C°

Mo

de

(@ Π

=20)

Compromized Mode (with Core #1 Fixed)Regular ModeFixed Mode (Core #1)

Fig. 9: The available thermal-resiliency of the system whenCore#1 is assigned a fixed mode is compared with the regular thermal-resiliency values for variousTenv andTref values. This is generatedfrom simulation results.

core. Observe that there is a momentary drop in performancemode; however, the system soon stabilizes.

Finally, we validate our thermal-resiliency calculation.Fig-ure 8 shows the predicted thermal-resiliency function for oursystem. Figure 9 shows the function when Core#1 is assigneda fixed mode of 15. Notice that when eitherTref decreases orTenv increases, the predicted mode decreases as the system isbecoming more thermally constrained.

Unfortunately, we do not have test equipment to accuratelyvary the environment temperature. Thus, we consider theenvironment temperature to be fixed at the room temperature,Tenv = 24◦C. Instead, we indirectly analyze the thermal-resiliency function via the inverse of the thermal-resiliencyfunctionΛ−1(MC,i, Tenv) = min{Tref | Tenv ≤ Λ(MC,i, Tref)}.Intuitively, a lower value ofΛ−1(MC,i, Tenv) means the systemcan operate at a lower temperature and thus is more resilient

2000 3000 4000 5000 6000 7000 8000 9000 10000 110000

20

40

60

80

100

Time (x 200 ms)

Tem

per

atu

re, C

° /Co

ntr

l In

pu

t, Θ

Testbed Behavior (All the Cores Shown)

2000 3000 4000 5000 6000 7000 8000 9000 10000 110000

20

40

60

80

100

Time (x 200 ms)

Tem

per

atu

re, C

° /Co

ntr

l In

pu

t, Θ

Testbed Behavior (Core #1)

Tref

Control Input, /ThetaT

cpu +T

env

3000 3500 4000 4500 50000

10

20

30

40

50

60

70

80

Time (x 200 ms)

Tem

per

atu

re, C

° /Co

ntr

l In

pu

t, Θ

Testbed Behavior (Core #1 (zoomed))

Tref

Control Input, /ThetaT

cpu +T

env

Fig. 10: The controller is capable to track theTref over considerablerange. This testbed result shows how the controller behaveswhenthe reference temperature,Tref varies from60◦C to 80◦C. TheTenvstays stable over the test run.

than a higher value of the function. We have calculated thisfunction multiple runs of the hardware testbed (to ensure thatminor fluctuations of the air temperature do not affect thesystem). Figure 12 shows a plot of the thermal-resiliency ofthe testbed runs when theTref is changed. The upper figureshows that the calculated inverse resiliency of the systemincreases with increasing operating mode. Most importantly,the calculated thermal-resiliency tracks the actual behavior ofthe testbed and provides a safe upper bound onTref in a largemajority of the cases which validates the effectiveness of theresiliency function. The lower figure shows that the calculatedinverse resiliency of the system increases with increasingoperating mode while a fixed mode is assigned to core#1.

VII. C ONCLUSIONS

In this paper, we have addressed the problem of obtainingperformance guarantees of multicore systems in an unpre-dictable thermal environment. Towards this challenge we havepresented a control-theoretic thermal-stress framework usingnested feedback control system, which is based on optimumcontrol theory.

For our system, we derive strong thermal-resiliency andhard-real-time guarantees for any real-time performance mode.Our method has the distinct advantage of being able to verifythe real-time thermal-resiliency of a system before it is putinto operation as previous approaches which have no formalguarantee on the thermal-resiliency. Further, our proposedmechanism also quantifies the thermal-resiliency on multicoresystems under certain capacity constraints on selected cores.

In future work, we plan to extend our framework to controldesigns that are more robust to model inaccuracies (e.g.,H∞

or model-predictive controllers). As a initial step in designinga framework for thermal stress analysis, our current design

9

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2

x 104

0

10

20

30

40

50

60

70

80

Time, (x 200 ms)

Tem

per

atu

re, C

° /Co

ntr

l In

pu

t, Θ

Testbed Bevarior at 22 C°

Tref

Tcpu

+Tenv

Control Input, Θ

Fig. 11: The controller is capable to maintain theTref over consid-erableTenv temperature variation. TheTref = 80◦C stays fixed andthe Tenv varies over22 to 32 ◦C randomly.

76 78 80 82 84 86 88 900

5

10

15

20

Prediction vs Tesbed Observations @ 22 ° C Tenv

Tref

° C

Cap

acit

y/M

od

e

76 78 80 82 84 86 88 900

5

10

15

20

Prediction vs Tesbed Observations @ 22 ° C Tenv

( Core #1 Mode Fixed at Highest Mode)

Tref

° C

Cap

acit

y/M

od

e

Sim Capacity (@Π=20) Testbed (@ Π=20) Sim Mode Testbed Mode

Fig. 12: Due to practical difficulty to control the environment temper-ature accurately, we calculateΛ−1(MC,i, Tenv) = min{Tref | Tair ≤Λ(MC,i, Tref)}, which is the inverse of the thermal-resiliency func-tion. The upper figure shows the available thermal-resiliency of thesystem when no restriction on core mode/capacity is enforced. Thelower figure shows the available thermal-resiliency of the systemwhen Core #1 is assigned a fixed mode. Each figure shows acomparison of our testbed observations with simulations results.

uses two RC circuits (for dynamic and leakage currents) tomodel the CPU temperature. We plan on extending our modelto permit multiple RC circuits for heterogeneous thermaldistributions and generalizing our thermal equations for morecomplex RC circuit layouts. We hope to derive a general-theoretic design framework that captures “resiliency” metricsfor other system properties (e.g., energy, noise, etc.) andextendour analysis to other hardware settings (e.g., DVFS).

ACKNOWLEDGMENTS

This research has been supported in part by the NSF (GrantNos. CNS-0953585, CNS-1116787, and CNS-1205338), theAir Force Office of Scientific Research (Grant No. FA9550-10-1-0210), and a grant from Wayne State University’s Office

of Vice President for Research.

REFERENCES

[1] M. Ahmed, N. Fisher, S. Wang, and P. Hettiarachchi. Minimizing peaktemperature in embedded real-time systems via thermal-aware periodicresources.Sustainable Computing: Informatics and Systems, 1(3):226 –240, 2011.

[2] E. Bini and G. Buttazzo. Biasing effects in schedulability measures. InProceedings of the 16th Euromicro Conference on Real-Time Systems,pages 196–203. IEEE Computer Society, 2004.

[3] D. Brooks and M. Martonosi. Dynamic thermal management for high-performance microprocessors. InInternational Symposium on High-Performance Computer Architecture, 2001.

[4] R. C. Dorf and R. H. Bishop.Modern Control Systems. Prentice-Hall,Inc., Upper Saddle River, NJ, USA, 2000.

[5] N. Fisher and M. Ahmed. Tractable real-time schedulability analysisfor mode changes under temporal isolation. InProceedings of the9th IEEE Symposium on Embedded Systems for Real-Time Multimedia(ESTImedia). IEEE Computer Society, October 2011.

[6] X. Fu and X. Wang. Utilization-controlled task consolidation for poweroptimization in multi-core real-time systems. InEmbedded and Real-Time Computing Systems and Applications (RTCSA), 2011 IEEE 17thInternational Conference on, volume 1, pages 73 –82, aug. 2011.

[7] X. Fu, X. Wang, and E. Puster. Dynamic thermal and timelinessguarantees for distributed real-time embedded systems. InEmbeddedand Real-Time Computing Systems and Applications, 2009. RTCSA ’09.15th IEEE International Conference on, pages 403 –412, aug. 2009.

[8] Y. Fu, N. Kottenstette, C. Lu, and X. D. Koutsoukos. Feedback thermalcontrol of real-time systems on multicore processors. InProceedingsof the tenth ACM international conference on Embedded software,EMSOFT ’12, pages 113–122, New York, NY, USA, 2012. ACM.

[9] A. K. Gelig and . Churilov, Alexander N.Stability and oscillationsof nonlinear pulse-modulated systems / Arkadii Kh. Gelig, Alexander N.Churilov. Boston : Birkhauser, 1998. Includes bibliographical references(p. [343]-359) and index.

[10] S. Ghosh, R. Rajkumar, J. Hansen, and J. Lehoczky. Integratedqos-aware resource management and scheduling with multi-resourceconstraints.Real-Time Syst., 33(1-3):7–46, July 2006.

[11] P. M. Hettiarachchi, N. Fisher, M. Ahmed, L. Y. Wang, S. Wang,and W. Shi. The design and analysis of thermal-resilient hard-real-time systems. InIEEE Real-Time and Embedded Technology andApplications Symposium, pages 67–76, 2012.

[12] P. M. Hettiarachchi, N. Fisher, and L. Y. Wang. Achieving ther-mal resiliency for multicore hard-real-time systems (extended version).Technical report, Wayne State University, 2013. Availableat http://www.cs.wayne.edu/∼fishern/papers/ECRTS-2013-Thermal-TR.pdf.

[13] C. Liu and J. Layland. Scheduling algorithms for multiprogramming ina hard real-time environment.Journal of the ACM, 20(1):46–61, 1973.

[14] C. Lu, X. Wang, and X. Koutsoukos. Feedback utilizationcontrolin distributed real-time systems with end-to-end tasks.Parallel andDistributed Systems, IEEE Transactions on, 16(6):550 – 561, june 2005.

[15] A. K. Mok. Fundamental Design Problems of Distributed Systems forThe Hard-Real-Time Environment. PhD thesis, Laboratory for ComputerScience, Massachusetts Institute of Technology, 1983. Available asTechnical Report No. MIT/LCS/TR-297.

[16] N. S. Nise. Control Systems Engineering. John Wiley & Sons, Inc.,New York, NY, USA, 2000.

[17] K. Ogata. Discrete-time control systems (2nd ed.). Prentice-Hall, Inc.,Upper Saddle River, NJ, USA, 1995.

[18] R. Rajkumar, C. Lee, J. Lehoczky, and D. Siewiorek. A resourceallocation model for qos management. InProceedings of the 18th IEEEReal-Time Systems Symposium, RTSS ’97, pages 298–, Washington, DC,USA, 1997. IEEE Computer Society.

[19] E. Seo, J. Jeong, S. Park, and J. Lee. Energy efficient scheduling of real-time tasks on multicore processors.Parallel and Distributed Systems,IEEE Transactions on, 19(11):1540 –1552, nov. 2008.

[20] I. Shin and I. Lee. Compositional real-time schedulingframework withperiodic model.ACM Transactions on Embedded Computing Systems,7(3), April 2008.

[21] K. Skadron, M. R. Stan, W. Huang, S. Velusamy, K. Sankaranarayanan,and D. Tarjan. Temperature-aware microarchitecture. InInternationalSymposium on Computer Architecture, 2003.

10

[22] J. Yao, X. Liu, Z. Gu, X. Wang, and J. Li. Online adaptive utilizationcontrol for real-time embedded multiprocessor systems.Journal ofSystems Architecture, 56(9):463 – 473, 2010.

APPENDIX

A. Multicore Thermal Model

To simplify the analysis, we consider a processor with 4adjacent cores. We apply the Kirchhoff’s circuit laws andget the following equations. We assume that the heat (powerdissipation in the equivalent system) distribution from core i

to corej asIij . It is easy to see thatPCcpu(t), C ∈ {1 . . .m}

(in this casem = 4) the individual core power dissipation,distributes in the own core and to the adjacent cores as follows:

P1cpu(t) = I11 + I12 + I13 + I14,

P2cpu(t) = I21 + I22 + I23 + I24,

P3cpu(t) = I31 + I32 + I33 + I34,

P4cpu(t) = I41 + I42 + I43 + I44.

(9)

Therefore, we get the thermal distribution for the 4 coresystem as,

I11 = Cdcpu11(t)

d

dtTcpu11(t) +

Tcpu11(t)

Rdcpu11

+ Clcpu11(t)

d

dtTcpu11(t) +

Tcpu11(t)

Rlcpu11

,

I22 = Cdcpu22(t)

d

dtTcpu22(t) +

Tcpu22(t)

Rdcpu22

+ Clcpu22(t)

d

dtTcpu22(t) +

Tcpu22(t)

Rlcpu22

,

I33 = Cdcpu33(t)

d

dtTcpu33(t) +

Tcpu33(t)

Rdcpu33

+ Clcpu33(t)

d

dtTcpu33(t) +

Tcpu33(t)

Rlcpu33

,

I44 = Cdcpu44(t)

d

dtTcpu44(t) +

Tcpu44(t)

Rdcpu44

+ Clcpu44(t)

d

dtTcpu44(t) +

Tcpu44(t)

Rlcpu44

.

(10)

Also, we can show that the thermal distribution between

cores as,

I21 = Ccpu12(t)d

dt(Tcpu22(t)− Tcpu11(t)) +

(Tcpu22(t)− Tcpu11(t))

Rcpu12,

I31 = Ccpu13(t)d

dt(Tcpu33(t)− Tcpu11(t)) +

(Tcpu33(t)− Tcpu11(t))

Rcpu13,

I41 = Ccpu14(t)d

dt(Tcpu44(t)− Tcpu11(t)) +

(Tcpu44(t)− Tcpu11(t))

Rcpu14,

I32 = Ccpu23(t)d

dt(Tcpu33(t)− Tcpu22(t)) +

(Tcpu33(t)− Tcpu22(t))

Rcpu23,

I42 = Ccpu24(t)d

dt(Tcpu44(t)− Tcpu22(t)) +

(Tcpu44(t)− Tcpu22(t))

Rcpu24,

I43 = Ccpu34(t)d

dt(Tcpu44(t)− Tcpu33(t)) +

(Tcpu44(t)− Tcpu33(t))

Rcpu34.

(11)

Therefore, we get systems of first order differential equations,

(Ccpu12+ Ccpu13+ Ccpu14+ Cdcpu11+ C

lcpu11)

d

dtTcpu11(t)

− Ccpu12d

dtTcpu22(t)− Ccpu13

d

dtTcpu33(t)− Ccpu14

d

dtTcpu44(t)

+ (1

Rcpu12+

1

Rcpu13+

1

Rcpu14+

1

Rdcpu11

+1

Rlcpu11

)Tcpu11(t)

−Tcpu22(t)

Rcpu12−

Tcpu33(t)

Rcpu13−

Tcpu44(t)

Rcpu14= P1

cpu(t), (12)

− Ccpu12d

dtTcpu11(t)

+ (Ccpu12+Ccpu23+Ccpu24+ Cdcpu22+ C

lcpu22)

d

dtTcpu22(t)

− Ccpu23d

dtTcpu33(t)− Ccpu24

d

dtTcpu44(t)−

Tcpu11(t)

Rcpu12

+ (1

Rcpu12+

1

Rcpu23+

1

Rcpu34+

1

Rdcpu22

+1

Rlcpu22

)Tcpu22(t)

−Tcpu33(t)

Rcpu23−

Tcpu44(t)

Rcpu24= P2

cpu(t), (13)

− Ccpu13d

dtTcpu11(t)− Ccpu23

d

dtTcpu22

+ (Ccpu13+Ccpu23+Ccpu34+ Cdcpu33+ C

lcpu33)

d

dtTcpu33(t)

− Ccpu34d

dtTcpu44−

Tcpu11(t)

Rcpu13−

Tcpu22(t)

Rcpu23

+ (1

Rcpu13+

1

Rcpu23+

1

Rcpu34+

1

Rdcpu33

+1

Rlcpu33

)Tcpu33(t)

−Tcpu44(t)

Rcpu34= P3

cpu(t), (14)

11

and

− Ccpu14d

dtTcpu11(t)− Ccpu24

d

dtTcpu22(t)− Ccpu34

d

dtTcpu33(t)

(Ccpu14+Ccpu23+Ccpu34+ Cdcpu44+ C

lcpu44)

d

dtTcpu44(t)

(15)

−Tcpu11(t)

Rcpu14−

Tcpu22(t)

Rcpu24−

Tcpu33(t)

Rcpu34

+ (1

Rcpu14+

1

Rcpu24+

1

Rcpu34+

1

Rdcpu44

+1

Rlcpu44

)Tcpu44(t)

= P4cpu(t). (16)

To simplify the analysis, we define the following,

X11 = Ccpu12+Ccpu13+ Ccpu14+ Cdcpu11+ C

lcpu11,

X12 = X21 = −Ccpu12,

X13 = X31 = −Ccpu13,

X14 = X41 = −Ccpu14,

Y11 =1

Rcpu12+

1

Rcpu13+

1

Rcpu14+

1

Rdcpu11

+1

Rlcpu11

Y12 = Y21 = −1

Rcpu12,

Y13 = Y31 = −1

Rcpu13,

Y14 = Y41 = −1

Rcpu14,

X22 = (Ccpu12+ Ccpu23+ Ccpu24+ Cdcpu22+ C

lcpu22),

X23 = X32 = −Ccpu23,

X24 = X42 = −Ccpu24,

Y22 =1

Rcpu12+

1

Rcpu23+

1

Rcpu34+

1

Rdcpu22

+1

Rlcpu22

,

Y23 = Y32 = −1

Rcpu23,

Y24 = Y42 = −1

Rcpu24,

X33 = Ccpu13+Ccpu23+ Ccpu34+ Cdcpu33+ C

lcpu33,

X34 = X43 = −Ccpu34,

Y33 =1

Rcpu13+

1

Rcpu23+

1

Rcpu34+

1

Rdcpu33

+1

Rlcpu33

,

Y34 = −1

Rcpu34,

X44 = Ccpu14+Ccpu23+ Ccpu34+ Cdcpu44+ C

lcpu44,

Y44 =1

Rcpu14+

1

Rcpu24+

1

Rcpu34+

1

Rdcpu44

+1

Rlcpu44

,

(17)

and

X = (Xij)4×4,

Y = (Yij)4×4,

A = (Aij)4×4,

B = (Bij)4×4,

A = −X−1Y,

B = X−1.

(18)

Therefore, we can simplify the system as,

XTcpu(t) + YTcpu(t) = Pcpu(t),

⇒ Tcpu(t) = ATcpu(t) + BPcpu(t), (19)

where, Tcpu(t) =

˙Tcpu11(t)˙Tcpu22(t)˙Tcpu33(t)˙Tcpu44(t)

, Tcpu(t) =

Tcpu11(t)Tcpu22(t)Tcpu33(t)Tcpu44(t)

, and

Pcpu(t) =

P1cpu(t)

P2cpu(t)

P3cpu(t)

P4cpu(t)

.

B. State-Space Model Basics

We use the standard state-space model to representcontinuous-time (ideal) system

x(t) = Ax(t) +Bu(t) + f,

y(t) = Cx(t), (20)

where x(t), u(t), and y(t) represent the state vector, theinput vector, and the output vector, respectively.A,B, andC represent the system matrices andf represents a constantvector. Both the state matrices and constant vector are time-invariant quantities.

Since we have a computer-controlled discrete-time system,we will use following state-space mode for the discrete-timecontroller for active/inactive modes. For a sampling intervalTs, u(t) is a constant and the sampled system of Equation(20) is

x((k + 1)Ts) = Gx(kTs) +Hu(kTs) + f ,

y(kTs) = Cx(kTs), (21)

where G = eATs , H =∫ Ts

0eAtBdt, C = C, and f =

∫ Ts

0 eAtfdt. The termeAt can be computed byL−1{(sI −A)−1}, whereL−1 is the inverse Laplace transform. In theremainder of the document, we abuse the notation by repre-sentingx(kTs) as x(k), x((k + 1)Ts) as x(k + 1), u(kTs)as u(k), and y(kTs) as y(k). The above definitions may befound in any textbook on discrete-time control theory [17].

C. Continuous Power Modes

As a first step towards our goal of designing a control-theoretic framework for thermal stress analysis, we employlinear quadratic (LQ) optimal control for real-time thermalmanagement. Our design consists of an optimal state feedbackand a servo that regulates the dynamics of the system. AnLQ controller enables us to design an efficient and low-overhead controller, derive the feedback parameters beforeruntime (used in thermal-resiliency analysis), and smoothlytrack our reference input. In the future, we plan on applyingmore complex and robust controllers (e.g.,H∞ controllers)to decrease the controller’s sensitivity to modeling inaccuracy

12

and noise. However, as observed in the simulations and exper-iments of Section VI, our current LQ design is appropriatelyresponsive to changes in environmental temperature.

In our system model, we specify the thermal power of theCPU as the control to the system. The controller is required towork as a servo and should follow the temperature reference,Tref. In our design, we considerTcpu(t) as one of the variableto be controlled andPd

cpu(t) as a manipulated variable. Thebasic control structure is given in Figure 3.

D. Stability Analysis and Optimal State Feedback

In our derivation of stability for our system, we will use thefollowing two results which can be found in any standard texton control theory [4], [16], [17].

Lemma 1 (from [4]): The system of Equation (20) iscom-pletely controllable if there exists an unconstrainedu(t) suchthat it can control any initial statex(t0) to any desired finalstate xf in a finite time, t0 ≤ t ≤ T . The property ofcompletely controllable can be determined by examining thealgebraic condition

rank[B AB A2B ... Am−1B] = m, (22)

where,A is m×m andB is m× r matrix.Lemma 2 (from [17]): A discrete-time linear time invariant

(LTI) system is asymptotically stable if and only if its alleigenvalues ofG lie inside the unit circle.

Now we derive the augmented system model that is used toobtain the optimality of the system. Equation (19) can be usedto describe the system dynamics at any time instance. Consideran instance where system is completely stable and has attainedto the steady state. We denote the system input, system states,and the servo error (described in Equation (2)) of this specialinstance of the system byPcpu(t∞)), Tcpu(t∞), Tenv(t∞) andve(t) respectively. Therefore, we get,

[

Tcpu(t∞)]

=[

A] [

Tcpu(t∞)]

+[

B]

Pcpu(t∞). (23)

Then, from the Equations (19) and (23) we get,[

Tcpu(t)− Tcpu(t∞)]

=[

A] [

Tcpu(t)− Tcpu(t∞)]

+[

B]

(Pcpu(t)− Pcpu(t∞)).

(24)

Definee(t),

e(t) =[

Tcpu(t)− Tcpu(t∞)]

, (25)

then we get,e(t) = Ae(t) +Bue(t). (26)

We select the feedback gainγ such that,

ue(t) = Pcpu(t)− Pcpu(t∞) (27)

= −K0e(t). (28)

The above state-space and the control gain parameters arevalid for a continuous-time controller. So, we may obtain thediscrete-time state-space matrices for the augmented model(i.e,G andH) from A andB via the transformation described

after Equation (21). In LQ optimal control, the objective istodesign the controller to minimize some performance index. Astandard LQ performance index is given by

Jdef=

1

2

∞∑

k=0

(

e(k)TQe(k) + uTe (k)Rue(k)

)

, (29)

whereQ andR are arbitrary symmetric matrices of sizem×m

and r × r such thatQ ≥ 0 (positive semi definite),R > 0(positive definite). Theue(k) is the difference of the feedbackvalue and it becomes zero at the steady state (whenTcpu(t)becomesTcpu(t∞) at the steady state).

It is easy to show that for a Linear Time Invariant (LTI)system, (Refer to Ogata [17]), the optimal state feedback canbe obtained as,

ue(k) = −Ke(k), (30)

whereK is the feedback gain defined as

K = (R+HTPH)−1HTPG, (31)

and whereP is the positive definite solution of the algebraicRiccati equation below,

P = Q+GTPG−GTPH(R+HTPH)−1HTPG.

From the above, it may be shown [17] that the optimalperformance index can be calculated as

Jmin =1

2eT (0)Pe(0). (32)

It is well known [17] that the feedback control (i.e.,K) resultsin an asymptotically stable closed-loop system according toLemma 2. Obviously, stable choices ofKo for the system canbe derived.

E. The Testbed Parameters

We run 20000 testbed intervals to generate IO data requiredfor the SI process. During SI process, the a randomΘC value isgenerated, and each CPU was allowed to execute a workloadfor a duration specified byΘC C ∈ {1 . . . 8} We found thefollowing parameter values: Also, theQ and R are 8 × 8identity matrices.

F. The Tcpu Temperature Calculation

In this section, we solve the Equation 19 to calculate thetemperature of the individual CPU cores. We assume thatthe system is stable and should have 4 real solutions forthe Equation 19. Taking the eigenvalues of the system asλi, i ∈ {1 . . . 4}, the general solution for the system ofequations (considering the homogeneous system) can be foundas follows,

Tcpu(t)c = c1V1eλ1t + c2V2e

λ2t + c3V3eλ3t + c4V4e

λ4t

, (34)

where,Vi =

v1,iv2,iv3,iv4,i

i ∈ {1 . . .4} are eigenvectors of the

system.

13

G =

0.984 0.035 0.019 −0.019 0.018 0.235 0.204 −0.5450.115 1.070 −0.068 −0.102 0.056 0.269 −0.902 0.8030.065 −0.166 0.842 0.179 0.050 0.313 0.276 0.9600.109 0.156 −0.037 0.814 0.061 0.564 −0.796 0.041−0.029 0.012 0.051 −0.029 0.951 −0.370 −0.127 0.005−0.112 0.029 0.120 −0.027 −0.066 0.507 0.330 −0.8040.063 0.129 0.014 −0.160 0.006 −0.120 0.026 0.426−0.034 0.089 0.068 −0.086 −0.012 −0.016 −0.030 0.321

H = 10−3 ×

0.035 0.012 0.024 0.022 0.017 0.032 0.019 0.025−0.132 0.244 −0.175 0.102 −0.026 0.117 −0.094 0.0460.097 0.269 0.019 −0.267 −0.023 0.194 −0.062 −0.251−0.124 0.245 −0.181 0.180 −0.002 0.133 −0.083 0.120−0.045 −0.150 0.003 0.059 −0.002 −0.119 0.022 0.0590.017 −0.322 0.096 0.077 0.024 −0.202 0.092 0.104−0.177 0.029 −0.156 0.209 −0.017 −0.041 −0.048 0.153−0.026 −0.106 0.002 0.126 0.018 −0.066 0.036 0.118

K0 = 10−3 ×

0.357 0.096 0.153 0.047 0.006 0.430 0.150 −0.5430.328 0.317 0.160 0.172 −0.074 0.218 −0.386 0.6520.319 −0.129 0.140 −0.201 0.139 −0.044 0.407 −0.2890.316 0.294 −0.172 0.005 0.079 0.134 −0.237 −0.2050.315 0.011 0.080 0.103 0.070 0.276 0.086 −0.3980.381 0.466 0.136 0.085 −0.039 0.379 −0.353 0.2790.202 0.043 0.061 −0.151 0.057 0.042 0.184 −0.1800.244 0.462 −0.202 0.052 0.047 0.361 −0.354 −0.463

(33)

Also the fundamental matrix is given by,

φ(t) =

v1,1eλ1t v1,2e

λ2t v1,3eλ3t v1,4e

λ4t

v2,1eλ1t v2,2e

λ2t v2,3eλ3t v2,4e

λ4t

v3,1eλ1t v3,2e

λ2t v3,3eλ3t v3,4e

λ4t

v4,1eλ1t v4,2e

λ2t v4,3eλ3t v4,4e

λ4t

(35)

.

We can further simplify the solution as,

Tcpu(t)c = φ(t)C,

⇒ Tcpu(t)c = φ(t)φ(t)−1Tcpu(0)c. (36)

Also, the particular solution is given by,

Tcpu(t)p = φ(t)

∫

φ(t)−1BPcpu(t)dt. (37)

Therefore, the general solution is,

Tcpu(t) = Tcpu(t)p + Tcpu(t)c. (38)

Assume that the controller calculates four capacitiesΘ1,Θ2,Θ3, such thatΘ4, Θ1 > Θ2 > Θ3 > Θ4. Then,

we can calculate the temperature as follows,

Tcpu(Θ4) = φ(t)φ(t)−1Tcpu(0)c + φ(t)

∫ Θ4

0

φ(t)−1BPcpu(t)1dt,

Tcpu(Θ3) = φ(t)φ(t)−1Tcpu(Θ4)c + φ(t)

∫ Θ3−Θ4

0

φ(t)−1BPcpu(t)2dt,

Tcpu(Θ2) = φ(t)φ(t)−1Tcpu(Θ3)c + φ(t)

∫ Θ2−Θ3

0

φ(t)−1BPcpu(t)3dt,

Tcpu(Θ1) = φ(t)φ(t)−1Tcpu(Θ2)c + φ(t)

∫ Θ1−Θ2

0

φ(t)−1BPcpu(t)4dt,

Tcpu(Π) = φ(t)φ(t)−1Tcpu(Θ1)c + φ(t)

∫ Π−Θ1

0

φ(t)−1BPcpu(t)5dt,

⇒ Tcpu(Π)) =(

φ(t)φ(t)−1)5Tcpu(0)c

+(

φ(t)φ(t)−1)4φ(t)

∫ Θ4

0

φ(t)−1BPcpu(t)1dt+

+(

φ(t)φ(t)−1)3φ(t)

∫ Θ3−Θ4

0

φ(t)−1BPcpu(t)2dt,

+(

φ(t)φ(t)−1)2φ(t)

∫ Θ2−Θ3

0

φ(t)−1BPcpu(t)3dt,

+(

φ(t)φ(t)−1)1φ(t)

∫ Θ1−Θ2

0

φ(t)−1BPcpu(t)4dt,

+ φ(t)

∫ Π−Θ1

0

φ(t)−1BPcpu(t)5dt.

14

Therefore,

⇒ Tcpu(nΠ)) =(

φ(t)φ(t)−1)5Tcpu((n− 1)Π)c

+ κ(

(

φ(t)φ(t)−1)4φ(t)

∫ Θ4

0

φ(t)−1BPcpu(t)1dt+

+(

φ(t)φ(t)−1)3φ(t)

∫ Θ3−Θ4

0

φ(t)−1BPcpu(t)2dt,

+(

φ(t)φ(t)−1)2φ(t)

∫ Θ2−Θ3

0

φ(t)−1BPcpu(t)3dt,

+(

φ(t)φ(t)−1)1φ(t)

∫ Θ1−Θ2

0

φ(t)−1BPcpu(t)4dt,

+ φ(t)

∫ Π−Θ1

0

φ(t)−1BPcpu(t)5dt)

,

where,

Pcpu(t)1dt =

PactPactPactPact

,

Pcpu(t)2dt =

PactPactPactPinc

,

Pcpu(t)3dt =

PactPactPincPinc

,

Pcpu(t)4dt =

PactPincPincPinc

,

Pcpu(t)5dt =

PincPincPincPinc

.

(39)

15