1

GPU NTC Process Variation Compensation withVoltage Stacking

Rafael Trapani Possignolo, Student Member, IEEE Elnaz Ebrahimi, Ehsan Khish Ardestani, AlameluSankaranarayanan, Jose Luis Briz, Member, IEEE, Jose Renau

Abstract—Near Threshold Computing (NTC) has the po-tential to significantly improve efficiency in high throughputarchitectures like GPGPUs. Nevertheless, NTC is more sensi-tive to process variation (PV) as it complicates power delivery.We propose GPU Stacking, a novel method based on voltagestacking, to manage the effects of PV and improve the powerdelivery simultaneously. To evaluate our methodology, we firstexplore the design space of GPGPUs in the NTC to find asuitable baseline configuration and then apply GPU Stacking tomitigate the effects of PV. When comparing with an equivalentNTC GPGPU without process variation management, weachieve 37% more performance on average. When consideringhigh production volume, our approach shifts all the chips closerto the nominal non-process variation case, delivering on average(across chips) ≈ 80% of the performance of nominal NTCGPGPU, whereas when not using our technique, chips wouldhave ≈ 50% of the nominal performance. We also show thatour approach can be applied on top of multi-frequency domaindesigns, improving the overall performance.

Index Terms—Process Variation, Voltage Stacking, Near-Threshold Computing, GPGPUs

I. IntroductionNear Threshold Computing (NTC) is a circuit design

technique used to reduce the power envelop used by adesign and thus improve energy efficiency by reducingthe operating voltage to near the threshold voltage of thetransistors employed [1]. Although more energy-efficient,NTC devices usually do not present as high performanceas their non-near threshold counterparts. However, ithas been shown that the performance impact resultingfrom NTC can be mitigated through parallelism. Anideal candidate for such operation is a GPU [1], [2], [3].Nonetheless, NTC makes the system more sensitive toprocess variation [4].

To manage the additional sensitivity to process vari-ation (PV) introduced by NTC, some researchers haveproposed frequency scaling [5] or having multiple voltagedomains [6]. Having multiple voltage domains requiresadditional power rails which will further exacerbate thecurrent delivery problem. Multi-frequency domains [7]have been shown to moderately mitigate the problem of

Part of this study has been supported by the National ScienceFoundation under grants CNS-1059442-003, CNS-1318943-001, CCF-1337278, and CCF-1514284. This work was supported in partby grants TIN2016-76635-C2-1-R (AEI/FEDER, UE), gaZ: T48research group (Aragón Gov. and European ESF), and HiPEAC4(European H2020/687698). Any opinions, findings, and conclusionsor recommendations expressed herein are those of the authors anddo not necessarily reflect the views of the funding institutions.

R.T. Possignolo, E. Ebrahimi, E.K. Ardestani, A. Sankara-narayanan and J. Renau are with the University of California, SantaCruz, Santa Cruz, CA, 95064 and J.L. Briz is with the UniversidadZaragoza, Pedro Cerbuna, 12, 50009 Zaragoza, Spain.

PV effects on energy and performance in many-core NTCdesigns.

Voltage stacking improves the efficiency of power de-livery [8]. When n units are stacked, they are placedin a series fashion, rather than the conventional parallelscheme. Thus, the current in the power delivery networkis reduced by a factor of n in a system. This allows voltageregulators (VRs) with increased efficiency, smaller areasand fewer package pins dedicated to power [9]. Voltagestacking has been applied to CPUs [8], GPUs [10], andSRAMs [11].

However, in this work we look at voltage stacking froma different perspective. We note that voltage stacking cancompensate for PV effects. The proposed GPU Stackingmethodology lets the voltage node between the stackedelements (VMID) float.1 This floating node is the key toPV compensation. GPU Stacking alleviates the currentdelivery challenges, and intrinsically mitigates PV effectswithout requiring multiple voltage domains. GPU Stack-ing automatically creates a voltage domain per level in thestack without the cost of multiple power rails. We buildon top of this premise, and discuss how it can be leveragedfor managing PV.

Voltage stacking of many cores has its own challenges,among which is the load mismatch between the stackedcores [9]. Cores go through different phases while runningapplications, which can result in transient impedancemismatch of the stacked cores, and yield timing failures.As a result, stacking is successful when the cores havea matching workload. GPGPUs are instances of such de-signs. Not only are the cores identical, but the applicationsrunning on them are roughly homogeneous2.

The evaluation of GPU Stacking is carried out in thenear threshold region. Although the use of this method inthe near threshold region is proposed, it is not a require-ment. The use of NTC in this research study is justifiedby the increased sensibility of NTC to process variationeffects. The first part of this experiment consists of findingthe ideal GPU configuration for the NTC region. Bycarefully sizing the GPU to NTC, power consumption isreduced by 43% with only 4.8% performance degradationcompared to the baseline.

Based on our experiments, there is a potential for selfbalancing in the stacked configuration. We observe thatstacking of cores with opposing PV trends is a better

1For safeguarding reasons, a few voltage regulators are used to capthe maximum and minimum levels of voltage but the voltage doesfloat between maximum and minimum values.

2Even though divergence exists in modern GPUs, the amount ofload mismatch observed in traditional GPGPUs benchmarks can behandled by our technique, as it will be shown in our evaluation.

2

choice to gain the best self balancing results. This studyproposes the stacking of SIMD Lanes, because having alarge number of lanes provides more opportunities forPV compensation. Analyzing several PV maps, GPUStacking shows that it is able to deliver, on average,≈ 80% of the nominal performance in a multi-frequencydomain scenario (as opposed to ≈ 60% in a non-stackedconfiguration). This represents a reduction of about 30%in the effects of process variation even after using a multi-frequency domain based compensation of process variationeffects. When not considering the multi-frequency domainscenario, the reduction in process variation impact is evenmore important, around 37% of the process variationeffects are compensated. This compensation is mainly dueto an uneven voltage distribution that considers the PVof cluster of lanes. For a nominal voltage of 0.6V, theobserved voltage was in the range of 0.602V to 0.65V forlanes negatively affected by PV, and in the range of 0.55Vto 0.598V for lanes positively affected by PV.

The floating Vmid node could be a source of problemsin cases of extreme load mismatch, or high temperatures.Thus, we propose the use of Dummy Activity (DA), anda small voltage regulator to keep the voltage within safeoperational margins (i.e., to avoid voltage starvation inone of the levels). We use SPICE simulations to verifythe reliability of the power delivery network (PDN), andshow that GPU Stacking does not incur extra voltagenoise. In our simulations, DA and the additional VR werenot required, given the stability of the operational voltage.

The main contributions of this paper are:• The first study to show how voltage stacking allevi-

ates PV effects.• The first study proposing stacking with an uneven

voltage division (VMID).• A study to propose a technique to make the post-

silicon configuration of the design feasible.The remainder of this paper is organized as follows.

In Section II, we discuss related work in a few differenttopics: voltage stacking, PV mitigation effects, NTC andenergy efficiency in GPUs. Then, in Section III, webriefly present background information needed to theunderstanding of this work. We present the GPU Stackingmodel in Section IV. Finally, we discuss the experimentalsetup in Section V and results in Section VI. We wrap-upon Section VII.

II. Related WorkWe divide our related work into a few categories.

We start discussing prior approaches for voltage staking.Then, we follow presenting other techniques to mitigatePV effects. Finally, we discuss work on NTC and energyefficiency for GPUs.

Voltage Stacking: As supply voltage decreases, theefficiency of power delivery components degrades [9]. On-chip voltage regulators [12] have been proposed to increasethe PDN efficiency, as well as the series configurationsof units rather than parallel [13], [14], [8]. Such config-urations are known as Multi-Story Power Delivery [13],charge recycling [14], or voltage stacked systems [8]. Ourproposed technique, GPU Stacking, depends on a seriesconfiguration of cores, yet for a different purpose. Note

that no previous research on voltage stacking exploits thestacking method to control or neutralize PV effect.

Process Variation mitigation: PV increases as featuresize shrinks. And lowering Vdd, as a power managementtechnique, further exacerbates the PV effects. Lee etal. study the impact of frequency variation on thethroughput of a GPGPU [15]. Adaptive Body Bias (ABB)leverages the power-performance trade-off to manage PVeffects [16]. Slower devices due to PV, can run faster byconsuming more power, and vice versa. Adaptive SupplyVoltage (ASV) is another technique, where supply voltageof a region in the design is adjusted to compensate forperformance loss due to variation. ABB and GPU Stackingare orthogonal techniques, and could be used together,although ABB efficiency is expected to reduce withtechnology scaling [7]. Similarly to ASV, our techniqueprovides a custom supply voltage to each stacking clusterto compensate for process variation.

NTC: An extensive amount of research targets increas-ing the power/energy efficiency of processors. Apart fromthe many proposed techniques of how to reduce theutilization of resources [17], [18], or how to promote theuse of more power efficient structures [19], [20], there are asignificant number of proposals that attempt to utilize thepower-performance trade-off. DVFS and Power Gating areamong the techniques widely studied at an architecturallevel, for the same purpose [21], [22]. Intel Turbo Boosttechnology is another example of utilizing voltage andfrequency scaling to adapt to runtime conditions.

Dreslinski et al. [1] study devices for near thresholdoperation, and Chang et al. propose the optimization ofdevice parameters for NTC [2]. They propose a slightlymodified SRAM cell to address the stability challenges in-troduced in near threshold regions. Lower Vdd exacerbatesthe effects of process variation. Another study of NTC inmany cores where they argued in favor of fine-grained coreassignment and DVFS [7].

Energy efficiency in GPGPUs: With the rising popular-ity of GPGPUs, several research groups discuss strategiesto make them more energy efficient [23], [24], [25], [26].Lee et al. study the impact of frequency variation on thethroughput of a GPGPU [15]. This methodology, however,is the first to extend the evaluation to NTC trade-offsand it is an extension of previous research addressingNTC challenges. Massive data parallelism and extremelyrepetitive nature of the GPGPU applications is leveragedto adapt the operational region and configuration to theruntime application demand.

III. BackgroundThis section briefly describes background concepts re-

lated to Voltage Stacking, PV and NTC that are needed tobetter understand this work. We also go over the micro-architecture of a GPU, giving special attention to thedetails relevant to this paper.

Figure 1 depicts two power delivery schemes, the con-ventional power delivery scheme (Figure 1-a) has all theelements in parallel. The elements can be interpreted asindividual gates, functional units within a core, a wholecore, and so forth. The stacked power delivery schemehas elements in series, in the case show in the figure,

3

the number of stacking levels is two, since there are twoelements stacked. In general, for voltage stacking, this isnot a requirement, but in this paper we only consider twolevels. In a voltage stacked system, the power deliveredis the same as in the conventional case, but the deliveryvoltage is multiplied by n, the number of stack levels, andthe current is divided by n, on average [8].

Assuming that the power consumed by each elementis the same, the voltage across each element is equal tothe nominal Vdd. Because roughly half the current flowsthrough the system, the power delivery subsystem couldoperate more efficiently [9]. However, this is not always thecase. When a full core is stacked on top of another, theactivity on each core will depend on the program running,and will create a mismatch between the stack levels andmake Vmid shift from Vdd. This problem has been solvedin different ways in the literature, for instance by insertingan extra voltage regulator [8], partitioning a core in unitsthat have correlated power consumption [9], or by stackingmemory arrays for SRAM application [11].

Pow

er D

eliv

ery

VZ1 Z2

Iin Vin = V1 =V2

I1=Vin

Z1I2= Vin

Z2

(a) Conventional

2VinPow

er D

eliv

ery

V

Iin(t) = 2Vin

Z1(t)+Z2(t)

Vmid(t) =Iin(t).Z2(t)

Stack

Levels

Z1(t)

Z2(t)

(b) Stacked

Fig. 1. Conventional vs. stacked power delivery mode

In this work, we use GPGPUs, which contain a largenumber of identical cores operating in lock-step, runningthe same program. This provides a very good startingpoint for voltage stacking. We note that “divergence” canoccur and we address divergence later in this paper.

A. General Purpose GPUIn this section, we explore the micro-architecture of a

GPU. General purpose computing on graphic processingunits (GPGPU) is becoming pervasive as it providesexcellent computing power for massively parallel appli-cations. GPGPUs are mainly designed as a cluster ofsimple processors, depicted in Figure 2. Identical simpleprocessors (lane3) operate in lock-step inside a streammultiprocessor cluster (SM), running identical threads(though processing different data).

The homogeneous structure of GPGPUs, both in hard-ware and application, makes them suitable for voltagestacking in order to manage process variation. GPUStacking stacks lanes inside SMs. All the other structuresremain in a conventional configuration. The choice ofstacking lane provides room for more configurability, dueto the larger number of lanes. It also provides a fine-grained mitigation of process variation, while at the SMlevel, techniques like multi-clock and multi-voltage domainare possible.

3Some authors use the term lane to refer to each of the smallexecution cores within what we call a SIMD lane. We use the samedefinition as Gebhart et al. [27], and thus count one lane perload/store unit. This leads to a lane count 4× smaller than what isadvertised for commercial GPUs.

L1L1Shared MemoryShared Memory

...

RF-0

Coalescing

Scratchpad DL1G

Lane-0

RF-1

Lane-1

RF-2

Lane-2

RF-3

Lane-3

RF-n

Lane-n

SM-0 1

(a) GPGPU

ClusteredRegister File

Operand Buffer

ALUsSFU

MEM

TEX

(b) SIMD Lanes

Fig. 2. GPGPUs present a large number of identical SIMD lanes,ideal for stacking.

B. Process Variation Model and EffectsSo far, we have mentioned PV without specifying how

PV can affect a chip. Process variation is the deviationfrom nominal specifications of a chip due to the difficultyof precisely control the fabrication process of a chip,especially at small feature sizes, e.g., lithographic lens,dopant density fluctuations, and others. Variation canbe divided in two categories: systematic and random.From a design point of view, they differ on the gran-ularity at which they occur. Random variation occursat the transistor level. Systematic variation occurs at amuch coarser granularity: within-die (WID) and die-to-die (D2D). Significant variation can be seen on the orderof half the chip length [4]. Overall, random variationends up being averaged out across a few gates in a pathsince statistically, positive and negative variation will beobserved in that path.

D2D variation cannot be addressed with on die tech-niques, since the whole die is biased towards a processcorner. Therefore, this work aims to address WID varia-tion. WID affects blocks in different regions of the designdifferently. This makes that each clock region is limitedby the slowest block within it.

For a MOS transistor gate delay can be formulated asfollows, where Vdd is the supply voltage to the core, Leff

is the effective channel length, K and M are fabricationconstants [4].

delay ∝ Vdd × Leff

K × ln2(eM×(Vdd−Vth) + 1)(1)

Due to the exponential component with the (Vdd−Vth)term, the gate delay exhibits more sensitivity to Vth

variation at supply voltages close to Vth. The delayvariation with regard to Leff however is linear. This isparticularly important for NTC applications, since Vth willhave more effect in those cases.

C. NTC and Power Performance Trade-offThe main argument for the adoption of NTC is its

energy-efficiency in terms of energy per-operation [4]. Thisis because, frequency has a linear relationship with thevoltage down to the near threshold region, while poweris reduced in a cubic relationship with voltage. Thus, forinstance, going from 1V to 0.5V, the device delay increasesby factor of 2, while the power is lowered to (1/2)

3 ofthe original value which results in a reduction of energy

4

consumption, as a product of power and delay, to (1/2)2

of its original value.The performance loss can be compensated with extra

resources to support more parallelism. If we consider 2×more resources to compensate for the 2× increase indelay, the power would increase by a factor of 2, and thedelay would decrease by half, leaving the energy reductionunchanged. So ideally, without considering the impact ofPV and faults, energy consumption can be cut to (1/2)

2

for the same performance target.However, as mentioned, the effects of PV, in particular

of Vth variation, are exponential with (Vdd−Vth), meaningthat the effect of variation is more significant at lowervoltages. Therefore, our approach is particularly useful inNTC applications.

IV. GPU StackingIn this section, we take a bottom-up approach to

construct the GPU Stacking model. We start by analysisthe fundamentals of how stacking can help mitigate PVeffects, and build upon that to get to the stacked GPU.

A. Process Variation Compensation with Voltage StackingGPU Stacking provides a unique opportunity to manage

the effects of process variation. An increased channellength (Leff ) or an increased threshold voltage (Vth) dueto process variation will result in higher impedance of thechannel and slower device.

In a conventional parallel power delivery system, theadverse process variation results in a lowered current Iini

through the corei, as Iini = Vdd/Zi4. Since gate delay is

inversely proportional to the Iini, it will result in a higherdelay and a slower core. To compensate for the lower Iini,higher voltage can be applied to the core, or the body biascould be adjusted to reduce the gate delay. In short, theadverse effect of process variation can be compensated bydelivering higher voltage.

In a stacked configuration, the same current Ic passesthrough different stack levels. Therefore, higher impedanceof corei, due to adverse process variation, results in ahigher voltage across it (Vi = IcZi).

Equation 2 shows the voltage across each core. Index iis the core or lane number. Depending on the switchingactivity of the circuit, the equivalent supply to groundimpedance of a core changes during execution. This will bereferred to as Z(t). Process variation will bias Z accordingto the magnitude of the variation. Switching activity (orthe running application) will change the transient aspectof Z. Using a performance and power simulator, we canobtain the core power traces and when physical dimensionsof the PDN are given, using a circuit simulator, we cananalyze the impedance change over time [28]. In ourexperiments, since the power consumption changes overtime, the impedance of the circuit has also varied overtime.

4Z(t) = R(t) + jX(t). The real component R defines the rela-tionship of the magnitudes of I and V , and the complex componentX affects the phase of V and I. In VLSI circuits the switching ofthe core will cause complex effects and thus phase change. To avoidimpedance effects we discuss adding extra coupling capacitors.

Vi(t) = Vin(t)×Zi(t)∑Z(t)

(2)

Utilizing the inherent feature of stacking is a keycontribution of this work. The core with higher impedancedue to adverse process variation will have a higher voltagedrop across its power terminals. This results in a corespeed up, relative to its speed without the higher voltageand with respect to a conventional power supply system,as delay is inversely proportional to the voltage. This, ofcourse, comes at the cost of a lower speed for the other corein the stack. So stacking enables the slower core to runfaster, relative to its speed in a conventional configuration,at the cost of the faster core running slower. In otherwords, the effects of process variation are intrinsicallybalanced in a stacked configuration.

Ideally, the variation effects in a stacked configurationconverge to an average variation. For this simplistic exam-ple, let us assume linear effects of PV5, then one can expectthe frequency of the stacked cores to converge to theaverage of the two cores in a conventional configuration.For example, a core with 10% variation compared to thenominal value runs at 0.9f in a conventional configuration,and a core with -10% variation runs at 1.1f . Stacking thesecores would result in both cores exhibiting delay propertiessimilar to the nominal values and run at about the nominalfrequency (+1.1f+0.9f

2 = f). After all, the nominal valueis nothing more than the mean properties across all thesamples. A non-symmetrical example, would have non-ideal compensation, e.g., c1 = 0.8f and c2 = 1.1f wouldresult in an average of 0.9f , which is better since the coreswould have to run at 0.8f if stacking was not applied.

Now, we apply this reasoning to a SPICE simulation ofa toy circuit, to test how this compensation works in a sim-ple circuit. SPICE simulations (at 45nm technology [29])are performed for an example case where inverters areconfigured in conventional and series with two stack levels.The test circuit consists of an inverter driving 4FO4, toexacerbate the delay effects with such a small circuit. Thestacked configuration is supplied with 1.2V . We test threeconfigurations: nominal (ref), not stacked with PV (PV),stacked with PV (s-xxxx). For the stacked configuration,the variation is set to affect the header inverter positively(i.e., shorter Leff , thus faster), and the footer negatively(i.e., longer Leff , thus slower).

0.8

0.6

1

1.2

0 4e-10

refPV

S-headVmid

2e-10Time (s)

Volta

ge (

V)

0

0.2

0.4

0.6

0 2e-10 4e-10

Voltage(V)

Time (s)

refPV

S-headVmid

Fig. 3. Stacked configuration (S − foot and S − head) intrinsicallymitigates the variation effects. These signals are closer to the casewithout process variation ref , than the non-stacked baseline PV .Since Vmid has shifted, there is more voltage available for the slowerpart of the design.

In this example, there is a 20% process variation ineffective channel length (Leff ). If the channel length

5This is not considered in our evaluation.

5

increases, the response time will be slower than thenominal non-variation case (lines PV and ref at thebottom of Figure 3). And if the channel shortens, theopposite effect will be observed (PV will be faster thanref at the top of Figure 3). However, this also impliesdifferent power consumption for each, as seen above.

The simulation results show that with voltage stacking,the voltage rail between the two levels of the stack(Vmid−pv) settles around 0.63V. This is because the headertransistors have less resistance due to the shorter Leff .The header transistors are effectively supplied with 1.2V- 0.63V = 0.57V, and the footer transistors with 0.63V.In this case, there is a mitigation of the delay variation,shown by S − head and S − foot, which are closer to thescenario without process variation. This is a reduction ofmore than half the delay variation introduced by processvariation. Next, we evaluate the effects of Vth variationon the inverter delays. The same scenarios are simulated,except that this time the variation is on Vth. The transientresponse for this case is similar to that of Figure 3, andthus the graph is not included.

Figure 4 summarizes all the experiments for differentvariation values from -20% to +20%. We evaluate bothLeff and Vth. Note how the delay variation is smallerwith the use of stacking. Only one line is presented forthe stacked configuration, since the stack position (headeror footer) does not change the result.

Leff Variation %

ConventionalStacked

-20

-10

10

20

De

lay

Va

ria

tio

n%

-20 -15 -10 -5 0 5 10 15 20

0

Ideal

(a) Leff

-20

-10

0

10

20

-20 -15 -10 -5 0 5 10 15 20

ConventionalStackedIdeal

Vth Variation %

De

lay V

ari

atio

n %

(b) Vth

Fig. 4. Mitigation of the variation effects compared to conventionalconfiguration.

The SPICE model confirms the premise of intrinsicmitigation of process variation in stacked configuration.Throughout the text, “variation” refers to the combinationof the effects of different sources of process variation,and its total effect on the frequency is measured, unlessotherwise specified. However, as could be expected, thecompensation is not ideal, being specially suboptimal forVth, due to the non-linearity dependency between delayand Vth.

B. Which Lanes to Stack?Now that we established how PV effects can be com-

pensated by voltage stacking, we look into the problemof how to decide which GPU lanes should be stacked foroptimal results. Stacking neighboring lanes can mitigatethe process variation effects regionally. For example,consider the die shown in Figure 5. The figure shows fourlanes with differing amounts of variation compared tothe nominal properties. Stacking the lanes based on theiradjacency (i.e., lane 1 and 2 as one stack, and lane 3 and4 as another) will help mitigate the worst case variation

(i.e., 20%). The 1-2 stack would operate at about -15%of the nominal frequency ((1.1f + 1.2f)/2 = 1.15f , inpractice the attenuation would be less). The 3-4 stackwould operate at about 15% of the nominal frequency((0.9f + 0.8f)/2 = 0.85f). The actual frequency must bethat of the slowest stack.

1

-20%

3

10%

2

-10%

4

20%

Fig. 5. Sample die with 4 lanes and different variations.

For the best results, however, the stacking configurationhas to be determined based on the observed variation,i.e., to stack a lane adversely affected by variation witha lane positively affected. For example, stacking lane 1and lane 4 together, and lane 2 and lane 3 as anotherstack would result in each running at about the nominalfrequency. The best strategy is to cluster, in each sideof the stack, lanes minimizing the standard deviation ofvariation (the rationale is that lanes with similar processvariation require approximately the same compensation).Then, the cluster with maximum process variation averagein the header should be stacked with the one withminimum negative variation in the footer. However, thismight not be trivial to find, since the number of possiblecombinations is large. A simpler approach, used in thisstudy, is to have the same number (for instance N)of lanes in all the clusters. The clustering is made bysimply picking the N lanes with maximum variation inthe header and clustering them with the N lanes withnegative variation in the footer. The process is repeatedfor the remaining lanes. This simpler approach works finein the level of lanes, since their spatial proximity causessimilar variations. Note that although this example showsan optimal case with opposite variations having the samemagnitude, this is not a requirement.

Since the variation is not known until after fabrication,a configurable fabric is needed to group and stack thelanes based on the observed variation. We adapt the ideaof a configurable power delivery fabric [14], [30] to allowthe connection of lanes that are not neighbors, becausein terms of variation management, neighboring lanes aremost likely, affected similarly. Our proposed fabric capi-talizes on the fact that GPU Stacking is actually bettersuited for the connections of logic with opposite variationeffects. Note that this may cause a problem, given thatit increases the path from Vdd to Gnd. Thus, there isa design trade-off here: on one hand, the compensationwould be better if the stacked logic were farther apartin the chip, on the other hand, if this distance is larger,the voltage droop due to voltage rails and switches isincreased. That is another reason why stacking lanes is agood design choice as opposed to stacking SMs. Lanes arewell constricted in space, within the SM, while SMs willbe farther apart in the chip, still an SM is large enoughto be used for the purposes of GPU Stacking, i.e., thereis enough variation in the SM to allow for the type of

6

compensation we aim, as observed in our evaluation.To simplify the design, we propose to cluster the

stacking of lanes. The clustering is a trade-off between costand complexity. To cluster lanes for stacking, we defineShared Net. Shared Net is a common net that connects anumber of lanes. For example, Vdd is conceptually a sharednet. However, we specifically use the term Shared Net foran intermediate net that connects a number of lanes in astacked configuration (Vmid). The number of Shared Netsis a design parameter and changes the trade-off betweenarea and compensation granularity (more Shared Netsresult in a more fine-grained compensation).

Vdd

Gnd

C1 C2 C3

C4 C5 C6

(a) Stacked

Vdd

Gnd

C1 C2 C3

C4 C5 C6

S-NetVR

(b) 3 Shared NetsFig. 6. Shared Net configuration simplifies stacking and supportspost-fabrication configurability.

Figure 6a shows a two-level stacked configuration ofsix cores with no shared nets. The stacking configurationis static, and is determined at the design time. As thevariation is not known at the design time, and thereis a spatial correlation in the variability, this schemeis not likely to provide the compensation opportunities.Figure 6b shows the design with three Shared Nets. Tosimplify the proposed design, each lane is fixed to eitherheader or footer at design time. This choice certainlyreduces the freedom of the system, but because there areseveral lanes per SM, the loss is minimal. The resistance inFigure 6b Shared Nets is not modeled in this study. Thescheme is similar to a multi-power domain case, wheremultiple voltage rails are present and each part of thecircuit connects to a different rail. The configurability onlyapplies to the intermediate nets.

The number of Shared Nets is determined at designtime based on the expected variation of the technology,or the severity of variation effects on the design metricsand to the level they need to be managed. This decisionhas to be based on the variation profile for the fabricationtechnology. Given the knowledge on the variation, it ispossible to calculate the expected power consumptionfor a block (in relative terms), e.g., using Varius-NTVmodels [4], the power information can then be used tocalculate the expected Vmid voltage for a given stackingconfiguration, which can finally be used to calculate theexpected performance. The decision on the number ofShared Nets is then a trade-off between the cost of addingan extra Shared Net and the extra performance boostgained. This is explored in our evaluation.

During post-silicon testing, each lane can be testedto characterize the observed variation for that particularlane. This increases binning time, but current chips al-ready undergo this type of testing for speedgrade purposes.

Once the effect of variation is known for each die, the lanescan join a cluster based on their observed variation. Notethat the clustering is static and done once in the lifetimeof the chip in a calibration step, right after fabrication.To allow for the post-silicon configurability, an array ofpower transistors or fuses can be used between each laneand each Shared Net. Power transistors are present inmodern designs for power gating purposes. Although wedid not consider them in this design, the reduced currentpassing through them (due to voltage stacking) will largelyreduce their impact on the circuit.

When multiple Shared Nets are present, the clusteringis done by stacking the same number of lanes (namely n)on each side (foot or head) of the Shared Net. The lanesare sorted by variation (minimum delay), for both headand foot groups. The n first lanes in the head group arestacked with the n last lanes in the foot group. The nextn lanes in each group are stacked together, and so forth.This configuration will have the maximum compensationwithin each SM, as lanes with opposed variation trendstend to be in opposite sides of the same stack, replicatingthe behavior observed in Section IV-A.

C. Divergence and Extreme ConditionsThe last component that needs to be addressed is the

presence of diverging code executing in each stack level,which has the potential to cause power mismatch andmake Vmid to diverge too much from safe levels.

However, Vmid cannot be fixed by the means of anextra traditional voltage regulator, as it is usually thecase in voltage stacking designs. The process variationcompensation comes from the fact that Vmid is “floating”,i.e., is not at a fixed frequency. Footer and header groupshave different voltages instead. Nevertheless, it is possiblethat due to load unbalance (caused by GPU divergence),or e.g., , extreme temperature conditions, the voltagedifference is such that either level has not enough voltageto guarantee correctness. We call this “voltage starvation”.We propose different mechanisms to handle such scenarios.

Dummy-activity is inserted through the activation ofparts of the lane that are not being used. For instance, ifdivergence is observed for long periods of time, this couldshift the voltage towards the most active lanes. This shiftcan be canceled by adding activity in inactive lanes, orinactive parts of lanes. When Dummy-activity is inserted,the lane does not commit any change to the architecturalstate, nor does it execute stores, for obvious reasons.

“Lane turn-off” is a more drastic measure for extremecases. In this case, there is no scheduling for one of thelanes in the level consuming more power than expected.This can only be done in architectures where each lanewithin the SM can execute different code, and wouldrequire awareness in the scheduler to be able to maintaincorrectness. The actual “turning-off” may be done in termsof power-gating, which requires lane level power-gatingor in terms of scheduling/clock-gating. If after the firstlane is turned off, there would still be deviation, a newlane is then turned-off. This could lead to big impact onperformance, thus Dummy-activity is a preferable solutionwhenever possible.

7

Scratchpad DL1

lane-j

lane-j+1 lane-2jlane-j+2

lane-0

RF

lane-1SNETs

VDD2

GNDVDD1

GND

IL1 L2 L3

VR

Fig. 7. GPU Stacking allows a more fine-grained voltage adjustmentper lane.

Additional VRs are used for extreme cases. In thestacked logic, it is expected to have Vmid floating, however,within a range that guarantees its functionality. Two smallintegrated voltage regulators are used, one pull-up and onepull-down, which would be activated when Vmid falls below0.4V or rises above 0.8V, to guarantee correct behaviorand avoid bit flips. Since these regulators only cap extremecases, they are not used in regular operations, therefore,they can be small and their potential inefficiency is notproblematic in the overall design. Since dummy-activityand lane turn-off have to be implemented in the micro-architecture level, they are naturally slower, and thus extraVRs are only used to give enough time for those to beactivated.

The natural candidate for triggering these mechanismsis Vmid, so in the case of the architectural mechanisms,the scheduler needs to be aware of the voltage during theregular operation of the GPU. VR is always connectedand is triggered without any architectural intervention. Inour experiments (details in Section VI), those mechanismswere never activated for any of the benchmarks tested,even when divergence was observed (for instance in BFS).

D. Final DesignIn this section, we present our GPU Stacking final

design. GPU Stacking does not change GPU organization(Figure 2), nor does it affect the placement. GPU Stackingdivides the GPU into different power domains, one fornon-stacked structures (e.g., caches, shared memory), andone “super” power domain with the stacked lanes (andassociated register files). It is a “super” power domainbecause, to be precise, each stack is a power domainof its own, but this is not known before fabrication.Figure 7 shows the proposed micro-architecture, from apower delivery perspective. The figure shows a solutionwith four Shared Nets.

We adopt the stacked SRAMs previously proposed [11](RF in Figure 7). Note that the SRAM stacked design hasread and write ports at nominal voltage, and thus levelshifters need to be added for read and write to the registerfile. There are multiple options available for level shifterssuitable for voltage stacking [31], with different trade-offsbetween area, power and performance.

One concern about GPU Stacking is the area overheaddue to Shared Nets. Introducing extra Shared Nets could

increase the total amount of metal dedicated to powerrails. On the other hand, GPU Stacking decreases theoverall current of the chip and metal from Vdd and Gndrails could be reduced.

Let mb be the total metal budget for power railsin a chip. For a 2-level stacked system (as the oneproposed here), the overall current is reduced by halfIstack = Ibase/2. Thus, Vdd and Gnd could have roughlyhalf the metal budget as needed in the baseline6 (mb/4for each). For a system with n Shared Nets, each SharedNet takes Istack/n current. Thus, each Shared Net needs≈ n times less metal than Vdd (mb/4/n). Consequently inGPU Stacking, the total metal budget for power rails ismvdd+mgnd+msnets = mb/4+mb/4+n·mb/4n = 3/4·mb.In other words, in a GPU Stacking configuration it ispossible to reduce the amount of metal dedicated to powerrails. In our evaluation, we assume that the metal budgetis kept constant instead (i.e., Vdd and Gnd metal is reducedby 1/3 only), which in turns helps improving the PDN byreducing the resistance.

This solution is suitable for systems not bound by powerdensity. However, if power density becomes an issue, amore elaborated power delivery network would be needed.It could be achieved by the observation that each laneonly taps either Vdd and Shared Net or Gnd and SharedNet. Thus, it is possible to reduce the amount of tracksdedicated to the unused rail on top of each lane andincrease the density of the relevant rails.

In any case, a more thorough analysis of the areaoverhead would be required when implementing a designthat uses GPU Stacking. However, since area overhead isusually not a major concern in modern chip designs, wedo not put much emphasis in this analyse and leave it asfuture work.

V. Experimental SetupTo evaluate GPU Stacking, we start by determining

what is the optimal configuration in terms of SMs andlanes for a NTC GPU. Non-NTC commercial GPUs varyin size from 2 to 16 SMs, and can have between 16 and 64lanes per SM. After deciding on the baseline GPU for ourexperiments, we evaluate the potential of GPU Stackingto compensate for process variation both in performanceand power. Then, we evaluate the reliability of the PDN,as it is a main concern in voltage stacking proposals.

A. Baseline GPU ChoiceA modified version of ESESC [32] is used to simulate

a GPGPU. For power estimation, we use a GPGPUmodel developed based on McPAT [33], very similar toGPUSimPow [34]. McPAT, takes the microarchitecturalactivity statistics from ESESC, and calculates the powerconsumption of each component. This simulation setupprovides both dynamic and leakage power as well asperformance for the applications. The temperature depen-dency of leakage is also taken into account by ESESC.

We simulate GPGPUs with a range of configurations,created either by varying the number of SMs or the

6For clarity, we are referring to the amount of tracks dedicated torails, not their pitch.

8

structures within each (e.g., number of lanes in each SM).This is summarized in Table I. McPAT [33] tool estimatesthe power consumption of the GPGPU model and onlythe on chip structures are modeled for this experiment.Since the stacking is applied within the SMs, the numberof SMs does not affect the result per SM.

Benchmarks used are from popular suites Rodinia [35],Parboil [36] and CUDA SDK (bfs, cfd, convolution,hotspot, backprop, lbm, transpose, srad and sgemm).

TABLE ISimulation parameters

Parameter 1x 1.5x 2xlanes per SM 32 32 64RFs per SM 32K 64K 64K

DL1G-Scratchpad memory per SM 32KB 32KB 64KBMaximum Frequency 1.5 GHz

Streaming Multiprocessors (SM) up to 8Threads per warp 32

Maximum Warps per SM 24L2 256KB 16wL3 4MB 32w

Memory access latency 180 cycVdd 0.4-1.0 VVth 0.30 VδV 0.1

Ambient Temperature 25C

B. Process Variation ModelingTo evaluate the impact of GPU Stacking to compensate

for the performance loss due to PV, we model PV followingan existing methodology [37]. Briefly, VARIUS-NTV [4](planar) and VARIUS-TC [38] (FinFET) are used togenerate process variation maps. VARIUS models bothfine grained (Within Die) and coarser grained (Die-to-Die)variation – both systematic and random components [39].The systematic component is modeled using a multivariatedistribution with a spherical spatial correlation structureand the random component, which occurs at the transistorlevel, is modeled analytically. VARIUS divides the chipinto n small equally sized rectangles. Each grid point hasa systematic variation of Leff and Vth which are assumedto have normal distribution. The random variation ofLeff and Vth is treated differently because of the levelof granularity at which it occurs and it is assumed to bedistributed normally and without any correlation [39].

Given the GPU floorplan as the input, VARIUS-NTVprovides die maps each with a specific process variationcase. The goal is to use VARIUS-NTV to consider theworst process variation over maps or die maps andunderstand how Vmid behaves in extreme cases of processvariation effects or an application. VARIUS-NTV alsooutputs normalized delay, normalized Leff , and effectiveVth for each component of the GPU die (lanes, caches,register files, etc.). The information from VARIUS-NTV isthen used to calculate the expected power for each elementin the stack and then the expected voltage on each SharedNet. The calculated Vmid is then fed back into VARIUS-NTV to calculate the delay and power after compensationin the stacked configurations.

This experiment is performed for different number ofShared Nets, where we compare our scheme with a conven-

tional non-stacked baseline and against multi-frequencydomain, which has been shown to have promising resultsin mitigating process variation effects [7].

C. Power Delivery Simulation and Technology NodeTo evaluate the Vmid noise and voltage noise behavior

in the GPGPU PDN, we adapt the methodology proposedby Leng et al. [28]. In short, a off-chip and on-chippower delivery network is simulated with cores modeledas current-controlled current sources, where the transientcurrent is estimated by cycle-accurate micro-architecturalsimulation for each core. This type of approach has beenused in multiple studies of this sort [40], [37], [41]. Thus,we use the power traces from ESESC for each lane.

Our SPICE simulations model the printed circuit board(PCB), the package [28], and the on-chip PDN, usingthe IBM Power Grid benchmark (ibmpg1t) [42]. Figure 8summarizes the complete PDN with the simulation pa-rameters. The grid is represented by the four resistors inthe box named “PDN”, but the simulation is performedusing the full grid. On-package capacitors (Cp) are usedto stabilize the voltage on Shared Nets and smaller on-die capacitors (Cd) are used to eliminate fast transientresponse due to mismatch. Although the figure only showsone set of on-package caps with respective C4 bump,we use one set per Shared Net. The cores are modeledas variable resistances based on the power traces fromESESC. This methodology is compatible with currentindustry practices, and short of fabricating a chip, it is thebest available method for this type of low-level analysis.For technology node, we use planar CMOS at 45nm [29],and FinFETs devices at 15nm [43].

Vdd

C4 Bumps

Rpcb,s

Rpcb,p

Cpcb

bump

Nbump

R

PDN

Lpcb,s

Rpkg,s Lpkg,s

0.1mΩ

21pH

Lpcb,s

Rpcb,s

0.27mΩ

240μF

0.55mΩ

Rpkg,sLpkg,s60pH

Rpkg,p0.54mΩ

Lpkg,p2.8pHCpkg52μF

20mΩ/N

PCB Package

0.36pH/N

corecd

cd

cd

cd

bump

Nbump

R

bump

Nbump

Rbump

Nbump

R

core

core

core

cp

cp

Fig. 8. The complete power delivery model used for simulationsincluding on small on-die decoupling caps (Cd = 100fF per SharedNet) and larger on-package decoupling caps (Cp = 5uF per SharedNet).

For between level communication, we use the levelshifter proposed in [13] which has been shown to providegood performance/energy/area trade-offs [31]. Our SPICEsimulations show that this circuit has one FO4 delay over-head at the NTC voltage, when communicating betweendifferent stacks, however these level shifters can substitutebuffers that were present in the design, minimizing theperformance impact. Memory and caches are not stackedin this study, and since the process variation is beingmitigated due to the stacking in the cores, it is expectedthat they will end up achieving a higher frequency thanmemory. To keep the improvement on the logic side, weconsider an increase in the number of access cycles ratherthan reducing the frequency of the core.

9

We consider both single-clock domain, and multi-clockdomain, in which case, each SM runs at a single frequency.In order to meet timing, this will be the frequency of theslowest lane (after compensation). It is possible to havedifferent frequencies for different SMs, but in this studyonly one frequency is considered for the whole GPU.

VI. EvaluationWe first look into the baseline selection results, since

all other results will be built upon this GPU. Then,we present the main results on how GPU Stacking canmitigate PV effects and present other advantages of ourapproach. Then, we evaluate different aspects related tothe stability of our approach to get a comprehensive ideaof how GPU Stacking affects the power delivery of a chip.Finally, we investigate how the trade-offs presented byGPU Stacking are changed when FinFET are used andfinish our evaluation discussing design and fabricationaspects of this new approach.

A. GPU Sizing and BaselineWe start our evaluation with a careful analysis of

performance, energy and area trade-offs of GPUs usingNTC. We consider different voltages and number of SMs.Figure 9 summarizes energy-delay (ED) and energy-delay-area (EDA) products for 1x with different number of SMs.The y-axis shows the normalized value for each metricwith reference to the 1x/4SM configuration at 1V supply.The x-axis is Vdd. In general, energy decreases as the Vdd

approaches near threshold region. Then the delay starts todegrade more rapidly, increasing the energy consumption,mainly due to clocked logics and leakage.

0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

2.4

0.4 0.5 0.6 0.7 0.8 0.9 1.0

Norm

alized

ed 4SM

eda 4SM

ed 6SM

eda 6SM

ed 8SM

eda 8SM

Fig. 9. Designs with more SMs become more efficient in terms ofenergy-delay product, as well as energy-delay-area product as Vdddecreases.

To understand the effect of a NTC GPU, let us examinethe 4SM and 8SM configurations at both 1.0V and atthe NTV voltage of 0.6V. Clearly, 8SM has a higherEDA than 4SM at 1.0V . However, as the Vdd approachesthe near threshold, EDA of both configuration decreases.Moreover, at around 0.6V , 8SM configuration has lowerEDA than 4SM. This means that the 8SM configurationis more efficient at lower voltages. Also the delay metricfor 8SM configuration at this point is the same as thedelay of the baseline 4SM running at 1.0V . This showsthat investment in extra resources pays off as the Vdd

approaches the near threshold region by maintainingthe performance within 4.8% of 1x/4SM configuration

operating at 1.0V , while reducing the power consumptionto about 43% of baseline. At the cost of more area.

Another observation is that the optimal configurationfor different metrics changes by changing the Vdd. Forexample Figure 10a shows the design space for EDat 1.0V . The optimal configuration is 1x/6SM. Biggerstructure sizes for cache or number of lanes could increasethe performance, but the increased power makes such atrade off less desirable due to power budget constraintsand possible thermal issues. Figure 10b shows the designspace for the same metrics at 0.6V . At this condition, theoptimal configuration is 8SM. Also the relative efficiency ofbigger structure sizes (e.g., 1.5×) increases. This impliesthat the architectural parameters, such as cache or RFsize, should be reconsidered for maximum efficiency asthe operating voltage changes.

1x1.5x

2x 4

6

8

1.1

E.D

Best configuration 1x/6SMwith E.D = 0.85

SM Size

#SM

E.D

(a) Typical

1x1.5x

2x 4

6

8

1.1

E.D

Best configuration 1x/8SMwith E.D = 0.82

SM Size

#SM

E.D

(b) Near Threshold

Fig. 10. The optimized baseline in near threshold region is differentfrom the typical super threshold region. Larger structure sizes forcache or register file or number of lanes could become more desirablein near threshold. This demands reconsidering the architecturalparameters to obtain the best energy efficiency, rather than justlowering the voltage.

B. Benefits of GPU Stacking on PVNow that we established the baseline configuration of

the NTC GPU, we look into the positive effect of GPUStacking on PV. We used the variation maps generate byVARIUS-NTV, where each map correspond to a die. Thenwe estimated performance and power for each die. Eachdie presents a unique PV, and thus GPU Stacking willhave different effects on each. We also look at the effectof different numbers of Shared Nets in the design.

0

5

10

15

20

25

0.2 0.4 0.6 0.8 1

Perc

enta

ge

Normalized Power

no-stack

1SN

2SN

4SN

8SN

0

5

10

15

20

25

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Perc

enta

ge

Normalized Performance

no-stack

1SN

2SN

4SN

8SN

Fig. 11. The proposed techniques shift the performance and powertowards the ideal scaling with no process variation.

Figure 11 shows histograms for performance and power,i.e., the y-axis is the percentage of chips (out of 10K chips)and the x-axis is the performance/power. Performance andpower are normalized to the value obtained in the casewith no process variation. Both Non-Stacked and Stacked(xSN, where x is the number of Shared Nets) methods

10

are shown. The nominal case, i.e., with no PV, would bein the x = 1 mark. We notice that without stacking andwith PV, the average performance is around 50% of thenominal. Using one Shared Net per SM does not improvethe non-stacked case much, since all the lanes share thesame Shared Net. Two, four and eight Shared Nets providegood design solutions, with eight being pretty close to fourin terms of power, but with slightly lower power. The bestconfiguration is arguably eight Shared Nets in this case,but two Shared Nets present a good trade-off betweendesign complexity and results.

Overall, GPU Stacking delivers about 75% of the per-formance, with 75% power, compared to the no variationconditions. This represents a reduction in the degradationdue to process variation: 37% in performance, and 39% inpower compared to the conventional configuration. Theincrease in power is due to the increased frequency, butEnergy-per-instruction remains roughly the same. Thismay seem like a no-gain approach, but means that GPUStacking is able to reduce the effects of process variation,delivering a chip that is closer to ideal scaling. For thesake of comparison, when using ASV with four powerdomains, it would be possible to deliver only about ≈ 62%of nominal performance, on average, while with 4 SharedNets, it would be possible to deliver ≈ 70% of nominalperformance, on average.

Another way of mitigating the effects of PV is to usemulti-clock domain [15]. Since each SM in a GPUs operatein lock-step, the maximum number of clock domainspossible is equals to the number of SMs. However, ourtechnique is orthogonal to multi-clock domains and couldbe applied in combination to it. Thus, we repeat theprevious experiment, but considering multi-clock domains.Our results (Figure 12) show that, by only using multi-clock domains, it is only possible to slightly improve theperformance to close to 60% of the nominal performance.When also applying GPU Stacking, there is a smallimprovement with 2SN . 4SN and 8SN have very similarperformance, but four Shared Nets have better power con-sumption. Thus, 4SN seems to be the best configuration,with two Shared Nets providing a good trade-off point.In summary, combining multi-clock domain and GPUStacking delivers about 80% of the nominal performance,with 70% of the power, which is a 20% improvement overmulti-clock domain only.

0

6

12

18

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Perc

enta

ge

Normalized Performance

1SN

2SN

4SN

8SN

no-stack

1SN

2SN

4SN

8SN

no-stack

0

5

10

15

20

25

0.2 0.4 0.6 0.8 1

Perc

enta

ge

Normalized Power

Fig. 12. When multi-clock domain architecture is used, GPUStacking shifts the performance further towards the ideal scalingwith no process variation.

C. Other advantages of GPU StackingBesides compensating PV effects, using GPU Stacking

has other advantages. The increase in the supply voltage

TABLE IIGPU Stacking delivers better performance and power than a

non-stacked configuration under process variation. It also allowsbetter VR efficiency, and reduced number of power pins.

Parameter Expected VariationPerformance vs. PV not-stacked +37%Power vs. PV not-stacked +39%VR area -41%VR efficiency +10%Supply pins -41%

has some indirect advantages, some of which are quantifiedin this section. They share two main causes: voltageregulators are more efficient at higher voltages and theoverall current in the circuit is reduced to roughly half [13],[8]. Using results from Hong et al. research [44], weestimate memory power as 17% of the total GPU power.The memory power is not affected by GPU Stacking.

Some of the advantages of GPU Stacking are related tovoltage regulators: VR area is reduced by roughly 2× withhalf the current [45]. In the Intel Haswell integrated VR,each 2.8mm2 cell can delivery a 25A maximum current. Amodern low-end GPU consumes ≈ 55W [46]. VR efficiencyis a function of both Vdd and output current [47]. Consid-ering both the efficiency from increasing Vdd from ≈ 0.6V(at the near-threshold region) to ≈ 1.2V , and reducingby half the total current, we expect an improvement inVR efficiency of 12%. Also, there is a ≈ 50% reduction incurrent drawn by stacked logic, that can yield a ≈ 41%reduction in VR area.

The number of pins and pads is mainly determined bythe total amount of current flowing through them. To keepthe current per pin constant, it is now possible to reducethe number of pins. Again the current related to the logicdecreases to roughly half, but the current in memories isthe same. Once more, this yields a reduction of 41% inthe total number of power pins. Note that the numberof pads dedicated to Vdd can be also decreased, but padsare now needed for Shared Nets, since on-package decapsare used. The overall number of pads is not expected tochange. Table II summarizes the expected variation inmultiple chip parameters due to GPU Stacking.

D. Analysis of PDN StabilityWe also looked into the PDN stability under different

optics to make sure that GPU Stacking can reliablyexecute different applications keeping Vmid stable and bylooking at power distributions concerns across the chip.

The main concern is that divergence between applica-tion threads could cause Vmid to diverge from acceptablelevels. In typical GPGPU applications, threads exhibitvery similar activity rates. This minimizes the possibilityof a load mismatch in the stack. In our experiments, weobserve that the power consumption of lanes are within5% of each other 98% of the time, and within 10%, 99.2%of the time. Powerlane1/Powerlane2 averages 1.000024,with standard deviation of 0.070. The sampling rate forour measurement is on the order of 1-10 MHz.

One source of concern is peak difference, which is ashigh as 40% in our experiments. We observe that mis-matches higher than 30% only occurred for the backpropbenchmark, but only 0.1% of the time for that benchmark.

11

0

0.2

0.4

0.6

0 0.5 1 1.5 2

Voltage

(V)

Execution Time (us)

stack1-bfs-headstack1-bfs-foot

stack2-bfs-headstack2-bfs-foot

stack3-backprop-headstack3-backprop-foot

Fig. 13. It is possible to maintain the voltage in each level of thestacks with on package decoupling capacitors .

Since the elevated mismatches are observed in a very shortinterval, they can be handled by decoupling capacitors.

To examine whether the achievable range of decouplingcapacitance in the design is enough for the observedmismatches, we ran a SM-like design through synthesisand back-end design down to GDS. We then extractedthe capacitance of the supply nets. Our experimentsshow that such a design using standard cell decouplingcapacitance cells (dcaps) would have total capacitance of1.4nF/mm2. At the super threshold region, with a powerdensity of about 1W/mm2, the time constant for the powersupply RC circuit would be on the order of a couple ofnanoseconds. Such a small time constant is not enough tosustain the transient mismatches that appear on the orderof 100ns to 1us. At the near threshold region, with smallerpower density, the time constant would be on the orderof 10 nanoseconds which is still not enough. Therefore, weuse of on-package capacitors, which are commonly use forPDN stabilization. Adding additional capacitors for theShared Nets will not increase the package pins [9].

In the PV case, Vmid is expected to deviate from thenominal voltage, and this behavior is desired, since it isthe source of the PV compensation. We evaluate howthe load mismatch affects the voltage available for bothlevels of the stack by carrying a SPICE simulation ofthe model presented in Figure 8. Each stack contains 16lanes, 8 in the header and 8 in the footer, a total of 4stacks (2 SMs with 32 lanes each) are hooked in the grid,equally distributed, each Vmid has two 5µF on packagedecoupling capacitor (one between Vmid and Gnd and theother between Vmid and Vdd). BFS and backprop are run,one in each SM (those where the two benchmarks withhigher mismatch between lanes).

Figure 13 shows the on-chip transient voltage for eachstack level during execution, one of the stacks is omittedfor clarity. The voltage source is 1.2V . Instead of plottingthe voltage with relation to the global Gnd, we plot thelocal voltage difference, which is more meaningful. Voltagefor each stack level stays within 10% the expected voltagefor compensation, showing a very good balance. This alsoimplies that the minimum voltage for our technique towork is Vth +10%.

Another concern regarding the PDN is lateral currentand IR drop. The main concern here is that if stacked lanesare further apart within the chip, there will be increasedresistance for current to traverse that distance.

The total current for Vdd in GPU Stacking is roughlyhalf the current than the conventional case, therefore, onewould expect decreased voltage droop. On the other hand,given a fixed budget for power delivery, the insertion of

80

60

40

20

00 20 40 60 80 100 120

(a) Conventional

80

60

40

20

00 20 40 60 80 100 120

1.20

1.18

1.16

1.14

1.12

1.10

1.08

(b) GPU Stacking

Fig. 14. GPU Stacking reduces the IR drop by reducing the totalcurrent flowing through Vdd. The stacked configuration is feed with1.2V , while the conventional was scaled up for better comparison.

new power rails requires that some of the resources usedby Vdd and Gnd in the conventional scenario be usedto Shared Nets instead. Since the current in this rails isexpected to be much smaller, the resource reduction inVdd and Gnd is small. We consider that 1/3 of the metalused to power delivery is used to Shared Nets. Figure 14shows a 2D color map of the IR drop for Vdd, both inthe conventional configuration and in GPU Stacking forthe BFS benchmark from our SPICE simulations, alreadyconsidering the reduced resources for Vdd, the contour linesare traced every 0.02V . The conventional configurationwas scaled up from 0.6V to 1.2V for better comparison.

For Vmid, we start by finding the worst case fromthe variation maps, i.e., the case with maximum powerdifference between stack levels. We note that, since weare stacking lanes, within an SM, and given the spatialcorrelation of the variation, in most cases the variation iswell below this maximum. To estimate the resistance, weuse the IBM Power Grid Benchmark (ibmpg1t), whichis properly scaled to estimate the equivalent grid ofone Shared Net. The transient simulation considers themaximum variation case. The maximum observed IR dropin Vmid was 65mV in our simulations for all benchmarks,which is 5% of the whole supply voltage (1.2V ), or 10% ofthe NTC voltage of 0.6V , since we are using over 200mVfrom Vth, this is well within acceptable margins.

E. Stacking FinFETs vs. Planar CMOSNow that we looked into the benefits of GPU Stacking

and have analyzed the stability and the impacts on thePDN, we look into how FinFETs affect the trade-offsobserved for GPU Stacking. We want to make sure thatour proposal is still useful in newer technology nodes.

The first observation is that FinFETs seem to be lesssensitive to PV effects than CMOS devices. Figure 15shows the Energy vs. Delay for both CMOS and FinFETs,with Leff variation from -15% to 15%. As we increaseLeff variation in FinFETs, the effect on energy-delay ismuch smaller than planar CMOS, which indicates that theeffect of PV in energy consumption of FinFETs is not assignificant as it is in planar CMOS. Since GPU Stackingrelies on the difference of power to shift Vmid and mitigatePV, it cannot be directly applied in that case. Also, thereis intrinsically less PV to compensate in this case.

Nevertheless, it is still possible to use GPU Stackingto compensate for PV effects on FinFET devices. To doso, we propose the use of the extra VRs to force the Vmid

12

40.00

45.00

50.00

55.00

60.00

3.32 3.36 3.40 3.44 3.48

Dela

y (

ps)

Energy (uJ)

-15%

-10%

-5%

0%

5%

10%

15%

(a) Planar CMOS 45nm

15.00

15.50

16.00

16.50

17.00

17.50

18.00

3.22 3.24 3.26 3.28

Dela

y (

ps)

Energy (uJ)

-15%

-10%

-5%

0%

5%

10%

15%

(b) FinFET 15nm

Fig. 15. Delay increases more rapidly for Planar CMOS than FinFETas we vary Leff .

voltage to a beneficial level. GPU Stacking is designed andevaluated in the same way, but during the binning process,a voltage level is chosen for each Shared Net based on thevariation of the lanes assigned to it.

Now, we look into how much PV compensation canbe obtained by applying GPU Stacking to a FinFETbased GPU design. We start from the variation mapsand calculate the expected performance and power foreach chip, considering the best forced voltage for eachShared Net. Figure 16 shows histograms for performanceand power. The y-axis is the percentage of chips (out of50K) and the x-axis is performance/power.

In Figure 16 performance and power are normalizedto the values of the no-process-variation case. We notethat in this case, the no stacking version already presentsperformance and power numbers very close to nominal(≈ 90%), still GPU Stacking is able to shift the curveabove to 95% performance, on average. Although theseresults are less impressive than the CMOS counterpart, thecompensation effect is still observed with FinFET devices.The multi-clock region experiment yielded very similarresults and was omitted for the sake of space.

0

2

4

6

8

10

12

0.8 0.85 0.9 0.95 1

Perc

enta

ge

Normalized Power

Perc

enta

ge

Normalized Performance

0

2

4

6

8

10

12

0.8 0.85 0.9 0.95 1

1SN

2SN

4SN

8SN

no-stack

1SN

2SN

4SN

8SN

no-stack

Fig. 16. The proposed techniques shift the performance and powertowards the ideal scaling with no process variation.

F. GPU Stacking Practicality IssuesFinally, we finish our evaluation by looking into some

practical aspects regarding fabrication, design, power-gating. We briefly discuss those issues in this section.

1) Extreme variation cases: In some extreme cases, thedifference in variation between the stack levels may besignificantly elevated, reducing the voltage in one of thelevels to non-acceptable levels (compared to VTH). Afterfabrication, chips are divided in performance/power gradesor discarded according to the variation. This should also bethe case in the GPU Stacking case. In practice, this is done

during binning, where timing characteristics of the chipare assessed. In this paper, to evaluate how GPU Stackingaffects yield, we use VARIUS-NTV [4] to calculate theexpected Vmid for each stack. We then compare this valueto the Vth of each lane in the stack. If the supply voltagein the lane is smaller than 1.2·Vth for at least one of thelanes, we consider the chip discarded. Our simulationsshow that, in such scenario, 1.8% of the chips would bediscarded. A more aggressive approach would be to disablelanes which do not meet such requirement, but we do notconsider this case.

2) In-Rush current and power gating: GPU Stackingreduces the current drawn from the PDN by roughlyhalf during the regular operation and start-up phase. OurSPICE models were simulated for both stacked and non-stacked configurations, and showed that in-rush is indeedsmaller in GPU Stacking (data is not included).

Power gating can be applied in addition to the GPUStacking. The only requirement is that pairs of lanes needto be powered off together (one in the head and one inthe foot group), but there is no need to power gate oneentire SM at a time.

3) Implementation flow: GPU Stacking operates in thephysical level only and does not make changes in the RTLlevel. But the physical implementation flow needs to bealtered. GPU Stacking creates voltage rails that will beconnected to the “local” GND rail of some portions of logicand to the “local” Vdd rail of others. The idea of havingmultiple power rails is similar to multiple power domains.Power transistors can then be used to open or close thecircuit and choose which Shared Net each core will tap.Another option is to crack open some connections, sincethe clustering decision do not change after fabrication.

GPU Stacking also requires isolated wells to avoidunwanted body bias effects between stack levels, thuseither Fully-Depleted SOI or triple-well technology areneeded [8]. Although this is restrictive, FD-SOI has highavailability and triple-well only requires an extra maskduring fabrication.

Sign-off of the stacked configuration may also be aconcern. However, we note that a corner based approachcan be used to determine the possible ranges of powerand timing of a stacked system. The procedure shouldtake into account what happens when stacking clusters ofdifferent types, i.e., FF and SS, FF and FF, SS and SS.Although this increases the effort for sign-off, it can betrivially implemented in EDA flows. This may be a bitof a pessimistic approach, since it will show worst-caseconfigurations that will likely not happen.

VII. ConclusionWe present GPU Stacking as a method to manage

the effects of process variation. We show that the stack-ing of cores with opposite variations tend to balancethe variation effects that would have appeared in aconventional configuration. To maximize the balancingeffect, cores with opposite variations should be stacked,which requires post-silicon configurability. We proposea clustering technique to make such a configurabilityfeasible. The homogeneous nature of GPGPU applicationsmake them suitable candidates for GPU stacking. Previous

13

voltage stacking publications only analyzed multicores andrequired complex circuitry to stabilize the voltage. Thiswork is the first to use stacking in the context of processvariation, and to propose a floating middle rail. We showthat the stable nature of GPGPUs allows for the use ofonly decoupling capacitors to stabilize the power delivery.

This research provides a detailed evaluation of NTCwith GPGPUs, and the idea of GPU stacking. We firstcarefully size a GPU for NTC operation, achieving 43%power savings, with only 4.8% performance degradation.We then apply GPU Stacking to manage process varia-tion, which impacts NTC circuits more than circuit inthe super-threshold region. The homogeneous nature ofGPGPU architectures and applications, make them a veryinteresting candidate for exploration in the extreme do-mains, with both low voltages and small feature sizes. Weshow that stacking can increase performance under processvariation at near threshold, on average, by 37% comparedto the traditional (not stacked) configuration, delivering80% of the performance compared to the no variation(ideal) conditions. Even when using multi-frequency do-main, GPU Stacking is able to further improve PV com-pensation by about 30%. Although this technique is moresuited to GPUs due to their homogeneous nature, it couldbe adapted to use in a scheme similar to CoreUnfolding [9],where authors leverage power consumption correlationamong parts of a single core.

References[1] R. G. Dreslinski, M. Wieckowski, D. Blaauw, D. Sylvester, and

T. Mudgi, “Near-threshold computing: Reclaiming moore’s lawthrough energy efficient integrated circuits,” February 2010.

[2] L. Chang, D. Frank, R. Montoye, S. Koester, B. Ji, P. Coteus,R. Dennard, and W. Haensch, “Practical strategies for power-efficient computing technologies,” Proceedings of the IEEE,vol. 98, no. 2, pp. 215–236, 2010.

[3] P. Basu, H. Chen, S. Saha, K. Chakraborty, and S. Roy, “Swift-gpu: Fostering energy efficiency in a near-threshold gpu througha tactical performance boost,” in 2016 53nd ACM/EDAC/IEEEDesign Automation Conference (DAC), June 2016, pp. 1–6.

[4] U. Karpuzcu, K. Kolluru, N. Kim, and J. Torrellas, “Varius-ntv:A microarchitectural model to capture the increased sensitivityof manycores to process variations at near-threshold voltages,”in Dependable Systems and Networks (DSN), 2012 42nd AnnualIEEE/IFIP International Conference on. IEEE, 2012, pp. 1–11.

[5] T. N. Miller, X. Pan, R. Thomas, N. Sedaghati, and R. Teodor-escu, “Booster: Reactive core acceleration for mitigating theeffects of process variation and application imbalance in low-voltage chips,” in High Performance Computer Architecture(HPCA), 2012 IEEE 18th International Symposium on. IEEE,2012, pp. 1–12.

[6] C. Silvano, G. Palermo, S. Xydis, and I. Stamelakos, “Voltageisland management in near threshold manycore architecturesto mitigate dark silicon,” in Proceedings of the conference onDesign, Automation & Test in Europe. European Design andAutomation Association, 2014, p. 201.

[7] U. R. Karpuzcu, A. Sinkar, N. Sung Kim, , and T. J.,“EnergySmart: Toward Energy-Efficient Many cores for Near-Threshold Computing,” in Proceedings of the 2013 internationalsymposium on high performance computer architecture. IEEEComputer Society, 2013.

[8] S. Lee, D. Brooks, and G. Wei, “Evaluation of Voltage Stack-ing for Near-threshold Multicore Computing,” in Low PowerElectronics and Design (ISLPED), 2012 IEEE InternationalSymposium on. ACM, 2012, pp. 373–378.

[9] E. K. Ardestani, R. T. Possignolo, J. L. Briz, and J. Renau,“Managing mismatches in voltage stacking with coreUnfolding,”ACM Trans. Archit. Code Optim., vol. 12, no. 4, pp. 43:1–43:26,Nov. 2015.

[10] Q. Zhang, L. Lai, M. Gottscho, and P. Gupta, “Multi-storypower distribution networks for GPUs,” in Design, Automation,and Test in Europe (DATE), Proceedings of, Mar 2016.

[11] E. Ebrahimi, R. T. Possignolo, and J. Renau, “SRAM voltagestacking,” in 2016 IEEE International Symposium on Circuitsand Systems (ISCAS), May 2016, pp. 1634–1637.

[12] W. Kim, D. Brooks, and G. Wei, “A fully-integrated 3-leveldc/dc converter for nanosecond-scale dvs with fast shunt regu-lation,” in Solid-State Circuits Conference Digest of TechnicalPapers (ISSCC), 2011 IEEE International. IEEE, 2011, pp.268–270.

[13] J. Gu and C. H. Kim, “Multi-story Power Delivery for SupplyNoise Reduction and Low Voltage Operation,” in Proceedingsof the 2005 International Symposium on Low Power Electronicsand Design, ser. ISLPED ’05. New York, NY, USA: ACM, 2005,pp. 192–197.

[14] S. Rajapandian, K. Shepard, P. Hazucha, and T. Karnik, “High-voltage Power Delivery Through Charge Recycling,” Solid-StateCircuits, IEEE Journal of, vol. 41, no. 6, pp. 1400–1410, 2006.

[15] J. Lee, P. Ajgaonkar, and N. Kim, “Analyzing throughputof GPGPUs exploiting within-die core-to-core frequency vari-ation,” in Performance Analysis of Systems and Software (IS-PASS), 2011 IEEE International Symposium on. IEEE, 2011,pp. 237–246.

[16] H. Wann, C. Hu, K. Noda, D. Sinitsky, F. Assaderaghi,and J. Bokor, “Channel doping engineering of mosfet withadaptable threshold voltage using body effect for low voltageand low power applications,” in VLSI Technology, Systems,and Applications, 1995. Proceedings of Technical Papers. 1995International Symposium on. IEEE, 1995, pp. 159–163.

[17] M. J. Lyons and D. Brooks, “The design of a bloom filter hard-ware accelerator for ultra low power systems,” in Proceedings ofthe 14th ACM/IEEE international symposium on Low powerelectronics and design, ser. ISLPED ’09, 2009, pp. 371–376.

[18] E. Donkoh, T. S. Ong, Y. N. Too, and P. Chiang, “Register filewrite data gating techniques and break-even analysis model,” inProceedings of the 2012 ACM/IEEE international symposiumon Low power electronics and design, ser. ISLPED ’12, 2012,pp. 149–154.

[19] J. Cong, M. A. Ghodrat, M. Gill, B. Grigorian, and G. Reinman,“Charm: a composable heterogeneous accelerator-rich micropro-cessor,” in Proceedings of the 2012 ACM/IEEE internationalsymposium on Low power electronics and design, ser. ISLPED’12, 2012, pp. 379–384.

[20] J.-H. Choi, J.-H. Lee, S.-W. Jeong, S.-D. Kim, and C. Weems,“A low power tlb structure for embedded systems,” IEEEComput. Archit. Lett., pp. 3–3, 2002.

[21] M. Huang, J. Renau, and J. Torrellas, “A framework fordynamic energy efficiency and temperature management,” inJournal on Instruction Level Parallelism, Aug 2002.

[22] C. Isci, A. Buyuktosunoglu, C.-Y. Cher, P. Bose, andM. Martonosi, “An analysis of efficient multi-core global powermanagement policies: Maximizing performance for a givenpower budget,” in Proceedings of the 39th Annual IEEE/ACMInternational Symposium on Microarchitecture, ser. MICRO 39,2006, pp. 347–358.

[23] J. Lee, V. Sathisha, M. Schulte, K. Compton, and N. S.Kim, “Improving throughput of power-constrained gpus usingdynamic voltage/frequency and core scaling,” in Proceedings ofthe 2011 International Conference on Parallel Architectures andCompilation Techniques, ser. PACT ’11, 2011, pp. 111–120.

[24] J. Zhao, G. Sun, G. H. Loh, and Y. Xie, “Energy-efficientgpu design with reconfigurable in-package graphics memory,” inProceedings of the 2012 ACM/IEEE international symposiumon Low power electronics and design, ser. ISLPED ’12, 2012,pp. 403–408.

[25] A. M. Gebhart, “Energy-Efficient Mechanisms for ManagingOn-Chip Storage in Throughput Processors,” Ph.D. disserta-tion, The University of Texas at Austin, May. 2012.

[26] A. Sankaranarayanan, E. K. Ardestani, J. L. Briz, and J. Re-nau, “An energy efficient gpgpu memory hierarchy with tinyincoherent caches,” in International Symposium on Low-PowerElectronics and Design, Beijing, China, Sept. 2013.

[27] M. Gebhart, D. R. Johnson, D. Tarjan, S. W. Keckler, W. J.Dally, E. Lindholm, and K. Skadron, “A hierarchical threadscheduler and register file for energy-efficient throughput pro-cessors,” ACM Trans. Comput. Syst., vol. 30, no. 2, pp. 8:1–8:38,Apr. 2012.

[28] J. Leng, Y. Zu, M. Rhu, M. Gupta, and V. J. Reddi, “Gpuvolt:Modeling and characterizing voltage noise in gpu architectures,”in Proceedings of the 2014 International Symposium on LowPower Electronics and Design, ser. ISLPED ’14. New York,NY, USA: ACM, 2014, pp. 141–146. [Online]. Available:http://doi.acm.org/10.1145/2627369.2627605

14

[29] W. Zhao and Y. Cao, “New generation of predictive technologymodel for sub-45nm early design exploration,” Electron Devices,IEEE Transactions on, vol. 53, no. 11, pp. 2816–2823, 2006.

[30] S. Kanev, “Motivating software-driven current balancing in flex-ible voltage-stacked multicore processors,” Ph.D. dissertation,Harvard University Cambridge, Massachusetts, 2012.

[31] E. Ebrahimi, R. T. Possignolo, and J. Renau, “Level shifterdesign for voltage stacking,” in Circuits and Systems (ISCAS),Proceedings of the 2017 IEEE International Symposium on,May 2017.

[32] E. K. Ardestani and J. Renau, “ESESC: A Fast Multicore Simu-lator Using Time-Based Sampling,” in International Symposiumon High Performance Computer Architecture, ser. HPCA’19,2013.

[33] S. Li, J. Ahn, R. Strong, J. Brockman, D. Tullsen, andN. Jouppi, “McPAT: an integrated power, area, and timingmodeling framework for multicore and manycore architectures,”in Microarchitecture. 42nd IEEE/ACM Int’l Symp. on. IEEE,2009, pp. 469–480.

[34] J. Lucas, S. Lal, M. Andersch, M. Alvarez-Mesa, and B. Ju-urlink, “How a Single Chip Causes Massive Power BillsGPUSimPow: A GPGPU Power Simulator,” in IEEE Inter-national Symposium on Performance Analysis of Systems andSoftware, 2013.

[35] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee,and K. Skadron, “Rodinia: A benchmark suite for heterogeneouscomputing,” in Proceedings of the 2009 IEEE InternationalSymposium on Workload Characterization (IISWC), ser. IISWC’09. Washington, DC, USA: IEEE Computer Society, 2009, pp.44–54.

[36] J. Stratton, C. Rodrigues, I. Sung, N. Obeid, L. Chang,N. Anssari, G. Liu, and W. Hwu, “Parboil: A revised benchmarksuite for scientific and commercial throughput computing,”Center for Reliable and High-Performance Computing, 2012.

[37] R. Thomas, K. Barber, N. Sedaghati, L. Zhou, and R. Teodor-escu, “Core tunneling: Variation-aware voltage noise mitigationin gpus,” in 2016 IEEE International Symposium on HighPerformance Computer Architecture (HPCA), March 2016, pp.151–162.

[38] S. K. Khatamifard, M. Resch, N. S. Kim, and U. R. Karpuzcu,“VARIUS-TC: A Modular Architecture-Level Model of Para-metric Variation for Thin-Channel Switches,” in 2016 IEEE 34thInternational Conference on Computer Design (ICCD), vol. 00.Los Alamitos, CA, USA: IEEE Computer Society, 2016, pp.654–661.

[39] S. Sarangi, B. Greskamp, R. Teodorescu, J. Nakano, A. Tiwari,and J. Torrellas, “VARIUS: A model of process variationand resulting timing errors for microarchitects,” SemiconductorManufacturing, IEEE Transactions on, vol. 21, no. 1, pp. 3–13,2008.

[40] J. Leng, Y. Zu, and V. J. Reddi, “GPU Voltage Noise: Charac-terization and Hierarchical Smoothing of Spatial and TemporalVoltage Noise Interference in GPU Architectures,” in 2015 IEEE21st International Symposium on High Performance ComputerArchitecture (HPCA), Feb 2015, pp. 161–173.

[41] W. Kim, M. S. Gupta, G.-Y. Wei, and D. Brooks, “SystemLevel Analysis of Fast, per-Core DVFS Using on-Chip SwitchingRegulators,” in 2008 IEEE 14th International Symposium onHigh Performance Computer Architecture, Feb 2008, pp. 123–134.

[42] S. Nassif, “Power grid analysis benchmarks,” in Design Automa-tion Conference, 2008. ASPDAC 2008. Asia and South Pacific,2008, pp. 376–381.

[43] S. Khandelwal, J. Duarte, N. Paydavosi, D. Lu, C.-H. Lin,Mohan, S. Yao, T. Morshed, A. Niknejad, and C. Hu, “BSIM-CMG 108.0.0 Multi-Gate MOSFET Compact Model,” 2014.

[44] S. Hong and H. Kim, “An integrated GPU power andperformance model,” SIGARCH Comput. Archit. News,vol. 38, no. 3, pp. 280–289, Jun. 2010. [Online]. Available:http://doi.acm.org/10.1145/1816038.1815998

[45] I. Corporation. (2010) A 400 amp fully integrated silicon voltageregulator with in-die magnetically coupled embedded inductors.Palm Spring, CA.

[46] (2017) NVIDIA GeForce GTX750 Specifications. NVIDIACorporation. Available on http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-750/specifications.

[47] I. Technologies. (2013) High performance DrMos TDA21220.Available on http://www.infineon.com/cms/en/product/power/dc-dc-converter/dc-dc-integrated-power-stage/drmos-integrated-power-stage/TDA21220/productType.html?productType=db3a3044243b532e0124de3165386adc.

Rafael Trapani Possignolo is a Ph.D. candi-date at UC Santa Cruz, working with com-puter architecture and VLSI design automa-tion. His main research interest lies in tech-niques and tools to improve design produc-tivity, such as automated pipelining and fastsynthesis methods. Rafael is also interested inarchitecture and circuit level techniques thatenable better power and performance trade-offs, such as voltage stacking. Rafael holdsa B.S. in Computer Engineering an M.S. in

Electrical Engineering from University of Sao Paulo, an M.S. in Em-bedded Systems Engineering, and an Engineering degree from ÉcoleNationale Supérieure des Techniques Avancees (ENSTA-ParisTech).

Elnaz Ebrahimi is a Ph.D. candidate at UCSanta Cruz, part of Santa Cruz MicroAr-chitecture (MASC) laboratory. Her primaryresearch is in architecture- and circuit-levelSRAM designs that ease the conventional de-sign paradigms. She is also interested in powermodeling and power-efficient design of proces-sor microarchitecutres. Elnaz holds an M.S. inComputer Architecture form UC Santa Cruzand a B.S. in Computer Engineering from SanJose State University.

Ehsan K.Ardestani is a CPU Architect andPhD Alumni of Computer Engineering at theUniversity of California, Santa Cruz. His re-search interests include computer architecture,and simulation methodology. He received hisB.S. and M.S. degree in Computer Engineer-ing and Computer Architecture from IsfahanUniversity and Amirkabir University of Tech-nology, Iran, respectively.

Alamelu Sankaranarayanan obtained herPh.D. in Computer Engineering from UCSanta Cruz in 2016. Her areas of interestinclude processor microarchitectures, energyefficient design and architectural simulationmethodologies.

José Luis Briz is an Associate Professor at theUniversity of Zaragoza (UZ). His research in-terests include memory hierarchy and proces-sor microarchitecture, embedded systems andreal-time scheduling on MPSoCs. He as alsocontributed to the application of computersto Structural Geology. Briz has a combinedBS/MS degree in Geology, a MS degree inComputer Science, and a Ph.D. in ComputerEngineering, all of them from UZ. He is amember of the gaZ group, the I3A Research

Institute, and Affiliate of the HiPEAC European Network of Excel-lence. He is also a member of the ACM, the IEEE, and of the SpanishSociety of Computer Architecture (SARTECO)

Jose Renau is a professor of computer en-gineering at the University of California,Santa Cruz (http://masc.soe.ucsc.edu/). Hisresearch focuses on computer architecture,including design effort metrics and models,infrared thermal measurements, low-powerand thermal-aware designs, process variability,thread level speculation, FPGA/ASIC design,Fluid Pipelines, and Pyrope (a modern hard-ware description language) and Live flows thataim at productivity in hardware design. Renau

has a Ph.D. in Computer Science from the University of Illinois atUrbana-Champaign.