Balancing DRAM Locality and Parallelism in Shared Memory CMP Systems

Min Kyu Jeong*, Doe Hyun Yoon†, Dam Sunwoo‡, Michael Sullivan*, Ikhwan Lee*, and Mattan Erez*

* Dept. of Electrical and Computer Engineering, The University of Texas at Austin† Intelligent Infrastructure Lab, Hewlett-Packard Labs

‡ ARM Inc.

{mkjeong, mbsullivan, ikhwan, mattan.erez}@mail.utexas.edudoe-hyun.yoon@hp.com dam.sunwoo@arm.com

Abstract

Modern memory systems rely on spatial locality toprovide high bandwidth while minimizing memory devicepower and cost. The trend of increasing the number of coresthat share memory, however, decreases apparent spatial lo-cality because access streams from independent threads areinterleaved. Memory access scheduling recovers only afraction of the original locality because of buffering lim-its. We investigate new techniques to reduce inter-threadaccess interference. We propose to partition the internalmemory banks between cores to isolate their access streamsand eliminate locality interference. We implement this byextending the physical frame allocation algorithm of theOS such that physical frames mapped to the same DRAMbank can be exclusively allocated to a single thread. Wecompensate for the reduced bank-level parallelism of eachthread by employing memory sub-ranking to effectively in-crease the number of independent banks. This combinedapproach, unlike memory bank partitioning or sub-rankingalone, simultaneously increases overall performance andsignificantly reduces memory power consumption.

1 Introduction

Modern main memory systems exploit spatial locality toprovide high bandwidth while minimizing memory devicepower and cost. Exploiting spatial locality improves band-width and efficiency in four major ways: (1) DRAM ad-dresses can be split into row and column addresses whichare sent to the memory devices separately to save addresspins; (2) access granularity can be large to amortize con-trol and redundancy overhead; (3) page mode is enabled,in which an entire memory row is saved in the row-buffer

c©2012 IEEE HPCA-18 New Orleans, Louisiana USA. Personal useof this material is permitted. Permission from IEEE must be obtainedfor all other uses, in any current or future media, including reprint-ing/republishing this material for advertising or promotional purposes, cre-ating new collective works, for resale or redistribution to servers or lists,or reuse of any copyrighted component of this work in other works.

so that subsequent requests with the same row address needonly send column addresses; and (4) such row-buffer hit re-quests consume significantly less energy and have lower la-tency because the main data array is not accessed.

Many applications present memory access patterns withhigh spatial locality and benefit from the efficiency and per-formance enabled by page mode accesses. However, withthe increasing number of cores on a chip, memory accessstreams have lower spatial locality because access streamsof independent threads may be interleaved at the memorycontroller [2, 23]. Figure 1 shows, for example, the impactof access interleaving on the spatial locality of the SPECCPU2006 benchmark lbm. The DRAM row-buffer hit rateof one lbm instance is shown for three different workloadmixes. When lbm runs alone, 98% of the accesses hit in therow-buffer. When 4 instances of lbm run together, however,the hit rate drops to 50%. The spatial locality of an inter-leaved stream can drop even further depending on the ap-plication mix, because some applications are more intrusivethan others. The SPEC CPU2006 benchmark mcf, for ex-ample, is memory intensive but has very poor spatial local-ity, and thus severely interferes with the memory accesses ofother applications. In a multi-programmed workload con-taining mcf, the row-buffer hit rate of lbm drops further to35%. Loss of spatial locality due to inter-thread interferenceincreases energy consumption because of additional row ac-tivations, increases the average access latency, and typicallylowers the data throughput of the DRAM system.

Out-of-order memory schedulers reorder memory oper-ations to improve performance and can potentially recoversome of the lost spatial locality. However, reclaiming lostlocality is not the primary design consideration for theseschedulers. Instead, they are designed either to simplymaximize the memory throughput for a given interleavedstream (e.g., FR-FCFS [20]), or to maximize some notionof fairness and application throughput (e.g., ATLAS [9] andTCM [10]). In other words, improved scheduling can some-what decrease the detrimental impact of interleaving butdoes not address the issue at its root. Furthermore, the effec-tiveness of memory access scheduling in recovering local-ity is constrained by the limited scheduling buffer size and

1

0%

20%

40%

60%

80%

100%

lbm lbm x 4 lbm,libq,mcf,omnetpp

DR

AM

RB

Hit

Rat

e

Figure 1: DRAM row-buffer hit rate of lbm whenrun alone, with 4 instances, and when run withother applications. FR-FCFS scheduling is usedfor higher hit rate. See Section 4 for details of thesimulation setup.

the (often large) arrival interval of requests from a singlestream. As the number of cores per chip grows, reorderingwill become less effective because the buffer size does notscale and contention for the shared on-chip network can in-crease the request arrival interval of each thread. With anincreased arrival interval, subsequent spatially adjacent re-quests have a higher chance of missing the scheduling win-dow, and the row is precharged prematurely.

We investigate a fundamentally different approach thataddresses locality interference at its root cause. Ratherthan recovering limited locality with scheduling heuristics,we propose to reduce (and possibly eliminate) row-bufferinterference by restricting interfering applications to non-overlapping sets of memory banks. In some cases, however,restricting the number of banks available to an applicationcan significantly degrade its performance as fewer banksare available to support concurrent overlapping memory re-quests. This is of particular concern because the total num-ber of banks available to a processor is growing more slowlythan the total number of threads it can concurrently execute.Therefore, we combine bank partitioning with memory sub-ranking [24, 27, 3, 2, 4], which effectively increases thenumber of independent banks. Together, bank partitioningand memory sub-ranking offer two separate tuning knobsthat can be used to balance the conflicting demands for row-buffer locality and bank parallelism. We show that this pow-erful combination, unlike memory sub-ranking or bank par-titioning alone, is able to simultaneously increase overallperformance and significantly reduce memory power con-sumption.

The rest of this paper is organized as follows: Section2 discusses how the DRAM address mapping affects local-ity interference and describes our bank partitioning mecha-nism. Section 3 describes how bank partitioning reduces theavailable DRAM bank-level parallelism, and investigateshow sub-ranking can recover the lost parallelism. Section4 describes our evaluation methodology, and Section 5 dis-cusses results. Section 6 reviews related work, and Sec-tion 7 concludes.

2 Bank Partitioned Address MappingRow-buffer locality interference in multicore processors

results from the mechanisms used to map the addresses ofthreads onto physical memory components. Current map-

ping methods evolved from uniprocessors, which do notfinely interleave independent address streams. These meth-ods are sub-optimal for chip multi-processors where mem-ory requests of applications can interfere with one another.In this section, we first describe the mechanism commonlyin use today and then discuss our proposed mapping mech-anism that simultaneously improves system throughput andenergy efficiency.2.1 Current Shared-Bank Address Mapping

In traditional single-threaded uniprocessor systems withno rank-to-rank switching penalty, memory system perfor-mance increases monotonically with the number of avail-able memory banks. This performance increase is due tothe fact that bank-level parallelism can be used to pipelinememory requests and hide the latency of accessing the in-herently slow memory arrays. To maximize such overlap,current memory architectures interleave addresses amongbanks and ranks at fine granularity in an attempt to distributethe accesses of each thread across banks as evenly as possi-ble. However, because row-buffer locality can be exploitedto improve efficiency and performance, it is common to mapadjacent physical addresses to the same DRAM row so thatspatially close future accesses hit in the row-buffer. There-fore, the granularity of bank interleaving in this case is equalto the DRAM row size.

Figure 2 gives an example of the typical row-interleavedmapping from a virtual address, through a physical address,to a main memory cell. The mapping given in Figure 2(a)is for a simple memory system (1 channel, 1 rank, and4 banks) and is chosen to simplify the following discus-sion. The mapping in Figure 2(b) is given for a more re-alistic DDR3 memory system with 2 channels, 4 ranks, and8 banks. When two threads concurrently access memory,both threads may access the same memory bank becausethe address space of each thread maps evenly across allbanks. Figure 2(c) depicts the full address translation pro-cess and shows how two threads with the simple mapping(Figure 2(a)) can map to the same bank simultaneously.When thread P0 and P1 accesses their virtual pages 2 and3, they conflict in banks 2 and 3. Bank conflicts, such asthis one, thrash the row-buffers and can significantly reducethe row-buffer hit rate. The likelihood of this type of in-terference grows with the number of cores as more accessstreams are interleaved.2.2 Bank Partitioned Address Mapping

If independent threads are not physically mapped to thesame bank, inter-thread access interference cannot occur.We call this interference avoidance mechanism bank par-titioning as proposed by Mi et al. [13]. Figure 3(a) givesan example address mapping scheme for a simple systemwhich utilizes bank partitioning. In this example, bit 14 ofthe physical address denotes the core ID. Core 0 is allocatedphysical frames {0, 1, 4, 5, 8, 9, ...}, and core 1 is allocatedframes {2, 3, 6, 7, 10, 11, ...}. This mapping partitions thebanks into two sets such that processes running on differ-ent cores are constrained to disjoint banks, thus avoidingrow-buffer interference. Figure 3(c) shows the correspond-ing change in virtual page layout in DRAM and illustrates

2

Virtual Address

Physical Address

DRAM Address Bit-mask

… 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Virtual Page Number Page Offset

Physical Frame Number Frame Offset

Row Bank Column

Bit index

(a) Virtual to physical to DRAM address mapping scheme for a simple memory system.

Virtual Address

Physical Address

DRAM Address Bit-mask

… 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Virtual Page Number Page Offset

Physical Frame Number Frame Offset

Row Rank Bank Column Ch Column

Bit index

(b) Generalized Virtual to physical to DRAM address mapping scheme.

Page # Frame # 0 x00 1 x01 2 x02 3 x03

Frame Table Page Table P0

Frame # DRAM Addr x00 Bank 0, Row 0 x01 Bank 1, Row 0 x02 Bank 2, Row 0 x03 Bank 3, Row 0 x04 Bank 0, Row 1 x05 Bank 1, Row 2 … ….

x40 Bank 0, Row 16 x41 Bank 1, Row 16 x42 Bank 2, Row 16 x43 Bank 3, Row 16

x00

x04

x40

Bank 0 Bank 1 Bank 2 Bank 3

Row 0

Row 1

…

Row 16

x01

x05

x41

x02

x06

x42

x03

x07

x43

Page # Frame # 0 x04 1 x05 2 x42 3 x43

Page Table P1

Physical Frame Layout in DRAM

(c) The layout of virtual pages allocated to each process in the DRAM banks according to the map (a).

Figure 2: Conventional address mapping scheme

how bank interference is eliminated. Virtual pages in P0and P1 are mapped to only banks {0, 1} and {2, 3}, respec-tively. Therefore, they do not share row-buffers and cannotinterfere with one another. Figure 4 gives an example ofmemory behavior with and without bank partitioning. P0and P1 concurrently access their virtual page #2 in an inter-leaved manner; bank partitioning eliminates the prechargecommands and the second activate command for P0. Thisimproves both throughput and power efficiency relative tothe memory behavior using conventional frame allocation.

Figure 3(b) shows a generalized bank partitioning ad-dress mapping scheme for a typical DDR3 memory systemwith 2 channels, 4 ranks, and 8 banks. Different colors rep-resent independent groups of banks. Given a number of col-ors, there is a wide decision space of allocating colors toprocesses. We can imagine an algorithm that allocates col-ors depending on the varying applications’ need for bank-level parallelism. However, this study uses a static parti-tion which assigns an equal number of colors to each core.Such static partitioning does not require profiling of work-loads and is robust to dynamic workload changes. We findthat there is a diminishing return in increasing the numberof banks that a modern out-of-order core can exploit frommore bank-level parallelism. For realistic modern memorysystem configurations which have internal banks and con-sist of multiple channels and ranks, sub-ranking can com-pensate for reduced per-thread bank count. More details onbank-level parallelism will be discussed in Section 3.

Note that the mapping between physical and DRAM ad-dresses stays the same, allowing the address decoding logic

in the memory controller to remain unchanged. Bank iso-lation is entirely provided by the virtual to physical addressmapping routine of the operating system. Therefore, when acore runs out of physical frames in colors assigned to it, theOS can allocate different colored frames at the cost of re-duced isolation. The decision of whose color the frames arespilled to can be further explored, but is beyond the scopeof this paper.

XOR-permuted bank indexing is a common memory op-timization which reduces bank conflicts for cache-inducedaccess patterns [26]. Without any permutation, the set in-dex of a physical address contains the bank index sent toDRAM. This maps all cache lines in a set to the same mem-ory bank, and guarantees that conflict-miss induced trafficand write-backs will result in bank conflicts. XOR-basedpermutation takes a subset of the cache tag and hashes itwith the original bank index such that lines in the samecache set are mapped across all banks.

XOR-permuted bank indexing nullifies color-based bankpartitioning and re-scatters the frames of a thread across allbanks. We use a technique proposed by Mi et al.[13] tomaintain bank partitioning. Instead of permuting the mem-ory bank index, we permute the cache set index. In thisway, a single color still maps to the desired set of banks. Atthe same time, cache lines within a set are mapped acrossmultiple banks and caches sets are not partitioned.

3 Bank-Level ParallelismBank partitioning reduces the number of banks available

to each thread. This section examines the impact that re-

3

Virtual Address

Physical Address

DRAM Address Bit-mask

… 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Virtual Page Number Page Offset

Physical Frame Number CID PFN Frame Offset

Row Bank Column

Bit index

(a) Virtual to physical to DRAM address mapping scheme for a simple memory system.

Virtual Address

Physical Address

DRAM Address Bit-mask

… 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Virtual Page Number Page Offset

Physical Frame Number Colors PFN Frame Offset

Row Rank Bank Column Ch Column

Bit index

(b) Generalized Virtual to physical to DRAM address mapping scheme.

Page # Frame #0 x001 x012 x043 x05

Frame TablePage Table P0

Frame # DRAM Addrx00 Bank 0, Row 0x01 Bank 1, Row 0x02 Bank 2, Row 0x03 Bank 3, Row 0x04 Bank 0, Row 1x05 Bank 1, Row 2… ….

x40 Bank 0, Row 16x41 Bank 1, Row 16x42 Bank 2, Row 16x43 Bank 3, Row 16

x00

x04

x40

Bank 0 Bank 1 Bank 2 Bank 3

Row 0

Row 1

…

Row 16

x01

x05

x41

x02

x06

x42

x03

x07

x43

Page # Frame #0 x021 x032 x423 x43

Page Table P1

Physical Frame Layout in DRAM

(c) The layout of virtual pages allocated to each process in the DRAM banks according to the map (a).

Figure 3: Address mapping scheme that partitions banks between processes.

A R

D

P A R

D

P A R

D

A R

D

A R

D

R

D

P0 P1 P0

Bank 2 Row 0

Bank 2 Row 16

Bank 2 Row 0

Bank 0 Row 1

Bank 2 Row 16

(a) Conventional Frame Allocation

(b) Bank Partitioned Frame Allocation

CMD

DATA

CMD

DATA

Figure 4: Benefits of bank partitioning. The upperdiagram corresponds to Figure 2(c), and the lowerdiagram corresponds to Figure 3(c). A, R, P, andD stand for activate, read, precharge, and data, re-spectively. The row-buffer is initially empty.

duced bank-level parallelism has on thread performance,and investigates the ability of memory sub-ranking to addmore banks to the system.

3.1 Bank-level Parallelism

The exploitation of bank-level parallelism is essential tomodern DRAM system performance. The memory systemcan hide the long latencies of row-buffer misses by overlap-ping accesses to many memory banks, ranks, and channels.

In modern out-of-order processor systems, a main mem-ory access which reaches the head of the reorder buffer(ROB) will cause a structural stall. Given the disparityof off-chip memory speeds, most main memory accessesreach the head of the ROB, negatively impacting perfor-mance [8]. Furthermore, memory requests that miss inthe open row-buffer require additional precharge and acti-vate latencies which further stall the waiting processor. If athread with low spatial locality generates multiple memoryrequests to different banks, they can be accessed in parallelby the memory system. This hides the latency of all but thefirst memory request, and can dramatically increase perfor-mance.

A thread, especially one with low spatial locality, canbenefit from many banks up to the point where its latencyis completely hidden. With DDR3-1600, a row-buffer misstakes about 33 DRAM clock cycles before it returns data (inthe following 4 DRAM clock cycles). Therefore, to com-pletely hide the latency of subsequent requests, a threadneeds to have 8 banks working in parallel 1 2. In general,as more requests are overlapped, the CPU will stall for lesstime and performance will increase.

While bank-level parallelism undoubtedly improves per-formance, its benefits have a limit. After the available par-

1tFAW limits the number of activate commands to 6 per row-buffermiss latency, due to power limitations. Also, tRRD has increased so thatactivates should be separated by more than 4-cycle data burst, causing idlecycles on the data bus in the case that all requests miss in the row-buffer.

2Since a row-buffer miss takes a constant latency, more bank-level par-allelism is needed with faster interfaces such as DDR3-1866 and 2133.

4

0.9

1.1

1.3

1.5

1.7

1.9

2x1 4x1 8x1 8x2 8x4 8x8 8x16 Spee

du

p o

ver

2 b

anks

sys

tem

Number of banks X Number of ranks lbm milc soplex libquantum mcf omnetpp leslie3d sphinx3 sjeng bzip2 astar hmmer h264ref namd

Figure 5: Normalized application execution timewith varying number of banks.

allelism can hide memory latencies, there is no further ben-efit to increasing the number of banks. Figure 5 shows ap-plications’ sensitivity to the number of banks in the mem-ory system. Most applications give peak performance at8 or 16 banks. After this point, applications show no fur-ther improvement or even perform worse, due to the 2 cyclerank-to-rank switching delay. lbm is an outlier that benefitsbeyond 16 banks. lbm shows a higher row-buffer hit-rateas the number of banks increases– it accesses multiple ad-dress streams concurrently, and additional banks keep morerow-buffers open and can fully exploit each stream’s spatiallocality.

3.2 Sub-ranking

The most straightforward way to retain sufficient bank-level parallelism with bank partitioning is to increase thetotal number of banks in the memory system. The totalnumber of banks in the system is equal to the product ofthe number of channels, the number of ranks per channel,and the number of internal banks per chip. Unfortunately,increasing the number of available banks is more expensivethan simply adding extra DIMMs.

There are a number of factors which complicate the addi-tion of more memory banks to a system. First, and most fun-damentally, paying for additional DRAM chips and powerjust to increase the number of banks is not a cost-efficientsolution for system designers. Also, the number of channelsis limited by the available CPU package pins. Furthermore,when fine-grained channel interleaving is used to maximizedata link bandwidth, the contribution of the channel countto the effective number of banks is limited. In addition,the number of ranks per channel is constrained by signalintegrity limitations. Point-to-point topologies as used inFB-DIMMs [12] can overcome this problem, but come at acost of additional latency and power consumption. Finally,adding internal banks to each DRAM chip requires expen-sive circuitry changes. Considering the extreme competi-tion and low margin of the DRAM market, more internal

banks would be difficult to achieve in a cost effective man-ner. For these reasons, with continued device scaling andthe recent trend of increasing core count, future systems areexpected to have higher core to bank ratio than current sys-tems.

We propose an alternative approach to increase the bank-level parallelism without adding expensive resources. Weemploy DRAM sub-ranking [2, 27, 24, 25] to effectively in-crease the number of banks available to each thread. DRAMsub-ranking breaks a wide (64-bit) rank into narrower (8-bit, 16-bit, or 32-bit) sub-ranks, allowing individual controlof each sub-rank. In the conventional memory system, allbanks within a rank work in unison and hold the same row.In the 32-bit sub-ranked system, two groups of four banksare independent and can hold different rows. By controllingthe sub-ranks independently, multiple long-latency missescan overlap, effectively increasing the available bank-levelparallelism. However, since fewer devices are involved ineach request, sub-ranked DRAM takes more reads (and thusa longer latency) to complete the same size of request. Al-though narrow 16-bit and 8-bit sub-ranks can provide abun-dant bank-level parallelism, additional access latencies dueto sub-ranking become prohibitive. Given a 4GHz proces-sor, 16-bit and 8-bit sub-ranks take 40 and 80 additionalCPU cycles, respectively, to access main memory. In addi-tion, narrow sub-ranks put greater pressure on the addressbus, which could impact performance or necessitate expen-sive changes to DRAM. Accordingly, we use 32-bit sub-ranks to balance bank-level parallelism and access latency,and avoid the need for extra address bus bandwidth.

4 Evaluation Methodology

We evaluate our proposed bank partitioning and sub-ranking scheme using Zesto, a cycle-level x86 out-of-orderprocessor simulator [11], and a detailed in-house DRAMsimulator [7]. Zesto runs user space applications bare-metaland emulates system calls. We extend the physical frameallocation logic of Zesto to implement bank partitioningamong cores as presented in Section 2.2. Our DRAM simu-lator supports sub-ranked memory systems. It models mem-ory controllers and DRAM modules faithfully, simulatingthe buffering of requests, scheduling of DRAM commands,contention on shared resources (such as address/commandand data buses), and all latency and timing constraints ofDDR3 DRAM. It also supports XOR interleaving of bankand sub-rank index to minimize conflicts [26], which isused for the baseline shared-bank configuration and re-placed with cache-set index permutation described in 2.2for bank-partitioned configurations.

System Configuration

Table 1 summarizes the system parameters of our simu-lated systems. We simulate the following 4 configurationsto evaluate our proposed mechanism.

5

Table 1: Simulated system parameters

Processor 8-core, 4GHz x86 out-of-order, 4-wideL1 I-caches 32KiB private, 4-way, 64B line size, 2-cycleL1 D-caches 32KiB private, 8-way, 64B line size, 2-cycleL2 caches 256KiB private for instruction and data, 8-way

64B line size, 6-cycleL3 caches 8MiB shared, 16-way, 64B line size, 20-cycleMemory FR-FCFS scheduling, open row policycontroller 64 entries read queue, 64 entries write queueMain memory 2 channels, 2 ranks / channel, 8 banks / rank

8 x8 DDR3-1600 chips / rankAll parameters from the Micron datasheet [14]

1. shared: Baseline shared-bank system.2. bpart: Banks partitioned among cores. Each core

receives consecutive banks from one rank.3. sr: Ranks split into two independent sub-ranks.4. bpart+sr: Bank partitioning with sub-ranking used

together.

Power modelsWe estimate DRAM power consumption using a powermodel developed by the Micron Corporation [1]. For pro-cessor power, we use the IPC-based estimation presentedin [2]: the maximum TDP of a 3.9GHz 8-core AMD bull-dozer processors is reported as 125W; half of the maximumpower is assumed to be static (including leakage) and theother half is dynamic power that is proportional to IPC. Ourstudy focuses on main memory and, as such, our mecha-nisms have a minimal impact on the core power.

WorkloadsWe use multi-programmed workloads consisting of bench-marks from the SPEC CPU2006 suite [22] for evaluation.Our evaluation is limited to multi-programmed workloadsdue to the limitation of our infrastructure. The principle ofplacing memory regions that interfere in access schedulingin different banks to better exploit spatial locality, however,applies to both multi-programming and multi-threading.Identifying conflicting regions can be more challengingwith multi-threading, yet doable (e.g., parallel domain de-composition can indicate bank partitions; and thread-privatedata is often significant).

To reduce simulation time, we use SimPoint [5] and de-termine a representative 200 million instruction region fromeach application. Memory statistics from identified regionare used to guide the mix of applications in our multi-programmed workloads. Table 2 summarizes the charac-teristics of each application. Some benchmarks are not in-cluded in our evaluation due to limitations of Zesto.

Two key application characteristics which are perti-nent to our evaluation are the memory access intensityand the row-buffer spatial locality. The memory accessintensity, represented by last-level cache misses-per-kilo-instructions(LLC MPKI), is an indicator of how much an

Table 2: Benchmarks statistics when running onthe baseline. IPC: Instructions per cycle, LLCMPKI: Last level-cache misses per 1000 instruc-tions, RB Hit Rate: Row-buffer hit rate, and MemBW: Memory bandwidth used in MiB/S.

Benchmark IPC LLC MPKI RB Hit Rate Mem BW

lbm 0.71 12.80 97% 3394.56milc 0.69 16.15 82% 3576.32soplex 0.53 20.88 90% 3246.08libquantum 0.53 15.64 98% 2967.04mcf 0.54 16.15 10% 2258.94omnetpp 0.69 9.51 49% 2136.83leslie3d 0.73 8.04 89% 1942.53sphinx3 1.03 0.66 77% 171.72sjeng 0.81 0.68 17% 141.49bzip2 0.94 0.49 82% 114.48gromacs 1.35 0.54 96% 88.76astar 0.91 0.17 62% 36.07hmmer 1.00 0.13 97% 34.20h264ref 0.71 0.16 81% 28.03namd 1.09 0.09 91% 23.25

application is affected by the memory system performance.We focus most our evaluation on workloads that includememory intensive applications.

The spatial locality of application, represented by therow-buffer hit rate, is another key characteristic. Applica-tions with high row-buffer hit rates suffer the most fromrow-buffer locality interference; conversely, they stand tobenefit the most from bank partitioning. On the other hand,applications with low row-buffer hit rates rely on bank-levelparallelism, and may be adversely affected by any reductionin the number of available banks.

Among the memory-intensive benchmarks, mcf andomnetpp have low spatial locality. libquantum, lbm,milc, soplex and leslie3d all demonstrate high spa-tial locality in the absence of interference.

Table 3 shows the mix of benchmarks chosen for ourmulti-programmed workloads. Unless otherwise noted, allbenchmark mixes contain programs with high memory in-tensity. Group HIGH consists of workloads with high spa-tial locality. Group MIX is a mixed workload of pro-grams with both high and low spatial locality. GroupLOW consists of workloads with low spatial locality. Fi-nally, group LOW BW is composed of workloads that con-tain non-memory-intensive benchmarks to see the effect ofour mechanisms on compute-bound workloads. Each mixin the table contains one, two, or four benchmarks. Thesebenchmark mixes are replicated to have eight benchmarksto run on each of the eight cores in our simulated system.

Metrics

To measure system throughput, we use Weighted Speedup(WS) [21], as defined by Equation 1. IPCalone andIPCshared are the IPC of an application when it is runalone and in a mix, respectively. The number of applica-tions running on the system is given by N .

6

Table 3: Multi-programmed workloads composi-tion. Workloads are then replicated to have 8 pro-grams per mix. RBH : Row-buffer hit rate.

Memory Intensive Memory non-intensive

Workload High RBH Low RBH High RBH Low RBH

HIGHH1 lbm,milc,soplex,libqH2 lbm,milc,libq,leslie3dH3 lbm,milc,soplex,leslie3dH4 lbm,soplex,libq,leslie3dH5 milc,soplex,libq,leslie3d

MIXM1 milc,libq,leslie3d mcfM2 lbm,milc,libq omnetppM3 soplex,leslie3d mcf,omnetppM4 lbm,libq mcf,omnetppM5 milc,leslie3d mcf,omnetppM6 soplex,milc mcf,omnetppM7 libq,soplex mcf,omnetpp

LOWL1 mcfL2 mcf,omnetppL3 omnetpp

LOW BWLB1 leslie3d,libq,soplex bzip2LB2 milc,leslie3d omnetpp sphinx3LB3 milc,soplex sphinx3 sjengLB4 lbm mcf bzip2 sjengLB5 libq omnetpp bzip2 sjeng

WS =

N−1∑i=0

IPCsharedi

IPCalonei

(1)

System power efficiency, expressed in terms of through-put (WS) per unit power, is also reported. System poweris the combined power consumption of cores, caches, andDRAM. Since we adopt a multi-programmed simulation en-vironment, we report power efficiency, rather than energyefficiency. Each application starts executing at its predeter-mined execution point. We simulate the mix until the slow-est application in the mix executes the desired number of in-structions. Statistics per application are gathered only untileach core (application) reaches the fixed number of instruc-tions. However, we keep executing the faster applicationsto correctly simulate the contention for shared resources.When IPCs are compared, the IPCs for the same number ofinstructions across different configurations are used. Con-sequently, the same application in a different configurationcan complete a different amount of work. For this reason, itis difficult to make fair energy comparisons across configu-rations; we compare power efficiency instead.

We also use minimum speedup (i.e. maximum slow-down) to determine the fairness of the system [9]. The min-imum speedup of a workload is defined by Equation 2

MinimumSpeedup =N−1mini=0

IPCsharedi

IPCalonei

(2)

The minimum speedup of a workload helps to identifywhether a technique improves overall system throughputby improving the performance of some applications signif-icantly while degrading the rest. The harmonic mean of

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

DR

AM

Ro

w B

uff

er H

it R

ate

shared bpart sr bpart+sr

Figure 6: Average DRAM row-buffer hit rate ofeach benchmark.

speedups is also widely used as a system fairness metric; wefavor the minimum speedup, however, to highlight the dis-advantage that a program can suffer due to each approach.

5 Results5.1 Row-buffer Hit Rate

We first present the effect of bank partitioning on row-buffer hit rate. Figure 6 shows the row-buffer hit rate ofeach benchmark with varying configurations. We gatherthe per-core (and thus per-benchmark) row-buffer hit rateand average the row-buffer hit rate for the same benchmarkacross different workloads. For most benchmarks, the row-buffer hit rate improves with bank partitioning. The mostimproved benchmark is libquantum, which recovers halfof the locality lost due to inter-thread interference. On theother hand, the row-buffer locality of lbm is degraded. Thismay be surprising considering that lbm has a very high row-buffer hit rate (98%) when run alone. As seen in Figure5, the spatial locality of lbm is sensitive to the number ofbanks. With a reduced number of banks due to bank parti-tioning, lbm cannot keep enough concurrent access streamsopen and loses row-buffer locality.

With sub-ranking, the row-buffer hit rate drops since thegranularity of sub-rank interleaving is 64B and the numberof independent, half-sized banks that a program accesses in-creases. For example, two consecutive 64B accesses wouldnormally result in 50% hit rate, but result in a 0% hit rateonce sub-ranking is introduced. Note that this is a start-upcost and is amortized over the subsequent accesses if theapplication has sufficient spatial locality. However, whensub-ranking is applied on top of bank partitioning, the row-buffer hit rate drops slightly due to this effect.

5.2 System ThroughputFigure 7 shows the system throughput for each mem-

ory configuration and workload. As expected, the workloadgroup with many high spatial locality benchmarks (HIGH)benefits most from bank partitioning. These benchmarksrecover the locality lost due to interference and their in-creased row-buffer hit rate results in improved throughput.Since many of the benchmarks in group HIGH have highspatial locality, they do not gain much from the additional

7

0.9$

0.95$

1$

1.05$

1.1$

1.15$

1.2$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$ Avg$

HIGH$ MIX$ LOW$ LOW_BW$

Normalized

+Weighted+Speedu

p+

shared$

bpart$

sr$

bpart+sr$

Figure 7: Normalized throughput of workloads.

banks that sub-ranking provides. Sub-ranking alone doesnot affect (or slightly improves) performance for the HIGHbenchmarks, but it degrades the improvements of bank par-titioning when both are applied. With shared banks, inter-thread interference results in many long-latency row-buffermisses. Additional latencies due to sub-ranking are in-significant compared to row-buffer miss latencies (4 cyclesvs. 37 cycles). With bank partitioning, however, many re-quests are now row-buffer hits and take little time (11 cy-cles). Therefore, the extra latency from sub-ranking be-comes relatively more costly and affects performance.

Workload group MIX enjoys less benefit from bank par-titioning since it includes benchmarks with low spatial lo-cality. MIX is also moderately improved by sub-ranking.When both techniques are applied together, however, mostMIX workloads show synergistic improvement. Exceptionsto this observation include workloads M1, which has onlyone application with low spatial locality to benefit from sub-ranking, and M4, whose performance with sub-ranking isslightly degraded due to the presence of lbm (for reasonsdescribed above). M2 suffers from both afflictions, and itsperformance relative to bank partitioning degrades the mostfrom the addition of sub-ranking, though it is still within3% of the maximum throughput.

Workload group LOW shows interesting results. Intu-itively, workloads composed of only low spatial localitybenchmarks would suffer performance degradation withbank partitioning due to reduced bank-level parallelism.However, benchmarks L1 and L3 show a healthy 5%throughput improvement from bank partitioning. This isbecause bank partitioning load-balances among banks. Asmost of the requests are row-buffer misses occupying a bankfor a long time waiting for additional precharge and acti-vate commands, the banks are highly utilized. Therefore,all the bank-level parallelism the DRAM system can offeris already in use and bank load-balancing becomes criticalfor the DRAM system throughput. Figure 8 shows the av-erage queueing delay of requests at the memory controllerscheduling buffer. When banks are partitioned among lowspatial locality threads, the requests are evenly distributedamong banks, lowering the average queueing delay.

Workloads in group LOW BW contain non-memory-intensive benchmarks, thus the collective impact on

0

5

10

15

20

25

30

L1 L2 L3

LOW A

vera

ge q

ue

uei

ng

late

ncy

in

DR

AM

cyc

les

shared

bpart

sr

bpart+sr

Figure 8: Average queueing delay of requests forworkload group LOW

throughput is smaller but has a similar trend. WorkloadLB4 is an outlier that shows a 7.6% throughput drop frombank partitioning. It consists of lbm, mcf, bzip2, andsjeng, where lbm and mcf are ill-affected by bank par-titioning as shown in Figure 6; bzip2 and sjeng arenon-memory-intensive and, thus, remain unaffected. Al-together, no improvement results from bank partitioningfor this workload. However, memory sub-ranking recov-ers some of the lost bank-level parallelism and performance.Bank partitioning combined with sub-ranking gives averagethroughput improvements of 9.8%, 7.4%, 4.2%, and 2.4%for HIGH, MIX, LOW, and LOW BW, respectively. The max-imum throughput improvement for each group is 12.2%,11.4%, 4.5%, and 5%.

Workload throughput alone only shows half of the story.Figure 9 shows the minimum speedup of workloads. Al-though bank partitioning alone could provide almost all ofthe system throughput improvements, it does so at the ex-pense of benchmarks with low spatial locality. Workloadsin group MIX and LB4 and LB5 in group LOW BW show asignificant reduction in minimum speedup when bank parti-tioning is employed. In all such cases, the benchmark whichsuffers the lowest speedup is either mcf or omnetpp;both benchmarks have low spatial locality. By partitioningbanks, these programs suffer from reduced bank-level par-allelism. When sub-ranking is employed along with bankpartitioning, the minimum speedup recovers from this dropin addition to providing system throughput improvements.Bank partitioning and memory sub-ranking optimize con-

8

0.2$0.25$0.3$0.35$0.4$0.45$0.5$0.55$0.6$0.65$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$ Avg$

HIGH$ MIX$ LOW$ LOW_BW$

Minim

um&&Spe

edup

&

shared$

bpart$

sr$

bpart+sr$

Figure 9: Minimum speedup of workloads.

0"

2"

4"

6"

8"

10"

12"

14"

16"

H1" H2" H3" H4" H5" Avg" M1" M2" M3" M4" M5" M6" M7" Avg" L1" L2" L3" Avg" LB1" LB2" LB3" LB4" LB5" Avg"

HIGH" MIX" LOW" LOW_BW"

DRAM

%Pow

er%(W

)%

shared"

bpart"

sr"

bpart+sr"

Figure 10: DRAM power consumption.

flicting requirements, locality and parallelism, and employthem together results in a more robust and fairer solution.

5.3 Power and System Efficiency

As described in Section 4, we estimate the system powerby modeling the DRAM power and the power consumed bythe processor. Figure 10 shows the DRAM power compo-nent. Although the DRAM power reduction due to bankpartitioning only appears to be modest, the actual energyreduction is significantly greater since bank partitioninggreatly reduces activations by improving the row-buffer hitrates, and thus performance, as shown in Figures 6 and 7.Sub-ranking reduces DRAM power consumption by onlyactivating a fraction of the rows. Naturally, bank partition-ing and sub-ranking, combined, brings additive benefits.With both schemes together, the DRAM power consump-tion is reduced dramatically, averaging at a 21.4% reductionacross all benchmark mixes.

System efficiency is measured by throughput (WS)per system power (Watts), where system power includesDRAM and CPU power. Since the dynamic power con-sumption of the CPU is modeled as a function of IPC (Sec-tion 4), the CPU power slightly increases with the improvedIPC. However, the combined system power (DRAM+CPU)remains fairly constant across benchmark mixes.

Figure 11(a) shows the normalized throughput per sys-tem power. In group HIGH, most of the benefit comes frombank partitioning alone. As mentioned earlier, this grouphas mixes of benchmarks that all have high row-buffer hitrates and does not benefit much by sub-ranking. On aver-

age, the improvement in system efficiency is 10%. In groupMIX, the overall system efficiency is best enhanced by thesynergy of bank partitioning and sub-ranking. This is theresult of the throughput improvement from bank partition-ing combined with the dramatic power savings from sub-ranking. Efficiency improves by 9% on average, whereasbank partitioning and sub-ranking alone improve efficiencyonly 6% and 4%, respectively. In group LOW BW, althoughthe improvement is relatively lower, using bank partition-ing and sub-ranking together still brings the best benefit.Workload LB4 again shows degraded power efficiency withbank partitioning because of its impact on throughput forthis workload.

Current processor and memory scaling trends indicatethat DRAM power is a significant factor in system power,especially in servers with huge DRAM capacities [6]. Suchsystems will see greater improvement in system efficiencywith combined bank partitioning and sub-ranking. Figure11(b) verifies this trend with the normalized throughput perDRAM power. Only taking DRAM power into account,bank partitioning with sub-ranking shows improvements ofup to 45% over the baseline.

5.4 Bank limited systems

Processor and memory scaling trends also indicate thatCMP core count increases faster than available memorybanks in the system, especially for embedded or many-coresystems. To approximate a system with limited memorybanks per core, we conduct experiments on a system witha single rank per channel instead of the two ranks previ-

9

0.8$

0.85$

0.9$

0.95$

1$

1.05$

1.1$

1.15$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$ Avg$

HIGH$ MIX$ LOW$ LOW_BW$

Normalized

+Throu

ghpu

t+/++

System

+Pow

er+

shared$

bpart$

sr$

bpart+sr$

(a) Normalized throughput per system power.

0.8$

0.9$

1$

1.1$

1.2$

1.3$

1.4$

1.5$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$Avg$

HIGH$ MIX$ LOW$ LOW_BW$

!Normalized

!Throu

ghpu

t!/!

DRAM

!Pow

er!

shared$

bpart$

sr$

bpart+sr$

(b) Normalized throughput per DRAM power.

Figure 11: System efficiency of two-rank systems.

ously shown. Figure 12(a) illustrates the system efficiencyfor this bank-limited machine. The results show that thebank-limited system enjoys greater benefit from bank par-titioning with sub-ranking. The average improvements insystem efficiency on the bank-limited system are 18%, 19%,25%, and 7% for groups HIGH, MIX, LOW and LOW BW, re-spectively. Figure 12(b) shows the system efficiency perDRAM power for a single rank system; combining bankpartitioning and sub-ranking again provides significant im-provements.

6 Related Work

6.1 DRAM bank partitioning

DRAM bank partitioning is first proposed by Mi et al.[13], who use page coloring and XOR cache mapping to re-duce inter-thread interference in chip multiprocessors. Theydevelop a cost model and search the entire color assignmentspace for each workload off-line. While this approach canapproximate a near-optimal color assignment for a specificmemory system and workload mix, the cost of this searchis prohibitive for dynamically changing workload mixes.Also, Mi et al. do not consider the impact of reduced bank-level parallelism; rather, they assume a very large numberof banks, exceeding the practical limit of modern DRAMsystems. The number of banks can be increased by addingmore ranks to a system, but rank-to-rank switching penaltiescan limit performance as we show in Figure 5. We proposea sub-ranking-based alternative which is both cost-effective

as well as power efficient, and provide in-depth analysis ofthe interaction between locality and parallelism.

Another existing approach by Muralidhara et al. parti-tions memory channels instead of banks [15]. Their par-titioning method is based on the runtime profiling of ap-plication memory access patterns combined with a mem-ory scheduling policy to improve system throughput. How-ever, they do not account for or analyze the impact of re-duced bank-level parallelism. Also, their method interfereswith fine-grained DRAM channel interleaving and limitsthe peak memory bandwidth of individual applications. Tothe best of our knowledge, our work is the first to preserveboth spatial locality and bank-level parallelism.

6.2 Memory scheduling policiesA number of memory scheduling policies have recently

been proposed that aim at improving fairness and systemthroughput in shared-memory CMP systems.

Computer-network-based fair queuing algorithms usingvirtual start and finish times are proposed in [18] and [19],respectively, to provide QoS to each thread. STFM [16]makes scheduling decisions based on the stall time ofeach thread to achieve fairness. A parallelism-aware batchscheduling method is introduced in [17] to achieve a bal-ance between fairness and throughput. ATLAS [9] pri-oritizes threads that attained the least service from mem-ory controllers to maximize system throughput at the costof fairness. TCM [10] divides threads into two separateclusters and employs a different memory request schedul-ing policies to each cluster. The authors claim that both

10

0.8$

0.9$

1$

1.1$

1.2$

1.3$

1.4$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$ Avg$

HIGH$ MIX$ LOW$ LOW_BW$

Normalized

+Throu

ghpu

t+/++

System

+Pow

er+

shared$

bpart$

sr$

bpart+sr$

(a) Normalized throughput per system power

0.8$0.9$1$

1.1$1.2$1.3$1.4$1.5$1.6$1.7$1.8$

H1$ H2$ H3$ H4$ H5$ Avg$ M1$ M2$ M3$ M4$ M5$ M6$ M7$ Avg$ L1$ L2$ L3$ Avg$ LB1$ LB2$ LB3$ LB4$ LB5$ Avg$

HIGH$ MIX$ LOW$ LOW_BW$

!Normalized

!Throu

ghpu

t!!/!

DRAM

!Pow

er!

shared$

bpart$

sr$

bpart+sr$

(b) Normalized throughput per DRAM Power

Figure 12: System efficiency of one-rank systems.

throughput and fairness are enhanced with the proposedscheme.

These previous works address the same problem of inter-thread interference in shared memory, but focus on improv-ing the scheduling of shared resources to threads consid-ering their memory access behavior and system fairness.Our approach is orthogonal to the existing scheduling pol-icy work, as it focuses on improving efficiency by preserv-ing row-buffer locality, and on balancing parallelism withaccess latency. Bank partitioning can benefit from betterscheduling, and better scheduling policies can benefit fromimproved row-buffer spatial locality.

6.3 Sub-rankingRecently, several schemes to reduce the active power of

DRAM have been proposed. Sub-ranking is a techniquethat breaks the conventional wide (64-bit) rank into nar-rower (8-bit, 16-bit or 32-bit) sub-ranks. As each accessinvolves fewer DRAM chips, it reduces the power consump-tion significantly.

Ware and Hampel [24] propose the Threaded MemoryModule, in which the memory controller itself controls in-dividual sub-ranks with chip-enable signals. They analyt-ically present the potential performance improvement dueto the effective increase in the number of memory banks.Zheng et al. [27] propose Mini-Rank memory. They keepthe memory controller module interface the same, and ac-cess sub-ranks using a module-side controller. They fo-cused on savings in activate power from sub-ranking. Ahn

et al. [3, 2] propose MC-DIMM. Their work is similar to theThreaded Memory Module, except demultiplexing and therepetitive issue of sub-rank commands are handled on themodule side. Also, Ahn et al. empirically evaluate MC-DIMM on multi-threaded applications, with full-systempower evaluation. Their target is a heavily threaded sys-tem which can naturally tolerate the additional latency fromnarrow sub-ranks. Unlike the previous works, we focus onthe additional bank-level parallelism that sub-ranking canprovide in the context of latency sensitive, general purposeout-of-order processor systems.

7 ConclusionIn this paper, we present a mechanism that can funda-

mentally solve the problem of interleaved memory accessstreams at its root. Our technique balances the conflictingneeds of locality and parallelism. We rely on bank parti-tioning using OS-controlled coloring to assign a set of in-dependent banks to each potentially conflicting thread. Tocompensate for the reduced bank parallelism available toeach thread we use DRAM sub-ranking to effectively in-crease the number of banks in the system without increasingits cost. This combined bank partitioning and sub-rankingtechnique enables us to significantly boost performance andefficiency simultaneously, while maintaining fairness. Ourevaluation demonstrates the potential of this technique forimproving locality, which leads to performance and effi-ciency gains. Moreover, we show that our technique is evenmore important when considering system configurations

11

that inherently have a small number of banks per thread. Weexpect future systems to have such bank-constrained con-figurations because of the relatively high cost of increasingmemory banks compared to the cost of increasing the num-ber of concurrent threads in the CPU.

Bank partitioning works best when all applications ina multi-programmed workload have high spatial locality.In such mixes, partitioning eliminates interference and im-proves performance and power efficiency by 10 − 15%.When applications with low spatial locality, which requirea larger number of banks to achieve best performance, areadded to the application mix then combining partitioningwith sub-ranking yields superior results. The combina-tion yields execution throughput improvements (weightedspeedup) of 7% and 5% on average for mixed workloadsand workloads consisting of only low spatial locality appli-cations respectively. Our combined technique even slightlyimproves the performance and efficiency of applicationsthat rarely access memory. The importance of combiningpartitioning with sub-ranking is more apparent when fair-ness and power efficiency are considered. The minimumspeedup (a fairness metric) improves by 15% with the com-bined technique. In addition, our experiments show thatsystem power efficiency improves slightly more than themeasured improvement in system throughput.

The previously given power efficiency improvements areshown for systems that are dominated by CPU power, inwhich DRAM power is only 20% or less. When lookingjust at DRAM power, power efficiency improvements are20−45% for the combined technique as opposed to roughly10−30% when the techniques are applied separately. For abank-constrained configuration, improvements are roughly50% better, relative to the whole-system measurements.

8 AcknowledgementsThis work is supported, in part, by the following orga-

nizations: The National Science Foundation under Grant#0954107, Intel Labs Academic Research Office for theMemory Hierarchy Innovations program, and The TexasAdvanced Computing Center.

References[1] Calculating memory system power for DDR3. Technical Report TN-

41-01, Micron Technology, 2007.[2] J. H. Ahn, N. P. Jouppi, C. Kozyrakis, J. Leverich, and R. S.

Schreiber. Future scaling of processor-memory interfaces. In Proc.the Int’l Conf. High Performance Computing, Networking, Storage,and Analysis (SC), Nov. 2009.

[3] J. H. Ahn, J. Leverich, R. Schreiber, and N. P. Jouppi. Multi-core DIMM: An energy efficient memory module with independentlycontrolled DRAMs. IEEE Computer Architecture Letters, 8(1):5–8,Jan. - Jun. 2009.

[4] T. M. Brewer. Instruction set innovations for the Convey HC-1 com-puter. IEEE Micro, 30(2):70–79, 2010.

[5] G. Hamerly, E. Perelman, J. Lau, and B. Calder. SimPoint 3.0: Fasterand more flexible program analysis. In Proc. the Workshop on Mod-eling, Benchmarking and Simulation (MoBS), Jun. 2005.

[6] U. Hoelzle and L. A. Barroso. The Datacenter as a Computer: AnIntroduction to the Design of Warehouse-Scale Machines. Morganand Claypool Publishers, 1st edition, 2009.

[7] M. K. Jeong, D. H. Yoon, and M. Erez. DrSim: A platform forflexible DRAM system research. http://lph.ece.utexas.edu/public/DrSim.

[8] T. Karkhanis and J. E. Smith. A day in the life of a data cache miss.In Workshop on Memory Performance Issues, 2002.

[9] Y. Kim, D. Han, O. Mutlu, and M. Harchol-Balter. ATLAS: A scal-able and high-performance scheduling algorithm for multiple mem-ory controllers. In Proc. the 16th Int’l Symp. High-PerformanceComputer Architecture (HPCA), Jan 2010.

[10] Y. Kim, M. Papamichael, O. Mutlu, and M. Harchol-Balter. Threadcluster memory scheduling: Exploiting differences in memory accessbehavior. In Proc. the 43rd Ann. IEEE/ACM Int’l Symp. Microarchi-tecture (MICRO), Dec. 2010.

[11] G. H. Loh, S. Subramaniam, and Y. Xie. Zesto: A cycle-level simu-lator for highly detailed microarchitecture exploration. In Proc. theInt’l Symp. Performance Analysis of Software and Systems (ISPASS),Apr. 2009.

[12] M. McTague and H. David. Fully buffered DIMM (FB-DIMM) de-sign considerations. Intel Developer Forum (IDF), Feb. 2004.

[13] W. Mi, X. Feng, J. Xue, and Y. Jia. Software-hardware cooperativeDRAM bank partitioning for chip multiprocessors. In Proc. the 2010IFIP Int’l Conf. Network and Parallel Computing (NPC), Sep. 2010.

[14] Micron Corp. Micron 2 Gb ×4, ×8, ×16, DDR3 SDRAM:MT41J512M4, MT41J256M4, and MT41J128M16, 2011.

[15] S. Muralidhara, L. Subramanian, O. Mutlu, M. Kandemir, andT. Moscibroda. Reducing memory interference in multicore systemsvia application-aware memory channel partitioning. In Proc. the 44thAnn. IEEE/ACM Int’l Symp. Microarchitecture (MICRO), Dec. 2011.

[16] O. Mutlu and T. Moscibroda. Stall-time fair memory access schedul-ing for chip multiprocessors. In Proc. the 40th IEEE/ACM Int’l Symp.Microarchitecture (MICRO), Dec. 2007.

[17] O. Mutlu and T. Moscibroda. Parallelism-aware batch scheduling:Enhancing both performance and fairness of shared DRAM systems.In Proc. the 35h Ann. Int’l Symp. Computer Architecture (ISCA), Jun.2008.

[18] K. Nesbit, N. Aggarwal, J. Laudon, and J. Smith. Fair queuing mem-ory systems. In Proc. the 39th Ann. IEEE/ACM Int’l Symp. Microar-chitecture (MICRO), Dec. 2006.

[19] N. Rafique, W. Lim, and M. Thottethodi. Effective management ofDRAM bandwidth in multicore processors. In Proc. the 16th Int’lConf. on Parallel Architecture and Compilation Techniques (PAC),Sep. 2007.

[20] S. Rixner, W. J. Dally, U. J. Kapasi, P. R. Mattson, and J. D. Owens.Memory access scheduling. In Proc. the 27th Ann. Int’l Symp. Com-puter Architecture (ISCA), Jun. 2000.

[21] A. Snavely and D. M. Tullsen. Symbiotic job scheduling for a si-multaneous multithreaded processor. In Proc. the 9th Int’l Conf.Architectural Support for Programming Languages and OperatingSystems (ASPLOS), Nov. 2000.

[22] Standard Performance Evaluation Corporation. SPEC CPU 2006.http://www.spec.org/cpu2006/, 2006.

[23] A. N. Udipi, N. Muralimanohar, N. Chatterjee, R. Balasubramonian,A. Davis, and N. P. Jouppi. Rethinking DRAM design and organiza-tion for energy-constrained multi-cores. In Proc. the 37th Ann. Int’lSymp. Computer Architecture (ISCA), Jun. 2010.

[24] F. A. Ware and C. Hampel. Improving power and data efficiency withthreaded memory modules. In Proc. Int’l Conf. Computer Design(ICCD), Oct. 2006.

[25] D. H. Yoon, M. K. Jeong, and M. Erez. Adaptive granularity memorysystems: A tradeoff between storage efficiency and throughput. InProc. the 38th Ann. Int’l Symp. Computer Architecture (ISCA), Jun.2011.

[26] Z. Zhang, Z. Zhu, and X. Zhang. A permutation-based page inter-leaving scheme to reduce row-buffer conflicts and exploit data local-ity. In Proc. the 33rd Ann. IEEE/ACM Int’l Symp. Microarchitecture(MICRO), Dec. 2000.

[27] H. Zheng, J. Lin, Z. Zhang, E. Gorbatov, H. David, and Z. Zhu. Mini-rank: Adaptive DRAM architecture for improving memory power ef-ficiency. In Proc. the 41st IEEE/ACM Int’l Symp. Microarchitecture(MICRO), Nov. 2008.

12