Strider: Architectures for Scalable Memory Centric ... · Strider: Architectures for Scalable...

Strider:

Architectures for Scalable Memory

Centric Reduction of Sparse Data Streams

Sriseshan Srikanth, Tom Conte, Erik DeBenedictis

Reduction of Sparse Data Streams

Graph

Analytics

Cyber-security

HPC

ML

Abstraction: (key, value, ⊕)

Associative Array Reduction

Sriseshan Srikanth

Example of Sparse Reduction

SpGEMM

1 - 4 - -

- - - - -

2 3 - 1 -

- - 5 - -

- - 2 2 2

1 - 4 - -

- - - - -

2 3 - 1 -

- - 5 - -

- - 2 2 2

1 - 4 - -

- - - - -

2 - 8 - -

- - - - -

- - - - -

8 12 - 4 -

- - - - -

- - - - -

10 15 - 5 -

4 6 - 2 2

- - - - -

- - - - -

- - 5 - -

- - - - -

- - 10 - -

- - - - -

- - - - -

- - - - -

- - - - -

- - 4 4 4

Key = Matrix Index

Value = Partial Product

⊕ = Summation

Sriseshan Srikanth

Dense vs Sparse Applications

Application Typical LLC Miss Ratio

Potential Application

Speedup due to Optimized

DRAM Row Performance

SPEC 2006 < 10% < 10%

SpGEMM * > 50% 2x – 10x

0 10 20 30 40 50 60

DRAM - cache bypass - row hit

DRAM - row miss

DRAM - row empty

DRAM - row hit

L3

L2

L1

Latency (ns)

* SpGEMM: Key kernel in graph analytics, HPC

Optimized DRAM Row Performance results in order of

magnitude speedup for sparse applications

Sriseshan Srikanth

DRAM-Centric Design for Sparse Reductions

• Bypass caches

• Maximize DRAM row buffer utilization

• Increase algorithmic granularity of

operation to 1 DRAM row, to avoid wasted

reads and activations

– Conventional approach: activate 2048 byte

row, read 64 byte-worth cache block, perhaps

use 8 bytes out of that

Sriseshan Srikanth

DRAM-Aware Sparse Representation

• Two-level hierarchy

• Lower Level:

– (At most) K sorted (by key) non-zeros: node

– Each node has a logical pivot

• Higher Level:

– Pivots form a binary tree

– Elements in left subtree have keys less that pivot

• Operation granularity: node

Pivot

Key

L

Subtree

Addr

R

Subtree

Addr

(key,

value)

(key,

value)…

(key,

value)

Metadata K non-zeros

Sriseshan Srikanth

Addvec()

Vector Insertion with Local Reduction

pivot: 40row: 0

min: 0, max: 1482

New 0 39 40 78 1482

Row 0 0 39 40 78 1482

New 40 79 936 1131 1132

Row 0 0 39 40 78

Row 1 79 936 1131 1132 1482

New 40 79 936 1131 1132

0 39 40 40 78 79 936 1131 1132 1482

pivot: 40row: 0

min: 0, max: 78

node_1pivot: 1131

row: 1min: 79, max: 1482

Send K higher records to

right subtreeSriseshan Srikanth

pivot: 118row: 2

min: 79, max: 664

pivot: 40row: 0

min: 0, max: 78

pivot: 1131row: 1

min: 936, max: 1482

pivot: 118row: 2

min: 39, max: 119

pivot: 40row: 0

min: 0, max: 78

pivot: 1131row: 1

min: 936, max: 1482

pivot: 510row: 3

min: 120, max: 665node_6

New 40 79 936 1131 1132

Row 0 0 39 40 78

Row 1 79 936 1131 1132 1482

New 79 80 118 119 664

Row 0 0 39 40 78

Row 1 936 1131 1132 1482

Row 2 79 80 118 119 664

New 39 120 509 510 665

Row 0 0 39 40 78

Row 1 936 1131 1132 1482

Row 2 39 79 80 118 119

Row 3 120 509 510 664 665

Addvec()

Vector Insertion with Local Reduction

Sriseshan Srikanth

GlobalReduce()

• Addvec() recurses over exactly one of L/R

subtrees

– Necessary for performance as tree grows

– Disadvantage: duplicates may occur along

different paths of tree

• However, this is NOT the common case

• GlobalReduce(): Final dedup pass

– Upon DFS traversal, detect and fix violations:if (max of L subtree) > (min of R subtree), then

extract argmax row from L subtree and perform Addvec()

Sriseshan Srikanth

GlobalReduce() Overhead

• Testing for violation requires row ACT

– This is redundant in the common case

– Solution:

• Include min, max info in metadata

• Decouple metadata

99.7%

overhead

120.9%

overhead

0

200000

400000

600000

800000

1000000

1200000

2cu bel caC mar net p2p pat roa AVE as2 tl2 as3 tl3 as1 tl1 AVE

# D

RA

M R

ow

AC

T

GlobalReduce() Addvec()

Sriseshan Srikanth

Architecture v1 – Bitonic Merge

CPU L2

MetaStrider

Mem CtrlMemory

Controller

L

1

System Interconnect

(Crossbar)

NDP

MetaStrider Port

Near Data

Processing Unit

Merge

Compress

Unit

Control

State

Machine

MetaData

Store

Indexable

SRAM 1

(K records)

Indexable

SRAM 2

(K records)

Buffer

(C records)

Buffer

(C records)

Mux Mux

Buffer

(C records)

Buffer

(C records)

FU FU FU FU

FU FU FU FU

FU FU FU FU

FU FU FU FU

FU FU FU FU

FU FU FU FU

Sriseshan Srikanth

Architecture v2 – Linear-Merge

V1: Bitonic network can be used for efficient sorting. Merging in O(N lg N).

V2: If input vectors are pre-sorted, merging in O(N).

• However, observe that most of the input to bitonic network is from

existing nodes (due to Addvec() recursion)

• Each node is sorted

• Don’t need bitonic network, simple linear merge sufficient

• Linear-merge

interleaves logic

and memory at

fine granularity

(DRAM burst)• Linear speedup

with increased

burst width

(DDR vs HBM)

Sriseshan Srikanth

Strider Variants

• SuperStrider

– Bitonic merge network

– No decoupled metadata

– No tree balancing

• MetaStrider

– Linear merge network

– Decoupled metadata

– Tree balancing

– Hashing

– Can be implemented with or without dedicated

hardware at memory controller (NDP)

Sriseshan Srikanth

API

Sriseshan Srikanth

Evaluation

• Workloads

– (8) SpGEMM – Undirected/directed graphs, 2D/3D,

Electromagnetics, Road networks

– (9) Firehose – Cybersecurity benchmark

• Key: IPv6 address

• Value: bias bit, ground truth (for reference)

• Reduction: – Count incidence of key (+1 logic)

– If count exceeds static threshold, flag future incidence and test bias (compare logic)

• Requires support for incremental updates

• Gem5 - single HBM channel simulation

• In-house – rapid design space exploration

Sriseshan Srikanth

Space of Reducers

• Software

– Scalar balanced red-black binary tree

(std::map) – CPU only (baseline)

– Kokkos hashmap accumulator – CPU or GPU

• Hardware

– GraFBoost – hierarchical merge-sort – unable to

handle incremental updates (limited to SpGEMM)

– SuperStrider

– MetaStrider

Sriseshan Srikanth

DRAM Performance

0

10

20

30

40

50

60

Re

du

cti

on

in

Ro

w

Acti

va

tio

ns

Scalar Binary Tree Kokkos SuperStrider MetaStrider

0

2

4

6

8

2cu

be

l

ca

C

ma

r

net

p2

p

pa

t

roa

HM

pl1

pl2

pl3

as1

as2

as3

tl1

tl2

tl3

HM

w/o

pl

Red

ucti

on

in

DR

AM

Tra

ffic

MetaStrider improves DRAM performance and energy efficiency by

an order of magnitude

Sriseshan Srikanth

Uncore Energy

0

20

40

60

80

100

Scalar Binary

Tree

Kokkos SuperStrider Non-NDP

MetaStrider

MetaStrider

Ave

rage

Un

-co

re E

ne

rgy

(mJ)

L1-L2 Bus

Merger Unit

Metadata Store

LLC

System Interconnect

DRAM

Any energy overheads due to MetaStrider’s Merger and Metadata are

overshadowed by energy reductions due to efficient data management

Sriseshan Srikanth

System Performance

0

10

20

30

40

SpGEMM Firehose

Ke

rne

l S

pe

ed

up

(H

M a

cro

ss a

ll w

ork

loa

ds)

Scalar Binary Tree Kokkos SuperStrider MetaStrider

Order of magnitude performance improvement on the reduction kernel

renders application performance that is within 8% of an ideal

hypothetical accelerator that performs reduction in zero time

Sriseshan Srikanth

Batch Mode – SpGEMM only

0.80.9

11.11.21.31.41.51.6

Reduction in

Row Activations

Reduction in

DRAM Traffic

Kernel Speedup Reduction in Un-

core Energy

Pa

ram

ete

rs n

orm

aliz

ed to

Gra

fboost

(HM

acr

os

all

work

loads)

Grafboost MetaStrider (Batch) 14.9

Even when ability to handle incremental updates is discarded,

MetaStrider is relatively faster and more efficient

Sriseshan Srikanth

Data Movement on System Interconnect

1.543.24

34.82

75.19

0

20

40

60

80

100

120

2cu

bel

ca

C

ma

r

ne

t

p2

p

pa

t

roa

HM

pl1

pl2

pl3

as1

as2

as3

tl1

tl2

tl3

HM

w/o

pl

Red

ucti

on

in

Cro

ssb

ar Tra

ffic Non-NDP MetaStrider MetaStrider

MetaStrider reduces data movement even without extra hardware

With NDP, over an order of magnitude reduction is seen

Sriseshan Srikanth

Scaling with MLP, Parallel Front-end

Pipelining Partitioning

Fanout Grouping

Sriseshan Srikanth

Scaling with MLP, multi-core

• Takeaway:

Performance ↑

~linearly using our

strategies.

• Pipelining: 80% of

available MLP achievable using

simple heuristics.

• Grouping: Requires

high frequency front-end cores.

• Partitioning vs

Fanout: see chart on

right. Partitioning is better if parallel front-

end available.Energy (DRAM rows accessed in total)

Sriseshan Srikanth

Conclusion

• Sparse data applications widely used in graph

analytics, cybersecurity, HPC, ML.

– Low compute to communicate ratio, low locality of

reference.

• Traditional architectures NOT energy-efficient:

– Significant / redundant data-movement through

memory hierarchy.

• Solution: Scalable memory-centric architectures

– SuperStrider

– MetaStrider

• Scales with tighter logic-memory integration, MLP

Sriseshan Srikanth

Thank [email protected]

Sriseshan Srikanth

Typical Memory Latencies

• L1: 1ns

• L2: 1 + 3 = 4ns

• L3: 1 + 3 + 10 = 14ns

• DRAM (no cache bypass)

– 14 + 14 = 28ns (hit)

– 14 + 28 = 42ns (empty)

– 14 + 42 = 56ns (miss)

• Similar trends for energy (nJ); off-chip data

movement ~2 orders of magnitude higher than

FLOP.

Sriseshan Srikanth

Superstrider Architecture

https://www.amd.com/en/technologies/hbm

DRAM bank:

Ctl. information K Records

Control logic:Scalar arithmetic, includes stack and length

Accumulator

ALU: Sort/merge, split, add

Sriseshan Srikanth

Superstrider Data Organization

Addr:(size) Pivot L subtree R subtree Records (K = 5)

3:(2) 28 0(0) 0(0) [27]=0.20 [28]=1.56

1:(7) 20 0(0) 3(2) [20]=1.31 [21]=0.26 [22]=0.45 [25]=0.07 [26]=1.59

0:(22) 17 2(10) 1(7) [14]=0.09 [16]=0.77 [17]=2.75 [18]=1.46 [19]=0.47

2:(10) 13 4(5) 0(0) [8]=0.59 [10]=0.14 [11]=1.66 [12]=0.27 [13]=0.61

4:(5) 6 0(0) 0(0) [1]=0.52 [2]=0.02 [3]=0.80 [6]=0.28 [7]=1.61

DRAM row 0

13

-

2-

4

Sriseshan Srikanth

Superstrider: Key Principles

• Memory rows organized as a binary tree with K

records (sorted by key) and a pivot (key) per row.

• Memory access and computation granularity:

– 1 memory row of sorted records.

– SIMD-style operation tightly integrated with wide

memory words.

• Sorted invariant

– Two N/2 length pre-sorted vectors can be merged in

log2(N) stages.

– Novel algorithms: Fast insertion/lookup into binary

tree; fast compression (later).

Sriseshan Srikanth

Superstrider Operation


Accumulator [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45


0:(5) 17 0(0) 0(0) [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45

Accumulator [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45

Pre-sorted input [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45

Pre-sorted input [13]=0.50 [13]=0.04 [18]=0.63 [20]=0.61 [27]=0.20


0:(5) 17 0(0) 0(0) [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45

Accumulator [13]=0.50 [13]=0.04 [18]=0.63 [20]=0.61 [27]=0.20

Sriseshan Srikanth

Addvec: Merge and Compress

• Row 0 and Accumulator:

• Merge:

• Compress


0:(5) 17 0(0) 0(0) [2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45

Accumulator [13]=0.50 [13]=0.04 [18]=0.63 [20]=0.61 [27]=0.20

[2]=0.02 [14]=0.09 [17]=0.49 [20]=0.27 [22]=0.45 [13]=0.50 [13]=0.04 [18]=0.63 [20]=0.61 [27]=0.20

[2]=0.02 [13]=0.50 [13]=0.04 [14]=0.09 [17]=0.49 [18]=0.63 [20]=0.27 [20]=0.61 [22]=0.45 [27]=0.20

[2]=0.02 [13] = 0.50 + 0.04 [14]=0.09 [17]=0.49 [18]=0.63 [20] = 0.27 + 0.61 [22]=0.45 [27]=0.20

Sriseshan Srikanth

Addvec: Recurse

• Red: 3

• Green: 5

• Keep red in node, propogate (Addvec) green to

“>” child.

[2]=0.02 [13] = 0.54 [14]=0.09 [17]=0.49 [18]=0.63 [20] = 0.88 [22]=0.45 [27]=0.20


1:(5) 20 0(0) 0(0) [17]=0.49 [18]=0.63 [20] = 0.88 [22]=0.45 [27]=0.20

0:(8) 17 0(0) 1(5) [2]=0.02 [13] = 0.54 [14]=0.09 [20]=0.27 [22]=0.45

DRAM row 0

1

Sriseshan Srikanth

Need for Normalization

• Row 0 and accumulator:

• Merge and Compress

• #(Red) > #(Green) => Propogate (Addvec) red to “<“ child.

• Performance advantage: Addvec accesses L subtree OR R subtree,

but NOT both.


1:(5) 20 0(0) 0(0) [17]=0.49 [18]=0.63 [20] = 0.88 [22]=0.45 [27]=0.20

0:(8) 17 0(0) 1(5) [2]=0.02 [13] = 0.54 [14]=0.09 [20]=0.27 [22]=0.45

Pre-sorted input [7]=0.68 [13]=0.07 [18]=0.82 [26]=0.25 [28]=0.99

[2]=0.02 [13] = 0.54 [14]=0.09 [7]=0.68 [13]=0.07 [18]=0.82 [26]=0.25 [28]=0.99

[2]=0.02 [7]=0.68 [13]=0.61 [14]=0.09 [18]=0.82 [26]=0.25 [28]=0.99


1:(5) 20 0(0) 0(0) [17]=0.49 [18]=0.63 [20] = 0.88 [22]=0.45 [27]=0.20

0:(12) 17 2(4) 1(5) [18]=0.82 [26]=0.25 [28]=0.99 [20]=0.27 [22]=0.45

2:(4) 13 0(0) 0(0) [2]=0.02 [7]=0.68 [13]=0.61 [14]=0.09

Sriseshan Srikanth

Superstrider ArchitectureAddr:(size) Pivot L subtree R subtree Records (K = 5)

3:(2) 28 0(0) 0(0) [27]=0.20 [28]=1.56

1:(7) 20 0(0) 3(2) [20]=1.31 [21]=0.26 [22]=0.45 [25]=0.07 [26]=1.59

0:(22) 17 2(10) 1(7) [14]=0.09 [16]=0.77 [17]=2.75 [18]=1.46 [19]=0.47

2:(10) 13 4(5) 0(0) [8]=0.59 [10]=0.14 [11]=1.66 [12]=0.27 [13]=0.61

4:(5) 6 0(0) 0(0) [1]=0.52 [2]=0.02 (3)=0.80 (6)=0.28 (7)=1.61

Control information Open row buffer

DRAM row 0

1

3

-

2

-

4

Control logic

sets which row to access nextAccumulator

Sorted stream of input records, with possibly repeated keys

Merge networks

- Forward: Merges two

sorted vectors.

- Backward: Deletes empty

records after compression.

Function unit pool

Applies a collision

function to records with

identical keys

Sriseshan Srikanth

Superstrider Architecture

Sriseshan Srikanth

SuperStrider / MetaStrider

• Memory-centric architecture operates on

DRAM rows directly.

• Two-level, in-memory representation for

efficient insertion and associative

reduction.

pivot: 118row: 2

min: 79, max: 664

pivot: 40row: 0

min: 0, max: 78

pivot: 1131row: 1

min: 936, max: 1482

Row 0 0 39 40 78

Row 1 936 1131 1132 1482

Row 2 79 80 118 119 664

Example: 5 (key, value) pairs fit in each DRAM

row, forming a node in a balanced binary tree

Sriseshan Srikanth

Fundamental Operation

ReduceAndDedup() is called recursively along one path of the tree until all

input records are inserted / reduced.

Sriseshan Srikanth

Realtime Capability

• Experiments performed at regular intervals

of Addvec():

– Exact match: 96.5%

– Partial match: 2%

• Omitted if GlobalReduce() called

– No match: 1.5%

• Omitted if no tree balance done

• Omitted if GlobalReduce() called

Sriseshan Srikanth

Benefits of Decouple, Balance

Sriseshan Srikanth

Partition vs Fanout

Sriseshan Srikanth

Scalability Analysis

Sriseshan Srikanth

Strider: Architectures for Scalable Memory Centric ... · Strider: Architectures for Scalable...

Documents

Transcript of Strider: Architectures for Scalable Memory Centric ... · Strider: Architectures for Scalable...