Peta-Cache, Mar30 , 2006

V1 1

Peta-Cache:Peta-Cache:

Electronics Discussion II

Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov

Gunther Hallerhaller@slac.stanford.edu(650) 926-4257

Peta-Cache, Mar30 , 2006

V1 2

Flash Storage OptionFlash Storage Option• Skim Builder as in option discussed earlier• Event server (cache box) as in other option• Shown as two boxes for simplicity, could be in one

box (there are pro’s and con’s)• Issue is again interconnect speed• Up to 16 1-Terra-Byte Flash boxes for each event

server– Each lane PCI-E 256 MByte/sec– 16-lanes gives total of 4-Gbytes/sec

bandwidth • Each Flash box has only fraction of total event

store• Flash has limited write-cycles so can’t frequently

rewritten (need to enforce with some policy which is most important)

• But don’t really want to “burn” results of skim in flash, since goal is to make own lists (and flash can’t be reburned at will anyways)

• Flexibility:– One event server can have a sub-set of the

list and events go to client– Or, better, have total of event server as “one”

cache” and event store is managed so that parts of the list which are in other pizza boxes are kept in that cache as opposed to discarded

• Question is again how to populate the Flash most effectively

• Decompression in event server• Flash bad-block management in event server• Reed-Solomon EDAC in event server• Can consider without cache box: 4000 clients

going after the same block, the last one to get data is ~ 300 msec later.

Flash Storage

Event Server

Ethernet/PCI-E/etc

Client (1, 2, or 4 core)

Skim builder (s)

Client (1, 2, or 4 core)

Up to 1,500 cores in ~ 800 units?

Disk Storage

Event Server

Tape

Client (1, 2, or 4 core)

Optionally direct IO

Disk Storage

PCI-E

116

Flash Storage1

16

Peta-Cache, Mar30 , 2006

V1 3

Flash-Box, Event BoxFlash-Box, Event Box

• Flash Memory Box• 8-, 16-, or 32-Gbit NAND devices • For 1 Terra-Byte need 250 each 32-Gbit devices

– All on board, or– 32 G-Byte memory cards (DIMM)

» Need > 30 DIMM’s

• Preliminary placement on 19-inch rack PCB shows that we can fit 1 Terra-byte on single board

– PCI-E to PCI-X bridge (to get 64-bit addressing space )– No smarts in here

• Event (Pizza) Box – 8 F40 Xilinx (each has 2 450-MHz PPC’s)– 16 GBytes of RLDRAM2– 8 PLX8508 PCI-E switch 5-ports– 2 PLX8532 8-port switch (32 lanes)

Peta-Cache, Mar30 , 2006

V1 4

Flash & Event Server BoxesFlash & Event Server Boxes

• 4Gbit chips: $30, 8Gbit = $60• 4 Gbyte device quote: $110 min qty 1000 (is 4-1GB die stack)• 1 Peta-Byte: 1,000 boxes total $27 Mill

Bridge Chips (total of 16) $500

Misc (Box, board, loading, regulators, etc)

$400

1 TByte Flash (250 x (4-GByte ~ $110)) $27,000

Xilinx’s (8 each) $500

Local RLDRAM2 (16Gbytes) $3,200

Misc (Box, board, loading, regulators, etc)

$400

Misc Switches $500

• Flash Box

• Event Server

Peta-Cache, Mar30 , 2006

V1 5

Pizza box block diagram (needs some Pizza box block diagram (needs some modification)modification)

(Out) PCI Express

x4

PPC 405

XILNIX XC4VFX40

RLDRAM II IGbyte

PLX8508

PLX8532

x16

(In) PCI Express

x16

x4

PLX8532

PLX8508

Peta-Cache, Mar30 , 2006

V1 6

Event Processing CenterEvent Processing Center

switch

file system fabric

sea of cores fabric

switch(s)

switch (s)

HPSS

pizza box

out protocol conversion

Event processing node

disks pizza box as skim builder

in protocol conversion