Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main...

24
Facebook OCP Server Hardware

Transcript of Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main...

Page 1: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Facebook OCP Server Hardware

Page 2: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Facebook Server Hardware

• Facebook uses four main building blocks to create its Infrastructure HW– Compute

– Storage

– Network

– Rack & Power

• Varying the ratios of compute to storage, and the internal configuration of the servers provides Facebook with all its SKUs.

Page 3: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Standard

Systems

I

Web

III

Database

IV

Hadoop

V

Haystack

VI

Feed

VII Cold

Storage

CPU

High

2 x Haswell

(12c)

2.5 GHz

High

2 x Haswell

(12c)

2.5 GHz

High

2 x Haswell

(12c)

2.5 GHz

Low

1 x Avoton

(8c)

2.7 GHz

High

2 x Haswell

(12c)

2.5GHz

High

2 x Haswell

(12c)

2.5 GHz

MemoryLow

32GB

High

256GB

Medium

128GB

Low

32GB

High

256GB

Medium

128GB

DiskLow

500GB

SATA

High IOP

2 x 3.2TB

Flash

128 GB

mSata

High

15 x 4TB (or

6TB)

NL SAS

High

30 x 4TB (or

6 TB)

NL SAS

Medium

2TB SAS

(1.8 TB

FLASH

optional)

Very High

240 X 4TB

16 active

disks

Services Web DatabaseData

Warehouse

(Hadoop)

Object Store

(Photos)

Cache

(Newsfeed)Archival

Page 4: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type I

• Front end/web service machines.

• Compute servers only

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Switch

Empty

Compute Empty Compute

Compute Empty Compute

Page 5: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type III

• High performance database applications

• Compute nodes w/ PCI-E Flash Cards

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Switch

Empty

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Page 6: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type IV

• Data warehouse storage and analytics (for ex. Hadoop)

• Powerful Compute + Mass Storage

Storage

Power Shelf

Storage

Storage

Storage

Storage

Storage

Compute Compute Compute

Storage

Power Shelf

Compute Compute Compute

Storage

Compute Compute Compute

Storage

Compute Compute Compute

Power Shelf

Compute Compute

Compute Compute Compute

Switch

Empty

Compute

Page 7: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type V

• Object storage (for ex. Photo and Video storage)

• Cost Efficient Compute + Mass Storage

uServer + Storage

Power Shelf

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

Power Shelf

uServer + Storage

Power Shelf

uServer + Storage

Switch

Empty

uServer + Storage

uServer + Storage

Page 8: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type VI

• Cache heavy applications (for ex. News Feed)

• Compute w/ large memory

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Switch

Empty

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Page 9: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Type VII

• Archival back up storage

• Single compute node connected to 240HDDs.

Storage

Storage

Storage

Storage

Storage

Storage

Empty Compute Empty

Storage

Storage

Storage

Storage

Empty Compute Empty

Storage

Storage

Storage

Power Shelf

Switch

Empty

Storage

Storage

Storage

Page 10: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Rack

• Wide equipment bay (21”) for maximum space efficiency

– Same external width (24”) as standard 19” rack.

• Shared power supply architecture for better resiliency

• Design optimized for serviceability and efficiency

Page 11: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Rack

• 1 OpenU = 48mm height

• Power shelf feeds 12V busbar that connects to servers

– Standardized mechanical interface between servers and rack

Page 12: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Compute Server• Dual Socket Xeon E5-2600 V3/V4

(Haswell/Broadwell)

• 16 DIMMs (2 per channel/4 channel per socket/2400MHz)

• Motherboard dimensions 6.5”x20”

• 3 PCI Gen 3 add in slots (x24)

• OCP Mezzanine Card (x8 PCI Gen 3)

• M.2/mSATA/HDD for boot

• ORV1/ORV2 compatible – 2OU 1/3 width

• BMC

• Manufactured by Quanta and WiWynn

Page 13: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Yosemite Compute Platform

• Platform for 4 single socket compute nodes

• ORV1/ORV2 compatible – 2OU 1/3 width

• Multi-host (shared) 40G/50G OCP Mezzanine NIC

• Shared BMC

Page 14: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Mono Lake

• Yosemite compute node card

• Xeon D-1500 SoC

• 4 DIMMs (2 per channel/2 channel/2400MHz/up to 128GB)

• 2 M.2 (NVMe/SATA)

• 210mm x 110mm

• Manufactured by Quanta

Page 15: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Big Sur GPU Server• Full Size Dual Socket Xeon E5-2600

V3/V4 (Haswell/Broadwell) Motherboard

• 8 x16 3 PCI Gen 3 GPU card slots– NVIDIA Tesla, AMD FirePro and Intel Phi

• OCP Mezzanine Card (x8 PCI Gen 3)– Ethernet and Infiniband

• 8x 2.5” SATA HDD/SSD

• 4OU ORV1/ORV2 compatible

• BMC

• Manufactured by Quanta

Page 16: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Vault Storage

• JBOD storage attach to Open Compute Server

• 2OU ORV1/ORV2 compatible • 30 HDDs (2 trays x 15 HDDs)• LSI SAS2x28 6Gb expander

per tray• 1 Ext. MiniSAS + 2 Int.

MiniSAS front facing ports per tray

• Manufactured by WiWynn

Page 17: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Microserver Storage

• Light weight server inside Open Vault• Same storage capacity as Open Vault• Microserver

– Intel Avoton C2750– DDR3 SO-DIMM (2 per channel/2

channel/1600 MHz/up to 32GB)– M.2 (NVMe/SATA) and mSATA

• Storage SubSystem– 12G SAS IOC + Expander (SAS 3008

+ SAS3x24)– BMC– OCP Mezzanine NIC

• Manufactured by WiWynn

Page 18: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Lightning (NVMe in Open Vault)

• JBOF storage attach to Open Compute Server over PCIe

• 2OU ORV1/ORV2 compatible

• Up to 60 SSDs (2 trays of 15 x4 PCI Gen 3 slots)

• 4 x4 PCI Gen 3 external upstream ports per tray

• OpenBMC

• Manufactured by WiWynn

Page 19: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Wedge 40

• Broadcom Trident II

• 16 x 40Gb QSFP ports (can be split to 4x10Gb)

• FBOSS

• OpenBMC

• Microserver card used for control plane

• Compatible with 19” and OR

• Manufactured by Accton

Page 20: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Wedge 100• Broadcom Tomahawk (BCM56960)

• 32 x 100G QSFP28 Ports (1x100G, 2x50G, 4x25G, 1x40G and 4x10G)

• FBOSS

• OpenBMC

• RackMon for ORV2

• COM-E CPU module for control plane

• Support DAC and optical transceiver

• Compatible with 19” and OR

• Manufactured by Accton

Page 21: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Rack V1

• 2100mm x 600mm x 1067mm

• 950kg weight handling capability

• 3 power zones of 10OU for servers

– Total 45 compute server slots per rack

• 3OU per zone for PSUs

• 3 busbars

• Manufactured by Rittal and Delta

Page 22: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

Open Rack V2

• 2210mm x 600mm x 1067mm• 1450kg weight handling

capability• 2 power zones of 16OU for

servers– Total 48 compute server slots per

rack

• 3OU per zone for PSUs• 1 busbar (3 optional)• Manufactured by Rittal and

Delta

Page 23: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

ORV1 Power• 3 power zones per rack

• 6 700W PSUs per zone + 1 redundant

• 4200W per shelf, 12.6kW per rack

• 94% efficiency (@50% load)

• 480/277VAC or 400/230VAC primary AC input, 12VDC output

• Optional 48V DC battery back up input

– 90 sec hold up time, 1 Lead Acid battery rack per 6 server racks

• 208/120VAC dual input option also available (no DC input)

• Manufactured by Delta and Power One

Page 24: Facebook OCP Server Hardware - Circle B · Facebook Server Hardware •Facebook uses four main building blocks to create its Infrastructure HW –Compute –Storage –Network –Rack

ORV2 Power• 2 power zones per rack

• 2 3.3kW PSUs per zone + 1 redundant

• 6600W per shelf, 13.2kW per rack

• 96% efficiency (@50% load)

• 480/277VAC or 400/230VAC primary AC input, 12VDC output

• Optional 52V DC battery back up input

– 90 sec hold up time, Li Ion batteries built into power shelf (no external rack)

• 208/120VAC dual input option also available (no DC input)

• Remote monitoring

• Manufactured by Delta and Power One

Batteries

PSUs