Facebook OCP Server Hardware

Facebook Server Hardware

• Facebook uses four main building blocks to create its Infrastructure HW– Compute

– Storage

– Network

– Rack & Power

• Varying the ratios of compute to storage, and the internal configuration of the servers provides Facebook with all its SKUs.

Standard

Systems

I

Web

III

Database

IV

Hadoop

V

Haystack

VI

Feed

VII Cold

Storage

CPU

High

2 x Haswell

(12c)

2.5 GHz

High

2 x Haswell

(12c)

2.5 GHz

High

2 x Haswell

(12c)

2.5 GHz

Low

1 x Avoton

(8c)

2.7 GHz

High

2 x Haswell

(12c)

2.5GHz

High

2 x Haswell

(12c)

2.5 GHz

MemoryLow

32GB

High

256GB

Medium

128GB

Low

32GB

High

256GB

Medium

128GB

DiskLow

500GB

SATA

High IOP

2 x 3.2TB

Flash

128 GB

mSata

High

15 x 4TB (or

6TB)

NL SAS

High

30 x 4TB (or

6 TB)

NL SAS

Medium

2TB SAS

(1.8 TB

FLASH

optional)

Very High

240 X 4TB

16 active

disks

Services Web DatabaseData

Warehouse

(Hadoop)

Object Store

(Photos)

Cache

(Newsfeed)Archival

Type I

• Front end/web service machines.

• Compute servers only

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Power Shelf

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Compute Empty Compute

Switch

Empty

Compute Empty Compute

Compute Empty Compute

Type III

• High performance database applications

• Compute nodes w/ PCI-E Flash Cards

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Power Shelf

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Switch

Empty

Compute +

FlashEmpty

Compute +

Flash

Compute +

FlashEmpty

Compute +

Flash

Type IV

• Data warehouse storage and analytics (for ex. Hadoop)

• Powerful Compute + Mass Storage

Storage

Power Shelf

Storage

Storage

Storage

Storage

Storage

Compute Compute Compute

Storage

Power Shelf

Compute Compute Compute

Storage

Compute Compute Compute

Storage

Compute Compute Compute

Power Shelf

Compute Compute

Compute Compute Compute

Switch

Empty

Compute

Type V

• Object storage (for ex. Photo and Video storage)

• Cost Efficient Compute + Mass Storage

uServer + Storage

Power Shelf

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

uServer + Storage

Power Shelf

uServer + Storage

Power Shelf

uServer + Storage

Switch

Empty

uServer + Storage

uServer + Storage

Type VI

• Cache heavy applications (for ex. News Feed)

• Compute w/ large memory

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Power Shelf

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Switch

Empty

Compute +

High MemEmpty

Compute +

High Mem

Compute +

High MemEmpty

Compute +

High Mem

Type VII

• Archival back up storage

• Single compute node connected to 240HDDs.

Storage

Storage

Storage

Storage

Storage

Storage

Empty Compute Empty

Storage

Storage

Storage

Storage

Empty Compute Empty

Storage

Storage

Storage

Power Shelf

Switch

Empty

Storage

Storage

Storage

Open Rack

• Wide equipment bay (21”) for maximum space efficiency

– Same external width (24”) as standard 19” rack.

• Shared power supply architecture for better resiliency

• Design optimized for serviceability and efficiency

Open Rack

• 1 OpenU = 48mm height

• Power shelf feeds 12V busbar that connects to servers

– Standardized mechanical interface between servers and rack

Open Compute Server• Dual Socket Xeon E5-2600 V3/V4

(Haswell/Broadwell)

• 16 DIMMs (2 per channel/4 channel per socket/2400MHz)

• Motherboard dimensions 6.5”x20”

• 3 PCI Gen 3 add in slots (x24)

• OCP Mezzanine Card (x8 PCI Gen 3)

• M.2/mSATA/HDD for boot

• ORV1/ORV2 compatible – 2OU 1/3 width

• BMC

• Manufactured by Quanta and WiWynn

Yosemite Compute Platform

• Platform for 4 single socket compute nodes

• ORV1/ORV2 compatible – 2OU 1/3 width

• Multi-host (shared) 40G/50G OCP Mezzanine NIC

• Shared BMC

Mono Lake

• Yosemite compute node card

• Xeon D-1500 SoC

• 4 DIMMs (2 per channel/2 channel/2400MHz/up to 128GB)

• 2 M.2 (NVMe/SATA)

• 210mm x 110mm

• Manufactured by Quanta

Big Sur GPU Server• Full Size Dual Socket Xeon E5-2600

V3/V4 (Haswell/Broadwell) Motherboard

• 8 x16 3 PCI Gen 3 GPU card slots– NVIDIA Tesla, AMD FirePro and Intel Phi

• OCP Mezzanine Card (x8 PCI Gen 3)– Ethernet and Infiniband

• 8x 2.5” SATA HDD/SSD

• 4OU ORV1/ORV2 compatible

• BMC

• Manufactured by Quanta

Open Vault Storage

• JBOD storage attach to Open Compute Server

• 2OU ORV1/ORV2 compatible • 30 HDDs (2 trays x 15 HDDs)• LSI SAS2x28 6Gb expander

per tray• 1 Ext. MiniSAS + 2 Int.

MiniSAS front facing ports per tray

• Manufactured by WiWynn

Microserver Storage

• Light weight server inside Open Vault• Same storage capacity as Open Vault• Microserver

– Intel Avoton C2750– DDR3 SO-DIMM (2 per channel/2

channel/1600 MHz/up to 32GB)– M.2 (NVMe/SATA) and mSATA

• Storage SubSystem– 12G SAS IOC + Expander (SAS 3008

+ SAS3x24)– BMC– OCP Mezzanine NIC

• Manufactured by WiWynn

Lightning (NVMe in Open Vault)

• JBOF storage attach to Open Compute Server over PCIe

• 2OU ORV1/ORV2 compatible

• Up to 60 SSDs (2 trays of 15 x4 PCI Gen 3 slots)

• 4 x4 PCI Gen 3 external upstream ports per tray

• OpenBMC

• Manufactured by WiWynn

Wedge 40

• Broadcom Trident II

• 16 x 40Gb QSFP ports (can be split to 4x10Gb)

• FBOSS

• OpenBMC

• Microserver card used for control plane

• Compatible with 19” and OR

• Manufactured by Accton

Wedge 100• Broadcom Tomahawk (BCM56960)

• 32 x 100G QSFP28 Ports (1x100G, 2x50G, 4x25G, 1x40G and 4x10G)

• FBOSS

• OpenBMC

• RackMon for ORV2

• COM-E CPU module for control plane

• Support DAC and optical transceiver

• Compatible with 19” and OR

• Manufactured by Accton

Open Rack V1

• 2100mm x 600mm x 1067mm

• 950kg weight handling capability

• 3 power zones of 10OU for servers

– Total 45 compute server slots per rack

• 3OU per zone for PSUs

• 3 busbars

• Manufactured by Rittal and Delta

Open Rack V2

• 2210mm x 600mm x 1067mm• 1450kg weight handling

capability• 2 power zones of 16OU for

servers– Total 48 compute server slots per

rack

• 3OU per zone for PSUs• 1 busbar (3 optional)• Manufactured by Rittal and

Delta

ORV1 Power• 3 power zones per rack

• 6 700W PSUs per zone + 1 redundant

• 4200W per shelf, 12.6kW per rack

• 94% efficiency (@50% load)

• 480/277VAC or 400/230VAC primary AC input, 12VDC output

• Optional 48V DC battery back up input

– 90 sec hold up time, 1 Lead Acid battery rack per 6 server racks

• 208/120VAC dual input option also available (no DC input)

• Manufactured by Delta and Power One

ORV2 Power• 2 power zones per rack

• 2 3.3kW PSUs per zone + 1 redundant

• 6600W per shelf, 13.2kW per rack

• 96% efficiency (@50% load)

• 480/277VAC or 400/230VAC primary AC input, 12VDC output

• Optional 52V DC battery back up input

– 90 sec hold up time, Li Ion batteries built into power shelf (no external rack)

• 208/120VAC dual input option also available (no DC input)

• Remote monitoring

• Manufactured by Delta and Power One

Batteries

PSUs