T5-DeepDive-Part_1 (1)


Drag picture to placeholder or click icon to add

SPARC T5 Servers – Deep Dive NDA – Part 1Insert Presenter’s Name HereInsert Presenter’s Title Here

Drag picture to placeholder or click icon to add


The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


•Scales to 8 sockets using directory

•Minimizes latency •Avoids congestion•Maximize bandwidth

•Double cores and cache

•Balance single thread and throughput

•Dynamically thread

•Oracle workloads •Engineered Systems•Extends

ü on-chip crypto acceleration

ü RAS

•Maximizes peak performance

•Manages thermal and current loads

•Scales elastically

Optimize

SystemsMultiply

Performance

SPARC T5

Scale Efficiently

Advance Power

Management

Design Objectives Achieved


Next Generation SPARC T5 ServersFaster. Optimized. Secure.

SPARC T5-1B

SPARC T5-8

SPARC T5-4

SPARC T5-2


SPARC T5-1B SPARC T5-2 SPARC T5-4 SPARC T5-8

Processor SPARC T5 3.6GHz SPARC T5 3.6GHz

SPARC T5 3.6GHz

SPARC T5 3.6GHz

Max Processor Chips 1 2 4 8

Max Cores/Threads 16, 128 32, 256 64, 512 128, 1024

DIMM Slots 16 32 64 128

Max Memory 128GB or 256GB 256GB or 512GB 1TB or 2TB 2TB or 4TB

Drive Bays 2 6 8 8

I/O Slots 2 x PCIe 2.0 EM, 2 NEM,1 FEM slots

8 LP x8 PCIe 3.0, 4 x 10GbE ports

16 LP x8 PCIe 3.0,4 x 10GbE ports

16 LP x8 PCIe 3.0,4 x 10GbE ports

Form Factor/RU Blade Rack 3RU Rack 5RU Rack 8RU

Max Power Consumption 689W 1927W 2410W 4850W

SPARC T5 ServersProduct Line Overview


New l SPARC T5-1B Blade Server

Compute1x SPARC T5 16-core CPU

16x DDR3 DIMMs

128GB (8GB DIMMs) or 256GB (16GB DIMMs) memory

I/O and Storage2x hot-plug PCIe 2.0 x8 Express Modules

2x 2.5” SAS HDD or SSD drives

Availability and ManagementBuilt-in RAID 0, 1, 1E

Hot plug disks

Integrates with Sun Blade 6000 network architecture

Oracle ILOM service processor

Next Generation 1-socket SPARC Blade Server


T5-1B Board

T5 CPU

16 DDR3DIMMs

ServiceProcessor

REM

BoBs

PCIeSwitch


New l SPARC T5-2 Enterprise Server

Compute2x SPARC T5 16-core CPU32x DDR3 DIMMs

256GB (8GB DIMMs) or 512GB (16GB DIMMs) of memory

I/O and Storage8x PCIe 3.0 x8 slots4x 10G-baseT ports6x 2.5” SAS HDD or SSD drives

Availability and ManagementBuilt-in RAID 0, 1, 1EHot-plug disksHot-swap and redundant fans and power suppliesOracle ILOM service processor

Next Generation 2-socket SPARC Server


T5-2 ChassisT5 CPU

ServiceProcessor

MemoryRisers


T5-2 Front

DISK 1

DISK 0

DISK 3DISK 2

DISK 5DISK 4

RFID/SerialNumber

2x USB 3.0Ports

HD-15 VGA Port

DVD

LocatorLED/Button

Fault LEDPowerButton

StatusLED

Over Temp Indicator

Status LEDs


T5-2 Rear

2x USB 3.0

HD-15 VGA Port

4x 10GbE Ports

SP Serial SP Network 10/100

PC

Ie 1

PC

Ie 2

PC

Ie 3

PC

Ie 4

PC

Ie 5

PC

Ie 6

PC

Ie 7

PC

Ie 8

System RearIndicators

AC0AC1

PS

U 0

PS

U 1




64x DDR3 DIMMs

1TB (16GB DIMMs) or 2TB (32GB DIMMs) memory

I/O and Storage16x PCIe 3.0 x8 slots, w/ carriers

4x 10G-baseT ports



Hot-plug disks, PCI cards

Hot-swap and redundant fans and power supplies



Front

Rear


SPARC T4-4 Comparison SPARC to T5-4

Feature SPARC T5-4 SPARC T4-4Form Factor 5RU, 31.5” deep 5RU, 28” deep

CPU 4x SPARC T53.6 GHz (512 threads)

4x SPARC T43.0 GHz (256 threads)

MemoryDDR3, 2TB MAX

64x SlotsDDR3, 2TB MAX

64x slots

Network 4x 10GbE 4 x 1GbE + 8x 10GbE (XAUI) Requires 2 Separate QSFP Connectors

Internal Storage Up to 8 x 2.5” SAS 3.0 or SSD, hot-plugUp to 8 x 2.5” SAS 2.0, can use up to

4x SATA SSDs, hot-plug

Removable Media 1x DVD-RW (via rKVMS; not local) 1x DVD-RW (via rKVMS; not local)

Serial 1x RS-232, 4x USB, 1x VGA 1x RS-232, 4x USB, 1x VGA

PCI Express slots16x x8 slots (Hot-Plug Low Profile slots

with carrier card, ver 3.0)16x x8 slots (Hot-Plug Express Module slots,

ver 2.0)

Power Supply 2 x 3000 Watt AC, N+NRedundant/Hot-Swap

4 x 2060 Watt AC, N+NRedundant/Hot-Swap

Fans 5 x Redundant Hot-Swap 5 x Redundant Hot-Swap


SPARC T5-4 Front Panel

DISK 0DISK 1

DISK 2DISK 3

DISK 4DISK 5 DISK 7

DISK 6

PSU 0 PSU 1

LocatorLED/Button

Over Temp Indicator

Processor Module

Fault LED

PowerButton

Status LED

Processor Module Status LEDs

Main Module(Entire Board)

Dual USB 2.0PortsSP Serial

Port

RFID/SerialNumber

PSU StatusLEDs

Main Module &SP Status LEDs

HD-15 VGA Port

Rear Fan/EM Indicator

PM 0PM 1 Main

Module


AC3 AC0

SPARC T5-4 Rear Panel

Fan Module 0 Fan Module 2 Fan Module 3 Fan Module 4

PC

Ie 1

PC

Ie 2

PC

Ie 3

PC

Ie 4

PC

Ie 5

PC

Ie 6

PC

Ie 7

PC

Ie 8

PC

Ie 9

PC

Ie 10

PC

Ie 11

PC

Ie 12

PC

Ie 13

PC

Ie 14

PC

Ie 15

PC

Ie 16

Rear I/OModule (RIO)

AC3 OKLED

System RearIndicators

PCIe CarrierHot-Plug Button, LEDs

Fan Module 1

HD-15 VGA Port

AC0 OK LED

SP Serial SP Network 10/100 2x USB 3.04x 10GbE Ports

C19 PlugC19 Plug




128x DDR3 DIMMs

2TB (16GB DIMMs) or 4TB (32GB DIMMs) memory

I/O and Storage16x PCIe 3.0 x8 slots, w/ carriers

4x 10G-baseT ports



Hot-plug disks, PCI cards

Hot-swap and redundant fans and power supplies



Front

Rear


Comparing T3, T4, and T5 SPARC T3 SPARC T4 SPARC T5

Processor Clock1.65GHz 2.85GHz, 3.0GHz 3.6GHz

# of cores 16 8 16

Core Architecture S2 S3# of memory controllers 2 4

DIMMs/BoB

2 BoBs/memory controller4 DIMMs/BoB

4 BoBs/CPU socket

2 BoBs/memory controller2 DIMMs/BoB

8 BoBs/CPU socketPCIe Gen support 2.0 3.0

PCIe Card Form Factor

Blade: Express Modules1 and 2 socket rack server: LP

4 socket server: Express Modules

Blade: Express Modules1-2 socket rack server: LP

4-8 socket rack server: LP on carrier card


T5-4 and T5-8 Processor Module

T5 Processor 1

T5 Processor 0

16 DIMMs per T5

16 DIMMs per T5

2 BoBs per Memory Controller in the T5


C0 C1 C2 C3 C4 C5 C6 C7

SSI, Present, 12C(FPGA connectivity)

FPGA-T5-48

C0 C1 C2 C3 C4 C5 C6 C7


FPGA-T5-48

C0 C1 C2 C3 C4 C5 C6 C7


FPGA-T5-48

C0 C1 C2 C3 C4 C5 C6 C7


FPGA-T5-48

6

6

6

6

6

6

66

5 5

5 5

5

5

5 5

4

4

4

4

4

4

4

4

3 3

3 3

3

3

3

3

2

2

2

2

2

2

22

1

1 1

1 1

1

1

1

00

0

0 0

0

0 0

PM3

PM2

PM1

PM0

C6 C7

C4 C5

C2 C1

C0 C1

6

6

6

6

66

5 5

5

5

5 5

4

4

4

4

4

4

3 3

3

3

3

3

2

2

2

2

22

1

1 1

1

1

1

00

0 0

0 0

PM3

PM1

PM0

C6 C7

C2 C1

C0 C1

6

6

66

5 5

5 5

4

4

4

4

3 3

33

2

2

22

1

1 1

1

00

0 0

PM3

PM0

C6 C7

C0 C16

6

5 5

4

4 33

22

110 0

PM0

C0 C1

PFM not connected (required for airflow/EMI only

CL routing CL routing CL routing CL routing1 CL between the 8 nodes 1 CL between the 6 nodes 2 CL between the 4 nodes 2 CL between the 2 nodes

T5-8 (8P option) T5-8 (6P option) T5-8 (4P option) T5-8 (2P option)

C00 1

C10 1

PMOC20 1

C30 1

PM1C40 1

C50 1

PM2C60 1

C70 1

PM3

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP8?

PCIe switches on MB in commonT5-4/8 Main Module

PCIe upstream routing (PSR)

0 0 0 01 1 2 22 1 1 03 3 3 1

C00 1

C10 1

PMOC20 1

C30 1

PM1 PFM2C60 1

C70 1

PM3

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP8?



0 0 0 01 1 2 23 3 3 1

C00 1

C10 1

PMO PFM1 PFM2C60 1

C70 1

PM3

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP8?



0 0 0 03 3 3 1

C00 1

C10 1

PMO PFM1 PFM2 PFM3

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP8?



0 0 0 0

PFM1

PFM (not required)Or Empty PM slot

PFM (not required)Or Empty PM slot

Note: the T5-8 2P option isnot a shipping configuration

PFM1

PFM2PFM2

T5-8 Processor Connectivity


C0 C1 C2 C3 C4 C5 C6 C7


FPGA-T5-48

C0 C1 C2 C3 C4 C5 C6 C7


FPGA-T5-48

6

6

66

5 5

5 5

4

4

4

4

3 3

33

2

2

22

1

1 1

1

00

0 0

PM1

PM0

C6 C7

C0 C16

6

5 5

4

4 33

22

110 0

PM0

C0 C1

CL routing CL routing2 CL between the 4 nodes 2 CL between the 2 nodes

T5-8 (4P option) T5-8 (2P option)

C00 1

C10 1

PMOC60 1

C70 1

PM3

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP4



0 0 0 03 3 3 1

C00 1

C10 1

PMO PFM1

Part #Sw 0

Partition #Switch 1

Partition #Switch 2

Partition #Switch 2

Part #Sw 4

MP4



0 0 0 0

PFM1

T5-4 Processor Connectivity


x2

x8

x8LSI SAS

x1

x1

Disk1

Disk2

x4

NEM0 NEM1 PCI-EM1

x8x8

FEM0 FEM1 Nalia Niantic

1x4

? here

USB 3.0Host Ctrl USB 1.0

Hub Ctrl

Sideband Mgmt

USB 2.0 EnetStorage Emulex

Pilot 3

VGA(HD15)

Front UCP (Dongle Cable)

Serial(RJ45)

Ethernet Mgmt(to CMM)

NEM0(4:7)

BoB BoBBoB BoBBoB BoB BoB BoB

DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM

T5CPU0

LPC 12C

FPGA

Host &Data Flash

TPM

Dual GigE10/100/1000

x8

NEM0(0:3)

NEM1(0:3)

NEM1(4:7)

1x4 1x4 1x4

PCIeSwitch 0

PCIeSwitch 1

CPUDC/DCs

CPUDebug

PortPCIe

x8PCIe Gen3PCIe Gen2

DBG

SP Module

USB 3.0

USB 2.0

USB

MIDPLANE

T5-1B Block Diagram

USB 1.1 Keyboard/Mouse

PCI-EM0

USB

0 1


T5-0

CPUDC/DCs

CPUDebugPort

PCIe

USB 2.0 Storage

get rid of all inside small boxes

EnetMgmt

10/100

SerialMgmt

BoB BoB

DIMM DIMM

T5-1

CPUDC/DCs

CPUDebug

Port

PCIe

BoB BoB

DIMM DIMM

BoB BoB

DIMM DIMM

BoB BoB

DIMM DIMM

BoB BoB

DIMM DIMM

BoB BoB

DIMM DIMM

Host &Data Flash

TPM

DBG

Service Processor

SP ModuleFRUID

DRAM

SPIFlash

NAND

USB 3.0 Host

VGADB15

Sideband Mgmt

USB0 USB1

VGA

USB 2.0 Hub

USB

PCIeSwitch 0

Slo

t 1

(8)

Slo

t 2

(8

)

Slo

t 3

(8)

Slo

t 4

(8)

Slo

t 5

(8)

Slo

t 6

(8)

Slo

t 7

(8

)

Slo

t 8

(8)

x8 x8

x8x8

x8x8

x8 x8

x1x1

HDD0

HDD0

HDD0

HDD0

HDD0

HDD0

SATA DVD

USB 3.0Hub

USB

VGA

SAS/SATAIO Controller

SAS/SATAIO Controller

Quad 10Gig Enet

BoB BoB

DIMM DIMM

BoB BoB

DIMM DIMM

x4

x4

FPGA

USB2 USB3

REAR IOFAN BOARD

x4

T5-2 Block Diagram

Internal USB

REAR IO Board

USB 1.1 Keyboard Mouse

VGA

0 1 0 1

PCIeSwitch 1


SAS 2308SAS1

SW6

DC-DCCONVERTERS

CLOCK SYNTH.&

BUFFERS

SW1 SW2 SW3SW0

Debug C

onn

4-DISK BACKPLANEDISKS 4-7SBP1 SBP04-DISK BACKPLANE

DISKS 0-3

VIDEOMUX

SerialMUX

SERVICE PROCESSOR SP

FPGA, TOD, TPM

10GB NIC 0 10GB NIC 1RIO Rear IO Module[VGA/DB15][Serial Mgmt][Enet Mgmt 10/100]

Quad 10Gig Enet (10GBase-T Copper)

1x USB (Gen2)

TO RIO TO RIO

Enet MgmtNC-SI

2x USB (Gen3)[VGA/DB15][Serial Mgmt]

HDD[4:7] HDD[0:3]

MP4, MP8T5-4 or T5-8

Midplane

CPUPCIE PORTS

CPUPCIE PORTS

CPUPCIE PORTS

CPUPCIE PORTS

1 2 3 4 5 6 7 8 9 10 11 12 13 14

CPUPCIE PORTS

15 16

SW4

EBPCI Express Backplane

MONITOR &CONTROL

MMMain Module

MBMotherboard

T5-4 & T5-8 I/O Block Diagram

RIO

Net 0

REAR USBMOST CTRLLR.

2x USB (Gen3)

SAS 2308SAS0

FRONT USBHOST CTRLLR.

FIO + VGAFRONT IOVGA, SERIAL MGMT, USB

x2 USB Ports

Debug C

onn

SW5

Net 1


CLR0

CLR0

Memory – Processor Module Block Diagram (T5-8 and T5-4)

L0FSR0

L1FSR1

L0FSR2

L1FSR3

L0FSR4

L1FSR5

L0FSR6

L1FSR7

T5 (CM1)MCU0 MCU1 MCU2 MCU3

IOS0

IOS1

CLR1

CLR2

CLR3

CLR4

CLR5

BoB0

C C0 1

BoB2

C C1 0

BoB4

C C0 1

BoB6

C C0 1

BoB1

C C0 1

BoB3

C C1 0

BoB5

C C1 0

BoB7

C C1 0

CLR6

CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5

L0FSR0

L1FSR1

L0FSR2

L1FSR3

L0FSR4

L1FSR5

L0FSR6

L1FSR7

T5 (CM0)MCU0 MCU1 MCU2 MCU3

IOS0

IOS1

CLR1

CLR2

CLR3

CLR4

CLR5

BoB2

C C0 1

BoB1

C C1 0

BoB4

C C0 1

BoB5

C C0 1

BoB3

C C0 1

BoB0

C C1 0

BoB7

C C1 0

BoB6

C C1 0

CLR6

CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5

Xcede HD 6-Row Midplane Connector


T5 PCIe Subsystem

Dual x8 PCI Express Gen 3 ports provide 32 GB/s peak b/w Supports Atomic Fetch-and-Add, Unconditional-Swap and Compare-and-Swap operationsAccelerates virtualized I/O with Oracle Solaris VMs

128k virtual function address spaces ensure direct SR-IOV access for all logical domains 64-bit DVMA space reduces IO mapping overhead, improving network performanceGuarantees fault and performance isolation among guest OS instances

Supports PCI Express Power Management


T5 PCIe Progression

T4 T5

PCI Express revision Gen 2 (dual x8 ports) Gen 3 (dual x8 ports)

Throughput full duplex 16 GBs 32 GBs

Data Management Unit Single shared unit for both x8 PCIe ports

Two independent units one for each x8 PCIe port

Physical Address Support 44 bit 48 bit

Transaction Id Identification on MSI and MSI-X No Yes

PCIe Atomic Transactions No Yes

TLP Processing Hints No Yes, directs data to L3 cache

PCIe 2.0 compliance (ECN “Internal Error Reporting”) Signaled via MSI interrupt Signaled via PCIe message


T5 and M5 PCIe Carrier Card

Supports standard low-profile PCIe cards

x16 Connector(x8 electrical)

PCIe Retimer

Air Flow


About the F40 Flash Card (Aura2)

Supported on T5-2, T5-4, T5-8, and M5-32Available on T5-2 and M5-32 at RRAvailable on T5-4 and T5-8 at RR + 1Q

Special carrier card for T5-4 and T5-8 that will span 2 PCIe slots. Required due to thermal characteristics.

Hot-plug support on T5-4, T5-8, and M5-32 at RR + 1QCan not be placed in Slot 8 of IOU on M5-32 due to thermal issues.


CPU0 CPU1

0 1 0 1

PM0

x8 up.Part. 0

x8 up.Part. 0

x8 up.Part. 0

x8 up.Part. 0

x8 up.Part. 0

NET1 x8

SAS0 x8

Front USB x1

Debug Slot x1

SP VGA x1

Rear USB x1

SAS1 x8

NET0 x8

Debug Slot x1

1x8c0

2x8c0

4x8c0

3x8c0

9x8c1

10x8c1

11x8c1

12x8c1

5x8c0

6x8c0

7x8c1

8x8c1

13x8c1

14x8c1

15x8xx

16x8xx

MP

MP

EBPCI-ExpressLow Profile

Hot Plug Slots

Dotted linedevicesresideon RIO

T5-4/8 Native 2-Socket Configuration with One Root Domain• Single non-redundant Domain• Second level Switch 6 is partitioned

differently from other configs

• Block fill color identifies Root Domain ownership• Block outline color identifies association to PM• Switch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

{

Switch 0 Switch 1 Switch 2 Switch 3 Switch 4

Sw

itch 5S

witch 6

Native Config(T5-4 only)

7Config

ID

PM1

PFM

x8 up.Part. 1

T5-4: 1 PM PCIe Connectivity

Slot #8 lanesCPU #


CPU0 CPU1

0 1 0 1

PM0

CPU2 CPU3

0 1 0 1

PM1

x8 up.Part. 0

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 1

NET1 x8

SAS0 x8

Front USB x1

Debug Slot x1

SP VGA x1

Rear USB x1

SAS1 x8

NET0 x8

Debug Slot x1

1x8c0

2x8c0

4x8c0

3x8c0

9x8c1

10x8c1

11x8c1

12x8c1

5x8c2

6x8c2

7x8c2

8x8c2

13x8c3

14x8c3

15x8c3

16x8c3

MP

MP


Hot Plug Slots

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

{


Sw

itch 5S

witch 6

Native Config6Config

ID


T5-4/8 Native 4-Socket Configuration with One Root Domain• Single non-redundant Domain• Second level Switch 6 is partitioned

differently from other configs



Slot #8 lanesCPU #


CPU0 CPU1

0 1 0 1

PM0 PM1 PM2

CPU6 CPU7

0 1 0 1

PM3

x8 up.Part. 0

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 3

x8 up.Part. 1

NET1 x8

SAS0 x8

Front USB x1

Debug Slot x1

SP VGA x1

Rear USB x1

SAS1 x8

NET0 x8

Debug Slot x1

1x8c0

2x8c0

4x8c0

3x8c0

MP

MP


Hot Plug Slots

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

{


Sw

itch 5S

witch 6


ID

PFMPFM


T5-8 Native 4-Socket Configuration with Two Root Domains• Two slot per Root Complex• Two Redundant Path Root Domains


9x8c1

10x8c1

11x8c1

12x8c1

5x8c6

6x8c6

7x8c6

8x8c6

13x8c7

14x8c7

15x8c7

16x8c7


Slot #8 lanesCPU #


CPU0 CPU1

0 1 0 1

PM0

CPU2 CPU3

0 1 0 1

PM1 PM2

CPU6 CPU7

0 1 0 1

PM3

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 2

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 2

x8 up.Part. 3

x8 up.Part. 1

NET1 x8

SAS0 x8

Front USB x1

Debug Slot x1

SP VGA x1

Rear USB x1

SAS1 x8

NET0 x8

Debug Slot x1

1x8c0

2x8c2

4x8c2

3x8c0

9x8c1

10x8c1

11x8c1

12x8c1

5x8c6

6x8c6

7x8c3

8x8c6

13x8c3

14x8c7

15x8c7

16x8c7

MP

MP


Hot Plug Slots

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

{


Sw

itch 5S

witch 6


ID

PFM





Slot #8 lanesCPU #


CPU0 CPU1

0 1 0 1

PM0

CPU2 CPU3

0 1 0 1

PM1

CPU4 CPU5

0 1 0 1

PM2

CPU6 CPU7

0 1 0 1

PM3

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 2

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 2

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 2

x8 up.Part. 3

x8 up.Part. 0

x8 up.Part. 1

NET1 x8

SAS0 x8

Front USB x1

Debug Slot x1

SP VGA x1

Rear USB x1

SAS1 x8

NET0 x8

Debug Slot x1

1x8c0

2x8c2

4x8c2

3x8c0

9x8c1

10x8c3

11x8c1

12x8c3

5x8c4

6x8c6

7x8c4

8x8c6

13x8c5

14x8c7

15x8c5

16x8c7

MP

MP


Hot Plug Slots

x8 up.Part. 0

x8 up.Part. 1

x8 up.Part. 0

x8 up.Part. 1

{


Sw

itch 5S

witch 6

Native Config 00Config

ID





Slot #8 lanesCPU #


PCIe Data Paths: Full System

Two root complexes per T5 processorEach PCIe port on a T5 processor controls a single PCIe slot


Root Complex

Each processor’s root complexes must be assigned to the same domainExample:

/pci@300/pci@1/pci@0/pci@6 & /pci@340/pci@1/pci@0/pci@6 from CPU0 are assigned to the same logical domain

PM CPU Switch I/O Slot Root Complex Path

0 0 0 1 /pci@300/pci@1/pci@0/pci@6

0 0 1 3 /pci@340/pci@1/pci@0/pci@6

0 1 2 9 /pci@380/pci@1/pci@0/pci@a

0 1 3 11 /pci@3c0/pci@1/pci@0/pci@e

1 2 0 2 /pci@400/pci@1/pci@0/pci@c

1 2 1 4 /pci@440/pci@1/pci@0/pci@c

1 3 2 10 /pci@480/pci@1/pci@0/pci@4

1 3 3 12 /pci@4c0/pci@1/pci@0/pci@8

2 4 1 5 /pci@500/pci@1/pci@0/pci@e

2 4 2 7 /pci@540/pci@1/pci@0/pci@e

2 5 3 13 /pci@580/pci@1/pci@0/pci@a

2 5 4 15 /pci@5c0/pci@1/pci@0/pci@8

3 6 1 6 /pci@600/pci@1/pci@0/pci@8

3 6 2 8 /pci@640/pci@1/pci@0/pci@8

3 7 3 14 /pci@680/pci@1/pci@0/pci@4

3 7 4 16 /pci@6c0/pci@1/pci@0/pci@4


Zakim (ZK) – M5/T5 Systems Memory Interface

Also referred to as a BoB (Buffer-on-Board)Features and Technology:

4 ZKs per M5 memory controller2 ZKs per T5 memory controllerSupports DDR3 DIMMsMemory Link to DDR Interface2 Memory Link Ports, 2 DDR ports8 Write FIFOsPass through commands to DIMMsCore frequency: 1066 MHz


T5-4 and T5-8 Memory Config Rules

• All DIMMs must have the same Oracle Part Number on a PM (Processor Module).

• All PM's must be fully populated (32 DIMMs)• No partial memory configs available at RR

• All PM's in the system must be configured the same (ATO specific rule)


T5-4 and T5-8 Airflow/Cooling

T5-8(top 3U)

T5-4(5U)


SPARC T5 Disk Drives

• Disk drives are 2.5” form factor• Chassis supports SAS HDD and SATA SSDs• SAS-2 HDDs

• 300GB @ 10K RPM• 600GB @ 10K RPM• 900GB @ 10K RPM

• SATA SSDs• 100GB• 300GB

Ready to Remove

FaultStatus

Disk LED's


SPARC T5 Disk Controller

T5 has Dual LSI SAS2008 8port SAS3/SATA2 controllersSupport for RAID 0 (striping) and RAID 1 (mirroring) using 'raidctl'


The SPARC T5 Processor


SPARC T5 Processor

• Features• 16 S3 cores, 16-128 Strands

@ 3.6Ghz• Single or multi-threaded

operation per core• System scalability to 8

sockets• SPARC Core S3

• 1-8 Strand Dynamically Threaded Pipeline

• ISA-based Crypto-acceleration

• 8MB Shared L3$

• Integrated I/O• Double I/O bandwidth over T4• 2 x8 Lane PCIe 3.0 @ 8GT/s

• System Scalability• 7 Coherence Ports for scalability

to 8S• Power Management

• Dynamic Voltage Frequency Scaling

• Downclock, Overclock


SPARC T5 CPU Block Diagram

2 x

8 P

CIe

3.0

@ 8

GB

ps16

GB

ps e

ach

dir

ectio

n

8 threads per Core

IOSubsystem

BoB BoB BoBBoB BoB BoB BoB

DDR3 – 1066 MHz

Memory Control

Co

he

ren

cy 4

x4

Sw

itch

Memory Control

Memory Control

Memory Control

8 x 9 Crossbar (~1TBps bandwidth)

C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15

L3$ B0L3$ B01MB,16-way

L3$ B01MB,16-way

L3$ B21MB,16-way

L3$ B11MB,16-way

L3$ B31MB,16-way

L3$ B0L3$ B01MB,16-way

L3$ B41MB,16-way

L3$ B61MB,16-way

L3$ B51MB,16-way

L3$ B71MB,16-way

128 KB L2$

16 KB L1I$

16 KB L1D$

FGU

Crypto

SPARC S3Core

BoB

DDR3 – 1066 MHz DDR3 – 1066 MHz DDR3 – 1066 MHz

Coherence Unit Coherence Unit Coherence Unit Coherence Unit Link 0

Link 1

Link 2

Link 3

Link 4

Link 5

Link 6

Coherency Links 12.8 Gbps per lane - 12 lanes per link


T5 Processor Overview§ 16 S3 cores @ 3.6GHz

§ 8MB shared L3 Cache

§ 8 DDR3 BL8 Schedulers providing 80 GB/s BW

§ 8-way 1-hop glueless scalability

§ Integrated 2x8 PCIe Gen 3

§ Advanced Power Management with DVFSSPAR

C Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

SPARC

Core

Cross Bar

MCU

Coherence

L3 L3 L3 L3

L3 L3 L3 L3

SerDes

SerDesSerDes MI/O

SerD

es

SerD

es

MCU

MCU

MCU

PCIe Gen3

Pwr


SPARC T5: Processor Key Features

SPARC T5 based on SPARC S3 core: Same core as used in T416 S3 cores, dual PCI Express 3.0 root complexes

Up to eight T5 processors per system, 16 cores x 8 threads → 128 (on T5-1B) or up to 1024 threads (on T5-8)

Clock frequency is 3.6 GHz

Each SPARC S3 core contains:

● 2 Integer pipelines

● 1 FGU pipeline (consisting of 3 physical sub-pipelines):

– FPX pipeline– FGX pipeline– FPD pipeline

● 1 Load-Store (Memory) pipeline



Each Core has two Level 1 cache memories, one for data and one for instructions, each 16KB in sizeEach Core has a Level 2 unified cache, 128KB in size

• Caches are all inclusive: L3 inclusive of L2; L2 inclusive of L1 (in this context, ‘inclusive’ refers to the fact a cached entry is always present in the next higher level of cache)

• Each core on SPARC T5 is capable of OoO execution, dual-issue of instructions but in order commit.

• Each core on SPARC T5 also includes cryptographic acceleration hardware, accessible via user-level instructions.



Memory – systems based upon SPARC T5 use DDR3 Quad Rank DIMMsDIMMs used are Registered DIMMS (RDIMMs)

16 DIMM slots per Processor→ memory capacity

depends on size of DIMM chosen for system

8 GB, 16 GB DIMM

ECC DIMMs required to be used at 1066 MT/s

Newly designed BoB between MCU and DIMMs

Four MCUs per processor; each maintains a memory link speed of 12.8 Gbps

Protocol between each MCU and its two BoBs is proprietary in nature


S3 Core Recap

28nm port from 40nm T4Out-of-order, dual-issueHigh frequency achieved with 3.6GHz 16 stage integer pipelineDynamically threaded, one to eight strandsAccelerates 16 encryption algorithms and random number generation


SPARC T4/T5/M5 Leads in On-Chip Encryption Acceleration

Processor / Mechanisms SPARC T4, T5, and M5 IBM Power7 IBM Power7+Intel Westmere/

Sandybridge

Operational ModelUserland unprivileged access

to on-core cryptographic functions

none3 accelerators shared

across 8 cores Userland

Asymmetric /Public Key Encryption RSA, DH, DSA, ECC none RSA, ECC RSA, ECC

Symmetric Key / Bulk Encryption AES, DES, 3DES, Camellia, Kasumi none AES AES

Message Digest / Hash Functions

CRC32c, MD5, Sha-1, SHA-224, SHA-256, SHA-

384, SHA-512none

MD5, SHA-1, SHA-256, SHA-512 none

Random Number Generation Supported none Supported none

API Support PKCS#11, Ucrypto APIs, JCE none PKCS#11 Intel IPP libraries

Virtualization SupportSolaris Zones

Oracle VM for SPARCnone ?? Intel VT

Crypto acceleration now available from 1 processors systems up to 32-way systems!


SPARC Core Roadmap

SPARC64 VII/VII+ Core

M-Series

T3 Servers

M5 Servers

T4 Servers

M6 ServersT5 Servers

S2 Core

S3 Core

S5 Core


S3 Core Overview

8-way threaded, dual-issue, OoO execution, in order commitDynamically threaded with hardware-optimized resource sharingSupport for Critical ThreadsDeep pipeline for high frequency operation (3 GHz in 40 nm)Balanced single-thread and multi-thread performance

- 5X better single-thread than SPARC T3 with equivalent multi-thread performance

Enhanced instruction set to accelerate Oracle SW stack- PAUSE, fused compare-branch

Integrated user-level cryptographic acceleration- DES/3DES, AES, Kasumi, Camellia, MD5, SHA-1, SHA-

224/256/384/512,RSA, DSA, CRC32cFoundation core for future technology / product nodes


Oracle SPARC S3 CoreEnables T4, T5, M5

• Design Goals- Develop a common replacement core for T-series processors

- Significantly improve the single strand performance of the T3 processor

- Significantly improve the throughput performance of the M3 processor

- Improve the RAS and power management capabilities- Maintain backward ISA, Solaris and OVM for SPARC compatibility

Core S1: M2, M3 Core S2: T2, T2+, T3 Core S3: T4, T5, M5

Frequency2.4 – 3.0 GHz 1.4 – 1.65 GHz T4: 2.85 – 3.0 GHz

T5, M5: 3.6 GHz

L1 Instruction Cache 64KB 16KB 16KB

L1 Data Cache 64KB 8KB 16KB

L2 Cache - - 128KB

# of Pipelines 1 2 1

# of Threads per Pipeline 2 4 1-8 Dynamic

Instructions per Thread 4 per cycle 1 per cycle 2 per cycle

Out of Order Issue Yes No Yes (36 instr window)

Cryptography Acceleration None SPU ISA Based

OVM for SPARC Compatible No Yes Yes

SPARC V9 ISA Compatible Yes Yes Yes


S3 Core: Dynamic Threading

While software can activate up to 8 strands on each core at a time, hardware dynamically and seamlessly allocates core resources such as instruction, data, and L2 caches and TLBs, and out-of-order execution resources such as the 128-entry re-order buffer in the core among the active strands. Software activates strands by sending an interrupt to a HALTed strand. Software deactivates strands by executing a HALT instruction on each strand it wants to deactivate. No strand has special hardware characteristics; all strands have identical hardware capabilities.


S3 Core: Dynamic Threading

Since the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate. If software effectively halts all strands except one on a core via Critical Thread Optimization, the core devotes all of its resources to the sole running strand. Thus, that strand will run as quickly as possible


S3 Core: Critical Thread Optimization

The S3 core, starting with Oracle Solaris 10 8/11, can optimize performance by assigning one software thread exclusive access to all of a core's hardware resources. That software thread is considered to be a "critical thread." Solaris automatically detects opportunities to perform this assignment: when one software thread has high CPU utilization and there are more cores than runnable threads. We recommend that users allow Solaris to automatically perform Critical Thread assignment. A privileged user can tell Solaris that a particular software thread should be a critical thread, via the nice(1) command. Solaris will then assign that thread to a core, even if there are more runnable threads than cores.



Solaris Critical Threads optimization for S3 core, tries to provide exclusive access of certain hardware resources to certain application threadsSince the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivateThere is no new API for declaring threads as ‘critical’; that would require significant changes to source codeRather, to invoke Critical Thread Optimization, use the following CLI or system calls to flag a thread as critical by raising its priority to 60:

priocntl(1)priocntl(2)priocntlset(2)



Starting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised to priority 60; the thread can be in any scheduling classIn Oracle Solaris 11, to be considered critical by the scheduler, a thread must be:

in the FX (Fixed Priority) or RT (Real-Time) scheduling classes

be raised to priority 60 by one of the previously mentioned mechanisms

In either of the above instances, this one thread will run as quickly as possible as it has exclusive access to all core resources


Critical Threads for key applications

Applicability Opportunity Current Status

Database Logwriter, LMS Up to 30% improvement in efficiency

LMS is already CT ready. LGWR planned for 12c

JAVA (JVM)Compiler threads, GC and priority mapping

support

Up to 2x improvement for app startup, Smooth GC

Support for JVM and JAVA apps to be CT aware is integrated in JDK7U4

Coherence Packet writer, service thread

Up to 20% improvement in throughput

Integrated in Coherence version 3.7.1 Patch 1

Solaris S11U1 / S10U11Improve CT perf to be within

10% of best case (hand optimized)

Optimizations for decayed PG util and stealing being

integrated in S11U1


Internode Coherency Overview

Glueless 1-hop scaling to eight socketsGlueless means no hub required to connect all 8 CPUs

A precise directory tracks all L3s in the system

striped across all processors stored in on-chip SRAMsflexible for different socket counts

Higher BW efficiency than snoop-based protocols enables better scaling

50% more effective bandwidth than comparable snoopy implementation

T -

T5 T5

T5 T5

T5 T5

T5T5


Internode Performance Optimizations

Speculative memory reads prior to cache line serialization in the directoryCache-to-cache line transfers between nodesDynamic congestion avoidance routes inter-node data around congested links


T5 System Interconnects

1-WaySingle Socket

2-WayDual Socket

4-Way

6-Way

8-Way


8 Socket Local Coherency & Data Interconnect

DIMMS

M5/T5

M5/T5

M5/T5

M5/T5

M5/T5

M5/T5

M5/T5 M5/T5 DIMMSDIMMS

DIMMS

DIMMS

DIMMS

DIMMS

DIMMS

POINT-TO-POINTLOCAL

INTERCONNECT

All-to-All InterconnectWhere the Node-to-Node Fabric is 12 diff pairs per link in each direction.

T5 interconnect bandwidth= 157.5 GB/sec T5-8 interconnect bandwidth= 1260 GB/sec

DDR3-1066 Memory BandwidthT5 is 133 GB/secT5-8 is 1064 GB/s

PCIe Gen3 Bandwidth8 diff pairs per portsAt 8Gb/sec ~8GB/sec/direction

~8Gb/sec/lane X 8 lanes = 64 Gb/sX 2 directions X 2 Ports/chip= 256Gb/s/chip= 32 GB/s per chip


Latency for T5

T5-2 T5-4 T5-8

Local Memory 136ns

Remote Memory 209ns

Cache to Cache 127ns 146ns 155ns


Scalability of T4 vs T5

T4 – Snoopy BasedCoherence Protocol

T5 – Directory BasedCoherence Protocol

By numbers8 node snoops will consume 25% Link B/W & increases linearly w/more nodes

8 node directory based will consume 5% of Link B/W

Feature

Address serialization is done at Home Node. Home Node broadcasts snoop request to all nodes. All nodes except the requesting node require to participate the snoop operation and provide snoop response back to requesting node.

Address serialization is done at Directory Node. Directory Node keeps track of which node hold each cache line. Eliminating the need for broadcasting, and relieve the L3$ from unnecessary foreign snoop operation.

Link BandwidthMessage broadcast and response consume a lot of link bandwidth.

Directory filter the snoops sent to the share nodes. Allow link bandwidth to be used more efficiently.

L3$ PerformanceL3$ need to participate every snoop request from any other node. The L3$ performance can be dropped due to lots of foreign snoop requests.

Only the L3$ from the selective node require to participate the foreign snoop operation. L3$ has less distraction from foreign snoop request.

Scalability Limited to small scale of system.Easy to scale to large number of processor environment.


Memory Controller of T4 vs T5

T4 MCU T5 MCU

Memory Link Speed 6.4 Gb/s 12.8 Gb/s

Memory Link ProtocolLegacy Intel FBDIMM2 Protocol Advanced In-house Link Protocol

Memory Link Low Power Feature

Not Supported L0s, L1

Memory Buffer Intel Milbrook2 MB Advanced In-house MB

DDR3 Protocol Burst length of 4 Burst length of 8

DDR3 Speed 800/1066 1066

DDR3 Device 1Gb/2Gb 2Gb/4Gb


RAS


Definition of Terms

• Hot-plug:• refers to the fact that a component can be plugged and unplugged without

powering down the platform. It applies to both hot swap and hot service.• Hot service:

• refers to the ability to perform hot-plug operations, with the additional necessity of some operator actions (invocation of a CLI or actuating a hot service button on the component to be removed).

• The system will notify the user when it is safe to remove the component. • Typical examples would be PCIe Express modules.

• Hot swap: • refers to an operation where a component is unplugged and plugged in with

no interaction with the ILOM or domain required. • Typical examples here are a single RAID disk or a power supplies.


SPARC T5 System RAS Overview

• Designed to minimize part count and operating temperature to enhance reliability

• End-to-end data protection detecting and correcting errors throughout server – ECC everywhere

• Processor and Memory protection• CPU core and thread off-lining• Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failover

• Major components redundant & hot-pluggable• Fan, Power Supply, and internal disks• RAID capability for internal disks

• Fault Management Architecture (FMA) support on ILOM


End-to-End RAS

Built RAS from the inside outStart with the processor, then memory, system and IO, virtualization layer, and the OSAdd Oracle Solaris Cluster software for additional service availabilityFault Management Architecture (FMA) binds all the layers together


T5/M5 Systems RASS11 FMA• Diagnosis engine on SP• Auto reconfigure on failure• Soft Error Rate Discrimination (SERD)• Bad page retirement• OS and SP watchdogs• FMA Component hot-upgradeable

T5/M5 Processor• L1$ Tag, Status $ Data

• Parity protection• Retry on error• L2$/L3$ Data• SEC/DED protection• Cache-line Sparing• L2$/L3$ Tags• SEC/DED protection• Inline Correction• Cache-line Sparing• L2$/L3$ Status & Directory• SEC/DED protection• Inline Correction• Architectural RegistersL2 Cache• SEC/DED protection• Precise Trap andHypervisor Correction and Retry

System• Redundant SPs with automatic failover• Redundant clock boards• Diagnosis to the FRU level on first fault

Power and Cooling• Advanced Power Management• Redundant hot-swap fans• Redundant hot-swap AC/DC• Dual grid power

System I/O• PCI-Express end-to-end CRC• PCI Express link retry• Hot-plug low profile PCI Express cards• Redundant, hot-plug boot disks• Alternate connections between M5 and IOcontrollersMemory• SDRAM Soft Errors

• ECC Protection and Correction• Extended ECC Protection• 4-bit Correction• Pin Steering

• Channel Interconnect• CRC protection/Message Retry• Lane Sparing

Hypervisor• Enables software partitioning (LDoms)virtualization and failure containment• Processor support for error clearing, correction and collection

Central Directory and Switch• SEC/DED protection with in line correction• Physical domain isolation • CRC protected System Interconnect with message retry and lane sparing• Deconfigurable directory chips, no loss of functionality, minimized bandwidth loss• Redundant Scalability Switch BoardsUnique to M5


RAS Features T4-1/T4-2 T4-4 M4000/M5000 T5-2 T5-4/T5-8 M8000/M9000 M5-32

Dynamic Reconfiguration X X X X X X X

Processor L2$ Degradation X X X X X X X

Processor L1$ Degradation X X

L3$ Line Sparing X X X

Hot-plug PCIe X X X X X

Memory Mirroring X X

Memory Lane Sparing X X X

CPU interconnect Lane Sparing X X X

Hard Domains X X X

Logical Domains X X X X X

Redundant System Interconnect X X X X

Hot-plug CPU and Memory X

Redundant/Hot-plug SP X X

Redundant Clock X X

•Hot-swap power supplies, fans•Hot-plug disks•ECC Memory

•Extended-ECC Memory•Instruction Retry•Partial ASIC Recovery

Common Features

SPARC RAS Features


Fault Management

• Knowledge Articles in MOS• ILOM fdd Diagnosis• Faults and Alerts• No ALOM Compatibility• ILOM FMA Captive Shell• Sideband Service Processor Network Connection• New ILOM Fault Notification (SNMP Trap)• ASR Support• FMA on M5 ILOM also applies to T5 ILOM, except for M5 specific features


FMA – restricted shell in SPSH

To get FMA details in SPSHstart -script /SP/faultmgmt/shellReturns a faultmgmtsp> prompt

Available build-in commands:echo - Display information to user.

● Typical use: echo $?

helpExit – exits restricted shell

External commands:fmadm - Administers the fault management servicefmdump - Displays contents of the fault and ereport/error logsfmstat - Displays statistics on fault management operationsetcd - ereport injector


Fault Management on T5 systems

T5 CPU and Memory faults are now diagnosed by ILOM FMA's Fault Proxy is used to keep ILOM's fault manager in sync with Solaris' fault

manager. Both will display the sum of all faults in the system. Faults can be repaired from either side. Fault Proxy communicates via the Ethernet Over USB connection. IO faults are still diagnosed by Solaris.

Disabled Database (DDB) owned by ILOM For faults which diagnose resources as unusable, ILOM will add those resources to

the DDB. Resources excluded on next host reset. When faults are repaired, ILOM automatically updates the DDB. Bringing

components back online requires a host reset. Extended SP-POST (Power on Self Test)

Runs at SP boot. Tests devices on the SP FRU and its Ethernet port. Status stored and converted to ereports after ILOM boots.


Fault proxy

IO ereports are “forwarded” from the SP to the control domain, and then on to any relevant IO domainFaults are “proxied” between the SP, the control domain and any IO domains to provide a single view of faults in the system.

Non-servicable faults such as memory faults are not proxied.

The SP and the control domain can view and manage all faults in the system.An IO domain can only view and manage faults local to the domain.

Control Domain IO Domain

LDC LDC

TCP/IP

SP

hostd

FETDip-transprt

ETM ETM

ETM

ETM

ETMip-transport

ereports ereports

faultsfaultsLDC


Degraded HW Configurations – T5-8

If a CPU has no configurable memory, i.e., all memory links are unconfigurable, then the CPU itself is unconfigurable

A T5-8 will not operate with only 5 or 7 CPUs configuredOne more CPU(s) must be chosen to be deconfiguredFor an 8-way, if we fault a CPU, we will offline the other CPU on the same PMFor a 6-way, if we fault a CPU, we will also offline the other CPU on the same PM


ASR Support

• SPARC T5 servers will be supported by ASR (Automatic Service Request) at release

• Continues use of sunHwTrapFaultDiagnosed SNMP notification

• Telemetry for ILOM fdd diagnosis

• Supports platform and FRU identity

• Supports multi-suspect list

T5-DeepDive-Part_1 (1)

Documents

Transcript of T5-DeepDive-Part_1 (1)