2 zEnterprise Technical Details - ITSOWW2011

yourdotcom

International Technical Support Organization and Authoring Services

© 2011 IBM Corporation

www.ibm.com/redbooks

zEnterprise Technical Details

Parwez HamidExecutive IT ConsultantIBM UK Limited


yourdotcomibm.com International Technical Support Organization and Authoring Services

2

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*BladeCenter®, DB2®, e business(logo)®, ESCON, eServer, FICON, IBM®, IBM (logo)®, MVS, OS/390®, POWER6®, POWER6+, POWER7, S/390®, System p, System p5, System x, System z, System z9®, System z10®, WebSphere®, z9®, z10, z/Architecture®, z/OS®, z/VM®, z/VSE, zSeries®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.

Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.



3

zEnterprise Technical Details – z114

Capacity on Demand14

Capacity and Performance Planning15

Upgrades16

Summary17

Coupling Link Connectivity and STP8

I/O Summary9

Sysplex Consisderations10

STP Support and Enhancements11

RAS13

Crypto12

Network Connectivity Features7

1 IBM System z Family

2 Model, Frame and Processor Drawer

3 Memory Design

4 I/O Infrastruture

5 I/O Subsystem Enhancements

6 Storage Connectivity Features



4

IBM System z Family



5

IBM System z10 family

�Announce 2/08 – Server w/ up to 77 PU cores�5 models – Up to 64-way�Granular Offerings for up to 12 CPs�PU (Engine) Characterization

– CP, SAP, IFL, ICF, zAAP, zIIP

�On Demand Capabilities – CoD, CIU, CBU, On/Off CoD, CPE

�Memory – up to 1.5 TB for Server and up to 1 TB per LPAR

– 16 GB Fixed HSA

�Channels– Four LCSSs– 2 Subchannel Sets– MIDAW facility– 63.75 subchannels– Up to 1024 ESCON® channels– Up to 336 FICON® channels– FICON Express2, 4 and 8– zHPF– OSA 10 GbE, GbE, 1000BASE-T– InfiniBand® Coupling Links

�Configurable Crypto Express3�Parallel Sysplex clustering�HiperSockets™ – up to 16�Up to 60 logical partitions�Enhanced Availability�Operating Systems

– z/OS, z/VM, z/VSE™, TPF, z/TPF, Linux on System z

IBM System z10 EC (2097) IBM System z10 BC (2098)

�Announced 10/08 – Server w/ 12 cores �Single model – Up to 5-way CPs�High levels of Granularity available

– 130 Capacity Indicators

�PU (Engine) Characterization– CP, SAP, IFL, ICF, zAAP, zIIP

�On Demand Capabilities – CoD, CIU, CBU, On/Off CoD. CPE

�Memory – up to 256 GB for Server– 8 GB Fixed HSA

�Channels– Two LCSSs– 2 Subchannel Sets– MIDAW facility– 63.75 subchannels– Up to 480 ESCON channels– Up to 128 FICON channels– FICON Express2, 4 and 8 – zHPF– OSA 10 GbE, GbE, 1000BASE-T– InfiniBand Coupling Links

�Configurable Crypto Express3�Parallel Sysplex clustering�HiperSockets – up to 16�Up to 30 logical partitions�Enhanced Availability�Operating Systems

– z/OS, z/OS.e, z/VM, z/VSE, TPF, z/TPF, Linux on System z



6

IBM zEnterprise family

�Announced 7/10 – Server w/ up to 96 PU cores�5 models – Up to 80-way�Granular Offerings for up to 15 CPs�PU (Engine) Characterization

– CP, SAP, IFL, ICF, zAAP, zIIP�On Demand Capabilities

– CoD, CIU, CBU, On/Off CoD, CPE�Memory – up to 3 TB for Server and up to 1 TB per LPAR

– 16 GB Fixed HSA�Channels

– Four LCSSs– 3 Subchannel Sets– MIDAW facility– Up to 240 ESCON channels– Up to 288 FICON channels– FICON Express8 and 8S– zHPF– OSA 10 GbE, GbE, 1000BASE-T– InfiniBand Coupling Links

�Configurable Crypto Express3�Parallel Sysplex clustering�HiperSockets – up to 32�Up to 60 logical partitions�Enhanced Availability�Unified Resource Manager�Operating Systems

– z/OS, z/VM, z/VSE, z/TPF, Linux on System z

�Announced 7/10�Model 002 for z196 or z114�zBX Racks with:

– BladeCenter Chassis– N + 1 components– Blades– Top of Rack Switches– 8 Gb FC Switches– Power Units– Advance Management Modules

� Up to 112 Blades– IBM Smart Analytics Optimizer Solution– POWER7 Blades– IBM System x Blades– IBM WebSphere DataPower Integration

Appliance XI50 for zEnterprise (M/T 2462-4BX)

–Operating Systems– AIX 5.3 and higher– Linux for x Blades– Microsoft Windows for x Blades

–Hypervisors– PowerVM Enterprise Edition– Integrated Hypervisor for System x

IBM zEnterprise 196 (2817) IBM zEnterprise Blade Extension (2458)

IBM zEnterprise 114 (2818)

�Announced 07/11 �2 models – M05 and M10

� Up to 5 CPs�High levels of Granularity available

– 130 Capacity Indicators�PU (Engine) Characterization

– CP, SAP, IFL, ICF, zAAP, zIIP�On Demand Capabilities

– CoD, CIU, CBU, On/Off CoD. CPE�Memory – up to 256 GB for Server

– 8 GB Fixed HSA�Channels

– Two LCSSs– 2 Subchannel Sets– MIDAW facility– Up to 240 ESCON channels– Up to 128 FICON channels– FICON Express8 and 8S – zHPF– OSA 10 GbE, GbE, 1000BASE-T– InfiniBand Coupling Links

�Configurable Crypto Express3�Parallel Sysplex clustering�HiperSockets – up to 32�Up to 30 logical partitions�Unified Resource Manager�Operating Systems

– z/OS, z/VM, z/VSE, TPF, z/TPF, Linux on System z



7

z114 continues the CMOS Mainframe heritage

0

500

1000

1500

2000

2500

3000

3500

1997Multiprise®

2000

1999Multiprise

3000

2002z800

2004z890

2006z9 BC

2008z10 BC

MH

z

139MHz

413 MHz

625 MHz

1.0 GHz

1.4 GHz

� z800 - Full 64-bit z/Architecture®

� z890 - Superscalar CISC pipeline� z9 BC - System level scaling

3.5 GHz

� z10 BC - Architectural extensions �Higher frequency CPU

� z114 – Additional Architectural extensions and new cache structure

� Multiprise 2000 – 1st full-custom Mid-range CMOS S/390

� Multiprise 3000 – Internal disk, IFL introduced on midrange

2011z114

3.8 GHz

4000



8

IBM zEnterprise 114 (z114)

Models, Frame and Processor Drawers



9

� Machine Type

– 2818

� 2 Models

– M05 and M10

– Single frame, air cooled

– Non-raised floor option available

– Overhead Cabling and DC Power Options

� Processor Units (PUs)

– 7 PU cores per processor drawer (One for M05 and two for M10)

– Up to 2 SAPs per system, standard

– 2 spares designated for Model M10

– Dependant on the H/W model - up to 5 or 10 PU cores available for characterization

• Central Processors (CPs), Integrated Facility for Linux (IFLs), Internal Coupling Facility (ICFs), System z Application Assist Processors (zAAPs), System z Integrated Information Processor (zIIP), optional - additional System Assist Processors (SAPs)

• 130 capacity settings

� Memory

– Up to 256 GB for System including HSA

• System minimum = 8 GB (Model M05), 16 GB (Model M10)

• 8 GB HSA separately managed

• RAIM standard

• Maximum for customer use 248 GB (Model M10)

• Increments of 8 or 32 GB

� I/O

– Support for non-PCIe Channel Cards

– Introduction of PCIe channel subsystem

– Up to 64 PCIe Channel Cards

– Up to 2 Logical Channel Subsystems (LCSSs)

� STP - optional (No ETR)

z114 Overview



10

� M/T 2818 – Model M05

– Air cooled– Single Frame– Non-raised floor option available– 30 LPARs

� Processor Units (PUs)– New processor drawer design (1

processor drawer)– 7 per system

• 2 SAPs standard• Up to 5 CPs• Up to 5 specialty engines• Up to 2 zIIPs/zAAPs• 0 spares when fully configured

� M/T 2818 – Model M10– Air cooled– Single Frame– Non-raised floor option available– 30 LPARs

� Processor Units (PUs)– New processor drawer design (2

processor drawers)– 14 per system

• 2 SAPs standard• Up to 5 CPs• Up to 10 specialty engines• Up to 5 zIIPs/zAAPs• 2 dedicated spares

� When Model M10 (requires the 2nd processor drawer)?

– > 5 Customer PUs– > 120 GB memory– > 4 Fanouts for additional I/O connectivity – especially PSIFB links

• Depends - numbers vary for drawers, I/O features and PSIFB links

zEnterprise 114 Models M05 and M10



11

zEnterprise 114 Functions and Features (GA Driver 93 – September, 2011)

Two hardware models

Up to 10 processors configurable as CPs, zAAPs, zIIPs, IFLs, ICFs, or

optional SAPs

Up to 130 subcapacity settings across a maximum of 5 CPs

Up to 256 GB of Redundant Array of Independent Memory (RAIM) for System

Dedicated Spares on the Model M10

Increased capacity processors

Out of order instruction execution

Improved processor cache design

New and additional instructions

On Demand enhancements

CFCC Level 17 enhancements

Cryptographic enhancements

6 and 8 GBps interconnects

2 New OSA CHPIDs – OSX and OSM

Doubled HiperSockets to 32

Additional STP enhancements

Doubled Coupling CHPIDs to 128

Improved PSIFB Coupling Link

Physical Coupling Links increased to 72 (Model M10)

New 32 slot PCIe Based I/O Drawer

Increased granularity of I/O adapters

New form factor I/O adapters i.e FICON Express8S and OSA-Expres4S

Humidity and altimeter smart sensors

Optional High Voltage DC power

Optional overhead I/O cable exit

zBX-002 with ISAOPT, POWER7, DataPower XI50z and IBM System x

Blades

NRF Support with either top exit or bottom exit I/O and power

Reclassification from “general business”environment to “data center”

z114



12

zEnterprise zBX Functions and Features

One hardware model

zBX is controlled by one specific z196 or z114

Up to 4 Racks (B, C, D and E)

2 BladeCenters Chassis per rack

Non-acoustics doors standard

Optional Rear Door Heat Exchanger

Optional Rear acoustic door

Redundant Power, Cooling and Management Modules

10 GbE and 1000BASE-T Network modules

8 Gb FC modules

Advance Management Module

1000BASE-T and 10 GbE TORs

Up to 112 Blades

IBM Smart Analytics Optimizer

POWER7 Blades

IBM System x Blades

IBM WebSphere DataPowerIntegration Appliance XI50 for zEnterprise

HMCs required for Unified Resource Manager

Additional zBX owned HMC required if System maintained by Third Party

zBX



1313

Internal

Batteries

(optional)

Power

Supplies

I/O Drawer

2 x Processor Drawers, Memory

& HCAs

FICON & ESCON®

FQC

Ethernet cables for internal System LAN

connecting Flexible Service Processor

(FSP) cage controller

cards (not shown)

PCIe I/O drawers

Rear View Front View

2 x Support

Elements

z114 Model M10 – Under the covers



14

CEC Drawers Placement

14



15

One z10 BC DrawerTwo z114 Drawers (Model M10)

� System resources split between 2 drawers

(Model M10)� Second processor drawer (Model 10) for:

– Increased specialty engine capability– Increased memory capability– Increased I/O capability

•More coupling links than z10 BC•More I/O features than z10 BC

Note: Unlike the z196 Books, add/remove/repair of the processor drawer is disruptive

Processor / Memory Subsystem Drawers (Model M05 and M10)



16

z114 Processor drawer packaging

PU SCM

SC SCM

PU SCM

HCA

HCA

HCA

HCA

inter procdrawer connect

2 x I/O Fanouts

2 x OSC

2 x FSP

2 x I/O Fanouts

2x DCA

10 x MemoryDimms

Two PU Single Chip Modules (SCMs) & One Storage Control (SC) SCM

Drw to DrwInterconnect



17

z114 Processor drawers Infrastructure (M10)

Processor drawer #2

Processor drawer #1

HCA2 or 3s OSC FSP HCA2 or 3s

Interconnect (in Brown) - C-shape flex cable SC to SC SMP Signals and Oscillator (OSC) Signals

DCA Power



18

Oscillator / Fabric Flex Cable

18

The “Oscillator / Fabric Flex Cable” which connects Drawer 0 with Drawer 1

Used to transport Oscillator signals (from Drawer 0 to Node1) and to interconnect the Fabric buses



19

z114 Processor drawer #1 (front) with fanout cards

2 Flexible Support Processor (FSP) card slots providing support for the Service Network subsystem

4 fanout card slots providing support for the I/O subsystem and/or coupling

2 x oscillators with Pulse Per Second (PPS)



20

z114 Processor drawer #2 (front) with fanout cards

2 Flexible Support Processor (FSP) card slots providing support for the Service Network subsystem

4 fanout card slots providing support for the I/O subsystem and/or coupling

2 x oscillators – No PPS



2121

Master Oscillator Card for Model 05 with PPS connector

BNC Connector for Pulse Per Second

NTP Ethernet LAN

Note: If the NTP LAN goes through the HMC, PPS can not be used.



22

z114 Model Structure and Upgrades

22

Model CPs

IFLs

Unassigned IFLs

zAAPs zIIPs ICFsStd.

SAPs

Add’I

SAPsSpares

M05 0-5 0-5 0-2 0-2 0-5 2 0-2 0

M10 0-5 0-10 0-5 0-5 0-10 2 0-2 2

� Model structure based on number of drawers

� M05 sparing based on prior Business Class (BC) offerings – no dedicated spares

� M10 sparing based on Enterprise Class (EC) offerings – dedicated spares

� SAP and PU Allocation/Sparing in the M10

� Default assignment is one SAP per drawer; one Spare per drawer. Spill and fill CP low to high; spill and fill specialty engines high to low

� Two defective PUs may cause the default assignment to spill and fill into the second processor drawer. LPAR has the capability to request PU of a specified type to be grouped together in a book/drawer (i.e. LPAR may change the default assignment)

� Disruptive upgrade from M05 to M10

– No model downgrades

� Upgrades from z9 BC and z10 BC into either model M05 or M10

� Only the M10 will upgrade to z196 Model M15 (Air cooled only)



23

z114 PU Conversions

23

To

From……….. IFL

Unassigned

IFLICF zAAP zIIP

Additional SAP

IFL x Yes Yes Yes Yes Yes

Unassigned IFL Yes x Yes Yes Yes Yes

ICF Yes Yes x Yes Yes Yes

zAAP Yes Yes Yes x Yes Yes

zIIP Yes Yes Yes Yes x Yes

Additional SAP Yes Yes Yes Yes Yes x

Specialty engines are converted "any to any", including Additional SAPs and unassigned IFLs.

CP capacity conversions are handled separately from the specialty engines under the following rules:

� CP Capacities are converted "any to any"...allow MIPS upgrades, downgrades and horizontal movement.

� CP Capacities are handled separately from specialty engines, meaning that in the conditions where CP capacity on the target machine is constructed using a lesser amount of CPs, the remaining CPsare not available to be converted to specialty engines.

.



24

z114 Conversions (continued)

24

Any to Any is allowed (upgrades or downgrades)– CP Capacities are handled separately from specialty engines.

• When CP capacity on the target machine is ordered using fewer CPs, the remaining CPs are not available to be converted to specialty engines without a fee.

F01 F02 F03 F04 F05

E01 E02 E03 E04 E05

D01 D02 D03 D04 D05

C01 C02 C03 C04 C05

B01 B02 B03 B04 B05

A01 A02 A03 A04 A05

1-way 2-way 3-way 4-way 5-way

Examples:

2-way to 1-way capacity downgrade

The PU in the green column must maintain its high water mark and can not be purchased as a specialty engine.

2-way to 1-way capacity upgrade

The high water mark is now in the red column

and the PU in the green column can be used during the purchase of a specialty engine.



25

IBM System z Business Class Configuration Comparisons (1 of 3) z9 BC

R07

z9 BC

S07

z10 BC

E10

z114

M05

z114

M10

Uniprocessor Performance 470 MIPS 673 MIPS 782 MIPS

z/OS Capacity 26-172 MIPS 193-1748 MIPS 26-2760 MIPS 26 - 3139 MIPS

Total System Memory 64 GB 256 GB 128 GB 256 GB

Configurable Engines 7 7 10 5 10

Dedicated Spares 0 0 0 0 2

Configurable CPs 1-3 0-4 0-5 0-5

LPARS/LCSS 15/1 30/2 30/2 30/2

HiperSockets 16 16 32

I/O Cages/Drawers 1 1 Up to 4 Up to 3(1) Up to 3(1)

I/O slots per Cage/Drawers 16 28 8 8/32(2)

FICON® Channels 64 112 128 128(3)

OSA Ports (10GbE/1GbE) 16/32 24/48 48/96 48/96

ESCON® Channels 240 420 480 240(4)

STI (z9), IFB (z10) BandwidthPCIe (z114 BC) Bandwidth

2.7 GB/sec 6.0 GB/sec 6.0 GB/sec8.0 GB/sec

ICB-4/ISC-3/PSIFB 16/48/0 12/48/12 0(5)/48/8 -16(6) 0(5)/48/16 - 32(7)

zIIP/zAAP Maximum Qty 3 3 5 2 5

IFL Maximum Qty 6 7 10 5 10

Capacity Settings 20 53 130 130 130

Upgradeable Upgrade to z10 and z114 Upgrade to z114 Upgrade From M10 to z196 (M15, Air cooled only)

See next chart for foot notes



26

(1) Up to 3 channel drawers standard, a combination of I/O drawers and PCIe I/O drawers as defined. Up to 4 I/O drawers via RPQ

(2) 8 card slots per I/O drawer, 32 per PCIe I/O drawer

(3) FICON count is based on 2 PCIe I/O drawers or 4 I/O drawers

(4) Limit of 240 ESCON channels is consistent with Statement of Direction

(5) Limit of 0 ICB-4 links is consistent with Statements of Direction

(6) 8 ports of 12x, 16 ports of 1x PSIFB links available on model M05 based on 4 HCA capabilities

(7) 16 ports of 12x, 32 ports 1x PSIFB links available on model M10 based on 8 HCA capabilities

M05 M10

I/O Drawer PCIe I/O Drawer

I/O Drawer PCIe I/O Drawer

0 0 0 0

0 1 0 1

0 2 0 2

1 0 1 0

1 1 1 1

1 2

2 0 2 0

2 1 2 1

3 0 3 0

4 0

3rd and 4th I/O drawers are offered via an RPQ

Notes for Configuration comparisons chart (2 of 3)



27

IBM System z Config Comparisons, z114 vs. z196 Model M15 (3 of 3)

See previous chart for foot notes

Upgrade from M10 to z196 (M15, MRU cooled only)

130

5

2

0(5)/48/8 -16(6)

6.0 GB/sec8.0 GB/sec

240(4)

48/96

128(3)

8/32(2)

Up to 3(1)

32

30/2

0 - 5

5

120 GB

26 - 3139 MIPS

782 MIPS

z114

M05

130

10

5

0(5)/48/16 - 32(7)

Up to 3(1)

10

248 GB

z1114

M10

z196 M32, M49, M66, M80MRU and Water Cooled

60

15

7

0(5)/48/16 - 32

6.0 GB/sec8.0 GB/sec

240

48/96

256

28/8/32(2)

3/6

32

60/4

0 - 15

15

704 GB

240 – 13,660 MIPS

1200 MIPS

z196

Model M15

Total System Memory

ICB-4/ISC-3/PSIFB

IFL Maximum Qty

I/O slots per Cage/Drawers

I/O Cages/Drawers

Upgradeable

Capacity Settings

zIIP/zAAP Maximum Qty

STI (z9), IFB (z10) BandwidthPCIe (z114/z196) Bandwidth

ESCON® Channels

OSA Ports (10GbE/1GbE)

FICON® Channels

HiperSockets

LPARS/CSS

Configurable CPs

Configurable Engines

z/OS Capacity

Uniprocessor Performance



28


Processor Unit Design



29

Heatsink

PU Chip

LGA

Processor drawer with

2 PU SCMs, 1 SC SCM

Memory DIMMS and DCA Power

z114 PU/SC Single Chip Module (SCM) Components

Thermal Interface Material



30

z114 SCM Vs z196 MCM Comparison – Same PU and SC Chip

Single PU Chip

without heatsink

� MCM

– 96mm x 96mm in size

– 6 PU chips per MCM

• Quad core chips with 3 or 4 active cores• PU Chip size 23.498 mm x 21.797 mm

– 2 SC chips per MCM• 96 MB L4 cache per chip

• SC Chip size 24.427 mm x 19.604 mm

– Up to 4 MCMs for System

z196 Multi Chip Module (MCM)

� PU SCM

– 50mm x 50mm in size – fully assembled– Quad core chip with 3 and 4 active cores

– 2 PU SCMs for M05 and 4 PU SCMS for M10

– PU Chip size 23.498 mm x 21.797 mm

� SC SCM

– 61mm x 61mm in size – fully assembled– 1 SC SCM for M05, 2 SC SCMs for M10

– 96 MB L4 cache per chip

– SC Chip size 24.427 mm x 19.604 mm

z114 SCMs

Single SC Chip

without heatsink

PU 0PU 2

SC 0SC 1

PU 1

S00

S01

PU 5PU 3 PU 4

S10

S11



31

z114 PU Chip Details

� 12S0 45nm SOI Technology

– 13 layers of metal

� Chip Area – 512.3mm^2

– 23.5mm x 21.8mm

– 8093 Power C4’s

– 1134 signal C4’s

� 1.4 Billion Transistors

� 4 Core Design

– 3.8 GHz

– Full out-of-order capability

– 1.5MB L2 cache

– 12MB shared L3 cache

� Two Co-Processors

– Data Compression & encryption

� Memory controller

– Supports 5-channel DDR3 RAIM

� GX Controller

� 3 PLLs, 5+ frequency domains

� 5 unique chip voltage supplies

L2(1.5MB)

CoPMCU

L2(1.5MB)

L2(1.5MB)

CoP GX

L2(1.5MB)

L3_0 Controller

L3_1 Controller

L3B

L3B

Core 0

Core 1

Core 2

Core 3

MCIOs

MCIOs

GXIOs

GXIOs

L3 Cache(12MB)

L3 Cache(12MB)



32

z114 SC Chip Details

�12S0 45nm SOI Technology–13 layers of metal

�Chip Area – 478.8mm^2 –24.4mm x 19.6mm–7100 Power C4’s–1819 signal C4’s

32

�1.5 Billion Transistors

–1 Billion cells for eDRAM

�eDRAM Shared L4 Cache–96 MB per SC chip

�6 CP chip interfaces

–2 used

� 3 Fabric interfaces

– 1 used

� 2 clock domains

� 5 unique chip voltage supplies



33

z114 System Topology

SC00

SC10

CP02

CP12

CP01

CP11

SC0 CP2 CP1

SC0

SC0

SC0CP1CP2

GX1 GX0 GX1 GX0

GX1

GX06 GX02

GX17 GX11

FP2

FP2

Drawer #1

GX0

GX07 GX01

PSI Redundant

5 DIMMs 5 DIMMs

5 DIMMs

PSI Redundant

5 DIMMs

GX1 GX0

GX16 GX12 Drawer #2

Color legend:

Same as in z196 models

Different

� 2-drawer System

� Up 14 Processing Units at 3.8 GHz

� 3 or 4 cores active, same as z196

� Up to 16 HCA ports (M10)

� Support for Legacy and PCIe HCAs

� Up to 4 Memory Controllers

� Up to 20 DIMM slots

� 5-channel Memory RAIM protection

� 4-level cache hierarchy– Split 64K I-L1 and 128K D-L1 per core– 1.5 MB L2 per core– 12 MB on chip L3 half that of z196– 96 MB node level L4 half that of z196

� eDRAM (embedded) caches – L3, L4 and SP Key Cache



34

z114 Internal System Structure – Compared to the z196

Book

Book

Book

Book

FBC

FBC

FBC

FBC

FBC

FBC

PU1PU2

CP4 CP5CP3

SC0SC1

GX5PSI

GX4GX3

Mem1

GX1

Mem0

GX0

FBC2

FBC0FBC1FBC2

Mem2

GX2

GX6 GX7

BOOK

CP0

Topology

FBC0FBC1

� z114 is built from subset of the z196 design and chipset– Half the L3 and L4 cache sizes– Half the chip to chip bandwidth



35

z114 Cache Hierarchy

35



3636

Cache Hierarchy – z196 (4 books) vs. z114 (2 processor drawers) Comparison

L2L1

L2L1

L2L1

L2L1

L1: 64KI + 128KD8w DL1, 4w IL1256B line size

L2 Private 1.5MB Inclusive of L1s12w Set Associative256B cache line size

L3 Shared 12MB Inclusive of L2s12w Set Associative256B cache line size

L4 96MB Inclusive24w Set Associative256B cache line size

z114

2 L4 Caches

96MB

Shared eDRAM L4

2 L3s,7 L1 / L2s

L2L1

12MB ShreDRAM L3

L2L1

L2L1

L2L1

L2L1

L2L1

L2L1

L2L1

L1: 64KI + 128KD8w DL1, 4w IL1256B line size

L2 Private 1.5MB Inclusive of L1s12w Set Associative256B cache line size

L3 Shared 24MB Inclusive of L2s12w Set Associative256B cache line size

L4 192MB Inclusive24w Set Associative256B cache line size

z196

4 L4 Caches

192MB

Shared eDRAM L4

6 L3s,24 L1 / L2s

L2L1

24MB ShreDRAM L3

L2L1

L2L1

L2L1

24MB ShreDRAM L3

12MB ShreDRAM L3



37

z114 Cache Structure

� On the processor chip there are 3 levels of cache

– L1 I Instruction Cache of 64KB – L1 D Data Cache of 128KB – L2 Private Cache of 1.5MB (per core) – L3 Share Cache of 12MB

� L3 Cache– 12MB eDRAM– Gear Ratio: 2:1 dataflow, 4:1 control flow – 2 address slices, address bit 55 selection – 16 octword cache interleaves – Address and Data bit stacking – Store-in cache, store dual-piped per L3 slice – 16B 2:1 store/fetch buffers planned between

L2&L3 – Extension of z10 cache coherency scheme – CP/SC merged transaction bussing

(addr/control + data) – IO fetch reserved cache slots

37



38

z114 Cache Structure – L4 Cache

� L4 directory is used to filter snoop traffic from the remote drawer

� Enhanced Fabric Protocol for improved latency and cache management

� Horizontal Cache Persistence for local cache extension

� LRU install on available cache slots in the remote L4 cache

� 96 MB eDRAM per SC chip, 24 Set Associatively.

� Gear ratio: 2:1 dataflow, 4:1 control flow. Synchronous fabric port

� 2 address sliced L4 Directories within the SC chip, address bit 54 selection

� 8 256B cache banks per logical directory

� z10 derived cache and storage coherency scheme

38



39

zEnterprise System PU core

� Each core is a superscalar, out of order processor with these characteristics:

– Six execution units

• 2 fixed point (integer), 2 load/store, 1 binary floating point, 1 decimal floating point

– Up to three instructions decoded per cycle (vs. 2 in z10)

– 211 complex instructions implemented by multiple internal operations

• 246 of the most complex z/Architecture instructions are implemented via millicode

– Up to five instructions/operations executed per cycle (vs. 2 in z10)

– Execution can occur out of (program) order

• Memory address generation and memory accesses can occur out of (program) order

• Special circuitry to make execution and memory accesses appear in order to software

– Each core has 3 private caches

• 64KB 1st level cache for instructions, 128KB 1st level cache of data

• 1.5MB L2 cache containing both instructions and data



40

� OOO yields significant performance benefit for compute intensive apps through

–Re-ordering instruction execution

• Later (younger) instructions can execute ahead of an older stalled instruction

–Re-ordering storage accesses and parallel storage accesses

� OOO maintains good performance growth for traditional apps

zEnterprise System Out of Order (OOO) Value

L1 miss

Instrs

1

2

3

4

5

Time

In-order core execution Out-of-order core execution

L1 miss

TimeExecution

Storage access



41

zEnterprise System Out of Order Detail

� Out of order yields significant performance benefit through

– Re-ordering instruction execution

• Instructions stall in a pipeline because they are waiting for results from a previous instruction or the execution resource they require is busy

• In an in-order core, this stalled instruction stalls all later instructions in the code stream

• In an out-of-order core, later instructions are allowed to execute ahead of the stalled instruction

– Re-ordering storage accesses

• Instructions which access storage can stall because they are waiting on results needed to compute storage address

• In an in-order core, later instructions are stalled

• In an out-of-order core, later storage-accessing instructions which can compute their storage address are allowed to execute

– Hiding storage access latency

• Many instructions access data from storage

• Storage accesses can miss the L1 and require 10 to 500 additional cycles to retrieve the storage data

• In an in-order core, later instructions in the code stream are stalled

• In an out-of-order core, later instructions which are not dependent on this storage data are allowed to execute



42

zEnterprise System Compression and Cryptography Accelerator

� Data compression engine

– Static dictionary compression and expansion

– Dictionary size up to 64KB (8K entries)

• Local 16KB cache per core for dictionary data

� CP Assist for Cryptographic Function (CPACF)

– Enhancements for new NIST standard

– Complemented prior ECB and CBC symmetric cipher modes with XTS, OFB, CTR, CFB, CMAC and CCM

– New primitives (128b Galois Field multiply) for GCM

� Accelerator unit shared by 2 cores

– Independent compression engines

– Shared cryptography engines

Core 0 Core 1

IB IBOB OBTLBTLB

2nd LevelCache

CmprExp

CmprExp

16K 16K

CryptoCipher

Crypto Hash



43

z196 and z114 Comparison

z114z196 Bi-NodalFully InterconnectedNodal Topology

2 drawers4 booksNumber of books / drawers

1496Max # of PU’s

2 PUs, 1 SC6 PUs, 2 SCsNumber of chips per Book/Drawer

64KB I, 128KB D64KB I, 128KB D1st level cache (PU core)

1.5MB private per core (L2)1.5MB private per core (L2)2nd level cache (PU core)

12MB Shared per 4 cores (L3)24MB Shared per 4 cores (L3)3rd level cache (PU nest)

96MB Shared per node (SC L4)192MB Shared per node (SC L4)4th level cache (SC)

1 Main, 1 Store2 Main, 2 Store# of PU L3 pipes

1 Main1 Main (per SC)# of SC cache pipes

x81 DIMMsx81 DIMMsMemory DIMMs

256 Gigabytes on 20 DIMMs3 Terabytes on 120 DIMMsMax System Memory Size



44


Memory Design



45

z114 RAIM Memory

� Memory technology introduced on the z196 is used on the z114

– Redundant Array of Memory (RAIM) which in the Disk industry is known as RAID

– Protection from Unscheduled Incident Repair Actions (UIRAs)caused by a DIMM failure

• DIMM failures include all components on the DIMM

• Portions of the memory controller or card failure isolated to one memory channel

� Flexible memory option not available on z114

– Used for Enhanced Book Availability on z196

DATA

CHECK

DATA

CHECKECCRAIM Parity

Level 4 Cache

Key Cache

MCU 0

Key Cache

MCU 1

Extra column provides RAIM

function

16B 16B 16B 16B

2B 2B



46

z114 Memory Topology

� 10 DIMMSs/drawer

� No cascading

� All DIMMs in drawer must be the same

� DIMMs plugged in quints– 5 DIMMs

46



47

Physical base memory

• M05 = 40GB, 24GB for concurrent ordering before a physical (disruptive) card change (minus HSA/RAIM)

• M10 = 80GB, 56GB for concurrent upgrade ordering before a physical (disruptive) card change (minus HSA/RAIM)

FC1903 - Memory Capacity Increment

• Used to determine memory billing for new memory being added server configurations

• Indicates 8GB (or 32 GB in larger configurations) LICC’d increments of Memory Capacity.

• Charged by increments when Plan Ahead memory is enabled (conversion)

• Subsequent memory upgrade orders will use up the Plan Ahead memory first

FC 1904 - 16GB Feature Converted Memory

• Used to determine billing for memory on MES upgrades coming from System z9 and z10 servers..

FC1993 – Preplanned Memory

• Tracks the quantity of 8GB physical increments.

• Charged (half price) when physical memory is installed

z114 Memory Details

Model Increment (GB) Standard (GB) Plan Ahead (GB)

M05 8 8 – 120 24 – 120

M10 8 16 – 120 56 – 120

M10 32 152 - 248 152 - 248



48

z114 Model M05 Memory Features

� Plan Ahead Memory

– Pre-plugged memory based ontarget capacity specified bythe customer.

– Enabled by LICCC, concurrently.– FC1993 tracks the quantity of 8GB

physical increments.• Charged (half price) when physical

memory is installed– FC1903 generally indicates 8GB

(or 32 GB in larger configurations)LICC’d increments of Memory Capacity.

• Charged by increments when Plan Ahead memory is enabled• Subsequent memory upgrade orders will use up the

Plan Ahead memory first

10 x 4 GB DIMMs

10 x 8 GB DIMMs10 x 16 GB

DIMMs

Feature Size Feature Size Feature Size

8 32 64

16 40 72

24 48 80

56 88

96

104

112

120

Physical memory upgrades are DISRUPTIVE



49

z114 Model M10 Memory Features

4 GB/4GB4GB/8GB8GB/4GB

8GB/8GB4GB/16GB16GB/4GB

8GB/16GB16GB/8GB

16GB/16GB

Feature Size Feature Size Feature Size Feature Size Feature Size Feature Size

16 64 96 152 184 216

24 72 104 248

32 80 112

40 88 120

48

56

Physical memory upgrades are DISRUPTIVE

32 GB increments



50

z114 Memory Offerings

FC GB Increment M05 M10 (2 CPC drawers)

Dial Max Dimm (GB) # plugged Dial Max Dimm (GB) # plugged

3609 8 8 24 4 10 N/A N/A N/A

3610 16 8 4 10 56 4/4 10/10

3611 24 8 4 10 4/4 10/10

3612 32 8 56 8 10 4/4 10/10

3613 40 8 8 10 4/4 10/10

3614 48 8 8 10 4/4 10/10

3615 56 8 8 10 4/4 10/10

3616 64 8 120 16 10 88 4/8 10/10

3617 72 8 16 10 4/8 10/10

3618 80 8 16 10 4/8 10/10

3619 88 8 16 10 4/8 10/10

3620 96 8 16 10 120 8/8 10/10

3621 104 8 16 10 8/8 10/10

3622 112 8 16 10 8/8 10/10

3623 120 8 16 10 8/8 10/10

3624 152 32

N/A

152 4/16 10/10

3625 184 32 184 8/16 10/10

3626 216 32 248 16/16 10/10

3627 248 32 16/16 10/10

Memory upgrades within the same color (except white)are concurrent without the need for Memory Plan Ahead.



51

System z HSA Comparisons

� HSA significantly larger than pre-z10 Servers

� Fixed HSA, does not affect customer purchased memory

� Size of HSA on prior Servers (dependant on defined configuration)

– Multiprise® 2000 From 12 MB up to 40 MB

– 9672 G4 From 48 up to 64 MB

– Multiprise 3000 From 38 MB up to 136 MB

– 9672 G5/G6 From 64 MB up to 192 MB

– z800 From 160 MB up to 256 MB

– z900 From 288 MB up to 512 MB

– z890 From 768 MB up to 1.9 GB

– z990 From 1 GB MB up to 2 GB

– z9 BC From 896 MB up to 2.7 GB

– z9 EC From 1.2 GB up to 4.2 GB

– z10 BC 8 GB – Fixed

– z10 EC 16 GB – Fixed

– z196 16 GB – Fixed

– z114 8 GB - Fixed

� HSA Estimator on Resource Link not relevant



52


I/O Infrastructure



53

Glossary for System z I/O on z196 and z114

Acronym Full Name Description / Comments

N/A I/O drawerI/O drawer introduced with z10 BC and also supported on z196 and z114; has 8 I/O card slots

N/A I/O cageI/O cage available since z900 (not supported on z10 BC or z114);has 28 I/O card slots

N/A PCIe switchIndustry standard PCIe switch Application-specific integrated circuit(ASIC) used to fanout (or multiplex) the PCI bus tothe I/O cards within the PCIe I/O drawer

N/A PCIe I/O drawer New I/O drawer that supports PCIe bus I/O infrastructure; has 32 I/O card slots

PCI-IN PCIe interconnectCard in the PCIe I/O drawer that contains the PCIe switch ASIC z10 uses IFB-MP; z9 uses STI-MP

N/A PCIe fanout

Card on front of processor book that supports PCIe Gen2 bus; used exclusively to connect to the PCIe I/O drawer; PCIe fanout supports FICON Express8S and OSA-Express4SUsed instead of an HCA2-C fanout for I/O which continues to support the cards inthe I/O cage and I/O drawer

HCA3or

HCA3-O LR

HCA3-O LR fanoutfor 1x IFB

For 1x InfiniBand at unrepeated distances up to 10 km; 5 Gbps link data rate; 4 ports per fanout; may operate at 2.5 Gbps or 5 Gbps. Based upon capability of DWDM. Exclusive to z196 and z114; can communicate with an HCA2-O LR fanout; third generation Host Channel Adapter

HCA3or

HCA3-O

HCA3-O fanout for 12x IFB

For 12x InfiniBand at 150 meters; supports 12x IFB and 12x IFB3 protocols; increased service times when using 12x IFB3 protocol; 6 GBps link data rate; two ports per fanout; can communicate with an HCA2-O fanout on z196 or z10; cannot communicate with an HCA1-O fanout on z9; third generation Host Channel Adapter



54

Glossary for System z InfiniBand Coupling

Acronym Full name Comments

AID Adapter identification HCA fanout has AID instead of a PCHID

CIB Coupling using InfiniBand CHPID type z196, z114, z10, System z9

HCA Host Channel Adapter Path for communication

PSIFB Parallel Sysplex using InfiniBand InfiniBand Coupling Links

12x IFB 12x InfinBand 12 lanes of fiber in each direction

1x IFB 1x InfiniBand Long Reach - one pair of fiber

12x IFB3 12x InfiniBand3Improved service times of 12x IFB on HCA3-O

Type z10 z196/z114

HCA1-O fanout N/A (z9 only) N/A (z9 only)

HCA2-C fanoutCopper – Connects to I/O Cage (z10

EC only) or I/O DrawerCopper - Connects to I/O Cage (z196

only) or I/O Drawer

HCA2-O fanout Optical - Coupling 12x InfiniBand Optical - Coupling 12x InfiniBand

HCA2-O LR fanout Optical - Coupling 1x InfiniBand Optical - Coupling 1x InfiniBand

PCIe fanout N/A Copper - Connects to PCIe Drawer

HCA3-O fanout N/A Optical - Coupling 12x InfiniBand

HCA3-O LR fanout N/A Optical - Coupling 1x InfiniBand

MBA fanout Copper - Coupling (ICB-4) N/A



55

System z I/O Interface Evolution

eSTI(IBM)

InfiniBandDDR

(Industry Standard)

Extended Link(processor to IO Cage)

z9

IO Cage Backplane

mSTI(IBM)

mSTI(IBM)

ISC(IBM)

InfiniBandDDR

(Industry Standard)

Coupling Link

z10

z196

z196z114

Sep 2011

PCI Express(Ind. Std.)

PCI Express(Ind. Std.)

InfiniBandDDR

(Ind. Std.)



56

STIz990/z890

STIz9

InfiniBandz10/z196/z114

STIz900/z800

STI: Self-Timed Interconnect

6 GBps

2.7 GBps

2 GBps

1 GBps

System z I/O Subsystem Internal Bus Interconnect Speeds (GBps)

PCIez196/z114Sep 2011

8 GBps



57

PCIe Internal I/O Infrastructure

� System z continues the I/O evolution with a new internal I/O infrastructure change on

the platform to continue to support the industry direction for new, high performance I/O

– Increase the internal bandwidth of I/O infrastructure backbone from InfiniBand (6 GBps) to PCIe (8 GBps)

– Allow for future I/O performance growth requirements– Provides future power management capability to reduce overall power consumption– Adopt industry standard technology (PCIe)

• System z strategic direction to utilize industry standards without compromising robustness and System z qualities of service.

• Enables improved I/O capability and flexibility by leveraging the technology investments of the industry

– Allows for the future possibility of integration of industry standard, native PCIe adapters and accelerators



58

PCIe I/O drawer 1 Of 2

� Compact

– Two PCIe I/O drawers occupy the same space as one I/O cage

� Increased granularity

– Two FICON Express8S channels per feature– One or two OSA-Express4S ports per feature

� I/O card density – 14% more capacity– 32 I/O card slots

� Peripheral Component Interconnect Express (PCIe Gen2) – 8 GBps bus – Infrastructure from processor book/drawer to I/O cards

� Reduced power consumption

� Designed for improved RAS

– Symmetrical, redundant cooling across all cards and power supplies– Temperature monitoring of critical Application-specific integrated circuits (ASICs) – Concurrent add/repair of PCIe I/O drawer and adapters



59

PCIe I/O drawer 2 of 2

� Designed to

– Support concurrent add and repair– Support for a new industry standard I/O

• Attachment of select industry standard PCIe cards– Provide a balanced system – Designed to provide improved performance for new and traditional workloads– Lower power requirements while increasing connectivity for all workloads

� Increased infrastructure bandwidth

– I/O bus infrastructure data rate up to 8GBps• Positioned for higher bandwidth in the future

� Power Save Mode– Power can be reduced on unused ports and unused area of the drawer

� Fewer ports per I/O card, but support for four times as many slots.– Intended for new I/O features (FICON Express8S, OSA-Express4S)– Legacy I/O technology (ESCON, ISC-3, PSC, Crypto Express3 and carry forward) still

supported on I/O Drawer (FC4000)



60

z196/z114 Internal I/O Infrastructure Overview

� The z196/z114supports two different internal I/O infrastructures

– The z196 supports:• InfiniBand based infrastructure for I/O cages and I/O drawers • PCIe based infrastructure for PCIe I/O drawers with a new form factor

drawer and I/O features

– The z114 supports:• InfiniBand based infrastructure for I/O drawers • PCIe based infrastructure for PCIe I/O drawers with a new form factor

drawer and I/O features



61

z114 and z196 at GA2 support two different internal I/O infrastructures

� The InfiniBand I/O infrastructure first made available on z10

– InfiniBand fanouts supporting the current 6 GBps InfiniBand I/O interconnect– InfiniBand I/O card domain multiplexers with Redundant I/O interconnect in:

• The 14U, 28-slot, 7-domain I/O cage (z196 only)• The 5U, 8-slot, 2-domain IO drawer (z114 and z196)

– Selected legacy I/O feature cards• Carry forward and new build

� And a new PCI Express 2 I/O infrastructure

– PCIe fanouts supporting a new 8 GBps PCIe I/O interconnect– PCIe switches with Redundant I/O interconnect in for I/O domains in a new 7U, 32-slot,

4-domain I/O drawer (z114 and z196 GA2)– New FICON Express8S and OSA-Express4S I/O feature cards

• Based on selected industry standard PCIe I/O• Designed to:

– Reduce purchase granularity (fewer ports per card)

– Improve performance

– Increase I/O port density

– Save energy



62

8-slot I/O Drawer (Introduced with z10 BC)

� Supports all z10 BC and z196 GA1 I/O and

Crypto Express3 cards

� Supports 8 I/O cards, 4 front and 4 back, horizontal orientation, in two 4-card domains (shown as A and B)

� Requires two IFB-MP daughter cards, each connected to a 6 MBps InfiniBand

interconnect to activate both domains.

� To support Redundant I/O Interconnect

(RII) between the two domains, the two interconnects must be from two different

InfiniBand fanouts. (Two fanouts can support two of these drawers.)

� Concurrent add, repair.

� Requires 5 EIA Units of space

(8.75 inches ≈ 222 mm)

DCA

I/O - B

I/O - A

IFB-MP A

I/O - B

I/O - A

IFB MP B

DCA

I/O - A

I/O - B

I/O - B

I/O - A

Front

Rear

5 U

5 U

RII



63

7U

7U

Rear

Front� Supports only the new PCIe I/O cards

introduced with z114 and z196 GA2.

� Supports 32 PCIe I/O cards, 16 front and

16 rear, vertical orientation, in four 8-card domains (shown as 0 to 3).

� Requires four PCIe switch cards (��), each connected to an 8 MBps PCIe I/O

interconnect to activate all four domains.

� To support Redundant I/O Interconnect(RII) between front to back domain pairs 0-1 and 2-3 the two interconnects to

each pair must be from 2 different PCIefanouts. (All four domains in one of

these cages can be activated with two fanouts.)

� Concurrent field install and repair.

� Requires 7 EIA Units of space

(12.25 inches ≈ 311 mm)

Domain 0 Domain 2

� �

� �

Domain 3 Domain 1

New 32 slot PCIe I/O drawer for z114 and z196



64

zEnterprise I/O Drawer comparison

• 8 horizontal slots• 4 Front/4 Back• 32 FICON Ports

• 2 DOMAINS

• 32 vertical slots

• 16 Front/16 Back• 64 FICON Ports• 4 DOMAINS

PCIe I/O DrawerFC4003

I/O DrawerFC4000

(same as z10 BC)

22.75 in

8.7

5 i

n30

in

midplane

22.75 in12

.25

.75

in

30 in

midplane

I/O Cards I/O Cards

FANOUT &

FAILOVER

PROCESSOR

FANOUT &

FAILOVER

HCA

PROCESSOR

PROCESSOR DRAWER

I/O Infrastructure

PROCESSOR DRAWER

FAIL-

OVER

GX I/O Bus

EXTENDED LINKS

CAGE BUSES

GX I/O Bus

DRAWER

HCA



65

Top views – I/O cage, I/O drawer, PCIe I/O drawer Domains

Front – 16 Rear – 16

FSP-1, 1

PCIeinterconnect

FSP-1, 2

RII

PCIeinterconnect

PCIeinterconnect

PCIeinterconnect

I/O drawer – 8 slots2 I/O domains

I/O cage – 28 slots7 I/O domains

Front – 16 Rear – 12

16D

15C

14D

13C

12D

11C

10D

9C

8B

7A

6B

5A

4B

3A

2B

1A

28F

27E

26F

25E

24F

23E

22F

21E

20G

19G

18G

17GW

I

R

N

G

IFB-MP B

C D

G G’

E F

DCA 1

I/O-A

I/O-B

I/O-B

I/O-AI/O-A

I/O-B

DCA 2

I/O-B

I/O-A

IFB-MP B

IFB-MP A

Front - 4 Rear - 4

PCIe I/O drawer – 32 slots4 I/O domains

RIIRIIRIIRII

Redundant I/O interconnect = RII

3

3

3

3

0

0

0

0

0

0

0

0

2

2

2

2

1

1

1

1

1

1

1

1

3

3

3

3

2

2

2

2



66

I/O Drawer

� No changes to as used on 2098,and

2817

� 7u I/O drawer

� The addition or removal of a drawer is a concurrent MES

� IFB-MP cards as used on I/O Cage

� DCA's are unchanged from 2098 z10-BC

� Each houses 8 I/O cards over 2

domains

� Min/Max is 0/4 for new build and 0/4 for

MES

� All card assemblies in the can be added

concurrently.

66



67

PCIe I/O Drawer

� 7u I/O drawer

� The PCIe drawer minimum/maximum is 0/2

� The addition or removal of a PCIedrawer is a concurrent MES

� Each PCIe houses 32 I/O cards over 4 domains

� The PCIe drawer is for new PCIe based I/O only. ESCON, ISC-3 and PSC24V

cannot be plugged in a PCIe drawer

� All card assemblies in the PCIe can be

added concurrently

67



68

PCIe

switch

OSA-Express4S10 GbE

PCIe

switchPCIe

switch

FICON Express8S

PCIe

switch

4 GB/s PCIe

Gen2 x8

…

IFB-MP IFB-MP

I/O D

raw

er 1

I/O

Dra

we

r 1

PCIe (8x)

Memory

SC1, SC0 (FBC)

PU PU

PU PU PU

PU

PCIe (8x)

ESCON

Channels

OSA-Express31000BASE-T

Ports

2GBpsmSTI

.

FP

GA

FP

GA

FP

GA

FP

GAESCON ESCON ESCON ESCON

HCA2-C (6x))HCA2-C (6x))

Processordrawer 0

IO Drawer 2

PCIe Fanout Support•1 PCIe drawer = 2 PCIe fanouts•2 PCIe drawers = 4 PCIe fanouts

HCA2-C Fanout Support•1 I/O drawer = 2 HCA2-C•2 I/O drawer = 2 HCA2-C•3 I/O drawer = 4 HCA2-C (RPQ)•4 I/O drawer = 4 HCA2-C (RPQ)

Max of 4 fanouts per processor

drawer !

z114 Fanout Usage for I/O (Mixed PCIe and I/O drawers)



69

PCIe

FANOUT

PCIe

FANOUT

Front

Domain 0 Domain 2

Domain 3 Domain 1

� Different PCIe Fanouts Support Domain Pairs:

– 0 and 1– 2 and 3

� Normal operation each PCIeinterconnect in a pair supports the eight I/O cards in its domain.

� Backup operation: One PCIeinterconnect supports all 16 I/O cards in the domain pair.

Rear

PCIe switch cards (��)

��

��

z114 M05 Redundant I/O Interconnect Example – One PCIe Drawer



70

z114 Fanout Usage for the I/O and PCIe drawers

Number and types of drawers Model 05 Model 10

Zero I/O drawers 0 0

One I/O drawer 2 x HCA2-C 2 x HCA2-C

One PCIe I/O drawer 2 x PCIe fanouts 2 x PCIe fanouts

Two I/O drawers 2 x HCA2-C 2 x HCA2-C

One I/O drawer + One PCIe drawer 2 HCA2-C & 2 x PCIe fanouts 2 HCA2-C & 2 x PCIe fanouts

Three I/O drawers 4 x HCA2-C 4 x HCA2-C

Two PCIe drawers 4 x PCIe fanouts 4 x PCIe fanouts

Two I/O drawers + One PCIe drawer 2 HCA2-C & 2 x PCIe fanouts 2 HCA2-C & 2 x PCIe fanouts

Four I/O drawers (RPQ) 2 x HCA2-C N/A – no drawer slots available

One I/O drawer + Two PCIe drawers N/A – not enough fanouts 2 HCA2-C & 4 x PCIe

Notes:

� Each HCA2-C and PCIe fanout has 2 ports

� Each I/O drawer has 2 I/O domains (4 I/O features per domain). All 4 domains are plugged to the HCA2-C

� Each PCIe drawer has 4 I/O domains (8 I/O features per domain). 2 domains at a time are plugged to the PCIe fanout

� 4th I/O drawer uses one of the processor drawer locations



71

Description F/C Ports Comments

HCA2-C copper fanout 0162 2 To I/O Drawers (FC4000)

HCA2-O 12x IB-DDR 0163 2 PSIFB Coupling (up to 150 meters)

HCA1-O 12x IB-SDR 0167 2 z9 PSIFB coupling to zEnterprise/z10

HCA2-O LR 1x IB-DDRCarry Forward only

0168 2 PSIFB Coupling – up to 10 km (unrepeated)*

PCIe copper fanout 0169 2 To PCIe I/O Drawers (FC4003)

HCA3-O LR 1x IFB 0170 4 PSIFB Coupling - up to 10 km (unrepeated)*

HCA3-O 12x IFB 0171 2 PSIFB Coupling 150 meters

HCA3-O can communicate with the z10 via HCA2-O

HCA3-O can not communicate with the z9

HCA2 on zEnterprise can communicate with z9 HCA1

*The 1X links support a distance of up to 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support.Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs), goingover 100 km requires RPQ 8P2263 or 8P2340

z114 HCA Fanout Cards – Summary



72

� Fanout cards – Two-port InfiniBand host channel adapters dedicated to function

– HCA2-C fanout – I/O Interconnect

• Supports FICON, ESCON, OSA, ISC-3 and Crypto Express3 cards in I/O drawer and I/O cage domains. Always plugged in pairs.

– HCA2-O fanout – 12x InfiniBand coupling links

• CHPID type – CIB for Coupling

– Fiber optic external coupling link – 150 m

– HCA2-O LR fanout – 1x InfiniBand coupling links –Long Reach (carrry forward only for z114)


• Fiber optic external coupling link – 10 km (Unrepeated), 100 km repeated

• PCIe fanout – I/O Interconnect

• Supports FICON Express8S, OSA-Express4S in PCIedrawer. Always plugged in pairs

– HCA3-O fanout – 12x InfiniBand coupling links


– Fiber optic external coupling link – 150 m

– HCA3-O LR fanout – 1x InfiniBand coupling links –Long Reach


• Fiber optic external coupling link – 10 km (Unrepeated), 100 km repeated

* Performance considerations may reduce the number of CHPIDsper port

HCA2-C IFB IFB

HCA2-O IFB IFB

Up to 16 CHPIDs – across 2 ports

HCA2-O LR IFB IFB


z196 and z114 Connectivity for Coupling and I/O - Summary

Up to 16 CHPIDs – across 2 ports*

HCA3-O IFB & IFB3 IFB & IFB3


HCA3-O LR IFB IFB IFB IFB

PCIe fanoutPCIe PCIe



73

All z114 Drawer Combinations

I/O Drawers

PCIe I/ODrawers

0 1 2 3 4

0

M05 M05 M05 M05 M05

M10 M10 M10 M10

1

M05 M05 M05

M10 M10 M10

2

M05

M10 M10

z114 Drawer Combinations



74


I/O Subsystem Enhancements



75

Three subchannel sets per LCSS (z196 only)

� Addition of a subchannel set of 64K devices to the existing two subchannel sets

� Value– Extends the amount of addressable storage for the largest System z customers– Provides a means to provide consistent device address definitions to help simplify

addressing scheme for congruous devices• Can utilize same device number in different subchannel sets

� The first subchannel set (SS 0) allow definitions of any type of device as is allowed today, (i.e. bases, aliases, secondaries, and those other than disk that do not implement the concept of associated aliases or secondaries)

� Second and third subchannel sets (SS1 and SS2) now be designated for use for disk alias devices (of both primary and secondary devices) and/or Metro Mirror secondary

devices only

� CHPID types: ESCON - CNC, FICON - FC (both native FICON and zHPF paths)

� Features: ESCON, FICON Express8, FICON Express4

� z/OS V1.12

� z/OS V1.10 and V1.11 with PTFs



76

IPL for alternate subchannel set

� Allows to IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to

subchannel set 0.

� Devices used early during IPL processing can now be accessed using subchannel set

1 or subchannel set 2. This is intended to allow the users of Metro Mirror (PPRC) secondary devices defined using the same device number and a new device type in an alternate subchannel set to be used for IPL, IODF, and standalone dump volumes

when needed.

� IPL from an alternate subchannel set is supported by z/OS V1.13, as well as V1.12 and

V1.11 with PTFs and applies to the FICON and zHPF protocols.

Note: SS2 for z196 only



77


I/O Connectivity Features



78

z114 I/O Features supported

I/O cage and I/O drawer

– Crypto Express3 (1P for z114 only)

– ESCON (240 or fewer)

– FICON Express8 (Carry forward or RPQ)

– FICON Express4 (Carry forward only for 4 port cards)

– FICON Express4-2C (Carry forward or RPQ – z114)

– ISC-3

– OSA-Express3 1000BASE-T (Includes 2P for z114)

– OSA-Express3 (Carry forward or RPQ)

• 10 GbE LR and SR, GbE LX and SX (2P for z114 only)

– OSA-Express2 (Carry forward only)

• GbE LX and SX, 1000BASE-T

– PSC (Carry forward or new build, no MES add)

PCIe I/O drawer

– FICON Express8S

• SX and 10 km LX

– OSA-Express4S

• 10 GbE LR and SR

• GbE SX and LX

8 slot I/O drawer

32 slot PCIe I/O drawer

Supported features

Note: PSIFB links are direct from the HCAs and not from the I/O cage or drawers



79

� I/O Channels

FICON (before FICON Express4)

FCV – ESCD Model 5 Bridge Card

� Networking

OSA-Express2 10 GbE LR

OSA-Express (pre OSA-Express2)

� Coupling Links

ICB-4 and earlier ICB

� Crypto

Crypto Express2 and earlier

� ETR

Sysplex Timer® (ETR) Attachment

Non-Supported Channel Types

z196 Non-Supported Features



80

FICON Express8S – SX and 10KM LX in the PCIe I/O drawer

� For FICON, zHPF, and FCP environments– CHPID types: FC and FCP

• 2 PCHIDs/CHPIDs

� Auto-negotiates to 2, 4, or 8 Gbps

� Increased performance compared to FICON Express8

� 10KM LX - 9 micron single mode fiber– Unrepeated distance - 10 kilometers (6.2

miles)– Receiving device must also be LX

� SX - 50 or 62.5 micron multimode fiber– Distance variable with link data rate and

fiber type– Receiving device must also be SX

� 2 channels of LX or SX (no mix)

� Small form factor pluggable (SFP) optics– Concurrent repair/replace action for each SFP

LX SX

FC 0409 – 10KM LX, FC 0410 – SX

OR

2, 4, 8 Gbps

2, 4, 8 Gbps

FLASH

SFP+

SFP+IBM ASIC

IBM ASIC

PCIeSwitch

HBAASIC

HBAASIC

FLASH



81

FICON Express8 in the I/O Drawer

� Auto-negotiate to 2, 4, or 8 Gbps

� Connector – LC Duplex

� 10KM LX – 9 micron single mode fiber

– Unrepeated distance - 10 kilometers (6.2 miles)

– Receiving device must also be LX

� SX – 50 or 62.5 micron multimode fiber

– Variable with link data rate and fiber type

– Receiving device must also be SX

� 4 channels of LX or SX (no mix)

LX SX

FC 3325 – 10KM LX, FC 3326 – SX

OR

PCIe

PCIe

PCIe

PCIe2, 4, 8 Gbps

2, 4, 8 Gbps

2, 4, 8 Gbps

2, 4, 8 Gbps

Small Form Factor Pluggable (SFP) optics. Concurrent repair/replace action for each SFP



82

2, 4, 8Gbps

2, 4, 8Gbps

2, 4, 8Gbps

2, 4, 8Gbps

FC 332510KM LX

LC Duplex

FICON Express8 10KM LX in the I/O Drawer

� FICON Express8

– Can be shared among LPARs, and defined as spanned

– High Performance FICON for System z (zHPF)

� 8 Gbps with Auto-negotiate capability (2, 4, or 8 Gbps)

– 1 Gbps NOT supported

� LX – 9 micron single mode fiber

– Unrepeated distance - 10 kilometers (6.2 miles)

– Receiving device must also be LX

� Small Form Factor pluggable optics for concurrent repair/replace

� Personalize as:

– FC

• Native FICON

• Channel-To-Channel (CTC)– z/OS, z/VM, z/VSE, z/TPF, TPF, Linux on System z

– FCP (Fibre Channel Protocol)

• Support of SCSI devices– z/VM, z/VSE, Linux on System z



83

FC 3326 SX

LC Duplex

FICON Express8 SX in the I/O Drawer

� FICON Express8

– Can be shared among LPARs, and defined as spanned

– High Performance FICON for System z (zHPF)

� 8 Gbps with Auto-negotiate capability (2, 4, or 8 Gbps)

– 1 Gbps NOT supported

� SX - 50 or 62.5 micron multimode fiber

– Unrepeated distance (20 to 500 meters) varies with speed and fiber type

– Receiving device must also be SX

� Small Form Factor pluggable optics for concurrent repair/replace

� Personalize as:

– FC

• Native FICON

• Channel-To-Channel (CTC)

–z/OS, z/VM, z/VSE, z/TPF, TPF, Linux on System z

– FCP (Fibre Channel Protocol)

• Support of SCSI devices

–z/VM, z/VSE, Linux on System z

2, 4, 8Gbps

2, 4, 8Gbps

2, 4, 8Gbps

2, 4, 8Gbps



84

zEnterprise zHPF supports data transfers larger than 64 k bytes

� zHPF multi-track data transfers are no longer limited to 64 k bytes– Up to 256 tracks can be transferred a single operation– Eliminating the 64 k byte limit is designed to allow a FICON Express8

channel to fully exploit its available bandwidth– This enhancement is exclusive to z196 and z114

� Designed to help provide

– Higher throughput for zHPF multi-track operations– With lower response time

� Requires: – FICON Express8 and 8S or FICON Express4 channel– CHPID TYPE=FC definition– Control unit support for zHPF

� z/OS operating system support

White Paper: “High Performance FICON (zHPF) for System z Analysis”http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101789

High Performance FICON (zHPF) for DS8000 System z Attached Analysis: AG Storage ATS Offeringhttp://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10668



85

FICON performance on System z

1200

14000

31000

20000

52000

23000

92000

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

I/O driver benchmark I/Os per second4k block sizeChannel 100% utilized

FICONExpress4

andFICON

Express2

zHPF

FICON Express8

zHPF

FICON Express8FICON

Express4and

FICONExpress2

ESCON

zHPF

FICON Express8S

FICON Express8S

z10 z10z196z10

z196z10

z196z114

z196z114

350

520

620

770

620

1600

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

FICON Express44 Gbps

I/O driver benchmarkMegaBytes per secondFull-duplexLarge sequentialread/write mix

FICONExpress44 Gbps

FICON Express88 Gbps

FICON Express88 Gbps FICON

Express8S8 Gbps

FICON Express8S

8 Gbps

z10z9 z10

z196z10

z196z10

z196z114

z196z114

zHPF

zHPF

zHPF

77% increase

108% increase



86

FCP performance on System z

15750

31500

60000

84000 84000

92000

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

FE8 FE8 FE8S

8 Gbps 8 Gbps 8 Gbps

I/Os per secondRead/writes/mix 4k block size, channel 100% utilized

z9 z9 z10 z10 z196

z196z114

520

770

1600

0

200

400

600

800

1000

1200

1400

1600 MegaBytes per second (full-duplex)Large sequential Read/write mix

z10z196z10

z196z114

108% increase

10% increase

FE44 Gbps

FE44 Gbps

FE44 Gbps

FE8S8 Gbps

FE88 Gbps

FE44 Gbps



87

Buffer credits for FICON Express8 and 8S

� 40 buffer credits available per channel

� Conforms to Fibre Channel standard flow control– Avoids overwriting the buffer– 40 credits is 80 KB of buffer at the receiver – Sufficient to hold the maximum data payload on the fiber for 20 km (round trip) of data. Anything less

will not fill the buffer

� Receiver side of port should contain sufficient buffer credits to ensure channel is fully utilized

– Receiver must contain = > 4 buffer credits per km of fiber distance at 8 Gbps

� Buffer credits are directly related to the distance supported– 8 Gbps needs about 4 credits/km... – 10 km = about 40 credits. 100 km = about 400 credits– 4 Gbps needs about 2 credits/km...

• 10 km = about 20 credits. 100 km = about 200 credits

� In the short block case, the concern is response time, not data bandwidth. Where the buffer-to-buffer credits (BtoB credits) becomes an issue is when large blocks per I/O

are being transferred, so link bandwidth becomes the factor of interest. In this case, the frames will be almost always full (2K).



88

z/OS Discovery and Auto-Configuration (zDAC) for z196/z114

� Provides automatic discovery for FICON disk and tape control units

� Reduces level of IT skill and time required to configure new I/O devices

� Ensures system (host) and control unit definitions are compatible with each other

� Automatically discovers storage devices accessible to the system but not currently configured and proposes host definition values

– For those discovered control units, explore for defined logical control units and devices – Compare discovered logical control units and devices against those configured previously– Add missing logical control units and devices to the configuration, proposing control unit and device

numbers, and proposing paths to reach them• Channel paths chosen using algorithm to minimize single points of failure

� Integrated into existing System z host configuration tools (HCD & HCM)

� Requirements:– z114 or z196 server with FICON Express8S, FICON Express8 or FICON Express4 – Switch/director attached fabric (no direct attachment)– z/OS V1.12 (at least 1 LPAR for Dynamic I/O capability)– First exploiter is IBM System Storage DS8700

• DS8000 licensed machine code level 6.5.15.xx (bundle version 75.15.xx.xx), or later• see IBM PSP buckets

– Suggested: FICON DCM (z/OS dynamic channel management) to help manage performance



89

zDAC Value

Without I/O discovery and auto configuration, using HCD **

With I/O discovery and autoconfiguration**

Defining the fibre channel network

Control unit address and ports specified on paper; requires many people and transcribing from paper to HCD.

Steps: multiple iterative steps

HCD: several panels per control unit

Time: hours

Discover the connections in the fiber channel network.

HCD invokes IOS to discover I/O hardware in the current configuration based on autoconfiguration policies.

Steps: 1

HCD: 5 panels for whole fibre channel network; policy definition and running auto discovery.

Time: a few minutes

Defining the control units: building the I/O configurations

Control Unit addresses and ports are specified manually one at a time with ‘new’ or ‘’new based on’ function in HCD.

Steps: multiple iterative steps

HCD: 10 panels per control unit

Time: hours

From the discovered control units, HCD retrieves and proposes control unit and device types and numbers, channel path assignments, partition access and OS device parameters.

Definitions are automatically written into a specified target work IODF which is created as a copy of the active or accessed IODF.

Steps: 1

HCD: 5 panels for whole fibre channel network

Error checking Errors not found until IODF is activated or device is brought online, requiring rework.

Manual error checking

Addresses and ports discovered and defined electronically, using best practices ad existing configurations.

Errors discovered in working IODF and can be corrected before activation

** Based on IBM laboratory results, your results may vary



9090

FCP channels to support T10-DIF for enhanced reliability

� System z Fibre Channel Protocol (FCP) has implemented support of the American

National Standards Institute's (ANSI) T10 Data Integrity Field (DIF) standard. – Data integrity protection fields are generated by the operating system and

propagated through the storage area network (SAN). – System z helps to provide added end-to-end data protection between the operating

system and the storage device

� An extension to the standard, Data Integrity Extensions (DIX), provides checksum

protection from the application layer through the host bus adapter (HBA), where cyclical redundancy checking (CRC) protection is implemented

� T10-DIF support by the FICON Express8S and FICON Express8 features, when defined as CHPID type FCP, is exclusive to z196 and z114.

� Exploitation of the T10-DIF standard requires support by the operating system and the storage device

– z/VM 5.4 with PTFs for guest exploitation– Linux on System z distributions:

• IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases. .



91

Storage Connectivity Options

Description F/C Ports Available Comments

FICON Express8S 10KM LX 0409 2 New Initial orders

FICON Express8S SX 0410 2 New Initial orders

FICON Express4-2C SX 3318 2 Carry Forward z114 Only

FICON Express4 10KM LX 3321 4 Carry Forward

FICON Express4 SX 3322 4 Carry Forward

FICON Express4-2C 4KM LX 3323 2 Carry Forward z114 Only

FICON Express4 4KM LX 3324 4 Carry Forward

FICON Express8 10KM LX 3325 2 NewOrder with RPQ 8P2534 if FC4000 slots are open

FICON Express8 SX 3326 2 NewOrder with RPQ 8P2534 if FC4000 slots are open

Maximum FICON features varies with mix of Drawers types and Model of the System

All use LC Duplex connectors



92

OM1 62.5 micron at 200 MHz-km, OM2 50 micron at 500 MHz-km, OM3 50 micron at 2000 MHz-kmInter-Switch Links (ISLs) is the link between two FICON directors; FICON features do not operate at 10 Gbps* The link loss budget is the channel insertion loss as defined by the standard. # This distance and dB budget applies to FICON Express4 4KM LX features

1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps ISLs

FiberCore (µ )

Light source

Distancemeters

feet

* Linkloss

budget

Distancemeters

feet

* Linkloss

budget

Distancemeters

feet

* Linkloss

budget

Distancemeters

feet

* Linkloss

budget

Distancemeters

feet

* Linkloss

budget

9µ SMLX laser

10 km6.2

miles7.8 dB

10 km6.2 miles

7.8 dB10 km

6.2 miles7.8 dB

10 km6.2 miles

6.4 dB10 km

6.2 miles6.0 dB

9µ SMLX laser

4 km #2.5

miles4.8 dB #

4 km #2.5 miles

4.8 dB #4 km #

2.5 miles4.8 dB # N/A N/A N/A N/A

50µ MM OM3SX laser

860 m2822 ft

4.62 dB500 m1640 ft

3.31 dB380 m1247 ft

2.88 dB150 m492 ft

2.04 dB300 m984 ft

2.6 dB

50µ MM OM2SX laser

500 m1640 ft

3.85 dB300 m984 ft

2.62 dB150 m492 ft

2.06dB50 m164 ft

1.68 dB82 m269 ft

2.3 dB

62.5µ MM OM1SX laser

300 m984 ft

3.00 dB150 m492 ft

2.10 dB70 m230 ft

1.78 dB21 m69 ft

1.58 dB33 m108 ft

2.4 dB

Fibre Channel Physical Interface (FC-PI-4) standard (Rev. 8.00)

� Applies to FICON Express8 and 8S (2, 4, 8 Gbps), FICON Express4 (1, 2, 4 Gbps) FICON Express2 (1, 2 Gbps), and FICON Express (1, 2 Gbps) features

� CHPID types FC (FICON, zHPF, CTC) and FCP (Fibre Channel Protocol)

� Unrepeated distances in kilometers (km), meters (m), and feet (ft)



93


Network connectivity



94

The above statements are based on OSA-Express3 performance measurements performed in a test environment on a System z10 EC and do not represent actual field measurements. Results may vary.

System z OSA-Express3 and OSA-Express4S

� Double density of ports compared to OSA-Express2

– Reduced CHPIDs to manage

– Reduced I/O slots

– Reduced I/O cages or I/O drawers

– Up to 96 LAN ports versus 48

� New microprocessor and hardware data router – compared to OSA-Express2– Large send for IPv4 and IPv6 traffic– Checksum offload– Large send packet construction, inspection and routing performed in hardware

� Designed to reduce the minimum round-trip networking time between z196/z114 systems (reduced latency)

– Designed to improve round trip at the TCP/IP application layer• OSA-Express3 and 4S10 GbE

45% improvement compared to the OSA-Express2 10 GbE

• OSA-Express3 and 4S GbE

45% improvement compared to the OSA-Express2 GbE

– Designed to improve throughput (mixed inbound/outbound)

• OSA-Express3 and 4S 10 GbE1.0 GBytes/ps @ 1492 MTU1.1 GBytes/ps @ 8992 MTU3-4 times the throughput of OSA-Express2 10 GbE0.90 of Ethernet line speed sending outbound 1506-byte frames1.25 of Ethernet line speed sending outbound 4048-byte frames



95

OSA-Express4S GbE and 10 GbE fiber for the PCIe I/O drawer

� 10 Gigabit Ethernet (10 GbE)

– CHPID types: OSD, OSX– Single mode (LR) or multimode (SR)

fiber– One port of LR or one port of SR

• 1 PCHID/CHPID

� Gigabit Ethernet (GbE)

– CHPID types: OSD (CHPID OSN not supported)

– Single mode (LX) or multimode (SX) fiber

– Two ports of LX or two ports of SX• 1 PCHID/CHPID

� Small form factor optics – LC Duplex

FC 0406 – 10 GbE LR, FC 0407 – 10 GbE SR

PCIeIBM ASIC

FPGA

PCIeIBM ASIC

FPGA

FC 0404 – GbE LX, FC 0405 – GbE SX



9696

Open Systems Adapter performance

� OSA processor becomes more efficient as throughput increases

� Window size

– TCP window determines amount of data that the sender can transmit to receiver without needing an acknowledgment from the receiver

– Faster and longer networks require larger windows to keep data flowing smoothly

� Blocking

– Performance is affected by the amount of data blocked together for transfer between OSA and TCP

� Frame size– Larger frames perform better– Larger frames reduce host and OSA processing costs– Size of frame depends on LAN type, MTU setting, size of data sent

� Measure: throughput, transaction response time, server utilization– Bulk data transfer and interactive transactions

� QDIO and jumbo frames (8992 byte MTUs) yield the highest streams



9797

OSA-Express4S 10 GbE performance (laboratory)Mixed Streams – 1492 Byte MTUs

1240

2080

0

500

1000

1500

2000

2500

MB

ps

680

1180

0

200

400

600

800

1000

1200

1400

MB

ps

615

1120

0

200

400

600

800

1000

1200

1400

MB

ps

11801680

0

500

1000

1500

2000

2500

MB

ps

Inbound Streams – 1492 Byte MTUs

OSA-E3 OSA-E4S

80% increase

70% increase 70% increase

40% increase

Mixed Streams – 8000 Byte MTUsInbound Streams – 8000 Byte MTUsOSA-E3 OSA-E4S

OSA-E3 OSA-E4S OSA-E3 OSA-E4S

Notes:� AWM on z/OS� z/OS is doing checksum� 1 megabyte per second(MBps) is 1,048,576bytes per second

MBps represents payloadthroughput (doesnot count packetand frame headers)

Maximum transmission units (MTUs)



98

OSA-Express4S - CHPID type controls operation

CHPID type Purpose / Traffic Operating Systems

OSDAll OSA features

z196, z114 z10, z9, zSeries

Queue Direct Input/Output (QDIO) architectureTCP/IP traffic when Layer 3 (uses IP address)

Protocol-independent when Layer 2 (uses MAC address)

z/OS, z/VMz/VSE, z/TPF

Linux on System z*

OSX10 GbE

z196, z114

OSA-Express for zBX. Connectivity and access control to intraensemble data network

(IEDN) from z196 or z114 to zBXz/OS, z/VM

*IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases.



99

PCIe

PCIe

OSA-Express3 – GbE and 10 GbE ‘fiber’ for the I/O Drawer

� Double the port density of OSA-Express2

� Reduced latency & improved throughput

– Ethernet hardware data router

� Improved throughput – standard & jumbo frames– New microprocessor– New PCIe adapter

� CHPID types– 10 Gigabit Ethernet – OSD, OSX– Gigabit Ethernet – OSD and OSN

• CHPID type OSN is for LPAR-to-LPAR communication. Does not use ports

CHPID shared by two ports4 ports - FC 3362 GbE LX, FC 3363 GbE SX

2 ports FC 3373 GbE SX for z10 BC and z114 only

FC 3370 10 GbE LR FC 3371 10 GbE SR

PCIe

PCIeOSA-Express2 OSA-Express3

Microprocessor500 MHz – 10 GbE448 MHz – 1 GbE

667 MHz

PCI bus PCI-X PCIe G1



100

OSA-Express3 1000BASE-T ‘copper’ for I/O Drawer

� Auto-negotiation to 10, 100, 1000 Mbps

� Double the port density of OSA-Express2

� Reduced latency & improved throughput

– Ethernet hardware data router

� Improved throughput – standard & jumbo frames

– New microprocessor– New PCIe adapter

� Usage of ports– OSC, OSD, OSE can exploit four ports– OSM, Port 0 only– CHPID type OSN is for LPAR-to-LPAR

communication. Does not use ports

OSA-Express2 OSA-Express3

Microprocessor 448 MHz 667 MHz

PCI bus PCI-X PCIe G1

CHPID shared by two ports4 ports - FC 3367 1000BASE-T

2 ports - FC 3369 for z10 BC and z114 only

PCIe

PCIe

Mode CHPID Description

OSA-ICC OSC TN3270E, non-SNA DFT, IPL CPCs, and LPARs, OS system console operations

QDIO OSD TCP/IP traffic when Layer 3, Protocol-independent when Layer 2

Non-QDIO OSE TCP/IP and/or SNA/APPN/HPR traffic

OSA for NCP (LP-to-LP)

OSN NCPs running under IBM Communication Controller for Linux (CDLC)

Unified Resource Manager

OSM Connectivity to intranode management network (INMN) from z196/z114 to zBX



101

OSA solutions – how do they measure up?

� Want to compare?

� Use IBM Application Workload Modeler (AWM)– Replaces Netmarks, and is now externally available– For z/OS and Linux on System z

� Can model your current network configuration in your environment

� Can help you allocate existing system resources more efficiently by modeling, generating real traffic on your network, and evaluating the network performance of

existing workloads

� Rerun the same workload against a proposed alternative (e.g. OSA gigabit)

� Access information at– http://www.ibm.com/software/network/awm/index.html



102

� Two new OSA CHPID types to support new types of z196/z114 networks� A z196/z114 System can have up to 6 types of OSA CHPID’s

– External (customer managed) networks• Defined as OSC,OSD,OSE, & OSN • Existing customer provided and managed OSA ports used for access to the current customer external

networks - no changes– Intranode management network (INMN)

• Defined as CHPID type OSM, OSA-Express for Unified Resource Manager • When the PCIe adaptor on 1000BASE-T is defined as CHPID type OSM, the second port cannot

be used for anything else • OSA-Express3 1000BASE-T configured as CHPID type OSM for connectivity to INMN from z196/z114

to Unified Resource Manager functions• OSA connection via the Bulk Power Hub (BPH) on the z196/z114 to the Top of the Rack (TORs)

switches on zBX– Intraensemble data networks (IEDN)

• Defined as CHPID OSX, OSA-Express for zBX• OSA-Express3 or 4S 10 GbE configured as CHPID type OSX for connectivity and access control to

IEDN from z196/z114 to zBX

� Functions Supported:– Dynamic I/O support– HCD– CP Query capabilities– Ensemble Management for these new channel paths and their related subchannels.

� z/VM 5.4 – CHPID types OSX and OSM cannot be varied online

Open Systems Adapter CHPIDs types:OSM (Express3 1000BASE-T)) and OSX (Express3 and 4S 10 GbE)



103

CHPID Types OSX and OSM – Usage

OSM (INMN)

Supports IOCP CHPID types:OSC, OSD, OSE, OSN, and OSM (ONLY 1000BASE-T).

PCHID = xx0 & xx1

OSA-Express3 10 GbE2 CHPIDs1 PORT/CHPID

OSX (IEDN)

Supports IOCP CHPID types:

OSD and OSX (ONLY 10 GbE).PCHID = xx0 & xx1

OSX IOCDS EXAMPLE:•CHPID PCHID=5E1,PATH=(CSS(0,1,2,3),2F),TYPE=OSX,SHARED, ….•CNTLUNIT CUNUMBR=09F0,PATH=((CSS(0),2F)),UNIT=OSX • IODEVICE ADDRESS=(09F0,15),CUNUMBR=(09F0),UNIT=OSA,UNITADD=00, *

MODEL=X,DYNAMIC=YES,LOCANY=YES

OSM IOCDS EXAMPLE:• CHPID PCHID=191,PATH=(CSS(0,1,2,3),23),TYPE=OSM,,SHARED,• CNTLUNIT CUNUMBR=0910,PATH=((CSS(0),23)),UNIT=OSM• IODEVICE ADDRESS=(0910,15),CUNUMBR=(0910),UNIT=OSA,UNITADD=00, *

MODEL=M,DYNAMIC=YES,LOCANY=YES

OSA-Express31GbE 2 CHPIDs2 PORTS/CHPID

FC3367

CAT 6 ETH CABLE

IEDN Distances•MM (Short Range Optics)50 micron at 2000 MHz-km: 300 meters (984’) 50 micron at 500 MHz-km: 82 meters (269’) 62.5 micron at 200 MHz-km: 33 meters (108’)

•SM (Long Range Optics) 10 km (6.2 miles)

MUST USE CHPID PORT 0

CHPID PORT 0

CHPID PORT 0

OSA-Express4S 10 GbE1 CHPID1 PORT/CHPID

FC3370 and FC0406 (LR)SM 9 micron LC duplex

FC3371 and FC0407 (SR)MM 50/62.5 micron LC duplex



104

z196/z114 HiperSockets – doubled the number

HiperSockets

L3

z/VM

LP4

L2

z/VM

LP5

z/VSE

LP3

Linux

LP2z/OS

LP1

� High-speed “intraserver” network

� Independent, integrated, virtual LANs

� Communication path – system memory

� Communication across LPARs– Single LPAR - connect up to 32 HiperSockets

� Support for multiple LCSS's & spanned channels

� Virtual LAN (IEEE 802.1q) support

� HiperSockets Network Concentrator

� Broadcast support for IPv4 packets

� IPv6

� HiperSockets Network Traffic Analyzer (HS NTA)

� No physical cabling or external connections required



105105

Inbound Workload Queuing for Enterprise Extender

� For improved scalability and performance

� Differentiate and separate inbound EE traffic to a new input queue

� Each input queue represents a unique type of workload

– Unique service and processing requirements– z/OS makes decision on processing resources required

• Avoid resource contention• “Off the wire” network traffic separation

� OSA-Express4S and OSA-Express3 (CHPID types OSD and OSX)

� z196 and z114

� z/OS V1.13.

� z/VM V5.4 with PTFs for guest exploitation



106106

CPU CPU CPU CPU

Other Bulk SD EE

Q1 Q2 Q3 Q4

z/OS Instance

(TCP/IP stack)

OSA-Express

Device

WAN

IWQ

EE traffic separation optimizes the host inbound EE processing

Inbound Workload Queuing for Enterprise Extender

� OSA separates inbound network traffic to

multiple input queues (based on type of workload) on the same host (device) interface

� Each input queue can be serviced concurrently by separate processors

� Stack receives pre-sorted packets and host processing is optimized based on the unique requirements for each unique type of workload

� New – Enterprise Extender traffic is added as a

new input queue (new IWQ)!



107107

� Checksum offload for IPv6 packets is now available for z/OS environments: – When the checksum function is offloaded from the host, CPU cycles are reduced, improving

performance. With the introduction of OSA-Express4S, the checksum offload function is now performed for IPv6 packets as well as IPv4 packets, whether the traffic goes out to the local area network (LAN), comes in from the LAN, or flows logical partition-to-logical partition through OSA-Express4S.

– Checksum offload provides the capability of calculating the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) header checksums for Internet Protocol Version 4 (IPv4) packets and now, IPv6 packets. Checksum verifies the correctness of files.

– When checksum offload was introduced in May of 2003, it was limited to IPv4 packets. Checksum offload for IPv6 packets is exclusive to OSA-Express4S features (CHPID types OSD and OSX) on the z196 and z114. It is supported by z/OS. Refer to the Software requirements section. Checksum offload for IPv4 packets is currently available for all in-service releases of z/OS and Linux on System z.

– Checksum offload for LPAR-to-LPAR traffic in the z/OS environment is included in the OSA-Express4S design for both IPv4 and IPv6 packets.

� Large send for IPv6 packets is now available for z/OS environments: – Large send (also referred to as TCP segmentation offload) is designed to improve performance by

offloading outbound TCP segmentation processing from the host to an OSA-Express4S feature by employing a more efficient memory transfer into OSA-Express4S.

Note: Large Send for IPv6 packets is not supported for LPAR-to-LPAR packets.

� Large send support for IPv6 packets applies to the OSA-Express4S features (CHPID type OSD and OSX), and is exclusive to the z196 and z114. It is supported by z/OS and z/VM for guest exploitation

� Large send for IPv4 packets is currently available for all in-service releases of z/OS and Linux on System z.

IPv6 Support



108

Open Systems Adapter in the PCIe I/O Drawer (FC4003)

Description F/C Ports Available CHPID

OSA-Express4S GbE LX 0404 2 New Build OSD

OSA-Express4S GbE SX 0405 2 New Build OSD

OSA-Express4S 10 GbE LR 0406 1 New Build OSD, OSX

OSA-Express4S 10 GbE SR 0407 1 New Build OSD, OSX

Note: OSA-Express3 1000BASE-T requires an I/O Drawer (FC4000)



109

Open Systems Adapter in the I/O Drawer (FC4000)

Description F/C Ports Available CHPID

OSA-Express 23xx/13xx 2 NO

OSA-Express2 GbE LX 3364 2 Carry Forward OSD, OSN

OSA-Express2 GbE SX 3365 2 Carry Forward OSD, OSN

OSA-Express2 1000BASE-T 3366 2 Carry Forward OSD, OSE, OSC, OSN

OSA-Express2 10 GbE LR 3368 1 NO

OSA-Express3 GbE LX 3362 41 Carry Forward OSD, OSN

OSA-Express3 GbE SX 3363 41 Carry Forward OSD, OSN

OSA-Express3 1000BASE-T 3367 41 New/Carry Forward OSD, OSE, OSC, OSN, OSM

OSA-Express3-2P 1000BASE-T 3369 21 New/Carry Forward OSD, OSE, OSC, OSN

OSA-Express3 10 GbE LR 3370 2 Carry Forward OSD, OSX

OSA-Express3 10 GbE SR 3371 2 Carry Forward OSD, OSX

OSA-Express3-2P GbE SX 3373 21 Carry Forward OSD, OSN

OSA-Express3 cards will be shown as selections via 8P2534 only if FC4000 slots are open.

1 two ports per CHPID



110

Summary: OSA-Express CHPID types to control operation

CHPID type Purpose / Traffic Operating Systems

OSDAll OSA features

z196, z114, z10, z9, zSeries

Supports Queue Direct Input/Output (QDIO) architectureTCP/IP traffic when Layer 3 (uses IP address)

Protocol-independent when Layer 2 (uses MAC address)


Linux on System z

OSE1000BASE-T

z196, z114, z10, z9, zSeries

Non-QDIO; for SNA/APPN/HPR trafficand TCP/IP “passthru” traffic

z/OS, z/VMz/VSE

OSC1000BASE-T

z196, z114, z10, z9, z990, z890

OSA-Integrated Console Controller (OSA-ICC)Supports TN3270E, non-SNA DFT to IPL CPCs & LPs

z/OS, z/VMz/VSE

OSM1000BASE-T

z196 and z114

OSA-Express for Unified Resource ManagerConnectivity to intranode management network (INMN)

from z196 or z114 to Unified Resource Manager functions

z/OS, z/VMLinux on System z

OSNGbE, 1000BASE-Tz196, z114, z10, z9

No OSN support for OSA-Express4S GbE

OSA-Express for NCPAppears to OS as a device supporting CDLC protocol

Enables Network Control Program (NCP) channel-related functionsProvides LP-to-LP connectivity

OS to IBM Communication Controller for Linux (CCL)


Linux on System z

OSX10 GbE

z196 and z114

OSA-Express for zBXConnectivity and access control to intraensemble data network (IEDN)

from z196 or z114 to zBX

z/OS, z/VMLinux on System z



111


Coupling Links Connectivity and STP



112

ISC-3 coupling links ISC-3 coupling links

� InterSystem Channel-3 (ISC-3)

– ISC-3 links ordered in increments of one– Activated links balanced across features

� Peer mode only – 2 Gbps– FC 0217 (ISC-M), FC 0218 (ISC-D / ISC link)– Activate link – FC 0219– Four links per ISC-M

• Two links per ISC-D

– Supports 9µ single mode fiber

� Up to 48 links per machine

� Distances:– Maximum distance is 10 km, RPQ 8P2197 is required for 20 km support– Extended distances of up to 100 km are possible using Dense Wavelength Division

Multiplexors (DWDMs)– Over 100 km requires RPQ 8P2263 or 8P2340

ISC-3

ISC-M (hot-plug) ISC-D (hot-plug)

LC Duplex connector

ISC link

EP

The z114 and z196 will be the last servers to offer ordering of ISC-3. Customers should start migrating to PSIFB coupling links.



113

Parallel Sysplex using InfiniBand (PSIFB)ready for even the most demanding data sharing workloads

� Simplify Parallel Sysplex connectivity

Do more with less

– Can share physical links by defining multiplelogical links (CHPIDs)

– Can consolidate multiple legacy links (ISC and/or ICB)– Can more easily address link constraints

• Define another CHPID to increase availablesubchannels instead of having to add physical links

� More flexible placement of systems in a data center

– 12x InfiniBand coupling links (FC 0171 HCA3-O and FC 163 HCA2-O)• Support optical cables up to 150 meters. No longer restricted to 7 meters between

System z CPCs– 1x InfiniBand coupling links (FC 0170 HCA3-O LR and FC FC 0168 HCA2-O LR)

• Use the same single mode fiber optic cables as ISC-3 and FICON/FCP for unrepeated distances of up to 10 km, and metropolitan distances with qualified DWDM solutions



114

HCA3 for Coupling Links� New 12x InfiniBand and 1x InfiniBand fanout cards

� Exclusive to z196 and z114

– HCA3-O fanout for 12x InfiniBand coupling links

• CHPID type – CIB

– Improved service times with 12x IFB3 protocol

– Two ports per feature

– Fiber optic cabling – 150 meters

– Supports connectivity to HCA2-O

– Link data rate of 6 GBps

– HCA3-O LR fanout for 1x InfiniBand coupling links


– Four ports per feature

– Fiber optic cabling

– 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support

– Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs)

– Over 100 km requires RPQ 8P2263 or 8P2340

– Supports connectivity to HCA2-O LR

– Link data rate server-to-server 5 Gbps

– Link data rate with DWDM; 2.5 or 5 Gbps

* Performance considerations may reduce the number of CHPIDs per port



IFB IFB

HCA3-O for 12x IFB & 12x IFB3

IFB

HCA3-O LR for 1x IFB

IFB IFB IFB

Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload.



115

HCA3-O fanout for 12x InfiniBand coupling links


– Two ports per feature

– Fiber optic cabling – 150 meters

– Supports connectivity to HCA2-O

– Link data rate of 6 GBps

Two protocols (IFB & IFB3)

1. 12x IFB = HCA3-O to HCA2-O

2. 12x IFB3 = HCA3-O to HCA3-O (see below)

• Improved service times, 12x IFB3 service times are

designed to be 40% faster than 12x IFB

12x IFB3 protocol activation requirements

– Maximum of four CHPIDs per HCA3-O port

• If more than four CHIPDs are defined per port, links will run at normal 12x IFB service times

• IFB3 protocol activated as long as 4 CHPIDs or less are defined. No configuration settings required.

HCA3-O HCA3-O


New 12x InfiniBand fanout cards, exclusive to z196 and z114


*Up to 16 CHPIDs – across 2 ports

System z10 z196

HCA3-OHCA2-O

IFB IFB3

IMPORTANT !

Attachment to System z9 HCA1 not supported

HCA3-O 12x Details



116


New 1x InfiniBand fanout cards, exclusive to z196 and z114


z196

HCA3-O LR

System z10

HCA2-O

LR

HCA3-O LR

*Up to 16 CHPIDs – across 4 ports

HCA3-O LR fanout for 1x InfiniBand coupling links


– Four ports per feature

– Fiber optic cabling

– Link sharing between multiple Sysplexes

– Dstance of up to 10 km without repeaters, RPQ 8P2263 or 8P2340 is required for 20 km support

– Extended distances of up to 100 km are also possible using Dense Wavelength Division Multiplexors (DWDMs), going over 100 km requires RPQ 8P2263 or 8P2340

– Supports connectivity to HCA2-O LR

– Link data rate server-to-server 5 Gbps

– Link data rate with DWDM; 2.5 or 5 Gbps

CHPID / Subchannel Options

– 32 subchannels per CHPID (default)

– Option to define 32 or 7 subchannels

– z114 or z196 GA2 to z114 or z196 GA2

HCA3-O LR

HCA3-O LR

HCA3-O

LR

DWDM2.5 Gbps

z196

HCA3-O LR

DWDM5 Gpbs

Server5 Gpbs

System z9(not supported)

z114TLLB116

HCA3-O LR 1x Details



117

InfiniBand coupling links on z196 and z114 Summary

DDR = double date rate, SDR = single data rate* For z114 carry forward only** Performance considerations may reduce the number of CHPIDsper port

Type Speed Distance Fanout Cabling

12x InfiniBand 6 or 3 GBps 150 metersHCA2-O

HCA3-O50µ MM (OM3) fiber

1x InfiniBand 5 or 2.5 Gbps 10 kmHCA2-O LR*

HCA3-O LR9µ SM fiber

� Ports exit from the front of a book/processor drawer with HCAs. Unlike ISC-3 links, does not use I/O card slots

� 12x InfiniBand – z196, z114, z10, z9– DDR at 6 GBps

• z196/z114 and z10 – SDR at 3 GBps

• z196/z114 & z10 to z9• z9 to z9 connection not supported

� 1x InfiniBand – z196/z114 and z10 (not available for z9)

– DDR at 5 Gbps (Server to Server and with DWDM)

– SDR at 2.5 Gbps (with DWDM)

HCA2-O IFB IFB


HCA2-O LR* IFB IFB


Up to 16 CHPIDs – across 2 ports**

HCA3-O IFB & IFB3 IFB & IFB3

Up to 16 CHPIDs – across 4 ports**

HCA3-O LR IFB IFB IFB IFB



118

InfiniBand Connectivity Options

zEnterprise to

zEnterprise orz10 (HCA2-O LR)

zEnterprise to

zEnterprise (HCA3) or

z10 (HCA2)

zEnterprise to

zEnterprise (HCA3) or

z10 (HCA2)



119

12x InfiniBand coupling links - Multiple channel paths

� Up to 16 CHPIDs across two ports

– More subchannels per physical link – CF Receiver CHPIDs can share links– NOT more subchannels per CHPID– MIF uses same address, 7 subchannels per CHPID

CHPID FF7 subchannels

CHPID FE7 subchannels

Single 12x IFB link

14 subchannels

7 SubChannels per channel CHPID

Up to 16 channel paths across two ports

___________________

112 subchannels across two ports

Eg.)

HCA3-O



120

� Up to 16 CHPIDs using same physical links – More subchannels per physical link– Link sharing by different Sysplexes

� Now more subchannels per CHPID– 32 subchannels per CHPID

– Option to define 32* or 7 subchannels– z114 or z196 GA2 to z114 or z196 GA2

CHPID FF32 subchannels

CHPID FE32 subchannels

One 1x IFB link

64 subchannels

32* subchannels per CHPID (default)

Up to 16 CHPIDs per HCA3-O LR___________________

512 subchannels per HCA3-O LR

z114 and z196 GA2 1x InfiniBand Coupling LinksMultiple CHPIDs per link, 32 or 7 subchannels per CHPID (HCA2-O LR and HCA3-O LR)

HC

A3-O

LR

For Example:

Carry forward only for z114

*HCD defaults to 32 subchannels.



121

12x InfiniBand SDR

12x InfiniBand

HCA2-O*

HCA2-C I/O Drawer

or I/0 Cage

ISC-3

ISC-3

ISC-3

ISC-3

z196, z114, z10 EC, z10 BC

IFB-MP

z196/z114

ISC-3

z196, z114, z10 EC, z10 BC, z9 EC, z9 BC

HCA2-O LR*

Up to 150 meters

Up to 150 meters

Up to 10/100 km

.... ....

HCA2-O**

.... ....

1x InfiniBand DDR

Up to 10/100KM

z9 EC and z9 BC

Fanouts

HCA3-O HCA3-O LR

.... ....

12x InfiniBand

Up to 150 meters

1x InfiniBand DDR

Up to 10/100KM

** Not supported on the z114 ** Carry forward only

z196/z114 Coupling Links

� Fanout, not I/O slot, used for InfiniBand

� ICB-4 – No longer supported

� ETR – No longer supported

� All coupling links support STP

� Sysplex Coexistence – z10 EC and BC and z9 EC and BC only



122

System z InfiniBand HCA Adapter Options – Summary

� Host Channel Adapter Optical Fanouts for PSIFB – z196 supports HCA3O, HCA3-O LR, HCA2-O and HCA2-O LR fanouts– z114 supports HCA3-O, HCA3-O LR and HCA2-O (carried forward only) fanouts

� 12X InfiniBand-SDR (3 GBps)– z196/z114 to z9 – Uses 12 lanes for a total link rate of 3 GBps and is a point-to-point connection, maximum length of 150 meters– New 50 micron OM3 (2000 MHz-km) multimode fiber with MPO connectors

� 12X InfiniBand-DDR (6 GBps)– z196/z114 to z196/z114– z196/z114 to z10– Uses 12 lanes for a total link rate of 6 GBps and is a point-to-point connection, maximum length of 150 meters– 50 micron OM3 (2000 MHz-km) multimode fiber with MPO connectors

� 1X InfiniBand-DDR LR (5 Gbps)– z196/z114 to z196/z114– z196/z114 to z10 – Uses one lane for a total link rate of 5 Gbps and supports an unrepeated distance of 10 km– Repeated distances at up to 100 km when attached to a Dense Wavelength Division Multiplexer (DWDM) qualified by

System z– Speed may be auto-negotiated if the attached DWDM is capable of operating at SDR or DDR

• Supports SDR at 2.5 Gbps when connected to a DWDM capable of SDR speed • Supports DDR at 5 Gbps when connected to a DWDM capable of DDR speed

– 9 micron single mode fiber optic cables with LC Duplex connectors (same cable as ISC3)

Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. With InfiniBand coupling links, while the link data rate may be higher than that of ICB (12x InfiniBand (SDR or DDR) or ISC-3 (1x InfiniBand (SDR or DDR), the service times of coupling operations are greater, and the actual throughput may be less than with ICB links or ISC-3 links.



123

Coupling links using InfiniBand Trade Association standard

� 1x InfiniBand

� LC Duplex connector

� One pair of fiber (1x) – one fiber for transmit and one fiber for receive

• 1x InfiniBand operating at 2.5 Gbps or 5 Gbps is exclusive to System z10 and z196/z114 servers

• All attachments to an outside cable plant (including public "dark fiber") are supported only through a patch panel or Wavelength Division Multiplexer (WDM) product.

• 10 to 100 km repeated with an InfiniBand DDR or SDR qualified DWDM. (STP qualification is also required if STP is in use.)

2.5 Gbps (SDR) 5 Gbps (DDR)

FiberCore (µ )

(Light source)@ wavelength

Unrepeateddistance

Opticalpassive

loss

Unrepeateddistance

Opticalpassive

loss

9µ SMLR laser

@ 1310 nm10 km

6.2 miles5.66 dB

10 km6.2 miles

5.66 dB



124

1x InfiniBand9/125 micrometer single mode fiber optic cabling

� Cables available from:

– IBM Global Technology Services (GTS)– Your preferred cable provider

� Fiber core – 9µ single mode

� Light source – LX laser @ wavelength: @ 1310 nm

Note: the fiber optic cabling is the same

as used with ISC-3, FICON LX, 10 GbE LR, and GbE LX

LC Duplex connector

LC Duplex harness



125

Coupling links using InfiniBand Trade Association standard

� 12x Multi-fiber Push-On (MPO) connector

� 24-fiber cable with Duplex MPO connectors– 12 fibers for transmit and 12 fibers for receive

1.12x InfiniBand SDR links operating at 3 GBps (2.5 Gbps per lane) are used to connect a z196/z114 to a z9 or z196/z114 to z10

2.12x InfiniBand DDR links operating at 6 GBps (5.0 Gbps per lane) are used to connect z196/z114 servers or z196/z114 to z10

3 GBps (SDR) 6 GBps (DDR)

FiberCore (µ )

(Light source)

FiberBandwidth

@ wavelength

Unrepeateddistance

Opticalpassive

loss

Unrepeateddistance

Opticalpassive

loss

50µ MM(SX laser)

2000 MHz-km@ 850 nm

150 meters492 feet

2.06 dB150 meters

492 feet2.06 dB



126

� Cables available from:– IBM Global Technology Services (GTS)– Anixter www.anixter.com/– Computer Crafts Inc. www.computer-crafts.com/– Tyco www.tycoelectronics.com/– Fujikura www.fujikura.com/

� Fiber core – 50u multimode

� Light source – SX laser

� Fiber bandwidth @ wavelength: 2000 MHz-km @ 850 nm

� IBM cable part numbers highly recommended

Supported 12x InfiniBand DDR cable lengthsOM3 50/125 micrometer multimode fiber optic cabling

Cable Cable

Item Cable Length Length Connector

Description IBM P/N Meters Feet Type

Duplex 24-fiber cable Assembly 41V2466 10.0 m 32.8 f MPO - MPO

Duplex 24-fiber cable Assembly 15R8844 13.0 m 42.7 f MPO - MPO

Duplex 24-fiber cable Assembly 15R8845 15.0 m 49.2 f MPO - MPO






Duplex 24-fiber cable Assembly 42V2083 Custom N/A MPO - MPO



127

Non-PSIFB Coupling Links

Description F/C Ports Available Comments

ISC-D 0217 N/A New/Carry Forward Mother Card

ISC-D 0218 1 to 2 New/Carry Forward ISC-D (Daughter Card)

ISC-3 Link 0219 1 to 4 New/Carry Forward Port(s) Enabled

RPQ 8P2197 2 New/Carry Forward ISC-3 20KM

ICB-3 0993 Not Available

ICB-4 3393 Not Available

The z114 is the last server to offer ordering of new ISC-3 features.

ISC-3 requires an I/O Drawer.

Server Time Protocol (STP) Required



128

� Theoretical maximum rates shown� 9672 / z900 / z800 / z990 / z890 Connectivity to z196/z114 Not Supported� z196 and z114 does not support ICB connectivity� 1x InfiniBand supports single data rate (SDR) at 2.5 Gbps when connected to a DWDM capable of SDR

speed and double data rate (DDR) at 5 Gbps when connected to a DWDM capable of DDR speed or when point-to-point with another z196/z114 or z10

� System z9 does NOT support 1x InfiniBand DDR or SDR InfiniBand Coupling Links

System z – CF Link Connectivity (Peer Mode only)

Connectivity Optionsz196/z114

ISC-3

z196/z114

1x InfiniBand

z196/z114

12x InfiniBand

z196 / z114 / z10 / z9

ISC-3 (RPQ 8P2197 – 20 km)1 Gbps N/A N/A

z196 / z114 / z10 / z9

ISC-3 (10/100 km)2 Gbps N/A N/A

z9 EC or z9 BC with 12x InfiniBand SDR (150 m) N/A N/A 3 GBps

z196 / z114/ z10 1x-InfiniBand DDR(10/100 km) N/A 5 Gbps N/A

z196 / z114 /z10 12x InfiniBand DDR

(150 m)N/A N/A 6 GBps

Note: The InfiniBand link data rates of 6 GBps, 3 GBps, 2.5 Gbps, or 5 Gbps do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. With InfiniBand coupling links, while the link data rate may be higher than that of ICB (12x InfiniBand (SDR or DDR) or ISC-3 (1x InfiniBand (SDR or DDR), the service times of coupling operations are greater, and the actual throughput may be less than with ICB links or ISC-3 links.



129

System z – Maximum Coupling Links and CHPIDs

Server1x IFB

(HCA3-O LR)

12x IFB & 12x IFB3(HCA3-O)

1x IFB(HCA2-O LR)

12x IFB (HCA2-O)

IC ICB-4 ICB-3 ISC-3Max

External Links

MaxCoupling CHPIDs

z19648

M15 – 32*

32

M15 – 16*

32

M15 – 16*

CF only

32

M15 – 16*32 N/A N/A 48 104 (1) 128

z114M10 – 32*

M05 – 16*

M10 – 16*

M05 – 8*

M10 – 12

M05 – 8*

CF only

M10 – 16*

M05 – 8*32 N/A N/A 48

M10 (2)

M05 (3) 128

z10 EC N/A N/A32

E12 – 16*

32

E12 – 16*32

16

(32/RPQ)N/A 48 64 64

z10 BC N/A N/A 12 12 32 12 N/A 48 64 64

z9 EC N/A N/A N/AHCA1-O

16S08 - 12

32 16 16 48 64 64

z9 BC N/A N/A N/AHCA1-O

1232 16 16 48 64 64

* Uses all available fanout slots. Allows no other I/O or coupling.

1. A z196 M49, M66 or M80 supports a maximum 104 extended distance links (48 1x IFB and 48 ISC-3) plus 8 12x IFB links. A z196 M32 supports a maximum 96 extended distance links (48 1x IFB and 48 ISC-3) plus 4 12x IFB links*. A z196 M15 supports a maximum 72 extended distance links (24 1x IFB and 48 ISC-3) with no 12x IFB links*.

2. z114 M10 supports a maximum of 72 extended distance links (24 1x IFB and 48 ISC-3) with no 12x IFB links*.3. z114 M05 supports a maximum of 56 extended distance links (8 1x IFB and 48 ISC-3) with no 12x IFB links*.



130


I/O Connectivity Summary



131

z114 I/O Connectivity

FeaturesOffered

AsMaximum # of features

Maximumchannels

Incrementsper feature

Purchaseincrements

ESCON NB 16 240 channels 1 - 15 active 4 channels

FICON

FICON Express8S NB 64 128 channels 2 channels 2 channels

FICON Express8 CF* 16 64 channels 4 channels 4 channels

FICON Express4 CF 16 64 channels 4 channels 4 channels

FICON Express4-2C CF* 16 32 channels 2 channels 2 channels

ISC-3 NB 12 48 links 4 links 1 link

OSA-Express

OSA-Express4S 10 GbE, GbE NB 48 96 ports (48 for 10 GbE) 1 (10 GbE) / 2 ports 1 feature

OSA-Express3 1000BASE-T NB 16 64 ports 4 ports 1 feature

OSA-Express3-2P 1000BASE-T NB 16 32 ports 2 ports 1 feature

OSA-Express3 10 GbE, GbE CF* 16 64 ports (32 for 10 GbE) 2 (10 GbE) / 4 ports 1 feature

OSA-Express3-2P GbE CF* 16 32 ports 2 ports 1 feature

OSA-Express2** CF 16 32 ports 2 ports 1 feature

Crypto

Crypto Express3 NB 8 16 PCIe adapters 2 PCIe adapters 2 features***

Crypto Express3-1P NB 8 8 PCIE adapters 1 PCIe adapter 2 features***

* Can be carried forward or ordered on MES using RPQ 8P2534** OSA-Express2 10 GbE LR is not supported as a carry forward*** Two features initially, one thereafter

NB = New BuildCF = Carry Forward



132

z114 Coupling Link connectivity summary

FeaturesMinimum # of features

Maximum # of features

Maximum connections

Incrementsper feature

Purchase increments

ISC-3 0 12 48 links 4 links 1 link

HCA2-O LR* (1x) 04 – M05**6 – M10**

8 links – M0512 links – M10

2 links 2 links

HCA2-O (12x) 04 – M05**8 – M10**


2 links 2 links

HCA3-O LR (1x) 04 – M05**8 – M10**


4 links 4 links

HCA3-O (12x) 04 – M05**8 – M10**


2 links 2 links

* Carry forward only ** Uses ALL the available fanout slots for PSIFB. Allows no other I/O or coupling.



133

IBM zEnterprise

Server Time Protocol Support and Enhancements



134

Glossary for System z Server Time Protocol (STP)

Acronym Full name Comments

Arbiter ArbiterServer assigned by the customer to provide additional means for the Backup Time Server to determine whether it should take over as the Current Time Server.

BTS Backup Time ServerServer assigned by the customer to take over as the Current Time Server (stratum 1 server) because of a planned or unplanned reconfiguration.

CST Coordinated Server TimeThe Coordinated Server Time in a CTN represents the time for the CTN. CST is determined at each server in the CTN.

CTN Coordinated Timing NetworkA network that contains a collection of servers that are time synchronized to CST.

CTN IDCoordinated Timing Network Identifier

Identifier that is used to indicate whether the server has been configured to be part of a CTN and, if so, identifies that CTN.

CTS Current Time ServerA server that is currently the clock source foran STP-only CTN.

Going Away SignalA reliable unambiguous signal to indicate that the CPC is about to enter a check stopped state.

PTS Preferred Time ServerThe server assigned by the customer to be the preferred stratum 1 server in an STP-only CTN.



135

Current STP Recovery – in a nut shell

� CANNOT have two Stratum 1 servers in timing network

� Backup Time Server (BTS) can take over as Current Time Server (CTS), active Stratum 1, only if either:

– Preferred Time Server (PTS) can indicate it has “failed”– BTS can unambiguously determine the PTS has “failed”

� Coordinated Timing Network (CTN) with only 2 servers/CFs– Based on combination of:

• Server Offline Signal (OLS- Channel going away signal) and • Console Assisted Recovery (CAR)

– Server Offline signal (OLS) transmitted on a channel by the server to indicate that the channel is going offline

– CAR initiated when OLS not received• Uses HMC/SE LAN path to determine if BTS can take over as CTS

� Coordinated Timing Network (CTN) with 3 or more servers/CFs– If BTS loses communication on all established paths to CTS

• BTS and Arbiter communicate to establish if Arbiter also has lost communication on all established paths to CTS

– If both BTS and Arbiter cannot communicate with CTS• BTS takes over as CTS (S1)



136

Server Time Protocol (STP) Recovery Enhancements

�The new generation of host channel adapter, HCA3-O

�Going away signal

�Server on which it is running is about to enter a failed (check stopped) state.



137



138



139



140

How it works

• Going Away Signal sent by the Current Time Server (CTS) in an STP-only Coordinated Timing Network (CTN)

• Received by the Backup Time Server (BTS)

• BTS can safely take over as the CTS

• without relying on Offline Signal (OLS) in a 2 server CTN or theArbiter in a CTN with 3 or more servers

• Current recovery design is still used when going away signal is not received by BTS and for other failures



141

Pre-reqs

• This enhancement is only available if you have a HCA3-O on the CTS communicating with a HCA3-O on the BTS.

• The STP recovery design is still available for the cases when a Going Away Signal is not received or for other failures besides a server failure

• Infiniband (IFB) links using • HCA3-O to HCA3-O 12x IFB to 12x IFB • HCA-O LR to HCA3-O LR 1x IFB



142

Dependencies

• Dependencies on OLS and CAR removed in a 2 server CTN –Going Away Signal has priority over OLS in a 2 server CTN

• Dependencies on BTS-->Arbiter communication removed in CTNswith 3 or more servers

–BTS can also use going away signal to take over as CTS for CTNswith 3 or more servers without communicating with Arbiter



143

Improved time coordination for virtual servers in zBX

� Network Time Protocol (NTP) clients running on blades and appliances in zBX synchronize their time to the NTP server provided by the Support Element (SE) every 2 to 4 hours

� It is important for the SE’s clock to maintain good time accuracy

� this enhancement provides the capability for the components in zBXto maintain an approximate time accuracy of 100 milliseconds to an NTP server if they synchronize to the SE’s NTP server at least once an hour

� The IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z) synchronizes to the SE’s BOC once an hour.

� The POWER7 blade and Optimizer synchronize to the SE every 4 hours



144

SE Enhancement

� An enhancement has been made to improve the time accuracy of theSE’s Battery Operated Clock (BOC) by synchronizing the SE’s BOC to the server’s Time of Day Clock (TOD) every hour, instead of the previous synchronization which took place every 24 hours.

� This enhancement will allow the SE’s clock to maintain a time accuracy of 100 milliseconds to an NTP server configured as the External Time Source in an STP-only CTN.



145

SE BOC Update

� At IML SE BOC used to set the CPC TOD

� Prior DR93, Primary SE synchronizes with CPC TOD every 24 hours since SE BOC may have a significantly larger accumulated drift in time compared to CPC TOD

� Could result in repeated timestamps in logs

� To ensure that new Primary SE running with accurate CEC time–Primary SE BOC synchs with the CPR TOD –Alternate SE BOC synchs with Primary SE BOC –Alternate SE switches to Primary SE



146

zBX Virtual Servers improved time



147

NTP thresholds

� Allows you to specify threshold settings in order to suppress generation of hardware and operating system messages related to changes in the NTP server stratum level or source ID.

� Select the stratum level threshold value to indicate the NTP server stratum level that must be reached before generating messages.

� Select the source ID time threshold value to indicate the amount of time that must pass before a change in the source ID generates messages.

– Messages issued if the source ID does not return to the original value within the specified time period.

� Example:– If NTP server stratum level changes

from 2 to 3 and back from 3 to 2, no messages will be generated

– If NTP server stratum level changes from 2 to 4, messages will be generated



148

STP Server Role Assignments (3 or more servers)

• STP server role assignments should be accounted for and included in ALL installation / migration planning activities !

• Process documented in techdoc WP101833 should be followed any time a server is take down for planned or unplanned outages. Failure to follow these procedures can result in a sysplex wide outage!

• Server role reassignments are easy and can be changed concurrently. Roles assignments performed under the “Network Configuration”panel via the “System (Sysplex) Time” task.

• All server roles assignments must be done from the CPC that is assigned as the PTS.

– For PTS reassignments only, the PTS assignment to another CPC must be done from the CPC that will become the new PTS.

New whitepaper released - WP101833http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101833



149

References for Additional Information

� Redbooks– Server Time Protocol Planning Guide, SG24-7280– http://www.redbooks.ibm.com/redpieces/abstracts/sg247280.html– Server Time Protocol Implementation Guide, SG24-7281– http://www.redbooks.ibm.com/redpieces/abstracts/sg247281.html

� Education– Introduction to Server Time Protocol (STP)

• Available on Resource Link at General Availability (GA)• www.ibm.com/servers/resourcelink/hom03010.nsf?OpenDatabase

� STP Web site– www.ibm.com/systems/z/pso/stp.html

� Systems Assurance– The IBM team is required to complete a Systems Assurance Review (SAPR Guide, SA06-012) and to

complete the Systems Assurance Confirmation Form via Resource Link– http://w3.ibm.com/support/assure/assur30i.nsf/WebIndex/SA779

� For further information on NTP and the NTP Public services project, refer to the Web sites:– http://www.ntp.org– http://support.ntp.org– http://www.faqs.org/rfcs/rfc1305.html



150

IBM zEnterprise

Cryptography



151

� Cryptographic Coprocessor Facility – Supports “Secure key” cryptographic processing � PCICC Feature – Supports “Secure key” cryptographic processing � PCICA Feature – Supports “Clear key” SSL acceleration� PCIXCC Feature – Supports “Secure key” cryptographic processing � CP Assist for Cryptographic Function allows limited “Clear key” crypto functions from any CP/IFL

– NOT equivalent to CCF on older machines in function or Crypto Express2 capability� Crypto Express2 – Combines function and performance of PCICA and PCICC� Crypto Express3 – PCI-e Interface, additional processing capacity with improved RAS

2001 2002 2003 2004 20062005 2007 2008 2010/112009

Crypto Express3 z10 EC/BC z196/z114

z9 EC z9 BC z10 EC/BCCrypto Express2

Cryptographic Coprocessor Facility (CCF)

PCI Cryptographic Coprocessor (PCICC)

PCI Cryptographic Accelerator (PCICA)

PCIX Cryptographic Coprocessor (PCIXCC)

CP Assist for Cryptographic Functions

System z Crypto History

z990/z890

G3, G4, G5, G6, z900, z800

G5, G6, z900, z800

z800/z900

z9 EC z9 BC z10 EC/BCz990 z890

z990

z990

z890

z890



152

z196/z114 Cryptographic Enhancements

� Elliptic Curve Cryptography Digital Signature Algorithm (ECC), an emerging public key algorithm expected eventually to replace RSA cryptography in many applications. ECC is capable of providing digital signature functions and key agreement functions. The new CCA functions provide ECC key generation and key management and provide digital signature generation and verification functions compliance with the ECDSA method described in ANSI X9.62 "Public Key Cryptography for the Financial Services Industry: The Elliptic Curve Digital Signature Algorithm (ECDSA)". ECC uses keys that are shorter than RSA keys for equivalent strength-per-key-bit; RSA is impractical at key lengths with strength-per-key-bit equivalent to AES-192 and AES-256. So the strength-per-key-bit is substantially greater in an algorithm that uses elliptic curves.

� ANSI X9.8 PIN security which facilitates compliance with the processing requirements defined in the new version of the ANSI X9.8 and ISO 9564 PIN Security Standards and provides added security for transactions that require Personal Identification Numbers (PIN).

� Enhanced Common Cryptographic Architecture (CCA), a Common Cryptographic Architecture (CCA) key token wrapping method using Cipher Block Chaining (CBC) mode in combination with other techniques to satisfy the key bundle compliance requirements in standards including ANSI X9.24-1 and the recently published Payment Card Industry Hardware Security Module (PCI HSM)standard.

� Secure Keyed-Hash Message Authentication Code (HMAC), a method for computing a message authentication code using a secret key and a secure hash function. It is defined in the standard FIPS 198, "The Keyed-Hash Message Authentication Code". The new CCA functions support HMAC using SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 hash algorithms. The HMAC keys are variable-length and are securely encrypted so that their values are protected.

� Modulus Exponent (ME) and Chinese Remainder Theorem (CRT), RSA encryption and decryption with key lengths greater than 2048-bits and up to 4096-bits.



153

Summary of Cryptographic Enhancements (2011)

� Crypto Express3 and Crypto Express3 -1P (z114 only)

� AES Key Encrypting Keys (AES KEKs) and AES typed keys, in general

� TR-31 key wrapping method for interoperable secure key exchange

� Elliptic Curve Cryptography (EC-DH key agreement protocol)

� PIN Decimalization table protection

� (Rivest-Shamir-Adleman) RSA-Optimal Asymmetric Encryption Padding (OAEP) with SHA-256



154

TKE 7.1 LIC

� Support of new hardware, firmware, software

� Support requirements for standards – Includes using stronger/newer types of encryption inside the TKE

� Simplification of tasks



155

TKE 7.1 LIC features

Security

� New smart card reader support

� Ease of Use/Simplification

� Key Data Migration Wizard update – A single Process for Loading an Entire Key

Hardware Support

� New Operational AES Key type – Users can load and manage the new AES EXPORTER,

IMPORTER, and CIPHER operational keys from the TKE’s crypto module notebook

� PIN Decimalization Table Support– CCA version 4.2 for the Crypto Express3 feature

includes support for 100 decimalization tables for each domain on a host cryptographic adapter.



156

TKE 7.1 LIC features

RAS

� HW Crypto module Status Display – Users will be able to display the current status of the HSM that is being managed – Users in privileged access mode will be able to Display Active IDs on the TKE Console

� Dynamic PKA Master Key Change Support

Standards/ Compliance

� PCI HSM Standards Compliance – design

� Configuration Profile Support



157

Description F/C Engines Available Comments

CPACF enablement 3863 N/A New/Carry Forward No Charge – DES/TDES

Crypto Express3 0864 2 New/Carry Forward

Crypto Express3-1P 0871 1 New z114 /Carry Forward z10 BC and z114 only

TKE 7.0 LIC 0860 N/A New/Carry Forward Upgrade available to TKE 7.1

TKE 7.1 LIC 0867 N/A New Required for z196/z114

TKE Workstation 0841 N/A New/Carry ForwardRequired for z196/z114 and ships with 7.1 LIC

Smart Card Reader 0885 N/A New/Carry Forward (7.0)EC to change from NXP JCOP 41 to NXP JCOP 31

Smart Card 0887 N/A Yes

Additional S-Cards 0884 N/A New/Carry Forward10-pack. Requires TKE 7.0 LIC or higher.

Summary of Crypto Features for z196/z114

Note: Crypto Express3 requires a I/O Cage (z196 only) or I/O Drawer (FC4000)



158

System z Evaluations & Certifications

The Common Criteria

program establishes

an organizational and

technical framework

to evaluate the

trustworthiness of IT

Products and

protection profiles

z/OS z/VM

Linux Linux Linux

Virtualization with partitions

Cryptography

• Common Criteria EAL4+ • with CAPP and LSPP• z/OS 1.7 + RACF• z/OS 1.8 + RACF• z/OS 1.9 + RACF• z/OS 1.10 + RACF (OSPP)• z/OS 1.11 + RACF (OSPP)

• z/OS 1.10 IPv6 Certification by JITC

• IdenTrust™ certification for z/OS PKI Services

• FIPS 140-2• z/OS System SSL 1.10

z/OS System SSL 1.11• z/OS ICSF PKCS#11

Services – z/OS 1.11• Statement of Integrity

z/OS

• System z9 EC and z9 BC System z10 EC and z10 BC

• Common Criteria EAL5 with specific target of evaluation -- LPAR: Logical partitions

• System z196• Common Criteria EAL5+ with specific target of

Evaluation -- LPAR • z114 pending

• Crypto Express2 & Crypto Express3 Coprocessors- FIPS 140-2 level 4 Hardware Evaluation- Approved by German ZKA

• CP Assist- FIPS 197 (AES) - FIPS 46-3 (TDES)- FIPS 180-3 (Secure Hash)

� Common Criteria

� z/VM 5.3

• EAL 4+ for CAPP/LSPP

• System Integrity Statement

• z/VM 6.1 Under evaluation

z/VM

� Common Criteria

� SUSE SLES10 certified at EAL4+ with CAPP

� Red Hat EL5 EAL4+ with CAPP and LSPP

� OpenSSL - FIPS 140-2 Level 1 Validated

� CP Assist - SHA-1 validated for FIPS 180-1 -DES & TDES validated for FIPS 46-3

Linux on System z



159


RAS



160

Scheduled (CIE+Disrupt ive Pat ches + ECs)

Planned - (MES + Dr iver Upgrades)

Unscheduled (UIRA)

Sources of Outages - Pre z9 -Hrs/Year/Syst- Prior

ServersIBM System z9® IBM System z10™ zEnterprise

Unscheduled Outages

Scheduled Outages

Planned Outages

Preplanning requirements

Power & Thermal

Management

Increased Focus over time

� � �

� ��

� �

�

�

�

�

�

Temperature = Silicon Reliability Worst EnemyWearout = Mechanical Components Reliability Worst Enemy.

IBM System z Overall RAS Strategy…..Continuing our RAS focus helps avoid outages

Imp

ac

t of O

uta

ge



161

Preventing All Outages

� Unscheduled Outages

– Advanced Memory RAIM design

• Enhanced Reed-Solomon code (ECC) – 90B/64B

• Protection against Channel/DIMM failures

• Chip marking for fast DRAM replacements

– Mirrored Key cache

– ECC on all Store-in caches (including new EDRAM caches)

– Improved SER resistant latches

– FCP end-to-end checking at OS level

– Integrated TCP/IP checksum generation/checking

– Enhanced STP take over architecture

– Integrated EPO switch cover (protecting the switch during repair actions)

– Continued focus on Firmware



162


� Scheduled Outages

– Double memory data bus lane sparing (reducing repair actions)

– Single memory clock bus sparing

– Double DRAM chipkill tolerance

– Field Repair of interface between processor chip and cache chip and between cache chips (fabric bus)

– Fast bitline delete on L3/L4 cache (largest caches)

– Redundant (N+2) Humidity Sensors

– Redundant (N+2) Altimeter Sensors

– PCIe based I/O infrastructure with Failover capability

– Concurrent replace of existing and new PCIe I/O drawer

– Concurrent blower repair on PCIe I/O drawer

– Light strip to guide repairs on PCIe IO drawer



163


� Planned Outage / Preplanning

– Concurrent add of existing and new PCIe I/O drawer

– Eliminate disruption on Cryptography Adapter (level 3 and higher) patches



164

Improving Reliability

� Power and Thermal Optimization and Management

– Static Power Save Mode under z/OS control

– FICON dynamic sub-chip level mini power down

– Smart blower management by sensing altitude and humidity



165

z114 – Memory Overview

Ch4Ch3

Ch2

Ch1

AS

IC

Ch0

DIMM

CLKDiff

CLKDiff

CRC

CRC

DRAM

X

X

X

X

MCU1

EC

C

RAIM

XX

X

Layers of Memory Recovery� ECC

– Powerful 90B / 64B Reed Soloman code

� DRAM Failure

– Marking technology; no half sparing needed

– 2 DRAM can be marked

– Call for replacement on third DRAM

� Lane Failure

– CRC with Retry

– Data – lane sparing

– CLK – RAIM with lane sparing

� DIMM Failure (discrete components, VTT Reg.)

– CRC with Retry

– Data – lane sparing

– CLK – RAIM with lane sparing

� DIMM Controller ASIC Failure

– RAIM Recovery

� Channel Failure

– RAIM Recovery



166

DATA

CHECK

DATA

CHECKECC

Cache

Key Cache

MCU 1

Key Cache

MCU 2

16B 16B

IBM System z10 BC Memory Structure



167

DATA

CHECK

DATA

CHECKECCRAIM Parity

Cache

Key Cache

MCU 0

Key Cache

MCU 1

Extra column provides RAIM

function

z114 Industry Exclusive RAIM Memory Structure



168

� 2 drawer configuration

� New Flex cable interconnect

� Clocked from primary drawer redundant oscillators via cable and pass-thru card

� Concurrent drawer add & repair not supported

� Limited degrade capabilities

� 2 dedicated IBM spares

� PU assignment and sparing across drawers

PUPUPU PU PU PU PU

I/O Hub I/O Hub I/O Hub I/O Hub

PUPUPU PU PU PU PU

I/O Hub I/O Hub I/O Hub I/O Hub

Primary Drawer

Secondary Drawer

osc

z114 Model M10

osc



169

processor

Failover

I/O Mux

I/O Hub I/O Hub

I/O Mux I/O Mux I/O Mux

Failover

I/O

a

dap

ter.

I/O

a

dap

ter.

Ne

two

rk

ad

ap

ter.

Ne

two

rk a

dap

ter.

Cry

pro

Acce

lera

tor

Cry

pto

Acce

lera

tor

ISC

L

inks

ISC

Lin

ks

Alternate Path

I/O Switch

ControlUnit

I/O Switch

ControlUnit

NetworkSwitch

End User

NetworkSwitch

EndUser

z114 RAS Design ……..Fully Redundant I/O Subsystem – of existing IO cage and drawers

� Fully Redundant I/O Design– SAP / CP sparing– SAP Reassignment– I/O Reset and Failover– I/O Mux Reset / Failover– Redundant I/O Adapter– Redundant Network Adapters– Redundant Crypto Accelerators– I/O Switched Fabric– Network Switched/Router Fabric– High Availability Plugging Rules– Channel Initiated Retry– High Data Integrity Infrastructure– Type 2 Recovery– I/O Alternate Path– Network Alternate Path



170

processor

Failover

I/O Mux

I/O Hub I/O Hub

I/O Mux I/O Mux I/O Mux

Failover

I/O

a

dap

ter.

I/O

a

dap

ter.

Ne

two

rk

ad

ap

ter.

Ne

two

rk a

dap

ter.

Cry

pro

Acce

lera

tor

Cry

pto

Acce

lera

tor

ISC

L

inks

ISC

Lin

ks

Alternate Path

I/O Switch

ControlUnit

I/O Switch

ControlUnit

NetworkSwitch

End User

NetworkSwitch

EndUser

z114 RAS Design ……..Fully Redundant I/O Subsystem – of new PCIe drawer

� Fully Redundant I/O Design– SAP / CP sparing– SAP Reassignment– I/O Reset and Failover– I/O Mux Reset / Failover– Redundant I/O Adapter– Redundant Network Adapters– Redundant Crypto Accelerators– I/O Switched Fabric– Network Switched/Router Fabric– High Availability Plugging Rules– Channel Initiated Retry– High Data Integrity Infrastructure– Type 2 Recovery– I/O Alternate Path– Network Alternate Path



171

Front

Back

Domain

0

Domain

2

Domain

3

Domain

1

Failover is

Front

To

Back

Hub cards on processor

drawer

FS

P-1

ca

rd

AMD 1 AMD 2

. . . . . . . . . . . . . . . . . . .

Front View – Conceptual packaging

Classic or Native I/O Adapters

PCIe Switch Cards

Light Strip

z114 PCIe IO Drawer

� Drawer Technology

– Concurrent Adapter Repair

– Light strip for adapter service and upgrades

– Concurrent Switch card Repair

– Concurrent DCA repair

– Concurrent FSP repair

– Concurrent Drawer Add

� New Technology

– Concurrent Drawer Repair

– Concurrent upgrade from legacy drawer to new PCIedrawer



172

zEnterprise HMC IntegrationSystem z brings Mainframe Systems Management and Service to the Blade Environment

HMC

Ethernet

� Familiar IBM BladeCenter attached through Ethernet on z196/z114

� System z HMC style operation and control of blades

� Redundant HMC with mirroring of critical ensemble data and switchover

� Dynamic provisioning of blade resources

� Trained IBM Service personnel for service and installation. Directed repair via Primary HMC

� Call home on error

� Dynamic failure trend monitoring

� Firmware management and Upgrade



173

z114 RAS

� The z114 Server continues to reduce customer down time by attack all sources of

outages: unscheduled outages, scheduled outages and planned outages. Power and cooling requirements were reduced while still managing reliability.

– Major new Memory design for preventing outages

– Introducing new IO drawers, IO technologies with full concurrent service

– Introducing System z management to the mixed computing environment

– Delivering Green functions and RAS together



174


Capacity on Demand



175

CoD Offerings

� On-line Permanent Upgrade– Permanent upgrade performed by customer (previously referred to Customer Initiated Upgrade - CIU)

� Capacity Backup (CBU) – For disaster recovery– Concurrently add CPs, IFLs, ICFs, zAAPs, zIIPs, SAPs– Pre-paid

� Capacity for Planned Event (CPE)– To replace capacity for short term lost within the enterprise due to a planned event such as a facility upgrade or

system relocation– Predefined capacity for a fixed period of time (3 days)– Pre-paid

� On/Off Capacity on Demand (On/Off CoD) – Production Capacity to satisfy periods of peak demand for computing resources– Supported through software offering – Capacity Provisioning Manager (CPM)– Payment:

• Post-paid or Pre-paid by purchase of capacity tokens• Post-paid with unlimited capacity usage• On/Off CoD records and capacity tokens configured on Resource Link

� Customer Initiated Upgrade (CIU)– Process/tool for ordering temporary and permanent upgrades via Resource Link– Permanent upgrade support:

• Un-assignment of currently active capacity• Reactivation of unassigned capacity• Purchase of all PU types physically available but not already characterized • Purchase of installed but not owned memory



176

Basics of CoD

On/Off CoD with tokensNo expirationCapacity- MSU %- # Engines

Tokens- MSU days- Engine days

Pre-paidCPE

On/Off CoD180 days expirationCapacity

- MSU %- # Engines

Using pre-paid unassigned

capacity up to the limit of the HWMNo expirationCapacity- MSU %

- # Engines

Post-paid

Permanent Upgrade

CBU

Capacity on Demand

Temporary Upgrade

Billable Capacity

(On/Off CoD)Replacement Capacity



177

Capacity on Demand Provisioning Architecture

Base Model

Dormant Capacity

Purchased Capacity

Change permanent capacity via CIU or MES order

Up to 8 records installed and active on the System and up to 200 records staged on the SE

Enforce Terms and Conditions and physical model limitations

Customer defined policy or user commands

CPM (z/OS 1.9 or higher)

APIHMC application

Query Activation

Authorization Layer

Manual operations

* Only one On/Off CoDrecord can be active

CBU – CPE – On/Off CoD

R1* R2* R3* R4* R5* R6* R7* R8*



178

Capacity on Demand Enhancements

z10 z196/z114

Unassigned engine purchase via CIU

Auto replenishment of On/Off CoDrecords

Manufacturing install of up to 4 CoDrecords with system ship. This is for new z196s/z114s only

On/Off CoD Administrative tests

Separate orders for purchase of unassigned engines

On/Off CoD records must be replenished manually

CoD records staged on machine deliver

No On/Off CoD administrative test



179

System z Capacity Resource Availability by Server

Capacity on Demand Server

Capacity BackUp z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800

Capacity Upgrade on Demand z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800

Customer Initiated Upgrade z196, z114, z10 EC, z9 EC, z9 BC, z990, z890, z900, z800

On/Off Capacity on Demand z196, z114, z10 EC, z9 EC, z9 BC, z990, z890

Capacity for Planned Event z196, z114, z10 EC, z10 BC

Capacity BackUp Capacity Upgrade on Demand

Customer Initiated Upgrade – for

ordering

On/Off Capacity on Demand

Capacity for Planned Event

z900Yes (CP only) Yes (CP, I/O, IFL, ICF,

Memory)Yes (CP, IFL, ICF, Memory)

No No

z800 Yes (CP only) Yes (CP, I/O, IFL, ICF) Yes (CP, IFL, ICF) No No

z890/z990 Yes (CP only) Yes (CP, I/O, IFL, ICF, zAAP, Memory*)

Yes (CP, IFL, ICF, zAAP, Memory*)

Yes (CP, IFL, ICF, zAAP)

No

z9Yes (CP, ICF, IFL zAAP, zIIP)

Yes (CP, I/O, IFL, ICF, zAAP, zIIP, Memory*)

Yes (CP, IFL, ICF, zAAP, zIIP, Memory*)

Yes (CP, IFL, ICF, zAAP, zIIP)

No

z10/z196/z114 Yes (CP, ICF, IFL zAAP, zIIP, SAP)

Yes (CP, I/O, IFL, ICF, zAAP, zIIP, SAP, Memory*)

Yes (CP, IFL, ICF, zAAP, zIIP, SAP, Memory*)

Yes (CP, IFL, ICF, zAAP, zIIP, SAP)

Yes (CP, IFL, ICF, zAAP, zIIP SAP)

* Not supported for some memory upgrades

Note: Upgrades are non-disruptive only where there is sufficient hardware resource available and provided pre-planning has been done



180

IBM zEnterprise 114 (z114)Capacity and Performance Planning



181

z114 Capacity Matrix

z114

� Granularity levels similar to z10 BC to facilitate upgrades and incremental growth

� Nomenclature: XYY

– X = Capacity level– YY= Number of processors – A00 = ICF or IFL only

� Any to any capacity upgrade/downgrade capability within the Model

� CBU capability from smallest to largest capacities within the Model

� On/Off CoD within the Model

� Linux only and ICF only servers

� Model M10 provides specialty engine scale out capabilities

Additional engines available on the M10

Z01 Z02 Z03 Z04 Z05

Y01 Y02 Y03 Y04 Y05

X01 X02 X03 X04 X05

W01 W02 W03 W04 W05

V01 V02 V03 V04 V05

U01 U02 U03 U04 U05

T01 T02 T03 T04 T05

S01 S02 S03 S04 S05

R01 R02 R03 R04 R05

Q01 Q02 Q03 Q04 Q05

P01 P02 P03 P04 P05

O01 O02 O03 O04 O05

N01 N02 N03 N04 N05

M01 M02 M03 M04 M05

L01 L02 L03 L04 L05

K01 K02 K03 K04 K05

J01 J02 J03 J04 J05

I01 I02 I03 I04 I05

H01 H02 H03 H04 H05

G01 G02 G03 G04 G05

F01 F02 F03 F04 F05

E01 E02 E03 E04 E05

D01 D02 D03 D04 D05

C01 C02 C03 C04 C05

B01 B02 B03 B04 B05

A01 A02 A03 A04 A05

1-way 2-way 3-way 4-way 5-way

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine

SpecialtyEngine



182

z114 Sub-capacity Processor Granularity

A B C D E F G H I J K L M N O P Q R S TU V

W XY

Z

� The z114 has 26 CP capacity levels (26 x 5 = 130)

– Up to 5 CPs at any capacity level

• All CPs must be the same capacity level

� The one for one entitlement to purchase one zAAPand/or one zIIP for each CP purchased is the same for CPs of any speed.

– All specialty engines run at full speed

– Processor Unit Value for IFL = 100

Number of z114

CPsBase Ratio Ratio z10 BC

to z114

1 CP z10 BC Z01 1.182 CPs z10 BC Z02 1.163 CPs z10 BC Z03 1.14

4 CPs z10 BC Z04 1.13

5 CPs z10 BC Z05 1.12

Capacity level # Engines

PCI – Processor Capacity Index

5-Way4-Way

3-Way2-Way

1-Way

1-way

(sub-capacity26 PCIs)

5-way 3139 PCIs

FULL size

Specialty Engine

1-way782 PCIs



183

Don’t use “one number” capacity comparisons!Work with IBM technical support for capacity planning!

Customers can now use zPCR

?HiperDispatch

GHz

MIPs

tables

MSUs

ratings

The IBM Processor Capacity Reference (zPCR) is a free tool available for download that can be used to size your System z processors.

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1381

System z Capacity Planning in a nutshell



184

What’s Changing for z114

� Single set of LSPR data, based on z/OS 1.11

� Multi-Image LSPR Table (typical partition configurations assumed based on N-way)– Supports up to 5 General Purpose CPs on z114– Supports up to 80 General Purpose CPs (z196)

� Single-Image LSPR Table (single partition configurations assumed)– Supports up to 5 General Purpose CPs or 10 IFLs in single image on z114– Supports up to 80 General Purpose or IFL CPs in a single image (z196) *

� LSPR workloads – Three hardware-characteristic-based workload categories– Low, Average, and High categories, based on “Relative Nest Intensity”

• Previous workload primitives and Mixes were replaced

� Reference Processor Typical Setting– 2094-701 @ 593 MIPS for a single-partition configuration– Equates to 560 MIPS for a 5-partition configuration (multi-image LSPR table)

� Capacity planning tools affected– zPCR, zCP3000, BWATOOL, zPSG, zTPM

* Note: Whenever considering a z/OS partition with more than 48 CPs in a single partition, send a note to [email protected] for guidance.



185

LSPR Tables

� Multi-image (MI) Processor Capacity Ratio table

– Average complex LPAR configuration for each model based on customer profiles– Most representative for vast majority of customers– Same workload assumed in every partition– z/OS only– Use for “high level” sizing– Used to develop the MSU rating

� Single-image (SI) Processor Capacity Ratio table *

– One z/OS partition equal in size to N-way of model (up to 80-way)– This table is ONLY relevant for truly single image z/OS cases– Includes z/VM and Linux workloads on both GP and IFL CPs

*Note: Likely to be dropped in a future release of zPCR

� Single-Image table is used as the basis for IBM capacity planning tools such as zPCR’s LPAR Configuration Capacity Planning function.

– Multi-Image table is a generalization of relative capacity, and is not useful for capacity planning purposes.

� Capacity for z/OS on z10 and later processors is represented with HiperDispatchturned ON.



186

LSPR Versions Supported

� LSPR Data (supporting z114)

– z114, and IBM System z (measured in LPAR-mode)• z/OS 1.11 (Low, Average, High)*• z/VM (High/LV)• Linux (WASDB/L)

– zPCR supplements published workload categories• Low-Avg• Avg-High

� Prior LSPR Data and Tables– Removed from zPCR July 2010

• (z/OS 1.9, z/OS 1.8, z/OS 1.6 and z/OS 1.4)

� All processor data prior to System z (G6 and before)

– Removed from zPCR November 2009– Removed from zCP3000 May 2010

� All capacity planning results will be based on the most recently published data

� *Note: Measured using HiperDispatch; applies to z10, z196 and z114 only



187

LPAR configuration Considerations

� z/OS 1.11– Supports up to 80 CPs– LSPR data for 1 through 80 CPs assumes a single partition– Supports zAAP/zIIP Logical CPs– Measured using HiperDispatch*

• Minimizes cross book effects• Attempts to keep important Logical CPs on same Real CPs• Improves high speed buffer efficiency

� Other z/OS releases may be specified– z/OS 1.4 through z/OS 1.10 and z/OS 1.12

• z/OS 1.11 LSPR capacity data will be used• Enforce the SCP rules

� z/VM 5.4– Supports up to a maximum of 32 CPs– LSPR measured up to 32 CPs

� Linux– zPCR allows up to up to a maximum of 32 CPs– LSPR measured up to 16 CPs

� z/VSE– zPCR allows up to a maximum of 4 CPs

� CFCC– Supports up to a maximum of 16 CPs

*Note: ”Capacity Planning for z10 Upgrades” document is available TD104738



188

New LSPR Workload Categories

� Various combinations of prior workload primitives are measured on which the new

workload categories are based– Applications include CICS, DB2, IMS, OSAM, VSAM, WebSphere, COBOL, utilities

� Low (relative nest intensity)– Workload curve representing light use of the memory hierarchy– Similar to past high scaling workload primitives

� Average (relative nest intensity)– Workload curve expected to represent the majority of customer workloads– Similar to the past LoIO-mix curve

� High (relative nest intensity)– Workload curve representing heavy use of the memory hierarchy– Similar to the past DI-mix curve

� zPCR extends published categories– Low-Avg

• 50% Low and 50% Average– Avg-High

• 50% Average and 50% High



189

Choosing a Workload Category

� Average is the default

– when no other information is available

� Based on DASD I/O Rate

– From RMF Workload Activity report – Captured in EDF using CP3KEXTR* (one file per partition)

� Based on SMF 113 Data (alternate view currently provided as hint)– Based on CPU MF counter data

• Captured by partition• Intended to eventually provide a more reliable way to choose a workload• Not yet validated – likely to be modified based on feedback

– Captured in EDF using CP3KEXTR* (one file per partition)– Not required

*Note: CP3KEXTR is a Load and Go program written in Assembler. Runs under z/OS.Available to customers since September 2010



190

System z New Workloads Sizing

� SAP

– Sizing for any IBM platform should be done through Techline (e.g., [email protected])

– Can also be used if you need a z114 sizing (less accurate) before announce (under Letter 112)

� zPSG– Sizing for new applications running on System z.

• Supports WebSphere (Application Server, Portal Server, Process Server, Message Broker and MQ), DB2 for z/OS Transaction sizing

– Engage Global Techline Solutions Sizing Support (choose your geography from the drop down)

• Additional support for workloads not found in zPSG• Cognos for Linux on System z• IBM Content Manager• Domino Mail

– Will zPSG be available to IBM BPs via Partner World



191

� Size the z196 portion or z114 portion– Engage a Techline specialist to help you collect the data and do the sizing via Deal Hub Connect– Use zPCR or zCP3000

• Use CP2KEXTR and CP3KVMXT to create an EDF file for z/OS and z/VM• Complete data collection guides located here:

– http://w3.ibm.com/techdocs/PRS2664 - for z/OS http://w3.ibm.com/techdocs/PRS2875 - for z/VM– IBM Business Partners will be able to obtain the tools directly from PartnerWorld refer to slide 24.

� Size the POWER7™ portion – allow at least one week.– Currently a manual process– 20-50 LPARS should take a week– More complex environments would take longer– Working towards a more automated process

� Sizing when migrating from competitive machines to POWER7 blades– Engage a Techline specialist via Deal Hub Connect to help you collect the data and do the sizing– Identify which machines and which time periods the customer would like to consider– Collect data from the competitive machines covering the time frames

• Server consolidation data collection guidance located here:• http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1423

� Sizing new applications running on POWER7 blades– Engage Global Techline Solutions Sizing Support via Deal Hub Connect

• Software sizing questionnaires located here: • http://w3-03.ibm.com/support/techline/sizing/tg_im_sizing.html

Cross Platform sizing including POWER7 blades



192

Cross Platform sizing including POWER7 blades

� Sizing when the customer has an existing set of IBM servers they would like to

migrate to POWER7 blades (go ahead and collect the data now)– Identify which machines and which time periods the customer would like to consider in

the proposal – Collect data from AIX covering the time frames

• Work hand in hand with a POWER7 Specialist to collect the data and do the sizing

– OR• Engage a Techline specialist via Deal Hub Connect to help you collect the data and do the sizing • Sizing questionnaires located here: • http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4034

� We're working with the WLE (Workload Estimator) team for support

– Targeted to be available at announce– Techline will deliver World Wide education when the support is available

� IBM Business Partners

– PWCS (PartnerWorld Contact Services)



193

Cross Platform sizing including Select IBM System x blades

� Size the z196 portion or z114 portion

– Engage a Techline specialist to help you collect the data and do the sizing via Deal Hub Connect

– Use zPCR or zCP3000• Use CP2KEXTR and CP3KVMXT to create an EDF file for z/OS and z/VM• Complete data collection guides located here:

– http://w3.ibm.com/techdocs/PRS2664 - for z/OShttp://w3.ibm.com/techdocs/PRS2875 - for z/VM

– IBM Business Partners will be able to obtain the tools directly from PartnerWorld.

� we're working with the WLE (Workload Estimator) team for support– Targeted to be available at announce– Techline will deliver World Wide education when the support is available



194

IBM Systems Workload Estimator

� Welcome to the IBM Systems Workload Estimator – IBMers access: http://wletest.rchland.ibm.com/wleProd/EstimatorServlet– IBM Business Partners access: http://www-912.ibm.com/wle/EstimatorServlet



195

IBM WebSphere DataPower XI50 for zEnterprise (DataPower XI50z)

� IBM

– DataPower XI50z can perform a tremendous variety of use cases, and in almost all real world use cases, there are other servers involved, which introduce their own latency.

– Currently DataPower XI50z sizing typically is performed on a per client basis, often based on results of a proof of concept.

– If a System z seller identifies the opportunity, they should bring in a SWG WebSphereseller who can then decide whether to bring in a SWG DataPower Tech Sales Specialist.

� IBM Business Partners– Must be certified to sell DataPower XI50z – Should already posses the skills to do the sizing



196


Upgrades



197

z9 BC

z10 BC

M10

M05

z114z114

z196 Model M15

Yes

Air Cooled onlyNO

M10 upgrade path only

M05 upgrade Not Supported

� Upgrade paths from z9 BC and z10 BC

� Upgrade path to z196 Model M15 (Air cooled only)

� Disruptive upgrade M05 to M10 and from M10 to z196 M15

z114 Upgrade paths



198

z114

z890

z9 BC

z10 BC

zEnterprise 114 Product Positioning

� New modular based packaging construct.

� Increased capacity in a single footprint

– Designed for up to 1.12 times the z10 BC in total system capacity

– out-of-order instruction processing– higher clock frequency– larger caches– compiler enhancements

� New I/O subsystem infrastructure

� Connectivity improvements include bandwidth and throughput

� RAIM fault tolerant memory

� Federated capacity and hybrid computing capabilities via integration with zBX, application server blades, optimizers and Unified Resource Manager.

MIP

S c

ap

ac

ity i

n a

sin

gle

fo

otp

rin

t

PUs for Customer use



199

Summary



200

zEnterprise 114 in a nutshell:

� Integration with the zEnteprise BladeExtension

� Two Models

� Increased capacity in a single footprint

• 12s0 technology• out-of-order instruction processing• higher clock frequency• larger cache

� Robust Memory

� Upgrades

– Investment protection with upgrades from two previous families• z10 BC • z9 BC

– Upgradeability to z196 (M15)

� I/O Improvements– new I/O features and New I/O Drawer



201201

Extending the reach and strategic role of the mainframe across the enterprise; simplify and reduce the range of skills necessary for managing the datacenter;

break down cultural divides.

� Scale without adding complexity to meet the growing demands on the infrastructure

� Reduce cost through centralized data centermanagement and automation

� Fit for purpose application deployment

� Unify and optimize multiple architectures to work as a single system

� Improve performance with co-location of data and applications

� Improve resiliency through a private network

� Enable innovation through flexibility and breaking down silos

Summary:Deploy Multi-tier workloads on a single integrated system Create an integrated infrastructure that aligns to end-to-end business processes across technology boundaries



202

Questions ?

Parwez Hamid

[email protected]

Luiz Fadel

[email protected]

IBM zEnterprise™ System



203

Thank You

TakDanishDanke

German

Dank uDutch

ObrigadoBrazilian

Portuguese

��Thai

GrazieItalian

go raibh maith agatGaelic

TrugarezBreton

MerciFrench

GraciasSpanish

СпаСпаСпаСпаcибоибоибоибо

Russian

ந�றிTamil

ध�यवादHindi

��ًاArabic

감사합니다감사합니다감사합니다감사합니다Korean

תודה רבה Hebrew

Tack så mycketSwedish

DankonEsperanto

ありがとうございます

Japanese

谢谢谢谢谢谢谢谢Chinese

děkujiCzech



204

End of Presentation



205

I/O Cage/Drawer Migration



206

z114 & z196•4 I/O domains•32 I/O slots (PCIe I/O cards only)• At least two PCIe fanouts (4 ports per drawer)•7 EIA Units

z114, z196, & System z10 BC•2 I/O domains•8 I/O slots (legacy I/O cards only)• At least 2 HCA2-C fanouts (2 ports per drawer)•Up to two drawers on a pair of fanouts•5 EIA Units

z196 & earlier (except z10 BC)•7 I/O domains•28 I/O slots (legacy I/O cards only)•Up to 4 fanouts for all 7 domains•14 EIA Units

z114

Not Supported

I/O drawer

FC 4000

PCIe I/O drawer

FC 4003

z114 & z196 GA2 I/O Drawers and Cages



207

z196/z114 I/O Cage/Drawer Migration

� With the introduction of the PCIe I/O drawer, a strategy has been developed to help

migrate customer’s I/O from the current I/O cage and I/O drawer to the PCIe I/O drawer

� I/O cage/drawer migration goals

– Eliminate the need for I/O cages and I/O drawers by two generations after the z196

� I/O cage/drawer migration execution

– I/O feature ordering – I/O feature carry forward



208

I/O Feature Ordering and Carry Forward*

� Customers cannot order I/O cages (z196) or I/O drawers, they order I/O features and

eConfig will determine the proper mix of I/O cage (z196), I/O drawer, and the PCIe I/O drawer

– New builds, migration offerings, and z exchange programs • All FICON and OSA features (except 1000 BASE-T) will be PCIe I/O based• Crypto Express3, ESCON, ISC-3, OSA-Express3 1000BASE-T and PSC will be I/O

drawer based– Box MES and loose piece MES

• RPQ to order the I/O drawer based cards to fill empty I/O cage and I/O drawer slots until they are filled

– RPQ is only available if there are no open PCIe I/O drawer slots

– Current I/O drawer based cards are FICON Express8 and OSA-Express3, Crypto Express3 and ISC-3 coupling links

– Once the empty slots (i.e. I/O cage & I/O drawer) are filled, eConfig will drive out a PCIe I/O drawer with the associated features

� Customers will not be allowed to carry forward the oldest generation of I/O cards in future severs*

– z114 will not support (i.e. allow the carry forward) of FICON Express4, OSA-Express2 and ESCON features

– Future servers will not support FICON Express8, OSA-Express3, ISC-3, or PSC features

*Refer to the Statements of Direction



209

*All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. Any reliance on

these statements of general direction is at the relying party's sole risk and will not create liability or obligation for IBM.

ESCON Statement of Direction* - February 15, 2011

� The IBM zEnterprise 196 (z196) will be the last high-end server to support ESCON

channels: IBM plans not to offer ESCON channels as an orderable feature on high-end System z servers which follow the z196 (machine type 2817). In addition, ESCON channels cannot be carried forward on an upgrade to such a follow-on server.

� This plan applies to channel path identifier (CHPID) types CNC, CTC, CVC, and CBY and to feature code numbers 2323 and 2324. System z customers should continue

migrating from ESCON to FICON. Alternate solutions are available for connectivity to ESCON devices.

� IBM Global Technology Services offers an ESCON to FICON Migration solution, This offering should help customers to simplify and manage a single physical and

operational environment.

� Notes: – This new Statement of Direction supersedes the previous ESCON SOD in Announcement letter 110-

170 of July 22, 2010. It also confirms the SOD in Announcement letter 109-230 of April 28, 2009 that “ESCON Channels will be phased out.”

– This SOD does NOT say that the z10 BC will be the last midrange server to support ESCON channels or the last to offer ESCON channels as an orderable feature.



210

IBMIBMIBMIBM

Old Current

Partnership

Prizm is available from IBM GTS Site & Facilities as part of the EFM Service (ESCON to FICON Migration - offering # 6948-97D)

http://www-935.ibm.com/services/us/index.wss/itservice/igs/a1026000?cm_re=masthead-_-itservices-_-site

The order process for Prizm is the same as it is for IBM cabling systems

FICON Convertor Support



211

Optica Technologies PRIZM FICON Converter http://www.opticatech.com/ To help reduce or eliminate ESCON channels on the host while maintaining ESCON and Bus/Tag-based devices and applications

What is PRIZM?� A purpose-built appliance designed exclusively for IBM System z to enable a smooth transition from use of

ESCON channels on the host to use of FICON channels on the host

� Allows ESCON devices to connect to FICON channels and FICON fabrics/networks – Also designed to support attachment of parallel (bus/tag) devices to FICON channels via ESBT

module

� Converts 1 or 2 FICON channels into 4, 8 or 12 ESCON channels– Replace aging ESCON Directors with PRIZM– Achieve streamlined infrastructure and reduced Total Cost of Ownership

� Qualified by the IBM Poughkeepsie, NY Vendor Solutions Lab in July 2009 for all ESCON devices– Refer to:

• http://www-03.ibm.com/systems/z/hardware/connectivity/index.html• Products -- > FICON / FCP Connectivity -- > Other supported devices

� PRIZM is available via IBM Global Technology Services: ESCON to FICON Migration offering (#6948-97D)

BackESCON Ports:

MT-RJ

FICON Ports: LC Duplex Front



212

Topologies supported by PRISM�Local: direct attached or switched�Remote: ISL (cascaded) or IP channel extended

FICON Disk

FICON

FICON

ESCON

FICON Tape

ESCON Tape

FICON

ESBT Module

B/T

ESCON Tape

Parallel (B/T) Printers

Prizm supportslocal OR remote FICON attachment via ISL or IP WAN

Where Does PRISM Fit in the Data Center?



213

Additional Material

213



214

PCHID Ranges

I/O Card Slot Drawer Type A02B A09B A16BA26B

(M05 only)

LG01I/O Drawer DCA 1

PCIe I/O Drawer 100 - 103 180 - 183 X X

LG02I/O Drawer 100 - 10F 180 - 18F 200 - 20F 280 - 28F

PCIe I/O Drawer 104 - 107 184 - 187 X X

LG03I/O Drawer 110 - 11F 190 - 19F 210 - 21F 290 - 29F

PCIe I/O Drawer 108 - 10B 188 - 18B X X

LG04I/O Drawer 120 - 12F 1A0 - 1AF 220 - 22F 2A0 - 2AF

PCIe I/O Drawer 10C - 10F 18C - 18F X X

LG05I/O Drawer 130 - 13F 1B0 - 1BF 230 - 23F 2B0 - 2BF

PCIe I/O Drawer PCIe Switch 0 X X

LG06I/O Drawer DCA 2

PCIe I/O Drawer 110 - 113 190 - 193 X X

LG07I/O Drawer 140 - 14F 1C0 - 1CF 240 - 24F 2C0 - 2CF

PCIe I/O Drawer 114 - 117 194 - 197 X X

LG08I/O Drawer 150 - 15F 1D0 - 1DF 250 - 25F 2D0 - 2DF

PCIe I/O Drawer 118 - 11B 198 - 19B X X

LG09I/O Drawer IFB MP

PCIe I/O Drawer 11C - 11F 19C - 19F X X

LG10I/O Drawer 160 - 16F 1E0 - 1EF 260 - 26F 2E0 - 2EF

PCIe I/O Drawer FSP-1,1 X X



215

PCHID Ranges


(M05 only)

LG11I/O Drawer 170 - 17F 1F0 - 1FF 270 - 27F 2F0 - 2FF

PCIe I/O Drawer 120 - 123 1A0 - 1A3 X X

LG12I/O Drawer X X X X

PCIe I/O Drawer 124 - 127 1A4 - 1A7 X X


PCIe I/O Drawer 128 - 12B 1A8 - 1AB X X


PCIe I/O Drawer 12C - 12F 1AC - 1AF X X




PCIe I/O Drawer 130 - 133 1B0 - 1B3 X X


PCIe I/O Drawer 134 - 137 1B4 - 1B7 X X


PCIe I/O Drawer 138 - 13B 1B8 - 1BB X X


PCIe I/O Drawer 13C - 13F 1BC - 1BF X X


PCIe I/O Drawer 140 - 143 1C0 - 1C3 X X



216

PCHID Ranges


(M05 only)


PCIe I/O Drawer 144 - 147 1C4 - 1C7 X X


PCIe I/O Drawer 148 - 14B 1C8 - 1CB X X


PCIe I/O Drawer 14C - 14F 1CC - 1CF X X




PCIe I/O Drawer 150 - 153 1D0 - 1D3 X X


PCIe I/O Drawer 154 - 157 1D4 - 1D7 X X


PCIe I/O Drawer 158 - 15B 1D8 - 1DB X X


PCIe I/O Drawer 15C - 15F 1DC - 1DF X X


PCIe I/O Drawer FSP-1,2 X X


PCIe I/O Drawer 160 - 163 1E0 - 1E3 X X



217

PCHID Ranges


(M05 only)


PCIe I/O Drawer 164 - 167 1E4 - 1E7 X X


PCIe I/O Drawer 168 - 16B 1E8 - 1EB X X


PCIe I/O Drawer 16C - 16F 1EC - 1EF X X




PCIe I/O Drawer 170 - 173 1F0 - 1F3 X X


PCIe I/O Drawer 174 - 177 1F4 - 1F7 X X


PCIe I/O Drawer 178 - 17B 1F8 - 1FB X X


PCIe I/O Drawer 17C - 17F 1FC - 1FF X X

2 zEnterprise Technical Details - ITSOWW2011

Documents

Transcript of 2 zEnterprise Technical Details - ITSOWW2011