ucr_na-CYBA5074_90001.pdf-1

V2500/V2600 Familiarization Guide

HP 9000 Servers

Final Edition

A5074A

A5074-90001

(available via HP-Web)

November 1999

Preface

The V2500 was HPs most powerful computer to date utilizing up to 32 of HPs most powerful PA8500 PCXW processor chip.

HPs V2500 is mostly an extension to the previous V-Class product’s performance with few visible changes to existing FRUs. The most notable change is adding a 2nd CPU to each processor board; the most visible is adding 2 more cooling fans to accommodate for the additional heat and for n+1 support.

A increase in processor speed for the V-Class product line was then needed for the V2500 product and the V2600 was created.

The V2600 is a field and factory upgrade to the V2500 product which itself was a field and factory upgrade to the V2200. Relative to the V2500 product, the V2600 is a processor board upgrade, utilizing the PA8600, 552 MHz, PCXW+ processor chip. It will allow customers to simply replace their V2500 processor board assemblies with V2600 processor board assemblies and will require a newer version of HPUX (11.0, Extension Pack 9905, CD-ROM Date Code of 3917) to be installed. The V2600 product is conceptually similar to the V2250 upgrade for the V2200 product. The major differences are:

• The V2250 processor board used the same processor as the V2200 processor board, the V2250 was simply clocked to operate at a higher frequency.

• The V2600 uses a different revision of the PCXW processor with functional differences from the processor used for the V2500.

V2500 and V2600 systems are very similar to the V22x0 system, this Familiarization Guide is mostly review of what you already know and will be use learn what the differences are.

How to use this document

This document contains five chapters and one appendix to provide training information to all levels of HP field service personnel. For easy references, each chapter’s material is organized under the following headings:

• Processor

• Memory

• I/O

• Mid-plane

• Power

• Cooling

• Mass Storage

• Service Support Processor (formerly Test Station)

Chapters 1 - 3 contain information primarily for CEs; chapters 4 - 5 contain information that assists troubleshooting and enhances functional knowledge for all (CEs, RCEs, CECs, WTEC). There is one appendix which contains both old and new V-Class server acronym definitions.

Contents

V2500/V2600 H/W Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Processor board (EWPB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Memory board (EWMB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

V-Class Memory Interleaves . . . . . . . . . . . . . . . . . . . . . . . . 3I/O Subsystem (SIOB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Mid-Plane (MIB, SCUB, MIBPB) . . . . . . . . . . . . . . . . . . . . . . . 6

Mid-plane Board (MIB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Utilities Board (SCUB). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Mid-plane Power Board (MIBPB) . . . . . . . . . . . . . . . . . . . . 6

Power Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Higher capacity +48VDC power supply (NPS) . . . . . . . . . . 7AC front-end changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Cooling Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8N+1 fan system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Mass Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

System Management Hardware . . . . . . . . . . . . . . . . . . . . . . 10System management Software . . . . . . . . . . . . . . . . . . . . . . . 10Universal Service Support Processor . . . . . . . . . . . . . . . . . . 10

V2500/V2600 H/W Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Processor Board Configurations . . . . . . . . . . . . . . . . . . . . . . . . . 14PCXW and PCXW+ Configurations . . . . . . . . . . . . . . . . . . 14Processor Board Installation . . . . . . . . . . . . . . . . . . . . . . . . . 15Single CPU in Processor Board Configuration. . . . . . . . . . . 15Mixed Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 CPUs per Processor Board Configuration . . . . . . . . . . . . . 17

Memory Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19V2500/V2600 Memory Configurations . . . . . . . . . . . . . . . . . . . 20

V2500/V2600 DIMM Configuration Rules . . . . . . . . . . . . . 22Using dcm to Check V2500/V2600 Memory Configuration 23V2500/V2600 Memory Board Configuration Rules . . . . . . 25

I/O Subsystem Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . 27Mid-Plane (MIB, SCUB, MIBPB) . . . . . . . . . . . . . . . . . . . . . . . 28

MIB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Contents

SCUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28MIBPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Power Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Cooling Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Mass Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Configuration Limitations/Exclusions . . . . . . . . . . . . . . . . . 33Upgradability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Actions and Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Processor Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Task: Deconfiguring Processors using xconfig . . . . . . . . . . 36Task: Deconfiguring Processors with CPUconfig . . . . . . . . 37Task: Deconfiguring Processors with SPPDSH . . . . . . . . . . 37Task: Interpreting POST messages. . . . . . . . . . . . . . . . . . . . 42Task: Interpreting the LCD Display during POST . . . . . . . . 42Task: COPMOD changes to a V2500 EWPB & V2600 ELPB 48

Memory Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Task: Deconfiguring Memory using xconfig . . . . . . . . . . . . 50Task: Deconfiguring Memory with SPPDSH. . . . . . . . . . . . 50SPDSH Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50SPPDSH Configuration Codes . . . . . . . . . . . . . . . . . . . . . . . 53Task: Interpreting POST messages. . . . . . . . . . . . . . . . . . . . 59

I/O Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Task: Determine I/O Hardware Path . . . . . . . . . . . . . . . . . . 63

Mid-Plane (MIB, SCUB, MIBPB) Tasks . . . . . . . . . . . . . . . . . . 64MIB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64SCUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64MIBPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Power Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Task: Determining the State of Power Subsystem . . . . . . . . 65Task: Hot Swap Replacement of Node Power Supplies. . . . 65

Cooling Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Task: On-Line replacement of Cooling Fans . . . . . . . . . . . . 67

Contents

Mass Storage Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Service Support Processor Tasks . . . . . . . . . . . . . . . . . . . . . . . . 69

Task: do_reset (new features) . . . . . . . . . . . . . . . . . . . . . . . . 69Task: pce vs. pce_util . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Task: Loading Firmware using PDCFL . . . . . . . . . . . . . . . . 71Task: Downloading Firmware to SCUB. . . . . . . . . . . . . . . . 73

Troubleshooting Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Some General Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Processor FRUs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

EWPB & ELPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77PCXW & PCXW+ (or CPU) . . . . . . . . . . . . . . . . . . . . . . . . 77Mesa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77CPU3000 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78Executing CPU3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Memory FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81V-Class DIMM Slot Identification . . . . . . . . . . . . . . . . . . . . 81Mesa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Using the dcm script to troubleshoot a memory error . . . . . . . . 84MEM3000 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Executing MEM3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

I/O Subsystem FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90SIOB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90IO3000 Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Executing IO3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Power Subsystem FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97+48VDC Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Cooling Subsystem FRUs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98N+1 Cooling Fan Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Mass Storage FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99PCI Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99SCSI Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Service Support Processor Utilities . . . . . . . . . . . . . . . . . . . . . . 100CCMD Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100EST Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Hard_Logger Master ID Differences . . . . . . . . . . . . . . . . . . 102

Contents

Theory of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Processor Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108PA8500/PA8600 Adaptations for V2500/V2600 . . . . . . . . . 108Runway Bus Arbitration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110V-Class Memory Addressing Formats . . . . . . . . . . . . . . . . . 111V22x0 Memory Interleaving . . . . . . . . . . . . . . . . . . . . . . . . 114V2500/V2600 Memory Interleaving . . . . . . . . . . . . . . . . . . 115Commercial vs. Technical Systems . . . . . . . . . . . . . . . . . . . 116

I/O Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117Key Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Mid-Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119Functional review of main computing FRUs . . . . . . . . . . . . 119

Power Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Processor module power supply . . . . . . . . . . . . . . . . . . . . . . 120

Cooling Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121Processor module thermal. . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Mass Storage Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Universal SSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125V22x0 Acronym List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126V2500/V2600 Acronym List . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

List of FiguresEWPB with Two PCXW CPUs and their Power Supplies ........................2SIOB Card Cage Cover for up to Seven PCI Slots....................................5CPU Locations on V2500/V2600 Processor Board...................................14DIMM Locations on V2500/V2600 EWMB .............................................22Example dcm Script -- partial listing .........................................................24DIMM Locations for Example dcm Script ................................................25Xconfig SSP (or Test Station) Screen........................................................36Menu Mode command “CPUconfig” ........................................................37Example clist command.............................................................................39Example cget command.............................................................................40Example cput command.............................................................................41POST processor messages .........................................................................42LCD Display during POST........................................................................43LCD Display ..............................................................................................47COPMOD example....................................................................................49Example clist command.............................................................................51Example cget command.............................................................................52Example query command ..........................................................................56Example cput command.............................................................................57Example toggle command .........................................................................57POST messages from a V2500/V2600 system with 32 GB of memory....60POST mapping of Quadrants, Buses and Rows ........................................60POST messages from a V2500/V2600 system with 16 GB of memory....61POST mapping of Quadrants, Buses and Rows ........................................61Physical EWMB Layout ............................................................................62SIOB Card Cage .......................................................................................63Steps performed when hot swapping a nps................................................66Example output from the pce command ....................................................70Command line interface.............................................................................80V2500/V2600 EWMB DIMM Slot Groupings: ........................................83Example V2500/V2600 dcm script, partial listing ....................................85Executing MEM3000 from the command line interface ...........................89Parameter Words 8 through 19 ..................................................................94Executing IO3000 from the command line interface.................................95V22x0 Processing Module pair and EPAC................................................103V2500/V2600 Processing Module pairs and SPAC ..................................103

V22x0 System Block Diagram ..................................................................106V2500/V2600 System Block Diagram ......................................................107Runway Bus Signal Sharing between PCXW CPUs .................................108V22x0 Coherent Memory Address Modification ......................................112V2500/V2600 Coherent Memory Address Modification ..........................113V22x0 Memory Block Diagram ................................................................114V2500/V2600 Memory Block Diagram ....................................................115

HP Company Confidential 1

1) V2500/V2600 H/W Enhancements

Introduction

The V2500/V2600 server contains the next generation of updated hardware compatible with the existing V22x0 platform. This consists of enhancements to each of these major subsystems in the platform: processor, memory and I/O.

The mechanical design for V2500/V2600 is virtually identical to V22x0 except for addition of a card slot in the I/O card cage.

Changes in some areas of the basic infrastructure hardware were required, including midplane, utilities board, power and thermal subsystems. These enhancements are bundled together to comprise the new V2500/V2600 server.

Key Points

Two keys to improving performance are increase parallelisms through twice as many CPUs and memory throughput through more banks.

In total, these enhancements are intended for both new server deliveries and field upgrades to existing V22x0 servers.

When upgrading a V22X0 server to V2500/V2600 functionality, all enhancements must be implemented. Separate configurability of one or more of the enhancements is not supported.

See Chapter 5 “Theory of Operations” for details on improvements to parallelism and memory interleaving.

2 HP Company Confidential

V2500 H/W EnhancementsProcessor board (EWPB/ELPB)

Processor board (EWPB/ELPB)The V2500 platform includes the new PA8500 PCXW CPU (processor chip) on the EWPB processor board. The V2600 platform includes the new PA8600 PCXW+ CPU (processor chip) on the ELPB processor board. Each processor board contains up to two CPU’s (processor chips) interconnected to the same Runway bus.

With two CPU’s per processor board, single cabinet configurations of up to 32 CPUs are supported. The CPU for the V2500 operates at 440MHz and the CPU for the V2600 operates at 552 MHZ, while the Runway bus frequency remains at 120MHz.

Figure 1 EWPB with two PCXW CPUs and their Power Supplies

Key Points

Both Instruction and Data Caches were moved from external to internal Caches within the PCXW and PCXW+. This makes room for installing one or two CPU’s on a processor board. There is 0.5 MB of parity-protected I-Cache and 1.0 MB of ECC-protected D-Cache inside each PCXW or PCXW+ CPU chip.

PCXU+ chips used in V22x0 EVPBs have 4.0 MB of external parity-protected Level 1 Cache.

SAGA is an enhancement of the EPIC chip; more details are forthcoming later in this chapter and in later chapters.

PnLAgent

0

1

PBnR

PBnL

V25040.ppt

toX-Bar

to SAGA

fromX-Bar

RUNWAY

PCXWCPU A

PCXWCPU B

DC to DCPower Supply


PCXWCPU A

PCXWCPU B




V2500 H/W EnhancementsMemory board (EWMB)

Memory board (EWMB)The V2500/V2600 memory board visibly looks just like the V22x0 EMB but consists of new memory board EWMB, new memory controller ASIC SMAC plus a new NBC chip that multiplexes four DIMMs to the SMAC, and a modified CTI controller ASIC (STAC).

The new memory controller (SMAC) provides higher memory bandwidth by accessing more memory banks in parallel.

This feature required substantial design changes to the layout of the memory board, but preserved compatibility with existing 32MB and 128MB DIMMs. CTI controller (STAC) changes were generally limited to those required for compatibility with the (SMAC) memory controller.

Key Points

Existing 32MB and 128MB DIMM types, and a new 256MB DIMM are supported. (There is no longer a restriction by HP-UX when only using 32MB DIMMs as contiguous memory is greater than 32MB.)

The 256MB DIMM is only supported on V2500/V2600 servers. Use of this DIMM allows per-cabinet memory configurations up to 32GB.

V-Class Memory Interleaves

This section compares V22x0 and V2500/V2600 memory interleaving options.

V22x0 Interleaving Options

V22x0’s memory boards (or EMBs) each support four banks; with up to four DIMMs per bank. Each EMB configuration allows 4/8/12/16 DIMMs per board; 4 banks per EMB provide 8/16/32 way interleaving.


V2500 H/W EnhancementsMemory board (EWMB)

Sequential memory addresses are interleaved across 1/2/4 pairs of memory boards and their banks yielding the following memory interleaves:

• 4 DIMMs provide 8/16/32 way interleaving




NOTE Mixing DIMM sizes has no performance degradation with this interleaving format. Adding more or larger DIMMs per bank will increase performance.

V2500/V2600 Interleaving Options

V2500/V2600 memory boards (or EWMBs) each support up to 32 banks; using eight individual buses with up to two DIMMs per bus. EWMB configurations also allow 4/8/12/16 DIMMs per board.

Sequential memory addresses are interleaved across 1/2/4 pairs of memory boards, their buses and banks yielding the following memory interleaves:

• 4 same size DIMM s provide 16/32/64 way interleaving




NOTE Mixing DIMM sizes has performance degradation with this interleaving format. Adding larger DIMMs increases performance; adding DIMMS to every bus increases both interleaving factor and performance.


V2500 H/W EnhancementsI/O Subsystem (SIOB)

I/O Subsystem (SIOB)The V2500/V2600 I/O subsystem features a new I/O card cage supporting both 32/64-bit PCI I/O buses. A new ASIC (SAGA) replaces the EPIC and a new I/O backplane design provides the 64-bit PCI I/O adapter functionality while preserving compatibility with the current 32-bit PCI programming model.

Key Points

Shared I/O memory is expanded from 192K to 448K per I/O module.

The new I/O chassis features one additional I/O controller card slot, for a total of 7 PCI slots in the chassis. SAGA supports +5V PCI signalling, providing compatibility with I/O controllers from V22x0.

A new EMI cover for the I/O card cage allows more clearance for the cables due to the additional card slot.

Figure 2 SIOB Card Cage Cover for up to Seven PCI Slots

See Chapter 5 “Theory of Operations“ for details of SAGA (A) and SAGA (B).

V25055.ppt

NEW

0 1 2 3 0 1 2

SAGA (B) SAGA (A)


V2500 H/W EnhancementsMid-Plane (MIB, SCUB, MIBPB)

Mid-Plane (MIB, SCUB, MIBPB)

Mid-plane Board (MIB)

The V2500/V2600 mid-plane remains visually the same as the ENRB used by the V22x0 except that SPAC ASICs replace EPAC ASICs; ERAC ASICs have not changed for V2500/V2600, however. Some minor changes to processor module (EWPB or ELPB) interconnect, clock distribution, and other functions made necessary by the processor, memory and I/O improvements.

Utilities Board (SCUB)

The V2500/V2600 Utilities board includes increased (NVRAM) and (SRAM) storage for firmware additions required for the increased processor population.

A partial list of (SCUB) changes, compared to the V22x0 (ECUB) utilities board includes:

• SRAM expanded from 128k to 256k

• NVRAM expanded from 32k to 128k

• 32 CPU support in CPU Report CSR in SMUC

• 32 CPU support in Semaphore CSR in SMUC

• 4 new resource semaphores in the SMUC

• Support of n+1 redundant fan operation and graceful shutdown

Mid-plane Power Board (MIBPB)

No functional changes between V22x0 and V2500/V2600.


V2500 H/W EnhancementsPower Subsystem

Power SubsystemSome V2500/V2600 power system changes include redesign of the processor module dc-dc converter power supply, development of a 2400 Watt +48VDC power supply (NPS), and minor changes to the ac front end.

Current V22x0 servers employ three 2000 Watts front end supplies to generate the internal +48VDC distribution. An optional fourth supply provides n+1 power supply redundancy.

Higher capacity +48VDC power supply (NPS)

A 2400 Watts supply, in a form factor that is compatible with new and existing chassis, was developed as initial power calculations indicated that three 2000 Watt power supplies did not satisfy the +48VDC demand for a fully configured (32 processors, 32GB memory, internal peripherals) V2500/V2600 server.

All V2500/V2600 servers employ four 2400 Watt front end power supplies to generate the internal +48VDC distribution. The fourth supply provides n+1 power supply redundancy.

AC front-end changes

Although both the V22x0 and V2500/V2600 incorporate modifications to support a 60 Amp AC input service, they both require the same 50 Amp AC input presently used by V22x0 servers today.


V2500 H/W EnhancementsCooling Subsystem

Cooling SubsystemThe V2500/V2600 cooling system features a redesigned processor heat pipe on each CPU, and increased airflow to cabinet FRUs with the deployment of six (instead of four) cooling fans.

The addition of two more fans and mechanical interlocks enables the n+1 feature that supports (and encourages) on-line fan replacement.

N+1 fan system

A new fan panel design provides n+1 fan capability. This fan panel increases the number of fans in the server from 4 to 6, and implements features permitting power-down and replacement of individual fans without interrupting operation of the server.

Key Points

Operation with a failed fan will be supported for a finite duration sufficient to schedule a service call. Indefinite operation with a failed fan is not recommended due to long-term reliability degradation.

The 6-fan panel is the only fan configuration supported in the V2500/V2600 cooling subsystem.


V2500 H/W EnhancementsMass Storage

Mass StorageTwo new internally-mountable (or embedded) product offerings on V2500/V2600 servers include the following:

• New embedded DVD-ROM replaces the CD-ROM (sometime after SR).

• New Low Voltage Differential (LVD) type of 18GB Internal Disks.

Key Points

The new LVD type disks are ULTRA 2-SCSI and cannot be mixed on the same SCSI bus with earlier HVD type ULTRA-SCSI disks supported by V22x0 servers.

All drives within the same disk tray must be all LVD type (requiring A5149A PCI SCSI card) or all HVD type (requiring A4800A PCI SCSI card).

All LVD drives are labelled to indicate this difference.


V2500 H/W EnhancementsService Support Processor

Service Support Processor

System Management Hardware

A faster Service Support Processor (SSP or Test Station), HP model # B180L with more memory, is provided with new V2500/V2600 servers.

Key Points

Upgrades to V2500/V2600 keep their existing SSP and just update their software for V2500/V2600.

System management Software

The version of HP-UX supported on the SSP may be 10.20 or 11.x; in any case, the format used to display the V2500/V2600 screens now uses CDE instead of X.11 used with V22x0 screens.

There are three distinct V2500/V2600 configurations offered to customers:

• Individual V2500/V2600 with service support processor;

• Multiple V2500/V2600 single nodes sharing the same service support processor;

• Multiple V2500/V2600 single/multi nodes sharing the same service support processor.


All of the above configurations share the same System Management Software, utilities, user interface, and Firmware. See the Service Support Processor Installation and Service Guide for a description of these features and for examples of SSP connections that support two of the three configurations.



Additional Security

All B180L SSP include a utility called xsecure that provides added security for customers that are hesitant about remote support.

NOTE Syntax & man page information about the X-based security manager xsecure:

xsecure [-on | -off | -check] | [-display display_device_name]

xsecure may be run as a script or as an X-based program. It is used to separate the test station (SSP) from any modems or external ethernet buses in order to provide additional security into the system. In the secure mode, all network LANs other than the tsdart bus are disabled and the optional modem on the second serial port will be disabled. When in normal mode, all networks and modems are re-enabled.

If the [-on | -off | -check] options are used, xsecure does not use the GUI interface and instead remains test based. These options allow the user to turn the secure mode on, off or allow the user to check the secure mode status.

A simple button with a red or green secure mode indicator provides the user with secure mode status information. The red indicator shows that the SSP is secure. A green indicator shows that the network is available and the SSP may be accessed through the ethernet port.

In order for xsecure to work properly, the SSP, console cables, terminal mux and modems must be configured in specific ways. The SSP’s JTAG connections, OBP connections and an optional terminal mux must be connected to lan0 and identified in the /etc/hosts file as “tsdart-d.” The sppconsole serial cable from node 0 must be connected to SSP’s serial port 0. An optional modem may be connected to SSP’s serial port 1. If a modem is connected directly to the node’s utility board, it remains unsecure.


2) V2500/V2600 H/W Configurations

Introduction

Hardware configuration guidelines are quite similar to those for V2200, except as follows:

• New processor count configuration rules are necessary due to two processors per processor module and differing module requirements.

• New DIMM positioning configurations.

• Additional DIMM configuration rules for the new 256MB DIMM module.

• Maximum I/O controller configurations have increased by one controller per I/O chassis, due to the additional card slot.

See Chapter 5 “Theory of Operations” for details on balanced configurations through board slot and DIMM slot choices.


V2500 H/W ConfigurationsProcessor Board Configurations

Processor Board ConfigurationsThe processor board for a V2500 or V2600 can contain up to two CPUs. Upgrade of a single-processor board to a dual-processor board will be supported through the addition of a second socketed processor to the board.

PCXW and PCXW+ Configurations

The following figure defines the physical locations of CPU A and CPU B on a processor board

Figure 3 CPU Locations on V2500/V2600 Processor Board

CONNECTOR

CPU A

CPU B

Power Brick for CPU A

Power Brick for CPU B



Processor Board Installation

When installing additional processor boards it is important to install them in the proper order for optimum performance.

The processor boards will be populated with one CPU per board for the first 16. Additional CPUs will be installed on each board to create dual CPU boards in the same order as shown in Table 2.

NOTE The orders are recommended for optimum performance. The system will still operate if this order is not followed.

Single CPU in Processor Board Configuration

Guidelines for a single CPU per processor board configurations:

• The cabinet may have 2 through 16 CPUs

• There may be 2 through 16 processor boards

• Every processor board must have exactly 1 CPU in socket A

Table 1 shows the order that processors with a single CPU should be added to the system.

Table 1 Single CPU per Processor Board Installation Sequence

Order Slot NameMenu Mode

Proc. IDCPU

Location

1st PB0L 0 A

2nd PB4L 8 A

3rd PB1R 2 A

4th PB5R 10 A

5th PB2R 5 A

6th PB6R 13 A

7th PB3L 7 A

8th PB7L 15 A



Mixed Configuration

Guidelines for the mixed configuration of a single CPU per processor board and 2 CPUs per processor board configurations:


• There must be exactly 16 processor boards

• Every processor board must have a minimum of 1 CPU

Table 2 shows the order that processors of a mixed configuration should be added to the system.

Table 2 Mixed Processor Board Installation Sequence

9th PB0R 1 A

10th PB4R 9 A

11th PB1L 3 A

12th PB5L 11 A

13th PB2L 4 A

14th PB6L 12 A

15th PB3R 6 A

16th PB7R 14 A


Proc. IDCPU

Location


Proc. IDCPU

Location

17th PB0L 16 B

18th PB4L 24 B

19th PB1R 18 B

20th PB5R 26 B



2 CPUs per Processor Board Configuration

Guidelines for 2 CPUs per processor board configurations:


• There may be 1 through 16 processor boards

• Every processor board must have exactly 2 CPU

Table 3 shows the order that processors with a 2 CPUs should be added to the system.

21st PB2R 21 B

22nd PB6R 29 B

23rd PB3L 23 B

24th PB7L 31 B

25th PB0R 17 B

26th PB4R 25 B

27th PB1L 19 B

28th PB5L 27 B

29th PB2L 20 B

30th PB6L 28 B

31th PB3R 22 B

32nd PB7R 30 B


Proc. IDCPU

Location



Table 3 Dual CPUs per Processor Board Installation Sequence


Proc. IDCPU

Location

1st & 2nd PB0L 0 & 16 A & B

3rd & 4th PB4L 8 & 24 A & B

5th & 6th PB1R 2 & 18 A & B

7th & 8th PB5R 10 & 26 A & B

9th & 10th PB2R 5 & 21 A & B

11th & 12th PB6R 13 & 29 A & B

13th & 14th PB3L 7 & 23 A & B

15th & 16th PB7L 15 & 31 A & B

17th & 18th PB0R 1 & 17 A & B

19th & 20th PB4R 9 & 25 A & B

21st & 22nd PB1L 3 & 19 A & B

23th & 24th PB5L 11 & 27 A & B

25th & 26th PB2L 4 & 20 A & B

27th & 28th PB6L 12 & 28 A & B

29th & 30th PB3R 6 & 22 A & B

31th & 32nd PB7R 14 & 30 A & B


V2500 H/W ConfigurationsMemory Configurations

Memory ConfigurationsAs with the V22x0, memory board configurations are limited to 2, 4 or 8 memory boards in a single cabinet system (multi-cabinet configurations require all 8 memory boards in each cabinet). These configurations may be selected independent of the number of processors present in the cabinet.

DIMM configurations are possible with 1, 2, 3, or 4 sets of DIMMs populated per memory board. One set of DIMMS, consists of 4 DIMM modules that are identical in size. All memory boards must be populated identically.

NOTE Mixing of 32MB and 256MB DIMM module configurations are not supported in production configurations. 32MB DIMMs provide lower performance than the other DIMM modules.

The V2500/V2600 memory configuration is different from the existing V22x0 memory configuration in several ways. This section discusses those differences and how to check your current configuration.


V2500 H/W ConfigurationsV2500/V2600 Memory Configurations

V2500/V2600 Memory ConfigurationsIn the V2500/V2600 server, DIMMs are installed on Exemplar W-Based Memory Boards (EWMBs). For the remainder of this document the EWMB will be referred to as the memory board. This section contains information about DIMMs and their positioning on memory boards.

A V2500/V2600 memory board is organized by quadrants, rows, and buses. Each memory board has 4 quadrants, 4 rows and 8 buses.

The following terms are used to describe a V2500/V2600 memory board, as shown in Table 4, 5 and Figure 4:

Slot The connector’s location into which DIMMs are installed. There are 16 DIMM slots, each with a unique designator which denotes the slot’s quadrant and bus.

Quadrant A group of four DIMM slots staggered across the memory board.

Buses 8 buses span the four rows. Each DIMM within a quadrant is on a different bus.

Rows Each DIMM has SDRAMs on each side and represents two rows. For instance, the first DIMM installed in the system would represent row 0 bus 0 and row 1 bus 0. All DIMMs have the same SDRAMs on both sides. Therefore, rows 0 and 1 will have the same SDRAM size. Rows 2 and 3 will have the same SDRAM size. Bus interleaving can be configured to either 4 way or 8 way bus interleaving. 8 way provides the best performance. To achieve 8 way bus interleaving, all buses on a row must be populated with DIMMs having the same SDRAM size.

Table 4 shows the correlation between a DIMM slot and a row bus intersection. The first DIMM to be installed in a memory board, Q0B0, occupies row 0 bus 0 and row 1 bus 0 in quadrant 0.



Table 4 DIMM row/bus Relationships

V2500/V2600 DIMM Quadrant Designations

Memory boards can be populated in increments of 4 DIMMs called quadrants. A quadrant is a group of four DIMM slots staggered across the memory board. Quadrants are similar to banks on the V22x0.

• 4 DIMMS provides 1/4 population



• 16 DIMMS provides full population

Table 5 shows the rows and buses associated with each quadrant (0-3) and Figure 4 shows how they are laid out on the memory board.

Table 5 Quadrant Assignments

Rows Buses

0 1 2 3 4 5 6 7

0 Q0B0 Q0B1 Q0B2 Q0B3 Q1B4 Q1B5 Q1B6 Q1B7

1

2 Q2B0 Q2B1 Q2B2 Q2B3 Q3B4 Q3B5 Q3B6 Q3B7

3

Rows Buses

0 1 2 3 4 5 6 7

0 Quadrant 0 Quadrant 1

1

2 Quadrant 2 Quadrant 3

3



Figure 4 DIMM Locations on V2500/V2600 EWMB

Example:

Q2B3: Quadrant 2, Bus 3

V2500/V2600 DIMM Configuration Rules

Use the following rules to plan the memory board DIMM configuration:

• All memory boards must be populated identically.

• Single cabinet memory boards may be populated in 1/4, 1/2, 3/4, or full increments.

• Multi cabinet memory boards may be populated in only 1/4, 1/2, or full increments.

• All DIMMs within a quadrant must be of the same size: 32 Mbyte, 128 Mbyte or 256 Mbyte.



• DIMMs in quadrant 0 can be of a different size than DIMMs in quadrant 2 or 3 without degrading performance.

• DIMMs in quadrant 1 can be of a different size than DIMMs in quadrant 2 or 3 without degrading performance.

• DIMMS in quadrant 0 and 1 should be the same size for maximum performance.

• DIMMS in quadrant 2 and 3 should be the same size for maximum performance.

• DIMMs in quadrant 0 can be of a different size than DIMMs in quadrant 1. To allow this memory to be fully utilized, the bus interleave span will be reduced to 4 way bus interleaving. This will degrade performance.

• DIMMs in quadrant 2 can be of a different size than DIMMs in quadrant 3. To allow this memory to be fully utilized, the bus interleave span will be reduced to 4 way bus interleaving. This will degrade performance.

• Mixing of 32 Mbyte DIMMS and 256 Mbyte DIMMs is not supported.

• All quadrants on a given memory board do not have to be populated with DIMMs.

NOTE Appendix A of V2500/V2600 Upgrade Guide contains diagrams of all possible DIMM configurations supported by the V2500/V2600.

Using dcm to Check V2500/V2600 Memory Configuration

Use the dcm script to check the existing memory configuration. Redirect the output of the dcm script to a file and print it out for future reference.

Select the tsh window and enter:

dcm # > filename

where # is the cabinet number and filename is where you want to redirect the dcm script’s output. If there is only one cabinet, # should be 0.

Print out the dcm script output. Enter:

lp filename



Figure 5 is an example of the part of a dcm script output showing the memory configuration of a system.

Figure 5 Example dcm Script -- partial listing

In the example dcm script above, a partial listing of the physical and logical memory information codes are defined. Physical memory represents memory that was not hardware or software deconfigured. Logical memory is memory that was altered by hardware or software to meet the memory configuration rules.

The memory board location is displayed after the physical and logical memory size information. In Figure 5, two boards are shown, EWMB0 and EWMB1. Each DIMM slot location is also displayed with the status of all quadrants, buses and rows.

Located to the right of each quadrant and bus number is the letter representing the size of the DIMM installed in that slot. Because these are Dual In-line Memory Modules, a 256 MB DIMM is represented as

Memory:=======

Physical: L=128MB, M=64MB, S=16MB Logical: l=128MB, m=64MB, s=16MB

(If logical memory not specified, then it matches physical memory size)

* = Software Deconfigured - = Not In Use

EWMB0:======EWMB0: Q0B0 L/L Q1B4 L/L Q2B0 -/- Q3B4 -/-EWMB0: Q0B1 L/L Q1B5 L/L Q2B1 -/- Q3B5 -/-EWMB0: Q0B2 L/L Q1B6 L/L Q2B2 -/- Q3B6 -/-EWMB0: Q0B3 L/L Q1B7 L/L Q2B3 -/- Q3B7 -/-

EWMB1:======EWMB1: Q0B0 L/L Q1B4 L/L Q2B0 -/- Q3B4 -/-EWMB1: Q0B1 L/L Q1B5 L/L Q2B1 -/- Q3B5 -/-EWMB1: Q0B2 L/L Q1B6 L/L Q2B2 -/- Q3B6 -/-EWMB1: Q0B3 L/L Q1B7 L/L Q2B3 -/- Q3B7 -/-

MemoryBoard

Quadrant # Bus #

Row 0 / Row 1

Quadrant # Bus #Quadrant # Bus #Quadrant # Bus #

Row 2 / Row 3Row 2 / Row 3Row 0 / Row 1



L/L and contains ten 128 MB DRAM chips on each side of the DIMM. A 32 MB DIMM would be listed as S/S and contains ten 16 MB DRAM chips on each side of the DIMM.

Figure 6 shows a V2500/V2600 memory board configured as reported by this dcm script example.

Figure 6 DIMM Locations for Example dcm Script

V2500/V2600 Memory Board Configuration Rules

The V-Class system supports up to eight memory boards. Valid configurations of memory boards include two, four, and eight. (A six memory board configuration is not supported.) When performing an upgrade, you could add either two, four, or six memory boards. The first two memory boards, as shown in Table 6, are located in slots MB0L and MB1L.



Table 6 Memory Board Configurations

Order Slot locations

Minimum memory board configuration MB0LMB1L

Upgrade to four memory boards MB6RMB7R

Upgrade to eight memory boards MB2RMB3RMB4LMB5L


V2500 H/W ConfigurationsI/O Subsystem Configurations

I/O Subsystem ConfigurationsUnlike the V22x0, all four I/O card cages are installed, regardless of selected processor and memory configuration. Each card cage (or chassis) houses two 64-bit PCI I/O adapter ASICs, one of which will host a 3-slot 64-bit PCI bus, and the other a 4-slot 64-bit PCI bus, for a maximum of 7 controller cards per chassis.

Due to power supply limitations in the I/O chassis, there are power-budget configuration rules for controller population in each chassis. V2500/V2600 will not be released with EPIC support. Each I/O chassis shipped with V2500/V2600 will be the 64-bit PCI chassis. Field upgrades of a V22X0 to a V2500/V2600 include replacing each EPIC I/O chassis with a SAGA I/O chassis. All upgraded and new V2500/V2600 systems will have 4 I/O chassis installed, providing the customer with 28 expansion slots.


V2500 H/W ConfigurationsMid-Plane (MIB, SCUB, MIBPB)

Mid-Plane (MIB, SCUB, MIBPB)

MIB

No visible configuration changes between v22x0 and V2500/V2600. The MIB will always have 12 ASICs (8 SPACs and 4 ERACs) installed.

SCUB

The SCUB replaces the ECUB for the V2500/V2600. There are no visible configuration changes between ECUB and SCUB except the n+1 switch on the SCUB should always be in the down (enabled) position.

MIBPB

No visible configuration changes between v22x0 and V2500/V2600.


V2500 H/W ConfigurationsPower Subsystem

Power SubsystemHot swapping of a +48 VDC node power supply (nps) is supported for the V2500/V2600. All V2500/V2600 systems will have 4 nps installed.


V2500 H/W ConfigurationsCooling Subsystem

Cooling SubsystemThere are visible configuration changes between v22x0 and V2500/V2600 as there are now six cooling fans in each of the two cooling fan tray assemblies. This arrangement supports n+1 on-line fan replacement.


V2500 H/W ConfigurationsMass Storage

Mass StorageLow Voltage Differential (LVD) SCSI devices will be supported on the V2500/V2600.


V2500 H/W ConfigurationsService Support Processor

Service Support ProcessorThe Common Desktop Environment (CDE) is implemented at the Service Support Processor (SSP or Test Station) software release version 4.3 vs. the X11 windows environment. The CDE windows environment will also be utilized on the SSP for V2500/V2600 systems.

B180L

For V2500/V2600 nodes, the Test LAN that connects between the SSP and the node is derived from lan0 on the B180L and not from lan1. This is not the case for V22x0 nodes as the Test LAN on the 712 SSP connects to lan1.

In any case, the new SSP software for V2500/V2600 sorts this out automatically by routing to the correct LAN port for either model SSP that connects to a V2500/V2600 node.


V2500 H/W ConfigurationsMiscellaneous

Miscellaneous

Configuration Limitations/Exclusions

The V2500/V2600 Stingray Hardware Program includes the first release of single-cabinet and multi-cabinet V2500/V2600 hardware platforms, with 440 MHz PCXW and 552 MHz PCXW+ processor chips.

PCXW processors are supported in the V2500 and the PCXW+ processors and supported in the V2600.

Upgradability

The V2500/V2600 is field upgradable from previous versions of the V22x0 product line. The number of subassemblies to be swapped is substantial, including in all cases:

• The mid-plane and utility boards.

• All processor modules.

• All memory modules (DIMMs are re-usable).

• All EPIC I/O chassis' (controllers are reusable).

• All +48VDC power supplies.

• All fans.

• A 50 Amp AC service is required for V2500/V2600 systems.

The V2600 is also field upgradable from the previous version of the V2500 product line. The assemblies to replace for this type of upgrade are:

• All processor modules.


V2500 H/W ConfigurationsMiscellaneous

Quality

Due to the similarities in hardware technologies, the V2500/V2600 hardware has greatly improved the AFR (Analyzed Failure Rate) characteristics over the V22x0. A significant reduction in AFR is anticipated with the introduction of the PCXW and PCXW+ processor chips which allow the elimination of external caches. Also a small reduction in AFR is achieved by the n+1 fan and hot swapping of the +48VDC power subsystem.


3) Actions and Tasks

Introduction

The intent of this chapter is to identify the new tasks required for V2500/V2600 systems and those V22x0 tasks that had to be modified because of changes or enhancements made for V2500/V2600 support.


Actions and TasksProcessor Tasks

Processor Tasks

Task: Deconfiguring Processors using xconfig

Minor changes were necessary for the xconfig utility to support 32 processors. Xconfig operates in the same manner as with the V22x0 hardware. The xconfig utility now displays the status of up to 32 processors as seen in Figure 7.

Figure 7 Xconfig SSP (or Test Station) Screen



Task: Deconfiguring Processors with CPUconfig

OBP’s menu mode command CPUconfig is used in the same manner as with V22x0 hardware. Figure 8 displays an example of the output from the CPUconfig command.

Figure 8 Menu Mode command “CPUconfig”

NOTE Tables provided in the previous chapter (Configuration) assist mapping of the Menu Mode Proc. ID to the slot name where the processor is installed in the chassis.

Task: Deconfiguring Processors with SPPDSH

To deconfigure or reconfigure processors when dialed into a V22x0 system using a modem, it is necessary to use the CCMU utility. All of the CCMU functionality is now incorporated within SPPDSH for the V2500/V2600.

Command Description------- -----------CPUconfig [<cpu>] [ON|OFF|SHOW] (De)Configure/Show Processor

Command: CPU config show

Proc type Proc# Proc Rev Speed State Dcache Icache I-prefetch---------- ----- -------- ------- ------- ------- ------- ----------HP,PA85000 0 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 1 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 2 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 3 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 4 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 5 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 6 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 7 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 8 1.0 440 MHz Active 1024 KB 512 KB OffHP,PA85000 9 1.0 440 MHz Active 1024 KB 512 KB Off



The definition of the acronym SPPDSH is “Symmetrical Parallel Processing Diagnostic Shell. SPPDSH is a modified ksh with a large number of built-in functions to allow the reading and writing of V2500/V2600 node memory, configuration parameters and CSRs.

NOTE To use the SPPDSH functions it is necessary to execute the sppdsh command on the Service Support Processor (SSP) or use the pull down menu option and open a SPPDSH window.

SPPDSH Commands

A man page is located on the SSP that contains detailed information about the commands available inside SPPDSH.

Retrieve

When modifying a configuration parameter, the first SPPDSH command necessary to use is retrieve. The retrieve command uploads the configuration parameters of the nodes NVRAM into a buffer in the SSP. Once the parameters have been uploaded they may be modified by using the SPPDSH command cput.

NOTE Syntax & man page information about the retrieve command:

retrieve <node#>

Uploads node configuration parameters to the SSP from NVRAM.

Clist

The clist command is used to provide a listing of the configuration parameters and will accept wildcards as a part of the parameter.

NOTE Syntax & man page information about the clist command:

clist <parameter>

Parameters are names representing POST configurable data.Wherever possible, these parameters should have the same names as those used in OBP.

The example in Figure 9 displays how the clist command can be used to list parameters which start with the characters cpu.



Figure 9 Example clist command

Cget

To obtain the value from a configuration parameter, the cget command is used. These values are reported in hexadecimal format. Table 7 provides a definition of the hex code values associated with processors.

NOTE Syntax & man page information about the cget command:

cget [-f | -s] <node #> <parameter>

Get the value stored in the configuration parameter name of the node from the SSP’s buffer. [-f] forces a read overwriting current data. [-s] forces a read but does so silently.

Figure 10 shows an example of how the cget command can be used to obtain the value stored in all configuration parameters that start with the letters cpu.

sppdsh$ clist cpu*cpu_0cpu_1cpu_10cpu_11cpu_12cpu_13cpu_14cpu_15cpu_16cpu_17cpu_18cpu_19cpu_2cpu_20cpu_21cpu_22cpu_23cpu_24cpu_25cpu_26cpu_27cpu_28cpu_29cpu_3cpu_30* * *(This is not a complete listing of every parameter!!)* * *



Figure 10 Example cget command

SPPDSH Configuration Codes

Table 7 Configuration Codes

sppdsh$ cget 0 cpu*cpu_0 0x01cpu_1 0x01cpu_10 0x01cpu_11 0x01cpu_12 0x01cpu_13 0x01cpu_14 0x01cpu_15 0x01cpu_16 0x30cpu_17 0x30cpu_18 0x30cpu_19 0x30cpu_2 0x01cpu_20 0x30cpu_21 0x30* * *(This is not a complete listing of every parameter!!)* * *

Hex Code Value

Hex Code Value Definition

0xff UNKNOWN

0x00 RESERVED

0x01 PASS

0x10 FAIL

0x20 DECONFIG

0x30 EMPTY

0x40 SW_DECONFIG

0x50 INSTALLED



Cput

Using the cput command allows the value of the configuration parameter to be modified in the SSP’s buffer. Table 7 can be used to determine the correct hex value to be used during the parameter modification.

NOTE Syntax & man page information about the cput command:

cput <node #> <parameter> <value>

Set the node configuration parameter to a specific value in the SSP’s buffer.

Figure 11 displays an example of using the cput command to modify the hex value stored in parameter cpu_0 to contain 0x40. This will force the processor associated with parameter cpu_0 (physical location of PB0L, CPU socket # A) to become software deconfigured. The software deconfiguration will not occur until the replace command is executed and the system has been reset.

Figure 11 Example cput command

WARNING If a processor is deconfigured in either socket of a processor board, the processor which is sharing this runway bus will also be deconfigured. The same tools that were used on the V22x0 system will be used to differentiate which processor was hardware deconfigured vs. software deconfigured for the V2500/V2600 systems.

Replace

When all modifications to the configuration parameters in the SSP’s buffer are complete, the SPPDSH command replace is used. The replace command downloads the configuration parameters from the SSP’s buffer to NVRAM. In order for the system to operate with the new values, a reset must be performed on the V2500/V2600.

sppdsh$ cput 0 cpu_0 0x40cpu_0 0x40sppdsh$




replace <node#>

Downloads node # configuration parameters from the SSP’s buffer to NVRAM.

Task: Interpreting POST messages

In the console window, POST messages will be displayed when a system is booting. By reading the POST messages during bootup, the amount of installed and configured processors can be determined. A failing processor could also be identified if it fails to pass POST.

Figure 12 has an example of a V2500 with 32 processors that passed POST.

Figure 12 POST processor messages

Task: Interpreting the LCD Display during POST

POST formats the LCD into three sections:

Node Information Line

CPU Status Lines

Message Line

Figure 13 displays where the three sections are positioned on the LCD Display.

POST Hard Boot on [0:18]

Stingray POST, Revision 0.6.1.0 1998/08/20 11:04:54 LAB #0006 Probing CPUs: PB0L_A PB0R_A PB1R_A PB1L_A PB2L_A PB2R_A PB3R_A PB3L_A PB4L_A PB4R_A PB5R_A PB5L_A PB6L_A PB6R_A PB7R_A PB7L_A PB0L_B PB0R_B PB1R_B PB1L_B PB2L_B PB2R_B PB3R_B PB3L_B PB4L_B PB4R_B PB5R_B PB5L_B PB6L_B PB6R_B PB7R_B PB7L_B



Figure 13 LCD Display during POST


The first line is used to display node identification. This information is primarily used when two or more systems are connected together to form a multinode system. A single node system always has 0 (0,0) as the output on the LCD display. It is not the intent of this manual to provide training on a multinode system.

H/W training on the multinode system will be presented in the form of a lecture lab course.The format of the Node Information Line is:

<node id> (<x>,<y>)

CPU Status Lines

The two middle rows of the LCD display are dedicated to CPU status information. One character in each row corresponds to a unique CPU. The V22x0 hardware has sixteen characters for the sixteen possible CPUs that could be installed and the V2500/V2600 hardware now has thirty two.

Table 8 defines the different states of the CPU when the status code is displayed on the LCD.

0 (0,0)

- - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

abcdefghijklm

Message Line


CPU Status

Lines



Table 8 LCD CPU Status Line States

character Description

0 CPU internal diagnostic register initialization.

1 CPU early data cache initialization.

2 CPU scratch RAM tests.

3 CPU scratch RAM initialization.

4 CPU BIST-based instruction cache initialization.

5 CPU BIST-based data cache initialization.

6 CPU internal register final initialization.

7 CPU basic instruction set testing.

8 CPU basic instruction cache testing.

9 CPU basic data cache testing.

a CPU basic TLB testing.

b CPU post-selftest internal register cleanup.

R RUN: Performing system initialization operations.

M MONARCH: The main POST initialization CPU.

I IDLE: CPU is in an idle loop, awaiting a command

H HPMC: CPU has detected a high priority machine check (HPMC).



Message Line

Information about what state the system is in can be obtained in the message line. Table 9 contains a listing of the codes that define which test the system is performing during POST.

When the system has finished: running POST, loading all of the necessary PDC files, and booted HP-UX, the message line will be used to display the system run codes as it did with the V22x0 hardware.

Table 9 Message Line States

T TOC: CPU has detected a transfer of control (TOC).

S SOFT_RESET: CPU has detected a soft RESET.

D Dead (failing) CPU

d Deconfigured CPU

- Empty CPU slot

? Unknown CPU status

character Description

Message Description

a Utilities board (SCUB) hardware initialization.

b CPU initialization/selftest rendezvous.

c Utilities board (SCUB) SRAM test.

d Utilities board (SCUB) SRAM initialization.



LCD Command:

Executed from the Test Station (SSP), the LCD command will print the contents of an LCD display for node 0 of current complex. If no arguments are specified, lcd will default to using the all parameter.

NOTE Syntax information about the lcd command:

e Reading Node ID and serial number.

f Verifying non-volatile RAM (NVRAM) data structures.

g Probing system hardware (ASICs).

h Initializing system hardware (ASICs).

i Probing CPUs.

j Initialing, and optionally testing, remaining SCUB SRAM.

k Probing main memory.

l Initializing main memory.

m Verifying multi-node hardware configuration.

n Multinode initialization starting synchronization.

o Multinode hardware initialization.

p Multinode hardware verification.

q Multinode initialization ending synchronization.

r Enabling system error hardware.

Message Description



lcd [<node id> | <complex name> | all]

Determining Processor Hardware Paths

The LCD display can be used to determine which slot the processors are installed in. Using the information provided in Figure 14 you can define the following:

CPU number used by menu mode.

Physical slot location of the processor board.

Locations of processor A and processor B.

SPAC the processors connect to.

A formula that can be used to calculate the hardware path of a processor using the path provided by the HPUX ioscan command.

Figure 14 LCD Display

As used by Menu Mode:0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

- - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

0 1 2 3 4 5 6 7

/ \ / \ / \ / \ / \ / \ / \ / \

- - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - -

L R R L L R R L L R R L L R R L

CPU #:

LCD Display, A Proc.:

LCD Display, B Proc.:

CPU #:

SPAC #:

LCD Display, A Proc.:

LCD Display, B Proc.:

CPU Location:

Using the LCD to determine a physical slot:

Formula used to calculate Hardware as viewed by HPUX:

CPU# + 16 = HPUX Hardware path



Task: COPMOD changes to a V2500 EWPB & V2600 ELPB

There are two types of configurations for a V2500 EWPB: one configuration is as a single processor EWPB (Assembly Part Number - A5491-60001) and the other as a dual processor EWPB (Assembly Part Number - A5492-60001). The EWPB will use the PA8500 PCXW processor chip.

There are two types of configurations for a V2600 ELPB: one configuration is as a single processor ELPB (Assembly Part Number - A5825-60001) and the other as a dual processor ELPB (Assembly Part Number - A5826-60001). The ELPB will use the PA8600 PCXW+ processor chip.

If a EWPB or ELPB is converted from a single processor to a dual processor or visa versa, it will be necessary to change the assembly part number to the corresponding configuration.

This is a new task for the V2500/V2600 only. Changing an assembly part number requires the use of the SPPDSH command copmod to update the final part number of an EWPB/ELPB after a CPU was installed or removed.

The COP chip allows for electronic tracking of the EWPB/ELPB FRU for future references, etc. The SPPDSH command cop reads information stored inside a COP chip and the copmod command writes data into the COP chip.

NOTE Syntax & man page information about the copmod command:

copmod <node#> <cop_id> [<-c> | <-n node#> | <-b board_serial_number> |

<-a assembly_part_number> | <-r assembly_revision>]

Reading/writing of COP information is via JTAG logic on the SCUB over the Test LAN from the SSP. Note that data base information on the SSP’s screen is derived from COP data.



Figure 15 displays an example of the steps that where taken to change a EWPBs assembly part number from a single processor board to a dual processor board. Using the appropriate part number, the same steps can also be used to update an ELPBs assembly part number.

Figure 15 COPMOD examplesppdsh$sppdsh$ cop 0 pb1l Node Device Part Number Board Serial Number EDC Scan Artwork Rev--------------------------------------------------------------------------- 0 pb1l A5491-60001 1321478 3843 00 asppdsh$sppdsh$ sppdsh$ copmod 0 pb1l -p A5492-60001sppdsh$sppdsh$sppdsh$ cop 0 pb1l Node Device Part Number Board Serial Number EDC Scan Artwork Rev--------------------------------------------------------------------------- 0 pb1l A5492-60001 1321478 3843 00 asppdsh$sppdsh$sppdsh$ do_reset 0 1


Actions and TasksMemory Tasks

Memory Tasks

Task: Deconfiguring Memory using xconfig

Minor changes were necessary for the xconfig utility to support the new positioning of DIMMS. The new slot naming convention of quadrants, buses and rows will be used within the xconfig utility. Xconfig operates in the same manner to display and deconfigure memory in the V2500 as with the V22x0. Figure 25 will assist determining were the physical slot is located on the EWMB.

Task: Deconfiguring Memory with SPPDSH

To deconfigure or reconfigure memory when dialed into a V22x0 system using a modem, it was necessary to use the CCMU utility. All of the CCMU functionality is now incorporated within SPPDSH for the V2500/V2600.

NOTE To use the SPPDSH functions it is necessary to execute the sppdsh command on the SSP or use the pull down menu option and open a SPPDSH window.

SPDSH Commands

A man page is located on the SSP that contains detailed information about the commands available inside SPPDSH.

Retrieve

When modifying a configuration parameter, the first SPPDSH command necessary to use is retrieve. The retrieve command uploads the configuration parameters of the nodes NVRAM into a buffer on the SSP. Once the parameters have been uploaded to a buffer on the SSP they may be modified by using the SPPDSH command cput.

NOTE Syntax & man page information about the retrieve command:



retrieve <node#>

Uploads node configuration parameters to the SSP from NVRAM.

Clist

The clist command is used to provide a listing of the configuration parameters and will accept wildcards as a part of the parameter.

NOTE Syntax & man page information about the clist command:

clist <parameter>

Parameters are names representing POST configurable data.Where ever possible these parameters should have the same names as used in OBP.

The example in Figure 16 displays how the clist command can be used to list parameters which start with the characters “mb”.

Figure 16 Example clist commandsppdsh$ clist mb*mb0l.q0b0r0mb0l.q0b0r1mb0l.q0b1r0mb0l.q0b1r1mb0l.q0b2r0mb0l.q0b2r1mb0l.q0b3r0mb0l.q0b3r1mb0l.q1b4r0mb0l.q1b4r1mb0l.q1b5r0mb0l.q1b5r1mb0l.q1b6r0mb0l.q1b6r1mb0l.q1b7r0mb0l.q1b7r1mb0l.q2b0r0mb0l.q2b0r1mb0l.q2b1r0mb0l.q2b1r1mb0l.q2b2r0mb0l.q2b2r1mb0l.q2b3r0mb0l.q2b3r1* * *(This is not a complete listing of every parameter!!)* * *



Cget

There are two commands cget and query which can be used to determine what the status of the hardware is. Examples of both commands will be provided.

To obtain the value from a configuration parameter, the cget command is used. These values are reported in hexadecimal format. Table 10 provides a definition of the hex code values.

NOTE Syntax & man page information about the cget command:

cget [-f | -s] <node #> <parameter>

Get the value stored in the configuration parameter name of the node from the SSP’s buffer. [-f] forces a read overwriting current data. [-s] forces a read but does so silently.

Figure 17 shows and example of how the cget command can be used to obtain the value stored in all configuration parameters that start with the characters “mb0l”.

Figure 17 Example cget command

sppdsh (hw2b):cget 0 mb0l*mb0l.q0b0r0 0x8amb0l.q0b0r1 0x8amb0l.q0b1r0 0x8amb0l.q0b1r1 0x8amb0l.q0b2r0 0x8amb0l.q0b2r1 0x8amb0l.q0b3r0 0x8amb0l.q0b3r1 0x8amb0l.q1b4r0 0x8amb0l.q1b4r1 0x8amb0l.q1b5r0 0x8amb0l.q1b5r1 0x8amb0l.q1b6r0 0x8amb0l.q1b6r1 0x8amb0l.q1b7r0 0x8amb0l.q1b7r1 0x8amb0l.q2b0r0 0x30mb0l.q2b0r1 0x30mb0l.q2b1r0 0x30mb0l.q2b1r1 0x30mb0l.q2b2r0 0x30mb0l.q2b2r1 0x30mb0l.q2b3r0 0x30* * *(This is not a complete listing of every parameter!!)* * *



SPPDSH Configuration CodesTable 10 Configuration Codes

Hex Code Value


0xff UNKNOWN

0x00 RESERVED

0x01 PASS

0x10 FAIL

0x20 DECONFIG

0x30 EMPTY

0x40 SW_DECONFIG

0x50 INSTALLED

0x04 16MB_DECONFIG

0x24 16MB_DECONFIG_88_TO_80

0x34 16MB_DECONFIG_88

0x44 SW_16MB_DECONFIG

0x64 SW_16MB_DECONFIG_88_TO_80

0x74 SW_16MB_DECONFIG_88

0x08 64MB_DECONFIG

0x28 64MB_DECONFIG_88_TO_80

0x38 64MB_DECONFIG_88

0x48 SW_64MB_DECONFIG

0x68 SW_64MB_DECONFIG_88_TO_80

0x78 SW_64MB_DECONFIG_88

0x0c 128MB_DECONFIG



0x2c 128MB_DECONFIG_88_TO_80

0x3c 128MB_DECONFIG_88

0x4c SW_128MB_DECONFIG

0x6c SW_128MB_DECONFIG_88_TO_80

0x7c SW_128MB_DECONFIG_88

0x85 16MB

0xa5 16MB_88_TO_80

0xb5 16MB_88

0x89 64MB_TO_16MB

0xa9 64MB_TO_16MB_88_TO_80

0xb9 64MB_TO_16MB_88

0xc9 SW_64MB_TO_16MB

0xe9 SW_64MB_TO_16MB_88_TO_80

0xf9 SW_64MB_TO_16MB_88

0x8d 128MB_TO_16MB

0xad 128MB_TO_16MB_88_TO_80

0xbd 128MB_TO_16MB_88

0xcd SW_128MB_TO_16MB

0xed SW_128MB_TO_16MB_88_TO_80

0xfd SW_128MB_TO_16MB_88

0x8a 64MB

0xaa 64MB_88_TO_80

0xba 64MB_88

Hex Code Value




Query

To obtain the status of a memory DIMM, the query command could also be used. The query command reports the status of hardware using an english format.

NOTE Syntax & man page information about the query command:

query <node id> <A|C|M|I|D#|S>

The A stands for agents or PACs, C stands for cpus, M stands for MACs, I stands for I/O devices, D stands for the dimms on memory board #, and S stands for SCI devices.

The example in Figure 18 displays how the query command can be used to obtain the status of DIMMs located on memory board zero.

0x8e 128MB_TO_64MB

0xae 128MB_TO_64MB_88_TO_80

0xbe 128MB_TO_64MB_88

0xce SW_128MB_TO_64MB

0xee SW_128MB_TO_64MB_88_TO_80

0xfe SW_128MB_TO_64MB_88

0x8f 128MB

0xaf 128MB_88_TO_80

0xbf 128MB_88

Hex Code Value




Figure 18 Example query command

Cput

The two commands cput and toggle, can be used to deconfigure hardware. Examples of both commands will be provided.

Using the cput command allows the value of the configuration parameter to be modified in the SSP’s buffer. Table 10 can be used to determine the correct hexadecimal value needed for parameter modification.


cput <node #> <parameter> <value>

Set the node configuration parameter to a specific value in the SSP’s buffer.

Figure 19 displays an example of using the cput command to modify the hex value stored in parameter mb0l.q0b0r0 to contain 0x48. This will force the memory row associated with parameter mb0l.q0b0r0 (physical location of Q0B0r0) to become a 64 MB software deconfigured row. The software deconfiguration will not occur until the replace command is executed and the system has been reset.

sppdsh (hw2b):query 0 D0mb0l.q0b0r0: configured to 64Mmb0l.q0b0r1: configured to 64Mmb0l.q0b1r0: configured to 64Mmb0l.q0b1r1: configured to 64Mmb0l.q0b2r0: configured to 64Mmb0l.q0b2r1: configured to 64Mmb0l.q0b3r0: configured to 64Mmb0l.q0b3r1: configured to 64Mmb0l.q1b4r0: configured to 64Mmb0l.q1b4r1: configured to 64Mmb0l.q1b5r0: configured to 64Mmb0l.q1b5r1: configured to 64Mmb0l.q1b6r0: configured to 64Mmb0l.q1b6r1: configured to 64Mmb0l.q1b7r0: configured to 64Mmb0l.q1b7r1: configured to 64M* * *(This is not a complete listing of the query output!!)* * *



Figure 19 Example cput command

WARNING When a memory row is software deconfigured, POST will deconfigure the rows on this memory board and all other memory boards installed in the system for the most optimal memory interleaving. This assures that all memory boards within the system are populated identically.

Toggle

To deconfigure a part in the system the toggle command could be used. The toggle command is only capable of activating or deactivating a component. toggle is not capable of deconfiguring a 128 Mb row to 64 Mb.

NOTE Syntax and man page information about the toggle command

toggle <node id> <A#|C#|M#|D#|I#|S#>

The toggle command is used to change the state of a part in the system between active and disabled. The A stands for agents or PACs, C stands for cpus, M stands for MACs, I stands for I/O devices, D stands for the dimms on memory board #, and S stands for SCI devices.

Figure 20 displays an example of using the toggle command to mark parameter mb0l.q0b0r0 to be software disabled. This will force the memory row associated with parameter mb0l.q0b0r0 (physical loca-tion of Q0B0r0) to become a 64 MB software deconfigured row. The software deconfiguration will not occur until the replace command is executed and the system has been reset.

Figure 20 Example toggle command

sppdsh$ cput 0 mb0l.q0b0r0 0x48mb0l.q0b0r0 0x48sppdsh$

sppdsh$ toggle 0 Dmb0l.q0b0r0mb0l.q0b0r0: deconfigured (0x48)sppdsh$



WARNING When a memory row is software deconfigured, POST will deconfigure the rows on this memory board and all other memory boards installed in the system for the most optimal memory interleaving. This assures that all memory boards within the system are populated identically.

Replace

When all modifications to the configuration parameters in the SSP’s buffer are complete, the SPPDSH command replace is used. The replace command downloads the configuration parameters from the SSP’s buffer to NVRAM. In order for the system to operate with the new values, a reset must be performed on the V2500/V2600.


replace <node#>

Downloads node # configuration parameters from the SSP’s buffer to NVRAM.



Task: Interpreting POST messages

With the new orientation of DIMMS on the memory board, an enhanced set of row status character definition codes were adopted. Table 11 provides a description for each of the row status characters.

Table 11 Row Status Character DIMM Definitions

Symbol Description

- 0 MB Installed

# Hardware Deconfigured to 0 MB

$ Software Deconfigured to 0 MB

. 32 MB Configured

: 64 MB Configured

| 128 MB Configured



Figure 21 displays the status of how a 32GB (256 Mb DIMMs) system was configured during POST without any memory deconfiguration problems.

Figure 21 POST messages from a V2500/V2600 system with 32 GB of memory

Figure 22 defines how a row maps to the quadrant and bus associated with each one of the status characters displayed in the POST message.

Figure 22 POST mapping of Quadrants, Buses and Rows

Probing Main Memory: MB0L MB1L MB2R MB3R MB4L MB5L MB6R MB7R Initializing Main Memory: Parallel memory initialization in progress

r0 r1 r2 r3 PB1R_B MB0L [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB1R_A MB1L [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB2L_A MB2R [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB3R_A MB3R [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB4L_A MB4L [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB5R_A MB5L [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB6L_A MB6R [|||| ||||][|||| ||||][|||| ||||][|||| ||||] PB7R_A MB7R [|||| ||||][|||| ||||][|||| ||||][|||| ||||]Building Main Memory Map:

Booting OBP

Row: r0 r1 r2 r3Status Character: PB1R_B MB0L [|||| ||||][|||| ||||][|||| ||||][|||| ||||]

Bus #: [0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7]

Quadrant #: 0 1 0 1 2 3 2 3

Q1B6R0



Figure 23 displays the status of how a 16GB (128 Mb DIMMs) system was configured during POST without any memory deconfiguration problems.

Figure 23 POST messages from a V2500/V2600 system with 16 GB of memory

Figure 24 defines how a row maps to the quadrant and bus associated with each one of the status characters displayed in the POST message.

Figure 24 POST mapping of Quadrants, Buses and Rows

Figure 25 can then be used to determine where the DIMM slot is located on the memory board. SPPDSH parameters are shown on the left and POST mapping is found on the right. The shaded slots represent one set of DIMMs installed in quadrant zero.

Probing Main Memory: MB0L MB1L MB2R MB3R MB4L MB5L MB6R MB7R Initializing Main Memory: Parallel memory initialization in progress

r0 r1 r2 r3 PB1R_B MB0L [:::: ::::][:::: ::::][---- ----][---- ----] PB1R_A MB1L [:::: ::::][:::: ::::][---- ----][---- ----] PB2L_A MB2R [:::: ::::][:::: ::::][---- ----][---- ----] PB3R_A MB3R [:::: ::::][:::: ::::][---- ----][---- ----] PB4L_A MB4L [:::: ::::][:::: ::::][---- ----][---- ----] PB5R_A MB5L [:::: ::::][:::: ::::][---- ----][---- ----] PB6L_A MB6R [:::: ::::][:::: ::::][---- ----][---- ----] PB7R_A MB7R [:::: ::::][:::: ::::][---- ----][---- ----]Building Main Memory Map:

Booting OBP

Row: r0 r1 r2 r3Status Character: PB1R_B MB0L [:::: ::::][:::: ::::][---- ----][---- ----]

Bus #: [0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7]

Quadrant #: 0 1 0 1 2 3 2 3

Q0B2R1



Figure 25 Physical EWMB Layout

7XHVGD\��'HFHPEHU��HULFY�PHP��SSW

Q0B3R0-Q0B3R1Q2B3R2-Q2B3R3








mbXl.q2b0r0 - mbXl.q2b0r1mbXl.q1b4r0 - mbXl.q1b4r1

mbXl.q3b4r0 - mbXl.q3b4r1

mbXl.q0b3r0 - mbXl.q0b3r1mbXl.q2b3r0 - mbXlq2b3r1






Memory DIMM (EWMB) Slot LocationsX=memory board number (0-7)

mbXl.q0b0r0 - mbXl.q0b0r1

60$&

67$&

POST MappingSPPDSH Parameter

Q0B3

Q2B3

Q1B7

Q3B7

Q0B1

Q2B1

Q1B5

Q3B5

Q0B2

Q2B2

Q1B6

Q3B6

Q0B0

Q2B0

Q1B4

Q3B4

(Even numbered rowscorrespond to a side ofthe DIMM which isclosest to theconnector.)

(Even numbered rowscorrespond to a side ofthe DIMM which isclosest to theconnector.)

Connector


Actions and TasksI/O Subsystem Tasks

I/O Subsystem Tasks

Task: Determine I/O Hardware Path

With the addition of a slot to each card cage, Figure 26 displays what the values are for the SAGA ASICs and the PCI slot numbers.

Figure 26 SIOB Card Cage


Actions and TasksMid-Plane (MIB, SCUB, MIBPB) Tasks

Mid-Plane (MIB, SCUB, MIBPB) Tasks

MIB

When replacing the MIB it is necessary to configure the customers SWID value into the new FRU.

SCUB

When replacing the SCUB it is necessary to restore the original OBP parameters to the state they were in before the SCUB was replaced. The SCUBs firmware versions must be verified to match those on the SSP and reload when applicable.

MIBPB

No functional changes between V22x0 power board for the ENRB and V2500/V2600 power board for the MIB.


Actions and TasksPower Subsystem Tasks

Power Subsystem TasksThe V2500/V2600 will allow hot swapping of a +48VDC node power supply when the n+1 rules apply. All V2500/V2600 systems will have 4 node power supplies. In order for n+1 to be in effect, the redundancy switch on the SCUB must be in the down position.

Task: Determining the State of Power Subsystem

This is a new task for V2500/V2600. Visual identification of the redundancy switch position on the SCUB is no longer necessary as this information is available from the T/S using the SPPDSH command pswitch.

NOTE Syntax & man page information about the pswitch command:

pswitch <node #>

Task: Hot Swap Replacement of Node Power Supplies

This is a new task for V2500/V2600. Hot swap replacement of a nps (node power supply) for the V2500/V2600 is given to the customer as a new feature. The V22x0 does not support hot swapping of a node power supply.

The first step is to use the SPPDSH command pce. pce identifies a node power supply failure via four status codes 0x70 - 73. These status codes are the same values used for a V22x0 system and can be found in the leds man page on the SSP. The pce command is explained in detail during the SSP tasks portion of this chapter.

Using the -r option of pce, set the system power flag to off. Figure 27 displays an example of using the -r option.

Move the on/off switch of the failed nps to the off position. Perform the necessary procedures to remove and replace the nps. Move the on/off switch of the new nps to the on position.


Actions and TasksPower Subsystem Tasks

Using the -r option of pce, set the system power flag to on. Figure 27 displays an example of using the -r option.

Verify the error has been removed from the system by executing the pce command.

The SPPDSH command blink is then used to toggle the state of light bar (ELEDB) to stop flashing after the node power supply is replaced.

NOTE Syntax information about the blink command:

blink <node #>

Figure 27 Steps performed when hot swapping a nps

sppdsh$ pce 0

Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4---- ------ ------------- ---- ------ -------- -------- -------- -------- 0 Normal ATTN 0x70 25 1 0000 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ pce 0 -r 0

Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4---- ------ ------------- ---- ------ -------- -------- -------- -------- 0 Normal ATTN 0x70 25 1 0000 Nominal Nominal Nominal Nominal sppdsh$sppdsh$ pce 0 -r 1

Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4---- ------ ------------- ---- ------ -------- -------- -------- -------- 0 Normal 0x00 25 1 0010 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ pce 0

Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4---- ------ ------------- ---- ------ -------- -------- -------- -------- 0 Normal 0x00 25 1 0010 Nominal Nominal Nominal Nominal sppdsh$sppdsh$ blink 0 sppdsh$


Actions and TasksCooling Subsystem Tasks

Cooling Subsystem TasksThe V2500/V2600 will allow hot swapping of cooling fan. All V2500/V2600 systems will have 6 cooling fans.

Task: On-Line replacement of Cooling Fans

This is a new task for V2500/V2600. Hot swapping of a cooling fan is made possible with newly designed fan trays.

The SPPDSH command pce identifies cooling fan failures via six status codes 0x5c - 61. The pce command is explained in detail during the SSP tasks portion of this chapter.

Mechanical interlocks within each fan assembly control DC power on/off switching for safety during individual fan assembly replacement.

The SPPDSH command blink is used to toggle the state of light bar (ELEDB) to stop flashing after the fan is replaced.

NOTE Syntax & man page information about the blink command:

blink <node #>

N+1 fans provided with V2500/V2600 will temporarily cool cabinet FRUs while replacing any 1 of 6 cooling fans. Each set of three fans in the Upper and Lower fan trays are marked (0, 1, 2) with silk screen.


Actions and TasksMass Storage Tasks

Mass Storage TasksNo functional changes between V22x0 and V2500/V2600 with the exception that newer I/O devices are now supported on V2500/V2600.


Actions and TasksService Support Processor Tasks

Service Support Processor Tasks

Task: do_reset (new features)

do_reset operates in the same manner on the V2500/V2600 hardware, however there are a few new features to the do_reset command.

NOTE Syntax information for the do_reset command:

do_reset [# | complex | all] [level] [boot option]

The new boot option feature allows the system boot up to different levels that could only be accomplished with the use of ccmu or xconfig on V22x0 hardware. Table 12 provides a listing of the parameter with a definition that may be used with the do_reset command.

Table 12 Boot Option Parameters

NOTE Since tc_standalone and tc_interactive are now boot option parameters to the do_reset command, their associated scripts have been removed from the /spp/scripts directory.

Parameter Definition

obp Reboots to OBP.

post_interactive Reboots to POST in the interactive mode.

tc_standalone reboots to the test controller in stand-alone mode

tc_interactive reboots to the test controller in interactive mode

loader reboots to the firmware loader



The loader parameter is a new feature in the V2500/V2600. When the loader option is used, the V2500/V2600 system will: reboot, execute the POST firmware, and then load a new firmware loading utility called PDCFL. See “Task: Loading Firmware using PDCFL” on page 69.

Task: pce vs. pce_util

For V22x0 hardware, pce_util is used to obtain the value of the environmental LEDs, current power, clock, temperature states and margin clocks in the node. This capability is still available but it has been incorporated within SPPDSH and has the new name of pce.

NOTE The command syntax for pce is provided for you and can also be obtained from the SPPDSH man page that resides in the SSP.

pce <node#| complex#|all> [-c <n|u|e>] [-r <on|off>] [-p <l|n|u>]

-c manipulate clocks on the current nodes.

n[ominal] nominal frequency

u[upper] upper frequency

e[xternal] external connector

-r sets the power flag to on or off

-p manipulate power on the current nodes.

n[ominal] nominal frequency

u[upper] upper frequency

e[xternal] external connector

Figure 28 displays an example of the output from the pce command.

Figure 28 Example output from the pce command

sppdsh$ pce 0Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4---- ------ ------------- ---- ------ -------- -------- -------- -------- 0 Normal 0x00 27 1 0000 Nominal Nominal Nominal Nominal



Task: Loading Firmware using PDCFL

PDCFL is a bootable module with the capabilities of loading other firmware modules into Flash RAM.

If a V22x0 system has a problem booting OBP firmware, the system can not be used. load_eprom was the only utility available to re-install the OBP firmware into Flash RAM and would take over two hours to install.

With PDCFL being a bootable module for V2500/V2600 systems, the system can be reset to boot PDCFL and re-install the OBP firmware. Re-installing OBP firmware with PDCFL will take approximately twelve minutes. Booting the system to load the PDCFL module can be performed by executing the following command on the SSP:

do_reset 0 1 loader

A successful boot of PDCFL displays a prompt similar to the shown below in the console window of the SSP:

PDCFL>

The fload command within PDCFL would then be used to re-install the OBP firmware into Flash RAM. Using the following command at the PDCFL prompt will reinstall OBP firmware:

fload obp2500.pdc 0xf0080000

After the fload command completes writing OBP to Flash RAM, resetting the system is necessary. The OBP parameters can then be restored using the nvrestore command at the OBP prompt.

PDCFL can be loaded into Flash RAM, via load_eprom command, through the JTAG port. Once PDCFL has booted, it can also be used to load other modules, such as POST and Test Controller, into Flash RAM through the SONIC port.

load_eprom supports a -f option for loading PDCFL into the appropriate sector in Flash RAM.

Two OBP parameters need to be setup to allow PDCFL to communicate with the SSP. These are ts-ip# and obp-ip#. The usual values are



ts-ip# 15.99.111.99

obp-ip# 15.99.111.150

The SSP needs to be setup to act as a tftp server for the files you wish to load into Flash RAM. This is accomplished for you via the /spp/scripts/ts.install command that is performed when the SSP software is upgraded. If additional information is required on how the SSP is configured as a tftp server, it can be obtained by reading the man page for PDCFL that resides on the SSP.

From the PDCFL prompt, the following commands are supported:

printenv [variable] - Prints configuration variables from NVRAM.

setenv [variable] [value] - Allows setting configuration variables in NVRAM.

reset [post] - Resets the node, optionally changing the boot vector to point to the POST module.

lifls - Prints a listing of the LIF volume into Flash RAM. The listing includes the name of the module, the Flash RAM address at which the module starts, the size in LIF units, the date the module was last written, and the sectors included by the module.

Example output from the PDCFL command lifls:

PDCFL> lifls

LIF Volume FLASH4

Name Addr Size Date Sectors

-----------------------------------------------------------------------

POST 0xF0020000 0x400 04/09/97 4-5

TC 0xF0140000 0x300 04/09/97 16-17

CPU3000 0xF0170000 0x300 04/09/97 17-18

DIODC 0xF01A0000 0x300 04/09/97 19-20

MEM3000 0xF01D0000 0x2F0 04/09/97 20-21

RDR_DUMPE 0xF01FF000 0x10 04/09/97 21

IO3000 0xF0260000 0x500 04/09/97 25-27

INTER3000 0xF02B0000 0x300 04/09/97 27-28

PDCFL 0xF02E0000 0x200 04/09/97 29

DFDUTIL 0xF0300000 0x200 04/09/97 30



fload [file] [location] - loads a file from the SSP tftp directory to the address in Flash RAM specified in the lif directory by name. Location can also be a specific address given in hex to allow loading files that have not yet been entered in the LIF directory. If this form is used, the LIF directory will not be updated.

Example output of installing POST firmware using the fload command:

PDCFL> fload post.fw POST

TFTP server : 15.99.103.191

CUB IP : 15.99.111.150

Reading : post.fw

Writing : POST (each ’.’ represents 4K copied)

Sector erased 0xF0020000

.......................................

Sector erased 0xF0040000

............

148384 bytes transferred

Task: Downloading Firmware to SCUB

This task is accomplished using the same commands on the V2500/V2600 as used on the V22x0. The primary commands used are load_eprom from the SSP and the OBP command source that uses the fwcp command from the /spp/scripts/dl-diags script.

Specific steps for downloading the utilities can be obtained from the SSP software release notes or by reading the man page for fw_install.


4) Troubleshooting Hints

Introduction

This chapter contains a little information about many things. Since this document won’t be updated after its release, we are providing as much information as is known at this time.


Troubleshooting HintsSome General Thoughts

Some General Thoughts

Stand alone Diagnostics

Just about every V22x0 Service Support Processor (SSP or Test Station) utility and offline diagnostic had to be revised and some even scrapped for the V2500/V2600 new CPUs and ASICs. Among the discarded were: ARCH3000 (some of its code now lives in CPU3000 and MEM3000); and INTRA3000 (some of its code now lives in INTER3000).

What this all means is that it is likely that some (yet to be discovered) utility errors may occur that are not caused by H/W faults or configuration errors.

Cheer up! There are still many H/W faults that are detected by the utilities and diagnostics. This is just a reminder that there are still some H/W faults that are not easy to troubleshoot even when the utilities find them.H/W faults during POST are the first that comes to mind; IO3000 is next; then there are always those hard errors from a “Responseless Request.”

One very positive point is that the V2500/V2600 hardware supports HP-UX and Mesa in a way that V22x0 H/W could not and may never.

HP-UX Chassis Codes

Chassis codes displayed in the control panel’s LCD while HP-UX is running must be able to indicate greater than 16 CPUs.

Chassis Code example: FnnF

where (n) = (% CPU utilization) and (n)=(# of CPUs)

Displaying greater than 16 CPUs requires stealing a bit from the previous digit. 16 CPUs is represented as F0FF; 32 CPUs as F1FF.

Web-based Troubleshooting for V2500/V2600

URL: http://fesweb.corp.hp.com/dewold/v2500/v-2500.htm


Troubleshooting HintsProcessor FRUs

Processor FRUs

EWPB & ELPB

Much more attention must now be paid to EWPB and ELPB faults as each PA8500 or PA8600 processor chip on the processor board is a Field Replaceable Unit and the EWPB or ELPB must be COPed when a CPU is both added or removed.

The difficulty is not identifying that a CPU failed, but in interpreting the error data’s naming convention that identifies CPU A/B on Runway 0/1 on PBnL or PBnR.

PCXW & PCXW+ (or CPU)

Thanks to a very robust cache design, D-Cache errors detected by PA8500 and PA8600 CPUs no longer HPMC.

Oh, in case you didn’t know, the floating point (FP) logic within every PA-RISC CPU has no error circuitry. Since POST does not test FP to any extent, it is necessary to run (CPU3000, subtest 570) periodically to ensure FP functionality.

Also, PCXW and PCXW+ chips have one more reason to HPMC as they detect a new Runway Bus estat code: 0xC (ESTAT_SYNCHRONIZER_ERR) The Runway synchronizer has detected a synchronize rule violation. As such, there are minor changes to the PCXW or PCXW+ Status 0 and 1 register definitions. See “Web-based Troubleshooting for V2500/V2600” (URL is on previous page).

Mesa

All PCXW and PCXW+ recoverable cache errors are logged as LPMCs by LOGTOOL within Mesa.



CPU3000 Tests

Class 1 subtests verify the most basic functionality of the processor chips. When testing processors it is recommended to execute all of the subtests within each class. The format for test times is (HH:MM:SS).

Table 13 CPU Class 1 Subtests (General)

Table 14 CPU Class 2 Subtests (Icache)

Table 15 CPU Class 3 Subtests (Dcache)

Table 16 CPU Class 4 Subtests (TLB)

Subtest Description 32 CPU test time100 CPU Minimal Functionality Test 00:00:05101 CPU ALU Test 00:00:05102 CPU Branch Test 00:00:05103 CPU Arithmetic Conditions Test 00:00:05104 CPU Bitwise Operations Test 00:00:05105 CPU Control Registers Test 00:00:05111 CPU Interval Timer Test 00:00:05120 CPU Multimedia Test 00:00:05130 CPU Shadow Register Test 00:00:05140 CPU Local Diagnostic Registers Test 00:00:05141 CPU Remote Diagnostic Registers Test 00:00:05150 CPU Register Bypass Test 00:00:05

Subtest Description 32 CPU test time210 Icache Ram Test 00:05:00

Subtest Description 32 CPU test time310 Dcache Ram Test 00:05:00

Subtest Description 32 CPU test time400 TLB Ram Test 00:00:30



Table 17 CPU Class 5 Subtests (Functional Tests)

Executing CPU3000

Executing the offline test controller diagnostics for a V2500/V2600 system is performed the same way as it is on a V22x0 system.

When the system is booted into test controller interactive mode, the CPU3000 diagnostic can be executed from the interactive menu driven interface within the console window.

If the system is booted into test controller standalone mode, the CPU3000 diagnostic can be executed from the GUI interface running the command cxtest -d from a SSP shell window. The cxtest command also has a command line interface. Detailed information about the cxtest command can be found in the cxtest man page on the SSP.

The figure on the following page shows an example of booting the system into test controller standalone mode and executing all of the CPU3000 subtests via the command line interface.

Subtest Description 32 CPU test time500 Re-executes early selftests (excluding cache) from icache 00:01:00510 Verifies access widths 00:00:30520 Verifies cache flush and purge operations 00:00:30530 Verifies instructions can be encached from coh. mem. 00:00:30540 Verifies data can be encached from coh. mem. 00:00:30550 Verifies dcache parity errors can be detected and handled 00:00:30560 Verifies functionality of the dcache store queue 00:00:30560 Test functionality of the TLB 00:00:30570 Verifies the floating point unit 00:00:30



Figure 29 Command line interface

fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone0:0xf000d00a90 0x010000000x01000000fesmesst:/users/sppuser$ cxtest -cpu1 Nodes foundCommand line 0 : -cpu : Number of args 1 Subtest 100 - CPU Minimal Functionality Test 0:0 0:00:05 passed Subtest 101 - CPU ALU Test 0:0 0:00:05 passed Subtest 102 - CPU Branch Test 0:0 0:00:05 passed Subtest 103 - CPU Arithmetic Conditions Test 0:0 0:00:05 passed Subtest 104 - CPU Bitwise Operations Test 0:0 0:00:06 passed Subtest 105 - CPU Control Registers Test 0:0 0:00:05 passed Subtest 111 - CPU Interval Timer Test 0:0 0:00:05 passed

*****This is not a complete listing*****


Troubleshooting HintsMemory FRUs

Memory FRUs

V-Class DIMM Slot Identification

The DIMM slot connectors on the EWMB of the V2500/V2600 look similar to those on the EMB of the V22x0.

However, each DIMM slot connector is identified with a different marking. This is because V22x0 memory boards are not compatible with V2500/V2600 memory boards.

V22x0 EMB DIMM Slot Accesses

With V22x0 EMBs, the connector markings use “Bn” to indicate Banks 0 - 3; and use “Sn” to indicate Slot 0 - 7.

Table 18 V22x0 DIMM Slot Markings

• All Row 0/1 accesses to DIMMs in Banks 0 - 3 go to slots labeled B0S0, B1S0, B2S4, and B3S4.




B0S0 B0S1 B0S2 B0S3

B1S0 B1S1 B1S2 B1S3

B2S4 B2S5 B2S6 B2S7

B3S4 B3S5 B3S6 B3S7



V2500/V2600 EWMB DIMM Slot Accesses

With V2500/V2600 EWMBs, the connector marking use “Qn” to indicate the Quadrant 0 - 3; and use “Bn” to indicate Bus 0 - 7.

Table 19 V2500/V2600 DIMM Slot Markings

• All Row 0/1 accesses to Banks 0/1 in DIMMs on Buses 0 - 3 are within Quadrant 0; slots are labeled Q0B0 through Q0B3.




NOTE The two banks (or sides) of each DIMM are not needed to identify the slot number.

See the following figure to understand how “Quadrant” is used as a DIMM slot identifier.

Q0B0 Q0B1 Q0B2 Q0B3

Q1B4 Q1B5 Q1B6 Q1B7

Q2B0 Q2B1 Q2B2 Q2B3

Q3B4 Q3B5 Q3B6 Q3B7



Figure 30 V2500/V2600 EWMB DIMM Slot Groupings:

Each box within the above figure represents a DIMM slot #.

This memory design allows mixed DIMM sizes to be managed on a Quadrant basis instead of by a Bank basis like V22x0. Both memory designs provide limited DIMM loading flexibility.

The first rule is that all DIMMs within a Quadrant must be the same size on all EWMBs.

The second rule is that all EWMBs must have identical Quadrant configurations.

Mesa

All single-bit ECC memory errors detected while running HP-UX are logged by Mesa and are no longer reported in DMesg buffer.

Double-bit ECC memory errors are logged into PDT after a hard error is signalled.

Q0 Q1

Q3Q2

Rows 0/1

0 1 2 3 4 5 6 7

V25020.ppt

Rows 2/3

Bus #

Q0B0

Q0B1

Q0B2

Q0B3

Q1B4

Q1B5

Q1B6

Q1B7

Q2B0

Q2B1

Q2B2

Q2B3

Q3B4

Q3B5

Q3B6

Q3B7


Troubleshooting HintsUsing the dcm script to troubleshoot a memory error

Using the dcm script to troubleshoot a memory error

This section discusses one way to troubleshoot a memory board or DIMM error.

If a screen similar to the following appears during POST run the dcm script to determine which DIMM (or DIMMs) may be faulty.

The “#” character in the following example indicates memory has been deconfigured.

For example:

r0 r1 r2 r3

PB0L_A MB0L [#### ....][#### ....][____ ____][____ ____] PB1R_A MB1L [.... ####][.... ####][____ ____][____ ____] PB2L_A MB2R [.... ####][.... ####][____ ____][____ ____] PB3R_A MB3R [.... ####][.... ####][____ ____][____ ____] PB4L_A MB4L [.... ####][.... ####][____ ____][____ ____] PB5R_A MB5L [.... ####][.... ####][____ ____][____ ____] PB6L_A MB6R [.... ####][.... ####][____ ____][____ ____] PB7R_A MB7R [.... ####][.... ####][____ ____][____ ____]

To run the dcm script:

Bring up the sppdsh prompt at a teststation window by entering:

$ sppdsh

Run the dcm script from the sspdsh prompt by entering:

sppdsh$ dcm 0

Output similar to the following example will be displayed:



Figure 31 Example V2500/V2600 dcm script, partial listing

The above figure shows a DIMM labeled -/-. This indicates that the DIMM has not been recognized during the boot process. POST typically marks only one DIMM at a time as -/-. Because of the failure in Q0B0 on EWMB0, POST automatically deconfigured the DIMMs in Q0B1, Q0B2, and Q0B3. The code L-/L- indicates that the DIMMs have been deconfigured. Because all of the memory boards must be configured identically, POST will also deallocate the equivalent amount of memory from every other memory board installed in this system leaving Q1B4, Q1B5, Q1B6, and Q1B7 deallocated.

MEM3000 Tests

The estimated test times are based on a system with 32 processors and 32 GB of memory.

Unrecognized DIMM

Deallocated DIMMS

Physical: L=128MB, M=64MB, S=16MB Logical: l=128MB, m=64MB, s=16MB

(If logical memory not specified, then it matches physical memory size)

* = Software Deconfigured - = Not In Use

EWMB0:======EWMB0: Q0B0 -/-EWMB0: Q0B1 L-/L-EWMB0: Q0B2 L-/L-EWMB0: Q0B3 L-/L-

Q1B4 L/LQ1B5 L/LQ1B6 L/LQ1B7 L/L

Q2B0 -/-Q2B1 -/-Q2B2 -/-Q2B3 -/-

Q3B4 -/-Q3B5 -/-Q3B6 -/-Q3B7 -/-

EWMB1:======EWMB1: Q0B0 L/LEWMB1: Q0B1 L/LEWMB1: Q0B2 L/LEWMB1: Q0B3 L/L

Q1B4 L-/L-Q1B5 L-/L-Q1B6 L-/L-Q1B7 L-/L-

Q2B0 -/-Q2B1 -/-Q2B2 -/-Q2B3 -/-

Q3B4 -/-Q3B5 -/-Q3B6 -/-Q3B7 -/-



Table 20 Memory Class 1 Subtests (Basic Functionality)

Table 21 Memory Class 2 Subtests (TAG Bytes)

Subtest Description 32 CPU test time100 Diagnostic CSR Read/Write Test 00:00:08101 Other EMAC CSR Read/Write Test 00:00:05110 Memory Data Read/Write Test 00:03:00120 Memory ECC Read/Write Test 00:00:33130 Memory Tag Read/Write Test 00:03:00140 Memory Initialization Test 00:00:30150 First 32 Memory Lines Test 00:00:30

Subtest Description 32 CPU test time200 Tag Bank Test 00:00:10210 Tag Addressing Test 00:05:30211 Tag Byte Uniqueness Pattern Test 00:05:00230 Tag March-C Pattern 1 Test 00:11:00231 Tag March-C Pattern 2 Test 00:11:00232 Tag March-C Pattern 3 Test 00:11:00233 Tag March-C Pattern 4 Test 00:11:00234 Tag March-C Pattern 5 Test 00:11:00235 Tag March-C Pattern 6 Test 00:11:00236 Tag March-C Pattern 7 Test 00:11:00237 Tag March-C Pattern 8 Test 00:11:00238 Tag User Data Pattern Test 00:11:00



Table 22 Memory Class 3 Subtests (Data Bytes)

Table 23 Memory Class 4 Subtests (Coherency and Semaphores)

Table 24 Memory Class 5 Subtests (ECC)

Subtest Description 32 CPU test time300 Memory Bank Test 00:04:00310 Memory Addressing Test 00:06:30311 Memory Byte Uniqueness Pattern Test 00:06:20330 Memory March-C Pattern 1 Test 00:08:50331 Memory March-C Pattern 2 Test 00:05:40332 Memory March-C Pattern 3 Test 00:05:40333 Memory March-C Pattern 4 Test 00:05:40334 Memory March-C Pattern 5 Test 00:05:40335 Memory March-C Pattern 6 Test 00:05:40336 Memory March-C Pattern 7 Test 00:05:40337 Memory March-C Pattern 8 Test 00:05:40338 Memory User Data Pattern Test 00:05:40

Subtest Description 32 CPU test time400 Memory Data Flush Transaction Test 00:00:10410 Memory Load/Store Test 00:00:15420 Memory Semaphore Transaction Test 00:00:15

Subtest Description 32 CPU test time500 ECC Single Error Correction Test 00:00:20510 ECC Double Error Detection (coherent) Test 00:00:45520 ECC Double Error Detection (non-coherent) Test 00:00:20530 ECC Disable Test 00:00:20



Table 25 Memory Class 6 Subtests (Miscellaneous Functionality)

Executing MEM3000

Executing the offline test controller diagnostics for a V2500/V2600 system is performed the same way as it is on a V22x0 system.

When the system is booted into test controller interactive mode, the MEM3000 diagnostic can be executed from the interactive menu driven interface within the console window.

If the system is booted into test controller standalone mode, the MEM3000 diagnostic can be executed from the GUI interface by running the command cxtest -d from a SSP shell window. The cxtest command also has a command line interface. Detailed information about the cxtest command can be found in the cxtest man page on the SSP.

The figure on the following page shows an example of booting the system into test controller standalone mode and executing all of the MEM3000 subtests via the command line interface.

Subtest Description 32 CPU test time600 Memory Access Protection Test 00:00:40610 Memory Tag Test I 00:00:30620 EMAC System Configuration CSR Test 00:00:30630 80 vs. 88 Bit DIMM Test 00:00:05



Figure 32 Executing MEM3000 from the command line interface

It is estimated that testing memory with MEM3000 will take approximately one quarter of the time to run on a V2500/V2600 as compared to a V22x0. This is due to the memory subsystem redesign.

fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone0:0xf000d00a90 0x010000000x01000000fesmesst:/users/sppuser$ cxtest -mem1 Nodes foundCommand line 0 : -mem : Number of args 1 Subtest 100 - Diagnostic CSR Read/Write Test 0:0 0:00:05 passed Subtest 101 - Other EMAC CSR Read/Write Test 0:0 0:00:05 passed Subtest 110 - Memory Data Read/Write Test 0:0 0:00:30 passed Subtest 120 - Memory ECC Read/Write Test 0:0 0:00:11 passed Subtest 130 - Memory Tag Read/Write Test 0:0 0:00:30 passed Subtest 140 - Memory Initialization Test 0:0 0:00:07 passed Subtest 150 - First 32 Memory Lines Test 0:0 0:00:11 passed

*****This is not a complete listing*****


Troubleshooting HintsI/O Subsystem FRUs

I/O Subsystem FRUs

SIOB

SIOB card cages are not interchangeable with previous EIOB card cages. Both, however, have the same DC power budget rules.

IO3000 Tests

Test times are estimates based on one SAGA and one peripheral selected. Test times are with parameters defaulted, this includes disk write enables set to False. Selecting more SAGA/devices or enabling write operations will increase test times.

Table 26 IO3000 Class 1 Subtests (SAGA CSRs)

Table 27 IO3000 Class 2 Subtests (Shared & Prefetch Memory)

Subtest Description Test time (1 SAGA)100 CSR reset test 00:00:10105 CSR read/write test 00:00:15110 Error CSR test 00:00:10

Subtest Description Test time (1 SAGA)200 Context/shared memory read/write test 00:00:10205 Context/shared memory access width test 00:00:10210 Context/shared memory march C- test 00:03:00215 Context/shared memory pattern test 00:03:00220 Context/shared memory parity detection test 00:00:10225 Prefetch memory read/write test 00:00:10230 Prefetch memory access width test 00:00:10235 Prefetch memory march C- test 00:01:30240 Prefetch memory pattern test 00:01:30245 Prefetch memory parity detection test 00:00:10



Table 28 IO3000 Class 5 Subtests (SCSI Disks)

Table 29 IO3000 Class 6 Subtests (Channel Builder)

Table 30 IO3000 Class 7 Subtests (DMA)

Subtest Description Test time (1 SAGA)500 SCSI Disk test unit ready test 00:00:15505 SCSI Disk inquiry test 00:00:15510 SCSI Disk read capacity test 00:00:15515 SCSI Disk read test 00:00:15520 SCSI Disk write test 00:00:15

Subtest Description Test time (1 SAGA)600 Channel init, ATPR = 0x0 00:00:15605 Channel build, ATPR = 0x0 00:00:15610 Channel init, data prefetch, ATPR = 0x2 00:00:15615 Channel build, ATPR = 0x2 00:00:15620 Channel init, write tlb, ATPR = 0x8 00:00:15625 Channel init, write tlb, data prefetch, ATPR 00:00:15630 Channel init, tlb prefetch, ATPR = 0xc 00:00:15635 Channel build, ATPR = 0xc 00:00:15640 Channel init, tlb & data prefetch, ATPR = 0xe 00:00:15645 Channel build, ATPR = 0xe 00:00:15650 Channel context access test 00:00:15

Subtest Description Test time (1 SAGA)700 External interrupt test 00:00:15705 DMA across page and channel 00:00:15710 Jump forward within a page 00:00:15715 Jump backward within a page 00:00:15720 Jump outside of a page (TLB encached) 00:00:15725 Jump outside of a page (TLB not encached) 00:00:15730 Jump outside of a channel 00:00:15735 Non contiguous TLB’s 00:00:15740 DMA byte swapping 00:00:15



Table 31 IO3000 Class 8 Subtests (Multi-Disks)

Table 32 IO3000 Class 12 Subtests (Symbios SCSI PCI Cards)

Executing IO3000

Using IO3000 test controller diagnostic requires the use of a switch and few options to the cxtest command. These are:

IO3000 Switch

io - This switch causes cxtest to execute the IO3000 diagnostic.

Options

c - To select an entire class of subtests use the -c <number> option. The user can specify a range of classes by using a hyphen between the numbers. As an example -c 2-4, would run classes 2,3 and 4.

The user can also specify a list of classes by placing a "," between the numbers. An example would be -c 4,7,2. This would run class 4 then class 7 and finally class 2.

s - To select a subtest use the -s <number> option. The user can specify a range of subtests by using a hyphen between the numbers. As an example -s 100-150, would run subtests 100 through 150.

Subtest Description Test time (1 SAGA)800 Multi-disk non-mixed traffic 00:03:30805 Multi-disk mixed traffic, ATPR = 0xe 00:03:30810 Multi-disk mixed traffic, ATPR = 0xf 00:03:30

Subtest Description Test time (1 SAGA)1200 Symbios SCSI PCI configuration space test 00:00:101205 Symbios SCSI PCI I/O and Memory space test 00:00:101230 Symbios SCSI Scripts RAM test 00:00:101240 Symbios SCSI Interrupt test 00:00:401250 Symbios SCSI DMA engine test 00:00:10



The user can also specify a list of subtests by placing a "," between the numbers. An example would be -s 100,150,140. This would run subtest 100 then subtest 150 and finally subtest 140.

pa# - To select a parameter word use the -pa# <number> option. The # symbol represents the parameter word to use.

The test controller environment allows IO3000 to have 20 words of user parameters. Words 8 –19 are used for device specification. Due to Core Logic SRAM space limitations, only 20 devices per SAGA can be tested at a time. Up to 24 devices can be specified via parameter words 8-19. Each of the parameter words is broken down into 2 device specifications as explained in Figure 33.

Devices are numbered according to their position in the parameter list. A device can be specified in any of the device specification locations in user parameter space. An unused device parameter should be initialized such that the slot field is 0xf (i.e. device spec of 0x0f00). Therefore if both device parameters in a given parameter word are unused, the parameter word would be set to 0x0f000f00.

As an example, to specify a disk on SAGA 0x4, slot 0x2, SCSI id 0xa, SCSI lun 0x0, one might set parameter word 8 to 0x42a00f00. Note that the lower (right) half of the parameter word has the slot field set to the 0xf. The device number is 0 since it was entered in device 0 parameter location.



Figure 33 Parameter Words 8 through 19

The figure on the following page shows an example of booting the system into test controller standalone mode and executing IO3000 using the command line interface. The command line is using parameter word 8 to execute the class 12 Symbios SCSI Card tests for a the device installed on SAGA 1 in slot 0 and subtests 500 through 515 to test the SCSI disk drive which is installed on SAGA 1 in slot 0 at address 3 with a LUN number of 0.

WARNING In Table 28, notice that subtest 520 is a SCSI disk write test. If this test is selected, data on the disk drive will be destroyed. Also notice that the example in Figure 34 is not executing subtest 520.

Parameter Word 8 DEV 0 DEV 1


**************THROUGH**************


Selecting Devices with Parameter Words 8 through 19

Parameter Word 8 = 0 1 0 0 0 F 0 0

Parameter Word 9 = 4 2 A 0 5 1 6 0

LUN #

SCSI ID #

SLOT #

EPIC #LUN #

SCSI ID #

SLOT #

EPIC #

0F00 = No device selected



Figure 34 Executing IO3000 from the command line interface

Table 33 provides a listing of values that could be used for parameter word 8 to test a single disk device for each one of the possible SAGA and slot numbers in a V2500/V2600.

NOTE In the following table, x represents the SCSI ID number; y represents the LUN number

Table 33 Possible values for Parameter word 8.

fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone0:0xf000d00a90 0x010000000x01000000fesmesst:/users/sppuser$ cxtest -io -pa8 0x10300f00 -c 12 -s 500-5151 Nodes foundCommand line 0 : -io : Number of args 7 Class 12 Subtest 1200 - Symbios SCSI PCI configuration space test 0:0 0:00:10 passed Subtest 1205 - Symbios SCSI PCI I/O and Memory space test 0:0 0:00:09 passed Subtest 1230 - Symbios SCSI Scripts RAM test 0:0 0:00:09 passed Subtest 1240 - Symbios SCSI Interrupt test 0:0 0:00:26 passed Subtest 1250 - Symbios SCSI DMA engine test 0:0 0:00:09 passed Class 12 passed Subtest 500 - SCSI Disk test unit ready test 0:0 0:00:15 passed Subtest 505 - SCSI Disk inquiry test 0:0 0:00:15 passed Subtest 510 - SCSI Disk read capacity test 0:0 0:00:16 passed Subtest 515 - SCSI Disk read test 0:0 0:00:12 passed

SAGA Slot pa8 value

4 0 0x40xy0f00

4 1 0x41xy0f00

4 2 0x42xy0f00

0 0 0x00xy0f00

0 1 0x01xy0f00

0 2 0x02xy0f00



0 3 0x03xy0f00

5 0 0x50xy0f00

5 1 0x51xy0f00

5 2 0x52xy0f00

1 0 0x10xy0f00

1 1 0x11xy0f00

1 2 0x12xy0f00

1 3 0x13xy0f00

6 0 0x60xy0f00

6 1 0x61xy0f00

6 2 0x62xy0f00

2 0 0x20xy0f00

2 1 0x21xy0f00

2 2 0x22xy0f00

2 3 0x23xy0f00

7 0 0x70xy0f00

7 1 0x71xy0f00

7 2 0x72xy0f00

3 0 0x30xy0f00

3 1 0x31xy0f00

3 2 0x32xy0f00

3 3 0x33xy0f00

SAGA Slot pa8 value


Troubleshooting HintsPower Subsystem FRUs

Power Subsystem FRUs

+48VDC Power Supplies

New SPPDSH command pswitch reports on the state of the n+1 switch on the SCUB.

The SPPDSH command pce is used to identify if an environmental error exist for a node power supply (NPS).The table below identifies the failed NPS when any of the four hexadecimal codes are found.

Table 34 Node Power Supply Status Code Identification

Hex CodeNode Power

Supply

0x70 Upper Left

0x71 Upper Right

0x72 Lower Left

0x73 Lower Right


Troubleshooting HintsCooling Subsystem FRUs

Cooling Subsystem FRUs

N+1 Cooling Fan Panels

The table below identifies the failed cooling fan when any of the six hexadecimal codes are found using the SPPDSH command pce. Upper Right implies a fan on the right, inside the top most fan panel when viewed from the rear of the system. Lower Left implies a fan on the left, inside the bottom most fan panel when viewed from the rear of the system.

Table 35 Cooling Fan Status Code Identification

NOTE Hexadecimal codes 5d and 60 are not implemented in V22x0 servers as they only use four cooling fans (two fans per panel).

Hex Code Cooling Fan

5c Upper Right

5d Upper Middle

5e Upper Left

5f Lower Right

60 Lower Middle

61 Lower Left


Troubleshooting HintsMass Storage FRUs

Mass Storage FRUs

PCI Cards

Two different choices here: F/W SCSI card for HVD devices and Ultra 2 SCSI card for LVD devices.

SCSI Devices

Both LVD and HVD disk types cannot be mixed on the same SCSI Bus.


Troubleshooting HintsService Support Processor Utilities

Service Support Processor Utilities

CCMD Differences

ccmd is the daemon that maintains a shared memory buffer of information on the V2500/V2600 hardware. ccmd also monitors the system and reports any significant complex events back to the SSP.

There are two types of related information in the shared memory buffer: cabinet information (node numbers and IP addresses) of the complex and scan information, which is derived from the hardware itself. ccmd periodically broadcasts to determine what cabinets are available. If ccmd cannot talk to a cabinet that it previously reached, it sends an event to the console and the log. If it establishes or re-establishes contact, or if a cabinet powers up, ccmd will read hardware information in EEPROM.

At power on of the system, ccmd performs a scan operation to build the boot configuration map. The boot configuration map contains information about the installed components. POST uses the boot configuration map to test the installed components.

Once running, ccmd checks for power-ups, power-downs, errors and environmental conditions at regular intervals. If at any time ccmd detects a change in the configuration, it changes the database, and writes the /spp/data/config/complex.cfg file for each applicable cabinet. These files are required for Excalibur System Test. If ccmd detects an error condition, it kicks off an error interrogator that logs and diagnoses error conditions.

ccmd is no longer responsible for building the node_0.cfg database file that was used by EST. This responsibility has been replaced by the est_config command.



EST Differences

EST, est_config & the node_0.cfg file

Though nothing about how EST works should have changed, there are some things that EST users need to be aware of for use on the V2500/V2600. EST will now use the est_config command to generate the node_0.cfg file EST uses to get information about the system under test. This is a change in the way the node_0.cfg file is generated.

To scan a system, EST needs various information about that system. ccmd must be running on a SSP to gather certain data about the system. When ccmd is done checking a system, EST can be run. EST will then decide whether or not to run est_config to generate the node_0.cfg file in /spp/data. When ccmd has done its job and EST has decided (by itself or with user input) whether or not to run est_config, a user should be ready to start scanning.

EST decides whether or not to run est_config by looking at the node_0.cfg and node_0.pwr files. If the machine was power cycled since est_config was last run, EST will automatically run est_config to generate the node_0.cfg file. If the node_0.cfg was updated after the power file, est_config will NOT be run. The node_0.cfg file is generated by est_config when run, and the node_0.pwr file is generated by ccmd when the machine is power cycled. Both the node_0.pwr and node_0.cfg files usually live in /spp/data.

A user can tell EST to run est_config with the -Y option (ex. prompt% est -Y 0). This option runs est_config regardless of the timestamps on the power and config files. A user can also tell EST not to run est_config with the -Z option (ex. prompt% est -Z 0). Though using this option is not recommended, -Z will tell EST not to run est_config even if the power file is newer than the config file.



Hard_Logger Master ID Differences

The Master ID is a 6-bit field extracted from the failed MAC’s Error Info CSR bits[31:36] and must be manually decoded to identify the “Initiator” of the request.

The Master ID value is placed into a hard_logger output when appropriate. In the case of the hard_logger event “Responseless Request or Response to incorrect state,” the extractor “r_incorrstate_resples_req” provides the user with the Master ID field.See “Web-based Troubleshooting for V22x0/V2500/V2600.”

The 6-bit field is really two separate 3-bit fields: source (bits[31:33]) and device # (bits[34:36]). See the two tables and figures below for specific details.

Table 36 Decoding the Master ID Value for V22x0 FRUs

Source (Bits[31:33])Device #

(Bits[34:36])

Value = 0 is Runway 0 (PBnL) Value = 0 is ASIC #0

Value = 1 is Runway 1 (PBnR) Value = 1 is ASIC #1

Value = 2 is I/O (EPIC) Value = 2 is ASIC #2

Value = 3 is (unused) Value = 3 is ASIC #3

Value = 4 is Data Mover Value = 4 is ASIC #4

Value = 5 is JTAG Value = 5 isASIC #5

Value = 6 is ETAC Value = 6 is ASIC #6

Value = 7 is EMAC Value = 7 is ASIC #7



Figure 35 V22x0 Processing Module pair and EPAC

Table 37 Decoding the Master ID Value for V2500/V2600 FRUs

Figure 36 V2500/V2600 Processing Module pairs and SPAC

Source (Bits[31:33])Device #

(Bits[34:36])

Value = 0 is Runway 0, CPU A (PBnL) Value = 0 is ASIC #0

Value = 1 is Runway 1, CPU A (PBnR) Value = 1 is ASIC #1

Value = 2 is I/O (SAGA) Value = 2 is ASIC #2

Value = 3 is Runway 1, CPU B (PBnR) Value = 3 is ASIC #3

Value = 4 is Data Mover Value = 4 is ASIC #4

Value = 5 is JTAG Value = 5 isASIC #5

Value = 6 is STAC Value = 6 is ASIC #6

Value = 7 is Runway 0, CPU B (PBnL) Value = 7 is ASIC #7

PnLAgent

0

1

PBnR

PBnL

V25035.ppt

toX-Bar

to EPIC

fromX-Bar

RUNWAY

D-Cache

PCXU+

DC to DCPowerSupply

DC to DCPowerSupply I-Cache

D-Cache

PCXU+

DC to DCPowerSupply

DC to DCPowerSupply I-Cache

PnLAgent

0

1

PBnR

PBnL

V25040.ppt

toX-Bar

to SAGA

fromX-Bar

RUNWAY

PCXWCPU A

PCXWCPU B



PCXWCPU A

PCXWCPU B




5) Theory of Operation

Introduction

This chapter is not intended to explain all V2500/V2600 functions in detail but to give some explanation of some important functions to better understand the differences from V22x0 in the following areas:

• V-Class H/W block diagrams showing balanced parallelism

• PA8500/PA8600 adaptations

• V-Class memory addressing formats

• V-Class Memory Board block diagrams showing DIMM accessing

• Miscellaneous V2500/V2600 subsystem functions


Theory of Operations

Figure 37 V22x0 System Block Diagram

The above figure shows all the V22x0’s main computing FRUs, their inter-connections and FRU numbering. DIMM slot locations are shown on the right side of the slide as the 16 short vertical lines within the eight Memory Boards. Note that the relative location of the first DIMM for Banks 0 - 3 are identified using 4 bolder vertical bars within each Memory Board.

Key Points

All EPIC (B) memory accesses use R0L and R2R X-Bar chips; all EPIC (A) memory accesses use R1L and R3R X-Bar chips.

Maximum configuration of PCI cards and EIOBs are evenly distributed (or balanced) across all available memory configurations.

Memory accesses are the bottleneck to increased performance.

1XPIC

1XPIC

P0LAgent

P1RAgent

U U

U U

1XPIC

1XPIC

P2LAgent

P3RAgent

U U

U U

1XPIC

1XPIC

P4LAgent

P5RAgent

U U

U U

1XPIC

1XPIC

P6LAgent

P7RAgent

U U

U U

0

0

1

1

0

1

0

1

1

0

1

0

1

1

0

0

R2RXBar

R0LXBar

R3RXBar

R1LXBar

CTI

CTI

CTI

CTI

CTI

CTI

CTI

CTI

IORF

IOLF

IORR

IOLR

0

4

2

6

1

5

3

7

1

3

4

6

EMAC

EMAC

EMAC

EMAC

EMAC

EMAC

EMAC

EMAC

32-bit PCIEIOBs

Single-PCXU+EVPBs MIDPLANE

(ENRB) MEMORYEMBs

v25015.ppt

(B)

(A)

(B)

(A)

(B)

(A)

(B)

(A)

MB0L

MB2R

MB4L

MB6R

MB1L

MB3R

MB5L

MB7R

PB0R

PB0L

PB1L

PB1R

PB2R

PB2LPB3R

PB3LPB4R

PB4LPB5R

PB5LPB6R

PB6LPB7R

PB7L

B0S0

B2S4

B3S4

B1S0

012

012

012

012

012

012

012

012


Theory of Operation

Figure 38 V2500/V2600 System Block Diagram

The above figure shows all the V2500/V2600 main computing FRUs, their inter-connections and FRU numbering. DIMM slot locations are shown on the right side of the slide as the 16 short vertical lines within the eight Memory Boards. Note that the relative location of the four DIMMs within Quadrant 0 are identified using 4 bolder vertical bars within each Memory Board.

Key Points

All SAGA (B) memory accesses use R0L and R2R X-Bar chips; all SAGA (A) memory accesses use R1L and R3R X-Bar chips.

Maximum configuration of PCI cards and SIOBs are not evenly distributed (or balanced) across all available memory configurations.

X-Bar accesses are the bottleneck to increased performance.

2XPIC

2XPIC

P0LAgent

P1RAgent

W W W W

W W W W

2XPIC

2XPIC

P2LAgent

P3RAgent

W W W W

W W W W

2XPIC

2XPIC

P4LAgent

P5RAgent

W W W W

W W W W

2XPIC

2XPIC

P6LAgent

P7RAgent

W W W W

W W W W

0

0

1

1

0

1

0

1

1

0

1

0

1

1

0

0

R2RXBar

R0LXBar

R3RXBar

R1LXBar

CTI

CTI

CTI

CTI

CTI

CTI

CTI

CTI

IORF

IOLF

IORR

IOLR

0

4

2

6

1

5

3

7

1

3

4

6

SMAC

SMAC

SMAC

SMAC

SMAC

SMAC

SMAC

SMAC

64-bit PCISIOBs

Dual-PCXWEWPBs MIDPLANE

(MIB) MEMORYEWMBs

v25010.ppt

(B)

(A)

(B)

(A)

(B)

(A)

(B)

(A)

MB0L

MB2R

MB4L

MB6R

MB1L

MB3R

MB5L

MB7R

PB0R

PB0L

PB1L

PB1R

PB2R

PB2LPB3R

PB3LPB4R

PB4LPB5R

PB5LPB6R

PB6LPB7R

PB7L

Q0B0

Q0B2

Q0B3

Q0B1

0123

0123

0123

0123

012

012

012

012


Theory of OperationsProcessor Modules

Processor Modules

PA8500/PA8600 Adaptations for V2500/V2600

Several considerations are needed to allow sharing a common Runway Bus by two PCXW or PCXW+ CPUs on the same EWPB.

For the V2500/V2600 implementation, a single PCXW/PCXW+ chip per board must be loaded in the socket for CPU A. The socket for CPU B does not function when the socket for CPU A is not loaded. See figure below for details.

Figure 39 Runway Bus Signal Sharing between PCXW CPUs

When both CPUs are installed, a different set of 32 signal lines from the shared Runway Bus are terminated by each processor chip to equalize skew, loading, etc. This is not the case, however, when only CPU A socket is loaded as all 64 signal lines are terminated by CPU A.

For some CPU A faults, the entire processor board may appear to not function and would be de-configured by POST.

PnLAgent

0

1

V25045.ppt

toX-Bar

to SAGA

fromX-Bar

RUNWAY

CPU A

CPU B



T

T

T

T 64

32

32


Theory of OperationProcessor Modules

Runway Bus Arbitration

Unlike K-Series where the host manages arbitration for all of Runway’s clients, Runway arbitration is determined by the CPU pairs. The Processor Agent Controller (SPAC), as the host for both Runway Buses, is not involved.

Since Runway buses have not changed, single threaded applications that are Runway Bus limited today are not expected to have any significant performance increase. Applications that are not Runway Bus limited will see increased performance that is only limited by memory latency and internal CPU instruction execution rate.

The increased CPU clock rate increases the internal execution rate by a proportionate amount relative to V22x0, although there is some degradation due to the smaller data and instruction caches within the PCXW and PCXW+.

The maximum application performance increase is 1.8x (the ratio of the PCXU+ to PCXW core clock rate increase). Other internal CPU related changes that result in performance increases are: 4-way set associative data and instruction caches, more TLB entries, and a larger branch history table.


Theory of OperationsMemory Modules

Memory ModulesThe V2500/V2600 memory module’s design greatly improves sustainable memory bandwidth. These improvements are a result of allowing more memory operations to occur concurrently as well as increasing the number of managed banks of coherent memory.

The result of these improvements is an approximate 2x increase in maximum sustainable memory bandwidth (~1.7x for real applications) compared to SPP2000/V22x0 hardware.

The latency of the memory system as measured from the processor for an idle system remains at roughly 500 nano-seconds. There is a small latency improvement due to the PCXW and PCXW+ clock rate increase (and thus a corresponding decrease in the amount of time to issue memory requests). There’s a large decrease in latency when the system is not idle due to the increase in sustainable bandwidth of the memory system.

This latency decrease is observed as better scalability of multi-threaded applications. The design requirement for the memory system yields a 2x sustainable memory bandwidth increase when comparing the latency versus bandwidth aspects of the SPP2000/V22x0 versus the V2500/V2600.

With all the changes to increase memory bandwidth, the format for the 40-bit coherent memory address also changes for V2500/V2600.


Theory of OperationMemory Modules

V-Class Memory Addressing Formats

Like the PA8200 CPU, the PA8500 and PA8600 CPU conforms to the same 40-bit Runway Bus addressing format.

Unlike the V22x0 before it, the V2500/V2600 architecture does not support the same 40-bit addressing format, it uses yet another format.

Unfortunately, Runway’s addressing format is different than the 40-bit addressing formats used by V-Class systems.

Each V-Class’s implementation is streamlined to get maximum performance from PA-RISC processors. As such, memory configurations only support balanced memory board and DIMM slot configurations that get maximum performance.

The V22x0’s addressing format maps a modified 40-bit Runway Bus memory address to a 16GB coherent memory address space; the V2500/V2600 addressing format maps it to a 32GB coherent memory space.

Memory configurations with less than the maximum number of memory boards and or less than the maximum number and size of DIMMs will leave gaps or holes within the memory space. But non-contiguous coherent memory space does not inhibit performance.

Both V-Class memory addressing formats require some knowledge of the memory interleaving algorithms used by EPAC/SPACs to modify the original 40-bit Runway Bus address to correctly map 32 byte cache lines into the appropriate DIMM on the appropriate board, bus/bank, and row for storage and retrieval.

Utilities are available either from the SSP or the web to decode either 40-bit coherent memory address formats.

Figures 40 and 41 show the V22x0 and V2500/V2600 40-bit addressing formats and how they differ from Runway’s format.

Figures 42 and 43 show the V22x0 and V2500/V2600 memory board DIMM accesses.



Figure 40 V22x0 Coherent Memory Address Modification

All V22x0 EPACs must modify the Runway 40-bit coherent memory addresses from up to two CPUs to interleave all banks on all EMBs. This is accomplished prior to routing the memory addresses through the X-Bar to a specific (even/odd) EMB.

Two decisions are derived by the interleaving algorithm that examines bits[33:37] and bits[52:58] of the 40-bit Runway address on top. Not shown are the different bit combinations chosen within the Page Offset bits dependent upon the total number of EMB pairs.

The EMB # and the bank #, derived by the algorithm, are written into bits[33:37] and the bank value is repeated in bits[59:61] of the 40-bit V22x0 address. This new 40-bit address is included within the packet along with the transaction type and sent to the EMB # derived from bits[33:35].

Each EMAC, upon receiving the packet, uses Row # from bits[30:32] and Bank # from bits[59:61] to access the appropriate DIMM slot.

The EPAC uses the unchanged value of bit [58] to determine which of the two X-Bar ports (even/odd) to route the packet to the EPAC.

V25025.ppt

Node ID - Row - VR- VB - Memory Page - Page Offset - B - E - H- 00

24 28 -30 32-33 -36 -38 51-52 58-59-60-61- 63

Mem BlockMem BankInterleave

MemoryBoard Map

5

VR RI VB BI

3

3 25 3

- Memory Page - Page Offset - Line Offset

24 -30 51-52 58-59 63

14 7 5

Note: Bit 29 of theV22x0’s 40-bit memory adddress is not used.

0

NotUsed



Figure 41 V2500/V2600 Coherent Memory Address Modification

All V2500/V2600 SPACs must modify the Runway 40-bit coherent memory addresses from up to four CPUs to interleave all buses and banks on all EWMBs. This is accomplished prior to routing the memory addresses through the X-Bar to a specific (even/odd) EWMB.

Four decisions are derived by the interleaving algorithm that examines bits[33:40] and bits[51:58] of the 40-bit Runway address on top. Not shown are the different bit combinations chosen within the Page Offset bits dependent upon the total number of EWMB pairs.

The EWMB #, the bus #, and the bank #, derived by the algorithm, are written into bits[33:40] of the 40-bit V2500/V2600 address. This new 40-bit address is included within the packet along with the transaction type and sent to the EWMB # derived from the algorithm.

Each SMAC, upon receiving the packet, uses Row # from bits[31:32], Bank # from bits[33:34], Bus # from bits[35:37] and Board # from bits[38:40] to access the appropriate DIMM slot and side.

V25030.ppt

MemoryBank

Interleave

8

24 2

- Memory Page

11 7

NodeID

UpperPage

InterleaveBase

LowerPage

PageOffset

LineOffset

ForceNode IDFunction

MemoryRowMaps

2

- Line Offset

- Page Offset

Row0

Note: Bit 24 of the V2500’s 40-bit memory address is not used.

MemoryBus

Interleave

MemoryBoard

Interleave

MemoryBoardMap

Interleave Base

Interleave Index8

3

3

5

3

24 28-29 30-31 33 40-41 51-52 58-59 63

24 -29 51-52 58-59 63Not

Used



The SPAC uses the unchanged value of bit [58] to determine which of the two X-Bar ports (even/odd) to route the packet to the SMAC.

Figure 42 V22x0 Memory Block Diagram

V22x0 Memory Interleaving

Sequential addresses are interleaved up to 32-way in the following order: first by Board #, then by (Even/Odd) Bank #, and lastly by Row #.

The two sides of a DIMM (Even/Odd) have a logical association with Row numbers (shown within each DIMM above).

NOTE The Row bits provide the starting point for interleaving’s modulo sequencing.

Supports 2/4 bank interleaving for both like/mixed DIMM sizes up to 32-way.

The V22x0’s memory boards can support up to two memory accesses (one Even Bank and one Odd Bank) simultaneously to any Row.

O E

V25060.ppt

EMAC

A

B

Rows: 0 1

O E

O E

O E

2 3

O E

O E

O E

O E

4 5

O E

O E

O E

O E

6 7

O E

O E

O E

O E

Banks

0

2

1

3

DIMMs

Even

Odd

ETAC

Y

Y

X

XB0 S0 B0 S1 B0 S2 B0 S3

B1 S0 B1 S1 B1 S2 B1 S3

B2 S4 B2 S5 B2 S6 B2 S7

B3 S4 B3 S5 B3 S6 B3 S7



EMBs have many memory holes when all DIMM slot are not occupied with the largest DIMM size. Contiguous memory available to HP-UX is only 32 MB with the smallest DIMM size and is therefore not supported in Rows 0/1.

Figure 43 V2500/V2600 Memory Block Diagram

V2500/V2600 Memory Interleaving

Sequential addresses are interleaved up to 256-way in the following order: first by Board #, then by Bus #, and lastly by (Even/Odd or by Odd/Even) Bank #. The two sides of a DIMM (0 & 1) have a logical association with Bank numbers (shown within each DIMM above).

A single DIMM on a Bus represents 2 Banks (0 & 1); two like sized DIMMs on the same Bus represent 4 Banks (0-3); two different sized DIMMs on the same Bus represent Banks (0 & 1) for each DIMM.

NOTE The Row bits provide the starting point for interleaving’s modulo sequencing.

V25065.ppt

SMAC

A

Rows:01

DIMMs

STAC

Y

Y

X

XBus:

0

1

0

1

0

1

0

1

N B C

1 2 30

Quad 0

0

1

0

1

0

1

0

1

N B C

5 6 74

Quad 1

B

0

1

0

1

0

1

0

1

N B C

Quad 2

210 3

0

1

0

1

0

1

0

1

N B C

Quad 3

654 7

Rows: 2

3

Bus:

Q2B

0

Q2B

1

Q2B

2

Q2B

3

Q0B

0

Q0B

1

Q0B

2

Q0B

3

Q1B

4

Q1B

5

Q1B

6

Q1B

7

DIMMs

Q3B

4

Q3B

5

Q3B

6

Q3B

7



For mixed DIMM sizes: supports 32 MB and 64 MB size DIMM combinations or 64 MB and 128 MB DIMM combinations or 32 MB and 128 MB DIMM combinations.

For like DIMM sizes: supports Quadrants 0-3 (4/8-bus 2/4 bank) interleaving up to 256-way.

Quadrants 0-3 may contain any supported DIMM size combination, however, the best performance is achieved with the most interleaves.

The V2500/V2600 memory boards can support up to eight memory accesses (one per Bus) simultaneously to any Row or logical Bank.

EWMBs have fewer memory holes when all DIMM slots are not occupied with the largest DIMM size. Contiguous memory available to HP-UX is now 64 MB with the smallest DIMM size and is supported in Rows 0/1 of Quadrant 0.

Commercial vs. Technical Systems

V2500/V2600 servers uses two modes of operation of the coherency protocol to optimize system performance for commercial versus technical systems.

1.) Commercial systems use two new Runway bus transactions. These transactions are notification of private clean cast outs and cache-to-cache writes for 3rd party access to private clean data.

The notification of private clean cast outs results in a 10-12% increase in performance for TPC-C applications. The cache-to-cache writes for 3rd party access to private clean data results in additional TPC-C application performance due to decreased access latency observed by a processor for 3rd party private clean accesses.

2.) For technical systems the coherency protocol includes the V22x0’s read hint mechanism to increase sustainable memory bandwidth. Technical systems disable notification of private clean cast outs. These changes result in an increase in usable memory bandwidth of roughly 30%. This usable memory bandwidth increases SPECrate_fp95 results by 10%.


Theory of OperationI/O Modules

I/O ModulesThe increase in peak I/O bandwidth for the SAGA I/O adapter allows a single PCI controller to achieve a larger peak bandwidth. However, in order to increase the aggregate system I/O throughput of the V2500/V2600 requires reworking system software at a later date.

The I/O subsystem supports the 2x PCI bus. This will double the peak I/O bandwidth capabilities of the hardware, to approximately 174MB/sec peak sustainable data transfer rate (bus transfer rate is 240MB/sec, but doesn’t count address and other overheads).

I2O features for PCI cards are not supported by SAGA. The extensions required for supported 64-bit PCI cards are documented in the I/O chapter of the V2500/V2600 Architecture Specification.

Key Points

SAGA supports PCI to PCI bridges, albeit, with limited performance. The SAGA DMA engine is designed as a pipe between a PCI controller and main memory. Every time a controller switches its address stream, it incurs a start-up penalty, equivalent to the time it takes to re-prime or flush the pipe.

For PCIbound DMA Writes, re-priming the pipe consists of updating SAGA’s channel context state, fetching page table pointers from system memory, and then fetching data from system memory.

For MEMbound DMA Reads, the flow consists of coherently flushing the previous data, updating the bridges channel context state, and fetching page table pointers from system memory. Due to these large overheads per address jump, the best performance is demonstrated by long transfers on PCI where the transfer time of the DMA Reads make this start-up overhead insignificant.

A PCI to PCI bridge or multi-port controller implemented using a PCI to PCI bridge, has just the opposite transfer pattern from the preferred pattern described above. Since the PCI bridge is arbitrating between


Theory of OperationsI/O Modules

multiple controllers on a secondary PCI bus, it turns long bursts from multiple controllers on the secondary bus into multiple small bursts (64 bytes is common) on the primary bus. The transfer time of these small bursts on the primary bus is dwarfed by the address switching start-up overhead, limiting the achievable performance through the bridge. This effect has been calculated to be between 7 to 15 MB per second maximum throughput when multiple controllers are active behind a bridge, as compared to a single controller’s maximum supported throughput rate of 174 MB per second.

Although SAGA will functionally support a PCI/PCI bridge controller, deliverable bandwidth will be limited to less than 15MB per second when switching between multiple addressing streams through the bridge.


Theory of OperationMid-Plane

Mid-Plane

Functional review of main computing FRUs

There are up to 32 CPUs in a V2500/V2600 server with 1 or 2 CPUs per processor board. Two CPUs on the same board must share the Runway Bus. Each SPAC must accommodate up to two processor boards, via their Runway Buses, and a SAGA I/O ASIC.

The X-Bar, implemented on the MIB or mid-plane, allows up to eight SPACs to simultaneously transfer 32 byte cache line packets to any of up to eight SMACs.

The core V2500/V2600 operations are centered around the ASIC interactions that shuttle packets (of instructions or data cache lines) between coherent memory space and each CPU’s cache memory. Other operations include SAGA DMA cache line transactions to coherent memory space and CPU (LOAD/STORE) and SPAC "unwedger" (get/put) transactions to CSR space.

The V-Class version of Scalable Computing Architecture (SCA, formerly ccNUMA) implements cache coherency checking within each SMAC. The CPUs assume they are interacting with another CPU or Memory module over their shared Runway Bus. However, each CPU is only interacting directly with an SPAC. Each SPAC has access to all SMACs via the X-Bar.

In order to maintain cache coherency, each SMAC must occasionally request a change of ownership of a private data cache line from one CPU to another CPU. This requires the SMAC to have access to all CPUs via the X-Bar.


Theory of OperationsPower Modules

Power Modules

Processor module power supply

The V2500/V2600 processor module required a total redesign of the +48VDC to low voltage power conversion hardware on the module. The change removes the existing +3.3VDC @ 50 Amp, +1.5VDC @ 6.5 Amp, and -1.9VDC @ 1.5 Amp converters. Power for the new processor pair will be provided by two +2.0V @ 50 Amp converters or an equivalent single converter. A +1.6V @ 3 Amp converter will power termination and a +3.3VDC @ 0.5 Amp converter will power the SPHYR clock chip. The redesign lives within available power, thermal, and control line constraints and is compatible with the existing system’s infrastructure.


Theory of OperationCooling Modules

Cooling Modules

Processor module thermal

Processor boards are air cooled using forced convection with 250 fpm (minimum) across the (EWPBs). Each board may have up to two processors cooled by low cost aluminum heat sinks with heat pipes imbedded in the base.

To account for the additional CPU on each module, the cooling area was expanded to utilize the space above the CPUs (in addition to the fin area used on V22x0 processor heat pipe). This compensates for the preheated air generated by the 3 upstream CPUs.


Theory of OperationsMass Storage Modules

Mass Storage Modules


Theory of OperationService Support Processor


Universal SSP


Theory of OperationsService Support Processor


A) Acronyms

Introduction

The release of the V2500/V2600 hardware platform introduces a new HP 9000 Series CPU for Hewlett Packard and new set of acronyms. During the design phase of this product a code name of Stingray was adopted and used in the acronyms.

V2500/V2600 use the same system architecture as the V22x0 except that it has been enhanced. The acronyms for the V2500/V2600 have also been enhanced.

The following tables list the acronyms for the V2500/V2600 and the V22x0 hardware platform. The tables provide you with a fully qualified explanation of the acronym, where the component is located within the system and some supporting text.


AcronymsV22x0 Acronym List

V22x0 Acronym List

AcronymFully

QualifiedIs Located

OnSupporting Text

ECUB Exemplar Core Utility Board

The Midplane

The ECUB is the heart of the power on self test, environmental monitoring, scanning, system clocks, as well as many other support issues.

ENRB Exemplar Node Routing Board

The Midplane

The ENRB is the largest circuit board in the V-Class system and is also known as the Midplane. This is where the HyperPlane Crossbar backplane is located as well as all of the Processor Agent Controllers (EPACs).

EMB Exemplar Memory Board

The Memory Board

The EMB has several plug-on modules to complete the functionality of the memory sub-system.

EPAC Exemplar Processor Agent Controller

The Midplane

Each pair of processors and one half (3 PCI slots) of an average I/O card cage connect to a Processor Agent Chip (or PAC). Each PAC chip communicates directly with the HyperPlane with NO bus waiting.

EMAC Exemplar Memory Access Controller

The Memory Board

There is a custom VLSI chip for the Memory Access Controller and there may be an optional VLSI chip for the V-Class technical SPU called Toroidal Access Controller (TAC).

EPIC Exemplar PCI Controller

The EIOB Each I/O Card Cage contains six total PCI slots for connecting PCI based I/O device adapter cards. There are two EPIC chips located inside each I/O card cage. Each EPIC chip controls three PCI slots and is connected to a Processor Agent Controller (PAC) chip which is located on the HyperPlane Crossbar Backplane.



E48VB Exemplar 48 Volt Board

The Upper Rear Chassis

The Node Power Supply (NPS) provides +48V DC to the two +48V Boards (E48VB) which simply route the +48V throughout the node to the fans and the DC - DC converters. The two E48VBs attach together to form a single "bus bar" type connection. The E48VBs are located in the rear chassis, behind the ENRB.

EVPB Exemplar Vulcan Processor Board

The Processor Board

The EVPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (ECUB). The EVPB contains an overtemp sensor and reports overtemp conditions to the ECUB. The DC - DC converters are part of the EVPB. Failure of one of the converters will require replacement of the EVPB.

EMBPB Exemplar Memory Board Power Board

The Memory Board

The EMBPB sits atop the Memory Board and connects to the EMB through the Memory Power Board Extender (EMBPX). The EMBPX is a field replaceable unit.

EMUC Environmental Monitor Utility Controller

The Utility Board (ECUB)

The EMUC is a non field replaceable FPGA located on the ECUB. It functions as the environmental monitor and power controller. Both the EMUC and power-on circuit work together to control power.

EPUC Exemplar Processor Utility Controller

The Utility Board (ECUB)

The EPUC is a non field replaceable FPGA located on the ECUB. Core logic functions are controlled by the EPUC via the 8 EPAC ASICs. System initialization and CPU interrupt handing are the EPUC’s main functions.

AcronymFully

QualifiedIs Located

OnSupporting Text



ERAC Exemplar Routing Access Controller

The Midplane

The ERAC is a field replaceable ASIC which is located on the Midplane. Four ERACs make up the HyperPlane XBAR. Each ERAC has 16 discrete ports with NO bus waiting!

EIOB Exemplar I/O Board

The I/O Card Cage

Each EIOB contains six PCI card slots. Up to four EIOBs can be installed in a V-Class system for a total of 24 PCI card slots.

ETAC Exemplar Torrodial Access Controller

The Memory Board

The ETAC ASIC will only be installed in X-Class (multi-node) systems. Globally shared memory is accessed between nodes via the ETAC. Off node memory accesses are routed from the ETAC to either main memory or the XBAR through the EMAC.

EPMB Exemplar Plug-in Memory Board

The 80 Bit DIMMs

Up to 16 DIMMs can be installed on each memory board (EMB). 80 bit DIMMS can only be used in single node systems.

EPMBM Exemplar Plug-in Memory Board (Multi-node)

The 88 Bit DIMMs

88 bit DIMMs are required in all multi-node systems. The extra 8 tag bits are needed for cache coherency between hypernodes. These DIMMs will function along side the 80 bit DIMMs in a single node system.

EMBPX Exemplar Memory Board Power Extender

The Memory Board

The EMBPX is a small extension board that buses the 48VDC input and logic level voltages between the EMB and the EMBPB.

ECCB Exemplar Cable Connector Board

the Upper Rear Chassis

The ECCB connects the keyswitch to all four power supplies and the RS232 ports from the bulk head connector to the ECUB. Environmental monitoring signals from the power supplies and fans also route through the ECCB to the ECUB.

AcronymFully

QualifiedIs Located

OnSupporting Text



ELEDB Exemplar LED Board

The left front side of Chassis

The ELEDB is the attention light for the system. The normal state is solid on across all segments. When flashing either an environment error or system crash has occurred. Test station utilities must be used to determine the error state when flashing.

ASIC Application Specific Integrated Circuit

12 on ENRB 1-2 on each EMB

All ASICs are field replaceable and require a specified torque and torque sequence when installing. V-Class ASICs include the EMAC, EPAC and ERAC chips. The ETAC ASIC is only installed in multi-node systems.

XBAR Crossbar The Midplane

The XBAR is a non-blocking fully muxed data switch located on the ENRB. It is comprised of four ERAC (routing attachment chip) ASICs. Connected directly to the XBAR are up to eight EMACs (memory controllers) and eight EPACs (processor agent chips). With 16 discrete ports per ERAC, the XBAR has a total of 64 discrete ports with NO bus waiting!

AcronymFully

QualifiedIs Located

OnSupporting Text


AcronymsV2500/V2600 Acronym List

V2500/V2600 Acronym List

Acronym Fully QualifiedIs Located

OnSupporting Text

SCUB Stingray Core Utility Board

The Midplane

The ECUB is the heart of the power on self test, environmental monitoring, scanning, system clocks, as well as many other support issues.

MIB Midplane Interface Board

The Midplane

The MIB is the largest circuit board in the V-Class system and is also known as the Midplane. This is where the HyperPlane Crossbar backplane is located as well as all of the Processor Agent Controllers (EPACs).

EWMB Exemplar W-Based Memory Board

The Memory Board

The EMB has several plug-on modules to complete the functionality of the memory sub-system.

SPAC Stingray Processor Agent Controller

Midplane Interface Board

Each pair of processors and one half (3 PCI slots) of an average I/O card cage connect to a Processor Agent Chip (or PAC). Each PAC chip communicates directly with the HyperPlane with NO bus waiting.

SMAC Stingray Memory Access Controller

The Memory Board

There is a custom VLSI chip for the Memory Access Controller and there may be an optional VLSI chip for the V-Class technical SPU called Toroidal Access Controller (TAC).



SAGA Just another cool acronym that has no meaning.

The SIOB Each Stingray I/O Card Cage contains seven total PCI slots for connecting PCI based I/O device adapter cards. There are two SAGA chips located inside each I/O card cage. One SAGA chip controls 4 slots (A side of the SIOB, e.g. iolf_a) and the other controls 3 slots (B side of the SIOB, e.g. iolf_b). Each SAGA chip connects to a Processor Agent Controller (PAC) chip which is located on the HyperPlane Crossbar Backplane.

E48VB Exemplar +48 Volt Board


The Node Power Supply (NPS) provides +48V DC to the two +48V Boards (E48VB) which simply route the +48V throughout the node to the fans and the DC - DC converters. The two E48VBs attach together to form a single "bus bar" type connection. The E48VBs are located in the rear chassis, behind the ENRB.

EWPB Exemplar W-Based Processor Board

The Processor Board

The EWPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (SCUB). The EWPB contains an overtemp sensor and reports overtemp conditions to the SCUB. The DC - DC converters are part of the EWPB. Failure of one of the converters will require replacement of the EWPB. The EWPB operates at 440 MHZ and is the processor board used in a V2500 system


OnSupporting Text



ELPB Exemplar Lanshark Processor Board

The Processor Board

The ELPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (SCUB). The ELPB contains an overtemp sensor and reports overtemp conditions to the SCUB. The DC - DC converters are part of the ELPB. Failure of one of the converters will require replacement of the ELPB. The ELPB operates at 552 MHZ and is the processor board used a V2600 system.

EWMBPB Exemplar W-Based Memory Board Power Board

The Memory Board

The EWMBPB sits a top the Memory Board and connects to the EWMB through the Memory Power Board Extender (EMBPX). The EMBPX is a field replaceable unit.

EMUC Environmental Monitor Utility Controller

The Utility Board (SCUB)

The EMUC is a non field replaceable FPGA located on the SCUB. It functions as the environmental monitor and power controller. Both the EMUC and power-on circuit work together to control power.

EPUC Exemplar Processor Utility Controller

The Utility Board (SCUB)

The EPUC is a non field replaceable FPGA located on the SCUB. Core logic functions are controlled by the EPUC via the 8 EPAC ASICs. System initialization and CPU interrupt handing are the EPUC’s main functions.

ERAC Exemplar Routing Access Controller

Midplane Interface Board

The ERAC is a field replaceable ASIC which is located on the Midplane. Four ERACs make up the HyperPlane XBAR. Each ERAC has 16 discrete ports with NO bus waiting!

SIOB Stingray I/O Board

The I/O Card Cage

Each SIOB contains seven PCI card slots. Up to four SIOBs can be installed in a V-Class system for a total of 28 PCI card slots.


OnSupporting Text



STAC Stingray Torrodial Access Controller

The Memory Board

Globally shared memory is accessed between nodes via the STAC. Off node memory accesses are routed from the STAC to either main memory or the XBAR through the SMAC.

EPMB Exemplar Plug-in Memory Board

The 80 Bit DIMMs

Up to 16 DIMMs can be installed on each memory board (EWMB). 80 bit DIMMS can only be used in single node systems.

EPMBM Exemplar Plug-in Memory Board (Multi-node)

The 88 Bit DIMMs

88 bit DIMMs are required in all multi-node systems. The extra 8 tag bits are needed for cache coherency between hypernodes. These DIMMs will function along side the 80 bit DIMMs in a single node system.

EMBPX Exemplar Memory Board Power Extender

The Memory Board

The EMBPX is a small extension board that buses the 48VDC input and logic level voltages between the EWMB and the EWMBPB.

ECCB Exemplar Cable Connector Board


The ECCB connects the keyswitch to all four power supplies and the RS232 ports from the bulk head connector to the SCUB. Environmental monitoring signals from the power supplies and fans also route through the ECCB to the SCUB.


OnSupporting Text



ELEDB Exemplar LED Board

the left front side of Chassis

The ELEDB is the attention light for the system. The normal state is solid on across all segments. When flashing either an environment error or system crash has occurred. Test station utilities must be used to determine the error state when flashing.

ASIC Application Specific Integrated Circuit

12 on ENRB 1-2 on each EMB

All ASICs are field replaceable and require a specified torque and torque sequence when installing. V-Class ASICs include the SMAC, SPAC and ERAC chips. The STAC ASIC is only installed in multi-node systems.

XBAR Crossbar Midplane Interface Board

The XBAR is a non-blocking fully muxed data switch located on the MIB. It is comprised of four ERAC (routing attachment chip) ASICs. Connected directly to the XBAR are up to eight SMACs (memory controllers) and eight SPACs (processor agent chips). With 16 discrete ports per ERAC, the XBAR has a total of 64 discrete ports with NO bus waiting!


OnSupporting Text

ucr_na-CYBA5074_90001.pdf-1

Documents

Transcript of ucr_na-CYBA5074_90001.pdf-1