Reliability, Availability and

Serviceability (RAS) Integration

and Validation Guide for the

Intel® Xeon® Processor E7- v3

Family

Memory Address Range Mirroring

April 2015

2

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of

merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course

of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information

provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast,

schedule, specifications and roadmaps.

The products and services described may contain defects or errors known as errata which may cause deviations

from published specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling

1-800-548-4725 or by visiting www.intel.com/design/literature.htm.

Intel, the Intel logo, and Intel Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others

© 2015 Intel Corporation.

3

Contents 1 Introduction ........................................................................................ 4

2 Address Range Mirroring Overview ..................................................... 6

3 System Configuration .......................................................................... 8

4 Software Integration ........................................................................... 9

4.1 UEFI Variables for Memory Address Mirroring ................................... 9

4.2 Management Interface Flow – BIOS .............................................. 13

4.3 Management Interface Flow – OS ................................................. 13

5 Intel® DFx Validation Recipe ............................................................ 14

5.1 BIOS Setup ............................................................................... 15

5.2 UCE Injection to TAD0 Primary Channel ........................................ 17

5.3 UCE Injection to Non-Mirroring Range ........................................... 20

5.4 Memory Mirror Failover Validation ................................................ 22

Figures

Figure 2-1. Typical Memory Layout ........................................................... 7

Figure 2-2. Address Range Memory Mirroring............................................. 7

Figure 5-1. Memory Mirroring Transaction Flow ........................................ 15

Figure 5-2. BIOS Setup Menu for Interleave configuration setting .............. 15

Figure 5-3. Enable Address Range Mirroring ............................................ 16

Figure 5-4. Set Address Range Mirroring Size (size unit is 64 MB) .............. 16

Figure 5-5. Independent Mode Memory Map ............................................ 17

Figure 5-6. Address Range Mirror Enabled Memory Map. ........................... 18

Figure 5-7. TAD Register of Address Range Mirror Enabled ........................ 19

Tables

Table 1-1. Definition, Business Value, and User Experience ......................... 4

Table 1-2. Memory Address Range Mirroring Support Platform Scope .......... 5

Table 3-1. Minimum System Configuration ................................................ 8

Table 4-1. UEFI Run-time Services ......................................................... 10

4

1 Introduction

The document describes the software and hardware integration and validation

methodologies associated with Memory Address Range Mirroring RAS feature

into Intel® Xeon® Processor E7 – 8800/4800/2800 v3 based systems (code

named Haswell EX). For those who are responsible for BIOS development,

system and platform validation, or product marketing, this document provides

integration and validation guidance for the Memory Address-based Range

Mirroring feature.

In channel memory mirroring feature, mirroring is implemented across two

memory channels. It requires allocating a significant amount of memory as

redundant memory thus increasing the cost and power as compare to a

nonmirrored system.

The Memory Address Range Mirroring offers more granular mirroring controls

allowing a cost-effective and power optimized solutions with the benefit of

mirroring which ensures system uptime even when memory uncorrected fatal

error event is detected.

At a high-level, business-value and user-experience are described in Table 1-1.

Table 1-1. Definition, Business Value, and User Experience

Definition Business Value User Experience

Memory Address Range Mirroring

feature provides further granularity

to mirror memory by allowing the firmware or OS to determine a range of

memory addresses to be mirrored,

leaving the rest of the memory in the

socket in non-mirror mode. Dynamic

(without reboot) failover to the

mirrored memory is transparent to the

OS and applications. The processor

supports up to two mirror ranges, one

mirror range per iMC. Each mirror

range size is using 64MB granularity.

More cost-effective

mirroring solutions by

mirroring just the

critical portion of

memory versus

mirroring the entire

memory space which

ensures system

uptime even when

memory uncorrected

fatal error event is

detected

Only mirror user defined

memory address range

through BIOS or OS for the

critical software execution

portion. For example, users

may choose to mirror

only the OS kernel space

leaving rest of the

memory configured in

independent mode

The Memory Address Range Mirroring feature description and platform support

scope are highlighted in Table 1-2 below:

5

Table 1-2. Memory Address Range Mirroring Support Platform Scope

Description Intel® Xeon® Processor E7 –V2

Intel® Xeon® Processor E7 –V3

Intel® Xeon® Processor E5 –V2

Intel® Xeon® Processor E5 –V3

The Memory Address

Range Mirroring offers a

more cost and power

efficient mirroring option.

It requires communication

between OS/VMM and

BIOS to determine the

address range for

mirroring.

No Yes No No

Integration guides in this document contain Unified Extensible Firmware (UEFI)

BIOS and the OS interface setup flows for reporting and configuring mirror

memory region during system boot and runtime.

Validation guides in this document contain Memory Address Range Mirroring

validation recipes using Intel® validation tool CScripts which is developed

based on Intel® Design for Test/Debug/Manufacturing (DFx) Technologies.

6

2 Address Range Mirroring

Overview

Memory address range mirroring is a new memory RAS feature on Intel®

Xeon® Processor E7 – 8800/4800/2800 v3 based systems (code named

Haswell EX) that allows greater granularity in choosing how much memory is

dedicated for redundancy. In normal channel mirroring, the memory is split into

two identical mirrors (primary and secondary). Half of all installed memory is

reserved for redundancy and not reported in total system memory size.

To reduce the amount of memory lost for redundancy, partial memory

mirroring can be used. For partial memory mirroring, mirroring can be enabled

selectively. If mirroring is not enabled, memory will be configured in

Independent mode and will be part of the total system memory. Once mirroring

is enabled, the memory is split into two identical mirrors. Half of the installed

memory with mirroring enabled will be reserved for redundancy, and not be

included in total system memory.

To further reduce the amount of memory reserved for redundancy, memory

address range mirroring can be used. It is similar to partial memory mirroring

and can be enabled selectively. The size (range) of the primary and secondary

mirrors can be defined using 64MB intervals. The range is defined by the value

programmed in the Target Address Decoder 0 (TAD0) register. TAD0 defines

the size of the primary and secondary mirror ranges. The secondary mirror

range is reserved for redundancy and not reported in the total memory size.

Because TAD0 is used to define the range for memory address range mirroring,

there is a TOLM limitation that contains memory (where system address 0x0

begins). On the first memory segment, the largest range allowed is limited by

TOLM (~2GB). This is because TAD1 will begin at system address 0x100000000

(4GB). TAD0 will cover 0-4GB. However only 0GB – TOLM (~2GB) will be

decoded to system memory. Note that TOLM to 4GB address range is assigned

to MMIO. Refer to Figure 2-1 and Figure 2-2 for overview of the memory map

and mirror region allocation.

When the memory address range mirroring feature is correctly configured, once

a multi-bit ECC error is injected to mirror region TAD0, the mulit-bit ECC error

is treated as an uncorrected error (UCE) in independent mode will be

downgraded to corrected error (CE) until mirror failover. Refer to Chapter 5 for

the detail validation flows.

7

Figure 2-1. Typical Memory Layout

Figure 2-2. Address Range Memory Mirroring

8

3 System Configuration

The system environment in this document for Address Range Mirroring

validation is based on the Intel® Xeon® Processor E7 – 8800/4800/2800 v3

Family. Minimum hardware configurations are shown in Table 3-1.

Table 3-1. Minimum System Configuration

Component Version/stepping Quantity Unit Comments

CPU Intel® Xeon

Processor E7 –

8800/4800/2800 v3

4 ea Target is to use latest stepping

of the processor

Memory 8 GB DDR4 RDIMM 32 ea Each channel has one DIMM.

BIOS 63.R00 1 ea Latest BIOS at the time of

releasing this doc.

OS N/A

Intel® DFx

Validation Tool

492130 Latest CScripts version at the

time of releasing this doc.

Intel® In-

Target Probe

Intel® DFx

Abstraction

Layer

1.9.4450.400

9

4 Software Integration

This chapter describes BIOS and OS integration and run time Memory Address

Range Mirroring setup through defined interface. Contact Intel® local

representative for further detail interface specification and supports.

The legacy memory mirroring feature is transparent to the OS; however, the

address range mirroring feature requires a firmware-OS interface for users to

specify the desired subset of memory to mirror. The OS needs the following

supports to fully utilize Address Range Mirroring feature:

• Present partial or total mirrored memory on the platform to the OS

• Provide the OS a method to request the amount of mirrored memory

that takes effect on subsequent boots.

Partial memory mirrored ranges are indicated by an attribute field in the

EFI_MEMORY_DESCRIPTOR. Any mirrored memory will need to be associated

with the following attribute.

Not only address range mirroring but also total mirrored memory can be

reported to the OS.

The OS uses the UEFI Call GetMemoryMap() returns to the OS all the address

ranges presented by the platform and is proposed to be modified to include

partially mirrored memory. Note that UEFI Call GetmemoryMap() is part of the UEFI BOOT services and hence cannot be invoked after ExitBootServices().

4.1 UEFI Variables for Memory Address Mirroring

UEFI Variables are used to allow the OS or system software to identify the

memory blocks currently mirrored by the platform and also to allow the OS to

request mirroring on a subsequent boot.

Supported features are:

• Enable and disable mirroring of memory below 4GB for a subsequent

boot

• Specify amount (up to 50%) of memory to be mirrored for a subsequent

boot

• Indicate whether memory below 4GB is mirrored for current boot

10

• Indicate amount of memory mirrored for current boot: The status of

requested memory redundancy (% and below 4GB) during last boot and

status of request – SUCCESS, FAILURE, PARTIAL…

Table 4-1 lists the UEFI run-time services available for managing variables.

Table 4-1. UEFI Run-time Services

Use the following global unique identification data type (GUID) to associate

with variables:

#define ADDRESS_RANGE_MIRROR_VARIBLE_GUID {0x7b9be2e0, 0xe28a,

0x4197, 0xad, 0x3e, 0x32, 0xf0, 0x62, 0xf9, 0x46, 0x2c}

The variable data contained in the following structure:

Typedef struct{

UNIT8 MirrorVersion;

BOOLEAN MirrorMemoryBelow4GB;

UNIT16 MirroredAmountAbove4GB

EFI_STATUS MirrorStatus;

}ADDRESS_ RANGE_MIRROR_VARIBLE_DATA;

MirroredAmountAbove4GB is the amount of available memory above 4GB that

needs to be mirrored measured in basis points (hundredths of percent e.g.

12.75%=1275)

In a multi-socket system, the platform is required to distribute the mirrored

memory range such that the amount mirrored is approximately propotional to

the amount of memory on each NUMA node. For example, on a two-mode

machine with 64GB on node 0 and 32GB on node 1, a request for 12GB of

mirrored memory should be allocated with 8GB of mirrored on node 0 and 4GB

on node1.

For another example: if total memory in the system is 48GB and 12GB of

memory above 4GB of mirrored memory is requested, the percentage of

memory above 4GB that needs to be mirrored is:

11

Mirrored Memory Above 4GB / Total Memory above 4GB. Assume TOLM = 2GB,

the percentage is (12-(4-TOLM))/44=22.72% = 2272 basis points

Size of the ADDRESS_RANGE_MIRROR_VARIABLE_DATA is calculated as

follows:

# define ADDRESS_RANGE_MIRROR_VARIBLE_SIZE

Sizeof(ADDRESS_RANGE_MIRROR_VARIBLE_DATA)

The Variables will be associated with the following attributes. Note that BIOS

may additionally specify authenticated write access attribute on “MirrorCurrent”

to prevent the OS from updating or deleting it.

#define

ADDRESS_RANGE_MIRROR_VARIABLE_ATTRITUBE{EFI_VARIABLE_N

ON_VOLATILE | EFI_VARIABLE_BOOTSERVICE_ACCESS |

EFI_VARIABLE_RUNTIME_ACCESS}

The following two commands are used in UEFI run-time service:

GetVariable() is invoked using these parameters:

Prototype

12

SetVariable() is invoked using these parameters:

Prototype

13

4.2 Management Interface Flow – BIOS

On the first boot of a system that supports memory address range mirroring,

the BIOS must create the MemoryCurrent variable and initialize it to values

supplied by BIOS setup options. If there are none, then it uses: [1,

false,0,EFI_SUCCESS]. Note that once “MemoryCurrent” is corrupted or does

not exist in a subsequent boot, the BIOS must create it.

On each subsequent boot, the BIOS should first check if “MemoryRequest” has

been set by the OS to request a change in mirror configuration. If it is, then

the BIOS should try to set up mirroring using the new parameters:

• If successful, copy parameters from “MemoryCurrent”, set

MemoryCurrent.MirrorStatus=EFI_SUCCESS

• If not successful, set MemoryRequest.MirrorStatus with an error code

and fall through to normal path to set mirroring using the old

parameters

If MemoryRequest was not set (or if it was and could not be used as mentioned

above), then the BIOS should set up mirroring from parameters in

MemoryCurrent and set MemoryCurrent.MirrorStatus to indicate the

success/fail status.

4.3 Management Interface Flow – OS

During early boot, the OS uses the attribute bits returned by GetMemoryMap()

to locate which ranges of memory are mirrored (same data also available via

_ATT attributes).

During hot plug memory operations the OS uses _ATT attribute field to

discover changes to memory mirror configuration.

Later it examines MemoryCurrent and MemoryRequest to check for errors in

setting up mirror ranges.

For the OS to change the mirror configuration, it sets MemoryRequest to

[1,desired-below-4GB,EFT_WARN_STALE_DATA] and then reboots to allow the

BIOS to apply the new settings.

14

5 Intel® DFx Validation Recipe

This chapter describes the methodology recommended for validating the

Address Range Mirroring Feature. Error injection tools and libraries, data

collector and analyzer are the key components described here. Primary

objective is to validate end-to-end flow from the time a hardware error is

injected to the time the error recovers successfully. Contact Intel® local

representative regarding CScripts installation process, system integration, and

detail recipes with CScripts command types and results. Regarding signal

transaction flows in mirror processing, refer to Figure 6-1.

15

Figure 5-1. Memory Mirroring Transaction Flow

5.1 BIOS Setup

The reference BIOS setup shown in Figure 5-2, Figure 5-3, and Figure 5-4 is

based on the BIOS version used in Intel® Xeon® Processor E7 –

8800/4800/2800 v3 based Customer Reference Platform. For different

platforms, users need to check with their BIOS vendors for the correct system

configurations.

The BIOS menu setup is shown below (also see Figure 5-2 through 5-4):

EDKII Menu -> Advanced -> Memory RAS Configuration: � Memory Interleaving: NUMA(1-way) Node Inter leave or 2-way Node

Interleave

� Memory Mirroring: Partial CH Mirroring.

� Memory Mirroring SCKx MCx: Enabled

� Partial Mirroring Size: select size here, 64MB based unit

Figure 5-2. BIOS Setup Menu for Interleave configuration setting

16

Figure 5-3. Enable Address Range Mirroring

Figure 5-4. Set Address Range Mirroring Size (size unit is 64 MB)

17

5.2 UCE Injection to TAD0 Primary Channel

For precise exeution commands, contact Intel® local representative for the

details.

Step 1: Configure Address Range Mirroring range

Step 2: Check Memory Map

Under EFI shell, current memory map can be shown using the memmap

command. If the Address Range Mirroring feature is enabled, the memory

map will be changed with mirrored size reduction. With the BIOS

configuration in Section 5.1, the top memory address is 0x87fffffff (34 GB) in

independent mode shown in Figure 5-5. In Address Range Mirroring mode,

the top memory address is 0x83fffffff(33 GB) shown in Figure 5-6. The

memory size is reduced 1 GB (16 x 64 MB = 1 GB). As there is a 2 GB MMIO

size under 4 GB address space, memory size in independent mode is 32 GB

(34 GB – 2 GB) and memory size in Address Range Mirroring mode is 31

GB(33 GB – 2 GB).

Figure 5-5. Independent Mode Memory Map

18

Figure 5-6. Address Range Mirror Enabled Memory Map.

Step 3: Get TAD0 System Address

TAD0 channel address is recorded in TAD0 registers. The physical mirroring

address range can be obtained by translating the address from its channel

address and its socket address recorded in SAD registers.

TAD0 can be read by CScripts as below

>> sv.socket0.uncore0.ha0_tad_0.show

Or through the register dump as shown in Figure 5-7, TAD0 records 0xF3E4.

based on the definition, the channel address limit is calculated as (0x0f + 1) x

64 MB= 1 GB. As its node base address is 0, the physical range of the mirror

area is 0 ~ 1 GB. Each node base address is recorded in SAD registers. SAD

19

registers describe each node’s limit address. In case there is no node level

interleave, the base address of node[n] is equal to the limit address of

node[n-1] plus 1. In this test configuration, since there is no DIMM installed

on the other node, the base address of this node is 0.

Figure 5-7. TAD Register of Address Range Mirror Enabled

Step 4: Inject multi-bit ECC error to TAD0

An uncorrectable error within mirroring range will be treated as a correctable

error.

In this validation recipe, one uncorrectable error will be injected to mirror

address range. The error should be treated as a correctable error. If not, the

test fails.

Step 5: Identify if UCE downgraded to corrected error CE

The new Event ID is produced and the error type shows a corrected error in

Windows Event Viewer.

20

5.3 UCE Injection to Non-Mirroring Range

To compare multi-bit ECC error injection to TAD0, UCE injection to the non-

mirroring address range, the memory controller detects uncorrectable error

and triggers machine check exception (MCE) to system software, and system

goes to BSOD with 124 codes under Windows Server 2012.

Step 1 ~3: The same as the step 1 through 3 in section 5.2

Step 4: Inject multi-bit ECC error to the non-mirror range above 0x3fff_ffff

Note that if inject multi-bit ECC error to TAD0 boundary 0x3ffffffe, this

UCE type of error should downgrade to CE type of error as well.

Once UCE type of error is injected to TAD1+, the system BSOD will be

expected with error code ID 124.

Step 5: Check system BSOD.

After injecting UCE successfully, let system go then check

Windows Event Viewer on Windows Server 2012

21

Step 6: check BSOD 124 event

After a system reboot, go back to Windows 2012 and check if the

system event generated a BSOD 124 event.

22

5.4 Memory Mirror Failover Validation

To validate the memory mirror failover function, the mirror scrub should be

enabled in the BIOS setup menu. When persistent UCE type of error is injected

until mirror failover, the memory controller loses mirror redundancy and the

system will operate in independent mode. At this stage, if one more multi-bit

ECC error is injected to TAD0 region, the error will be treated as an

uncorrectable error. Following steps are the detail validation procedure.

Step 1 ~3: Refer to step 1 to step 3 in section 5.2

Step 4: Inject mirror failover on TAD0

Inject mirror_failover error type at TAD0 address.

Step 5: Check system status

Let system go. At this point, system should be still alive.

Then check event log in event view. There is ID 47 event generated by

OS CMCI handler.

Step 6: Inject multi-bit ECC error after mirror failover occurs

The System goes to BSOD with ID 124 error which is WHEA type.

23