SAN Storage

© 2005 EMC Corporation. All rights reserved.

CLARiiON Installation and Maintenance

Back-End Loop Architecture


Back-End Loop Architecture


FC Series

EXP

PRI

Shunt

9876543210 DAE

EXP

MID

Shunt

9876543210 DPE

EXP

PRI

Shunt

9876543210 DAE

SP


EXP

PRI

1413123210 DAE2

Shunt

EXP

PRI

1413123210 DAE2

CX Series

EXP

PRI

or

MID

1413123210

DAE2

(OS)

or

DPESP

Shunt

Shunt

© 2005 EMC Corporation. All rights reserved. Stiletto Enclosure - 4

PBC

Switching Logic

Switch / Loop Switch

Port Bypass Device vs. Loop Switch

DAE2

DAE2P


CLARiiON Tools and SoftwareCLARiiON Tools and Software


CLARiiON Tools and Software

� Navisphere Manager

- GUI for configuration control and monitoring

� Navisphere CLI

- Command line for configuration control and monitoring

� CLARiiON Procedure Generator

- Utility for generating highly detailed installation and repair instructions

� CAP2

- GUI utility for displaying CLARiiON Array Properties

� SPLAT

- GUI utility for displaying CLARiiON event logs

� FBI

- GUI utility for analysis of CLARiiON backend loops

� Release Bundles

- Array based software, version determines functionality, field upgradeable

� Recovery Images

- Array based packages used to recover/restore Storage Processor Boot Partitions


CLARiiON Installation & Maintenance

Enhanced Install Procedure (EIP)

CLARiiON Installation & Maintenance




• The Enhanced Install Procedure (EIP) is incorporated into the CAP2 tool and replaces the earlier Healthcheck_install tool. It should be run at a very specific point in the installation of a CX-Series array. It performs the following tasks:

• Checks validity of supplied IP Addresses

• Checks connectivity to the IP Addresses

• Checks for privileged user access to the array

• Checks for any bound LUNs

• Resets and turns on statistics

• Saves and clears the event logs

• Cleans the dump directory on both Storage Processors

• Launches FBI (Fibre Backend Isolation)

• Prompts for power tests

• Runs zero_disk on all drives

• Executes and retrieves SPCollect on both Storage Processors

• Displays the results in CAP2


EIP Start


EIP Continued (2)


EIP Continued (3)


FBI – Configuration


FBI – Configuration (cont.)


Sizing


Initial sized screen


EIP Continued (3)


EIP Continued (4)


EIP Continued (5)


EIP Continued (6)



CLARiiON Troubleshooting Tools


CLARiiON Troubleshooting Tools


Objectives

� Execute SPCollect and retrieve files via Navisphere CLI

� Execute SPCollect and retrieve files via Navisphere

Manager (Release 19)

� Identify failing FRUs

� Execute CAP Version 2

� Capture SPCollects via CAP Version 2

� Execute the Enhanced Install Procedure

� Examine event logs using SPLAT

� Execute Host-based and Arrayside FBI


Tools

� SPCollect

� CAP2

– CLARiiON Array Properties version 2

� SPLAT2

– SP Log Analysis Tool version 2

� FBI

– Fibre Backend Isolation


spcollect Introduction

� SP-based data collect script

– Navisphere CLI

– Navisphere Manager

� Revision 5

� Invoked from management station


spcollect Basic Operation

� Produces a data zip file, filename format is:

SPx_ChassisSerNum_SpSig_Date_Time_data.zip

� Dump files not included

� C:\dumps folder on the SP

� Run spcollect on both Storage Processors


Running spcollect via Navisphere CLI

E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 spcollect -messner

E:\Program Files\EMC\Navisphere CLI>

E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 spcollect -messner



Monitoring spcollect with CLI

E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 managefiles -list

Index Size in KB Last Modified Filename

0 4 11/01/03 14:08:26 CIMOM_HangInfo0.txt

1 5 08/16/03 14:45:37 DeletedVolume.txt

2 676 09/15/04 15:42:22 ktdump-040915-194222.txt

3 1002 09/15/04 16:28:59 ktdump-040915-202858.txt

4 1 09/15/04 16:36:14 ktdump_log.txt

5 0 09/15/04 15:45:28 naviagent_Sep-15-04_19-45-15.log


7 16 09/28/04 15:26:34 SPA_09-28-2004_19-26-34_runlog.txt

8 0 09/28/04 15:27:02 SPA_09-28-2004_19-26-34_stderr_log.txt

9 0 09/28/04 15:27:40 SPA_09-28-2004_19-26-34_stdout_log.txt

10 196 09/28/04 15:27:30 SPA_cfg_info.txt

11 0 09/28/04 15:26:34 SPA_collect_name_rev.txt

12 356 09/28/04 15:27:04 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_evt.zip

13 570 09/28/04 15:27:05 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_ktd.zip

14 693 09/28/04 15:26:41 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_nav.zip

15 8 09/28/04 15:26:41 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_psm.zip

16 15 09/28/04 15:27:03 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_rtp.zip

17 608 09/28/04 15:27:31 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_sus.zip





2 676 09/15/04 15:42:22 ktdump-040915-194222.txt

3 1002 09/15/04 16:28:59 ktdump-040915-202858.txt

4 1 09/15/04 16:36:14 ktdump_log.txt



7 16 09/28/04 15:26:34 SPA_09-28-2004_19-26-34_runlog.txt

8 0 09/28/04 15:27:02 SPA_09-28-2004_19-26-34_stderr_log.txt

9 0 09/28/04 15:27:40 SPA_09-28-2004_19-26-34_stdout_log.txt

10 196 09/28/04 15:27:30 SPA_cfg_info.txt

11 0 09/28/04 15:26:34 SPA_collect_name_rev.txt

12 356 09/28/04 15:27:04 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_evt.zip

13 570 09/28/04 15:27:05 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_ktd.zip

14 693 09/28/04 15:26:41 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_nav.zip

15 8 09/28/04 15:26:41 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_psm.zip

16 15 09/28/04 15:27:03 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_rtp.zip

17 608 09/28/04 15:27:31 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_sus.zip






2 676 09/15/04 15:42:22 ktdump-040915-194222.txt

3 1002 09/15/04 16:28:59 ktdump-040915-202858.txt

4 1 09/15/04 16:36:14 ktdump_log.txt



7 2659 09/28/04 15:31:09 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_data.zip





2 676 09/15/04 15:42:22 ktdump-040915-194222.txt

3 1002 09/15/04 16:28:59 ktdump-040915-202858.txt

4 1 09/15/04 16:36:14 ktdump_log.txt



7 2659 09/28/04 15:31:09 SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_data.zip

spcollect Completion


Transferring spcollect Data via CLI

E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 managefiles -retrieve -file

SPA__WRE00021500537_90a1f_09-28-2004_19-26-34_data.zip

Files selected to be retrieved are


Do you want to continue (y/n)? y


E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 managefiles -retrieve -file


Files selected to be retrieved are


Do you want to continue (y/n)? y



spcollect Data Location


Running spcollect via Navisphere Manager


Transferring spcollect Data via Manager


spcollect Data Contents



Sniffer


Sniffer


Flare Verify

� Basic operation (Verify) performed by Flare

– Read 64K - each drive in LUN

– Check flare generated checksum is correct

– Check flare metadata – ensures consistency of parity and data (Parity units only)

– XOR data and parity - ensures parity consistent with the data (Parity and mirror units only)

� If any single error reported - attempts to correct

� Error correction can be one of the following.

– Re-map bad sector and reconstruct data

– Correct a sector with bad checksum – reconstruct sector

– Metadata and parity inconsistencies fixed

� If multiple errors reported across a stripe

– data and or parity may not be correctable

– message reported to Event Log


Two Levels of Verify

� Sniff

– Low priority check of an entire storage system

– Very low impact on performance

� Background Verify

– High priority check of a LUN,

– may be initiated automatically by system (non-vol verify)

– or manually by operator

– Once BV started, will run to completion and cannot be

interrupted.


Sniffer Management

� Navisphere CLI

- setsniffer, getsniffer

� Navisphere Manager

- When binding LUN

- LUN properties


Setsniffer Command Format

Release 16 and Prior

-setsniffer lun sniff [-bv] [-bvtime priority] [-cr] [-snrate rate]

Release 19

- setsniffer lun [-rg rg-number] [-all] sniff [-bv] [-bvtime priority] [-cr] [-snrate

rate]

where:

Lun Lun number

rg-number Raid Group number

Sniff 0 disables sniffer, 1 enables sniffer (no longer available for Release 19

arrays)

-bv Starts a full unit background verify

-bvtime priority Specifies the background verify priority

-cr Clears background verify reports

-snrate rate Specifies the sniff rate

Note: This command must be executed from the owning SP


Getsniffer Command Format

Release 16 and Prior

- getsniffer lun [-alltot] [-curr] [-nonv] [-rec]

Release 19

- getsniffer lun [-rg rg-number] [-all] [-alltot] [-curr] [-nonv] [-rec]

where:

Lun Lun number

rg-number Raid Group number

-alltot Displays historical totals

-curr Displays currently running

-nonv Displays non-volatile historical totals

-rec Displays most recently completed

Note: This command may be executed from either SP


Modified Sniff Verify (Release 19)

� Motivation

– At the default sniff rate of one request (of 64kb) every 0.5 second, a

100GB disk takes about 9~10 days to finish one sniff verify pass.

– FC or ATA disks with larger capacities, Sniff Verify will take even

longer to finish.

– Disks also seem to be becoming more susceptible to soft/hard media

errors and a slow Sniff Verify doesn’t help to catch disk errors in a

timely fashion. These disk errors eventually might result in data

corruption or loss.

– Modified Sniff Verify has been speed up, mainly to help detect media

error on drives as early as possible. It also will no longer check data

integrity and stripe consistency of the unit.



� Performance

– The performance goal of the Modified Sniff Verify is to keep its impact to

host I/O at about the same degradation level as the current Sniff Verify with

their respective default settings

– Default Sniff rate is changed from 64KB requests every 0.5 seconds to

512KB requests every second

556 Hours142 Hours256 GB



Rate = 64KB/0.5secRate = 512KB/secDrive Capacity

Current SniffModified Sniff



� Limitation

– Sniff Verify will no long operate to check data integrity and stripe

consistency of the unit. It will only issue disk verify operations to all

the disks in a unit and have the disks read their data without

returning them back and verifying their contents. The user can run a

Background Verify on a bound unit at any time to inspect the unit’s

data integrity and coherency.

– Based on feedback from ongoing quality improvement teams in the

DU/DL area, it was decided that the sniff verify functionality in R19

and subsequent releases will no longer be user adjustable. In

normal usage, the user will not be able to disable sniffing and the

rate will be controlled by the Flare driver. An undocumented setting

will allow sniffing to be disabled in customer situations where

performance is prioritized over data integrity.


Error Levels

� Disk Recovered Error

820 (status 0x22) – read successful due to internal drive

heroics.

820 (status 0x3a) – same success but for ATA drive.

� RAID Correctable Error

820 with status 0x05 – read command successful due to RAID

� RAID Uncorrectable Error

0x957 (Uncorrectable Sector)

0x840 (Data Sector Invalidated)


Verify report

� Provide verify results for

specific lun

� Information returned

� Four log sections

� Shows corrected and

uncorrectable errors

� Types of errors

VERIFY RESULTS FOR UNIT 1Sniffing state: ENABLEDSniffing rate (100 ms/IO): 5Background verify time (hours): HighHistorical Total of all Non-Volatile Recovery Verifies (0 passes)----------------------------------------------------------------------

Corrected UncorrectableChecksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Currently Running Full Unit Verify----------------------------------------------------------------------Verify State: Sniff RunningPercent Complete: 10

Corrected Uncorrectable Checksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Most Recently Completed Full Unit Verify----------------------------------------------------------------------

Corrected Uncorrectable Checksum error 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Historical Total of All Full Unit Verifies (5 passes)----------------------------------------------------------------------

Corrected Uncorrectable Checksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0

VERIFY RESULTS FOR UNIT 1Sniffing state: ENABLEDSniffing rate (100 ms/IO): 5Background verify time (hours): HighHistorical Total of all Non-Volatile Recovery Verifies (0 passes)----------------------------------------------------------------------

Corrected UncorrectableChecksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Currently Running Full Unit Verify----------------------------------------------------------------------Verify State: Sniff RunningPercent Complete: 10

Corrected Uncorrectable Checksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Most Recently Completed Full Unit Verify----------------------------------------------------------------------

Corrected Uncorrectable Checksum error 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0Historical Total of All Full Unit Verifies (5 passes)----------------------------------------------------------------------

Corrected Uncorrectable Checksum errors 0 0Write Stamp errors 0 0Time Stamp errors 0 0Shed Stamp errors 0 0Coherency errors 0 0

E:\Program Files\EMC\Navisphere CLI>navicli -h 192.168.1.23 getsniffer 1


Verify Management with GUI


CLARiiON Troubleshooting

CAP (CLARiiON Array Properties)

CLARiiON Troubleshooting

CAP (CLARiiON Array Properties)


CAP Version 2 Introduction

� Java based application

– GUI

– Can be scripted

� Data collection tool

– Spcollect files

– Maxscript files

– Navicli getall files

– Online collection (local)

– Offline collection (remote)

� Current version

– 2.15.16


CAP Version 2 Requirements

� Windows NT 4.0 or later

� Java 1.4.2_5 or later

– JRE 5 is NOT supported

� 128 MB of RAM


Supported Hardware

� Storage Systems

- FC4500, FC4700, CX Series

� Switches

McData

- DS16M, DS32M, DS64M

Brocade

- DS8B, DS16B, DS16B2, DS32B2, ED12000

� Hosts

Requirements- Navisphere Agent 6.2 or higher

- Host must be registered with storage system


CAP Interface

Menu Bar

Main Window

Message Pane


File Menu


Edit Menu


View Menu


Tools Menu


Help Menu


Open Files


Open Files (2)


CAP Display


Tools Applications Integration


Capturing Array Data with CAP2


Capture Wizard (steps 1 and 2)


Capture Wizard (steps 3 and 4)


Capture Wizard (step 5)


Enhanced Install Procedure


EIP Wizard Configuration


EIP Wizard Execution


EIP Wizard - FBI


EIP Wizard – Power Cycling


EIP Wizard – Zerodisk


EIP Wizard – Finish



SPLAT (SP Log Analysis Tool)


SPLAT (SP Log Analysis Tool)


SPLAT Introduction

� GUI log file viewer

– Launched from Start menu or from within CAP2

� Requires

– Java JRE 1.4.2 or later

� Current Version – 2.0


SPLAT Launch


SPLAT InterfaceMenu Bar

Main DisplayStatus Pane


File Menu


Edit Menu


BookMark Menu


Mode Menu


Filter Menu


Colors Menu


Search MenuText input field


Selecting Filters


Displaying Filtered Data


Analyzing the Display


Display With Analysis


Analyze an Entry


Entry With Analysis



CLARalert


CLARalert


Trapping events with Event Monitor

� Receive automatic notification in the event of array or component failures– Email

– Pager

– SNMP

– Other

� Configured Through Navisphere Manager, and run from a Windows Monitoring Station on the network

� One Monitoring Station can monitor multiple arrays, each with different filter settings.


Event Monitor Architecture

Monitoring Station

Ethernet

TCP/IP

Network

CX Series

CX Series CX Series

Legacy Array

Legacy Host

FC

The Agent running on

the monitoring station

needs to be able to

reach the Agents on

the monitored arrays

(or hosts, for legacy

systems) through the

TCP/IP network


CLARalert Methods

� Centralized monitor Email-Home for Windows– Requires modem

� Centralized monitor Email-Home with Webex Remote Access

� Distributed monitor Email-Home with Remote Access– Requires modem

� Centralized monitor Email-Home for Solaris

� Centralized monitor Dial-Home for Windows– Requires modem


CLARalert Centralized Monitor Email-Home with Webex

Keys to Success

� Ensure that all the storage system, Windows workstation, and Host Agent environment and system requirements are met.

� Follow this entire procedure.

� Do NOT deviate from the installation procedures.

� Ensure that the proper Java Runtime Environment is installed on the Windows workstation.

� Ensure that there is a usable SMTP gateway mail server in the customer’s network environment.

� Ensure that the customer’s management workstation has internet access.


Installation from the CLARalert Remote Service CD


EMCRemote Introduction

� Windows based client/server application

– Server is SP resident

– Client is PC resident

� Two modes of access

– Dial-up

– LAN

� Client becomes ‘terminal’ on host (server)


EMCRemote Requirements

Operating Systems

� EMCRemote Host/Client supports the following operating systems:

– Version 4.02

�Windows 98

�Windows NT Workstation with SP 6A

�Windows NT Server with SP 6A

�Windows 2000 Professional with SP 2, 3, 4

�Windows 2000 Server with SP 3, 4

�Windows 2000 Advanced Server with SP 3, 4

– Version 5

�Windows XP Professional with SP1

�Windows 2003 Server Enterprise Edition


Installing EMCRemote (TPM)


Installing EMCRemote (TPM) - continued


EMCRemote Installation


EMCRemote Execution


EMCRemote Configuration Wizard


EMCRemote Configuration Wizard (2)


EMCRemote Address Book Wizard


EMCRemote Address Book Wizard (2)


EMCRemote Address Book Wizard (3)


EMCRemote Call Manager


Installing Navisphere CLI - continued


Installing Navisphere Agent


Installing Navisphere Agent - continued


Configuring Navisphere Agent


Portal Configuration


Portal Configuration - continued


Portal Configuration - completed


Selecting Monitored Systems


Importing the Monitoring Template


Editing the Monitoring Template


Editing the Monitoring Template - continued


Applying the Monitoring Template


Verifying The Webex Connectivity


Test Email-Home

C:\Program Files\EMC\Navisphere CLI\navicli –h <SP IP> insertteseventC:\Program Files\EMC\Navisphere CLI\navicli –h <SP IP> inserttesevent

Note: Enter the storage system in clarify before executing this command


Viewing Message File


Viewing Message File - continued


Viewing Response Log


Closing Slide

SAN Storage

Documents

Transcript of SAN Storage