0_d9cdf

Copyright © 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Symmetrix HW & SW Architecture - 1

© 2007 EMC Corporation. All rights reserved.

Symmetrix Monitoring and Management Using Solutions EnablerSymmetrix Monitoring and Management Using Solutions Enabler

Course Overview

Welcome to the first module in the Symmetrix Monitoring and Management course. In this module we will review the hardware and software architecture of the Symmetrix and the process for installing and configuring Solutions Enabler on a open systems host.



© 2007 EMC Corporation. All rights reserved. Symmetrix HW & SW Architecture - 2

Revision HistoryRev Number Course Date Revisions

1.0 September, 2007 Complete

Copyright © 2007 EMC Corporation. All rights reserved.

These materials may not be copied without EMC's written consent.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR APARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.




Course ObjectivesProvide a software view of the Symmetrix Architecture

Install and configure Solutions Enabler and Symmetrix Management Console

Using the appropriate commands query the Symmetrix to determine the physical and logical configuration

Modify the configuration to change system parameters, configure new devices, set port attributes, and map devices to front-end directors

Understand host to Symmetrix connectivity

Secure access to the Symmetrix using access controls

Gather real time performance statistics on the Symmetrix

The objective of this course is to provide an understanding of the Symmetrix hardware and software architecture for a host prospective using Solutions Enabler.




Symmetrix Engineering Curriculum

Symmetrix Internals(4 days)

Symmetrix Internals(4 days)

Advanced SymmetrixInternals(2 days)

Advanced SymmetrixInternals(2 days)

TimeFinder & SRDFWorkshop

(3 days)

TimeFinder & SRDFWorkshop

(3 days)

Symmetrix Monitoring & Management

(2 days)

Symmetrix Monitoring & Management

(2 days)

Configuration & ManagementSymmetrix Replication Technology

(2 days)

Configuration & ManagementSymmetrix Replication Technology

(2 days)

Service Processor Solutions Enabler

For EMC Internal Engineering there are two Symmetrix training Paths.

On the left is the Symmetrix Internals class that discuss the architecture, configuration, and monitoring of the Symmetrix from the hardware and Service Processor. Included is the use of SymWin to create an initial bin file and basic inlines commands. It also includes a brief introduction to Solutions Enabler.

Advanced Symmetrix Internals is a follow-on course that traces an IO operation from the host through the Symmetrix for debugging and performance purposes.

The TimeFinder & SRDF Workshop discusses the setup, monitoring, and control of the Symmetrix local and remote replication products using SymWin and Inlines commands on the Service Processor and Solutions Enabler.

On the right is the new curriculum path and was designed for an internal audience who needs an understanding of the Symmetrix hardware and software architecture from a software prospective.




Day 1

Hardware and Software Architecture review– Lab: Installing Solutions Enabler

Understanding Symmetrix Configuration– Lab: Understanding Physical and Logical Configuration

Configuring Symmetrix Devices– Lab: Configuring Symmetrix Devices

Symmetrix Management Console– Lab: SMC Installation and Operations




Day 2

Host Connectivity– Lab: Host Connectivity

Monitoring Symmetrix Activity– Lab: Monitoring Symmetrix Performance Statistics

Symmetrix Security– Lab: Symmetrix Security




Closing Slide





Module 1:Symmetrix Hardware and Software Architecture

Welcome to the first module in the Symmetrix Monitoring and Management course. In this module we will review the hardware and software architecture of the Symmetrix, and the process for installing and configuring Solutions Enabler on a open systems host.





1.0 July, 2007 Complete









Course ObjectivesUpon completion of this course, you will be able to:

Describe the hardware and software architecture of a Symmetrix DMX

List the Symmetrix configuration information, where it is located, and how it is maintained

Describe the communications path from a host to the Symmetrix for monitoring and management

List the Solutions Enabler daemons and briefly describe the functions they perform

Describe the SYMAPI database, where it resides, how it is created, and how it is accessed

Describe the Solutions Enabler software environment including directory structure, environment variables, options, and log files

Install Solutions Enabler

License Solutions Enabler features

We will start the module with a quick overview of the hardware and software architecture of the Symmetrix but the primary focus is on Solutions Enabler, the Command Line Interface (CLI) for monitoring and managing the Symmetrix.




Symmetrix DMX-3/4

Front-end– Host Connectivity

Fibre ChannelFICON/ESCONiSCSI

Global Cache– Memory

Back-end– Disk Directors– DAE-LCC– Disk Drives

All members of the Symmetrix family share the same fundamental architecture. The modular hardware framework allows rapid integration of new storage technology, while supporting existing configurations.

There are three functional areas:Shared Global Memory - provides cache memory Front-end - the Symmetrix connects to the hosts systems using Channel Adapter a.k.a Channel Directors. Each director includes multiple independent processors on the same circuit board, and an interface-specific adapter board. Celerra Data Movers connect to the storage through the front-end.Back-end – is how the Symmetrix controls and manages its physical disk drives, referred to as Disk Adapters or Disk Directors. Like front-end directors, each director includes multiple independent processors on the same circuit board.

What differentiates the different generations and models is the number, type, and speed of the various processors, and the technology used to interconnect the front-end and back-end with cache. With the current generation, the interconnections between the front-end, back-end, and global memory is based on a Direct Matrix Architecture – Direct connections between every front-end/back-end director and every memory board.




Modem Batteries CoolingServiceProcessor

Environmentalcontrol and

status signals

Environmentalcontrol and

status signals

Control andcommunications

signals

Control andcommunications

signals

Hardware Architecture

64 GBmemory

64 GBmemory

64 GBmemory

64 GBmemory

64 GBmemory

64 GBmemory

64 GBmemory

64 GBmemory

ESCON host attachFibre Channel host attach

FC Controller

DirectMatrix

DirectMatrix

CntlFC Controller

DirectMatrix

DirectMatrix

CntlESCON Controller

DirectMatrix

DirectMatrix

CntlESCON Controller

DirectMatrix

DirectMatrix

Cntl

Multi-protocolCD

DirectMatrix

DirectMatrix

Cntl

Multi-protocolCD

DirectMatrix

DirectMatrix

Cntl

Multi-protocolCD

DirectMatrix

DirectMatrix

Cntl

Multi-protocolCD

DirectMatrix

DirectMatrix

Cntl

FICON, Gigabit Ethernet, iSCSI

Fibre Channel back-end directors

FC (back-end)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (back-end)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (BE or FE)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (BE or FE)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (BE or FE)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (BE or FE)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (back-end)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

CntlFC (back-end)

A

A

B

B

A

A

B

B

DirectMatrix

DirectMatrix

Cntl

The advantage of Direct Memory Architecture can be appreciated when you visualize it as in the picture above. The Global Memory technology supports multiple regions and 16 connections on each global memory director. In a fully configured Symmetrix system, each of the sixteen directors connects to one of the sixteen memory ports on each of the eight global memory directors. These 128 individual point-to-point connections facilitate up to 128 concurrent global memory operations in the system.



© 2007 EMC Corporation. All rights reserved. Symmetrix HW & SW Architecture - 6 6

System BayTotal of 16 independent front-end and back-end directors– Eight PowerPC processors per director– Up to 8 disk directors

Installed in pairsUp to 480 drives per disk-director pair

– Up to 12 channel directors Eight-port Fibre Channel 2Gb/4GbEight-port ESCON Four-port multi-protocol - FICON, iSCSI, and Gigabit Ethernet (for SRDF)

Up to 512 GB Global Memory– Mirrored DDR technology – Memory Vault protection

The Symmetrix System Bay has:Either two, four, six. or eight disk directorsUp to 12 channel directors (combined total of all directors is 16)Up to eight power supplies, each of which has:−A dedicated 2.2-kilowatt standby power supply (SPS)−A 1µ Service Processor with KVM (keyboard, video screen, and mouse) and dedicated

uninterruptible power supply (UPS)− Three cooling-fan assemblies, each containing three fans

Two power zones with independent power cables, each zone capable of powering the fully configured System Bay

Symmetrix DMX-3/4 systems can be configured with up to 512 GB of total memory (256 GB useable). These models support extended Global Memory capabilities, using industry-standard DDR RAM technology DDR is Double data-rate Random Access Memory that achieves greater bandwidth by transferring data on the rising and falling edges of the clock signal. Effectively, it nearly doubles the transfer rate without increasing the frequency of the bus. Thus, a system with a 100 MHz front side bus has an effective clock rate of 200 MHz when DDR SDRAM memory is installed. The same system using SDR (single data rate) SDRAM, will not have its front side bus rate doubled and be limited to a 100 MHz front side bus speed.



© 2007 EMC Corporation. All rights reserved. Symmetrix HW & SW Architecture - 7 7

Storage BayStorage bays support up to 240 disk drives

– Two groups of 120 drives– Each group connects to a separate disk-director pair– Groups can be daisy-chained linearly across cabinets– Fully configured system supports up to 2,400 disks

4Gb dual-ported Fibre Channel drives – 2Gb with DMX-3

Tiered storage with the Symmetrix DMX– High performance

73 GB and 146 GB 15,000 rpm Fibre Channel– Good price/performance

146 GB and 300 GB 10,000 rpm Fibre Channel– Low cost

500 GB 7,200 rpm Fibre Channel750GB SATA II (DMX-4)

Online upgradeability and scalability– Add storage bays, directors, and disk drives

120 drives

120 drives

Symmetrix DMX-3/4 series configurations allow scalable capacity and performance to consolidate systems. Base configurations are composed of a System Bay and one or more independent storage bays.

The Storage Bay is configured for capacities of 120 or 240 disk drives. Each drive bay:Has redundant power supplies with battery backups to provide standby power to all componentsHas two power zones with independent power cables—each zone capable of powering the fully configured drive bayCan be populated with any combination of 73, 146, and 300 GB 10,000 rpm drives; and 73 and 146 GB 15,000 rpm drives500 GB low-cost Fibre Channel drivesWith DMX-4, 750GB SATA II drives for low cost archival requirements

Additional capacity (upgrade) can be added online by adding drives, Disk Array Enclosures (DAEs), and/or additional drive bays. Disk directors can also be added non-disruptively.

One of the distinctions between the between the DMX-3 and DMX-4 is that DMX-4 uses 4Gb Disk Array Enclosures that uses a cut-through-switching technology creating point-to-point connections to each driver. Earlier generations used arbitrated look technology.




Symmetrix DMX-3/4 950Entry-level Symmetrix DMX-3

– 128 GB Memory – Two or four front-end/back-end Fibre

Channel directors– Native Gigabit Ethernet for replication

and iSCSI– FICON with DMX-4

Flexible expandability– Nondisruptive upgrades– Scale capacity and performance– Up to 360 drives with one add-on

storage bayBay 1

120 DrivesBay 2 (Optional)

240 Drives

The Symmetrix DMX-3 950 provides an entry point into high-end availability and functionality for open-systems environments. It contains a six-slot backplane, supporting two memory directors with up to 64 GB of usable cache (128 GB raw) and two or four front-end/back-end directors.

It can be configured with system components and up to 120 drives in the single bay—60 drives above and 60 drives below the system’s directors, which are housed in the middle of the cabinet. When needed, you can add one storage bay to the main bay, increasing the drive count by 240 for a total of 360 drives—180 TB—in a fully loaded Symmetrix DMX-3 950.

The Symmetrix DMX-3 950 leverages Symmetrix DMX-3 technology and uses the latest release of Enginuity Operating Environment, with full support for nondisruptive upgrades—from disks and storage bays to Fibre Channel directors and memory directors. This means the Symmetrix DMX-3 950 fully supports all of EMC’s leading local and remote array-based replication software like the TimeFinder and SRDF families and Open Replicator for Symmetrix.

And because the Symmetrix DMX-3 950 contains a six-slot backplane, it is optimized for the most efficient power and cooling in a compact footprint, providing significant operational savings. It is therefore optimized for data centers with limited space, power, and cooling. It is also ideally suited for:

Enterprise consolidation and tiering in a compact footprintDedicated array for a large application Extended-distance replication, including full support for the SRDF familyHigh-performance, high-capacity backup to disk




DMX 950 DMX-3/4: High-End Storage Array

Disk adapters 1 (DA) pair 1 DA pair 2 DA pairs 3 DA pairs 4 DA pairs

Drive channels 8–16

32–360

2

64 GB

• 16 x Fibre Channel (FC)

• 8 x Gigabit Ethernet remote replication

• 10 x Gigabit Ethernet iSCSI

16 32 48 64

Number of disks 96–240 192–960 360–1,440

4–8

256 GB

480–2,400

• 64 x FC• 32 x FICON• 64 x ESCON• 8 x Gigabit

Ethernet remote replication


4–8

216 GB

• 64 x FC• 40 x FICON • 64 x ESCON• 8 x Gigabit



Memory directors 2–8 2–8

Maximum Global Memory 72 GB 144 GB

Host connectivity • 48 x FC• 24 x FICON• 48 x ESCON• 8 x Gigabit



• 64 x FC• 48 x FICON• 64 x ESCON• 8 x Gigabit



Scalable and Expandable Configuration

INCREMENTAL SCALABILITYEXPANSION

The Symmetrix DMX-3 950 is an ideal entry point for high-end configurations requiring one DA pair and 32 to 360 drives. The Symmetrix DMX-3 models are ideal for high-end configurations requiring performance and scaling capability to start as small as one DA pair and 96 drives, and grow to a maximum of four DA pairs and 2,400 disks.

The incremental scalability that the Symmetrix DMX-3 provides allows you to meet your growth requirements by adding Disk Adapters, disk channels, and disk drives nondisruptively to the existing frame. This enables true pay-as-you-grow economics for high-growth storage environments.




Software ArchitectureSymmetrix architecture can be compared to MPP Server– Multiple independent processors

Front-end directors service host requestsBack-end directors stage and destagedata in and out of cache Intelligent disk drives process SCSI commandsIntelligent Environmental and Communications ModulesService Processor

– All working together to service host request and protect data– Parallelism provides nearly infinite scalable performance

Enginuity Operating Environment is the micro code that runs eachprocessor in the Symmetrix– Allows the independent processors on each director to work together

The Symmetrix architecture is often compared to Symmetrical Multiprocessor (SMP) architecture used by server manufacturers, but it might be more correct to compare it to the Massively Parallel Processor (MPP) used for computational intensive applications where multiple independent processors are working in parallel to solve a problem. A well-known example of this is “Deep-Blue” the chess champion of a few years back.

In the Symmetrix we have multiple independent front-end and backend processors working together to service host requests and protect data. The Direct Matrix Architecture and parallelism, with multiple independent processors provides nearly infinite scalable performance.

Enginuity, a.k.a firmware or microcode, is the code that is loaded in the processors on the directors that allows the independent processors to work together. Components of Enginuity are also loaded in the XCM boards and in the Service Processor.

Provides all system functionality including advanced features like local and remote replication

Emulation code is loaded into each processor on each directorDownloaded from service processor to directors during code load Zipped code persistently stored in EEPROMLoaded to SDRAM (control store) on each processor on IMPL




Symmetrix ConfigurationInitial configuration is created using SymmWin to build an IMPL.bin file

– a.k.a “the bin file”, contains all the configuration information for a Symmetrix

Physical hardware configurationDirectorsMemoryPhysical Drives

Logical storage configurationEmulation, number, size anddata protection schemes for logical volumesSpecial volume attributesVolume front-end assignmentsDirector flags

Operational parameters and features

After initial configuration subsequent changes are made on-line– EMC can update the IMPL.bin– Users can make changes on-line

EMC Control Center Configuration ManagerSolutions Enabler CLISymmetrix Management Console (SMC)

The Symmetrix is configured using a static configuration file called the IMPL.bin. The file is created initially using SymmWin and loaded into each director in the Symmetrix. When modifying a configuration, the current IMPL.bin file is pulled from the Symmetrix and edited use Symmwin.

SymmWin is an EMC written graphical-based application for managing a Symmetrix. Capabilities include:

Building and modifying system configuration files (IMPL.bin)Issuing Inlines commands, diagnostic, and utility scriptsMonitoring performance statisticsAutomatically performs periodic error polling for errors and events. Certain errors will cause the service processor to “Call Home”.

SymmWin runs locally on a Symmetrix Service Processor or on a standalone PC. Running on the service processor allows communications with an operational Symmetrix. Running it on a standalone system allows you to build a new configuration or view and modify an archived configuration file.




System Information Maintained by Enginuity

Symmetrix File System (SFS)– Two SFS volumes are automatically defined when the bin file is

created6-Gigabyte FBA volumesNot host addressable

– Contains:Error codes are copied to the SFS from NVD for long-term storageTrace informationHistorical access information used by Dynamic Mirror Service PolicyOptimizer data is used to plot a thermograph of hot and cold spindleDevice masking database (VCMDB)Global Name Services (GNS) device group databaseAccess Control (ACL)Audit information

Enginuity automatically reserves two volumes for internal use as a Symmetrix File System (SFS). This space is automatically allocated while initially loading the Enginuity Operating Environment on Symmetrix systems and is not visible to the host environment. The SFS is contained on two mirrored 6GB volumes.

The SFS stores statistical data to provide a number of benefits:Dynamically adjusting performance algorithms including dynamic mirror service policyLoad balancing and optimized performanceProblem identification and recoveryEnhanced system audit and investigationSecurity and Access Control

Enginuity also allows Quality of Service (QoS), giving the ability to set varying priority levels to applications residing within a Symmetrix to meet varying customer needs or agreements.




Host View of the Symmetrix

The Symmetrix emulates disk drives– Opens Systems Host simply see the Symmetrix

as one or more FBA SCSI disk drives– Mainframe systems see the Symmetrix as a

Logical Control Unit and one or more CKD disk drives

Other than basic query data, the host has no knowledge of the Symmetrix internal configuration

EMC provided integration tools provide visibility and control

From a host’s perspective, the Symmetrix is simply seen as one or more FBA or CKD devices.

Standard SCSI commands such as SCSI INQUIRY and SCSI READ CAPACITY return low-level physical device data, such as vendor, configuration, and basic configuration, but have very limited knowledge of the configuration details of the storage system. Vendor specific information such as director configuration, cache size, number of devices, mapping of physical-to-logical, port status, flags, etc. are accessible using these commands.




EMC Solutions Enabler Introduction

Symmetrix Command Line Interface (SYMCLI)

Provides a host with a comprehensive command set for managing a Symmetrix storage environment– Invoked from the host OS command line– Scripts that may provide further

integration with OS and application

Separate components licenses

Security and access controls– Monitor only – Host-based and user-based controls

Detailed Configuration Information

Status

On-line ConfigurationChanges

Performance

Control

SYMCLI can be used to perform ad-hoc operations or incorporated into user developed scripts to integrate Symmetrix management and control with the application and host environment.

.




Documentation

EMC Powerlink

Solutions Enabler documentation can be downloaded from Powerlink one document at a time or the full library can be downloaded as a CD image.




Downloading Solutions Enabler

Solutions Enabler is provided to the customer on CD ROM but can also be downloaded from Powerlink

Also require license key(s)

Solutions Enabler software is host operating system specific and can be downloaded as an installation package from Powerlink.




Solutions Enabler ComponentsComponent License CommandsBase symacl

symcli symgate symreturnsyminq

symaudit symcfg symdevsymdrvsympd

symauthsymcgsymdgsymeventsymlabel

symbcv symdisk symld symstatsymqos

SYMAPI Server symapisrv storsrvd

symlvsymvg

Configuration Manager symconfigure

DeltaMark (Change Tracker) symchg

Device Masking symconnect symmask symmaskdb

Double Checksum (Oracle PAK) symchksum

Dynamic Cache Partitioning symqos -cp

Mapping Solution (SRM) symhost

sympart

symhostfs

symrdb

symioctl

symrslv

Open Replicator/LM

Open Replicator/DM

symrcopy

To enable Solutions Enabler features, license keys must be installed to enable specific functionality.




Solutions Enabler Components (cont.)Component License CommandsOptimization (Control) symoptmz

SRDF/Synchronous symrdf symioctl

TimeFinder/Snap symsnap

TimeFinder/Consistency Groups symreplicate start -consistent

symsnap activate -consistent

symclone activate-consistent

SRDF/A (Asynchronous mode support)

symrdf set mode async

SRDF/Automated Replication symreplicate

SRDF/Consistency Groups symcg -cg CgName enable

SRDF/Star symstar

Symmetrix Priority Control symqos -pst

TimeFinder/Mirror symioctl symmir symreturn

TimeFinder/Clone symclone symmir




Solutions Enabler Integration with Enginuity

Third Party Applications

SYMAPI

SIL (Symmetrix Interface Layer)

Enginuity Operating Environment

SYMCLIEMC Software Applications

Symmetrix

Host

SYMCLI commands are built on top of SYMAPI library functions– Use system calls that generate low-level I/O SCSI commands that are sent

to the Symmetrix

This illustrates the software layers and where each component resides.

EMC’s Solution Enabler APIs are the storage management programming interfaces that provide an access mechanism for managing the Symmetrix. They can be used to develop storage management applications. SYMCLI resides on a host system to monitor and perform control operations on Symmetrix arrays. SYMCLI commands are invoked from the host operating system command line (shell). The SYMCLI commands are built on top of SYMAPI library functions, which use system calls that generate low-level I/O SCSI commands to the storage arrays.




SYMAPI - SYMCLI: What’s the Difference? Both are used to query, configure, and control the Symmetrix

– APIs are used by application and systems management programmers and are linked when the programs are compiled

– CLIs are typically used for ad hoc operations or called using scripting languages

Both use the same dynamically-linked libraries– APIs are composed of C Programming-like function calls, with arguments and flags– The CLIs are composed of command-line commands, with options and parameters

API provide finer granularity

Each SYMCLI command invokes an API session– Calls SymInit() with RW access to the database – Executes API call(s)

Execution logged in SYMAPI log file– SymExit() is called at the end of each command, and all dynamically allocated

memory is automatically freed

SYMAPI and CLI communicate from host to Symmetrix using the samemechanism

Sometimes the terms API and CLI are used interchangeably but there are core differences.




SYMCLI commands

SYMAPI Database

Symmetrix configuration and status information is a SYMAPI database on the management host– Reduces the number of system calls from the host– Commands either act on information in the database or query

the Symmetrix directly

SYMAPIDatabase

To reduce the number of inquiries from the host to the storage arrays, configuration and status information is maintained in a Symmetrix host database file called the Symmetrix configuration database (default file name: symapi_db.bin).

SYMCLI commands can either run in two modesOnline mode−Query current status−Use used for configuration and control operations−Automatically attempt to gather the latest state information from the arrays−Update the in-memory database and the configuration database file on the host− In the event a configuration change has occurred, commands will load any updated

configuration informationCommands that can execute in offline mode, such as symcfg list, retrieve data exclusively from the configuration database




Configuration Database Overview

A collection of binary data– Flat file of binary data – Physical and logical configurations

information– Operational status

Also user-defined information– User-defined logical grouping of devices (device groups) – User-supplied physical and logical device names

Location– In-memory database in process virtual address space per session– In host file system when saved from in-memory database

<Root directory>\EMC\symapi\db\symapi_db.bin

SYMAPIDatabase

Configuration and status information is maintained in a binary configuration file on the local host. Also contained is user defined management information such as device grouping and logical names that are assigned to help document the configuration. The database resides on disk and portions of it are called into memory as needed.




Discovery

SYMAPI database is created/updated using the Discovery and/or Synchronization process

SYMAPIDatabase

symcfg discover

The SYMCLI commands are built on top of SYMAPI library functions, which use system calls that generate low-level I/O SCSI commands to the storage arrays. To reduce the number of inquiries from the host to the storage arrays, configuration and status information is maintained in a Symmetrix host database file (called the Symmetrix configuration database; symapi_db.bin by default).

During initial command line session, or if a configuration change has occurred, the configuration database must be built, or rebuilt, with the current information for all physical devices connected to your host. To scan the hardware and rebuild the database, enter: symcfg discover. This command scans all SCSI buses, collects information about all the devices found, and rebuilds the database with the collected device information and parameters from all local and remotely attached Symmetrix devices. symcfg discover scans all SCSI buses on the host, not just those connected to Symmetrix arrays. This can take a significant amount of time to complete. Key points on Discovery:

Builds a database containing the most complete and current information for all physical devices connected to the hostThe scan can take a significant amount of timeRe-run if Symmetrix configuration has changedNo significant output on screen




Solutions Enabler DaemonsOptional daemons that support SYMCLI/SYMAPI operations

– storapid – Base Daemon– storsrvd – storevntd– storgnsd– storrdfd– storstpd– storwatchd– Other daemons related to SRM and 3rd party

database operations

Daemons execute as root, therefore applications do not need to be privileged

Start, listed and stopped using stordaemon commands

– Example: stordaemon start storsrvd

SYMAPIDatabase

storapid

storsrvd

storevtd

storwatchd

storgnsd

others

storapid (Base daemon)Coordinates gatekeeper selection, Symmetrix locks, and parallel application syscalls to the operating system kernel

storevntd (Event Daemon)Monitors the Symmetrix and forwarded events to any client applications that have registered an interest. Can be configured to automatically log events using local log files, Unix syslog, Windows event log, and SNMP traps to an SNMP management application.

storgnsd (GNS Daemon)Provides support for global, distributed repository for DG and CG group definitions across Symmetrix arrays that are visible to all locally attached hosts.

storsrvd (SYMAPI server)Provides support for remote clients allowing them to execute symcli commands without having a direct connection to a Symmetrix or copy of the SYMAPI DB

storstpdUsed to collect raw performance counters for a Symmetrix, directors, devices, disks, ports, LRU, and RDF/A.

storrdfdProvides consistency for SRDF/A MSC and RDF-ECA composite groups (CGs) in multi-Symmetrix environments.

storwatchdMonitor the core Solutions Enabler daemons and automatically restart them if they crash.

Note: Not all daemons are supported on all platforms.




In-Band Communicates

Host to the Symmetrix communication is performed using standard SCSI Write Buffer/Read Buffer commands

Devices designated to receive commands are called Gatekeepers– Typically minimum sized volumes– Simply used to pass commands and return response

Gatekeeper

SYMCLICommands

The SCSI command architecture was originally defined for parallel SCSI. Today, the same command structure is also leveraged for Fibre Channel and iSCSI communications.

Basically, in SCSI protocol, the initiator sends a SCSI command to the target which then responds. SCSI commands are sent in a Command Descriptor Block (CDB). The CDB consists of a one byte operation code followed by five or more bytes containing command-specific parameters. Solutions Enabler communicates to a device using the standard Write Buffer (3B) and Read Buffer Commands (3C). This device is called a Gatekeeper device but in reality, a Logical Unit (LUN) with a address like any other device. Once a gatekeeper has been successfully obtained, SYMCLI determines if a semaphore exists; if so, a lock is obtained on the device. Once the CDB sequence is processed, the gatekeeper is closed and the lock released, freeing the device for other processing.




Gatekeeper DevicesA Gatekeeper can be any volume accessible to the host– Appear like any other volume– Usually configured as a small device – Should not be used by the host for normal data processing

Best practice is to dedicate devices as Gatekeepers– When a SYMCLI session is started, gatekeeper and database locks

are used to avoid conflictsSemaphore

– Once the CDB sequence is processed, the gatekeeper is closed andthe lock released, freeing the device for other processing

Solutions Enabler commands executed on the Symmetrix Service Processor uses a pseudo Gatekeeper device– Storage Processor does not have direct access to any devices

Gatekeeper(Typically<10MB)

When a Symmetrix array is installed, it is usually configures a number of Symmetrix small devices for use as gatekeeper devices. By standard convention, these are 6 cylinders in size, but need to be at least as large as the minimum volume size accessible by your host. Consult your host documentation for the minimum device size accessible by your particular host to determine the minimum gatekeeper device size for your environment.




SYMCLI Gatekeeper SelectionAny device that is visible to the host can be used as a Gatekeeper

– Gatekeeper devices should not be used by the host system for normal IO

Selection order:1. Defined and Associated gatekeepers 2. Defined gatekeepers 3. Small (< 10 cylinders) non-PowerPath devices, 4. Any small device in the following priority:5. Standard non-RDF and non-meta devices 6. RDF R1 devices7. RDF R2 devices

Best Practice is to define gatekeeper device – Places them higher on the gatekeeper candidate priority list making them

more likely to be used as a gatekeepers

SYMCLI selects a gatekeeper based upon a preestablished priority list. 1. Defined and associated gatekeepers These devices have been explicitly designated as

gatekeeper devices and associated with specific device groups.2. Defined gatekeepers — These devices have been explicitly designated as gatekeeper

devices.3. Small (< 10 cylinders) non-PowerPath devices – Marked by the array with the inquiry

gatekeeper flag.4. Any small device in the following priority:

PowerPath Count key data format

5. Standard non-RDF and nonmetadevices in the following priority:Non-PowerPathPowerPath

6. RDF R1 devices in the following priority:Non-PowerPathPowerPath

7. RDF R2 devices in the following priority:Non-PowerPathPowerPath




Gatekeeper ManagementBest Practice: Configure multiple gatekeepers per host– At least on gatekeeper for each application that is run concurrently– Manage Gatekeepers to avoid conflicts

Automatically by the Base Daemon (storapid)Manually mange Gatekeepers using the SYMCLI symgate command

Define a device as a gatekeeper– Adds the specified physical device name to the list of gatekeeper devices

Information maintained in SYMAPI DB on local host– Examples:

symgate define pd c2t0d2 (UNIX)symgate define pd physicaldrive2 (Windows)

Associate a gatekeeper with a device group– Explicitly designates one or more gatekeepers to be used – SYMCLI routes SCSI commands to any device in the device group via the

associated gatekeeper with a device group– Example:

symgate -g Oracle_app1 associate pd c2t0d2

While optional, configuring the base daemon (storapid) alleviates contention when there are limited gatekeeper resources available and also eliminates the need for every client to constantly select, open, lock, and ping for an available gatekeeper device for every online function. Additionally, the base daemon monitors Symmetrix External Locks (SEL) and Device External Locks (DEL), and will automatically release any lock(s) held by a crashed application. The base daemon also eliminates the need for Solutions Enabler applications to run as root.

Alternatively, gatekeepers can be manually managed using the symgate command to define and associate gatekeepers for operations on specific devices.

Gatekeeper definition and association information is managed in the SYMAPI database on the local host.

Another option for altering the default gatekeeper selection criteria is configuring the following configuration files on the local host.

gkavoidgkselect




SYMAPI Server (storsrvd)Two types of SYMAPI configurations:

– Local host directly connected to Symmetrix and uses local SYMAPI DB

– Remote hosts direct SYM CLI commands to a remote server that is connected to the Symmetrix.

Secure communications– Socket Layer (SSL)– Optional Kerberos Authenticated & encrypted

connection

On the server, storsrvd runs as a background process

– stordaemon start storsrvd

On the client:– Set Environment Variable:

SYMCLI_CONNECT=SYMAPI_SERVERSYMCLI_CONNECT_TYPE=REMOTE

– Edit the netcnfg file to point the server(s)

SYMCLIclient

SYMAPIDatabase

storapid

TCP/IPnetwork

SCSI or FC connected

SYMCLIclient

SYMCLIclient

storsrvd

There are basically two types of SYMAPI configurations: Local and remote. With Local, the SMYCLI commands are issued directly on the host attached to the Symmetrix. With this type of configuration, the local SYMAPI DB is used. With remote, the SYMCLI commands are directed to a remote server that is connected to the Symmetrix.

To configure remote operations:

1) Install Solutions Enabler software in the machine designated as the client,

2) Install the same Solutions Enabler software in the machine designated as the server. Invoke symlmf and apply the SYMAPI server license key.

3) Edit the netcnfg file in the client machine to include the host name or IP address of the server.

4) Set environment variables SYMCLI_CONNECT and SYMCLI_CONNECT_TYPE.

5) Issue a stordaemon start storsrvd command on the server machine.




EnvironmentEnvironment variables can be preset to streamline and expedite command line session– Set common argument values for

a series of associated commands to eliminate repeated key strokes

To view a list of environment variables that can be set for a given SYMCLI session:– symcli -env

To view the environment variables that you currently have set:– symcli –def

Examples:– SYMCLI_DG

Default device group name– SYMCLI_NOPROMPT

Set =1 to disable verification prompt

– SYMCLI_OFFLINESet=1 to access SYMAPI DB only

– SYMCLI_SIDSpecifies a default Symmetrix ID

Reference Solutions Enabler Commend Reference Guide

In addition to Solutions Enabler Environment variables, it is useful to include the symcli executables to the path.

For UNIX:# set path=$path /usr/symcli/bin) - Unix C Shell# PATH=$PATH:/usr/symcli/bin; export PATH - Korn or Bourn shell

For Windows, ensure the following SYMCLI directories are appended to the MS-DOS variable path: C:\Program Files\EMC\SYMCLI\bin




OptionsThe options file contains parameters that change the default behavior of SYMCLI operations, SYMAPI calls, and control actions– Global parameters and restrictions – Customize and streamline command

line coding to your specific environment

Examples:– SYMAPI_TRACK_SIZE_32K_COMPATIBLE= DISABLE | ENABLE– SYMAPI_USE_GNS= ENABLE | DISABLE– SYMAPI_MAX_CLIENTS=– SYMAPI_LOGFILE_FORMAT

Reference Solutions Enabler Installation Guide

/var/symapi/db/options

In addition to the Environment variables, the options file can be used to modify the default behavior of Solutions Enabler.




Return CodesAll SYMCLI commands return status or error codes – To view return code in UNIX:

echo $status (C shell)echo $? (bourn/korn shell)

– To view return code in Windows:echo %ERRORLEVEL%

– For all commands:0= CLI call completed successfully1= CLI call failed

– Others command specific returnExample: symcfg return code 24 Indicates:“The Symmetrix configuration and the database file are NOT in sync”

Reference Solutions Enabler Commend Reference Guide




Solutions Enabler Debug

Environment Variables SYMCLI_DEBUG=<value> – SYMCLI_DEBUG=32

For arguments passed to SYMCLI commands– SYMCLI_DEBUG=64

Lots of output – can be very hard to read– SYMCLI_DEBUG=128

SRDF processing– SYMCLI_DEBUG=256

TimeFinder processing

SYMAPI_DEBUG=<value>– Work for both SYMCLI and applications

written using the SYMAPI– SYMAPI_DEBUG_FILENAME

Allows you to control the output

Verify Environment:symcli -def

Third Party Applications

SYMAPI

SIL (Symmetrix Interface Layer)

Enginuity Operating Environment

SYMCLIEMCApplications

Where you enable debugging you will see extra information you do not normally see. Type something like "symdg create test".

There are a number of possible values for the SYMAPI_DEBUG environment variable (not all of which are published); different values accomplish different debugging goals.




XML (Extensible Markup Language) OutputFacilitates the automated processing of SYMCLI output

The data returned is identical to the standard command output, but “marked-up” with tags

Advantage:– Call individual pieces of

data by tag– Relationship between

different objects

Enabled using environmentvariable or CLI option– SYMCLI_OUTPUT_MODE=

xml – Attribute-based tagsxml_element – element based tagsstandard – default symcli output

– <symcli command> -output <xml|xml_element|standard>

symcfg list -output xml_element

<?xml version="1.0" standalone="yes" ?>

<SymCLI_ML>

<Symmetrix>

<Symm_Info>

<symid>000190102254</symid>

<attachment>Local</attachment>

<model>DMX3-24</model>

<microcode_version>5771</microcode_version>

<cache_megabytes>32768</cache_megabytes>

<devices>65</devices>

<physical_devices>10</physical_devices>

</Symm_Info>

</Symmetrix>

</SymCLI_ML>

Output options provide a mechanism to facilitate the automated processing of SYMCLI command output. XML is a deterministic parsing tool that eases the processing of output data and provides advantages over screen-scraping tools like awk or perl.

The XML industry standard is based on the experience of SGML( Standard Generalize Markup Language) and is endorsed by the World Wide Web Consortium. Details on XML can be found at: http://w3.org/XML.




Symmetrix Event Logs

SYMCLI and SYMAPI log significant events and actions – UNIX:/var/symapi/log/symapi-yyyymmdd.log

– Windows: C:\Program Files\EMC\Symapi\log\symapi-yyyymmdd.log

The log entries include the following event information:– Time tag of the event occurrence– Process ID (PID)– Source of the event (application name)– Related (internal) API function call– Name of the specific operation or event– Variable event field that describes in detail the event or error

The Log files accumulate over time and can consume needed disk space. Periodically, you may need to purge the log files to conserve space. For a detailed list of possible Symmetrix events, refer to the EMC Solutions Enabler Symmetrix CLI Command Reference.

Change date formats in the log entries by setting the environment variableDate Formats

Allows the creation of undated SYMAPI log files by setting the environment variable SYMAPI_DATED_LOGFILE_NAMELog File Names

Disable logging by setting the environment variable SYMCLI_NOLOGGING to 1. Enable/Disable Logging

Specify the number of days to retain the log files.• Maximum value: 1825 (or 5 years)• Minimum value: 6• Default value: 0 maintains the log file forever (service processor default is 30)

Log File Retention

Name and path of the default log file. Log File Name

DescriptionLogging Option




Event Daemon (storevntd)Monitor Symmetrix, detecting and reporting events as they happen– UNIX, LINUX, and Windows

environments

The event daemon continually collects Symmetrix event information– Monitors in real-time– Filters the events by severity and type

Responds:– Alerting client event applications– Logs events to specified targets

Log filesUNIX syslog or Windows Application LogSends SNMP traps to SNMP management application

storevntd

client

WARNING!Hot Spare Invoked

against disk!

In UNIX, LINUX, and Windows environments, the event daemon (storevntd) enables you to monitor Symmetrix operations by detecting and reporting events as they happen. The event daemon continually collects Symmetrix event information in real-time, filters the events by severity and type, and responds by doing the following:

Alerting client event applications and/or: Logging events to specified targets

When using the daemon with a client event application (for example, the Symmetrix Management Console), the application registers with the event daemon, specifying the events in which it is interested. When used in this manner, the daemon will automatically start when the client application requests its services. When configuring the daemon to log events, you can specify to log the events to the UNIX Syslog, the Windows Event log, SNMP, and/or a file on disk. When used in this manner, you should configure the daemon to automatically start at system boot.




Auditing

All control and configuration operations initiated by host applications are logged in a common audit log stored in the Symmetrix File System (SFS)– Maximum size of 40MB– Circular log will overwrite oldest entries– Audit records are read-only

Auditing is a key security feature– Who, is doing what, when, from where– Also logs attempted operations

symaudit command used to examine and filter audit log entries




Directory Structure

config – Includes the license file and other files that determine the operation of the SYMAPI

daemons – Includes storapidhost daemon

db – The SYMAPI database is in this directory

log – Includes the STORAPI and SYMAPI log files

With normal installation some of the above directories would be created. These directories contain the extra tools and sample code the code the programmers uses to develop applications.




Directory Structure

bin – Command executables

daemons – Daemon executables

doc – Special Notes

man – Manual pages

With normal installation some of the above directories would be created. These directories contain the extra tools and sample code the code the programmers uses to develop applications.




Installation OverviewAll components are packaged on distribution CD

Interactive install script

Installation options include– db, log, and config directories

Select components to install

Enter license keys

Post Installation tasks– Edit path– Environment variables– Options file

The Solution Enabler software is installed from a distribution CD. Depending on the platform, mount or load the CD so the installation software can be accessed from the host. Start the installation process. During the installation process, you are required to enter the installation directories for such structures as the db, log, and configuration directories. The installation process also requires you to identify any installation options required. Detailed installation instructions are available for supported platforms in the Solution Enabler Installation Guide.

SYMCLI provides environment variables that can be preset to streamline and expedite your command line session. These environment variables can be preset to common argument values for a series of associated commands, which eliminates repeated keystrokes for your given session.

The options file, located in the SYMAPI configuration directory, contains behavior parameters that can be set to change defaults of certain behavior options within various SYMCLI components and associated SYMAPI calls. It can be used by advanced SYMCLI and SYMAPI users to customize and streamline command line coding to your specific environment.




Solutions Enabler Unix Installation ScriptMount CD

Change directory to the location of the Solutions Enabler Kit

Run the install script

# cd /<CD_ROM Mountpoint>/UNIX

# ./emc_install.csh

# ----------------------------------------------------------

# EMC Installation Manager

# ----------------------------------------------------------

Copyright 2007, EMC Corporation

All rights reserved.

The terms of your use of this software are governed by

The applicable contract.

# cd /<CD_ROM Mountpoint>/UNIX

# ./emc_install.csh

# ----------------------------------------------------------

# EMC Installation Manager

# ----------------------------------------------------------

Copyright 2007, EMC Corporation

All rights reserved.

The terms of your use of this software are governed by

The applicable contract.

The first step to installing Solutions Enabler on a UNIX platform is to mount the installation media. Each operation system will have it’s own mount commands for the cdrom file system.




Solutions Enabler Installation DirectoriesDefault Installation and working directories– Install root directory – Default /opt/emc– Working root directory – Default /usr/emc

The install root directory /opt/emc does not exist.

This directory is necessary for the installation to complete

Successfully

It will be created now. Is that OK? [y] y

./emc_install : Creating Directory ----/opt/emc

The working root directory /usr/emc does not exist.


Successfully


./emc_install : Creating Directory ----/usr/emc

The install root directory /opt/emc does not exist.


Successfully


./emc_install : Creating Directory ----/opt/emc

The working root directory /usr/emc does not exist.


Successfully


./emc_install : Creating Directory ----/usr/emc

If the root and working directories do not exist, the installation script will create them.




Solutions Enabler Products to be Installed

Installation options

Install All Solutions Enabler Shared Libraries and Run Time Environment? [y]:

Install Solutions Enabler 64-bit shared libraries ? [N]:

…

Install Symmetrix Command Line Interface (SYMCLI) ? [y}:

Install Solutions Enabler SRM Database and Run Time Components ? [N]:

…

Install Options to enable JNI interface for EMC Solutions Enabler APIs ? [N]:

…

Install All Solutions Enabler Shared Libraries and Run Time Environment? [y]:

Install Solutions Enabler 64-bit shared libraries ? [N]:

…

Install Symmetrix Command Line Interface (SYMCLI) ? [y}:

Install Solutions Enabler SRM Database and Run Time Components ? [N]:

…

Install Options to enable JNI interface for EMC Solutions Enabler APIs ? [N]:

…

After selecting the installation directories, two lists of Solutions Enabler products are displayed:

The first list displays the Solutions Enabler that the install script detects that are already installed on the host.

The second list is the products that are available on the install media to be installed.

You will be prompted for additionall installation options for libraries and other host environment considerations.




Solutions Enabler Windows Installation

Insert the CD into the CD-ROM drive– If autorun is enabled, the installation starts automatically– If autorun is not enabled, run <CD-ROM drive>:wsk63rt.exe

For a Windows environment, you will process through a series of dialogs.




License

License keys are installed to enable functionality– Using symlmf command

# export PATH=$PATH:/usr/symcli/bin

# symlmf

E M C S O L U T I O N S E N A B L E R

SOLUTIONS ENABLER LICENSE MANAGEMENT FACILITY

Register License Key (y/[n]) ? y

Enter License Key : ####-####-####-$$$$

# export PATH=$PATH:/usr/symcli/bin

# symlmf

E M C S O L U T I O N S E N A B L E R

SOLUTIONS ENABLER LICENSE MANAGEMENT FACILITY

Register License Key (y/[n]) ? y

Enter License Key : ####-####-####-$$$$




Licensed Components

License keys are stored in:/var/symapi/config/symapi_license.dat

# cat /var/symapi/config/symapi_license.dat

License Key: ####-####-####-#### SYMAPI Feature: BASE / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: CMODE / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ConfigChange / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ConnectivityBase / All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: ConnectivityZoning /All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: DeltaMark / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: DevMasking / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: OPTMZR / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ORAPAK / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ResourcePak / All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: ResourcePak / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SERVER / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SOLUTION_1 / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SOLUTION_4 / Symmetrix

…

# cat /var/symapi/config/symapi_license.dat

License Key: ####-####-####-#### SYMAPI Feature: BASE / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: CMODE / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ConfigChange / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ConnectivityBase / All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: ConnectivityZoning /All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: DeltaMark / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: DevMasking / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: OPTMZR / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ORAPAK / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: ResourcePak / All+Conn+SRMLicense Key: ####-####-####-#### SYMAPI Feature: ResourcePak / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SERVER / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SOLUTION_1 / SymmetrixLicense Key: ####-####-####-#### SYMAPI Feature: SOLUTION_4 / Symmetrix

…




Help!In addition to the full set of documentation, there is extensive online information – man symcli– symcli

Provides the version number – symcli -h

Describes how to use the SYMCLI command.– symcli -v

Displays the list of SYMCLI commands and a brief description of each command.

– symcli -envDisplays a list of the environment variables that can be set for a SYMCLI session.

– symcli -defDisplays a list of the environment variables that are set for the current SYMCLI session.

For more information about Solutions Enabler symcli commands, reference the Solutions Enabler Symmetrix CLI Command Reference.




Inquiry - syminq

Performs a SCSI Inquiry and displays information about disk devices visible to host– Note Gatekeeper devices

C:>syminq

Device Product Device-------------------------- --------------------------- ---------------------Name

Type Vendor ID Rev Ser Num Cap (KB)-------------------------- --------------------------- ---------------------\\.\PHYSICALDRIVE1 EMC SYMMETRIX 5771 5400020080 919680\\.\PHYSICALDRIVE2 EMC SYMMETRIX 5771 5400021080 919680\\.\PHYSICALDRIVE3 EMC SYMMETRIX 5771 5400022080 919680\\.\PHYSICALDRIVE4 EMC SYMMETRIX 5771 5400023080 919680\\.\PHYSICALDRIVE5 EMC SYMMETRIX 5771 5400024080 919680\\.\PHYSICALDRIVE6 EMC SYMMETRIX 5771 5400025080 919680\\.\PHYSICALDRIVE7 EMC SYMMETRIX 5771 5400026080 919680\\.\PHYSICALDRIVE8 EMC SYMMETRIX 5771 5400027080 919680\\.\PHYSICALDRIVE9 GK EMC SYMMETRIX 5771 5400028080 9600\\.\PHYSICALDRIVE10 GK EMC SYMMETRIX 5771 5400029080 9600

C:>syminq

Device Product Device-------------------------- --------------------------- ---------------------Name

Type Vendor ID Rev Ser Num Cap (KB)-------------------------- --------------------------- ---------------------\\.\PHYSICALDRIVE1 EMC SYMMETRIX 5771 5400020080 919680\\.\PHYSICALDRIVE2 EMC SYMMETRIX 5771 5400021080 919680\\.\PHYSICALDRIVE3 EMC SYMMETRIX 5771 5400022080 919680\\.\PHYSICALDRIVE4 EMC SYMMETRIX 5771 5400023080 919680\\.\PHYSICALDRIVE5 EMC SYMMETRIX 5771 5400024080 919680\\.\PHYSICALDRIVE6 EMC SYMMETRIX 5771 5400025080 919680\\.\PHYSICALDRIVE7 EMC SYMMETRIX 5771 5400026080 919680\\.\PHYSICALDRIVE8 EMC SYMMETRIX 5771 5400027080 919680\\.\PHYSICALDRIVE9 GK EMC SYMMETRIX 5771 5400028080 9600\\.\PHYSICALDRIVE10 GK EMC SYMMETRIX 5771 5400029080 9600

The syminq command issues a SCSI INQUIRY and optionally SCSI READ Capacity.Because it uses standard SCSI commands, it will return information about all devices seen by the host including CLARiiON and non-EMC storage.

syminq –h will provide online help information

Note the Device Serial Number. For systems running Enginuity 5671 and higher, the maximum number of logical devices per Symmetrix array is 64,000 devices. In the output format of syminq command, the device serial number is in the format of SSNNNNNDDP. If the port number (P) is greater than 8 subtract 8 from the P field.




Discover and Verify

C:\>symcfg discover

This operation may take up to a few minutes. Please be patient...

C:\>symcfg list

S Y M M E T R I X

Mcode Cache Num Phys Num Symm

SymmID Attachment Model Version Size (MB) Devices Devices

000190102254 Local DMX3-24 5771 32768 10 65

C:\>symcfg verify -sid 254

The Symmetrix configuration and the database file are in sync.

C:\>symcfg discover


C:\>symcfg list

S Y M M E T R I X

Mcode Cache Num Phys Num Symm

SymmID Attachment Model Version Size (MB) Devices Devices

000190102254 Local DMX3-24 5771 32768 10 65

C:\>symcfg verify -sid 254


Before issuing any SYMCLI commands, you need to initialize the SYMAPI database. This is done using the symcfg discover command. If the database is already initialized, a quick way to verify if it is current is to issue the “verify” command.

The symcfg list command displays a list of all Symmetrix systems that are attached to the host. If it is part of an SRDF environment, Symmetrix systems that are on the other side of a SRDF link will be displayed as Remote.




Configuring Distributed (Remote) Operations

Verify that the SYMAPI Server is running

On the local host set the Connection environmentC:\>symcli -def

Symmetrix Command Line Interface (SYMCLI) Version V6.4.1.0 (Edit Level: 827)

built with SYMAPI Version V6.4.1.0 (Edit Level: 827)

Current settings of the SYMCLI environmental variables:

SYMCLI_CONNECT : SYMAPI_SERVER

SYMCLI_CONNECT_TYPE : REMOTE_CACHED

C:\>symcli -def

Symmetrix Command Line Interface (SYMCLI) Version V6.4.1.0 (Edit Level: 827)


Current settings of the SYMCLI environmental variables:

SYMCLI_CONNECT : SYMAPI_SERVER

SYMCLI_CONNECT_TYPE : REMOTE_CACHED

C:\>symcfg –services list

S Y M A P I N E T S E R V I C E S

Port Security

Name Domain Type Node Name Address Number Level

------------ ------ -------- -------------------- --------------- ------ ---------

SYMAPI_SERVER TCPIP w2k3-39-106 10.127.39.106 2707 ANY

C:\>symcfg –services list

S Y M A P I N E T S E R V I C E S

Port Security

Name Domain Type Node Name Address Number Level

------------ ------ -------- -------------------- --------------- ------ ---------

SYMAPI_SERVER TCPIP w2k3-39-106 10.127.39.106 2707 ANY

For remote operations:

1. On the host system directly connected to the Symmetrix, the storsrvd daemon should be running.

2. On the client host, set the environment variable SYMCLI_CONNECT to SYMAPI_SERVER.

3. On the client edit the configuration file <installdir>\emc\symapi\config\netcnfg and add the IP address of the host that is running the storsrvd daemon.




Software Architecture SummaryCommunicates with the Symmetrix through a gatekeeper device– Small Symmetrix Logical Volume

presented to host– SYMCLI Sends low level commands

through these devices– When running SYMCLI on the

Service Processor, a pseudo gatekeeper is used

Maintains a symapi database on the local host – Contains configuration and status information– Created by running symcfg discover command

The default location and name of the database file:

– <Root directory>\EMC\symapi\db\symapi_db.bin

SYMCLI SYMAPI

Environment Variables

SYMAPIDB

GK

The SYMCLI commands are built on top of SYMAPI library functions, which use system calls that generate low-level SCSI commands to the storage arrays. To reduce the number of inquiries from the host to the storage arrays, configuration and status information is maintained in a Symmetrix host database file (called the Symmetrix configuration database; symapi_db.bin by default).

Commands are invoked from the host command line or script.

The communications path is through a Gatekeeper devicesSmall devices, 6 cylinder minimum configuration for a gatekeeperBest Practice is to have several gatekeepers− Specific devices can be defined as gatekeepers−A gatekeeper can be explicitly associated to a specific device group

Host-resident symapi databaseContains configuration and status informationHost specific Groups and/or relationships created on one host are not universal to all hosts in the Symmetrix environmentCreated by running symcfg discover command

Solutions Enabler includes Group Name Services (GNS). This enable group definitions to be shared and maintained across multiple hosts, including mainframes.




Closing Slide


Understanding Symmetrix Configuration - 1



Module 2:Understanding Symmetrix Configuration

Welcome to the second module of Symmetrix Monitoring and Management. In this module we will be looking closer at the Symmetrix architecture from the prospective of a host using Solutions Architecture. We will start by looking at the physical configuration including front-end, back-end, memory and disk and finish by looking at the logical configuration including logical volumes.



© 2007 EMC Corporation. All rights reserved. Understanding Symmetrix Configuration - 2


1.0 August, 2007 Complete









Lesson 1 Physical Configuration

Objective:

Map your understanding of the physical Symmetrix configuration to the output of various SYMCLI commands

Understand the various device naming conventions from a host, Symmetrix, and Solutions Enabler perspective

Determine if a front-end director is online and if there is a physical cable connected

Describe the concept of disk groups

Describe the concepts of sparing and determine how the spare pool is configured and if there are spares invoked




Symmetrix ArchitectureRemember this Picture!– Front-end– Back-end– Memory

While the host may see it as a Black Box, the Symmetrix is really a complex and sophisticated intelligent storage system

As you look at the configuration of the Symmetrix, keep this block diagram in mind.




Host and Symmetrix PerspectivesFrom a host perspective, a Symmetrix array appears as a number of physical disk devices connected to one or more I/O controllersHost applications address devicesusing a physical device name– /dev/dsk/c1t1d1– hdisk2– PHYSICALDRIVE4

Symmetrix devices map to slices of a physical disk called a hyper– Typically a Symmetrix device maps to multiple hypers for redundancy

Symmetrix configuration, and mapping to host device names is maintained in the SYMAPI DB on the management host SYMCLI commands are used to query device details from either the SYMAPI database or directly from the Symmetrix– Configuration and status information – Back-end information - Disk Directors and Physical Drives– Front-add director and channel addresses

A host views a Symmetrix device as a physical disk and will assign it a logical name. In the Symmetrix, a device is a logical abstraction of a disk and maps to slices of physical disks called hypers. On the management host, the SYMAPI database contains a copy of configuration and status information. The SYMNAPI database also maps the host device name to the Symmetrix device number.




Terminology

Symmetrix Frame

Physical Disk Drive

Hyper-volumes (Splits)

Symmetrix Logical Volume (SLV)

Symmetrix ID (sid or symmid)

Physical Device (Pdev)

Logical Device (Ldev)

Symmetrix Device (Dev)

SolutionsEnabler

Symmetrix Hardware

Depending on your point of view, a Symmetrix device can have a number of different names.




Help!In addition to the full set of documentation, there is extensive online information – man symcli– symcli

Provides the version number – symcli -h

Describes how to use the SYMCLI command.– symcli -v

Displays the list of SYMCLI commands and a brief description of each command.

– symcli -envDisplays a list of the environment variables that can be set for a SYMCLI session.

– symcli -defDisplays a list of the environment variables that are set for the current SYMCLI session.

For more information about Solutions Enabler symcli commands, reference the Solutions Enabler Symmetrix CLI Command Reference.




symcli

C:\Program Files\EMC\SYMCLI\bin> symcli -v

Symmetrix Command Line Interface (SYMCLI) Version V6.0.2.0 (Edit Level:642)


SYMCLI BASE Commands

...

symcfg - Discover or display Symmetrix configuration information. Refresh the host's Symmetrix database file or remove Symmetrix info from the file. Can also be used to view or release a 'hanging‘Symmetrix exclusive lock.

...

symdev - Perform operations on a device given the device's Symmetrix name. Can also be used to view Symmetrix device locks.

symdisk - Display information about the disks within a Symmetrix.

syminq - Issues a SCSI Inquiry command on one or all devices.

...

symlmf - Registers SYMAPI license keys.

...

C:\Program Files\EMC\SYMCLI\bin> symcli -v

Symmetrix Command Line Interface (SYMCLI) Version V6.0.2.0 (Edit Level:642)


SYMCLI BASE Commands

...

symcfg - Discover or display Symmetrix configuration information. Refresh the host's Symmetrix database file or remove Symmetrix info from the file. Can also be used to view or release a 'hanging‘Symmetrix exclusive lock.

...

symdev - Perform operations on a device given the device's Symmetrix name. Can also be used to view Symmetrix device locks.

symdisk - Display information about the disks within a Symmetrix.

syminq - Issues a SCSI Inquiry command on one or all devices.

...

symlmf - Registers SYMAPI license keys.

...

The symcli –v command provides the version number and a brief description of the available commands. In this example the output was filtered to show the commands that we will be discussing here.

Note: the current release of Solutions Enabler fir 5771 is 6.02.




symcfg -help

C:\Program Files\EMC\SYMCLI\bin> symcfg -help

NAME

symcfg - Discover, change or display Symmetrix configurationinformation. Refresh the host's Symmetrix database file orremove Symmetrix information from the database file.Can also be used to view or release a Symmetrix exclusive lock,or to online or offline RDF (RA) directors.

Scan and rebuild the set of devices seen by a host system.

Display network service file entries.

Display UNIX database and gatekeeper semaphores.

Display host and application registrationinformation.

Display mainframe CU image information.

SYNOPSIS

...

C:\Program Files\EMC\SYMCLI\bin> symcfg -help

NAME

symcfg - Discover, change or display Symmetrix configurationinformation. Refresh the host's Symmetrix database file orremove Symmetrix information from the database file.Can also be used to view or release a Symmetrix exclusive lock,or to online or offline RDF (RA) directors.

Scan and rebuild the set of devices seen by a host system.

Display network service file entries.

Display UNIX database and gatekeeper semaphores.

Display host and application registrationinformation.

Display mainframe CU image information.

SYNOPSIS

...

All the SYMCLI commands support the Help option.




Discover and Verify (symcfg discover)

Build, update, and verify the SYMAPI database

C:\Program Files\EMC\SYMCLI\bin>symcfg discover


C:\Program Files\EMC\SYMCLI>symcfg verify -sid 172


C:\Program Files\EMC\SYMCLI>

C:\Program Files\EMC\SYMCLI\bin>symcfg discover


C:\Program Files\EMC\SYMCLI>symcfg verify -sid 172


C:\Program Files\EMC\SYMCLI>

During your first command line session, or if a configuration change has occurred, the configuration database must be built, or rebuilt, with the most complete and current information for all physical devices connected to your host. The following are the discovery syntax options:symcfg

discover [-all | -symmetrix | -clariion |

-pdev [-sid SymmID] | -sid SymmID]

Example: To scan the hardware and rebuild the database, enter:symcfg discover

This command scans all SCSI buses, collects information about all the arrays and devices found, and rebuilds the database with the collected device information and parameters from all local and remotely attached devices. You can limit the discovery to just Symmetrix arrays by specifying the -symmetrix or -clariion option, only physical device information by specifying the -pdev option, or just a specific Symmetrix array by specifying the -sid option. Note: symcfg discover scans all SCSI buses on the host, not just those connected to Symmetrix arrays. This can take a significant amount of time to complete. Only use symcfg discover if you have added or removed devices seen by the host. Also, if you had previously run discover and had subsequently removed one or more array(s), a later execution of discover will not remove information from the database relating to the removed Symmetrix array(s). A symcfg -sync is less performance intensive than a full discover operation




Listing Attached Symmetrix (symcfg list)

symcfg list provides high level details of Symmetrix attached to local host

Will also provide information on remote Symmetrix in SRDF configurations

C:\Program Files\EMC\SYMCLI\bin>symcfg list

S Y M M E T R I X

Mcode Cache Num Phys Num SymmSymmID Attachment Model Version Size (MB) Devices Devices

000190100172 Local DMX3-24 5771 32768 0 126

C:\Program Files\EMC\SYMCLI\bin>symcfg list

S Y M M E T R I X

Mcode Cache Num Phys Num SymmSymmID Attachment Model Version Size (MB) Devices Devices

000190100172 Local DMX3-24 5771 32768 0 126

The first thing that you need to do before using the SYMCLI commands is to initialize the symapi database. This is done using the symcfg discover. If the database has already been initialized, a quick way to verify that it is current and correctly reflects the actual configuration is to run the “Verify” command.

The symcfg list command will provide a list of Symmetrix that are attached to the host. If running from the service processor, you will see the local system and if it is part of a SRDF configuration, it will query across the link and provide information about the remote Symmetrix as well.

In the above example, we initialized the database, listed high level information about the attached Symmetrix, and finally verified that the symapi database is in sync with the actual configuration.




Verbose Output (symcfg list -v)C:\Program Files\EMC\SYMCLI\bin>symcfg list -v | moreSymmetrix ID: 000190100172 (Local)Time Zone : Eastern Standard Time

Product Model : DMX3-24Symmetrix ID : 000190100172Microcode Version (Number) : 5771 (168B0000)Microcode Date : 07.21.2005Microcode Patch Date : 07.21.2005Microcode Patch Level : 59

Cache Size (Mirrored) : 16384 (MB)# of Available Cache Slots : 460798# of PermaCache Slots In Use : 6Max # of System Write Pending Slots : 368998Max # of DA Write Pending Slots : 184498Max # of Device Write Pending Slots : 18440

...Symmetrix Total Operating Time : 1 days, 14:28:55Number of Configured (Sym) Devices : 126Number of Visible (Host) Devices : 0Number of Configured Actual Disks : 80Number of Configured Hot Spares : 4Number of Unconfigured Disks : 0Maximum number of hypers per disk : 64

...

C:\Program Files\EMC\SYMCLI\bin>symcfg list -v | moreSymmetrix ID: 000190100172 (Local)Time Zone : Eastern Standard Time

Product Model : DMX3-24Symmetrix ID : 000190100172Microcode Version (Number) : 5771 (168B0000)Microcode Date : 07.21.2005Microcode Patch Date : 07.21.2005Microcode Patch Level : 59

Cache Size (Mirrored) : 16384 (MB)# of Available Cache Slots : 460798# of PermaCache Slots In Use : 6Max # of System Write Pending Slots : 368998Max # of DA Write Pending Slots : 184498Max # of Device Write Pending Slots : 18440

...Symmetrix Total Operating Time : 1 days, 14:28:55Number of Configured (Sym) Devices : 126Number of Visible (Host) Devices : 0Number of Configured Actual Disks : 80Number of Configured Hot Spares : 4Number of Unconfigured Disks : 0Maximum number of hypers per disk : 64

...

The –v option of the symcfg list command will display verbose information about the configuration. To keep from scrolling the output off the screen, pipe it to more and this will allow you to display the output page at a time.

In the example above we see microcode levels, status and information about the configuration. Some of the output was deleted to allow it to fit on a single page.




Environment (symcfg list -env_data)C:\Program Files\EMC\SYMCLI\bin>symcfg -sid 097 list -env_data

Symmetrix ID : 000190100097

Timestamp of Status Data : 09/29/2005 12:09:54

System BayBay Name : SystemBayNumber of Fans : 3Number of Power Supplies : 8Number of Standby of Power Supplies : 8Summary Status of Contained ModulesAll Fans : NormalAll Power Supplies : NormalAll Standby Power Supplies : Failed (1)

Drive BaysBay Name : Bay-1ANumber of Standby of Power Supplies : 8Number of Drive Enclosures : 16Summary Status of Contained ModulesAll Enclosures : NormalAll Link Control Cards : NormalAll Power Supplies : NormalAll Standby Power Supplies : Normal

Bay Name : Bay-1B

…

C:\Program Files\EMC\SYMCLI\bin>symcfg -sid 097 list -env_data


Timestamp of Status Data : 09/29/2005 12:09:54

System BayBay Name : SystemBayNumber of Fans : 3Number of Power Supplies : 8Number of Standby of Power Supplies : 8Summary Status of Contained ModulesAll Fans : NormalAll Power Supplies : NormalAll Standby Power Supplies : Failed (1)

Drive BaysBay Name : Bay-1ANumber of Standby of Power Supplies : 8Number of Drive Enclosures : 16Summary Status of Contained ModulesAll Enclosures : NormalAll Link Control Cards : NormalAll Power Supplies : NormalAll Standby Power Supplies : Normal

Bay Name : Bay-1B

…

Environment data including fans and power supplies.




Summary of all Directors (symcfg list –dir ALL)

C:\Program Files\EMC\SYMCLI\bin>symcfg list -dir ALL

Symmetrix ID: 000190100172 (Local)

S Y M M E T R I X D I R E C T O R SIdent Symbolic Numeric Slot Type Status

DF-1A 01A 1 1 DISK OnlineDF-2A 02A 2 2 DISK OnlineFA-7A 07A 7 7 FibreChannel OnlineFA-8A 08A 8 8 FibreChannel OnlineFA-9A 09A 9 9 FibreChannel OnlineFA-10A 10A 10 10 FibreChannel OnlineDF-15A 15A 15 15 DISK OnlineDF-16A 16A 16 16 DISK OnlineDF-1B 01B 17 1 DISK OnlineDF-2B 02B 18 2 DISK OnlineFA-7B 07B 23 7 FibreChannel OnlineFA-8B 08B 24 8 FibreChannel OnlineFA-9B 09B 25 9 FibreChannel OnlineFA-10B 10B 26 10 FibreChannel OnlineDF-15B 15B 31 15 DISK OnlineDF-16B 16B 32 16 DISK OnlineDF-1C 01C 33 1 DISK OnlineDF-2C 02C 34 2 DISK Online

C:\Program Files\EMC\SYMCLI\bin>symcfg list -dir ALL


S Y M M E T R I X D I R E C T O R SIdent Symbolic Numeric Slot Type Status

DF-1A 01A 1 1 DISK OnlineDF-2A 02A 2 2 DISK OnlineFA-7A 07A 7 7 FibreChannel OnlineFA-8A 08A 8 8 FibreChannel OnlineFA-9A 09A 9 9 FibreChannel OnlineFA-10A 10A 10 10 FibreChannel OnlineDF-15A 15A 15 15 DISK OnlineDF-16A 16A 16 16 DISK OnlineDF-1B 01B 17 1 DISK OnlineDF-2B 02B 18 2 DISK OnlineFA-7B 07B 23 7 FibreChannel OnlineFA-8B 08B 24 8 FibreChannel OnlineFA-9B 09B 25 9 FibreChannel OnlineFA-10B 10B 26 10 FibreChannel OnlineDF-15B 15B 31 15 DISK OnlineDF-16B 16B 32 16 DISK OnlineDF-1C 01C 33 1 DISK OnlineDF-2C 02C 34 2 DISK Online

In this example we are listing the directors in the system and the high level status.

Note the different ways to identify a director: Symbolic, Numeric, and Slot number. Also note that the Slot numbers are in decimal starting with one. Whereas Symmetrix slots numbers are in hexadecimal starting with zero.




Back-end Directors (symcfg -DA)

C:\Program Files\EMC\SYMCLI\bin>symcfg -DA ALL list

Symmetrix ID: 000190100172 (Local)S Y M M E T R I X D I S K D I R E C T O R S

Num Of ServicedIdent Symbolic Numeric Slot Type Hyper Volumes Status

DF-1A 01A 1 1 DISK 12 OnlineDF-2A 02A 2 2 DISK 7 OnlineDF-15A 15A 15 15 DISK 8 OnlineDF-16A 16A 16 16 DISK 11 OnlineDF-1B 01B 17 1 DISK 11 OnlineDF-2B 02B 18 2 DISK 8 OnlineDF-15B 15B 31 15 DISK 9 OnlineDF-16B 16B 32 16 DISK 13 OnlineDF-1C 01C 33 1 DISK 12 OnlineDF-2C 02C 34 2 DISK 6 OnlineDF-15C 15C 47 15 DISK 8 OnlineDF-16C 16C 48 16 DISK 10 OnlineDF-1D 01D 49 1 DISK 11 OnlineDF-2D 02D 50 2 DISK 10 Online

C:\Program Files\EMC\SYMCLI\bin>symcfg -DA ALL list

Symmetrix ID: 000190100172 (Local)S Y M M E T R I X D I S K D I R E C T O R S

Num Of ServicedIdent Symbolic Numeric Slot Type Hyper Volumes Status

DF-1A 01A 1 1 DISK 12 OnlineDF-2A 02A 2 2 DISK 7 OnlineDF-15A 15A 15 15 DISK 8 OnlineDF-16A 16A 16 16 DISK 11 OnlineDF-1B 01B 17 1 DISK 11 OnlineDF-2B 02B 18 2 DISK 8 OnlineDF-15B 15B 31 15 DISK 9 OnlineDF-16B 16B 32 16 DISK 13 OnlineDF-1C 01C 33 1 DISK 12 OnlineDF-2C 02C 34 2 DISK 6 OnlineDF-15C 15C 47 15 DISK 8 OnlineDF-16C 16C 48 16 DISK 10 OnlineDF-1D 01D 49 1 DISK 11 OnlineDF-2D 02D 50 2 DISK 10 Online

In this example we are looking only at the back-end directors or the Disk Adapters.

Note the output shows the number of hyper volumes or splits serviced by each director.




Open Systems Front-end Directors (symcfg –SA)

C:\Program Files\EMC\SYMCLI\bin>symcfg -SA ALL list


S Y M M E T R I X D I R E C T O R S

Ident Symbolic Numeric Slot Type Status

FA-7A 07A 7 7 FibreChannel OnlineFA-8A 08A 8 8 FibreChannel OnlineFA-9A 09A 9 9 FibreChannel OnlineFA-10A 10A 10 10 FibreChannel OnlineFA-7B 07B 23 7 FibreChannel OnlineFA-8B 08B 24 8 FibreChannel OnlineFA-9B 09B 25 9 FibreChannel OnlineFA-10B 10B 26 10 FibreChannel OnlineFA-7C 07C 39 7 FibreChannel OnlineFA-8C 08C 40 8 FibreChannel OnlineFA-9C 09C 41 9 FibreChannel OnlineFA-10C 10C 42 10 FibreChannel Online

C:\Program Files\EMC\SYMCLI\bin>symcfg -SA ALL list


S Y M M E T R I X D I R E C T O R S

Ident Symbolic Numeric Slot Type Status

FA-7A 07A 7 7 FibreChannel OnlineFA-8A 08A 8 8 FibreChannel OnlineFA-9A 09A 9 9 FibreChannel OnlineFA-10A 10A 10 10 FibreChannel OnlineFA-7B 07B 23 7 FibreChannel OnlineFA-8B 08B 24 8 FibreChannel OnlineFA-9B 09B 25 9 FibreChannel OnlineFA-10B 10B 26 10 FibreChannel OnlineFA-7C 07C 39 7 FibreChannel OnlineFA-8C 08C 40 8 FibreChannel OnlineFA-9C 09C 41 9 FibreChannel OnlineFA-10C 10C 42 10 FibreChannel Online

In this example we are looking only at the front-end directors. The SA option will display all open systems front end directors.

Note the different ways to identify a director: Symbolic, Numeric, and Slot number. Also note that the Slot numbers are in decimal starting with one. Whereas Symmetrix slots numbers are in hexadecimal starting with zero.




Director and Port StatusC:\Program Files\EMC\SYMCLI\bin>symcfg -fa all list -port -sid 097

Symmetrix ID: 000190100097

S Y M M E T R I X D I R E C T O R P O R T S

Director Port Status ConnectionStatus

Ident Type Status P0 P1 P2 P3 P0 P1 P2 P3

FA-3A FibreChannel Online ON ON N/A N/A X - - -

FA-4A FibreChannel Online ON ON N/A N/A - - - -





FA-3B FibreChannel Online ON ON N/A N/A X - - -

< . . .>

C:\Program Files\EMC\SYMCLI\bin>symcfg -fa all list -port -sid 097



Director Port Status ConnectionStatus

Ident Type Status P0 P1 P2 P3 P0 P1 P2 P3







FA-3B FibreChannel Online ON ON N/A N/A X - - -

< . . .>

In this example we are listing the front-end directors, on-line status and if there is a cable connection..




Front-end Channel Director Details (-v)C:\Program Files\EMC\SYMCLI\bin> symcfg -SA 07a -v list...Director Identification: FA-7A

Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags{

Tagged_Commands : EnabledLinked_Commands : EnabledSync_Transfer : EnabledWide_Transfer : Enabled...

C:\Program Files\EMC\SYMCLI\bin> symcfg -SA 07a -v list...Director Identification: FA-7A

Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags{

Tagged_Commands : EnabledLinked_Commands : EnabledSync_Transfer : EnabledWide_Transfer : Enabled...

For a more detailed information about a specific director you can use the –v option to get verbose output. In this example, we are looking at a fibre Channel director which has two ports per processor. Some of the output was deleted so that it would fit on a single screen.

Note: SA identifies SCSI Adapters and includes Parallel SCSI, (SA), Fibre Channel (FA), and iSCSI (SE).




Global Memory (symcfg -memory)C:\Program Files\EMC\SYMCLI\bin>symcfg list -memory

Symmetrix ID : 000190100172Number of Memory Boards : 4

Slot Number Capacity(MB)

0 163841 163842 163843 16384

--------Total (Usable Cache) 32768

--------

C:\Program Files\EMC\SYMCLI\bin>symcfg list -memory

Symmetrix ID : 000190100172Number of Memory Boards : 4

Slot Number Capacity(MB)

0 163841 163842 163843 16384

--------Total (Usable Cache) 32768

--------

Here we are looking at the memory that is configured in the system. The reason we have (4) 16GB boards but only 32GB of usable capacity is this is a DMX3 with Mirrored Cache.




Physical Disk Drives (symdisk list)

C:\Program Files\EMC\SYMCLI\bin>symdisk list

Symmetrix ID : 000190100172Disks Selected : 84

Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------DF-1A 01A C 0 SEAGATE C73LDFX 2 70007 43283 70007DF-1A 01A C 2 SEAGATE C73LDFX 2 70007 43283 70007DF-1A 01A C 10 SEAGATE C146LDF 3 140014 133771 140014DF-1A 01A C 12 SEAGATE C146LDF 2 140014 133780 140014DF-1A 01A D 1 SEAGATE C146LDF 3 140014 134885 140014DF-1A 01A D 3 SEAGATE C146LDF 2 140014 134885 140014DF-2A 02A C 1 SEAGATE C146LDF 3 140014 133953 140014DF-2A 02A C 3 SEAGATE C146LDF 2 140014 133953 140014DF-2A 02A D 0 SEAGATE C146LDF 3 140014 132073 140014DF-2A 02A D 2 SEAGATE C146LDF 2 140014 132073 140014DF-15A 15A C 0 SEAGATE C146LDF 3 140014 129598 140014DF-15A 15A C 2 SEAGATE C146LDF 2 140014 133953 140014DF-15A 15A D 1 SEAGATE C146LDF 3 140014 132073 140014DF-15A 15A D 3 SEAGATE C146LDF 2 140014 132073 140014DF-16A 16A C 1 SEAGATE C73LDFX 2 70007 43283 70007DF-16A 16A C 3 SEAGATE C73LDFX 2 70007 43283 70007DF-16A 16A C F SEAGATE C146LDF 3 140014 136255 140014DF-16A 16A C 11 SEAGATE C146LDF 2 140014 133780 140014DF-16A 16A D 0 SEAGATE C146LDF 3 140014 134885 140014

C:\Program Files\EMC\SYMCLI\bin>symdisk list


Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------DF-1A 01A C 0 SEAGATE C73LDFX 2 70007 43283 70007DF-1A 01A C 2 SEAGATE C73LDFX 2 70007 43283 70007DF-1A 01A C 10 SEAGATE C146LDF 3 140014 133771 140014DF-1A 01A C 12 SEAGATE C146LDF 2 140014 133780 140014DF-1A 01A D 1 SEAGATE C146LDF 3 140014 134885 140014DF-1A 01A D 3 SEAGATE C146LDF 2 140014 134885 140014DF-2A 02A C 1 SEAGATE C146LDF 3 140014 133953 140014DF-2A 02A C 3 SEAGATE C146LDF 2 140014 133953 140014DF-2A 02A D 0 SEAGATE C146LDF 3 140014 132073 140014DF-2A 02A D 2 SEAGATE C146LDF 2 140014 132073 140014DF-15A 15A C 0 SEAGATE C146LDF 3 140014 129598 140014DF-15A 15A C 2 SEAGATE C146LDF 2 140014 133953 140014DF-15A 15A D 1 SEAGATE C146LDF 3 140014 132073 140014DF-15A 15A D 3 SEAGATE C146LDF 2 140014 132073 140014DF-16A 16A C 1 SEAGATE C73LDFX 2 70007 43283 70007DF-16A 16A C 3 SEAGATE C73LDFX 2 70007 43283 70007DF-16A 16A C F SEAGATE C146LDF 3 140014 136255 140014DF-16A 16A C 11 SEAGATE C146LDF 2 140014 133780 140014DF-16A 16A D 0 SEAGATE C146LDF 3 140014 134885 140014

The symdisk command allows access to the configuration information of the physical disks (spindles) that make up a Symmetrix array. It can be used to list all of the disks in a Symmetrix array, or only those that match certain criteria.

In this example we are displaying only summary information. Displayed is the Physical disks that are configured in the system, and how much of each drive is actually being used.

Note the interface is showing as C & D but with a DMX-3 they are really 0 & 1.

The selection criteria allows the user to return only data about the disks on a certain disk director (DA), disk interface (INT), or disk SCSI target ID (TID).

The -hypers flag can be used with -v to display additional information about each of the logical hypers on a given disk, including which Symmetrix devices they make up.

The -by_diskgroup option will organize the disks by disk group number. The -disk_group option will print only disks within that disk group.




Physical Disk Drive Details (symdisk list –v)C:\Program Files\EMC\SYMCLI\bin> symdisk list –v | more ...Symmetrix ID : 000190100172Disks Selected : 84

Director : DF-1AInterface : CTarget ID : 0Disk Group Number : 1

Vendor ID : SEAGATEProduct ID : SX373307FCProduct Revision : C73LDFXSerial ID : 3HZ7FYLM

Disk Blocks : 143374737Block Size : 512Actual Disk Blocks : 143374737Total Disk Capacity (MB) : 70007Free Disk Capacity (MB) : 43283Actual Disk Capacity (MB) : 70007Hypers : 2

...

C:\Program Files\EMC\SYMCLI\bin> symdisk list –v | more ...Symmetrix ID : 000190100172Disks Selected : 84

Director : DF-1AInterface : CTarget ID : 0Disk Group Number : 1

Vendor ID : SEAGATEProduct ID : SX373307FCProduct Revision : C73LDFXSerial ID : 3HZ7FYLM

Disk Blocks : 143374737Block Size : 512Actual Disk Blocks : 143374737Total Disk Capacity (MB) : 70007Free Disk Capacity (MB) : 43283Actual Disk Capacity (MB) : 70007Hypers : 2

...

Again, the –v or verbose option can display much more information about each disk.




Disk Groups

When the Symmetrix is configured, disk drives are assigned to disk groups– Type– Capacity– Performance

Isolate and organize disk drives– Work load isolation– Tiered storage

When Symmetrix Volumesare created, space is allocated from a Disk Group

When physical disk drives are added to the Symmetrix, they can be organized into disk groups. When Symmetrix Logical devices are configured, you can specify a disk group to use.




Disk Groups (symdisk list –by_diskgroup)C:\>symdisk list -by_diskgroupSymmetrix ID : 000190102254Disks Selected : 64

Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------Disk Group 1:DF-1A 01A C 0 SEAGATE C300LPF 2 286102 272535 286102DF-1A 01A C 2 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A C 4 SEAGATE C300LPF 2 286102 276583 286102DF-1A 01A C 6 SEAGATE C300LPF 2 286102 281445 286102DF-1A 01A D 1 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A D 3 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A D 5 SEAGATE C300LPF 1 286102 285200 286102DF-1A 01A D 7 SEAGATE C300LPF 1 286102 277655 286102DF-16A 16A C 1 SEAGATE C300LPF 2 286102 272535 286102DF-16A 16A C 3 SEAGATE C300LPF 2 286102 277227 286102DF-16A 16A C 5 SEAGATE C300LPF 2 286102 276583 286102DF-16A 16A C 7 SEAGATE C300LPF 2 286102 282383 286102DF-16A 16A D 0 SEAGATE C300LPF 2 286102 272535 286102DF-16A 16A D 2 SEAGATE C300LPF 2 286102 277227 286102DF-16A 16A D 4 SEAGATE C300LPF 2 286102 281445 286102DF-16A 16A D 6 SEAGATE C300LPF 2 286102 273900 286102DF-1B 01B C 1 SEAGATE C300LPF 2 286102 272535 286102DF-1B 01B C 3 SEAGATE C300LPF 2 286102 277227 286102…

C:\>symdisk list -by_diskgroupSymmetrix ID : 000190102254Disks Selected : 64

Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------Disk Group 1:DF-1A 01A C 0 SEAGATE C300LPF 2 286102 272535 286102DF-1A 01A C 2 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A C 4 SEAGATE C300LPF 2 286102 276583 286102DF-1A 01A C 6 SEAGATE C300LPF 2 286102 281445 286102DF-1A 01A D 1 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A D 3 SEAGATE C300LPF 2 286102 277227 286102DF-1A 01A D 5 SEAGATE C300LPF 1 286102 285200 286102DF-1A 01A D 7 SEAGATE C300LPF 1 286102 277655 286102DF-16A 16A C 1 SEAGATE C300LPF 2 286102 272535 286102DF-16A 16A C 3 SEAGATE C300LPF 2 286102 277227 286102DF-16A 16A C 5 SEAGATE C300LPF 2 286102 276583 286102DF-16A 16A C 7 SEAGATE C300LPF 2 286102 282383 286102DF-16A 16A D 0 SEAGATE C300LPF 2 286102 272535 286102DF-16A 16A D 2 SEAGATE C300LPF 2 286102 277227 286102DF-16A 16A D 4 SEAGATE C300LPF 2 286102 281445 286102DF-16A 16A D 6 SEAGATE C300LPF 2 286102 273900 286102DF-1B 01B C 1 SEAGATE C300LPF 2 286102 272535 286102DF-1B 01B C 3 SEAGATE C300LPF 2 286102 277227 286102…

To display how physical disk are organized into disk groups, execute the symdisk listcommand using the –by_diskgroup option.




Data Protection - Sparing

Spares– When the Symmetrix is setup, a number of disk drives are

configured as hot sparesNot User-addressableMember of Spare Pool

When a drive has failed or the number of soft error indicate that a hard failure is probable, a replacement is initiated– Proactive monitoring through disk scrubbing

Two sparing policies– Temporary– Permanent

The sparing function is automatically initiated by Enginuity when it determines that the number of errors logging on a volume is excessive, and that a hard failure is probable. By default, invoking a spare invokes a call-home. Sparing is designed to be transparent to the user and host level processing. In fact, the only indication a user may have is a EMC Customer Engineer arrives on site to replace the failed drive.

Sparing can also be initiated by a script on the service processor.




Spare PoolWhen the binfile is created, a number of drives can be set aside for the Spare pool– Minimum of two spares of each

speed & capacity– Plus an additional spare for

every 100 drives

Drives in the pool are used for either temporary or permanent sparing

Spare

Spare

Spare

Spare pool

Spare

When the Symmetrix is configured, spare drives are set up through the SymmWin DiskMap wizard. Members in the pool of spares can be used either as temporary spares or permanent spares.

While drives are very reliable, the best practice is to configure a minimum of 2 host spares plus one additional spare for every 100 drives of each capacity and speed.




Dynamic Sparing - Temporary Replacement “Hot spare” is invoked against bad drive– Takes up a mirror position.– Data is copied to the Hot spare and bad

drive is made Not Ready (NR)Host I/O processed at higher priority

When bad drive is replaced, data is copied back to new drive– EMC Support initiates replacement

Hot spare is released and is ready to be used again

Dynamic Sparing– (+) Data Protection– (+) Maintains back-end configuration– (-) Two Copy operations required

M2

Spare

M1 X

With Dynamic sparing, a hot spare is allocated as a spare and returned to the spare pool after the failing physical is replaced. This functionality is implemented by Enginuity. This process increase data protection in the event of a media failure by minimizing the time that a volume is in a degraded state. In nearly all cases, Symmetrix logical volumes will be configured with either mirrored or parity based protection. While these schemes provide protection in the event of a single drive failure, if a second failure occurs, data could be loss. By configuring Spares, if a failure occurs, or even if there is an increase in the number of soft (recoverable) errors, a Spare drive will be invoked and the data will be copied from the failing driver or the other members in the volume. After a failed drive is repaired, data is resynchronizes to the new disk.

The process chooses an available spare drive of equal or larger capacity. There is no restriction preventing a 7.2K/10K/15K spare from being invoked against any other speed failing drive. For example, it is possible that a 7.2K spare will be invoked against a 15K failing drive, which may affect performance.

Sparing notes:Disk Scrubbing is a feature of Enginuity where DA’s read disk blocks looking for media defectsMicrocode invokes sparing when error threshold exceeded.Spare becomes the next available mirrorSpare will emulate either CKD or FBACopying data (or rebuilding data) is a lower priority background task but priority can be increased using Inlines command




Permanent Sparing Dynamic (Temporary) Spare is invoked only for unprotected volumes– Quickly copy data to Dynamic spare before total drive failure– Dynamic spare not used if volume is RAID 1, 5, or 6 protected

Spare is identified and permanent configuration change is made– Spare must be same capacity and speed as failed drive– Spare must be on a different director/port than good mirror or RAID member

Failed is added to the Spare pool as “failed”– When replaced, will become available as a spare– Customer Engineer initiates action

After M2Before

Data Flow

M1xAvailable spare in

spare poolBad Drive in Spare Pool

Permanent Sparing Advantages:Reduces the amount of time required to replace the failing drive− If the volume being spared has a ready mirror, there is no need for a dynamic spare−Reduces the time the system is ‘locked’ from configuration changes and other operations

Time of exposure to a second drive failure in the RAID group is reducedReduces the need for EMC Customer Service to replace the failed drive immediately

DisadvantagesMore spares are required− 2 + (1 per 100 physicals) for every drive type

Must follow the same configuration rules for distributing volumes− Same capacity, speed, and block size−Cannot be on the same disk director or loop of any mirrors of the failed disk−Not all drive failures will be candidates for permanent sparing




Hot Spares Configuration

Spare configuration informationC:\>symdisk list -hotspares


Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------DF-1A 01A C 4 SEAGATE C300LPF 0 0 0 286102DF-1B 01B D 4 SEAGATE C300LPF 0 0 0 286102DF-16C 16C D 4 SEAGATE C300LPF 0 0 0 286102…

C:\>symdisk list -spare_info -v…Director : DF-1AInterface : CTarget ID : 4Disk Group Number : 0…Hot Spare : True

Failed Director : N/AFailed Interface : N/AFailed Target ID : N/A

Failed Disk : False

C:\>symdisk list -hotspares


Capacity(MB)Ident Symb Int TID Vendor Type Hypers Total Free Actual------ ---- --- --- ---------- ---------- ------ -------- -------- --------DF-1A 01A C 4 SEAGATE C300LPF 0 0 0 286102DF-1B 01B D 4 SEAGATE C300LPF 0 0 0 286102DF-16C 16C D 4 SEAGATE C300LPF 0 0 0 286102…

C:\>symdisk list -spare_info -v…Director : DF-1AInterface : CTarget ID : 4Disk Group Number : 0…Hot Spare : True

Failed Director : N/AFailed Interface : N/AFailed Target ID : N/A

Failed Disk : False

The symdisk list -hotspare command will return information about hot spare devices. If a hot spare is invoked against a failed disk, the symdisk list-v -spare_info option can be used to return information about the failed disk that has been replaced.




Hot Spares & Dynamic Spares

Determining if hot spare is invoked

C:>/symdev list -hotspare


Could not select any Symmetrix devices to list.C:>/symdev list -hotspareSymmetrix ID: 000184500120Device Name Directors Device------------------------- ------------ ----------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)------------------------- ------------ -------------------------------------0010 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 1030098 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700AE Not Visible 13A:0 01A:D0 2-Way Mir N/Grp'd RW 431500C7 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700DF Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700F5 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 470108 Not Visible ???:? 01A:D0 2-Way Mir N/Grp'd RW 469

C:>/symdev list -hotspare


Could not select any Symmetrix devices to list.C:>/symdev list -hotspareSymmetrix ID: 000184500120Device Name Directors Device------------------------- ------------ ----------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)------------------------- ------------ -------------------------------------0010 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 1030098 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700AE Not Visible 13A:0 01A:D0 2-Way Mir N/Grp'd RW 431500C7 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700DF Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 4700F5 Not Visible ???:? 01A:D0 Unprotected N/Grp'd NR 470108 Not Visible ???:? 01A:D0 2-Way Mir N/Grp'd RW 469

When a disk fails, a hot spare is invoked against it. All devices that have mirrors on the failed disk against which the hot spare was invoked, are shown in the symdev list –hotsparedisplay as having a hot spare invoked against them. The first example shows that there are no hot spares invoked against failed disks. The second example shows one hot spare invocation against disk 01A:D0. Unprotected devices with a hot spare invoked against them become not ready (NR) because unprotected devices have no mirror to use to rebuild the data on the spare. Mirrored or RAID 5/6 protected devices continue to be ready (RW) when a hot spare is invoked against them.




Lesson 2 – Logical Volume Configuration

Objectives

Describe the relationship between physical disks and logical devices within the Symmetrix, and how they are viewed by and attached host

Briefly describe the different types of logical volumes

Compare and contrast the supported Symmetrix data protection schemes and how they are implemented

Describe how mirror positions are used

Describe the concept of meta devices

The first part of this module was focused on the physical aspects of the Symmetrix. This lesson is focused on the logical volume configuration.




Symmetrix Logical VolumesLogical abstraction of a disk drive– Industry term is LUN – Logical Unit– EMC terms often used are hyper-volume, slice, split, device, or volume– Assigned a Volume Identifier

Enginuity maps Logical Volumes to locations on Physical Disk on back-end– Protection from media failure

RAID 1 MirroringRAID 5 and RAID 6 parity based protection

– ReplicationLocalRemote

Symmetrix Logical Volumes are made available to a host– Physical Connectivity– Front-end Channel Address– Device Masking

One of the most confusing aspects of learning the Symmetrix is the different terms that are used to describe the same thing.




Symmetrix Logical Volume IdentifiersSymmetrix Device Name (SymDevName)– Unique hex identifier assigned when a volume is created

0000-FFFF

Physical Device Name (PevName)– Name used by the host to uniquely identify a disk drive– Maintained in the SYMAPI Database– Will not be assigned if not visible to host

Example: /dev/rdsk/c1t2d0

Symmetrix Logical Device Name (LdevName)– Assigned name – Useful for documenting a system configuration

Example: ldev00c7 or ora_redo_1

Fibre Channel World Wide Name (WWN)– Used by Fibre Channel to uniquely identify a device– Example: 60060480000190100172533030303441

Every Symmetrix Logical Volume has a number of different ways to identify a Symmetrix Logical Volume. Some are used internally by the Symmetrix and others are used by host and applications.

In addition a Symmetrix Logical Volume may be assigned a unique label or signature. This is normally done by the host operating system but can also be assigned using the symlabelSMYCLI command for Windows hosts.




Volume Geometry A Symmetrix Logical Volume is an emulation of a Physical Disk and uses similar terminology– Sector

(8) 512 byte Blocks(16) 512 byte Block (DMX3/4)

– Track (R/W Head)(8) Sectors

– Cylinders(15) Tracks

Volume sizes are typically specified in Cylinders

9000 Cyl Device = 8.8GB (9000 X 15 X 8 X 16 X 512 = 8847360000)

Host I/O operation are managed by the Enginuity operating environment, which runs in the Symmetrix I/O subsystem (channel directors and disk directors). Because each of the physical disks are indirectly seen as part of the I/O protocol, Symmetrix devices are presented to the host with the following configuration or emulation attributes:

Each device has N cylinders. The number is configurable (blocks ÷ 960)Each cylinder has 15 tracks (heads)Each device track in a fixed block architecture (FBA) has 128 blocks of 512 bytes (64K)Note: previous to DMX3, the track sizes for FBA devices 32KMainframe hosts use Count Key Devices (CKD) uses variable block sizes.

Maximum Volume size that can be configured is 65520 Cylinders. If host applications require larger volumes, multiple volumes can be combined to form a metavolume.




Symmetrix Logical Volume EmulationsOpen Systems hosts use Fixed Block Architecture (FBA)– Each block is a fixed size of 512 bytes– A host sees a Symmetrix device as

a number of 512 byte blocks

Mainframes use Count Key Data (CKD) devices

– Count field indicates the data record’s physical location (cylinder and head) record number, key length, and data length

– Key field is optional and contains information used by the applicationFor example it could be used as an index

– Data field is the area which contains the user data and is variable in length

DataCount Key

512 Byte Data Block

The Symmetrix supports two emulation modes: FBA and CKD. Open systems use FBA emulation and mainframe uses CKD. CKD data fields are variable in length and can extend beyond the 512 byte blocks used internally in the Symmetrix. The size of a volume is specified by the number of cylinders with one cylinder equaling 15 tracks. A track is the unit of cache allocation. With open systems, the track size was historically 32K and a mainframe track was 57K. With the DMX-3, a track size is now 64K. The reason for this change is more efficient cache utilization..

A notable exception to the “512-byte” open systems rule is AS/400 (iSeries). It uses 520 bytes per block.

Regardless of the emulation, internally, the Symmetrix stores both FBA and CKD data on physical disk in 512 byte FBA format.




Symmetrix Device TypesStandard Devices

A Symmetrix device configured for normal Symmetrix operation

Configured using protection method such as RAID 1, RAID-5, RAID 6

Gatekeeper Devices

SCSI commands executed by SYMAPI are transferred to the Symmetrix array via a Symmetrix device that is designated as a gatekeeper device

Any device can be used for a gatekeeper but typically a number of vsmall volumes are configured specifically to be used as gatekeepers

Metadevices Multiple Standard Devices can be concatenated to create larger devices

Consists of a metahead and its member devices. The metahead is the first device in the metadevice sequence and is responsible for receiving all incoming commands. It also identifies the entire metadevice. When an incoming command for the metahead is processed, the Symmetrix determines which metadevice member should execute the command.

Potential Performance benefits

BCV Device Specialized devices used by the Symmetrix TimeFinder application to create a local copy of data contained in a standard Symmetrix device

Typically used for backup, restore, decision support, and application testing

BCV device have own host address, and is configured as a stand-alone Symmetrix device




Symmetrix Device TypesSRDF Devices

Type of volumes used by Symmetrix Remote Data Facility (SRDF) to create a device-level mirror in a remote Symmetrix

Devices are configured as either RDF1 or RDF2 where during normal operation data is mirrored from the RDF1 to RDF2 devices

RDF1 are the local volume used for production

RDF2 devices are in configured in a remote Symmetrix to support disaster recovery and other applications

Dynamic RDF Devices

Enables the dynamic creation, deletion, and swapping of RDF1 and RDF2 relationships online

Dynamic RDF must be enabled via the Configuration Manager and the devices must be designated as Dynamic RDF-capable devices

Virtual Devices

Used in TimeFinder/Snap operations

A host-accessible device containing track-level location information (pointers), which indicates where the copy session data is located in the physical storage.

Virtual devices consume minimal physical disk storage, as they store only the address pointers to the data stored on the source device or a pool of save devices.




Symmetrix Device TypesSave Devices Special devices (not mapped to the host) that provide physical storage space for pre-

update images or changed tracks during a virtual copy session of TimeFinder/Snap and SRDF/A DSE operations

Save devices are a predefined pool of storage devices and must be configured for this purpose.

Save devices are assigned a Symmetrix device number and can be unprotected, mirrored, or parity RAID

Device Masking

(VCM) Devices)

The device masking database (VCMDB) holds device masking records

Symmetrix devices that have been masked for visibility only to certain hosts.

DRV Devices A non-user-addressable Symmetrix device used by the Symmetrix Optimizer in logical volume swapping operations

Temporarily hold user data while reorganization of the devices is being executed




Data Protection Options for DMX3/4

Option Characteristics Protection Performance Cost

RAID 1 Write to two separate physical drives

Read from single drive – DMSP

Highest

High

Higher

Low Cost

RAID 5 Parity based protection

Striped data and parity– 3+1 and 7+1

Configurations

Fastest

Fast ReadGood Write

Fast ReadFair write

Lower Cost

RAID 6 Two parity drives– 6 + 2 and 14 + 2

Data Availability is primary

Performance is a secondary consideration

New with Enginuity 5772

Lowest Cost

Unprotected Not recommended

RAID 5 is based on the industry standard algorithm and can be configured with three data and one parity, or seven data and one parity. While the latter will provide more capacity per $, there is a greater performance impact when in degrade mode where a drive has failed and all surviving drives must be read in order to rebuild the missing data.

RAID 6 is new with 5772 and is focused on availability. With the new larger capacity disk drives, rebuilt times may take multiple days increasing the exposure to a second disk failure. Works nicely with tiered storage and where you have data that is written once and read occasionally.

Other data protection schemes include remote replication using SRDF.




Physical Disk and Hyper VolumesPhysical Disk

146 GB

10 GB

8 GB

9 GB

36 GB

6 GB

8 GB

11 GB

8 GB

Hyper Volumes

The software used to “split” physical disks into volumes is called Hyper Volume Extension. Symmetrix physical disks are split into logical Hyper Volumes. Hyper Volumes (disk slices) are then defined as Symmetrix Logical Volumes (SLV). SLVs are internally labeled with hexadecimal identifiers (0000-FFFF). The maximum number of host addressable logical volumes per Symmetrix configuration is 64,000 using Enginuity 5771+.

While “Hyper Volume” and “split” refer to the same thing (a portion of a Symmetrix physical disk), a “Symmetrix Logical Volume” is a slightly different concept. A Symmetrix Logical Volume is an abstraction of a disk drive that is presented to a host via a Symmetrix channel director port. As far as the host is concerned, the Symmetrix Logical Volume is a physical drive.

Symmetrix Logical Volumes are defined in the Symmetrix Configuration (BIN File). From the Symmetrix perspective, physical disk drives are partitioned into hyper volumes. A hyper volume could be used as an unprotected Symmetrix logical volume, a mirror of another hyper volume, a Business Continuance Volume (BCV), a member for RAID 5 or RAID 6 volume, a remote mirror using SRDF, and other uses.

Volume Table of Contents (VTOC) on disk are used to map logical volumes to physical disks. These data structures are created during initial installation.

Maximum hyper volumes per physical disk varies with software version - currently 255 maximumHyper volumes can be of variable size




Mirror Positions

Internally each Symmetrix Logical Volume is represented by four mirror positions – M1, M2, M3, M4 – Data structure

Each mirror positions represents a mirror copy of the volume data protection– Local Replication– Remote Replication– Dynamic Spare– Drive Relocation

Symmetrix Logical Volume

M2M1M1 M3 M4M4

Within the Symmetrix, each logical volume is represented by four mirror positions – M1, M2, M3, and M4. These Mirror Positions are actually data structures that point to a physical location of a data mirror and the status of each track of data. Each position either represents a mirror or is unused. For example, an unprotected volume will only use the M1 position to point to the only data copy. A RAID-1 protected volume will use the M1 and M2 positions. If this volume was also protected with SRDF, three mirror positions would be used, and if we add a BCV to this SRDF protected RAID-1 volume, all four mirror positions would be used.




Mirror Position Usage Examples

Mirror positions may “float”

M1 M2 M3 M4

Unprotected

Raid 1

RAID 5

RAID 6 Data Not Used Not Used Not Used

RAID 1 + BCV

RAID 5 + BCV

RAID 1 + BCV+ SRDF

RAID1 with Dynamic Spare Invoked

Data Not Used Not Used Not Used

Data Data Not Used Not Used

Data Parity Not Used Not Used

Data Data BCV Not Used

Data Parity BCV Not Used

Data Data BCV RDF

Data Data Dynamic Spare

Not Used




Mirroring: RAID-1

Two physical “copies” of the same data– Different disk– Different back-end director Physical

DriveLV 04B M1

Physical Drive

LV 04B M2

Logical Volume 04B

Host WriteData Slot

Disk Director

Disk Director

Disk Director

Disk Director

Cache

Mirroring provides applications both high performance and availability. Mirroring maintains a duplicate copy of a logical volume on two physical drives. The Symmetrix maintains these copies internally by writing all modified data to both physical locations. The mirroring function is transparent to attached hosts, as the hosts view the mirrored pair of hypers as a single logical volume.

Key points about RAID-1 Mirroring:Identical data is stored on a redundant mirrored diskMirrored devices by default will reside on different back-end directorsShould either disk fail, the other disk provides continuous availabilityTransparent to the host systemSymmetrix will call home if one of the mirrored pair failsAll data on the good mirror is copied to replacement driveFor read intensive applications, mirroring provides even higher performance than non-mirrored disk because there are two possible source for the read and the Dynamic Mirror Service Policy (DMSP) will choose the copy that provides the best performance

Mirroring is the best form of data protection available, but it is also the most expensive of all protection options (50% reduction in total storage capacity).




Physical Drive Physical Drive

Logical Volume 000

Logical Volume004

Logical Volume 008

Logical Volume 00C

LV 000 M1

LV 004 M2

LV 008 M1

LV 00C M2 LV 00C M1

LV 008 M2

LV 004M1

LV 000 M2

Mirror Service PoliciesInterleave Service Policy

Split Service Policy

Dynamic Mirror Service Policy (DMSP)

During a read operation, the Mirror Service Policy choose the best hyper in the mirrored pair based upon three service policies:

Interleave Service Policy - Shares the read operations of the mirrored pair by reading tracks from both logical hypers in an alternating method: a number of tracks from the primary volume (M1) and a number of tracks from the secondary volume (M2). The interleave policy is designed to achieve maximum throughput.Split Service Policy - Read operations are assigned to either the M1 or the M2 logicalvolume, but not to both. Split is designed to minimize head movement.Dynamic Mirror Service policy (DMSP) - Utilizes both Interleave and split for maximum throughput and minimal head movement. Dynamic Mirror Service policy adjusts the policy for each logical volume dynamically, based on access patterns detected. This is the default mode within the Enginuity operating environment.




Dynamic Mirror Service Policy (DMSP)

Determines which disk in a set of mirrored disks should service a read request such that:– Workload balancing is optimized– Spindle Seek times are minimized– Statistically, a disk will seek no more than ½ it’s maximum

DMSP collects samples of logical volume activity and stores them in the Enginuity SFS.– Uses this data to make informed performance decisions

The Symmetrix uses one of the three policies when reading from mirrored devices. The M1 and M2 policies tend to be better for random I/O workloads. By their nature, random workloads tend to spread the traffic over many devices, so even with the M1 or M2 policy, many disks are working simultaneously to retrieve data. The Interleave policy tends to be better for sequential I/O workloads. The interleaving allows both mirrors to service reads at the same time.

Each device can be permanently locked into using one of the policies, or the Dynamic Mirror Service Policy (DMSP) can be enabled. With DMSP enabled, the Symmetrix analyzes the workload on the DAs, drive interfaces, and drives and runs simulations to predict how each policy would change the situation. It chooses the policy that best balances the workload and also minimizes the disk latency and seek times for each device. After a short delay, the process of choosing a policy is repeated to keep up with dynamically changing workloads.

The DMSP algorithm optimizes READ performance in two ways:

1. DMSP balances the load among the physical drives, so that even when the workload is skewed, the loads on the physical drives are as balanced as possible. For example, when a LUN is very busy, the DMSP will use both mirrors to service the read operations of this LUN.

2. DMSP minimizes the seek operations on the disks. Whenever the load balancing allows it, each disk reads from one half of its platters: either from the outermost cylinders, or from the innermost cylinders. Because of this, the seek distances are significantly shortened. Seek minimization is a critical optimization because, as disk capacity increased over the years, the data transfer time was improved much more significantly than seek time.




RAID 1 Layout

Full Mirror

Best Overall Performance

Best AvailabilityDisk D1 D2 D3 D4 D5 D6 D7 D8DA

Track0123456789AB… … … … … … … … …

16b15a01a02b02a16a15b01b

DataDataDataDataDataDataDataDataDataDataData

Data’ Data’Data’Data’

Data

Data’Data’

Data’

Data’Data’

Data

Data’Data’

Data’

Data’

Data’

Data’

DataData DataDataDataData

Data’

DataDataData

Data

Data

Data’

DataData

Data’Data’ DataData’Data’Data’Data’

Data’

Data’Data’Data’

Data’

Data’

DataDataDataData

Data

DataDataData

DataData’

Data’Data’ Data’Data’Data’Data’

Data’

DataData

DataDataData

Data

Data’Data’Data’

Data’

Data’

DataDataDataDataData

Data’Data’Data’Data’Data’Data’Data

Volume 1 Volume 2 Volume 3 Volume 4

Two copies of each track is maintained on different disks.




RAID 5 Parity Protection

Parity is calculated using the Boolean Exclusive OR logic function

Truth Table:

Data: 111 Data: 101 Data: 001 Parity: 011

Bit 1 Bit 2 Result0 0 00 1 11 0 11 1 0

111 Data 101 Data 010 PartialXOR

010 Data 001 Data 011 ParityXOR

In the above example we are only using 3 bits to represent a block of data on disk, however, the XOR works the same way whether 3 bits or 512.

The advantage of RAID 5 is more of the disk space is available for data storage. With a 4+1 configuration, 75% of the available disk space is used to store data and only 25% is used for data protection. When compared with RAID1, where 50% of the available storage is used for data protection, you can see that this is a cost effective approach. Effective utilization is even higher when using 7+1 configuration where 87.5% of available space is used for data storage.




Data Reconstruction

If a track or whole disk is lost, data can be recovered through parity calculationsIf the disk containing “001” is lost, the parity is XOR’ed with the surviving members to reconstruct the missing data

Data: 111 Data: 101 Data: 001 Parity: 011X

111 Data 101 Data 010 PartialXOR

010 Data 011 Parity 001 Reconstructed DataXOR

During normal read operations, the performance of RAID 5 is comparable to RAID1. However during degraded mode, where one member drive has failed, performance is impacted as all surviving members must be read in order to reconstruct the data on a failed drive. The performance impact of a drive failure is even greater with a 7+1 configuration.




RAID 5 Layout

3 + 1 Striped Parity and DataDisk D1 D2 D3 D4 D5 D6 D7 D8DA

2/6 2/A 2/E P 2/6 2/A 2/E P

… … … … … … … …

2/7 2/B 3/0 P 2/7 2/B 3/0 P2/8 2/C 3/1 P 2/8 2/C 3/1 P2/9 2/D 3/2 P 2/9 2/D 3/2 P

16b15a01a02b02a16a15b01b

PPPP

0/C0/D0/E1/01/91/A1/B

0/0 0/80/10/2

0/4

0/3P

0/0

PP

P

P1/D

0/8

1/E

0/9

2/0

0/40/5 0/50/60/71/1

0/1

1/21/31/4

P

P

0/A

PP

0/90/A 0/60/B1/51/61/7

0/2

1/82/22/3

0/B

2/1

PP

0/C0/D

0/7

0/E1/01/9

P2/4

0/3P 1/5PPP

2/5

1/A1/B

1/11/21/3

1/C

1/D1/E2/0

1/6

2/1

1/4PPPP

1/71/82/22/32/42/51/C

Volume 1 Volume 1

RAID 5 device are made up of four or eight hyper volumes, each containing both data and rotating parity. The stripe width is the amount of data stored on a RAID 5 member. The Symmetrix uses a stripe width of 4 by default, which means 4 tracks are written to a member before writing to the next member. The Table above represents how data might be laid out.




RAID 5 Random Write

Host WriteData Slot

Cache

Parity Slot

XOR Data Write

XOR Old & New Data

XOR Parity Write

Parity

Data

Disk drives assist in parity calculation

The added step of XORing the old and new data, and XORing it with the old parity to create new parity, and then writing the new data and new parity is referred to as the RAID 5 write penalty.




RAID 5 Large Sequential Write

Parity

Host Write

Data Slot

Cache

Parity Slot

Write new Data

Data SlotData Slot

Data

Data

Data

Write new Data

Write new Data

Write new Parity

XOR in Cache

Parity calculation performed in cache

Large sequential writes have a performance advantage in that parity can be calculated in cache eliminating an XOR operation.




RAID 6

Similar to Raid 5 but two parity drives– Independent horizontal & diagonal parity– 6 + 2 and 14 + 2 configurations– Both FBA and CKD emulation supported

Data Availability is the primary motivation– All other protection schemes cause data lost in dual drive failures– Data lost if unrecoverable read error during rebuild– Larger capacity drives increase the exposure because of increased rebuild

time

Performance is a secondary– Response time– Rebuild times

Supported with Enginuity 5772

Data Data Data Data Data Data Data Data

RAID level 6 was not among the original raid levels. It adds an additional parity block to a RAID 5 array. It needs at least four disks (2 disks for the capacity, 2 disks for redundancy). Calculations are much more difficult than the simple XORs used with RAID 5. Like with RAID 5, parity and data are on different disks, for each block. The two parity blocks are also located on different disks.

This should be positioned as an availability solution. RAID 6 is slower than RAID 5, but it allows the RAID to continue with any two disks failed. The EMC implementation of RAID 6 yields performance is competitive with the industry. And is ideally suited for archival purposes where data is written once and read occasionally.




RAID 6 Layout

6+2 Striped data and parity - Dual ParityDisk D1 D2 D3 D4 D5 D6 D7 D8DA

DP 11/6 11/A 11/E 12/3 12/7 12/B P

16b15a01a02b02a16a15b01b

PPPP

1/91/A1/B1/C3/33/43/53/6

9/9

DPDPDPDP

1/5

PP

1/1

PP

0/C

3/73/8

0/8

3/93/A

0/4

9/D

0/00/1 1/60/20/3DP

1/2

DPDPDP

0/D

PPP

0/9

P

10/2

0/51/70/6

0/71/D1/E

1/3

2/02/1DPDP

0/E

DPDP

0/A0/B 1/82/22/32/42/53/B

1/4

3/C3/D3/E

10/6

0/12/62/72/82/9

2/E

4/04/14/2

10/A

2/A2/B2/C2/D4/44/5

4/3

10/E

3/03/13/24/8

4/64/7

P

4/94/A4/B

DP

Volume

This algorithm calculates two types of parity — Horizontal Parity (HP) and Diagonal Parity (DP), in order to provide protection against the loss of two members of the RAID group. The two parities are calculated using different sets of the user data, and are independent of one another. Horizontal Parity is the equivalent of RAID 5 parity, and is calculated from the data across all the data disks. Diagonal Parity (DP) is made up of segments; each DP segment is calculated on a selected group of data segments. Each DP segment skips a different data drive in its calculation, which is key to the ability of this algorithms to reconstruct data after a double drive failure. Another important requirement of this algorithm is that the number of data drives used in the DP calculations be a prime number. Symmetrix offers RAID 6 (6+2) and RAID 6 (14+2) protection (6 data or 14 data, with 2 parities each), neither of which is 'prime'. The Symmetrix implementation of Even-Odd assumes there to be 17 data drives in the RAID group -named D1-D17. For each of the protection schemes, the 'real' data drives take up positions D1-D6 (for 6+2) or D1-D14 (for 14+2) in the DP calculations; the remaining drives (Dx up to D17) are 'NULL disks'. These NULL disks exist only for the sake of the DP calculations, and the 'data' on these NULL disks is assumed to be all zero when calculating parity. These NULL disks do not consume any system resources (memory, cache, or disks).




RAID-10 CKD Volume

Mainframe CKD devices support RAID-10– RAID-10 - Mirrored Stripes– Each group consists of 8 hypers– Data is striped across four hypers and then mirrored

Cyl 17Cyl 13Cyl 9Cyl 5Cyl 1








M1 M2

RAID-10 Meta Volume

This is a diagram of a RAID-10 CKD volume. These are sometimes referred to as CKD Meta Volumes. The portion of the logical volume which resides on one physical volume is called a stripe. Each RAID-10 stripe group consists of four stripes distributed across four volumes. These are mirrored to consist of eight total hypers. The stripe group is constructed by alternately placing one cylinder across each of the four volumes. These volumes cannot be on the same disk director. The eight volumes are distributed across the Symmetrix back end for additional availability and improved performance.




Meta VolumesMaximum Symmetrix Logical Volume size is 65520 Cylinders– Multiple Logical Volumes can be combined to create Meta Volumes

CapacityPerformance

Logical Volume001

Logical Volume002

Logical Volume 003

Logical Volume 004

Meta Volume

LV 001

LV 002

LV 003

LV 004

Configured as a Meta Volume, appears to the host as a single 64 GB disk

Example: /dev/dsk/c1t1d0

Four individual Symmetrix16 GB logical volumes

Two or more Symmetrix Logical Volumes can be grouped into a Meta Volume configuration and presented to a host as a single disk. Meta Volumes allow customers to present larger Symmetrix logical volumes than the current maximum hyper volume size(65520 Cylinders) and satisfies requirements for environments where there is a limited number of host addresses or volume labels available. For example Windows only has 26 Drive Letters available.

With open systems Meta Volumes, data is either striped or concatenated.

Notes:An Open Systems Meta volume may have a maximum of 255 members (SLVs)Meta size up to 1.1 Terabyte (up to 7.5TB with RPQ)Members can be of different sizeDoes not have to be contiguous devicesMembers may be mirrored, RAID-5, or Parity RAID volumesMembers of a Meta Volume may also be configured for remote mirrors using SRDFDynamic Spares may be invoked for any member of a Meta Volume Refer to the following website for further Meta Volume information: http://www.cs.isus.emc.com/config/Configuration/Features/Meta/meta.htm




SYMCLI Logical Volume Info

Three commands to obtain a list of devices:– syminq

Only devices visible to hostUses SCSI Query commandDoes not use the SYMAPI Database

– sympdLists Symmetrix devices that are host-visibleObtains information from the SYMAPI database

– symdevDisplays information about all Symmetrix devices, including those that are not visible to the hostInformation obtained from SYMAPI database




SCSI Querysyminq uses standard SCSI query commands

Does not use read or update the SYMAPI database

C:>syminqDevice Product Device

-------------------------- --------------------------- ---------------------Name Type Vendor ID Rev Ser Num Cap (KB)-------------------------- --------------------------- ---------------------\\.\PHYSICALDRIVE1 EMC SYMMETRIX 5771 5400020080 919680\\.\PHYSICALDRIVE2 EMC SYMMETRIX 5771 5400021080 919680\\.\PHYSICALDRIVE3 EMC SYMMETRIX 5771 5400022080 919680\\.\PHYSICALDRIVE4 EMC SYMMETRIX 5771 5400023080 919680\\.\PHYSICALDRIVE5 EMC SYMMETRIX 5771 5400024080 919680\\.\PHYSICALDRIVE6 EMC SYMMETRIX 5771 5400025080 919680\\.\PHYSICALDRIVE7 EMC SYMMETRIX 5771 5400026080 919680\\.\PHYSICALDRIVE8 EMC SYMMETRIX 5771 5400027080 919680\\.\PHYSICALDRIVE9 GK EMC SYMMETRIX 5771 5400028080 9600\\.\PHYSICALDRIVE10 GK EMC SYMMETRIX 5771 5400029080 9600

C:>syminqDevice Product Device

-------------------------- --------------------------- ---------------------Name Type Vendor ID Rev Ser Num Cap (KB)-------------------------- --------------------------- ---------------------\\.\PHYSICALDRIVE1 EMC SYMMETRIX 5771 5400020080 919680\\.\PHYSICALDRIVE2 EMC SYMMETRIX 5771 5400021080 919680\\.\PHYSICALDRIVE3 EMC SYMMETRIX 5771 5400022080 919680\\.\PHYSICALDRIVE4 EMC SYMMETRIX 5771 5400023080 919680\\.\PHYSICALDRIVE5 EMC SYMMETRIX 5771 5400024080 919680\\.\PHYSICALDRIVE6 EMC SYMMETRIX 5771 5400025080 919680\\.\PHYSICALDRIVE7 EMC SYMMETRIX 5771 5400026080 919680\\.\PHYSICALDRIVE8 EMC SYMMETRIX 5771 5400027080 919680\\.\PHYSICALDRIVE9 GK EMC SYMMETRIX 5771 5400028080 9600\\.\PHYSICALDRIVE10 GK EMC SYMMETRIX 5771 5400029080 9600




Physical Device (sympd list)

C:\Program Files\EMC\SYMCLI\bin>sympd list


Device Name Directors Device

------------------------- ------------- -------------------------------Cap

Physical Sym SA :P DA :IT Config Attribute Sts (MB)

--------------------------- ------------- ---------------------------------

/dev/rdsk/c1t0d0s2 0000 02A:1 01C:C6 Unprotected N/Grp'd VCM WD 47

/dev/rdsk/c1t0d1s2 0040 02A:1 16A:D4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d2s2 0041 02A:1 01D:D4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d3s2 0042 02A:1 16B:C4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d4s2 0043 02A:1 01A:C4 Unprotected N/Grp'd RW 188

C:\Program Files\EMC\SYMCLI\bin>sympd list


Device Name Directors Device

------------------------- ------------- -------------------------------Cap

Physical Sym SA :P DA :IT Config Attribute Sts (MB)

--------------------------- ------------- ---------------------------------

/dev/rdsk/c1t0d0s2 0000 02A:1 01C:C6 Unprotected N/Grp'd VCM WD 47

/dev/rdsk/c1t0d1s2 0040 02A:1 16A:D4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d2s2 0041 02A:1 01D:D4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d3s2 0042 02A:1 16B:C4 Unprotected N/Grp'd RW 188

/dev/rdsk/c1t0d4s2 0043 02A:1 01A:C4 Unprotected N/Grp'd RW 188

sympd displays information about Symmetrix Logical Volumes that are visible to the attached host

Queries and update the SYMAPI database




Symmetrix Devices (symdev list)

C:\>symdev list


Device Name Directors Device----------------------- ------------- ---------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)----------------------- ------------- ----------------------------------

0020 \\.\PHYSICALDRIVE1 08A:0 16A:C7 2-Way Mir N/Grp'd RW 8980021 \\.\PHYSICALDRIVE2 08A:0 16B:C6 2-Way Mir N/Grp'd RW 8980022 \\.\PHYSICALDRIVE3 08A:0 16C:C5 2-Way Mir N/Grp'd RW 8980023 \\.\PHYSICALDRIVE4 08A:0 16D:C4 2-Way Mir N/Grp'd RW 8980024 \\.\PHYSICALDRIVE5 08A:0 01B:D4 2-Way Mir N/Grp'd RW 8980025 \\.\PHYSICALDRIVE6 08A:0 01A:D5 2-Way Mir N/Grp'd RW 8980026 \\.\PHYSICALDRIVE7 08A:0 01D:D4 2-Way Mir N/Grp'd RW 8980027 \\.\PHYSICALDRIVE8 08A:0 01C:D5 2-Way Mir N/Grp'd RW 8980028 \\.\PHYSICALDRIVE9 08A:0 16C:C7 2-Way Mir N/Grp'd RW 90029 \\.\PHYSICALDRIVE1008A:0 16D:C6 2-Way Mir N/Grp'd RW 9002C Not Visible ???:? 16C:D6 2-Way Mir N/Grp'd RW 8438…0033 Not Visible ???:? 01D:D0 2-Way Mir N/Grp'd RW 84380034 Not Visible ???:? 16B:C0 2-Way Mir N/Grp'd RW 84380035 Not Visible ???:? 16A:C1 2-Way Mir N/Grp'd RW 84380036 Not Visible ???:? 01A:C2 RAID-5 N/Grp'd (M) RW 450000037 Not Visible ???:? 16A:D2 RAID-5 N/Grp'd (m) RW -0038 Not Visible ???:? 16A:C3 RAID-5 N/Grp'd (m) RW -0039 Not Visible ???:? 01A:D1 RAID-5 N/Grp'd (m) RW -

C:\>symdev list


Device Name Directors Device----------------------- ------------- ---------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)----------------------- ------------- ----------------------------------

0020 \\.\PHYSICALDRIVE1 08A:0 16A:C7 2-Way Mir N/Grp'd RW 8980021 \\.\PHYSICALDRIVE2 08A:0 16B:C6 2-Way Mir N/Grp'd RW 8980022 \\.\PHYSICALDRIVE3 08A:0 16C:C5 2-Way Mir N/Grp'd RW 8980023 \\.\PHYSICALDRIVE4 08A:0 16D:C4 2-Way Mir N/Grp'd RW 8980024 \\.\PHYSICALDRIVE5 08A:0 01B:D4 2-Way Mir N/Grp'd RW 8980025 \\.\PHYSICALDRIVE6 08A:0 01A:D5 2-Way Mir N/Grp'd RW 8980026 \\.\PHYSICALDRIVE7 08A:0 01D:D4 2-Way Mir N/Grp'd RW 8980027 \\.\PHYSICALDRIVE8 08A:0 01C:D5 2-Way Mir N/Grp'd RW 8980028 \\.\PHYSICALDRIVE9 08A:0 16C:C7 2-Way Mir N/Grp'd RW 90029 \\.\PHYSICALDRIVE1008A:0 16D:C6 2-Way Mir N/Grp'd RW 9002C Not Visible ???:? 16C:D6 2-Way Mir N/Grp'd RW 8438…0033 Not Visible ???:? 01D:D0 2-Way Mir N/Grp'd RW 84380034 Not Visible ???:? 16B:C0 2-Way Mir N/Grp'd RW 84380035 Not Visible ???:? 16A:C1 2-Way Mir N/Grp'd RW 84380036 Not Visible ???:? 01A:C2 RAID-5 N/Grp'd (M) RW 450000037 Not Visible ???:? 16A:D2 RAID-5 N/Grp'd (m) RW -0038 Not Visible ???:? 16A:C3 RAID-5 N/Grp'd (m) RW -0039 Not Visible ???:? 01A:D1 RAID-5 N/Grp'd (m) RW -

The symdev command displays information about Symmetrix Logical Volumes. In this example the volumes are not presented to a front end directors so *** are displayed for the SA:P column and they are not visible to the local host.

The symdev command provides similar output as the sympd command, but includes all Symmetrix devices and lists them by Symmetrix device names for all devices on all or a given Symmetrix array(s). The corresponding output shows the Symmetrix device names (Sym), physical device (Physical) names, director information, and device-specific information for all devices on the Symmetrix array. Note: The physical device names are not known for those devices that are not visible to the host making the request. A value of ???:? for the SA means there is no mapping to a front-end director port. A value of ***:* means there are multiple mappings.




Device Details (symdev show)

C:\Program Files\EMC\SYMCLI\bin> symdev show 04a


Device Physical Name : Not Visible

Device Symmetrix Name : 004ADevice Serial ID : N/ASymmetrix ID : 000190100172

Attached BCV Device : N/A

Attached VDEV TGT Device : N/A

Vendor ID : EMCProduct ID : SYMMETRIXProduct Revision : 5771Device WWN : 60060480000190100172533030303441Device Emulation Type : FBADevice Defined Label Type: N/ADevice Defined Label : N/ADevice Sub System Id : 0x0001

Device Block Size : 512

C:\Program Files\EMC\SYMCLI\bin> symdev show 04a


Device Physical Name : Not Visible

Device Symmetrix Name : 004ADevice Serial ID : N/ASymmetrix ID : 000190100172

Attached BCV Device : N/A

Attached VDEV TGT Device : N/A

Vendor ID : EMCProduct ID : SYMMETRIXProduct Revision : 5771Device WWN : 60060480000190100172533030303441Device Emulation Type : FBADevice Defined Label Type: N/ADevice Defined Label : N/ADevice Sub System Id : 0x0001

Device Block Size : 512

The symdev show command provides details on a specified volume. The output is continued on the following page.




symdev show (cont.)Device Capacity

{Cylinders : 12280Tracks : 184200512-byte Blocks : 11788800MegaBytes : 5756KiloBytes : 5894400}

Device Configuration : RAID-5 (Non-Exclusive Access)Device is WORM Enabled : NoDevice is WORM Protected : NoSCSI-3 Persistent Reserve: DisabledDynamic Spare Invoked : NoDynamic RDF Capability : NoneDevice Service State : NormalDevice Status : Ready (RW)Device SA Status : Ready (RW)

Front Director Paths (4):-------------------------------------------------------------------

POWERPATH DIRECTOR PORT LUN--------- ---------- ---- -------- ---------

PdevName Type Type Num Sts VBUS TID SYMM Host----------------------------------------------------------------------Not Visible N/A FA 07A:0 RW 000 00 00A N/ANot Visible N/A FA 08A:0 RW 000 00 00A N/ANot Visible N/A FA 09A:0 RW 000 00 00A N/ANot Visible N/A FA 10A:0 RW 000 00 00A N/A }

Device Capacity{Cylinders : 12280Tracks : 184200512-byte Blocks : 11788800MegaBytes : 5756KiloBytes : 5894400}

Device Configuration : RAID-5 (Non-Exclusive Access)Device is WORM Enabled : NoDevice is WORM Protected : NoSCSI-3 Persistent Reserve: DisabledDynamic Spare Invoked : NoDynamic RDF Capability : NoneDevice Service State : NormalDevice Status : Ready (RW)Device SA Status : Ready (RW)

Front Director Paths (4):-------------------------------------------------------------------

POWERPATH DIRECTOR PORT LUN--------- ---------- ---- -------- ---------

PdevName Type Type Num Sts VBUS TID SYMM Host----------------------------------------------------------------------Not Visible N/A FA 07A:0 RW 000 00 00A N/ANot Visible N/A FA 08A:0 RW 000 00 00A N/ANot Visible N/A FA 09A:0 RW 000 00 00A N/ANot Visible N/A FA 10A:0 RW 000 00 00A N/A }

This is a continuation of the symdev show command. Here we see the capacity and other configuration information as well as the front-end directors that it is presented to. In this case we see it is presented to four different front end directors. Because the local host is not configured to “see” the volume, the PdevName shows as Not Visible. The out put is continued on the next page.

Beginning with Enginuity 5771, track sizes for FBA devices has been increased from 32K to 64K. To make comparing device sizes easier between Enginuity 56xx and 5771 architectures, SYMCLI will present size information in the Enginuity 56xx architecture format, 32K track size, by default. This default can be changed by enabling (default) or disabling the following value in the options file: SYMAPI_TRACK_SIZE_32K_COMPATIBLE = ENABLE | DISABLE

Note: In this example this environment variable was NOT set.




symdev show (cont.)Mirror Set Type : [RAID-5,RAID-5,N/A,N/A]

Mirror Set DA Status : [RW,RW,N/A,N/A]

Mirror Set Inv. Tracks : [0,0,0,0]

Back End Disk Director Information{Hyper Type : RAID-5Hyper Status : Ready (RW)Disk [Director, Interface, TID] : [N/A,N/A,N/A]Disk Director Volume Number : N/AHyper Number : N/A

Hyper Type : RAID-5Hyper Status : Ready (RW)Disk [Director, Interface, TID] : [N/A,N/A,N/A]Disk Director Volume Number : N/AHyper Number : N/A}

Mirror Set Type : [RAID-5,RAID-5,N/A,N/A]

Mirror Set DA Status : [RW,RW,N/A,N/A]

Mirror Set Inv. Tracks : [0,0,0,0]

Back End Disk Director Information{Hyper Type : RAID-5Hyper Status : Ready (RW)Disk [Director, Interface, TID] : [N/A,N/A,N/A]Disk Director Volume Number : N/AHyper Number : N/A

Hyper Type : RAID-5Hyper Status : Ready (RW)Disk [Director, Interface, TID] : [N/A,N/A,N/A]Disk Director Volume Number : N/AHyper Number : N/A}

This is a continuation of the symdev show command. Here we are looking at the mirror positions, and invalid tracks. The Back end director information for RAID 5 volumes shows as N/A. The back-end for RAID 5 is shown on the next page.




symdev show (cont.)RAID-5 Device Information

{Number of Tracks in a Stripe : 4Overall Ready State of RAID-5 Device : ReadyNoSpareOverall WriteProtect State of RAID-5 Device : EnabledNoSpareMember Number of the Failing Device : NoneMirror Number of the Failing Device : NoneMirror Number of the Data Device : 0Member Number that Invoked the Spare : NoneMirror Number of the Spare Member : NoneDisk Director (DA) that Owns the Spare : NoneCopy Direction : N/ARAID-5 Hyper Devices (3+1):

{Device : 004A

{-----------------------------------------------------------Disk DA Hyper Member Spare DiskDA :IT Vol# Num Cap(MB) Num Status Status Grp# Cap(MB)-----------------------------------------------------------02A:C1 2 2 1921 2 RW N/A 3 14001416A:Cf 5 1 1921 4 RW N/A 5 14001401B:Cf 5 1 1921 1 RW N/A 5 14001415B:C1 2 2 1921 3 RW N/A 3 140014

RAID-5 Device Information{Number of Tracks in a Stripe : 4Overall Ready State of RAID-5 Device : ReadyNoSpareOverall WriteProtect State of RAID-5 Device : EnabledNoSpareMember Number of the Failing Device : NoneMirror Number of the Failing Device : NoneMirror Number of the Data Device : 0Member Number that Invoked the Spare : NoneMirror Number of the Spare Member : NoneDisk Director (DA) that Owns the Spare : NoneCopy Direction : N/ARAID-5 Hyper Devices (3+1):

{Device : 004A

{-----------------------------------------------------------Disk DA Hyper Member Spare DiskDA :IT Vol# Num Cap(MB) Num Status Status Grp# Cap(MB)-----------------------------------------------------------02A:C1 2 2 1921 2 RW N/A 3 14001416A:Cf 5 1 1921 4 RW N/A 5 14001401B:Cf 5 1 1921 1 RW N/A 5 14001415B:C1 2 2 1921 3 RW N/A 3 140014

This is a continuation of the symdev show command. Here we are looking at the RAID5 specific configuration Note there are 4 hyper volumes associated with the one RAID5 volume.




Device Status and Service State

Device status:– Ready– Not Ready – Not available for host read/write operations– Write Disabled

Can be changed using the symdev command

Device service states:– Normal– Failed– Degraded – one or more mirror positions to the protected device is

Not Ready but the device is still available to the host for read/write

The service state of interest can be easily queried:symdev list –service_state notnormal

The symdev list and symdev list pd commands provides options that allows the returned data to be limited to a specific service state (-service_state option) . Service states can be degraded, failed, or normal, or all service states except one by preceding the service state value with a not, such as -service_state notfailed.




Device GroupsManagement and control operations are typically performed on a multiple devices in a single operations– Host Operating System volume group– Application– Dataset

Two types of Device Groups – Device Groups (DG)

Single Symmetrix array Single RDF (RA) group.

– Composite Groups (CG)One or more locally-attached Symmetrix arrays One or more RDF (RA) groups within a Symmetrix

By default DG information is stored in a hosts SYMAPI DB file– Group Name Services (GNS) Provides the ability to maintain group

definitions in a shared repository located on the Symmetrix

Device GroupDevice Group

A collection of devices can be assigned to a named group to provide a more manageable object to query status and impart control operations. Groups can be used to identify and work with a subset of available Symmetrix devices, obtain configuration, status, and performance statistics on a collection of related devices, or issue control operations that apply to all devices in the specified device group. You can use device groups to identify and work with a subset of available Symmetrix devices, obtain configuration, status, and performance statistics on a collection of related devices, or issue control operations that apply to all devices in the specified device group.

Two types of Device Groups:Device Groups (DG)− User-defined group comprised of devices that belong to a single Symmetrix array and a single RDF (RA)

group. − A control operation can be performed on the group as a whole, or on the individual device pairs that

comprise it. − By default, a device cannot belong to more than one device group and all of the STD devices in a group

must reside on the same Symmetrix array. − If the Symmetrix options file parameter SYMAPI_ALLOW_DEV_IN_MULT_GRPS is enabled, a device

can be added to multiple groups.Composite Groups (CG)− User-defined group comprised of devices that can belong to one or more locally-attached Symmetrix

arrays − One or more RDF (RA) groups within a Symmetrix.− An RDF consistency group is a CG comprised of RDF devices enabled for RDF consistency. The RDF

consistency group acts in unison to preserve dependent write consistency of a database distributed across multiple SRDF systems. It maintains this consistency via PowerPath or Multisession Consistency (MSC), which respects the logical relationships between dependant I/O cycles.




Device Group Example

Create a Device Groupsymdg create –type <Regular|RDF1|RDF2> <DgName>

Add Device using Symmetrix Device namesymld –g <DgName> add dev <SymDevName>

Add Device using Physical Device Namesymld –g <DgName> add pd <PDevName>

Associate Gatekeepers with Device Groupsymgate -g prod associate pd c2t0d2

Display Device Groupsymld –g <DgName> listsymdg show <DgName>

The symdg and symld commands group Symmetrix devices for status, monitoring and control purposes. A device group can be set up to contain all devices used by a particular host or that are used in a particular application.

Device groups, as well as the devices in a device group, are assigned names that facilitate reference in a session. You assign a device group name at the time you create it. The name can have up to 31 characters and must be unique for a given configuration database.

When you add a device to a device group, it is given a logical name. This name allows you to refer to the device independently of its physical device name or Symmetrix device name. The name can have up to 31 characters and must be unique within the device group. It is known only within the context of the device group to which the device belongs.

Three types of device groups: REGULAR(non-RDF)RDF1 (RDF source deviceRDF2 (RDF target device).

When you create a device group in an RDF configuration, you must specify the type of the device group (RDF1or RDF2). Otherwise, the group type defaults to REGULAR when no type is specified. The following device lists can be maintained in device groups:

SRDF device listTimeFinder/BCV device listTimeFinder/Snap virtual device listTimeFinder/Clone target listGatekeeper device list




Module SummaryKey points to remember:

A Symmetrix device is a logical emulation of a physical disk– FBA architecture for Open Systems– CKD for mainframe systems

A Symmetrix device has multiple identifiers that are used depending on the perspective

– Host– Symmetrix– Solutions Enabler

The DMX supports three data protection schemes:– RAID 1 Mirroring– RAID 5 Parity based protection– RAID 6 Dual parity protection

There are multiple ways of approaching the Symmetrix configuration– From the physical disk looking at the logical device configuration– From the logical device looking the physical storage




Reference Documentation

PowerLink




Closing Slide


Configuring Symmetrix Devices - 1



Module 3:Configuring Symmetrix Devices

Welcome to the third module of Symmetrix Monitoring and Management In this module we will examine the symconfigure command and how it can be used to create new mirrored, RAID5, RAID6, and meta devices.




Configuring Symmetrix Devices Objective:

Describe how the symconfigure command is used to perform on-line configuration changes

Describe the purpose and structure of the command file

Use the appropriate command to determine if there is available free disk space to create new devices

Create new mirrored, RAID 5, and RAID 6 devices

Describe the concept of Meta DevicesUse the symconfigure command to form a meta device and add members

Configure hot spare drives

Query the state of configuration changes




Device Configuration OverviewSymmetrix devices are configured a number of different ways– When the Symmetrix IMPL.bin file is first created

a number of devices are initially configured– Modify the bin file and perform on-line

configuration change from service processor– EMC ControlCenter Configuration Manager– Symmetrix Management Console– Solutions Enabler symconfigure command

Executed from management hostCreate, convert, modify, and delete nearly all types of Symmetrix devicesMap, unmap, and mask devices to front-end portsAdd or remove drives to the spare poolConfigure BCV and SRDF devices and other systems attributes

While historically, all configuration changes were accomplished by EMC Engineers modifying the IMPL.bin file and performing an off-line/online configuration change, for quite sometime now, configuration changes can be performed by the customer. This capability has greatly reduced the time to provision new capacity and has resulted in greater efficiency as customers can provision the appropriate size, protection, and other attributes when they need it.




symconfigure ArchitectureConfiguration commands invoked on a management host actually are executed as procedures on the service processor– Symmetrix Configuration Server– Initiates change session– Service processor performs configuration change on-line

Change sessions invokes an external configuration lock, allowing only a single change session at a time

Device locks are also placed on specificdevices during configuration operations– Prevents control operations (TF &SRDF)– To determine if a lock is currently heldsymdev list –lockn 15

GK

The Symmetrix configuration server engages a configuration lock known as lock 15 or lock F on the Symmetrix during the changes session, effectively blocking others from attempting to change the configuration. The lock is released at the end of the session or if the operator chooses to abort the operation. Information about a lock including how long it has been held can be viewed using the command: symcfg list –lock15.

Symmetrix device level locks are also used to prevent control operations such as TimeFinder establish or split operations from being performed on a device that is being modified by a configuration session. Device locks are engaged by the SYMAPI during the preview stage. Device locks are retained until the configuration change session has completed and the change has been committed on the Symmetrix.

SYMCLI provides the ability to release a lock on the Symmetrix. This is not a recommended procedure and is only useful for locks which have been confirmed as stranded. To release a stranded configuration lock:

symcfg –lockn 15 release

Note: It is rarely necessary to perform this operation as the lock is associated with a change session. Releasing a lock in the middle of an active session will cause problems with the processing of the change.




Symmetrix

FAsyscalls

RA

Architecture Details

HostSYMCLI ECC

SYMAPI

SIL FC

Service Processor

SYMWIN

SYMWINScripts

RA

Symmetrix

SYMWIN

Service Processor

SYMWINScripts

Ethernet Ethernet

The Symmetrix Configuration Manager deploys a a client-server architecture where the SYMAPI on the management host is the client and SYMWIN on the service processor is the server.

Note the communication paths also include remote Symmetrix if part of a SRDF environment.




Prerequisites

Requires Configuration Manager License key– Installed using symlmf

Verify that the current Symmetrix environment supports host-initiated configuration change

C:\>symconfigure verify

A Configuration Change Verification is in progress. Please wait...

Establishing a configuration change session...............Established.

Verifying configuration...................................Verified.

Terminating the configuration change session..............Done.

The configuration verification session has succeeded.

C:\>symconfigure verify

A Configuration Change Verification is in progress. Please wait...

Establishing a configuration change session...............Established.

Verifying configuration...................................Verified.

Terminating the configuration change session..............Done.

The configuration verification session has succeeded.

Before attempting to perform a configuration change using Solutions Enabler, execute the symconfigure verify command to verify the feature is licensed, there is a communications path to the Symmetrix, the service processor is on-line, and there are no other configuration sessions active.




Configuration Change Session

Configuration changes progress through three stages– Preview

Verifies syntax and command structure– Prepare

Runs preview checks and verifies appropriateness of configuration change against current configuration

– CommitPerforms preview, prepare checks and then executes the change on the Symmetrix

Stages can be executed individually

Preview

Prepare

Commit

Configuration changes progress through three stages: preview, prepare, commit. While it is possible to execute the commit directly and have it progress through the preview and prepared. It is often appropriate to create the change session, preview and prepare to verify that the change is appropriate and then commit the change later during a scheduled maintenance window where there is less activity on the Symmetrix.




Command Files Define Configuration ChangesCommand file provides specific details of configuration change to be processed

Multiple operations can be defined in a single file– Executed sequentially

Example:symconfigure commit –file changes.cmd

In Unix, may redirect standard insymconfigure commit <<EOFcreate dev…;EOF

#Sample commandfile

create dev…;

convert dev…;

form meta …;

map dev …;

create spare…;

delete dev …;

Example: changes.cmd

Command files are used to define the operations to be performed during a the configuration session. This file is created using an editor such as vi or notepad. Multiple operations can defined in a single file with each operation delimitated with a semicolon

Optionally, for UNIX, you can redirect the command as standard input and not create a command file.




Before Creating New Devices

Determine the available space on the physical disksC:\>symdev list -da all -space


DA Capacity (MegaBytes) Details------------- -------------------------------------- ------------Ident Int TID Total Configured Unconfigured Hypers Forma----- --- --- ------------ ------------ ------------ ------ -----

01A C 0 286102 13558 272535 2 FBA01A C 2 286102 8874 277227 2 FBA01A C 4 286102 9510 276583 2 FBA01A C 6 286102 4652 281445 2 FBA01A D 1 286102 8874 277227 2 FBA01A D 3 286102 8874 277227 2 FBA…16D C 2 286102 8874 277227 2 FBA16D C 4 286102 4652 281445 2 FBA16D C 6 286102 3763 282335 2 FBA16D D 1 286102 8874 277227 2 FBA16D D 3 286102 8874 277227 2 FBA16D D 5 286102 898 285200 1 FBA16D D 7 286102 8438 277655 1 FBA

Total ------------ ------------ ------------ ------ -----18310546 531464 17778747 120 FBA

C:\>symdev list -da all -space


DA Capacity (MegaBytes) Details------------- -------------------------------------- ------------Ident Int TID Total Configured Unconfigured Hypers Forma----- --- --- ------------ ------------ ------------ ------ -----

01A C 0 286102 13558 272535 2 FBA01A C 2 286102 8874 277227 2 FBA01A C 4 286102 9510 276583 2 FBA01A C 6 286102 4652 281445 2 FBA01A D 1 286102 8874 277227 2 FBA01A D 3 286102 8874 277227 2 FBA…16D C 2 286102 8874 277227 2 FBA16D C 4 286102 4652 281445 2 FBA16D C 6 286102 3763 282335 2 FBA16D D 1 286102 8874 277227 2 FBA16D D 3 286102 8874 277227 2 FBA16D D 5 286102 898 285200 1 FBA16D D 7 286102 8438 277655 1 FBA

Total ------------ ------------ ------------ ------ -----18310546 531464 17778747 120 FBA

There are a number of ways to determine the amount of free disk space that is available for use in configuring new devices. symdev list –da all –space above displays configured and unconfigured (free) disk space on a disk by disk basis as well as a summary. symconfigure list -freespace provides a summary of available disk space.




Creating New DevicesCommand file includes:– create dev

– count= <number of devices>– size= <size in cylinders>

1Cyl = 1920 512-byte blocksor .98MB

– emulation=<emulation type>FBA, Celera FBA, VME512FBACKD-3380 or CKD-3390

– config=<Protection type>2-WAY-MirRAID 5

datamember_count-3 or 7

RAID 6datamember_count=6 or 14

– disk_group_number=

create dev

count=100,

size=9000,

emulation=FBA,

config=2-WAY-Mir,

disk_group=1;

Example command file:

Syntax for command file:Hatch marks “#” are used for commentsWhite space is ignoredEach statement must end with a semicolon “;”Commas “,” are optionalCommand file is case insensitiveWhile you can have multiple statements in a single command file, they must be in the appropriate order




Creating RAID 5 and RAID 6 Devices

Example command files

create dev

count=24,

size=16000,

emulation=CKD-3390,

config=RAID5,

datamember_count=3,

disk_group=1;

create dev

count=100,

size=35000,

emulation=FBA,

config=RAID6,

datamember_count=6,

disk_group=1;

RAID 5 and RAID 6 devices are configured in a similar way. Note, you must also specify the number of data members. For example in a RAID 5 3+1 configuration, there are three data members.




symconfigure Example

C:\>symconfigure -file new_devices.cmd -v commit

Establishing a configuration change session...............Established.Processing symmetrix 000190102254{create dev count=10, size=9000, emulation=FBA,

config=2-Way Mir, mvs_ssid=0000, disk_group=0;}Performing Access checks..................................Allowed.Checking Device Reservations..............................Allowed.Submitting configuration changes..........................SubmittedValidating configuration changes..........................Validated.New symdevs: 0041:004AInitiating PREPARE of configuration changes...............Queued.PREPARE requesting required resources.....................Obtained.Step 013 of 017 steps.....................................Executing.Local: PREPARE...........................................Done.Initiating COMMIT of configuration changes................Queued.COMMIT requesting required resources......................Obtained.Step 019 of 116 steps.....................................Executing.…Step 209 of 255 steps.....................................Executing.Step 209 of 255 steps.....................................Executing.Step 212 of 255 steps.....................................Executing.Step 212 of 255 steps.....................................Executing.Local: COMMIT............................................Done.Terminating the configuration change session..............Done.

The configuration change session has successfully completed.

C:\>symconfigure -file new_devices.cmd -v commit

Establishing a configuration change session...............Established.Processing symmetrix 000190102254{create dev count=10, size=9000, emulation=FBA,

config=2-Way Mir, mvs_ssid=0000, disk_group=0;}Performing Access checks..................................Allowed.Checking Device Reservations..............................Allowed.Submitting configuration changes..........................SubmittedValidating configuration changes..........................Validated.New symdevs: 0041:004AInitiating PREPARE of configuration changes...............Queued.PREPARE requesting required resources.....................Obtained.Step 013 of 017 steps.....................................Executing.Local: PREPARE...........................................Done.Initiating COMMIT of configuration changes................Queued.COMMIT requesting required resources......................Obtained.Step 019 of 116 steps.....................................Executing.…Step 209 of 255 steps.....................................Executing.Step 209 of 255 steps.....................................Executing.Step 212 of 255 steps.....................................Executing.Step 212 of 255 steps.....................................Executing.Local: COMMIT............................................Done.Terminating the configuration change session..............Done.

The configuration change session has successfully completed.

In the example above, the verbose option was used and the processing is shown in detail. Much of the output was removed to fit it on the slide. The processing details is also logged in the SYMAPI log file.




Device 001

Device 002

Device 003

Device 004

Meta Device

Dev 001

Dev 002

Dev 003

Dev 004

Meta Devices A.K.A Meta VolumesMaximum Symmetrix device size is 65520 Cylinders– Multiple devices can be combined to create Meta devices– Presented to host as single device

CapacityPerformance

Configured as a Meta Volume, appears to the host as a single

64 GB diskFour individual Symmetrix

16 GB logical volumes

Two or more Symmetrix devices can be grouped to form a meta device configuration and presented to a host as a single disk. Meta devices allow customers to present larger Symmetrix devices than the current maximum hyper volume size (65520 Cylinders).

With Meta devices, data is organized either striped or concatenated.

Notes:An Open Systems Meta volume may have a maximum of 255 membersMeta size up to 3.825 Terabyte – Larger with RPQMembers can be of different sizeDoes not have to be contiguous devicesMembers may be mirrored, RAID-5, or RAID-6Members of a Meta Volume may also be configured for remote mirrors using SRDFDynamic Spares may be invoked for any member of a Meta Volume




Meta Device Data Layout

Meta Device organization determines how a host accesses information– Concatenated

– StripedPotential increased performance for sequential writesDefault and recommended stripe size is 2 cylinders

Meta Head Meta Member Meta Member Meta Tail

Meta Head Meta Member Meta Member Meta Tail

A Meta Device is two or more Symmetrix volumes presented to a host as a single addressable device. It consists of a meta head device, some number of member devices, and a meta tail device. The meta head is the first device in the meta device and receives commands from the host. When a incoming command is processed, Enginuity software determines which meta device should execute the command. With Open Systems, Meta device data is either striped or concatenated.

Concatenated devices are organized with the first byte at the beginning of the first device. Addressing continues to the end of the first device before data on the next device is referenced. When writing to a concatenated device, the first meta device member receives all the data until it is full, then data is directed to the next member and so on.

Striping divides each meta device into a series of stripes, addressing a stripe from each device before advancing to the next stripe on the first device. When writing to a striped volume, equal size stripes of data from each member are written alternately to each member of the set. Striping benefits sequential writes by avoiding stacking multiple writes to a single spindle and disk director, This scheme creates large volumes, but additionally spreads the workload across multiple disk drives and directors on the back-end and can potentially have very good performance impact.




Creating Meta DevicesCreate member devices as usual– Mirrored– RAID-5– RAID-6

Form the meta head and add members– Two Approaches:

Specify a countForm Meta head and add membersas separate operations

– Specify config=ConcatenatedStriped

Stripe_size12920 (512) byte blocks2 Cylinders

form metafrom dev 0c9config=concatenatedcount=4;

---or---

form metafrom dev 0DEconfig=stripedstripe_size=2 cyl;

add dev 0DF:0E1to meta 0DE;


You can form or dissolve a meta device, or add or remove members. To initially create a meta device, the head and members are specified in a command file. You can add or remove meta members from an existing concatenated meta devices or add members to a striped meta device. If using the count option, no members need to be specified. Members will automatically be selected from the pool of unmapped devices that match the size, emulation, and configuration type of the meta head.

Historically, stripe size can be expressed in blocks or cylinders. Today, the default and only supported value is (1920) 512-byte blocks or 2 Cylinders.

Restrictions:The member device must be unmapped (not visible to a host)The meta must contain at least one meta member. When a meta device is formed, at least one member must be added.To create a striped meta device, all members must be the same size and have the same type of protectionMeta devices must be composed of devices that are the same emulation typeMainframe meta devices are created using the create dev command, not the form meta command.

You can also dissolve a meta device which releases all it’s members making them available as regular devices.




Expanding Meta Devices

The capacity of a meta device can be expanded while the device is on-line

With concatenated meta device expansion, the new capacity is added to the endadd dev 113 to meta 0CE

With striped meta device expansion, the data is re-striped across all members– Protected – creates a BCV copy to preserve the dataadd dev 0113:0014 to meta 0CE,protect_data=true,bcv_meta_head=0DE;

– Non-protected – DATA IS LOSTprotect_date=False

Meta device can be reconfigured to add additional capacity while the device is online and available for host I/O. An administrator can:

Expand both concatenated and striped meta devicesConvert an unused device to a concatenate or striped meta deviceConvert a populated device to a concatenated or striped meta device

When performing a protected expansion, a BCV_meta must have been previously created and be the same size, configuration, and member count as the original meta device.




Converting Meta Devices

Convert from:– Concatenated to striped– Striped to concatenated– Example:convert meta 0c2, config=striped,protect_data=TRUE, bcv_meta_head=113;

Conversion can be done while the meta device is on-line

For protected operations, the BCV meta device must be identical to the original meta device in capacity, member count, and stripe size

Meta Devices can be reconfigured while the device is online to the host.




Removing Members and Dissolving Meta Devices

Removing a member from a meta device effectively makes the device smaller to the host– For concatenated meta devices:

Data is preserved up to where the member is removedHost must be able to handle this geometry change

– Removing a member of a striped meta device will result in data loss– Example: remove dev 113 from meta 0c2;

Dissolving a meta device– Data is preserved on the member devices but would be inaccessible

to the host– Example: dissolve meta dev 0ce;

While it is possible to reduce the size of a meta device by removing a member, what we are effectively doing is truncating the device. Many host systems will not be able to handle this type of change.




Managing Spare Pool

Physical drives can be reserved as hot spares and added to the spare pool– Disk cannot contain any hypers– Example: create spare count=2, format=512

To remove a hot spare and make it available for normal use– Can only remove a hot spare that is currently not invoked – To view existing hot spares:symdisk list –hotspare

– Specify the director, Interface, and disk id– Example delete spare disk=[16A,D,8]

The spare pool contains disk drives that can be invoked to either temporarily or permanently replace a failing or failed drive.




Managing Configuration Change Sessions

Configuration changes may take a while to execute– Depending on the activity on the Symmetrix– Extent of the changes

Session status can be monitored– Example: symconfigure query –i 10

Aborting a configuration is possible but not recommended– May not be possible if the session is passes a “point of no return”– Frees configuration session lock– Example: symconfigure –sid 254 abort

When config changes complete, update the SYMAPI DBsymcfg discover

Depending on the Symmetrix workload and the extend of the changes, Configuration change sessions can take quite some time to complete. Progress caqn be monitored using the query option.




SYMAPI Logsymconfigure progress is logged in the file:../emc/SYMAPI/log/symapiyyymmmddd.log

C:/>cd program files/emc/symapi/log

C:/>More symapi2070817.log

08/16/2007 06:32:58.546 2624 3336 EMC:SYMCONFIGURE iCfgChgSessionStart Starting a local CfgChg session for SID 000190102254, symapi V6.3-771 (0.0) ucode 5771

08/16/2007 06:33:02.031 2624 3336 Establishing session with Local cfg srvr (000190102254)...Established.

08/16/2007 06:33:08.687 2624 3336 Verifying configuration...................................Verified.

08/16/2007 06:33:10.921 2624 3336 Terminating session with configuration server.............Done.

08/16/2007 11:25:04.843 3748 176 EMC:SRVdaemon srvShowReadyMessage ANR0020I SYMAPI server listening on port 2707 over IPv4 only

…

C:/>cd program files/emc/symapi/log

C:/>More symapi2070817.log

08/16/2007 06:32:58.546 2624 3336 EMC:SYMCONFIGURE iCfgChgSessionStart Starting a local CfgChg session for SID 000190102254, symapi V6.3-771 (0.0) ucode 5771

08/16/2007 06:33:02.031 2624 3336 Establishing session with Local cfg srvr (000190102254)...Established.

08/16/2007 06:33:08.687 2624 3336 Verifying configuration...................................Verified.

08/16/2007 06:33:10.921 2624 3336 Terminating session with configuration server.............Done.

08/16/2007 11:25:04.843 3748 176 EMC:SRVdaemon srvShowReadyMessage ANR0020I SYMAPI server listening on port 2707 over IPv4 only

…

Details of a configuration change session is logged in the SYMAPI log.




Module SummaryKey points to remember:

Solutions Enabler provides a CLI that allow most Symmetrix configuration changes to be made on-lineThe symconfigure command uses a command file to define the operations to be performed in a session

Three stages of progression when processing a config change:– Preview– Prepare– Commit

Meta devices allow the creation of logical devices that are larger the largest Symmetrix device– Additional benefit is potential improved performance.





PowerLink




Closing Slide


Symmetrix Management Console - 1


Symmetrix Monitoring and ManagementUsing Solutions EnablerSymmetrix Monitoring and ManagementUsing Solutions Enabler

Module 4:Symmetrix Management Console

While this course is focused on using the Solutions Enabler CLI to monitor and manage the Symmetrix, it is appropriate that we briefly discuss the Symmetrix Management Console SMC) which is often described as a GUI front-end to the CLI.



© 2007 EMC Corporation. All rights reserved. Symmetrix Management Console - 2








All other trademarks used herein are the property of their respective owners.




Objectives

Upon completion of this module, you will be able to:

Describe the SMC Architecture

Install and configure the SMC Server

Using a supported web-browser, connect to the SMC console

Use SMC to monitor and manage a Symmetrix

SMC is designed to be self-documenting. Other than the Installation notes, and the on-line help text, there is no other documentation available (or required).




Symmetrix Management Console (SMC)

Device manager for the Symmetrix– Light-weight, web-based application– Intuitive browser interface

Monitor, configuration and operational control of one or more arrays– Supports all configuration capabilities

as Solutions Enabler CLI– Real-time reporting – No historical data– No SAN or Host monitoring and control– Manage one Symmetrix at a time

Bundled with EMC Control Center Symmetrix Manager

The Symmetrix Manager Console is a light-weight, web-based graphical user interface for performing Symmetrix monitoring and management tasks. Almost anything you can do with the CLI can be done using SMC GUI. SMC is supported on all Symmetrix starting with Enginuity 5568 and newer.

SMC is an independent application that includes a server that runs on a Linux or windows server. The client is a web-browser, and is therefore supported on a wide variety of client systems with network connectivity to the SMC server.

SMC GUI features closely match Solutions Enabler in both terminology and the information presented. Even die-hard CLU users will find the filtering features useful and the interface for performing common tasks such as configuring new devices is extremely intuitive.




SMC Management CapabilitiesDiscovery

Array and internals configLogical and physical devices

MonitoringPropertiesStatusAlerts

AdministrationUser-level security, logging, auditingAccess-control management

ConfigurationDevice creation, configuration, maskingSupports Solutions Enabler code for Open Systems and MainframeDynamic Cache PartitioningSymmetrix Priority Controls

ReplicationConfiguration and controlDiscovery of objects and statusSRDF monitoringCommand Line Generator

5

Symmetrix Management Console can be used to perform all the typical management and configuration tasks such as device creation and configuration, along with volume masking and front-end port configuration. In addition, SMC provides an easy to use interface for performing local and remote replication setup and control.




Client Server ArchitectureBrowser-based client

SMC Server operates within Solutions Enabler environment– Connectivity to the Symmetrix

Gatekeeper– Shares the SYMAPI DB– Feature Licenses

SMC License

Two deployment options:– SMC Server on the Solutions Enabler host– Stand-alone SMC server remotely communicates

to Solutions Enabler host using storapisrvdaemon

GK

SMC Server

SYMAPI DB

storsrvd

storapid

SYMAPI DB

SMC Server

storapid

The SMC Client runs on a supported web-browser with network connectivity to the SMC server.Internet Explorer 6.0+ and Firefox 1.0.5+ are supportedJava RTE

The SMC ServerWindows 2000 Server SP4+, Windows 2003 Standard or Enterprise EditionLinux Redhat 4, 32 bit or SUSE 9, 32 bitRequires Solutions enabler environmentRequires at least the SMC and Base License keys. Other and key for all SE functionality as appropriate




Installation PrerequisitesSMC Server requirements– Solutions Enabler V6.2+ must be installed on the same system as

the SMC ServerUse symlmf to enter SMC License keyThe storevntd (SYMAPI Event Daemon) is required for SMC to receive Symmetrix alerts

– Server host must have at least 512MB of memory for the SMC application

Client browser requirements– Javascript must be enabled– Pop-up blockers must be disabled

Or set to not block all pop-ups from the SMC server address– Java browser plug-in is required

Version 1.5 or above is highly recommended– Minimum browser resolution is 1024 x 768

SMC is designed to require a light footprint on both the Server and client systems.




Installation Process1. Click the SMC install executable

2. Select destination locations

3. Enter Super User Name and Connection TypeSuper User can create initial additional accountsLocal Symmetrix connection or Remote using storsrvd

4. Enter http and https ports (default 7070 and 8443)

5. Start Copying Files verification screen

6. Start Symmetrix Management Console Service query

7. Installation Complete screen, click Finish

Installation Process details are available in the Installation Guide, Chapter 1: Installing the Symmetrix Management Console.

Super User Name− The initial user that will have rights to do all things in SMC including creating additional

user accounts and setting permissions and LDAP-SSL−Default is user lowercase "smc" with password lowercase "smc"

SECURITY WARNING: After installation has completed, you are strongly advised to change this default password as a security precaution!

−Non-default user must be an authorized user on the SMC server hostConnection Type− Local is the default and requires no additional information−Remote provides a means to specify the remote symapisrv location

Node Name (or IP address)Net Port, defaults to 2707, the default port for symapisrv start

Must match port that symapisrv was started on

(4.)




Invoking the Client Browser

Open browser and point to hostname/IP address of SMC Server– https://<host>:port– Example:

https://localhost:8443

To start the SMC client, open a web browser, and point to the host name (SMC server) and the port entered (or accepted) during installation

If the installation defaults were accepted and running client local to the SMC Server, enter http://localhost:7070If the installation defaults were accepted, but running client remote from the SMC Server, enter http://<host>:7070If the port was changed from the default specify the <http_port> instead of 7070For a secure connection: https://<host>:<https_port>The first time the client browser is invoked there may be a lengthened display of a progress bar as the client is loaded.

At the Web Console login screen, enter the Super Username and PasswordEnter the Super User Username, default smc, or as specified during the installationEnter the Password, default smc If a correct Username/Password is entered, the SMC Web Console will appear

Reminder: pop-up blockers must be disabled or set to allow all pop-ups from the specified host in order to display the SMC Web Console




SMC Interface

(Properties) View (Details)(Properties) View (Details)

Navigation Tree

Menu BarMenu BarView BarView Bar

Menu BarContains multiple sub-menus: File, Control, Administration and HelpContains actions: Export, Logout, and Refresh View (also in File menu)Alert counter if clicked will select Alert View (also in View Bar)

Navigation TreeContains folders made up of sub-folders and objectsEach folder can be expanded or collapsedSelect a single object for monitoring and control operations Selecting a folder containing multiple objects is the only way to "select" multiple objects in the Navigation Tree

View BarToggle View Pane between different views: Properties, Config Session, Alerts, and Command History

(Properties) View Color-coded to match current View with the View Bar, e.g Blue for Properties View, as shown on this slideDisplay will depend on Tree Selection, View Bar selection, and Object selection within the View areaView can split horizontally 1 or 2 times to display additional detail




Symmetrix Properties

Similar to output of CLI commands– symcfg list -v

Note the similarities in the properties view of the Symmetrix with the output of the symcfg list –v command




Device Properties

Similar to output of CLI commands– symdev list & symdev show

The Properties view of devices is similar to the output of symdev list. When you select a specific device and display the properties, it has similar information as the symdev showcommand.




Create Device

Creates configuration session– Same parameters as symconfigure

Again, the SMC provides similar options as the CLI but in an intuitive GUI.




Task List

Configuration session added to task list– Actions: Preview, Commit, Deactivate

Configuration sessions are created and displayed in the Task list for execution when appropriate




Alerts View and Acknowledge

The Alerts View displays a list of alerts for the selected Symmetrix arraySeverity: the alert’s severity, as defined by SYMAPI.Object: the object to which the alert is related.Message: a description of the alert.Created: timestamp for when the alert happened.Last Modified: timestamp.Acknowledged By: a user name.Category: SYMAPI category.Code: SYMAPI error code.

Alerts will only appear if first enabled via the Administration menu.




Command History View

The Command History View displays a list of actions taken by the users using SMC.Command History displays for the selected Symmetrix arrayCommand History information is view-only, there are no actions




Device GroupsAll device groups in the local SYMAPI database or GNS (if active) are available in SMC

Additional Device Groups can be created

Add onedevice typeto the DG at a time

No logicaldevice namesupport

SMC Device Group creation is a multi-step processFirst, the Device Group is created and populated with devicesUsing the Create Device Group dialog box, add or remove devices to the group with the Add/Remove function− This dialog box can only accept changes to a device group one type at a time. To add

multiple types of devices to a device group, find and accept all devices of one type, by clicking OK

−Open the dialog box again, select a different type, and continue to find devices of that type




SMC Local User Accounts

The installation dialog includes the specification of the SMC Super User account– Default user "smc" with password "smc"

Recommended to create other Administrators in place of Super User– Can specify different Super User name in installation dialog

Menu: Administration Local User Accounts– List existing usernames in a table– Add new user information (name and password)– Edit user information (enter a new password)– Delete selected user account(s)

Accounts only apply to SMC and do not require any user credentials outside SMC, with the exception of an initial Super User and default "smc“ password.




Permissions and RolesTo Modify a Users permissions:– Menu: Administration Permissions– List Symmetrix ID, Username, and Role entries– Add new permission entry– Edit permission entry (change role)– Delete permission entry

Five Roles– Administrator - StorageAdmin and SecurityAdmin combined– StorageAdmin - full Symmetrix control– Monitor - view Symmetrix only, no manipulation of array– SecurityAdmin - add users and set permissions, no Symmetrix access– None - no access other than login (the default)

Super User – Created during installation– Only user with permission to set Symmetrix Preferences and LDAP-SSL– smc/smc username/password is a potential security hole if unchanged

Permissions must be explicitly added for any newly defined users, default permission is none.




SMC and SYMCLI DifferencesSMC and SYMCLI are very similar in functionality– Both have the same code base and provide the same feature set– Similar output and terminology

Symmetrix configuration is simpler in SMC– Pull down menus and context sensitive actions

Device filtering simplifies device grouping

SMC allows acknowledgement and tracking of alerts– No similar feature in CLI

SYMCLI operations default to online mode, always refreshing the latest Symmetrix data unless -offline is specified– SMC views do not automatically refresh when selected in the tree– SMC views refresh at the end of a control operation or upon explicitly

selecting Refresh View




Potential Common Issues

Installation Configuration Errors– Incorrect information for remote symapisrv

symapisrv not running at remote site– Incorrect HTTP or HTTPS port assignments– Super User name specified cannot be authenticated in Windows

Pop-up blockers preventing SMC Console from displaying

Firewall blocking ports used for HTTP, HTTPS or symapisrv

SMC not licensed

SMC is a simple product to install and operate, so there are few potential problems. The installation verifies key information, such as the presence of prerequisite software and minimum memory requirements.

Error message will clearly point out if one of these are the problems. For example, the message for the symapisrv not detected cannot tell if the host or port assignment is wrong or if symapisrv simply has not been started, but the message should be sufficient to lead the user to make the appropriate checks and to ensure it is running on the correct host and port.

Pop-up blockers will prevent the SMC Console from displaying. This setting is controlled in a drop-down menu in your browser utility (e.g. Internet Explorer).

Some pop-up blockers will provide an error message, making the user aware of the issueSome pop-up blockers tested provide no message whereby the screen just vanishes.




Module Summary

Key points covered in this module:

SMC was designed to be a lightweight device manager

Easy to install and minimal infrastructure requirements

Provides similar functionality as Solutions Enabler– Requires appropriate SE license key

Local or remote deployment options– SMC Server running on a host that is attached to the Symmetrix– SMC Communicates via the storapisrv to host attached to the

Symmetrix

Intuitive browser-based client




Closing Slide


Host Connectivity - 1



Module 5:Host Connectivity

Welcome to the fourth module of Symmetrix Monitoring and Management In this module we will take a look at what is involved with making Symmetrix devices available to an attached host.




Host ConnectivityUpon completion of this module, you will be able to:

Describe requirements for connecting a Open Systems host to a Symmetrix

Using the appropriate host, switch and Solutions Enabler commands, determine if a Fibre Channel fabric and port logins have occurred

Describe concept of port flags, determine how they should be set for a specific host, and use the appropriate Solutions Enabler commands to set them

Make a Symmetrix device visible to a host by setting the channeladdress using Solutions Enabler

Describe the concept of device masking and how masking is configured on the Symmetrix

There is more to connecting a host to a Symmetrix than simply plugging in the cables. This module starts by reviewing the I/O stack and all the components between the host application and the Symmetrix. To fully understand what is happening under the covers, we will spend a bit of time reviewing the Fibre Channel initialization process and then explore in detail what it takes to configure a Symmetrix for host access. Included will be configuring the operational characteristics of a front-end port, device mapping, and device masking.




IO Stack Open Systems

Host– Operating System– Application and/or Database– Logical Volume Manager– PowerPath– HBA

Storage Area Network (SAN)– Switches & Cables– Fibre Channel or TCP/IP

Symmetrix– Front-end Director

HBA HBAPowerPathOS/ LVM

App/Database

Back-end DirectorsCache

FA/SE Adapter

Host

SAN

Symmetrix

The diagram above illustrates the IO stack of a typical open systems host. Starting for the top, the host initiates a read or write operation. The I/O operation is passed by the I/O handlers in the operating system to the Logical Volume Management layer. PowerPath is between the Host Bus adapters and the upper layers of the IO stack. It is an optional component but is part of most open systems environments. It performs two functions: load balancing and path failover.

While it is possible to connect a HBA directly to a Symmetrix frond-end director port, more likely, you will be connected through a Storage Area Network or a SAN. A SAN consists of one or more interconnected switches. Originally SANs were based on Fibre Channel protocols but today we are starting to see TCP/IP based SANs for applications that don’t require the performance and availability of Fibre Channel. SANs provide greater connectivity by allowing more than one host to share the same front-end port on the Symmetrix. When compared to legacy SCSI connectivity which used parallel cables, SANs provide greater distances between the host and the Symmetrix. Depending on the SAN design, distances of 10’s of kilometers are possible. This allows consolidation and management efficiency.

In the Symmetrix, devices are presented to either a Fibre Channel or iSCSI front-end director and assigned a channel addresses. When more than one host is connected to the same front-end port (most-likely) Volume Logix is configured to restrict which host has access to which specific volumes.

The front-end directors were designed to support a large number of SCSI variants, therefore each port must be configured to support a subset of the SCSI and Fibre Channel protocol required for specific hosts.




Connectivity Requirements

Requirements for connecting an Open Systems host to a Symmetrix:– HBA and device drivers must be installed and configured– Path management software (PowerPath, DMP, MPIO, etc)– SAN connection between the Host Bus Adapter (HBA) and the

Symmetrix front-end director portPhysical cable connectionLogical connection – zoning

– On the Symmetrix:1.Director flag settings2.Channel address assignment on multiple Front-end ports

– Availability– Performance

3.Device masking

There are three things that must be configured when connecting a open systems host to the Symmetrix. HBA, SAN, and Symmetrix. We will look briefly at the HBA and the SAN but the focus of this discussion is the Symmetrix.




Host PerspectiveThe typical host configuration includes two or more Host Bus Adapters (HBA)

– HBAs are SCSI Initiator devices– Unique identifier

Fibre Channel WorldWide Name (WWN)iSCSI Qualified Name (iqn)

If more than one HBA, PowerPath or other path management software is installed to provides:

– Availability – Path failover– Performance – Load balancing

HBA connect to target devices and discover Logical Units (LUN)

– Symmetrix Devices– Identified by

LUN AddressesWWN

Reference Host Connectivity Guides on Powerlink

Typically a host will be configured with two or more HBAs, with each HBA configured to see the same set of devices on the Symmetrix. The Symmetrix has an Active-Active architecture and the same devices may be visible on more than one front-end port. When you add EMC PowerPath or other path management software to coordinate access to the devices, you get a higher level of availability and performance by spreading the workload across multiple HBAs and multiple front-end ports.

In a Fibre Channel world all devices are identified by a unique identifier called a WWN. The picture to the left is taken from the Emulex HBAnywhere utility and you can see on the host WIN1, there are two HBAs and each HBA sees four different front-end ports. On each port, seven devices called LUNs are mapped. Again, in environments where the same LUNs are visible on different front-end ports and HBAs, path management software such as PowerPath is required.

Reference Host Connectivity Guides on Powerlink.

The HBAnyware is an Emulex tool that allows easy verification of the FC environment on a host.




PowerPathMonitors path availability and workloadIf multiple paths, PowerPath dynamically selects the optimal pathAdjusts for spikes and changes in workloadTransparently redirects IO in the event of a path failure

Without PowerPath

AppsApps AppsApps AppsApps AppsApps AppsApps AppsApps

With PowerPath

SymmetrixFA FA

SymmetrixFA FA

AppsApps AppsApps

PowerPathAppsApps AppsApps

PowerPathAppsApps AppsApps

PowerPath

PowerPath is an EMC host-resident software product that works with any storage system to deliver intelligent I/O path management, and intelligent I/O balancing. PowerPath software improves performance and availability by intelligently sending I/O down the least busy path. This ensures consistently channel utilization across all paths. PowerPath also automatically detects and restores server-to-storage path failures.

Without PowerPath, applications direct I/O to devices as needed. In the illustration, some of the devices have more I/Os in their queues, leading to longer delays on those devices. If devices that have more I/O are crowded on to one HBA, that channel will be utilized more than the others.

PowerPath balances I/O over multiple paths to the same device. Host applications write to devices usual and PowerPath drivers intercept the I/O, and directs it to the device through the least busy path.




Fibre Channel InitializationPhysical cable connection– FA Port and the Switch– HBA Ports and the switch

When the HBA and FA ports are initialized, they will log in to the switch– Identifies itself using the World Wide

Name (WWN)– Updates the FC Name Server– Fabric Login (FLOGI)

The Initiator (HBA) will query the switch for a list of Target devices (FA ports)The HBA logs in to the FA ports– Port Login (PLOGI)– Device discovery

Host

HBA HBA

Symmetrix

FA FA

SwitchF-port F-port

F-port F-port

FLOGIFLOGI

FLOGIFLOGI

PLOGI PLOGI

The first step in connectivity is to have a connections from the Host Bus Adapter to the FA port. This requires physical cabling, and a logical path controlled by Fabric Zoning.

When the HBA and FA ports are initialized , or when the cables are connected, a Fabric Log In (FLOGI) occurs. This is a standard part of the Fibre Channel protocol where devices identify themselves to the fabric using their World Wide Name (WWN), a unique 128 bit hex identifier. The switch will update the Name Server database and assign the device a Fibre Channel Address. After the Fabric Log In, the HBA will ask the switch for the address of target devices (FA ports) and then attempt to log into each port. This is called a Port Log In or PLOGI. After the HBA has logged into the FA port, it will do a device discovery.

Note: The FA port maintains a persistent login tables on the FA ports so you can identify what HBAs have logged in in the past and the current state of the connection.




HBA Status (syminq hba)

The status of the HBA can be queried using syminq– Port WWN – Online state

# syminq hba -fibre

Host Name : diamondHBA Type : FibreChannelHBA Name : Emulex-LP8000-1Vendor : Emulex CorporationModel : LP8000Serial Number : 0000c9238053Firmware Version : 3.20 (D2D3.20X3)Driver Version : 5.00f; HBAAPI v1.3, 3-29-02Host WWN : 20000000c9238053Number of Ports : 1Port WWN : 10000000c9238053Port name : /devices/pci@1f,4000/fibre-channel@2Port type : NLPortPort FCID : 2493971Port speed : 1gbitSupported speed : 1gbitPort state : Online

# syminq hba -fibre

Host Name : diamondHBA Type : FibreChannelHBA Name : Emulex-LP8000-1Vendor : Emulex CorporationModel : LP8000Serial Number : 0000c9238053Firmware Version : 3.20 (D2D3.20X3)Driver Version : 5.00f; HBAAPI v1.3, 3-29-02Host WWN : 20000000c9238053Number of Ports : 1Port WWN : 10000000c9238053Port name : /devices/pci@1f,4000/fibre-channel@2Port type : NLPortPort FCID : 2493971Port speed : 1gbitSupported speed : 1gbitPort state : Online

HBA vendors provide tools to view the HBA. In addition, the Solutions Enabler symingcommand can be used to query the configuration and login status of an HBA. Note that in Fibre channel, all devices are identified by the World Wide Name. Actually there are two World Wide Names; the Node WWN and the Port WWN. The port WWN is what is typically used when configuring Fabric zoning and device masking.

In the example above, the Port state of Online indicates that the FLOGI process was successful and the HBA has logged into the switch.




Front-end Director Status

Determine if the FA port is on-line and a cable connection– Port status– Connection status# symcfg -fa all list -port -sid 097



Director Port Status Connection StatusIdent Type Status P0 P1 P2 P3 P0 P1 P2 P3

FA-3A FibreChannel Online ON ON N/A N/A X - - -FA-4A FibreChannel Online ON ON N/A N/A - - - -FA-7A FibreChannel Online ON ON N/A N/A X - - -FA-10A FibreChannel Online ON ON N/A N/A - - - -FA-13A FibreChannel Online ON ON N/A N/A - - - -FA-14A FibreChannel Online ON ON N/A N/A - - - -FA-3B FibreChannel Online ON ON N/A N/A X - - -

Legend for Connection Status:(X) : Fibre Port is Connected to a Fibre Port (HBA, Switch or RF Director)(-) : Fibre Port is Not Connected.

# symcfg -fa all list -port -sid 097



Director Port Status Connection StatusIdent Type Status P0 P1 P2 P3 P0 P1 P2 P3

FA-3A FibreChannel Online ON ON N/A N/A X - - -FA-4A FibreChannel Online ON ON N/A N/A - - - -FA-7A FibreChannel Online ON ON N/A N/A X - - -FA-10A FibreChannel Online ON ON N/A N/A - - - -FA-13A FibreChannel Online ON ON N/A N/A - - - -FA-14A FibreChannel Online ON ON N/A N/A - - - -FA-3B FibreChannel Online ON ON N/A N/A X - - -

Legend for Connection Status:(X) : Fibre Port is Connected to a Fibre Port (HBA, Switch or RF Director)(-) : Fibre Port is Not Connected.

On the Symmetrix, you can also determine the status of the FA port. The port status tells us if the Port is online and the connection status tells us if there is a cable connected. If the Connection status shows an “X” under a port, host likely it is connected to a switch or directly to an HBA.




Front-end Director Details

FA Port details– WWN– Port and connection status

# symcfg -SA 07a -v list...Director Identification: FA-7A

Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags...


Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags...

The verbose option provides similar information on the port and connection status. Note. If a director is not on-line, you can use the symcfg command to place a port on-line or off-line. Example: symcfg -SA 07a –p 0 online




Switch Perspective - Fabric Login (FLOGI)

Online indicates the HBA and FA Ports have logged into the switchswitchshow

(Brocade)

switch118:admin> switchshow…Port Media Speed State=========================

0 id N2 Online F-Port 50:06:04:61:00:60:02:421 id N2 Online F-Port 50:06:04:60:00:60:02:422 id N2 Online F-Port 50:06:04:68:00:60:02:423 id N2 Online F-Port 50:06:04:69:00:60:02:424 id N2 No_Light5 id N2 No_Light6 id N2 No_Light7 id N2 No_Light8 id N2 Online F-Port 10:00:01:69:30:60:2e:d29 id N2 Online F-Port 10:00:01:60:30:60:2e:d2

10 id N2 Online F-Port 10:00:01:68:30:60:2e:d211 id N2 Online F-Port 10:00:01:61:30:60:2e:d212 id N2 Online F-Port 10:00:01:68:30:60:2f:3b13 id N2 Online F-Port 10:00:01:60:30:60:2f:3b14 id N2 Online F-Port 10:00:01:61:30:60:2f:3b15 id N2 Online F-Port 10:00:01:69:30:60:2f:3b

switch118:admin> switchshow…Port Media Speed State=========================

0 id N2 Online F-Port 50:06:04:61:00:60:02:421 id N2 Online F-Port 50:06:04:60:00:60:02:422 id N2 Online F-Port 50:06:04:68:00:60:02:423 id N2 Online F-Port 50:06:04:69:00:60:02:424 id N2 No_Light5 id N2 No_Light6 id N2 No_Light7 id N2 No_Light8 id N2 Online F-Port 10:00:01:69:30:60:2e:d29 id N2 Online F-Port 10:00:01:60:30:60:2e:d2

10 id N2 Online F-Port 10:00:01:68:30:60:2e:d211 id N2 Online F-Port 10:00:01:61:30:60:2e:d212 id N2 Online F-Port 10:00:01:68:30:60:2f:3b13 id N2 Online F-Port 10:00:01:60:30:60:2f:3b14 id N2 Online F-Port 10:00:01:61:30:60:2f:3b15 id N2 Online F-Port 10:00:01:69:30:60:2f:3b

You can take a number of different approaches to verifying connectivity status; starting at the host and working toward the Symmetrix or starting at the Symmetrix and look toward the host. Another approach is to start in the middle. In Fibre Channel SANs, the switch provides a login service. When a HBA or FA port is connected to the switch, it identifies itself by it’s WWN, specifies operating parameters such as buffer credits, link speed, and class of services and is assigned a Fibre Channel address.

In the example above, we see that both the HBA and FA ports have logged into the switch. Note, the example is for a Brocade switch but other vendors have commands that provide similar information.




Switch Zoning

Switch zoning determines which FA ports an HBA “sees”zoneshow

(Brocade) Switch118:admin> zoneshow...Effective configuration:

cfg: Production_cfg

zone: HBA0_FA07A_Port010:00:01:60:30:60:2f:3b50:06:04:60:00:60:02:42

zone: HBA0_FA10A_Port0 10:00:01:60:30:60:2f:3b50:06:04:68:00:60:02:42



…

Switch118:admin> zoneshow...Effective configuration:

cfg: Production_cfg

zone: HBA0_FA07A_Port010:00:01:60:30:60:2f:3b50:06:04:60:00:60:02:42




…

Zoning controls access in a Fibre Channel network by allowing or restricting access between HBAs and front-end ports. There are several ways of defining zones, but the most common is using the World Wide Port name which uniquely identifies a device in a Fibre Channel network. In the example above, we see several different zones defined. Each zone includes a single HBA and a FA port. While it is possible to have a big zone with multiple members, our best practice is to configure single initiator zoning which is a single HBA per zone. This approach eliminates unnecessary interactions between devices.




FA Port Login

HBA –> FA Log in– FLOGI

# symmask list logins –dir 02a –p 1


Director Identification : FA-2A

Director Port : 1

User-generated Logged OnIdentifier Type Node Name Port Name FCID In Fabric

---------------- ----- --------------------------------- ------ ------ -----

10000000c9238053 Fibre diamond i@1f,4000,@2 260e13 Yes Yes10000000FEEBDAED Fibre diamond i@1f,4000,@2 260e13 No No

# symmask list logins –dir 02a –p 1



Director Port : 1

User-generated Logged OnIdentifier Type Node Name Port Name FCID In Fabric

---------------- ----- --------------------------------- ------ ------ -----

10000000c9238053 Fibre diamond i@1f,4000,@2 260e13 Yes Yes10000000FEEBDAED Fibre diamond i@1f,4000,@2 260e13 No No

If the both the HBA and FA ports have logged into the switch, and the switch is zoned correctly, the HBA should log in to the FA.

The identifier field indicates which HBA is communicating with the Symmetrix array. User-generated node and port names are identified as the AWWN or AISCSI alias associated with it. Columns labeled On Fabric and Logged In indicate whether the HBA is connected to a fabric and whether it is logged in to the Symmetrix system You can use the verbose (-v) option to view the last active login information.




Symmetrix Connectivity RequirementsOnce there is connectivity between the HBA and the front-end director port, there are three things that need to be configured on the Symmetrix:

0. Assessment of host requirements & planning - Critical!1. Director port characteristics

SCSI and Fibre Channel operating parametersHost Operating Systems Specific

2. Device mappingAssign a channel address to a device on a SP portLogical Unit Number

3. Device maskingA single FA port can be shared my many hostsControls access so specific hosts see specific devicesMasking information is maintained in Volume Logix Database

There are three things that must be configured on the Symmetrix in order to connect a open systems host: port flags, device mapping, and device masking

However one of the most important thing is to have a plan.What host is going to connect to what portWhat operating systems and versionsNumber, type, and firmware levels for Host Bus Adapters (HBA)Is PowerPath or other multi-pathing failover software usedHow many, what protection, what size volumes are requiredPerformance considerations, special Symmetrix features

Normally this is done as part of the pre-site survey and is again validated prior to installation. In Engineering labs, you may not have a formal site planning document but you still need a plan.




Front-end Port Characteristics

Front-end directors can be configured to support a wide variety of host operatingsystemsCalled flagsParameters for:– Fibre Channel– SCSI– Gigbit Ethernet

(iSCSI)


Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags{Tagged_Commands : EnabledLinked_Commands : EnabledSync_Transfer : EnabledWide_Transfer : Enabled

...


Director Type : FibreChannelDirector Status : OnlineNumber of Director Ports : 2Director Ports Status : [ON,ON,N/A,N/A]Director Connection Status : [N/A,N/A,N/A,N/A]Director Symbolic Number : 07ADirector Numeric Number : 7Director Slot Number : 7Director Port: 0WWN Node Name : 50060482d52cb306WWN Port Name : 50060482d52cb306Fibre Channel Loop ID : N/AFibre Adapter Type : N/ASCSI Flags{Tagged_Commands : EnabledLinked_Commands : EnabledSync_Transfer : EnabledWide_Transfer : Enabled

...

Protocols used to communicate between hosts and storage systems are designed to support a wide variety of applications and operating system requirements. SCSI is a standards-based protocol that has been around for over twenty years and the command set and nexus is flexible enough to support many different types of storage devices and host operating systems. Nearly every server vendor supports SCSI, unfortunately not every vendor implements SCSI in exactly the same way. For example, while both HP-UX and IBM AIX support the SCSI protocol, they support a different subset of the operational parameters. Fibre Channel and IP are transport protocol used with the SCSI protocol and they too has a number of configurable protocol and link parameters.

The good news is that the emulation code in the Symmetrix front-end director port is implemented in software and we have the flexibility to configure the front-end port on to support diversity of host configurations. These settings are called port flags.




Setting Front-end Port Flags

Port operating characteristics, called flags, are configured in the binfile using SymmWin on the Service ProcessorAlso configured using Solutions Enabler symconfigure commandReference the EMC Support Matrix for specific settings for a host environment

# This is a command file for# setting FA port flags

set port 07A:0SCSI_3=enabled,Volume_Set_Addressing=enabledUnique_WWN=enabled,Auto_Negothiate=enabled;

set port 10A:0copying port 07A:0;


You cannot simply plug a cable into a free FA port and have it work correctly. Various host operating systems have different protocol requirements.

Specific port flags are configured in the binfile when the Symmetrix is first configured but can also be configured using the symconfigure command. This allows a storage administrator to dynamically reconfigure the operating characteristics of a port without having to depend on EMC to change the binfile.

It is important to understand what the individual flags do before changing them. An incorrect setting can cause errors or performance problems. Reference the EMC Support matrix or the e-Lab Navigator for specific configuration requirements for each operating system type and configuration.




Heterogeneous Environments

If there multiple host sharing a front-end port and they have different operational parameter requirements, the port setting can be overwritten using the symmask command– Specify the host operating system – Specify the specific flag setting (SE 6.4)

# symmask -wwn 1234567891234567 set heterogeneous on IBM_AIX_DMP -dir 7a -p 0

# symmask -wwn 1234567891234567 set hba_flags on C,SC3 dir 7a -p 0

# symmask -wwn 1234567891234567 set heterogeneous on IBM_AIX_DMP -dir 7a -p 0

# symmask -wwn 1234567891234567 set hba_flags on C,SC3 dir 7a -p 0

Setting the flags for a specific FA port is a good approach if all host that share the port have the same requirements. If you have a heterogeneous environment and different hosts have different requirements, the flag requirements can be set in the Volume Logix database and they are enforced on a connection basis.

Two ways to do this: By setting the Operating system type or setting the specific flags that will override the settings on the port. Setting the specific flags is the preferred way starting with Solutions Enabler 6.4 Volume Logix and the VCM database are discussed later in this module.

Note: Setting the heterogeneous host configuration has been superseded by Setting the HBA port flags. The heterogeneous host configuration types continue to be valid, but will not be expanded. To switch to setting HBA port flags, the heterogeneous host configuration must be disabled on the array, and all flags must be reset.




E-Lab Configuration Information

The E-lab Navigator is accessible from the Powerlink and/or other EMC websites. Host specific director flag settings information can be found here. The support matrix refers to these flags as bit settings as they were originally set by toggling specific bits




Director Bit Information Table

This is an excerpt from the full Director Bit Information table. The table includes all operating systems, both clustered and non-clustered, and all supported Enginuity levels. There are also references to extensive footnotes.




Mapping a Device to a Port Symmetrix Devices are made available on a FA port by assigning them a channel address– Referred to as mapping– Configure in the binfile but also can be

configured using Solutions Enabler

Typically the same devices are mapped to two or more FA ports– Availability – Load balancing– Requires host-based path

management software

The channel address is used by an attached host access the device– Often reflected in the c#t#d# device

naming convention– Example channel address 0003 may be

seen as c1t0d3

Host

HBA HBA

Symmetrix

FA FA

00 0201 03

04 FF…

C#

T#D#

A Symmetrix can have over 64000 devices configured. Not all devices are accessed by every front-end port. Instead, specific devices are “mapped” to specific ports by assigning a channel address. Host systems discover and access Symmetrix devices using these Channel Addresses. For open systems hosts, the Channel address is the SCSI ID. Normally a host uses a combination of the Controller, Target, and Logical Unit Number to address a disk device. The Controller number is the Host Bus Adapter, the Target is the port on the Storage System and the Logical Unit Number is the Channel Address we assign.




Mapping Requirements

1. Identify available devices– Devices that are currently not mapped to other hosts– Normally only a single host will have access to device– Exception would be clustered environments

2. Identify available channel address– Valid addresses: 000-FFF (Maximum of 4096 addressable devices

per port)Not all addresses may be valid for a host

– Address must be unique on a port

Before performing a mapping operation, some research and planning is required. First what is a valid address range for a host. Secondly, what addresses are currently available.

Possible host operating system addressing issues:Some hosts only support addresses 00-FFSome hosts require range of addresses to start with 00Some hosts do not allow holes in address range− Example devices 00-10, 20-2F

Later in this module we will discuss an alternative way of assigning addresses that can allow multiple hosts to use the same address for different devices on the same FA port.




Mapping - Identify Available Devices

C:\>symdev list -noport


Device Name Directors Device------------------------ ------------- -------------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)------------------------ ------------- -------------------------------------002C Not Visible ???:? 16C:D6 2-Way Mir N/Grp'd RW 8438002D Not Visible ???:? 16A:D6 2-Way Mir N/Grp'd RW 8438002E Not Visible ???:? 16B:D7 2-Way Mir N/Grp'd RW 8438002F Not Visible ???:? 16D:D7 2-Way Mir N/Grp'd RW 84380030 Not Visible ???:? 16C:C1 2-Way Mir N/Grp'd RW 84380031 Not Visible ???:? 16D:C0 2-Way Mir N/Grp'd RW 84380032 Not Visible ???:? 01B:D0 2-Way Mir N/Grp'd RW 84380033 Not Visible ???:? 01D:D0 2-Way Mir N/Grp'd RW 84380034 Not Visible ???:? 16B:C0 2-Way Mir N/Grp'd RW 84380035 Not Visible ???:? 16A:C1 2-Way Mir N/Grp'd RW 84380036 Not Visible ???:? 01A:C2 RAID-5 N/Grp'd (M) RW 45000003A Not Visible ???:? 01A:C4 RAID-5 N/Grp'd RW 11250003B Not Visible ???:? 16A:D4 RAID-5 N/Grp'd RW 11250003C Not Visible ???:? 16A:C5 RAID-5 N/Grp'd RW 11250…

C:\>symdev list -noport


Device Name Directors Device------------------------ ------------- -------------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)------------------------ ------------- -------------------------------------002C Not Visible ???:? 16C:D6 2-Way Mir N/Grp'd RW 8438002D Not Visible ???:? 16A:D6 2-Way Mir N/Grp'd RW 8438002E Not Visible ???:? 16B:D7 2-Way Mir N/Grp'd RW 8438002F Not Visible ???:? 16D:D7 2-Way Mir N/Grp'd RW 84380030 Not Visible ???:? 16C:C1 2-Way Mir N/Grp'd RW 84380031 Not Visible ???:? 16D:C0 2-Way Mir N/Grp'd RW 84380032 Not Visible ???:? 01B:D0 2-Way Mir N/Grp'd RW 84380033 Not Visible ???:? 01D:D0 2-Way Mir N/Grp'd RW 84380034 Not Visible ???:? 16B:C0 2-Way Mir N/Grp'd RW 84380035 Not Visible ???:? 16A:C1 2-Way Mir N/Grp'd RW 84380036 Not Visible ???:? 01A:C2 RAID-5 N/Grp'd (M) RW 45000003A Not Visible ???:? 01A:C4 RAID-5 N/Grp'd RW 11250003B Not Visible ???:? 16A:D4 RAID-5 N/Grp'd RW 11250003C Not Visible ???:? 16A:C5 RAID-5 N/Grp'd RW 11250…

Identifying unmapped devices can be easily done using the symdev list –noportcommand which identifies devices that are not currently assigned to any port.




Mapping - Identify Available Channel AddressC:\>symcfg list -FA 8a -address -available


Director Device Name Attr Address---------------------- ----------------------------- ---- --------------Ident Symbolic Port Sym Physical VBUS TID LUN ----- -------- ---- ---- ----------------------- ---- --- ---FA-8A 08A 0 - AVAILABLE 0 00 000*

0020 \\.\PHYSICALDRIVE1 0 00 050 0021 \\.\PHYSICALDRIVE2 0 00 051 0022 \\.\PHYSICALDRIVE3 0 00 0520023 \\.\PHYSICALDRIVE4 0 00 053 0024 \\.\PHYSICALDRIVE5 0 00 0540025 \\.\PHYSICALDRIVE6 0 00 055 0026 \\.\PHYSICALDRIVE7 0 00 0560027 \\.\PHYSICALDRIVE8 0 00 0570028 \\.\PHYSICALDRIVE9 0 00 0580029 \\.\PHYSICALDRIVE10 0 00 059- AVAILABLE 0 00 05A*

Total ----Mapped Devices: 10Including Metamembers: 10Available Addresses: 4086 (s)

C:\>symcfg list -FA 8a -address -available


Director Device Name Attr Address---------------------- ----------------------------- ---- --------------Ident Symbolic Port Sym Physical VBUS TID LUN ----- -------- ---- ---- ----------------------- ---- --- ---FA-8A 08A 0 - AVAILABLE 0 00 000*

0020 \\.\PHYSICALDRIVE1 0 00 050 0021 \\.\PHYSICALDRIVE2 0 00 051 0022 \\.\PHYSICALDRIVE3 0 00 0520023 \\.\PHYSICALDRIVE4 0 00 053 0024 \\.\PHYSICALDRIVE5 0 00 0540025 \\.\PHYSICALDRIVE6 0 00 055 0026 \\.\PHYSICALDRIVE7 0 00 0560027 \\.\PHYSICALDRIVE8 0 00 0570028 \\.\PHYSICALDRIVE9 0 00 0580029 \\.\PHYSICALDRIVE10 0 00 059- AVAILABLE 0 00 05A*

Total ----Mapped Devices: 10Including Metamembers: 10Available Addresses: 4086 (s)

Before performing a mapping operation, some research and planning is required. First what is a valid address range for a host. Secondly, what addresses are currently available.

(*)indicates a gap in the address assignments or are the next available address in the run . In this example address 000 -4F and 05A and above are available addresses.




Mapping a Device to a Port Configure in the binfile but also configured using Solutions Enabler symconfigure commandIn most environments, the same devices are mapped to more than one FA port – Availability – Performance

Example:symconfigure -file mycmdfile commit

Also possible to perform masking in same operation

# This is a command file for# mapping a device to a port

map dev 00cd to dir 14A:0target=0, lun=7;

map dev 00cd to dir 3A:0target=0, lun=7;

map dev 00e0:00e7 to dir 14A:0starting target= 1, lun=0;

map dev 00e0:00e7 to dir 3A:0starting target= 1, lun=0


Mapping can be performed when the bin files is created or can be modified by editing the binfile and doing an on-line configuration change but it is often easier to perform mapping using the symconfigure command.

When configuring mapping, you specify the LUN address and optionally the target address. You can assign a specific address to specific LUNS or specify a range of devices and a starting address. It is a best practice to specify the same channel address when configuring a LUN on multiple FA ports.




Displaying Device Mapping

Device address from the device perspective

C:\>symdev show 21


Device Physical Name : \\.\PHYSICALDRIVE2

Device Symmetrix Name : 0021

…

Front Director Paths (2):

{

----------------------------------------------------------------------

POWERPATH DIRECTOR PORT LUN---------- ---------- ---- -------- ---------

PdevName Type Type Num Sts VBUS TID SYMM Host ----------------------------------------------------------------------\\.\PHYSICALDRIVE2 N/A FA 08A:0 RW 000 00 051 000 Not Visible N/A FA 09A:0 RW 000 00 051 N/A }

…

C:\>symdev show 21


Device Physical Name : \\.\PHYSICALDRIVE2

Device Symmetrix Name : 0021

…

Front Director Paths (2):

{

----------------------------------------------------------------------

POWERPATH DIRECTOR PORT LUN---------- ---------- ---- -------- ---------

PdevName Type Type Num Sts VBUS TID SYMM Host ----------------------------------------------------------------------\\.\PHYSICALDRIVE2 N/A FA 08A:0 RW 000 00 051 000 Not Visible N/A FA 09A:0 RW 000 00 051 N/A }

…

The symdev show command can be used to determine what devices are mapped to what port. In most environments, high availability is important and mapping the same device to multiple ports is a requirement.




Host Addressing Considerations

An FA port supports a maximum of 4096 device addresses

Typically, multiple host systems share the same port

Masking will effectively allow exclusive assess to a subset of devices

May be desirable to have each host address range start with 00

Solution: Configuring Dynamic LUN Addressing

Host A

HBA HBA

Symmetrix

FA FA

00 0201 03

04 FF…

Host B

HBA HBA

Host C

HBA HBA

The channel address assigned to a logical is used by the host when it discovers and accesses a Symmetrix Logical Volume. Typically a disk is addresses through a Controller instance, a Target device, and Device on that target. This is often represented as a c#t#d#. For example /dev/rdsk/c1t0d3 references Host Bus Adapter 1, Target 0 (first port accesses through the HBA), and device 3 (channel address 3).

LUN offset is an enhanced visibility feature that allows any host type to adjust host visibility by offsetting (renumbering) LUN addresses. This is useful for host types that need to see LUN 0000 or transform a noncontiguous LUN sequence to a contiguous sequence. In a case where two hosts access the same Symmetrix director port and need to see a LUN 0000 but not the same device, you can use LUN offset so that one host sees the devices mapped from LUN “x” as starting from LUN 0000, and the other host sees devices from LUN “y” as starting from LUN 0000. To account for noncontiguous device LUN addresses, specify a LUN base and offset as hexadecimal values to adjust for the break in the LUN sequence. The base hex value represents the first LUN in a renumbered LUN sequence. The offset hex value added to the base value determines where to begin renumbering. For example, if a host needs to detect LUN 0000 but you want your host to detect only LUNs 0005 through 0008, you can specify a LUN base address of 0000 and an offset of 0005 The use of the Solutions Enabler symmask command is discussed later in this module but below is an example of a command to renumbers LUNs 0005 through 0008 as LUNs 0000 through 0003:: symmask set lunoffset on 0005 0000 –dir 16A -p 0 –wwn 10000000c920b484




Device MaskingDevice maskingallows multiplehosts to effectively share the same front-end ports– FA port can “see” up to 256 HBAs

Restrict access to specific host and/or host clusters

Implemented in theSymmetrix with Volume Logix– Fibre Channel– iSCSI

Switch

Host A

HBA HBA

Host B

HBA HBA

Host C

HBA HBA

Host D

HBA HBA

Host X

HBA HBA

Host Y

HBA HBA

Host Z

HBA HBA

VCMDB

Symmetrix

FA or SEp0 p1

FA or SEp0 p1

Storage Area Networks provide a fan-out capability were it is likely that more than one host is connected to the same Symmetrix port. The actual number of HBA’s that can be configured to a single port is operating system and configuration dependent but fan-out ratios as high as 256:1 are currently supported. Reference the support matrix for specific configuration limitations.

Each port may have as many as 4096 addressable volumes presented. When several hosts connect to a single Symmetrix port, an access control conflict can occur because all hosts have the potential to discover and use the same storage devices. However, by creating entries in the Symmetrix’s device masking database (VCMDB), you can control which host “sees” which volume.

Device Masking is independent from zoning but they are typically used together in an environment. Zoning provides access control at the port level and restricts which host bus adapter “sees” which port on the storage system and device masking restricts which host sees which specific volumes presented on a port.

With Fibre Channel, Device Masking uses the UWWN (Unique Worldwide Name) of Host Bus Adapters and a VCM database device. In iSCSI, the iSCSI Qualified Name (IQN) is used. Regardless of the protocol, the concepts are the same. The device-masking database (VCMDB) on each Symmetrix unit specifies the devices that a particular WWN or IQN can access through a specific Fibre port.




Device MaskingHost AHBA0WWN

HBA1WWN

Symmetrix

FA3a:0 FA14a:0

00 0201 …

VCMDB

HBA-to-FA connection records are created and maintained in the Device Masking Database (VCMDB)– HBA0 WWN -> FA03a:0 - dev 000-010– HBA0 WWN -> FA14a:0 - dev 000-010– HBA1 WWN -> FA03a:0 - dev 000-010– HBA1 WWN -> FA14a:0 - dev 000-010

Entries in the VCMDB define relationship between masked connections and devices– FA consults VCMDB to resolve access rights

Same approach for both FC and iSCSI

Device Masking controls host access to a set of devices by maintaining a set of entries in the VCMDB on the array that defines the relationship between masked connections and devices. These entries are sometimes called initiator records.

Each entry includes a host's HBA identity (WWN or iSCSI Qualified Name), its associated FA port, and a range of devices mapped to the FA port that should be visible only to the corresponding HBA.

Once you make this VCMDB entry and activate the configuration, the Symmetrix makes visible to a host those devices that the VCMDB indicates are available to that host's initiator through that FA port.

Volume Logix is the brand name for the software in the Symmetrix that performs the device masking function. The capability is built into Enginuity but its use is optional.




Volume Logix Database - VCMDBDatabase types are Symmetrix model specific and sized to handle the maximum number of logical volumes – Originally a separate Symmetrix Logical Volume– Starting with the DMX-3, it is stored in the Symmetrix File System

VCM DB is maintained using Solutions Enabler symmask and symmaskdb commands

Solutions Enabler accesses the VCM database using a VCM gatekeeper device– VCM gatekeeper is host accessible– The SFS is not directly accessible– VCM GK only needs to be mapped

to the management host

Symmetrix File System Symmetrix File System

VCM GK

VCM GK

VCMDBManagement Host

The Volume Logix Database persistently maintains the device masking information. Originally the database was located directly on a Symmetrix Logical Volume. On DMX-3 it is maintained in the Symmetrix File System (SFS). Rather than create the actual VCMDB device, today we create a VCM Gatekeeper device which is used by the Solutions Enabler to access the database on the SFS, as the SFS volumes are not host addressable. The VCM Gatekeeper is a 6-cyl device. Solutions Enabler 5.3+ is required for DMX-3 running 5771 code.

By default, the device masking VCMDB is accessible to all HBAs that log into the director port where the database is configured. Thus, any host with access privileges can effectively modify the contents of the database if it has device masking commands are installed. Beginning with Enginuity Version 5670, the VCMDB can be unmapped from any director that is not being used for masking control.




Setting up Device Masking

There are four steps in setting up device masking on the Symmetrix

1. Configure a VCM gatekeeper device2. Set the VCM attribute for the device3. Set the VCM flag for front-end director port4. Assign the VCM gatekeeper to front-end port

After the VCM is setup, the database is maintained using the SE symmask and symmaskdb commands

– Add and remove masking entries– Initialize, query, backup, and restore database – May also be configured with EMC ControlCenter or SMC– External configuration locks are used to coordinate updates

Typically performed when the bin file is initially created

To set up device masking, you must create a database volume (VCMDB), and set flags on the Fibre Channel or iSCSI ports to enable the uses. Once the database is setup and enabled, the Solutions Enabler symmask command can be used to configure entries granting specific hosts access to specific volumes.

During the execution of the symmask or symmaskdb commands, the SYMCLI sets a Symmetrix External Lock on the Symmetrix where the device masking database (VCMDB) resides. This lock ensures that only one host can make changes to the database at any one point in time. If during the processing of a symmask or symmaskdb command, the host fails, or a Ctrl/C is performed in the middle of the command, the lock might not release and could lock out further needed changes or control actions. If a device masking command is interrupted and the lock is not released, future invocations of a device masking command will display the following error message: The operation failed because another process has an exclusive lock on the local Symmetrix.

To further examine the presence of this lock, use the following form:

symcfg -sid SymmID list -lock -lockn ALL

The command will list Symmetrix external locks being held..

To release this lock, use the following form:

symcfg -sid SymmID -lockn # release




DiscoveryThe symmask discover hba command– Verifies VCM is enabled for port– Identifies paths to the VCMDB– Assigns alias names to the HBAs AWWN

Host AHBA0WWN

HBA1WWN

Symmetrix

FA3a:0 FA14a:0

00 0201 …

VCMDBVCMGK

C:\>symmask discover hba


Device Masking Status : Success

Identifier Type User-generated Name

---------------- ----- -------------------------------

10000000c93da44a Fibre W2K3-39-106/10000000c93da44a10000000c93da45c Fibre W2K3-39-106/10000000c93da45c

C:\>symmask discover hba


Device Masking Status : Success

Identifier Type User-generated Name

---------------- ----- -------------------------------

10000000c93da44a Fibre W2K3-39-106/10000000c93da44a10000000c93da45c Fibre W2K3-39-106/10000000c93da45c

The symmask discover command can be run on the management host and or other attachedhosts. The symmask discover identifies paths to the device masking database (VCMDB) and assigns alias names (AWWN/AISCSI) to the HBAs residing on the host on where the command was executed.

When the symmask discover finds a host HBA, it reads the login history table and performs the following:

1) Checks whether an alias exists in the device masking VCMDB. If one does, this command writes it to the login history table. If there is no alias in the device masking VCMDB record, or the login history table, it creates an ASCII alias and writes it to the login history table.

2) Outputs the initiator identifier (WWN/iSCSI) of the HBAs that are connected to the masked channel and Symmetrix array.

Alias names can be used in the command line, replacing the cumbersome numeric identifiers. These names, which are stored in the Symmetrix array’s login history table, identify the HBAs connected to the network interface. Alias names can be shorter in length and much more recognizable than the cryptic WWNs/iSCSIs.

ASCII alias names generated by the discover action consists of two parts: the name of the host and the name of the HBA. For example: the AWWN for a host whose TCP/IP hostname is john4554b, on adapter 10000000c920cf87, would be john4554b/10000000c920cf87.




Initializing and Formatting the VCMDB Database# symmaskdb list database

The Symmetrix device masking database has not been initialized

C:>symmaskdb init -file VCMdbBackup070707

Initialize Symmetrix SymMask database on Symmetrix 000190100172 (y/[n])? y

Symmetrix SymMask database on Symmetrix 000190100172 initialized

# symmaskdb list database

The Symmetrix device masking database has not been initialized

C:>symmaskdb init -file VCMdbBackup070707

Initialize Symmetrix SymMask database on Symmetrix 000190100172 (y/[n])? y

Symmetrix SymMask database on Symmetrix 000190100172 initialized

During the initial setup of any device masking environment, you must initialize the database. In the process of formatting the VCMDB, the current data is cleared. In most cases, you do not want to clear the data of an existing VCMDB. If you are unsure whether a VCMDB currently exists, issue the following command::

symmaskdb list database

To initialize and clear the VCMDB database, you must specify a backup file name to safeguard against clearing data in the database that should not be lost. For example, the following command creates a file called MyInitBackup and attempts to write any current data to it prior to initializing and formatting the VCMDB:

symmaskdb init –file MyInitBackup

Note: The Solutions Enabler device masking function requires a license key. This is installed using the symlmf command.




Adding HBA Access to Symmetrix DevicesC:> symmask list hba

Identifier Type Adapter Physical Device Path Dir:P

---------------- ----- ---------------- ------------------------ -----

10000000c9274156 Fibre i@1f,4000,@2 Physicaldrive0 7A:0

Physicaldrive1 7A:0

10000000c92741a1 Fibre i@1f,4000,@4 Physicaldrive2 8A:0

Physicaldrive3 8A:0

C:> symcfg list -FA 7a -addr


Director Device Name Attr Address ---------------------- ----------------------------- ---- --------------

Ident Symbolic Port Sym Physical VBUS TID LUN

------ -------- ---- ---- ----------------------- ---- --- ---

FA-7A 07A 0 0040 Not Visible 0 00 000 0041 Not Visible 0 00 0010042 Not Visible 0 00 0020043 Not Visible 0 00 0030044 Not Visible 0 00 004

C:> symmask list hba

Identifier Type Adapter Physical Device Path Dir:P

---------------- ----- ---------------- ------------------------ -----

10000000c9274156 Fibre i@1f,4000,@2 Physicaldrive0 7A:0

Physicaldrive1 7A:0

10000000c92741a1 Fibre i@1f,4000,@4 Physicaldrive2 8A:0

Physicaldrive3 8A:0

C:> symcfg list -FA 7a -addr


Director Device Name Attr Address ---------------------- ----------------------------- ---- --------------

Ident Symbolic Port Sym Physical VBUS TID LUN

------ -------- ---- ---- ----------------------- ---- --- ---

FA-7A 07A 0 0040 Not Visible 0 00 000 0041 Not Visible 0 00 0010042 Not Visible 0 00 0020043 Not Visible 0 00 0030044 Not Visible 0 00 004

When configuring device masking, there are three pieces of information that are needed. First, you need to know the World Wide Port Number for the Host Bus Adapter(s). Second, you need to know how the HBA is connected to the Symmetrix; specifically you need to know the front-end director number and port. Finally, you need to know what Symmetrix devices that are mapped to the port. Available devices are those that have a channel address assigned.

There are a number of ways to identify the WWN of a HBA. One way is to use the symmask list hba command. Not only will this display the WWN for each HBA in the host, it also displays the director and port of the connection to the Symmetrix and the device file for the gatekeeper device. In the example above we see the host actually has two HBAs: one with the WWN of 10000000c9274156, and the other 10000000c92741a1.We also see that one is connected through director 7a port0 and the other through 8a por0.

To get a list of Symmetrix Logical Volumes on a port you could use the Solutions Enabler symcfg list command as shown in the example above. He we see there are a number of volumes presented to Director7A Port 0 starting with Symmetrix Logical Volume number 0040.




Adding HBA Access to Symmetrix Devices

C:> symmask add dev 40,41,42 -wwn 10000000c9274156 -dir 7a -p 0

C:> symmask add dev 40,41,42 -wwn 10000000c92741a1 -dir 8a -p 0

C:> symmask refresh

Refresh Symmetrix FA directors with contents of SymMask database000190100172 (y/[n]) ? y

Symmetrix FA directors updated with contents of SymMask Database000190100172

C:>

C:> symmask add dev 40,41,42 -wwn 10000000c9274156 -dir 7a -p 0

C:> symmask add dev 40,41,42 -wwn 10000000c92741a1 -dir 8a -p 0

C:> symmask refresh

Refresh Symmetrix FA directors with contents of SymMask database000190100172 (y/[n]) ? y

Symmetrix FA directors updated with contents of SymMask Database000190100172

C:>

To make an entry for the HBA-to-FA connection in the VCMDB and specifying devices that the HBA can access, use the symmask command shown above. On the first line we are specifying that volumes 0040, 0041, and 0042 are accessible to the first HBA through FA 7A port0. The second command enables access to the same volumes through the other HBA and the other Symmetrix port.

After making changes to the VCM database, you must tell the Symmetrix to refresh the access control tables in the director. This is done using the symmask refresh command.

Note: In addition to explicitly listing the devices to be added. If you are using devices that have been reserved, you must supply the device reservation ID. For example, to add reserved device 0014 for access to Host3b using director 16a, port 0, enter:

symmask -wwn 1234567890123456 add dev 0014 -dir 16a -p 0 -reserve_id 5

You can also use a previously defined device group to specify the devices to mask to a host. For example:symmask –wwn 1234567890123456 add -g prodB -std -dir 16a -p 0




Dynamic AddressingWhen devices are mapped to front-end ports, a channel address is assigned

The LUN address can be specified in the connection record overriding the mapped address

Allows each host sharing a port to used the same addresses to access different devices

Example:symmask add devs 15,18,20 -lun 0 -wwn 20000000c920b484 -dir 14C -p 1

Host A

HBA HBA

Symmetrix

FA FA

00 0201 03

04 FF…

Host B

HBA HBA

Host C

HBA HBA

When adding devices you can specify the starting LUN address for each device using the -lun option. Or have SYMAPI assign the LUN address using the -dynamic_lun option.

-lun = Specifies starting LUN addresses. You can specify a single starting LUN or multiple LUNs to match the given ranges. For example:

symmask add devs 15,18 -lun 0 -wwn 20000000c920b484 -dir 4C -p 1

-dynamic_lun = Specifies the use of dynamic LUN addressing. The application assigns the addresses based on what is already in use for the host HBA.-

symmask add devs 2C,2E,30 -dynamic_lun -wwn 20000000c920b484 -dir 2a -p 1




Displaying the Contents of the VCMDBC:> cd \Program Files\EMC\SYMCLI\bin

C:> symmaskdb list database


Database Type : Type6

Last updated at : 01:40:29 PM on Thu Sep 01,2007


Director Port : 0

User-generated

Identifier Type Node Name Port Name Devices

---------------- ----- --------------------------------- ---------

10000000c92741a1 Fibre 10000000c92741a1 10000000c92741a1 0040:0042

C:> cd \Program Files\EMC\SYMCLI\bin

C:> symmaskdb list database


Database Type : Type6

Last updated at : 01:40:29 PM on Thu Sep 01,2007


Director Port : 0

User-generated

Identifier Type Node Name Port Name Devices

---------------- ----- --------------------------------- ---------

10000000c92741a1 Fibre 10000000c92741a1 10000000c92741a1 0040:0042

You can display the entire contents of the VCMDB or use options to restrict the display to your area of interest. In the example above we are displaying access control records for the entries we previously added. Note: the entire output is not displayed.

You can restrict the output to a specific HBA. For example:

symmaskdb -list devs -wwn 10000000c9238053

You can also view which HBAs have been assigned to specific devices. For example:

symmaskdb list assignment -dev 0040:0043




VCM Database Maintenance

Access control is tied to the WWN of an HBA

If a HBA fails and must be replaced, the VCMDB must be updated to reflect the change– Example:symmask –wwn 20000000c920b484

replace 20000000c920b393

– Updates all masking records

Backup the VCM database– Periodically and before making changes– Example:symmaskdb –sid 123 –file VCM_backup073007 backup

Because device masking is tied to the WWN of an HBA, if it must be replaced, the VCMDB bust e updated to reflect the new WWN.




Module SummaryKey points covered in this module:

Making a Symmetrix Device available to a host involves simply connecting a cable!– Host

Install HBA and DriversPath management software

– SwitchZoning

– SymmetrixPort flagsDevice mappingDevice masking

Planning is critical!– Capacity, availability, & performance

Connecting a host to the Symmetrix involves more than connecting a cable between the host and the Symmetrix. On the host, it is necessary to physically install a supported HBA and load the appropriate device driver. The authorative list of what we support can be The EMC Support Matrix..

In most environments, a switch will be used. This is what provides the fan-out capability and allows a single FA port to be shared among many HBAs. The switch is zoned to allow or restrict access between specific HBAs and specific FA ports. Zoning is normally done using Fibre Channel World Wide Names.

On the Symmetrix, Three things need to be configured; port flags, mapping, and masking. Port flags define th4 SCSI and Fibre channel operating characteristics. Mapping assigns channel addresses to devices on FA ports. Masking grants access for specific devices to specific HBAs.

While the focus of this module has been on Fibre Channel connectivity in a Open Systems environment, the same concepts apply to iSCSI environments as well.





PowerLink




Closing Slide


Monitoring Symmetrix Activity - 1



Module 6:Monitoring Symmetrix Activity

In this module we will take a look at the Solutions Enabler commands that allow you to monitor performance activity on the Symmetrix.




Monitoring Symmetrix Activity

Upon completion of this module, you will be able to:

List objectives for performance analysis

Discuss the relationships between IO size, IO Operations per Second, and data throughput

Describe performance metrics that can be queried on the Symmetrix

Use the symstat command to display activity on the Symmetrix




Performance Monitoring

Performance monitoring is the process used to identify what is normal and to identify bottlenecks in a system

Optimal performance is achieved when workload is spread evenly across all components

Factors outside the Symmetrix significantly impact application performance

BE

FECache

Symmetrix

Performance tuning is all about identifying bottlenecks in the systems so they can be eliminated. Bottlenecks occur when one or more components are over utilized and new work becomes queued up and waits before being processed. A physical disk drive is a good example of a potential bottleneck. Because of mechanical and electrical limitations, a physical disk drive is only capable of processing a certain number of IOs in a given timeframe. If there is more work than can be processed, some work will be held up. Performance analysis is about analyzing the workload to identify these bottlenecks or hotspots. Once a bottleneck is identified, it can be removed by rebalancing the workload. Quite often, after a bottleneck is removed, another one will appear. Therefore performance analysis is a iterative process of monitor, identify, resolve.

While our focus is on the Symmetrix, there are many other factors that can be the root cause of perceived performance problems. Often times, we monitor to qualify that the problem is not with the Symmetrix. Other times real bottlenecks can be identified and tuning is necessary to resolve.

Performance can be defined by raw numbers, it is usually the user experience in real-world environments that will prompt the investigation of performance.




IO Operation - End to End Different prospective: Application, Host, SAN, Storage System

I/O might be broken into smaller units, or re-combined depending on the protocol of the end points

1 File(File System)

6 Blocks(HBA)

12 Frames(Fibre)

6 Blocks(FA)

2 Slots(Cache)

4 Writes(DA->Disk)

An I/O is a unit of work, or a complete transfer, between two end points. There are many end points in a single pathway of the Enterprise storage environment. Each one uses a different protocol which may break the received I/O into multiple I/Os before passing it on to the next end point. When considering an I/O at a particular level, always keep in mind that more than one complete transfer might be taking place at a different level to transmit that same data.

The illustration shows how a single file might be broken up during the process of storing it on a Symmetrix. The file will frequently be broken up into multiple I/Os (or blocks) when the File System Manager transfers it to the Host Bus Adapter. The HBA perceives the activity as a series of smaller work units.

The HBA will conform to Fibre Channel protocol, and transfer each I/O it receives as a series of frames of up to 2 KB to the next connectivity device. The frames are routed through the fabric to the fibre adapter (FA) on a Symmetrix, where they are re-assembled into the original I/O form seen by the HBA.

The FA will transfer the data into Symmetrix cache slots. For this example, let’s assume that the file data fits smoothly into two slots. The cache slots will be treated as single units from this point; a disk adapter (DA) will transfer each slot to the physical disk (called “destaging”) in one write operation. However, if the data is mirrored, twice as many writes will be needed to commit the data to disk.




I/O Components

All I/O operations include four stages:1. Negotiation – Communications used by both endpoints to agree to and

manage the operation.2. Header – Identifying or addressing information. 3. Data – Actual transferred data.4. Acknowledgement – Final communication to terminate the action.

Time required for negotiation, header, and acknowledgement typically fixed in size– Larger I/Os have the same overhead as smaller I/Os

DataHeaderNeg Ack

time

Transferring a single I/O between any points involves more than just transferring the data. In every data transfer protocol, several other tasks must be performed when sending an I/O. These other tasks add time to the overall process and increase the amount of data on the transmission channel. They can be thought of as additional parts or components of the I/O transfer process. The components are:

Negotiation, Acknowledgement – Both endpoints must agree to the transfer and manage the operation. This includes any “handshaking” tasks that processors use to schedule and organize the activity. Most protocols require some “setup” negotiation to start the I/O and a final “finish”message to terminate it.

Header – Some identifier or address value that the receiving endpoint uses to properly utilize the data. Any CRC or parity bits used to error check the data can be considered header also, since this overhead adds to the information load on the channel.

Data – The actual data. All of the other components are added to the I/O during transfer and removed before the data can be utilized.

The Negotiation, Header, and Acknowledgement components are largely fixed in size regardless of I/O size. A 2 KB I/O requires about the same negotiation and header information as a 32 KB I/O. A change in the I/O size typically means a change in the Data component only.

The illustration graphically shows the parts of an I/O on a time graph. At the start of the I/O, some time is taken for the Negotiation, then time is taken for the I/O header, etc.




I/O Per Second

I/O Per Second (IOPS)– Number of transfers per second– An important measure of I/O, especially when large numbers

must be transferred to meet service levels, e.g. database.

I/O Per Second

Time to completely transmit I/O, or channel cost of

transmission

I/O per second (IOPS) measures the number of transfers per second between end points. The most common end points considered for this measurement are a host and its storage. As data is broken into blocks and transferred to and from disk, each is counted as an I/O. IOPS can also be measured at more focused points in the data transfer, such as between a caching system and the physical disks.

IOPS is a good measure when the I/O sizes are small, especially when large numbers must be transferred to meet service levels. This is often true in OLTP (On Line Transaction Processing) environments: a high IOPS measure indicates that many database transactions (typically one or more I/O each) can be serviced per second.




ThroughputThroughput

– Volume of Data transferred per secondOverhead for negotiation, header, etc ignoredLarger I/Os , provide higher throughput

– Frequently used as the capacity rating of channele.g. 200 MBps Fibre, 100 MBps iSCSITheoretical maximum for all traffic on the channel, including headers and negotiation

Measured Throughput

Actual cost of data transfer

Throughput measures the volume of data transferred per second through an I/O channel. All commonly used performance measurement tools report only the data transferred when measuring throughput, since this is the “useful” part of the I/O. The header and negotiation “overhead” are ignored even though bytes of content must be transferred across the channel to complete these tasks as well.

Throughput is a useful measure when I/O is large, especially when the time taken to transfer the overall volume of data needs to be minimized. Backup is a good example of this sort of activity. The total number of individual I/Os is not a concern, the total time to move the backup data archive is.

A “throughput” value is often used to rate the speed of a channel: 200 MBps Fibre, 100 MBpsiSCSI, etc. This is more accurately termed “bandwidth.” This number is a theoretical maximum for all traffic on the channel, including headers and negotiation. Since performance measurement tools report the data volume and ignore header and negotiation, measured throughput can not reach the rated maximums in practice. Additionally, processors at the endpoints will often require some calculation time between requests, further ensuring that the maximum ratings can not be achieved under real conditions.




IOPS, Throughput, and I/O Size Relationship

IOPS, and throughput are inversely related– Block size is a key factor

Small I/O

IOPS: 5

Through:

Large I/O

IOPS: 2

Through:

As I/O size increases, IOPS decreases

As I/O size increases, Throughput increasesI/O Size

IOPS Throughputhigh

low

The I/O per second, throughput, and I/O size measures are closely related.

When small I/Os are transferred less time is taken on each I/O, increasing the number that can be moved in a given time period. The IOPS measure is typically large for small I/Os. However, the data payload is small when compared to the header and negotiation factors. So while many I/Os are being exchanged, the total data throughput is small.

When large I/Os are transferred more time is taken on each I/O, decreasing the IOPS measure. However, a larger percentage of the channel resources are spent on data rather than header and negotiation. This increases the overall throughput. This is illustrated in the block diagrams above. The combined data parts of the large I/O example is roughly 50% larger than the small I/O example.

The bottom graph in the illustration shows this relationship. As the I/O size being transferred increases the IOPS measure drops. If the number of transfers per second is the important measure of performance in this environment, then smaller I/O size is beneficial. Databases typically set a size for their I/O transfers that maximizes this benefit.

The graph also illustrates how throughput will increase as the I/O size increases, even as the total number of I/Os goes down. File copy and backup operations will take advantage of this relationship by increasing the I/O size.




Response TimeLength of time taken to process an I/OOften the only real measure of performance problems– If response time increases to noticeable levels, you have a problem

Symmetrix response time measured by front-end directorHost response time measured by host operating system

App Host queue

HBA Connectivity FA Cache DAOS Disk

Symmetrix measuresHost measures

Response time is another measure of performance that is often used in parallel with measures like I/O per second and throughput. Response time measures the total time taken to process an I/O. This is often very useful in detecting application slowdowns. The speed of an application that relies on disk access is defined by the access response time, so when it increases, the application necessarily must slow down.

Symmetrix response time is measured by the front end directors. The time of the start of the I/O negotiation by the director to the time of the final acknowledgement is the Symmetrix response time. Therefore, these measures omit any time taken in SAN connectivity devices, HBA delays, or host queuing.

Host response times are measured by the host operating system. The time of the start of the application’s request to the time of the operating system’s final acknowledgement to the application is the host response time. This measure includes queuing, HBA and connectivity delays, as well as the entirety of the Symmetrix time.




Symmetrix StatisticsThe Symmetrix maintains counters that provide statistics on system activity – IO per second and throughput– Front-end directors and ports– Back-end directors and ports– Disks drives– Cache hits and misses– Write Pendings– Other metrics

Solutions Enabler symstat command querying these counters allow real-time reporting of activity

Counters are leveraged by other tools such as Work Load Analyzer to provide historical view of activity

The Symmetrix array maintains statistical counters for performance-critical operations. Using the symstat command, you can view the performance statistics derived from these counters.

Real-time vs. historical:Real Time−What’s the current workload on the Symmetrix?

Historical−When do peaks occur?−Will peaks overlap?−Will peaks be distributed




Reporting Objectssymstat reports statistics on how object perform in an environment:– Devices

Throughput and I/O statistical on a device or set of device within a device groupRead/Write OperationsCache statistics including hits and misses Other statistics including dynamic mirroring service policy (DMSP)

– Directors Similar statistics but on director, director port, or type of director basisPrefetching activity

– Disk Reports Back-end I/O requests and throughput for selected disks

– SRDF/A sessions CYCLE, CACHE, and REQUESTS

The statistics command (symstat) performs the following:Queries Symmetrix devices to capture raw performance counts and store them in memory.Retrieves the performance counts for the Symmetrix array as a whole.Retrieves the performance counts for a director or director port. Retrieves the performance counts for one or more Symmetrix devices.Retrieves the performance counts for one or more Symmetrix device groups, composite groups, or RDF groups.Retrieves the performance counts for a selection of, or all, Symmetrix disks.Retrieves the timestamp of the performance count sample.Retrieves and displays replication session statistics for SRDF/A.Retrieves GigE iSCSI network statistics.




Types of StatisticsREQUESTS

– Reports I/O requests and throughput for selected devices, directors, or SRDF/A sessions– Default type if no type is specified

BACKEND – Reports back-end I/O requests and throughput for selected devices

PORT – Reports performance statistics for a director port

ISCSI – Report GigE network statistics

CACHE – Reports cache activity for selected front-end or remote link directors, or SRDF/A sessions

MEMIO – Reports cache memory to disk activity for selected devices

DISK – Reports back-end I/O requests and throughput for selected disks

PREFETCH – Reports track prefetch disk activity for selected back-end directors only

DMSP – Reports dynamic mirroring service policy (DMSP) statistics for the selected device(s)

Other symstat statistics include:

CYCLEReport cycle summary information for SRDF-A sessions.

RDFReports SRDF statistics

PATH Report R-Copy path information for non incremental sessions.




REQUESTS – Front-end ActivityMost common type of statistic – Default for symstatcommand– I/O requests –I/O Operations per second– Throughput - Data transferred

Real-time statistical information for:– Entire storage system– Device– Device group, – SRDF/A sessions

Specify the interval and the number of samples (counts)

Example: symstat -i 5 -c 5 -sid 054

The default type of statistics for the for the symstat command is requests, therefore if a type is not specified with the –type option, requests are presented. The -c argument defines the number of samples. The default for this argument is continuous sampling. If you do not specify this argument, but you specify an -i value, the command produces a continuous statistical output, requiring a cancel (Ctrl-C) to stop the process. If no sample interval is specified, the default sample interval is 10 seconds; the minimum is 5 seconds. It is recommended that 30 seconds or greater be used for effective statistical sampling.

To filter the information to display only information of interest, you can use the –g and specify a device group.

symstat -i 5 -c 3 -g production_dg




I/O Requests - symstat –type request

C:\>symstat -i 5 –c 5

DEVICE IO/sec KB/sec % Hits %Seq Num WP

07:07:01 READ WRITE READ WRITE RD WRT READ Tracks

07:07:06 0021 (DRIVE2) 538 255 8617 4080 100 100 0 20810

0022 (DRIVE3) 488 241 7808 3865 98 100 0 19086

0023 (DRIVE4) 566 265 9062 4249 100 100 0 20002

0024 (DRIVE5) 195 98 3139 1571 77 100 0 1860

------ ------ ------- ------- --- --- --- ------

Total 1787 859 28626 13765 97 100 0 61758

07:07:12 0021 (DRIVE2) 527 253 8441 4051 100 100 0 20610

0022 (DRIVE3) 500 242 8006 3872 99 100 0 19020

0023 (DRIVE4) 538 268 8608 4294 100 100 0 20470

0024 (DRIVE5) 224 108 3593 1731 80 100 0 2072

------ ------ ------- ------- --- --- --- ------

Total 1789 871 28648 13948 97 100 0 62172

C:\>symstat -i 5 –c 5

DEVICE IO/sec KB/sec % Hits %Seq Num WP

07:07:01 READ WRITE READ WRITE RD WRT READ Tracks

07:07:06 0021 (DRIVE2) 538 255 8617 4080 100 100 0 20810

0022 (DRIVE3) 488 241 7808 3865 98 100 0 19086

0023 (DRIVE4) 566 265 9062 4249 100 100 0 20002

0024 (DRIVE5) 195 98 3139 1571 77 100 0 1860

------ ------ ------- ------- --- --- --- ------

Total 1787 859 28626 13765 97 100 0 61758

07:07:12 0021 (DRIVE2) 527 253 8441 4051 100 100 0 20610

0022 (DRIVE3) 500 242 8006 3872 99 100 0 19020

0023 (DRIVE4) 538 268 8608 4294 100 100 0 20470

0024 (DRIVE5) 224 108 3593 1731 80 100 0 2072

------ ------ ------- ------- --- --- --- ------

Total 1789 871 28648 13948 97 100 0 62172

The default type of statistics for the for the symstat command is requests, therefore if a type is not specified with the –type option, requests are presented.




Understanding Write PendingGlobal Memory is not an infinite resource

Some memory is used by the system, the rest is available for cache slots – With DMX3 Cache is allocated in 64K slots– Read Cache– Write Cache

No more than 80% of total cache slots

Cache is a shared resource– All I/O operations goes through cache– Allocated with fairness to prevent a few

very active devices from consuming all available cache

Global DataTrack Tables

CacheSlots

Global Memory

Cache is the heart of a Symmetrix. Every I/O must go through cache, whether it is a read or a write. From a users data perspective, cache serves two primary functions.

Maintains recently accessed data making it readily available. Statistically, users will access data they most recently used. The longer data resides untouched, the less likely it is to be accessed again. The system uses Least Recently Used (LRU) algorithms to determine which cache slots to replace and which ones to keep.Buffer writes until they can be destaged to a persistent location on disk.. When a host writes data to the DMX, it is stored in cache, and the host is notified the data is saved. Data is then destaged to disk as a background process while the Symmetrix systems is less utilized.

The total size of cache is determined by the number and size of the memory boards in the Symmetrix. DMX systems can have a maximum of 8 boards that are up to 64GB each for a total of ½ TB.,

The IMPL.bin that defines how cache is laid out and allocates three area; the Common Area, Device Table Area and Cache Slot area.

There is one Device Table for every configured logical in the Symmetrix. The tables are used as a look up space for each track in the system. The aim of the tables is to provide direct access to a given block of data, either in cache (if it is there) or on the drives, in the shortest time possible. The table space starts at the end of the common area of global memory and does not end until each configured logical volume is mapped. This part of global memory is directly mapped, and the structures are built during the IMPL of the Symmetrix. As we have just mentioned the logical volumes are mapped into the direct mapping area of cache.




Write Pending Limits

Three types of write pending limits:– System– DA– Device

symcfg list -v

<...>

Symmetrix ID: 000190300180Time Zone : EDTProduct Model : DMX3-6Symmetrix ID : 000190300180Microcode Version (Number) : 5771 (168B0000)Microcode Date : 06.22.2006Microcode Patch Date : 06.22.2006Microcode Patch Level : 86Cache Size (Mirrored) : 16384 (MB)# of Available Cache Slots : 460798# of PermaCache Slots In Use : 6Max # of System Write Pending Slots : 368998Max # of DA Write Pending Slots : 184498Max # of Device Write Pending Slots : 18440

…

symcfg list -v

<...>

Symmetrix ID: 000190300180Time Zone : EDTProduct Model : DMX3-6Symmetrix ID : 000190300180Microcode Version (Number) : 5771 (168B0000)Microcode Date : 06.22.2006Microcode Patch Date : 06.22.2006Microcode Patch Level : 86Cache Size (Mirrored) : 16384 (MB)# of Available Cache Slots : 460798# of PermaCache Slots In Use : 6Max # of System Write Pending Slots : 368998Max # of DA Write Pending Slots : 184498Max # of Device Write Pending Slots : 18440

…

When a user writes to any device in the Symmetrix system, the data is write pending (WP) until it is destaged from the cache to the physical disk. This acts as a buffer for the writes, and enables the system to accept writes faster then disk speeds. Under normal circumstances, the Symmetrix system will not destage data to disk immediately after it’s written to cache. This enables time for the intelligent algorithms to optimize the order of writes to the disk. Since most applications write with a high locality of reference, it can dramatically reduce the I/O traffic to the back-end disks.

When write requests coming into the Symmetrix arrive faster than they can be destaged on the back-end, performance problems can occur. This is especially true when there are a large number of writes to a few volumes.

The system imposes a limits on the:Logical Volume Write-pending (LVWP), about 5% of available write cache. When a devicereaches the LVWP ceiling, each new write to a track that is not already write pending for the device will trigger a special task. This special task waits for write pending data to be destaged to disk and a slot to become free before the new write is accepted. System-wide write-pending (80% of usable cache). When the Symmetrix is at the systemwide write-pending limit, and you write to a track that is not already write pending, each new write will trigger a special task. This task waits for write pending data to be destaged to disk and a slot to become free before the new write is completed.




Type Memorysymstat -i 5 -c 3 -type memio

C:\>symstat -i 5 -c 3 -type memio

DEVICE Tracks/sec % Dev07:17:12 WP Tracks Prefchd Destgd WPmax

07:17:17 0021 (DRIVE2) 4490 0 0 110022 (DRIVE3) 4402 0 0 110023 (DRIVE4) 4548 0 0 110024 (DRIVE5) 4416 0 0 110025 (DRIVE6) 1406 4 98 30026 (DRIVE7) 1042 2 86 20027 (DRIVE8) 1032 8 93 2

------ ------ ------ ------21336 14 277 7


------ ------ ------ ------26670 14 250 9

C:\>symstat -i 5 -c 3 -type memio

DEVICE Tracks/sec % Dev07:17:12 WP Tracks Prefchd Destgd WPmax


------ ------ ------ ------21336 14 277 7


------ ------ ------ ------26670 14 250 9

Type Memory IO shows cache statistics including Write pendings, and prefetch activities




Type Cachesymstat -i 5 -sa all -type cache

C:\>symstat -i 5 -sa all -type cacheDIRECTOR Misses/sec Disconnects/sec

07:19:48 Cache Read System Device07:19:54 FA-8A 217 0 007:19:59 FA-8A 207 0 007:20:04

DIRECTOR Misses/sec Disconnects/sec07:20:09 Cache Read System Device07:20:09 FA-8A 196 0 007:20:1507:20:20 FA-8A 184 0 007:20:2507:20:30 FA-8A 174 0 0

DIRECTOR Misses/sec Disconnects/sec07:20:35 Cache Read System Device07:20:35 FA-8A 164 0 007:20:4107:20:48 FA-8A 106 0 007:20:53 FA-8A 133 0 0

C:\>symstat -i 5 -sa all -type cacheDIRECTOR Misses/sec Disconnects/sec

07:19:48 Cache Read System Device07:19:54 FA-8A 217 0 007:19:59 FA-8A 207 0 007:20:04

DIRECTOR Misses/sec Disconnects/sec07:20:09 Cache Read System Device07:20:09 FA-8A 196 0 007:20:1507:20:20 FA-8A 184 0 007:20:2507:20:30 FA-8A 174 0 0

DIRECTOR Misses/sec Disconnects/sec07:20:35 Cache Read System Device07:20:35 FA-8A 164 0 007:20:4107:20:48 FA-8A 106 0 007:20:53 FA-8A 133 0 0

Cache misses often occur with random workload and require the date be staged from disk beforthe I/O operation completes.




Front-end DirectorFront-end director board has four independent processors– Loaded with specific emulation code

ESCON – EAFICON – EFFibre Channel – FAISCSI – SE

– Each processor has one or more ports – Every director has a direct connection to

every global memory board

Performance activity can be measured at the Processor level or the port level

8D

8C

8B

8A

9D

9C

9B

9A

Direct access channels to cache

Front-end directors, also called channel directors are normally installed in pairs, providing redundancy and continuous availability in the event of repair or replacement. Each front-end director has multiple microprocessors and supports multiple independent data paths to the global memory to and from the host system.




Director Request symstat -i 10 -c 3 -dir all

C:\>symstat -i 10 -c 3 -dir allDIRECTOR IO/sec Cache Requests/sec % RW

07:25:42 READ WRITE RW Hits

07:25:52 DF-1A 264 0 254 255 0FA-8A 3435 2133 2120 4254 100DF-16A 140 0 133 133 0DF-1B 171 0 164 164 0DF-16B 150 0 143 144 0DF-1C 272 2 257 259 0DF-16C 145 1 136 138 0DF-1D 303 2 285 288 0DF-16D 284 2 269 271 0

------ ------ ------ ------ ---Total 5164 2140 3761 5906 72


------ ------ ------ ------ ---Total 5123 2229 3686 5919 75

C:\>symstat -i 10 -c 3 -dir allDIRECTOR IO/sec Cache Requests/sec % RW

07:25:42 READ WRITE RW Hits


------ ------ ------ ------ ---Total 5164 2140 3761 5906 72


------ ------ ------ ------ ---Total 5123 2229 3686 5919 75

Front-end and back-end requests




Portsymstat -type PORT -dir -port

C:\>symstat -i 10 -type PORT -dir 8a -port 0

07:39:44 DIRECTOR PORT IO/sec Kbytes/sec07:39:44 FA-8A 0 4675 7477007:39:54 FA-8A 0 4612 7364007:40:05 FA-8A 0 3244 5180207:40:15 FA-8A 0 4715 7540407:40:25 FA-8A 0 3293 52668


C:\>symstat -i 10 -type PORT -dir 8a -port 0



Individual port statistics are valuable when analyzing response time issues.




Back-end Components

Logical Device– Traffic routed to that device, no matter

how it is physically stored

Disk Director – Traffic handled by the processor for

all disks, including protection writes and business continuity synchronization

Disk– Traffic to/from all hyper volumes on

the disk, including protection, BC, and internal scrubbing

DA DA

device

directors

ports

disks

hyper volumes

DA-1A DA-16A

There are three components involved with processing an I/O from the back-end prespective:

The logical device

For the most part, I/Os from the host to a particular logical device are observed and measured at the device level. So a standard measure like I/O per second indicates the number of I/O generated by the hosts accessing the device. However, some device measures show the back end traffic associated with the device. These measures are typically prefixed with “DA.”

The Disk Director

Traffic measured at the disk director level shows all traffic to/from all of the physical disks controlled by the director. This includes host I/O, protection writes to mirrored and RAID devices, and internal synchronizations like those performed by TimeFinder and Optimizer.

The Disk

Physical disk traffic is measured at the disk level. All traffic for the hyper volumes stored on a physical disk are measured here. As with the disk adapter level, traffic due to protection and internal synchronizations will also be visible in these metrics. Additionally, traffic performed by internal scrubbing (error detection and correction) will also be apparent here.




BACKEND – Back-end Activitysymstat -type backend

C:\>symstat -type backend -i 10DEVICE IO/sec KB/sec Prefetched Tracks

07:43:50 READ WRITE READ WRITE Tracks Used

07:44:00 0021 (DRIVE2) 0 57 0 5435 0 00022 (DRIVE3) 0 66 0 5700 0 00023 (DRIVE4) 0 85 0 5928 0 00024 (DRIVE5) 0 91 0 6076 0 00025 (DRIVE6) 0 92 0 5871 0 00026 (DRIVE7) 0 65 0 5679 0 00027 (DRIVE8) 0 100 0 7829 0 00036 (Not Visible ) 0 0 51 0 0 00037 (Not Visible ) 0 0 76 0 0 00038 (Not Visible ) 0 0 25 0 0 00039 (Not Visible ) 0 0 57 0 0 0003A (Not Visible ) 0 0 76 0 0 0003B (Not Visible ) 0 0 128 0 0 0003D (Not Visible ) 0 0 57 0 0 0003F (Not Visible ) 0 0 76 0 0 0

------ ------ ------- ------- ------ ------Total 0 556 546 42518 0 0

C:\>symstat -type backend -i 10DEVICE IO/sec KB/sec Prefetched Tracks

07:43:50 READ WRITE READ WRITE Tracks Used

07:44:00 0021 (DRIVE2) 0 57 0 5435 0 00022 (DRIVE3) 0 66 0 5700 0 00023 (DRIVE4) 0 85 0 5928 0 00024 (DRIVE5) 0 91 0 6076 0 00025 (DRIVE6) 0 92 0 5871 0 00026 (DRIVE7) 0 65 0 5679 0 00027 (DRIVE8) 0 100 0 7829 0 00036 (Not Visible ) 0 0 51 0 0 00037 (Not Visible ) 0 0 76 0 0 00038 (Not Visible ) 0 0 25 0 0 00039 (Not Visible ) 0 0 57 0 0 0003A (Not Visible ) 0 0 76 0 0 0003B (Not Visible ) 0 0 128 0 0 0003D (Not Visible ) 0 0 57 0 0 0003F (Not Visible ) 0 0 76 0 0 0

------ ------ ------- ------- ------ ------Total 0 556 546 42518 0 0

Displays performance statistics at the back-end DA




Symmetrix Prefetch

DA detects two sequential reads, considers probability of sequence continuing, begins prefetch task for device

Can have many simultaneous prefetch tasks

Two tracks are read into cache ahead of read stream

As sequential stream continues, more read ahead: up to 32 tracks

Prefetch task remains for an additional 120 seconds after sequential I/O stops to avoid thrashing

Enginuity’s Prefetch algorithm collects statistics about the sequential patterns of each logical volume. Prefetch calculates the probabilities that the host will read subsequent tracks (those that follow the track that the system just read) in the very near future. Based on these probabilities and system utilization factors, Enginuity decides whether to prefetch more data, how many tracks to prefetch, when to prefetch, and for how long to keep the prefetched data in cache. The system collects statistics on an ongoing basis so that the algorithm can adjust itself to rapid changes in the workload.

The Symmetrix prefetching algorithm is carried out by the Disk Directors. When two sequential reads are detected, a prefetch task is started for the device.

Depending on the Enginuity level, eight or more prefetch tasks can be active on a single device at a time. Considering the number of devices in a modern Symmetrix array, this means that the disk directors can be monitoring a considerable number of prefetch tasks at once.

Once the task begins, the disk directors try to prefetch data into cache at least two tracks in advance of the current read address. As the sequential stream continues, more read ahead will be performed up to a maximum of 32 tracks.

Once the sequential read stream stops, the prefetch task will remain monitoring the activity for another 120 seconds before terminating. If the sequence begins again before the 120 seconds has elapsed, the same task will continue the prefetch work. This prevents tasks from being a performance issue by starting and stopping frequently.




Prefetchsymstat -type prefetch

C:\>symstat -i 5 -c 4 -sid 254 -da 2a -type prefetch

DIRECTOR Preftch Preftch Trk/sec

12:08:20 Trk/sec Used Unused

12:08:25 DA-2A 1 1 0

12:08:30 DA-2A 7 6 1

12:08:36 DA-2A 19 19 0

C:\>symstat -i 5 -c 4 -sid 254 -da 2a -type prefetch

DIRECTOR Preftch Preftch Trk/sec

12:08:20 Trk/sec Used Unused

12:08:25 DA-2A 1 1 0

12:08:30 DA-2A 7 6 1

12:08:36 DA-2A 19 19 0




Physical Disks

Physical disk are the slowest link in the I/O Chain– Positional delay– Rotational delay

Physics limits the speed of a drive– Parallelism overcome physical limits– Best performance is achieved using

more smaller drives

Key to performance is balance– Utilize as many as possible

But the best performance is achieved by servicing I/O request from memory rather than disk– Benefits of cache

Any given worlkoad is going to put some level of stress on the backend. Understanding the backend activity and the workload characteristics is critical in determining the number of spindles necessary.




Disksymstat -type disk

C:\>symstat -type disk -i 10 -sid 254DISK IO/sec KB/sec

08:02:40 READ WRITE READ WRITE08:02:50

01A:C2 1 0 76 001A:C4 0 0 51 001A:C6 0 151 38 586401A:D1 0 0 32 001A:D3 0 0 25 001A:D5 0 149 0 574916A:C3 1 0 76 016A:C5 0 0 51 016A:D2 0 0 32 016A:D4 0 120 6 4574

…01C:C4 0 163 6 630201C:C6 0 0 25 001C:D1 1 0 76 001C:D3 0 0 51 001C:D5 0 151 0 574416C:C3 0 0 32 016C:C5 0 139 25 532016C:D2 1 0 76 0…16D:D5 0 166 0 6332

Total ------ ------ ------- -------8 1994 1648 76750

C:\>symstat -type disk -i 10 -sid 254DISK IO/sec KB/sec

08:02:40 READ WRITE READ WRITE08:02:50

01A:C2 1 0 76 001A:C4 0 0 51 001A:C6 0 151 38 586401A:D1 0 0 32 001A:D3 0 0 25 001A:D5 0 149 0 574916A:C3 1 0 76 016A:C5 0 0 51 016A:D2 0 0 32 016A:D4 0 120 6 4574

…01C:C4 0 163 6 630201C:C6 0 0 25 001C:D1 1 0 76 001C:D3 0 0 51 001C:D5 0 151 0 574416C:C3 0 0 32 016C:C5 0 139 25 532016C:D2 1 0 76 0…16D:D5 0 166 0 6332

Total ------ ------ ------- -------8 1994 1648 76750

The type DISK displays I/O requests and throughput on a physical disk. The drive is identified by the DA director number, interface ID, and SCSI ID of the disk drive





The Symmetrix maintains statistical counters of activity in the array– Querying these counters allow real-time reporting of activity

Solutions Enabler symstat command

– Allows real-time monitoring of activity on the Symmetrix– Complements other tools such as Workload Analyzer

Keep in mind…– Individual statistics are of little value when taken alone

Analysis Relationships

– What is good in one environment could be a problem in another– There is more in the I/O stack than just the Symmetrix that can impact

performance





PowerLink




Closing Slide


Symmetrix Security - 1



Module 7:Symmetrix Security

Welcome to the last module of Symmetrix Monitoring and Management In this module we will take a look at the Solutions Enabler security features including user-based and host-based access controls, and auditing facilities.




Symmetrix Approach to SecurityThree prong approach:– Physical Security

Restricted data center accessPasswordsSecure Service Credentials (SSC) on the Service Processor

– Access controls for configuration and controlHost based using symaclUser based using symauth

– Monitoring - Who did what when?symauditsymevent

In addition to access controls provided by Volume Logix and fabric zoning

The first defense in security is to control physical access to the systems. All the access control and auditing will not help if the Symmetrix is open to physical threat. Nearly all enterprise customers restrict access to the data center and today, with Secure Service Credentials on the service processor, even if a person could physically get to the Symmetrix, they couldn’t log into the Symmetrix and invoke configuration changes or control operations.

Not all threats require physical access to the Symmetrix. An end user on a host attached to the Symmetrix could invoke commands to cause accidental or malicious data loss or unauthorized access. The Symmetrix includes host-based and user-based access control that enables only the required level of access but not more access than required.

Also important to security is understanding who is doing (or attempting) to do what operations on the system.

These protections are in addition to the host access controls provided by device masking using Access Logix and fabric zoning.




Host-based Access Control - symaclCaution! Anyone with host access to a can execute any configuration and control function on any Symmetrix device– Fibre Channel connection to the Symmetrix– At least on device visible to serve as a Gatekeeper– Solutions Enabler installed and appropriate license

Solution:– Initialize and manage Symmetrix Access Control – Restrict management access so only specific hosts can perform specific

configuration and operations on specific devices– Does not limit or restrict normal I/O Operations

Results:– Secure sharing of a system– Eliminate accidental or intentional tampering– Support change control process by restricting ad hoc operations

Anyone with access to Symmetrix-based management software can execute any function on any Symmetrix device. Applications can issue management commands through any device in a Symmetrix and invoke configuration or control commands. Open Systems hosts can manipulate mainframe devices, Windows hosts can manipulate UNIX volumes, and vice versa. Shared storage systems, are vulnerable to either accidentally or intentionally tampering.

To prevent this, the symacl command can be used by an Symmetrix administrator to restrict host access to defined sets of devices and specific operations

When configured and an unauthorized host attempts to run a control operation, the SYMCLI command will return a message similar to the following: Symmetrix Access Control Denied the Request, to the consol and log the unauthorized use in the SYMAPI log, and the Symmetrix Audit log.

SYMACL is a feature of the Symmetrix and Enginuity operating environment beginning with Version 5x67 and Solutions Enabler 4.3 and later.

Note: Once Access Control have been implemented, additional steps must be added to customers Change Management Process. Provisioning operations require access control changes!




symacl Key ComponentsUnique host identifier (Access ID)– Encrypted hexadecimal number– Derived from the physical hardware

and software configuration – Created using the symacl –unique command

PIN number known only to authorized Access Control Administrators

Access Control database in the Symmetrix– Maintains the definitions of what hosts can do what

operations on what devices– Resides in the Symmetrix File System

No direct user access– Managed only through an authorized host and by a

administrator that has the PIN

HostAccess ID:

12345678-ABCDEF12-87654321

Symmetrix

Access Control Database

Access Control Database

PIN

Hosts with management rights to the Symmetrix are identified with an Access ID, an encrypted hexadecimal number. These unique identifiers are derived from the system configuration. An example of an Access ID is: 46712F2A-AF54C32A-FE365D1C. By uniquely identifying a host, it is possible to target specific rights to specific host on a system-by-system basis. If a host cannot produce a unique ID, it will be considered Unknown and will be provided limited or no configuration and control access.

Because this unique ID is derived from the physical configuration, any significant changes to host hardware may result in a change in ID, requiring the regeneration of the ID and updating the access control entries.

When Access Controls is first configured, an access pin is created. When an Administrator attempts to execute an access control configuration command, the user will be prompted to enter the PIN. The PIN can also be set in the environment variable SYMCLI_ACCESS_PIN.

All access control data is maintained in a Access Control database that resides in the Symmetrix File System.




Configuration Objects

Access Control GroupHosts identified by unique ID

Access Control GroupHosts identified by unique ID

Access PoolSet of Symmetrix Devices

Access PoolSet of Symmetrix Devices

111111112112

113113114114

115115

121121122122

123123124124

125125

131131132132

133133114114

135135

Access Control List (ACL)

ACEACEACE

Access Control List (ACL)

ACEACEACE

Control

Access Control Group– Group of one or more host– Similar access requirements– Each host identified by unique

Access ID Access Control List– Access Control Entries (ACE)– Level of Access (Permissions)

Allowing hosts in Access Control Group to control devices defined in Access Pools

Access Pool– List of Devices

Host A Host B Host C ...

When configuring access control, there are three objects that are created: Access Control Groups, Access Pools, and Access Control Entries.\

Access Control Groups — Are logical grouping of hosts with similar needs (determined byan Administrator) identified by Unique access IDs and names. Access groups are allowed to act on Access Pools based on permissions (access types) granted by the Administrator. The unique host ID for open systems can be viewed by running symacl -unique.Access pools —A set of Symmetrix devices. Permissions (or level of access), are assigned to allow a host to perform certain Solutions Enabler functions on a specified set of devices. These sets of devices are referred to as Access Pools or accpool.Access Control Entries (ACEs) — Once the group and access pool mechanisms are established, the access control entries (ACEs) are created, which grant permissions to these pools. The ACEs along with Access Control Groups and Access Control Pools are managed using the symacl command and stored in the access control database. Access Control Lists (ACLs) consist of Access Control Entries (ACEs




Access Control DatabaseManagement

Host

Symmetrix

FA FA

Symmetrix File SystemSymmetrix File System

GKGK

ACLDB

Resides in the Symmetrix File System– No direct user access– Initial configured on the Service Processor– Configured and maintained using symacl commands

From authorized hosts Only by administrators with the Access PIN

The size of the SymACL database is currently 32KB. This is large enough for 320 entries based on an average entry size of 100bytes. This is large enough for most environments. The actual space used can be seen with the symacl list –v command.

The entries that are high lighted are what is normally created as part of the initial default set up.




Access Controls Setup

Initial setup is performed by an EMC Engineer on site 1. Enable access control in the binfile (5771 and prior)2. Initialize the Access Control database in the Symmetrix File System

– Creates a default Access Groups (AdminGrp and UnknwGrp)– Creates default Access Pools (ALL-DEVS and !INPOOLS)

3. Generate a unique host identifier for the administration host(s)– Add a minimum of one administrative host (two or more would be better) – Add Host Access Control ID to AdminGrp Access Group

4. Create an access ID PIN– Add User Access Control ID

SymACL requires an EMC CE using SymmWin to enable SymACL to set up the firstAdministrator Group. The following procedure must be completed on every Symmetrix. To implement a minimal SymACL configuration:1. For Enginuity code 5x71 and earlier, perform an online configuration change to the BIN file

with the 'Enable Access Control - Yes' initialization setting. Note: This step is not required for Enginuity releases 5x72 and later where this setting is enabled by default.

2. Initialize the SymACL database on the Symmetrix system via the SymmWin Procedure Wizard.

3. Obtain a unique identifier from a host (note two admin hosts recommended) that will be the primary and/or secondary admin host(s). This is done by running the symacl –uniquecommand on the admin host.

4. Create an administrative user from the SymmWin Procedure Wizard.This procedure configures the admin host and creates a PIN. At this point you've enabled ACLand configured an ACL admin host. By default, all other hosts have full access (via theUnknown group), unless expressly restricted via access controls.




Enable Access Control

Must be done in the binfile with 5771 and prior– Enabled by default with 5772– Change requires Online Configuration Change

The initial setup is normally performed by EMC Engineers onsite but is shown here for reference only. Starting with 5772, the step of enabling access control is not required as it is enabled by default.




Initialize Access Control

From SymWin Procedure Wizard– Initializes the ACL database– Creates default Access Control Groups and Access Control Pools

The initial setup is normally performed by EMC Engineers onsite but is shown here for reference only. This steps creates the ACL database in the Symmetrix File system and initializes it with the default access control groups and Access Control pools.




Add Administration Host Access ID

Identify Access ID for Administration host

Using symwin on the Service Processor

C:\>symacl -unique

The unique id for this host is: 2C5F00A6-54408FC0-92397573

C:\>symacl -unique

The unique id for this host is: 2C5F00A6-54408FC0-92397573

The initial setup is normally performed by EMC Engineers onsite but is shown here for reference only. At least one host must be identified as the administration host and added to the Admin Access Control Group.




Add Admin User Access ID (PIN)

Create a User entry and PIN

User BasedHost BasedQuit

The initial setup is normally performed by EMC Engineers onsite but is shown here for reference only. An administrator Personal Identification Number (PIN) is also created during initialization.




Verify Access Control Status

Access control must be enabled in the bin file– On-line Config

Change– Cannot use symconfigure

C:\>symacl list –v

SYMMETRIX ACCESS CONTROL STATUS


Access Control : EnabledSession Locked : NoTime Held in Seconds : 0Lock Identifier : NoneTime Enabled : Fri Aug 10 08:31:17 2007Time Disabled : Thu Jan 01 00:00:00 2004Time Updated : Fri Aug 10 08:31:17 2007ADMIN priv : YesADMINRD priv : N/ADB space remaining : 31756 bytes

C:\>symcfg list –v

Symmetrix ID: 000190102254 (Local)…

Access Control Configuration State : Enabled…

C:\>symacl list –v

SYMMETRIX ACCESS CONTROL STATUS


Access Control : EnabledSession Locked : NoTime Held in Seconds : 0Lock Identifier : NoneTime Enabled : Fri Aug 10 08:31:17 2007Time Disabled : Thu Jan 01 00:00:00 2004Time Updated : Fri Aug 10 08:31:17 2007ADMIN priv : YesADMINRD priv : N/ADB space remaining : 31756 bytes

C:\>symcfg list –v

Symmetrix ID: 000190102254 (Local)…

Access Control Configuration State : Enabled…

symacl –v is a quick way of determining if ACL is enabled. You can also see this attribute by performing a symcfg list –v command.




Default Access Control Environment On the administration host verify that default Access Control configuration has been enabled and the host has administrative authority– AdminGrp

Admin permissionsto All DevicesAll Permissionsto All Devices

– Unknown GroupBase Permissionsto All DevicesAll Permissions to All Devices that are not currentlyin Access Pools

C:\>symacl list -acl


Group Name Pool Name Access Type-------------- ------------ -----------AdminGrp ALL_DEVS ADMIN AdminGrp ALL_DEVS ALLUnknwGrp ALL_DEVS BASE UnknwGrp !INPOOLS ALL



Group Name Pool Name Access Type-------------- ------------ -----------AdminGrp ALL_DEVS ADMIN AdminGrp ALL_DEVS ALLUnknwGrp ALL_DEVS BASE UnknwGrp !INPOOLS ALL

With the default Access Control configuration, until other access controls are configured, all hosts would continue to have full permissions to perform all operations on all devices. The recommended implementation strategy is to gradually phase in Access Controls one application at a time and continue to add appropriate access controls until the Symmetrix is locked down appropriately. Not all devices need to be placed under control of Access Control.




Securing the Symmetrix

Planning is critical!– Full assessment of Host and Application environment required

Best Practices is to gradually phase in implementation1. Create Access Control Groups

Add one or more hosts2. Create Access Control Pools

Add devices to pool3. Create Access Control Entries

Grant BASE rights to the Access Group for All DevicesGrand specific feature access rights to the Access Group for theAccess Pool

When access controls are full configured, remove the “Unknown” Group preventing access by undefined hosts

It cannot be over stated: Planning is the key to a successful implementation of Access controls. If not thought out completely and implemented precisely, applications will fail and customer disaster positions may be compromised.

The last step of removing the Unknown group may not be necessary as it might be appropriate to only provide access controls on critical application devices and all other devices would not be members of any access pool and thus open for command and control operations.




Configuring Access Control DataThe symacl command is used to manage access control database entries

Uses a command file to define specific operations– Similar to symconfigure– Command file is processed

to manage access control entriesPreviewPrepareCommit

– Example:symacl commit

–file myacl.cmd

Access Group

Access ID

Access Pools

Devices

ACE

Create

Delete

Remove

Add

Remove

Command File

The symacl command works like the symconfigure command in that it uses a command file to define the management function to perform. When executing a symacl command it performs three progressive operations on the command file.

Preview operation verifies the syntax and correctness of the contents of the entries in the command file.Prepare operation (also before you commit) performs the preview checks, but also verifies the appropriateness of the requested access control modifications against the current state of the access control database in the Symmetrix array.Commit operation performs both the preview and prepare checks and then commits the contents of the command file to the Symmetrix Access Control database.

A lock is taken out by the specified Symmetrix during an access control change session. Only one access control session can be active within a Symmetrix at a time.




Create Access Control Group

1. Generate unique Access IDs for each hostsymacl –unique

2. Create Access Group

3. Add hosts to Access Group

Example:symacl commit –file mycmd

create accgroup OraClstr;

add host accid12345678-87654321-ABCDEFABname node1 to accgroup OraClstr;

add host accid 87654321-12345678-ABCDEFABname node2 to accgroup OraClstr;

add host accid ABCDEFAB-12345678-87654321name node3 to accgroup OraClstr;

add host accid 12345678-87654321-ABCDEFABname node4 to accgroup OraClstr;

mycmd

Note: You will be prompted for theaccess PIN if environment variable SYMCLI_ACCESS_PIN is not set

The typical approach is to implement access control by host. To do this, the first step is to generate unique access ID for each host using the command symacl –unique. Next, create an access group and add all host that share similar access requirements. In the example above, in a command file, we added the commands to created a group for a cluster of Oracle servers and called the group OraClster. In the same file we added command entries to add four servers called node 1-4. Once the command file has been written and saved we can run the symacl -commit command to process the command file and add the entries to the access control database. All commit operations must be run from a host that is a member of the AdminGrp, and the user must enter the correct Access PIN that was generated when the system was configured for access control.

Once a group is created additional hosts can be added, remover, or moved from one group to another. A group can also be deleted.




Example:C:\>symacl commit -f diamond.cmd

Enter Access PIN:

Command file: (diamond.cmd) PREVIEW......................................................Started. PREVIEW......................................................Done. PREPARE......................................................Started. Creating group OraClstr......................................Done. Adding Host access id node1 to group OraClstr................Done. Adding Host access id node2 to group OraClstr................Done. Adding Host access id node3 to group OraClstr................Done. Adding Host access id node4 to group OraClstr................Done. PREPARE......................................................Done. Starting COMMIT..............................................Done.

C:\>symacl list -accgroup

Symmetrix ID: 000190102254Number of Number of

Group Name Access IDs ACLs---------------------------------- ---------- ---------AdminGrp 2 2UnknwGrp 1 2OraClstr 4 0

C:\>symacl commit -f diamond.cmd

Enter Access PIN:

Command file: (diamond.cmd) PREVIEW......................................................Started. PREVIEW......................................................Done. PREPARE......................................................Started. Creating group OraClstr......................................Done. Adding Host access id node1 to group OraClstr................Done. Adding Host access id node2 to group OraClstr................Done. Adding Host access id node3 to group OraClstr................Done. Adding Host access id node4 to group OraClstr................Done. PREPARE......................................................Done. Starting COMMIT..............................................Done.

C:\>symacl list -accgroup

Symmetrix ID: 000190102254Number of Number of

Group Name Access IDs ACLs---------------------------------- ---------- ---------AdminGrp 2 2UnknwGrp 1 2OraClstr 4 0

Example above creates an access control group and adds the unique Host IDs of four systems.




Creating Access Pools

Access pools are logical groupings of Symmetrix devices

– Permissions are assigned to allow a host to perform functions on a sets of devices defined in access pools

1. Identify devices related to hosts and applications that require the same access control

2. Create a Access Pool3. Add devices to the pool


create accpool Orapool;

add dev 003a:03f to accpool Orapool;

myfile.cmd

Access pools are logical groupings of Symmetrix devices. Permissions are assigned to allow a host to perform functions on a sets of devices defined in access pools. The first step is to plan access control by identifying devices that are related to an application and or set of hosts. Create a access pool and add devices using a command file.

Again, when the commit is executed, the user will be prompted for the PIN if the environment variable SYMCLI_ACCESS_PIN is not set appropriately.

Devices can be added or removed from the access pool and the pool can be deleted if no longer needed.




ACL Permissions (A.KA. Access Types)

ADMIN*ADMINRD*ALL*BASEBASECTRLBCVCACHCTRLCFGDEVCFGSYM*CHECKSUMCREATEDV*

DIRCTRL*ECCOPTMZRPOWRPATHQOSRCOPYSDDFSDRSNAPVLOGIX*

* Access Control Entries associated with all Devices.All others are directed at Device Pools

After creating access groups and access pools, permissions are set by creating Access Control Entries (ACE’s) in the Access Control List (ACL) that grant permission to execute specific Solutions Enabler functions on a group of devices defined in access pools, by specific host defined in access groups.

Reference pages 91-93 of Solutions Enabler Symmetrix Array Management CLI version 6.4 Product Guide for full description of the type of controls each ACL allows and a list of the commands effected. Note, most ACL permissions are associated with a Device Pool except those in bold which are associated with ALL DEVICES.




Access Control Entries (ACE)

Permissions are set by creating Access Control Entries– Grant permission to control specific Solutions Enabler functionality

On groups of devices defined in Access PoolsBy specific host defined in Access Groups

1. Identify Solutions Enabler functions to be controlled

2. Create a Access Control EntryGrant permissions to devices in Access PoolGrant permissions to all devicesGrant permissions to devicesnot in a device pool


grant access=BASE to accgroup OraClstr for ALL devs;

grant access=BCVto accgroup OraClstrfor accpool OraPool;

grant access=VLOGIX, SDRto accgroup OraClstrfor NON_POOLED devs;

remove access=rdffrom accgroup OraClstrfor accpool OraPool;

myfile.cmd

Note: Before you added permissions for specific Solutions Enabler functionality, it is first required that BASE permissions be granted.




Service Processor Considerations

The Service Processor on the Symmetrix is often used to execute SYMCLI command

Access Control must be configured like any other host– Create Access Group containing

only the Service Processor– Grant:

BASE rights to ALL DEVICESOPTIMIZER to ALL DEVICESADMINRD to ALL DEVICESOther Permissions as Appropriate

create accgroup EMCSP;

add host accid12345678-87654321-ABCDEFABname storprocto accgroup EMCSP;

grant access=BASE to accgroup EMCSP for ALL devs;

grant access=OPTIMIZERto accgroup EMCSP for ALL devs;

grant access=ADMINRD to accgroup EMCSP for ALL devs;

myfile.cmd

The Symmetrix Service Processor also execute Solutions Enabler commands and therefore must be configured with the appropriate access controls.




ACL DatabaseWhen querying Access Control Database, you will only see objectsassociated with the access group that the host belongs– Unless host has ADMIN or ADMINRD (Read-only) permissions



Group Name Pool Name Access Type

------------------------ ----------------------- -----------AdminGrp ALL_DEVS ADMIN

AdminGrp ALL_DEVS ALL

UnknwGrp ALL_DEVS BASE

UnknwGrp !INPOOLS ALL

OraClstr ALL_DEVS VLOGIX

OraCLstr OraPool SDR

OraClstr OraPool BCV

OraClstr ALL_DEVS BASE. . .



Group Name Pool Name Access Type

------------------------ ----------------------- -----------AdminGrp ALL_DEVS ADMIN

AdminGrp ALL_DEVS ALL

UnknwGrp ALL_DEVS BASE

UnknwGrp !INPOOLS ALL

OraClstr ALL_DEVS VLOGIX

OraCLstr OraPool SDR

OraClstr OraPool BCV

OraClstr ALL_DEVS BASE. . .

Unless the host that you are running the symacl list command from has ADMIN permissions, you will only see a subset of the access control database.




ACL Database Details -symacl show

C:\>symacl show accpool MilfPool -acl

Access Pool: MilfPool


Number of Access Control Entries: 2

Number of Member Devices : 5

Access Control Entries (2):{------------------------------------Group Type--------------------------------------MILFORD SDR

BCV}

Member Devices (5):{

Device Name Device--------------------------- --------------------------------------

CapSym Physical Config Attribute Sts (MB) ---------------------------- --------------------------------------

0030 Not Visible 2-Way Mir N/Grp'd RW 84380031 Not Visible 2-Way Mir N/Grp'd RW 8438 0032 Not Visible 2-Way Mir N/Grp'd RW 8438 0033 Not Visible 2-Way Mir N/Grp'd RW 8438 0034 Not Visible 2-Way Mir N/Grp'd RW 8438 }

C:\>symacl show accpool MilfPool -acl

Access Pool: MilfPool


Number of Access Control Entries: 2

Number of Member Devices : 5

Access Control Entries (2):{------------------------------------Group Type--------------------------------------MILFORD SDR

BCV}

Member Devices (5):{

Device Name Device--------------------------- --------------------------------------

CapSym Physical Config Attribute Sts (MB) ---------------------------- --------------------------------------

0030 Not Visible 2-Way Mir N/Grp'd RW 84380031 Not Visible 2-Way Mir N/Grp'd RW 8438 0032 Not Visible 2-Way Mir N/Grp'd RW 8438 0033 Not Visible 2-Way Mir N/Grp'd RW 8438 0034 Not Visible 2-Way Mir N/Grp'd RW 8438 }




Access Control Database Backup

Best Practice is to backup access control database prior to making any configuration changes– Backup file is saved on local administration host– Host must have ADMIN permissions– File contains encrypted access IDs

Example:symacl –sid 123 backup –file mybackup

Note: You will be able to view the backup files, however, Access IDs are encrypted

Like any other critical configuration database, it is recommended that it be backed up before any configuration change.




Other ConsiderationsOnce Access Control Management have been implemented, additional steps must be added to customers Change Management Process– Provisioning operations require access control changes

After making access control changes for a host, update the host’s SYMAPI databasesymcfg discover

Processing a command file with a Prepare and Commit action invokes a session– Locks ensure only a single ACL change at a time– If a host files and a session is locked, the lock can be released

symacl list –vsymacl release

After making access control changes for a host, update the host’s SYMAPI database. Because the discover simply updates the database with changes, a better approach might be to delete the database file altogether and recreate it using the command. symcfg discover




User Based Access Controls

Users, based on usernames, are mapped to specific roles which define permitted operations– Uses the symauth command

Independent facility– Can be used with Symmetrix based Access Controls– By itself

An alternative to host-based access control is user-based. With user-based access control, a role is associated with a user name. Based on the role, the user will have the specified access level for all devices in the Symmetrix. Host-based and user-based access control, while completely separate and independent, can be used together to create the highest level of security.




RolesPredefined set of permissions that determine what operations a user can perform– Role is assigned to a user for ALL devices in the Symmetrix– Roles are predefined - symauth list -roles

NoneNo capabilities

MonitorRead only for all facilities except Audit log and Access Control Database

StorageAdminPerform all management and control functions

SecurityAdminPerform security functions including symaudit, symacl and symauth

AdminPerform all management and control plus security

AuditorAble to view but not modify all security setting including symaudit, symacl and symauth

Unlike host based authentication, use roles are for all devices in the Symmetrix.




UsersUsers are defined using the following format:

– Examples:D:emc.com\diamojD:*\diamondH:oraserver1\jimH:*\diamondjim

Type:Qualifier\Name

D = Domain user

H = Host

Domain or Realm name

Hostname

User Name

User-based authentication requires that users be identified by Role. The user definition is in the format above:

When the hostname or domain portion of a <Username> is an asterisk, the asterisk is treated as a wildcard meaning any host. Examples of this include: H:*\user, D:*\user, and *\user. In all other cases, the asterisk is treated as a regular character.




Configuring User Role-based Authentication

1. Create user-to-role mapping– Using symauth and command file– Example:

symauth commit –file myfile.cmd

2. Enable User Authentication

3. Set enforcement policy

assign user H:QA\laura to role Monitor;

assign user D:Eng\hardy to role Monitor;

assign user D:Eng\moeto role Admin;

assign user D:Eng\larry to role Admin;

assign user D:Eng\curlyto role StorageAdmin;

assign user D:Eng\gilligan to role StorageAdmin;

assign user D:Eng\skipper to role SecurityAdmin;

myfile.cmd

The first step is to create a user-to-role mapping using a command file.




Enabling User Role-based Authentication

2. Enable User Authentication– symauth –sid 254 enable

3. Set enforcement policy– symauth set enforcement <advise|enforce>

C:\>hostname 2K3-39-106

C:\> symauth -sid 254 enable

Perform an Authorization change on Symmetrix unit 000190102254 (y/[n]) ?

C:\>symauth set enforcement enforce

Perform an Authorization change on Symmetrix unit 000190102254 (y/[n]) ? y

C:\>hostname 2K3-39-106

C:\> symauth -sid 254 enable

Perform an Authorization change on Symmetrix unit 000190102254 (y/[n]) ?

C:\>symauth set enforcement enforce

Perform an Authorization change on Symmetrix unit 000190102254 (y/[n]) ? y

Be default, User Authorization is disabled. As such, any user can make changes to the authorization control data, including creating and removing user-to-role mappings. Once User Authorization is enabled for an array, only users with the Admin or SecurityAdmin role can change authorization control data. Therefore it is recommended that, prior to enabling User Authorization, you first create your user-to-role mappings and map at least one user to the Admin or SecurityAdmin role on an array.

A enforcement policy of advise will generate a warning message but allow the operation to proceed. An entry is placed in the SYMAPI log on the host and in the Symmetrix Audit log. A policy of enforce will cause the operation to fail.




User Authentication Status

Enable User level authentication– Must previously have granted security admin privileges to execute

the change

symauth -sid 1254 list

S Y M M E T R I X A U T H O R I Z A T I O N S T A T U S


Authorization Control : Enabled

Time Enabled : Mon Aug 13 11:52:03 2007

Time Disabled : Fri Aug 10 11:47:10 2007

Time Updated : Mon Aug 13 11:52:03 2007

Enforcement Mode : Advise

Server Policy : Trust clients

symauth -sid 1254 list

S Y M M E T R I X A U T H O R I Z A T I O N S T A T U S


Authorization Control : Enabled

Time Enabled : Mon Aug 13 11:52:03 2007

Time Disabled : Fri Aug 10 11:47:10 2007

Time Updated : Mon Aug 13 11:52:03 2007

Enforcement Mode : Advise

Server Policy : Trust clients

The default is to trust user names sent from clients . For additional security when operating in client/server mode, Solutions Enabler can be set to require a secure cross-host authentication (validate_client), such as Kerberos or Windows. If validate_client is set, the server is required to validate a user's identity via a distributed authentication mechanism.

To change the server policy to trust clients, enter:

symauth -sid 1234 set server_policy trust_client




Listing Users and Roles

Identify defined users and roles

symauth -sid 1234 list -users


Role name Username----------------- ----------------------------------------

Admin D:Eng\moe

Admin D:Eng\larry

StorageAdmin D:Eng\curly

StorageAdmin D:Eng\gilligan

Monitor D:\Eng\laura

Monitor D:\Eng|hardy

symauth -sid 1234 list -users


Role name Username----------------- ----------------------------------------

Admin D:Eng\moe

Admin D:Eng\larry

StorageAdmin D:Eng\curly

StorageAdmin D:Eng\gilligan

Monitor D:\Eng\laura

Monitor D:\Eng|hardy




Monitoring

A second part of security is monitoring activity– Who?– What?– When?

On the host– The SYMAPI log tracks all syscalls that fail due to access control or

user authentication activity

On the Symmetrix– The Audit log tracks all successful and unsuccessful access

attemptssymaudit command

Nearly as important as restricting access is understanding who is attempting to do what operation, from what system, when.




Symmetrix Audit Log

Data is written to the audit log during control operations initiated by host applications – Secure, persistent circular log– Activity from all hosts into one file

The symaudit command is used to retrieve audit information – Show – High level details about log– List - Filter the audit data to match criteria;

date/time, application, host, etc– Monitor - Run in the foreground polling the

Symmetrix for new audit log records using Interval and Count

Symmetrix

FA FA

Symmetrix File SystemSymmetrix File System

GKGK

AuditLog

Host Host Host

Enginuity collects chronological list of host-initiated Symmetrix actions and activities. Manual activities (for example, physically removing/replacing a component) as well as automatically initiated scripts and EMC’s Solutions Enabler activities (for example, TimeFinder or SRDF routines), are tracked and recorded in the SFS. This provides a means to oversee and historically recall how and when a Symmetrix device is being used.




Symmetrix Audit – symaudit High Level

Persistent

Circular log file

C:\>symaudit show

A U D I T L O G D A T A

Symmetrix ID : 000190102254Starting date : 08/08/2007 16:58:44Ending date : 08/14/2007 04:19:13Starting record number : 1Ending record number : 151Total record count : 151

C:\>symaudit show

A U D I T L O G D A T A

Symmetrix ID : 000190102254Starting date : 08/08/2007 16:58:44Ending date : 08/14/2007 04:19:13Starting record number : 1Ending record number : 151Total record count : 151

The Audit log is a circular log file that will overwrite the oldest entry. symaudit showdisplays a summary of the contents of the audit log.




Symmetrix Audit – symaudit Verbose

Extensive filtering # symaudit -sid 04 list -v -start_time 8/13:12:00 -end_time 8/14:12:00

A U D I T L O G D A T A…

Record Number : 139Records in Seq : 2Offset in Seq : 2Time : 08/13/07 13:49:39Vendor ID : EMC CorpApplication ID : SYMMASKApplication Version : 6.4.1.0API Library : SEKAPI Version : V6.3.0.0 (Edit Level: 771)Host Name : W2K3-39-106OS Name : WinNTOS Revision : 5.2.3790SeClient Host : usxxdiamojProcess ID : 00001304Task ID : 00001520Function Class : DevMaskAction Code : AddText : [ 0036 ]Username : D:CORP\diamojActivity ID : SEa29861bf5c

…

# symaudit -sid 04 list -v -start_time 8/13:12:00 -end_time 8/14:12:00

A U D I T L O G D A T A…

Record Number : 139Records in Seq : 2Offset in Seq : 2Time : 08/13/07 13:49:39Vendor ID : EMC CorpApplication ID : SYMMASKApplication Version : 6.4.1.0API Library : SEKAPI Version : V6.3.0.0 (Edit Level: 771)Host Name : W2K3-39-106OS Name : WinNTOS Revision : 5.2.3790SeClient Host : usxxdiamojProcess ID : 00001304Task ID : 00001520Function Class : DevMaskAction Code : AddText : [ 0036 ]Username : D:CORP\diamojActivity ID : SEa29861bf5c

…

Filtering allows you to specify a specific record number, a time range, application, host, user and other selection criteria.





Host based and user based Access Controls provides options for “locking down” and enabling secure sharing of a Symmetrix

– Grant a host only the required access level on specified volumes– Prevents accidental or malicious tampering– Supports change control procedures and limits ad-hoc operations

Managed using the symacl and symauth commands that processes a command file through the review, prepare, and commit phases symacl requires initial setup including enabling the feature on the Service Processor, creating an administrator PIN and define initial groups

Successful implementation requires planning

Post implementation requires monitoring of API, Event, and Audit logs

Can also be setup and administered using SMC

Before any type of Access Control is configured, extensive planning is required to ensure the implementation is not disruptive to the customers application environment. This includes a survey of all user, host and application requirements for controlling Solutions Enabler functions, and then organizing the requirements into host access groups, device pools, and access control entries.





PowerLink




Closing Slide

0_d9cdf

Documents

Transcript of 0_d9cdf