FC Standar

ICSNS

Implementing Cisco Storage Networking Solutions Volume 3 Version 3.0

Student Guide

Text Part Number: 67-2461-01

DISCLAIMER WARRANTY: THIS CONTENT IS BEING PROVIDED “AS IS.” CISCO MAKES AND YOU RECEIVE NO WARRANTIES IN CONNECTION WITH THE CONTENT PROVIDED HEREUNDER, EXPRESS, IMPLIED, STATUTORY OR IN ANY OTHER PROVISION OF THIS CONTENT OR COMMUNICATION BETWEEN CISCO AND YOU. CISCO SPECIFICALLY DISCLAIMS ALL IMPLIED WARRANTIES, INCLUDING WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE. This learning product may contain early release content, and while Cisco believes it to be accurate, it falls subject to the disclaimer above.

Table of Contents Volume 3 Appendix A: The Fibre Channel Protocol AA-1

Overview AA-1 Module Objectives AA-1

The SCSI Protocol AA-3 Overview AA-3

Objectives AA-3 SCSI Protocol Overview AA-4 SCSI Architecture Model AA-7 SCSI Parallel Technology AA-10

Multidrop Topology and Addressing AA-11 SCSI Operation AA-13 SCSI Commands and Status AA-15 SCSI Messages AA-18

Error Handling AA-19 Summary AA-20

FC Protocol Concepts AA-21 Overview AA-21

Objectives AA-21 Fibre Channel Overview AA-22

Fibre Channel: The Best of Both Worlds AA-23 Advantages of Serial Architecture AA-24

Fibre Channel Performance AA-25 Fibre Channel Topologies AA-29

What is the Point-to-Point Topology? AA-30 What is the Arbitrated Loop Topology? AA-31 What is the Switched Fabric Topology? AA-32

Fibre Channel Ports AA-34 Fibre Channel HBAs AA-36 Fibre Channel Classes of Service AA-37 Summary AA-39

FC Layers AA-41 Overview AA-41

Objectives AA-41 Fibre Channel Layers AA-42

FC-0: Physical Interface AA-44 FC-1: Encoding AA-46 FC-2: Framing and Flow Control AA-49 FC-3: Common Services AA-50 FC-4: Upper-Layer Protocol Interfaces AA-51

Fibre Channel Data Constructs AA-53 Fibre Channel Frames AA-54 Frame Headers AA-56

SCSI-FCP Operations AA-58 Link Services AA-60

Types of Link Services AA-61 Basic Link Services AA-62 Extended Link Services AA-63

Summary AA-64 FC Flow Control AA-65

Overview AA-65 Objectives AA-65

Fibre Channel Flow Control AA-66 Credit-Based Flow Control AA-67 Types of Flow Control AA-68

ii Implementing Cisco Storage Networking Solutions (ICSNS) v3.0 © 2007 Cisco Systems, Inc.

Buffer-to-Buffer Flow Control and End to End Flow Control AA-69 Credit Management Methods AA-71 The Base Credit Management Method AA-72

Allocating Buffer Credits AA-73 Example AA-74 Example (Cont.) AA-75 Example (Cont.) AA-76 Example (Cont.) AA-77

Fibre Channel Addressing AA-79 The Switched Fabric Address Space AA-79 The FC-AL Address Space AA-81

World-Wide Names AA-82 Summary AA-84

FC Login AA-85 Overview AA-85

Objectives AA-85 Fabric Login AA-86 Port Login AA-94

Port and Address Discovery AA-97 Process Login AA-98 Loop Initialization and Arbitration AA-103

The Loop Initialization Protocol AA-103 The Loop Arbitration Protocol AA-103 The Loop Port State Machine AA-105

Summary AA-106 FC Error Recovery AA-107

Overview AA-107 Objectives AA-107

FC-1 Errors AA-108 R_T_TOV AA-109

FC-2 Errors AA-111 E_D_TOV AA-113 Sequence Recovery AA-114 R_A_TOV AA-117

SCSI-FCP Error Recovery AA-118 Summary AA-122

FC Switched Fabric AA-125 Overview AA-125

Objectives AA-125 Fabric Configuration Overview AA-126 FSPF AA-128

FSPF Protocol Operations AA-130 Stage 1—The Hello Protocol AA-131 Stage 2—Initial Database Synchronization AA-133 Stage 3—Database Maintenance AA-133 Stage 4—Path Discovery AA-135 Stage 5—Path Computation AA-135 Limitations of FSPF AA-137

The RSCN Process AA-139 Fabric State Changes AA-139 The RSCN Process AA-139

Standard Fabric Services AA-143 The Domain Manager AA-144 The Name Server AA-145 Name Server Operations AA-146 The Management Server AA-147 Well-Known Addresses AA-148

Summary AA-149

© 2007 Cisco Systems, Inc. Implementing Cisco Storage Networking Solutions (ICSNS) v3.0 iii

Appendix B: Installation and Configuration Reference AB-1 Overview AB-1

Module Objectives AB-1 Switch Hardware Installation Reference AB-3

Overview AB-3 Installation Guidelines AB-4 Cabinet and Rack Options AB-8 Configuring Power Supplies AB-12 Installing Fan Modules AB-26 Supervisor and Line Card Modules Installation AB-30 Summary AB-40

Appendix A

The Fibre Channel Protocol

Overview This appendix provides an overview of the SCSI and Fibre Channel protocols.

Module Objectives Upon completing this module, you will be able to describe the SCSI and Fibre Channel protocols. This includes being able to meet these objectives:

Describe the basic characteristics of the SCSI protocol

Explain the role of Fibre Channel in a storage environment

Describe the Fibre Channel layered model, data constructs, SCSI-FCP read and write operations, and Link Services

Explain Fibre Channel flow control and addressing

Describe the Fibre Channel device login process

Explain how Fibre Channel recovers from errors

Describe the Fibre Channel Switched Fabric protocol

Lesson 1

The SCSI Protocol

Overview The Fibre Channel Protocol (FCP) is based on the serial Small Computer Systems Interface (SCSI) protocol. This lesson covers the fundamentals of the SCSI protocol family.

Objectives Upon completing this lesson, you will be able to describe the basic characteristics of the SCSI protocol. This includes being able to meet these objectives:

Explain the function of the SCSI protocol in a storage environment

Describe the SCSI architecture model

Explain the operations and limitations of SCSI parallel technology

Describe the SCSI operational phases

Identify the most common SCSI commands and status messages

Explain the role of SCSI messages in error handling


SCSI Protocol Overview This topic provides an overview of SCSI technology.

© 2007 Cisco Systems, Inc. All rights reserved. ICSNS v3.0—7-4

SCSI Protocol Overview

SCSI performs passing commands, status, and block data between initiators and targetsSCSI is a hierarchy of functions to assemble raw data blocks into application readable files

Delivery SubsystemParallel or FCP or IP

LUNs

DeviceServer

ApplicationClient

Requests

Responses

Tasks

Initiator Target

The Small Computer System Interface (SCSI) performs the heavy lifting of passing commands, status, and block data between platforms and storage devices. One function of operating systems is to hide the complexity of the computing environment from the end user. Management of system resources including , memory, peripheral devices, display, context switching between concurrent applications, and son on, are generally concealed behind the user interface. The internal operations of the OS must be robust, closely monitor changes of state, ensure that transactions are completed within the allowable time frames, and automatically initiate recovery or retires in the event of incomplete or failed procedures. For I/O operations for peripheral devices such as disk, tape, optical storage, printers, and scanners, these functions are provided by the SCSI protocol, typically embedded in a device driver or logic onboard a host adapter.

Because the SCSI protocol layer sits between the operating system and the peripheral resources, it has different functional components. Applications typically access data as files or records. Although these may be ultimately stored on disk or tape media in the form of data blocks, retrieval of the file requires a hierarchy of functions to assemble raw data blocks into a coherent file that can be manipulated by an application.

SCSI architecture defines the relationship between initiators (hosts) and targets (for example, disks) as a client/server exchange. The SCSI-3 application client resides in the host and represents the upper layer application, file system, and operating system I/O requests. The SCSI-3 device server sits in the target device, responding to requests.

© 2007 Cisco Systems, Inc. Appendix A: The Fibre Channel Protocol AA-5


SCSI Protocol Overview (Cont.)

The file system presents an abstraction of data to the user application.Physical storage devices are presented as an abstraction to the file system.

Hierarchy of logical-to-physical SCSI mapping

InterconnectPhysical Layer

SCSI Command or DataStorage Network Transport

Device DriverStorage Network Interface

SCSI MappingBlock Transition

Logical DrivesVolume Management

Files or RecordsOperating System

Files or RecordsFile System or Database

Files or RecordsUser Applications

When a user application opens a file, a series of processes are launched that rely on lower SCSI commands and controls to transport the appropriate data blocks from storage safely into memory. A translation between file representation and block I/O thus occurs in the file system layer.

Just as the file system presents an abstraction of data to the user application, the physical storage devices are presented as an abstraction to the file system. An E: drive in Windows or a /dev/dsk2 in UNIX may be a single disk, a partition on a larger disk, or a striped array of multiple disks. The file system depends on a volume management function to present sometimes diverse storage devices as coherent and easily addressable resources. Device virtualization turns physical storage into logical storage, and assumes the intricate tasks necessary for placement of data blocks on disks. This file/block translation and mapping function can be as sophisticated as a separate volume management application or as straightforward as an adaptor card device driver interface to an operating systems disk utility.

This hierarchy of logical abstractions descends to the physical world of actual SCSI devices and their connectivity to the host system. Common access methods at the OS level allow uniform treatment of SCSI devices regardless of their physical attachment. In saving a file, the file system does not need to be concerned with whether the logical drive identifier fronts a direct SCSI-attached unit, a Fibre Channel array or an IP storage device somewhere on the Gigabit Ethernet network.

Regardless of the underlying plumbing, the operating system’s view of the physical storage is defined by the bus/target/LUN triad inherited from parallel SCSI technology. The mapping between the bus/target/LUN designation and the logical drive identifier provides the portal between physical devices and the upper layer file system. Because Fibre Channel and IP storage are serial transports and have no bus component, the bus identifier is fabricated for compatibility with the operating system’s SCSI nomenclature. Two IP storage NICs in a single server, for example, may have different bus designations to mimic SCSI adapter configuration.


The bus/target/LUN identifier may be further mapped to the addressing requirements of a specific transport. FCP, for example, maps bus/target/LUN to a device identification (ID)/LUN pair. Consequently, the representation of physical storage has two components:

1. One directed to the operating system, to establish a familiar, addressable entity based on the SCSI triad

2. The other is directed at the specific transport, to accommodate the addressing requirements of that topology


SCSI Architecture Model This topic reviews the Layers of the SCSI Architecture Model.


SCSI Architecture Model

PhysicalInterconnect

TransportProtocol

SharedCommand Set

FC-PH

FCP

SPI

SPC

SBC SSC SES SMC

SBP

IEEE1394

SSP

SSA

SCSI

Arc

hite

ctur

e M

odel

CAMApplicationLayer

The SCSI Architecture Model (SAM) consists of four layers of functionality:

1. The physical interconnect layer specifies the characteristics of the physical SCSI link:

— FC-PH is the physical interconnect specification for Fibre Channel.

— Serial Storage Architecture (SSA) is a storage bus aimed primarily at the server market.

— IEEE1394 is the FireWire specification.

— SCSI-3 Parallel Interface (SPI) is the specification used for parallel SCSI buses.

2. The transport protocol layer defines the protocols used for session management:

— FCP is the transport protocol specification for Fibre Channel.

— Serial Storage Protocol (SSP) is the transport protocol used by SSA devices.

— Serial Bus Protocol (SBP) is the transport protocol used by IEEE1394 devices.

3. The shared command set layer consists of command sets for accessing storage resources:

— SCSI Primary Commands (SPC) are common to all SCSI devices.

— SCSI Block Commands (SBC) are used with block-oriented devices, such as disks.

— SCSI Stream Commands (SBC) are used with stream-oriented devices, such as tapes.

— SCSI Media Changer Commands (SMC) are used to implement media changers, such as robotic tape libraries and CD-ROM carousels.


— SCSI Enclosure Services (SES) defines commands used to monitor and manage SCSI device enclosures, such as RAID arrays.

4. The SCSI Common Access Method (CAM) defines the SCSI device driver application programming interface (API).



SCSI Architecture Model (cont.)

*SCSI-3Class Drivers

*SCSI-3Class Drivers

SCSI-3 FCPPort Driver

SCSI-3 FCPPort Driver

SCSI ParallelPort Driver

SCSI ParallelPort Driver

SAS Port Driver

SAS Port Driver

iSCSI IPPort DriveriSCSI IP

Port DriverSCSI Serial BusProtocol (SBP-2)

Port Driver

SCSI Serial BusProtocol (SBP-2)

Port Driver

Fibre ChannelPort

Fibre ChannelPort

SCSI Parallel Port

SCSI Parallel Port

SAS Serial Port

SAS Serial Port

EthernetPort

EthernetPort

IEEE-1394(Firewire)

Port

IEEE-1394(Firewire)

Port

FC Serial AttachedSCSI Interface

FirewireInterface

FC CardSCSI Card NIC

*SCSI-3: Separation of physical interface, transport protocols, and SCSI Command Set

SPC

The SCSI_3 family of standards introduced several new variations of SCSI commands and a protocol, including serial SCSI-3 and special command sets for streaming and media handling required for tape. As shown in the diagram, the command layer is independent of the protocol layer, which is required to carry SCSI-3 commands between devices. This enables more flexibility in substituting different transports beneath the SCSI-3 command interface to the operating system.


SCSI Parallel Technology This topic reviews parallel-bus SCSI technology and explains why SCSI has been supplanted by Fibre Channel in high-performance environments.


SCSI Parallel Technology

SCSI uses a parallel architecture in which data is sent simultaneously over multiple wires.SCSI is half-duplex—data travels in one direction at a time.On a SCSI bus, a device must assume exclusive control over the bus in order to communicate. (SCSI is sometimes referred to as a “simplex”channel because only one device can transmit at a time).

• Half-duplex• Parallel• Shared bus

The bus/target/LUN triad is defined from parallel SCSI technology. The bus represents one of several potential SCSI interfaces installed in the host, each supporting a separate string of disks. The target represents a single disk controller on the string. And the LUN designation allows for additional disks governed by a controller – for example, a RAID device.

The following are characteristics of parallel SCSI technology:

SCSI uses a parallel architecture in which data is sent simultaneously over multiple wires.

SCSI is half-duplex—data travels in one direction at a time.

On a SCSI bus, a device must assume exclusive control over the bus in order to communicate. (SCSI is sometimes referred to as a “simplex” channel because only one device can transmit at a time).



Multidrop Topology and Addressing

SCSI Initiator

(I/O Adapter)

Data/Address Bus

Control Signals

Interface Interface Interface

Terminator

IDID

77IDID

66IDID

55IDID

00…

Multidrop Topology

Priority

Multidrop Topology and Addressing All of the devices on a SCSI bus are connected to a single cable. This is called a multidrop topology:

Data bits are sent in parallel on separate wires. Control signals are sent on a separate set of wires.

Only one device at a time can transmit—a transmitting device has exclusive use of the bus.

A special circuit called a terminator must be installed at the end of the cable. The cable must be terminated to prevent unwanted electrical effects from corrupting the signal.

A multidrop topology has inherent limitations:

Parallel transmission of data bits allows more data to be sent in a given time period but complicates transmitter-receiver synchronization. The fact that control signals, such as clock signals, are sent on a separate set of wires also makes synchronization more difficult.

It is an inefficient way to use the available bandwidth, because only one communication session can exist at a time.

Termination circuits are built into most SCSI devices, but the administrator often has to set a jumper on the device to enable termination.

Incorrect cable termination can cause either a severe failure or intermittent, difficult-to-trace errors.



Multidrop Topology and Addressing (Cont.)

LUN 0LUN 1

LUN 2LUN 3

LUN 0LUN 1

LUN 2LUN 3

LUN 0LUN 1

Address = BUS : Target ID : LUN

Addressing

SCSI Initiator

(I/O Adapter)

Data/Address Bus

Control Signals

Interface Interface Interface

Terminator

…

SCSI was designed to support a few devices at most, so its device addressing scheme is fairly simple—and not very flexible. SCSI devices use hard addressing:

Each device has a series of jumpers that determine the device’s physical address, or SCSI ID. The ID is software-configurable on some devices.

Each device must have a unique ID. Before adding a device to the cable, the administrator must know the ID of every other device connected to the cable and choose a unique ID for this new device.

The ID of each device determines its priority on the bus. For example, the SCSI target with ID 7 always has a higher priority than the SCSI initiator with ID 6. Because each device must have exclusive use of the bus while it is transmitting, ID 6 must wait until ID 7 has finished transmitting. Fixed priority makes it more difficult for administrators to control performance and quality-of-service.


SCSI Operation This topic provides an overview of SCSI protocol operations.


SCSI Operation

SCSI includes three phases of operation:Command – send the required command and parameters via a Command Descriptor Block (CDB)Data – Transfer data in accordance with the commandStatus – Receive confirmation of command execution

TargetInitiatorFC

FC

HBA

Every communication on the SCSI bus is formed by sequences of events called bus phases. Each phase has a purpose and is linked to other phases to execute SCSI commands and transfer data and messages back and forth.

The majority of the SCSI protocol is controlled by the target. The initiator only initiates a SCSI task by selecting a target.

Once the target is selected, it (the target) controls the bus.

It does this by picking up the command from the initiator, executing it and delivering a status back to the initiator.



SCSI Operation (Cont.) SCSI Bus Arbitration:

Arbitrate for the SCSI Bus and take controlSCSI Device Selection:

Address the target by its SCSI ID and select it

Connection , Arbitrate and select

Command Phase

Data In Phase

Status Phase

Disconnect

SCSI Command:Send required command and parameters via a Command Descriptor Block (CDB)

SCSI Data (optional):Transfer Data in accordance with the command

SCSI Status or Response:Receive confirmation of command execution

SCSI Message:Send ‘Command Complete’ messageRelease the Bus

SCSI ReadCMD (28h)

RSP

DATA

FCFCHBA

A simple SCSI task can be described by using the following example.

1. The host adapter is the initiator. A host adapter wants to read a logical block of data from a disk drive.

2. The host adapter waits until the bus is free.

3. When the bus is free, the host adapter uses the arbitration phase to acquire initial control over the bus.

4. The disk drive that will be the target is selected. The disk accepts the selection by taking over control of the bus.

5. The host sends a SCSI READ command to the disk.

6. The disk picks up the command from the host adapter. The disk reads its data from the media and enters the data phase to send it across the bus to the host adapter

7. After the data transfer, the disk enters a status phase and sends the status code GOOD.

8. The SCSI task is finished, so the disk sends a COMMAND COMPLETE message to the host adapter and releases the bus to the BUS FREE phase.


SCSI Commands and Status A simple SCSI task can be described by using the following example.

1. The host adapter is the initiator. A host adapter wants to read a logical block of data from a disk drive.

2. The host adapter waits until the bus is free.

3. When the bus is free, the host adapter uses the arbitration phase to acquire initial control over the bus.

4. The disk drive that will be the target is selected. The disk accepts the selection by taking over control of the bus.

5. The host sends a SCSI READ command to the disk.

6. The disk picks up the command from the host adapter. The disk reads its data from the media and enters the data phase to send it across the bus to the host adapter

7. After the data transfer, the disk enters a status phase and sends the status code GOOD.

8. The SCSI task is finished, so the disk sends a COMMAND COMPLETE message to the host adapter and releases the bus to the BUS FREE phase.


SCSI Commands and Status

A command is executed by sending a CDB to a target.

Byte 7 6 5 4 3 2 1 00123456789

Reserved

Transfer LengthControl

Service ActionLogical Block AddressLogical Block AddressLogical Block AddressLogical Block Address

ReservedTransfer Length

LSB

LSB

MSB

MSB

First Byte – Operation Code

Transfer Data starting at this LBA

Number of SCSI Blocks to be transferred

Last Byte – Control Byte

SCSI Command Descriptor Block (CDB)

Command CodeGroup Code


SCSI commands are built from a common structure:

Operation Code byte

N bytes of parameters

Control byte

The Operation Code consists of a Group Code and a command Code

Group Code establishes the total command length.

Command Code establishes the command function.

The number of bytes of parameters (“N”) can be determined from the Operation Code byte which is located in byte 0 of the Command Descriptor Block (CDB).

The Control Byte, which is located in the last byte of a Command Descriptor Block, contains control bits that define the behavior of the command.



SCSI Commands and Status (cont.)

Mode Sense5Ah

Mode Select55hLog Sense4DhLog Select4Ch

Read Buffer3ChWrite Buffer3BhCopy and Verify3Ah

Compare39hSend Diagnostic1DhReceive Diagnostic Results1Ch

Mode Sense1AhCopy18hMode Select15h

Inquiry12hRequest Sense03HTest Unit Ready00h

Command NameOp Code

..ACA Active30

Task Set Full28Reservation Conflict18Intermediate-Condition Met14

Intermediate10Busy08Condition Met04

Check Condition02Good00StatusHex

After a SCSI command is sentto a target, the initiator expects a status.

Standard SCSI commands are usedon all devices.

SCSI defines commands for all devices as well as commands for specific devices. For example:

The OpCode for the General Command “Write Buffer” is 3Bh.

The OpCode for the General Command “Read Buffer” is 3Ch.

The OpCode for the Disk Command “Write(6)” is 0Ah, “Write(10)” is 2Ah.

The OpCode for the Disk Command “Read(6)” is 08h, “Read(10)” is 28h.

The numbers in the parenthesis (6) and (10) refer to the type of CDB utilized.


SCSI Messages This topic introduces the functions of SCSI messages.


SCSI Messages

SCSI messages are an additional way in which the initiator and the target communicate with each other.

Some SCSI transmission parameters are not tied to a specific command, but to the relationship between a specific initiator and target.– Transfer speed– Data width

Other asynchronous events such as:– To abort a SCSI command that is currently executed by a

target– RESTORE POINTERS

SCSI messages are an additional way in which the initiator and the target communicate with each other.

Some SCSI transmission parameters are not tied to a specific command, but to the relationship between a specific initiator and target:

Transfer speed

Data width

Other asynchronous events such as:

To abort a SCSI command that is currently executed by a target

RESTORE POINTERS



SCSI Messages (Cont.)

Error Handling:Parity BitUsed for each group of eight data bitsReceiving device calculates and compares the parityDevice that detected the parity error forces a retransmission:– RESTORE POINTERS message

U160 supports CRC

Error Handling Parallel SCSI is not as efficient in detecting transmission errors as, LAN protocols or SAN protocols.

SCSI uses a parity bit. The receiving device calculates the parity and compares it with the parity bit. If they don’t match, a parity error has occurred. Consequently, the device that detected the parity error sends a RESTORE POINTERS message that causes the data transfer counter to be reset to the value at the last disconnect so that the transfer of data is repeated from that point on.


Summary This topic summarizes the key points that were discussed in this lesson.


Summary

The SCSI protocol was originally based on parallel technology and modeled after a bus topology.The SCSI Architecture Model is a reference for the SCSI functional layers and the SCSI Transport Interfaces.To communicate, the SCSI protocol operates in phases.The SCSI protocol has a set of command codes and status codes.SCSI messages are used for error handling.

Lesson 2

FC Protocol Concepts

Overview Fibre Channel (FC) has characteristics of both I/O channels and data networks, and this unique blend of features is what makes FC ideal for storage area networks (SANs). This lesson takes a close look at the features and capabilities of FC, and compares these features and capabilities with those of traditional I/O channels such as SCSI, and data networks such as Ethernet and ATM.

Objectives Upon completing this lesson, you will be able to explain the role of Fibre Channel in a storage environment. This includes being able to meet these objectives:

Describe the basic characteristics of Fibre Channel

Describe Fibre Channel performance characteristics

Identify the three basic Fibre Channel topologies

Define a Fibre Channel port

Explain the functions of a Fibre Channel HBA

Explain the Fibre Channel Classes of Service


Fibre Channel Overview This topic describes FC as a data transport technology that is well-suited to storage networks.


Fibre Channel Overview

Fibre Channel is a technology for transporting data between devicesFibre Channel is the transport technology most commonly used for SANs today

FC

FC

IPNetwork

Fibre ChannelFabric

FCHBA

FCHBA

FC

HBA

FC

FC is a technology for transporting data between devices. It is the network interconnect technology that is most commonly used for SANs today.

Traditional storage technologies, such as SCSI, are designed for controlled, local environments. They support few devices and only short distances, but they deliver data quickly and reliably. Traditional data network technologies, such as Ethernet, are designed for chaotic, distributed environments. They support many devices and long distances, but delivery of data can be delayed.

FC combines the best of both worlds. It supports many devices and longer distances, and it provides reliable data delivery

In the diagram, the network on the right, consisting of servers and storage devices, is an FC SAN. The SAN consists of servers and storage devices connected by an FC network.



Fibre Channel: The Best of Both Worlds

Many devices

Dynamic

x High latency

Long distances

x Software-baseddelivery management

x Few devices

x Static

Low latency

x Short distances

Hardware-baseddelivery management

Many devices

Dynamic

Low latency

Long distances

Hardware-baseddelivery management

NetworkI/O Channel Fibre Channel

FC

HBA

Fibre Channel: The Best of Both Worlds This section compares FC capabilities to traditional I/O buses and data networks.

FC is designed to incorporate the best features of both channel and network architectures:

FC’s serial architecture allows it to support many devices and flexible configurations.

Like a network, FC is designed to dynamically adjust to changing environments. For example, FC allows devices to be added to the SAN with minimal disruption to ongoing communication sessions.

Storage networks are generally somewhat simpler and more “stable” than data networks (such as the Internet). Therefore, FC compromises between flexibility and efficiency in order to minimize end-to-end latency.

FC’s serial architecture supports long distances at high speeds.

FC HBAs incorporate application-specific integrated circuits (ASICs) that perform delivery management services in hardware, minimizing host CPU load and further reducing latency.



Advantages of Serial Architecture

Benefits of serial architectures:Reduces the cost and complexity of cablingUse either copper cables or optical fiberClock synchronization and data transmission are performed in one signalSimplifies product design, allowing faster evolution:– Wire speed of SCSI increases by 2x– Wire speed of Ethernet and FC increases by 10x

Advantages of Serial Architecture In a serial data transmission, the data bits are sent sequentially along a single wire. This architecture offers several advantages over the parallel architecture:

A serial architecture reduces the cost and complexity of cabling. Unlike SCSI, FC does not require terminators, and it uses a network architecture (hubs and switches) rather than a multidrop (single cable) architecture.

Serial networks can use either copper cables or optical fiber. This allows customers to choose cheaper copper cables where distance is not a requirement, and to choose more expensive optical cable when longer distances must be supported.

Because clock synchronization and data transmission are performed in one signal, rather than on separate wires, synchronization can be more easily maintained at higher link rates and longer distances.

Overall, a serial architecture simplifies product design, allowing faster evolution. For example, the wire speed of SCSI doubles with each release, while the wire speed of Ethernet and FC increases by a factor of 10 with each release.

One of the most significant advantages of serial networks is that serial networks can support longer link distances. A single-mode fiber optic FC or Gigabit Ethernet link can support links over 400 times as long as on the longest SCSI bus.


Fibre Channel Performance This section describes the performance characteristics of FC technology and compares to bus as well as networking technologies.


Fibre Channel Performance

Bandwidth:

Mode:

Maximum# of Nodes:

Link Distance:

Reliability: Bit Error Ratio < 10-12

100,200, 400, 1000 MBps(sustained, each direction)

Full duplex, serial

126 arbitrated loop~16 million switched fabric

Up to 30 m/link copperUp to 10 km/link optical

The performance characteristics of FCs are as follows:

Bandwidth: 100, 200, 400, and 1000 MBps (sustained, each direction)

Mode: Full duplex, serial

Maximum number of nodes: 126 for arbitrated loop, >16 million for switched fabric

Link Distance: Up to 30 m/link copper, to 10 km/link optical

Bit Error Ratio (BER): < 10-12

Note that 100MBps, 200MBps, 400MBps, and 1000MBps are the half-duplex rates for Fibre Channel, but Fibre Channel is actually a full-duplex technology. In other words, Fibre Channel supports up to 1000MBps between two ports in both directions simultaneously.



Fibre Channel Performance (Cont.)

The Bit Error Ratio (BER) is calculated by dividing the number of erroneous bits by the total number of bits transmittedA BER of 10-12 corresponds to one error every 8 minutes at 2 GbpsDue to some stringent applications, the industry is working on a BER of 10-15, or one error every 5.5 days at 2 Gbps

The Bit Error Ratio (BER) is calculated by dividing the number of erroneous bits by the total number of bits transmitted, received, or processed over some stipulated period. For example, 2.5 erroneous bits out of 100,000 bits transmitted would be 2.5 divided by 100,000 or 2.5 × 10-5.

The minimum and maximum values of average received power range determine the input power range required to maintain a BER less than 10-12. This value takes into account worst case signal characteristics.

A BER of 10-12 corresponds to one error every 8 minutes at 2Gbps. This might seem like a very low error rate, but due to some stringent applications, the industry is working on achieving a BER of 10-15, which results in one error every 5.5 days at 2Gbps.




Bandwidth:

Mode:

Maximum# of Nodes:

Link Distance:

Protocol Model: Monolithic (SCSI)

20-320 MBps (burst)

Half duplex,parallel, shared bus

16

1.5–25m

Layered (SCSI, IP,FICON, ESCON, etc.)

100, 200, 400, 1000 MBps (sustained)

Full duplex,serial, packet-based

126 arbitrated loop>16 million switched

Up to 30m copperUp to 10Km optical

SCSI Bus Fibre Channel

The table compares the characteristics of FC to those of SCSI. Significant differences between FC and SCSI include:

Bandwidth: FC is capable of delivering the published data rates in a sustained manner. The maximum SCSI bit rate is the peak rate, and cannot be sustained for long periods of time.

Mode: SCSI uses a parallel bus, with half duplex capability (transmission in one direction at a time), while the FC serial connection has full duplex capability.

Maximum number of nodes: 16 for SCSI, up to 16 million for FC.

The SCSI cable length limitations results in a maximum link distance of 25 meters, while FC, using optical cable, has a maximum link distance of 10 kilometers.

Software protocols: FC supports multiple protocols simultaneously. A version of the SCSI command set is often used on FC SANs, but the same SAN can simultaneously carry IP traffic and other protocols.

Note that the storage market typically measures data rates in megabytes-per-second (MBps), whereas the network market typically measures data rates in megabits-per-second (Mbps) or gigabits-per-second (Gbps). The Fibre Channel market measures data rates in both MBps and Gbps, so you must be able to quickly translate between both units of measure. In Fibre Channel, 100MBps equals 1Gbps. Note that this conversion assumes that each byte equals 10 bits. This is actually true—Fibre Channel uses a bit encoding scheme in which each 8-bit byte is encoded as 10 bits for transmission.




Bandwidth:

Mode:

Link Distance:

1000 MBps (burst)


100m copper5Km optical

100, 200, 400, 1000 MBps (sustained)


30m copper10Km optical

Gigabit Ethernet Fibre Channel

Average ContinuousData Flow:

~ 40% ~ 95%

Protocol Model: Layered Layered

The table compares the characteristics of FC to those of Gigabit Ethernet.

One notable difference is in the Average Continuous Data Flow of each network. This figure represents how well the different technologies utilize their link bandwidth, and is stated as a percentage of the maximum link bandwidth. Ethernet has significant system overheads in processing the data from high speed links, so the Average Continuous Data Flow is typically far less than the maximum bandwidth. FC, however, maximizes efficiency by implementing many functions in hardware instead of in its software drivers, and is able to achieve an Average Continuous Data Flow of up to 95 percent of maximum bandwidth.

Note that all link distances stated here are according to the specifications. Many vendors support longer distances. For example, Finisar sells long-wave GBICs that support up to 30km on single-mode optical fiber.

FC and Gigabit Ethernet support similar link distances. However, IP is the most common protocol used on Ethernet networks, and there is a global WAN infrastructure that supports IP, so people tend to think of Ethernet as spanning longer distances than FC. FC’s “distance barrier” is not its physical specification but its incompatibility with the global IP infrastructure. Today, however, emerging technologies like FCIP allow IP networks to carry FC data, breaking down that distance barrier.


Fibre Channel Topologies This section defines and compares the three FC topologies.


Fibre Channel Topologies

Arbitrated LoopSwitched Fabric

Point-to-Point

FC

FCHBA

FC

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

FC

FC

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

Fibre Channel Protocol includes three basic SAN topologies:

Point-to-point

Arbitrated loop

Switched fabric



What Is the Point-to-Point Topology?

Dedicated 1-to-1 connection between two nodesThis is really a DAS architecture, but offers better performance and flexibility than SCSI

FC

FC

HBA

What is the Point-to-Point Topology? The Point-to-Point topology is the simplest FC storage configuration. As its name suggests, a Point-to-Point configuration is a 1-to-1 connection between a host and a storage device.

The preceding graphic illustrates an example of the simplest Point-to-Point implementation: one server connected via an FC link to a storage device.



What Is the Arbitrated Loop Topology? Limitations of Arbitrated Loops

Performance:– One data path means only one pair of devices can

communicate at a time: shared bandwidth– Higher latency than fabrics

Scalability:– 127 addressable ports:

126 available for nodes1 reserved for fabric-attach port

– About a dozen nodes in practice

Reliability:– If one device fails, the entire loop

can fail

Hub

FC

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

What is the Arbitrated Loop Topology? The key characteristics of the Arbitrated Loop topology are:

All servers can have access to all storage.

I/O speeds depend upon the location of the initiator and target in the loop.

Arbitrated loops are practical for 12 or fewer devices.

The key limitations of the Arbitrated Loop topology are:

Loops suffer from poor performance. Because there is only one data path, only one pair of devices can communicate at a time. This means that all the devices on the loop must share the available bandwidth.

Loops have a higher latency than fabrics. Devices must negotiate for control of the loop.

Loops are not very scalable. Because loops were designed to support a small number of devices, the FC-AL protocol provides only 127 unique addresses: 126 addresses are usable for nodes (hosts and storage devices), and 1 address is reserved for attaching the loop to an FC switched fabric.

In reality, only about a dozen devices can be connected in a loop before performance drops below acceptable levels.

Loop configurations are susceptible to device failures.



What Is the Switched Fabric Topology?

FC

FC

FC

FC

FC

FCFC

FC

FC

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

FCHBA

FC

HBA

FCHBA

FCHBA

What is the Switched Fabric Topology? The Switched Fabric topology incorporates a high-bandwidth FC switch, instead of a hub, to handle data traffic among host and storage devices.

The diagram illustrates a sample switched fabric topology. The scalability of the fabric is indicated by the fact that multiple switches can be linked together to support many devices.



The FC Switched Fabric Protocol (FC-SW)

Supports multiple “conversations” at full link speedAllows over 16,000,000 device addressesEnhanced management capabilities:– Security services– Multicast and broadcast– Remote management

The majority of modern organizations choose to implement a fabric

The Fibre Channel Switched Fabric (FC-SW) protocol differs from the arbitrated loop topology in several important ways:

Switches can support multiple simultaneous “conversations”. Each “conversation” between two devices can use the full link bandwidth.

The FC-SW device addressing scheme allows over 16,000,000 ports. Existing implementations can support hundreds and even thousands of nodes using large director-class switches.

The FC-SW protocol defines several management services that increase the scalability, manageability, and security of the SAN.

Due to the limitations of the Arbitrated Loop topology, the majority of modern organizations choose to implement a Switched Fabric topology because it offers greater scalability, performance, reliability, and manageability.


Fibre Channel Ports This section describes an FC port, and the role it plays in a SAN.


Fibre Channel Ports

Ports are intelligent interface points on the Fibre Channel network:

Embedded in an I/O adapterEmbedded in an array or tape controllerEmbedded in a fabric switchPorts understand Fibre Channel

FCPorts

ServerStorage

SwitchI/O Adapter Arraycontroller

Tape deviceFC

FC

HBA

In data networking terminology, ports are often thought of as just physical interfaces where you plug in the cable. In FC, however, ports are intelligent interfaces, responsible for actively performing critical network functions.

The preceding graphic contains several ports. There are ports in the host I/O adapter (host bus adapter [HBA]), ports in the switch, and ports in the storage devices.

FC terminology differentiates between several different types of ports, each of which performs a specific role on the SAN. You will encounter these terms often as you continue to learn about FC, so it is important that you learn to recognize the different port types. In addition to the common ports defined for FC, Cisco has developed some proprietary port types.



FC

HBA

FC

HBA

Fibre Channel Ports (Cont.)

Hub

StorageArray

StorageArray

Host

Host

FC

FC

WAN Bridge

Standard Ports

N_Port F_Port

E_PortB_Port

NL_Ports

FL_Port

E_Ports

An N_Port (Node Port) is a port on a node that connects to a fabric:

I/O adapters and array controllers contain one or more N_Ports

N_Ports can also directly connect two nodes in a point-to-point topology

An F_Port (Fabric Port) is a port on a switch that connects to an N_Port.

An E_Port (Expansion Port) is a port on a switch that connects to an E_Port on another switch.

An FL_Port (Fabric Loop Port) is a port on a switch that connects to an arbitrated loop.

Logically, an FL_Port is considered part of both the fabric and the loop.

FL_Ports are always physically located on the switch.

Note that FC hubs, although they obviously have physical interfaces, do not contain FC ports. Hubs are basically just passive signal splitters and amplifiers. They do not actively participate in the operation of the network. On an arbitrated loop, the node ports manage all FC operations.

Not all switches support FL_Port operation. For example, some McDATA switches do not support FL_Port operation.

An NL_Port (Node Loop Port) is a port on a node that connects to another port in an arbitrated loop topology. There are two types of NL_Ports:

Private NL_Ports can communicate only with other loop ports.

Public NL_Ports can communicate with other loop ports and with N_Ports on an attached fabric.

Note that the term L_Port (Loop Port) is sometimes used to refer to any port on an arbitrated loop topology. “L_Port” can mean either “FL_Port” or “NL_Port”. In reality, there is no such thing as an L_Port.


Fibre Channel HBAs This section describes typical and differentiating features of HBAs.


Fibre Channel HBAs

Ethernet NIC

HBA

Flow ControlSequencingSegmentationError Correction

Fibre ChannelHBATCP Driver

I/O Subsystem

OS

FC Driver

I/O Subsystem

OS

Flow ControlSequencingSegmentationError Correction

HBAs are I/O adapters that are designed to maximize performance by performing protocol processing functions in silicon. HBAs are roughly analogous to network interface cards, but HBAs are optimized for storage networks, and provide features that are specific to storage.

The figure contrasts HBAs with NICs, illustrating that HBAs offload protocol processing functions into silicon.

With NICs, protocol processing functions such as flow control, sequencing, segmentation and reassembly, and error correction are performed by software drivers. HBAs offload these protocol processing functions onto the HBA hardware itself—usually some combination of an application-specific integrated circuit (ASIC) and firmware. Offloading these functions is necessary to provide the performance required by storage networks.

NICs can utilize over 80 percent of a server’s CPU capacity (measured with a 1Ghz Intel Pentium CPU) to deliver 50-80MBps on a Gigabit Ethernet link. I/O processing adds considerable real cost to what may appear to be an inexpensive NIC.

HBAs manage I/O transactions with little or no involvement of the server CPU. FC HBAs can provide throughput at nearly 95 percent of link speed with less than 10 percent server CPU utilization.


Fibre Channel Classes of Service This section identifies the Classes of Service that are commonly used on FC SANs and are supported by commercially available FC products.


Fibre Channel Classes of ServiceCharacteristics Use

Class 1

Class 2

Class 3

Class 4

Class 6

Class F

Connection-orientedConfirmed delivery

Specialized applications; not widely supported

Packet-switchedConfirmed delivery

Generally supported but not widely used

Packet-switchedNo delivery confirmation

Most commonly used Class of Service

Fractional bandwidth virtual circuitConfirmed delivery

Specialized applications; not supported in SAN products

Connection-oriented multicastConfirmed delivery

Specialized applications; not supported in SAN products

Packet-switchedConfirmed delivery

Used for inter-switch communication

The table displays uses of the FC Classes of Service:

Few commercially available FC SAN products currently support Class 1.

Many FC products support Class 2, but it is not widely used.

Class 3 is by far the most commonly used Class of Service on fabrics, and it is often the only class supported on arbitrated loops. All FC SAN products support Class 3.

No commercially available FC SAN products currently support Class 4.

No commercially available FC SAN products currently support Class 6.

Class F is always used for inter-switch communication.

Note that Class 5 is not yet defined. Class 5 was intended to enable isochronous transactions by multiple ports, but has not been completed. An isochronous connection is one in which bandwidth and data delivery rate are guaranteed. Class 5 would be appropriate for video delivery services.



Fibre Channel Classes of Service

Attribute Class 1 Class 2 Class 3 Class 4 Class 6

Connection Oriented Yes No No Yes Yes

Bandwidth Reserved 100% No No Fractional 100%

Guaranteed Latency Yes No No Yes(QoS) Yes

Guaranteed Delivery Order Yes No No Yes Yes

Delivery Confirmation Yes Yes No Yes Yes

Packet-switched No Yes Yes No No

The preceding table summarizes the features of the Classes of Service. Although Classes 2 and 3 are the only options currently available in Fibre Channel products today, customers might have specialized applications that call for the features of other classes—and might be willing to investigate specialized products that support those applications.




Summary

Fibre Channel supports many devices, dynamic network reconfiguration, low latency, long distances, and hardware-baseddelivery management.Fibre Channel currently supports 100, 200, 400 and 1000 MBps.The Fibre Channel Protocol supports three topologies: Point to Point, Arbitrated Loop and Switched Fabric.Ports are intelligent interface points on the Fibre Channel network.Fibre Channel HBAs offload flow control, sequencing, segmentation, and error correction into the HBA hardware, increasing performance.Fibre Channel has defined classes of service similar to the Class of Service models in LAN networks, however the Fibre Channel implementation is different.

Lesson 3

FC Layers

Overview Like nearly all modern networks, Fibre Channel (FC) is designed with a modular, layered architecture. This architecture is designed to carry other protocols, as well as new native protocols. A layered architecture provides benefits for both vendors and users because it enhances the clarity and flexibility of the architecture. This lesson describes the five layers of the FC layered model, and the upper layer protocols that FC supports.

Objectives Upon completing this lesson, you will be able to describe the Fibre Channel layered model, data constructs, SCSI-FCP read and write operations, and Link Services. This includes being able to meet these objectives:

Describe the FC layered model

Describe the Fibre Channel Data constructs

Describe SCSI-FCP protocol operations

Describe Link Services


Fibre Channel Layers This section introduces the FC layered architecture.


Fibre Channel Layers

Ethernet

Application

Presentation

Session

Transport

Network

Data link

Physical

TCP • SPX • ...

HTTP • FTP •SNMP • ...

OSI Layers

Physical

Logical

NTFS • CIFS •NFS • DAFS • ...

SCSI • IP • VI •HiPPI • ...

Fibre Channel

IP • IPX • ...

Fibre Channel IP

The OSI model defines seven layers of functionality for network protocols. While FC does not map directly to the OSI model, it does use a layered model. FC’s lower layers relate closely to the lower layers of OSI:

FC defines the lower three layers (approximately) of the OSI model: Physical, Data link, and Network

Other protocols, such as SCSI, are responsible for the upper layers

If you are familiar with data networking, you probably understand the difference between physical-layer protocols, such as Ethernet, and “logical-layer” protocols, such as TCP and IP:

Ethernet defines how data is physically transmitted.

Protocols like TCP and IP define aspects of the network such as flow control and addressing.

The preceding graphic shows that FC defines both the physical layer and part of the logical layer, and then interfaces with ULPs that perform the functions of the upper layers of the OSI model.



Fibre Channel Layers (Cont.)

NTFS • CIFS •NFS • DAFS • ...

Fibre Channel

SCSI • IP • VI •HiPPI • ...

FC-0 Physical interface

FC-1 Encoding

FC-2 Framing and flow control

FC-3 Common Services

Upper-layerprotocols

FC-4

FC-FS

FC-PI

The five layers of FC are:

FC-0: Physical interface, transmission and signaling

FC-1: 8b/10b encode/decode, link control, ordered set specifications

FC-2: Framing, flow control, exchange/sequence management

FC-3: Application-specific layer for fabric services

FC-4: Upper-layer protocol mapping specification

The lower three layers (FC-0, FC-1, and FC-2) are collectively known as the FC Physical Layer (FC-PH), even though they also implement logical functions such as framing and flow control. The FC-3 layer provides a framework for implementing new SAN-wide services, while the FC-4 layer interfaces with the ULPs and maps them to the FC.

The FC-PH specification was the original document that defined layers FC-0, FC-1, and FC-2. The final version of the FC-PH specification was FC-PH-3. However, FC-PH was then superceded by two additional documents:

Fibre Channel Physical Interface (FC-PI) defines FC-0.

Fibre Channel Framing and Signaling (FC-FS) defines FC-1 and FC-2




FC-1 Encoding



ULPsFC-4

FC-FS

FC-PI

FC-0: Physical Interface

FC-0 specifies the physical characteristics of the data link:

Cables and connectorsTransmitter and receiver functionsSignaling protocol

FC-0: Physical Interface The FC-0 layer specifies the characteristics of the physical links. FC-0 is responsible for:

Specifying signaling protocols for transmitting and receiving a signal at different transfer rates

Specifying data rates and maximum transmission distances



Optical Media Types

TwinAx, QuadAx 33m

59m

300m

500m

10km†

30m–100m

†30km is supported by some manufacturers

150m

300m

10km†

70m

150m

10km

Coaxial

Shielded Twisted Pair

Multimode 62.5µ

Multimode 50µ

Single-mode 9µ

4Gb

1Gb2Gb4Gb

2Gb1Gb

1Gb2Gb

4Gb Optical Cables

Electrical Cables

Optical media types:

Multimode fiber uses a 780nm short-wave laser.

Single-mode fiber uses a 1300nm long-wave laser.

Maximum link distances vary by data rate.

The current specification states a minimum 2m distance for optical fiber. This is to allow for a build-up of photons that occur in the first 2m of cable after the laser fires, and, in multimode cables, to eliminate problems associated with some modes of light which, due to their steep angle of reflection, do not travel very far down the cable.



FC-1: Encoding

FC-1 defines the bit encoding scheme:Encoding and decoding of serial signalsBit-level error detectionClock synchronizationLink initialization andrecovery


FC-1 Encoding



ULPsFC-4

FC-FS

FC-PI

FC-1: Encoding The FC-1 layer specifies how data is encoded at the bit and byte levels for transmission across the link. FC-1 is responsible for:

Taking data from the transmitter’s I/O bus and encoding into a serial signal for transmission

Taking a serialized signal and decoding it into a signal that can be sent to the receiver’s I/O bus

Bit-level error detection

Clock synchronization between the transmitter and receiver

Link initialization and recovery



FC-1: Encoding (Cont.)

Each 8-bit data byte is encoded into a 10-bit character before it is transmitted over the link:

Transmitter

Parallel Input Serial Output Media Output

8b/10bEncoder

Tx Byte

Parallel/SerialConverter(SERDES)

The preceding diagram shows the FC-1 encoding process:

The data from the transmitting I/O bus is encoded using the 8b/10b encoding scheme.

The parallel data is converted to serial format.

The serial data is transmitted across a link.

The receive end decodes the serial data and forwards it to the receiver’s I/O bus.



FC-1: Encoding (Cont.)

8b/10b Encoding Scheme:– Transmission Characters always have either:

6 ones and 4 zeros = Positive disparity4 ones and 6 zeros = Negative disparity5 ones and 5 zeros = Neutral disparity

0101000110

1010110110

-

+110111110xDF

6 31 = D31.6

DisparityEncodedBitsByte

Transmission characters always have either:

Positive disparity: 6 ones and 4 zeros

Negative disparity: 4 ones and 6 zeros

Neutral disparity: 5 ones and 5 zeros

The 8b/10b scheme defines multiple transmission characters for each 8-bit data byte. Because the encoder can choose between multiple 10-bit representations for each 8-bit byte, it can balance the number of ones and zeros in the data stream. The imbalance between the number of 1s and 0s—known as the running disparity—is continually reevaluated. To balance the number of ones and zeros, every transmitted byte is encoded into one of two possible 10-bit representations depending on the current running disparity.

In the FC-1 specification, every 10-bit character is represented using a special notation:

Dxx.y: Used for data characters that map to 8-bit characters; xx is the decimal value of the lowest 5 bits and y is the decimal value of the highest bits.

Kxx.y: Used for special control characters; xx and y are defined as for data characters.



FC-2: Framing and Flow Control

FC-2 defines the structure, organization, and delivery of the data:

Constructs and manages framesInserts addressing andheader informationManages flow controlFrame error detection


FC-1 Encoding



ULPsFC-4

FC-FS

FC-PI

FC-2: Framing and Flow Control The FC-2 layer specifies how the data is packaged for transmission:

To transport a stream of data from one port to another, the data must be packaged into discrete packets, or frames. At the receiving end, the data must then be extracted from the frames.

A frame carries identification information, such as addressing information and information that identifies how the network should serve, deliver and respond to this particular frame. The FC-2 layer is responsible for inserting this information into each frame.

The flow of frames across the network must be also be controlled so that the sending port does not send data faster than the receiving port can receive it. The FC-2 layer performs this flow control function by initiating or preventing transfer of frames.

FC-2 is also responsible for detecting frame-level errors.

FC-2 is the “workhorse” of FC. It is the most complex of the FC layers.



FC-3: Common Services

FC-3 is an “expandable” layer that is designed to support services, such as:

Name serverSecure key serverManagement serverTime server

Future services?CompressionEncryptionLink multiplexing


FC-1 Encoding



ULPsFC-4

FC-FS

FC-PI

FC-3: Common Services The FC-3 layer defines a generalized structure for implementing new services:

“Generic” services are functions that can span multiple ports and can be applied to multiple upper-layer protocols.

FC defines generic fabric services that include a name server, a secure key distribution server, a management server, and a time server.

Examples of potential future services include data compression, encryption services, or multiplexing multiple links to form one aggregated virtual link.

Generic services use the Fibre Channel Common Transport (FC-CT) protocol to communicate and distribute functions between switches.



FC-4: Upper-Layer Protocol Interfaces

FC-4 maps upper-layer protocols to the FC protocol:SCSI, HiPPI, ESCON/FICON for storageIP, VI, ATM and othersAllows multiple protocolsto be transported over thesame physical interface


FC-1 Encoding



ULPsFC-4

FC-FS

FC-PI

FC-4: Upper-Layer Protocol Interfaces The FC-4 layer defines how the ULPs map to the lower layers of FC. It allows multiple protocols to be transported over the same physical interface. FC-4 makes sure that the ULP data or commands are broken down appropriately and packaged correctly into FC frames.

When a sender transmits a block of data:

The FC-4 layer receives the data from the ULP drivers and passes the data down to the FC-3 layer.

The FC-3 layer performs any required manipulation on the data, such as compressing or encrypting the data, or it simply passes the data directly to the FC-2 layer.

The FC-2 layer then packages the data into frames and passes it down to the FC-1 layer.

The FC-1 layer encodes each byte into 10-bit characters using the 8b/10b scheme and passes the data down to the FC-0 layer.

The FC-0 layer encodes the data into physical signals and transmits those signals across the link.

Only the FC-4 layer needs to know which ULP is used. The protocol-independence of the FC-3 through FC-0 layers allows FC to be easily adapted to new ULPs.



FC-4: Upper-Layer Protocol Interfaces (Cont.)

95% of FC network utilization is for SCSI applications

FCP-SCSI FC-SB-2 FC-LE FC-IP FC-FP FC-BB-2

SCSI-3 FICON,ESCON

IEEE802.2 LLC IP HiPPI ATM,

SONET


FC-1 Encoding


FC-2 Framing and Flow Control

The Small Computer System Interface (SCSI) command set is widely used among storage devices. Even though SCSI bus technology is not suitable for SANs, the SCSI command set is well-suited for many types of storage applications.

The use of the SCSI command set enables the use of inexpensive SCSI disks and SCSI tape drives in FC SAN storage devices. SCSI-FCP also enables compatibility with existing operating systems and legacy storage applications. In fact, most operating systems and applications are not “aware” of the FC SAN—FC devices appear to the host and its applications as SCSI devices.

The mapping of the SCSI protocol to FC is called SCSI-Fibre Channel Protocol (SCSI-FCP), or sometimes simply FCP. SCSI-FCP is the ULP command set used on most FC SANs. SCSI-FCP provides the command set for reading and writing data to and from storage devices.

The fact that FC supports a wide range of protocols allows FC to meet the needs of diverse applications and integrate with heterogeneous platforms. FC supports the following existing ULP protocols:

The Enterprise Systems Connection (ESCON) protocol is a storage interconnect used in IBM mainframe computing environments. The Fibre Connection (FICON) protocol allows ESCON assets to be used within an FC SAN infrastructure. The FC-SB-2 standard maps FICON to FC-2.

The IEEE 802.2 standard defines the generic logical link control (LLC) layer in the OSI Reference Model. The FC-LE standard helps map IEEE 802.2-based protocols to FC.

IP is the protocol that drives the Internet. FC-IP allows FC to carry the IP protocol. Servers can use IP to communicate with each other over the SAN.

High Performance Parallel Interface (HiPPI) connects devices at short distances and high speeds. HiPPI is used primarily to connect supercomputers and to provide high-speed backbones for LANs. The FC-FP standard maps HiPPI to FC.

The FC-BB-2 standard enables FC to exchange data with ATM and Synchronous Optical Network (SONET) networks for long-haul transport of FC data.


Fibre Channel Data Constructs This section describes the Fibre Channel Data constructs: Frames, Sequences, Exchanges, and Ordered Sets.


Fibre Channel Data Constructs

FCSequence

Word

Frame

Exchange

Initiator Target

FC

HBA

The preceding graphic shows a transaction between a host (initiator) and a storage device (target):

The smallest unit of data is a word. Words consist of 32 bits (4 bytes) of data that are encoded into a 40-bit form by the 8b/10b encoding process.

Words are packaged into frames. An FC frame is equivalent to an IP packet.

A sequence is a series of frames sent from one node to another node. Sequences are unidirectional—in other words, a sequence is a set of frames that are issued by one node.

An exchange is a series of sequences sent between tow nodes. The exchange is the mechanism used by two ports to identify and manage a discrete transaction. The exchange defines an entire transaction, such as a SCSI read or write request. An exchange is opened whenever a transaction is started between two ports and is closed when the transaction ends. An FC exchange is equivalent to a TCP session.



Fibre Channel Frames

Payload CRC EOF

SOF

Header

14

624

0–5280–2112

14

1 = 5374 = 2148

OptionalHeaders Data or commands Fill

Bytes

0–160–64

WordsBytes

= 5280-3 = 2112

0–5120–2048

WordsBytes

Fibre Channel Frames The maximum total length of an FC frame is 2148 bytes, or 537 words. This consists of:

A 4-byte SOF delimiter

A 24-byte header

A data payload that can vary from 0 to 2112 bytes

A 4-byte (32-bit) CRC that is used to detect bit-level errors in the payload

A 4-byte EOF delimiter

The frame payload consists of 3 elements:

The payload itself, containing data or commands, can be up to 2112 bytes.

The first 64 bytes of the payload can be used to incorporate optional headers. This would reduce the data payload size to 2048 bytes (2KB).

The payload ends with 1-3 fill bytes. This is necessary because the smallest unit of data recognized by FC is a 4-byte word. However, the ULP is not aware of this FC requirement, and the data payload for a frame might not end on a word boundary. FC therefore adds up to 3 fill bytes to the end of the payload—as many as are needed to ensure that the payload ends on a word boundary.



Frame Channel Frames (Cont.)

FC protocol trace – FLOGI example:

Ordered Sets (OS)• R-Rdy• Idle

Frames (F)

The screen image displays an FC protocol trace. A single FC frame—Fabric Login (FLOGI)—is displayed in the right-hand window. Each word in the frame is depicted on a separate line, beginning with the SOF Frame Delimiter (SOFi3) and ending with the EOF Frame Delimiter (EOFt). The display shows the 6 words in the frame header, 29 words in the payload, and the 32-bit CRC.



Frame Headers

Payload CRC EOF

SOF

Header

Bit24 23 16 15 8 7 031Word

R_CTLCS_CTL

TYPESEQ_ID

OX_IDDF_CTL

RX_IDSEQ_CNT

F_CTL

Parameter

S_IDD_ID0

12345

Frame Headers These are the header fields of an FC frame:

R_CTL (Routing Control, 8 bytes): Frame type and function; used by the switch to route frames

CS_CTL (Class Specific Control, 8 bytes): Class specific control information for Class 1, 4 & 6

D_ID (Destination ID, 24 bytes): 24-bit address of the destination port

S_ID Source ID, 24 bytes): 24-bit address of the source port

TYPE (Data Structure Type, 8 bytes): Type of Information Unit & ULP carried by this frame

F_CTL (Frame Control, 24 bytes): Specifies number of fill bytes and sequence control information

SEQ_ID (Sequence ID, 8 bytes): Unique ID for each sequence

SEQ_CNT (Sequence Count, 16 bytes): Frame count identifying each frame in the sequence

DF_CTL (Data Frame Control, 8 bytes): Information about optional headers

OX_ID (Originator ID, 16 bytes): Unique ID set by the exchange originator

RX_ID (Receiver ID, 16 bytes): Unique ID set by the exchange responder

Parameter (Parameter or Offset, 32 bytes): Used for multi-purpose parameters, such as buffer offset.



Ordered Sets

Transmission Word

Ordered SetK28.5, Dxx.y, Dxx.y, Dxx.y

Data WordDxx.y, Dxx.y, Dxx.y, Dxx.y

Primitive SequenceFrame Delimiter Control SignalFill Word

Primitive Signal

Start-of-FrameEnd-of-Frame

IdleArbitrate

Receiver ReadyVirtual Circuit ReadyCloseOpenDynamic Half-DuplexMarkSynchronize

Non-Operational StateOffline StateLink ResetLink Reset ResponseLoop InitializationLoop Port BypassLoop Port Enable

Ordered Sets are FC words (5 bytes) that are used for link-level functions. They are used because they are fast and light, and because commands sometimes need to be exchanged before devices have been assigned FC addresses.

The first byte of an Ordered Set is always the K28.5 character, which defines the word as an Ordered Set. The second byte identifies the Ordered Set type, and the last two bytes can be used to transmit other parameters.

There are three types of Ordered Sets:

Frame Delimiters are used to mark the beginning and end of frames.

Primitive Signals are used to initiate, synchronize, and terminate communication sessions, and to maintain synchronization when no other information is being transmitted on the link. The two types of Primitive Signals are fill words and control signals.

Primitive Sequences are similar to Primitive Signals, but are transmitted repeatedly until a response is received. They are used for link and loop initialization.


SCSI-FCP Operations This section provides a brief overview of SCSI-FCP protocol operations.


ULP Information Transfer

FCP_RSP 5

FCP_DATA 3

FCP_CMD1 Sequence 1

Target

Sequence 3

Sequence 2

SCSI-FCP Read Operation

Initiator Fabric

Frame2

Frames

4

Frame 6 IU 3

IU 1

IU 2Exchange

The preceding diagram illustrates a SCSI-FCP read operation:

1. The initiator node generates a SCSI read request (FCP_CMD), which is packaged as IU 1.

2. The initiator FC-2 layer converts IU 1 to a single command chunk and sends it across the fabric as a single frame. This constitutes Sequence 1.

3. The target node processes IU 1, retrieves the requested data (FCP_DATA) from storage and packages the data as IU 2.

4. The target FC-2 layer converts IU 2 to one or more data chunks and sends them across the fabric. This constitutes Sequence 2.

5. The target node then generates a status command (FCP_RSP) that informs the initiator that the requested data transmission is complete. The status command is packaged as IU 3.

6. The target FC-2 layer converts IU 3 to a single command chunk and sends it across the fabric. This constitutes Sequence 3.

At this point, the I/O operation is complete. The collection of three sequences constitutes a single exchange.



ULP Information Transfer (Cont.)

FCP_RSP 7

FCP_DATA5

FCP_XFR_RDY 3

FCP_CMD1 Sequence 1

Target

SCSI-FCP Write Operation

Initiator

Exchange

Fabric

Sequence 3

Sequence 2

Sequence 4

Frame2

Frame 4

Frames

6

Frame 8

IU 1

IU 2

IU 3

IU 4

The preceding diagram illustrates a SCSI-FCP write operation:

1. The initiator node generates a SCSI write request (FCP_CMD), which is packaged by the FC-4 layer as IU 1.

2. The initiator FC-2 layer converts IU 1 to a single command chunk and sends it across the fabric as a single frame. This constitutes Sequence 1.

3. The target node responds with a SCSI write request response (FCP_XFR_RDY), which is packaged as IU 2. The write request response is required for synchronization between the initiator and target.


5. The initiator node retrieves the data (FCP_DATA) from its ULP buffers and packages it as IU 3.

6. The initiator FC-2 layer converts IU 3 to one or more data chunks and sends them across the fabric. This constitutes Sequence 3.

7. The target then generates a status command (FCP_RSP) to confirm the end of the exchange. The command is packaged as IU 4.


The collection of four Sequences constitutes a single Exchange.


Link Services This section describes what a Link Services command is, the role of Link Services, and how Link Services differ from Ordered Sets.


Fibre Channel Link Services

ULP-independent Fibre Channel commandsImplement session management functions:– Fabric and port login– Address resolution– Error recovery

Defined within the FC-CT framework– Thus carried within a frame construct

N_Port N_Port

ABTS

BA_ACC

Link Services are upper-layer protocol (ULP) independent FC commands. Link Services are used to implement control functions used in session management, such as fabric and port login, address resolution, and error recovery. Link Services are defined within the Fibre Channel Common Transport (FC-CT) framework.

Link Services are transparent to ULPs. In other words, Link Services frames are generated by the initiator N_Port, not by the ULP driver. Upon receiving a Link Services command, the target N_Port processes and discards all Link Services frames.

The preceding diagram shows an example of a Link Services exchange:

The N_Port on the left has sent an ABTS Link Services command to attempt to terminate the current FC sequence.

The N_Port on the right receives the ABTS request and responds with the BA_ACC Link Services command to indicate that the N_Port has successfully processed the ABTS request.

Note Ordered Sets are not Link Services. Ordered Sets are short, one-word (four-byte) commands that can carry, at the most, two bytes of parameters, whereas Link Services commands consist of one or more FC frames. Ordered Sets are typically used at the physical layer to perform basic link management functions. Link Services comprise a higher-level command set that is essential to performing session management and error recovery.



Types of Link Services

Basic Link Services:Implement a handful of basic control functionsMust be supportedSingle frame (can be inserted into a sequence)

Extended Link Services:Perform a variety of control functions at the FC-2 layerNew commands are added regularly to provide additional control functionsOnly a few must be supportedAlways sent as a separate exchange

Types of Link Services There are two types of link services:

Basic Link Services implement just a handful of basic control functions. All Basic Link Services must be supported by all FC devices. Each Basic Link Service command is transmitted as a single frame—either as a single-frame sequence or as a frame inserted into a longer sequence.

Extended Link Services are used to perform a variety of control functions between ports at the FC-2 layer. Extended Link Services are added regularly by the ANSI committee to provide additional control functions. Only a few Extended Link Services are required by the FC specification; most are optional. The number and functionality of Extended Link Services are constantly evolving. Extended Link Services are always sent as a separate exchange.



Basic Link Services

Pre-empt Dedicated Connection (Class 1 or 6)PRMT

Remove Dedicated Connection (Class 1 or 6)RMC

Basic Link Service RejectBA_RJT

Basic Link Service AcceptBA_ACC

Abort SequenceABTS

No OperationNOP

Basic Link Services The Basic Link Services are as follows:

NOP performs no specific operation but can be used to carry control information in the header, such as initiating a Class 1 or Class 6 connection.

ABTS is used to abort the current sequence or an entire exchange. Only the ABTS command has a reply: BA_ACC if accepted and BA_RJT if rejected.

BA_ACC is the normal response to an ABTS command

BA_RJT is an abnormal response to an ABTS command if the ABTS request cannot be processed. There are several reasons why this may occur, including an invalid OX_ID or RX_ID.

RMC is used to request an immediate Class 1 or 6 disconnection. The recipient will abort any current sequences and exchanges resulting in possible frame loss and will send a final ACK frame.

PRMT is used to terminate a Class 1 or 6 connection if it is required for a higher priority connection.



Extended Link Services

Required byFibre Channelstandard

Supported byall SANimplementations

Used for FC Ping and FC TraceECHO

Extended Link Services AcceptLS_ACC

Extended Link Services RejectLS_RJT

State Change RegistrationSCR

Registered State Change NotificationRSCN

State Change NotificationSCN

Process LogoutPRLO

Process LoginPRLI

Port LogoutLOGO

Port LoginPLOGI

Fabric LoginFLOGI

Examples:

Extended Link Services The Fibre Channel standard requires support for only a few Extended Link Services:

FLOGI is used by an N_Port to log in to an F_Port and obtain a Fibre Channel address.

PLOGI is used by one N_port to log in to another N_Port, open a session, and exchange service parameters.

LOGO is used to terminate a login session and free its associated resources.

LS_ACC notifies the sender of an Extended Link Service command that the command has been accepted.

LS_RJT notifies the sender of an Extended Link Service command that the command was not accepted. The command might have been rejected because the port was busy at the time, the command was invalid or malformed, or the command is not supported by the receiver. The reason code in the payload gives a more precise reason for the rejection.




Summary

Fibre Channel has a layered model that is similar in some respects to the OSI model.Fibre Channel constructs include words, frames, sequences and exchanges. The Fibre Channel protocol is broken down into control and data units. The control units in Fibre Channel are called Ordered Sets.Link Services are additional communications from port to port used to relay logins, state changes, error situations and other administration messages.

Lesson 4

FC Flow Control

Overview Like any network protocol, Fibre Channel (FC) must define how the flow of data is managed. FC defines two flow control processes that are used either individually or together. FC uses a unique receiver-based flow control strategy that ensures that data is delivered efficiently and with a minimum of delivery errors.

Objectives Upon completing this lesson, you will be able to explain Fibre Channel flow control and addressing. This includes being able to meet these objectives:

Explain the Fibre Channel flow control process

Calculate the number of buffer credits needed for a FC link

Explain the Fibre Channel addressing schemes

Explain the function of World Wide Names


Fibre Channel Flow Control This section reviews the fundamental objectives of flow control and provides an example of a flow control strategy.


Fibre Channel Flow Control

How data interchange is controlled in a networkThe flow control strategy used by Ethernet and other data networks can degrade performance:– Transmitter does not stop transmitting packets until after the receiver’s buffers

overflow– Lost packets must be retransmitted– Degradation can be severe under heavy traffic loads

PAUSERx

Flow Control in Ethernet

DataDataData DataData Data

Lost packets

Tx

Flow control is a mechanism for ensuring that frames are sent only when there is somewhere for them to go. Just as traffic lights are used to control the flow of traffic in cities, flow control manages the data flow in an FC fabric.

Some data networks, such as Ethernet, use a flow-control strategy that can result in degraded performance:

A transmitting port (Tx) can begin sending data packets at any time.

When the receiving port’s (Rx) buffers are completely filled and cannot accept any more packets, Rx “tells” Tx to stop or slow the flow of data.

After Rx has processed some data and has some buffers available to accept more packets, it “tells” Tx to resume sending data.

This strategy results in lost packets when the receiving port is overloaded, because the receiving port tells the transmitting port to stop sending data after it has already “overflowed”. Lost packets must be retransmitted, which degrades performance. Performance degradation can become severe under heavy traffic loads.



Credit-Based Flow Control

Fibre Channel uses a credit-based strategy:– Transmitter does not send a frame until the receiver “tells” the

transmitter that the receiver can accept another frame– The receiver is always in control

Benefits:– Prevents loss of frames due to buffer overruns– Maximizes performance under high loads

DATA

READYREADYTx Rx

Port Rx has0101

free buffers

Flow Control in Fibre Channel

Credit-Based Flow Control To improve performance under high traffic loads, FC uses a credit-based flow control strategy in which the receiver must issue a credit for each frame that is sent by the transmitter before that frame can be sent.

A credit-based strategy ensures that the receiving port is always in control. The receiving port must issue a credit for each frame that is sent by the transmitter. This strategy prevents frames from being lost when the receiving port runs out of free buffers. Preventing lost frames maximizes performance under high traffic load conditions because the transmitting port does not have to resend frames.

The preceding diagram illustrates a credit-based flow control process:

The transmitting port (Tx) counts the number of free buffers at the receiving port (Rx).

Before Tx can send a frame, Rx must notify Tx that Rx has a free buffer and is ready to accept a frame. When Tx receives the notification, it increments its count of the number of free buffers at Rx.

Tx only sends frames when it knows that Rx can accept them.



Types of Flow Control

Fibre Channel defines two types of flow control:Buffer-to-buffer (port-to-port)End-to-end (source-to-destination)

Buffer-to-buffer flow control

N_Port F_Port E_Port E_Port F_Port N_Port

End-to-end flow control

Types of Flow Control FC defines two types of flow control:

Buffer-to-buffer flow control takes place between two ports that are connected by a FC link, such as an N_Port and an F_Port, or two E_Ports, or two L_Ports.

End-to-end flow control takes place between the source node and the destination node.

Note that buffer-to-buffer is performed between E_Ports in the fabric, but it is not performed between the incoming and outgoing ports in a given switch. In other words, FC buffer-to-buffer flow control is not used between two F_Ports or between an F_Port and an E_Port within a switch. FC does not define how switches route frames across the switch.

Buffer-to-buffer flow control is used in the following situations:

Class 1 connection request frames use buffer-to-buffer flow control, but Class 1 data traffic uses only end-to-end flow control.

Class 2 and Class 3 frames always use buffer-to-buffer flow control.

Class F service uses buffer-to-buffer flow control.

In an Arbitrated Loop, every communication session is a virtual dedicated point-to-point circuit between a source port and destination port. Therefore, there is little difference between buffer-to-buffer and end-to-end flow control. Buffer-to-buffer flow control alone is generally sufficient for arbitrated loop topologies.

End-to-end flow control is used in the following situations:

Classes 1, 2, 4, and 6 use end-to-end flow control.

Class 2 service uses both buffer-to-buffer and end-to-end flow control.



Buffer to Buffer and End-to-End Flow Control

R_RDY

End-to-end flow control

N_PortA

N_PortB

R_RDY

Fabric

F_Port F_Port

Buffer-to-bufferflow control

Buffer-to-bufferflow control

ACK

R_RDYData

R_RDY

12

34

5

Buffer-to-Buffer Flow Control and End to End Flow Control The preceding preceding diagram illustrates buffer-to-buffer flow control in Class 3:

1. Before N_Port A can transmit a frame, it must receive the primitive signal R_RDY from its attached F_Port. The R_RDY signal tells N_Port A that its F_Port has a free buffer.

2. When it receives the R_RDY signal, N_Port A transmits a frame.

3. The frame is passed through the fabric. Buffer-to-buffer flow control is performed between every pair of E_Ports, although this is not shown here.

4. At the other side of the fabric, the destination F_Port must wait for an R_RDY signal from N_Port B.

5. When N_Port B sends an R_RDY, the F_Port transmits the data frame.

End-to-end flow control is designed to overcome the limitations of buffer-to-buffer flow control. The preceding preceding diagram illustrates end-to-end flow control in Class 2:

1. Standard buffer-to-buffer flow control is performed for each data frame.

2. After the destination N_Port B receives a frame, it waits for an R_RDY from the F_Port.

3. When N_Port B receives an R_RDY, it sends an acknowledgement (ACK) frame back to N_Port A.

4. At the other side of the fabric, the initiator F_Port must wait for an R_RDY signal from N_Port A.

5. When N_Port A sends an R_RDY, the F_Port transmits the ACK frame.

End-to-end flow control involves only the port at which a frame originates and the ultimate destination port, regardless of how many FC switches are in the data path.


When end-to-end flow control is used, the transmitting port is responsible for ensuring that all frames are delivered. Only when the transmitting N_Port receives the last ACK frame in response to a sequence of frames sent does it know that all frames have been delivered correctly, and only then will it empty its ULP data buffers. If a returning ACK indicates that the receiving port has detected an error, the transmitting N_Port has access to the ULP data buffers and can resend all of the frames in the sequence.



Credit Management Methods

Base credit management method:Transmitting port knows how many buffers the receiving port hasTransmitting port can begin sending immediately

Alternate credit management method:Transmitting port knows only how many free buffers the receivingport can guaranteeTransmitting port must wait for R_RDY

Credit Management Methods There are two types of credit management used on FC SANs:

In the base credit management method, the transmitting port knows how many buffers the receiving port has. The transmitting port can therefore begin sending frames immediately after a session is established.

In the alternate credit management method, the transmitting port knows only how many free buffers the receiving port can guarantee. The transmitting port must therefore wait for the receiving port to send an R_RDY signal before sending a frame.



The Base Credit Management Method

1. At login, Rx tells Tx how many buffers Rx has (BB_Credit)2. Tx sets BB_Credit_CNT = 0 at login3. Tx increments BB_Credit_CNT when it sends a frame4. Rx sends an R_RDY when it processes the frame5. Tx decrements BB_Credit_CNT when it receives R_RDYTx sends only when BB_Credit_CNT < BB_Credit

Base Credit Management Method

Tx

BB_Credit:BB_Credit_CNT:

40

PLOGIACC DATA

1

DATA

2

DATA

3

DATA

4

R_RDY

3

Rx

The Base Credit Management Method The base credit management method works as follows:

When the transmitting port sends a port login request, the receiver responds with an accept frame (ACC) that includes information on the size and number of frame buffers it has (BB_Credit). The transmitting port stores the BB_Credit value in a table.

The transmitting port also stores another value called BB_Credit_CNT, which represents the number of “used” buffer credits. BB_Credit_CNT is set to zero after the ports complete the login process.

Each time the transmitting port sends a frame, it increments BB_Credit_CNT.

Upon receiving the frame, the receiver processes the frame and moves it to upper-layer protocol (ULP) buffer space. The receiving port then sends an R_RDY acknowledgement signal back to the transmitting port, informing it that a buffer is available.

When the transmitting port receives the R_RDY signal, it then decrements its BB_Credit_CNT.

To prevent overrunning the receiving port’s buffers, the transmitting port can never allow BB_Credit_CNT (the count of frames which have not yet been acknowledged) to exceed BB_Credit (the total number of buffers in the receiving port). In other words, if it cannot confirm that the receiving port has a free buffer, it does not send any more frames.


Allocating Buffer Credits This lesson explains how to calculate the number of buffer credits needed for a FC link of a given distance.


Allocating Buffer Credits

Credit requirement depends on link length:Calculated based on frame size, propagation delay, and end-to-end latencyLatency is deterministic for “pure” Fibre Channel linksDefault credit allocation is generally sufficient for intra-datacenter linksAllocation must often be increased for long-haul linksFCIP links require additional credits due to IP latencyCisco 16-port switch modules support up to 255 credits

The number of buffer-to-buffers required for a link depends on the physical length of that link. The number of credits required is calculated based on frame size, propagation delay (speed of light in fiber), and the end-to-end latency of the link; of all of these factors, latency is the only variable. On a “pure” FC link, latency is deterministic and depends primarily on the length of the link and the number of hops. On an FC WAN link (such as FCIP), latency depends on the characteristics of the WAN.

The default credit allocation on most vendors’ switches is generally sufficient for intra-datacenter links. However, the credit allocation often must be increased for long-haul links. FC WAN links, including FCIP, typically require additional buffer credits due to the increased latency of the IP network.

Cisco 16-port switch modules support up to 255 credits per port, which provides ample credits for most applications.



Allocating Buffer Credits (Cont.)

Frame serialization time:– Link rate of 1.0625 Gb/s = 9.41ns/byte– Frame size = 2048 data + 36 header + 24 IDLE = 2108 bytes– Frame serialization time = 19.84µs ≈ 20µs

(Round_Trip_Time + Processing_Time)Serialization_Time

Credits =

Initiator N_Port

10Km

Target N_Port

Frame

20µs

You can calculate the number of credits required on a link to maintain optimal performance using the following formula:

Credits = (Round_Trip_Time + Processing_Time) / Serialization_Time

Example This diagram and the following two diagrams illustrate how the required number of BB_Credits are calculated for a 10km, 1Gb/s FC link:

At a link rate of 1.0625 Gb/s, the time required to serialize (transmit) each byte is 9.41ns. (Note that each byte is 10 bits due to 8b/10b encoding.)

The maximum Fibre Channel frame size is 2048 bytes. The frame size used in an actual customer environment would be based on the I/O characteristics of the customer’s applications. You also need to account for the frame header, which is 36 bytes, and the number of IDLEs between frames, which is usually 6 IDLEs, or 24 bytes. This gives a total of 2108 bytes.

The total serialization time for a 2108-byte frame (including idles) is 19.84µs, or approximately 20µs.




Propagation delay:– Speed of light in fiber ≈ 5µs/Km– Time to transmit frame across 10Km ≈ 50µs

Processing time:– Assume same as deserialization time ≈ 20µs

Frame

20µs

Initiator N_Port

10Km

Target N_Port

Frame

20µs50µs

The speed of light in a fiber optic cable is approximately 5µs per kilometer, so each frame will take about 50µs to travel across the link.

The receiving port must then process the frame, free a buffer, and generate an R_RDY. This processing time can vary—for example, if the receiver ULP driver is busy, the frame might not be processed immediately. In this case, we can assume that the receiving port will process the frame immediately, so the processing time is equal to the time it takes to deserialize the frame. The deserialization time is equal to the serialization time: 20µs




Response time:– Time to transmit R_RDY across 10Km ≈ 50µs

Total latency ≈ 50µs + 20µs + 50µs = 120µs

Frame

20µs

Initiator N_Port

10Km

Target N_Port

R_RDY 50µs

Frame

20µs50µs

The receiving port then transmits a credit (R_RDY) back across the link. This response takes another 50µs to reach the transmitter.

The total latency on the link is equal to the frame serialization time plus the round-trip time across the link, or about 120µs.




Given frame serialization time ≈ 20µs and total latency ≈ 120µs, there could be up to 6 frames on the link at one timeBuffer-to-buffer credits required = 6

FrameFrame Frame Frame Frame Frame

Initiator N_Port

10Km

Target N_Port

Given a frame serialization time of 20µs, and a total round-trip latency of 120µs, there could be up to 6 frames on the link at one time. In other words, six buffer-to-buffer credits are required to make full use of the bandwidth of the link.




Credit requirement also depends on frame size:

All figures based on 10Km link

2Gb/s1Gb/s

116

87

58

35

20

11

6

Credits Required

0.43

0.58

0.89

1.49

2.69

5.11

9.93µs

Serialization Time

2320.8732

1721.1764

1141.77128

682.97256

385.38512

2110.201024

1119.84µs2048 bytes

Credits Required

Serialization Time

Payload Size

The formula for calculating required credits is Credits = (Round_Trip_Time + Processing_Time) / Serialization_Time. Serialization time is proportional to frame size, so the number of credits required varies with frame size.

For example, with a 10Km link at 2Gb/s, only 11 credits are required if the average frame size is the maximum (2048 payload bytes). However, if the average payload is 32 bytes, 232 credits are required.


Fibre Channel Addressing This section describes the addressing mechanism used by Fibre Channel.


Fibre Channel Addressing

DomainDomain AreaArea PortBit 23 16 15 08 07 00

239 Domains(01–EF)

Hub

Nodes

Switch

FC

FC FC

FCHBA

FCHBA

FCHBA

FCHBA

The FC point-to-point topology uses a 1-bit addressing scheme. One port assigns itself an address of 000000 and then assigns the other port an address of 000001.

The FC Arbitrated Loop topology uses an 8-bit addressing scheme:

The Arbitrated Loop Physical Address (AL_PA) is an 8-bit address, which provides 256 potential addresses. However, only a subset of 127 addresses are available due to 8b/10b encoding requirements.

One address is reserved for an FL_Port, so there are 126 addresses available for nodes.

Addresses are cooperatively chosen during loop initialization.

The Switched Fabric Address Space The 24-bit FC address consists of three 8-bit elements:

The Domain ID is used to define a switch. Each switch receives a unique Domain ID.

The Area ID is used to identify groups of ports within a Domain. Areas can be used to group port ports within a switch, and are also used to uniquely identify fabric-attached arbitrated loops. Each fabric-attached loop receives a unique Area ID.

The Port ID is used to identify each individual port within an Area.

Although the Domain ID is an 8-bit field, only 239 Domains are available to the fabric:

Domains 01–EF are available

Domains 00 and F0–FF are reserved for use by switch services


Each switch must have a unique Domain ID, so there can be no more than 239 switches in a fabric. The largest director-class switch available today has 256 ports, so the practical limit on the number of nodes that can be supported in a fabric is 61184 ports (239 domains x 256 ports). With 16-port switches the total port count is reduced to 3824 (239 domains x 16 ports), minus the number of ports used for ISLs. Note that these calculations do not take into account ports consumed by inter-switch links (ISLs)—which reduces the number of ports—or the fact that an arbitrated loop multiple L_Ports can be attached to a single FL_Port—which increases the potential number of ports.



The FC-AL Address Space

00000000 00000000 AL_PAPrivate Loop

AL_PADomainDomain AreaAreaPublic Loop

DomainDomain AreaArea PortFabric000708151623Bit

The FC-AL Address Space In a public (fabric-attached) loop:

Public NL_Ports are assigned a full 24-bit fabric address when they log into the fabric.

There are 126 AL_PA addresses available to NL_Ports in an arbitrated loop; the AL_PA 0x00 is reserved for the FL_Port (which is logically part of both the fabric and the loop).

The Domain and Area fields are identical to those of the FL_Port to which the loop is connected.

In a private (isolated) loop :

Private NL_Ports can communicate with each other based upon the AL_PA, which is assigned to each port during loop initialization.

Private NL_ports are not assigned a 24-bit fabric address, and the Domain and Area segments are not used.


World-Wide Names This section introduces WWNs, a second addressing scheme used on FC SANs.


World-Wide Names

Every Fibre Channel port and node has a hard-coded address called a World Wide Name (WWN):

Allocated to manufacturer by IEEECoded into each device when manufactured64 or 128 bits (128 bits most common today)

Switch Name Server maps WWNs to FC addresses:World-Wide Node Names (WWNNs/NWWNs) uniquely identify devicesWorld-Wide Port Names (WWPNs/PWWNs) uniquely identify each port in a device

Example WWNExample WWN

20:00:00:45:68:01:EF:25WWN 21:00:00:45:68:01:EF:25WWPN A22:00:00:45:68:01:EF:25WWPN B

20:00:00:45:68:01:EF:25WWNNExample WWNs from a Dual-Ported DeviceExample WWNs from a Dual-Ported Device

WWNs are unique identifiers that are hard-coded into FC devices. Every FC port has at least one WWN. Vendors buy blocks of WWNs from the IEEE and allocate them to devices in the factory.

WWNs are important for enabling fabric services because they are:

Guaranteed to be globally unique

Permanently associated with devices

These characteristics ensure that the fabric can reliably identify and locate devices, which is an important consideration for fabric services. When a management service or application needs to quickly locate a specific device:

1. The service or application queries the switch Name Server service with the WWN of the target device

2. The Name Server looks up and returns the current port address that is associated with the target WWN

3. The service or application communicates with the target device using the port address


There are two types of WWNs:

WWNNs uniquely identify devices. Every host bus adaptor (HBA), array controller, switch, gateway, and FC disk drive has a single unique WWNN.

WWPNs uniquely identify each port in a device. A dual-ported HBA has three WWNs: one WWNN and a WWPN for each port.

WWNNs and WWPNs are both needed because devices can have multiple ports. On single-ported devices, the WWNN and WWPN are usually the same. On multi-ported devices, however, the WWPN is used to uniquely identify each port. Ports must be uniquely identifiable because each port participates in a unique data path. WWNNs are required because the node itself must sometimes be uniquely identified. For example, path failover and multiplexing software can detect redundant paths to a device by observing that the same WWNN is associated with multiple WWPNs.

Cisco MDS switches use the following acronyms:

PWWN (Port WWN)

NWWN (Node WWN)




Summary

Fibre Channel uses a credit-based strategyTwo types of flow control:– Buffer-to-buffer (port-to-port)– End-to-end (source-to-destination)

Credit requirements depend on frame size, RTT, serialization, and processing timeFC addressing is a 24-bit number:– 3 bytes represent: [ domain ] [ area ] [ port ]

Every Fibre Channel port and node has a hard-coded address called a World Wide Name (WWN).

Lesson 5

FC Login

Overview The Fabric Login, Port Login, and Process Login protocols define how fabric ports behave when they are brought online and when they want to establish a communication session. This lesson provides a detailed examination of each of the login protocols. It explains the role that each login protocol serves, and identifies the commands that are exchanged during each phase of each protocol.

Objectives Upon completing this lesson, you will be able to describe the Fibre Channel device login process. This includes being able to meet these objectives:

Identify the phases of the Fabric Login protocol

Identify the phases of the Port Login protocol

Identify the phases of the Process Login protocol

Identify the phases of the Loop Initialization and Arbitration protocols


Fabric Login This section provides an overview of the session establishment protocols that are performed by N_Ports and F_Ports in a fabric topology.


Fabric Login

ProcessProcess

NodeNodeN_Port A N_Port BF_Port A F_Port B

Process A Process B

FLOGIFLOGI FLOGIFLOGI

PLOGIPLOGI

PRLIPRLI

Fabric

Before an N_Port can begin exchanging data with other N_Ports, three processes must occur:

The N_Port must log in to its attached F_Port. This process is known as Fabric Login (FLOGI).

The N_Port must log in to its target N_Port. This process is known as Port Login (PLOGI).

The N_Port must exchange information about ULP support with its target N_Port to ensure that the initiator and target process can communicate. This process is known as Process Login (PRLI).



Fabric Login (Cont.)

F_PortF_

Port

Switch

F_Po

rt

F_PortN

_Por

t

Node

N_P

ort

NodeNOS

Hey! I am connected to something – I must tell it I am here and in the Not

Operational State.

1

Hey! I am connected to something – I must tell it I am here and in

an Off-Line State.

OLS

2

FLOGI is the initial “bootstrap” process that occurs when an N_Port is connected to an F_Port. FLOGI is mandatory for N_Ports, and optional for NL_ports.The N_Port uses Fabric Login to discover if a fabric is present. Communication with other N_Ports may not be attempted until the Fabric Login process is complete.

The FLOGI protocol follows this process:

1. The F_Port sends a primitive sequence of NOS (Not Operational) to the N_Port.

2. When the N_Port receives the NOS, it responds with a primitive sequence of OLS (Offline State) to begin link initialization.




F_PortF_

Port Switch

F_Po

rt

F_PortN

_Por

t

Node

N_P

ort

NodeLR

A new connection!

I will try to reset the link.

3

LRR

This F_Port is trying to initialize the link. I will

respond.

4

3. After the N_Port begins the initialization process with by sending OLS, the F_Port tries to reset the port by sending an LR (Link Reset) command.

4. The N_Port responds with an LRR (Link Reset Response) command.




F_PortF_

Port Switch

F_Po

rt

F_PortN

_Por

t

Node

N_P

ort

NodeIDLE

This is just IDLE talk, filling the time with

words.

IDLE

Let us engage in some IDLE conversation until

something happens.

5

5. From this point on, the link is active and IDLE fill words flow in both directions on the link.

6. Following link initialization, a new N_Port uses an S_ID of 000000 or 0000[AL_PA] to indicate that the port is unidentified during FLOGI. An existing N_Port uses its existing port address as its S_ID.




F_PortF_

Port Switch

F_Po

rt

F_Port

Logi

nS

erve

r

N_P

ort

Node

N_P

ort

NodeFLOGI

Great, I have established a link with the switch!

Now I need to request a port address.

6

LS_ACC

Ok, here is a unique port address.

7

7. After the N_Port has established a link to its F_Port, the N_Port obtains a port address by sending a FLOGI Link Services command to the switch Login Server (at Well-Known Address 0xFFFFFE).

8. The Login Server sends an ACC reply that contains the N_Port address in the D_ID field.

When an N_Port is performing FLOGI and receives ACC frame that indicates that the ACC came from another N_Port, then the N_Port that is logging in assumes that it is in a point-to-point configuration. In this case, the N_Port immediately initiates PLOGI with the other N_Port after completing FLOGI.




F_PortF_

Port Switch

F_Po

rt

F_Port

Nam

e S

erve

r

N_P

ort

Node

N_P

ort

NodePLOGI

Now that I have a port address I will log in to the Name Server and tell it

about me.

8

LS_ACC

Thank you for your information.

9

9. After receiving a port address, the N_Port logs into the Fabric Name Server at address 0xFFFFFC and transmits its service parameters, such as the number of buffer credits it supports, its maximum payload size, and supported Classes of Service.

10. The Name Server responds with an LS_ACC frame.



LS_ACC Frame Format

Bits 7-0Bits 15-8Bits 23-16Bits 31-24Word000000Command 040

1-4 Common Service parameters (16 bytes)5-6 N_Port Name (8 bytes)7-8 Node Name (8 bytes)

9-12 Class 1 Service Parameters (16 bytes)13-17 Class 2 Service Parameters (16 bytes)18-21 Class 3 Service Parameters (16 bytes)22-25 Class 4 Service Parameters (16 bytes)26-29 Vendor Version Level (16 bytes)

This table shows the format of the LS_ACC frame.




The preceding image shows an analyzer trace that displays part of a fabric login sequence. The top of the trace shows the OLS-LR-LRR sequence that occurred while the link was being initialized. The right-hand panel shows the contents of the FLOGI frame from the N_Port to the F_Port (FFFFFE).

Useful information can be obtained by studying these analyzer traces:

Notice that at this time the N_Port does not yet have an address.

Notice also that the World Wide Port Name is the same as the World Wide Node Name. This is common in single ported nodes.

The N_Port does not support Class 1, but it does support Classes 2 and 3.

The N_Port supports Alternate Buffer Credit Management Method and can guarantee 2 BB_Credits at its receiver port.

You can see that this is a single-frame Class 3 sequence because the Start of Frame is SOFi3 and End of Frame is EOFt, meaning that this initial first frame is also the last one in the sequence.


Port Login This section provides a description of the PLOGI protocol. Each command used during PLOGI is identified; however, the parameters exchanged during each command are not described in detail.


Port Login

F_Port

F_Po

rt

F_Po

rt

F_Port

N_P

ort

Node

N_P

ort

NodePLOGI PLOGI

I want to exchange data with another N_Port. I will tell them I am here and find out what their

capabilities are.

1

After completing the FLOGI process, the N_Port can log in to another N_Port using the PLOGI protocol. PLOGI must be completed before the nodes can perform any ULP operations.

The PLOGI protocol follows this process:

1. The initiator N_Port sends a PLOGI frame that contains the N_Port’s operating parameters encapsulated in the payload.



Port Login (Cont.)

F_Port

F_Po

rt

F_Po

rt

F_Port

N_P

ort

Node

N_P

ort

Node

I see that this port hassome limitations, so I willoperate in Class 3 with

small frame sizes.

LS_ACCLS_ACC

Hello. I must tell you that I support only

Class 3 and cannot accept large frames.

2

2. The target N_Port responds to the initiator N_Port by sending an ACC frame that specifies the target N_Port’s operating parameters. The operating system driver that manages the initiator N_Port stores this information in a parameter block.

An N_Port can be logged into multiple N_Ports simultaneously. N_Ports typically perform Port Logout (PLOGO) only when one of the nodes go offline.



Port Login (Cont.)

The image shows the N_Port logging in to another N_Port a PLOGI command. Note that the N_Port has provided the same data that it provided when it logged in to the Name Server.



Port and Address Discovery

Discover Address (ADISC):– Confirm port addresses– Discover the whether the other port has a hard-coded address

Discover F_Port Service Parameters (FDISC):– Verify fabric service parameters

Discover N_Port Service Parameters (PDISC):– Verify service parameters between two N_Ports

Port and Address Discovery After FLOGI is and PLOGI are complete, the N_Port can use the following Extended Link Services commands to retrieve updated information or verify information about port addresses and service parameters:

Discover Address (ADISC) can be used to confirm another port’s address or to discover whether the other port has a hard-coded address.

Discovery Fabric Service Parameters (FDISC) can be used to verify fabric service parameters.

Discover N_Port Service Parameters (PDISC) can be used to verify the service parameters of another N_Port.

These commands allow ports to query and verify fabric and port parameters without performing PLOGI and thus forcing logout of the current session.


Process Login This section provides a description of the PRLI protocol. Each command used during PRLI is identified; however, the parameters exchanged during each command are not described in detail.


Process Login

F_Port

F_Po

rt

F_Po

rt

F_Port

N_P

ort

Node

N_P

ort

Node

I am going to be using the SCSI protocol. I

wonder if the target can support the same functions as I can.

PRLI PRLI

I must tell the target what SCSI functionality I can

support and find out what the target can support.

1

After completing the PLOGI protocol, both N_Ports knows about the other’s Fibre Channel (FC) operating parameters capabilities. At this point, the driver for the initiator port can open a channel with the driver associated with the target port using the PRLI protocol. The PRLI protocol is used to establish a session between two FC-4 level logical processes.

The PRLI protocol follows this process:

1. The initiator sends a PRLI frame that contains information about its ULP support.



Process Login (Cont.)

F_Port

F_Po

rt

F_Po

rt

F_Port

N_P

ort

Node

N_P

ort

Node

Now we have information about each other, we will

only talk in protocols that we can both understand

LS_ACCLS_ACC

Yes, I have SCSI support! Thanks for

your information. Here are my details.

2

2. The target port responds with an ACC frame that contains details about its ULP support. At this point, a channel has been successfully opened and communication can take place. The relationship between the initiator process and the target process is known as an image pair.




F_Port

F_Po

rt

F_Po

rt

F_Port

N_P

ort

Node

N_P

ort

NodePRLO PRLO

Ok, I am done exchanging data

now. See you later!

3

Thanks for the memories…

LS_ACC LS_ACC

4

3. When the initiator has finished exchanging data with the target, the initiator sends a Process Logout (PRLO) frame.

4. The target responds with an ACC frame, and the image pair is then terminated.

At this point, the image pair must be established again before further communication can take place.




The image shows the N_Port performing process login (PRLI) with its target N_Port. The payload data in a PRLI is relevant to the ULP, which in this case is SCSI-FCP. For example:

This N_Port can function as an initiator.

The ULP driver does not use the SCSI-3 XFER_RDY command during SCSI Read operations.



SP=00000012


The image shows the target N_Port responding to the PRLI command shown on the previous page.


Loop Initialization and Arbitration This section provides a description of the Loop Initialization and Arbitration protocol. Each command used during Loop Initialization is identified; however, the parameters exchanged during each command are not described in detail.


Loop Initialization and Arbitration

Informall portsInformall ports

Selecta loop master

Selecta loop master

SelectAL_PAsSelect

AL_PAsPosition MapReporting

Position MapReporting

Position MapDistribution

Position MapDistribution

A

B

C D

E

Arbitratefor ownership

Arbitratefor ownership

Opena channel

Opena channel

Transferdata

Transferdata

Closethe channel

Closethe channel

A B

C

D

Initialization Arbitration

The Loop Initialization Protocol There are five phases in the Loop Initialization protocol:

A. The port that has just come online informs the other ports that it is initiating the initialization process.

B. The ports cooperatively select one port to manage the remainder of the process. This port is known as the loop master.

C. The ports cooperatively select their own AL_PAs.

D. The ports collect an “address book” of port addresses known as the position map or position bitmap.

E. The ports circulate and save a copy of the completed position map.


The Loop Arbitration Protocol There are four phases in the Loop Arbitration protocol:

A. Arbitrate for ownership of the loop

B. Open a logical connection to form an image pair

C. Transfer data

D. Close the connection

The Open-Transfer-Close process used on arbitrated loops is similar to the Select-Transfer-Release process used by SCSI.



The Loop Port State Machine


FC-1 Encoding

FC-2 Framing & flow control


Upper-layerprotocols

FC-4

LPSMLPSM

FC-FS

FC-PI

The Loop Port State Machine The Loop Port State Machine (LPSM) is an FC-2-layer function in each L_Port that protocols the loop-specific commands used for flow control on arbitrated loops. The LPSM is responsible for managing the Loop Initialization and Loop Arbitration protocols. It is implemented in hardware for speed.




Summary

Fabric Login (FLOGI) is performed by all connecting devices to acquire an FCIDPort Login (PLOGI) is performed to pass identification and capability to the name serverProcess Login (PRLI) is performed between end devices to exchange operating parameters capabilities and establish a sessionLoop initialization is performed by all device connecting to a loop hub or device to acquire an AL_PA (arbitrated loop physical address)

Lesson 6

FC Error Recovery

Overview Each Fibre Channel (FC) layer plays a role in error management. In this lesson, you will learn about how each layer detects and recovers from errors. You will also learn about configuration parameters that affect the way a FC SAN responds to error conditions.

Objectives Upon completing this lesson, you will be able to explain how Fibre Channel recovers from errors. This includes being able to meet these objectives:

Explain how FC handles frame-level errors detected by the FC-1 layer

Explain how FC handles sequence-level errors detected by the FC-2 layer

Explain how the SCSI-FCP protocol handles error conditions


FC-1 Errors This section describes the error recognition protocol in the FC-1 layer of Fibre Channel.


FC-1 Errors

Four consecutive invalid transmission words trigger an FC-0 loss-of-synchronization error

Synchronization AcquiredLoss of Sync

1 2 3 4

Invalid word detectedValid word detectedSync (re)gained

Four consecutive invalid transmission words must occur to trigger an FC-0 loss-of-synchronization error. This requirement prevents transient errors from causing loss of synchronization.

The preceding graphic shows the trigger conditions required to cause a loss-of-synchronization error:

The system starts in state 1.

When an invalid word is detected, the system moves to state 2.

If the next word is valid, the system moves back to state 1.

After three consecutive invalid words, the system is in state 4. The next consecutive invalid word will trigger a loss-of-synchronization error.



Link Control Protocols and R_T_TOV

Receiver-Transmitter Timeout Value (R_T_TOV) applies to link-level events, such as:– Loss of synchronization– Link initialization, reset, and failure protocols

Exceeding R_T_TOV during these events results in link failureDefault value is 100msCannot be changed on MDS switches

Link failure occurs in the following situations:

Loss of synchronization occurs and synchronization cannot be reestablished within a specified timeout period.

An expected Primitive Sequence is not received within a specified timeout period during the link initialization, reset, and failure protocols.

R_T_TOV The timeout period that governs both of these cases is the Receiver-Transmitter Timeout Value (R_T_TOV). The default value of R_T_TOV is 100ms. R_T_TOV cannot be changed on MDS switches.

R_T_TOV is an FC-1 layer timer. This timer is used to detect loss of synchronization between the transmitter and receiver, and is also used to time link reset events. If R_T_TOV is too low, the transmitter and receiver will experience repeated loss of synchronization and link reset events. If R_T_TOV is too low for the link reset process to complete, the link will not come up.



R_T_TOV (Cont.)

All ports in the fabric must have the same R_T_TOV valueFabric will segment if R_T_TOV is not consistentShorter R_T_TOV (100µs) has been proposed to provide faster error detection required for real-time systems, such as avionics environments

R_T_TOV is a fabric-wide timeout value. All ports in the fabric must have the same value. If R_T_TOV is not the same on two connected switches, the fabric will segment.

The default value of 100ms is acceptable in most situations. However, R_T_TOV might need to be adjusted in some environments. Real-time environments like FC-AE require very fast responses, fast error recovery, and low latency. For applications with these requirements, 100ms is a long time to wait; 5000 2KB frames could be sent in that time. (Each 2KB frame takes approx 20µs to serialize, so 5000 x 20µs = 100ms.) Some FC developers have proposed reducing the default R_T_TOV to 100µs (1000 times shorter) for certain environments.


FC-2 Errors This section describes the error recognition protocol in the FC-2 layer of Fibre Channel.


FC-2 Errors

Delimiter Errors Delivery ErrorsResource Errors

Frame ErrorsInvalid D_ID

Invalid OX_ID

Invalid S_ID

Invalid RX_ID

Invalid SEQ_ID

Invalid SEQ_CNT

Invalid R_CTL Invalid F_CTL Invalid DF_CTL

Unsupported ULPor invalid TYPE

Invalid Offset

Invalid SOF/EOF

UnsupportedClass of Service

Too manySequences

Cannot establishExchange

Missing frameor ACK

Out-of-orderSEQ_CNT

Undeliverableframes

Sequence or linktimeout

Only type of errors detected in Class 3 operation

The FC-2 layer detects four general types of errors:

Frame Errors occur when any of the frame header fields are invalid, such as a frame with an invalid D_ID or unsupported ULP.

Resource Errors occur when the sequence count exceeds the maximum number of sequences within an exchange (256) or when a valid exchange cannot be established.

Delimiter Errors occur when either SOF or EOF are invalid or if a frame is received with an unsupported Class of Service.

Delivery Errors occur when frames arrive out of sequence, are missing, or fail to arrive within a specified time period.



FC-2 Errors—Invalid Frames (Cont.)

Reject and busy frames are sent when a Class 1, 2, 4, or F frame could not be processedClass 3 service is unacknowledged

Receiver cannot accept frame

Frame is invalid

P_BSYF_BSY

P_RJTF_RJT

Sent by an N_PortSent by an F_Port

In Classes 1, 2, 4, and F, which all provide acknowledged delivery, a RJT or BSY response will be sent to the transmitting port when a frame is invalid or cannot be delivered:

The fabric will reply with F_BSY if the destination switch port had no free buffers.

The fabric will reply with F_RJT if the frame had an invalid D_ID or S_ID, or if the port is unavailable.

The receiver will reply with P_BSY if the receiver port had no free buffers.

The receiver will reply with P_RJT if the requested Class of Service or ULP is not supported.

In Class 3, frames will be discarded without notification if the receiver port has no buffers, is unavailable, or does not support the requested Class of Service or ULP.



E_D_TOV

E_D_T_V is an FC-2 layer timer:– Determines how long a receiver waits for a response before

declaring an error condition

Default value of 2 secondsOn FCIP links you might need to increase E_D_TOV if RTT exceeds 2 seconds:– Unusually high latency in the IP WAN– Dropped packets that need to be retransmitted– Congestion at the FCIP gateway (low-bandwidth IP)

E_D_TOV E_D_TOV is an FC-2 layer timer. E_D_TOV determines how long a receiver waits for an expected response before declaring an error condition. For example, if a frame arrives out of sequence, the receiver waits E_D_TOV before it declares an error and aborts the sequence.

The default value of E_D_TOV is usually 2 seconds on FC switches. This value is always sufficient for DWDM and almost always sufficient for SONET/SDH. On FCIP links, however, you might need to increase E_D_TOV due to two factors:

There might be unusually high latency in the IP WAN, or there might be dropped packets that need to be retransmitted. In either case, it is possible—although unlikely in a well-designed IP network—for the total round-trip latency to exceed 2 seconds.

The bandwidth of the IP link might be less than the bandwidth of the FC fabric, so frames could pile up in the fabric if the IP link becomes congested.



Sequence RecoverySequence Error

Detected

BA_ACCreceived?

Send ABTS

Discard all framesin Sequence

Reply BA_ACC

Discard all framesin Sequence

NOYES

Retry ABTS

??

??

BA_ACCreceived?

YES

Implicit logoutof other port

NO

Sequence Recovery This diagram illustrates the first part of the sequence recovery process:

When a sequence error occurs, the N_Port that detected the error sends the Abort Sequence (ABTS) Extended Link Services command to abort the sequence. ABTS can be transmitted as part of the current sequence or as a new sequence.

The other N_Port responds with the Basic Accept (BA_ACC) command..

Both ports discard all frames in the Sequence.

If the N_Port that sent ABTS does not receive BA_ACC, it assumes that the other ports is no longer available and performs an implicit port logout.

In some cases, the entire exchange is aborted with the Abort Exchange (ABTX) command, and the entire exchange must be reestablished.



Sequence Recovery (Cont.)

When a sequence is aborted:– Frames from the aborted sequence could still be in transit for

an undetermined period of time– SEQ_ID can be reused, so receiver cannot distinguish “old”

frames from “new” frames– This could cause data errors at the ULP level– The initiator therefore waits before allowing retransmission of

the sequence

The wait time is determined by the Resource Allocation Timeout Value (R_A_TOV):– Default value is 10 seconds in a fabric– Default value is 2 * E_D_TOV (4 sec) for point-to-point

When ABTS is issued to abort a sequence, the fabric must be purged of all frames in the sequence before the sequence can be re-transmitted; otherwise, old frames could arrive out of sequence. The receiver might not be able to differentiate between the old frames and the retransmitted frames, and data errors could result at the ULP level.

Therefore, before the sequence is resent, the initiator waits for a specified period of time before retransmitting the sequence. This time period is determined by the Resource Allocation Timeout Value (R_A_TOV):

In a fabric, the default value of R_A_TOV is 10 seconds.

In a point-to-point topology, the default value of R_A_TOV is twice the value of E_D_TOV, or 4 seconds.



Sequence Recovery (Cont.)Sequence Error

Detected

BA_ACCreceived?

Send ABTS

Discard all framesin Exchange

Reply BA_ACC

Discard all framesin Exchange

NOYES

Retry ABTS

BA_ACCreceived?

YES

Implicit logoutof other port

NO

Wait R_A_TOV

Send RRQ

This diagram continues the description of the sequence recovery process:

After BA_ACC is received, the originating N_Port waits for a time equal to R_A_TOV (10 seconds in a fabric).

After R_A_TOV has expired, the originating N_Port sends the Resource Recovery Qualifier (RRQ) command.

After RRQ is sent, the ports can begin retransmission of the failed sequence or exchange.



R_A_TOV

R_A_TOV is an FC-2 layer timer:– Specifies how long a frame can be in transit– Used to determine how long a sender must wait before it can

begin resending an aborted sequence

Default value of R_A_TOV is usually 10 seconds

R_A_TOV R_A_TOV is an FC-2 layer timer. R_A_TOV specifies how long a FC frame can be in transit. This value is used to determine how long a sender must wait before it can begin resending a sequence after the sequence was aborted after an error occurred. The sender must wait for R_A_TOV because if a sender begins to resend a sequence before the frames from the old aborted sequence have been received, discarded or expired, frames from the old and new sequences might arrive intermixed. Because the FC protocol provides no way for the receiver to guarantee that the new sequence ID will be different than the old sequence ID, the sender waits until there is no chance that frames could still be in transit.

The default value of R_A_TOV is usually 10 seconds on FC switches. This value is sufficient for any type of WAN link except some IP links.


SCSI-FCP Error Recovery This section describes the basic and enhanced error recovery methods used by the SCSI-FCP protocol.


SCSI-FCP Error Recovery

Basic error recovery:SCSI-FCP uses the “Abort, Discard Multiple Sequences” error policy by default:– A failed sequence results in the entire exchange being retransmitted

Discarding the entire exchange is often not desirable:– Initiator must wait a minimum of R_A_TOV (10 seconds) before

retrying the aborted exchange– Impact can be greater on some media, e.g. tape devices that must

rewind media when the stream is aborted

By default, the SCSI-FCP protocol uses the “Abort, Discard Multiple Sequences” exchange error policy, in which all sequences in the exchange are retransmitted. However, discarding the entire exchange is often not the most desirable solution. The initiator must wait for the R_A_TOV timeout period (10 seconds by default) to expire before retrying the aborted exchange. In addition to reducing overall performance, this long wait time can have greater impact in some situations. For example, if the failed operation is a backup application streaming frames to a tape drive, then the tape buffer will empty and the drive will stop. When the buffer begins to fill again, the tape will rewind, run up to speed, and continue streaming from the last file mark.



SCSI-FCP Error Recovery (Cont.)

Initiator TargetFCP_CMND

XFCP_RSP

ABTS

BA_ACC

Retry FCP_CMND

E_D

_TO

VR

_A_T

OV 12 seconds

Basic Error Recovery

The preceding diagram shows an example of basic SCSI-FCP error recovery:

An FCP command (FCP_CMND) is issued by the initiator,

Something goes wrong during the exchange and the FCP status sequence (FCP_RSP) does not arrive.

The initiator waits for E_D_TOV (2 seconds) for the missing FCP_RSP to arrive.

The initiator sends ABTS to abort the exchange.

The target responds with BA_ACC.

The initiator then waits for R_A_TOV (10 seconds) for all frames to be purged from the fabric before retrying the FCP command.

A total of about 12 seconds elapses until the FCP command is resent.



Enhanced Error Recovery

Enhanced error recovery:The following FC-4 Link Services commands can be used to recover from an error without aborting the entire exchange:– Read Exchange Concise (REC)– Sequence Retransmission Request (SRR)

The use of REC and SRR allows faster recovery:– Read Exchange Concise Timeout Value (REC_TOV) determines how long

initiator waits before sending REC– Default value of REC_TOV is E_D_TOV + 1 sec = 3 sec

Many vendors implement enhanced error recovery

Some ports are capable of using an enhanced recovery technique that allows nodes to recover from sequence errors without having to abort the entire exchange. This enhanced recovery technique is defined by FC-4 Link Services commands. FC-4 Link Services are similar to Extended Link Services, but FC-4 Link Services are defined by the ULP, whereas Extended Link Services are defined by FC-2.

The Read Exchange Concise (REC) Extended Link Service command allows the initiator to ask the target to report the status of the exchange.

The Sequence Retransmission Request (SRR) Extended Link Service command requests retransmission of the exchange beginning at a specific sequence.

The Read Exchange Concise Timeout Value (REC_TOV) determines how long the initiator waits before sending the REC command. The default value of REC_TOV is equal to the value of E_D_TOV (2 seconds) plus 1 second.

The REC and SRR commands are not defined in the FC-PH specification; rather, they are defined in the FCP-2 ULP specification. Many vendors take advantage of this technique.



Enhanced Error Recovery (Cont.)

Initiator TargetFCP_CMND

REC

_TO

V

Enhanced Error Recovery

3 secondsX

FCP_RSP

REC

FC-4_ACC

SRR

FC-4_ACC

Resend FCP_RSP

The preceding diagram shows an example of enhanced SCSI-FCP error recovery:

An FCP_CMND is issued by the initiator, but something goes wrong during the exchange.

The FCP_RSP does not arrive.

The initiator waits REC_TOV (3 seconds) for the missing frame to arrive

The initiator sends REC to request information about the status of the exchange.

The target acknowledges REC by sending FC-4_ACC.

When the initiator receives FC-4_ACC, the initiator knows where in the exchange the failure occurred. The initiator then sends SRR to request retransmission of the sequence.

The target resends the missing FCP_RSP sequence.

A total of about 3 seconds (the default value of REC_TOV) elapses until the initiator sends REC. This is about one quarter of the time elapsed during basic error recovery (12 seconds) in a similar situation.




Summary

Four consecutive invalid transmission words trigger an FC-0 loss-of-synchronization errorAt FC-1, all ports in the fabric must have the same R_T_TOV valueFC-2 layer detects four general types of errors:1. Frame Errors 2. Resource Errors3. Delimiter Errors 4. Delivery Errors

Initiator must wait a minimum of R_A_TOV (10 seconds) before retrying the aborted exchange



Summary (Cont.)

Receiver Transmitter Time-Out Value (R_T_TOV):– Short timer used to detect link-level failures

Error Detect Time-Out Value (E_D_TOV):– Medium-length timer used to time events at the sequence level

Resource Allocation Time-Out Value (R_A_TOV):– Long timer used to delay sequence recovery

Lesson 7

FC Switched Fabric

Overview This lesson explains three important protocols in a Fibre Channel switched fabric. The fabric configuration protocol, the FSPF protocol, and the RSCN protocol. You will also learn about fabric services and how they are addressed.

Objectives Upon completing this lesson, you will be able to describe the Fibre Channel Switched Fabric protocol. This includes being able to meet these objectives:

Describe the high-level phases of the fabric configuration protocol

Explain the FSPF protocol

Explain the RSCN protocol

Identify the standard fabric services and their well-known addressed


Fabric Configuration Overview This section describes the steps taken during fabric initialization.


Fabric Configuration OverviewDetermine PortOperating Mode

Domain IDAssigned?

Principal Switch Selected(DIA) Domain Identifier

Assigned

Request Domain Identifiers(RDI)

Exchange FabricParameters (EFP)

YES

Word SyncAcquired

E_Port Mode?

NOYES

Wait for Fabric Login

Exchange Link Parameters (ELP)

Exchange SwitchCapabilities (ESC)

Build Topology Database(FSPF)

Compute Least Cost PathsBuild Routing Table

NO

The diagram describes the steps taken, when a fabric is first initialized, a new switch is added to an existing, or a link becomes active.

A switch port detects a valid signal on its attached link and achieves word synchronization.

The switch port begins link initialization. If the port is capable of operating at more than speed, it may perform speed negotiation.

The switch port determines the proper operating mode; FL_Port, F_Port or E_Port.

Exchange Link Parameters (ELP). When two E_Ports are connected and the link initialized, the ports exchange link parameters. This is accomplished by using a set of switch internal link service (SW_ILS) parameters called Exchange Link Parameters (ELP). The ELP is sent from the Fabric Controller (x’FFFFFD’) in one switch to the Fabric Controller in the neighbor switch using Class-F service.

Exchange Switch Capabilities (ESC). Next, ESC is sent between neighboring Fabric Controllers to agree upon a common routing protocol.

Exchange Fabric Parameters (EFP). The principal switch is selected using the Exchange Fabric Parameters (EFP) (SW_ILS). The EFT is sent between Fabric Controllers in neighbor switches.

Domain ID Identifier (DIA). After a principal switch has been selected, Domain_IDs are assigned to the switches. The Principal Switch assigns itself a Domain ID, then floods the fabric with this information.

Request Domain Identifier (RDI). After a switch receives a Domain Identifier Assigned (DIA) switch internal link service, it can request a Domain_ID from the principal switch by sending a Request Domain identifier (RDI) to the principal switch.


Fabric Shortest Path First (FSPF). After the Domain_ID assignment phase is complete, routing tables are built. The switch may use the standardized FSPF protocol or a vendor-unique routing protocol

Build Routing Tables. Finally , each switch computes the paths it will use to deliver frames to other switches.


FSPF This section provides an overview of the FSPF protocol.


FSPF

Fabric Shortest Path First(FSPF):

Computes the least-cost path through the fabric, based on:– Link speed– Number of hops

Avoids looping of framesAll frames follow the same pathEnsures in-order delivery in a stable SAN

FC

Frame 1

Frame 2

Frame 3

Single pathIn-order delivery

FCHBA

FCHBA

The FSPF protocol is the routing protocol used on FC SAN fabrics.

The preceding diagram shows that FSPF selects a single path for a given I/O transaction, avoiding looping and ensuring in-order delivery.

The FSPF algorithm is a cost-based routing algorithm that computes the most efficient path between two connected nodes. The cost of a given path is based on two factors:

The speed of each of the ISLs along the path

The number of hops on the path

Routing using a single fixed path prevents looping of frames and, in a stable SAN, ensures in-order delivery. In other words, if routes are stable, frames always follow the same path. However, if the least-cost route changes while a session is in progress, frames sent after the route change might take the new route.



FSPF (Cont.)

Three protocols used in FSPF:The Hello protocol is used to establish communication between two connected switchesInitial Link State Record (LSR) database synchronizationLSR database maintenance

These protocols use Switch Internal Link Services (SW_ILS) with Class F frames

The FSPF protocol maintains a Topology Database which is distributed to every switch in the fabric. If a switch detects a lost connection, either to a Node or to another switch (ISL), it will update the Topology Database and send a Link State Update frame to all other switches directly connected to it. Each of these switches will update their Topology Database and pass the LSR frame onto other switches. In this way the fabric is flooded with updates to the Topology Database. Any LSR frames already received are discarded to stop duplicate LSRs from being distributed throughout the fabric.



FSPF Protocol Operations

Five stages of the FSPF protocol:1. Hello protocol2. Initial topology database synchronization3. Topology database maintenance4. Path discovery5. Path computation

FSPF Protocol Operations There are five stages associated with the FSPF protocol:

1. Hello protocol

2. Initial topology database synchronization

3. Topology database maintenance

4. Path discovery

5. Path computation

Each of these FSPF stages uses switch internal link services (SW_ILS) and Class F service. However, unlike other SW_ILS operations, there are no expected reply sequences. The SW_ILS request is both the first and last sequence of the exchange. Responses are communicated in a separate SW_ILS request sequence using a new exchange.

Operation of the FSPF protocol stages can be represented using a finite state machine. A separate instance of this state machine operates in every E_Port in the fabric.



FSPF Protocol Operations (Cont.) Stage 1: The Hello Protocol

After a switch acquires a Domain_ID, it begins the process of building a routing table:

Hello messages act as a ‘heartbeat’:– Default Hello Interval = 20s– Default Hello Dead Interval = 80s

After a switch acquires a Domain_ID, it begins the process of building a routing table:

1. Does not know if neighbor switch has acquired a Domain_ID

2. Begins transmitting Hello messages to its neighbors on all initialized ISLs

3. Exchanges Domain_IDs with all neighbors

After two switches have exchanged Domain_IDs, the ISL is active and FSPF topology database synchronization can begin.

Stage 1—The Hello Protocol The first stage of the FSPF protocol is called the Hello protocol. The Hello protocol is used to determine the status of the link connected to the switch’s immediate neighbor. The switches use the Hello protocol to exchange Domain IDs with each other.

After two switches have exchanged Domain_IDs, the ISL is active and the switches can proceed to the next stage of the FSPF protocol.

Hello protocol messages are transmitted on a periodic basis on each interswitch link, even after two-way communication is established. Periodic Hello messages provide a mechanism to detect a switch that has failed. In effect, the Hello messages act as a heartbeat between the switches. If a switch fails to receive a Hello in the expected time, it assumes the neighbor switch is no longer operational:

The Hello Interval is the time in seconds between Hello messages sent by this port. Its default value is 20 seconds.

The Dead Interval is the time in seconds this port will wait for a Hello message from the attached port before removing the route to that port from the LSD. Its default value is 80 seconds.


Note that the default values of these intervals mean that FSPF can take up to 100s to become aware of a link failure. You can lower these values to promote faster recovery when a link fails, but you should also keep in mind that Hello messages are flooded, so smaller Hello Interval values increase congestion. The Hello Dead Interval should generally be set to 4 times the Hello Interval to avoid triggering unnecessary FSPF route computation if Hello messages are lost due to congestion.



FSPF Protocol Operations (Cont.) Stages 2 and 3

Database Sync and Maintenance:

After communication established, switches exchange LSRs to synchronize topology databasesLink State Update (LSU) used to exchange entire LSD Recipients respond with Link State Acknowledgement (LSA)After database in sync, LSUsissued only upon topology changes, which are flooded throughout the entire fabricLSUs retransmitted by a mechanism called “Reliable Flooding”

BALSU(DB-A)

LSU(DB-B)

LSA(DB-B)

LSA(DB-A)

BALSU(LSR-A)

LSU(LSR-B)

LSA(LSR-B)

LSA(LSR-A)

Stage 2—Initial Database Synchronization After two-way communication has been established between two switches using the Hello protocol, the switches begin to synchronize their topology databases. This is accomplished by exchanging LSRs between the switches:

During topology database synchronization, each switch sends its entire LSD topology database to its neighbor.

Switches synchronize databases by sending LSRs in a Links State Update (LSU) SW_ILS extended link service command.

An LSU can contain one or more LSRs. An LSU with zero LSRs signals the end of the database transmission.

When a switch receives an LSU, it compares each LSR in the LSU with its current topology database. If the new LSR is not present in the switch’s LSD, or if the new LSR is newer than the existing LSR, the LSR is added to the database.

Each LSR is acknowledged with a Link State Acknowledgment (LSA) SW_ILS command or with a newer instance of the LSR.

Stage 3—Database Maintenance After the initial database synchronization is complete, the topology database must be maintained to ensure that all switches in the fabric contain identical information in their databases.

Events that cause an LSR to be transmitted include:

An ISL fails (or the switch associated with that ISL fails). A new LSR is transmitted to remove the failed link(s) from the topology database.

An ISL reverts to the ‘one-way’ communication state. A new LSR is transmitted to remove the one-way link from the topology database.


A new ISL completes link initialization (stage 1) and initial database synchronization (stage 2). One or more LSRs are transmitted to notify other switches to add the new information to their databases.

This process by which LSRs are propagated through the fabric is known as “reliable flooding”. When a switch receives an LSR, it retransmits the LSR on other links. After the LSR is acknowledged, the switch stops transmitting that LSR on that link. The switch continues to send the LSR on other links until acknowledgement is received on those links.



FSPF Protocol Operations (Cont.) Stages 4 and 5

Stage 4: Path Discovery:As frames arrive at a switch:– The frame’s Domain_ID is compared to the Domain_ID in the switch’s LSD– If the LSD does not contain that Domain_ID, the least-cost path is then

calculated– The switch always forwards frames on the least-cost path

Stage 5: Path Computation:Link Cost = S * (1.0625e12 / R):– S represents an administratively defined factor

(default value = 1)– R is the bit rate of the link

Examples:– Default link cost for 1Gb/s link:

1 * (1.0625e12 / 1.0625e9) = 1000– Default link cost for 2Gb/s link:

1 * (1.0625e12 / 2.1250e9) = 500FSPF considers ISLs only

Stage 4—Path Discovery As frames arrive at a switch, the frame’s destination Domain_ID is compared to the relevant LSR in the LSD.

If the LSD does not contain the destination Domain_ID, the Path Selector computes the cost of each path to the destination Domain_ID and selects the least-cost path.

Switches must forward frames on the least-cost path.

Stage 5—Path Computation The switch runs the path selection algorithm when it is notified of a physical change to the fabric. It is notified through the process of receiving a new or updated LSU.

The link cost for each individual link is calculated based on the bit rate of the link and an administratively defined weighting factor. The weighting factor allows an administrator to adjust link cost based on particular circumstances. By default, the weighting factor is set to 1.

The cost calculation is based on the bit rate of a 1Gb/s FC link, and is represented by the formula S * (1.0625e12 / R), where S is the administrative weight and R is the bit rate of the link. Assuming that the weighting factor is the default 1, the cost calculation will arrive at the following costs:

Cost of a 1Gb/s link: 1000




The calculation is performed on a link-by-link basis, so each link in a data path can be advertised with a different cost. These costs are used by the path selection algorithm to determine the most efficient paths. When a path contains multiple links, the costs of each link are added up to determine the total cost of the path. In the case of two or more paths of equal cost, the decision of which path to use is not specified and is determined by the switch vendor.

Note that FSPF only considers the ISLs along the data path—it does not consider the node-to-switch link at either end of the path. FSPF routes frames between domains only.



Limitations of FSPF

FSPF algorithm does not account for traffic loadAll frames in an exchange follow the same pathPath changes only in response to changes in the fabric topology

Host 2FC

Storage 2

Host 1FC

Storage 1

Switch A Switch C2Gb/s

Cost=500

2Gb/sCost=500

2Gb/sCost=500

FCHBA

FCHBA

Switch B

Limitations of FSPF The FSPF protocol supports load sharing, but it does not support load balancing. Load sharing is significantly different than load balancing, and the distinction can have significant effects for fabric design, especially when tuning performance:

Load sharing simply means that multiple paths can be used

Load balancing means that traffic load is balanced across multiple paths

FSPF does not account for actual path utilization. In other words, an unused path with a cost of 1000 will be disregarded in favor of an overutilized path with a cost of 500. All frames in an exchange must follow the same path, and paths are recomputed only when the physical ISL configuration changes.

The preceding diagram shows a simple SAN with two data paths:

Path A→C has a total cost of 500

Path A→B→C has a total cost of 1000

FSPF will never use path A→B→C, even if path A→C is congested.



Limitations of FSPF (Cont.)

The least-cost path is not always the best path:– Path A→B→C: cost=1000, bandwidth=2Gb/s– Path A→C : cost=1000, bandwidth=1Gb/s

Load-sharing occurs but cannot be optimized

Host 2FC

Storage 2

Host 1FC

Storage 1

Switch A Switch C

Switch B

1Gb/sCost=1000

2Gb/sCost=500

2Gb/sCost=500

FCHBA

FCHBA

The FSPF least-cost path algorithm does not necessarily select the best path. For example, in the preceding diagram, links A→B and B→C are 2Gb/s links, with a default cost of 500 per link. Link A→C is a 1Gb/s link with a default cost of 1000. (Note that this diagram differs from the previous diagram only in that link A→C is a 1Gb/s link in this diagram.)

There are two paths available from Switch A to Switch C:

Path A→B→C has a total cost of 1000 and supports 2Gb/s along the entire path

Path A→C also has a total cost of 1000 but supports only 1Gb/s

FSPF will weight both paths identically. When a single pair of devices (Host 1 and Storage 1) are attached to the SAN, FSPF might select path A→C even though that path supports only half the bandwidth of path A→B→C. (Path A→B→C does have greater latency than path A→C, but latency is a far less significant performance factor than bandwidth.)

When a second pair of devices (Host 2 and Storage 2) are attached to the same switches, the switch will use the second equal-cost data path in an attempt to distribute the load evenly. In other words, Host 1→Storage 1 will be assigned one path, and Host 2→Storage 2 will be assigned the other path. Both data paths will be used.

In both situations, the administrator can force path selection by adjusting the administrative weighting factor.


The RSCN Process This section explains the Registered State Change Notification (RSCN) process.


RSCN

Registered State Change Notification

FC

Path failure

Host Storage

Switch

FabricController

SCRSCRRSCNRSCN

LS_ACCLS_ACCFC

HBA

Fabric State Changes Changes to the state of the fabric can affect the operation of ports. Examples of fabric state changes include:

A node port is added or removed from the fabric

Inter-switch links (ISLs) are added or removed from the fabric

A membership change occurs in a zone

Ports must be notified when these changes occur.

The RSCN Process The FC-SW standard provides a mechanism through which switches can automatically notify ports that changes to the fabric have occurred. This mechanism, known as the RSCN process, is implemented by a fabric service called the Fabric Controller. The RSCN process works as follows:

Nodes register for notification by sending a State Change Registration (SCR) frame to the Fabric Controller.

The Fabric Controller transmits RSCN commands to registered nodes when a fabric state change event occurs. RSCNs are transmitted as unicast frames because multicast is an optional service and is not supported by many switches.

Only nodes that might be affected by the state change are notified. For example, if the state change occurs within Zone A, and Port X is not part of Zone A, then Port X will not receive an RSCN.


Nodes respond to the RSCN with an LS_ACC frame.

The RSCN message identifies the ports that were affected by the state change event, and it identifies the general nature of the event. After receiving an RSCN, the node can then use additional Link Services commands to obtain more information about the event. For example, if the RSCN specifies that the status of Port Y has changed, the nodes that receive the RSCN can attempt to verify the current (new) state of Port Y by querying the Name Server.



The RSCN Process

Conditions that result in RSCN:An Nx_Port logs in to the fabricPath between two Nx_Ports has changed (e.g., E_Portinitialization or failure)An implicit logout of an Nx_Port (e.g. link failure)Any other fabric-detected state change of an Nx_PortLoop initialization of an L_PortAn Nx_Port issues an RSCN request to the Fabric Controller

The Fabric Controller will generate RSCNs in the following circumstances:

A fabric login (FLOGI) from an Nx_Port.

The path between two Nx_Ports has changed (e.g., a change to the fabric routing tables that affects the ability of the fabric to deliver frames in order, or an E_Port initialization or failure)

An implicit fabric logout of an Nx_Port, including implicit logout resulting from loss-of-signal, link failure, or when the fabric receives a FLOGI from a port that had already completed FLOGI.

Any other fabric-detected state change of an Nx_Port.

Loop initialization of an L_Port, and the L_bit was set in the LISA Sequence.

An Nx_Port can also issue a request to the Fabric Controller to generate an RSCN. For example, if one port in a multi-ported node fails, another port in that node can send an RSCN to notify the fabric about the failure.



The RSCN Process (Cont.)

Port_ID of affected Nx_Port

Area_ID of affected Nx_Port

Domain_ID of affected Nx_Port

Address FormatEvent QualifierReserved

0001 — Changed Name Server object0010 — Changed port attribute0011 — Changed fabric service object0100 — Changed switch configuration

0

1

2

3

7 6 5 4 23 0Bit

Byte 1

Not veryinformative

An RSCN frame payload contains one or more Port_ID Pages. Each Port_ID page is a 4-byte page that describes a single state change that has occurred with respect to a single Nx_Port. Each Port_ID page contains the following fields:

The Domain_ID, Area_ID, and Port_ID of the affected Nx_Port (bytes 1-3)

The Event Qualifier (bits 2-5 of byte 0)

The Event Qualifier is a 4-bit code that specifies the general nature of the event:

0001 — A Name Server object has changed; for example, a port came online or went offline.

0010 — A port attribute has changed; for example, the number of buffer credits assigned to that port was changed.

0011 — A fabric service object has changed; for example, an Alias_ID was added. In this case, the Port_ID page will refer to the Well-Known Address of the affected fabric service.

0100 — The switch configuration has changed; for example, a time-out value was changed.

Note that the Event Qualifiers do not communicate much information. For example, Event Qualifier code 0001 indicates a change to a Name Server object. This could signify that a port came online, went offline, or changed zones. The ports that receive the RSCN must then query the Name Server to determine the specific change that occurred.


Standard Fabric Services This section lists the standard fabric services defined by the FC specification.


Standard Fabric Services

Common Transport

Link Services


FC-1 Encoding

FC-2 Framing & flow control

FC-3 Generic Services

FC-4 ULP Mapping

Generic ServicesFabric Controller

Alias Server

Con

f igur

atio

nS

erve

r

Management Server

Zone

S

e rve

r

Unz

one d

Nam

e S

e rve

r

Name Server

Domain Manager

Key Server

Time Server

The FC-SW-2 specification defines several services that are required for fabric management. These services include:

Name Server

Login Server

Address Manager

Alias Server

Fabric Controller

Management Server

Key Distribution Server

Time Server

The FC-SW-2 specification does not require that switches implement all of these services; some services can be implemented as an external server function. However, the services discussed in this lesson are typically implemented in the switch, as in Cisco MDS 9000 Family Switches.



The Domain Manager

Domain Manager

Principal SwitchSelection

FabricConfiguration

Domain IDAllocation

FCID Databaseand Cache

FC_ID Allocation

Port Manager Login Server

WWN ManagerVSAN Manager

Management Services

The Domain Manager The Domain Manager is the logical function of a switch that is responsible for the assignment of addresses in a fabric. The Domain Manager is responsible for:

Allocating domain IDs (requesting a domain ID, and assigning domain IDs to other switches if this switch is the Principal Switch)

Allocating port addresses (FC_IDs)

Participating in the Principal Switch selection process

Performing the Fabric Build and Reconfiguration processes when the topology changes

The Domain Manager supports the Fabric Port Login Server, which is the service that N_Ports use when logging in to the fabric. When an N_Port logs into the fabric, it sends a FLOGI command to the Login Server. The Login Server then requests an FC_ID from the Domain Manager and assigns the FC_ID the N_Port in its ACC reply to the FLOGI request.

The preceding diagram shows how the Domain Manager interacts with other fabric services:

The VSAN Manager provides the Domain Manager with VSAN configuration and status information.

The WWN Manager tells the Domain Manager what WWN is assigned to the VSAN.

The Port Manager provides the Domain Manager with information about the fabric topology (a list of E_Ports) and notifies the Domain Manager about E_Port state changes.

The Login Server receives N_Port requests for FC_IDs during FLOGI.

The Domain Manager interacts with management services to allow administrators to view and modify Domain Manager parameters.



The Name Server

Name Server stores data about nodes, such as:– FC_IDs– WWNs– Fibre Channel operating parameters

Supports soft zoningProvides information only about nodes in the requestor’s zoneDistributed Name Server (dNS) resides in each switchResponsible for entries associated with that switch’s domainMaintains local data copies and updates via RSCNsSends RSCNs to the fabric when a local change occurs

The Name Server FC Name Server is a database implemented by the switch that stores information about each node, including:

FC_IDs

WWPN and WWNNs

FC operating parameters, such as supported ULPs and Classes of Service

The Name Server:

Supports soft zoning by performing WWN lookups to verify zone membership

Enforces zoning by only providing information about nodes in the requestor’s zone

Is used by management applications that need to obtain information about the fabric

Each switch in a fabric contains its own resident name server, called a distributed Name Server (dNS). Each dNS within a switch is responsible for the name entries associated with the domain assigned to the switch. The dNS instances synchronize their databases using the RSCN process.

When a client Nx_Port wants to query the Name Service, it submits a request to its local via the Well Known Address for the Name Server. If the required information is not available locally, the dNS within the local switch responds to the request by making any necessary requests of other dNS instances contained in the other switches. The communication between switches that is performed to acquire the requested information is transparent to the original requesting client.

Partial responses to dNS queries are allowed. If an entry switch sends a partial response back to an Nx_Port, it must set the partial response bit in the CT header.



Name Server Operations

FC-4 Features

FC-4 Descriptors

Hard Address

Fabric Port NamePort IP Address

Port Type

Symbolic Port Name

Symbolic Node NameFC-4 TYPEs

Initial Process AssociatorClass of Service

Node IP AddressNode NamePort NamePort Identifier

Indexed FieldsSecondary KeyIndexed FieldsPrimary Key

Name Server Database Objects

Name Server Operations When ports and nodes register with the Name Server, their characteristics are stored as objects in the Name Server database. The table in the preceding graphic shows how the objects are organized in the Name Server database.

The Port Identifier is the Fibre Channel port address identifier (FC_ID) assigned to an N_Port or NL_Port during fabric login (FLOGI). The Port Identifier is the primary key for all objects in a Name Server record. All objects are ultimately related back to this object. Because a node may have more than one port, the Node Name is a secondary key for some objects.

There are three types of Name Server requests:

Get Object: This request is used to query the Name Server

Register Object: Only one object at a time can be registered with the Name Server. A Client registers information in the Name Server database by sending a registration request containing a Port Identifier or Node Name.

Deregister Object: Only one global deregistration request is defined for the Name Server.

Name Server information is available, upon request, to other nodes, subject to zoning restrictions. If zones exist within the fabric, the Name Server restricts access to information in the Name Server database based on the zone configuration.

When a port logs out of a fabric, the Name Server deregisters all objects associated with that port.



The Management Server

Management Server:Information is provided without regard to zone—single access point for information about the fabric topologyRead-only accessServices provided:– Fabric Configuration Service (FCS)– Zone Service– Unzoned Name Service

The Management Server The FC Management Server provides a single access point for obtaining information about the fabric topology. Whereas the Name Server only provides information about ports configured within the zone of the port requesting information, the Management Server provides information about the entire fabric, without regard to zone. The Management Server allows SAN management applications to discover and monitor SAN components, but it does not allow applications to configure the fabric—the Management Server provides read-only access to its data.

The Management Server provides the following services:

The Fabric Configuration Service (FCS) supports configuration management of the fabric. This service allows applications to discover the topology and attributes of the fabric.

The Zone Service provides zone information for the fabric to either management applications or directly to clients.

The Unzoned Name Service provides access to provide information about the fabric without regard to zones. This service allows management applications to see all the devices on the entire fabric.



Well-Known Addresses

Well-known addresses are the highest 16 addresses in the 24-bit fabric address space

FFFFF4 – FFFFF0ReservedOptionalFFFFF5Multicast Server OptionalFFFFF6Clock Synchronization ServerOptionalFFFFF7Key Distribution ServerOptionalFFFFF8Alias ServerOptionalFFFFF9QoS FacilitatorOptionalFFFFFAManagement Server OptionalFFFFFBTime ServerOptionalFFFFFCName Server

MandatoryFFFFFDFabric ControllerMandatoryFFFFFEFabric Login ServerMandatoryFFFFFFBroadcast Alias

Well-Known Addresses Well-known Addresses allow devices to reliably access switch services. All services are addressed in the same way as an N_Port is addressed. Nodes communicate with services by sending and receiving Extended Link Services commands (frames) to and from Well-Known Addresses

Well-known addresses are the highest 16 addresses in the 24-bit fabric address space:

FFFFFF - Broadcast Alias

FFFFFE - Fabric Login Server

FFFFFD - Fabric Controller

FFFFFC - Name Server

FFFFFB - Time Server

FFFFFA - Management Server

FFFFF9 - Quality of Service Facilitator

FFFFF8 - Alias Server

FFFFF7 - Key Distribution Server

FFFFF6 - Clock Synchronization Server

FFFFF5 - Multicast Server

FFFFF4–FFFFF0 - Reserved




Summary

The FSPF protocol is the routing protocol used on FC SAN fabricsFive stages of the FSPF protocol: Hello protocol; Initial topology database synchronization; Topology database maintenance; Path discovery; Path computationFC Name Server is a database implemented by the switch that stores information about each nodeThe Fabric Controller service provides a mechanism for state change notification through the Registered State Change Notification (RSCN) process Well-known addresses are the highest 16 addresses in the 24-bit fabric address space

Appendix B

Installation and Configuration Reference

Overview This appendix reviews the installation and configuration guidelines for MDS 9000 switches.

Module Objectives Upon completing this module, you will be able to describe installation and configuration guidelines. This includes being able to meet these objectives:

Explain the process used to install and power up the switch

Lesson 1

Switch Hardware Installation Reference

Overview This lesson describes the interfaces, functions, and installation practices of MDS 9000 Family system components to provide knowledgeable answers to questions related to these system component topics. You must be familiar with the system components before installing them in a MDS 9000 Family environment.

Objectives Upon completing this lesson, you will be able to explain the process used to install and power up the switch. This includes being able to meet these objectives:

Describe the installation guidelines for the MDS 9000 platform

Describe the types of rack and cabinet installations that are compatible with the MDS 9000 platform

Describe the power supply configuration options for the MDS 9000 platform, and state the power requirements of individual modules

Describe the characteristics and installation requirements of fan modules for the MDS 9000 Series

Describe the functions, interfaces, and installation requirements of MDS 9000 supervisor modules


Installation Guidelines This topic describes the installation guidelines for the MDS 9000 platform.


Installation Guidelines

Prepare the site:Space evaluationWeight distribution and floor loadingEnvironmental evaluation Power evaluationGrounding evaluation Cable and interface equipment evaluation EMI evaluation Gather network-related information

Verify the cabinet and rack: Hardware specificationsDepth, airflow, and vertical clearance

Confirm site power:UPS type Circuit sizes

Gather required equipment:ToolsESD strapsGrounding items

Unpack and inspect switch:Check for transportation damage and missing items Document chassis and module informationVerify contents of rack mount kit

Installation Guidelines Follow these guidelines when installing the Cisco MDS 9500 Series:

Plan your site configuration and prepare the site before installing the chassis. It is recommended that you use the site planning tasks listed in the Cisco MDS Series Hardware Installation Guide.

Ensure there is adequate space around the switch to allow for servicing the switch and for adequate airflow.

Ensure the air conditioning meets the heat dissipation requirements listed in the Cisco MDS Series Hardware Installation Guide.

Ensure the cabinet, or rack, meets the requirements listed in the Cisco MDS Series Hardware Installation Guide.

Note Jumper power cords are available for use in a cabinet.

Ensure the chassis is adequately grounded. Grounding the chassis is recommended in all cases, and it is mandatory for Cisco MDS 9506 Directors that have a DC power supply installed. If the switch is not mounted in a grounded rack or cabinet, it is recommend connecting both the system ground on the chassis and the power supply ground to an earth ground, regardless of whether the power supplies are AC or DC.

Ensure the site power meets the power requirements listed in the Cisco MDS Series Hardware Installation Guide. If available, you can use an uninterruptible power supply (UPS) to protect against power failures.

© 2007 Cisco Systems, Inc. Appendix B: Installation and Configuration Reference AB-5

Note Avoid UPS types that use ferroresonant technology. These UPS types can become unstable with systems such as the Cisco MDS 9000 Family, which can have substantial current draw fluctuations because of fluctuating data traffic patterns.

Ensure circuits are sized according to local and national codes.

If you are using 200/240 VAC power sources in North America, the circuits must be protected by two-pole circuit breakers.

Note To prevent loss of input power, ensure that the total maximum loads on the circuits supplying power are within the current ratings of the wiring and breakers.

Record your installation and configuration information as you work. See “Site Planning and Maintenance Records” in the Cisco MDS Series Hardware Installation Guide.

Screw Torques Use the following screw torques when installing the switch:

Captive screws: 4 in-lb

M3 screws: 4 in-lb

M4 screws: 12 in-lb

10-32 screws: 20 in-lb

12-24 screws: 30 in-lb

Required Equipment Gather the following items before beginning the installation:

Number 1 and number 2 Phillips screwdrivers with torque capability

3/16-inch flat-blade screwdriver

Tape measure and level

ESD wrist strap or other grounding device

Antistatic mat or antistatic foam

In addition to the grounding items provided in the accessory kit, you need the following items:

Grounding cable (6 AWG recommended), sized according to local and national installation requirements; the required length depends on the proximity of the Cisco MDS 9500 to proper grounding facilities.

Crimping tool large enough to accommodate girth of lug

Wire-stripping tool

For DC power supplies in a Cisco MDS 9506 Director, you need two 10-32 ring lugs for each DC power supply.



Weight Distribution and Floor Loading

Installation of the Cisco MDS 9513 Director in a rack requires a mechanical lift to place the chassis in the rack:

Chassis weighs 101.0 lbsComponents:

– 6 KW power supply: 32.5 lbs– System front-fan tray: 18.0 lbs– Fabric rear-fan tray: 2.25 lbs– Fabric card: 5.75 lbs– Four-port 10-Gbps line card : 8.5 lbs– 12-port line card: 7.5 lbs – 48-port line card : 11.0 lbs– 24-port line card : 7.75 lbs– Supervisor-2 line card : 7.25 lbs– Line card blank panels: 0.50 lbs

Front view

14 RU form factor, 28” deep

Rear view

Powe

r Sup

plie

s

Fabr

ic c

ards

Fan-

R

Cloc

kMod

ules

Installations with three 9513s per rack should include a floor-loading assessment to support 296.75 lbs as pictured.

Installation of the Cisco MDS 9513 Director in a rack requires a mechanical lift to place the chassis in the rack. Make sure you have access to the lift during the installation process. A fully loaded 9513 can weigh about 300 pounds. Installations with three 9513s per rack should include a floor-loading assessment to evaluate the static-load rating of the flooring as part of the site evaluation process. For additional information about floor loading requirements, consult the UL document GR-63-CORE, Network Equipment-Building System (NEBS) Requirements: Physical Protection.

Module Weights The following components are listed with their weights:

Crossbar switching module 6 lbs ( 2.7 kg)

48-port 4-Gbps switching module 11.0 lbs ( 4.99 kg)

24-port 4-Gbps switching module 7.75 lbs (3.52 kg)



32-port FC switching module 9 lbs (4.1 kg)

16-port FC switching module 9 lbs (4.1 kg)

Source-specific multicast (SSM) 11 lbs (5 kg)

Advanced Services Module (ASM) 11 lbs (5 kg)

Cisco Security Manager (CSM) 11.5 lbs (5.2 kg)

IPS-8 10 lbs (4.5 kg)

IPS-4 9 lbs (4.1 kg)

MPS-14/2 10 lbs (4.5 kg)

Supervisor-2 for MDS 9500 Series 7.25 lbs ( kg)


Supervisor-1 for MDS 9500 Series 9 lbs (4.1 kg)

Supervisor for MDS 9200 Series 9 lbs (4.1 kg)

Crossbar module fan tray 2.25 lbs (1.13 kg)

Module blank panels 0.50 lbs ( 0.25 kg)


Cabinet and Rack Options This topic describes the types of rack and cabinet installations that are compatible with the MDS 9000 platform.


Installation Options

Standard telco rack (no side panels):Not intended for use with the Cisco MDS 9513Minimum of 6 inches (15.2 cm) of clearance between chassis is recommendedMinimum of 2.5 inches (6.4 cm) of distance between the chassis air vents and any walls is required

Cabinet with solid side panels and doors:Roof-mounted fans delivering a minimum of 500 cubic feet per minuteBottom-to-top airflow

Cabinet with perforated front and back doors:Roof-mounted fans are recommended but not requiredFront-to-back airflow

The Cisco MDS 9506 and MDS 9509 directors can be installed using the following methods:

In an open EIA rack, using:

The rack mount kit shipped with the switch.

The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the rack mount kit shipped with the switch.

In a perforated or solid-walled EIA cabinet, using:


The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the rack mount kit shipped with the switch.

In a two-post telco rack, using:


The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the front brackets shipped with the switch.

The Cisco MDS 9509 Director can also be installed in a four-post nonthreaded cabinet or rack, using the optional 9500 Shelf Bracket Kit.

The Cisco MDS 9513 Director can be installed in solid or perforated walled cabinets, but not in two-post telco racks.

Note The Telco and EIA Shelf Bracket Kit is optional and is not provided with the switch. To order the kit, contact your switch provider.


Note The Telco and EIA Shelf Bracket Kit is not intended for use with a Cisco MDS 9509 Director in a two-post telco rack. The MDS 9513 exceeds telco rack load ratings.

Requirements and recommendations for perforated cabinets are:

The front and rear doors must have at least a 60-percent open-area perforation pattern with at least 15 square inches of open area per rack unit of door height.

The roof should be perforated with at least a 20-percent open-area perforation pattern.

An open or perforated cabinet floor is recommended to enhance cooling.

Requirements and recommendations for solid-walled cabinets are:

A roof-mounted fan tray with bottom-to-top airflow that has a minimum of 500 cfm of airflow exiting the cabinet roof through the fan tray.

Non-perforated (solid and sealed) front and back doors and side panels so that air travels predictably from bottom to top.

A cabinet depth of 36 to 42 inches (91.4 to 106.7 cm) to allow the doors to close and adequate airflow is recommended.

A minimum of 150 square inches (968 sq. cm) of open area must be at the floor air intake of the cabinet.

The lowest piece of equipment should be installed a minimum of 1.75 inches (4.4 cm) above the floor openings to prevent blocking the floor intake.

Requirements and recommendations for telco racks are:

Minimum of 6 inches (15.2 cm) of clearance between chassis is recommended.

Minimum of 2.5 inches (6.4 cm) of distance between the chassis air vents and any walls is required.

Not intended for use with the Cisco MDS 9513 director.



General rack and cabinet requirements:The minimum vertical rack space for a Cisco MDS 9513 chassis is 24.5 inches (64.23 cm)The width between the rack mounting rails must be at least 17.75inches (45.09 cm)The minimum spacing for four-post EIA cabinets (perforated or solid-walled) is a minimum of 3 inches (7.62 cm)

Cabinet and Rack Options

The cabinet or rack must conform to the following:

Standard 19-inch four-post EIA cabinet, or rack, with mounting rails that conform to English universal hole spacing per section 1 of ANSI/EIA-310-D-1992.

Standard two-post telco racks are not intended for use with the 9513.

General Rack Requirements The minimum vertical rack space per chassis is as follows:

Cisco MDS 9513 chassis; 24.5 inches (62.2 cm) or 14 RU

Height with required rack mount support is 15 RU

The width between the rack-mounting rails must be at least 17.75 inches (45.1 cm). For four-post EIA racks, this is the distance between the two front rails and rear rails.

The minimum spacing for four-post EIA cabinets (perforated or solid-walled) is as follows:

To ensure the minimum bend radius for fiber optic cables, the front mounting rails of the cabinet should be offset from the front door by a minimum of 3 inches (7.6 cm) and a minimum of 5 inches (12.7 cm) if cable management brackets are installed on the front of the chassis.

A minimum of 2.5 inches (6.4 cm) of clear space between the side edge of the chassis and the side wall of the cabinet; no sizeable flow obstructions should be immediately in the way of the chassis air intake or exhaust vents.

Telco racks are not intended for use with the Cisco MDS 9513 Director. The MDS 9513 exceeds telco rack load ratings.



Cabinet and Rack Options (Cont.)

Open rack requirements:The minimum width between two front-mounting rails must be 17.75 inches (45.1 cm)The minimum vertical rack space for each Cisco MDS 9513 chassis must be 24.5 inches (62.2 cm), or 14 RUThe rack mount support brackets provided with the Cisco MDS 9513 Director require an additional height of 0.75 inches (1.9 cm). They are required for the installation and can not be removedThe horizontal distance between the chassis and any adjacent chassis should be 6 inches (15.2 cm)The distance between the chassis air vents and any walls should be 2.5 inches (6.4 cm)

If mounting the chassis in an open rack (no side panels or doors), ensure the rack meets two requirements:

The minimum width between two front mounting rails must be 17.75 inches (45.1 cm).

The minimum vertical rack space per chassis must be at least:

For the Cisco MDS 9513 chassis 24.5 inches (62.2 cm), or 14 RU.



Note The rack-mount support brackets provided with the Cisco MDS 9513 Director require an additional height of 0.75 inches (1.9 cm). They are required for the installation of the Cisco MDS 9513 Director and can not be removed.

Note The side rail mount brackets provided with the Cisco MDS 9509 Director require an additional height of 0.75 inches (1.9 cm). They are required only for the installation of the Cisco MDS 9509 Director and can be removed, or left installed, after the front rack mount brackets are securely fastened to the rack-mounting rails.


Configuring Power Supplies This topic describes the power supply configuration options for the MDS 9000 platform, and the power requirements of individual modules.


MDS 9513 Power Supply Installation

Installing an AC power supply in the Cisco MDS 9513 Director

To install an AC power supply in the Cisco MDS 9513 Director, follow these steps:

Step 1 Ensure that the system (earth) ground connection has been made.

Step 2 If a filler panel is installed, remove the filler panel from the power supply bay by loosening the captive screw.

Step 3 Ensure that the power switch is in the off (0) position on the power supply that is being installed.

Step 4 Grasp the power supply handles, one with each hand. Orient the power supply and align it with the bay.

Note There is a handle at the top rear of the power supply you can also use to tilt the power supply into the bay.

Step 5 Slide the power supply into the power supply bay. Ensure that the power supply is fully seated in the bay.

Step 6 Secure all four 6-32 panel fasteners and tighten to 8 in-lbs.



MDS 9513 Power Supply Installation (Cont.)

Providing power to an AC power supply

1. Power switch 2. Cable retention

Step 1 Plug the power cable into the power supply. Tighten the screw on the cable retention device to ensure the cable can not be pulled out.

Step 2 Connect the other end of the power cable to an AC power source.

Step 3 Turn the power switch to the on (1) position on the power supply.

Step 4 Verify power supply operation by checking that the power supply LEDs are in the following states:

Input OK: LEDs are green.

Fans OK: LED is green.

Output Fail: LED is off.



MDS 9513 Power Supply Configuration

The dual 6000 W AC power supplies for the Cisco MDS 9513 Director are designed to provide an output power for the modulesand fans. Each power supply has two AC power connections.

Power Supply 1 and Power Supply 2 with 2x220 V inputs

Status LEDs with 2x220 V inputs

Configuring Power Supplies The 6000-W AC power supplies for the Cisco MDS 9513 Director are designed to provide an output power for the modules and fans. Each power supply has two AC power connections and provides power as follows:

One AC power connection at 110 VAC: No output

Two AC power connections at 110 VAC: 2900-W output

One AC power connection at 220 VAC: 2900-W output

Two AC power connections at 220 VAC: 6000-W output



MDS 9513 Power Supply Configuration (Cont.)

Removing an AC Power Supply:Warning! Voltage is present on the backplane when the system is operating. Keep hands and fingers out of the power supply bays and backplane areas!

Use both hands to install and remove power supplies. Each power supply weighs 34.2 lbs (15.5 kg).

Removing an AC Power Supply Follow these steps to remove an AC power supply:

Step 1 Turn the power switch on the power supply to the off (0) position. There is an internal-lock mechanism that prevents you from removing the power supply if it is not set to the off position.

Step 2 Disconnect the power cables from the power source.

Step 3 Loosen the screw on the cable retention device, and disconnect the power cable from the power supply.

Step 4 Loosen all four panel fasteners at the corners of the power supply.

Step 5 Grasp the power supply handles and slide the power supply partially out of the chassis, about 4 to 5 inches.

Step 6 If the power supply is at your waist or chest level, place your other hand underneath the power supply and slide the power supply completely out of the chassis.

Note To avoid damage to the panel fasteners, do not place the power supply down on the perforated ends.

Step 7 Install a filler panel over the opening. Tighten the captive screws if the power supply bay is to remain empty.




1. Verify the power supply switch is off and no power is connected to the unit.

2. Use both hands to slide the power supply into the unit so it is seated properly.

3. Tighten the captive screw.4. Ensure the power source can

supply enough power to drive the installed supply.

5. Connect the power input cables to the power source.

MDS 9509 AC Power Supply Installation To install an AC power supply, follow these steps:

Step 1 Remove the blank power-supply filler plate from the chassis power-supply bay opening by loosening the captive installation screw, if necessary.

Step 2 Turn the power switch to the off (0) position on the power supply you are installing.

Step 3 Grasp the power supply handle with one hand. Place your other hand underneath the power supply, as shown in the figure. Slide the power supply into the power supply bay. Make sure that the power supply is completely seated in the bay.

Step 4 Tighten the power supply captive installation screw.

Step 5 Plug in the power cord to the power supply and tighten the screw on the cable retention device.

Step 6 Turn the power switch to the on (1) position on the power supply you are installing.

Step 7 Check LED status indicators for proper operation.

MDS 9509 DC Power Supply Installation To install a DC power supply, follow these steps:

Step 1 Remove the blank power-supply filler plate from the chassis power-supply bay opening by loosening the captive installation screw, if necessary.

Step 2 Turn the power switch to the off (0) position on the power supply that you are installing.

Step 3 Grasp the power supply handle with one hand. Place your other hand underneath the power supply, as shown in the figure. Slide the power supply into the power supply bay. Make sure that the power supply is fully seated in the bay.

Step 4 Tighten the power supply captive installation screw.


Step 5 Remove two screws securing the terminal block cover. Slide the cover off the terminal block.

Step 6 Attach appropriate lugs to the DC-input wires. The maximum width of a lug is 0.300 inch (7.6 cm). The wire should be sized according to local and national installation requirements.

Note Use only copper wire.

Step 7 Connect DC-input wires to the terminal block in the following order: (1) ground, (2) negative (-), (3) positive (+).

Step 8 Turn the power switch to the off (0) position on the power supply that is being installed.

Step 9 Check LED status indicators for proper operation.

Note For redundant or combined power requirements, the number and type of line cards and supervisor modules determine the amount of power needed by the chassis. If each power supply in the chassis is capable of supplying the total chassis power, then the power supplies can be redundant. If each power supply is unable to supply the total chassis power, then the power supplies are shared, and the loss of one supply results in some of the cards being inoperable.



MDS 9509 Power Supplies

Dual power supply modules:RedundantHot swappableCondition LEDsInput:– 100 to 240 VAC nominal– 16 A maximum

Output:– 1400 W (100 VAC at 16 A)– 3000 W (200 VAC at 16 A)

3000 Watt AC Power Supply1. AC power connection 2. Power cable 3. Power supply switch 4. Power supply LEDs5. Captive screws

The MDS 9500 series supports redundant hot swappable power supplies that support AC or DC input voltages. Each power supply is capable of supplying sufficient power to the entire chassis should one fail. The power supplies monitor their output voltage and provide status to the supervisor module. To prevent the unexpected shutdown of an optional module, the power management software only allows a module to power up if adequate power is available.

The power supplies can be configured to be redundant or combined. By default, they are configured as redundant so that if one fails, the remaining power supply can still power the entire system. Condition LEDs give visual indications of the installed modules and their operation.



MDS 9506 Power Supplies

Dual power supply modules:RedundantHot swappableCondition LEDsInput:– 100 to 240 VAC nominal– 12 A maximum

Output:– 1900 W at 200 to 240 VAC– 1050 W at 100 to 120 VAC

1900 W AC or DC1. Power supply LEDs2. Captive screws

As with the MDS 9509 power supplies, the MDS 9506 Director supports redundant AC hot-swappable power supplies, each of which is capable of supplying sufficient power to the entire chassis should one power supply fail. The power supplies monitor their output voltage and provide status to the supervisor module. Also, they can be configured to be redundant or combined. By default, they are configured as redundant. Condition LEDs are also available on the power supply modules.

The 1900-watt supply provides full output capabilities when powered by 220 VAC; however, that output is reduced to 1050 watts when powered by a 110 VAC input. It has a current rating of 15 amps, but a maximum draw of 12 amps under normal conditions.



MDS 9506 Power Supply PEMs

1. Captive screws2. PEM switch3. AC power connection

MDS 9506 Power Supply PEMs The MDS 9506 Director uses power entry modules (PEMs) that are installed in the front of the chassis to provide power to its power supplies in the back. There is no power connector on the power supply itself. The PEM provides current protection, surge and EMI suppression, and filtering functions.

The PEM that is on the left when viewed from front of the switch (PEM 1) connects the site power source to power supply 1 (upper power supply) in the back of the chassis; the PEM on the right (PEM 2) connects the site power source to power supply 2 (lower power supply), also in the back of the chassis.

To provide power to an AC power supply in an MDS 9506 Director, follow the same connection and power-on steps as indicated in MDS 9509 procedures, but only after the MDS 9506 AC PEM has been securely installed in the chassis front, and the AC power supply has been installed in the rear of the chassis.

In addition, the PEM provides current protection, surge and EMI suppression, and filtering functions. The PEM that is on the left when viewed from front of the switch (PEM 1) connects the site power source to power supply 1 (upper power supply) in the back of the chassis; the PEM on the right (PEM 2) connects the site power source to power supply 2 (lower power supply), also in the back of the chassis.

To provide power to an AC power supply in an MDS 9506 Director, follow the same connection and power-on steps as indicated in MDS 9509 procedures, but only after the MDS 9506 AC PEM has been securely installed in the front of the chassis, and the AC power supply has been installed in the rear of the chassis.


Power to DC Power Supplies To provide power to a DC power supply in an MDS 9506 Director, follow these steps:

Step 1 Ensure that all power to the DC circuit is off.

Step 2 Ensure that the system (earth) ground connection is made.

Step 3 Loosen the captive screws on the DC PEM. Pull the PEM part way out of the chassis to provide access to the PEM terminal block screws.

Step 4 The process for connecting the positive and negative DC cables to the DC PEM with a 10-32 ring lug for each cable is as follows:

1. Identify the positive and negative DC cables and ensure that both are copper and sized according to local and national installation requirements.

2. Strip the cable ends to allow for metal-to-metal contact. Insert each cable into a separate ring lug. Crimp the lugs around the cables.

3. Insert each cable and lug into the appropriate hole in the front of the PEM. Fasten the lugs to the appropriate terminal block screws in the following order: (1) negative (-), (2) positive (+).

4. Secure the cables in place by tightening the terminal block screws.




845W AC power supplies:Dual and hot swappable– 100-240 VAC– 15 A current rating– 12 A at 100-120 VAC– 5 A at 200-240 VAC

Self-monitoring of output voltages– Status provided to supervisor– +3.3V at 10 A– +5.0V at 16.2 A

1. Power supply switch2. AC power connection3. Power supply handle4. Power supply LEDs

The Cisco MDS 9216 switch supports dual hot swappable 845-watt AC power supplies. Each supply is autoranging on the input voltage and can provide sufficient power to the entire chassis should one of them fail. They also monitor their own output voltage and provide status to the system’s supervisor module. MDS 9216 power supplies can be configured to be redundant or combined. By default, they are configured as redundant, so that if one fails, the remaining power supply can still power the entire system.

The MDS 9216 power supplies are field-replaceable units (FRUs), are installed, and can be removed easily from the rear of the chassis using pull handles. They also provide condition LEDs for operational status.

The MDS 9216 supports AC voltage inputs only, not DC, ranging from 100 to 240 VAC. The power supplies have a current rating of 15 amps for circuit breakers but draw a maximum of 12 amps at 110 VAC and only 5 amps at 220 VAC.

MDS 9216 Power Supply Installation To install an MDS 9216 power supply, follow these steps:


Step 2 If the power-supply bay has a filler panel, loosen the screws holding it on and remove the panel.

Step 3 Verify that the power switch is in the off (0) position on the power supply you are installing.

Step 4 Orient the power supply as shown in the figure. Hold it by the handle and slide the power supply into the chassis power supply bay.

Step 5 Ensure that the power supply is completely seated in the bay. Tighten the power supply captive screws.

Step 6 Plug the power cable into the power supply. Tighten the screw on the power cable retainer to ensure that the cable can not be pulled out.


Step 7 Connect the other end of the power cable to an AC power source.


Step 9 Verify power supply operation by checking that the power supply LEDs are in the following states:

Input OK: LED is green.

Fan OK: LED is green.

Output Fail: LED is off.

Note In a system with dual power supplies, connect each power supply to a separate power source. In case of a power source failure, the second source will most likely still be available.




The 300-W power supplies require a 20 A circuitWhen using a 200- or 240-VAC power source in North America, the circuit must be protected by a two-pole circuit breakerAC input voltage:

– Minimum = 85 VAC – Nominal = 100-240 VAC – Maximum = 264 VAC

AC input current rating (max):– 4.7 A at 85 VAC– .6 A at 110 VAC– .8 A at 220 VAC

Dual 300-W AC Input Power Supplies Installation in MDS 9100 Fabric Switches

To install a dual 300-W AC input power supply in an MDS 9100 fabric switch, follow these steps:


Step 2 Make sure the power cord is disconnected before installing the power supply.

Step 3 Verify that the power switch is in the off (0) position on the power supply you are installing.

Step 4 Slide the power supply into the power supply bay. Make sure that the power supply is completely seated in the bay.

Step 5 Tighten the power supply captive screw.

Step 6 Plug in the power cord to the power supply.

Step 7 Connect the other end of the power cord to an AC input power source.

Note Depending on the outlet receptacle on your power distribution unit, you might need the optional jumper power cord to connect the Cisco MDS 9216 switch to your outlet receptacle.


Step 9 Verify the power supply’s operation by checking that the power supply (P/S) LED in the front panel is green.



Power Management Modes

Redundant mode:– Default mode– Power capacity of the lower-capacity supply– Sufficient power is available in case of failure

Combined mode:– Is nonredundant– Twice the power capacity of the lower-capacity supply– Sufficient power might not be available in case of a power supply failure– System reset if power requirements exceed capacity– Only modules with sufficient power are powered up– If no reset, no modules down but no new modules up– Should not be used for director-class switches

Power reserved for the supervisor and fan assembliesPower failure triggers Syslog, Call Home, and SNMP trap

Power supplies are configured in redundant mode by default, but they can also be configured in a combined, or nonredundant, mode:

Redundant mode: The chassis uses the power capacity of the lower-capacity power supply so that sufficient power is available in case of a single power supply failure.

Combined mode: The chassis uses twice the power capacity of the lower-capacity power supply. Sufficient power might not be available in case of a power supply failure in this mode. If there is a power supply failure and the real power requirements for the chassis exceed the power capacity of the remaining power supply, the entire system is reset automatically to prevent permanent damage to the power supply.

In both modes, power is reserved for the supervisor and fan assemblies. Each supervisor module has roughly 220 watts in reserve, even if there is only one installed; and the fan module has 210 watts in reserve. In the case of insufficient power, after supervisors and fans are powered, line card modules are given power from the top of the chassis down.

After the reboot, only those modules that have sufficient power are powered up.

If the real power requirements do not trigger an automatic reset, no module is powered down. Instead, no new module is powered up.

In all cases of power supply failure or removal, the following occur:

A Syslog message is printed

A Call Home message is sent if configured

A Simple Network Management Protocol (SNMP) trap is sent

Note Combined mode should not be used for director-class switches.


Installing Fan Modules This topic describes the characteristics and installation requirements of fan modules for the MDS 9000 Family.


MDS 9500 Series Fan Modules and Airflow

Fans are hot swappable: In the event of a fan failure, remaining fans increase RPM to maintain optimal airflow Sensors monitor system temperatureTemperature rise or fan failure generates an eventReplace fan tray at earliest opportunity

MDS 9513 rear airflow bottom to top, at rear of chassis

MDS 9506, 9509, and 9513 airflow

Air Flow

The MDS 9500 Series supports hot swappable fan modules that are easily installed or removed from the from of the chassis. They provide 85 cfm of airflow per slot with 410 watts of power dissipation per slot. The MDS 9506 has a fan module with 6 fans, the Cisco MDS 9509 has a fan module with 9 fans, and the Cisco MDS 9513 has a fan module with 15 fans.

Sensors on the supervisor module monitor the internal air temperature. If the air temperature exceeds a preset threshold, the environmental monitor displays warning messages. If one or more fans within the module fails, the Fan Status LED turns red, and the module must be replaced. When all fans are operating properly, the LED is green. If the fan LED is red, the fan assembly might not be seated properly in the chassis. If this happens, remove the fan assembly and reinstall. After reinstalling, if the LED is still red, then a failure on the fan assembly has occurred. Fan LED status indication is provided on a per-module basis. If one fan fails, then the module is considered failed.

The switch can continue to run when the fan module is removed for a maximum of 5 minutes if the temperature thresholds are not exceeded. In this way, you can swap out a fan module without having to bring the system down. The fan module is designed to be removed and replaced while the system is operating without presenting an electrical hazard or damage to the system, provided the replacement is performed promptly. Install the fan module in the front chassis cavity with the status LED at the top. Push the fan module to ensure that the power supply connector mates with the chassis. Tighten the captive installation screws. If the switch is powered on, listen for the fans. You should hear them operating immediately.

No automated shutdown sequence is associated with the removal of the crossbar module fan tray. Shutdown is initiated when temperature thresholds for the crossbar modules are exceeded. Replacement of the crossbar module fan tray should be performed promptly.



Fan Module Installation and Removal

MDS 9513

Fan Module Installation To install a fan module, follow these steps:

Step 1 Hold the fan module so that the Fan Status LED is at the top.

Step 2 Place the fan module in the front chassis cavity so that it rests on the chassis. Lift the fan module up slightly to align the top and bottom chassis guides.

Step 3 Push in the fan module to the chassis until it seats in the backplane and the captive screws make contact with the chassis. The fan module snaps in.

Step 4 If the switch is powered on, listen for the fans. You should hear them operating immediately.

Note If you do not hear the fans, ensure that the fan module is inserted completely in the chassis and the outside surface is flush with the outside surface of the chassis.

Step 5 Verify that the Fan Status LED is green. If the LED is not green, one or more fans are faulty.

Fan Module Removal To remove a fan module, follow these steps:

Step 1 Push the button on the top fan-module latch to release the fan module from the midplane.

Step 2 Repeat this on the bottom fan-module latch.

Step 3 Grasp the fan module with both hands and pull it outward. Rock it gently, if necessary, to unseat the power connector from the backplane.

Step 4 Pull the fan module clear of the chassis.


Fan Assembly Module Removal and System Shutdown To protect the MDS 9000 system from overheating and causing undue damage, an automatic system-shutdown sequence can take place under certain fan-module conditions.

If the fan module is removed from the chassis, a fan-removal sequence is started for the following 3 minutes.

If the fan module is still not reinstalled within that 3-minute time frame, a system-shutdown sequence is started for a period of 2 minutes. At the end of this 2-minute time frame, the system is shut down.

If the fan module is reinstalled at any time in the total 5-minute sequence, the shutdown sequences are stopped, and the system remains on.



Crossbar Module Fan Tray Installation and Removal

Crossbar Module Installation To install a crossbar module fan tray, follow these steps:

Step 1 Orient the crossbar module fan tray in the chassis by positioning the module in the slot, and then sliding the module carefully into the slot until the fan tray is completely inserted in the chassis.

Step 2 Tighten the two captive screws on the crossbar module fan tray to 8 in-lb screws.

Crossbar Module Removal To remove a crossbar module fan tray, follow these steps :

Step 1 Loosen the two captive screws on the fan tray.

Step 2 Hold the two captive screws and pull the fan tray out of the chassis with both hands.

Step 3 Take one hand and hold the face of the fan tray while supporting it with the other hand.

Step 4 Pull the fan module clear of the chassis.


Supervisor and Line Card Modules Installation This topic describes the functions, interfaces, and installation requirements of MDS 9000 Family supervisor modules.


Supervisor Module and Line Card Installation

1. Slot guides2, 3. EMI gaskets4. Ejector level (fully extended)

You must install at least one supervisor module before installing any switching modules.

In a Cisco MDS 9513 Director, slots 7 and 8 are reserved for the Supervisor-2 modules. In the Cisco MDS 9506 and 9509 Directors, slots 5 and 6 are reserved for the supervisor modules. A supervisor module should be installed before installing any switching modules.

Supervisor Module Installation To install a supervisor module, follow these steps:

Step 1 Before installing any modules in the chassis, it is recommended that you install the chassis in the rack.

Step 2 Verify that there is enough clearance to accommodate any cables or interface equipment that you want to connect to the module.

Step 3 Verify that the captive screws are tightened to 8 in-lb on all modules already installed in the chassis. This ensures that the EMI gaskets are fully compressed and maximizes the opening space for the module being installed.

Step 4 If a filler panel is installed, remove the two Phillips pan-head screws from the filler panel and remove the panel.

Step 5 Open completely both ejector levers on the new or replacement module.


Line Card Installation To install a line card module, follow these steps:

Step 1 Slide the module carefully into the slot until the EMI gasket along the top edge of the module contacts the module in the slot above it and both ejector levers close to approximately 45 degrees with respect to the front of the module.

Step 2 Grasp the two ejector levers with the thumb and forefinger of each hand, and then press down to create a small 0.040-inch (1-mm) gap between the module's EMI gasket and the module above it.

Step 3 While pressing down, simultaneously close the left and right ejector levers to completely seat the supervisor module or switching module in the backplane connector. The ejector levers are completely closed when they are flush with the front of the module.

Step 4 Tighten the two captive screws on the supervisor module or switching module to 8 in-lb.



MDS 9500 Series Supervisor Module Installation

Supervisor module locations are fixed (slots 5 and 6 for the 9506 and 9509, slots 7 and 8 for the 9513)Ensure supervisor internal CompactFlash disk is seated properly before installingConnect management port to LAN network with standard Category 5 UTP cableConnect VT100 terminal to the console connectorUse adaptor and cable that are provided with the switch

To install an MDS 9500 Series supervisor module, follow these steps:

Note Supervisor module locations are fixed (slots 5 and 6).

Note Be sure the CompactFlash disk is seated properly on the supervisor module before installing in the chassis.

Step 1 Connect the management port to the LAN using a Category 5 unshielded twisted-pair (UTP) cable.

Step 2 Connect the supplied RJ-45 to DB-9 female adaptor to the computer serial port.

Note It is recommend to use the adaptor and cable provided with the switch.

Step 3 Then connect the console cable (a rollover RJ-45 to RJ-45 cable) to the console port and to the RJ-45 to DB-9 adapter at the computer serial port. Configure the terminal emulator program to match the following default port characteristics:

9600 baud

8 data bits

1 stop bit

No parity

Note See the Hardware Installation Guide for all cable pinouts.



Supervisor Module Management

Migrating from Supervisor-1 modules to Supervisor-2 modules in the Cisco MDS 9500 Series Directors:

Supervisor-1 port indexing is based on the chassis slot number, whereas Supervisor-2 does not share this requirementDifferences between how Supervisor-1 and Supervisor-2 treat port indexes creates the following upgrade qualifiers: – Supervisor-1 and Supervisor-2 modules can not be used in the

same switch (9506 or 9509) except for migration purposes– The migration process is disruptive– Supervisor-2 to Supervisor-1 migration is not supported

Supervisor-2 is required for the 9513 chassisSupervisor-1 is not supported and is hardware blocked

Generation-1 Supervisor Module Management Supervisor-1 and Supervisor-2 treat port indexing differently. Supervisor-1 creates port indexes based on the chassis slot number. Supervisor-2 does not share this behavior, that is, any available port index in a range can be used regardless of the chassis slot number.

Due to the difference in the way they handle port indexing, the Supervisor-1 module and Supervisor-2 module can not be used in the same switch, except for migration purposes. Both the active and standby supervisor modules must be of the same type, either Supervisor-1 or Supervisor-2 modules. For Cisco MDS 9513 Directors, both supervisor modules must be Supervisor-2 modules.

Supervisor-2 Module Migration Follow these guidelines when migrating from Supervisor-1 modules to Supervisor-2 modules:

Ensure that the configured domain ID is the same as the current domain ID for every VSAN on the switch.

Save the configuration with the copy run start command.

Verify that the switch is running SAN-OS 3.0(1), or later.

Determine which Supervisor-1 module is standby with the show module command.

Take the standby Supervisor-1 module out of service with the out-of-service module command.

Remove the standby Supervisor-1 module, and install the Supervisor-2 module in the chassis.

Establish a console session on the standby Supervisor-2 module.

Verify that the standby Supervisor-2 module is in the warm standby state using the show system redundancy status command.


If necessary, copy the contents of the SSM NVRAM to the standby Supervisor-2 module.

Initiate a switchover on the active Supervisor-1 module to power it down and cause the standby Supervisor-2 module to become the active supervisor module with the system switchover command.

Install the other Supervisor-2 module in the chassis.

Run the install all command to update the image versions and boot variables.



MDS 9500 Series Supervisor Module Interfaces

1. Status LEDs (green, orange, red)2. Reset button3. Console port4. 10/100 Ethernet management port5. COM1 serial port6. CompactFlash LED7. CompactFlash eject button9. CF1 slot

COM1 modem cable adaptor (modem adaptor not shown)

This supervisor module is installed in the MDS 9500 Series chassis and has the following interfaces:

Status LEDs: Status, System, Active/Standby, and Power Management

Module reset button: Used for a warm start.

Console port: RS-232 (RJ-45) for local command line interface (CLI) management.

10/100 Ethernet interface: Out-of-band (OOB) management access with integrated link and activity LEDs.

COM1 serial port: DB-9 interface is an RS-232 port that you can use to connect to an external serial communication device such as a modem.

CompactFlash LED: This LED is lit when a CompactFlash (CF) card is installed into slot 0.

CompactFlash eject button: Push to eject a CompactFlash card.

CF1: Slot you can use for a CompactFlash card.

The Status LED states are:

Green: OK

Orange: Initializing or over temperature

The System LED states are:

Green: System OK

Orange: Environmental error, incompatible power supply, or redundant clock failure

Red: Major temperature threshold has been exceeded

The Active LED states are:

Green: Active

Orange: Standby


The Power/Mangement LED states are:

Green: Good power, that is, sufficient power for all modules

Orange: Not enough power, that is, insufficient power for all modules

Connect the modem to the COM1 serial port with the adaptors and cables provided with the accessory kit as follows:

Step 1 Connect the DB-9 serial adapter to the COM1 port.

Step 2 Connect the RJ-45 to DB-25 modem adaptor to the modem.

Step 3 Connect the adapters using the RJ-45 to RJ-45 rollover cable (or equivalent crossover cable).



MDS 9216 Series Supervisor Module

Supervisor and interface module locations are fixed and not removableConnect management port to LAN network with standard Category 5 unshielded twisted-pair (UTP) cableConnect VT100 terminal to the console connector

Fixed interface module

Connect to Ethernet LAN

Connect to terminal

The MDS 9216 supervisor module has 2 slots:

Slot 1: This slot is reserved for the supervisor module with its integrated 16-port switching module.

Slot 2: This slot can contain an optional 16- or 32-port switching module or a services module such as an 8-port IP Storage Services (IPS) module.



Verify Module Status in Device Manager

Reset any module

Device Manager Module Status Verification To verify the status of all modules using Device Manager, click the Physical menu item, and then select Modules.

Using the display, you can reset any module by selecting the reset box and clicking the Apply button.

Note Resetting modules should only be performed when necessary, for example if a software upgrade or downgrade has failed.



Module Shutdown

Use the poweroff module command to power down a moduleVerify status with the show module commandRemove module safely without shutting down entire switch

# conf t(config)# poweroff module 1(config)# do show moduleMod Ports Module-Type Model Status--- ----- -------------------------- --------------- ----------1 16 1/2 Gbps FC Module DS-X9016 ok2 4 IP Storage Services Module powered-dn5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby

Caution Even though you can hot-swap MDS 9000 modules, it is recommended that you shut down a module before removal.

To shut down any module, use the poweroff module command in config mode:

<config># poweroff module 2

To verify the status of a module at any time, use the show module command in EXEC mode.

To view information on one module only, you can specify a module slot number.

Example The command show module 1 returns the status information of only the module installed in slot 1.

# show module

Mod Ports Module-Type Model Status

--- ----- -------------------------- --------------- ----------

1 16 1/2 Gbps FC Module DS-X9016 ok

2 4 IP Storage Services Module powered-dn

5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *

6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby




Summary

Installation guidelines include recommendations for evaluating site preparedness, rack hardware, and power requirements. MDS 9000 switches can be installed in a standard telco rack or cabinets with solid panels. The MDS 9513 can not be used with a telco rack. Power supply installation and configuration should be carefully considered to ensure high-availability. MDS 9000 fan modules are hot swappable and provide for easy installation and replacement. Install at least one supervisor module before installing any line card modules, and use the poweroff module command to shut down individual line cards prior to removal.

FC Standar

Documents

Transcript of FC Standar