Deep dive storage networking the path to performance

Storage NetworkingThe Path to Performance

Howard Goldstein, President HGAI

© Copyright 2010 Howard Goldstein Associates, Inc. Visit www.hgai.com for more information

Session Outline

• What’s Important?• Performance Elements, Storage Devices Matter• Bit Rate, Bandwidth, Latency & Throughput• Circuit Switching vs. Packet Switching• Spraying, Striping, Ports & PHYs, Load Sharing• TCP/IP Offload Engines• Virtualization• FC Fabric Gateways & Routers• Virtual Fabrics• FCoE• Summary


Information

Repository

Access

Processing

Sharing Information

Price Performance

ScalabilityReliability

What’s Important?


Performance Elements

Year Storage Network Penalty1992 10 MBPS .1 MBPS 100-11994 20 MBPS 1 MBPS 20-11996 40 MBPS 10 MBPS 4-11998 100 MBPS 100 MBPS 1-1

Wires to Disk Wires to NetworkSCSI 10bT shared

Fast-Wide SCSI FDDI sharedUltra SCSI 100bT switched

Fibre Channel Gigabit Switched

IP / Ethernet networks are fast enough for storage


Performance Myths

• FC at 4 Gbps is four times the Ethernet bit rate at 1 Gbps so FC must be 4x faster than iSCSI

• TCP/IP protocols is much less efficient than FC, so iSCSI must be much slower than FC

• TOEs are required when using iSCSI to reduce server CPU utilization consumed by TCP/IP overhead

• All “Enterprise” applications require high performance


Disk Technology

PlatterSectorTrack Head


Disk Technology

PlatterSector Track HeadActuator Arm


Disk Technology

PlatterSector Track HeadActuator ArmCylinder

SCSI/ATA Performance may depend on• Revolutions per minute, Seek time, • Latency, Data transfer time• Caching mechanisms• Application I/O Types – Random vs. SequentialSSD changes the SCSI/ATA performance game!


Cache Types

• Read Caching– Recently Used Cache: Data hot spots– Read-Ahead Cache: Sequential pre-fetch

• Write Caching– Write-Through Cache: Direct to non-volatile memory.

Optimized for integrity– Write-Back (Write-Behind) Cache: Faster response to

applications. Optimized for performance


Technical Performance Drivers

• Bit Rate• Bandwidth• Latency• Throughput


Bit Rate, Bandwidth, & Throughput


Parallel vs. Serial

Does increasing bit rate increase speed?

A recent student metaphor:


Parallel vs. SerialSerial Data Transfers

Clock RecoveryCircuit

Serial Data InClocked Data

Recovered Clock

Skew causes misalignment at the receiver

All bits are aligned at transmitting device

Parallel Data Transfers

Example of Cross Talk

Electromagnetic Coupling

Example of Signal Skew

Serial protocol:• Eliminates Signal Skew• Eliminates Cross-talk• Simplifies Interconnect• Allows Higher Data Rate• Point to Point

Source: Maxtor


What About Speed?

• Signal Speed for light or voltage is the same!– 299,792,458 Meters per Second

• .03 Meters per Nanosecond (ns)

– 186,282.397 Miles per Second • 1 Foot per Nanosecond (ns)


Which is faster?

• Copper • Fiber


Which is faster?

• Copper

• Fiber

Copper CAT5 approximately with 4.72 ns/m

Fiber approximately 5 ns/m

Media Propagation Delay


The Rise and Fall

• Copper

• Fiber

The Rise Time

The Fall Time

The Bit Time

The Rise Time

The Fall Time

The Bit Time


Bit Rate – Serial Scaling

Does increasingbit rate increasespeed?


Serial Link Bit Rate Scaling

40 Bits = Tword

1 Bit

1 Bit @ 1.5 Gbps = .666ns

Transmit Receive

1 Bit @ 3.0 Gbps = .333 ns

1 Bit40 Bits = Tword

Assume 40 Folds


Change the tortoise and the hare fable

Throughput does not equal Bandwidth

Improve bit rate


Parallel to Serial?

Back to that student metaphor

Shorter Cars! Both Directions!


Serial - Higher bit rates & Even Shorter Cars!


• Multi-Lane, serial striping techniques goes “back to the future”!

Parallel the Serial – Aggregate for Higher bit rates


Semantic “Anal”ism

• High Speed• High Bandwidth


iSCSI Overhead

iSCSIStandard Ethernet

Ethernet with Non-Standard Jumbo Frames

iSCSI/TCP/IP/ Standard Ethernet Ovehead

iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames Standard Ethernet



iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames

iSCSI Data Out or Data in PDU Header Bytes 52 52 52 52

TCP Header Bytes 20 20 20 20

IPv4 Header Bytes 20 20 20 20

Inter Frame Gap Bytes 12 12 12 12 12 12 12 12

Preamble Bytes 7 7 7 7 7 7 7 7

SFD Bytes 1 1 1 1 1 1 1 1

Destination Address Bytes 6 6 6 6 6 6 6 6

Source Address Bytes 6 6 6 6 6 6 6 6

Length/Type Bytes 2 2 2 2 2 2 2 2

Maximum Data/Pad Bytes 1,500 9,000 1,500 9,000 1,500 9,000 1,500 9,000

FCS Bytes 4 4 4 4 4 4 4 4

Ethernet Frame overhead including Preamble and SFD fields Bytes 38 38 38 38 38 38 38 38

Total Overhead Bytes 76 76 168 168 76 76 168 168


iSCSI Performance

Payload Bytes 1,500 9,000 1,500 9,000 1,500 9,000 1,500 9,000

Efficiency % 97.53% 99.16% 89.93% 98.17% 97.53% 99.16% 89.93% 98.17%

Bit Rate gbps 1.25 1.25 1.25 1.25 12.5 12.5 12.5 12.5

Data Rate MBps 121.91 123.95 112.41 122.71 1219.12 1239.53 1124.10 1227.09

Encode/Decode Penalty 0.80 0.80 0.80 0.80 0.80 0.80 0.80 0.80

Effective Data Rate MBps 97.53 99.16 89.93 98.17 975.29 991.63 899.28 981.68

iSCSIStandard Ethernet



iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames Standard Ethernet



iSCSI/ TCP/IP/ Ethernet Ovehead with Non-Standard Ethernet Jumbo Frames


Fibre Channel PerformanceBytes Bytes Bytes Bytes Bytes Bytes

Fibre Channel

Fibre Channel Class 3

SCSI FC DATA Out or IN / Fibre Channel Class 3





SOF Bytes 4 4 4 4 4 4Frame Header Bytes 24 24 24 24 24 24Maximim Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048CRC Bytes 4 4 4 4 4 4EOF Bytes 4 4 4 4 4 4

InterFrame Word Minimum Bytes Bytes 24 24 24 24 24 24

Acknowledgement Frame Overhead 60 60Total Overhead Bytes 60 60 120 60 60 120

Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048

Efficiency % 97.24% 97.15% 91.92% 97.24% 97.15% 91.92%

Bit Rate gbps 1.0625 1.0625 1.0625 2.125 2.125 2.125

Data Rate MBps 103.31 103.23 97.67 206.63 206.45 195.33

Encode/Decode Penalty 0.8 0.8 0.8 0.8 0.8 0.8

Effective Data Rate MBps 82.65 82.58 78.13 165.30 165.16 156.27


Fibre Channel Performance

Fibre Channel

SOF BytesFrame Header BytesMaximim Payload BytesCRC BytesEOF BytesInterFrame Word Minimum Bytes BytesAcknowledgement Frame OverheadTotal Overhead Bytes

Payload Bytes

Efficiency %

Bit Rate gbps

Data Rate MBps

Encode/Decode Penalty

Effective Data Rate MBps

Bytes Bytes Bytes Bytes Bytes Bytes







4 4 4 4 4 424 24 24 24 24 24

2,112 2,048 2,048 2,112 2,048 2,0484 4 4 4 4 44 4 4 4 4 4

24 24 24 24 24 24

60 6060 60 120 60 60 120

2,112 2,048 2,048 2,112 2,048 2,048

97.24% 97.15% 91.92% 97.24% 97.15% 91.92%

4.25 4.25 4.25 8.5 8.5 8.5

413.26 412.90 390.66 826.52 825.81 781.33

0.8 0.8 0.8 0.8 0.8 0.8

330.61 330.32 312.53 661.22 660.65 625.06


Bandwidth does not equal bit rate

Bit Rate Efficiency Effective Data Rate

Gigabit Ethernet Standard MTU 1500

1.25 Gbps 97.5% 97.5 MBps

iSCSI / TCP / IP / Gigabit Ethernet Standard MTU

1500

1.25 Gbps 89.9% 89.9 MBps

iSCSI / TCP / IP / Gigabit Ethernet Jumbo Frames

MTU 9000

1.25 Gbps 98.2% 98.2 MBps

Standard SCSI FCP Fibre Channel Class 3

1.0625 Gbps 97.15% 83 MBps

Gigabit Ethernet Standard MTU 1500

12.5 Gbps 97.5% 975 MBps

iSCSI / TCP / IP / Gigabit Ethernet Standard MTU

1500

12.5 Gbps 89.9% 899 MBps

iSCSI / TCP / IP / Gigabit Ethernet Jumbo Frames

MTU 9000

12.5 Gbps 98.2% 982 MBps

Standard SCSI FCP Fibre Channel Class 3

8.5 Gbps 97.15% 661 MBps

SAN Performance Comparison – Effective Data Rate

Good

8% Better

18% Best

Good

36% Better

49% Best


FC vs. iSCSI Performance

• Payload Efficiency is not the whole story• Latency and Delay play a role too!• Offered Load• Off Load• Ethernet plays a big role for both


Harry Newton’s Telecom Dictionary • Delay

– The wait time between two events, such as the time from when a signal is sent to the time it is received.

– There are all sorts of reasons for delays• Propagation Delay• Queuing Delay• Processing Delay• Rotational Delay• Satellite Delay

• Latency– A fancy term for waiting delay. The time it takes to get

information through a network.– Accumulated delay


Circuit Switch – Packet Switch

• Circuit Switch - Fixed path whether moving or not• Packet Switch - Routing frames or packets on the

current best path as they arrive and leave


Circuit Switching

Circuit Switch Construction Circuit Switch Setting


Circuit Switch

SAS Expanders

SATA Port Multipliers

FC Class 1

POTS (Plain Old Telephone Network)


Frame Switch

Frame Direction


Packet (Frame) Switch

FC Class 2,3

PCI Express

IP Datagram (Router)

VoIP


Packet (Frame) Switch

RC

PCI Express Transaction


Parts of a Physical Transport Network?

1. 2. 3.Media or the “appearance of” media

Ports - 2 or more (transceivers)

Protocol


Port vs. PHY = Ports vs. Docks

Fibre Channel, Ethernet

1 Port – 1 PHY

SAS, Infiniband, PCIe

1 Port – Many PHYs


Spraying - Striping

1 Link - 4 Phy Lanes

Aggregate Bandwidth(PCIe, InfiniBand)


Striping the Transmission Word


0 1 2 30 1 2 3

0 1 2 3

PCIe, InfiniBand does Striping (4x, 8x, 12x, etc.)


Striping the Transmission Word


0 1 2 30 1 2 3

0 1 2 3

PCIe, InfiniBand does Striping (4x, 8x, 12x, etc.)SAS is not Striping – See connection path


Individual Pathways1 per Open Connection

1 Wide Link – 2 Connections

No Aggregate Bandwidth


Wide Ports – Wide Link



Wide Ports – Wide Link


SAS is not Striping – See connection path

1 2 30


SAS 2 Expander – Open, Arbitration, Open Accept

Node Node

Node NodeExpander

ID03

ID04

ID01

ID02

ARB 02, 03

OPEN 02, 03

AIP (Normal)

Frame Operation 1 Primitive

ExpanderARB 02, 03AIP (Normal)

OPEN 02, 03

AIP Status (WoD)

OPEN 02, 03

OPEN Accept

Frame Operation 2


SAS Expander – Move Revisited –Before Mux

Node Disk Node

HBA Node Disk NodeExpander

ID03

ID04

ID01

ID02

6 gbps 3 gbps

3 gbps

6 gbps Frame

3 gbpsFrame

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01

04-01


SAS Expander – Move Revisited –Before Mux

Node Disk Node


ID03

ID04

ID01

ID02

6 gbps 3 gbps

3 gbps

6 gbps Frame

3 gbpsFrame

04-0104-01

04-0104-01 04-01ALIGN ALIGN

04-01


SAS Expander – Move Revisited –After Mux

Node


ID03

ID01

ID02

6 gbps 3 gbps

3 gbps

6 gbps Frame

3 gbpsFrame

04-01

04-01 02-0102-01

04-01

04-01 02-0102-01

04-01

02-0104-0102-01

04-01

02-01

ID04

04-0102-01

04-01

02-0104-0102-01

04-01

02-0104-0102-01

04-01

02-0104-0102-01

04-01

02-01


SAS Expander – Move Revisited –After Mux

Node


ID03

ID01

ID02

6 gbps 3 gbps

3 gbps

6 gbps Frame

3 gbpsFrame

ID04

02-01

04-01

02-0104-01 02-01

04-01

02-01

04-01

04-01 02-01


Fabric Inter-Switch Links - FSPF

Fabric

N-PortE-Ports

N-Port

Node

F-PortF-Port

Node Node


Fabric Inter-Switch Links

N-PortE-Ports

N-Port

Node

F-PortF-Port

Node NodeStandard FSPF Routing


Round Robin


Fabric Routing – Round Robin Load Share• Fabric Routing involves Inter-Switch Links

(ISLs)1. Port-Based2. Device-Based3. Exchange-Based


Load Sharing –Port Based

Fabric

N-Port E-Ports N-Port

Node

F-PortF-Port

Node Node

1

3

2

4

5

Domain 1

Domain 2


Load Sharing – Device Based

Fabric


Node

F-PortF-Port

Node Node

1

3

2

4

5

Domain 1

Domain 2


Load Sharing – Exchange Based

Fabric


Node

F-PortF-Port

Node Node

1

3

2

4

5

Domain 1

Domain 2

Exchanges


Trunking

Fabric

N-PortE-Ports

N-Port

Node

F-PortF-Port

Node Node


SAS: The Evolutionary Tale….

Evolution:

- Replaces Ultra320 - Preserves Legacy

SCSI Software- Renews the SCSI

Roadmap- Continues the

Serial Trend

CPU

Storage

Interface

3G

6G

http://images.google.com/imgres?imgurl=www.integraltech.com/art/photos/disk drive.jpg&imgrefurl=http://www.integraltech.com/downloads.html&h=221&w=219&prev=/images%3Fq%3Ddisk%2Bdrive%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8

http://images.google.com/imgres?imgurl=www.duxcw.com/digest/guides/hd/hd2.ht2.jpg&imgrefurl=http://www.duxcw.com/digest/guides/hd/hd2.htm&h=149&w=219&prev=/images%3Fq%3D5%2Binch%2Bdisk%2Bdrive%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8

http://images.google.com/imgres?imgurl=www.pocketpcmag.com/Jan01/images/IBM-microdrive-coin.jpg&imgrefurl=http://www.pocketpcmag.com/Jan01/storage.stm&h=250&w=262&prev=/images%3Fq%3Dmicrodrive%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8

http://images.google.com/imgres?imgurl=www.maran.com/dictionary/p/pentium/image.gif&imgrefurl=http://www.maran.com/dictionary/p/pentium/&h=236&w=304&prev=/images%3Fq%3Dpentium%2Bcpu%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8

http://pcmuseum.de/images/80386intels.jpg

http://images.google.com/imgres?imgurl=www.xbitlabs.com/editorial/september2000-news/itanium.jpg&imgrefurl=http://www.xbitlabs.com/editorial/september2000-news.html&h=269&w=300&prev=/images%3Fq%3Ditanium%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8


Fibre Channel Industry Association FCIA Roadmap

Base2*

Product NamingThroughput

(MBps)Line Rate

(Gbaud)

T11 Spec Completed

(Year)

Market Availability

(Year)

1GFC 200 1.065 1996 1997

2GFC 400 2.125 2000 2001

4GFC 800 4.25 2003 2005

8GFC 1,600 8.5 2006 2008

16GFC 3200 17 2009 2011

32GFC 6400 34 2012 Market Demand



Base10**

10GFC 2400 10.52 2003 2004

FCoE


The Road to 100G Ethernet

Year Standard Description1999 802.3ab Gig Ethernet TP (Copper)June 2006

802.3an Gig Ethernet (10 Gbps) TP

July 2006 IEEE High Speed Study Group

Gig Ethernet (40 Gbps, 100 Gbps)

2007 IEEE will form 100G Ethernet Task Force

Gig Ethernet (100 Gbps)

2009 / 2010

IEEE 100G Standard Complete

Gig Ethernet (100 Gbps)


Traditional FC SAN

Host Applications

File Manager

Device Driver

SCSI Protocol

Operating SystemI/O Interface

HBA

FC2FC1

FC3

FC4

FC0

Transport

ApplicationServices

Internet

Internet Protocol Suite

Host NIC

NetworkInterfaceSublayer

Ethernet

FC2


Read 8K 6 FC frames (4 - 2K Frames)

Host

HBA

FC2FC1

FC3

FC4

FC0

FC2

Fibre Channel Multi-Frame Sequence Minimizes Server I/O interrupts - 2

Read CommandDataDataDataDataStatus


Traditional NAS

Host Applications

File Manager

Device Driver

SCSI Protocol


HBA

FC2

FC1

FC3

FC4

FC0

Transport

ApplicationServices

Internet


Host NIC


Ethernet

NFS/CIFS

FC2


8K 7 Ethernet Frames (6 - 1.5K Frames)

Host

Host IP Fragmentation - 6 IP Packet Fragments, 6 – 1500 byte MTU Ethernet Frames, Maximizes Server I/O interrupts - 7

NIC


Ethernet

Transport

ApplicationServices

Internet

Get FileDataDataDataData

DataData

NFS / CIFS / iSCSI


iSCSI Full Offload

Host Applications

File Manager

Device Driver

SCSI Protocol


Transport

ApplicationServices

Internet


Host IntelligentNIC/HBA


GigabitEthernet

iSCSI

TCP

IP

iSCSI


Read 8K 8 (6 – 1.5K Ethernet Frames)

Host Offload IP Fragmentation (Like FC Multi-Frame Sequence)Minimizes Server I/O interrupts - 2

Read CommandDataDataData

Status

IntelligentNIC/HBA


GigabitEthernet

TCP

IP

iSCSI

DataData

Data


Not All TOEs Are Created Equal!

Even Between Different ASICs …

Full HW State-Based TOE

Firmware ASIC TOEFirmware TOELittle TOE

TCP/IP Offload Engines (TOE)


The TOE Spectrum

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI“NIC + Driver”

Media Interface

Ethernet

Media Interface

Ethernet

Fast Path TCP/IP

Software iSCSIWith Partial TCP Off-load

TCP/IP

iSCSI

SCSI Port to OS

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCPand iSCSI Off-load

SCSI Port to OS

= SW = Hardware

Hos

tA

dapt

er

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP and Firmware iSCSI

Off-load

SCSI Port to OS

= FW


Game Changers

• PCIe no longer has “Hard I/O Interrupts”– Message Signaled Interrupts (MSI)

• Multi-core processers have cycles to spare– Message to Virtual Machines – Let us have a few cycles for

TCP/IP overhead!

• Parallel the Serial – Leverages physical barriers


Summary

• Achieving high bit rates are important for the data center

• Understand the information flow path• Asset utilization is key• Hope this helps• Thanks for coming

Fibre Channel for the Data CenterFC Virtualization and FCoE

Presented and Developed by Howard Goldstein of Howard Goldstein Associates, Inc.

© Copyright 2009 Howard Goldstein


FC and the Data Center

• Virtualization• N-Port ID Virtualization• Distributed Services• FC Fabric Gateways & Routers• Virtual Fabrics• FCoE



What is Virtualization?

• Virtualization is an Abstraction– Server– Network– Storage

• What is an Abstraction?



Virtualization is Everywhere

• Benefits– Pooling of resources– Rapidly deploy new applications– Increase resource utilization– Over-subscribe resources– Lower acquisition cost and TCO



Where is Virtualization?Servers

Storage

Network

Storage Storage

VirtualMachine

VM File System

Scale Servers & Storage

VirtualMachineVirtual

Machine

VirtualMachineVirtual

Machine VirtualMachine

VM File System VM File System

VirtualMachine

VirtualMachine

VirtualMachine

Storage



What is Storage Virtualization?

• An abstraction of storage that separates – Host view – Storage system implementation

• Makes invisible to host:– physical pathing– device characteristics– physical data location

• Provides Location and Implementation Transparency

• Dynamic– Enables transparent “on the fly” reconfiguration– Allow data location to change transparently to host environment

• There are a lot of different types and approaches and degrees of storage virtualization



Virtualization in the Network (In-Band)

Virtualization

Data center-wide

Provisioning

Host

RAID subsystem

SANHost accessDevice access

Network Device: Appliance

The appliance is in the data path; Plug and play from the host perspective


Virtualization in the Network (Out-of-Band)

Virtualization

Data center-wide

Provisioning

Host

RAID subsystem

“SAN appliance”separate boxIn-host Network

The appliance is not in the data path; Requires agent software on each host; Separates the data from the control path


Virtualization in the Network (In-Band with Switches)

Virtualization

Data center-wide

Provisioning

Host

RAID subsystem

NetworkDevice: Switch

SANHost accessDevice access

The FC switch is in the data path; Plug and play from the host perspective

Fibre Channel Names & Addresses NPIV


Fabric

World Wide (Port & Node) Names

FC-ALHub

Point-to-point

WWPN

WWPN

WWPNWWPN

FCIP gateway

WWPN

WWPN

WWPN

WWNN WWNN

WWNN WWNN

WWNN

WWNN


Fibre Channel Routing

010600 to 0302EF

Domain ID 01

Domain ID 03

Domain ID 02

Hub

010600

0302EFSwitch

Switch

Switch


Standard Fabric Login

Switch Domain

ID 02

Switch

WWPN 1

HBA Name Server

WWNN WWPN Port_ID

Fabric Login Controller

Port_ID Allocate

Area_IDs00 01 02 03 04 05 06 07Fabric Login

WWPN1

Login AcceptWWPN1 020000

Port_ID is tightly couple with switch port

Area_ID often represents switch port

Device_ID represents:

• “00” FL_Port for FCAL

• ALPA attached device for FCAL,

• “00” in most cases for Point-to-Point


NPort_ID Virtualization

Switch Domain

ID 02

Switch

WWPN 1

NPIV Switch

Name Server

WWNN WWPN Port_ID

Fabric Login Controller

Port_ID Pool

Area_IDs00 01 02 03 04 05Fabric Login

WWPN1

Login AcceptWWPN1 020000

Virtual HBAs

FDISCWWPN2

FDISC AcceptWWPN2 020001

FDISCWWPN3


FDISCWWPN4


Port_ID TableWWPN1 020000WWPN2 020001WWPN3 020002WWPN4 020003

WWPN 2 WWPN 3 WWPN 4


N_Port ID Virtualization

• N_Port virtualization enables an N_Port to acquire multiple N_Port IDs (addresses)– Each address has the appearance of a separate N_Port to other

ports– Separate WWN is used for each address– Each address maintains a separate context

• The initial address is acquired using FLOGI– Additional addresses are acquired by FDISC

• An address can be relinquished using LOGO• All virtual N_Ports share the same physical link

– BB_Credit is shared by all virtual N_Ports– Link Reset, Link Initialization or Link Failure affects all virtual

N_Ports on that link

Fibre Channel Gateways, Routers & Virtual Fabrics


Distributed Services


Fabric Expansion – 2 Separate Fabrics


Fabric Expansion – 2 Separate Fabrics

Fabric 1

Fabric 2


Fabric Expansion – Inter Switch Link

ISL


Merged Fabric

Fabric Expansion – Inter Switch Link

ISL


Fabric Initialization

• Fabric initialization includes a series of actions: – Negotiate the link speed, if supported – Determine the switch port operating mode – If an F_Port or FL_Port, wait for node port to initiate

login – If an E_Port, Exchange Link Parameters and Switch

Capabilities with neighbor – Select a principal switch – Request/Assign Domain_IDs – Build routing tables and select paths


IP FC Switch

Fibre Channel over IP (FCIP)

FC

FC

FC

FC

FC Switch

SCSI

SAN ASAN A

Virtual ISL

Merged Fabric


IP

Fibre Channel over IP (FCIP)

FC

FC

FC

FC Switch

FCIP Gateway

Blade

FCIP Gateway

Blade

SCSI

SAN ASAN AFC

Virtual ISLFC Switch

Merged Fabric


IP

Multipoint Fibre Channel over IP (FCIP)

FC

FC

FC

FC Switch

FCIP Gateway

Blade

FCIP Gateway

Blade

SAN ASAN AFC

Virtual ISL FC Switch

FC

FCIP Gateway

Blade

SAN A

FC

FC Sw

itch

Merged Fabric


IP

Multipoint FC-FC Router FCR with FCIP

FC

FC

FC

FC Switch

FCIP Gateway

Blade

FCIP Gateway

Blade

SAN BSAN AFC

Virtual IFL FC Switch

FC

FCIP Gateway

Blade

SAN C

FC

FC Sw

itch

FCR – Connectivity with Isolation

Not Merged Fabric


Fibre Channel Router


Multiple Fabric Address Translation

Page 326: Figure 140


Security, Zoning and Virtual Fabrics

• Zoning limits the ability to discover (soft zoning) or access (hard zoning) other ports– Does not limit access to well-known services– Does not provide authentication of entities

• Virtual fabrics provide isolation between instances of services– Separate Name Server, Fabric Controller, Zoning, FSPF

Database and routing tables• Security provides authorization, authentication and

encryption– Which devices can join a fabric and which ports can connect to

other ports– Means to authenticate the identity of entities– Encryption provides confidentiality


Virtual Fabrics

• Virtual fabrics enable a single physical fabric to appear as multiple virtual fabrics (VSANs, LSANs)

• Each virtual fabric has all the characteristics of a normal fabric. Each has its own:– Address space– FSPF routing database– Zoning– Name Server– RSCN registration and distribution

• Virtual fabrics enable consolidating multiple separate fabrics into a single physical fabric


Example Fabric Physical Configuration


Virtual Fabric 1


Virtual Fabric 2


Shared ISLs and Frame Tagging


Applications and Networks

• Today, each application class has its own interface– Networking: Ethernet– Storage: Fibre Channel (or SAS or SATA)– Clustering: Infiniband

• This results in three different networks– Three different adapters for each system or server– Three different fabrics, cables and switches– Three different skill sets and tools– Three different management facilities


I/O Consolidation in the Data Center

FC HBA IB HCA Ethernet NIC

FC IB Ethernet

Processor Memory

SAN IPC LAN

HBA HCA NIC

Unified Data Center FabricSAN, IPC, LAN Information Flow Requirements




FC IB Ethernet

Processor Memory

Unified Data Center Fabric


I/O Consolidation Benefits

– Adaptor: NIC for Ethernet/IP, HCA for InfiniBand, Converged Network Adaptor (CNA) for FCoE

– Customer Benefit: Fewer NIC’s, HBA’s and cables, lower CapEx, OpEx

CNA

CNA

FC HBA

FC HBA

IB HCA

FC Traffic

FC Traffic

IB Traffic

iSCSIorInfiniBand orFCoEIB HCA IB Traffic

NIC Ethernet Traffic

NIC Ethernet Traffic


The Drive for I/O Consolidation

• Multicore – Multisocket CPUs• Server Virtualization software (Hypervisor)• High demand for I/O bandwidth• Reductions in cables, power and cooling,

therefore reducing OpEx/CapEx• Limited number of interfaces for Blade

Servers• Consolidated Input into Unified Fabric




FC IB Ethernet

Processor Memory



Ethernet FC IB

CNA

CNA

CNA




Converged Network Adapter (CNA)

Processor Memory


Ethernet FC IB

CNA

CNA

CNA



Converged Enhanced Ethernet – 10 GE

Converged Network Adapter (CNA)

Processor Memory


Separate Interfaces and Networks


Ethernet as a “Fat Pipe”

• One barrier to using Ethernet as the basis for a converged network has been the limited bandwidth that Ethernet has historically provided

• As Ethernet bandwidth increases, more traffic can be carried via fewer physical links– With the advent of 10 gigabit Ethernet (10 GBE), the available

bandwidth now offers the potential to consolidate all of the traffic types over the same link


Converged Enhanced Ethernet

• Why not use a single “converged” network?– Fewer adapters, cables and switches– Especially important in the data center or blade servers


Benefits of a Converged Network Approach

• The benefits of a converged network are:– A server now has a single adapter replacing multiple

different types of adapters that were required when a different type of network was used for each class of application traffic.

– The datacenter has a single network Instead of two, or three, different networks. This means a single network to install, manage and maintain.

– The number of cables and connections is dramatically reduced.


Fibre Channel Levels


IP & Fibre Channel in SANs

IP Network FC NetworkIntegrated IP & FC Networks


IP Storage Protocol Options

Ethernet

IP

TCP

iSCSI

Ethernet

IP

TCP

FCP FC-4

Ethernet

IP

TCP

FC 0, 1, 2

FCP FC-4

Standard SCSI Command Set

Operating System

Applications

iSCSI iFCP FCIP


FC & IP Storage Protocol Options

Ethernet Phy

FC Base T

Ethernet Phy

Ethernet MAC

FCoE

FC 2

FCP FC-4

FC FC Base T FCoE

FC 2

FCP FC-4

FC 0FC 1

FC 2

FCP FC3

FCP FC-4

Standard SCSI

FCP FC3

FC 0FC 1

FCP FC3


The Death of Fibre Channel or iSCSI?

iSCSI lives!FC lives!


The Resurrection of Fibre Channel?


Fibre Channel DriversEthernet Drivers

Operating System

Fibre Channel DriversEthernet Drivers

Operating System

PCIe

Fibr

e C

hann

el

Ethe

rnet

10G

bEE

10G

bEE

Link

Fibr

e C

hann

el

PCIe

4GH

BA

4GH

BA

Link

PCIe

Ethe

rnet

10G

bE

10G

bE

Link

FCoE Enabler: Converged Network Adapter

LAN - NIC CNASAN - HBA


Fibre Channel

Host

FC HBA

FC

SCSI

Storage

FC SA

SCSI

FC ServicesLogin, Zone, Name

Server, etc. FCFC

Moving

Routing - Switching


Fibre Channel Over Ethernet (FCoE)

Host

FC HBA

FC

SCSI

Storage

FC SA

SCSI


Server, etc. FCCEERouting - Switching

Moving

MPS FCF


Host

FC HBA

FC

SCSI

Storage

FC SA

SCSI


Server, etc.Ethernet Ethernet

Moving

Routing - Switching

FCoE for Storage?

MPS FCF MPSFCF


Host

FC HBA

FC

SCSI

Storage

FC SA

SCSI


Server, etc.Ethernet Ethernet

Moving

Routing - Switching

FCoE Switches

FCoE


Host

FC HBA

FC

SCSI

Storage

FC SA

SCSIFC Services

Ethernet Ethernet

Moving

Ethernet

Login, Zone, Name Server, etc.

How about a FC software alternative

Why FC services in an expensive FC switch?


Host

iSCSI HBA

iSCSI

SCSI

Storage

iSCSI SA

SCSIiSNS ServicesLogin, Zone, Name

Server, etc.

Ethernet Ethernet

IP Routing (optional)

Moving

Ethernet

Why FC Switch? – Why not Server model


What is FCoE?

• Fibre Channel over Ethernet (FCoE) is the transport of encapsulated Fibre Channel frames over “lossless” Ethernet– Ethernet provides the physical interface

• Lossless Ethernet NICs or Converged Network Adapters (CNAs)• Lossless Ethernet switches and/or Fibre Channel switches make up

the “Fabric”– Fibre Channel provides the transport protocol

• Fibre Channel frame content is delivered in the Ethernet frames– Most environments will be a mixture of FCoE and FC devices.

• FCoE end devices and switches and FC end devices and switches


FCoE Has “Virtual” Ports

• FCoE ports do not meet the strict definitions in the Fibre Channel standards– They do not have a Fibre Channel Physical interface– They emulate the behavior of a native Fibre Channel

port• Because of this, FCoE ports are referred to a

“virtual” ports– Virtual Node Port, or VN_Port– Virtual Fabric Port, or VF_Port– Virtual Expansion Port, or VE_Port


Fibre Channel Roadmap


General FCoE Objectives

• Seamlessly and transparently replace the Fibre Channel physical interface with Ethernet– FCoE devices and Fibre Channel devices can communicate with

one another without awareness of the underlying physical link

• No changes to protocol mappings, information units, initialization, Fibre Channel services, etc.

• Develop a mapping that can be implemented totally in software– It should be possible to implement FCoE functions totally within

a software driver

• Enable high-performance implementations through the use of hardware assists


What’s Different About FCoE?

• There are existing storage mappings to Ethernet– Internet SCSI (iSCSI)– Internet FCP (iFCP)– Fibre Channel over IP (FCIP)

• All are based on the use of TCP/IP– TCP/IP adds complexity and overhead– Argument is that TCP/IP is not required in a local network

• TCP/IP is still required for the WAN or long-haul

• FCoE bypasses TCP/IP for efficiency and simplicity– Assumes a reliable, lossless Ethernet


Protocol “Stack” Comparison


Why Maintain Fibre Channel Content?

• Why not get rid of Fibre Channel altogether and use something else, such as iSCSI?– iSCSI has made inroads into storage– iSCSI is often used where Fibre Channel is not already in use

• Significant install base of Fibre Channel today– Fibre Channel is a proven technology– Customers don’t want to “rip and replace”– Fibre Channel supports protocols other than SCSI

• e.g., FICON, FC-SATA etc.

• Fibre Channel will likely continue to provide the highest performance for the data center


Rationale for FCoE

• FCoE preserves existing FC infrastructure– FCoE can blend seamlessly into an existing FC environment

• No need to “rip and replace”– FCoE is an evolutionary step– New FCoE enabled servers can communicate with existing FC

storage

• FCoE leverages Ethernet technology– Ethernet is ubiquitous – its everywhere– Ethernet is mature and well understood– As Ethernet speeds increase FCoE can seamlessly take

advantage of the increases– Ethernet should provide cost-effective transport with

performance (almost?) equivalent to native Fibre Channel


FCoE Configurations

• Many different FCoE configurations are possible, but all probably boil down to a few basic variations– Point-to-Point Ethernet– Point-to-Point Ethernet to Fibre Channel– FCoE devices directly connected to FCoE Forwarder (FCF)– FCoE devices connected to FCoE Forwarder (FCF) via

intervening Lossless Ethernet switches– Hybrid configurations consisting of:

• FCoE devices and switches• Fibre Channel devices and switches• Standard Ethernet devices and switches


Hybrid Configuration: Example 1

• Most FCoE will be deployed into existing environments– FC hosts, fabric and storage– Ethernet NICs, fabric and Network Attached Storage

• FCoE hosts and switches will co-exist with existing devices– New FCoE enabled devices can access legacy storage– Legacy FC hosts can access new FCoE enabled storage


Today’s Main Issue

• Is Ethernet convergence the right answer?• What happened to the separate networks

argument?• Are we ready politically and operationally

– Who handles the network now?• Economics and existing vendors will help

drive this decision• YES to convergence (over time)

The New Ethernet


Fundamental I/O Consolidation Technologies

• PCI-Express– I/O Bandwidth implements multi-lane bit rate

aggregation, • 2.5 & 5.0 Gbps• 1x, 2x,4x, 8x, 16x, 32x

• Gigabit Ethernet– I/O Bandwidth single-lane bit rate

• Evolved from a shared media (CSMA/CD) to a serial full duplex point-to-point network

• 10 Gbps & 100 Gbps• 1x• SFP, SFP+, SFP+ Cu


What does Ethernet need?

• Losslessness• Why are frames dropped?

– Frame Errors– Collisions (Old Ethernet)– Congestion

• PAUSE, FC Credit & Priority Flow Control (PFC)


Ethernet Flow Control

• Ethernet is connectionless– No concept of a virtual circuit– On bit error Ethernet throws frame away– Errors are infrequent

• Ethernet frame loss can also occur due to buffer unavailability too!– Ethernet “brute force” technique of high bit rate and

high switch memory has limits– TCP will retry, with a Positive Acknowledgement and

Retransmission (PAR) mechanism– But how much TCP is too much TCP?


Priority-Base Flow Control

Receivebuffers

Link-level flow control per priority• 8 priority levels • Traffic paused based on priority level• Protects from transient congestion• Enables Loss-Less Ethernet fabric for latency sensitive applications like FCoE and HPC

Only one priority queue is paused

Transmitqueues

Pause

8 VirtualLanes


Enhanced Transmission Selection

t1 t2 t3 t40

1

2

3

4

5

6

7

8

9

10

EmailHTTPHPCFCoE

For effective fabric consolidation, important traffic like storage and HPC needto be assigned high priorities and guaranteed bandwidths.

• To improve network efficiency 802.1Qaz allows lower priority traffic to use unused bandwidth from high priority queues.

Assigned Bandw idth0

1

2

3

4

5

6

7

8

9

10

Example:Email = best effortHTTP = 1Gb minHPC = 3Gb minFCoE = 5Gb min


Link Layer Congestion Management


Shortest Path Bridging

• An incremental advance to MSTP• Uses link state protocol (IS-IS) to share learned topologies between switches• Enables learning of shortest paths as the fabric changes

Ensures forward and reverse paths are aligned

IEEE – TRILL Internet draft is similar(Transparent Interconnect over Lots of Links )

Deep dive storage networking the path to performance

Education

Transcript of Deep dive storage networking the path to performance