EXTRAORDINARY EXTENSION Mark S. Detrick BCFP, BCAF, BCNE, CISSP, CCIE Global Solutions Architect...

EXTRAORDINARY EXTENSION

Mark S. DetrickBCFP, BCAF, BCNE, CISSP, CCIEGlobal Solutions ArchitectPortland, OR+1 (503) 645-2488July 2012

Brocade 7800 & FX - Brocade CONFIDENTIAL

Legal Disclaimer

All or some of the products detailed in this presentation may still be under development and certain specifications, including but not limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in this presentation.

NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD-PARTY RIGHTS WITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN.

Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, and VCS are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners.

March 2012 2


Brocade Extension InnovationsThe Leader in All Extension

1st with 10GE FCIP

1st with FastWrite

1st with FICON Emulation (XRC, Teradata, & Tape)

1st with PP-TCP-QoS

1st with FCIP Trunking

1st with OSTP

1st with FCR

1st with SO-TCP

1st with ARL

1st with IPsecMarch 2012 3

Brocade’s 4 Disciplines of Extension• Open Systems Extension (RDR & Tape)• FICON Extension (Emulation)• Native FC/FICON Extension• FC Routing


Next-Generation Extension Products

March 2012 4

High-touch services, product flexibility, scalability

Solu

tion

pri

ce r

an

ge (

hard

ware

an

d s

oft

ware

)

Global 1000/ Large

Enterprise

Director-ClassStandalone

7800 Base

Small/Medium Business

SMB/Enterprise

Brocade 7800 + PoD license

FX8-24 blade forBrocade DCX 8510

DCX 8510-4

DCX 8510-8

FX baseGE - 10 Gbps

FX 10GE license10GE - 20 Gbps

Brocade Network Advisor

RDR over IP Networks

Business Challenge• Create nonstop IT operations for

business continuance• Application failover and data

access

Solution• Leverage secondary site for

failover• Deploy direct-connected FX8-24

Extension Blades over MLX IP network

• Provide protected data, very small RPO and RTO

March 2012Brocade 7800 & FX - Brocade CONFIDENTIAL 5

Disk Array with RDR

Mainframe

and/or Open

Systems

IP WAN

Mainframe

and/or Open

Systems

Disk Array with RDR

Brocade DCXFX8-24 Extension

Blades

Brocade DCXFX8-24 Extension

Blades

Secondary Data Center

Primary Data Center

FICON

FC

MLXeL2/L3

Switch

MLXeL2/L3

Switch

XRC: Remote host p

ulls the

data over

Arrays replicate data between each other using FC


BROCADE EXTENSIONPRODUCT DETAILS

6September 2010 6March 2012


Brocade Extension Solutions

The fastest, most reliable, and most cost-effective network infrastructure for remote data replication, backup, and migration• FCIP-FW (FCIP FastWrite) accelerates SCSI write processing,

enabling synchronous and asynchronous disk replication over any distance

• OSTP accelerates read and write tape processing, minimizing backup and restore windows

• SO-TCP optimizes TCP window size and flow control, accelerating TCP transport for storage applications

• FICON Accelerator for accelerating disk and tape read and write

operations, maximizing FICON performance

• True interoperability with Brocade switches and management software simplifies deployment and administration

• Advanced Fabric Services address the most challenging requirements

Legacy Features

March 2012 7

The most comprehensive extension solutions

Brocade FX8-24 Extension Blade for

the Brocade DCX 8510

Brocade 7800Channel Extender


Brocade Extension Solutions

Accelerates and optimizes replication, backup, and migration over any distance using advanced Fibre Channel and FCIP networking technology

• FCIP Trunking creates an ISL spanning multiple physical ports for load balancing and network failure resiliency

• ARL dynamically adjusts bandwidth limits to ensure efficient utilization and sharing of available bandwidth

• PP-TCP-QoS prioritizes handling of initiator-target flows within an FCIP tunnel to optimize bandwidth and performance by application

• Advanced Compression architecture provides flexibility to optimizecompression ratios and maximize throughput

• 200 ms Round-Trip Time (RTT) of latency supports distances approaching 17,500 kilometers (nearly 11,000 miles)

• Brocade Network Advisor FCIP management unifies management of Fibre Channel and FCIP network infrastructures

Features Introduced with 7800/FX

March 2012 8

Brocade 7800Channel Extender

Brocade FX8-24 Extension Blade for

the Brocade DCX 8510


Brocade FX8-24 Extension Blade

• Brocade FX8-24 Extension Blade • Twelve 8G FC ports, ten GE interfaces

• Functionality enabled through optional licenses• 10GE Interfaces

• Enables 10GE interfaces and doubles FCIP bandwidth to 20 Gbps

• Supported 10GE interface configurations• Ten GE and one 10GE or• Two 10GE

• Advanced Extension: FCIP Trunking and ARL• Adaptive Networking: Activates FC and FCIP QoS

functionality• Enterprise Bundle: Advanced Performance Monitoring,

Adaptive Networking, Fabric Watch, Brocade Trunking, and Server Application Optimization (SAO)

• Advanced Accelerator for FICON: Enables high-performance FICON tape, XRC, and Teradata over distance

• FICON Management Server: Control Unit Port (CUP) enables host control of switches in mainframe environments

Blade Options

March 2012 9

Twelve 8 Gbps FC or FICON ports

Ten GE FCIP interfaces

Two 10GE FCIP interfaces


Brocade 7800 Extension Switch

• Two models• Brocade 7800 4/2: Four 8G FC ports, two GE interfaces

• Brocade 7800 16/6: Sixteen 8G FC ports, six GE interfaces

• Functionality enabled through optional licenses• Brocade 7800 16/6 Upgrade License: Enables all ports and OSTP

• Advanced Extension: FCIP Trunking and ARL

• Adaptive Networking: FC and FCIP QoS functionality

• Enterprise Bundle: Advanced Performance Monitoring, Adaptive Networking, Fabric Watch, Brocade Trunking, and SAO

• FICON Management Server: CUP enables host control of switches in mainframe environments

• Advanced Accelerator for FICON: Enables high-performance FICON tape and replication over distance

Platform Options

March 2012 10

Brocade 7800 & FX - Brocade CONFIDENTIAL March 2012 11

Brocade 7800 SwitchFront and Back Panels

Brocade 7800 16/616x 8G FC or FICON ports

Brocade 7800 16/66 GE FCIP interfaces(2 ports configurable as RJ-45 or SFP)

Brocade 7800 4/22 GE FCIP interfaces (Configurable as RJ-45 or SFP)

Brocade 7800 4/24x 8G FC or FICON ports

Brocade 7800Dual redundant, 150W hot-swappable power supplies with

integrated fans

Brocade 7800 Management access

ports(RJ-45, GE, and USB)



March 2012 13



March 2012 14



March 2012 15

Brocade 7800 & FX - Brocade CONFIDENTIAL 16

In-band Management

• Management communications via the FCIP interfaces

• Ethernet based

• Facilitates CLI, NMS and BNA management platforms

Management via FCIP Interfaces

March 2012

BNA server or NMS Site

Brocade7800

Brocade7800

Storage Array

Storage Array

IP CloudFCIP & Mgmt

FCIP & Mgmt

Mgmt

FCFC


• Fabric OS v7.0.1• FC and FICON features based on Brocade Condor2 ASIC

• Well-known

• Proven reliability

• Hugely successful

• Supported by all OEMs

• True interoperability with FOS and M-EOS• Fully supported by BNA

• Configuration

• Current and historical monitoring

• Seamless integration with other Brocade products

• Capabilities and manageability of FOS-based products increasing with every release

Brocade Fabric OS

March 2012 17


Storage Optimized TCP

A suite of TCP enhancements

1. Optimized Fast Recovery

2. Quick Start

3. Congestion Avoidance – Disable

4. Optimized Retransmit Timer

SO-TCP makes different assumptions about the network

• Assumptions appropriate to storage applications

• Results in increased overall throughput


FX ArchitectureFront Facing

12x 8G FC Ports

Backplane Facing8x 8G FC Ports

FCIP Complex 010x 4G FC VE Ports

FCIP Complex 110x 4G FC VE Ports

BlasterFPGA

BlasterFPGA

CaviumProc

CaviumProc

L2Switch

10x GE

10

GE-0

10

GE-1

Condor2

FCIP

Com

ple

x 1

FCIP

Com

ple

x 0


7800 ArchitectureFront Facing

16x 8G FC Ports

5x 4G FC VE Ports

BlasterFPGA

CaviumProc

L2Switch

6x GE

GoldenEye2

FCIP

Com

ple

x

FC

PCI

Ethernet


FC ROUTINGA Brocade Innovation

24March 2012


IR with 7800/FX8-24Best Practice Routing: Edge-BB-Edge

7800 7800

IP network

Brocade MLXe Brocade MLXeEdge Fabric

Edge Fabric

Backbone

Edge

Edge

EX_Ports EX_PortsIntegrated Routing License

Required


Fault IsolationFCIP + FCR provides fault isolation between fabrics

March 2012 26

• No routing—FICON cannot be routed

• Open Systems when not directly attached to array, going through fabric

N_Po

rt

F_Po

rt

VE_Po

rt

VE_Po

rt

F_Po

rt

N_Po

rtFCIP Tunnel

One merged fabricNo routing, no edge

fabrics

Isolated Edge Fabric

Isolated Edge Fabric

Backbone

IP Cloud

DCX w/ FX8-24

VE_Po

rt

VE_Po

rtFCIP Tunnel

E_Po

rt

EX

_Port

E_Po

rt

EX

_Port

Edge Fabric

Edge Fabric

7800 7800

IP Cloud

DCX w/ FX8-24

MLXeMLXe


FCIP TUNNEL ENHANCEMENTS

28March 2012


Designed to Maintain High Utilization

• Improved Protocol Efficiency• Batching of sequences

• 7800 and FX8-24 have FastWrite, FICON Emulation and SO-TCP

• Improved Buffering• Optimized data flow management ingress and egress

• 7800 and FX8-24 have huge buffers and new buffer management algorithms

• Enhanced Compression System• Capacity of compression engine to maintain data flow

• Maximize MTU of outgoing datagrams

• 7800 and FX8-24 compress at the most efficient stage and batch multiple FC frames per TCP segment

• TCP Windows• Storage-optimized TCP (SO-TCP)

• Proper TCP windowing (RFC 1323) and control of outstanding segments

• MTU Utilization• Fills datagrams to MTU, sending fewer larger datagrams for greater efficiency

• MTU optimization more efficiently utilizes BW with less overhead

• DF bit is set to eliminate fragmentation in IP network

Many factors determine FCIP utilization of available bandwidth

March 2012 29


Enhanced Compression System

• Developed by Brocade• Technology leveraged from Brocade Encryption Switch

(BES)

• Mode 1: Brocade-optimized LZ (Lempel-Ziv) algorithm• Typical 2:1 compression• High-speed 40 Gbps FC comp (per FCIP complex)• Very low latency added at about ≈ 10 µs• Can be used with synchronous applications

• Mode 2: Dynamic Huffman-Encoded LZ (Lempel-Ziv) algorithm• Typical 2.5:1 compression ratio• Maximum ingress rate is 8 Gbps

• Mode 3: Deflate (GZIP) algorithm• Typical 4:1 compression ratio• Maximum ingress rate is 2.5 GbpsMarch 2012 30

Brocade 7800 vs. Cisco MDS 9222i


FCIP performance with comparable settings and compressionMeasurements in MB/sBrocade FOS 7.0.1 and Cisco NX 5.2.1KGen compressible data generator (20%)2x 8G FC Ingress, 1x GE Egress, IPsec enabled

0 ms 10 ms

20 ms40 ms

60 ms120 ms

200 ms250 ms

300 ms

0

50

100

150

200

250

300

350

400

9222i

7800

9222i

7800



FCIP performance with comparable settings and compressionMeasurements in MB/sBrocade FOS 7.0.1 and Cisco NX 5.2.1KGen compressible data generator (50%)2x 8G FC Ingress, 1x GE Egress, IPsec enabled

0 ms 10 ms

20 ms40 ms

60 ms120 ms

200 ms250 ms

300 ms

0

50

100

150

200

250

9222i

7800

9222i

7800



FCIP performance with comparable settings and compressionMeasurements in MB/sBrocade FOS 7.0.1 and Cisco NX 5.2.1KGen compressible data generator (All 0)2x 8G FC Ingress, 1x GE Egress, IPsec enabled

0 ms 10 ms

20 ms40 ms

60 ms120 ms

200 ms250 ms

300 ms

0

50

100

150

200

250

300

350

400

9222i

7800

9222i

7800



FCIP performance with comparable settings and compressionMeasurements in MB/sBrocade FOS 7.0.1 and Cisco NX 5.2.1KGen compressible data generator (Random)2x 8G FC Ingress, 1x GE Egress, IPsec enabled

0 ms 10 ms

20 ms40 ms

60 ms120 ms

200 ms250 ms

300 ms

0

20

40

60

80

100

120

9222i

7800

9222i

7800


FCIP Encapsulation

March 2012 35

FC FC

End-to-End Devices Negotiate FC Frame Size (2048 Bytes for FX8-24 or 7800)

FC frames entering extension

FC Frame Encapsulation

RFC 3643 (Common Encapsulation)

Protocol Number 1 Byte

Version1 Byte

1’s Compliment of Protocol Number

1 Byte

1’s Compliment of Version1 Byte

Encapsulating Protocol Specific 2 Words

Flags6 Bits

Frame Length10 Bits

1’s Compliment of Flags6 Bits

1’s Compliment of Frame Length10 Bits

Time Stamp (Seconds) 1 Word

Time Stamp (Seconds Fraction) 1 Word

CRC 1 Word

The header below is prepended to batches of up to 8 FC frames creating a single FCIP Frame.

This is then sent to TCP for processing.

2080 Bytes

28-Byte

Header

IP MTU 1500 bytes

7800 or FX

7800 or FX

A Batch


FCIP Batches

• Up to 14 FC frames in a Open Systems batch, 4 for FICON

• Compression occurs first upon FC ingress

• All frames are from the same exchange

• Only data FC frames are batched

• Processed as a single entity

• End of Sequence bit in FC frame causes immediate send

March 2012

36

FCIP Head

er

FC Frame 8

FC Frame 6

FC Frame 4

FC Frame 2

FCIP Batch

FCIP Heade

r

FC Frame 16EoS bit set

FC Frame 15

FC Frame 7

FC Frame 5

FC Frame 3

FC Frame 1

FC Frame 14

FC Frame 12

FC Frame 10

FC Frame 13

FC Frame 11

FC Frame 9


FC to FCIP Method without Jumbo Frames

FCIP Frame 12108 bytes




TCP Segment656 bytes

TCP Header

28 bytes


TCP Header

28 bytes



TCP Header

28 bytes

IP Compression

IP Payload684 bytes

IP Header

20 bytes

IP Payload1480 bytes

IP Header

20 bytes


IP Header

20 bytes

IP Payload684 bytes

IP Header

20 bytes

IP Packet 1MTU 1500

bytes

FCIP frames to be sent

to EthernetIP

Payload352

bytes

IP Header

24 bytes

IP Payload750 bytes

IP Header

24 bytes

IP Payload750 bytes

IP Header

24 bytes

IP Payload

352 bytes

IP Header

24 bytes

IP Packet 1774 bytes


IP Packet 4

376 bytes


Assume 2:1

compression

Brocade 7500/FR4-18i Method


TCP Heade

r28

bytes






FC to FCIP Method with Jumbo Frames






TCP Header

28 bytes


TCP Header

28 bytes

TCP Header

28 bytes

IP Compression

IP Header

20 bytes

IP Header

20 bytes


IP Header

20 bytes


FCIP frames to be sent

to EthernetIP

Header24

bytes


IP Header

24 bytes


IP Header

24 bytes




Assume 2:1

compression

Brocade 7500/FR4-18i Method


FCIP to Ethernet

March 2012 39

Brocade 7800/FX8-24 MethodSeq 1, FC

Frame 52084 Bytes

Seq 1, FC Frame 6

2084 Bytes

Seq 1, FC Frame 7

2084 Bytes

Seq 1, FC Frame 8

2084 Bytes

HW FCcomp

≈ 2:1

In this example the MTU is 1500

bytes

FC Data Frames within an exchange to be sent…

Sent to Ethernet

IP Packet 1IP Packet 4

CreateFCIP Batch Frame

Fill MTU for maximum efficiency

Seq 1, FC Frame 1

2084 Bytes

Seq 1, FC Frame 2

2084 Bytes

Seq 1, FC Frame 3

2084 Bytes

Seq 1, FC Frame 4

2084 Bytes

TCP Segment

1460 B

TCP Heade

r20 B

TCP Segment

1460 B

TCP Heade

r20 B

TCP Heade

r20 B

TCP Segment

1460 B

TCP Heade

r20 B

TCP Segment

1460 B

1102

4B

21024

B

3102

4B

41024

B

FCIP Header

28 B

5102

4B

61024

B

7102

4B

81024

B

…

IP Payload

1480 B

IP Heade

r20 B

IP Payload

1480 B

IP Heade

r20 B

IP Heade

r20 B

IP Payload

1480 B

IP Heade

r20 B

IP Payload

1480 B

IP Packet 2IP Packet 3

Note:4 frames/batch with FICON


FCIP Tunnel Enhancements

• An FCIP Tunnel or FCIP Trunk is single logical ISL

• Brocade 7800/FX8-24: an FCIP tunnel can have one or more circuits• A circuit is an FCIP connection between two unique IP addresses

• One or more circuits may be assigned to any GE interface

• Brocade 7500/FR4-18i and 7800/FX8-24 are not FCIP-interoperable

Trunks, Circuits and Interoperability

March 2012 40

7800

WAN

FCIP TrunkVE_Port to VE_Port

GE0,1,2GE0,1,2

FCIP Circuits

FX8-24


VE_Port to VE_Port (Dedicated Ethernet Ports)

March 2012 41

VE_Port 16

FCIP Trunk1

Logical Source IP Interfaces

Logical Destination IP Interfaces

Physical Ethernet Interface

s

VE_Port 16

FCIP Trunk 1


sCircuits Circuits

VE_Port FCIP Trunk IP Interface Circuit Ethernet Interface

Unique IP Address

Unique IP Address

Unique IP Address

Unique IP Address

GE0

GE1

GE2

GE3

GE0

GE1

GE2

GE3

GE4

GE5

Unique IP Address

Unique IP Address

GE4

GE5

WANIP Cloud

• FCIP Trunks (multiple circuits) or Tunnels (single circuit) are logical entities• IP interfaces are logical entities• Circuits and Ethernet interfaces are physical entities


VE_Port to VE_Port (Shared Ethernet Ports)

March 2012 42

VE_Port 16

FCIP Trunk 1

Multiple Source IP Interfaces

Multiple Destination IP Interfaces


s

VE_Port 16

FCIP Trunk 1


sCircuits CircuitsUnique IP

Unique IP

Unique IP

Unique IP

GE0GE0

VE_Port 17

FCIP Trunk 2

VE_Port 17

FCIP Trunk 2

GE1

Unique IP

Unique IP GE1

The WANIP Cloud

VE_Port FCIP Tunnel IP Interface Circuit Ethernet Interface


FCIP Trunking

• Is an FCIP Tunnel with more than 1 circuit• FCIP Trunking provides

• Single logical ISL in routing table as a single link, ULP see single link

• Bandwidth Aggregation of each circuit• Load Balancing per batch across circuits• Failover to remaining circuits• Lossless Link Loss (LLL)

• Data in-flight is not lost when a link goes down, will be retransmitted same as with TCP

• In-Order-Delivery (IOD) Does not require IOD to be enabled• Data in-flight will be delivered in the correct order, even after data lost

in-flight

• Works with both FICON and FC• Supports FastWrite, OSTP, and FICON emulation over

multiple circuits

• FCIP Trunking is a proven technology• Leveraged from widely deployed McDATA/CNT technology

Brocade exclusive feature

March 2012 43


FCIP Trunking Patent pending

March 2012 44

Class-Fhighmedlow

Circuit 0GE0

Class-Fhighmedlow

Circuit 1GE1

Class-Fhighmedlow

Circuit 2GE2

Class-Fhighmedlow

Circuit 3GE3

FCIP trunk 16 (VE_Port 16)

Batches round robin across

member circuits within trunk

Four priority TCP sessions make up each

circuit

Two or more circuits make

up an FCIP Trunk

FCIP Trunks perform load balancing, lossless failover, and provide multiple links for FastWrite, OSTP, and

FICON emulation


FCIP Trunking Concept

Supervisor TCP Session• Encapsulates the circuits’ TCP sessions within a FCIP Trunk• Provides granular load-balancing• Retransmits lost frames due to circuit loss

• For operational links, the TCP sessions within the circuit handle loss

• Ensures FC frames are delivered in order (IOD)• Prevents IFCC on the mainframe

• Supports FastWrite, OSTP, and FICON emulation• FCIP Trunk appears as a single logical ISL

L3 Lossless Link Loss (LLL)

March 2012 45

13

4

X X XCircuit 1 with TCP session

Circuit 2 with TCP session

FCIP Batches

FCIP Batches3?

Supervisor TCP Session Data Received

3

FICON EmulationFastWriteOSTP

FICON EmulationFastWrite

OSTP

2


Protocol Optimization & State Machines

March 2012 46

Fabric can cause inconsistent path relative

to opposite side

Protocol Optimization State

Machines live within each FCIP tunnel

Path 1

Path 2

WRT_CMD

XFER_RDY

2 FCIP tunnels has2 different State

Machines

? ?

1 FCIP Trunk - 1 State Machine with multiple circuits

WRT_CMD

XFER_RDY

State Machine doesn’t exist for this exchange in

progress!Error! SCSI Broken

Protocol Optimization State Machines• FCIP-FastWrite• OSTP• FICON Emulation

State Machine Setup for this

Exchange


Path Definition

March 2012 47

VE_Port 16

FCIP Tunnel

1

Multiple Source IP Interfaces

Multiple Destination IP Interfaces


s

VE_Port 16

FCIP Tunnel

1


sCircuits CircuitsUnique IP Address

Unique IP Address

Unique IP Address

Unique IP Address

GE0

GE1

GE2

GE3

GE0

GE1

GE2

GE3

VE_Port 17

FCIP Tunnel

2

VE_Port 17

FCIP Tunnel

2

GE4

GE5

Unique IP Address

Unique IP Address

GE4

GE5

The WANIP Cloud

VE_Port FCIP Tunnel Circuits (IP Ports) Ethernet Interface

Traffic Isolation Zone or VF LS for VE_Port 16

Traffic Isolation Zone or VF LS for VE_Port 17


Circuit Metrics and Traffic Flow

• Circuits with a different metric• All traffic goes through circuits with

the lowest metric and no traffic goes through circuits with a higher metric

• If all circuits with the lowest metric fail, circuits with a higher metric are used

• All circuits have the same metric• Traffic flows on all circuits

• The remote end of a tunnel reorders frames to maintain in-order delivery

March 2012 48

FCIP Tunne

l

Metric = 0

Metric = 1

FCIP Tunne

l

Metric = 0

Metric = 1

X

FCIP Tunne

l

Metric = 0

Metric = 1

X

X


ADAPTIVE RATE LIMITINGA Brocade Innovation

49March 2012


Rate Limit Problem – OversubscribedHigh Availability Architecture

Channel Extender A1

Channel

Extender A2

Channel Extender B1

Channel Extender B2

Port Rate Limit = OC-12

RDR Application

Port Rate Limit = OC-12 Port Rate Limit = OC-12

Port Rate Limit = OC-12

IP Network OC-12Oversubscribed 2:1

OC-12 = 622 Mbps

Bad Solution – Do not do this, even with SO-TCP

• Aggregate Port Egress Bandwidth = 2 x OC-12 = 1.244 Gbps• Oversubscribed during normal operation• Results in a massive number of dropped frames – TCP flow contrrol

Buffer overflo

wSite A Site B


Rate Limit Problem – UndersubscribedHigh Availability Architecture

Channel Extender A1

Channel

Extender A2

Channel Extender B1

Channel Extender B2

Port Rate Limit ½ OC-12

RDR Application

Port Rate Limit½ OC-12



IP Network OC-12

½ OC-12 = 311 Mbps

Site A Site B

Better Solution – Not Perfect• Aggregate Egress Bandwidth = 2 x ½(OC-12) = 622 Mbps• No oversubscription during normal operation• Less than 622 Mbps at times of failure of equipment or when application is offline

At times of outage undersubscribed


Adaptive Rate Limiting – PerfectHigh Availability Architecture

Channel Extender A1

Channel

Extender A2

Channel Extender B1

Channel Extender B2

ARLMin ½ OC-12Max OC-12

RDR Application

ARLMin ½ OC-12Max OC-12

ARLMin ½ OC-12

Max OC-12

ARLMin ½ OC-12

Max OC-12

IP Network OC-12

½ OC-12 = 311 Mbps

Site A Site B

Best Solution• Aggregate Egress Bandwidth is always 622 Mbps• Not oversubscribed or underutilized• Rate Limiting automatically changes with network conditions

Fully Utilized


Adaptive Rate LimitingHigh Availability Architecture

March 2012 53

Channel Extender A1

Channel

Extender A2

Channel Extender B1

Channel Extender B2

RDR Application

ARLMin ½ OC-12

Max OC-12

ARLMin ½ OC-12

Max OC-12

IP Network OC-12

½ OC-12 = 311 Mbps

Site A Site B

Fully Utilized

Normal Operation• Link BW Number of Interfaces = BW of interface• Not oversubscribed or underutilized• Rate Limiting automatically changes with device or

link failures• Optics, smashed cables, FW upgrades,

configuration errors, defects…

Offline Path Operation• Link BW Number of Remaining Interfaces

= new BW of interface• Not oversubscribed or underutilized• Applies to the entire path from end device

to end device

ARLNot Operating

ARLOperating at OC-12

50%

50%100%

ARLOperating at ½ OC-12

ARLOperating at ½ OC-12


ARLOptimization after a path failure

March 2012 54

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

9000000

0 50000 100000 150000 200000 250000 300000 350000

With one interface online, all available

bandwidth is consumed up to the

maximum rate

When a second interface is enabled, the network experiences a

congestion event and the first interface drops its rate down to the

minimum configured rate. From here it will seek the ceiling again

The second interface starts off by claiming

its minimum configured rate

Offline equipment/link

results in available BW

This interface ramps up to fully utilize the available

WAN BW. It stops seeking a ceiling at the maximum

configured rate


Adaptive Rate Limiting—Varying BW Usage

Depending on the time of day, BW may become available for other applications• Aggregate egress bandwidth is always 622 Mbps

• If an application is not using BW, then it is free to be used by other applications

• Each GbE interface will get at least ¼ OC-12 up to a maximum of full OC-12

Multiple applications in HA architecture

March 2012 55

Channel Extender A1

Channel Extender A2

Channel Extender B1

Channel Extender B2 IP Network OC-

12

¼ OC-12 = 155 Mbps

Site A Site B

Tape Media Server Tape

LibraryRDR and Tape Applications

Blue is TIZ for Tape

ARLMin ¼ OC-12Max OC-12

ARLMin ¼ OC-12Max OC-12

ARLMin ¼ OC-

12Max OC-12

ARLMin ¼ OC-12

Max OC-12


ARLApplication bandwidth optimization

March 2012 56

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

0 50000 100000 150000 200000 250000 300000 350000

During nighttime, this application is

using less bandwidth

Traffic in TIZ for RDR

Traffic in TIZ for Tape

Tape ramps up using the spare bandwidth made

available by reduced RDR

demand


PER PRIORITY TCP QOSA Brocade Innovation

57March 2012


QoS Single TCP Stream Problem

• Does it make any sense to map multiple levels of service to one TCP stream? No!

• If one TCP stream is used, all levels of service are affected during congestion• Undesirable behavior• TCP is very rigid and puts everything back in the same order as it was

transmitted• Not an issue with autonomous TCP streams

• The same TCP stream cannot tolerate different paths and maintain adequate performance

• There is no way to treat each stream independently with a single TCP session• Priority (802.1P)• DSCP in the IP network

• That is, different DSCP values can be mapped to different MPLS labels within the service provider’s network

• Today this problem is solved using multiple tunnels to the same remote site• Difficult to implement• Difficult to manage each tunnel’s bandwidth and behavior• Each tunnel carries a different priority flow• Protocol optimization requires Traffic Isolation Zones

March 2012 58


Per Priority TCP QoS

• Three QoS priorities per FCIP tunnel for data• High (50%) — Medium(30%) — Low(20%)

• Four TCP streams per FCIP tunnel• There is an additional class-F TCP stream with strict queuing• Strict means all class-F data is sent first• Class-F is not optional and not user configurable

• Each stream is autonomous in the IP network• Each has its own TCP stack• Each has its own settings for 802.1P & DSCP

• Egress enforcement into WAN• QoS is enforced only during contention for

bandwidth• Priorities are mapped to VCs and the associated TCP session

automatically

March 2012 59


FC ISL

VirtualChannels

PP-TCP-QoS

FCIP CircuitFCIP Tunnel

TCP Sessions

DCX

QOSH_<zone name>High Priority

QOSL_<zone name>Low Priority

7800

7800

FC ASIC will put traffic

into priority VC

7800/FX will map

VCs into PP-TCP-QoS


FC CS_CTL Priority ClassificationEnd-to-End Granular QoS

March 2012

FC ISL

H/M/LVirtualChannels

FCIP Circuit

FCIPTrunk/Tunnel

F-class/H/M/LTCP Sessions

DCX 7800 7800

7800/FX will map

VCs into PP-TCP-QoS

Same FC portSame pWWNLUN1: CS_CTL = HighLUN2: CS_CTL = Med

QOSL_<zone> = Low

Tape

Array

FC ASIC will put traffic

into priority VC


Per Priority TCP QoS

TCP session for class-F

TCP session for High

TCP session for Medium

TCP session for Low

QoS enforcement at egress• Class-F – strict access to BW• High/Med/Low BW is balanced

according to user configurable weights for each level

Per Priority TCP QoS Sessions

• Incoming Virtual Channels are mapped to corresponding TCP session automatically

• Assigned by zoning for directly attached devices

• If any one TCP session experiences congestion, the other TCP sessions are not necessarily affected

• Each TCP session can be routed and treated independently of the others via autonomous settings for Priority Tagging (802.1P) and/or DSCP

• ARL is applied per tunnel, not to individual TCP sessions

An FCIP tunnel with 4 TCP sessions. It is possible to have

multiple circuits/tunnels per

interface

Local RemoteIP Network such as MPLS


IPsecNo Additional License

63March 2012


IPsec

• There is no license for IPsec

• It is included in the base unit

• It adds only 5 µs of latency

• It operates at line rate

• All it costs is the effort to configure• Uses PSK (Pre-Shared Keys)

March 2012


FICON EMULATIONA Brocade Innovation

65March 2012


Mainframe Business Continuity

• Host generates entire Read Record Set (RRS) channel program from Define Subsystem Operation (DSO):• DSO has all the info needed to

build local emulated command set

• Network-emulated protocol for XRC RRS channel program

• Keeps the IU pacing:• Open at host side when link

latency would normally close it

• Data frames and final CE/DE are queued at channel for SDM

XRC emulation

March 2012 66

System z Mainframe

PrimaryDASD

Brocade DCX w/ FX

PrimaryData Center

RecoveryData Center

XRC System

Data MoverSystem z

Mainframe

Target/Secondary

DASD

Brocade DCX w/ FX

IP Network


FICON Emulation: Tape Write

• Tape Write Pipelining:• Pre-acknowledges write sequences

received from the channel• Write chains sent to remote side for

actual writes to the control unit• Simultaneous outstanding write

command acknowledgments:• Configurable, default is 32

• Tape Write Pipelining mode works only with typical write channel programs:• Mode set• Write• No-op• Channel Path No-op

• Other commands are noncompliant, pipelining gracefully exits emulation sequence

FICON tape acceleration over any distance

March 2012 67

Writes sent async to remote side

IP Network

PrimaryData Center

Virtual Tape

Controller

TapeLibrary

1

1a

Write seqpre-acknow. sent to host and data is continuously sent to network

4a

4b

Write channel programs delivered to CU and CE/DE

LocalCE

EndingStatus DE

2 3

2a 3a

FinalProgramStatus

1b 42b 3b

EndingStatus DE

Brocade DCX w/ FX

System z Mainframe

Brocade DCX w/ FX


FICON Emulation: Tape Read

• Tape Read Pipelining • Enables industry-leading

performance for FICON read operations over extended distances

• Two modes of FICON read emulation

• Read channel program

• Read block operation

• Tape Read Emulation• Discovers the mode of a read

block or a read channel program

• Pre-reads data and sends it to host side for presentation

• Host channel is released and all read responses are presented to the channel when requested

FICON tape acceleration over any distance

March 2012 68

Multiple reads sentto CU

IP Network

PrimaryData Center

Virtual Tape

Controller

TapeLibrary

1

2a

3

2

Reads are performed and “piped” to host side

3b

3a Buffers“piped”tohost

Reads satisfiedlocally

Read program

s are analyzed

for trends in

size of reads

4 5

4b 5b

4a 5a

Read program is known and channel is suspended with retry

Retry

Brocade DCX w/ FX

System z Mainframe

Brocade DCX w/ FX


FICON Extension: Optica Prizm and ESBT

• New interoperability with Optica Prizm and ESBT• Useful for FICON Printers• Has to be a device type qualified by Optica and

Brocade• IBM’s has announced z196 will be last platform for

ESCON• Prizm is FICON to ESCON converter

• FICON is extended and converted at the device end

FICON Accelerator License

March 2012

FICON

FCIP: FICON Emulation

Prizm

ESCON

IP Cloud

Brocade7800 4/2

Local Mainframe

Remote ESCON Printer

Brocade7800 4/2 Optica

Remote ESCON Tape


FICON Extension: Optica Prizm and ESBTFICON Accelerator License

March 2012

FICON

FCIP

Prizm

Bus & Tag

IP Cloud

FICON DCX Director

w/FX

FICON DCX Director

w/FX

Local Mainframe Remote

B&T Printer

FICON

FCIP

Prizm

ESCON

IP Cloud

FICON DCX Director

w/FX

FICON DCX Director

w/FX

Local Mainframe

Remote ESCON Printer

ESBT

Optica Optica

ESCON

Optica

Remote ESCON Tape


FICON Extension: Teradata Emulation

• Teradata Emulation for both read & write

• Acceleration of Teradata FICON protocols

• Greatly reduces droop caused by latency

• Supported on: 7800 4/2 or 16/6, and FX

FICON Accelerator License

March 2012March 2011© 2011 Brocade Communications Systems, Inc. CONFIDENTIAL - Customer NDA Required 71

FICON

FCIP: FICON Emulation

IP Cloud

FICON DCX Director

w/FX

FICON DCX Director

w/FX

Local Mainframe

Remote Teradata


FICON Emulation: Teradata Write

• Teradata Write Pipelining:• Once a Teradata controller has been

discovered, we will automatically enter write pipelining (a process similar to Tape processing). This process optimizes performance of bulk database load operations or restore operations.

• Write chains pre-acknowledged (to a pipeline limit) and are sent to remote side for actual writes to the Teradata control unit

• Simultaneous outstanding write command acknowledgments:• Configurable, default is 32

• Teradata Write Pipelining mode accelerates only the bulk data transfer Teradata sequences (database load).

FICON Teradata Write acceleration over any distance

March 2012 72

Writes sent async to remote side

IP Network

PrimaryData Center

Teradata Controller

1

1a

Write seqpre-acknow. sent to host and data is continuously sent to network

4a

4b

Write channel programs delivered to CU and CE/DE

LocalCE/DE or CE

EndingStatus CE/DE

2 3

2a 3a

FinalProgramStatus – will update the write pipeline

1b 42b 3bEndingStatus CE/DE

Brocade DCX w/ FX

System z Mainframe

Brocade DCX w/ FX


FICON Emulation: Teradata Read

• Teradata Read Pipelining • Optimizes performance of

database backup type operations (bulk data read from the controller)

• Discovers when database bulk read operations are in effect and will pre-read a fixed number of blocks of data and sends it to host side for presentation. Read pipe is configurable with a default of 32 pre-read blocks.

• Host channel is presented the read data as read commands are received and pre-read data is present in the local (channel connected switch). When data is not available channel is presented a command retry status until data is received from the remote switch from the Teradata controller.

FICON Teradata acceleration over any distance

March 2012 73

Read Cmds are issued to device

IP Network

PrimaryData Center

Teradata Controller

1

2a

3

2

Reads are performed and “piped” to host side

3b

3a Buffers“piped”tohost

Reads satisfiedlocally

Teradata controlle

r Read chain is

received. Read

processing starts

4 5

4b 5b

4a 5a

If read data is available, present it, or issue Cmd Retry

Retry

Brocade DCX w/ FX

System z Mainframe

Brocade DCX w/ FX


DESIGN

74March 2012


Best-Practice RDR Network

March 2012 75

Array Array

Fabric 2A

Fabric 2B

Fabric 1A

Fabric 1B

Hosts Hosts

7800/FX 7800/FX

Separate RDR Network

Production Fabrics

Primary Secondary

WAN A

7800/FX 7800/FXWAN B

Dedicated RDR ports

MLX MLX


Infrastructure

March 2012 76

FICON

DASD

GDPS

10

GE FC

IP

GDPS 10

Gb

E FC

IP

10

Gb

E FC

IP

10

Gb

E FC

IP

Access Level

Aggregation Level

Core Switches/Routers

Access Level

Aggregation Level

Core Switches/Routers

FICON

Tape TapeStorage

Storage

Storage

Storage

IP IPIP IP

WAN Infrastructure Types

DWDM

SONET

TDM

MPLS/VPLS

DASD

A B A B


Using VF to Get to Edge-Backbone-Edge

• Both xISL and EX_Ports in the Base Switch are not supported

March 2012 77

Edge ALocal

Backbone FabricFCIP Tunnel

E_Port E_Port

EX EX

VE_Port VE_PortBase LS Base LS

Fab-A LS Fab-A LS

Edge ARemote


Using VF for FICON Production and Extension Isolation

March 2012 78

Extended FICON Fabric

VE_Port VE_Port

Base LS Base LS

Fab-A LS Fab-A LSProduction

Logical Switches

XRC SDM


DWDM 10GE Lambdas vs. FC or GE

March 2012 79

Native 2, 4, 8, and 10G FC are very expensive DWDM

10GE lambdas are relatively inexpensive DWDM

10GE satisfies speed, latency, distance, and cost

requirementsDWDM

GE not fast enough DWDM


FCIP vs. FC at 75 KM

• FX 10GbE FCIP at 75 km has:• Faster response times• More throughput• Capability for longer distances

March 2012 80

FX 10GE FCIP Native 10G FCResponse Time (µs)

Throughput (Gbps)

Response Time (µs)

Throughput (Gbps)

1282 16 1697 10

• Compression mode 1

• FastWrite enabled• 512-byte payload

• Compression n/a• FastWrite n/a• 512-byte payload

Tests performed at 75 KM


Designing an FCIP RDR Network

• Direct attach or fabric attached?• Best practice is direct attached• Scalable solution is fabric attached• Never preclude fabric attached if that is what makes sense

• Why is direct attach a best practice?• Firmware upgrades do not have to coincide with production fabric• Single router per site does not have to compromise separate A and B fabrics• Less risk of disruption to production fabric (even when a separate Brocade 7800 is used per fabric)• Less points of failure in the path• Less chance for configuration error (like zoning issues)• Less complexity• Less risk that a unforeseen bug can bring down more of the fabric than is absolutely necessary• Isolates the RDR network from the production SAN—because you can never be too cautious

Direct or fabric attached?

March 2012 81

Remote Storage

Subsystem

Direct Attached Direct Attached

Fabric Attached Fabric AttachedEdge Fabric Edge Fabric

Local Storage Subsystem


Best Practice: A and B Physical Separation

March 2012 82

Fab A

Fab B

Fab A

Fab BFor each edge fabric, FCR runs the following:• NS• FSPF• SCR/RSCN• Routing• PSS• DIA• Others

FOS is commonNOT PHYSICALLY ISOLATED

No redundant fabrics!

Merging fabrics A and B is not the point here!

Supported by Brocade and OEMs, but is considered high risk and

poor practice

VF is not any better


Typical Four-Brocade 7800 Solution

• Solution:• 2 FCIP Trunks• 1 FCIP Trunk per 7800

• 2 circuits in each trunk• Array sees 2 parallel paths

• ARL used on all 4 circuits• Max is link BW or interface line rate, whichever is smaller• In this example: Min = Max 4

March 2012 83

FC path A

FC path B

FC path A

FC path B

7800 4/2

7800 4/2

IP WAN Link

GE FCIP

GE FCIP

GE FCIP

GE FCIPService Provider

(Possibly carrier Ethernet)

7800 4/2

7800 4/2

MLX Core

MLX Core

MLX Core

MLX Core


High-Availability Design

• Four Brocade 7800/FX

• Two IP Networks

• Most RDR applications load-balance across multiple paths

• Most RDR applications will fail over to the remaining paths

• Supports FastWrite, OSTP, and FICON Emulation

Multiple paths

March 2012 84

FCIP

FCIP

Storage Local

Storage Remote

7800

7800

7800

7800

Fabric 1A

Fabric 1B

Fabric 2A

Fabric 2B

WAN A

WAN B


What Not To Do!

March 2012 85

RA

1R

A2

RA

3R

A4

Arr

ay

A

B

A

B

A

B

RA

1R

A2

RA

3R

A4

Arr

ay

A

B

A

B

A

B

RA

1R

A2

RA

3R

A4

Arr

ay

A

B

A

B

A

B

RA

1R

A2

RA

3R

A4

Arra

yA

B

A

B

A

B

RA

1R

A2

RA

3R

A4

Arra

y

A

B

A

B

A

B

RA

1R

A2

RA

3R

A4

Arra

y

A

B

A

B

A

B

78

00

78

00

78

00

78

00

GE Circuits

GE Circuits

ROUND TRIP LATENCY 40 – 60 milliseconds

Somewhere, NJ TheOtherEnd, TX

NETWORK MONITORING SERVICE

Brocade Exclusive


Brocade Network Monitoring Service (NMS)

Around-the-clock, remote monitoring of data center extension devices and networks

Proactive device polling allows Brocade experts to quickly identify potential problems

Performance statistics and reporting available via secure web portal, including real-time and historical data for trending analysis

• Reduces downtime 54 minutes per event on average

• Faster fault isolation and resolution, improving availability

• More effective capacity planning

• Improves resource utilization

• 7800 and FX blade were designed with NMS in mind

• Perfect Integration

• Have the Tier 1 Support organization monitor and support your RDR network


QUESTIONS?

88March 2012

THANK YOU


Extension Roadmap Update

March 2012


FR4-18i Product Life Cycle Plan

• FR4-18i is part of the EOL plan for all 4G blades• Announcement letter in Feb 2012• Last customer ship in Aug 2012

March 2012


Qualification of Additional SFPs in FOS 7.0.1• For 7800 and FX8-24 FC Ports

• 4G ELWL – 100km (SmartOptics)• 8G CWDM – 70km (SmartOptics) (delayed to post FOS 7.0.1)

• 8 wavelengths from 1470nm-1610nm

• For 7800 and FX8-24 GE Ports• GE CWDM – 80km (BR branded)

• 8 wavelengths from 1470nm-1610nm

• Brocade P/N: 00600-29x (x=1 to 8)

• For FX8-24 10GE Ports• 10GE ER – 40km (BR branded) (delayed to post FOS 7.0.1)

• Brocade P/N: 57-0000085-01

• 10GE CWDM – 70km (SmartOptics) (delayed to post FOS 7.0.1)• 8 wavelengths from 1470nm-1610nm

• FOS 7.0.1 accepts above listed 3rd party (SmartOptics) “boutique” SPFs by the P/N in SFPs’ EEPROM

March 2012


Emulation for Select ESCON and B&T Printers

• Emulation for the following printer models will be supported for extending z/OS host to printer connection• 3203, 3800-1, IPDS 3900, IPDS 4100

• Solution requires Optica Prizm/ESBT for ESCON/Bus & Tag to FICON conversion

• Solution allows customers to upgrade IBM System z hosts and upgrade ESCON to FICON channels while keeping legacy printers to avoid costly equipment and operation support upgrade• The IBM System zEnterprise 196 and 114 are the last system z

servers to support ESCON channels

March 2012

FICON

IBMSystem z

Prizm

Bus&Tag

IP NetworkFICON

ESCON

Printer

Tape

ESCON

Printer

Tape

Brocade 7800 Brocade 7800

ESBT


Non-Brocade Fabric Extension (FOS 7.1)

• Extend non-brocade fabric to non-Brocade or Brocade fabric

• Provide N_Port port group support with N_Port failover

• All 7800 FCIP extension features will be available• FastWrite, OSTP, QoS, FCR, etc…

• Devices to be connected to remote fabric need to be imported to remote 7800 manually• 7800 registers virtual device which has same pWWN and

nWWN as the imported device to connected fabric

March 2012

H1

H2

T1

Non-Brocade or Brocade Fabric 2

Non-Brocade or Brocade Fabric 1

T2

7800 7800FCIP

VSAN_Zone1N-port port group 1

N-port port group 2

vT1

vH2

N-port port group 1

N-port port group 2

vH1

vT2

VSAN_Zone2


Additional Features Planned for FOS 7.1

• VF (eight LS) support on 7800• Support 1 sec granularity in the counters• Support interval option in portshow

fciptunnel/fcipcircuit for real-time monitoring• Real-time counter for FCIP tunnel compression ratio• Cperf support (individual circuit performance)• All counters related traffic stats need to be able to

be reset to zero• Monitor individual TCP session within individual

circuits with BNA• USGv6/IPv6• IPsec over IPv6• Extend bottleneck detection to FCIP ports

(Subject to Change)

March 2012

EXTRAORDINARY EXTENSION Mark S. Detrick BCFP, BCAF, BCNE, CISSP, CCIE Global Solutions Architect...

Documents

Transcript of EXTRAORDINARY EXTENSION Mark S. Detrick BCFP, BCAF, BCNE, CISSP, CCIE Global Solutions Architect...