FC_FCoE Part 1 Flex SysFab CN4093 Deep Dive Part 1 10.23.12 V3_cattail

System Networking Technical EducationOctober 23, 2012

Delivering on the Promise of Smarter ComputingIBM Systems Technology Group 2011 IBM Corporation

Welcome to the Webinar!We will be starting shortly!

Webinar InstructionsQuestions During The Call?Should be submitted via the Go To Meeting Tool

This Presentation Will Be RecordedAll or a portion of the recording will be placed in the System Networking course catalogs of IBMs Education SMART Zone and PartnerWorlds System College when non confidential

Please Remember That Registration Is Required For Each Go To Webinar Event

Any other questionsEmail [email protected]

Todays SessionDATE: October 23, 2012TIME: 10:00 AM ESTEVENT: System Networking Technical Education SessionTITLE: IBM Flex System Fabric CN4093 FC/FCoE Capabilities Deep DiveSPEAKERs: Meenakshi Kaushik and Deepak KumarBios:

Meenakshi Kaushik: PLM (FC/FCoE, Fabric Architecture and Virtualization)

Deepak Kumar: Architect, Software Development, FC/FCoE Content in Todays Session is Confidential Until November Launch

Upcoming Sessions

Please Remember to REGISTER for the Events Just Click on Button in the Calendar Notice To Be Safe Register at least ONE HOUR PRIOR to Event

10.24 IBM System Networking Monthly Roadmap and Marketing Update10.30How to Leverage the STG Tech Support Model

11.05IBM Flex System Fabric EN4093 10Gb Scalable Switch 4Q Update11.13(TBC)11.20Blocked - for Thanksgiving11.27IBM System Networking Monthly Roadmap and Marketing Update11.29Overview of System Networking Environments TRILL (Comparison to other technologies: FabricPath, etc)Do you have a topic you would like covered? Send recommendations to [email protected]*

2012 IBM CorporationSystem Networking

IBM Flex System Fabric CN4093 FC/FCoE Capabilities Deep DiveMeenakshi Kaushik


AgendaCN4093 FC/FCoE feature overviewData I/O Convergence overviewFCoE Protocol Building BlocksFIP and FCoEFull Fabric FC/FCoE SwitchNPV Gateway CN4093 FC/FCoE Features and Scaling LimitsCN4093 Fabric Architecture


AgendaCN4093 FC/FCoE Configuration NPV Gateway Configuration CLIDisplay CLIDebugging and Trouble ShootingLogin Redistribution (Load balancing)Full Fabric FC/FCoE Configuration CLIDisplay CLIDebugging and Trouble ShootingGhost FC PortsDesign Tips

IBM Confidential

IBM Confidential


Networking InfrastructureNetworkingPay as you grow scalability Optimized for performance Designed for ConvergenceEnhanced virtualization intelligenceLower TCOSeamless interoperability IBM Flex System Fabric CN4093 10Gb Converged Scalable SwitchOverview

LeadershipOmniPorts allow flexibility 10GbE or 4/8Gb FCProven IBM Networking Operating SystemPerformance - Low latency, up to 1.28Tbps Scalable pay-as-you-grow design VM aware & VM Mobility with VMreadyVirtual Fabric carve up virtual NICs and pipesSeamless interoperability with other vendors Warranty is 1 year or will match the chassis warranty (Includes software upgrades)1GbE Mgmt12 x 10GbE OmniPorts (6 in base + 6 #2)#1 = 2x40GbE2X10Gb SFP+

Total Ports10Gb to ServerSFP+ PortsOmniPortsQSFP+ PortsBase System (00D5823)14260Upgrade #1 (00D5845)28262Upgrade #2 (00D5847)282120Upgrade #1 & #2422122


IBM Flex System Fabric CN4093 Converged Scalable Switch Industry leadership - price/performance, scalable 10/40Gb Ethernet switchesWhere to winTarget Customers: Clients wanting native FC connectivity from the chassisClients looking to implement convergence to reduce cost and complexityClients looking wanting features that help improve performance & available, energy efficiency and more automated virtualizationCompetes Against: HP FlexFabric Dell Force10 MXL Cisco UCS Fabric Extender and Fabric Interconnect Base Configuration 14 x 10Gb node port, 2 x SFP+ and 6 OmniPortsPart Number: 00D5823Switch upgrade 1:28 x 10Gb node ports, 2 x SFP+, 6 OmniPorts & 2 x 40Gb uplinksPart Number: 00D5845Switch upgrade 2: 28 x 10Gb node ports, 2 x SFP+, and 12 OmniPortsPart Number: 00D5847Switch with both upgrades: 42 x 10Gb node ports, 2 x SFP+, 12 OmniPorts & 2x40Gb Uplinks

* 6-port adapter not available at this time

Key FeaturesCustomer BenefitsConvergence Ability to reduce adapters and switches Full Fabric FC/FCoE and NPV Gateway can co-exist at the same timeFull Fabric FC/FCoE including end-to-end FCoE (target to the initiator). Ability to leverage integrated storage node. All done (storage and Compute) within the Pure Flex chassis.NPV gateway to connect to any SAN switch. Reduces cost while integrating to the existing SAN. Support for connectivity to existing FC SAN or FCoE StorageEasy & FlexibleReduce IT staff cost to deploy and manage, higher availability/reliability.Reduces cable clutter (no cables from server to switch).Flexible and Scalable ports up to 42 node ports (6 per server*), plus up to two SFP+ 10Gb uplinks (1/10GbE), 12 OmniPorts (10GbE, 4/8Gb FC) and 2x40Gb uplinksHigh availability via enterprise class features like failoverSeamless integration with 3rd party switches.PerformanceBetter end users/customer experience.Low Latency under 1 microsecondIndustries first 40Gb Ethernet embedded switchVirtualizationVMready makes the network VM aware for simpler management & VM mobility.Virtual Fabric allows for vNICs and virtual pipes between adapter and switchWorks with IBM DVS 5000V simple virtual & physical switches managementCost of OwnershipBetter and shorter return of investment, savings could be used elsewhere. Acquisition cost is less than separate Ethernet and FC adapters and switches


IBM Capabilities and Competitive advantagesCapabilities:Ability to provide flexibility of ports as either 10Gb or 4/8Gb Fiber ChannelAbility to provide cost effective FCoE clients connectivity into existing FC SAN (NPV Gateway)Ability to provide end to end FCoE (Full Fabric) compute node to integrated storage node

Competitive AdvantagesHigh Availability (Avoid down-time)No reboot of switch needed to convert the OmniPorts from Ethernet to Fibre Channel or Fibre Channel to EthernetNo reboot of switch needed to make it FC switch or NPV-GatewayReduce ComplexityOn a VLAN basis, it can be configured to be an Full Fabric FC/FCoE switch (only 1 VLAN)On a VLAN basis, configure the switch to be a FCoE->FC NPV-Gateway switch (up to 12 VLANs)Simultaneous FC switching and Gateway functionality (on different VLANs) Not a Interop Tested Configuration for Nov 2012


Traditional segmented fabric approach to networkingAdvantagesTraffic isolationIndependent administration ChallengesUnderutilization of resourcesHigher infrastructure costsComplexity of managementSeparate fabrics for local area networking (LAN) and storage area networking (SAN)EthernetFibre ChannelLANSAN


Convergence Options for the EnterpriseGatewayiSCSI Storage ArrayNFS/CIFS NASFC ArraysFCoEFibre ChannelLoss-LessEthernetGateway*This educational material is intended for your use in selling. It is NOT a deliverable for your clients.FCoE Storage Array


Why FCoE convergence is interesting nowData transmission speed - Ethernet is now able to meet or exceed the speeds available with Fibre Channel.Ethernet capabilities that are designed to make it more similar to Fibre Channel such as Data Center Bridging protocols, lower latency and lossless.Improved asset utilization and cost of ownership (procuring, installing, managing and operating ).*This educational material is intended for your use in selling. It is NOT a deliverable for your clients.


Why convergence versus separate LAN and SAN switching?Clients can see over a 25% acquisition cost savings through convergenceClients can see significant operational cost savings 50% fewer switches and less cablesLower power consumptionFewer components to manage Delivers separate LAN & SAN outside the chassis requirement by some clients* NTE price subject to change and confidential until announce

Using x240 nodesSeparate LAN & SANConverged SolutionI/O10Gb = Std. on x86 nodes8Gb FC ($935 x 14 = $13,090)SW upgrade to LOMCNA ($99 x 14 = $1,386)10G FabricEN4093 ($13,199 x 2 = $26,398)CN4093 ($20,899* x 2 = $41,798)FC FabricFC3171 ($9,899 x 2 = $19,798)Included with CN4093Acquisition Costs$59,286$43,184


FCoE Protocol ComponentsFCoE Standards


FCoE Network Layering and FCoE FrameSource: IBM Redbookshttp://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdf

Based on FC ModelNo FC Frame Fragmentation; Jumbo frames


Basic Components of FCoECN4093Full Fabric FC/FCoE ForwarderServices Login Request FLOGIsConsumes a Domain IDProvides FC Function and ServicesZoning, Name Server, RSCN (Registered State Change Notification)Also FSPF (not used in CN4093 Nov. Release)Forwarding based on FC Domain IDsFC/FCoE Encap/Decap (optional)

Source: IBM Redbookshttp://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdf


CN4093 in Full Fabric mode (End-to-End FCoE)LOM or CN4054 (pNIC, vNIC mode)CN4093 Flex System Fabric Converged SwitchKey value:Directly connects to internal storage node via midplane in FCoE modeIntegrated Storage and Compute

Black line is FCOEGreen line is EthernetStorage nodeEthernet NetworkEx: G8264/G8316CN4093 in Full Fabric


CN4093 in Full Fabric mode (End-to-End FCoE)LOM or CN4054 (pNIC, vNIC mode)Ext. FCoEStorageCN4093 Flex System Fabric Converged SwitchKey value:Reduce cost and complexity of external SAN network removes Cisco and Brocade SANIdeal for new storage deploymentsRemoves need for FC cables and network all togetherBlack line is FCOEGreen line is EthernetEthernet NetworkEx G8264/G8316

AdapterIntegrated SwitchStorage TargetOS levelsLOM & CN4054 4-port adapter (BE3) pNIC and vNIC modesCN4093 10Gb SwitchStorwise V7KWin2008, ESX 4/5, RHEL 5/6, SLES 10/11


FCoE Protocol Control and Data Plane

Destination MAC Address

Source MAC Address(IEEE 802.1Q Tag)ET = FCoEVerReservedReservedReservedReservedSOFEncapsulated FC Frame (minimum of 28 Bytes)(including FC-CRC)EOFReserved (Pad Bytes)Ethernet FCS


FCoE is Fibre Channel on Ethernet TransportNot inCN4093CNAFCF


*IBM Internal Use OnlyFCoE is Fibre Channel on Ethernet TransportStill have a one to one direct connection between host ports and switch portsCEE Switch capable of FCoE is known as a Fibre Channel Forwarder (FCF) in the standardThis is the config we wantNo longer a one to one direct connectionHave to discover FCoE switches (FCFs)They arent just at the other end of the link any moreMAC AddressesFCoE HBAFCoE SwitchFCoE HBAFCoE HBAFCoE HBAFC HBAFC HBAFC HBAFC HBAFC SwitchSANSANGoing from hereFCoE HBAFCoE SwitchFCoE HBAFCoE HBAFCoE HBASANEnet SwitchThis is harder!to here is easy!Source: Dan Eisenhauer and Roger Hathorn

IBM Internal Use Only


EnodeFCF (FC-BB-5)VLAN DiscoveryFIP(FCoE Initialization Protocol)Default VLAN; ALL-FCF-MACsFCFDiscoveryFCFDiscoverySolicitations/ Advertisement; FLOGI and FDISCFLOGI and FDISCAcceptVLAN DiscoveryFC CommandFIP and FCoEFC CommandResponseFCoE Protocol


FSB (FIP Snooping Bridge)FIP Snooping BridgeSnoops FIP messages (VLAN Disc, FCF Disc and FLOGI) Secure host-FCF path with dynamic ACLSA=MAC2, DA=FCF_MAC, Type=FCoE_Type, permit; SA = {FCFs}, deny; DA = ALL_FCF_MACs, Type = FIP_Type, permit; DA = {FCFs}, Type = FIP_Type, permit; Type = FIP_Type, deny; Type = FCoE_Type, deny; SApre = FC-MAP, deny; for FPMA fabric Any non-FCoE related ACEs.


Full Fabric FC/FCoE Switch (End-to-End FCoE)Full Fabric FC/FCoE SwitchServices Login Request FLOGIsConsumes a Domain IDProvides FC Function and ServicesZoning, Name Server, RSCN (Registered State Change Notification)Forwarding based on FC Domain IDsFC/FCoE Encap/Decap (optional)


NPV GatewayNPV - N_Port Virtualization (Extend port density of the Full Fabric Switch) Reduces port requirements on expensive SAN switchSAN switch agnostic; To the FC switch, the gateway NP_Ports appear to be native FC N_Ports that are capable of performing NPIVFC/FCoE GatewaySimple FC/FCoE Frame translation 1:1 frame mappingStatelessLow costFC Ports


Enode FC SwitchVLAN DiscoveryFIP(FCoE Initialization Protocol)Default VLAN; ALL-FCF-MACsFCFDiscoveryFCFDiscoverySolicitations/ Advertisement; FLOGI Send FDISC to FCF

Relay FC DISC via FIPVLAN DiscoveryFC CommandFC CommandResponseFCoE ProtocolNPV FLOGI and FDISCFLOGIAccept FDISCAcceptFC FCoE / FCEncap/DecapNPV Gateway


CN4093 in NPV Gateway

LOM or CN4054 (pNIC, vNIC mode)Ext.StorageCN4093 Flex System Fabric Converged SwitchKey value:Acquisition costsReduce number of director class SAN switch portsLeverage existing external SAN infrastructureBlack line is FCOERed line is Fibre ChannelGreen line is EthernetCisco or BrocadeSANEthernet NetworkEx. G8264/G8316

AdapterIntegrated SwitchFCoE Top Of Rack SwitchSAN SwitchStorage TargetOS levelsLOM & CN4054 4-port adapter (BE3) pNIC and vNIC modesCN4093 10Gb SwitchNPIV modeN/ACisco SANBrocade SANStorwise V7K, FC: SVC, DS3K/5K, DS8K, Tape, XIVWin2008, ESX 4/5, RHEL 5/6, SLES 10/11


LOM or CN4054 (pNIC, vNIC mode)CN4093 Flex System Fabric Converged SwitchKey value:More control of the overall solutions and leverage existing storageDirectly connects to internal storage node via midplane in FCoE modeCan connect to external storage via SAN switch also. Black line is FCOERed line is Fibre ChannelGreen line is EthernetStorage nodeEthernet NetworkEx: G8264/G8316Ext.StorageCisco or BrocadeSANCN4093 in NPV Gateway

AdapterIntegrated SwitchFCoE Top Of Rack SwitchSAN SwitchStorage TargetOS levelsLOM & CN4054 4-port adapter (BE3) pNIC and vNIC modesCN4093 10Gb SwitchNPIV modeN/ACisco SANBrocade SANFCoE: Storage V7000 node FC: Storwise V7000, SVC, DS3K/5K, DS8K, Tape, XIVWin2008, ESX 4/5, RHEL 5/6, SLES 10/11


CN4058 AdapterExt.StorageCN4093 Flex System Fabric Converged SwitchKey value:Integrated FCoE on compute nodeDirectly connects to internal storage node via midplane in FCoE modeCan connect to external storage via SAN switch also. Black line is FCOERed line is Fibre ChannelGreen line is EthernetCisco or BrocadeSANStorage nodeEthernet NetworkEx. G8264/G8316CN4093 in NPV Gateway (Power Compute Node)

AdapterIntegrated SwitchToo Of Rack SwitchSAN SwitchStorage TargetOS levelsIBM Flex System CN4058 8-port 10Gb Converged Adapter CN4093 10Gb SwitchNPIV modeNACisco & Brocade SANFCoE: V7000 nodeFC: Storwise V7000, SVC, DS3K/5K, Tape, XIVAIX 5/6/7VIOS 221/222IBM i6/i7


FSB, NPV and Full Fabric FC/FCoE Switch * == Not available in Nov. 2012

FeaturesFSBNPV GatewayFull Fabric FC/FCoE SwitchFCoE End Device Connectivity

(F) (F)

FC Connectivity (NP) and (NP-F)* (F)*

FCoE/FC Gateway (Encap/Decap)*Security(FIP ACL)(FIP ACL)(FIP ACL, Zoning)ZoningS_ID/D_ID Routing


NPV GatewayFC Port Speeds: 4/8 GbBridging to Fibre Channel SANsLogin load distributionEnd-to-end FCoE (initiator to target). FCoE initiator /target can be attached to any port that is configured as ethernetDirect Attachment of FCoE targets. Manageable via ISCLI / BBI ScalingLogins# of FCoE Logins/Port = 175# of FCoE Logins/Switch= 175

Full Fabric FC/FCoE SwitchFC Port Speeds: 4/8 GbFC-BB-5 Compliant Full Fabric FC/FCoEFabric Services: Name Server, Registered State Change Notification (RSCN), Login services, ZoningWWN, FCID or Alias based ZoningLogin load distributionFC classes of service: Class 2 and Class 3 Manageable via ISCLI / BBIScalingZoneMAX ZONESETS : 4MAX ZONES per ZONESET : 64MAX MEMBERS per ZONE : 20MAX ZONES : 200MAX ALIASES : 200MAX MEMBERS : 1000Logins# of FCoE Logins/Port = 175# of FCoE Logins/Switch= 175

FC/FCoE Features and Scaling


FC-BB-5 vs. FC-BB-6 ( Not in Nov. 2012) FC-BB-5FC-BB-6


CN4093 Architecture (Not for Customer ) Login Load DistributionThroughputPerformance


ReferencesT11 FC-BB-5http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdfIBM Redbookhttp://www.redbooks.ibm.com/redbooks/pdfs/sg247986.pdfUnderstanding NPIV and NPV Bloghttp://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/http://www.juniper.net/techpubs/en_US/junos12.2/topics/concept/fibre-channel-fip-functions-understanding.html


BACKUP SLIDES


*IBM Supported Interop SolutionsSystem Storage Interoperation Center (SSIC) http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss Provides End to End tested / supported IBM FCoE and iSCSI solutions

Outlines all testing building blocks Server Models Operating System Adapter (ASIC) Connection Protocol Storage

Supported Solution Requirement All Building Blocks = Supported One missing = Non Supported

Interop Reference Guides Networking Interop Guide BladeCenter Interop Guide

***CNA VLAN configuration decides

***A typical data center infrastructure today consists of several distinct network fabrics, to carry three different traffic types:One fabric for storage I/O (Fibre Channel-based)One network fabric for computing and management (Ethernet-based)One fabric for interprocess communications (InfiniBand)Each fabric is made of the respective network adapters on the host side, the switching devices and the physical interconnects.

Although this traditional approach of segmenting storage and data traffic has certain advantages (such as traffic isolation and independent administration), it nevertheless poses several disadvantages, including higher infrastructure costs, complexity of management and under-utilization of resources. Customers have to invest in separate infrastructures for LAN, SAN and IPC fabrics, including host adapters, cables, switching, routing and other device-specific equipment. From a manageability standpoint, each network fabric is managed as a separate entity, with dedicated personnel assigned to manage each fabric. At any one time, the individual fabrics tend to not be fully utilized because only a percentage of peak bandwidth is used in the respective networks, which can result in poor overall utilization of these resources, and consequently lower ROI.

*To the FC switch, the gateway NP_Ports appear to be native FC N_Ports that are capable of performing NPIV*-Performance - Sequential vs. random access =Input/Output oPerations perSecond (IOPS) that storage provides.*

FC_FCoE Part 1 Flex SysFab CN4093 Deep Dive Part 1 10.23.12 V3_cattail

Documents

Transcript of FC_FCoE Part 1 Flex SysFab CN4093 Deep Dive Part 1 10.23.12 V3_cattail