1. Converged Data Center: FCoE, iSCSI and the Future of Storage Networking David L. Black, Distinguished Engineer Office of the CTO Copyright 2011 EMC Corporation. All rights reserved.1

2. Agenda Network Convergence Protocols & Standards Solution Evolution Conclusion andSummary Copyright 2011 EMC Corporation. All rights reserved.2 3. 10Gb Ethernet Converged Data Center Maturation of 10 Gigabit Ethernet Replace multiple 1Gb adapters with fewer 10Gb adapters (start with 2) Single network simplifies mobility for virtualization/cloud deploymentsSingle Wire for Network SAN 10 GbE and StorageLAN 10 Gigabit Ethernet simplifies infrastructure Reduces number of cables and server adapters Lowers capital expenditures and administrative costs Reduces server power and cooling costs Blade servers and server virtualization drive consolidated bandwidthFCoE and iSCSI both leverage this inflection point Copyright 2011 EMC Corporation. All rights reserved. 3 4. Conventional Rack Servers Ethernet Fibre Channel iSCSI SAN Servers connect to LAN, NAS andiSCSI SAN with NICs 1 Gigabit Ethernet Servers connect to FC SAN withHBAs Many environments today are still 1 Gigabit Ethernet 1 Gigabit Ethernet Multiple server adapters, multiplecables, power and cooling costsFibre Ethernet LAN 1 Gigabit Storage is a separate network (includingChannel EthernetHBAsiSCSI) NICs Note: NAS will continue to be part of theFibre Channel SANsolution. Everywhere that you see Ethernet or 10Gb Ethernet in this presentation, NAS can be considered part of the unified storage solution StorageRack-mountedservers Copyright 2011 EMC Corporation. All rights reserved. 4 5. FCoE: Why a New Option for FCCustomers? FC: large and well managed installed base Leverage FC expertise / investment Other convergence options not incremental for existing FC Data Center solution for I/O consolidation Leverage Ethernet infrastructure and skill setFCoE allows an Ethernet-based SAN to be introduced into an FC-based Data Center without breaking existing administrative tools and workflows Copyright 2011 EMC Corporation. All rights reserved.5 6. FCoE Extends FC on a Single Network Server sees storage traffic as FC Network FCEthernet DriverDriverNetworkConvergedFC storageNetwork Adapter SAN sees host as FCLossless EthernetFC networkEthernet FCoEFC Switch Copyright 2011 EMC Corporation. All rights reserved.6 7. Time to Widespread Adoption1980 199020002010 10 Gigabit Ethernet 02 09EthernetStandardWidespread73 8393 iSCSIDefined Standard Widespread0004 08 DefinedWidespread Standard Fibre Channel 859403FCoE DefinedStandard Widespread07 09 ??DefinedStandardWidespread Copyright 2011 EMC Corporation. All rights reserved. 7 8. Further Developments: Whats Next?40/100 GbEthernet1632GFC GFC 40 & 100 Gb Ethernet (IEEE) standards completed in June 2010 16GFC (T11) standard completed in January 2011, 32GFC is next (2012 target) FC throughput doubles, encoding change optimizes analog bandwidth 8GFC: 800 MB/sec max, 8.5 Ghz 16GFC: 1600 MB/sec max, 14.025 Ghz Copyright 2011 EMC Corporation. All rights reserved. 8 9. Agenda Network Convergence Protocols & Standards Solution Evolution Conclusion and Summary Copyright 2011 EMC Corporation. All rights reserved. 9 10. iSCSI Introduction Transport storage (SCSI) over standard Ethernet Reliability through TCP More flexible than FC due to IP routingSCSI Good performanceiSCSI iSCSI has thrived Especially where the server, storage and networkTCPadministrators are the same personIPLink IP Network Copyright 2011 EMC Corporation. All rights reserved.10 11. iSCSI Introduction (continued) Standardized in 2004: IETF RFC 3720 Stable: No major changes since 2004 iSCSI Corrections and Clarifications: IETF RFC 5048 (2007) iSCSI Session: One Initiator and one Target Multiple TCP connections allowed in a session Important iSCSI additions to SCSI Immediate and unsolicited data to avoid round trip Login phase for connection setup Explicit logout for clean teardown Copyright 2011 EMC Corporation. All rights reserved. 11 12. iSCSI Read ExampleInitiator Target SCSI ReadData in PDU CommandTargetData in PDU ReceiveData in PDU DataStatus Command Complete Optimization: Good status can be included with last Data in PDU Copyright 2011 EMC Corporation. All rights reserved.12 13. iSCSI Write ExampleInitiatorTarget Ready toSCSI Write TransmitCommand (R2T)Optimization:Data out PDUImmediate or Receive Data unsolicited dataData out PDUavoids a round tripData out PDU R2TData out PDU Receive DataCommand StatusComplete Copyright 2011 EMC Corporation. All rights reserved.13 14. iSCSI and FCoE Framing iSCSI is SCSI functionality transported using TCP/IP for delivery and routing in a standard Ethernet/IP environmentEthernetDataiSCSI Frame IP TCPiSCSICRCHeader FCoE is FC frames encapsulated in Layer 2 Ethernet frames No TCP, so Lossless Ethernet required FC Frame No IP routing Header FCoE FrameFC Payload Ethernet Header Header CRC FCoE EOFFC FCS Copyright 2011 EMC Corporation. All rights reserved. 14 15. FCoE Frame FormatBit 0 Bit 31 Destination MAC Address 1:1 encapsulation of FC framesSource MAC Address No segmenting of FC frames across IEEE 802.1Q Tagmultiple Ethernet frames FCoE flow control is Ethernet based ET = FCoEVer Reserved Reserved FCoE requires Mini Jumbo Reserved frames Reserved SOF Max FC payload size: 2180 bytes Max FCoE frame size: 2240 bytes Encapsulated FC Frame (Including FC-CRC) FCoE Initialization Protocol (FIP) Discovery: VLAN and FCoE EOFReservedswitches FCS FC login to discovered FCoEswitches Copyright 2011 EMC Corporation. All rights reserved. 15 16. FCoE Initialization ProtocolEthernet is more than a cable Native Fiber Channel link: Fiber has exactly 2 endpoints (simple) Discovery: Who is at the other end? Response to FC login contains answer Liveness: Is the other end still there? Is the optical link lit and synchronized? FCoE virtual link: Ethernet LAN or VLAN has more than 2 endpoints Discovery: Choice of endpoints (FCoE switches) Where should the FC login be sent? Liveness: FCoE virtual link may span multiple Ethernet links What if attached link is ok, but some other link is not? FCoE configuration concern: Do mini jumbo frames (2.5k) work? FIP: FCoE Initialization Protocol Discover other endpoint, create and initialize virtual link with FCoE switch Mini jumbo frame support: Large frame is part of discovery Periodic LKA (Link Keep Alive) messages after initialization Copyright 2011 EMC Corporation. All rights reserved. 16 17. FCoE Switch Discovery FCoE/FCStep 1: FIP SolicitationSwitches Server DCB FC SAN Ethernet Solicitation Select FCoE VLAN first (pre-configured or use FIP) Solicitation is multicast: Server can discover multiple switches Solicitation identifies Server (FC WWN for FCoE CNA) CNA = Converged Network Adapter (FCoE analog of HBA) Switch may choose which servers to respond to (default: respond to all) Copyright 2011 EMC Corporation. All rights reserved.17 18. FCoE Switch Discovery FCoE/FCStep 2: FIP Advertisement Switches Server DCBFC SAN EthernetAdvertisement Priority = 1AdvertisementPriority = 25 Advertisement identifies switch (FC WWN) Multiple switches may respond, advertisement includes priority Server chooses FCoE switch by priority (smallest number wins) Advertisement padded to max FC frame size: Test mini jumbo frame support Copyright 2011 EMC Corporation. All rights reserved. 18 19. FIP Switch DiscoveryFCoE/FCStep 3: FIP-based FC LoginSwitches Server DCBFC SAN EthernetFLOGI Priority =FLOGI1 ACCPriority = 25 FIP encapsulated FC Login Server sends FC Fabric Login (FLOGI) to selected switch Switch responds with FC FLOGI ACC (accept) that contains assigned FCID Subsequent traffic: Standard FC frames (FCoE encapsulated) Copyright 2011 EMC Corporation. All rights reserved. 19 20. FCoE and Ethernet StandardsTwo complementary standards effortsFiber Channel over EthernetData Center Bridging Ethernet(FCoE)(DCB) Developed by International Developed by IEEE Data CenterCommittee for InformationBridging (DCB) Task GroupTechnology Standards (INCITS) DCB Ethernet drops frames asT11 Fiber Channel Interfaces rarely as Fiber ChannelTechnical Committee Technology commonly referred to as Fiber Channel over Ethernet allows Lossless EthernetFiber Channel traffic over Ethernet IEEE standards: final approval FC-BB-5 standard ratified June March 20112009 DCB: Required for FCoE FC-BB-6 in process to expand DCB: Enhancement for iSCSIsolution Companies working on the standard committees Key participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others Copyright 2011 EMC Corporation. All rights reserved.20 21. FC-BB-6 Major features Direct connection of servers to storage PT2PT [point to point]: Single cable VN2VN [VN_Port to VN_Port]: Dedicated Ethernet LAN or VLAN Better support for FC fabric scaling Distribute logical FC fabric switch functionality Enables every DCB Ethernet switch to participate in FCoE For more, see Erik Smiths (EMC E-Lab) presentation: FCoE - Topologies, Protocol, and LimitationsTues 5:00pm and Wed 4:15pm Copyright 2011 EMC Corporation. All rights reserved.21 22. Lossless Ethernet (DCB) IEEE 802.1 Data Center Bridging (DCB) Link enhancements: standardized, initial products available 1. Enhanced Transmission Selection (ETS) 2. Priority Flow Control (PFC) 3. Data Center Bridging Exchange Protocol (DCBX) DCB-enabled LAN: Network portion that must be lossless Generally limited to data center distances per link Can use long-distance optics, but uncommon in practice Enhanced Ethernet provides the Lossless Infrastructure that enables FCoE Copyright 2011 EMC Corporation. All rights reserved.22 23. Enhanced Transmission SelectionDCB part 1: IEEE 802.1QazManagement framework for link bandwidth Priority configuration and bandwidth reservation E.g., HPC & storage traffic: higher priority & reserved bandwidth Bandwidth utilizationOffered Traffic 10 GE Link Realized Traffic Utilization Unused higher priority 3G/sHPC Traffic2G/s 3G/s 3G/s 2G/sbandwidth available3G/sto other traffic 3G/s Storage Traffic 3G/s 3G/s 3G/s 3G/s 3G/s Low latency assured to higher priority traffic 3G/s 4G/s 6G/s3G/sLAN Traffic5G/s 4G/s t1t2t3 t1 t2 t3 Copyright 2011 EMC Corporation. All rights reserved.23 24. PAUSE and Priority Flow ControlDCB part 2: IEEE 802.1Qbb & 802.3bd Classic PAUSE can produce lossless Ethernet behavior Classic 802.3x PAUSE stops all traffic: Rarely implemented New PAUSE: Priority Flow Control (PFC) Pause per priority level No effect on traffic at other priority levels Creates lossless virtual lanes Per-priority link flow control Enabled/disabled by priority Only affect traffic that needs it More than 8-way 802.3x PAUSESwitch A Switch B Copyright 2011 EMC Corporation. All rights reserved.24 25. DCBX Data Center Bridging CapabilityeXchange FCoE/FCDCB part 3: IEEE 802.1Qaz (again)SwitchesServer DCB FC SAN EthernetDCB X Ethernet Link configuration (single link) Extends Link Layer Discovery Protocol (LLDP) Reliably enables lossless behavior (DCB) e.g., exchange Ethernet priority values for FCoE and FIP FCoE virtual links should not be instantiated without DCBX Copyright 2011 EMC Corporation. All rights reserved.25 26. Ethernet Spanning Trees and FCoE Reminder: FCoE is Ethernet only, no IP routing Ethernet (layer 2) is bridged, not routed Spanning Tree Protocol (STP): Prevents (deadly) forwarding loops Elects a Root Switch, disables redundant paths to create a tree Causes problems in large Ethernet networks No network multipathing Inefficient link utilization Root Switch SiSiSiSiSiSi Si Copyright 2011 EMC Corporation. All rights reserved.26 27. TRILL Transparent Interconnection ofLots of Links Layer 2 routing among Ethernet switches In contrast to IP routing at layer 3 IS-IS routing protocol for inter-switch Ethernet traffic Blocks Spanning Tree Protocol TRILL encapsulates Ethernet frames Not used with end systems (NICs) NICs use link teaming/aggregation SiSiAll links active Si SiSiSi Si Copyright 2011 EMC Corporation. All rights reserved.27 28. Ethernet Cabling ChoicesType /ConnectorCable1Gb10Gb 40/100GbCopper Cat6 orMajority ofSome products Not(10GBase-T) /Cat6aexisting on market, but supportedRJ-45 cablingnot for FCoE(e.g., Cat 5e) yet Cat 6 55m Cat 6a 100mOpticalOM2Rare for Most Expect shift(multimode) /(orange) Ethernet backbone to optical w/LC OM3 deployments40/100Gb (aqua) Typical forare opticalFCOM3 100m OM4 OM2 82m (aqua) OM4 125m OM3 300mCopper / SFP Twinax N/ALow powerDifferent+DA (direct short-attach)5-10mdistance distance (Rack option solution)(QSFP) Copyright 2011 EMC Corporation. All rights reserved.28 29. Virtual Machines and Storage Resources Private Storage Resources Accessed directly by VM Device driver in VMs OS Managed as part of VM Not visible to virtualization Privatemanagement (e.g., vCenter) If disk is local: No vMotion Shared Storage Resources Accessed by Hypervisor Device driver in hypervisorShared Managed as part ofvirtualization (e.g., vCenter) Copyright 2011 EMC Corporation. All rights reserved.29 30. Storage Drivers and Server Virtualization vNIC vSCSI vNIC vSCSI virtual switchHypervisorHypervisor driver NICFCNICFCHBAHBALAN traffic iSCSI traffic FC traffic*iSCSI initiator can also be in the VM (PrivateStorage) Copyright 2011 EMC Corporation. All rights reserved.30 31. Storage Drivers and Server Virtualization vNIC vSCSI vNIC vSCSI virtual switchHypervisorHypervisor driver NIC C FC NIC C FC N HBAN HBA AA LAN trafficiSCSI traffic FCoE follows FCpath*iSCSI initiator can also be in the VM (PrivateStorage) Copyright 2011 EMC Corporation. All rights reserved.31 32. Software FCoE and Server VirtualizationSW SW FCoE FCoE vNIC vSCSI vNIC vSCSIVirtual Switches in ESX/ESXi (including Ciscovirtual switchHypervisor HypervisorNexus 1000v) and driver Hyper-V are notLossless (not DCB) Not a problem foriSCSI, NFS orNIC FCNIC FC HBA HBA CIFS in a VirtualMachine FCoE software in VMs would send traffic through the virtual switch to the NICs Copyright 2011 EMC Corporation. All rights reserved. 32 33. Storage Virtual Appliance (SVA):Sharing Private StorageStorage Virtual Appliance Virtual Machine that providesstorage to hypervisors Direct physical storage access e.g., RDM (Raw DeviceMapping) Exports storage to hypervisors iSCSI ESX & ESXi: iSCSI (or NFS) Avoid vMotion for SVAs This SVA example: Simplified Availability: Multiple SVAsmirror or RAID across servers Scale: SVAs provide sharedstorage to vSphere servercluster Copyright 2011 EMC Corporation. All rights reserved. 33 34. Agenda Network Convergence Protocols & Standards Solution Evolution Conclusion and Summary Copyright 2011 EMC Corporation. All rights reserved. 34 35. FCoE and iSCSIFCoE iSCSI EthernetNo FC expertise neededFC expertise / install baseLeverageFC managementEthernet/IP expertiseSupports distanceLayer 2 Ethernet 10 Gigabit Ethernet (Layer 3 IP routing) Use FCIP for distanceLossless EthernetStrong virtualization affinity Copyright 2011 EMC Corporation. All rights reserved.35 36. iSCSI Deployment 10 Gb iSCSI solutions available Traditional Ethernet TCP recovers from dropped packets Lossless Ethernet (DCB) iSCSI: natively routable (IP) Can use VLAN(s) to isolate trafficEthernet iSCSI solutions: smaller scale thanFCiSCSI SAN Single FC director : larger than most iSCSI environments Copyright 2011 EMC Corporation. All rights reserved. 36 37. FCoE Server Phase (1) FCoE with Converged Network Switch at top of rack or end of row Tightly controlled solutionEthernet LAN Server 10 GE adapters may be CNA or NIC Storage still a separate network EthernetFC Converged Network SwitchFCAttach Fiber Channel SAN1 Gb NICs FC10 GbE CNAsHBAsStorageRack Mounted Servers Copyright 2011 EMC Corporation. All rights reserved. 37 38. FCoE Network Phase (2) Converged Network Switches move out ofrack into unified network Maintains existing LAN and SAN management Ethernet LANOverlapping admin domains may compel cultural adjustments Ethernet NetworkEthernet(IP, FCoE)Converged NetworkSwitchFCConverged NetworkSwitch FC AttachFiber Channel SAN10 GbE CNAsStorage RackMountedServers Copyright 2011 EMC Corporation. All rights reserved. 38 39. FCoE Storage Phase (3) Single Ethernet network for IP and storage traffic End-to-End Ethernet with native FCoE FC/FCoE configured and managed as an FC SANEthernet LAN Leverage FC management skills and procedures Converged NetworkEthernet SwitchFCoEFCStorage Fiber Channel & FCoEFC & FCoE SANattach10 GbE CNAsStorage RackMountedServers Copyright 2011 EMC Corporation. All rights reserved. 39 40. Convergence at 10 Gigabit Ethernet Two paths to a Converged Network iSCSI purely Ethernet FCoE enables mix of FC and Ethernet (or allEthernet)Ethernet LAN FC that you have today or buy tomorrow is compatible Choose based on scalability, management, Converged Network and skill setEthernet SwitchFCiSCSI/FCoE Storage Fiber Channel & FCoEattach10 GbE CNAs FC & FCoE SANRack Mounted Servers Copyright 2011 EMC Corporation. All rights reserved.40 41. EMC and Ethernet Best Practices Google FCoE Tech Book (FCoE & Ethernet) Services Design, Implementation, Performance and Security offerings for networks Products Ethernet equipment for creating Converged Network Environments Copyright 2011 EMC Corporation. All rights reserved.41 42. Agenda Network Convergence Protocols & Standards Solution Evolution Conclusion andSummary Copyright 2011 EMC Corporation. All rights reserved.42 43. Summary Converged data center environments can be built using 10Gb Ethernet Achieving a converged network requires consideration of technology, processes/best practices and organizational dynamics 10 Gigabit Ethernet solutions are maturing Active industry participation is creating standards that allowsolutions that can integrate into existing data centers Continued use of FC and adoption of FCoE can be flexible due toshared management FCoE and iSCSI will follow Ethernet roadmap to 40 and 100Gigabits/sec Copyright 2011 EMC Corporation. All rights reserved.43 44. Related Session and Resources FCoE - Topologies, Protocol, and Limitations Tuesday 5:00p & Wednesday 4:15p Birds of a Feather: The Future of Storage Networking Wednesday 8:30a Cisco - Building Cloud-Ready Storage with Cisco and EMC Tuesday 10:00a FCoE in the EMC Topology Guide http://elabnavigator.emc.com EMC FCoE Videos: Search for FCoE on YouTube EMC FCoE Introduction whitepaper http://www.emc.com/collateral/hardware/white-papers/h5916-intro-to-fcoe-wp.pdf FCoE Blog by Erik Smith (E-Lab) http://www.brasstacksblog.typepad.com Copyright 2011 EMC Corporation. All rights reserved. 44 45. Q&A Copyright 2011 EMC Corporation. All rights reserved. 45 46. Copyright 2011 EMC Corporation. All rights reserved. 46 47. THANK YOU Copyright 2011 EMC Corporation. All rights reserved. 47