The Unintended Consequences of Converged Datacenter

The Unintended Consequences of Converged Datacenter Deployment Models

Simon Gordon, Juniper Networks

The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 2 2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.


Abstract

The Unintended Consequences of Converged Datacenter Deployment Models

The last two years have seen FCoE move from buzz to production deployments. As this has happened many people have had to start dealing with the many of the unintended and unexpected consequences that arise as a result of the move to a converged infrastructure.

During this session we will examine these areas and review both the challenges and some potential solutions to meet the full datacenter requirements list. Customers and partners are seeing differences and challenges in a number of areas including:

Changes in the server redundant nic model due to io consolidation and CNA operating models Tension between storage dual independent rail and Ethernet/ip fully connected networks Differences in the HA assumptions around Ethernet/IP and FC platforms Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE Implications of multi protocol storage networks (mixed has iscsi fcoe big-data) Challenges in maintain the traditional FC Qualification Matrix / Firmware requirements model.

The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.

Agenda – NOT COVERED

What about iSCSI, NAS, Big Data Much of what I say here applies there as well

What about multiprotocol storage networks Aka all of the above at the same time Again much of what I say here applies even more so

In other words this presentation is mostly about network convergence but happens to use FCoE as the example given the many companies currently using native Fibre Channel

4


Agenda – NOT COVERED

FCoE using FC-BB-5 & IEEE DCB works as intended For both ‘Converged Access’ and ‘End to End FCoE’ Of course it depends on what products you look at And depends on what topologies (logical & physical)

FC-BB-6 is adding more capabilities (but it still depends on product and topology specifics)

802.1P, and DCB (inc. ETS) gives separation of traffic in some sense like having 8 virtual networks PFC Gives Lossless per priority for FCoE DCBX helps DCB & FCoE Setup FC-BB-5 FC encapsulation, L2 Discovery, etc

P0

P1

P2

P3

P4

P5

P6

P7

P3

FCoE


Agenda

I See No Ships ! Is it ok to just go right ahead ? Should you ?

The Elephant(s) in the Room … Certifications qualifications interoperability upgrades HA

How do you know there’s an Elephant in the Fridge ? Because there are footprints in the butter Monitoring tools

We don’t know what we don’t know And always will be


I See No Ships !

When asked to retreat… Nelson claimed he couldn’t see the enemies ships

Battle ensued… And Nelson was victorious

Sometimes we should get engaged despite the fact that we know there WILL be challenges…

7


I See No Ships

Fibre Channel over Ethernet IS NOT PERFECT We should not claim it is

HOWEVER Fibre Channel is not perfect Ethernet is not perfect, nor is IP perfect

BUT FCoE DOES WORK within well understood boundaries EVEN End to End FCoE works well as its ‘nothing special’ FCoE can be problematic if done badly (as can FC) Just so long as you apply best practices all will be well


I See No Ships

Is it the right time to deploy ? When are your natural replacement cycles ？ What are the current focuses for investment ？ What are your short medium and long term goals ？

Most companies do not have Enough people or enough budget

To do everything they might like to do

Work out what gives the best bang for the buck Does FCoE deployment align with other activities ？

Moving towards cloud / virtualization / orchestration

Don’t try to do ‘unnatural acts’ just for the sake of it ？ Just because you need to replace some kit it may not be the right time for you


I See No Ships

Maybe now is not the right time for you But start to understand so that you are ready when it is Choose not to converge rather than accidental…

Or you want to use convergence to increase potential connectivity to 100% attach

Aka converged access to on going dedicated SAN

Maybe its cap manage and wait for the next steps Aka converged access to ongoing dedicated SAN

Or maybe its cap and actively migrate Either to FCoE end to end Or to other Ethernet/IP storage protocols


The Elephant(s) in the Room…

Qualification Firmware Versions Interoperability High Availability Supported topologies


The Elephant in the Room

Qualification / Certification has value Verifies unidimensional interoperability Multidimensional feature testing Scale & performance testing End to End Solution testing Define best practices to make sure it works well Fibre Channel has been successful in part due to this testing

But it was never a perfect world Early on in Fibre Channel maybe we followed the qualification guidelines mostly because we bought the solution from one vendor These days as most of you have servers/storage from multiple vendors you end up breaking at least one vendors guidelines



The new FC hardware/software is tested by The manufacturer

The hardware/software development team The systems test team The solutions teams The partner pre-qualification team

The server & storage vendor Qualification test team

The systems integrator SI test lab

The customer Customer test lab vs production floor deployment

Yet sometimes you still find bugs in production…

13



Interop’ testing also done for Ethernet & IP Verifies unidimensional interoperability Multidimensional feature testing Scale & performance testing End to End Solution testing (though more limited) Define best practices (but customers diverge more) All vendors forced to support much broader interoperability Sheer volume of deployed ports leads to greater hardening App’s & high level protocols designed to run on unreliable networks Formal qualification / certification matrixes don’t tend to exist in for format Fibre Channel people are used too

Even for iSCSI and NAS in many cases Though you can argue even for iSCSI/NAS they should…



What are the issues ? What boxes need what testing

FC-FCoE Switch (FC-SW) / FCoE Forwarder (FCF) N_Port Virtualizing Gateway FIP Snooping Bridge (FSB)

When should a vendor retest ? Even when new code is not intended to affect FCoE ?

How deeply should they test ? More protocol complexity, new features, new standards, new topologies

How broad should they test Lots of features and hard to know all the interactions There are ALWAYS unintended or unanticipated consequences



Solutions… We all do risk analysis

Work out what we do or don’t need to test Work out best practices on how to deploy Prioritize where to do the testing and who with

Frankly we always did Its nothing new This is how the FC world always works This is how the Ethernet world works as well

You must still do this And I am sure you all do There is a lot of good information on FCoE best practices



Even more solutions… SAN and LAN team cooperation & coordination

Just like for dc connectivity Product selection Datacenter design Operational practices

Best practices Just because you can do something doesn't mean you should

– Remember there is a fine line between clever and stupid

Think about the implications as you come up with your design

Don’t forget most of the protocols are mature both Fibre Channel and Ethernet are mature Even DCB is very well defined and no ambiguity Testing can be focused on very specific areas of challenge


Oh No – ANOTHER Elephant…

18 18

Different views on High availability: Single Rail Vs Dual Rail Network Design

FC Single Rail vs Ethernet Dual Rail

Product HA Characteristics Fixed & Modular - Ethernet vs FC

Implications of L2 Domain on HA Physical L2 Domain

– full separation even from untagged broadcast

Logical L2 Domain (VLAN) – partial separation as untagged broadcast may cross over

To cover this properly would take entire separate session


Oh No –ANOTHER OTHER Elephant…

19 19

Implications of LAG LAG (LLCP) assumes packets can be sent to any link in the LAG

That is fine when the upper level protocol is AFTER the LAG

However for some devices the FCoE traffic is peeled off from the single physical interfaces before it is remerged in to the logical interface

LAG based NIC teaming can be problematic with HW offload CNA Any device that has the logic on the physical port not the logical port

Of course LAG is NOT the only load balancing or link level redundancy mechanism

There are other options that do not have these challenges

To cover this properly would take entire separate session


How do you know that there’s an elephant in the fridge ?

Because you can see footprints in the butter


Clearly Use the existing SAN monitoring and management tools Use the existing LAN monitoring and management tools

But are they complete ? Do the SAN tools properly understand the Ethernet piece Do the LAN tools give the data the SAN team needs

Not to mention dual admin Are there challenges with two sets of people using two sets of tools to monitor and manage a single set of equipment The tools, and indeed the standards, are still a work in progress

How do you know that there’s an elephant in the fridge ?


We Don’t Know what we Don’t Know

But then that’s always the case We do know there is value in convergence…


What are the scalability implications

Fibre Channel Networks getting larger And more complex

23 23

Ethernet Networks getting larger And more complex

Fibre Channel over Ethernet Merging networks means even larger networks

But better economies of scale And less diversity of equipment

That’s why FC-BB-6 is addressing scalability/topologies FCF/FDF Distributed switches VN2VN for direct layer 2 initiator to target traffic



Implications of multi protocol storage networks (mixed NAS iSCSI FCoE big-data)

24 24

FC SAN A

FC SAN B

Big Data Storage

End of Row Storage

In Rack Storage

In Server Storage

Data De-duplication



Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE

Bandwidth & Latency may be an issue for end to end converged networks Firewalls & Routers definitely get in the way of end to end converged networks

25 25

Ethernet

Servers NAS

Multi Tier Architecture

1 2 3 4

Red NetworkGreen Network



Simplify your network to an L2/L3 Fabric of some form

Avoid L2 scale issues Avoid criss crossing the fabric for VLAN Routing

MAN/WAN L3 pushed to the edge

L3 Scale & performance

Move Firewalls to the edge and only redirect those flows needing firewall services

Plus gateway services in hypervisor

26

Firewall

Storage

MAN WAN

Fin Mkt Eng Servers

Firewall Service NO Firewall Service

Core Routers

Datacenter Fabric

Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE



Many other changes underway Ethernet Fabrics

And what is the trade of between FCF vs FSB in the devices that make up the fabric

Network Virtualization & Overlay Networks What’s the impact on FCoE Far too early to tell

SDN What is the impact on FCoE May simplify here just as elsewhere

27



Summary

This stuff works and does brings benefits So you should plan your route forwards Go at the pace that makes sense for you Look to where you get the greatest value

However there are challenges Learn - before, during, and after - each step Keep your options open as to the final destination


Attribution & Feedback

Please send any questions or comments regarding this SNIA Tutorial to [email protected]

The SNIA Education Committee would like to thank the following individuals for their contributions to this Tutorial.

Authorship History

Original Author: Simon Gordon (Fall 2012) Updates:

Additional Contributors

Joseph White Erik Smith / J Metz

mailto:[email protected]�

The Unintended Consequences of Converged Datacenter

Documents

Transcript of The Unintended Consequences of Converged Datacenter