Looking Ahead to 10 Gigabit Ethernet and Storage Networks

Looking Ahead to 10 Gigabit Ethernet and Storage Networks

By Marc Farley September 2002

2

Copyright 2002 TidalWire Inc. All rights reserved.

With the ratification of the 10 Gigabit Ethernet (IEEE 802.3ae) standard this June and with products already being sold, it is only a matter of time before storage networking will be integrated with it one way or another. This white paper looks at the technology of 10 Gigabit Ethernet and attempts to predict some of the ways storage networks will make use of it. In addition, several business issues are discussed, pointing out some of the problems that will be encountered as network operators and storage network subscribers seek a common ground to utilize this new technology.

10 GIGABIT ETHERNET OVERVIEW

10 Gigabit Ethernet is the latest version of the most popular LAN technology in the world. The stated goal for 10 Gigabit Ethernet was to create a new high-speed Ethernet technology that compliments and expands the range and use of pre-existing types of Ethernet. Specifically, Gigabit Ethernet enables higher bandwidth LAN backbone networks as well as extending the range of Ethernet from the LAN directly into MANs (metropolitan area networks) and WANs (wide area networks). Transmission distances with 10 Gigabit Ethernet vary greatly, depending on the type of lasers and fiber optic cabling used. 850 nanometer lasers over 62.5 micron multimode fiber (MMF) can be limited to 25 meters, while 1550 nanometer lasers over 9 micron single mode fiber (SMF) can be extended to 40 kilometers – over a thousand-fold increase. There are several key differences between 10 Gigabit Ethernet and previous versions of Ethernet. For starters, the original half-duplex CSMA/CD access method is not supported in 10 Gigabit Ethernet. There were many who felt that CSMA/CD should not have been supported in Gigabit Ethernet, but the argument for access compatibility was too strong. Today, It is clear that new high bandwidth technologies will use switched, full duplex communications with separate send and receive channels between communicating ports. In general, 10 Gigabit Ethernet is viewed as a technology to be used for inter-switch links, as opposed to being used by systems. Another key difference is the specification for several types of fiber optic cabling to achieve the desired transmission reliability and link distances. This fits with the goal to use 10 Gigabit Ethernet as a network infrastructure technology in enterprise backbone and MAN/WAN networks. 10 Gigabit Ethernet is not generally considered for connecting individual systems to a network because there are so few systems that could potentially avail themselves of 10 Gigabit Ethernet’s bandwidth. As a result, it seems unlikely that there will be much development of 10 Gigabit Ethernet host adapters in the next several years. There are efforts underway to develop copper-based cabling for 10 Gigabit Ethernet with the goal of creating less expensive connectors and cables that would support links of 20 meters or less. The encoding scheme of a networking technology provides accurate and reliable data transmissions for the price of a certain amount of overhead. Most 10 Gigabit Ethernet media interfaces use an encoding scheme with a 64/66 ratio, which means that for every 64 bits of data transmitted, two extra bits-worth of encoding information are also transmitted – equal to approximately 3%. This is far less than the encoding scheme used for Gigabit Ethernet and Fibre

3


Channel, which both use an 8/10 ratio, or 25% overhead. This means that 10 Gigabit Ethernet can be extremely efficient in its use of the total available bandwidth.

DUAL NETWORK ADAPTATIONS

The most interesting development in 10 Gigabit Ethernet is its ability to be used on both traditional LAN environments as well as WAN/MAN environments. The LAN connection technology is referred to as the LAN PHY (“PHY” is engineering-speak for the lowest physical level in a network protocol stack) and the WAN/MAN connection technology is referred to as the WAN PHY. This dual-network functionality was one of main accomplishments of the 10 Gigabit Ethernet standards group. For planning purposes, it’s important to understand that 10 Gigabit Ethernet makes a distinction between LANs and WAN/MANs. The port hardware is different for both and must match the network type it is connecting to. The LAN PHY is designed to run over dark fiber or dark wavelengths with a data rate of 10 Gb/second. The term “dark” here means private, unshared, non-multiplexed transmissions. While this is certainly what one would expect for storage traffic, but this has not been the case historically in the telecommunications industry where the use of shared, multiplexed transmissions are practically synonymous with “network”. The term “wavelength” in “dark wavelength” refers to the technology of wave division multiplexing, or WDM, which maintains separate virtual private networks within high-speed, mult-iplexed networks. Wave division multiplexing is akin to the concept of virtual LANs, but with a physical implementation in physical network transceivers. There are several kinds of WDM technology: wide WDM (WWDM), dense WDM (DWDM) and coarse WDM (CDWM), which add a fair amount of complexity to the mix. WAN PHYs in 10 Gigabit Ethernet are designed to connect to existing or new SONET SDH networks, such as those used in voice networks by the telecommunications industry. The data rates for SONET SDH are 9.6 Gb/second – also commonly referred to as OC-192. SONET and SDH are similar standards used by the American and international telecommunications industries. In essence the WAN PHY is a superset of the LAN PHY, with the ability to interoperate with SONET SDH networks by framing and receiving SONET SDH transmissions, as well as by participating in basic network management functions. Unlike traditional SONET SDH applications that use synchronous transmissions, the 10 Gigabit Ethernet WAN PHY uses asynchronous transmissions and does not use some of the higher level transmission logic associated with SONET SDH networks. Cabling scenarios for 10 Gigabit Ethernet are bound to be confusing due to the fact that both LAN and WAN PHYs can use the same type of physical cables with potentially different transceivers. Therefore, extra care will need to be taken when configuring and ordering 10 Gigagbit Ethernet products to make sure everything will be compatible. Product configuration and part options are bound to be complicated. There are likely to be creative engineering solutions that will make their way into the market as the technology becomes embraced.

4


COMBINING STORAGE NETWORKS WITH 10 GIGABIT ETHERNET

Both SAN and NAS implementations can use the bandwidth made available by 10 Gigabit Ethernet. There are several scenarios that are likely be emerge that are discussed in the remainder of this article.

Remote Data Protection over High Speed, Long-Distance Connections

The most likely scenario for the intersection of 10 Gigabit Ethernet and storage is in WAN/MAN environments to achieve redundant, remote data protection at high speeds. While many IT organizations would like to be able to electronically copy data to a secure remote location, the price has been too high or the performance has been too low: either the job cannot be done as required or it costs too much. In many cases, the distances involved are too great to be cost effective. Another issue with remote storage is the potential to lose administrative control of valuable, sensitive data. There is no doubt that the most important data a company wants to protect is also often its most sensitive data. If this prized data is being turned over to external network and storage operators, there is some amount of risk the data could be stolen or accessed by hostile entities. So, it would seem that remote data protection requires new technology that reduces costs and boosts performance without turning over control and security of data resources. 10 Gigabit Ethernet may be able to provide some of the pieces of this puzzle. To begin with, 10 Gigabit Ethernet has excellent performance. In addition, this performance is supported to maximum distances of 40 kilometers. The cost of the long-distance technology will probably be high, but compared to Fibre Channel’s maximum link distance of 10 kilometers and Gigabit Ethernet’s maximum link distance of 5 kilometers, 10 Gigabit Ethernet might have the characteristics that make it appealing for remote storage. An important element working in 10 Gigabit Ethernet’s favor is the ability for network service providers to integrate it into their existing voice-oriented SONET SDH networks and to parcel the bandwidth out as multiple smaller data/storage networks to many different customers. 10 Gigabit Ethernet supports the 802.1Q standard for virtual LANs (VLANs), which allows service providers to establish unique virtual networks for storage subscribers. This is not the same as saying the network can be over-subscribed, as is common in voice networks, but it does mean that network service providers can spread the cost of an OC-192 SONET SDH network to many different clients. The tough spot about sharing bandwidth where storage networks are concerned is the large amount of bandwidth needed for storage I/O, especially when compared with other types of networks. For instance, most data networking applications work very well with 100 Mb connections between communicating nodes. This is approximately equivalent to only 10 MB of storage I/O, a number that was surpassed many years ago. Realistically, storage I/O probably requires performance levels of 30MB/sec and higher to be broadly applicable. So, VLANs used for storage need to have more bandwidth allocated to them than VLANs used for data networking applications. Unfortunately, it’s not clear yet if network operators understand this bandwidth discrepancy and whether they are designing their networks to meet the bandwidth requirements for storage WANS and MANS. If not, artificial bandwidth shortages in

5


10 Gigabit Ethernet networks could result in inadequate bandwidth for storage applications, which in turn would drive up the cost for storage subscribers and under-utilization of the technology. The suggestion here is that WAN/MAN network operators find other business models based or value-added services instead of bandwidth to build their storage network businesses. Another key capability of VLANs is their ability to provide private tunnels through the network backbone or core. In other words, a 10 Gigabit Ethernet VLAN is essentially a private network that is not shared by other subscribers, providing a basic level of security. VLANs do not provide strong security in the form of authentication and encryption, but the use of VLAN technology in Ethernet networks does reduce the number of ways a network can be hacked. It is not yet clear how 10 Gigabit Ethernet VLANs for storage will be managed. Subscribers will want to mange these network segments as if they are internal network resources – even if network operators are providing them. The ability to monitor and control a 10 Gigabit Ethernet VLAN as a private resource could turn out to be the most attractive characteristic of the technology. But network operators need to be able to manage their resources, especially if they have SLAs associated with them that they need to meet. Again, this gets back to the topic of business models for network service providers who market to storage network subscribers. It’s not clear how this will evolve, but there are considerable business hurdles to overcome.

Remote Data Protection Over 10 Gigabit Ethernet

There are basically two remote storage applications that can drive the adoption of 10 Gigabit Ethernet: remote backup and remote mirroring. Both have been used successfully with Fibre Channel networks providing the storage connectivity and there is no reason to believe that 10 Gigabit Ethernet will change things. One of the most effective applications of storage networks so far has been the consolidation of backup systems in a central site. Backup is chronically problematic for many organizations and the centralization of backup resources helps pinpoint problems as they occur, eliminating a slew of errors associated with distributed administration and operation. Where backup is concerned, the biggest impact for remote storage is probably not transmitting backup data to a remote location, but instead is transmitting backup data from remote locations to a central location. In other words, the remote sites could use electronic vaulting to copy data to a centrally controlled site with adequate round-the-clock administration resources. This is not to say the remote transmission of backup data from the central site is not valued. Today, companies use tape courier services to physically transport tapes to off-site facilities for safe-keeping. Electronic vaulting could be an immediate and cost-effective way to copy data to a secure off-site storage facility. This could be done as a real-time backup operation with tape mirroring technology, but would more likely be accomplished with tape-to-tape copy routines over a 10 Gigabit Ethernet VLAN network. Figure 1 below shows a 10 Gigabit Ethernet WAN/MAN VLAN that would be used to consolidate backup resources in a central site, and then to copy backup tapes to a remote tape storage service provider.

6


FIGURE 1 The other primary remote storage application to use 10 Gigabit Ethernet is mirroring. Whereas backup is normally done as a batch process and has the ability to maintain historical versions of files on tape, mirroring is done in real time and maintains a single complete copy of data as it exists at any given point in time. The great enemy of remote mirroring is network performance, especially latency. Disk I/O operations are tightly coupled processes that depend on an exchange of acknowledgements to verify correct execution of storage commands. In most disk mirroring solutions, both devices (or subsystems) must acknowledge completion of a command before the next command can be issued. There are also schemes that do not require tightly coupled acknowledgements, but there can be serious data integrity risks when network errors occur. In WAN and MAN environments, slow round-trip acknowledgements can have a detrimental effect on system performance because applications with rapid I/O requirements are constantly waiting for acknowledgements to be returned. In general, remote mirroring has not been affordable except in large companies who can afford their own dedicated Fibre Channel networks. Fibre Channel has proved that remote mirroring can work, but it has not proved that it can be affordable to the broad market. 10 Gigabit Ethernet may change the market dynamics of remote mirroring by enabling high-bandwidth, private, VLAN connections over less expensive public networks, as opposed to dark fiber. In addition to supporting 803.1Q VLANs, 10 Gigabit Ethernet also supports the 802.1p standard for prioritization of frames in an Ethernet network. 802.1p over 10 Gigabit Ethernet sets the stage for mirroring remote high-throughput transaction processing data to a remote site at the highest priority level. While it may be difficult to guarantee a particular round trip response

SONET Network

Remote Storage

Remote Tape

Storage 10 Gigabit Ethernet

WAN Switch

VLAN

VLAN

VLAN

VLAN VLAN

VLAN

VLAN Main Central Storage

Remote Storage

Remote Storage

Remote Storage

Remote Storage

Remote Storage

7


time, it may be possible to guarantee that mirrored transaction data will have the best performance in the network. An organization that wants to mirror data between two different data centers could do so over a single 10 Gigabit Ethernet connection. The full-duplex nature of the technology means that a large amount of data could be flowing in both directions simultaneously. For instance, a full 10 Gigabit Ethernet “pipe” could accommodate mirrored traffic for up to 20 heavily utilized servers at each end of the pipe.

DIRECT ACCESS TO REMOTE STORAGE OVER A 10 GIGABIT ETHERNET NETWORK

The ability to extend the reach of an Ethernet infrastructure allows organizations to extend the reach of their storage networks for accessing data directly.

Remote NAS and 10 Gigabit Ethernet

NAS clients and servers can be connected over WAN and MAN 10 Gigabit Ethernet links in exactly the same way as they are connected over an Ethernet campus backbone network today. The one big difference is that NAS systems are unlikely to have 10 Gigabit Ethernet NICs, the way some of them have Gigabit Ethernet NICs today. There are two advantages to doing this. The first is that storage administration can be consolidated at a central location in the network. There is nothing new about the cost benefits of administering resources over 10 Gigabit Ethernet. Previous forms of Ethernet have been providing cost efficiencies now for decades. The second advantage of extending the reach of NAS networks is that workers in physically separate locations can use the exact same data files without having to exchange them using an inefficient and error-prone process like email. There is no doubt that email attachments for data sharing create many problems for both system administrators and information workers alike. By bypassing email and accessing data directly on commonly available NAS servers, a great deal of confusion over file versions can be avoided, not to mention all the unnecessary redundant and wasted storage space associated with email attachments.

Remote SANs and 10 Gigabit Ethernet

SAN subsystems can also be consolidated and centralized, just as NAS servers and backup systems can be. While it may seem more likely to use 10 Gigabit WANs and MANs for mirroring, it is also possible to use them for primary access to data that is located in a remote location. This could an effective way of managing configuration changes in distributed application servers or as a way to provide regular data updates for certain applications that use data created by some other process. A system outside of the headquarters campus might have its own boot disk and storage for certain applications, but access centralized storage for other applications.

8


LOCAL SANS AND 10 GIGABIT ETHERNET

10 Gigabit Ethernet can also be used as a backbone to connect local SANs together. For instance, Fibre Channel SANs can be connected to an Ethernet/IP LAN through FC/IP gateway systems. Notice that it’s not necessary to have FC/IP gateways with 10 Gigabit Ethernet components to take advantage of a 10 Gigabit Ethernet backbone. Gateways with Gigabit Ethernet ports can be connected to 10 Gigabit Ethernet switches as shown below in Figure 2. FIGURE 2 iSCSI SANs can also be connected to 10 Gigabit Ethernet backbone LANs. This is a bit of a no-brainer as there is nothing needed to enable this other than switches and routers that can make the speed transition between slower speed Ethernet used for iSCSI (probably Gigabit Ethernet) and 10 Gigabit Ethernet. In either the Fibre Channel or iSCSI case, it is not evident that connecting many SANs together just for the sake of being connected is a good idea. There may be certain resources, such as tape resources that could be used by several different SANs, but that does not mean that its desired to have all servers be able to connect to all storage subsystems. For that reason, it seems more likely that 10 Gigabit Ethernet will be used in a WAN/MAN context, as opposed to in LANs.

SAN

SAN

SAN

SAN

FC/IP Gateway

FC/IP Gateway

FC/IP Gateway

FC/IP Gateway

10 Gigabit EthernetLAN Switch

10 Gigabit EthernetLAN Switch

10 Gigabit Backbone Network

Gigabit Ethernet

Gigabit Ethernet

Gigabit Ethernet

Gigabit Ethernet

FC

FC FC

FC

9


Future of 10 Gigabit Fibre Channel

Fibre Channel SANs have enabled remote storage connections over dark fiber for several years. The Fibre Channel industry has announced its intention to support the new 10 Gigabit Ethernet physical capabilities in its next revision of Fibre Channel. This would be a reversal of the situation with Gigabit Ethernet and gigabit speed Fibre Channel, where the Fibre Channel physical layer was developed first and then adopted by the Gigabit Ethernet community. If the Fibre Channel industry completes this work, it would most likely have similar characteristics to 10 Gigabit Ethernet, including the ability to transmit data over SONET SDH and WDM networks. Unlike 10 Gigabit Ethernet, which can be used for all types of networking applications, 10 Gigabit Fibre Channel would most likely be limited to storage applications in most installations. ABOUT THE AUTHOR Marc Farley is the author of Building Storage Networks and a recognized expert in the field of storage networking. He has over 10 years of experience in storage networking strategies and management including backup, storage appliances, virtualization software, storage subsystems, IP tunneling, routing and switching. His consulting company, “Building Storage, Inc.” provides storage services to vendors, financial institutions and end-user organizations.

Looking Ahead to 10 Gigabit Ethernet and Storage Networks

Documents

Transcript of Looking Ahead to 10 Gigabit Ethernet and Storage Networks