IBM Installation Tuning Guide -...

NetXen, Inc.

Intelligent NIC® Installation and Tuning GuideFor IBM Servers: NetXen Software Release 3.4.216

Version 3.5November 15, 2007

NetXen, Inc.3965 Freedom CircleFourth FloorSanta Clara, CA 95054Telephone: (408) 330-0010Web Site: http://www.netxen.com

Copyright© Copyright 2007. NetXen, Inc. — All Rights Reserved.

DisclaimerThis is a preliminary document and is subject to change without prior notification and does not represent a commitment on the part of NetXen, Inc. The information contained herein is the proprietary and confidential information of NetXen or its licensors, and is supplied subject to, and may be used only by NetXen’s customer in accordance with, a written agreement between NetXen and its customer. Except as may be explicitly set forth in such agreement, NetXen does not make, and expressly disclaims, any representations or warranties as to the completeness, accuracy or usefulness of the information contained in this document.

Please contact NetXen, Inc. to obtain the latest revision of this document.

TrademarksNetXen, the NetXen logo, and Intelligent NIC are trademarks or registered trademarks of NetXen, Inc. Third Party brands and names are the property of their respective owners.

Revision History

Version Date Editor Notes

3.0 6/20/2007 SR First release of document.

3.1 6/272007 SR Added Section 9, Teaming Driver, and all related subsections.Added Appendix C, PXE Boot with the NetXen CardAdded Appendix D, iSCSI Boot with the NetXen Card (place holder)

3.2 7/24/2007 SR Added Section B.2, PXE Configuration.Added Section 8.4, LSA Statistics.Added Appendix C, Using Accelerated iSCSI.

3.3 8/14/20078/20/20078/22/20078/23/2007

SR Added Section 6.5, Additional Support.Added Section 9.2, Linux Bonding/Teaming Mode and all related subsections.Added Section B.2, PXE Configuration for BIOS Setup Changes.Added Section B.3, PXE Configuration, Setting up Windows Server 2003.Added Section B.4, PXE Configuration, Setting up a Linux Server.Added iSCSI mode information at the beginning of Appendix C.

NetXen, Inc

3.4 10/18/07 SR Added Section 2.2.2, XFP Module.Added Table 7, “Quick Start Instructions — Enabling WIndows Chimney and Linux LSA”.Added TCPv4 Stateful Offload entry to Table 8.Added Section 4.5.3, “Uninstalling the NetXen NIC Driver in Linux”.Added Section 4.5.4, “Uninstalling the LSA Driver in Linux”.Removed Appendix B, PXE Boot.Added Section B.2.2, “Windows Remote Installation Services (RIS) Installation”.Added Section 8, NIC VLAN in Windows

3.5 11/13/07 SR Rewrote installation procedure for Section 4.2, “Installing and Connecting the NetXen Mezzanine Card for BladeCenter”.Added Section 4.4.4, Linux Kudzu.Updated Section 6.1, Linux Driver Installation.Updated steps 2 and 3 of Section 7.1.1, Windows Installation.Updated text in Section 10, User Diagnostics.

Version Date Editor Notes

NetXen, Inc

Table of Contents

1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 NIC Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Currently Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Software Release Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Hardware and Software Installation Requirements . . . . . . . . . . . . . . . . . . . 2

2.2.1 Cable Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2.2 XFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3

3 Quick-Start Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Detailed Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.1 Installing and Connecting the NetXen Intelligent NIC Adapter . . . . . . . . . . 74.2 Installing and Connecting the NetXen Mezzanine Card for BladeCenter . . 94.3 Windows Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.3.1 NIC Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.3.2 Configuring the Network Interface Card . . . . . . . . . . . . . . . . . . . 18

4.4 Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.4.1 NIC Driver Installation (TAR format) . . . . . . . . . . . . . . . . . . . . . . 204.4.2 Linux NIC Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.3 Configuring the NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.4 Configuring Under Linux Kudzu . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.5 Uninstalling the Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.5.1 Uninstalling Windows Drivers for Both NIC and Chimney . . . . . 224.5.2 Uninstalling the NetXen NIC Driver in Linux. . . . . . . . . . . . . . . . 234.5.3 Uninstalling the LSA Driver in Linux . . . . . . . . . . . . . . . . . . . . . . 23

5 Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1 Using the NetXen Flash Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2 Windows Firmware Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.3 Linux Firmware Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Updating the Flash on Multiple Standalone Cards in Windows. . . . . . . . . 26

6 Linux Socket Accelerator (LSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1 LSA Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.1.1 TAR Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.2 LSA Offload Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.2.1 LSA Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.2 LSA Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.3 nxoffload Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

6.3 Running LSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.4 LSA Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.4.1 Example Read Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Teaming Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.1 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.1.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.1.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357.1.3 Fault Tolerance Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

7.2 Linux Teaming/Bonding Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

NetXen, Inc. i

Table of Contents

7.2.1 Mode 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457.2.2 Mode 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467.2.3 Bonding Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477.2.4 Bonding Driver Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

8 NIC VLAN in Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498.1 Optional Configuration on the NIC and Switch Side . . . . . . . . . . . . . . . . . 498.2 NIC VLAN as Send Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498.3 NIC VLAN as Receive Side. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

9 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 529.2 Unable to ping the NIC Installed on the Host . . . . . . . . . . . . . . . . . . . . . . 529.3 Gather System Information and Check for Errors on Linux Host . . . . . . . 539.4 Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

9.4.1 MSI Interrupts for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549.4.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

9.5 Additional Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5510 User Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

10.1 Windows Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5610.2 Linux Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Appendix A Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1 Windows Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1.1 NTttcp (Windows) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1.2 nttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . . . . 60A.2 Windows Optimization — Tcp1323Opts . . . . . . . . . . . . . . . . . . . . . . 61A.3 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A.3.1 IPERF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A.4 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Appendix B Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

ii NetXen, Inc.

1. Overview

1 OverviewThis guide describes the system requirements, installation, and setup required for the NetXen Intelligent NIC® network interface card product release.

1.1 Related DocumentsRelease notes included in the software release.

Check NetXen web site for updates.

1.2 NIC VersionsThe NIC products support Windows, Linux, and other operating systems. See the release notes for an updated list.

1.3 Currently Supported Features

Note: Check with NetXen periodically for software updates and additional features.

Product Number Product Definition

43W4453 Dual 10-GE ports, IBM-MEZZNXB-10GSXR Single 10-GE port (optical fiber)NXB-10CX4 Single 10-GE port (copper)

Feature Windows Linux

TCP, UDP, and IP checksum offload Yes YesTCP segmentation offload (TSO / LSO) Yes YesInterrupt coalescing Yes YesInterrupt distribution No YesVLAN Yes YesSNMP Ethernet MIB Yes YesJumbo frames Yes YesTeaming and Failover Yes YesLinux Socket Accelerator (LSA) N/A YesChimney (TCP/IP Offlead) Yes N/A

NetXen, Inc. Intelligent NIC Installation and Tuning Guide 1

2. System Requirements

2 System RequirementsBefore installing the product, verify the minimum system configuration requirements outlined below.

2.1 Software Release Package Contents

2.2 Hardware and Software Installation Requirements

2.2.1 Cable Specifications

Contents Format Directory Tree

Documentation IBM Installation and Tuning GuideBest Practices Guide

Release NotesNIC Linux Driver [version].tar.gz or [version].rpm

NIC Windows Driver [version].zipNX Intercept Driver [version].-lsa.tar.gz or

[version]-lsa.rpmLSA Driver [version].-lsa.tar.gz or

[version]-lsa.rpmFirmware imez_romimage or xfp_romimage

Diagnostics nxudiagTools nxflash

Table 1 NXB-10GXSR Cable Specifications — Optical1

1. Check with NetXen for information on the NXB-10GXLR long range optical cable specifications.

ParameterOptical Core Thickness

50 μm 62.5 μm

Wavelength 850 nm 850 nmBandwidth (min) 500 MHz 500 MHzOperating Range 300 m 300 m

Table 2 NXB-10CX4 Cable Specifications — Copper

802.3 Physical Layout Network Type Cable Length

10GBASE-CX4 LAN 15m (max)

Windows3.4.216

imx64x86

SNPx64x86

win2k3x64x86

utils

3.4.216

bindocsdriverinclude

Linux3.4.216-lsa

bindriverincludenxoffload

Linux LSA

2 Intelligent NIC Installation and Tuning Guide NetXen, Inc.

2. System Requirements

2.2.2 XFP Module

2.2.3 System and Software Requirements

Table 3 NXB-10GXSR XFP Module Specifications

Manufacturer Name Part Number

JDS Uniphase Multi-rate 10 Gb/s Short Wave Optical XFP Transceiver

64P0194

PicoLight Multi-rate 10 Gb/s 850 nm Optical XFP Transceiver Module

PL-XXL-SC-S45-22

Table 4 System and Software Requirements

Hardware Systems Validated Platforms

Validated OS (32bit and x64) Firmware1

1. NIC and LSA use the same firmware.

Driver

PCIe: (x8)

Memory: Minimum

2GB

AMD, Intel Windows Server: 2003 SP1,SP2,R2

Linux Distros: RHEL3 U8,

RHEL4 U3,U4 SLES9 SP3,

SLES10, RHEL5

NICLSA

Chimney2

2. To enable Chimney, a license must be obtained from NetXen.

Linux:3 NIC, LSA

Windows:4NIC

Chimney2

3. For Linux NIC and LSA, install the kernel source and header files

4. For Windows NIC, SP1 must be installed. SP2 is recommended.


3. Quick-Start Installation Instructions

3 Quick-Start Installation InstructionsThis section is for users already familiar with the Windows or Linux operating systems. For those user’s requiring more detailed information, refer to Section 4.

Table 5 Quick Start Instructions — New NIC Driver Installation

NetXen NIC Hardware Installation Windows Driver Installation Linux Driver Installation

Pre-Installation1. Verify the system meets the requirements

described in Section 2.2. Power down the system.

Installing the NetXen NIC Adapter1. Insert the NetXen NIC adapter card into an

available PCI Express slot.2. Connect the appropriate cables to the

board.3. Power on the system.

Note that the NIC card comes with the firmware already installed.

Driver Installation1. Copy all of the provided NIC driver files

to a directory.2. From the Device Manager, scan for new

hardware.3. When prompted to connect to Windows

Update, Click No, not at this time, then click Next.

4. In the next window, click Install from a list of specific location and then click Next.

5. Select Search for the best driver in these locations. Select Include this location in the search.

6. Provide the path to the directory where the driver NXP2NIC.sys is located. For 32-bit operating systems, choose the x86 directory. For 64-bit operating systems, choose the x64 directory.

NIC Card Configuration1. Open the Network Connections window

from the Control Panel.2. Verify that NetXen NIC adapter icon says

Connected. If not, right-click on the NIC Adapter icon and select Properties, then assign the IP and subnet addresses and click OK.

Driver Installation (tar.gz)1. Untar the file, build, install, and load the

driver: tar xzf [version].tar.gz cd [version] make make install insmod ./driver/nx_nic.ko

Driver Installation (source rpm)1. Install the RPM as follows1

rpm -iv nx_nic-[version].src.rpm2. Create the binary RPM as follows:

rpmbuild -bb /usr/src/redhat/SPECS/ nx_nic-[version].spec, (Redhat) or

rpmbuild -bb /usr/src/packages/SPECS nx_nic-[version].spec (SuSE)

3. Load the nx_nic driver. modprobe nx_nic

NIC Card Configuration1. Identify the NetXen Ethernet interface:

ethtool -i eth[n]2. Assign an IP address:

ifconfig eth[n] [IP address]

1. x.yy.zz is an example of [version]



If a NetXen Intelligent NIC card is already installed on the system and a driver and/or firmware update is required, refer to the procedure in Table 6 below.Table 6 Quick Start Instructions — Updating from an Existing Driver and Firmware

Windows Driver/Firmware Update Linux Driver/Firmware Update

Verify Link Status1. With the existing driver and firmware installed,

verify that the interface is up before continuing with the firmware update.

Update to New Firmware1. Open a Command window.2. Go to the utils directory in the driver package

provided.3. The following binaries are required to be in the

same directory from where the Flash command will be executed: nxflash.exe xfp_ibm_romimage imezz_romimage

4. Run the following commands: nxflash -i NX_NIC --all1

5. Verify the firmware has been updated using the following command. nxflash -i NX_NIC --info

Update to the New Driver1. Install the new driver version.2. Verify the new driver version.

1. Writing of the Flash is a critical operation. Please do not interrupt or reboot during this time.

Verify Link Status1. With the existing driver and firmware installed, verify

that the interface is up using ifconfig before continuing with the firmware update.

Update to New Firmware1. Run the following command from the bin directory of

the driver: ./nxflash -i eth[n] --all1

Unload and Remove the Existing Driver1. Unload the existing driver.

ifconfig eth[n] down rmmod nx_nic2. Remove the existing driver module. rm -f /lib/modules/[kernel version]/kernel/ drivers/net/nx_nic.ko

Reboot the System1. Once the existing driver has been removed, reboot

the system.

Update to the New Driver2. Update to the new driver as follows:

tar xzf [version].tar.gz cd [version] make make install insmod driver ./nx_nic.ko

3. Ensure that the Ethernet interface is up. ifconfig eth[n] <ip addr> up



Once the WIndows or Linux NIC installation is complete as described in the previous tables, the following procedures in Table 7 can be used to enable Windows Chimney or Linux LSA. Table 7 Quick Start Instructions — Enabling WIndows Chimney and Linux LSA

Enabling Windows Chimney Enabling Linux LSA

The following steps can be used to enable Windows Chimney on the driver. Note that the Windows NIC driver must be installed before executing the procedure below.1

1. Open the Network Connections window.

2. Right click on the icon for the NetXen adapter and select Properties.

3. Click on the Configure button.4. Click on the Advanced tab.5. Change the “TCPv4 Stateful offload” from disabled

to enabled.6. To verify the offload has been enabled and is

operating: C:\>netsh int ip show offload

1.A key is required to be loaded onto the card before Chimney can be enabled. Please refer to the Re-lease Notes for more information on how to obtain and verify this key.

The following steps can be used to enable Linux LSA on the driver. Note that the Linux NIC driver must be installed before executing the procedure below.

1. Untar the file, build, and load the drivers.tar xzf [version]-lsa.tar.gzcd [version]-lsamakecd driverinsmod ./nx_intercept.koinsmod ./nx_lsa.ko

2. To verify the LSA driver is operating, execute the following command:cat /proc/net/nx_nic/lsa_1/stats

3. Example for offloading iperf.cd ..\nxoffload./nxoffload –a –n iperf –z

4. To display the current offload rules execute the following:./nxoffload -s


4. Detailed Installation and Configuration

4 Detailed Installation and Configuration

4.1 Installing and Connecting the NetXen Intelligent NIC Adapter1 Verify that the system is powered off.

2 Remove the system cover.

3 Locate the x8 PCIe slot if one is available.

4 Once the bracket is removed, insert the adapter into the PCI Express slot1. Apply uniform pressure on top of the board until it is firmly seated in the bus connector.

1. Due to space constraints in some systems, when installing an XFP board, it may be necessary to first remove theXPF module from the board, insert the board into the PCI slot, then reconnect the XFP module to the board.

Identify x8 slots for optimal performance



5 For optical installations, once the board is inserted into the slot, connect the XFP module. Make sure the optical module clicks when inserted into the cage.

6 Secure the bracket.

7 Connect the appropriate cables to the NIC and replace the cover.

8 Power on the system.

XFP Cable connection

CX4 Cable connection

XFP Module

CX4Connector



4.2 Installing and Connecting the NetXen Mezzanine Card for BladeCenter1 Verify that the Blade is powered off.

2 Remove the Blade server from the chassis.

3 Remove the Blade cover.

4 Locate the Combo Form Factor Horizontal (CFFh) PCI-Express connector on the Blade server.

5 Remove the PCI-express connector cover if needed.

6 Insert the NetXen Mezzanine card firmly into the CFFh connector. A picture of the mezzanine card is shown below.

7 The NetXen Mezzanine card should be inserted at an angle into the mounting posts. Align the holes in the back of the card with the mounting posts at an angle of approximately 30 - 45 degrees. The card should then be rotated down onto the Blade, and pressed into the CFFh PCI-e connector. Refer to the figure on the next page.

NOTE: If the card is not inserted properly, first into the mounting posts and then into the connector as described above, damage to the card could occur.



8 Replace the Blade cover.

9 Insert the Blade server back into the chassis slot.

10 Power on the Blade.

CFFh PCI-eConnectors

NetXen Mezzanine Card

Mounting Posts



4.3 Windows Driver InstallationThis section describes the Windows installation procedure. For the Linux driver installation procedure, refer to Section 4.4.

4.3.1 NIC Driver InstallationNote that the following procedure is for a new installation.

1 Copy the NIC release package to a directory on the host.

2 After the server system is powered on, Windows detects the new card and displays the Found New Hardware Wizard window.

When this window appears, select No, not at this time, and click Next.



3 Select Install from a list or specific location (Advanced) and click Next, as follows:

4 Select Search for the best driver in these locations.Select Include this location in the search.

Type the appropriate path to the NIC driver directory in the accompanying field and click Next. An example is shown below:

If the exact directory path is not known, click Browse and select the NIC driver directory.

C:\3.4.216\windows\win2k3\x86



The NIC card supports both 32-bit and 64-bit operating systems. The x86 directory contains 32-bit driver and the x64 directory contains the 64-bit driver. Select the appropriate directory (x86 or x64) and click OK. Then click Next in the Hardware Update Wizard window to install the driver.

5 The following screen is displayed while the driver is installed.



6 Once the driver installation is complete, the following window will appear. Click Finish to exit.

7 To verify that the driver was properly installed, Right-click on My Computer, then select Device Manager. A listing of devices appears in right side of the window. Under Network Adapters, the NetXen 10Gb Ethernet Adapter should be listed as shown below:



8 In the above window, right-click on the NetXen 10Gb Ethernet adapter and select Properties. The General tab will be displayed as shown.

9 There are four tabs — General, Advanced, Driver, and Resources. Click on the Advanced tab to display the driver properties as shown below.



The Advanced tab selections displayed above are defined as follows:Table 8 Description of Advanced Properties Parameters

Parameter Description

Flow Control Sets flow control parameters.Registry Key: FlowControlDefault: Enabled.Values: Enabled/Disabled

Large Send Offload TCP Large send offload – Enable/Disable options.Registry Key: LargeSendOffloadDefault: Enabled.Values: Enabled/DisabledSize of LSO: 64K

Locally Administered Address This is the Locally Administered Address (LAA) that users/administrators can set on this interface and this address overrides the permanent address of the adapter (that may have been flashed in to the hardware).Registry Key: NetworkAddressFormat: HexOptions: Value (a value is required, enter the 12 hex bytes of the MAC address to be used), or not present.

Max Ethernet Frame Size Ethernet frame size for packet transmission and receive (does not include MAC Header).Reg Key: MaxFrameSize)Default: 1500 (corresponds to 1514 bytes on the wire + 4 bytes of CRC)Max: 8000 Min: 1500

Max Jumbo Buffers Number of buffers per Jumbo frame.Reg Key: RxJumboRingSizeDefault: 4096 Values: 1024, 2048, 4096, 8192.

Maximum Receive buffers (in K) This is packet buffers used by the driver for receive (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: MaxRxBuffersMin: 1KMax: 16KDefault: 8192

802.1P/Q Packet Tagging For supporting 802.1p/q priority tagging. Should be enabled to set VLAN ID.Registry Key: PQTaggingDefault: EnabledValues: Enable/Disabled



Receive IP Checksum Offload This parameter is for enabling/disabling the receive IP checksum offload.Registry Key: RxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Receive TCP/UDP Checksum Offload This parameter is for enabling/disabling the receive TCP and receive UDP checksum offload.Registry Key: RxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Status Ring Size Receive status ring size from hardware.Registry Key: StsRingSizeDefault: 8192Values: 1024, 2048, 4096, 8192, 16384, 32768.

TCPv4 Stateful Offload TCPv4 stateful offload.Registry Key: EnableTcpOffloadDefault: DisabledValues: Enabled/Disabled

Transmit IP Checksum Offload This parameter is for enabling/disabling the transmit IP checksum offload.Registry Key: TxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Transmit Ring Size Transmit command ring size provided to hardware.Registry Key: TxRingSizeDefault: 1024Values: 1024, 2048.

Transmit TCP/UDP Checksum Offload

This parameter is for enabling/disabling the transmit TCP and transmit UDP checksum offload.Registry Key: TxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

VLAN ID For setting the VLAN ID for this interface (also exposed through the standard OID).Registry Key: VlanIdDefault: 0 (no VLAN)Range: 0 - 4094

Table 8 Description of Advanced Properties Parameters (Continued)

Parameter Description



10 Click on the Driver tab in the Advanced Properties window.

The driver can be updated, rolled back, or uninstalled from this window.

This completes the installation of the NetXen NIC software. Refer to the next section for information on configuring the NIC.

4.3.2 Configuring the Network Interface Card

1 Configure the network connections by opening the Network Connections window in the control panel, or directly from the desktop if the icon exists. The following window is displayed.



Note that if an IP address has not been assigned, the NetXen 10Gb Ethernet adapter entry above will say “Limited or no connectivity”.

If a red ‘X’ appears in the NetXen 10Gb Ethernet10 Adapter icon, this indicates that the link is down. If this occurs, check the following:

Verify that the XFP module is installed correctly.

Verify that the card is seated in the PCI slot properly.

Make sure the network cable is connected properly at both ends.

Is a known good cable being used?

Check the XFP module and replace if necessary.

Has an IP address been assigned to the interface?

To check the status of the connection, right-click on the NetXen 10 Gb NIC adapter icon above and select properties. This causes the Local Area Connection Properties window to be displayed.

2 To assign an IP address manually, select the Internet Protocol (TCP/IP) entry below and click the Properties tab.



3 In the Internet Protocol (TCP/IP) Properties window, assign the appropriate IP, subnet mask, and default gateway values.

4 Using the ping utility, verify connectivity with another card or switch.

4.4 Linux InstallationThe Linux sources can be installed in either TAR of RPM format depending on the release package you received as described below.

Note that the following procedure is for a new installation.

4.4.1 NIC Driver Installation (TAR format)

Note: Root level permissions are required for the driver installation.

1 Copy the file [version]1 tar.gz to the host system and follow the installation command sequence:

To build, install, and load the driver:

tar xvzf [version].tar.gz

cd [version]

make

make install

(driver module is installed in: /lib/modules/<kernel version>/kernel/drivers/net)

cd driver

1. Where [version] is the numbered release version of the software.



insmod ./nx_nic.ko

Verify that nx_nic driver is loaded:

lsmod | grep -i nx_nic

2 Identify the NetXen Ethernet interface:ethtool –i eth[n]

3 Assign an IP address and bring up the Ethernet interfaceifconfig eth[n] [IPaddress] up

4.4.2 Linux NIC Driver OptionsStep 1 above uses the insmod command. The following optional parameters can be passed to the driver with the modprobe or insmod commands:

insmod nx_nic <parameter> modprobe nx_nic <parameter>

The optional parameters are listed below.

4.4.3 Configuring the NIC on LinuxAssign an IP address to the interface from the command line. For example:

1 Identify the Ethernet interface (eth1, eth2, etc.) and the nx_nic driver:

ethtool -i eth[n]ifconfig -eth[n] -a

2 Assign an IP Address and activate the interface:ifconfig eth[n] downifconfig eth[n] <IP Address> <netmask> up

Table 9 insmod and modprobe Command Optional Parameters

Parameter Description Default Range

tx_desc1

1.Example usage: modprobe nx_nic tx_desc=1024 or insmod nx_nic tx_desc=1024

Sets the size of the command descriptor rings (Tx). 1024 256 - 4096 (must be power or two)

jumbo_desc2

2.Example usage: modprobe nx_nic jumbo_desc=1024 or insmod nx_nic jumbo_desc=1024

Sets the size of the receive ring for jumbo frames. 1024 64 - 4096(must be power or two)

rdesc_1g3

3.Example usage: modprobe nx_nic rdesc_1g=8192 or insmod nx_nic rdesc_1g=8192

Sets the size of the receive ring for normal packets (1 GbE ports).

8192 64 - 32,768

rdesc_10g4

4.Example usage: modprobe nx_nic rdesc_10g=32768 or insmod nx_nic rdesc_10g=32768

Sets the size of the receive ring for normal packets (10 GbE port).

32,768 64 - 32,768



3 Verify network services are running, and ping:service network restartping <IP address>

4.4.4 Configuring Under Linux KudzuAfter powering on the system the Linux Kudzu may report that the NetXen XG Mgmt hardware has been added to the system. If so, please use the Ignore option to continue with the booting of the Linux OS as shown below.

4.5 Uninstalling the SoftwareThis section describes how to uninstall the NetXen NIC software from Windows and Linux systems.

4.5.1 Uninstalling Windows Drivers for Both NIC and ChimneyTo uninstall the Windows driver, perform the following steps:

1 Right-click on My Computer and click on Manage. The Computer Management window is displayed.

2 Double click on Device Manager. The list of devices in the computer is displayed on the right side of the window.

3 Locate the NetXen Dual 10G Ethernet Server Adapter under the Network adapters section.

4 Right-click on NetXen 10Gb Ethernet Server Adapter.

5 Click Uninstall to perform the operation.



4.5.2 Uninstalling the NetXen NIC Driver in LinuxNote that if the LSA driver is installed, it must be uninstalled before the NetXen Linux NIC driver is uninstalled. Refer to Section 4.5.3 below for information on how to uninstall the LSA driver.

TAR FormatIn the Linux system, the TAR archive may be removed as follows:

rm -rf [version]

rm -f [version].tar.gz

ifconfig eth[n] downrmmod nx_nic.ko

The driver may be removed from /lib/modules as follows:

rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_nic.ko

4.5.3 Uninstalling the LSA Driver in Linux

TAR FormatNote that the LSA driver must be removed before the Linux NIC driver can be uninstalled as described in Section 4.5.2 above. The removal procedure for the LSA driver is shown below.

rm -rf [version]-lsa

rm -f [version]-lsa.tar.gz

rmmod nx_lsa.koThe driver may be removed from /lib/modules as follows:

rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_lsa.ko


5. Firmware Updates

5 Firmware UpdatesThe NetXen card is shipped with a default firmware version. When a newer version of driver and firmware is available, the card is updated to the latest GA version of the firmware as described in the Release Notes.

Typically the firmware and the driver version should match unless it is stated in the Release Notes that the supported firmware and driver versions in the specific release do not match.

Note that the firmware must be properly Flashed in order for the adapter to function properly. Check with NetXen periodically for firmware updates.

5.1 Using the NetXen Flash UtilityThe NetXen Flash utility can be used to update the firmware. When using the nxflash.exe utility, the following syntax should be used.

nxflash -i NX_NIC [command] (for Windows OS)1

./nxflash -i eth[n] [command] (for Linux OS)

In this command line, the -i indicates the interface command and the NX_NIC specifies the actual interface for Windows. The [n] indicates the number of the Ethernet interface for Linux.

Along with the interface, the available commands are listed in Table 10.

The following are some examples of how to use the nxflash utility:

Update firmware image

Get firmware information

Each of these functions is described in detail below.

1 Update firmware image

1. For Windows, the interface type MUST BE ALL CAPITALS to be recognized as a valid interface.

Table 10 NetXen Flash Utility Commands

Command Definition Description

-i--interface

interface Specifies the interface type [NX_NIC, NX_NIC1, etc.]

--info information Prints firmware version and MAC address.-a

--allFirmware update Complete update of BIOS and firmware.

-r--restore

restore Restores the existing firmware version on the card.

-h--help

help Displays the help menu.

-v--version

version Displays the version information.


5. Firmware Updates

The following files must be present in the directory from where nxflash is executed:

imez_romimage - This is the firmware image for the IBM IMEZZ card.

xfp_ibm_romimage - This is the firmware image for the IBM XFP card.

These files are supplied as part of the software release. To update the firmware, change directory where the rom_image files are located, and type the following:

nxflash -i NX_NIC --all (for Windows)

./nxflash -i eth[n] --all (for Linux)

This above update command displays the current firmware version on the card and the new firmware version to which the card will be updated upon completion. The Flash update takes a few minutes to complete and MUST NOT BE INTERRUPTED.

2 Get Flash related information

The --info option may be used to get information from the Flash. e.g.

nxflash -i NX_NIC --info (for Windows)

./nxflash -i eth[n] --info (for Linux)

The following is an example of a Windows output:

C:\3.4.216\utils>nxflash.exe -i NX_NIC --infoVerifying flash...Done.Board Type : XGb IMEZBoard Chip Rev : C1Serial Id : MZ76MK0238Firmware version : 3.4.216BIOS version : 1.0.2MAC Addr 0 : 00:0E:1E:00:1C:F6MAC Addr 1 : 00:0E:1E:00:1C:F7Subsystem ID : 0x00044040

5.2 Windows Firmware UpdateThe following steps are used for updating the Windows OS firmware.

1 Open Command window.

2 Go to the utils directory in the driver package provided.

3 The following binaries are required to be in the same directory from where the Flash command will be executed:

nxflash.exe imez_romimage xfp_ibm_romimage

4 Run the following command:nxflash -i NX_NIC --all

Please do not interrupt or reboot during this time.

5 Reboot the system.


5. Firmware Updates

Verify the firmware has been updated using the following command.

nxflash -i NX_NIC --info

5.3 Linux Firmware UpdatesThe following steps are used for updating the Linux OS firmware.

1 Using the existing firmware and driver on the host, bring the Ethernet interface up using the following command:

Ifconfig eth[n] <IP addr> netmask <netmask> up

2 Update to the new firmware from the bin directory:cd <new release version directory>/bin./nxflash -i eth[n] --all

3 Bring down the Ethernet interface.ifconfig eth[n] down

4 Remove the existing or old driver module.rmmod nx_nic

5 Build and install the new driver.cd <new release version directory>makemake install

6 Reboot the host system.

7 Load the new driver.cd <new release version directory>/driverinsmod ./nx_nic.ko

5.4 Updating the Flash on Multiple Standalone Cards in WindowsThe Flash can be updated on multiple cards on the same host using one of the following procedures.

1 If the updates are being done from the same directory tree, the following commands must be run sequentially after the Flash update has completed.

nxflash.exe -i NX_NIC0 --allnxflash.exe -i NX_NIC1 --all

2 Updates can be done simultaneously from two different directory trees. Note that the following two commands are being executed from two separate directory trees.

c:\[version]-a\utils\ nxflash.exe -i NX_NIC0 --allc:\[version]-b\utils\ nxflash.exe -i NX_NIC1 --all


6. Linux Socket Accelerator (LSA)

6 Linux Socket Accelerator (LSA)The purpose of LSA is to allow the user to offload a significant amount of TCP network processing on to the Netxen NIC, significantly reducing host CPU utilization. Note that the LSA does not require the user to rebuild the Linux kernel, as simple loadable modules are used.

As described below, the user has very fine-grained control over what can be offloaded. Offload decision can be based on application name, local TCP port number, or remote TCP port number.

6.1 LSA Driver InstallationThis section describes the LSA installation instructions for the tar and RPM formats.

NOTE: The LSA Intercept driver must be running at all times for the LSA driver to work. Whenever the server is rebooted, verify the NIC and intercept drivers are running (by using the lsmod command) before using the LSA driver. The LSA intercept driver cannot be removed with the command rmmod; Remove the driver file from /lib/modules/<kernel>/drivers/net/, and reboot the host.

6.1.1 TAR FormatThis section describes the LSA driver installation from the tar package.

1 First Load the NIC driver (nx_nic.ko)

2 Build and load the Intercept and LSA driver

% tar xvzf <version>-lsa.tar.gz

% cd <version>-lsa

% make

% make install

% insmod./driver/nx_intercept.ko

% insmod./driver/nx_lsa.ko

3 Use offload rules to offload the applications

4 Use rmmod nx_lsa to remove the LSA driver

6.2 LSA Offload ConfigurationLSA can be configured via the nxoffload utility. This utility works by maintaining a list of rules that dictate which applications to offload. Each rule can take one of the following parameters:

Application name

Local TCP port number

Local and Remote TCP port number



6.2.1 LSA Configuration OptionsA list of parameters to this utility can be obtained by typing ./nxoffload -help

usage: ./nxoffload <options>

The options are listed below.

6.2.2 LSA Configuration Examples

To offload the iperf application, type the following: ./nxoffload -a -n iperf

To offload the application that uses local TCP port 80, type the following: ./nxoffload -a -p80

To remove the application that uses local TCP port 80, type the following: ./nxoffload -r -p80

To list the current offload rules, type the following: ./nxoffload -s

To offload an application with zero copy enabled, type the following: ./nxoffload -a -z -n <application-name>.

Zero copy is a way to significantly reduce CPU utilization in the send path for some applications. Not all applications may benefit from using zero copy, and some reduction in throughput may be observed. Applications that do well with zero copy are those that have large write sizes and a large number of individual connections.

6.2.3 nxoffload Caveatsnxoffload has the following caveats:

To add or remove offload rules, nxoffload requires root access

Rules may be added or removed at any time. The addition or removal of a rule, however, will not affect connections that have already been established. For example,

Table 11 LSA Configuration Options

<option> Definition Comments

-p Port number to be offloaded.-n Application name to be offloaded.-t TCP tuple to be offloaded. Format: local IP, local port, remote

OP, remove port. Example: ./nxoffload -a -t'0, 0, 0, 5001'./nxoffload -a -t'0, 2000, 0, 5001'

-a Add an offload rule.-r Remove and offload rule.-z Enable zero copy.-s List all ports and applications offloaded.



if the user starts a non-offloaded iperf application and then types ./nxoffload -a -n iperf, this instantiation of iperf will NOT be subsequently offloaded. Future instantiations, however, will be offloaded.

6.3 Running LSAOnce the appropriate rule has been entered into nxoffload, no further user intervention is required. The offload occurs automatically. The offload statistics can be monitored by reading the file /proc/net/nx_nic/lsa_x/stats, where 'x' is NetXen card number.

6.4 LSA StatisticsTable 12 lists the different types of LSA statistics, along with the current support status for each type.

Table 12 LSA Statistics

Name Description Status

TCP StatisticsTcpMaxConn Total number of connections since initial power up. Not supported.

TcpActiveOpens Total number of connections opened via connect. Supported.TcpPassiveOpens Total number of connections opened via accept. Not supported.TcpAttemptFails Number of unsuccessful connection attempts. Supported.TcpEstabResets Resets received by connections in the established

state.Supported.

TcpCurrEstab Number of connections currently in the established state.

Supported.

TcpInSegs Number of received segments. Supported.TcpOutSegs Number of transmitted segments. Supported.

TcpSlowOutSegs Number of transmit segments processed by the slow firmware path.

Supported.

TcpRetransSegs Number of retransmitted Supported.TcpInErrs Number of illegal received segments. Supported.

TcpOutRsts Number of transmitted resets. Supported.TcpOutCollapsed Number of collapsed transmitted segments. A

collapsed segment is generated when smaller segments are combined to form a single large segment

Supported.

IP StatisticsIpInReceives The total number of input datagrams received from

interfaces, including those received in error.Supported.

IpInHdrErrors The number of input datagrams discarded due to errors in their IP headers, including bad checksums, version number mismatch, other format errors, time-to-live exceeded, errors discovered in processing their IP options, etc.

Supported.



IpInAddrErrors The number of input datagrams discarded because the IP address in their IP header's destination field was not a valid address to be received at this entity. This count. includes invalid addresses (e.g., 0.0.0.0) and addresses of unsupported Classes (e.g., Class E). For entities which are not IP routers and therefore do not forward datagrams, this counter includes datagrams discarded because the destination address was not a local address.

Not supported.

IpInNoRoutes (Not in ip-mib)

Number of input datagrams discarded because there was a problem with either determination of inbound interface or a problem with determination of a route back to the source of the datagram.

Supported.

IpInDiscards The number of input IP datagrams for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space). Note that this counter does not include any datagrams discarded while awaiting re-assembly.

Not supported.

IpInDelivers The total number of input datagrams successfully delivered to IP user-protocols (including ICMP).

Supported.

IpOutRequests The total number of IP datagrams which local IP user-protocols (including ICMP) supplied to IP in requests for transmission. Note that this counter does not include any datagrams counted in ipForwDatagrams.

Supported.

IpOutDiscards The number of output IP datagrams for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space). Note that this counter would include datagrams counted in ipForwDatagrams if any such packets met this (discretionary) discard criterion.

Supported.

IpOutNoRoutes The number of IP datagrams discarded because no route could be found to transmit them to their destination. Note that this counter includes any packets counted in ipForwDatagrams which meet this ̀ no-route' criterion. Note that this includes any datagrams which a host cannot route because all of its default routers are down.

Supported.

IpReasmTimeout The maximum number of seconds which received fragments are held while they are awaiting reassembly at this entity.

Not Supported.

IpReasmReqds The number of IP fragments received which needed to be reassembled at this entity.

Not Supported.

IpReasmOKs The number of IP datagrams successfully re-assembled.

Not Supported.

Table 12 LSA Statistics (Continued)




6.4.1 Example Read OutThe following is a sample of the TCP and IP statistics displayed when the stats command is executed.

cat /proc/net/nx_nic/lsa_1/stats

IpReasmFails The number of failures detected by the IP re-assembly algorithm (for whatever reason: timed out, errors, etc). Note that this is not necessarily a count of discarded IP fragments since some algorithms (notably the algorithm in RFC 815) can lose track of the number of fragments by combining them as they are received.

Not Supported.

IpFragOKs The number of IP datagrams that have been successfully fragmented at this entity.

Not Supported.

IpFragFails The number of IP datagrams that have been discarded because they needed to be fragmented at this entity but could not be, e.g., because their Don't Fragment flag was set.

Not Supported.

IpFragCreates The number of IP datagram fragments that have been generated as a result of fragmentation at this entity.

Not Supported.

TCP Statistics Port 0 Port 1 Port 2 Port 3

TcpMaxConn 0 0 0 0

TcpActiveOpens 180 0 0 0

TcpPassiveOpens 0 0 0 0

TcpAttemptFails 0 0 0 0

TcpEstabResets 0 0 0 0

TcpCurrEstab 0 0 0 0

TcpInSegs 54593130 0 0 0

TcpOutSegs 173931132 0 0 0

TcpSlowOutSegs 2690 0 0 0

TcpRetransSegs 743 0 0 0

TcpInErrs 0 0 0 0

TcpOutRsts 0 0 0 0

TcpOutCollapsed 0 0 0 0

TcpTimeWaitConns 0 0 0 0

Table 12 LSA Statistics (Continued)




IP Statistic Value

IpInReceives 54593098

IpInHdrErrors 0

IpInAddrErrors 0

IpInNoRoutes 0

IpInDiscards 0

IpInDelivers 54593130

IpOutRequests 173933823

IpOutDiscards 0

IpOutNoRoutes 0

IpReasmTimeout 0

IpReasmReqds 0

IpReasmOKs 0

IpReasmFails 0

IpFragOKs 0

IpFragFails 0

IpFragCreates 0


7. Teaming Driver

7 Teaming DriverThis section discusses the installation and configuration options for Windows and Linux.

7.1 Windows

7.1.1 InstallationOnce the Windows driver has been installed and the firmware updated, the IM (Intermediate driver) can be installed using the following procedure:

1 Install the Microsoft Visual C++ Distributable package (x86 or x64) -- vcredist_x86.exe or vcredist_x64.exe

The software in included in the NetXen driver release package, or it can be downloaded directly from Microsoft.

Microsoft Visual C++ Redistributable Package. The Microsoft Visual C++ 2005 Redistributable Package (x86) installs run time components of Visual C++ Libraries required to run applications developed with Visual C++ on a computer that does not have Visual C++ 2005 installed. This package installs run time components of C Run Time (CRT), Standard C++, ATL, MFC, OpenMP and MSDIA libraries

2 Select and open "Network Connections" window. Right Click on NetXen NIC Adapter icon to open the NetXen Properties window.

3 In the Local Area Connection Properties window, select Network Load Balancing from the list and click Install, then click OK.


7. Teaming Driver

4 The Select Network Component Type window is displayed.

5 Select Protocol and click the Add button. This causes the Select Network Protocol window to be displayed as shown below.

6 Click on the Have Disk button in the Select Network Protocol window above. This displays the Install From Disk window as shown below.

C:\NetXen\3.4.216\im\x64


7. Teaming Driver

7 In the Install From Disk window, open and select the NetXen IM subdirectory containing the nxteam driver as shown and click OK. The Select Network Protocol window is again displayed with the information shown below.

8 Select NetXen Network Configuration Utility 7 and click OK. After the driver is installed, Windows should be rebooted to complete the installation of the NetXen Network Configuration Utility.

7.1.2 Configuration

1 The icon for Launch NetXen Network is located on the toolbar icon tray. Double-click this icon to launch the Network Configuration GUI. Alternatively, you can open the NetXen NIC adapter properties, and click the Properties button to launch the NetXen Network Configuration Utility GUI to configure teaming. Either of these procedures causes the Local Area Connection 20 Properties window to be displayed as shown below.


7. Teaming Driver

2 Single click on the NetXen Network Configuration Utility 7 entry, then click Properties to display the NetXen Network Configuration 7 Properties window shown below.

The following window illustrates how to highlight the entries to ‘Team’ the appropriate adapters together.


7. Teaming Driver

Once the appropriate entries are highlighted, click the Team button above to team the adapters together. An example is shown below.

3 The NetXen Network Configuration Utility 7 Properties window contains all the configuration options to setup the teaming properties for the Intelligent NIC and its ports. Double-click on the NetXen Network Team #1 entry to display the Team Properties window shown below.


7. Teaming Driver

4 In the Team Properties window, click on the Information tab to display general information about the board as shown below.

5 After clicking OK in the Team Properties window above and OK in the NetXen Configuration Utility 7 Properties window, the user must click on the Close button in the Local Area Connections Properties window for the teaming to take effect.


7. Teaming Driver

Click on Continue Anyway as shown in the window below to permit the completion of the teaming setup.

6 During the teaming setup, the following window is displayed. The time required to configure the adapters varies depending on the system configuration.

7 Once the adapter configuration is complete, the click OK in the following windows.


7. Teaming Driver

7.1.3 Fault Tolerance OptionsThe NetXen adapter provides three fault tolerant options:

Manual: This setting allows the user to select between the Primary NIC and secondary NIC options.

Fail on Fault: Should the primary NIC fail, this option enables the system to automatically switch from the primary NIC to the secondary NIC.

Smart Switch: This option allows the user to designate a certain ‘team’ member as the preferred NIC. This team member is always used as long as it is enabled. However, if the preferred NIC fails, and then some time later is brought back on-line, the system automatically switches back to it since it is the preferred NIC. Smart Switch is the recommended choice for fault-tolerant applications.

Manual ModeManual Mode allows the user to switch between the primary and secondary adapters. Once the teaming control has been installed and configured as described in Sections 5.1.1 and 5.1.2 above, manual mode can be set as described below.

1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. Note that the Switch Now option is disabled until Manual is selected and OK is clicked.


7. Teaming Driver

2 After clicking OK in the above window, the following window is displayed. Perform a switch of primary adapter by clicking "Switch Now".

3 Finally, click on OK to complete the manual switch process.

Fail on FaultThe Fail on Fault mode causes the system to automatically switch to the secondary adapter when a failure occurs on the primary adapter.

1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. This step selects the primary adapter.


7. Teaming Driver

In the Teaming Properties windows, click on Fail on Fault as shown below, then click Change Mode, then click OK.

Smart SwitchThis option allows the user to designate a certain ‘team’ member as the preferred NIC. This team member is always used as long as it is enabled. However, if the preferred NIC fails, and then some time later is brought back on-line, the system automatically switches back to it since it is the preferred NIC. Smart Switch is the recommended choice for fault-tolerant applications.

A Smart Switch from the Current Primary Adapter [13] to the secondary adapter [14] is performed as follows:

1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. This step selects the primary adapter.


7. Teaming Driver

2 In the Team Properties window, click on Smart Switch as shown below, then click OK.

3 Click OK on the NetXen Network Configuration Utility 7 Properties


7. Teaming Driver

4 Click Close on the Local Area Connection Properties.

7.2 Linux Teaming/Bonding ModeThis section describes the Linux teaming/bonding mode. The steps for setting this mode are as follows:

1 cd <release>

2 insmod ./driver/nx_nic.ko

3 insmod bonding mode=1 miimon=100 updelay=50000 primary=eth16

4 ifconfig bond0 25.145.1.199 netmask 255.255.255.0 broadcast 25.145.1.255 up

5 ifenslave bond0 eth15 eth16

To bring down the bond please use: ifconfig bond0 down

Note: These commands and steps have been tested under RHEL5, but for other Linux distributions and versions there may be differences in how bonding is enabled and utilized.

7.2.1 Mode 1active-backup or mode=1

Active-backup policy: Only one slave in the bond is active.

A different slave becomes active if, and only if, the active slave fails.

By setting mode=1, when ifenslave is executed the bonding driver calls into

bond_enslave() and sets the slave link modes for eth15 and

eth16 to BOND_LINK_DOWN.


7. Teaming Driver

Since the bond mode is 1, ie. BOND_MODE_ACTIVEBACKUP, and primary device is indicated, one slave device is configured as active slave and dmesg will show "bonding: bond0: making interface eth16 the new active one 0 ms earlier." and this will result in bond->current_slave being updated and

MII Status: up will be reported for this link and this is what cat /proc/net/bonding/bond0 prints. For the other port, dmesg will display:

bonding: bond0: link status up for interface eth15, enabling it in 50000 ms.

and 'cat /proc/net/bonding/bond0' will print for eth15, MII Status: down;

After 50 seconds dmesg will print:

bonding: bond0: link status definitely up for interface eth15.

bonding: bond0: making interface eth15 the new active one.

'cat /proc/net/bonding/bond0' will print MII Status: up for eth15.

7.2.2 Mode 0If the mode is set to 0 i.e. not BOND_MODE_ACTIVEBACKUP, (in this case Round Robin)

insmod bonding mode=0 miimon=100 updelay=50000 primary=eth16

then both links delay for 50 secs before their states are set to BOND_LINK_UP state,

blade12$ cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0

Slave Interface: eth16

MII Status: down <========= down state

Link Failure Count: 0

Permanent HW addr: 00:0e:1e:00:11:d2


MII Status: down <========= down state



after 50 secs:

blade12$ cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0


7. Teaming Driver


MII Status: up <=========== up state




MII Status: up <=========== up state



And dmesg will print:



7.2.3 Bonding Driver OptionsOptions for the bonding driver are supplied as parameters to the bonding module at load time. They may be given as command line arguments to the insmod or modprobe command, but are usually specified in either the /etc/modules.conf or /etc/modprobe.conf configuration file, or in a distro-specific configuration file (some of which are detailed in the next section).

7.2.4 Bonding Driver ParametersThe available bonding driver parameters are listed below. If a parameter is not specified the default value is used. When initially configuring a bond, it is recommended "tail -f /var/log/messages" be run in a separate window to watch for bonding driver error messages.

It is critical that either the miimon or arp_interval and arp_ip_target parameters be specified, otherwise serious network degradation will occur during link failures. Very few devices do not support at least miimon, so there is really no reason not to use it.

Options with textual values will accept either the text name or, for backwards compatibility, the option value. E.g., "mode=802.3ad" and "mode=4" set the same mode.

The parameters are as follows:

arp_interval

Specifies the ARP link monitoring frequency in milliseconds. If ARP monitoring is used in an etherchannel compatible mode (modes 0 and 2), the switch should be configured in a mode that evenly distributes packets across all links.


7. Teaming Driver

If the switch is configured to distribute the packets in an XOR fashion, all replies from the ARP targets will be received on the same link which could cause the other team members to fail. ARP monitoring should not be used in conjunction with miimon. A value of 0 disables ARP monitoring. The default value is 0.

arp_ip_target

Specifies the IP addresses to use as ARP monitoring peers when arp_interval is > 0. These are the targets of the ARP request sent to determine the health of the link to the targets.

Specify these values in ddd.ddd.ddd.ddd format. Multiple IP addresses must be separated by a comma. At least one IP address must be given for ARP monitoring to function. The maximum number of targets that can be specified is 16. The default value is no IP addresses.

downdelay

Specifies the time, in milliseconds, to wait before disabling a slave after a link failure has been detected. This option is only valid for the miimon link monitor. The downdelay value should be a multiple of the miimon value. If not, it will be rounded down to the nearest multiple. The default value is 0.

lacp_rate

Option specifying the rate in which we'll ask our link partner to transmit LACPDU packets in 802.3ad mode. Possible values are:

- slow or 0: Request partner to transmit LACPDUs every 30 seconds

- fast or 1: Request partner to transmit LACPDUs every 1 second

- The default is slow.


8. NIC VLAN in Windows

8 NIC VLAN in WindowsThis section describes the NIC VLAN on Windows using Miniport Driver (NIC Driver). The IM Driver GUI can set multiple VLAN IDs, but the NIC Driver property sheet can set only one VLAN-ID. The VLAN decision table shown below describes how the packets are handled by the NIC driver.

By default, when the VLAN-ID is set on the NIC driver property, tagging is enabled. Ensure that the switch port connected to the NIC has tagging enabled.

8.1 Optional Configuration on the NIC and Switch SideThe NIC driver property also allows tagging to be disabled when VLAN-ID is

enabled.

There are 2 separate settings: VLANID and TAGGING.

1 Tagging must be enabled for the NIC to tag packets on either 802.1p or 802.1Q (Priority/VLAN) or both. Tagging is always enabled by default on the NIC. If the stack sends out a packet with 802.1p (priority setting), the NIC adds the VLAN tag of 0 along with this priority and send the packets with tagging.

The NIC will also processes packets with or without tagging and passes the information up to the stack with this setting.

2 VLANID is not really needed and is only useful when setting one VLANID on the NIC and tagging it with that VLAN ID. By default, 0 here means NO VLAN.

8.2 NIC VLAN as Send SideTo generate the components of the packet's tag header a miniport driver:

1 Set the tag header's CFI bit to zero

2 Proceed as indicated in the following table, based on the following values:Configured VLAN ID for the NIC.Priority or VLAN ID in the packet's OOB data.

Table 13 Windows VLAN Configuration Summary

VLAN-ID Tagging Packets Comments

Zero Enabled Untagged This is the behavior with default NIC on installation (If you don't configure it). Allows a user to set and configure multiple VLANs from the Windows IM GUI.

Zero Disabled UntaggedNon-Zero Disabled UntaggedNon-Zero Enabled Tagged



VLAN ID in the packet's tag header.

If the priority value that the miniport driver retrieves from the packet is zero, the miniport driver should not mark the packet with a priority value before transmitting the packet over the network. On Ethernet (802.3) networks, an unmarked packet is the same as a packet that is marked with the "best effort" priority value appended to it. Not marking packets ensures backward compatibility with devices on the network that do not support the tag header.

Note: If a send packet's Ethernet header specifies a protocol type (EtherType) of 802.3ad or GARP VLAN Registration Protocol (GVRP), the miniport driver should transmit the packet without modifying it.

The following table indicates the circumstances under which a miniport driver should not send an 802.1Q-marked packet.

Table 14 Windows NIC VLAN Send Side Action Table

Configured NIC VLAN ID

Priority or VLAN ID in the Packet's OOB Data

Packet Tag HeaderVLAN ID Miniport Driver Action

Zero Priority and VLAN ID are zero.

Any value or not present. Send the packet with no changes.

Zero Priority and VLAN ID are nonzero.

Not present Insert tag header into packet. Set VLAN ID in tag header to the value of VLAN ID in OOB data.Send packet.

Nonzero VLAN ID is zero or matches the configured VLAN ID for the NIC.

Not present Insert tag header into packet. Set VLAN ID in tag header to the value of configured VLAN ID for the NIC.Send packet.

Nonzero VLAN ID matches the configured VLAN ID for the NIC.

Zero Set VLAN ID in tag header to the value of configured VLAN ID for the NIC.Send packet.

Nonzero Not present. Matches the configured VLAN ID for the NIC.

Send the packet with no changes.

Table 15 Miniport Driver Packet Send Conditions


Priority or VLAN ID in Packet's OOB Data

Packet Tag Header VLAN ID Miniport Driver Action

Nonzero VLAN ID does not match configured VLAN ID for NIC.

Any value or not present

Fail the send.



8.3 NIC VLAN as Receive SideThe following table summarizes the actions that a miniport driver should perform on an 802.1Q receive packet.

Nonzero Not present Does not match the configured VLAN ID for the NIC.

Fail the send.

Any value Nonzero Nonzero Fail the send because VLAN ID should not be set in both OOB data and ag header.

Table 16 NIC VLAN as Receive Side

Configured NIC VLAN ID Tag Header in Packet Packet Tag Header

VLAN ID Miniport Driver Action

Zero Yes Any value. Remove priority and VLAN ID from the packet. Insert removed priority and VLAN ID into per-packet information.Indicate the packet.

Zero No Not applicable. Set the priority and VLAN ID in per-packet information to zero Indicate the packet.

Nonzero Yes Matches the configured VLAN ID.

Remove priority and VLAN ID From the packet.Insert removed priority and VLAN ID into per-packet Information.Indicate the packet.

Nonzero Yes Does not match the configured VLAN ID.

Do not indicate the packet.

Nonzero No Not applicable. Do not indicate the packet.

Table 15 Miniport Driver Packet Send Conditions


Priority or VLAN ID in Packet's OOB Data

Packet Tag Header VLAN ID Miniport Driver Action


9. Troubleshooting Guidelines

9 Troubleshooting Guidelines

9.1 Unable to Boot the Host System with the NIC Card InstalledRemove the NIC and reboot. Does the system boot without the NIC?

If so, power down the system, insert the NIC into a different PCIe slot and boot-up the system.

Is the recommended minimum memory installed on the host system?

Note any error message during the OS boot sequence.

Check for system error messages:

Dbgview.exe (Windows event viewer)

dmesg (Linux)

Use the NetXen diagnostics utility for further testing.

9.2 Unable to ping the NIC Installed on the HostType lspci -v and look for vendor ID ‘4040’.

- Is the NIC visible on the PCI bus?

- Is the NIC is placed in the PCIe x8 slot?

Check if the driver is loaded correctly and the card is initialized

- dmesg (display system messages on Linux)

- Windows Event Viewer tool

Run the NetXen user diagnostics

Check the NIC XFP module and optical cables; For CX4, check the IB cable

Check Network Configuration

- Check for valid IP Address and subnet

- Check for duplicate IP address and subnet numbers

- If connected via gateway, check gateway address and network path

- Check the NIC MAC address values

- Network link status as seen by the host OS

- Check for LED activity (using ping)

- Using network monitoring tools, analyze the Tx and Rx packets for the NIC

- Verify the ARP entry and response



9.3 Gather System Information and Check for Errors on Linux HostNetXen Product Information

- Card type

- NIC firmware and driver version

Operating System Information

- Windows OS (32 / 64) and Service Pack #

- Linux Distribution and Kernel Release (uname –a)

Host System Information

- CPU, Cores, Chipset – (Run System Information tool)

- BIOS information

- Host memory

- PCIe x8, x4, x1 …

- System setup description

- How many servers, clients etc

- Model of the host system

- Network setup (back to back, router, switch, backplane etc)

Check for error messages from the kernel:

- Type dmesg

- Type cat /var/log/messages

Check routing table, gateway, and IP Address conflicts for all interfaces:

- Type route

Perform a "cat /proc/net/nx_nic/dev0 | grep eth" and "cat /proc/net/nx_nic/dev1 | grep eth" to find the correct "ethx" device name for each of the two ports on the NetXen Mezzanine card.

Check network interface TX and RX counts during pings:

- Type ifconfig <network interface>

Use ethereal, tethereal, or tcpdump utilities to capture packets:

- Analyze the ping packets

9.4 CaveatsJumbo frames of maximum size 8000 are supported.

The current version of the Windows Win2k3 OS does not support MSI in the network drivers (NDIS 5.1). However, hardware support is provided for NDIS versions 5.2 and 6.0.



9.4.1 MSI Interrupts for LinuxThe Linux OS version supports MSI, and is used by the Intelligent NIC driver on the Linux OS.

On the Linux host, type cat /proc/interrupts to check that PCI-MSI is enabled for the Ethernet interface

Determine if the installed Linux kernel has the MSI configuration option enabled (earlier versions of the kernel may require an MSI patch).

Type make menuconfig from the directory /usr/src/kernels/<kernel version> and verify that MSI configuration option is supported/enabled.

NetXen device ID = 4040

$ lspci -d 4040

9.4.2 PCI-Express SlotThe NetXen reference card can be located on the PCI-Express bus using the following Linux procedures.

Determine Vendor and Device IDTo check if the PCIe slot has an x8 link for optimal PCIe performance, first determine the PCI vendor ID and device ID codes using the following command:

% lspci -n

A message similar to the following should be displayed:

06:00.0 0200: 4040:0001 (rev 24)

The NetXen vendor ID is 0x4040. A list of NetXen device ID’s is shown below.

Determine Interface AttributesTo determine the attributes of the interface, execute the following command:

lspci -n -d 4040:0004 -vvv | grep -i width

A message similar to the one below should be displayed.

Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0

Link: Speed 2.5Gb/s, Width x8

Table 17 NetXen Device IDs

Product Device ID Description

NXB-10GXSR 0x0001 10 Gbit Ethernet NIC with XFP optical interface.NXB-10GCX4 0x0002 10 Gbit Ethernet NIC with CX4 copper interface.



Determine Width ParametersCheck if the lspci output "negotiated width" (the second width in the output) reflects the actual "supported width". In an XFP system for example, execute the following command to determine if these two widths are the same:

% lspci -n -d 4040:0004 -xxx

where 4040 indicates the NetXen vendor ID and 0004 indicates an IBM IMezz card device ID. An ID of 0001 indicates a NetXen XFP card.

Examine the following hex output:

From the lspci hex output:

d0: 10 00 01 00 02 80 00 00 40 20 01 00 81 04 00 00 /\

||_________ capability. "81 04" = x8

e0: 01 00 81 00 00 00 00 00 00 00 00 00 00 00 00 00 /\

||_________ negotiated. "81 04" = x8. "41 04" = x4, "11 04" = x1, "01

05" = x16

Note that some PCI-e chip sets with a x16 link may not support a x8 PCI-Express link width. A link is only required to support its maximum width and x1. Check the motherboard specification to determine if its PCI-e x16 slot can autonegotiate to x8.

On some motherboards, the PCI-e slots are all the same physical size required by a x16 slot, but electrically they are different widths. Check the motherboard specification for the supported width of each of the PCI-e slots.

9.5 Additional SupportIf problems are encountered with server applications when either Chimney or RSS are enabled under Microsoft Win2K3 with SP2 (in Offload Mode), please follow the link below to see if Microsoft has addressed the specific issue.

http://support.microsoft.com/kb/936594

...... 81 04 00 00 01 00 81

Supported link width

Negotiated link width


10. User Diagnostics

10 User DiagnosticsThe user diagnostics can be invoked in Windows and Linux using the following command:

nxudiag.exe -i NX_NIC (Windows)

[version]/bin/nxudiag -i eth[n] (Linux)

NOTE: If the eth[n] number is not known, perform a "cat /proc/net/nx_nic/dev0 | grep eth" and "cat /proc/net/nx_nic/dev1 | grep eth" to find the correct "ethx" device name for each of the two ports on the IBM NetXen Mezzanine card.

10.1 Windows Tests The Windows user diagnostics utility performs the following tests using the nxudiag.exe command.

Table 18 NetXen Windows Diagnostic Utility Commands

Command Description

-i--interface

Specifies the interface type [NX_NIC, NX_NIC1, etc.]

-a--all

Perform all test, regardless of default.

-n--nodefault

No defaults, all tests will be set implicitly.

-R--CRegs

Test all control registers.

-F-Flash

Test the Flash.

-I--IRQS

Test interrupt mechanism.

-L--IntLB

Internal loopback test

-S--LinkST

Link status test.

-I--LED

LED test.

-nR--noCRegs

No control registers test.

-nF--noFLASH

No Flash test.

-nl--noIRQS

No interrupt test.

-nL--noIntLB

No internal loopback test.

-nS--noLinkSt

No Link status test.



10.2 Linux Tests The Linux user diagnostics utility performs the following tests using the nxudiag.exe -i eth[n] command.

-nl--noLED

No LED status test.

-E--ExtLB

External full duplex loop-back test.

-h--help

Prints the help message.

Table 19 NetXen Linux Diagnostic Utility Commands

Command Description

-i--interface

Specifies the interface type [eth1, eth2, etc]

-a--all

Perform all test, regardless of default.

-n--nodefault

No defaults, all tests will be set implicitly.

-s--silent

Silent mode. No output to console.

-v--verbose

Verbose mode. Output result to console.

-R--CRegs

Test all control registers.

-F-Flash

Test the Flash.

-I--IRQS

Test interrupt mechanism.

-L--IntLB

Internal loopback test

-H--Hw

Hardware test.

-S--LinkST

Link status test.

-nR--noCRegs

No control registers test.

-nF--noFLASH

No Flash test.

-nl--noIRQS

No interrupt test.

Table 18 NetXen Windows Diagnostic Utility Commands

Command Description



-nL--noIntLB

No internal loopback test.

-nH--noHw

No hardware test.

-nS--noLinkSt

No Link status test.

-E--ExtLB

External full duplex loop-back test.

-C--Cmdif

Command interface status test.

-h--help

Prints the help message.

Table 19 NetXen Linux Diagnostic Utility Commands

Command Description


Appendix A. Tuning Guidelines

Appendix A Tuning Guidelines

A.1 Windows Performance Tools

A.1.1 NTttcp (Windows)There are two switches on ntttcp that boost performance: -a and -rb.

Running ntttcp without an optimal value for the -a switch and without the -rb switch is acceptable for the Gigabit NIC product.

In the 10G NIC product, however, and in particular with fewer threads on Windows, performance is limited by the Receive window for the most part and by the outstanding I/O (-a) value, regardless of hardware limitations.

For receive, an optimal value is 16 for -a and 512000 for -rb (or something similar). With these values, a single thread should perform closer to a multi-thread configuration.

ntttcpr -m 16,0,<rcv ipaddr> -n <# of tcp segs for snd/rcv> -a 16

For transmit, -a should be a value less than 8. The more threads, the lower the number should be.

Note: The send and receive side NTTTCP commands must use the receive side’s IP address.

Steps for NIC Performance Testing using NTTTCP on Windows

1 Install Win2k3 SP2

2 Enable Windows Scaling

Set TCp1323Opts parameter value to 1, if the parameter does not exist, create it:

"regedit" to HKLM\system\currentcontrolset\services\Tcpip\Parameters

3 The drivers property sheet default is 1500. For Jumbo, set the frame size to 8000.

Table 20 ntttcpr Command Options

Command Optionsntttcpr Description

-l Message size in bytes. Default is 64K.-m Number of threads.0 CPU number.IP

AddressReceive IP address.

(Send side also uses receive side's IP Address).-n Number of TCP segments.-a Number of requests in parallel.

(Send and receive side values must match).-rb Receive window size on the receiver.



Unidirectional: One Server to One ClientReceiver:

ntttcpr.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -fr -n 200000

Sender:ntttcps.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -n 200000

The above commands can be used in both Chimney and non-Chimney modes. With Chimney, enabled performance is higher and CPU utilization is also reduced.

Unidirectional: One Server to One Client — Single Thread, Multiple ProcessorsIf the system has more than one CPU, the load can be distributed across two CPUs and port numbers.

Send Side

ntttcps -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5001ntttcps -m 1,1,192.168.20.2 -n 300000 -a 6 -p 5002

Receive Side

ntttcpr -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5001ntttcpr -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5002

A.1.2 nttcps and ntttcpr Command Line Options

C:\NTTTCP_NEW>ntttcps <parameters>orC:\NTTTCP_NEW>ntttcpr <parameters>

Version 2.6ntttcpr.exe: [-l|-n|-p|-a|-x|-rb|-sb|-i|-f|-u|-w|-d|-t|-v|-6|-fr|-mb] -m <mapping> [mapping]

-l <Length of buffer> [default: 64K] -n <Number of buffers> [default: 20K] -p <Port base> [default: 5001] -a [outstanding I/O] [default: 2] -x [PacketArray size] [default: 1] -rb <Receive buffer size> [default: 64K] -sb <Send buffer size> [default: 0] -i Infinite Loop [Only UDP mode] -f <File Name> [default: output.txt] -u UDP send/recv -w WSARecv/WSASend -d Verify Flag -t <Runtime> in seconds -v enable verbose mode -6 enable IPv6 mode



-fr Full buffers posted on reads -mb Multiple buffer post mode -m <mapping> [mapping] where a mapping is a session(s),processor,receiver IP set e.g. -m 4,0,1.2.3.4 sets up: 4 sessions on processor 0 to test a network on 1.2.3.4

A.2 Windows Optimization — Tcp1323OptsThis parameter determines whether TCP uses the time-stamping and window scaling features described in RFC 1323, TCP Extensions for High Performance.

Window scaling permits TCP to negotiate a scaling factor for the TCP receive window size, allowing for a very large TCP receive window of up to 1 Gbyte. The TCP receive window is the amount of data the sending host can send at one time on a connection.

To set the window scaling and timestamp parameters, click Start -> Run, then type regedit to start the registry editor. Open the following location:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Window ScalingFor more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes.

Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte.

When the value for window size is added to the registry and its size is larger than the default value, Windows attempts to use a scale value that accommodates the new window size.

Data Type Range Recommended Value

REG_DWORD 0 | 1 | 2 | 3 1

Hex Value Meaning

0 Timestamps and window scaling are disabled. 1 Window scaling is enabled. 2 Timestamps are enabled. 3 Timestamps and window scaling are enabled.



A.3 Linux Performance ToolsOnce connectivity between the two cards is established, common performance tools can be used for performance measurements.

Note: Various factors including tools, kernel, stacks, and related parameters affect the performance of a system. A discussion on performance criteria is beyond the scope for the getting started section.

Add the following for performance tuning on Linux systems under --

/etc/sysctl.conf

A.3.1 IPERFThe -P option determines the number of threads; default is one.

Server Side:

iperf -s -w256k -l 56k

Client Side:

iperf -c <ipAddr of Server> -w256k -P12 -l 56k

A.4 Linux OptimizationWhile optimization of the Linux operating system is a topic beyond the scope of this guide, below are a few common guidelines.

PCIe x8 provides optimal performance.

MSI interrupts are preferred over legacy interrupts.

Linux Kernel versions 2.6.15 or later provide better performance optimization for the NIC.

Check CPU utilization on the host for CPU-bound bottlenecks.

net.core.rmem_max = 16777216net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216

net.core.netdev_max_backlog = 250000

net.ipv4.tcp_timestamps = 0net.ipv4.tcp_sack = 1net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216net.ipv4.tcp_low_latency = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_moderate_rcvbuf = 1net.ipv4.route.flush = 1

net.core.optmem_max = 524287



The tunable performance parameters are system and application specific. Understanding the relationship between the parameter values, target workload, performance applications, OS and system configuration provides a better understanding of performance optimization and bottlenecks.


Appendix B. Glossary

Appendix B Glossary

Name Description

Bonding Used in a system with multiple ports to implement fault tolerance in Linux.

dbgview.exe Debug viewer for Windows

dmesg Linux system messages

ethereal Packet analyzer

ethtool Linux Ethernet tool for card settings. See man ethtool in Linux

imezz_romimage NetXen firmware image for IBM Mezzanine card.

iperf Linux performance tool

LKM Linux Kernel Module

Load balancing Allows the load to be balanced between 2 or more ports in a system.

LSA Linux Socket Accelerator

NIC Network Interface Card

nxflash NetXen Flash utility

nx_intercept.ko Linux LSA intercept driver

nx_lsa.ko Linux LSA driver

nx_nic.ko Linux NIC driver

nxp2nic.sys Windows NIC driver

nxudiag NetXen user diagnostic utility

Teaming Used in a system with multiple ports to implement fault tolerance in Windows.

xfp_romimage NetXen firmware image for an XFP card


Appendix B. Glossary


IBM Installation Tuning Guide -...

Documents

Transcript of IBM Installation Tuning Guide -...