Microsoft Windows 2016 Mellanox 100GbE NIC Tuning...
Transcript of Microsoft Windows 2016 Mellanox 100GbE NIC Tuning...
Microsoft® Windows® 2016 Mellanox
100GbE NIC Tuning Guide
Publication # 56288 Revision: 1.00 Issue Date: June 2018
Advanced Micro Devices
© 2018 Advanced Micro Devices, Inc. All rights reserved.
The information contained herein is for informational purposes only, and is subject to change without notice. While
every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions
and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced
Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the
contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement,
merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software
or other products described herein. No license, including implied or arising by estoppel, to any intellectual property
rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are
as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.
Trademarks
AMD, the AMD Arrow logo, EPYC, and combinations thereof, are trademarks of Advanced Micro Devices, Inc.
Other product names used in this publication are for identification purposes only and may be trademarks of their
respective companies.
Microsoft and Windows are registered trademarks of Microsoft Corporation.
PCIe is a registered trademark of PCI-SIG corporation.
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
Revision History 3
Revision History
Date Revision Description
June 2018 1.00 Initial public release.
Microsoft® Windows® 2016 Mellanox 100GbE NIC
Tuning Guide
56288 Rev. 1.00 June 2018
4
Contents
Overview ......................................................................................................................................... 5
Initial Test Systems Preparation ..................................................................................................... 5
Download and Install the Mellanox Driver .................................................................................... 5
Initial Steps ..................................................................................................................................... 6
Determine the NUMA Node That the Mellanox Card is Installed On .......................................... 7
Find the Index Value of the Network Interface .............................................................................. 8
Adding Registry Values .................................................................................................................. 9
Changing Advanced Settings in the Driver .................................................................................. 10
Test Optimizations ........................................................................................................................ 13
Conclusion .................................................................................................................................... 13
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
5
Overview
This guide is focused on tuning Mellanox ConnectX-4 cards and Windows® 2016 for high
bandwidth runs. First, the latest Mellanox driver will be installed, and then the driver is tuned.
Tuning the driver results in a significant performance improvement as we optimize the operating
system.
Initial Test Systems Preparation
1. Both systems should be configured as follows:
a. Hardware
i. AMD EPYC™ 7601 for best CPU performance.
ii. One DIMM per channel using the highest supported memory
configuration by your platform provider.
iii. ConnectX-4 in a 16-bit PCIe® slot running at PCIe Gen 3 (8GT/s) – one of
the steps below will show how to verify this
iv. Both ConnectX-4s directly connected and configured with IP addresses.
b. BIOS
i. C-States disabled (see next step)
c. Set the operating system power plan option found in the control panel to high
performance.
Download and Install the Mellanox Driver
1. Download the Mellanox driver from:
http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers
You will need the latest WinOF-2 Driver version.
2. Double-click on the downloaded file and accept all the defaults. It may update the firmware
if the ConnectX-4 card that is installed is behind on its firmware version.
3. After it completes reboot the machine.
Microsoft® Windows® 2016 Mellanox 100GbE NIC
Tuning Guide
56288 Rev. 1.00 June 2018
6
Initial Steps
1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this
example it is the ConnectX-4 #3.
2. Right-click on the card, select Properties, then select the Information tab.
3. Make sure the card negotiated to PCIe 8.0 GT/s x16 and it negotiated to 100.0 Gbps.
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
7
Determine the NUMA Node That the Mellanox Card is
Installed On
1. Go to Device Manager and select the Mellanox ConnectX-4 that you wish to tune, in this
example it will be the Mellanox ConnectX-4 Adapter #3:
2. Right click on the selected card, go to Properties, then select the Details tab.
3. From there click the Property drop down and press the letter “N” which will then display the
NUMA node property. Document the NUMA node that the card is attached to. This
example shows NUMA node 6. This number is dependent on the PCIe slot the ConnectX-4
is installed in.
Microsoft® Windows® 2016 Mellanox 100GbE NIC
Tuning Guide
56288 Rev. 1.00 June 2018
8
Find the Index Value of the Network Interface
1. Open Device Manager
2. Right click on the selected card, go to Properties and then select the Details tab.
3. Select the Driver key and document the value.
4. The driver key value in your system may be slightly different from the example shown.
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
9
Adding Registry Values
1. Adding these registry values is necessary with Mellanox v1.80 driver. The newer driver
v1.90 (and newer) automatically sets the registry values when installed
2. Add the registry values to the following key:
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-
08002be10318}\0006\
The part in bold text, 0006, will be specific to your machine.
3. Right-click on 0006 and then add the following new string values:
RscMaxPacketSize – 16384
RfdBufferSize - 16384
RscTimeoutPeriodUsecs – 64
Microsoft® Windows® 2016 Mellanox 100GbE NIC
Tuning Guide
56288 Rev. 1.00 June 2018
10
4. This example shows 0004. Select the folder which corresponds to driver key identified
earlier. In our case, it was 0006. The example photo shows 0004. It will likely be different
in your case.
Changing Advanced Settings in the Driver
1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this
example it is Mellanox ConnectX-4 Adapter #3.
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
11
2. Right-click on the selected card, go to Properties, then select the Advanced tab.
3. Change "Receive Buffers" from Advanced property page to 4096.
Microsoft® Windows® 2016 Mellanox 100GbE NIC
Tuning Guide
56288 Rev. 1.00 June 2018
12
4. Change "Receive Completion Method" to Polling in the Advanced property page.
56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE
NIC Tuning Guide
13
Test Optimizations
On the server side, from a command prompt window enter “start /node 2 /affinity 0xAAAA iperf
-s” from the folder in which iperf resides. This example shows Node 2. You should select the
node that corresponds to the NUMA node identified above.
On the TX side run “start /Node 2 /Affinity 0xAAAA iperf -c <server ip> -P 8 -t <number of
seconds>”. Run this for at least three minutes as there is an important step to complete before
iperf completes.
Conclusion
Following these procedures, AMD achieved transfer rates of up to 94.5 Gb/s in our laboratories
using version 2.09 of IPERF.