Sr Iov Guide

8/12/2019 Sr Iov Guide

1/8

SR-IOV

Version 3.0

Technology Preview Guide

Emulex Connects

P006796-01B Rev. A Servers, Storage and People


2/8

SR-IOV Technology Preview Guide

Copyright 201-20121 Emulex. All rights reserved worldwide. No part of this document may be reproduced by any

means or translated to any electronic medium without the prior written consent of Emulex.

Information furnished by Emulex is believed to be accurate and reliable. However, no responsibility is assumed by

Emulex for its use; or for any infringements of patents or other rights of third parties which may result from its use.

No license is granted by implication or otherwise under any patent, copyright or related rights of Emulex.

Emulex, the Emulex logo, AutoPilot Installer, AutoPilot Manager, BlockGuard, Connectivity Continuum,

Convergenomics, Emulex Connect, Emulex Secure, EZPilot, FibreSpy, HBAnyware, InSpeed, LightPulse,

MultiPulse, OneCommand, OneConnect, One Network. One Company., SBOD, SLI, and VEngine are trademarks of

Emulex. All other brand or product names referenced herein are trademarks or registered trademarks of their

respective companies or organizations.

Emulex provides this manual "as is" without any warranty of any kind, either expressed or implied, including but not

limited to the implied warranties of merchantability or fitness for a particular purpose. Emulex may make

improvements and changes to the product described in this manual at any time and without any notice. Emulex

assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties that may

result. Periodic changes are made to information contained herein; although these changes will be incorporated into

new editions of this manual, Emulex disclaims any undertaking to give notice of such changes.

Emulex, 3333 Susan Street

Costa Mesa, CA 92626


3/8

SR-IOV Technology Preview User Guide Page iii

Introduction ........................... ........................... .......................... ............................ ................................ . 1

SR-IOV Advantages ....................... ........................... ............................ ......................... .................. 1

Requirements ....................... ............................ ......................... ............................ ........................... 1

Setting Up SR-IOV .......................... ............................ ........................... .......................... ................... 2

Configuration ......................... ........................... .......................... ........................... .............................. ... 4

MAC Address for Virtual Function Interfaces......................... ........................... ............................ ....... 4

Configuring the MAC Address for the VF.... .......................... ........................... ............................ ..... 4

Configuring VLAN on VF Interfaces Using vconfig..................... ......................... ......................... ..... 4

Transparent VLAN Tagging ........................ .......................... ........................... ........................... 4

Virtual Guest Tagging (VGT) ........................... ............................ ......................... ...................... 5

Configuring Rate Limit on VF Interfaces.................... ............................ ......................... ..................... 5
http://-/?-http://-/?-


4/8

SR-IOV Technology Preview Guide Page 1

Introduction

Emulexoffers Single Root I/O Virtualization (SR-IOV) functionality as a Technology Preview feature.

This guide contains requirements and steps to use SR-IOV in a RHEL 6 K Virtual Machine (KVM)environment. This environment supports capabilities of OneConnect to enable multiple virtual

functions (VFs) for a physical function (PF). Each of these VFs can be assigned to virtual machines

(VMs). A VF enables the guest operating system direct access to OneConnect, such that guest

performance is not limited by the overheads of the hypervisor.

It is important to remember that a Technology Preview feature is not to be used in production

environments and may not be functionally complete. Emulex cannot provide standard support levels

(including rapid response, fixes, troubleshooting, and workarounds). However, Emulex is very interested

in your feedback and early experiences with this feature.

SR-IOV Advantages

Direct I/O throughput Reduced CPU utilization

Increased scalabil ity

Improved device performance for virtual guests

Requirements

To set up SR-IOV on your system, the following is required:

A server/blade with an SR-IOV-capable motherboard BIOS.

The server/blade must be supported by one of the following operating systems, whichare supported as hypervisor and VMs:

RHEL 6.1 (64-bit)

RHEL 6.0 (64-bit)

RHEL 5.6

SLES 11 SP1

OneConnect has been tested with a limited set of servers based on Intel CPU.

A OneConnect OCx11102-xT or higher Universal Converged Network Adapter (UCNA).

Note: Currently, SR-IOV is supported only in non-multichannel setups.

Note: Configuration mechanisms for parameters such as MAC address, VLAN and QoS for

VF are supported only in RHEL6 and RHEL 6.1 distributions.


5/8


Setting Up SR-IOV

1. Depending on your system, perform one or more of the following tasks to set up your BIOS. (For

more information, refer to the manufacturers instructions for your system.)

a. Enable SR-IOV in the system BIOS. This supports SR-IOV functionality.

b. Enable Intel Virtualization Technology support for Direct I/O VT-d. This enables support

of VMs.

2. To enable SR-IOV in the OneConnect firmware using the PXE Select menu, enter the PXE

Select utility menu by pressing +

when the PXE BIOS banner appears in during

system boot.

Figure 1: PXESelect Utility, SR-IOV enabled

3. Install Linux RHEL6.0 or 6.1.

4. Update the /boot/grub/menu.lst file to include the following command line load parameter for theLinux kernel:

intel_iommu=on iommu=pt

5. Install an Emulex driver for Linux that supports SR-IOV.

6. Reboot the server for the new firmware to become operational.

7. Use the "lspci -vvv" output command to check if SR-IOV is properly enabled:

lspci -vvv

This command returns an SR-IOV capability section for each OneConnect NIC PF, and the total/

initial VFs should be non-zero. Make a note of the lspci command output in the host. This output

is needed in step 12.

Example of the SR-IOV capabilities section output of the PF with SR-IOV enabled:

Capabilities: [180] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

IOVSta: Migration-

Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency

Link: 00

VF offset: 0, stride: 1, Device ID: 0710

Supported Page Size: 00000557, System Page Size: 00000001


6/8


Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)

VF Migration: offset: 00000000, BIR: 0

Example of the SR-IOV capabilities section output of the PF with SR-IOV disabled:

Capabilities: [180] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

IOVSta: Migration-

Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency

Link: 00

VF offset: 0, stride: 1, Device ID: 0710

Supported Page Size: 00000557, System Page Size: 00000001

Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)

VF Migration: offset: 00000000, BIR: 0

8. Create a virtual machine using the Virtual Machine Manager utility and install the RHEL6

operating system on the VM.

9. Boot into the VM and install version 3.0 of the OneConnect software installation package or

installation CD.

10. Shut down the VM.11. Reload the newly installed NIC driver in the host (hypervisor) with the number of virtual PCI

functions (VFs) / physical functions (PFs) as module parameter:

rmmod be2net

modprobe be2net num_vfs=X

whereXis the number of VF per PF.

12. Rerun the lspci command and note that X (as specified in step 11) is the number of virtual NIC

functions that have been added per PF in the output of lspci from step 7.

View the PCI bus:device:function details for the VFs that you want to assign to the VM by using

the lspci command, then detach these VFs from the host. For example, if the lspci output has an

entry similar to:

04:04.0 Ethernet controller:

Emulex Corp.Emulex OneConnect 10Gb NIC (be3) (rev 01)

use the following command:

virsh nodedev-dettach pci_0000_04_04_0

13. Use the Virtual Machine Manager GUI to attach the VF (from step 11) to the guest operating

system by using the add physical device option.

14. Start the RHEL6 guest operating system. Once the guest operating system is booted, use the

lspci command to confirm the visibility of the OneConnect NIC function. The output will show an

OneConnect NIC function, for example:

03:00.0 Ethernet controller: Emulex Corp. Emulex OneConnect 10Gb NI (be3)

15. The OneConnect NIC driver automatically loads and creates the network interfaces. Use the

ifconfig command to verify that the interface is created.

16. After configuring the network interfaces with proper IP addresses, you can send and receive

network traffic from the VM.

Note: The maximum number of virtual functions per physical function is 16.


7/8


Configuration

MAC Address for Virtual Function Interfaces

The Universal Converged Network Adapter (UCNA) is shipped with factory configured MAC addresses

for the network interfaces corresponding to the Physical Functions (PFs). The driver generates random

MAC addresses for the network interfaces corresponding to the Virtual Functions (VFs) using the factory

configured MAC address as seed. Other MAC addresses can be assigned for the interfaces

corresponding to the VF using the IP utility.

Configuring the MAC Address for the VF

To configure the MAC address for the VF, type:

# ip link set eth vf mac

where:

eth is the interface corresponding to the physical function.

is the VF number (0 based) corresponding to the interface for which you are configuring

the MAC address.

is the MAC address you are configuring.

Example:

# ip link set eth0 vf 0 mac 00:16:88:AA:BB:AA

Configuring VLAN on VF Interfaces Using vconfig

Transparent VLAN Tagging

When transparent tagging is configured for a VF, the NIC transparently tags all traffic from the VF with

the configured VLAN ID. The VM is not aware of the VLAN tag.

To assign a VLAN ID to the VF, run the following IP command in the host:

# ip link set eth vf vlan

where:


is the VF number (0 based) corresponding to the interface for which you are configuring

the VLAN.

is the VLAN ID you are configuring.

Example:

# ip link set eth0 vf 0 vlan 5


8/8


Virtual Guest Tagging (VGT)

When VGT is configured, the guest operating system can configure any VLAN ID on the VF. To configure

VGT, run the IP link command with the VLAN ID of 4095. For example:

# ip link set eth0 vf 0 vlan 4095

Configuring Rate Limit on VF Interfaces

Configure the rate limit on a VF interface from the hypervisor using the ip command:

# ip link set eth vf rate

where:


is the VF number (0 based) corresponding to the interface for which you are configuring the TX-

rate.

is the transmission rate in Mb/s that you are configuring. The TX-rate value is interpreted as

a percent of the maximum link speed (currently 10 Gb/s) irrespective of current PF QoS (Quality of

Service). The TX-rate value is not interpreted as an absolute QoS value.

For example, to set a transmit bandwidth rate limit of 5000 Mb/s for the VF 0, run the command:

# ip link set eth0 vf 0 rate 5000

The default rate for a VF is 100Mbps, which is the lowest rate you can program per function in

OneConnect.

Note: Only one VLAN ID per VF is currently supported.

Note: In the above example, the Tx rate value indicates that 50 percent of the PF QoS is

assigned to a VF.

Sr Iov Guide

Documents

Transcript of Sr Iov Guide