Network Interface Card & its Linux programming

Dr A SahuDept of Comp Sc & Engg.

IIT Guwahati

Outline• PCI Devices• NIC Cards• NIC card architecture• Access to NIC register –PCI access

Major Player of NIC Cards

• RealTek (Example RTK 8169S)• Broadcom (Example BCM5751)• Intel (Example 82573L NIC)

• Download manual from vender site–Or search “Realtek 8169 pdf” in google

Detecting PCI Devices : Early PCs

• Peripheral devices in the early PCs used fixed i/o-ports and fixed memory-addresses, e.g.:– Video memory address-range: 0xA0000-0xBFFFF – Programmable timer i/o-ports: 0x40-0x43– Keyboard and mouse i/o-ports: 0x60-0x64– Real-Time Clock’s i/o-ports: 0x70-0x71– Hard Disk controller’s i/o-ports: 0x01F0-01F7– Graphics controller’s i/o-ports: 0x03C0-0x3CF– Serial-port controller’s i/o-ports: 0x03F8-0x03FF– Parallel-port controller’s i/o-ports: 0x0378-0x037A

The PC’s evolution

• It became clear in the 1990s that there would be contention among equipment vendors for ‘fixed’ resource-addresses, which of course were in limited supply

• Among the goals that motivated the PCI Specification was the creation of a more flexible scheme for allocating addresses that future peripheral devices could use

PCI Configuration Space

PCI Configuration Space Body(48 doublewords – variable format)

64doublewords

PCI Configuration Space Header(16 doublewords – fixed format)

A non-volatile parameter-storage areafor each PCI device-function

PCI Configuration Header

StatusRegister

CommandRegister

DeviceID

VendorID

BISTCacheLineSize

Class CodeClass/SubClass/ProgIF

RevisionID

Base Address 0

SubsystemDevice ID

SubsystemVendor ID CardBus CIS Pointer

reserved capabilitiespointer Expansion ROM Base Address

MinimumGrant

InterruptPin reserved

LatencyTimer

HeaderType

Base Address 1

Base Address 2Base Address 3

Base Address 4Base Address 5

InterruptLine

MaximumLatency

31 0 31 0

16 doublewords

Dwords

1 - 0

3 - 2

5 - 4

7 - 6

9 - 8

11 - 10

13 - 12

15 - 14

Three IA-32 address-spaces

memoryspace(4GB)

i/o space(64KB)

PCIconfiguration

space(16MB)

accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation

accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses)

i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

reserved

Interface to PCI Configuration Space

CONFADD( 0x0CF8)

CONFDAT( 0x0CFC)

31 23 16 15 11 10 8 7 2 0EN

bus(8-bits)

device(5-bits)

doubleword (6-bits)

function(3-bits) 00

PCI Configuration Space Address Port (32-bits)

PCI Configuration Space Data Port (32-bits)31 0

Enable Configuration Space Mapping (1=yes, 0=no)

Reading PCI Configuration Data

• Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port

• Step two: Read the designated data from the Configuration-Space Data-Port

• Already discussed PCI-probes pciprobes.c– Lect 29..Showing vram, pciprobe.cpp

Example: network interface

• We can identify the network interface controller in PC’s by class-code 0x02

• The subclass-code 0x00 is for ‘ethernet’• We can identify the NIC from its VENDOR and DEVICE

identification-numbers:• VENDOR_ID = 0x14E4• DEVICE_ID = 0x1677

• You can use the ‘grep’ command to search for these numbers in this header-file:

</usr/src/linux/include/linux/pci_ids.h>

Vendor’s identity example

• The VENDOR-ID 0x14E4 belongs to the Broadcom Corporation

• Information about this firm may be learned from the corporation’s website

• The DEVICE-ID 0x1677 is used to signify Broadcom’s BCM5751 ethernet product

nic

Typical NIC

TX FIFO

RX FIFO

transceiver LANcableB

US

main memory

packet

buffer

CPU

Packet filtering capability

• Network Interface’s hardware needs to implement ‘filtering’ of network packets

• Otherwise the PC’s memory-usage and processor-time will be wasted handling packets not meant for this PC to receive

network packet’s layout

Destination-address (6-bytes) Source-address (6-bytes)

Each data-packet begins with the 6-byte device-address of the network interface which is intended to receive it

Your NIC’s unique address

• You can see the Hardware Address of the ethernet controller on your PC by typing:

$ /sbin/ifconfig

• Look for it in the first line of screen-output that is labeled ‘eth0’, for example:

eth0 Link encap: Ethernet HWaddr 00:11:43:C9:50:3A

Demo “nic_pci_info.c”

• Lets write a kernel module that lets users see certain register-values which pertain to the network interface in your system :– (1) the PCI Configuration Space registers– (2) the Media Access Controller’s address

• It also shows your machine’s node-name (in case you want to save the information)

How we got the MAC-address

• We do not have NIC’s programming datasheet -- but we do have Linux source code for the ‘nic_pci_info.c’ device-driver, which includes a header-file ‘tg3.h’ found here:

</usr/src/linux/drivers/net/>

• If you scroll through the #define directives you will see the offset where the hardware address is stored in the memory-mapped register-space of the ‘nic_pci_info.c’ interface

How we get PCI_NIC registers

StatusRegister

CommandRegister

DeviceID0x1677

VendorID0x14E4

BISTCacheLineSize

Class CodeClass/SubClass/ProgIF

RevisionID

Base Address 0

SubsystemDevice ID

SubsystemVendor ID CardBus CIS Pointer

reserved capabilitiespointer Expansion ROM Base Address

MinimumGrant

InterruptPin reserved

LatencyTimer

HeaderType

Base Address 1

Base Address 2Base Address 3

Base Address 4Base Address 5

InterruptLine

MaximumLatency

31 0 31 0

16 doublewords

Dwords

1 - 0

3 - 2

5 - 4

7 - 6

9 - 8

11 - 10

13 - 12

15 - 14

Linux helper-functions#include <linux/pci.h>struct pci_dev *devp;unsigned int iomem_base, iomem_size;void *io;

devp = pci_get_device( 0x14E4, 0x1677, NULL );if ( !devp ) return –ENODEV;

iomem_base = pci_resource_start( devp, 0 );iomem_size = pci_resource_len( devp, 0 );

io = ioremap( iomem_base, iomem_size );if ( !io ) return -EBUSY;

Big-Endian to Little-Endian

mac1

mac0

mac5

mac4

mac3

mac2

0x0410 0x0411 0x0412 0x0413 0x0414 0x0415 0x0416 0x0417Broadcom network interface storage-addresses

Intel IA-32 character-array storage

mac0

mac1

mac2

mac3

mac4

mac5

Little-Endian to Little-Endian

mac0

mac1

mac2

mac3

mac4

mac5

0x5400 0x5401 0x5402 0x5403 0x5404 0x5405 0x5406 0x5407 Intel network interface storage-addresses

Intel IA-32 character-array storage

mac0

mac1

mac2

mac3

mac4

mac5

For Intel NICs

• For Intel NICs :#define VENDOR_ID 0x8086 // Intel Corp#define DEVICE_ID 0x109A // 82573L NIC

• Intel’s filter-register at offset 0x5400 uses the ‘little endian’ storage-convention

Ethernet LAN

host-1 host-2 host-3

host-4

HUB“Collision Domain”

CSMA/CD = “Carrier Sense Multiple Access/Collision Detection”

Acronyms

• PCI = Peripheral Component Interconnect• MAC = Media Access Controller• Phy = Physical-layer functions• AMT = Active Management Technology• LOM = LAN On Motherboard

Intel 82573L NIC: Hardware Features

• 32K configurable RX and TX packet FIFO• IEEE 802.3x Flow Control support• Host-Memory Receive Buffers 16K/256K• IEEE 802.3ab Auto-Negotiation• TCP/UDP checksum off-loading• Jumbo-frame support (up to 16KB)• Interrupt-moderation controls

External Architecture

PCI/PCI-e Bus

10/100/1000 PHY

MAC/Controller

MDI interface

SM Businterface

EEPROM

Flashinterface

LEDindicators

S/W Defined pins

GMII/MII interface MDIO

interface

Access to PRO/1000 registers

• Device registers are hardware mapped to a range of addresses in physical memory

• You obtain the location (and the length) of this memory-range from a Base Add register in the nic device’s PCI Configuration Space

• Then you request the Linux kernel to setup an I/O ‘remapping’ of this memory-range to ‘virtual’ addresses within kernel-space

i/o-memory remapping

dynamicram

nic registers

vram

IO-APIC Local-APIC

userspace

APIC registers

kernel code/data

nic registersvram

‘virtual’ address-spacephysical address-space

1-GB

3-GB

portability syntax

• Linux provides device-driver writers with some macros for accessing i/o-memory:

#include <asm/io.h>unsigned int datum; iowrite32( datum, address ); datum = ioread32( address );

module_init()#include <linux/pci.h>#include <asm/io.h>#define E1000_STATUS 0x0008unsigned int iomem_base, iomem_size;void *io;

// remap the device’s i/o-memory into kernel spacedevp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );if ( !devp ) return –ENODEV;iomem_base = pci_resource_start( devp, 0 );iomem_size = pci_resource_len( devp, 0 );io = ioremap_nocache( iomem_base, iomem_size );if ( !io ) return –ENOSPC;

// read and display the nic’s STATUS registerdevice_status = ioread32( io + E1000_STATUS );printk( “ Device Status Register = 0x%08X \n”, status );

0

Device Status (0x0008)

? 0 0 0 0 0 0 0 0 0 0 0GIO

MasterEN

0 0 0

0 0 0 0 PHYreset ASDV

ILOS

SLU

0 TXOFF 0 0 F

D

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

FunctionID

LU

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16

SPEED

FD = Full-DuplexLU = Link UpTXOFF = Transmission PausedSPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved)ASDV = Auto-negotiation Speed Detection Value 82573L

some undocumented functionality?

Thanks