Building Gigabit-rate Routers with th N ... - wiki.netfpga.org · PCI NetFPGA Driver (Extensible)...
Transcript of Building Gigabit-rate Routers with th N ... - wiki.netfpga.org · PCI NetFPGA Driver (Extensible)...
Building Gigabit-rate Routersith th N tFPGAwith the NetFPGA:
NICTA Tutorial at UNSWPresented by:
John W. Lockwood, Jad Naous, Glen Gibb (Stanford University)(Stanford University)
Hosted by: Lavy Libman (NICTA) and Philip Allen (UNSW)
February 6, 2008: 9am-5pmLab 343A, Electrical Engineering Building (G17)Lab 343A, Electrical Engineering Building (G17)
Kensington Campus, University of New South WalesSydney, Australia
NICTA 2008 - NetFPGA Tutorial 1 S T A N F O R D U N I V E R S I T Y
http://NetFPGA.org
What is the NetFPGA?
CPU MemoryNetworkingSoftwareSoftwarerunning on a standard PC
1GE
PCI PC with NetFPGA
A hardware
FPGA1GE
1GE
1GE
acceleratorbuilt with Field Programmable Gate Array
Memory 1GENetFPGA Board
Gate Arraydriving Gigabit network links
NICTA 2008 - NetFPGA Tutorial 2 S T A N F O R D U N I V E R S I T Y
Introduction
Who uses the NetFPGA• Teachers• Students• Researchers
How they use the NetFPGA1. To run the Router Kit2 To build modular reference designs2. To build modular reference designs
• IPv4 router• 4-port NIC• Ethernet switch, …
3. To create new systems
NICTA 2008 - NetFPGA Tutorial 3 S T A N F O R D U N I V E R S I T Y
Running the Router Kit
User-space development, 4x1GE line-rate forwarding
CPU MemoryOSPF BGP
My Protocoluser
PCI
kernelRouting
Table
“Mi ”
FPGA1GE
1GE
1GE
1GE
FwdingTable
PacketBuffer
“Mirror”
Memory
1GE
1GE
1GE
IPv4Router
1GE
1GE
1GE
NICTA 2008 - NetFPGA Tutorial 4 S T A N F O R D U N I V E R S I T Y
Building Modular Router Modules
PW OSPFCPU Memory
Java GUIFront Panel
PW-OSPFVerilog
EDA Tools(Xilinx,
Mentor etc )
PCI NetFPGA Driver
(Extensible) Mentor, etc.)
1 Design
FPGA1GE
1GEIn Q
MgmtL2
ParseL3
Parse
1GE
1GE
1. Design2. Simulate3. Synthesize4. Download
Memory
1GE
1GE
1GE
Mgmt
IPLookup
ParseParse
Out QMgmt
1GE
1GE
1GE
MyBlock
NICTA 2008 - NetFPGA Tutorial 5 S T A N F O R D U N I V E R S I T Y
1GEVerilog modules interconnected by FIFO interfaces
Creating new systems
CPU MemoryVerilog
EDA Tools(Xilinx,
Mentor etc )
PCI NetFPGA Driver
Mentor, etc.)
1. Design2. Simulate3 Synthesize
FPGA1GE
1GE
1GE
1GEMy Design
3. Synthesize4. Download
Memory
1GE
1GE
1GE
1GE
1GE
1GE
My Design
(1GE MAC is soft/replaceable)
NICTA 2008 - NetFPGA Tutorial 6 S T A N F O R D U N I V E R S I T Y
1GE
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 7 S T A N F O R D U N I V E R S I T Y
Basic Operation of an IP RouterR3
AR1
R4 DA
B
R2
E
C R2FR5
R3DNext HopDestination
R5FR3ER3D
NICTA 2008 - NetFPGA Tutorial 8 S T A N F O R D U N I V E R S I T Y
What does a router do?R3
AR1
R4 DA16 3241
Total Packet LengthT.ServiceHLenVerB E
Header ChecksumProtocolTTL
Fragment OffsetFlagsFragment ID
0 by
tes
C R2FR5
R3DNext HopDestination
O ti (if )
Destination Address
Source Address20
R5FR3ER3D
Data
Options (if any)
NICTA 2008 - NetFPGA Tutorial 9 S T A N F O R D U N I V E R S I T Y
What does a router do?
R3
AR1
R4 D
B EB
C R2
E
CFR5
NICTA 2008 - NetFPGA Tutorial 10 S T A N F O R D U N I V E R S I T Y
Basic Components of an IP Router
Management& CLI
S
Control PlaneR ti
Routing Protocols
& CLI
Softw
are Control PlaneRoutingTable
e
Datapathper-packet SwitchingForwarding
Table
Hardw p p
processing
ware
NICTA 2008 - NetFPGA Tutorial 11 S T A N F O R D U N I V E R S I T Y
Per-packet processing in an IP Router
1. Accept packet arriving on an incoming link.2. Lookup packet destination address in the
forwarding table, to identify outgoing ( )port(s).
3. Manipulate IP header: e.g., decrement TTL, update header checksumupdate header checksum.
5. Buffer packet in the output queue.6 Transmit packet onto outgoing link6. Transmit packet onto outgoing link.
NICTA 2008 - NetFPGA Tutorial 12 S T A N F O R D U N I V E R S I T Y
Generic Datapath Architecture
Lookup Update
Header ProcessingData Hdr Data Hdr
QueueLookupIP Address
UpdateHeader
QueuePacket
Forwarding
IP Address Next Hop
BufferForwardingTable
BufferMemory
NICTA 2008 - NetFPGA Tutorial 13 S T A N F O R D U N I V E R S I T Y
CIDR and Longest Prefix MatchesThe IP address space is broken into line segments.Each line segment is described by a prefix.A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment.
Th fi 128 9/16 t th li te.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255.
128 9 0 0
128.9/16
128.9.0.0 142.12/1965/8
0 232-1216
128 9 16 14
NICTA 2008 - NetFPGA Tutorial 14 S T A N F O R D U N I V E R S I T Y
128.9.16.14
Classless Interdomain Routing (CIDR)
128 9 19/24
128 9 16/20 128 9 176/20
128.9.19/24128.9.25/24
128.9/16
128.9.16/20 128.9.176/20
0 232-1
128.9.16.14
Most specific route = “longest matching prefix”
NICTA 2008 - NetFPGA Tutorial 15 S T A N F O R D U N I V E R S I T Y
Techniques for LPM in hardware• Linear search• Direct lookup
– Currently requires too much memory– Updating a prefix leads to many changes
Tries• Tries– Deterministic lookup time– Easily pipelinedEasily pipelined– But requires multiple memories/references
• TCAM (Ternary CAM)( y )– Simple and widely used– But low-density, high-power
NICTA 2008 - NetFPGA Tutorial 16 S T A N F O R D U N I V E R S I T Y
– Gradually being replaced by new algorithms
An IP Router on NetFPGA
Management& CLI
S
i l l
R ti
Routing Protocols
& CLI
Softw
are
Linux user-levelprocesses
ExceptionP i Routing
Table
eVerilog on
Processing
SwitchingForwardingTable
Hardw
Verilog on NetFPGA PCI board
ware
NICTA 2008 - NetFPGA Tutorial 17 S T A N F O R D U N I V E R S I T Y
NetFPGA Router
Function – 4 Gigabit Ethernet ports
Fully programmable– FPGA hardware
Low cost
Open-source FPGA hardware – Verilog base design
Open-souce Software– Drivers in C and C++
NICTA 2008 - NetFPGA Tutorial 18 S T A N F O R D U N I V E R S I T Y
Drivers in C and C
NetFPGA PlatformMajor Components
– Interfaces4 Gi bit Eth t P t• 4 Gigabit Ethernet Ports
• PCI Host Interface
– Memories• 36Mbits Static RAM• 512Mbits DDR2 Dynamic RAM• 512Mbits DDR2 Dynamic RAM
– FPGA Resources• Block RAMs• Configurable Logic Block (CLBs)• Memory Mapped Registers
NICTA 2008 - NetFPGA Tutorial 19 S T A N F O R D U N I V E R S I T Y
• Memory Mapped Registers
NetFPGA System
Browser& VideoMonitor
SoftwareCADT l
Web &Video
User Space
Linux Kernel
ClientSoftwareTools Server
Linux Kernel
PCI-ePCI
Packet Forwarding Table
NI
NetFPGA Router
VI VI VI VI
CGGGGGG
NetFPGA RouterHardware
NICTA 2008 - NetFPGA Tutorial 20 S T A N F O R D U N I V E R S I T Y
GE
GE
GE
GE
GE
GE
(eth1 .. 2)(nf2c0 .. 3)
NetFPGA Hardware
NICTA 2008 - NetFPGA Tutorial 21 S T A N F O R D U N I V E R S I T Y
NetFPGA System Implementation• NetFPGA Blocks
– Virtex-2 Pro FPGA– 4.5MB ZBT SRAM– 64MB DDR2 DRAM 6– PCI Host Interface– 4 Gigabit Ethernet ports
• Intranet Test PortsIntranet Test Ports – Dual or Quad Gigabit
Etherents on PCI-e
• InternetInternet – Gigabit Ethernet
on Motherboard
• ProcessorProcessor – Dual-Core CPU
• Operating System– Linux CentOS 4 4
NICTA 2008 - NetFPGA Tutorial 22 S T A N F O R D U N I V E R S I T Y
Linux CentOS 4.4
NetFPGA Lab Setup
GEPClient Eth2 : Server
(eth1 .. 2)
Nf2c3 : Adj ServerCPU x2 Net FPGA
Dual NICGEPC
I-e
ClientGE
GE
Eth2 : Server
ServerEth1 : Local host
Nf2c3 : Adj. ServerNet-FPGAPC
INetFPGAControl SW
GE
GE
GE
GE
InternetRouter
HardwareNf2c1 : Adjacent
Nf2c2 : Local Host
Nf2c0 : AdjacentGECAD ToolsNf2c0 : Adjacent
NICTA 2008 - NetFPGA Tutorial 23 S T A N F O R D U N I V E R S I T Y
NetFPGA Hardware Set for Demo #1
Net-FPGA GE
GEInternet
CPU x2
PCI-e
PC
VideoServer
NICGE
PCI-e GE
GE
GE
InternetRouter
Hardware
CI
Server delivers
Net-FPGA GE
GE
GE
InternetRouter
Hardware
streaming HD videothrough a GEHardware
…
through a chain of NetFPGA R
Net-FPGACPU x2
NICGE
PCI-e
PCVideo
GE
GE
GEInternet
…Routers
NICTA 2008 - NetFPGA Tutorial 24 S T A N F O R D U N I V E R S I T Y
CI
Display
CAD Tools
GE
GE
te etRouter
Hardware
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 25 S T A N F O R D U N I V E R S I T Y
Topology of NetFPGA Routers
VideoServerServer
HDDisplayDisplay
NICTA 2008 - NetFPGA Tutorial 26 S T A N F O R D U N I V E R S I T Y
Setup for the Reference Router
Each NetFPGA card has four ports NetFPGA
Video Server
has four ports
Port 2 connected to Cli t / S
NetFPGA
Client / Server
Ports 0 and 3 connected to NetFPGAadjacent NetFPGA cards
NetFPGA
Video Client
NetFPGA
NICTA 2008 - NetFPGA Tutorial 27 S T A N F O R D U N I V E R S I T Y
Demo 1: Logical Topology
1 1 4 1 7 1 10 1 13 1 16 1.1.1
.1.2.3.1
.30.2
.4.1
.4.2
.6.1.3.2
.7.1
.7.2
.9.1.6.2
.10.1
.10.2
.12.1.9.2
.13.1
.13.2
.15.1.12.2
.16.1
.16.2.15.2
.17.1.5.1 .8.1 .11.1 .14.1 .18.1
.2.1.18.1
27 1
.30.1
.24.1.27.2
.21.1.24.2 .21.2
.18.2.20.1
.23.1.26.1
.29.1
.28.1
.28.2 .27.1
.25.1
.25.2.24.1
.22.1
.22.2.21.1
.19.1
.19.2
NICTA 2008 - NetFPGA Tutorial 28 S T A N F O R D U N I V E R S I T Y
Video ClientShortest Path
Video Server
Working IP Router
• ObjectivesObjectives – Become familiar with
Stanford Reference Router
– Observe PW-OSPF re-routing traffic around a failure
NICTA 2008 - NetFPGA Tutorial 29 S T A N F O R D U N I V E R S I T Y
Streaming Video through the NetFPGA
• Video server– Source files
/var/www/html/video
– Network URL : http://192.168.Net.Host/Video
Vid li• Video client– Windows Media Player– Linux mplayer
• Video traffic– MPEG2 HDTV (35 Mbps)( p )– MPEG2 TV (9 Mbps)– DVI (3 Mbps)– WMF (1.7 Mbps)
NICTA 2008 - NetFPGA Tutorial 30 S T A N F O R D U N I V E R S I T Y
( p )
Step 1 – Observe the Routing TablesThe router is already
configured and running on your machines
The routing table has converged to theconverged to the routing decisions with minimum number of hops
NICTA 2008 - NetFPGA Tutorial 31 S T A N F O R D U N I V E R S I T Y
Next, break a link …
Step 2 - Dynamic Re-routing
Break the link between videobetween video server and video client
.1.1
.1.2.3.1
.30.2
.4.1
.4.2
.6.1.3.2
.7.1
.7.2
.9.1
.6.2
.10.1
.10.2
.12.1
.9.2
.13.1
.13.2
.15.1
.12.2
.16.1
.16.2.15.2
.17.1.5.1 .8.1 .11.1 .14.
1.18.1
.2.1
Routers re-route t ffi d th
.28.2.27.1
.30.1
.25.2.24.1
.27.2
.22.2.21.1
.24.2
.19.2
.21.2.18.2
.20.1
.23.1.26.1
.29.1
traffic around the broken link and video continues
.28.1 .25.1 .22.1 .19.1
video continues playing
NICTA 2008 - NetFPGA Tutorial 32 S T A N F O R D U N I V E R S I T Y
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 33 S T A N F O R D U N I V E R S I T Y
Integrated Circuit Technology
Full-custom Design – Complementary Metal Oxide Semiconductor (CMOS)Complementary Metal Oxide Semiconductor (CMOS)
Semi-custom ASIC Design G t– Gate array
– Standard cell
Programmable Logic Device– Programmable Array Logic
Field Programmable Gate Arra s– Field Programmable Gate Arrays
Processors
NICTA 2008 - NetFPGA Tutorial 34 S T A N F O R D U N I V E R S I T Y
Look-Up TablesCombinatorial logic is stored
in Look-Up Tables (LUTs) – Also called
A B C D Z0 0 0 0 00 0 0 1 0Also called
Function Generators (FGs)– Capacity is limited only by
number of inputs, not complexity
0 0 0 1 00 0 1 0 00 0 1 1 1p , p y
– Delay through the LUT is constant 0 1 0 0 10 1 0 1 1
Combinatorial Logic
A
. . .1 1 0 0 01 1 0 1 0
B
CD
Z 1 1 1 0 01 1 1 1 1
NICTA 2008 - NetFPGA Tutorial 35 S T A N F O R D U N I V E R S I T Y
Diagram From: Xilinx, Inc
Xilinx CLB Structure
Slice 0
Each slice has four outputs– Two registered outputs,
two non-registered outputs Slice 0
LUT Carry D QCE
PRE
CLR
g p– Two BUFTs associated
with each CLB, accessible by all 16 CLB outputs
CLRy p
Carry logic run verticallyLUT Carry D
QCEPRE
CLR
Carry logic run vertically – Signals run upwards– Two independent
carry chains per CLBcarry chains per CLB
Diagram From: Xilinx Inc
NICTA 2008 - NetFPGA Tutorial 36 S T A N F O R D U N I V E R S I T Y
Diagram From: Xilinx, Inc (Courtesy Jeff Weintraub)
Field Programmable Gate ArraysDin Clk
CLB– Primitive element of FPGA
4 LUT
G4G3G2G1
G
4 LUT
F4F3F2F1
F
3 LUT
H
S
R
D Q
S
R
D Q
H1
YQ
Y
XQ
X
M
M
M
M
CLB
Routing Module– Global routing
Local interconnect
GRMLocal Routing
CLB PIP– Local interconnect
3rd Generation LUT-based FPGA
Macro Blocks– Block Memories– Microprocessor
...
... ...
Macro
I/O Block
... ...
......
Block(uP,Mem)
NICTA 2008 - NetFPGA Tutorial 37 S T A N F O R D U N I V E R S I T Y
Pad Routing CLB Matrix I/O
NetFPGA Block Diagram
NetFPGA platform
1M
18MSR
A1P
FPGA w/provided infrastructureFo 1GE
M
AC
1GE
MAC
Virtex II-Pro 50 FPGA withuser-defined network logic* Hardware specified with
- Verilog source code
Mb
AM
1GE
PH
Y1G
EPH
Y
64MD
DR
SDR
A
ur Gigabit E E
C1G
E
MA
C1M
g- Pre-generated cores
* Software written for - Embedded PowerPCs - Soft core processors
E Y
1GE
PH
Y1P
BR2AM
Ethernet Inte
18Mb
SRA
M
Board-B1G
E
MAC (Microblaze, LEON ..)
1GE
PH
Y
erfaces
M
FIFOpacket Control, PCI
Interface
3 Gb
SATA
Board Interc
Linux OS - NetFPGA Kernel driverHostcomputer
User defined software networking applications
buffers Interface
connect
NICTA 2008 - NetFPGA Tutorial 38 S T A N F O R D U N I V E R S I T Y
User-defined software networking applications
Details of NetFPGA
• Fits into Standard PCI slotStandard Bus : 32 bits 33 MHz– Standard Bus : 32 bits, 33 MHz
• Provides Interfaces for processing network packets– 4 Gigabit Ethernet Ports
• Allows hardware-accelerated processing
NICTA 2008 - NetFPGA Tutorial 39 S T A N F O R D U N I V E R S I T Y
• Allows hardware-accelerated processing – Implemented with Field Programmable Gate Array (FPGA) Logic
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 40 S T A N F O R D U N I V E R S I T Y
Hardware Description LanguagesC• Concurrent– By Default, Verilog statements
l t d tlevaluated concurrently
• Express fine grain parallelismExpress fine grain parallelism– Allows gate-level parallelism
• Provides Precise Description– Eliminates ambiguity about operation
• SynthesizableG t h d f d i ti
NICTA 2008 - NetFPGA Tutorial 41 S T A N F O R D U N I V E R S I T Y
– Generates hardware from description
Verilog Data Typesreg [7:0] A; // 8-bit register, MSB to LSB
// (Preferred bit order for NetFPGA)reg [0:15] B; // 16-bit register LSB to MSBreg [0:15] B; // 16 bit register, LSB to MSB
B = {A[7:0],A[0:7]}; // Assignment of bits
reg [31:0] Mem [0:1023]; // 1K Word Memory
integer Count; // simple signed 32-bit integerinteger K[1:64]; // an array of 64 integerstime Start Stop; // Two 64-bit time variablestime Start, Stop; // Two 64-bit time variables
NICTA 2008 - NetFPGA Tutorial 42 S T A N F O R D U N I V E R S I T Y
From: CSCI 320 Computer ArchitectureHandbook on Verilog HDL, by Dr. Daniel C. Hyde :
http://eesun.free.fr/DOC/VERILOG/verilog-manual.html
Signal Multiplexers
Two input multiplexer (using if / else)reg y;always @*
if (select)y = a;
elsey = b;
Two input multiplexer (using ternary operator ?:)p p ( g y p )
wire t = (select ? a : b);
NICTA 2008 - NetFPGA Tutorial 43 S T A N F O R D U N I V E R S I T Y
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
Larger Multiplexers
Three input multiplexer
reg s;always @*
begincase (select2)
2'b00: s = a;;2'b01: s = b;default: s = c;
endcaseend
NICTA 2008 - NetFPGA Tutorial 44 S T A N F O R D U N I V E R S I T Y
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
Synchronous Storage Elements• Values change at• Values change at
times governed by clock Clock
Din DoutQD
Clock Transition1Clock– Clockt=0 t=1 t=2
0Clock
time• Input to circuit
Clock Event
A B CDint 0
– Clock Event• Example: Rising edge
Clock Transition
S0Doutt=0
A B
t=0– Flip/Flop
• Transfers Value From Dinto Dout on Clock event
NICTA 2008 - NetFPGA Tutorial 45 S T A N F O R D U N I V E R S I T Y
t=0to Dout on Clock event
Finite State Machines
Copyright 2001, John W. Lockwood, All Rights Reserved
Inputs (X)Outputs (Z)=λ (X,S(t))
[Moore](S(t))=λ
[Mealy]-or-
Combinational Logic
N tQ D
S(t) δS(t+1)=(X,S(t)) State
Next
...Q D
State Storage
NICTA 2008 - NetFPGA Tutorial 46 S T A N F O R D U N I V E R S I T Y
Synthesizable Verilog : Delay Flip/Flops
D-type flip flopreg q;always @ (posedge clk)always @ (posedge clk)
q <= d;
D type flip flop with data enableD type flip flop with data enablereg q; always @ (posedge clk)
if (enable) q <= d;q < d;
NICTA 2008 - NetFPGA Tutorial 47 S T A N F O R D U N I V E R S I T Y
From: http://eesun.free.fr/DOC/VERILOG/synvlg.html
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 48 S T A N F O R D U N I V E R S I T Y
Reference Router Pipeline• Five stages
– InputMACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
– Input Arbitration– Routing Decision and Input Arbiter
packet modification– Output Queuing
O t tOutput Port Lookup
– Output• Packet-based module
interfaceOutput Queues
interface• Pluggable design
MAC CPU MAC CPU MAC CPU MAC CPU
SIGMETICS 2007 - NetFPGA Tutorial 49 S T A N F O R D U N I V E R S I T Y
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
Make your own router
Objectives: Object es– Learn how to build hardware– Run the software
E l t hit t– Explore router architecture
ExecutionExecution– Start synthesis– Rerun the GUI with the new hardware
T t ti it d t ti ti ith i– Test connectivity and statistics with pings– Explore pipeline in the details page– Explore detailed statistics in the details page
NICTA 2008 - NetFPGA Tutorial 50 S T A N F O R D U N I V E R S I T Y
p p g
Step 1 - Build the hardware
Start terminal cd toStart terminal, cd to “NF2/projects/tutorial_router/synth”
Start synthesis with “make”
SIGMETICS 2007 - NetFPGA Tutorial 51 S T A N F O R D U N I V E R S I T Y
Step 2 - Run Homemade Router
cd to “NF2/projects/tutorial_router/sw”
Type: “tutorial router gui pl” totutorial_router_gui.pl touse the just built router hardwarea d a e
The same interface shouldstart again
SIGMETICS 2007 - NetFPGA Tutorial 52 S T A N F O R D U N I V E R S I T Y
Step 4 - Connectivity and StatisticsPing any addresses
192.168.x.y where x is from 1-20 and y is 1 or 2
O th t ti ti t b iOpen the statistics tab in the Quickstart window to see some statisticssee some statistics
Explore more statistics inExplore more statistics in modules under the details tab
SIGMETICS 2007 - NetFPGA Tutorial 53 S T A N F O R D U N I V E R S I T Y
Step 5 - Explore Router Architecture
Click the Details tab of the Quickstart windowthe Quickstart window
This is the reference router pipeline – a canonical simple tocanonical, simple to understand, modular router pipelinep p
NICTA 2008 - NetFPGA Tutorial 54 S T A N F O R D U N I V E R S I T Y
Step 6 - Explore Output QueuesClick on the Output
Queues module in the Details tab
The page gives configuration details
…and statistics
SIGMETICS 2007 - NetFPGA Tutorial 55 S T A N F O R D U N I V E R S I T Y
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 56 S T A N F O R D U N I V E R S I T Y
Buffer Requirements in a RouterBuffer size matters:
– Small queues reduce delayLarge buffers are expensive– Large buffers are expensive
Theoretical tools predict requirements– Queuing theory– Large deviation theory– Mean field theoryMean field theory
Yet, there is no direct answer.Fl h l d l t– Flows have a closed-loop nature
– Question arises on whether focus should be on equilibrium state or transient state..
SIGMETICS 2007 - NetFPGA Tutorial 57 S T A N F O R D U N I V E R S I T Y
Rule-of-thumb
CRouterSource Destination
• Universally applied rule-of-thumb:A router needs a buffer size: CTB ×2
2T
– A router needs a buffer size:– 2T is the two-way propagation delay (or just 250ms)– C is capacity of bottleneck link
CTB ×= 2
p y• Context
– Mandated in backbone and edge routers.A i RFP d IETF hit t l id li– Appears in RFPs and IETF architectural guidelines.
– Already known by inventors of TCP • [Van Jacobson, 1988]
SIGMETICS 2007 - NetFPGA Tutorial 58 S T A N F O R D U N I V E R S I T Y
[ , ]– Has major consequences for router design
The Story So Far
10,000 20# packetsat 10Gb/s
1,000,000
(1) Assume: Large number of desynchronized flows; 100% utilization(2) A : L b f d h i d fl ; <100% tili ti
SIGMETICS 2007 - NetFPGA Tutorial 59 S T A N F O R D U N I V E R S I T Y
(2) Assume: Large number of desynchronized flows; <100% utilization
Using NetFPGA to explore buffer size
• Need to reduce buffer size and measure occupancy
• Alas, not possible in commercial routers• So, we will use NetFPGA instead
Objective:j– Use NetFPGA to understand how large a buffer
we need for a single TCP flow.
SIGMETICS 2007 - NetFPGA Tutorial 60 S T A N F O R D U N I V E R S I T Y
Why 2TxC for a single TCP Flow?
Rule for adjusting WIf ACK i i d W W+1/WOnly W packets – If an ACK is received: W ← W+1/W
– If a packet is lost: W ← W/2
y pmay be outstanding
SIGMETICS 2007 - NetFPGA Tutorial 61 S T A N F O R D U N I V E R S I T Y
Time Evolution of a Single TCP Flow
Time evolution of a single TCP flow through a router. Buffer is < 2T*CTime evolution of a single TCP flow through a router. Buffer is 2T*C
SIGMETICS 2007 - NetFPGA Tutorial 62 S T A N F O R D U N I V E R S I T Y
NetFPGA Hardware Set for Demo #2
NICGE
PC
…
Net-FPGACPU x2
NICI-ePC
IVideoClient
GE
GE
GE
GE
InternetRouter Server
GEHardware deliversstreaming HD video
PCI-e
VideoServer
NICGE
PCI-e GE
HD videoto adjacent client
CPU x2
NICTA 2008 - NetFPGA Tutorial 63 S T A N F O R D U N I V E R S I T Y
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 64 S T A N F O R D U N I V E R S I T Y
Setup for the Demo 2
Each NetFPGA card has four portshas four ports
Port 2 connected to L l H tLocal Host
Port 3 connected to AdjacentServer
adjacent Server
LocalHost
NetFPGA
NICTA 2008 - NetFPGA Tutorial 65 S T A N F O R D U N I V E R S I T Y
Topology for Second DemonstrationRouters connected point-to-point topologyPort 3 connects to local hostPort 1 connects to adjacent neighborPorts 0 and 2 n sedPorts 0 and 2 unused
.2.1 .5.1 .8.1 .11.1
.1.1 .1.2
.4.1
.4.2
.7.1
.7.2
.10.1
.10.2
.13.1
.13.2.2.2 .5.2 .8.2 .11.2
.14.2.29.1
.14.126 2
.29.2
.16.1.16.2
.28.1
.28.2
.25.1
.25.2
.22.1
.22.2
.19.1
.19.2.17.2
.17.1
.20.2
.20.1
.23.2
.23.1
.26.2
.26.1
SIGMETICS 2007 - NetFPGA Tutorial 66 S T A N F O R D U N I V E R S I T Y
Enhanced Router
ObjectivesObjectives – Observe router with new modules– New modules: rate limiting, delay, event capture
Execution– Run event capture router– Run event capture router– Look at routing tables– Explore details pane– Start tcp transfer, look at queue occupancy– Change rate/delay, look at queue occupancy
NICTA 2008 - NetFPGA Tutorial 67 S T A N F O R D U N I V E R S I T Y
Step 1 - Run Pre-made Enhanced Router
Start terminal and cd to “NF2/projects/tutorial roNF2/projects/tutorial_router/sw/”
Type “./tut adv router gui.pl”_ _ _g p
A familiar GUI should start
SIGMETICS 2007 - NetFPGA Tutorial 68 S T A N F O R D U N I V E R S I T Y
Step 3 - Explore Enhanced Router
Cli k th D t il t bClick on the Details tab
A similar Pipeline to the one seenthe one seen previously shows with some additions
SIGMETICS 2007 - NetFPGA Tutorial 69 S T A N F O R D U N I V E R S I T Y
Enhanced Router PipelineTwo modules added
1. Event CaptureMACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
to capture output queue events (writes reads
Input Arbiter
O t t P t L k(writes, reads, drops)
Output Port Lookup
Event Capture
2 Rate Limiter topOutput Queues
2. Rate Limiter to create a bottleneck MAC
TxQCPUTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
RateLimiter
SIGMETICS 2007 - NetFPGA Tutorial 70 S T A N F O R D U N I V E R S I T Y
MACTxQ
Step 4 - Decrease the Link RateTo create bottleneck and
show the TCP “sawtooth”, li k t i d dlink-rate is decreased.
I th D t il t b li k thIn the Details tab click the “Rate Limit” module
Check Enable
Set link rate to 1.953Mbps
SIGMETICS 2007 - NetFPGA Tutorial 71 S T A N F O R D U N I V E R S I T Y
Step 5 – Decrease Queue Size
Go back to the DetailsGo back to the Details Panel and click on “Output Queues”.
Select the “Output Queue 2” tab.
Change the output queues size in packets slider to 16
SIGMETICS 2007 - NetFPGA Tutorial 72 S T A N F O R D U N I V E R S I T Y
slider to 16
Step 6 - Start Event Capture
Click on the Event CaptureClick on the Event Capture module under the Details tab
This should start the configuration page
SIGMETICS 2007 - NetFPGA Tutorial 73 S T A N F O R D U N I V E R S I T Y
Step 7 - Configure Event Capture
Check Send to local h t t i thost to receive events on the local host
Check Monitor Queue 2to monitor output queueto monitor output queue of MAC port1
Check “Enable Capture” to start Event capture
SIGMETICS 2007 - NetFPGA Tutorial 74 S T A N F O R D U N I V E R S I T Y
Step 8 - Start TCP Transfer
We will use iperf to run aWe will use iperf to run a large TCP transfer and look at queue evolutionoo at queue e o ut o
Start a terminal and cd to“NF2/projects/tutorial_router/sw”
type “iperf.sh”
SIGMETICS 2007 - NetFPGA Tutorial 75 S T A N F O R D U N I V E R S I T Y
Step 9 - Look at Event Capture Results
Click on the Event Capture module underCapture module under the details tab.
The sawtooth patternThe sawtooth pattern should now be visible.
SIGMETICS 2007 - NetFPGA Tutorial 76 S T A N F O R D U N I V E R S I T Y
Queue Occupancy Charts
SIGMETICS 2007 - NetFPGA Tutorial 77 S T A N F O R D U N I V E R S I T Y
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 78 S T A N F O R D U N I V E R S I T Y
NetFPGA in the Classroom
• Stanford CS344: “Build an Internet Router”– Courseware will be available later in 2007– Students work in teams of three (2 software, 1
hardware)– Design and implement hardware and software
in 8 weeksS ft CLI PW OSPF– Software: CLI, PW-OSPF
– Show interoperability with other groups Add ne feat res in remaining t o eeks– Add new features in remaining two weeks
• Firewall, NAT, DRR, Packet capture, Data generator, …
NICTA 2008 - NetFPGA Tutorial 79 S T A N F O R D U N I V E R S I T Y
g ,
Networked FPGAs in Research1. RCP: Congestion control
• New module for parsing and overwriting new packet• New software to calculate explicit ratesp
2. Packet Monitoring (ICSI)• Network Shunt
3. Deep Packet Inspection (FPX)3. Deep Packet Inspection (FPX)• TCP/IP Flow Reconstruction• Regular Expression Matching• Bloom FiltersBloom Filters
4. Ethane: Network security• New switch (“managed flow-table”) deployed
5 Buffer Sizing5. Buffer Sizing• Reduce buffer size and measure effect on network
performance.• Need a way to set buffer size and measure buffer
NICTA 2008 - NetFPGA Tutorial 80 S T A N F O R D U N I V E R S I T Y
• Need a way to set buffer size, and measure buffer occupancy.
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 81 S T A N F O R D U N I V E R S I T Y
Enhance Your Router
ObjectivesObjectives – Add new modules to datapath– Synthesize and test routery
Execution– Open user_datapath.v, uncomment
delay/rate/event capture modulesS h i– Synthesize
– After synthesis, test the new system.
NICTA 2008 - NetFPGA Tutorial 82 S T A N F O R D U N I V E R S I T Y
An aside: xemacs TipsWe will be modifying the Verilog source codeSlides show xemacs, but vim also available.xemacs:
– To undo, use ctrl+shift+'-'T l lti k t k d j t t t l– To cancel a multi-keystroke command, just type ctrl+g
– To select lines, hold shift and press the arrow keys.– To comment some selected lines, type ctrl+c+c, yp– To uncomment a commented block, move the cursor to
one of the lines inside the commented block and type ctrl+c+uctrl+c+u
– To save type ctrl+x+s– To search, type ctrl+s search_pattern
NICTA 2008 - NetFPGA Tutorial 83 S T A N F O R D U N I V E R S I T Y
Step 1 - Open the SourceWe will modify the Verilog
source code to add event capture, rate limiter, and delay modules
We will simply comment and uncomment existing codeuncomment existing code
Open terminalpType “xemacs
NF2/projects/tutorial_router/sr/ d t th
SIGMETICS 2007 - NetFPGA Tutorial 84 S T A N F O R D U N I V E R S I T Y
c/user_data_path.v
Step 2 - Add wires
Now we need to add wires t t thto connect the new modules
Search for “new wires” (ctrl+s new wires) then(ctrl s new wires) then press Enter
Uncomment the wires (ctrl+c+u)
SIGMETICS 2007 - NetFPGA Tutorial 85 S T A N F O R D U N I V E R S I T Y
Step 3 - Connect Event Capture
Search for opl_output (ctrl+s opl_output) then press Enter
Comment the four lines above (up, shift + up + up + up + up ctrl+c+c)up, ctrl+c+c)
Uncomment the block below toUncomment the block below to connect the outputs (ctrl+s opl out, ctrl+c+u)
SIGMETICS 2007 - NetFPGA Tutorial 86 S T A N F O R D U N I V E R S I T Y
p _ , )
Step 4 - Add the Event Capture Module
Search for evt capture topSearch for evt_capture_top (ctrl+s evt_capture_top) then press Entere p ess e
Uncomment the block (ctrl+c+u)
SIGMETICS 2007 - NetFPGA Tutorial 87 S T A N F O R D U N I V E R S I T Y
Step 5 - Connect the Output Queue to the Rate Limiter
Search for port_outputs (ctrl+s ports outputs, Enter)( p _ p , )
Comment the 4 lines above (select the four lines by using shift+arrow keys, th t t l )then type ctrl+c+c)
U t th t dUncomment the commented block by scrolling down into the block and typing
SIGMETICS 2007 - NetFPGA Tutorial 88 S T A N F O R D U N I V E R S I T Y
the block and typing ctrl+c+u
Step 6 - Add Rate Limiter
Scroll down until you reachScroll down until you reach the next “Excluded” block
Uncomment the block containing the rate limiter instantiationsinstantiations. (scroll into the block and type ctrl+c+u)
Save (ctrl+x+s)
SIGMETICS 2007 - NetFPGA Tutorial 89 S T A N F O R D U N I V E R S I T Y
Step 7 - Build the hardware
Start terminal, cd to “NF2/projects/tutorial_router/synth”ter/synth”
Start synthesis with “make”Start synthesis with make
SIGMETICS 2007 - NetFPGA Tutorial 90 S T A N F O R D U N I V E R S I T Y
Tutorial OutlineB k d• Background
– Basics of an IP Router– The NetFPGA Platform
• The Stanford Base Reference Router• The Stanford Base Reference Router– Demo1 : Reference Router running on the NetFPGA– Inside the NetFPGA hardware– Breakneck introduction to Verilog– Exercise 1: Build your own Reference Router
• The Enhanced Reference Router– Motivation: Understanding buffer size requirements in a router– Demo 2: Observing and controlling the queue size– Using NetFPGA for research and teaching– Exercise 2: Enhancing the Reference Router
Th Lif f P k t Th h th N tFPGA• The Life of a Packet Through the NetFPGA
NICTA 2008 - NetFPGA Tutorial 91 S T A N F O R D U N I V E R S I T Y
Full System Components
Softwaref2 0 f2 1 f2 2 f2 3 i tl
PCI Bus
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
CPUR Q
CPUT Q
nf2_reg_grpCPU CPUCPU CPUCPU CPU
NetFPGARxQ TxQ
user data path
RxQ TxQCPURxQ
CPUTxQ
CPURxQ
CPUTxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
NICTA 2008 - NetFPGA Tutorial 92 S T A N F O R D U N I V E R S I T YEthernet
Life of a Packet through the hardware
port0 port2192.168.2.y192.168.1.x
NICTA 2008 - NetFPGA Tutorial 93 S T A N F O R D U N I V E R S I T Y
Router Stages Again
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
MACRxQ
CPURxQ
Input Arbiter
Output Port Lookup
Output Queues
MAC CPU MAC CPU MAC CPU MAC CPU
NICTA 2008 - NetFPGA Tutorial 94 S T A N F O R D U N I V E R S I T Y
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
MACTxQ
CPUTxQ
Inter-module Communication
Using “Module Headers”:Data WordCtrl Word
Module Hdrx Contain information such as packet
(64 bits)(8 bits)
Eth Hdr0Last Module Hdry
……p
length, input port, output port, …
IP Hdr…
00
Last word of packet0x10 Last word of packet0x10
NICTA 2008 - NetFPGA Tutorial 95 S T A N F O R D U N I V E R S I T Y
Inter-module Communication
data
ctrlwrwr
rdy
NICTA 2008 - NetFPGA Tutorial 96 S T A N F O R D U N I V E R S I T Y
MAC Rx Queue
NICTA 2008 - NetFPGA Tutorial 97 S T A N F O R D U N I V E R S I T Y
Rx Queue
Eth Hdr:
Pkt length,input port = 00xff
IP Hdr:IP Dst: 192 168 2 3
Dst MAC = port 0, Ethertype = IP
0
0
IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4
Data
0
0
NICTA 2008 - NetFPGA Tutorial 98 S T A N F O R D U N I V E R S I T Y
Input Arbiter
Pkt
PktPkt
NICTA 2008 - NetFPGA Tutorial 99 S T A N F O R D U N I V E R S I T Y
Pkt
Output Port Lookup
NICTA 2008 - NetFPGA Tutorial 100 S T A N F O R D U N I V E R S I T Y
Output Port Lookup1- Check input port matches
Dst MAC
5- Add output port module
Pkt length,input port = 00xff
output port = 40x04Dst MAC
2- Check TTL, checksum
6- Modify MAC Dst and Src
IP HdIP Hd
EthHdr: Dst MAC = 0Src MAC = x,Ethertype = IP
0
input port 0checksum
3- Lookup next hop IP &
addressesEthHdr: Dst MAC = nextHopSrc MAC = port 4,
Ethertype = IPIP Hdr:
IP Dst: 192.168.2.3, TTL: 64, Csum:0x3ab4
IP Hdr:IP Dst: 192.168.2.3,
TTL: 63, Csum:0x3ac2
D t
0
0
output port (LPM)
4- Lookup
7-Decrement TTL and update
checksumData04 Lookup next hop MAC address (ARP)
checksum
NICTA 2008 - NetFPGA Tutorial 101 S T A N F O R D U N I V E R S I T Y
Output Queues
OQ0OQ0
OQ4
OQ7
NICTA 2008 - NetFPGA Tutorial 102 S T A N F O R D U N I V E R S I T Y
MAC Tx Queue
NICTA 2008 - NetFPGA Tutorial 103 S T A N F O R D U N I V E R S I T Y
MAC Tx Queue
Pkt length,0xff
output port = 40x04
EthHdr: Dst MAC = nextHopSrc MAC = port 4,
Ethertype = IP0
input port = 00xff
IP Hdr:IP Dst: 192.168.2.3,
TTL: 64, Csum:0x3ab4
IP Hdr:IP Dst: 192.168.2.3,
TTL: 63, Csum:0x3ac2
Ethertype IP
0
Data0
NICTA 2008 - NetFPGA Tutorial 104 S T A N F O R D U N I V E R S I T Y
Exception Packet
• Example: TTL = 0 or TTL = 1• Packet has to be sent to the CPU which will
generate an ICMP packet as a response• Difference starts at the Output Port lookup p p
stage
NICTA 2008 - NetFPGA Tutorial 105 S T A N F O R D U N I V E R S I T Y
Exception Packet Path
Softwaref2 0 f2 1 f2 2 f2 3 i tl
PCI Bus
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
CPU CPU CPU CPU CPU CPU CPU CPU nf2_reg_grp
NetFPGA
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
user data path
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
NICTA 2008 - NetFPGA Tutorial 106 S T A N F O R D U N I V E R S I T YEthernet
Output Port Lookup1- Check input port matches
Dst MAC
Pkt length,input port = 00xff
output port = 10x04Dst MAC
2- Check TTL, checksum –
IP Hd
EthHdr: Dst MAC = 0,Src MAC = x,Ethertype = IP
0
input port 0checksum EXCEPTION!
IP Hdr:IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
D t
0
0
3- Add output port module
Data0
NICTA 2008 - NetFPGA Tutorial 107 S T A N F O R D U N I V E R S I T Y
Output Queues
OQ0OQ0
OQ1
OQ2
OQ7
NICTA 2008 - NetFPGA Tutorial 108 S T A N F O R D U N I V E R S I T Y
CPU Tx Queue
NICTA 2008 - NetFPGA Tutorial 109 S T A N F O R D U N I V E R S I T Y
CPU Tx Queue
Pkt length,0xff
output port = 10x04
EthHdr: Dst MAC = 0, Src MAC = x,Ethertype = IP
0
input port = 00xff
IP Hdr:IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
Ethertype IP
0
Data0
NICTA 2008 - NetFPGA Tutorial 110 S T A N F O R D U N I V E R S I T Y
ICMP Packet
• For the ICMP packet, the packet arrives at the CPU Rx Queue from the PCI Bus
• Follows the same path as a packet from the MAC until the Output Port Lookup.
• The OPL module seeing the packet is from the CPU Rx Queue 1, sets the output port directly to 0.
• The packet then continues on the same path as the non-exception packet to the Output Queues and then MAC Tx queue 0.
NICTA 2008 - NetFPGA Tutorial 111 S T A N F O R D U N I V E R S I T Y
ICMP Packet Path
Softwaref2 0 f2 1 f2 2 f2 3 i tl
PCI Bus
nf2c0 nf2c1 nf2c2 nf2c3 ioctl
CPU CPU CPU CPU CPU CPU CPU CPU nf2_reg_grp
NetFPGA
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
CPURxQ
CPUTxQ
user data path
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
MACTxQ
MACRxQ
NICTA 2008 - NetFPGA Tutorial 112 S T A N F O R D U N I V E R S I T YEthernet
NetFPGA-Host Interaction
• Linux driver interfaces with hardware– Packet interface via standard Linux network
stack
– Register reads/writes via ioctl system call (with convenience wrapper functions)
• readReg(nf2device *dev int address unsigned *rd data)• readReg(nf2device *dev, int address, unsigned *rd_data)• writeReg(nf2device *dev, int address, unsigned *wr_data)
eg:eg:readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);
NICTA 2008 - NetFPGA Tutorial 113 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
NetFPGA to host packet transfer1. Packet arrives –forwarding table sends to CPU queue
P2. Interrupt tifi 3 Driver sets upC
I Bus
notifies driver of packet arrival
3. Driver sets up and initiates DMA transfer
NICTA 2008 - NetFPGA Tutorial 114 S T A N F O R D U N I V E R S I T Y
NetFPGA-Host Interaction
NetFPGA to host packet transfer (cont)
PC4. NetFPGA t f
5. Interrupt signals C
I Bus
transfers packet via DMA
completion of DMA
6. Driver passes packet to network stack
NICTA 2008 - NetFPGA Tutorial 115 S T A N F O R D U N I V E R S I T Y
network stack
NetFPGA-Host Interaction
Host to NetFPGA packet transfers
PC
3. Interrupt signals
2. Driver sets up and initiates C
I Bus
completion of DMA
DMA transfer
1. Software sends packet via network sockets. Packet delivered to driver
NICTA 2008 - NetFPGA Tutorial 116 S T A N F O R D U N I V E R S I T Y
Packet delivered to driver.
NetFPGA-Host Interaction
Register access
PC
2. Driver performs C
I Bus
pPCI memory read/write
1. Software makes ioctl call on network socket. ioctl passed to driver
NICTA 2008 - NetFPGA Tutorial 117 S T A N F O R D U N I V E R S I T Y
ioctl passed to driver.
NetFPGA-Host Interaction
• Packet transfers shown using DMA interface
• Alternative: use programmed IO to transfer p gpackets via register reads/writes– slower but eliminates the need to deal with
network sockets
NICTA 2008 - NetFPGA Tutorial 118 S T A N F O R D U N I V E R S I T Y
Step 8 – Perfect the Router
If interested, go back to “Demo 2: Step 1” after synthesis is done and redo the steps with your ownsynthesis is done and redo the steps with your own router.
You can also change the bandwidth and queue size settings to see how that effects the queue occupancy evolutionevolution.
To run your router:1- cd NF2/projects/tutorial_router/sw2- type “./tut_adv_router_gui.pl --use_bin
/ / /bitfiles/tutorial router bit”
NICTA 2008 - NetFPGA Tutorial 119 S T A N F O R D U N I V E R S I T Y
../../../bitfiles/tutorial_router.bit
We’re done!We re done!
Congratulations!
NICTA 2008 - NetFPGA Tutorial 120 S T A N F O R D U N I V E R S I T Y
AcknowledgementsNetFPGA Team : January 2007
Jianying Luo, Glen Gibb, Nick McKeown, Greg Watson, Jim Weaver, Jad Naous, Ramanan Raghuraman,
Paul Hartke John Lockwood
NICTA 2008 - NetFPGA Tutorial 121 S T A N F O R D U N I V E R S I T Y
Paul Hartke, John Lockwood
Acknowledgements• Support for the NetFPGA project has been provided
by the following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in this
NICTA 2008 - NetFPGA Tutorial 122 S T A N F O R D U N I V E R S I T Y
y p g pmaterial do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.
Reference on the Web
NetFPGA homepagep g
http://NetFPGA.orgp g
NICTA 2008 - NetFPGA Tutorial 123 S T A N F O R D U N I V E R S I T Y