Multi Channel DMA for PCI Express IP Design Example User ......Multi Channel DMA for PCI Express IP...

Multi Channel DMA for PCI ExpressIP Design Example User Guide

Updated for Intel® Quartus® Prime Design Suite: 20.2

IP Version: 20.0.0

SubscribeSend Feedback

UG-20303 | 2020.08.05Latest document on the web: PDF | HTML

https://www.intel.com/content/www/us/en/programmable/bin/rssdoc?name=nki1590181067003

mailto:[email protected]?subject=Feedback%20on%20Multi%20Channel%20DMA%20for%20PCI%20Express%20IP%20Design%20Example%20User%20Guide%20(UG-20303%202020.08.05)&body=We%20appreciate%20your%20feedback.%20In%20your%20comments,%20also%20specify%20the%20page%20number%20or%20paragraph.%20Thank%20you.

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug20303.pdf

https://www.intel.com/content/www/us/en/programmable/documentation/nki1590181067003.html

Contents

1. Terms and Acronyms...................................................................................................... 3

2. Design Example Detailed Description..............................................................................42.1. Design Example Overview.......................................................................................42.2. Hardware and Software Requirements...................................................................... 42.3. Avalon-ST PIO using MCDMA Bypass mode............................................................... 5

2.3.1. Simulation Result.......................................................................................52.3.2. Hardware Test Result................................................................................. 6

2.4. Avalon-ST Packet Generate/Check........................................................................... 62.4.1. Simulation Results..................................................................................... 72.4.2. Hardware Test Results................................................................................ 9

2.5. Avalon-ST Device-side Packet Loopback.................................................................. 122.5.1. Simulation Results................................................................................... 142.5.2. Hardware Test Results.............................................................................. 15

2.6. Avalon-MM PIO using MCDMA Bypass mode.............................................................172.6.1. Simulation Results................................................................................... 172.6.2. Hardware Test Results.............................................................................. 18

2.7. Avalon-MM DMA...................................................................................................182.7.1. Simulation Results................................................................................... 192.7.2. Hardware Test Results.............................................................................. 19

3. Design Example Quick Start Guide................................................................................ 223.1. Design Example Directory Structure....................................................................... 223.2. Generating the Example Design using Intel Quartus Prime.........................................23

3.2.1. Procedure............................................................................................... 233.3. Simulating the Design Example..............................................................................25

3.3.1. Testbench Overview................................................................................. 253.3.2. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/

Check Design Example..............................................................................263.3.3. Run the Simulation Script..........................................................................273.3.4. View the Results...................................................................................... 28

3.4. Compiling the Example Design in Intel Quartus Prime............................................... 283.5. Running the Design Example Application.................................................................29

3.5.1. Program the FPGA....................................................................................293.5.2. Set Up the Linux Software.........................................................................293.5.3. Run the Test Application Software.............................................................. 31

4. Revision History............................................................................................................32

Contents

Multi Channel DMA for PCI Express IP Design Example User Guide Send Feedback

2


1. Terms and AcronymsTable 1. Acronyms

Term Definition

PCIe* Peripheral Component Interconnect Express (PCI Express*)

DMA Direct Memory Access

MCDMA Multi Channel Direct Memory Access

PIO Programmed Input/Output

H2D Host-to-Device

D2H Device-to-Host

H2DDM Host-to-Device Data Mover

D2HDM Device-to-Host Data Mover

QCSR Queue Control and Status register

GCSR General Control and Status Register

IP Intellectual Property

HIP Hard IP

PD Packet Descriptor

QID Queue Identification

TIDX Queue Tail Index (pointer)

HIDX Queue Head Index (pointer)

TLP Transaction Layer Packet

IMMWR Immediate Write Operation

MRRS Maximum Read Request Size

CvP Configuration via Protocol

PBA Pending Bit Array

Avalon®-MM Avalon Memory-Mapped Interface

Avalon-ST Avalon Streaming Interface

UG-20303 | 2020.08.05

Send Feedback

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/orother countries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.

ISO9001:2015Registered


https://www.intel.com/content/www/us/en/quality/intel-iso-registrations.html



2. Design Example Detailed Description

2.1. Design Example Overview

The Multi Channel DMA for PCI Express IP Design Examples demonstrate a MultiChannel DMA solution for Intel® Stratix® 10 devices using the H-Tile Gen3 x16 hard IPand soft IP implemented in the FPGA fabric.

You can generate the design example from the Example Designs tab of the MultiChannel DMA for PCI Express IP Parameter Editor. For user interface, you can chooseeither Avalon-ST or Avalon-MM Interface. You can allocate up to 8 DMA channels whenAvalon-MM Interface type is selected. For Avalon-ST Interface, DMA channel andAvalon-ST port has 1:1 mapping. You can also configure the PCIe BAR2 size that ismapped to the Avalon-MM PIO Master port.

2.2. Hardware and Software Requirements

• Intel Quartus® Prime Pro Edition Software version 20.2

• Modelsim, VCS, or NCSim

• Intel Stratix 10 MX or GX FPGA Development Kit

For details on the design example simulation steps and running Hardware test, refer tothe Quick Start Guide.

For more information on development kits, refer to Intel Stratix 10 FPGA DevelopmentKits on Intel website.

UG-20303 | 2020.08.05

Send Feedback



https://www.intel.com/content/www/us/en/programmable/products/boards_and_kits/all-development-kits.html

https://www.intel.com/content/www/us/en/programmable/products/boards_and_kits/all-development-kits.html





2.3. Avalon-ST PIO using MCDMA Bypass mode

Figure 1. Avalon-ST PIO using MCDMA Bypass mode

ninit_done

AVMM

Design Example Platform Designer System

Multi Channel DMAfor PCI Express

rx_pio_master

H2DDMA

PCIeHIP

Hosthip

_ser

ial

D2HDMA

resetIP

MEM_PIO

h2d_st_0h2d_st_1h2d_st_2h2d_st_3

d2h_st_0d2h_st_1d2h_st_2d2h_st_3

This design example enables Avalon-MM PIO master which bypasses the DMA path.The Avalon-MM PIO master allows application to perform single, non-bursting registerread/write operation with on-chip memory.

The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and following components:

• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until theentire Intel Stratix 10 FPGA fabric enters user mode

• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMAAvalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2

Transfer mode option supported in test application software (perfq_app) commandline:

• PIO test: -o

2.3.1. Simulation Result

Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. This design example testbench doesn’t simulate H2D/D2Hdata movers.

Figure 2. Simulation Log


UG-20303 | 2020.08.05

Send Feedback Multi Channel DMA for PCI Express IP Design Example User Guide

5


Figure 3. Simulation Waveform

2.3.2. Hardware Test Result

Figure 4. PIO Test-o option

2.4. Avalon-ST Packet Generate/Check

Figure 5. Avalon-ST Packet Generate/Check

ninit_done

AVMM

AVSTAVSTAVSTAVST



rx_pio_master

H2DDMA

PCIeHIP

Host

hip_s

erial

AVSTAVSTAVSTAVST

D2HDMA

resetIP

MEM_PIO


GEN_CHKd2h_st_0d2h_st_1d2h_st_2d2h_st_3

This design example performs H2D and D2H multi channel DMA via Avalon-STstreaming. The Multi Channel DMA for PCI Express IP core provides four independentAvalon-ST Source/Sink ports. DMA channel and Avalon-ST port has 1:1 mapping.

For H2D streaming, Multi Channel DMA sends the data to Avalon-ST packet checkervia four Avalon-ST Source ports. The Packet Checker validates the received data. ForD2H streaming, Multi Channel DMA receives the data from Avalon-ST packet generatorvia Avalon-ST Sink ports.


UG-20303 | 2020.08.05


6


In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block. Also, test application software, perfq_app,uses the Avalon-MM PIO Master port to configure the Packet Generator and Checker.




• GEN_CHK – Packet Generator and Checker for MCDMA. Connected to the MCDMAAvalon-ST Source (h2d_st_x) and Avalon-ST Sink (d2h_st_x) ports

Transfer mode Options supported in test application software (perfq_app) commandline:

• PIO test: -o

• DMA test: -t (Tx), -r (Rx), -z (Bidirection)

2.4.1. Simulation Results

Note: For detailed description about the testbench for this design example, refer to ExampleTestbench Flow for DMA Test with Avalon-ST Packet Generate/Check Design Exampleon page 26.

Figure 6. H2D Simulation Log


UG-20303 | 2020.08.05


7


Figure 7. H2D Simulation Waveform

Figure 8. D2H Simulation Log


UG-20303 | 2020.08.05


8


Figure 9. D2H Simulation Waveform

2.4.2. Hardware Test Results


Figure 11. H2D Avalon-ST Streaming-t option

Note: The perfq_app command transfers 1 GB of total transfer size with payload of 8192bytes in each descriptor in H2D direction (-t) for four channels. The File size is 127,i.e. sof is on the 1st descriptor and eof on the 127th descriptor. Without -v (datavalidtion) option, the command displays the bandwidth.


UG-20303 | 2020.08.05


9


Figure 12. H2D Avalon-ST Streaming with Data Validation Enabled-t with -v option

Figure 13. D2H Avalon-ST Streaming-r option


UG-20303 | 2020.08.05


10


Figure 14. D2H Avalon-ST Streaming with Data Validation Enabled-r with -v option

Figure 15. Bidirectional Avalon-ST Streaming-z option


UG-20303 | 2020.08.05


11


Figure 16. Bidirectional Avalon-ST Streaming with Data Validation Enabled-z with -v option

2.5. Avalon-ST Device-side Packet Loopback

Figure 17. Avalon-ST Device-side Packet Loopback

ninit_done

AVMM

AVSTAVSTAVSTAVST



rx_pio_master

H2DDMA

PCIeHIP

Host

hip_s

erial

AVSTAVSTAVSTAVST

D2HDMA

resetIP

MEM_PIO


FIFO_ST0FIFO_ST1FIFO_ST2FIFO_ST3

d2h_st_0d2h_st_1d2h_st_2d2h_st_3


UG-20303 | 2020.08.05


12


This design example performs H2D and D2H multi channel DMA via Avalon-STstreaming. The Multi Channel DMA for PCI Express IP core provides four independentAvalon-ST Source/Sink ports. DMA channel and Avalon-ST port has 1:1 mapping.

For H2D streaming, Multi Channel DMA sends the data to Avalon-ST loopback FIFOsvia four Avalon-ST Source ports. For D2H streaming, Multi Channel DMA receives thedata from Avalon-ST loopback FIFOs via Avalon-ST Sink ports.

In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block.




• FIFO_ST0, FIFO_ST1, FIFO_ST2, and FIFO_ST3 – Avalon-ST FIFOs forstreaming loopback. Connected to the MCDMA Avalon-ST Source (h2d_st_x) andAvalon-ST Sink (d2h_st_x) ports

Transfer mode options supported in test application software (perfq_app) commandline:

• PIO test: -o

• DMA test: -i (performance loopback operation where the Tx and Rx are run in twodifferent threads), -v (enable data validation)

— -i without -v flag displays the throughput per channel


UG-20303 | 2020.08.05


13





UG-20303 | 2020.08.05


14


Figure 19. H2D Simulation Waveform

Figure 20. D2H Simulation Waveform




UG-20303 | 2020.08.05


15


Figure 22. Performance Test-i option

Figure 23. Data Validation Test-i with -v option


UG-20303 | 2020.08.05


16


2.6. Avalon-MM PIO using MCDMA Bypass mode

Figure 24. Avalon-MM PIO using MCDMA Bypass mode

ninit_done

AVMM



rx_pio_master

h2ddm_master

H2DDMA

PCIeHIP

Hosthip

_ser

ial

d2hdm_master

D2HDMA

resetIP

MEM_PIO

This design example enables Avalon-MM PIO master which bypasses the DMA path.The Avalon-MM PIO master allows application to perform single, non-bursting registerread/write operation with on-chip memory.

The design example includes the Multi Channel DMA for PCI Express IP Core with theparameters you specified and other supporting components:



Transfer mode option supported in test application software (perfq_app) commandline:

• PIO test: -o


Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. This design example testbench doesn’t simulate H2D/D2Hdata movers.



UG-20303 | 2020.08.05


17





2.7. Avalon-MM DMA

Figure 28. Avalon-MM DMA

ninit_done

AVMM



rx_pio_master

h2ddm_master

H2DDMA

PCIeHIP

Host

hip_s

erial

d2hdm_master

D2HDMA

resetIP

MEM_PIO

AVMM

AVMM

MEM

This design example performs H2D and D2H multi channel DMA via Avalon-MMmemory-mapped interface. The Multi Channel DMA for PCI Express IP core providesone Avalon-MM Write/Read Master port. You can allocate up to eight DMA channelswhen generating this example design.

For H2D DMA, Multi Channel DMA H2D data mover writes the data to on-chip memoryvia Avalon-MM Write Master port. For D2H DMA, Multi Channel DMA D2H data moverreads the data from on-chip memory via Avalon-MM Read Master port.

In addition, the design example enables Avalon-MM PIO master which bypasses theDMA path. It allows application to perform single, non-bursting register read/writeoperation with on-chip memory block.


UG-20303 | 2020.08.05


18





• MEM – Dual port on-chip memory. One port is connected to the Avalon-MM WriteMaster (h2ddm_master) and the other port to Avalon-MM Read Master(d2hdm_master)

Transfer mode options supported in test application software (perfq_app) commandline:

• PIO test: -o

• DMA test: -t (Tx), -r (Rx)


Testbench writes 4 KB of incrementing pattern to on-chip memory and read back viaAvalon-MM PIO interface. In the current release, this design example testbench doesnot simulate data movement through Avalon-MM Write and Read Master ports.






UG-20303 | 2020.08.05


19


Figure 32. H2D Avalon-MM Write-t option

Figure 33. H2D Avalon-MM Write with Data Validation Enabled-t -v option


UG-20303 | 2020.08.05


20


Figure 34. D2H Avalon-MM Read-r option


UG-20303 | 2020.08.05


21


3. Design Example Quick Start GuideUsing Intel Quartus Prime software, you can generate a design example for the MultiChannel DMA for PCI Express (PCIe) IP core.

The generated design example reflects the parameters that you specify. The designexample automatically creates the files necessary to simulate and compile in the IntelQuartus Prime software. You can download the compiled design to your FPGADevelopment Board. To download to custom hardware, update the Intel Quartus PrimeSettings File (.qsf) with the correct pin assignments.

Figure 35. Design Example Development Steps

DesignExample

Generation

Compilation(Simulator)

FunctionalSimulation

HardwareTesting

Compilation(Quartus Prime)

3.1. Design Example Directory Structure

Table 2. Directory Structure

Directory / File Sub-directory /File

Sub-directory /File

Sub-directory /File

Sub-directory /File

Note

pcie_ed

sim

pcie_ed.v Design exampletop-level HDL

<simulators> <simulation scripts> pcie_edsimulationdirectory

synth pcie_ed.v Design exampletop-level HDL

<Components automatically generated by Platform Designer>

pcie_ed_tb pcie_ed_tb simpcie_ed_tb.v Testbench

including IntelFPGA BFM

continued...

UG-20303 | 2020.08.05

Send Feedback







Directory / File Sub-directory /File

Sub-directory /File

Sub-directory /File

Sub-directory /File

Note

<simulators> <simulationscript>

Testbenchsimulationdirectory

ip pcie_ed_tb DUT_pcie_tb_ip Intel FPGA BFM(RP)

pcie_ed_tb.qsys TestbenchPlatform Designerfile

pcie_ed.ipx

software

kernel

common

driver kmod <kernel driverfiles>

Kernel driver

Licenses

user

cli

perfq_app

<test applicationsoftware>

Test Application

README Readme file

sample ref.c Reference APIflow

common

include regs MCDMA and PktGen/Chk registers

mk

src

libmqdma <user space library files> User space library

Licenses

Readme Readme file

readme Readme file

ip pcie_ed <Design example IP components>

pcie_ed.qpf Quartus projectfile

pcie_ed.qsf Quartus settingfile

pcie_ed.qsys Design examplePlatform Designerfile

3.2. Generating the Example Design using Intel Quartus Prime

Figure 36. Design Example Generation

Start ParameterEditor

Specify IP Variationand Select Device

SelectDesign Parameters

SpecifyExample Design andSelect Target Board

InitiateDesign Generation

3.2.1. Procedure

3. Design Example Quick Start Guide

UG-20303 | 2020.08.05


23


1. In the Intel Quartus Prime Pro Edition software, create a new project (File → NewProject Wizard).

2. Specify the Directory, Name, and Top-Level Entity.

3. For Project Type, accept the default value, Empty project. Click Next.

4. For Add Files click Next.

5. For Family, Device & Board Settings, select Intel Stratix 10(GX/SX/MX/TX) and the Target Device for your design.

Note: The selected device is only used if you select None in Step 10f below.

6. Click Finish.

7. In the IP Catalog locate and add the Multi Channel DMA for PCI Express*which brings up the IP Parameter Editor.

8. In the New IP Variant dialog box, specify a name for your IP. Click Create.

9. On the IP Settings tabs, specify the parameters for your IP variation.

10. On the Example Designs tab, make the following selections:

a. For Currently Selected Example Design, select a design example from apulldown menu.

b. Available design examples depends on the Interface type setting in MCDMASettings under IP Settings tab. Available design examples for Avalon-STInterface type:

• PIO using DMA Bypass Mode

• Packet Generate/Check

• Device-side Packet Loopback

Available design examples for Avalon-MM Interface type:

• PIO using DMA Bypass Mode

• Avalon-MM DMA

c. For Example Design Files, turn on the Simulation and Synthesis options.If you do not need these simulation or synthesis files, leaving thecorresponding option(s) turned off significantly reduces the example designgeneration time.

d. For Select simulation Root Complex BFM, choose the appropriate BFM:

i. Intel FPGA BFM: This bus functional model (BFM) supports x16configurations by down training to x8.

ii. Third-party BFM: If you want to simulate all 16 lanes, use a third-partyBFM. If you have an Avery BFM installed and need information aboutsimulating with the Avery BFM, contact your local FAE or salesrepresentative.

e. For Generated HDL Format, only Verilog is available in the current release.

f. For Target Development Kit, select the appropriate option.

Note: If you select None, the generated design example targets the devicespecified. Otherwise, the design example uses the device on theselected development board. If you intend to test the design inhardware, make the appropriate pin assignments in the .qsf file.


UG-20303 | 2020.08.05


24


11. Select Generate Example Design to create a design example that you cansimulate and download to hardware. If you select one of the Intel Stratix 10development boards, the device on that board supersedes the device previouslyselected in the Intel Quartus Prime Pro Edition project if the devices are different.When the prompt asks you to specify the directory for your example design, youcan choose to accept the default directory ./intel_pcie_mcdma_0_example_design or choose another directory.

12. Click Close on Generate Example Design Completed message.

13. Close the IP Parameter Editor. Click File → Exit. When prompted with Savechanges?, you do not need to save the .ip. Click Don’t Save.

3.3. Simulating the Design Example

3.3.1. Testbench Overview

Figure 37. Testbench Platform Designer View

Testbench Platform Designer file path: pcie_ed_tb/pcie_ed_tb.qsys

The design example, pcie_ed_inst, is generated with x16. The Intel FPGA BFM,DUT_pcie_tb, can support up to x8 link. The BFM supports the testbench simulationby down-training to x8 link. If you want to simulate x16 link, you can use a third-partyBFM.

The testbench uses a Root Port driver module, altpcietb_bfm_rp_gen3_x8.sv(Path: pcie_ed_tb/ip/pcie_ed_tb/DUT_pcie_tb_ip/altera_pcie_s10_tbed_191/sim), to exercise the target memory and DMAchannel in the Endpoint. This is the module that you can modify to vary thetransactions sent to the example Endpoint design or your own design.

For more information about Intel FPGA BFM, refer to Intel Stratix 10 Avalon streamingand SR-IOV Interface for PCI Express Solutions User Guide (Section 9.3 Root Port BFMOverview).

Note: The Endpoint BAR0 is configured for 64-bit 4 MB address space used for the MultiChannel DMA control registers. The BAR2 is used for 64-bit PIO Access.


UG-20303 | 2020.08.05


25


Related Information

Intel Stratix 10 Avalon streaming and SR-IOV Interface for PCI Express Solutions UserGuide (Section 9.3 Root Port BFM Overview)

3.3.2. Example Testbench Flow for DMA Test with Avalon-ST PacketGenerate/Check Design Example

The DMA testbench for the Avalon-ST Packet Generate/Check design exampledemonstrates the following two major tasks:

• Host-to-Device: Transferring packets stored in the host memory to the PacketChecker in the design example user logic, where a checker module verifies theintegrity of the packet

• Device-to-Host: Packets generated from a Generator module are transferred to thehost memory where the host checks the packet integrity

Note: This testbench implements transfer of one packet with length of 4096 bytes.

The DMA testbench for the design example completes the following tasks for each ofthe 4 ports supported by the DUT:

1. Set up 4096 bytes of incrementing data pattern for testing data movement fromthe host to the device and then back to the host.

2. Write the expected packet length value (4096 bytes) to the Packet Generation andChecker in the design example user logic through the PIO. This value is used bythe Packet checker module for testing packet integrity.

3. MSI-X is enabled and configured for launching a memory write to signal the end ofeach descriptor’s DMA transaction. Write-Back function is kept disabled for thesimulation.

4. Set up the H2D (Host-to-Device) queue in the Multi Channel DMA.

5. Set up three H2D descriptors in the host memory, with the source addresspointing to the incrementing data pattern locations in the host memory. The startof packet (SOF) and end of packet (EOF) markers along with packet length areindicated in the descriptors.

6. At the last step of the Queue programming, the Multi Channel DMA tail pointerregister is written, which triggers the Multi Channel DMA to start the H2D DMAtransaction.

7. The previous step instructs the H2D Data Mover to fetch the descriptors from thehost memory.

8. The Multi Channel DMA H2D Data Mover reads the data from the host memory andforwards the packet to the Packet Generator and Checker through the AVSTStreaming interface.

9. The checker module receives the packet and checks for integrity by testing thedata pattern, length as expected and proper receipt of the “end of packet” marker.If the packet is found to be proper, the good packet count is incremented by 1 elsethe bad packet count is incremented.

10. The testbench does a PIO read access of the Good Packet Count and Bad PacketCount registers and displays the test success or failure status.

11. MSI-X write commands are triggered for every description or completion which arechecked by the testbench for proper receipt.


UG-20303 | 2020.08.05


26

https://www.intel.com/content/www/us/en/programmable/documentation/lbl1464284700752.html#nik1410565006482

https://www.intel.com/content/www/us/en/programmable/documentation/lbl1464284700752.html#nik1410565006482


12. Next, set up the D2H (Device-to-Host) Queue.

13. Setup three D2H descriptors in the host memory, with the destination addresspointing to a new address space in host memory which is pre-filled with all zeroes.

14. At the last step of the Queue programming, the Multi Channel DMA tail pointerregister is written, which triggers the Multi Channel DMA to start the D2H DMAtransaction.

15. The previous step instructs the H2D Data Mover to fetch the descriptors from thehost memory to start the D2H DMA transaction.

16. The Multi Channel DMA D2H Data Mover reads the incoming packet from thePacket Generator and writes the data to the host memory according to thedescriptors fetched in the previous step.

17. MSI-X write commands are triggered for every description completion which arechecked by the testbench for proper receipt.

18. Compares the data written back to the system memory in D2H task with thestandard incrementing pattern and declare test success/failure.

The simulation reports Simulation stopped due to successful completionif no errors occur.

3.3.3. Run the Simulation Script

Figure 38. Simulation Script

Change toTestbench Directory

Run<Simulation Script>

AnalyzeResults

1. Change to the testbench simulation directory, pcie_ed_tb/pcie_ed_tb/sim/<simulators>.

2. Run the simulation script for the simulator of your choice. Refer to the table below.

3. Analyze the results.

Table 3. Steps to run the simulation

Simulator Simulation Directory Instructions

ModelSim <example_design>/pcie_ed_tb/pcie_ed _tb/sim/mentor/

1. Invoke vsim (by typing vsim, whichbrings up a console window whereyou can run the followingcommands).

2. do msim_setup.tcl

continued...


UG-20303 | 2020.08.05


27


Simulator Simulation Directory Instructions

Note: Alternatively, instead ofdoing Steps 1 and 2, youcan type: vsim -c -domsim_setup.tcl

3. ld_debug

4. run -all

5. A successful simulation ends withthe following message:"Simulation stopped due tosuccessful completion!"

VCS <example_design> /pcie_ed_tb/pcie_ed _tb/sim/synopsys/vcs

1. sh vcs_setup.sh USER_DEFINED_COMPILE_OPTIONS="" USER_DEFINED_ELAB_OPTIONS="-xlrm\ uniq_prior_final" USER_DEFINED_SIM_OPTIONS=""


NCSim <example_design>/pcie_ed_tb/pcie_ed_tb/sim/cadence

1. sh ncsim_setup.sh USER_DEFINED_SIM_OPTIONS="" USER_DEFINED_ELAB_OPTIONS ="-timescale\ 1ns/1ps"


3.3.4. View the Results

To view the Simulation Logs, Simulation Waveforms and Hardware Test Results foreach design example, refer to Design Example Detailed Descirption chapter of thisdocument.

3.4. Compiling the Example Design in Intel Quartus Prime

To compile the example design, follow these steps:

1. Navigate to the design example directory,intel_pcie_mcdma_0_example_design, and open the Intel Quartus Primeproject file, pcie_ed.qpf in Intel Quartus Prime Pro Edition software.

2. On the Processing menu, select Start Compilation.

Figure 39. Design Example Compilation


UG-20303 | 2020.08.05


28


3.5. Running the Design Example Application

3.5.1. Program the FPGA

1. Connect a FPGA programming cable to the Intel Stratix 10 FPGA DevelopmentBoard

2. On the Tools menu, select Programmer

3. In the Programmer, click Hardware Setup and verify the Intel Stratix 10 FPGADevelopment Board is detected in Hardware Setting tab and JTAG Settings tab

4. Select Auto Detect to detect the JTAG device chain

5. Select the target FPGA device in the JTAG chain, select Change File, and selectthe pcie_ed.sof

6. Select Start to start programming

Figure 40. Programming Stratix 10 MX FPGA Development Board

3.5.2. Set Up the Linux Software

3.5.2.1. Set the default huge pages

1. Modify the default huge pages setting in grub files as follows:

Edit /etc/default/grub file. Add the following in GRUB_CMDLINE_LINUXparamenter:

"default_hugepagesz=1G hugepagesz=1G hugepages=40"

After the edit, the file will look as below:

GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved


UG-20303 | 2020.08.05


29


GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="default_hugepagesz=1G intel_iommu=on iommu=pt intel_pstate=disable hugepagesz=1G hugepages=40 panic=1 crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet" GRUB_DISABLE_RECOVERY="true"

2. Generate GRUB configuration files. Check if /sys/firmware/efi exists. If itexists, the system is EFI based. Otherwise, the system is a legacy system.

In case of legacy system execute following command

$ grub2-mkconfig -o /boot/grub2/grub.cfg

In case of EFI based system, execute following command

$ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

3. Reboot the system.

4. Verify the above changes

$ cat /proc/cmdline

The output should include the following:

default_hugepagesz=1G hugepagesz=1G hugepages=40

5. Set the huge pages

$ echo 10 > /proc/sys/vm/nr_hugepages

3.5.2.2. Install the Linux Kernel Driver

1. Install the UIO driver

$ modprobe uio

2. Build the Multi Channel DMA kernel driver and load

$ cd software/kernel$ make -C driver/kmod/

3. Install the kernel driver

$ insmod driver/kmod/ifc_uio.ko

4. Verify if the kernel driver is loaded

$ lspci -d 1172:000 -v | grep ifc_uio

Kernel driver in use: ifc_uio

3.5.2.3. Build and Install User Space Library

1. Build the library

$ cd software/user $ make -C libmqdma/

2. Load the library


UG-20303 | 2020.08.05


30


For 64 bit system:

$ rm -f /usr/lib64/libmqdmasoc.so$ cp libmqdma/libmqdmasoc.so /usr/lib64/

For 32 bit system:

$ rm -f /usr/lib/libmqdmasoc.so$ cp libmqdma/libmqdmasoc.so /usr/lib/

3. Verify that ldconfig output contains libmqdma

$ ldconfig -v | grep libmqdmasoc.so

3.5.3. Run the Test Application Software

1. Build the perfq_app test application software and check the available commandline options using -h.

$ cd cli/perfq_app/$ make clean$ make$ ./perfq_app -h

Note: For more information on perfq_app command options, refer to the READMEfile located in software/user/cli/perfq_app directory.

2. Perform the PIO test to check if the hardware setup is correct, if successful, theapplication will show as Pass status as shown below:

3. Perform the DMA test with design example. Refer to Hardware Test results foreach design example in the Design Example Detailed Description chapter.


UG-20303 | 2020.08.05


31


4. Revision HistoryTable 4. Revision History for Multi Channel DMA for PCI Express IP Design Example

User Guide

Date Intel QuartusPrime Version

IP Version Changes

2020.08.05 20.2 20.0.0 Initial Release

UG-20303 | 2020.08.05

Send Feedback







Multi Channel DMA for PCI Express IP Design Example User ......Multi Channel DMA for PCI Express IP...

Documents

Transcript of Multi Channel DMA for PCI Express IP Design Example User ......Multi Channel DMA for PCI Express IP...