Intel® Stratix® 10 Avalon-ST Interface for PCIe Solutions ...1 Introduction 1.1 Avalon-ST...

Intel® Stratix® 10 Avalon-STInterface for PCIe Solutions UserGuideUG-200322017.05.08

Last updated for Intel® Quartus® Prime Design Suite: Quartus Prime Pro v17.1 Stratix 10 Editions

SubscribeSend Feedback

https://www.altera.com/servlets/subscriptions/alert?id=UG-20032

mailto:[email protected]?subject=Feedback%20on%20Intel%20Stratix%2010%20Avalon-ST%20Interface%20for%20PCIe%20Solutions%20User%20Guide%20(UG-20032%202017.05.08)&body=We%20appreciate%20your%20feedback.%20In%20your%20comments,%20also%20specify%20the%20page%20number%20or%20paragraph.%20Thank%20you.

Contents

1 Introduction.................................................................................................................... 61.1 Avalon-ST Interface for PCIe Introduction.................................................................. 61.2 Features................................................................................................................71.3 Release Information ...............................................................................................81.4 Device Family Support ............................................................................................81.5 Recommended Speed Grades................................................................................... 91.6 Performance and Resource Utilization ....................................................................... 91.7 Transceiver Tiles................................................................................................... 101.8 PCI Express IP Core Package Layout........................................................................111.9 Channel Availability...............................................................................................15

2 Quick Start Guide...........................................................................................................172.1 Design Components.............................................................................................. 182.2 Directory Structure............................................................................................... 192.3 Generating the Design...........................................................................................202.4 Simulating the Design........................................................................................... 212.5 Compiling the Design............................................................................................ 23

3 Interface Overview........................................................................................................ 25

4 Parameters ................................................................................................................... 314.1 Stratix 10 Avalon-ST Settings................................................................................. 314.2 Base Address Registers..........................................................................................324.3 Device Identification Registers................................................................................324.4 PCI Express and PCI Capabilities Parameters............................................................ 33

4.4.1 Device Capabilities.................................................................................... 334.4.2 Link Capabilities .......................................................................................334.4.3 MSI and MSI-X Capabilities ....................................................................... 344.4.4 Slot Capabilities .......................................................................................344.4.5 Power Management ..................................................................................354.4.6 Vendor Specific Extended Capability (VSEC)................................................. 36

4.5 Configuration, Debug and Extension Options............................................................ 364.6 PHY Characteristics .............................................................................................. 364.7 Stratix 10 Example Designs....................................................................................37

5 Designing with the IP Core............................................................................................ 385.1 Generation...........................................................................................................385.2 Simulation........................................................................................................... 39

5.2.1 Selecting Serial or PIPE Simulation .............................................................405.2.2 Reducing Counter Values for Simulation....................................................... 40

5.3 Files Generated for Intel FPGA IP Cores and Qsys Pro Systems....................................405.4 Integration and Implementation..............................................................................43

5.4.1 Clock Requirements...................................................................................435.4.2 Reset Requirements.................................................................................. 44

5.5 Required Supporting IP Cores.................................................................................445.5.1 Hard Reset Controller................................................................................ 455.5.2 TX PLL.....................................................................................................45

5.6 Channel Layout and PLL Usage............................................................................... 45

Contents

Intel® Stratix® 10 Avalon-ST Interface for PCIe Solutions User Guide2

5.7 Compiling the Full Design and Programming the FPGA............................................... 50

6 Block Descriptions......................................................................................................... 516.1 Interfaces............................................................................................................52

6.1.1 TLP Header and Data Alignment for the Avalon-ST RX and TX Interfaces.......... 536.1.2 Avalon-ST RX Interface..............................................................................546.1.3 Avalon-ST TX Interface.............................................................................. 576.1.4 TX Credit Interface....................................................................................606.1.5 Interpreting the TX Credit Interface.............................................................616.1.6 Clocks..................................................................................................... 636.1.7 Update Flow Control Timer and Credit Release.............................................. 636.1.8 Resets.....................................................................................................636.1.9 Interrupts................................................................................................ 656.1.10 Transaction Layer Configuration Space Interface..........................................666.1.11 Configuration Extension Bus Interface........................................................686.1.12 Hard IP Status Interface...........................................................................686.1.13 Serial Data Interface............................................................................... 706.1.14 PIPE Interface.........................................................................................706.1.15 Hard IP Reconfiguration........................................................................... 736.1.16 Power Management Interface....................................................................746.1.17 Test Interface......................................................................................... 746.1.18 Message Handling................................................................................... 76

6.2 Errors reported by the Application Layer.................................................................. 776.2.1 Error Handling..........................................................................................78

6.3 Power Management...............................................................................................796.3.1 Endpoint D3 Entry ....................................................................................796.3.2 End Point D3 Exit...................................................................................... 806.3.3 Exit from D3 hot ...................................................................................... 806.3.4 Exit from D3 cold......................................................................................806.3.5 Active State Power Management................................................................. 80

6.4 Transaction Ordering............................................................................................. 806.4.1 TX TLP Ordering........................................................................................806.4.2 RX TLP Ordering....................................................................................... 81

6.5 RX Buffer.............................................................................................................816.5.1 Retry Buffer............................................................................................. 826.5.2 Configuration Retry Status......................................................................... 82

7 Interrupts...................................................................................................................... 837.1 Interrupts for Endpoints.........................................................................................83

7.1.1 MSI Interrupts .........................................................................................837.1.2 MSI-X .....................................................................................................867.1.3 Implementing MSI-X Interrupts.................................................................. 867.1.4 Legacy Interrupts .....................................................................................88

8 Registers....................................................................................................................... 898.1 Configuration Space Registers................................................................................ 89

8.1.1 Type 0 Configuration Space Registers.......................................................... 918.1.2 PCI Express Capability Structures................................................................928.1.3 JTAG Silicon ID ........................................................................................ 948.1.4 Intel Defined VSEC Capability Header ......................................................... 958.1.5 Intel Defined Vendor Specific Header........................................................... 968.1.6 Intel Marker ............................................................................................ 96

Contents


8.1.7 User Device and Board Type ID ..................................................................968.1.8 General Purpose Control and Status Register................................................ 968.1.9 Uncorrectable Internal Error Status Register ................................................ 968.1.10 Uncorrectable Internal Error Mask Register................................................. 978.1.11 Correctable Internal Error Status Register ..................................................988.1.12 Correctable Internal Error Mask Register ................................................... 98

8.2 Avalon-MM DMA Bridge Registers............................................................................ 998.2.1 PCI Express Avalon-MM Bridge Register Address Map.....................................99

9 Testbench and Design Example ...................................................................................1069.1 Endpoint Testbench ............................................................................................ 1079.2 Chaining DMA Design Examples ........................................................................... 108

9.2.1 BAR/Address Map ...................................................................................1139.2.2 Chaining DMA Control and Status Registers ............................................... 1149.2.3 Chaining DMA Descriptor Tables ............................................................... 116

9.3 Test Driver Module ............................................................................................. 1189.4 DMA Write Cycles ...............................................................................................1199.5 DMA Read Cycles ............................................................................................... 1219.6 Root Port BFM ....................................................................................................122

9.6.1 BFM Memory Map ...................................................................................1249.6.2 Configuration Space Bus and Device Numbering ......................................... 1249.6.3 Configuration of Root Port and Endpoint .................................................... 1249.6.4 Issuing Read and Write Transactions to the Application Layer ....................... 130

9.7 BFM Procedures and Functions .............................................................................1309.7.1 ebfm_barwr Procedure ............................................................................1309.7.2 ebfm_barwr_imm Procedure ....................................................................1319.7.3 ebfm_barrd_wait Procedure .....................................................................1319.7.4 ebfm_barrd_nowt Procedure ....................................................................1329.7.5 ebfm_cfgwr_imm_wait Procedure .............................................................1329.7.6 ebfm_cfgwr_imm_nowt Procedure ............................................................1339.7.7 ebfm_cfgrd_wait Procedure ..................................................................... 1339.7.8 ebfm_cfgrd_nowt Procedure .................................................................... 1349.7.9 BFM Configuration Procedures...................................................................1349.7.10 BFM Shared Memory Access Procedures .................................................. 1369.7.11 BFM Log and Message Procedures ...........................................................1389.7.12 Verilog HDL Formatting Functions ........................................................... 1419.7.13 Procedures and Functions Specific to the Chaining DMA Design Example....... 144

9.8 Setting Up Simulation..........................................................................................1489.8.1 Changing Between Serial and PIPE Simulation ............................................1489.8.2 Using the PIPE Interface for Gen1 and Gen2 Variants .................................. 1499.8.3 Viewing the Important PIPE Interface Signals..............................................1499.8.4 Disabling the Scrambler for Gen1 and Gen2 Simulations ..............................1499.8.5 Disabling 8B/10B Encoding and Decoding for Gen1 and Gen2 Simulations.......149

A Troubleshooting and Observing the Link......................................................................150A.1 Simulation Fails To Progress Beyond Polling.Active State...........................................150A.2 Hardware Bring-Up Issues ...................................................................................150A.3 Link Training ......................................................................................................150A.4 Link Hangs in L0 State.........................................................................................151A.5 Use Third-Party PCIe Analyzer ............................................................................. 152A.6 BIOS Enumeration Issues ....................................................................................152

Contents


B PCI Express Core Architecture..................................................................................... 154B.1 Transaction Layer ...............................................................................................154B.2 Data Link Layer ................................................................................................. 155B.3 Physical Layer ................................................................................................... 157

C TX Credit Adjustment Sample Code..............................................................................160

D Stratix 10 Avalon-ST Interface for PCIe Solutions User Guide Archive ........................162

E Document Revision History.......................................................................................... 163

Contents


1 Introduction

1.1 Avalon-ST Interface for PCIe Introduction

Intel Stratix® 10 FPGAs include a configurable, hardened protocol stack for PCIExpress® that is compliant with PCI Express Base Specification 3.0. The Stratix 10Hard IP for PCI Express IP Core supports Gen1, Gen2 and Gen3 data rates and x1, x2,x4, x8, or x16 configurations.

Figure 1. Stratix 10 PCIe Variant with Avalon-ST InterfaceIn the following figure EMIB abbreviates Embedded Multi-die Interconnect Bridge.

Channels to FPGAAvalon-ST Interface

MappingPLL

Embe

dded

Mult

i-die

Inte

rconn

ect B

ridge

Stratix 10 Avalon-ST Hard IP for PCIe IP Core (altpcie_s10_hip_pipen1b.v)

PCIe Variant with Avalon-ST Interface

H-Tile or L-Tile

PCIe Link

Gen3x16 PCI Express IP Core

Data Link LayerTransaction Layer

Avalon-ST TX

Avalon-ST RX

Hdr & Data Credits

MSI/MSI-X

Clocks

ResetsPCS and PMA

Table 1. PCI Express Data Throughput

The following table shows the theoretical link bandwidth of a PCI Express link for Gen1, Gen2, and Gen3 for 1,2, 4, 8, and 16 lanes. This table provides bandwidths for a single transmit (TX) or receive (RX) channel. Thenumbers double for duplex operation. The protocol specifies 2.5 giga-transfers per second (GT/s) for Gen1,5.0 GT/s for Gen2, and 8.0 GT/s for Gen3. Gen1 and Gen2 use 8B/10B encoding which introduces a 20%overhead.Gen3 uses 128b/130b encoding which introduces about 1.6% overhead.

Link Width

×1 ×2 ×4 ×8 ×16

PCI Express Gen1 (2.5 Gbps) 2 4 8 16 32

PCI Express Gen2 (5.0 Gbps) 4 8 16 32 64

PCI Express Gen3 (8.0 Gbps) 7.87 15.75 31.49 63 126

Related Links

• Introduction to Intel FPGA IP CoresProvides general information about all Intel FPGA IP cores, includingparameterizing, generating, upgrading, and simulating IP cores.

1 Introduction

Intel Corporation. All rights reserved. Intel, the Intel logo, Altera, Arria, Cyclone, Enpirion, MAX, Nios, Quartusand Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or othercountries. Intel warrants performance of its FPGA and semiconductor products to current specifications inaccordance with Intel's standard warranty, but reserves the right to make changes to any products and servicesat any time without notice. Intel assumes no responsibility or liability arising out of the application or use of anyinformation, product, or service described herein except as expressly agreed to in writing by Intel. Intelcustomers are advised to obtain the latest version of device specifications before relying on any publishedinformation and before placing orders for products or services.*Other names and brands may be claimed as the property of others.

ISO9001:2008Registered

https://www.altera.com/documentation/mwh1409960636914.html#mwh1409958250601

http://www.altera.com/support/devices/reliability/certifications/rel-certifications.html



• Generating a Combined Simulator Setup ScriptCreate simulation scripts that do not require manual updates for software or IPversion upgrades.

• Project Management Best PracticesGuidelines for efficient management and portability of your project and IP files.

• Creating a System QsysProvides information about the Qsys system integration tool that simplifies thetasks of defining and integrating customized IP cores.

1.2 Features

New features in the Quartus Prime Pro v17.1 Stratix 10 ES Editions Software:

• Support for the H-Tile transceiver for up to Gen3 x8 configurations. Refer toTransceiver Tiles for additional information.

• Design examples using the H-Tile available in the Quartus® Prime Pro Editioninstallation directory.

• Support for a Gen3 x16 simulation model that you can use in an Avery testbench.The Avery testbench simulates all 16 lanes. For more information about the Averytestbench, refer to AN-811: Using the Avery BFM for PCI Express Gen3x16Simulation on Intel Stratix 10 Devices. The Intel® simulation testbench supportsGen3 x16 variants by downtraining to Gen3 x8.

The Avalon-ST Stratix 10 Hard IP for PCI Express IP Core supports the followingfeatures:

• Complete protocol stack including the Transaction, Data Link, and Physical Layersimplemented as hard IP.

• ×1, ×2, ×4, ×8 and ×16 configurations with Gen1, Gen2, or Gen3 lane rates forNative Endpoints.

• Avalon®-ST 256-bit interface to the Application Layer.

• Instantiation as a stand-alone IP core from the Quartus Prime Pro Edition IPCatalog or as part of a system design in Qsys.

• Dynamic design example generation.

• PHY interface for PCI Express (PIPE) or serial interface simulation using IEEEencrypted models.

• Testbench bus functional model (BFM) supporting x1, x2, x4, and x8configurations. The x16 configuration downtrains to x8 for Intel (internallycreated) testbench. Alternatively, the Avery testbench that supports allconfigurations is available for a licensing fee.

• Dedicated 69.5 kilobyte (KB) receive buffer.

• End-to-end cyclic redundancy check (ECRC).

Related Links

• Transceiver Tiles on page 10

• AN-811: Using the Avery BFM for PCI Express Gen3x16 Simulation on Intel Stratix10 Devices

1 Introduction



https://www.altera.com/documentation/mwh1409960181641.html#esc1444754592005

https://www.altera.com/documentation/jbr1437426657605.html#mwh1409958596582

https://www.altera.com/documentation/zoz1492100248719.html#sjb1492100458182


1.3 Release Information

Table 2. Hard IP for PCI Express Release Information

Item Description

Version Quartus Prime Pro v17.1 Stratix 10 ES Editions Software

Release Date May 2017

Ordering Codes No ordering code is required

Related Links

Timing and Power ModelsReports the default device support levels in the current version of the QuartusPrime Pro Edition software.

1.4 Device Family Support

The following terms define device support levels for Intel FPGA IP cores:

• Advance support—the IP core is available for simulation and compilation for thisdevice family. Timing models include initial engineering estimates of delays basedon early post-layout information. The timing models are subject to change assilicon testing improves the correlation between the actual silicon and the timingmodels. You can use this IP core for system architecture and resource utilizationstudies, simulation, pinout, system latency assessments, basic timing assessments(pipeline budgeting), and I/O transfer strategy (data-path width, burst depth, I/Ostandards tradeoffs).

• Preliminary support—the IP core is verified with preliminary timing models forthis device family. The IP core meets all functional requirements, but might still beundergoing timing analysis for the device family. It can be used in productiondesigns with caution.

• Final support—the IP core is verified with final timing models for this devicefamily. The IP core meets all functional and timing requirements for the devicefamily and can be used in production designs.

Table 3. Device Family Support

Device Family Support Level

Stratix 10 Advance support.The Quartus Prime Pro v17.1 Stratix 10 ES Editions Software supports simulation and compilation.Programmer Object File (.pof) support is available for up to Gen3 x8 with an *.ini file.

Other device families No support.Refer to the Intel PCI Express Solutions web page on the Intel website for support information onother device families.

Related Links

• Timing and Power ModelsReports the default device support levels in the current version of the QuartusPrime Pro Edition software.

• PCI Express Solutions Web Page

1 Introduction


https://www.altera.com/documentation/ewa1443722509979.html#hco1416836656622

https://www.altera.com/documentation/ewa1443722509979.html#hco1416836656622

https://www.altera.com/solutions/technology/transceiver/protocols/pro-pci_exp.html

1.5 Recommended Speed Grades

Table 4. Stratix 10 Recommended Speed Grades for All Avalon-ST Widths andFrequenciesThe recommended speed grades are for production parts.

Lane Rate Link Width Interface Width Application ClockFrequency (MHz)

RecommendedSpeed Grades

Gen1 x1, x2, x4, x8, x16 256 bits 125 –1, –2, -3

Gen2 x1, x2, x4, x8 256 bits 125 –1, –2, -3

Gen2 x16 256 bits 250 –1, –2, -3

Gen3

x1, x2, x4 256 bits 125 –1, –2, -3

x8 256 bits 250 –1, –2 , -3

x16 256 bits 500 –1, –2

1.6 Performance and Resource Utilization

Because the PCIe protocol stack is implemented in hardened logic, it uses no coredevice resources (no ALMs and no embedded memory).

Related Links

Running the FitterFor information on Fitter constraints.

1 Introduction


https://documentation.altera.com/#/link/jbr1437426657605/jbr1443213804950/en-us

1.7 Transceiver Tiles

Stratix 10 introduces several transceiver tile variants to support a wide variety ofprotocols.

Figure 2. Stratix 10 Transceiver Tile Block Diagram

Transceiver Tile (24 Channels) E-Tile

L-Tile and H-Tile

Package Substrate

4

x 10

0GE

EMIB

XCVR

Bank

(6 Ch

anne

ls)

Trans

ceive

r PLL

s

XCVR

Ba

nk(6

Chan

nels)

XCVR

Bank

(6 Ch

anne

ls)PC

Ie H

ard I

P

Refer

ence

Cloc

k Net

work

Clock

Net

work

XCVR

Bank

(6 Ch

anne

ls)

Transceiver Tile (24 Channels) E-Tile

4

x 10

0GE

EMIB

Transceiver Tile (24 Channels) L-Tile/ H-Tile

PCIe

Gen3

Har

d IP

EMIB

®Tra

nsce

iver P

LLs

100G

Ethe

rnet

Har

d IP

Refer

ence

Cloc

k Net

work

Clock

Net

work

Trans

ceive

rs (2

4 Cha

nnels

)

E-Tile

Table 5. Transceiver Tiles Channel Types

Tile Channel TypeChannel Capability

Channel Hard IP accessChip-to-Chip Backplane

L-Tile GX 17.4Gbps (NRZ) 12.5 Gbps (NRZ) PCIe Gen3x16

H-TileGX 17.4 Gbps (NRZ) 17.4 Gbps (NRZ) PCIe Gen3x16

GXT 28.3 Gbps (NRZ) 28.3 Gbps (NRZ) N/A

E-Tile GXE30 Gbps (NRZ),56 Gbps (PAM-4)

30 Gbps (NRZ),56 Gbps (PAM-4)

100G Ethernet

L-Tile and H-Tile

Both L and H transceiver tiles contain four transceiver banks-with a total of 24 duplexchannels, eight ATX PLLs, eight fPLLs, eight CMU PLLs, a PCIe Hard IP block, andassociated input reference and transmitter clock networks. L and H transceiver tilesalso include 10GBASE-KR/40GBASE-KR4 FEC block in each channel.

L-Tiles have transceiver channels that support up to 17.4 Gbps chip-to-chip or12.5Gbps backplane applications. H-Tiles have transceiver channels to support 28Gbpsapplications. H-Tile channels support fast lock-time for Gigabit-capable passive opticalnetwork (GPON).

Stratix GX/SX devices incorporate L-Tiles or H-Tiles. Package migration is availablewith Stratix 10 GX/SX from bL-Tileb to bL-Tileb variants. In addition, device migrationbetween an L-Tile equipped Stratix 10 GX device and an Stratix 10 GX device issupported.

1 Introduction


E-Tile

E-Tiles are designed to support 56 Gbps with PAM-4 signaling or up to 30 Gbpsbackplane with NRZ signaling. E-Tiles do not include any PCIe Hard IP blocks.

1.8 PCI Express IP Core Package Layout

Stratix 10 devices have high-speed transceivers that are implemented on separatetransceiver tiles. The transceiver tiles are located on the left and right sides of thedevice.

Each 24-channel transceiver tile includes one Gen3 x16 PCIe IP Core implemented inhardened logic. The following figures show the layout of PCIe IP cores in Stratix 10devices. Both L- and H-tiles are orange. E-tiles are green.

Figure 3. Stratix 10 GX/SX Devices with 4 PCIe Hard IP Cores and 96 TransceiverChannels

Transceiver Tile(24 Channels)




Package SubstrateEM

IBEM

IB

EMIB

EMIB

Core Fabric

®

GX/SX 1650 UF50 (F2397B)GX/SX 2100 UF50 (F2397B)GX/SX 2500 UF50 (F2397B)GX/SX 2800 UF50 (F2397B)PC

Ie H

IP

PCIe

HIP

PCIe

HIP

PC

Ie H

IP

Figure 4. Stratix 10 GX/SX Devices with 2 PCIe Hard IP Cores and 48 TransceiverChannels

Package Substrate

EMIB

EMIB

Core Fabric

®

GX/SX 850 NF43 (F1760A)GX/SX 1100 NF43 (F1760A)GX/SX 1650 NF43 (F1760A)GX/SX 2100 NF43 (F1760A)GX/SX 2500 NF43 (F1760A)GX/SX 2800 NF43 (F1760A)GX/SX 850 NF48 (F2112A)GX/SX 1100 NF48 (F2112A)



PCIe

HIP

PCIe

HIP

1 Introduction


Figure 5. Stratix 10 GX/SX Devices with 2 PCIe Hard IP Cores and 48 TransceiverChannels - Transceivers on Both Sides

Package Substrate

Core Fabric

®

GX/SX 650 NF43 (F1760C)

(24 Channels)(24 Channels) EMIB

PCIe

HIP

PCIe

HIP

Transceiver TileTransceiver Tile

EMIB

Figure 6. Stratix 10 Migration Device with 2 Transceiver Tiles and 48 TransceiverChannels



Package Substrate

EMIB

EMIB

Core Fabric

®

GX 2800 NF45 (F1932)

PCIe

HIP

PCIe

HIP

Note: 1. Stratix 10 migration device contains 2 L-Tiles which match Arria 10 migrationdevice.

1 Introduction


Figure 7. Stratix 10 GX/SX Devices with 1 PCIe Hard IP Core and 24 TransceiverChannels

Package Substrate

Core Fabric

®

GX/SX 400 HF35 (F1152)GX/SX 650 HF35 (F1152)GX/SX 2500 HF55 (F2912A)GX/SX 2800 HF55 (F2912A)GX/SX 4500 HF55 (F2912A)GX/SX 5500 HF55 (F2912A)

Transceiver Tile(24 Channels) PC

Ie H

IP

EMIB

Figure 8. Stratix 10 TX Devices with 1 PCIe Hard IP Core and 144 Transceiver Channels






Package Substrate

EMIB

EMIB

EMIB

EMIB

EMIB

EMIB

Core Fabric

®

TX 1650 YF55 (F2912B)TX 2100 YF55 (F2912B)

Transceiver Tile(24 Channels) PC

Ie H

IP

Note: 1. Stratix 10 TX Devices use a combination of E-Tiles and H-Tiles.

2. Five E-Tiles support 56G PAM-4 and 30G NRZ backplanes.

3. One H-Tile supports up to 28.3G backplanes.

1 Introduction


Figure 9. Stratix 10 TX Devices with 1 PCIe Hard IP Core and 96 Transceiver Channels





Package Substrate

EMIB

EMIB

EMIB

EMIB

Core Fabric

®

TX 1650 UF50 (F2397C)TX 2100 UF50 (F2397C)TX 2500 UF50 (F2397C)TX 2800 UF50 (F2397C)

PCIe

HIP


2. Three E-Tiles support 56G PAM-4 and 30G NRZ backplanes.

3. One H-Tile supports up to 28.3G backplanes.

Figure 10. Stratix 10 TX Devices with 2 PCIe Hard IP Cores and 72 Transceiver Channels




Package Substrate

EMIB

EMIB

EMIB

Core Fabric

®

TX 1650 SF48 (F2212B)TX 2100 SF48 (F2212B)TX 2500 SF48 (F2212B)TX 2800 SF48 (F2212B)

PCIe

HIP

PC

Ie H

IP


2. One E-Tile support 56G PAM-4 and 30G NRZ backplanes.

3. Two H-Tiles supports up to 28.3G backplanes.

Related Links

Stratix 10 GX/SX Device OverviewFor more information about Stratix 10 devices.

1 Introduction


https://www.altera.com/documentation/joc1442261161666.html#joc1443027925492

1.9 Channel Availability

PCIe Hard IP Channel Restrictions

Each transceiver tile contains a PCIe Hard IP block. The following table and figureshow the possible PCIe Hard IP channel configurations, the number of unusablechannels, and the number of channels available for other protocols. For example, aPCIe x4 variant uses 4 channels and 4 additional channels are unusable.

Table 6. Unusable Channels

PCIe Hard IP Configuration Number of Unusable Channels Number of Channels Available forOther Protocols

PCIe x1 7 16

PCIe x2 6 16

PCIe x4 4 16

PCIe x8 0 16

PCIe x16 16 8

Figure 11. PCIe Hard IP Channel Configurations Per Transceiver Tile

PCIe Hard IP x1

7 ChannelsUnusable

PCIe x1

PCIe Hard IP x2

6 ChannelsUnusable

PCIe x2

PCIe Hard IP x4

4 ChannelsUnusable

PCIe x4

PCIe Hard IP x8

PCIe x8

PCIe Hard IP x16

PCIe x16

Transceiver Tile Transceiver Tile Transceiver Tile Transceiver Tile Transceiver Tile0 0 0 0 0

15

23 23 23 23 23

8 8 87 7

8

134

1

7

2

7

16 ChannelsUsable

16 ChannelsUsable

16 ChannelsUsable

16 ChannelsUsable

8 ChannelsUsable

16

The table below maps all transceiver channels to PCIe Hard IP channels in availabletiles.

Table 7. PCIe Hard IP channel mapping across all tiles

Tile ChannelSequence

PCIe Hard IPChannel

Indexwithin I/O

Bank

Bottom LeftTile BankNumber

Top Left TileBank Number

Bottom RightTile BankNumber

Top Right TileBank Number

23 N/A 5 1F 1N 4F 4N

22 N/A 4 1F 1N 4F 4N

21 N/A 3 1F 1N 4F 4N

continued...

1 Introduction


Tile ChannelSequence

PCIe Hard IPChannel

Indexwithin I/O

Bank

Bottom LeftTile BankNumber

Top Left TileBank Number

Bottom RightTile BankNumber

Top Right TileBank Number

20 N/A 2 1F 1N 4F 4N

19 N/A 1 1F 1N 4F 4N

18 N/A 0 1F 1N 4F 4N

17 N/A 5 1E 1M 4E 4M

16 N/A 4 1E 1M 4E 4M

15 15 3 1E 1M 4E 4M

14 14 2 1E 1M 4E 4M

13 13 1 1E 1M 4E 4M

12 12 0 1E 1M 4E 4M

11 11 5 1D 1L 4D 4L

10 10 4 1D 1L 4D 4L

9 9 3 1D 1L 4D 4L

8 8 2 1D 1L 4D 4L

7 7 1 1D 1L 4D 4L

6 6 0 1D 1L 4D 4L

5 5 5 1C 1K 4C 4K

4 4 4 1C 1K 4C 4K

3 3 3 1C 1K 4C 4K

2 2 2 1C 1K 4C 4K

1 1 1 1C 1K 4C 4K

0 0 0 1C 1K 4C 4K

PCIe Soft IP Channel Usage

PCI Express soft IP PIPE-PHY cores available from third-party vendors are not subjectto the channel usage restrictions described above. Refer to Intel FPGA ➤ Products➤ Intellectual Property for more information about soft IP cores for PCI Express.

1 Introduction


https://www.altera.com/products/intellectual-property/ip.html

https://www.altera.com/products/intellectual-property/ip.html

2 Quick Start GuideUsing Qsys Pro, you can generate a PIO design example for the Avalon-ST Stratix 10Hard IP for PCI Express IP core. The generated design example reflects theparameters that you specify. The PIO example transfers data from a host processor toa target device. It is appropriate for low-bandwidth applications. This design exampleautomatically creates the files necessary to simulate and compile in the Quartus PrimePro software. You can download the compiled design to the Stratix 10 DevelopmentBoard or your own custom hardware.

In addition to the dynamically generated design examples, you can download staticdesign examples from the following directories:

• For the L-Tile transceiver: <install_dir>/ip/altera/altera_pcie/altera_pcie_s10_ed/example_design/s10.

• For the H-Tile transceiver: <install_dir>/ip/altera/altera_pcie/altera_pcie_s10_ed/example_design/s10_htile

The static design examples are available for simulation and compilation.

Figure 12. Development Steps for the Design Example

DesignExample

Generation

Compilation(Simulator)

FunctionalSimulation

Compilation(Quartus Prime)

HardwareTesting

2 Quick Start Guide






2.1 Design Components

Figure 13. Block Diagram for the Qsys PIO Design Example Simulation Testbench

Avalon-STdata

Generated PCIeEndpoint

Variant (DUT)

On-ChipMemory

(MEM)

PCIe Example Design

PIO Application(Memory for Gen3 x16)

hip_serial

hip_pipe

OR

Root Port BFM(RP_BFM)

PCIe Example Design Simulation Testbench

Avalon-MMdata

Figure 14. Important Interfaces and Signals in the Stratix 10 PCIe PIO Design Example

tx_st_hip

rx_st_hip

reset

clk

APPS

hprxms1

reset1

clk1

On-Chip Memory(RAM or ROM)

NPOR

PERST#

REF_CLK

tx_st

rx_st

nreset_status

coreclkout_hip

DUT(Stratix 10 Hard IP

for PCI Express)Serial/PIPE

Interface

Avalon-MM Interface

256 256

256

2 Quick Start Guide


2.2 Directory Structure

Figure 15. Directory Structure for the Generated Design Example

pcie_s10_hip_0_example_design

pcie_example_design

<top-level design files>

pcie_example_design_tb

pcie_example_design_tb

<simulator script>

testbench component

sim

<simulator>

<simulator script>

<simulator>

ip

pcie_example_design

<design component>.ip

<design component 1>

internal component

sim

synth

<design component <n>

internal component

sim

synth

pcie_example_design.qpf

pcie_example_design.qsf

pcie_example_design.sdc

pcie_example_design.qsys

ip

sim

synth

2 Quick Start Guide


2.3 Generating the Design

Figure 16. Procedure

Start ParameterEditor

Specify IP Variationand Select Device

SelectDesign Parameters

InitiateDesign Generation

Specify Example Design

Follow these steps to generate your design:

1. In the Quartus Prime Pro software, create a new project (File ➤ New ProjectWizard).

2. Specify the Directory, Name, and Top-Level Entity.

3. For Project Type, accept the default value, Empty project.

4. For Add Files, click Next.

5. For Family, Device & Board Settings, under Family, select Stratix 10(GX/SX) and the Target Device for your design. The Stratix 10 Development Kitincludes the 1SG280HU3F50E3VG device. The following devices are compatiblewith the 1SG280HU3F50E3VG device:

• 1SG280LU3F50E3VGS1

• 1SG280LU3F50E3VG

• 1SG280HU3F50E3VGS1

• 1SG280HU3F50E3VG

In the current release, dynamic design example generation overwrites yourselection with the 1SG280HU3F50E3VG device. Consequently, you can downloadyour design to the Stratix 10 Development Kit.

6. Click Finish.

7. Select Tools ➤ Qsys Pro.

8. For Qsys System or IP Variant, click + and specify a File name. Click Create.Click create again.

9. On the Qsys System Contents tab, delete the clock_in and reset_incomponents that appear by default.

10. In the IP Catalog locate and add the Avalon-ST Stratix 10 Hard IP for PCIExpress.

11. On the IP Settings tabs, specify the parameters for your IP variation.

12. On the System Contents tab, connect refclk to coreclkout_hip.

13. On the Example Designs tab, select the PIO design.

2 Quick Start Guide


Figure 19. Example Design Tab

Initiates DesignGeneration

14. For Example Design Files, select the Simulation and Synthesis options.

15. For Generated HDL Format, only Verilog is available.

16. For Target Development Kit select the NONE option. Although the Stratix 10Development Kit is not listed, it is available in limited quantities to Betacustomers. Contact Intel.

17. Click the Generate Example Design button. The software generates all filesnecessary to run simulations. Click Close when generation completes.

18. Click Finish.

19. Save your .ip file when prompted.

20. The prompt, Recent changes have not been generated. Generate now?,allows you to create files for simulation and synthesis. Click No to continue tosimulate the design example you just generated.

2.4 Simulating the Design

Figure 20. Procedure

Change to Testbench Directory

Run<Simulation Script>

AnalyzeResults

1. Change to the testbench simulation directory.

2. Run the simulation script for the simulator of your choice. Refer to the table below.

3. Analyze the results.

2 Quick Start Guide


https://www.altera.com/about/contact/contact.html

Table 8. Steps to Run Simulation

Simulator Working Directory Instructions

ModelSim* <example_design>/pcie_example_design_tb/pcie_example_design_tb/sim/mentor/

1. do msim_setup.tcl

2. ld_debug

3. run -all

4. A successful simulation ends with the followingmessage, "Simulation stopped due tosuccessful completion!"

VCS* <example_design>/pcie_example_design_tb/pcie_example_design_tb/sim/synopsys/vcs

1. sh vcs_setup.shUSER_DEFINED_SIM_OPTIONS=""

2. A successful simulation ends with the followingmessage, "Simulation stopped due tosuccessful completion!"

This testbench simulates up to x8 variants. It supports x16 variants by down-trainingto x8. To simulate all lanes of a x16 variant, you can create a simulation model inQsys to use in an Avery testbench. For more information refer to AN-811: Using theAvery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10 Devices.

The automatically generated testbench completes the following tasks:

• Host allocation of memory

• Reads the DMA read status register

• Sets up two DMA write descriptors in host memory and performs DMA writes

• Checks the status for the DMA writes

• Verifies the data for the DMA writes

The simulation reports, "Simulation stopped due to successful completion" if no errorsoccur.

2 Quick Start Guide


Figure 21. Partial Transcript from Successful Simulation Testbench

Related Links

AN-811: Using the Avery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10Devices

2.5 Compiling the Design

You need an Initialization File (.ini) to compile your design in the Quartus Prime Prov17.1 Stratix 10 ES Editions Software.

Be sure that the following assignments are included in yourpcie_example_design.qsf before compiling:

set_instance_assignment -name VIRTUAL_PIN ON -to *pipe_sim_only*set_instance_assignment -name IO_STANDARD "3.0-V LVTTL" -to pcie_rstn_pin_perst

Note: These commands are included in the Quartus Prime Settings Files (*.qsf) for projectsincluded in the <install_dir>ip\altera\altera_pcie\altera_pcie_s10_ed\example_design\s10 and <install_dir>ip\altera\altera_pcie\altera_pcie_s10_ed\example_design\s10_htile directories.

2 Quick Start Guide




1. On the Processing menu, select Start Compilation.

2. When compilation completes, click Assembler (Generate programming files)to generate an SRAM Object File (.sof).

3. After successfully compiling your design, program the targeted device with theProgrammer and verify the design in hardware.

2 Quick Start Guide


3 Interface OverviewThe PCI Express Base Specification 3.0 defines a packet interface for communicationbetween a Root Port and an Endpoint. When you select the Avalon-ST interface,Transaction Layer Packets (TLP) transfer data between the Root Port and an Endpointusing the Avalon-ST TX and RX interfaces. The interfaces are named from the point-of-view of the user logic.

Figure 22. Stratix 10 Top-Level Interfaces


MappingPLL

Embe

dded

Mult

i-die

Inte

rconn

ect B

ridge



H-Tile or L-Tile

PCIe Link



Avalon-ST TX

Avalon-ST RX

Hdr & Data Credits

MSI/MSI-X

Clocks

ResetsPCS and PMA

The following figures show the PCIe hard IP Core top-level interfaces and theconnections to the Application Layer and system.

3 Interface Overview






Figure 23. Connections: User Application to Stratix 10 Avalon-ST Hard IP for PCIe IPCore

PCIeRX Port

Avalon-STMaster

rx_st_*

PCIeTX Port

Avalon-STSlave

tx_st_*

tx_ph_cdts[7:0]tx_pd_cdts[11:0]tx_nph_cdts[7:0]tx_npd_cdts[11:0]tx_cplh_cdts[7:0]tx_cpld_cdts[11:0]tx_hdr_cdts_consumedtx_data_cdts_consumedtx_cdts_type[1:0]tx_cdts_data_value[1:0]

PCIe Clocks and Resets

coreclkout_hip

refclk

nporcurrent_speed[1:0]

pin_perstpld_clk_inusepld_core_readyreset_statusserdes_pll_locked

PCIe MSI Interface

app_msi_*app_int_stsint_status[3:0]int_status_common[2:0]

PCIe Configuration Space Signals

PCIe Configuration Extension Bus

ceb_*

app_err_*

tl_cfg_*

PCIe Hard IP Status Interface

derr_*int_status[3:0]int_status_common[2:0]lane_act[4:0]link_upltssmstate[5:0]rx_par_errtx_par_err

Application RX Interface

Avalon-STSlave

Application TX Interface

Avalon-STMaster

Application Clocks and Reset Interface

Application MSI Interface

Application Configuration Space Interface

Application Configuration Extension Interface

Application HIP Status Interface

Avalon-ST Interface to the PCIe IP Core Application and System Interfaces



Figure 24. Connections: PCIe Link and Reconfiguration, Power and Test Interfaces

Hard IP Reconfiguration

(Optional)

Avalon-MMSlave

hip_reconfig_*

PCIe Power Management

pm_*

apps_pm_xmt_pmeapps_ready_entr_123apps_pm_xmt_turnoffapp_init_rstapp_xfer_pending

PCIe Serial Data Interfacerx_in

PCIe PIPE (Simulation Only)

Receive Data

tx_out[<n>-1:0]

[<n>-1:0]

tx *powerdown[1:0]

rate[1:0]rxpolarity

currentrxpreset0[2:0]

currentcoeff0[17:0]rxeqevalrxeqinprogress

rx *

phystatus0

dirfeedback[5:0]

simu_mode_pipesim_pipe_pclk_in

sim_pipe_mask_tx_pll

sim_pipe_rate [1:0]

sim_ltssmstate [5:0]

Hard IP Reconfiguation

(Optional)

Avalon-MMMaster

Application Power Management Interface

test_in[66:0]

testin_zero Application Test Interface

PCIe IP Core Interfaces Application InterfacesPCIe Link

Transmit Data

PCIe Test Interface

The following sections introduce these interfaces. Refer to the Interfaces section in theBlock Description chapter for detailed descriptions and timing diagrams.

Avalon-ST RX

The Transaction Layer transfers TLPs to the Application on this 256-bit interface. TheApplication must assert rx_st_ready before transfers can begin. The latency on thisinterface includes the three-cycle ready latency and the latency of the EmbeddedMulti-die Interconnect Bridge (EMIB) for a total of 17 cycles. Once rx_st_readydeasserts, rx_st_valid will deassert within 17 cycles. Once rx_st_readyreasserts, rx_st_valid will resume data transfer within 17 cycles. To achieve thebest performance the Application must include a receive buffer large enough to avoidthe deassertion of rx_st_ready. Refer to Avalon-ST RX Interface for moreinformation.

Note: Throughout this user guide the terms Application and Application Layer refer to thecustomer logic that interfaces with the PCIe Transaction Layer.



Avalon-ST TX

The Application transmits TLPs to the Transaction Layer of the IP core on this 256-bitinterface. The Transaction Layer must assert tx_st_ready before transmissionbegins. Transmission must be continuous when tx_st_ready is asserted. The readylatency is three coreclkout_hip cycles. If tx_st_ready deasserts, the Applicationmust stop transmitting data within three cycles. When tx_st_ready reasserts, theApplication must wait for three clock cycles before resuming data transmission. Formore detailed information about the Avalon-ST interface, refer to Avalon-ST TXInterface. The packet layout is shown in detail in the Block Description chapter.

TX Credit

The Transaction Layer TX interface transmits TLPs in the same order as they werereceived from the Application. To optimize performance, the Application can performcredit-based checking before submitting requests for transmission, allowing theApplication to reorder packets to improve throughput. Application reordering isoptional. The Transaction Layer always performs its own credit check beforetransmitting any TLP.

TX and RX Serial Data

This differential, serial interface is the physical link between a Root Port and anEndpoint. The PCIe IP Core supports 1, 2, 4, 8, or 16 lanes. Each lane includes a TXand RX differential pair. Data is striped across all available lanes.

PIPE

This is a 32-bit parallel interface between the PCIe IP Core and PHY. It carries the TLPdata before it is serialized. It is available for simulation only and provides morevisibility for debugging.

ClocksThe PCI Express Card Electromechanical Specification Revision 2.0 defines the inputreference clock. It is a 100 MHz ±300 ppm. The motherboard drives this clock to thePCIe card. The PCIe card is not required to use the reference clock received from theconnector. However, the PCIe card must provide a 100 MHz ±300 ppm to the refclkinput pin.

Table 9. Application Clock FrequencyThe interface to the Application is fixed at 256 bits; consequently, the Application clock frequency depends onlyon the data rate.

Data Rate Clock Frequency

Gen1 x1, x2, x4, x8, and x16 125 MHz

Gen2 x1, x2, x4, and x8 125 MHz

Gen2 x16 250 MHz

Gen3 x1, x2, and x4 125 MHz

Gen3 x8 250 MHz

Gen3 x16 500 MHz

ResetThe PCIe IP core receives two reset inputs:



• pin_perst is the active low, edge-triggered reset driven from the PCIemotherboard. Logic on the motherboard autonomously generates this fundamentalreset signal.

• npor is an active low, edge-triggered reset signal. The Application drives thisreset signal.

The PCIe IP core reset logic requires a free-running clock input. This free-runningclock becomes stable after the secure device manager (SDM) block assertsiocsrrdy_dly indicating that the I/O Control and Status registers programming iscomplete.

Reset StatusThis interface indicates when the clocks are stable and FPGA configuration is complete.

Hard IP Status

This optional interface includes the following signals that are useful for debugging

• Link status signals

• Interrupt status signals

• TX and RX parity error signals

• Correctable and uncorrectable error signals

Interrupts

The PCIe IP core support Message Signaled Interrupts (MSI), MSI-X interrupts, andLegacy interrupts. MSI, MSI-X, and legacy interrupts are mutually exclusive.

MSI uses the TLP single dword memory writes to implement interrupts. This interruptmechanism conserves pins because it does not use separate wires for interrupts. Inaddition, the single dword provides flexibility for the data presented in the interruptmessage. The MSI Capability structure is stored in the Configuration Space and isprogrammed using Configuration Space accesses.

The Application generates MSI-X messages which are single dword memory writes.The MSI-X Capability structure points to an MSI-X table structure and MSI-X PBAstructure which are stored in memory. This scheme is different than the MSI capabilitystructure, which contains all the control and status information for the interrupts.

Enable Legacy interrupts by programming the Interrupt Disable bit (bit[10]) ofthe Configuration Space Command to 1'b0. When legacy interrupts are enabled, the IPcore emulates INTx interrupts using virtual wires. The app_int_sts port controlslegacy interrupt generation.

Transaction Layer (TL ) tl_cfg ConfigurationThis interface provides time-domain multiplexed (TD) access to a subset of the valuesstored in the Configuration Space registers.

Hard IP ReconfigurationThis optional Avalon Memory-Mapped (Avalon-MM) interface allows you to dynamicallyupdate the value of read-only Configuration Space registers at run-time. It is availablewhen Enable dynamic reconfiguration of PCIe read-only registers is enabled inthe component GUI.



Power ManagementSoftware uses the power management interface to control the power managementstate machine. The power management output signals indicate the current powerstate. The IP core supports the two mandatory power states: D0 full power and D3preparation for loss of power. It does not support the optional D1 and D2 low-powerstates.

Related Links

• Avalon-ST RX Interface on page 54The Application Layer receives data from the Transaction Layer of the PCIe IPcore over the Avalon-ST RX interface. This is a 256-bit interface.

• Interfaces on page 52This section describes the top-level interfaces in the Avalon-ST Stratix 10 HardIP for PCIe IP core.

• PCI Express Base Specification 3.0



http://www.pcisig.com/

4 ParametersThis chapter provides a reference for all the parameters of the Stratix 10 Hard IP forPCI Express IP core.

Table 10. System Settings

Parameter Value Description

ApplicationInterface Type

Avalon-ST Selects the interface to the Application Layer.

Hard IP Mode Gen3x16, 256-bit interface, 500 MHzGen3x8, 256-bit interface, 250 MHzGen3x4, 256-bit interface, 125 MHzGen3x2, 256-bit interface, 125 MHzGen3x1, 256-bit interface, 125 MHz

Gen2x16, 256-bit interface, 250 MHzGen2x8, 256-bit interface, 125 MHzGen2x4, 256-bit interface, 125 MHzGen2x2, 256-bit interface, 125 MHzGen2x1, 256-bit interface, 125 MHz

Gen1x16, 256-bit interface, 125 MHzGen1x8, 256-bit interface, 125 MHzGen1x4, 256-bit interface, 125 MHzGen1x2, 256-bit interface, 125 MHzGen1x1, 256-bit interface, 125 MHz

Selects the following elements:• The lane data rate. Gen1, Gen2, and Gen3 are

supported• The Application Layer interface frequencyThe width of the data interface between the hard IPTransaction Layer and the Application Layerimplemented in the FPGA fabric interface is 256-bit.Note: If the Mode selected is not available for the

configuration chosen, an error message displaysin the Message pane.

Port type Native Endpoint Specifies the port type.The Endpoint stores parameters in the Type 0Configuration Space. Only Native Endpoints aresupported in the Quartus Prime Pro v17.1 Stratix 10 ESEditions.

Related Links

PCI Express Base Specification 3.0

4.1 Stratix 10 Avalon-ST Settings

Table 11. System Settings for PCI Express


Enable Avalon-STreset output port

On/Off When On, the generated reset output port clr_st has the samefunctionality as the hip_ready_n port included in the Hard IP Resetinterface. This option is available for backwards compatibility with Arria® 10devices.

Enable byte parityports onAvalon-STinterface

On/Off When On, the RX and TX datapaths are parity protected. Parity is odd. TheApplication Layer must provide valid byte parity in the Avalon-ST TXdirection.

continued...

4 Parameters








This parameter is only available for the Avalon-ST Stratix 10 Hard IP for PCIExpress.

4.2 Base Address Registers

Table 12. BAR Registers


Type Disabled64-bit prefetchable memory32-bit non-prefetchable memory

If you select 64-bit prefetchable memory, 2 contiguous BARsare combined to form a 64-bit prefetchable BAR; you mustset the higher numbered BAR to Disabled. A non-prefetchable 64-bit BAR is not supported because in a typicalsystem, the Root Port Type 1 Configuration Space sets themaximum non-prefetchable memory window to 32 bits. TheBARs can also be configured as separate 32-bit memories.Defining memory as prefetchable allows contiguous data tobe fetched ahead. Prefetching memory is advantageouswhen the requestor may require more data from the sameregion than was originally requested. If you specify that amemory is prefetchable, it must have the following 2attributes:• Reads do not have side effects such as changing the

value of the data read• Write merging is allowed

Size 16 Bytes–8 EBytes Specifies the size of the address space accessible to the BAR.

4.3 Device Identification Registers

Table 13. Device ID RegistersThe following table lists the default values of the read-only Device ID registers. You can use the parametereditor to change the values of these registers. At run time, you can change the values of these registers usingthe optional Hard IP Reconfiguration block signals.

Register Name Default Value Description

Vendor ID 0x00001172 Sets the read-only value of the Vendor ID register. This parameter cannotbe set to 0xFFFF per the PCI Express Base Specification.Address offset: 0x000.

Device ID 0x00000000 Sets the read-only value of the Device ID register.Address offset: 0x000.

Revision ID 0x00000001 Sets the read-only value of the Revision ID register.Address offset: 0x008.

Class code 0x00000000 Sets the read-only value of the Class Code register.Address offset: 0x008.

Subclass code 0x00000000 Sets the read-only value of the Subclass Code register.

continued...

4 Parameters


Register Name Default Value Description

Address offset: 0x008.

SubsystemVendor ID

0x00000000 Sets the read-only value of Subsystem Vendor ID register in the PCI Type0 Configuration Space. This parameter cannot be set to 0xFFFF per the PCIExpress Base Specification. This value is assigned by PCI-SIG to the devicemanufacturer.Address offset: 0x02C.

Subsystem DeviceID

0x00000000 Sets the read-only value of the Subsystem Device ID register in the PCIType 0 Configuration Space.Address offset: 0x02C

4.4 PCI Express and PCI Capabilities Parameters

This group of parameters defines various capability properties of the IP core. Some ofthese parameters are stored in the PCI Configuration Space - PCI CompatibleConfiguration Space. The byte offset indicates the parameter address.

4.4.1 Device Capabilities

Table 14. Capabilities Registers

Parameter PossibleValues

Default Value Description

Maximumpayload sizessupported

128 bytes256 bytes512 bytes

1024 bytes

128 bytes Specifies the maximum payload size supported. This parametersets the read-only value of the max payload size supported fieldof the Device Capabilities register (0x084[2:0]).Address: 0x084.

Supportextended tagfield

OnOff

On When you turn this option On, the core supports 256 tags,improving the performance of high latency systems. Turning thisoption on turns on the Extended Tag bit in the ConfigurationSpace Device Capabilities register.The IP core tracks tags for Non-Posted Requests. The trackingclears when the IP core receives the last Completion TLP for aMemRd.

4.4.2 Link Capabilities

Table 15. Link Capabilities


Link port number(Root Port only)

0x01 Sets the read-only value of the port number field in the LinkCapabilities register. This parameter is for Root Ports only. It shouldnot be changed.

Slot clockconfiguration

On/Off When you turn this option On, indicates that the Endpoint uses thesame physical reference clock that the system provides on theconnector. When Off, the IP core uses an independent clock regardlessof the presence of a reference clock on the connector. This parametersets the Slot Clock Configuration bit (bit 12) in the PCI ExpressLink Status register.

4 Parameters


4.4.3 MSI and MSI-X Capabilities

Table 16. MSI and MSI-X Capabilities


MSI messagesrequested

1, 2, 4, 8, 16, 32 Specifies the number of messages the Application Layer canrequest. Sets the value of the Multiple Message Capable fieldof the Message Control register,Address: 0x050[31:16].

MSI-X Capabilities

Implement MSI-X On/Off When On, adds the MSI-X functionality.

Bit Range

Table size [15:0] System software reads this field to determine the MSI-X Table size<n>, which is encoded as <n–1>. For example, a returned value of2047 indicates a table size of 2048. This field is read-only. Legalrange is 0–2047 (216).Address offset: 0x068[26:16]

Table offset [31:0] Points to the base of the MSI-X Table. The lower 3 bits of the tableBAR indicator (BIR) are set to zero by software to form a 64-bitqword-aligned offset. This field is read-only.

Table BAR indicator [2:0] Specifies which one of a function’s BARs, located beginning at 0x10in Configuration Space, is used to map the MSI-X table into memoryspace. This field is read-only. Legal range is 0–5.

Pending bit array (PBA)offset

[31:0] Used as an offset from the address contained in one of thefunction’s Base Address registers to point to the base of the MSI-XPBA. The lower 3 bits of the PBA BIR are set to zero by software toform a 32-bit qword-aligned offset. This field is read-only. 1

Pending BAR indicator [2:0] Specifies the function Base Address registers, located beginning at0x10 in Configuration Space, that maps the MSI-X PBA into memoryspace. This field is read-only. Legal range is 0–5.

4.4.4 Slot Capabilities

Table 17. Slot Capabilities


Use Slot register On/Off This parameter is only supported in Root Port mode. The slot capability isrequired for Root Ports if a slot is implemented on the port. Slot status isrecorded in the PCI Express Capabilities register.Defines the characteristics of the slot. You turn on this option by selectingEnable slot capability. Refer to the figure below for bit definitions.

Slot power scale 0–3 Specifies the scale used for the Slot power limit. The following coefficientsare defined:• 0 = 1.0x• 1 = 0.1x• 2 = 0.01x• 3 = 0.001x

continued...

1 Throughout this user guide, the terms word, dword and qword have the same meaning thatthey have in the PCI Express Base Specification. A word is 16 bits, a dword is 32 bits, and aqword is 64 bits.

4 Parameters



The default value prior to hardware and firmware initialization is b’00. Writesto this register also cause the port to send the Set_Slot_Power_LimitMessage.Refer to Section 6.9 of the PCI Express Base Specification Revision for moreinformation.

Slot power limit 0–255 In combination with the Slot power scale value, specifies the upper limit inwatts on power supplied by the slot. Refer to Section 7.8.9 of the PCI ExpressBase Specification for more information.

Slot number 0-8191 Specifies the slot number.

Figure 25. Slot Capability

31 19 18 17 16 15 14 7 6 5

Physical Slot Number

No Command Completed SupportElectromechanical Interlock Present

Slot Power Limit ScaleSlot Power Limit Value

Hot-Plug CapableHot-Plug Surprise

Power Indicator PresentAttention Indicator Present

MRL Sensor PresentPower Controller PresentAttention Button Present

04 3 2 1

4.4.5 Power Management

Table 18. Power Management Parameters


Endpoint L0sacceptable latency

Maximum of 64 nsMaximum of 128 nsMaximum of 256 nsMaximum of 512 nsMaximum of 1 usMaximum of 2 usMaximum of 4 usNo limit

This design parameter specifies the maximum acceptable latency thatthe device can tolerate to exit the L0s state for any links between thedevice and the root complex. It sets the read-only value of theEndpoint L0s acceptable latency field of the Device CapabilitiesRegister (0x084).This Endpoint does not support the L0s or L1 states. However, in aswitched system there may be links connected to switches that haveL0s and L1 enabled. This parameter is set to allow system configurationsoftware to read the acceptable latencies for all devices in the systemand the exit latencies for each link to determine which links can enableActive State Power Management (ASPM). This setting is disabled forRoot Ports.The default value of this parameter is 64 ns. This is the safest settingfor most designs.

Endpoint L1acceptable latency

Maximum of 1 usMaximum of 2 usMaximum of 4 usMaximum of 8 usMaximum of 16 usMaximum of 32 usNo limit

This value indicates the acceptable latency that an Endpoint canwithstand in the transition from the L1 to L0 state. It is an indirectmeasure of the Endpoint’s internal buffering. It sets the read-only valueof the Endpoint L1 acceptable latency field of the DeviceCapabilities Register.This Endpoint does not support the L0s or L1 states. However, aswitched system may include links connected to switches that have L0sand L1 enabled. This parameter is set to allow system configurationsoftware to read the acceptable latencies for all devices in the systemand the exit latencies for each link to determine which links can enableActive State Power Management (ASPM). This setting is disabled forRoot Ports.

continued...

4 Parameters



The default value of this parameter is 1 µs. This is the safest setting formost designs.

4.4.6 Vendor Specific Extended Capability (VSEC)

Table 19. VSEC


User ID registerfrom the VendorSpecific ExtendedCapability

Custom value Sets the read-only value of the 16-bit User ID register from the VendorSpecific Extended Capability. This parameter is only valid for Endpoints.

4.5 Configuration, Debug and Extension Options

Table 20. Configuration, Debug and Extension Options


Enable dynamicreconfiguration ofPCIe read-onlyregisters

On/Off When On, you can use the Hard IP reconfiguration bus to dynamicallyreconfigure Hard IP read-only registers. For more information refer to HardIP Reconfiguration Interface.

Related Links

Hard IP Reconfiguration on page 73The Hard IP reconfiguration interface is an Avalon-MM slave interface with a 21-bitaddress and an 8-bit data bus. You can use this bus to dynamically modify thevalue of configuration registers that are read-only at run time.

4.6 PHY Characteristics

Table 21. PHY Characteristics


Gen2 TX de-emphasis

3.5dB6dB

Specifies the transmit de-emphasis for Gen2. Intel recommends thefollowing settings:• 3.5dB: Short PCB traces• 6.0dB: Long PCB traces.

Enable soft DFEcontroller IP

OnOff

When On, the PCIe Hard IP core includes a decision feedbackequalization (DFE) soft controller in the FPGA fabric to improve the biterror rate (BER) margin. The default for this option is Off because theDFE controller is typically not required. However, short reflective linksmay benefit from this soft DFE controller IP.

VCCR/VCCT supplyvoltage for thetransceiver

1_1V1_0V

Allows you to select the preferred voltage for the transceivers.

4 Parameters


4.7 Stratix 10 Example Designs

Table 22. Example Designs


Available ExampleDesigns

DMAPIO

When you select the DMA option, the generated example design includes adirect memory access application. This application includes upstream anddownstream transactions. When you select the PIO option, the generateddesign includes a target application including only downstream transactions.

Simulation On/Off When On, the generated output includes a simulation model.

Synthesis On/Off When On, the generated output includes a synthesis model.

Generated HDLformat

Verilog Only Verilog HDL is supported.

Select Board None Stratix 10 development boards are not available at this time.

4 Parameters


5 Designing with the IP Core

5.1 Generation

Use Qsys Pro to define and generate an Avalon-ST Stratix 10 Hard IP for PCI ExpressIP Core custom component.

Qsys Pro requires you to define a Quartus Prime project before customizing the PCIeIP core in Qsys Pro.

1. In the Quartus Prime Pro software, create a new project (File ➤ New ProjectWizard).

2. Specify the Directory, Name, and Top-Level Entity.

3. For Project Type, accept the default value, Empty project.

4. For Add Files, click Next.

5. For Family, Device & Board Settings, under Family, select Stratix 10(GX/SX) and the Target Device for your design. Refer to Stratix 10 ProductTable for information about package options.

6. Click Finish.

7. To define your Qsys Pro project, select Tools ➤ Qsys Pro.

8. For Qsys System or IP Variant, click + and specify a File name. Click Create.Click create again.

Figure 26. Qsys Pro Create New System

9. In the IP Catalog, locate and select Avalon-ST Stratix 10 Hard IP for PCIExpress.







The parameter editor appears.

10. On the IP Settings tabs, specify the parameters for your IP variation.

Refer to the Parameters chapter for an explanations of all parameters.

11. Click Finish to exit the parameter editor.

— You can now generate a simulation testbench for your customized PCIe IP coreor continue to build your system design.

Related Links

• Parameters on page 31This chapter provides a reference for all the parameters of the Stratix 10 HardIP for PCI Express IP core.

• Creating a System with Qsys

• Stratix 10 Product Table

5.2 Simulation

After you install the Quartus Prime Pro v17.1 Stratix 10 ES Editions Software, you candownload Quartus Prime Archive Files (.qar) from <install_dir>ip\altera\altera_pcie\altera_pcie_s10_ed\example_design\s10. This directoryincludes example designs for the following Endpoint configurations:

• Gen1, Gen2, and Gen3

• x1, x2, x4, x8, and x16

• Avalon-ST, Avalon-MM, and Avalon-MM with DMA

The simulation testbench supports only the Verilog HDL and the following simulators:

• Mentor: ModelSim SE

• Synopsys: VCS

If you choose to simulate an example design from the Quartus Prime Pro installation,the testbench includes the following components:

• An Endpoint (DUT)

• A Root Port BFM

• Memory (MEM)

The examples provided with the Quartus Prime Pro – Stratix 10 Edition 17.1 InterimRelease software installation are static. Consequently, you cannot modify theparameters and regenerate the example design.

Note: The Intel testbench and Root Port BFM provide a simple method to do basic testing ofthe Application Layer logic that interfaces to the variation. This BFM allows you tocreate and run simple task stimuli with configurable parameters to exercise basicfunctionality of the example design. The testbench and Root Port BFM are not intendedto be a substitute for a full verification environment. Corner cases and certain trafficprofile stimuli are not covered. To ensure the best verification coverage possible, Intelsuggests strongly that you obtain commercially available PCI Express verification IPand tools, or do your own extensive hardware testing or both.




https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/pt/stratix-10-product-table.pdf

Refer to the Example Design for Avalon-ST Stratix 10 Hard IP for PCI Express chapterto create a simple custom example design using the parameters that you specify.

Related Links

Generating the Design

5.2.1 Selecting Serial or PIPE Simulation

The parameter serial_sim_hwtcl in <working_dir>/<variant>_example_design/pcie_example_design_tb/pcie_example_design_tb/altera_pcie_<dev>_tbed_<ver>/sim/altpcied_<dev>_hwtcl.sv determines the simulation mode. When 1'b1, thesimulation is serial. When 1'b0, the simulation runs in the 32-bit parallel PIPE mode.

5.2.2 Reducing Counter Values for Simulation

You can save simulation time by reducing the value of counters whose default valuesare set appropriately for hardware. The test_in[0] signal controls the values.Setting test_in[0]= 1'b, reduces counter values for simulation. (By default, thegenerated simulation testbench does set test_in[0]= 1'b.)

5.3 Files Generated for Intel FPGA IP Cores and Qsys Pro Systems

The Quartus Prime Pro Edition software generates the following output file structurefor IP cores and Qsys Pro systems. The Quartus Prime Pro Edition softwareautomatically adds the .ip files and the generated .qsys files when you open yourQuartus Prime Pro project.



Figure 27. Files generated for IP cores and Qsys Pro Systems

<Project Directory>

<your_ip>_inst.v or .vhd - Lists file for IP core synthesis

<your_ip>.qip - Lists files for IP core synthesis

<your_ip>.debuginfo - Post-generation debug data

<your_ip>_generation.rpt - IP generation report

<your_ip>.bsf - Block symbol schematic file

<your_ip>.ppf - XML I/O pin information file

<your_ip>.html - Memory map data

<your_ip>.cmp - VHDL component declaration

<your_ip>.sip - NativeLink simulation integration file

<your_ip>.spd - Combines individual simulation startup scripts

<your_system>.qsys - System File<your_subsystem>.qsys - Subsystem File

<your_system_directory>

<your_subsystem_directory>

<your_ip>.ipxact - IP XACT File

sim - IP simulation files

<simulator vendor> - Simulator setup scripts

<your_ip>.v or vhd - Top-level simulation file

synth - IP synthesis files

<your_ip>.v or .vhd - Top-level IP synthesis file

ip - IP files

<your_system> - - Your system directory

<your_system>.ip - Parameter file for system IP component

<your_subsystem> - Your Subsystem directory

<your_subsystem>.ip - Parameter file for subsystem IP component

<your_ip>_bb.v - Verilog HDL black box EDA synthesis file

<your_ip>.qgsimc - Simulation caching file

<your_ip>.qgsynthc - Synthesis caching file

Table 23. IP Core and Qsys Simulation Files

File Name Description

<my_system>.qsys The Qsys Pro system.

<my_subsystem>.qsys The Qsys Pro subsystem.

ip/ Contains the parameter files for the IP components in the system andsubsystem(s).

continued...




<my_ip>.cmp The VHDL Component Declaration (.cmp) file is a text file that contains localgeneric and port definitions that you can use in VHDL design files.

<my_ip>_generation.rpt IP or Qsys generation log file. A summary of the messages during IPgeneration.

<my_ip>.qgsimc Simulation caching file that compares the .qsys and .ip files with the currentparameterization of the Qsys Pro system and IP core. This comparisondetermines if Qsys Pro can skip regeneration of the HDL.

<my_ip>.qgsynth Synthesis caching file that compares the .qsys and .ip files with the currentparameterization of the Qsys Pro system and IP core. This comparisondetermines if Qsys Pro can skip regeneration of the HDL.

<my_ip>.qip Contains all the required information about the IP component to integrate andcompile the IP component in the Quartus Prime software.

<my_ip>.csv Contains information about the upgrade status of the IP component.

<my_ip>.bsf A Block Symbol File (.bsf) representation of the IP variation for use in BlockDiagram Files (.bdf).

<my_ip>.spd Required input file for ip-make-simscript to generate simulation scripts forsupported simulators. The .spd file contains a list of files generated forsimulation, along with information about memories that you can initialize.

<my_ip>.ppf The Pin Planner File (.ppf) stores the port and node assignments for IPcomponents created for use with the Pin Planner.

<my_ip>_bb.v Use the Verilog black box (_bb.v) file as an empty module declaration for useas a black box.

<my_ip>.sip Contains information required for NativeLink simulation of IP components. Addthe .sip file to your Quartus Prime Standard Edition project to enableNativeLink for supported devices. The Quartus Prime Pro Edition software doesnot support NativeLink simulation.

<my_ip>_inst.v or _inst.vhd HDL example instantiation template. Copy and paste the contents of this fileinto your HDL file to instantiate the IP variation.

<my_ip>.regmap If the IP contains register information, the Quartus Prime software generatesthe .regmap file. The .regmap file describes the register map information ofmaster and slave interfaces. This file complements the .sopcinfo file byproviding more detailed register information about the system. This file enablesregister display views and user customizable statistics in System Console.

<my_ip>.svd Allows HPS System Debug tools to view the register maps of peripheralsconnected to HPS within a Qsys system.During synthesis, the Quartus Prime software stores the .svd files for slaveinterface visible to the System Console masters in the .sof file in the debugsession. System Console reads this section, which Qsys can query for registermap information. For system slaves, Qsys can access the registers by name.

<my_ip>.v <my_ip>.vhd HDL files that instantiate each submodule or child IP core for synthesis orsimulation.

mentor/ Contains a ModelSim script msim_setup.tcl to set up and run a simulation.

aldec/ Contains a Riviera-PRO script rivierapro_setup.tcl to setup and run asimulation.

/synopsys/vcs

/synopsys/vcsmx

Contains a shell script vcs_setup.sh to set up and run a VCS simulation.Contains a shell script vcsmx_setup.sh and synopsys_ sim.setup file toset up and run a VCS MX® simulation.

continued...




/cadence Contains a shell script ncsim_setup.sh and other setup files to set up andrun an NCSIM simulation.

/submodules Contains HDL files for the IP core submodule.

<IP submodule>/ For each generated IP submodule directory, Qsys generates /synth and /simsub-directories.

5.4 Integration and Implementation

5.4.1 Clock Requirements

The Avalon-ST Stratix 10 Hard IP for PCI Express IP Core has a single 100 MHz inputclock and a single output clock. An additional clock is available for PIPE simulationsonly.

refclk

Each instance of the PCIe IP core has its own dedicated refclk input signal. Thisinput reference clock can be sourced from any reference clock in its transceiver tile.Refer to the Stratix 10 GX and SX Device Family Pin Connection Guidelines foradditional information on termination and valid locations.

coreclkout_hip

This output clock is a fixed frequency clock that drives the Application Layer. Theoutput clock frequency is derived from the maximum link width and maximum linkrate of the PCIe IP core.

Table 24. Application Layer Clock Frequency for All Combinations of Link Width, DataRate and Application Layer Interface Widths

Maximum Link Rate Maximum Link Width Avalon-ST InterfaceWidth

coreclkout_hip Frequency

Gen1 x1, x2, x4, x8, x16 256 125 MHz

Gen2 x1, x2, x4, x8 256 125 MHz

Gen2 x16 256 250 MHz

Gen3 x1, x2, x4 256 125 MHz

Gen3 x8 256 250 MHz

Gen3 x16 256 500 MHz

sim_pipe_pclk_in

This input clock is for PIPE simulation only. It is derived from refclk. It is the PIPEinterface clock used for PIPE mode simulation.



5.4.2 Reset Requirements

The Avalon-ST Stratix 10 Hard IP for PCI Express IP Core has two, asynchronous,active low reset inputs, npor and pin_perst. Both reset the Transaction, Data Linkand Physical Layers.

npor

The Application Layer drives the npor reset input to the PCIe IP core. If you choose todesign an Application Layer does not drive npor, you must tie this output to 1'b1. Thenpor signal resets all registers and state machines to their initial values.

pin_perst

This is the PCI Express Fundamental Reset signal. Asserting this signal returns allregisters and state machines to their initialization values. Each instance of the PCIe IPcore has its own dedicated pin_perst pin. You must connect the pin_perst of eachhard IP instance to the corresponding nPERST pin of the device. These pins have thefollowing location.

• NPERSTL0 : Bottom Left PCIe IP core and Configuration via Protocol (CvP)

• NPERSTL1: Middle Left PCIe PCIe IP core (When available)

• NPERSTL2: Top Left PCIe IP core (When available)

• NPERSTR0: Bottom Right PCIe IP core (When available)

• NPERSTR1: Middle Right PCIe IP core (When available)

• NPERSTR2: Top Right PCIe IP core (When available)

For maximum compatibility, always use the bottom PCIe IP core on the left side of thedevice first. This is the only location that supports CvP using the PCIe link.

Note: CvP is not available in the Quartus Prime Pro – Stratix 10 Edition 17.1 Interim Release

reset_status

When asserted, this signal indicates that the PCIe IP core is in reset. Thereset_status signal is synchronous to coreclkout_hip. It is active high.

clr_st

This signal has the same functionality as reset_status. It is provided for backwardscompatibility with Arria 10 devices. It is active high.

5.5 Required Supporting IP Cores

Stratix 10 Hard IP for PCI Express IP core designs always include the Hard ResetController and TX PLL IP cores. System generation automatically adds thesecomponents to the generated design.



5.5.1 Hard Reset Controller

The Hard Reset Controller generates the reset for the PCIe IP core logic, transceivers,and Application Layer. To meet 100 ms PCIe configuration time, the Hard ResetController interfaces with the SDM. This allows the PCIe Hard IP to be configured firstso that PCIe link training occurs when the FPGA fabric is still being configured.

5.5.2 TX PLL

For Gen1 or Gen2, the PCIe IP core uses the fractional PLL ( fPLL) for the TX PLL. ForGen3, the PCIe Hard IP core uses the fPLL when operating at the Gen1 or Gen2 datarate. It uses the advanced transmit (ATX) PLL when operating at the Gen3 data rate.

Note: The ATX PLL is sometimes called the LC PLL, where LC refers to inductor/capacitor.

5.6 Channel Layout and PLL Usage

The following figures show the channel layout and connection between the Hard ResetController (HRC) and PLLs and for Gen1, Gen2 and Gen3 x1, x2, x4, x8 and x16variants of the Stratix 10 Hard IP for PCI Express IP core. The channel layout is thesame for the Avalon-ST and Avalon-MM interfaces to the Application Layer.

Note: All of the PCIe hard IP instances in Stratix 10 devices are Gen3 x16. The Hard ResetController drives all 16 channels. However, channels 8-15 are available for otherprotocols when fewer than 16 channels are used. Refer to Channel Availability formore information.

Figure 28. Gen1 and Gen2 x1

PMA Channel 5PMA Channel 4PMA Channel 3PMA Channel 2PMA Channel 1PMA Channel 0

PCS Channel 5PCS Channel 4PCS Channel 3PCS Channel 2PCS Channel 1PCS Channel 0

fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1

Ch 15Ch 14Ch 13Ch 12Ch 11Ch 10Ch 9Ch 8Ch 7Ch 6Ch 5Ch 4Ch 3Ch 2Ch 1Ch 0

PCIe Hard IP

HRCconnects to

fPLL0






fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0




fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0






fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0




fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 middleXCVR bank



Figure 33. Gen3 x1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0ATXPLL0(Gen3)

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 &ATXPLL0

Figure 34. Gen3 x2



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0ATXPLL0(Gen3)

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 &ATXPLL0



Figure 35. Gen3 x4



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0ATXPLL0(Gen3)

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 &ATXPLL0

Figure 36. Gen3 x8



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0ATXPLL0(Gen3)

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 &ATXPLL0



Figure 37. Gen3 x16



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1



fPLL1

fPLL0ATXPLL0(Gen3)

ATXPLL1



fPLL1

fPLL0

ATXPLL0

ATXPLL1


PCIe Hard IP

HRCconnects to

fPLL0 &ATXPLL1middle

XCVR bank

Related Links

Channel Availability on page 15

5.7 Compiling the Full Design and Programming the FPGA

You can use the Start Compilation command on the Processing menu in the QuartusPrime software to compile your design. After successfully compiling your design,program the targeted Intel device with the Programmer and verify the design inhardware.

Related Links

Programming Intel Devices




6 Block DescriptionsThe Stratix 10 Hard IP for PCI Express implements the complete PCI Express protocolstack as defined in the PCI Express Base Specification. The protocol stack includes thefollowing layers:

• Transaction Layer—The Transaction Layer contains the Configuration Space, whichmanages communication with the Application Layer, the RX and TX channels, theRX buffer, and flow control credits.

• Data Link Layer—The Data Link Layer, located between the Physical Layer and theTransaction Layer, manages packet transmission and maintains data integrity atthe link level. Specifically, the Data Link Layer performs the following tasks:

— Manages transmission and reception of Data Link Layer Packets (DLLPs)

— Generates all transmission link cyclical redundancy code (LCRC) values andchecks all LCRCs during reception

— Manages the retry buffer and retry mechanism according to received ACK/NAKData Link Layer packets

— Initializes the flow control mechanism for DLLPs and routes flow control creditsto and from the Transaction Layer

• Physical Layer—The Physical Layer initializes the speed, lane numbering, and lanewidth of the PCI Express link according to packets received from the link anddirectives received from higher layers. The following figure provides a high-levelblock diagram.

Figure 38. Stratix 10 Hard IP for PCI Express Using the Avalon-ST Interface


MappingPLL

Embe

dded

Mult

i-die

Inte

rconn

ect B

ridge



H-Tile or L-Tile

PCIe Link



Avalon-ST TX

Avalon-ST RX

Hdr & Data Credits

MSI/MSI-X

Clocks

ResetsPCS and PMA

Each channel of the Physical Layer is paired with an Embedded Multi-die InterconnectBridge (EMIB) module. The FPGA fabric interfaces to the PCI Express IP core throughthe EMIB. The TX and RX interfaces each connect to four EMIB modules.

6 Block Descriptions






Related Links


6.1 Interfaces

This section describes the top-level interfaces in the Avalon-ST Stratix 10 Hard IP forPCIe IP core.

Figure 39. Avalon-ST Stratix 10 Hard IP for PCI Express Top-Level Signals

rx_st_data[255:0] rx_st_soprx_st_eoprx_st_readyrx_st_valid rx_st_bar_range[2:0]

Hard IP for Express, Avalon-ST Interface

tx_st_data[255:0] tx_st_soptx_st_eoptx_st_readytx_st_validtx_st_errtx_st_parity[31:0]

TX Port

PowerManagementt

tx_outrx_in Serial Data Interface

Avalon-STRX Port

Avalon-ST

Clocks refclkcoreclkout_hip

int_status[3:0]int_status_common[2:0]

MSI

app_msi_reqapp_msi_ackapp_msi_tc[2:0]app_msi_num[4:0]app_int_sts

tx_ph_cdts[7:0]tx_pd_cdts[11:0]tx_nph_cdts[7:0]tx_npd_cdts[11:0]tx_cplh_cdts[7:0]tx_cpld_cdts[11:0]tx_hdr_cdts_consumedtx_data_cdts_consumedtx_cdts_type[1:0]tx_cdts_data_value[1:0]

Avalon-STTX Credits

npor

reset_status

pin_perst

serdes_pll_locked

pld_core_readypld_clk_inuse

current_speed[1:0]

Reset

derr_cor_ext_rcvderr_cor_ext_rpl

derr_rplderr_uncor_ext_rcv

int_statusint_status_common[2:0]

lane_act[4:0]link_up

ltssmstate[5:0]rx_par_errtx_par_err

Hard IP StatusInterface Clocks

tl_cfg_add[4:0]tl_cfg_ctl[31:0]tl_cfg_func[1:0]

app_err_hdr[31:0]app_err_info[10:0]

ConfigurationSpace Signals

app_err_valid

txdata0[31:0]txdatak0[3:0]

rxdata0[31:0]rxdatak[3:0]

txdetectrxtxelectidle

txcompl

rxpolarity

powerdown[1:0]txmargin[2:0]

txswing

rxvalidphystatus0

rxelecidlerxstatus[2:0]

simu_mode_pipe

sim_pipe_rate[1:0]sim_pipe_pclk_in

sim_ltssmstate[5:0]

sim_pipe_mask_tx_pll

txdeemph

txblkst[3:0]txdataskip

Transmit Data

Receive Data

PIPE Interface for Simulation and HardwareDebug Usingdl_ltssm[4:0]

in SignalTap

rxdataskiprxblkst[3:0]

txsynchd[1:0]

currentcoeff0[17:0]currentrxpreset0[2:0]

rxsynchd[3:0]

rate[1:0]

rxeqevalrxeqinprogress

invalidreq

dirfeedback[5:0]

Hard IPReconfiguration(Optional)

hip_reconfig_clkhip_reconfig_rst_n

hip_reconfig_address[20:0]hip_reconfig_read

hip_reconfig_readdata[7:0]

hip_reconfig_writehip_reconfig_writedata[15:0]

hip_reconfig_readdatavalid

hip_reconfig_waitrequest

pm_linkst_in_l1pm_linkst_in_10s

pm_state[2:0]pm_dstate[2:0]

apps_pm_xmt_pmeapps_ready_entr_l23

apps_pm_xmt_turnoffapp_init_rst

app_xfer_pending

Configuration Extension Bus

ceb_reqceb_ackceb_addr[11:0]ceb_din[31:0]ceb_dout[31:0]ceb_wr[3:0]

TestInterface

test_int[66:0]testin_zero



http://www.pcisig.com

6.1.1 TLP Header and Data Alignment for the Avalon-ST RX and TXInterfaces

The TLP header and data is packed on the TX and RX interfaces.

The ordering of bytes in the header and data portions of packets is different. The firstbyte of the header dword is located in the most significant byte of the dword. The firstbyte of the data dword is located in the least significant byte of the dword on the databus. In the figure H0 to H3 are header dwords. D0 to D9 are data dwords.

Figure 40. Three and Four DWord Header and Data on the TX and RX Interfaces

Byte 3Byte 2Byte 1Byte 0

D3


D2


D1


D0


H3


H2


H1


H0

0

255


D9


D8


D7


D6


D5


D4

4DW Header and Data


D3


D2


D1


D0


H2


H1


H0

0

255


D9


D8


D7


D6


D5


D4

3DW Header and Data



Note: Unlike previous generations, data is not qword aligned. If you are porting yourapplication from an earlier implementation, you must update your application tocompensate for this change.

6.1.2 Avalon-ST RX Interface

The Application Layer receives data from the Transaction Layer of the PCIe IP coreover the Avalon-ST RX interface. This is a 256-bit interface.

Table 25. 256-Bit Avalon-ST RX Datapath

Signal Direction Description

rx_st_data[255:0] Output Receive data bus. The Application Layer receives data from the TransactionLayer on this bus.Refer to the TLP Header and Data Alignment for the Avalon-ST TX andAvalon-ST RX Interfaces for the layout of TLP headers and data.

rx_st_sop Output Marks the first cycle of the TLP when both rx_st_sop and rx_st_validare asserted.

rx_st_eop Output Marks the last cycle of the TLP when both rx_st_eop and rx_st_validare asserted.

rx_st_ready Input Indicates that the Application Layer is ready to accept data. TheApplication Layer deasserts this signal to throttle the data stream.

rx_st_valid Output Clocks rx_st_data into the Application Layer.The rx_st_ready to rx_st_valid latency for Stratix 10 devices is 17cycles. When rx_st_ready ready deasserts, rx_st_valid mustdeassert within 17 cycles. When rx_st_ready ready reasserts,rx_st_valid must reassert within 17 cycles if there is more data tosend. To achieve the best throughput, Intel recommends that you size theRX buffer to avoid the deassertion of rx_st_ready. Refer to Avalon-STTX Interface tx_st_ready Deasserts for a timing diagram that illustratesthis behavior.

rx_st_bar_range[2:0] Output Specifies the bar for the TLP being output. The following encodings aredefined:• 000: Memory Bar 0• 001: Memory Bar 1• 010: Memory Bar 2• 011: Memory Bar 3• 100: Memory Bar 4• 101: Memory Bar 5• 110: Reserved• 111: Reserved

6.1.2.1 Avalon-ST RX Interface Three- and Four-Dword TLPs

These timing diagrams illustrate the layout of headers and data for the Avalon-ST RXinterface.



Figure 41. Avalon-ST RX Interface Cycle Definition for Three-Dword Header TLPsIn this example, rx_st_empty specifies six empty dwords in the last cycle, whenrx_st_eop asserts.

rx_st_valid

rx_st_data[127:96]

rx_st_data[95:64]

rx_st_data[63:32]

rx_st_data[31:0]

rx_st_soprx_st_eop

rx_st_empty[2:0]

Data 0 Data 8

Header 2 Data 7

Header 1 Data 6 Data (n)

Header 0 Data 5 Data (n-1)

rx_st_data[255:224]

rx_st_data[223:192]

rx_st_data[191:160]

rx_st_data[159:128]

Data 4 Data 12

Data 3 Data 11

Data 2 Data 10

Data 1 Data 9

coreclkout_hip

60 0

Figure 42. Avalon-ST RX Interface Cycle Definition for Four-Dword Header TLPsIn this example, rx_st_empty specifies four empty dwords in the last cycle, whenrx_st_eop asserts.

coreclkout_hip

rx_st_valid

rx_st_data[127:96]

rx_st_data[95:64]

rx_st_data[63:32]

rx_st_data[31:0]

rx_st_sop

rx_st_eop

Header3 Data 7 Data n

Header 2 Data 6 Data n-1



rx_st_data[255:224]

rx_st_data[223:192]

rx_st_data[191:160]

rx_st_data[159:128]

Data 3 Data 11

Data 2 Data 10

Data 1 Data 9

Data 0 Data 8

rx_st_empty[2:0] 40 0



6.1.2.2 Avalon-ST RX Interface rx_st_ready Deasserts

This timing diagram illustrates the timing of the RX interface when the applicationthrottles the Stratix 10 Hard IP for PCI Express by deasserting rx_st_ready.Therx_st_valid signal deasserts within 17 cycles of the rx_st_ready deassertion. Therx_st_valid signal reasserts with 17 cycles after rx_st_ready reasserts if there ismore data to send. rx_st_data is held until the application is able to accept it.

To achieve the best throughput, Intel recommends that you size the RX buffer to avoidthe deassertion of rx_st_ready.

Figure 43. Avalon-ST RX Interface rx_st_ready Deasserts

coreclkout_hip

rx_st_data[255:0]

rx_st_sop

rx_st_eop

rx_st_ready

rx_st_valid

000 CC... CC... CC... CC...CC... CC... CC...

17 clks

6.1.2.3 Avalon-ST RX Interface rx_st_valid Deasserts

The rx_st_valid signal can deassert, even when rx_st_ready remains asserted.

Figure 44. Avalon-ST RX Interface rx_st_valid Deasserts

coreclkout

rx_st_data[255:0]

rx_st_sop

rx_st_eop

rx_st_ready

rx_st_valid

BB.. . . . . . . . . . . . . . . . . . .

6.1.2.4 Avalon-ST RX Back-to-Back Transmission

This timing diagram illustrates back-to-back transmission on the Avalon-ST RXinterface with no idle cycles between the assertion of rx_st_eop and rx_st_sop.



Figure 45. Avalon-ST RX Back-to-Back Transmission

coreclkout_hip

rx_st_data[255:0]

rx_st_sop

rx_st_eop

rx_st_ready

rx_st_valid

BB ... BB ... BB ... BB ... BB ... BB ... BB ... BB ... BB ... BB ... BB ... BB ... ...BB

6.1.2.5 Avalon-ST RX Interface Single-Cycle TLPs

This timing diagram illustrates two single-cycle TLPs.

Figure 46. Avalon-ST RX Single-Cycle TLPs

pld_clk

rx_st_data[255:0]

rx_st_sop

rx_st_eop

rx_st_ready

rx_st_valid

XX..BE ... XXXXXXXXXXXXXXXX. . . 4592001487DF08876210...

6.1.3 Avalon-ST TX Interface

The Application Layer transfers data to the Transaction Layer of the PCIe IP core overthe Avalon-ST TX interface. This is a 256-bit interface.

Table 26. 256-Bit Avalon-ST TX Datapath


tx_st_data[255:0] Input Data for transmission. The Application Layer must provide a properlyformatted TLP on the TX interface. Valid when tx_st_valid is asserted.The mapping of message TLPs is the same as the mapping of TransactionLayer TLPs with 4 dword headers. The number of data cycles must becorrect for the length and address fields in the header. Issuing a packetwith an incorrect number of data cycles results in the TX interface hangingand becoming unable to accept further requests.Refer to the TLP Header and Data Alignment for the Avalon-ST TX andAvalon-ST RX Interfaces for the layout of TLP headers and data.

tx_st_sop Input Indicates first cycle of a TLP when asserted together with tx_st_valid.

tx_st_eop Input Indicates last cycle of a TLP when asserted together with tx_st_valid.

tx_st_ready Output Indicates that the Transaction Layer is ready to accept data fortransmission. The core deasserts this signal to throttle the data stream.tx_st_ready may be asserted during reset. The Application Layer should

continued...




wait at least 2 clock cycles after the reset is released before issuingpackets on the Avalon-ST TX interface. The reset_status signal can alsobe used to monitor when the IP core has come out of reset.If tx_st_ready is asserted by the Transaction Layer on cycle <n> , then<n> + readyLatency is a ready cycle, during which the ApplicationLayer may assert valid and transfer data.The readyLatency is 3 coreclkout_hip cycles.

tx_st_valid Input Clocks tx_st_data to the core when tx_st_ready is also asserted.Between tx_st_sop and tx_st_eop, tx_st_valid must not bedeasserted in the middle of a TLP except in response to tx_st_readydeassertion. When tx_st_ready deasserts, this signal must deassertwithin 3 coreclkout_hip cycles. When tx_st_ready reasserts, andtx_st_data is in mid-TLP, this signal must reassert within 3 cycles. Thefigure entitled Avalon-ST TX Interface tx_st_ready Deasserts illustratesthe timing of this signal.To facilitate timing closure, Intel recommends that you register both thetx_st_ready and tx_st_valid signals.

tx_st_err Input Indicates an error on transmitted TLP. This signal is used to nullify apacket. To nullify a packet, assert this signal for 1 cycle with tx_st_eop.When a packet is nullified, the following packet should not be transmitteduntil the next clock cycle.

tx_st_parity[31:0] Output The IP core supports byte parity. Each bit represents odd parity of theassociated byte of the tx_st_data bus. For example, bit[0] correspondsto tx_st_data[7:0], bit[1] corresponds to tx_st_data[15:8], and soon.

6.1.3.1 Avalon-ST TX Three- and Four-Dword TLPs

These timing diagrams illustrate the layout of headers and data for the Avalon-ST TXinterface.

Figure 47. Avalon-ST TX Interface Cycle Definition for Three-Dword Header TLPsIn this example, tx_st_empty specifies six empty dwords in the last cycle, whentx_st_eop asserts.

tx_st_valid

tx_st_data[127:96]

tx_st_data[95:64]

tx_st_data[63:32]

tx_st_data[31:0]

tx_st_soptx_st_eop

tx_st_empty[2:0]

Data 0 Data 8

Header 2 Data 7

Header 1 Data 6 Data (n)

Header 0 Data 5 Data (n-1)

tx_st_data[255:224]

tx_st_data[223:192]

tx_st_data[191:160]

tx_st_data[159:128]

Data 4 Data 12

Data 3 Data 11

Data 2 Data 10

Data 1 Data 9

coreclkout_hip

60 0



Figure 48. Avalon-ST TX Interface Cycle Definition for Four-Dword Header TLPsIn this example, tx_st_empty specifies four empty dwords in the last cycle, whentx_st_eop asserts.

coreclkout_hip

tx_st_valid

tx_st_data[127:96]

tx_st_data[95:64]

tx_st_data[63:32]

tx_st_data[31:0]

tx_st_sop

tx_st_eop

Header3 Data 7 Data n




tx_st_data[255:224]

tx_st_data[223:192]

tx_st_data[191:160]

tx_st_data[159:128]

Data 3 Data 11

Data 2 Data 10

Data 1 Data 9

Data 0 Data 8

tx_st_empty[2:0] 40 0

6.1.3.2 Avalon-ST TX Interface tx_st_ready Deasserts

This timing diagram illustrates the timing of the TX interface when the Stratix 10 HardIP for PCI Express pauses the application by deasserting tx_st_ready. The timingdiagram shows a readyLatency of 3 cycles to facilitate timing closure. Theapplication deasserts tx_st_valid after three cycles.

Figure 49. Avalon-ST TX Interface tx_st_ready Deasserts

coreclkout_hip

tx_st_data[255:0]

tx_st_sop

tx_st_eop

tx_st_ready

tx_st_valid

tx_st_err

000 CC... CC... CC... CC... CC... CC... CC... CC... CC... CC...4 clks

3 clks

6.1.3.3 Avalon-ST TX Back-to-Back Transmission

This timing diagram illustrates back-to-back transmission on the Avalon-ST TXinterface with no idle cycles between the assertion of tx_st_eop and tx_st_sop.

Figure 50. Avalon-ST TX Back-to-Back Transmission



Figure 51.

coreclkout_hip

tx_st_data[255:0]

tx_st_sop

tx_st_eop

tx_st_ready

tx_st_valid

tx_st_err

.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1.4 TX Credit Interface

Before a TLP can be transmitted, flow control logic verifies that the link partner's RXport has sufficient buffer space to accept it. The TX Credit interface reports theavailable RX buffer space to the Application. It reports the space available in unitscalled Flow Control credits.

Flow Control credits are defined for the following TLP categories:

• Posted transactions - TLPs that do not require a response

• Non-Posted transactions - TLPs that require a completion

• Completions - TLPs that respond to non-posted transactions

Table 27. Categorization of Transactions Types

TLP Type Category

Memory Write Posted

Memory ReadMemory Read Lock

Non-posted

I/O ReadI/O Write

Non-posted

Configuration ReadConfiguration Write

Non-posted

Message Posted (The routing method depends on the Message type.)

CompletionsCompletions with DataCompletion LockedCompletion Lock with Data

Completions

Fetch and Add AtomicOp Non-posted



Table 28. Value of Header and Data CreditsThe following table translates credits to dwords. The PCI Express Base Specification defines a dword as fourbytes.

Credit Type Number of Dwords

Header Credit - completions 4 dwords

Header credit - requests 5 dwords

Data credits 4 dwords

Table 29. 256-Bit Avalon-ST TX DatapathThe IP core provides TX credit information to the Application Layer. To optimize performance, the Applicationcan perform credit-based checking before submitting requests for transmission, and reorder packets to improvethroughput. Reordering is optional. The IP core always checks for sufficient TX credits before transmitting anyTLP.


tx_ph_cdts[7:0] Output Header credit net value for the flow control (FC) posted requests.

tx_pd_cdts[11:0] Output Data credit net value for the FC posted requests.

tx_nph_cdts[7:0] Output Header credit net value for the FC non-posted requests.

tx_npd_cdts[11:0] Output Data credit net value for the FC non-posted requests.

tx_cplh_cdts[7:0] Output Header credit net value for the FC Completion. A value of 0 indicatesinfinite Completion header credits.

tx_cpld_cdts[11:0] Output Data credit net value for the FC Completion. A value of 0 indicates infiniteCompletion data credit.

tx_hdr_cdts_consumed Output Asserted for 1 coreclkout_hip cycle, for each header credit consumed.Note that credits the Hard IP consumes for internally generatedCompletions or Messages are not tracked in this signal.

tx_data_cdts_consumed Output Asserted for 1 coreclkout_hip cycle, for each data credit consumed.Note that credits the Hard IP consumes for internally generatedCompletions or Messages are not tracked in this signal.

tx_cdts_type[1:0] Output Specifies the credit type shown on the tx_cdts_data_value[1:0] bus.The following encodings are defined:• 2'b00: Posted credits• 2'b01: Non-posted credits2

• 2'b10: Completion credits• 2'b11: Reserved

tx_cdts_data_value[1:0] Output The value of tx_cdts_data_value[1:0]+1 specifies the data creditconsumed when tx_data_cdts_consumed is asserted.

6.1.5 Interpreting the TX Credit Interface

The Stratix 10 PCIe TX credit interface reports the number of Flow Control creditsavailable.

The following equation defines the available buffer space:

RX_buffer_space = (credit_limit - credits_consumed)+ released_credits

2 The PCIe IP core does not consume non-posted credits.



where:

credits_consumed = credits_consumedapplication + credits_consumedPCIe_IP_core

The hard IP core consumes a small number of posted credits for the following reasons:

• To send completion TLPs for Configuration Requests targeting its internal registers

• To transmit posted TLPs for error and interrupt Messages

The hard IP core does not report the internally consumed credits to the Application.Without knowing the hard IP core's internal usage, the Application cannot maintain acompletely accurate count of Posted or Completion credits.

The hard IP core does not consume non-posted credits. Consequently, it is possible tomaintain an accurate count of the available non-posted credits. Refer to the TX CreditAdjustment Sample Code for code that calculates the number of non-posted creditsavailable to the Application. This RTL recovers the updated Flow Control credits fromthe remote link. It drives value of RX_buffer_space for non-posted header and datacredits on tx_nph_cdts and tx_npd_cdts, respectively.

Figure 52. Flow Control Credit Update LoopThe following figure provides an overview of the steps to determine credits availableand release credits after the TLP is removed from the link partner's RX buffer.

Flow ControlGating Logic

(Credit Check)

CustomComputation:

RX_buf_space=((limit-consumed)

+ released)

FC UpdateDLLP Decode

FC UpdateDLLP Generate

CreditAllocated

RXBuffer

FU Update DLLP

Data PacketData

Packet

ApplicationLayer

TransactionLayer

Data LinkLayer

PhysicalLayer

PCIExpress

Link

ApplicationLayer

TransactionLayer

Data LinkLayer

PhysicalLayer

Data Source Data Sink (Link Partner)

Allow Decrement

Increment

1 2

7

6

5

43

TX CreditSignals

Note: To avoid a potential deadlock or performance degradation, the Application shouldcheck available credits before sending a TLP to the Stratix 10 Hard IP for PCI ExpressIP Core.

Related Links

TX Credit Adjustment Sample Code on page 160This sample Verilog HDL code computes the available credits for non-posted TLPs.



6.1.6 Clocks

Table 30. Clocks


refclk Input This is the input reference clock for the IP core as defined by the PCIExpress Card Electromechanical Specification Revision 2.0. Its frequency is100 MHz ±300 ppm.

coreclkout_hip Output This clock drives the Data Link, Transaction, and Application Layers. Forthe Application Layer, its frequency depends on the data rate and thenumber of lanes as specified in the table

Data Rate coreclkout_hip Frequency

Gen1 x1, x2, x4, x8, and x16 125 MHz

Gen2 x1, x2, x4, and x8, 125 MHz

Gen2 x16 250 MHz

Gen3 x1, x2, and x4 125 MHz

Gen3 x8 250 MHz

Gen3 x16 500 MHz

6.1.7 Update Flow Control Timer and Credit Release

The IP core releases credits from the RX Buffer on a per-clock-cycle basis as itremoves TLPs from the RX Buffer.

6.1.8 Resets

An embedded hard reset controller (HRC) generates the reset for PCIe core logic,PMA, PCS, and Application. To meet the 100 ms PCIe configuration time, theembedded reset controller connects to the SDM. This connection allows the FPGAperiphery to configure and begin operation before the FPGA fabric is programmed.

The reset logic requires a free running clock that is stable and available to the IP corewhen the SDM asserts iocsrrdy. A delayed version of iocsrrdy, (iocsrrdy_dly),asserts after the control and status registers are initialized and transceiverconfiguration completes.



Figure 53. Stratix 10 Hard IP for PCI Express Reset ControllerThe signals shown in the Hard IP for PCIe are implemented in hard logic and notavailable for probing.

EMIB Adaptor

Self Download Mode(SDM)

PCIe IP Core

Hard IPReset

Controller

PHY(PCSand

PMA)

SDM

user

_mod

eloc

sr_rd

y fref_dk[15:0]

Hard IP for PCIe

serdes_pll_locked[15:0]pma_cal_done[15:0]rx_pma_rstbrx/tx_pcs_rstn

rst_n <n>

pld_rst_n

pld_rst_n

nporreset_statuslink_req_rst_n

pipe_clk

pin_perst_n

FPGA

Table 31. Resets


currentspeed[1:0] Output Indicates the current speed of the PCIe link. The followingencodings are defined:• 2'b00: Undefined• 2'b01: Gen1• 2'b10: Gen2• 2'b11: Gen3

npor Input The Application Layer drives this active low reset signal. npor resets theentire IP core, PCS, PMA, and PLLs. npor should be held for a minimum of20 ns. Gen3 x16 variants, should hold npor for at least 10 cycles. Thissignal is edge, not level sensitive; consequently, a low value on this signaldoes not hold custom logic in reset. This signal cannot be disabled.

pin_perst Input Active low reset from the PCIe reset pin of the device. Resets the datapathand control registers.

pld_clk_inuse Output This reset signal has the same effect as reset_status. This signal isprovided for backwards compatibility with Arria 10 devices.

pld_core_ready Input When asserted, indicates that the Application Layer is ready. The IP corecan releases reset after this signal is asserted.

continued...




reset_status Output Active high reset status. When high, indicates that the IP core is not readyfor usermode. reset_status is deasserted only when npor isdeasserted and the IP core is not in reset. Use reset_status to drivethe reset of your application. Synchronous to coreclkout_hip.

clr_st Output clr_st has the same functionality as reset_status. It is provided forbackwards compatibility with previous device families.

serdes_pll_locked Output When asserted, indicates that the PLL that generates coreclkout_hip islocked. In pipe simulation mode this signal is always asserted.

6.1.9 Interrupts

6.1.9.1 MSI Interrupts

Message Signaled Interrupts (MSI) interrupts are signaled on the PCI Express linkusing a single dword Memory Write TLP.

Table 32. MSI


app_msi_req Input Application Layer MSI request. Assertion causes an MSI posted write TLPto be generated based on the MSI configuration register values and theapp_msi_tc and app_msi_num input ports. Deasserts whenapp_msi_ack acknowledges the request.

app_msi_ack Output Application Layer MSI acknowledge. Asserts for 1 cycle to acknowledge theApplication Layer's request for an MSI interrupt.

app_msi_tc[2:0] Input Application Layer MSI traffic class. This signal indicates the traffic classused to send the MSI (unlike INTX interrupts, any traffic class can be usedto send MSIs).

app_msi_num[4:0] Input MSI number of the Application Layer. This signal provides the low ordermessage data bits to be sent in the message data field of MSI messagesrequested by app_msi_req. Only bits that are enabled by the MSIMessage Control register apply.

6.1.9.2 Legacy Interrupts

Legacy interrupts mimic the original PCI level-sensitive interrupts using virtual wiremessages. The term, INTx, refers collectively to the four legacy interrupts, INTA#,INTB#, INTC# and INTD#.



Table 33. Legacy Interrupts (MSI)


app_int_sts Input Controls legacy interrupts. Assertion of app_int_sts causes anAssert_INTx message TLP to be generated and sent upstream. Deassertionof app_int_sts causes a Deassert_INTx message TLP to be generatedand sent upstream.

int_status[3:0] Output int_status[3:0] drives legacy interrupts to the Application Layer asfollows:• int_status[0]: interrupt signal A• int_status[1]: interrupt signal B• int_status[2]: interrupt signal C• int_status[3]: interrupt signal D

int_status_common[2:0] Output Specifies the interrupt status for the following registers. When asserted,indicates that an interrupt is pending:• int_status_common[0]: Autonomous bandwidth status register.• int_status_common[1]: Bandwidth management status register.• int_status_common[2]: Link equalization request bit in the link

status register.

6.1.10 Transaction Layer Configuration Space Interface

The Transaction Layer (TL) bus provides a subset of the information stored in theConfiguration Space. Use this information in conjunction with the app_err* signals tounderstand TLP transmission problems.

Table 34. Configuration Space Signals


tl_cfg_add[4:0] Output Address of the TLP register. This signal is an index indicating whichConfiguration Space register information is being driven ontotl_cfg_ctl. Refer to Multiplexed Configuration Register InformationAvailable on tl_cfg_ctl for the available information.

tl_cfg_ctl[31:0] Output The tl_cfg_ctl signal is multiplexed and contains a subset of contentsof the Configuration Space registers.

tl_cfg_func[1:0] Output Specifies the function whose Configuration Space register values are beingdriven tl_cfg_ctl[31:0]. The following encodings are defined:• 2'b00: Physical Function (PF0)• 2'b01: Reserved for L-tile; PF1 for H-Tile• 2'b10: Reserved for L-tile; Virtual Functions (VF) for H-Tile• 2'b11: Reserved

app_err_hdr[31:0] Input Header information for the error TLP. Four, 4-byte transfers send thisinformation to the IP core.

app_err_info[10:0] Input The Application can optionally provide the following information:• app_err_info[0]: Malformed TLP• app_err_info[1]: Receiver Overflow• app_err_info[2]: Unexpected Completion• app_err_info[3]: Completer Abort• app_err_info[4]: Completer Timeout• app_err_info[5]: Unsupported Request• app_err_info[6]: Poisoned TLP Received• app_err_info[7]: AtomicOp Egress Blocked

continued...




• app_err_info[8]: Uncorrectable Internal Error• app_err_info[9]: Correctable Internal Error• app_err_info[10]: Advisory Non-Fatal Error

app_err_valid Input When asserted, indicates that the data on app_err_info[10:0] is valid.

Figure 54. Configuration Space Register Access TimingInformation on the Transaction Layer (TL) bus is time-division multiplexed (TDM).When tl_cfg_func[1:0]= 2'b00, tl_cfg_ctl[31:0] drives PF0 ConfigurationSpace register values for eight consecutive cycles. The next 40 cycles driveinformation for PF1 and the VFs which are not supported in the current release. Then,the 48-cycle pattern repeats.

coreclkout_hip

Cycle:

tl_cfg_ctl[31:0]

tl_cfg_add[4:0]

tl_cfg_func[1:0]

0 1 2 3 4 5 6

1 2 3 4 5 6

7 0 6 7 0

0=PF0 0=PF0

Cycle4Cycle0 Cycle1 Cycle3Cycle2 Cycle5 Cycle6 Cycle8Cycle7 Cycle46 Cycle48Cycle47

0 8 487 4746

Table 35. Multiplexed Configuration Register Information Available on tl_cfg_ctl

TDM

31 24 23 16 15 8 7 0

0 [28:24] Device Number Bus Number Reserved Device Control[2:0]: Max payload size[5:3]: Max rd req size[6]: Extended tag enable[7]: Bus master enable

1 [28:24]AER IRQ msg num [16]: RCB cntl[17]: cfg_pm_no_soft_rst

PCIe capability IRQ msgnum[20:16]

[1:0]: Sys power ind. cntl[3:2]: Sys atten ind cntl[4]: Sys power cntl

2 Index of 1st VF Num VFs Reserved [0]: VF enable[2:1]: TPH[3]: Atomic req en[4:] AtomicOp egress block[5]: ARI forward enable[6]: End-End TLP prefixblocking

3 MSI Address Lower

4 MSI Address Upper

5 MSI Mask

6 MSI Data Reserved [0]: MSI enable[1]: 64-bit MSI[4:2]: Multiple MSI enable

7 Reserved



6.1.11 Configuration Extension Bus Interface

Use the Configuration Extension Bus to add capability structures to the IP core’sinternal Configuration Spaces. Configuration TLPs with a destination register byteaddress of 0xC00 and higher are routed to the Configuration Extension Bus interface.Report the Completion Status Successful Completion (SC) on the ConfigurationExtension Bus. The IP core then generates a Completion to transmit on the link.Use the app_err_info[8] signal included in the Transaction Layer ConfigurationSpace Interface to report uncorrectable internal errors.

Note: The IP core does not apply ID-based Ordering (IDO) bits to the internally generatedCompletions.


ceb_req Output When asserted, indicates a valid Configuration Extension Bus access cycle.Deasserted when ceb_ack is asserted.

ceb_ack Input Asserted to acknowledged ceb_req. The Application must implement thislogic.

ceb_addr[11:0] Output Address bus to the external register block. The width of the address bus isthe value you select for the CX_LBC_EXT_AW parameter.

ceb_din[31:0] Input Read data.

ceb_dout[31:0] Output Data to be written.

ceb_wr[3:0] Output Indicates the configuration register access type, read or write. For writes,CEB_wr also indicates the byte enables: The following encodings aredefined:• 4'b000: Read• 4'b0001: Write byte 0• 4'b0010: Write byte 1• 4'b0100: Write byte 2• 4'b1000: Write byte 3• 4'b1111: Write all bytes.Combinations of byte enables, for example,4'b 0101b are also valid.

6.1.12 Hard IP Status Interface

Hard IP Status: This optional interface includes the following signals that are useful fordebugging, including: link status signals, interrupt status signals, TX and RX parityerror signals, correctable and uncorrectable error signals.

Table 36. Hard IP Status Interface


derr_cor_ext_rcv Output When asserted, indicates that the RX buffer detected a 1-bit (correctable)ECC error. This is a pulse stretched output.

derr_cor_ext_rpl Output When asserted, indicates that the retry buffer detected a 1-bit(correctable) ECC error. This is a pulse stretched output.

derr_rpl Output When asserted, indicates that the retry buffer detected a 2-bit(uncorrectable) ECC error. This is a pulse stretched output.

derr_uncor_ext_rcv Output When asserted, indicates that the RX buffer detected a 2-bit(uncorrectable) ECC error. This is a pulse stretched output.

continued...




int_status[3:0] Output Drives legacy interrupts to the Application Layer as follows:• int_status[0]: interrupt signal A

• int_status[1]: interrupt signal B• int_status[2]: interrupt signal C• int_status[3]: interrupt signal D

int_status_common[2:0] Output Specifies the interrupt status for the following registers. When asserted,indicates that an interrupt is pending:• int_status_common[0]: Autonomous bandwidth status register.• int_status_common[1]: Bandwidth management status register.• int_status_common[2]: Link equalization request bit in the link

status register.

lane_act[4:0] Output Lane Active Mode: This signal indicates the number of lanes thatconfigured during link training. The following encodings are defined:• 5’b0 0001: 1 lane• 5’b0 0010: 2 lanes• 5’b0 0100: 4 lanes• 5’b0 1000: 8 lanes• 5'b1 0000: 16 lanes

link_up Output When asserted, the link is up.

ltssmstate[5:0] Output LTSSM state: The LTSSM state machine encoding defines the followingstates:• 6'h00 - Detect.Quiet• 6'h01 - Detect.Active• 6'h02 - Polling.Active• 6'h03 - Polling.Compliance• 6'h04 - Polling.Configuration• 6'h05 - PreDetect.Quiet• 6'h06 - Detect.Wait• 6'h07 - Configuration.Linkwidth.Start• 6'h08 - Configuration.Linkwidth.Accept• 6'h09 - Configuration.Lanenum.Wait• 6'h0A - Configuration.Lanenum.Accept• 6'h0B - Configuration.Complete• 6'h0C - Configuration.Idle• 6'h0D - Recovery.RcvrLock• 6'h0E - Recovery.Speed• 6'h0F - Recovery.RcvrCfg• 6'h10 - Recovery.Idle• 6'h20 - Recovery.Equalization Phase 0• 6'h21 - Recovery.Equalization Phase 1• 6'h22 - Recovery.Equalization Phase 2• 6'h23 - Recovery.Equalization Phase 3• 6'h11 - L0• 6'h12 - L0s• 6'h13 - L123.SendEIdle• 6'h14 - L1.Idle• 6'h15 - L2.Idle• 6'h16 - L2.TransmitWake• 6'h17 - Disabled.Entry• 6'h18 - Disabled.Idle• 6'h19 - Disabled• 6'h1A - Loopback.Entry

continued...




• 6'h1B - Loopback.Active• 6'h1C - Loopback.Exit• 6'h1D - Loopback.Exit.Timeout• 6'h1E - HotReset.Entry• 6'h1F - Hot.Reset

rx_par_err Output Asserted for a single cycle to indicate that a parity error was detected in aTLP at the input of the RX buffer. This error is logged as an uncorrectableinternal error in the VSEC registers. For more information, refer toUncorrectable Internal Error Status Register. If this error occurs, you mustreset the Hard IP because parity errors can leave the Hard IP in anunknown state.

tx_par_err Output Asserted for a single cycle to indicate a parity error during TX TLPtransmission. The IP core transmits TX TLP packets even when a parityerror is detected.

Related Links

Uncorrectable Internal Error Status Register on page 96This register reports the status of the internally checked errors that areuncorrectable.

6.1.13 Serial Data Interface

The IP core supports 1, 2, 4, 8, or 16 lanes.

Table 37. Serial Data Interface


tx_out Output Transmit serial data output.

rx_in Input Receive serial data input.

6.1.14 PIPE Interface

The Stratix 10 PIPE interface compiles with the PHY Interface for the PCI ExpressArchitecture PCI Express 3.0 specification.


txdata[31:0] Output Transmit data.

txdatak[3:0] Output Transmit data control.

txcompl Output Transmit compliance. This signal drives the TX compliance pattern. Itforces the running disparity to negative in Compliance Mode (negativeCOM character).

txelecidle Output Transmit electrical idle. This signal forces the TX output to electrical idle.

txdetectrx Output Transmit detect receive. This signal tells the PHY layer to start a receivedetection operation or to begin loopback.

powerdown[1:0] Output Power down. This signal requests the PHY to change its power state to thespecified state (P0, P0s, P1, or P2).

txmargin[2:0] Output Transmit VOD margin selection. The value for this signal is based on thevalue from the Link Control 2 Register.

continued...




txdeemp Output Transmit de-emphasis selection. The Stratix 10 Hard IP for PCI Expresssets the value for this signal based on the indication received from theother end of the link during the Training Sequences (TS). You do not needto change this value.

txswing Output When asserted, indicates full swing for the transmitter voltage. Whendeasserted indicates half swing.

txsynchd[1:0] Output For Gen3 operation, specifies the receive block type. The followingencodings are defined:• 2'b01: Ordered Set Block• 2'b10: Data BlockDesigns that do not support Gen3 can ground this signal.

txblkst[3:0] Output For Gen3 operation, indicates the start of a block in the transmit direction.pipe spec

txdataskip Output For Gen3 operation. Allows the MAC to instruct the TX interface to ignorethe TX data interface for one clock cycle. The following encodings aredefined:• 1’b0: TX data is invalid• 1’b1: TX data is valid

rate[1:0] Output The 2-bit encodings have the following meanings:• 2’b00: Gen1 rate (2.5 Gbps)• 2’b01: Gen2 rate (5.0 Gbps)• 2’b1X: Gen3 rate (8.0 Gbps)

rxpolarity Output Receive polarity. This signal instructs the PHY layer to invert the polarity ofthe 8B/10B receiver decoding block.

currentrxpreset[2:0] Output For Gen3 designs, specifies the current preset.

currentcoeff[17:0] Output For Gen3, specifies the coefficients to be used by the transmitter. The 18bits specify the following coefficients:• [5:0]: C-1

• [11:6]: C0

• [17:12]: C+1

rxeqeval Output For Gen3, the PHY asserts this signal when it begins evaluation of thetransmitter equalization settings. The PHY asserts Phystatus when itcompletes the evaluation. The PHY deasserts rxeqeval to abortevaluation.Refer to the figure below for a timing diagram illustrating this process.

rxeqinprogress Output For Gen3, the PHY asserts this signal when it begins link training. The PHYlatches the initial coefficients from the link partner.Refer to the figure below for a timing diagram illustrating this process.

invalidreq Output For Gen3, indicates that the Link Evaluation feedback requested a TXequalization setting that is out-of-range. The PHY asserts this signalcontinually until the next time it asserts rxeqeval.

rxdata[31:0] Input Receive data control. Bit 0 corresponds to the lowest-order byte ofrxdata, and so on. A value of 0 indicates a data byte. A value of 1indicates a control byte. For Gen1 and Gen2 only.

rxdatak[3:0] Input Receive data control. This bus receives data on lane. Bit 0 corresponds tothe lowest-order byte of rxdata, and so on. A value of 0 indicates a databyte. A value of 1 indicates a control byte. For Gen1 and Gen2 only.

phystatus Input PHY status. This signal communicates completion of several PHY requests.pipe spec

continued...




rxvalid Input Receive valid. This signal indicates symbol lock and valid data on rxdataand rxdatak.

rxstatus[2:0] Input Receive status. This signal encodes receive status, including error codesfor the receive data stream and receiver detection.

rxelecidle Input Receive electrical idle. When asserted, indicates detection of an electricalidle. pipe spec

rxsynchd[3:0] Input For Gen3 operation, specifies the receive block type. The followingencodings are defined:• 2'b01: Ordered Set Block• 2'b10: Data BlockDesigns that do not support Gen3 can ground this signal.

rxblkst[3:0] Input For Gen3 operation, indicates the start of a block in the receive direction.

rxdataskip Input For Gen3 operation. Allows the PCS to instruct the RX interface to ignorethe RX data interface for one clock cycle. The following encodings aredefined:• 1’b0: RX data is invalid• 1’b1: RX data is valid

dirfeedback[5:0] Input For Gen3, provides a Figure of Merit for link evaluation for H tiletransceivers. The feedback applies to the following coefficients:• dirfeedback[5:4]: Feedback applies to C+1

• dirfeedback[3:2]: Feedback applies to C0

• dirfeedback[1:0]: Feedback applies to C-1

The following feedback encodings are defined:• 2'b00: No change• 2'b01: Increment• 2'b10: Decrement• 2/b11: ReservedRefer to the figure below for a timing diagram illustrating this process.

simu_mode_pipe Input When set to 1, the PIPE interface is in simulation mode.

sim_pipe_pclk_in Input This clock is used for PIPE simulation only, and is derived from the refclk.It is the PIPE interface clock used for PIPE mode simulation.

sim_pipe_rate[1:0] Output The 2-bit encodings have the following meanings:• 2’b00: Gen1 rate (2.5 Gbps)• 2’b01: Gen2 rate (5.0 Gbps)• 2’b10: Gen3 rate (8.0 Gbps)

sim_ltssmstate[5:0] Output LTSSM state: The following encodings are defined:• 6'h00 - Detect.Quiet• 6'h01 - Detect.Active• 6'h02 - Polling.Active• 6'h03 - Polling.Compliance• 6'h04 - Polling.Configuration• 6'h05 - PreDetect.Quiet• 6'h06 - Detect.Wait• 6'h07 - Configuration.Linkwidth.Start• 6'h08 - Configuration.Linkwidth.Accept• 6'h09 - Configuration.Lanenum.Wait• 6'h0A - Configuration.Lanenum.Accept• 6'h0B - Configuration.Complete• 6'h0C - Configuration.Idle• 6'h0D - Recovery.RcvrLock

continued...




• 6'h0E - Recovery.Speed• 6'h0F - Recovery.RcvrCfg• 6'h10 - Recovery.Idle• 6'h20 - Recovery.Equalization Phase 0• 6'h21 - Recovery.Equalization Phase 1• 6'h22 - Recovery.Equalization Phase 2• 6'h23 - Recovery.Equalization Phase 3• 6'h11 - L0• 6'h12 - L0s• 6'h13 - L123.SendEIdle• 6'h14 - L1.Idle• 6'h15 - L2.Idle• 6'h16 - L2.TransmitWake• 6'h17 - Disabled.Entry• 6'h18 - Disabled.Idle• 6'h19 - Disabled• 6'h1A - Loopback.Entry• 6'h1B - Loopback.Active• 6'h1C - Loopback.Exit• 6'h1D - Loopback.Exit.Timeout• 6'h1E - HotReset.Entry• 6'h1F - Hot.Reset

sim_pipe_mask_tx_pll_lock

Input Should be active during rate change. This signal Is used to mask the PLLlock signals. This interface is used only for PIPE simulations.In serial simulations, The Endpoint PHY drives this signal. For PIPEsimulations, in the Intel testbench, The PIPE BFM drives this signal.

Related Links

PHY Interface for the PCI Express Architecture PCI Express 3.0

6.1.15 Hard IP Reconfiguration

The Hard IP reconfiguration interface is an Avalon-MM slave interface with a 21-bitaddress and an 8-bit data bus. You can use this bus to dynamically modify the valueof configuration registers that are read-only at run time.To ensure proper systemoperation, reset or repeat device enumeration of the PCI Express link after changingthe value of read-only configuration registers of the Hard IP.

Table 38. Hard IP Reconfiguration Signals


hip_reconfig_clk Input Reconfiguration clock. The frequency range for this clock is 100–125 MHz.

hip_reconfig_rst_n Input Active-low Avalon-MM reset for this interface.

hip_reconfig_address[20:0] Input The 21-bit reconfiguration address.

hip_reconfig_read Input Read signal. This interface is not pipelined. You must wait for thereturn of the hip_reconfig_readdata[7:0] from the current readbefore starting another read operation.

hip_reconfig_readdata[7:0] Output 8-bit read data. hip_reconfig_readdata[7:0] is valid on thethird cycle after the assertion of hip_reconfig_read.

continued...





hip_reconfig_readdatavalid Output When asserted, the data on hip_reconfig_readdata[7:0] isvalid.

hip_reconfig_write Input Write signal.

hip_reconfig_writedata[7:0] Input 8-bit write model.

hip_reconfig_waitrequest Output When asserted, indicates that the IP core is not ready to respond to arequest.

6.1.16 Power Management Interface

Software programs the Device into a D-state by writing to the Power ManagementControl and Status register in the PCI Power Management Capability Structure. The IPcore supports the required PCI D0 and D3 Power Management states. D1 and D2states are not supported.

Table 39. Power Management Interface Signals


pm_linkst_in_l1 Output When asserted, indicates that the link is in the L1 state.

pm_linkst_in_l0s Output When asserted, indicates that the link is in the L0s state.

pm_state[2:0] Output Specifies the current power state.

pm_dstate[2:0] Output Specifies the power management D-state for PF0.

apps_pm_xmt_pme Input Wake Up. The Application Layer asserts this signal for 1 cycle to wake upthe Power Management Capability (P MC) state machine from a D1, D2 orD3 power state. Upon wake-up, the core sends a PM_PME Message. Thisport is functionally identical to outband_pwrup_cmd. You can use thissignal or outband_pwrup_cmd to request a return from a low-powerstate to D0.

apps_ready_entr_l23 Input When asserted, the data on hip_reconfig_readdata[7:0] is valid.

apps_pm_xmt_turnoff Input Application Layer request to generate a PM_Turn_Off message. TheApplication Layer must assert this signal for one clock cycle. The IP coredoes not return an acknowledgment or grant signal. The Application Layermust not pulse this signal again until the previous message has beentransmitted.

app_init_rst Input Application Layer request for a hot reset to downstream devices.

app_xfer_pending Input When asserted, prevents the IP core from entering L1 state or initiatesexit from the L1 state.

6.1.17 Test Interface

The 256-bit test output interface is available only for x16 simulations. For x1, x2, x4,and x8 variants a 7-bit auxiliary test bus is available.




test_int[66:0] Input This is a multiplexer to selects the test_out[255:0] and aux_test_out[6:0]buses. Driven from channels 8-15.The following encodings are defined:• [66:1]: Reserved• [0]: Turns on diag_fast_link_mode to speed up simulation.

testin_zero Output When asserted, indicates accelerated initialization for simulation is active.



6.1.18 Message Handling

6.1.18.1 Endpoint Received Messages

Message Type Message Message Processing

Power Management PME_Turn_Off Forwarded to the Application Layer on Avalon-ST RX interface. Alsoprocessed by the IP core.

Slot Power Limit Set_Slot_Power_Limit Forwarded to the Application Layer on Avalon-ST RX interface.

Vendor Defined withor without Data

Vendor_Type0 Forwarded to the Application Layer on Avalon-ST RX interface. Notprocessed by core.You can program the IP core to drop these Messages using thevirtual_drop_vendor0_msg. When dropped, Vendor0 Messages arelogged as Unsupported Requests (UR).

ATS ATS_Invalidate Forwarded to the Application Layer on Avalon-ST RX interface. Notprocessed by the IP core.

Locked Transaction Unlock Message Forwarded to the Application Layer on Avalon-ST RX interface. Notprocessed by the IP core.

All Others --- Internally dropped by Endpoint and handled as an Unsupported Request.

6.1.18.2 Endpoint Transmitted Messages

The Endpoint transmits Messages that it receives from the Application Layer on theAvalon-ST TX interface and internally generated messages. The Endpoint maygenerate Messages autonomously or in response to a request received on the :app_pm_xmt_pme, app_int, or app_err_* input ports.


Power Management PM_PME, PME_TO_Ack The Application Layer transmits the PM_PME request via theapp_pm_xmt_pme input.The Application Layer must generate the PME_TO_Ack and transmit it onthe Avalon-ST TX interface.The Application Layer transmits the app_ready_entr_l23 to indicate that itis ready to enter the L23 state.


Vendor_Type0 The Application Layer must generate and transmit this on the Avalon-STTX interface.

ATS ATS_Request,ATS_InvalidateCompletion

The Application Layer must generate and transmit these Messages on theAvalon-ST TX interface.

INT INTx_Assert,INTx_Deassert

The Application Layer transmits the INTx_Assert and INTx_DeassertMessages using the app_int interface.

ERR ERR_COR,ERR_NONFATAL,ERR_FATAL

The IP core transmits these error Messages autonomously when it detectsinternal errors. It also receives and forwards these errors when receivedfrom the Application Layer via the app_err_* interface.

6.1.18.3 Root Port Received Messages

The Root Port forwards Messages received to the Application Layer on the Avalon-STRX interface. It also reflects legacy interrupts on the dedicated INTx interface anderror messages on the app_err_* interface.



Note: Root Ports are not supported in the Quartus Prime Pro – Stratix 10 Edition 17.1Interim Release.


PowerManagement

PM_PME,PME_TO_Ack

Forwarded to the Application Layer on the Avalon-RX interface. Alsoprocessed by the IP core.

Vendor Definedwith or withoutData

Vendor_Type0 Forwarded to the Application Layer on the Avalon-RX interface. Notprocessed by the IP core.You can program the IP core to drop these Messages using thevirtual_drop_vendor0_msg. When dropped, Vendor0 Messages arelogged as Unsupported Requests (UR).

ATS ATS_Request,ATS_InvalidateCompletion

Forwarded to the Application Layer on the Avalon-RX interface. Notprocessed by the IP core

INT INTx_Assert,INTx_Deassert

Processed by the IP core. The IP core transmits these Messages on theint_status_* interface. Forwarded to the Application Layer on the Avalon-RX interface.

ERR ERR_CORR,ERR_NONFATAL,ERR_FATAL

Processed by the IP core. The IP core transmits these Messages on theapp_err_* interface. Forwarded to the Application Layer on the Avalon-RXinterface.

All Others N/A The Root Port drops this messages and handles them as UnsupportedRequests (UR).

6.1.18.4 Root Port Transmitted Messages

The Root Port forwards Messages received from the Application Layer on its Avalon-STTX interface. It also transmits also generates Messages autonomously or in responseto Messages received on a dedicated Power Management interface.

Note: Root Ports are not supported in the Quartus Prime Pro – Stratix 10 Edition 17.1Interim Release.


Power Management PME_Turn_Off Application requests this Message via the app_pm_xmt_turnoff input.

Slot Power Limit Set_Slot_Power_Limit The IP core transmits this Message whenever the Slot Capabilities registeris written or when the LTSSM enters L0.


Vendor_Type0 The Root Port forwards Messages received from the Application Layer. TheApplication must assemble the TLP and transmit it via the Avalon-ST TXinterface.

ATS ATS_Invalidate The Root Port forwards Messages received from the Application Layer. TheApplication must assemble the TLP and transmit it via the Avalon-ST TXinterface.

Locked Transaction Unlock Message The Root Port forwards Messages received from the Application Layer. TheApplication must assemble the TLP and transmit it via the Avalon-ST TXinterface.

6.2 Errors reported by the Application Layer

The Application Layer uses the app_err_* interface to report errors to the IP core.



The Application Layer reports the following types of errors to the IP core:

• Unexpected Completion

• Completer Abort

• CPL Timeout

• Unsupported Request

• Poisoned TLP received

• Uncorrected Internal Error, including ECC and parity errors flagged by the core

• Corrected Internal Error, including Corrected ECC errors flagged by the core

• Advisory NonFatal Error

For Advanced Error Reporting (AER), the Application Layer provide the information tolog the TLP header and the error log request via the app_err_* interface.

The Application Layer completes the following steps to report an error to the IP core:

• Sets the corresponding status bits in the PCI Status register, and the PCIeDevice Status register

• Sets the appropriate status bits and header log in the AER registers if AER isenabled

• Indicates the Error event to the upstream component:

— Endpoints transmit an Message upstream

— Root Ports assert app_serr_out to the Application Layer if an error isdetected or if an error Message is received from a downstream component.The Root Port also forwards the error Message from the downstreamcomponent on the Avalon-ST RX interface. The Application Layer may chooseto ignore this information. (Root Ports are not supported in the Quartus PrimePro – Stratix 10 Edition 17.1 Interim Release.)

6.2.1 Error Handling

When the IP core detects an error in a received TLP, it generates a Completion. It setsthe Completion status set to Completer Abort (CA) or Unsupported Request (UR).



The IP Core completes the following actions when it detects an error in a received TLP:

• Discards the TLP.

• Generates a Completion (for non-posted requests) with the Completion status setto CA or UR.

• Sets the corresponding status bits in the PCI Status register and the PCIeDevice Status register.

• Sets the corresponding status bits and header log in the AER registers if AER isenabled.

• Indicates the Error event to the upstream component.

— For Endpoints, the IP core sends an error Message upstream.

— For Root Ports, the IP core asserts app_serr_out asserts to the ApplicationLayer) when it detects an error or receives an error Message from adownstream component.

Note: The error Message from the downstream component is also forwardedon the Avalon-ST RX interface. The Application Layer may choose toignore this information.

6.3 Power Management

The Stratix 10 Hard IP for PCI Express IP core supports the required PCI D0 and D3Power Management states. It does not support the optional D1 and D2 PowerManagement states.

Software programs the Device into a D-state by writing to the Power ManagementControl and Status register in the PCI Power Management Capability Structure.The pm_* interface transmits the D-state to the Application Layer.

6.3.1 Endpoint D3 Entry

This topic outlines the D3 power-down procedure.

All transmission on the Avalon-ST TX and RX interfaces must have completed beforeIP core can begin the L1 request (Enter_L1 DLLP). In addition, the RX Buffer must beempty and the Application Layer app_xfer_pending output must be deasserted.

1. Software writes the Power Management Control register to put the IP core to theD3hot state.

2. The Endpoint stops transmitting requests when it has been taken out of D0.

3. The link transitions to L1.

4. Software sends the PME_Turn_Off Message to the Endpoint to initiate power down.The Root Port transitions the link back to L0, and Endpoint receives the Messageon the Avalon-ST RX interface.

5. The End Point transmits a PME_TO_Ack Message to acknowledge the Turn Offrequest.

6. When ready for power removal, (D3cold), the End Point assertsapp_ready_entr_l23. The core sends the PM_Enter_L23 DLLP and initiates theLink transition to L3.



6.3.2 End Point D3 Exit

An End Point can exit the D3 state if the following two conditions are true: First, thePME_Support setting in the Power Management Capabilities (PMC) registermust enable PME notification. Second, software must set the PME_en bit in the PowerManagement Control and Status (PMCSR) register.

6.3.3 Exit from D3 hot

The Power Management Capability register must enable D3Hot PME_Support.In addition, software must set the PME_en bit in the Power Management Controland Status register.

6.3.4 Exit from D3 cold

1. To issue a PM_EVENT Message from the D3 cold state, the device must first issuea wakeup event (WAKE#) to request reapplication of power and the clock.The wakeup event triggers a fundamental reset which reinitializes the link to L0.

2. The Application Layer requests a wake-up event by asserting apps_pm_xmt_pme.Asserting apps_pm_xmt_pme causes the IP core to transmit a PM_EVENTMessage. In addition, the IP core sets the PME_status bit in the PowerManagement Control and Status register to notify software that it hasrequested the wakeup.

The PCIe Link states are indicated on the pm_* interface. The LTSSM state isindicated on the ltssm_state output.

6.3.5 Active State Power Management

Active State Power Management (ASPM) is not supported as indicated by the ASPMSupport bits in the Link Capabilities register.

6.4 Transaction Ordering

6.4.1 TX TLP Ordering

TLPs on the Avalon-ST TX interface are transmitted in the order in which theApplication Layer presents them. The IP core provides TX credit information to theApplication Layer so that the Application Layer can perform credits-based reorderingbefore submitting TLPs for transmission.

This reordering is optional. The IP core always checks for sufficient TX credits beforetransmitting any TLP. Ordering is not guaranteed between the following TLPtransmission interfaces:

• Avalon

• MSI and MSI-X interrupt

• Internal Configuration Space TLPs



6.4.2 RX TLP Ordering

TLPs ordering on the Avalon-ST RX interface is determined by the available credits foreach TLP type and the PCIe ordering rules. Refer to Table 2-34 Ordering RulesSummary in the PCI Express Base Specification Revision 3.0 for a summary of theordering rules.

The IP core implements relaxed ordering as described in the PCI Express BaseSpecification Revision 3.0. It does not perform ID-Based Ordering (IDO). TheApplication Layer can implement IDO reordering. It is possible for two different TLPtypes pending in the RX buffer to have equal priority. When this situation occurs, theIP core uses a fairness-based arbitration scheme to determine which TLP to forward tothe Application Layer.

Related Links


6.5 RX Buffer

The Receive Buffer stores TLPs received from the PCI Express link. The RX Bufferstores the entire TLP before it forwards it to the Application Layer.Storing the entire TLP allows the IP to accomplish two things:

• The IP core can rate match the PCIe link to the Application Layer.

• The IP core can store TLPs until error checking is complete.

The 64 KB RX buffer has separate buffer space for Posted, Non-Posted and CompletionTLPS. Headers and data also have separate allocations. The RX Buffer enables fullbandwidth RX traffic for all three types of TLPs simultaneously.

Table 40. Flow Control Credit AllocationThe buffer allocation is fixed.

RX Buffer Segment Number of Credits Buffer Size

Posted Posted headers: 127 creditsPosted data: 750 credits

~14 KB

Non-posted Non-posted headers credits: 115 creditsNon-posted data credits: 230 credits

~5.5 KB

Completions Completion headers: 770 creditsCompletion data: 2500 credits

~50 KB

The RX buffer operates only in the Store and Forward Queue Mode. Bypass and Cut-through modes are not supported.

Flow control credit checking for the posted and non-posted buffer segments preventsRX buffer overflow. The PCI Express Base Specification Revision 3.0 requires the IPcore to advertise infinite Completion credits. The Application Layer must manage itsRead Requests so as not to overflow the Completion buffer segment.

Related Links






6.5.1 Retry Buffer

The retry buffer stores a copy of a transmitted TLP until the remote deviceacknowledges the TLP using the ACK mechanism. In the case of an LCRC error on thereceiving side, the remote device sends NAK DLLP to the transmitting side. Thetransmitting side retrieves the TLP from the retry buffer and resends it.

Retry buffer resources are only freed upon reception of an ACK DLLP.

6.5.2 Configuration Retry Status

The Stratix 10 Hard IP for PCI Express is part of the periphery image of the device.After power-up, the periphery image is loaded first. The End Point can then respond toConfiguration Requests with Config Retry Status (CRS) to delay the enumerationprocess until after the FPGA fabric is configured.



7 Interrupts

7.1 Interrupts for Endpoints

The Stratix 10 Hard IP for PCI Express provides support for PCI Express MSI, MSI-X,and legacy interrupts when configured in Endpoint mode. The MSI, MSI-X, and legacyinterrupts are mutually exclusive. After power up, the Hard IP block starts in legacyinterrupt mode. Then, software decides whether to switch to MSI or MSI-X mode. Toswitch to MSI mode, software programs the msi_enable bit of the MSI MessageControl Register to 1, (bit[16] of 0x050). You enable MSI-X mode, by turning onImplement MSI-X under the PCI Express/PCI Capabilities tab using theparameter editor. If you turn on the Implement MSI-X option, you should implementthe MSI-X table structures at the memory space pointed to by the BARs.

Note:

Refer to section 6.1 of PCI Express Base Specification for a general description of PCIExpress interrupt support for Endpoints.

Related Links


7.1.1 MSI Interrupts

The IP core generates single dword Memory Write TLPs to signal MSI interrupts on thePCI Express link. The Application Layer MSI Handler Module app_msi_req output portcontrols MSI interrupt generation. When asserted, it causes an MSI posted MemoryWrite TLP to be generated. The IP core constructs the TLP using information from thefollowing sources:

• The MSI Capability registers

• The traffic class (app_msi_tc)

• The message data specified by app_msi_num

To enable MSI interrupts, the Application Layer must first set the MSI enable bit.Then, it must disable legacy interrupts by setting the Interrupt Disable, bit 10 ofthe Command register.

7 Interrupts



http://www.pcisig.com/home




Figure 55. MSI Handler Module

app_msi_reqapp_msi_tc[2:0]app_msi_num[4:0]app_int_sts

app_msi_ack

MSI HandlerModule

The following figure illustrates a possible implementation of the MSI handler modulewith a per vector enable bit. Alternatively, the Application Layer could implement aglobal interrupt enable instead of this per vector MSI.

Figure 56. Example Implementation of the MSI Handler Block

app_int_en0

app_int_sts0

app_msi_req0

app_int_en1

app_int_sts1

app_msi_req1

app_int_sts

MSIArbitration

msi_enable & Master Enable

app_msi_reqapp_msi_ack

Vector 1

Vector 0

IRQGeneration

App Layer

IRQGeneration

App Layer

R/W

R/W

There are 32 possible MSI messages. The number of messages requested by aparticular component does not necessarily correspond to the number of messagesallocated. For example, in the following figure, the Endpoint requests eight MSIs but isonly allocated two. In this case, you must design the Application Layer to use only twoallocated messages.

7 Interrupts


Figure 57. MSI Request Example

Endpoint

8 Requested2 Allocated

Root Complex

CPU

Interrupt Register

RootPort

InterruptBlock

The following table describes three example implementations. The first exampleallocates all 32 MSI messages. The second and third examples only allocate 4interrupts.

Table 41. MSI Messages Requested, Allocated, and Mapped

MSI Allocated

32 4 4

System Error 31 3 3

Hot Plug and Power Management Event 30 2 3

Application Layer 29:0 1:0 2:0

MSI interrupts generated for Hot Plug, Power Management Events, and System Errorsalways use Traffic Class 0. MSI interrupts generated by the Application Layer can useany Traffic Class. For example, a DMA that generates an MSI at the end of atransmission can use the same traffic control as was used to transfer data.

The following figure illustrates the interactions among MSI interrupt signals for theRoot Port. The minimum latency possible between app_msi_req and app_msi_ackis one clock cycle. In this timing diagram app_msi_req can extend beyondapp_msi_ack before deasserting. However, app_msi_req must be deasserted beforeor within the same clock as app_msi_ack is deasserted to avoid inferring a newinterrupt.

Figure 58. MSI Interrupt Signals Timing

clk

app_msi_req

app_msi_tc[2:0]

app_msi_num[4:0]

app_msi_ack

1 2 3 5 64 7

valid

valid

7 Interrupts


7.1.2 MSI-X

You can enable MSI-X interrupts by turning on Implement MSI-X under the PCIExpress/PCI Capabilities heading using the parameter editor. If you turn on theImplement MSI-X option, you should implement the MSI-X table structures at thememory space pointed to by the BARs as part of your Application Layer.

The Application Layer transmits MSI-X interrupts on the Avalon-ST TX interface. MSI-Xinterrupts are single dword Memory Write TLPs. Consequently, the Last DW ByteEnable in the TLP header must be set to 4b’0000. MSI-X TLPs should be sent onlywhen enabled by the MSI-X enable and the function mask bits in the MessageControl for the MSI-X Configuration register. These bits are available on thetl_cfg_ctl output bus.

Related Links

• PCI Local Bus Specification


7.1.3 Implementing MSI-X Interrupts

Section 6.8.2 of the PCI Local Bus Specification describes the MSI-X capability andtable structures. The MSI-X capability structure points to the MSI-X Table structureand MSI-X Pending Bit Array (PBA) registers. The BIOS sets up the starting addressoffsets and BAR associated with the pointer to the starting address of the MSI-X Tableand PBA registers.

Figure 59. MSI-X Interrupt Components

Host

RX

TX

RX

TX

PCIe with Avalon-ST I/F

MSI-X Table

IRQProcessor

MSI-X PBA IRQ Source

Application LayerHost SW Programs Addr,Data and Vector Control

Memory Write TLP

Memory Write TLP Monitor & Clr

Addr, Data

1. Host software sets up the MSI-X interrupts in the Application Layer by completingthe following steps:

a. Host software reads the Message Control register at 0x050 register todetermine the MSI-X Table size. The number of table entries is the <valueread> + 1.

The maximum table size is 2048 entries. Each 16-byte entry is divided in 4fields as shown in the figure below. The MSI-X table can reside in any BAR.The base address of the MSI-X table must be aligned to a 4 KB boundary.

b. The host sets up the MSI-X table. It programs MSI-X address, data, andmasks bits for each entry as shown in the figure below.

7 Interrupts




Figure 60. Format of MSI-X Table

Vector ControlVector ControlVector Control

Vector Control

Message DataMessage DataMessage Data

Message Data

DWORD 3 DWORD 2Message Upper AddressMessage Upper AddressMessage Upper Address

Message Upper Address

DWORD 1Message AddressMessage AddressMessage Address

Message Address

DWORD 0 Host Byte AddressesEntry 0Entry 1Entry 2

Entry (N - 1)

BaseBase + 1 × 16Base + 2 × 16

Base + (N - 1) × 16

c. The host calculates the address of the <nth> entry using the followingformula:

nth_address = base address[BAR] + 16<n>

2. When Application Layer has an interrupt, it drives an interrupt request to the IRQSource module.

3. The IRQ Source sets appropriate bit in the MSI-X PBA table.

The PBA can use qword or dword accesses. For qword accesses, the IRQ Sourcecalculates the address of the <mth> bit using the following formulas:

qword address = <PBA base addr> + 8(floor(<m>/64))qword bit = <m> mod 64

Figure 61. MSI-X PBA Table

Pending Bits 0 through 63Pending Bits 64 through 127

Pending Bits (( N - 1) div 64) × 64 through N - 1

QWORD 0QWORD 1

QWORD (( N - 1) div 64)

BaseAddressPending Bit Array (PBA)

Base + 1 × 8

Base + ((N - 1) div 64) × 8

4. The IRQ Processor reads the entry in the MSI-X table.

a. If the interrupt is masked by the Vector_Control field of the MSI-X table,the interrupt remains in the pending state.

b. If the interrupt is not masked, IRQ Processor sends Memory Write Request tothe TX slave interface. It uses the address and data from the MSI-X table. IfMessage Upper Address = 0, the IRQ Processor creates a three-dwordheader. If the Message Upper Address > 0, it creates a 4-dword header.

5. The host interrupt service routine detects the TLP as an interrupt and services it.

Related Links

• Floor and ceiling functions

• PCI Local Bus Specification, Rev. 3.0

7 Interrupts


http://en.wikipedia.org/wiki/Floor_and_ceiling_functions


7.1.4 Legacy Interrupts

The following figure illustrates interrupt timing for the legacy interface. In this figurethe assertion of app_int_sts instructs the Hard IP for PCI Express to send aAssert_INTx message TLP.

Figure 62. Legacy Interrupt Assertion

clk

app_int_sts

app_int_ack

The following figure illustrates the timing for deassertion of legacy interrupts. Theassertion of app_int_sts instructs the Hard IP for PCI Express to send aDeassert_INTx message.

Figure 63. Legacy Interrupt Deassertion

clk

app_int_sts

app_int_ack

7 Interrupts


8 Registers

8.1 Configuration Space Registers

Table 42. Correspondence between Configuration Space Capability Structures and PCIeBase Specification Description

Byte Address Hard IP Configuration Space Register Corresponding Section in PCIe Specification

0x000:0x03C PCI Header Type 0 Configuration Registers Type 0 Configuration Space Header

0x040:0x04C Power Management PCI Power Management Capability Structure

0x050:0x05C MSI Capability Structure MSI Capability Structure, see also and PCI LocalBus Specification

0x060:0x06C Reserved N/A

0x070:0x0A8 PCI Express Capability Structure PCI Express Capability Structure

0x0B0:0x0B8 MSI-X Capability Structure MSI-X Capability Structure, see also and PCILocal Bus Specification

0x0BC:0x0FC Reserved N/A

0x100:0x134 Advanced Error Reporting (AER) Advanced Error Reporting Capability

0x138:188 Reserved N/A

0x188:0x1B4 Secondary PCI Express Extended CapabilityHeader

PCI Express Extended Capability

0x188:0xB7C Reserved N/A

0xB80:0xBAC Intel-Defined VSEC Capability Header Vendor-Specific Header (Header only)

0xC00 Optional Custom Extensions N/A

Summary of Configuration Space Register Fields

0x000 Device ID, Vendor ID Type 0 Configuration Space Header

0x004 Status, Command Type 0 Configuration Space Header

0x008 Class Code, Revision ID Type 0 Configuration Space Header

0x00C Header Type, Cache Line Size Type 0 Configuration Space Header

0x010 Base Address 0 Base Address Registers



0x01C Base Address 3 Base Address Registers

continued...

8 Registers









0x028 Reserved N/A

0x02C Subsystem ID, Subsystem Vendor ID Type 0 Configuration Space Header

0x030 Reserved N/A

0x034 Capabilities Pointer Type 0 Configuration Space Header

0x038 Reserved N/A

0x03C Interrupt Pin, Interrupt Line Type 0 Configuration Space Header

0x040 PME_Support, D1, D2, etc. PCI Power Management Capability Structure

0x044 PME_en, PME_Status, etc. Power Management Status and Control Register

0x050 MSI-Message Control, Next Cap Ptr, Capability ID MSI and MSI-X Capability Structures

0x054 Message Address MSI and MSI-X Capability Structures

0x058 Message Upper Address MSI and MSI-X Capability Structures

0x05C Reserved Message Data MSI and MSI-X Capability Structures

0x0B0 MSI-X Message Control Next Cap Ptr CapabilityID

MSI and MSI-X Capability Structures

0x0B4 MSI-X Table Offset BIR MSI and MSI-X Capability Structures

0x0B8 Pending Bit Array (PBA) Offset BIR MSI and MSI-X Capability Structures

0x100 PCI Express Enhanced Capability Header Advanced Error Reporting Enhanced CapabilityHeader

0x104 Uncorrectable Error Status Register Uncorrectable Error Status Register

0x108 Uncorrectable Error Mask Register Uncorrectable Error Mask Register

0x10C Uncorrectable Error Mask Register Uncorrectable Error Severity Register

0x110 Correctable Error Status Register Correctable Error Status Register

0x114 Correctable Error Mask Register Correctable Error Mask Register

0x118 Advanced Error Capabilities and Control Register Advanced Error Capabilities and Control Register

0x11C Header Log Register Header Log Register

0x12C Root Error Command Root Error Command Register

0x130 Root Error Status Root Error Status Register

0x134 Error Source Identification Register CorrectableError Source ID Register

Error Source Identification Register

0x188 Next Capability Offset, PCI Express ExtendedCapability ID

Secondary PCI Express Extended Capability

0x18C Enable SKP OS, Link Equalization Req, PerformEqualization

Link Control 3 Register

0x190 Lane Error Status Register Lane Error Status Register

0x194 Lane Equalization Control Register Lane Equalization Control Register

0xB80 VSEC Capability Header Vendor-Specific Extended Capability Header

continued...

8 Registers



0xB84 VSEC Length, Revision, ID Vendor-Specific Header

0xB88 Intel Marker

Intel-Specific Registers

0xB8C JTAG Silicon ID DW0

0xB90 JTAG Silicon ID DW1



0xB9C User Device and Board Type ID

0xBA0::BAC Reserved

0xBB0 General Purpose Control and Status Register

0xBB4 Uncorrectable Internal Error Status Register

0xBB8 Uncorrectable Internal Error Mask Register

0xBBC Correctable Error Status Register

0xBC0 Correctable Error Mask Register

0xBC4:BD8 Reserved N/A

0xC00 Optional Custom Extensions N/A

Related Links



8.1.1 Type 0 Configuration Space Registers

Endpoints store configuration data in the Type 0 Configuration Space. TheCorrespondence between Configuration Space Registers and the PCIe Specificationlists the appropriate section of the PCI Express Base Specification that describes theseregisters.

Figure 64. Configuration Space Registers Address Map

PCI Header Type 0PCI Power Management

MSIPCI Express

MSI-XAER

VSECCustom Extensions

RequiredRequiredOptionalRequiredOptionalRequired

RequiredOptional

0x000x400x500x700xB0

0x100

0xB800xC00

End Point Capability Structure Required/Optional Starting Byte Offset

Secondary PCIe Required 0x188

8 Registers




Figure 65. Type 0 Configuration Space Registers - Byte Address Offsets and Layout

0x0000x0040x0080x00C0x0100x0140x0180x01C0x0200x0240x0280x02C0x0300x0340x0380x03C

Device ID Vendor IDStatus Command

Class Code Revision ID

0x00 Header Type 0x00 Cache Line SizeBAR RegistersBAR RegistersBAR RegistersBAR RegistersBAR RegistersBAR Registers

ReservedSubsystem Device ID Subsystem Vendor ID

ReservedReserved

Reserved

Capabilities Pointer

0x00 Interrupt Pin Interrupt Line

31 24 23 16 15 8 7 0

Related Links


8.1.2 PCI Express Capability Structures

The layout of the most basic Capability Structures are provided below. Refer to the PCIExpress Base Specification for more information about these registers.

Figure 66. Power Management Capability Structure - Byte Address Offsets and Layout

0x040

0x04C

Capabilities Register Next Cap Ptr

Data

31 24 23 16 15 8 7 0Capability ID

Power Management Status and ControlPM Control/StatusBridge Extensions

8 Registers



Figure 67. MSI Capability Structure

0x050

0x0540x058

Message ControlConfiguration MSI Control Status

Register Field DescriptionsNext Cap Ptr

Message AddressMessage Upper Address

Reserved Message Data

31 24 23 16 15 8 7 0

0x05C

Capability ID

Figure 68. PCI Express Capability Structure - Byte Address Offsets and LayoutIn the following table showing the PCI Express Capability Structure, registers that arenot applicable to a device are reserved.

0x070

0x0740x0780x07C0x0800x0840x0880x08C0x0900x0940x0980x09C0x0A00x0A4

0x0A8

PCI Express Capabilities Register Next Cap Pointer

Device CapabilitiesDevice Status Device Control

Slot Capabilities

Root StatusDevice Compatibilities 2

Link Capabilities 2Link Status 2 Link Control 2

Slot Capabilities 2

Slot Status 2 Slot Control 2

31 24 23 16 15 8 7 0PCI Express

Capabilities ID

Link CapabilitiesLink Status Link Control

Slot Status Slot Control

Device Status 2 Device Control 2

Root Capabilities Root Control

8 Registers


Figure 69. MSI-X Capability Structure

0x0B0

0x0B4

0x0B8

Message Control Next Cap Ptr

MSI-X Table Offset

MSI-X Pending Bit Array (PBA) Offset

31 24 23 16 15 8 7 0Capability ID

3 2

MSI-XTable BAR

IndicatorMSI-X

PendingBit Array

- BARIndicator

Figure 70. PCI Express AER Extended Capability Structure

Byte Offs et 31:24 23:16 15:8 7:0

0x100

0x104 Uncorrectable Error Status Register

PCI Express Enhanced Capability Register

Uncorrectable Error Severity Register

Uncorrectable Error Mask Register0x108

0x10C

0x110

0x114

0x118

0x11C

0x12C

0x130

0x134

Correctable Error Status Register

Correctable Error Mask Register

Advanced Error Capabilities and Control Register

Header Log Register

Root Error Command Register

Root Error Status Register

Error Source Identification Register Correctable Error Source Identification Register

Related Links



8.1.3 JTAG Silicon ID

This read only register returns the JTAG Silicon ID. The Intel Programming softwareuses this JTAG ID to make ensure that it is programming the SRAM Object File(*.sof) .

Table 43. JTAG Silicon ID - 0xB8C-0xB98

Bits Register Description Default Value3

Access

[127:96 JTAG Silicon ID DW3 Unique ID RO

[95:64] JTAG Silicon ID DW2 Unique ID RO



3 Because the Silicon ID is a unique value, it does not have a global default value.

8 Registers




8.1.4 Intel Defined VSEC Capability Header

The figure below shows the address map and layout of the Intel defined VSECCapability.

Figure 71. Vendor-Specific Extended Capability Address Map and Register Layout

31 24 23 16 15 8 7 0Next Cap Offset

VSEC LengthVersion

VSEC RevisionPCI Express Extended Capability ID

VSEC ID00h04h08h0Ch10h14h18h1Ch20h24h28h2Ch30h34h38h3Ch40h44h48h4Ch50h54h58h

Intel MarkerJTAG Silicon ID DW0JTAG Silicon ID DW1JTAG Silicon ID DW2JTAG Silicon ID DW3

Reserved User Configurable Device/Board IDReservedReservedReservedReserved

General-Purpose Control and StatusUncorrectable Internal Error Status RegisterUncorrectable Internal Error Mask RegisterCorrectable Internal Error Status RegisterCorrectable Internal Error Mask Register

ReservedReservedReservedReservedReservedReserved

Table 44. Altera-Defined VSEC Capability Header - 0xB80

Bits Register Description Default Value Access

[31:20] Next Capability Pointer: Value is the starting address of the next CapabilityStructure implemented. Otherwise, NULL.

Variable RO

[19:16] Version. PCIe specification defined value for VSEC version. 1 RO

[15:0] PCI Express Extended Capability ID. PCIe specification defined value forVSEC Capability ID.

0x000B RO

8 Registers


8.1.5 Intel Defined Vendor Specific Header

Table 45. Intel defined Vendor Specific Header - 0xB84


[31:20] VSEC Length. Total length of this structure in bytes. 0x5C RO

[19:16] VSEC. User configurable VSEC revision. Not available RO

[15:0] VSEC ID. User configurable VSEC ID. You should change this ID to yourVendor ID.

0x1172 RO

8.1.6 Intel Marker

Table 46. Intel Marker - 0xB88


[31:0] Intel Marker - An additional marker for standard Intel programmingsoftware to be able to verify that this is the right structure.

0x41721172 RO

8.1.7 User Device and Board Type ID

Table 47. User Device and Board Type ID - 0xB9C


[15:0] Allows you to specify ID of the .sof file to be loaded.From

configurationbits

RO

8.1.8 General Purpose Control and Status Register

This register provides up to eight I/O pins for Application Layer control and statusrequirements. This feature supports Partial Reconfiguration of the FPGA fabric. PartialReconfiguration only requires one input and one output pin. The other seven I/Osmake this interface extensible.

Table 48. General Purpose Control and Status Register - 0xBB0


[31:16] Reserved. N/A RO

[15:8] General Purpose Status. The Application Layer can read status bits. 0 RO

[7:0] General Purpose Control. The Application Layer can write control bits. 0 RW

8.1.9 Uncorrectable Internal Error Status Register

This register reports the status of the internally checked errors that areuncorrectable.When these specific errors are enabled by the UncorrectableInternal Error Mask register, they are forwarded as Correctable Internal Errors.This register is for debug only. Only use this register to observe behavior, not to drivelogic custom logic.

8 Registers


Table 49. Uncorrectable Internal Error Status Register - 0xBB4This register is for debug only. It should only be used to observe behavior, not to drive custom logic. Theaccess code RW1CS represents Read Write 1 to Clear Sticky.

Bits Register Description ResetValue

Access

[31:13] Reserved. 0 RO

[12] Debug bus interface (DBI) access error status. 0 RW1CS

[11] ECC error from Config RAM block. 0 RW1CS

[10] Uncorrectable ECC error status for Retry Buffer. 0 RO

[9] Uncorrectable ECC error status for Retry Start of the TLP RAM. 0 RW1CS

[8] RX Transaction Layer parity error reported by the IP core. 0 RW1CS

[7] TX Transaction Layer parity error reported by the IP core. 0 RW1CS

[6] Internal error reported by the FPGA. 0 RW1CS

[5:4] Reserved. 0 RW1CS

[3] Uncorrectable ECC error status for RX Buffer Header #2 RAM. 0 RW1CS

[2] Uncorrectable ECC error status for RX Buffer Header #1 RAM. 0 RW1CS

[1] Uncorrectable ECC error status for RX Buffer Data RAM #2. 0 RW1CS

[0] Uncorrectable ECC error status for RX Buffer Data RAM #1. 0 RW1CS

8.1.10 Uncorrectable Internal Error Mask Register

The Uncorrectable Internal Error Mask register controls which errors areforwarded as internal uncorrectable errors.

Table 50. Uncorrectable Internal Error Mask Register - 0xBB8The access code RWS stands for Read Write Sticky meaning the value is retained after a soft reset of the IPcore.

Bits Register Description Reset Value Access

[31:13] Reserved. 1b’0 RO

[12] Mask for Debug Bus Interface. 1b'1 RO

[11] Mask for ECC error from Config RAM block. 1b’1 RWS

[10] Mask for Uncorrectable ECC error status for Retry Buffer. 1b’1 RO

[9] Mask for Uncorrectable ECC error status for Retry Start of TLP RAM. 1b’1 RWS

[8] Mask for RX Transaction Layer parity error reported by IP core. 1b’1 RWS

[7] Mask for TX Transaction Layer parity error reported by IP core. 1b’1 RWS

[6] Mask for Uncorrectable Internal error reported by the FPGA. 1b’1 RO

[5] Reserved. 1b’0 RWS

[4] Reserved. 1b’1 RWS

[3] Mask for Uncorrectable ECC error status for RX Buffer Header #2 RAM. 1b’1 RWS

continued...

8 Registers


Bits Register Description Reset Value Access

[2] Mask for Uncorrectable ECC error status for RX Buffer Header #1 RAM. 1b’1 RWS

[1] Mask for Uncorrectable ECC error status for RX Buffer Data RAM #2. 1b’1 RWS

[0] Mask for Uncorrectable ECC error status for RX Buffer Data RAM #1. 1b’1 RWS

8.1.11 Correctable Internal Error Status Register

Table 51. Correctable Internal Error Status Register - 0xBBCThe Correctable Internal Error Status register reports the status of the internally checked errors thatare correctable. When these specific errors are enabled by the Correctable Internal Error Maskregister, they are forwarded as Correctable Internal Errors. This register is for debug only. Only use thisregister to observe behavior, not to drive logic custom logic.


Access


[11] Correctable ECC error status for Config RAM. 0 RW1CS

[10] Correctable ECC error status for Retry Buffer. 0 RW1CS

[9] Correctable ECC error status for Retry Start of TLP RAM. 0 RW1CS

[8] Reserved. 0 RO

[7] Reserved. 0 RO

[6] Internal Error reported by FPGA. 0 RW1CS

[5] Reserved 0 RO

[4] PHY Gen3 SKP Error occurred. Gen3 data pattern contains SKP pattern(8'b10101010) is misinterpreted as a SKP OS and causing erroneousblock realignment in the PHY.

0 RW1CS

[3] Correctable ECC error status for RX Buffer Header RAM #2. 0 RW1CS

[2] Correctable ECC error status for RX Buffer Header RAM #1. 0 RW1CS

[1] Correctable ECC error status for RX Buffer Data RAM #2. 0 RW1CS

[0] Correctable ECC error status for RX Buffer Data RAM #1. 0 RW1CS

8.1.12 Correctable Internal Error Mask Register

Table 52. Correctable Internal Error Status Register - 0xBBCThe Correctable Internal Error Status register controls which errors are forwarded as InternalCorrectable Errors.


Access


[11] Mask for correctable ECC error status for Config RAM. 0 RWS

[10] Mask for correctable ECC error status for Retry Buffer. 1 RWS

[9] Mask for correctable ECC error status for Retry Start of TLP RAM. 1 RWS

continued...

8 Registers



Access

[8] Reserved. 0 RO

[7] Reserved. 0 RO

[6] Mask for internal Error reported by FPGA. 0 RWS

[5] Reserved 0 RO

[4] Mask for PHY Gen3 SKP Error. 1 RWS

[3] Mask for correctable ECC error status for RX Buffer Header RAM #2. 1 RWS

[2] Mask for correctable ECC error status for RX Buffer Header RAM #1. 1 RWS

[1] Mask for correctable ECC error status for RX Buffer Data RAM #. 1 RWS

[0] Mask for correctable ECC error status for RX Buffer Data RAM #1. 1 RWS

8.2 Avalon-MM DMA Bridge Registers

8.2.1 PCI Express Avalon-MM Bridge Register Address Map

The registers included in the Avalon-MM bridge map to a 32 KB address space. Readsto undefined addresses have unpredictable results.

Table 53. Stratix 10 PCIe Avalon-MM Bridge Register Map

Address Range Registers

0x0040 PCIe Interrupt Status Register

0x0050 PCIe Interrupt Enable Register

0x0800-0x081F PCIe to Avalon Mailbox Registers

0x0900-0x091F Avalon to PCIe Mailbox Registers

0x1000-0x1FFF Address Translation Table for the Bursting Avalon-MM Slave

0x3060 Avalon-MM Interrupt Status Register

0x3070 Avalon Interrupt Enable Register

0x3A00–0x3A1F Avalon-MM to PCIe Mailbox Registers

0x3B00–0x3B1F PCIe to Avalon-MM Mailbox Registers

0x3C00-0x3C1F PCIe Configuration Information Registers

8.2.1.1 Avalon-MM to PCI Express Interrupt Status Registers

These registers contain the status of various signals in the PCI Express Avalon-MMbridge logic. These registers allow MSI or legacy interrupts to be asserted whenenabled.

Only Root Complexes should access these registers; however, hardware does notprevent other Avalon-MM masters from accessing them.

8 Registers


Table 54. Avalon-MM to PCI Express Interrupt Status Register, 0x0040

Bit Name Access Description

[31:24] Reserved N/A N/A

[23] A2P_MAILBOX_INT7 RW1C 1 when the A2P_MAILBOX7 is written to








[15:0] User Interrupts RW1C If the Application implements interrupts usingrxm_irq[15:0], this register records interruptstatus. Set to 1'b1 on the rising edge of thecorresponding bit of rxm_irq[15:0]. .

8.2.1.2 Avalon-MM to PCI Express Interrupt Enable Registers

The interrupt enable registers enable either MSI or legacy interrupts.

A PCI Express interrupt can be asserted for any of the conditions registered in theAvalon-MM to PCI Express Interrupt Status register by setting thecorresponding bits in the Avalon-MM to PCI Express Interrupt Enable register.

Table 55. Avalon-MM to PCI Express Interrupt Enable Register, 0x0050

Bits Name Access Description

[31:0] 1-for-1 enable mapping for the bits in thePCIe Interrupt Status register

RW When set to 1, indicates the setting of theassociated bit in the PCIe InterruptStatus register causes the legacy PCIe orMSI interrupt to be generated.

Table 56. Avalon-MM Interrupt Vector Register - 0x0060


[31:16] Reserved N/A N/A

[15:0] AVL_IRQ_Vector RO Stores the interrupt vector of the systeminterconnect fabric. When the host receives aninterrupt, it should read this register todetermine the servicing priority.

8.2.1.3 PCI Express Mailbox Registers

The PCI Express Root Complex typically requires write access to a set of PCIExpress to Avalon-MM Mailbox registers and read-only access to a set ofAvalon-MM to PCI Express mailbox registers. Eight mailbox registers areavailable.

8 Registers


The PCI Express to Avalon MM Mailbox registers are writable at the addressesshown in the following table. Writing to one of these registers causes thecorresponding bit in the Avalon-MM Interrupt Status register to be set to a one.

Table 57. PCI Express to Avalon-MM Mailbox Registers, 0x0800–0x081F

Address Name Access Description

0x0800 P2A_MAILBOX0 RW PCI Express-to-Avalon-MM Mailbox 0



0x080C P2A_MAILBOX3 RW PCI Express-to-Avalon-MM Mailbox 3




0x081C P2A_MAILBOX7 RW PCI Express-to-Avalon-MM Mailbox 7

The Avalon-MM to PCI Express Mailbox registers are read at the addressesshown in the following table. The PCI Express Root Complex should use theseaddresses to read the mailbox information after being signaled by the correspondingbits in the Avalon-MM to PCI Express Interrupt Status register.

Table 58. Avalon-MM to PCI Express Mailbox Registers, 0x0900–0x091F


0x0900 A2P_MAILBOX0 RO Avalon-MM-to-PCI Express Mailbox 0



0x090C A2P_MAILBOX3 RO Avalon-MM-to-PCI Express Mailbox 3




0x091C A2P_MAILBOX7 RO Avalon-MM-to-PCI Express Mailbox 7

8.2.1.4 Address Mapping for High-Performance Avalon-MM 32-Bit Slave Modules

Address mapping for 32-bit Avalon-MM slave devices allows system software to specifynon-contiguous address pages in the PCI Express address domain. All highperformance 32-bit Avalon-MM slave devices are mapped to the 64-bit PCI Expressaddress space. The Avalon-MM Settings tab of the component GUI allows you toselect the number and size of address mapping pages. Up to 10 address mappingpages are supported. The minimum page size is 4 KB. The maximum page size is 4GB.

When you enable address mapping, the slave address bus width is just large enoughto fit the required address mapping pages. When address mapping is disabled, theAvalon-MM slave address bus is set to 64 bits. The Avalon-MM addresses are used as-is in the resulting PCIe TLPs.

8 Registers


When address mapping is enabled, bursts on the Avalon-MM slave interfaces must notcross page boundaries. This restriction means:

(address + 32 * burst count) <= (page base address + page size )

Address Mapping TableThe address mapping table is accessible through the Control and Status registers.Each entry in the address mapping table is 64 bits (8 bytes) wide and is composed oftwo successive registers. The even address registers holds bits [31:0]. The oddaddress registers holds bits [63:32]. The lower-order address bits are passed throughto the PCIe TLPs unchanged and are ignored in the address mapping table. Thenumber of LSBs that are passed through defines the size of the page and is set atconfiguration time. If bits [63:32] of the resulting PCIe address are zero, TLPs with32-bit wide addresses are created as required by the PCI Express standard.

Table 59. Avalon-MM-to-PCI Express Address Translation Table, 0x1000–0x1FFF


0x1000 A2P_ADDR_MAP_LO0

RW Lower bits of Avalon-MM-to-PCI Express address map entry 0.

0x1004 A2P_ADDR_MAP_HI0

RW Upper bits of Avalon-MM-to-PCI Express address map entry 0.

0x1008 A2P_ADDR_MAP_LO1

RW Lower bits of Avalon-MM-to-PCI Express address map entry 0.This entry is only implemented if the number of address translationtable entries is greater than 1.

0x100C A2P_ADDR_MAP_HI1

RW Upper bits of Avalon-MM-to-PCI Express address map entry 1.This entry is only implemented if the number of address translationtable entries is greater than 1.

8.2.1.5 PCI Express to Avalon-MM Interrupt Status and Enable Registers forEndpoints

These registers record the status of various signals in the PCI Express Avalon-MMbridge logic. They allow Avalon-MM interrupts to be asserted when enabled. Aprocessor local to the interconnect fabric that processes the Avalon-MM interrupts canaccess these registers.

Note: These registers must not be accessed by the PCI Express Avalon-MM bridge masterports. However, nothing in the hardware prevents a PCI Express Avalon-MM bridgemaster port from accessing these registers.

The following table describes the Interrupt Status register for Endpoints. It records thestatus of all conditions that can cause an Avalon-MM interrupt to be asserted.

Table 60. PCI Express to Avalon-MM Interrupt Status Register for Endpoints, 0x3060


[31:24] Reserved N/A Reserved

[23] P2A_MAILBOX_INT7 RW1C Set to a 1 when the P2A_MAILBOX7 is writtento.

[22] P2A_MAILBOX_INT6 RW1C Set to a 1 when the P2A_MAILBOX6

continued...

8 Registers









[15:0] Reserved N/A Reserved

An Avalon-MM interrupt can be asserted for any of the conditions noted in theAvalon-MM Interrupt Status register by setting the corresponding bits in thePCI Express to Avalon-MM Interrupt Enable register.

PCI Express interrupts can also be enabled for all of the error conditions described.However, it is likely that only one of the Avalon-MM or PCI Express interrupts can beenabled for any given bit. Typically, a single process in either the PCI Express orAvalon-MM domain handles the condition reported by the interrupt.

Table 61. INTX Interrupt Enable Register for Endpoints, 0x3070


[31:0] 1-for1 enable mapping for the bits in theAvalon-MM Interrupt Status register.

RW When set to 1, indicates the setting of theassociated bit in the Avalon-MM InterruptStatus register will cause the Avalon-MMinterrupt signal, cra_irq_o, to be asserted.

8.2.1.6 Avalon-MM Mailbox Registers

A processor local to the interconnect fabric typically requires write access to a set ofAvalon-MM to PCI Express Mailbox registers and read-only access to a set ofPCI Express to Avalon-MM Mailbox registers. Eight mailbox registers areavailable.

The Avalon-MM to PCI Express Mailbox registers are writable at the addressesshown in the following table. When the Avalon-MM processor writes to one of theseregisters the corresponding bit in the Avalon-MM to PCI Express InterruptStatus register is set to 1.

Table 62. Avalon-MM to PCI Express Mailbox Registers, 0x3A00–0x3A1F


0x3A00 A2P_MAILBOX0 RW Avalon-MM-to-PCI Express mailbox 0

0x3A04 A2P_MAILBOX1 RW Avalon-MM-to-PCI Express mailbox 1

0x3A08 A2P _MAILBOX2 RW Avalon-MM-to-PCI Express mailbox 2

0x3A0C A2P _MAILBOX3 RW Avalon-MM-to-PCI Express mailbox 3


continued...

8 Registers





0x3A1C A2P_MAILBOX7 RW Avalon-MM-to-PCI Express mailbox 7

The PCI Express to Avalon-MM Mailbox registers are read-only at theaddresses shown in the following table. The Avalon-MM processor reads theseregisters when the corresponding bit in the PCI Express to Avalon-MMInterrupt Status register is set to 1.

Table 63. PCI Express to Avalon-MM Mailbox Registers, 0x3B00–0x3B1F

Address Name AccessMode

Description

0x3B00 P2A_MAILBOX0 RO PCI Express-to-Avalon-MM mailbox 0



0x3B0C P2A_MAILBOX3 RO PCI Express-to-Avalon-MM mailbox 3




0x3B1C P2A_MAILBOX7 RO PCI Express-to-Avalon-MM mailbox 7

8.2.1.7 PCI Express Configuration Information Registers

The PCIe Configuration Information duplicate some of the information foundin Configuration Space Registers. These registers provide processors residing in theAvalon-MM address domain read access to selected configuration information stored inthe PCIe address domain.

Table 64. PCIe Configuration Information Registers 0x0x3C00–0x3C1F


0x3C00 CONFIG_INFO_0 RO The following fields are defined:• [31:29]: Reserved• [28:24]: Device Number• [23:16]: Bus Number• [15]: Memory Space Enable• [14]: Reserved• [13:8]: Auto Negotiation Link Width• [ 7]: Bus Master Enable• [ 6]: Extended Tag Enable• [5:3]: Max Read Request Size• [2:0]: Max Payload Size

0x3C04 CONFIG_INFO_1 RO The following fields are defined:

continued...

8 Registers



• [31:29]: Reserved• [28:24] : AER Interrupt Msg No• [23:18]: Auto Negotiation Link

Width

• [17]: cfg_pm_no_soft_rs• [16]: Read Cpl Boundary (RBC)

Control

• [15:13]: Reserved• [12:8]: PCIe Capability Interrupt

Msg No

• [7:5]: Reserved• [4]: System Power Control• [3:2]: System Attention Indicator

Control

• [1:0]: System Power IndicatorControl

0x3C08 CONFIG_INFO_2 RO The following fields are defined:• [31:30]: Reserved• [29:24]: Index of Start VF[6:0]• [23:16]: Number of VFs• [15:12]: Auto Negotiation Link

Speed

• [11:8] ATS STU[4:1]• [7]: ATS STU[0]• [6]: ATS Cache Enable• [5]: ARI forward enable• [4]: Atomic request enable• [3:2]: TPH ST mode[1:0]• [1]: TPH enable[0]• [0]: VF enable

0x3C0C CONFIG_INFO_3 RO MSI Address Lower

0x3C10 CONFIG_INFO_4 RO MSI Address Upper

0x3C14 CONFIG_INFO_5 RO MSI Mask

0x3C18 CONFIG_INFO_6 RO The following fields are defined:• [31:16] : MSI Data• [15:7]: Reserved• [6]: MSI-X Func Mask• [5]: MSI-X Enable• [4:2]: Multiple MSI Enable• [ 1]: 64-bit MSI• [ 0]: MSI Enable

0x3C1C CONFIG_INFO_7 RO The following fields are defined:• [31:10]: Reserved• [9:6]: Auto Negotiation Link Speed• [5:0]: Auto Negotiation Link Width

8 Registers


9 Testbench and Design ExampleThis chapter introduces the Endpoint design example including a testbench, BFM, anda test driver module. You can create this design example for using design flowsdescribed in Example Design for the Avalon-ST Stratix 10 Hard IP for PCI Express IPCore.

This testbench simulates up to x8 variants. It supports x16 variants by down-trainingto x8. To simulate all lanes of a x16 variant, you can create a simulation model inQsys to use in an Avery testbench. For more information refer to AN-811: Using theAvery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10 Devices.

When configured as an Endpoint variation, the testbench instantiates a designexample and a Root Port BFM, which provides the following functions:

• A configuration routine that sets up all the basic configuration registers in theEndpoint. This configuration allows the Endpoint application to be the target andinitiator of PCI Express transactions.

• A Verilog HDL procedure interface to initiate PCI Express transactions to theEndpoint.

The testbench uses a test driver module, altpcietb_bfm_driver_chaining toexercise the chaining DMA of the design example. The test driver module displaysinformation from the Endpoint Configuration Space registers, so that you can correlateto the parameters you specified using the parameter editor.

This testbench simulates a single Endpoint DUT.

The testbench uses a test driver module, altpcietb_bfm_driver_rp, to exercise thetarget memory and DMA channel in the Endpoint BFM. The test driver module displaysinformation from the Root Port Configuration Space registers, so that you cancorrelate to the parameters you specified using the parameter editor. The Endpointmodel consists of an Endpoint variation combined with the chaining DMA applicationdescribed above.

Note: The Intel testbench and Root Port BFM provide a simple method to do basic testing ofthe Application Layer logic that interfaces to the variation. This BFM allows you tocreate and run simple task stimuli with configurable parameters to exercise basicfunctionality of the Intel example design. The testbench and Root Port BFM are notintended to be a substitute for a full verification environment. Corner cases andcertain traffic profile stimuli are not covered. Refer to the items listed below for furtherdetails. To ensure the best verification coverage possible, Intel suggests strongly thatyou obtain commercially available PCI Express verification IP and tools, or do yourown extensive hardware testing or both.

9 Testbench and Design Example






Your Application Layer design may need to handle at least the following scenarios thatare not possible to create with the Intel testbench and the Root Port BFM:

• It is unable to generate or receive Vendor Defined Messages. Some systemsgenerate Vendor Defined Messages and the Application Layer must be designed toprocess them. The Hard IP block passes these messages on to the ApplicationLayer which, in most cases should ignore them.

• It can only handle received read requests that are less than or equal to thecurrently set Maximum payload size option specified under PCI Express/PCICapabilities heading under the Device tab using the parameter editor. Manysystems are capable of handling larger read requests that are then returned inmultiple completions.

• It always returns a single completion for every read request. Some systems splitcompletions on every 64-byte address boundary.

• It always returns completions in the same order the read requests were issued.Some systems generate the completions out-of-order.

• It is unable to generate zero-length read requests that some systems generate asflush requests following some write transactions. The Application Layer must becapable of generating the completions to the zero length read requests.

• It uses fixed credit allocation.

• It does not support parity.

• It does not support multi-function designs which are available when usingConfiguration Space Bypass mode.

Related Links

AN-811: Using the Avery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10Devices

9.1 Endpoint Testbench

After you install the Quartus Prime software, you can copy any of the example designsfrom the <install_dir>/ip/altera/altera_pcie/altera_pcie_s10_ed/example_design/s10directory.

This testbench simulates up to an ×8 PCI Express link using either the PIPE interfaceof the Endpoint or the serial PCI Express interface. The testbench design does notallow more than one PCI Express link to be simulated at a time. The following figurepresents a high level view of the design example.





Figure 72. Design Example for Endpoint Designs

APPS altpcied_<dev>_hwtcl.v

Hard IP for PCI Express Testbench for Endpoints

Avalon-ST TXAvalon-ST RX

resetstatus

Avalon-ST TXAvalon-ST RXresetstatus

DUT<instance_name>_altera_pcie_a10_hip_<version>_<generated_string>.v

Root Port Modelaltpcie_tbed_<dev>_hwtcl.v

PIPE or Serial

Interface

Root Port BFMaltpcietb_bfm_rpvar_64b_x8_pipen1b

Root Port Driver and Monitoraltpcietb_bfm_vc_intf

The top-level of the testbench instantiates four main modules:

• <qsys_systemname>— This is the example Endpoint design. For more informationabout this module, refer to Chaining DMA Design Examples.

• altpcietb_bfm_top_rp.v—This is the Root Port PCI Express BFM. For moreinformation about this module, refer to Root Port BFM.

• altpcietb_pipe_phy—There are eight instances of this module, one per lane.These modules interconnect the PIPE MAC layer interfaces of the Root Port and theEndpoint. The module mimics the behavior of the PIPE PHY layer to both MACinterfaces.

• altpcietb_bfm_driver_chaining—This module drives transactions to the RootPort BFM. This is the module that you modify to vary the transactions sent to theexample Endpoint design or your own design.

In addition, the testbench has routines that perform the following tasks:

• Generates the reference clock for the Endpoint at the required frequency.

• Provides a PCI Express reset at start up.

Note: Before running the testbench, you should set the following parameters in<instantiation_name>_tb/sim/<instantiation_name>_tb.v:

• serial_sim_hwtcl: Set to 1 for serial simulation and 0 for PIPE simulation.

• enable_pipe32_sim_hwtcl: Set to 0 for serial simulation and 1 for PIPEsimulation.

9.2 Chaining DMA Design Examples

This design example shows how to create a chaining DMA native Endpoint whichsupports simultaneous DMA read and write transactions. The write DMA moduleimplements write operations from the Endpoint memory to the root complex (RC)memory. The read DMA implements read operations from the RC memory to theEndpoint memory.

When operating on a hardware platform, the DMA is typically controlled by a softwareapplication running on the root complex processor. In simulation, the generatedtestbench, along with this design example, provides a BFM driver module in VerilogHDL that controls the DMA operations. Because the example relies on no otherhardware interface than the PCI Express link, you can use the design example for theinitial hardware validation of your system.



The design example includes the following two main components:

• The Root Port variation

• An Application Layer design example

The DUT variant is generated Verilog HDL. The testbench files are only generated inVerilog HDL in the current release.

Note: The chaining DMA design example requires setting BAR 2 or BAR 3 to a minimum of256 bytes. To run the DMA tests using MSI, you must set the Number of MSImessages requested parameter under the PCI Express/PCI Capabilities page toat least 2.

The chaining DMA design example uses an architecture capable of transferring a largeamount of fragmented memory without accessing the DMA registers for every memoryblock. For each block of memory to be transferred, the chaining DMA design exampleuses a descriptor table containing the following information:

• Length of the transfer

• Address of the source

• Address of the destination

• Control bits to set the handshaking behavior between the software application orBFM driver and the chaining DMA module

Note: The chaining DMA design example only supports dword-aligned accesses. The chainingDMA design example does not support ECRC forwarding.

The BFM driver writes the descriptor tables into BFM shared memory, from which thechaining DMA design engine continuously collects the descriptor tables for DMA read,DMA write, or both. At the beginning of the transfer, the BFM programs the Endpointchaining DMA control register. The chaining DMA control register indicates the totalnumber of descriptor tables and the BFM shared memory address of the firstdescriptor table. After programming the chaining DMA control register, the chainingDMA engine continuously fetches descriptors from the BFM shared memory for bothDMA reads and DMA writes, and then performs the data transfer for each descriptor.



The following figure shows a block diagram of the design example connected to anexternal RC CPU. For a description of the DMA write and read registers, Chaining DMAControl and Status Registers.

Figure 73. Top-Level Chaining DMA Example for Simulation

Root Complex

CPU

Root Port

Memory

WriteDescriptor

Table

Data

Chaining DMA

Endpoint Memory

Avalon-MM interfaces

Hard IP forPCI Express

DMA Control/Status Register

DMA Read

Avalon-ST

Configuration

PCI Express DMA Write

DMA Wr Cntl (0x0-4)

DMA Rd Cntl (0x10-1C)

RC Slave

ReadDescriptor

Table

The block diagram contains the following elements:

• Endpoint DMA write and read requester modules.

• The chaining DMA design example connects to the Avalon-ST interface of theStratix 10 Hard IP for PCI Express. The connections consist of the followinginterfaces:

— The Avalon-ST RX receives TLP header and data information from the Hard IPblock

— The Avalon-ST TX transmits TLP header and data information to the Hard IPblock

— The Avalon-ST MSI port requests MSI interrupts from the Hard IP block

— The sideband signal bus carries static information such as configurationinformation

• The descriptor tables of the DMA read and the DMA write are located in the BFMshared memory.

• A RC CPU and associated PCI Express PHY link to the Endpoint design example,using a Root Port and a north/south bridge.

The example Endpoint design Application Layer accomplishes the following objectives:

• Shows you how to interface to the Stratix 10 Hard IP for PCI Express using theAvalon-ST protocol.

• Provides a chaining DMA channel that initiates memory read and writetransactions on the PCI Express link.

• If the ECRC forwarding functionality is enabled, provides a CRC Compiler IP coreto check the ECRC dword from the Avalon-ST RX path and to generate the ECRCfor the Avalon-ST TX path.



The following modules are included in the design example and located in thesubdirectory <qsys_systemname>/testbench/<qsys_system_name>_tb/simulation/submodules :

• <qsys_systemname> —This module is the top level of the example Endpointdesign that you use for simulation.

This module provides both PIPE and serial interfaces for the simulationenvironment. This module has a test_in debug ports. Refer to Test Signalswhich allow you to monitor and control internal states of the Hard IP.

For synthesis, the top level module is <qsys_systemname>/synthesis/submodules. This module instantiates the top-level module and propagates onlya small sub-set of the test ports to the external I/Os. These test ports can be usedin your design.

• <variation name>.v or <variation name>.vhd— Because Intel provides manysample parameterizations, you may have to edit one of the provided examples tocreate a simulation that matches your requirements. <variation name>.v or<variation name>.vhd— Because Intel provides many sample parameterizations,you may have to edit one of the provided examples to create a simulation thatmatches your requirements.

The chaining DMA design example hierarchy consists of these components:

• A DMA read and a DMA write module

• An on-chip Endpoint memory (Avalon-MM slave) which uses two Avalon-MMinterfaces for each engine

The RC slave module is used primarily for downstream transactions which target theEndpoint on-chip buffer memory. These target memory transactions bypass the DMAengines. In addition, the RC slave module monitors performance and acknowledgesincoming message TLPs. Each DMA module consists of these components:

• Control register module—The RC programs the control register (four dwords) tostart the DMA.

• Descriptor module—The DMA engine fetches four dword descriptors from BFMshared memory which hosts the chaining DMA descriptor table.

• Requester module—For a given descriptor, the DMA engine performs the memorytransfer between Endpoint memory and the BFM shared memory.



The following modules are provided in both Verilog HDL:

• altpcierd_example_app_chaining—This top level module contains the logicrelated to the Avalon-ST interfaces as well as the logic related to the sidebandbus. This module is fully register bounded and can be used as an incremental re-compile partition in the Quartus Prime compilation flow.

• altpcierd_cdma_ast_rx, altpcierd_cdma_ast_rx_64,altpcierd_cdma_ast_rx_128—These modules implement the Avalon-ST receiveport for the chaining DMA. The Avalon-ST receive port converts the Avalon-STinterface of the IP core to the descriptor/data interface used by the chaining DMAsubmodules. altpcierd_cdma_ast_rx is used with the descriptor/data IP core(through the ICM). a ltpcierd_cdma_ast_rx_64 is used with the 64-bit Avalon-ST IP core. altpcierd_cdma_ast_rx_128 is used with the 128-bit Avalon-ST IPcore.

• altpcierd_cdma_ast_tx, altpcierd_cdma_ast_tx_64,altpcierd_cdma_ast_tx_128—These modules implement the Avalon-STtransmit port for the chaining DMA. The Avalon-ST transmit port converts thedescriptor/data interface of the chaining DMA submodules to the Avalon-STinterface of the IP core. altpcierd_cdma_ast_tx is used with the descriptor/dataIP core (through the ICM). altpcierd_cdma_ast_tx_64 is used with the 64-bitAvalon-ST IP core. altpcierd_cdma_ast_tx_128 is used with the 128-bitAvalon-ST IP core.

• altpcierd_cdma_ast_msi—This module converts MSI requests from the chainingDMA submodules into Avalon-ST streaming data.

• alpcierd_cdma_app_icm—This module arbitrates PCI Express packets for themodules altpcierd_dma_dt (read or write) and altpcierd_rc_slave.alpcierd_cdma_app_icm instantiates the Endpoint memory used for the DMAread and write transfer.

• alt pcierd_compliance_test.v—This module provides the logic to perform CBBvia a push button.

• altpcierd_rc_slave—This module provides the completer function for alldownstream accesses. It instantiates the altpcierd_rxtx_downstream_intf andaltpcierd_reg_ access modules. Downstream requests include programming ofchaining DMA control registers, reading of DMA status registers, and direct readand write access to the Endpoint target memory, bypassing the DMA.

• altpcierd_rx_tx_downstream_intf—This module processes all downstreamread and write requests and handles transmission of completions. Requestsaddressed to BARs 0, 1, 4, and 5 access the chaining DMA target memory space.Requests addressed to BARs 2 and 3 access the chaining DMA control and statusregister space using the altpcierd_reg_access module.

• altpcierd_reg_access—This module provides access to all of the chaining DMAcontrol and status registers (BAR 2 and 3 address space). It provides addressdecoding for all requests and multiplexing for completion data. All registers are32-bits wide. Control and status registers include the control registers in thealtpcierd_dma_prg_reg module, status registers in thealtpcierd_read_dma_requester and altpcierd_write_dma_requestermodules, as well as other miscellaneous status registers.

• altpcierd_dma_dt—This module arbitrates PCI Express packets issued by thesubmodules altpcierd_dma_prg_reg, altpcierd_read_dma_requester,altpcierd_write_dma_requester and altpcierd_dma_descriptor.



• altpcierd_dma_prg_reg —This module contains the chaining DMA controlregisters which get programmed by the software application or BFM driver.

• altpcierd_dma_descriptor—This module retrieves the DMA read or writedescriptor from the BFM shared memory, and stores it in a descriptor FIFO. Thismodule issues upstream PCI Express TLPs of type MRd.

• altpcierd_read_dma_requester, altpcierd_read_dma_requester_128—Foreach descriptor located in the altpcierd_descriptor FIFO, this module transfersdata from the BFM shared memory to the Endpoint memory by issuing MRd PCIExpress transaction layer packets. altpcierd_read_dma_requester is used withthe 64-bit Avalon-ST IP core. altpcierd_read_dma_requester_128 is used withthe 128-bit Avalon-ST IP core.

• altpcie rd_write_dma_requester, altpcierd_write_dma_requester_128—For each descriptor located in the altpcierd_descriptor FIFO, this moduletransfers data from the Endpoint memory to the BFM shared memory by issuingMWr PCI Express transaction layer packets. altpcierd_write_dma_requester isused with the 64-bit Avalon-ST IP core. altpcierd_write_dma_requester_128is used with the 128-bit Avalon-ST IP core.ls

• altpcierd_cpld_rx_buffer—This modules monitors the available space of the RXBuffer; It prevents RX Buffer overflow by arbitrating memory read request issuedby the application.

• altpcierd_cplerr_lmi—This module transfers the err_desc_func0 from theapplication to the Hard IP block using the LMI interface. It also retimes thecpl_err bits from the application to the Hard IP block.

• altpcierd_tl_cfg_sample—This module demultiplexes the Configuration Spacesignals from the tl_cfg_ctl bus from the Hard IP block and synchronizes thisinformation, along with the tl_cfg_sts bus to the user clock (pld_clk)domain.

Related Links

Chaining DMA Control and Status Registers on page 114

9.2.1 BAR/Address Map

The design example maps received memory transactions to either the target memoryblock or the control register block based on which BAR the transaction matches. Thereare multiple BARs that map to each of these blocks to maximize interoperability withdifferent variation files. The following table shows the mapping.

Table 65. BAR Map

Memory BAR Mapping

32-bit BAR032-bit BAR164-bit BAR1:0

Maps to 32 KB target memory block. Use the rc_slave module to bypass the chainingDMA.

32-bit BAR232-bit BAR364-bit BAR3:2

Maps to DMA Read and DMA write control and status registers, a minimum of 256bytes.

32-bit BAR432-bit BAR5

Maps to 32 KB target memory block. Use the rc_slave module to bypass the chainingDMA.

continued...



Memory BAR Mapping

64-bit BAR5:4

Expansion ROM BAR Not implemented by design example; behavior is unpredictable.

I/O Space BAR (any) Not implemented by design example; behavior is unpredictable.

9.2.2 Chaining DMA Control and Status Registers

The software application programs the chaining DMA control register located in theEndpoint application. The following table describes the control registers which consistsof four dwords for the DMA write and four dwords for the DMA read. The DMA controlregisters are read/write.

In this table, Addr specifies the Endpoint byte address offset from BAR2 or BAR3.

Table 66. Chaining DMA Control Register Definitions

Addr Register Name Bits[31:]24 Bit[23:16] Bit[15:0]

0x0 DMA Wr Cntl DW0 Control Field Number of descriptorsin descriptor table

0x4 DMA Wr Cntl DW1 Base Address of the Write Descriptor Table (BDT) in the RC Memory–UpperDWORD

0x8 DMA Wr Cntl DW2 Base Address of the Write Descriptor Table (BDT) in the RC Memory–LowerDWORD

0xC DMA Wr Cntl DW3 Reserved Reserved RCLAST–Idx of lastdescriptor to process

0x10 DMA Rd Cntl DW0 Control Field (described in the next table) Number of descriptorsin descriptor table

0x14 DMA Rd Cntl DW1 Base Address of the Read Descriptor Table (BDT) in the RC Memory–UpperDWORD

0x18 DMA Rd Cntl DW2 Base Address of the Read Descriptor Table (BDT) in the RC Memory–LowerDWORD

0x1C DMA Rd Cntl DW3 Reserved Reserved RCLAST–Idx of the lastdescriptor to process

The following table describes the control fields of the of the DMA read and DMA writecontrol registers.

Table 67. Bit Definitions for the Control Field in the DMA Write Control Register andDMA Read Control Register

Bit Field Description

16 Reserved —

17 MSI_ENA Enables interrupts of all descriptors. When 1, the Endpoint DMA moduleissues an interrupt using MSI to the RC when each descriptor iscompleted. Your software application or BFM driver can use this interruptto monitor the DMA transfer status.

18 EPLAST_ENA Enables the Endpoint DMA module to write the number of each descriptorback to the EPLAST field in the descriptor table.

continued...




[24:20] MSI Number When your RC reads the MSI capabilities of the Endpoint, these registerbits map to the back-end MSI signals app_msi_num [4:0]. If there ismore than one MSI, the default mapping if all the MSIs are available, is:• MSI 0 = Read• MSI 1 = Write

[30:28] MSI Traffic Class When the RC application software reads the MSI capabilities of theEndpoint, this value is assigned by default to MSI traffic class 0. Theseregister bits map to the back-end signal app_msi_tc[2:0].

31 DT RC Last Sync When 0, the DMA engine stops transfers when the last descriptor hasbeen executed. When 1, the DMA engine loops infinitely restarting withthe first descriptor when the last descriptor is completed. To stop theinfinite loop, set this bit to 0.

The following table defines the DMA status registers. These registers are read only. Inthis table, Addr specifies the Endpoint byte address offset from BAR2 or BAR3.

Table 68. Chaining DMA Status Register Definitions

Addr Register Name Bits[31:24] Bits[23:16] Bits[15:0]

0x20 DMA Wr Status Hi For field definitions refer to Fields in the DMA Write Status High Registerbelow.

0x24 DMA Wr Status Lo Target Mem AddressWidth

Write DMA Performance Counter. (Clock cyclesfrom time DMA header programmed until lastdescriptor completes, including time to fetchdescriptors.)

0x28 DMA Rd Status Hi For field definitions refer to Fields in the DMA Read Status High Registerbelow.

0x2C DMA Rd Status Lo Max No. of Tags Read DMA Performance Counter. The number ofclocks from the time the DMA header isprogrammed until the last descriptorcompletes, including the time to fetchdescriptors.

0x30 Error Status Reserved Error Counter. Numberof bad ECRCs detectedby the ApplicationLayer. Valid only whenECRC forwarding isenabled.

The following table describes the fields of the DMA write status register. All of thesefields are read only.

Table 69. Fields in the DMA Write Status High Register


[31:28] CDMA version Identifies the version of the chaining DMA example design.

[27:24] Reserved —

[23:21] Max payload size The following encodings are defined:• 001 128 bytes• 001 256 bytes• 010 512 bytes• 011 1024 bytes• 100 2048 bytes

continued...





16 Write DMA descriptorFIFO empty

Indicates that there are no more descriptors pending in the write DMA.

[15:0] Write DMA EPLAS Indicates the number of the last descriptor completed by the write DMA.For simultaneous DMA read and write transfers, EPLAST is only supportedfor the final descriptor in the descriptor table.

The following table describes the fields in the DMA read status high register. All ofthese fields are read only.

Table 70. Fields in the DMA Read Status High Register



[23:21] Max Read Request Size The following encodings are defined:• 001 128 bytes• 001 256 bytes• 010 512 bytes• 011 1024 bytes• 100 2048 bytes

[20:17] Negotiated Link Width The following encodings are defined:• 4'b0001 ×1• 4'b0010 ×2• 4'b0100 ×4• 4'b1000 ×8

16 Read DMA Descriptor FIFOEmpty

Indicates that there are no more descriptors pending in the read DMA.

[15:0] Read DMA EPLAST Indicates the number of the last descriptor completed by the read DMA. Forsimultaneous DMA read and write transfers, EPLAST is only supported forthe final descriptor in the descriptor table.

9.2.3 Chaining DMA Descriptor Tables

The following table describes the Chaining DMA descriptor table. This table is stored inthe BFM shared memory. It consists of a four-dword descriptor header and acontiguous list of <n> four-dword descriptors. The Endpoint chaining DMA applicationaccesses the Chaining DMA descriptor table for two reasons:

• To iteratively retrieve four-dword descriptors to start a DMA

• To send update status to the RP, for example to record the number of descriptorscompleted to the descriptor header

Each subsequent descriptor consists of a minimum of four dwords of data andcorresponds to one DMA transfer. (A dword equals 32 bits.)

Note: The chaining DMA descriptor table should not cross a 4 KB boundary.



Table 71. Chaining DMA Descriptor Table

Byte AddressOffset to Base

Source

Descriptor Type Description

0x0 Descriptor Header Reserved

0x4 Reserved

0x8 Reserved

0xC EPLAST - when enabled by the EPLAST_ENA bit in the controlregister or descriptor, this location records the number of the lastdescriptor completed by the chaining DMA module.

0x10 Descriptor 0 Control fields, DMA length

0x14 Endpoint address

0x18 RC address upper dword

0x1C RC address lower dword

0x20 Descriptor 1 Control fields, DMA length

0x24 Endpoint address

0x28 RC address upper dword

0x2C RC address lower dword

. . .

0x ..0 Descriptor <n> Control fields, DMA length

0x ..4 Endpoint address

0x ..8 RC address upper dword

0x ..C RC address lower dword

The following table shows the layout of the descriptor fields following the descriptorheader.

Table 72. Chaining DMA Descriptor Format Map

Bits[31:22] Bits[21:16] Bits[15:0]

Reserved Control Fields (refer to Table 18–9) DMA Length

Endpoint Address

RC Address Upper DWORD

RC Address Lower DWORD

The following table shows the layout of the control fields of the chaining DMAdescriptor.

Table 73. Chaining DMA Descriptor Format Map (Control Fields)

Bits[21:18] Bit[17] Bit[16]

Reserved EPLAST_ENA MSI

Each descriptor provides the hardware information on one DMA transfer. The followingtable describes each descriptor field.



Table 74. Chaining DMA Descriptor Fields

Descriptor Field EndpointAccess

RC Access Description

Endpoint Address R R/W A 32-bit field that specifies the base address of the memorytransfer on the Endpoint site.

RC AddressUpper DWORD

R R/W Specifies the upper base address of the memory transfer onthe RC site.

RC AddressLower DWORD

R R/W Specifies the lower base address of the memory transfer onthe RC site.

DMA Length R R/W Specifies the number of DMA DWORDs to transfer.

EPLAST_ENA R R/W This bit is OR’d with the EPLAST_ENA bit of the controlregister. When EPLAST_ENA is set, the Endpoint DMA moduleupdates the EPLAST field of the descriptor table with thenumber of the last completed descriptor, in the form <0 – n>.Refer to Chaining DMA Descriptor Tables on page 116 for moreinformation.

MSI_ENA R R/W This bit is OR’d with the MSI bit of the descriptor header.When this bit is set the Endpoint DMA module sends aninterrupt when the descriptor is completed.

9.3 Test Driver Module

The BFM driver module, altpcietb_bfm_driver_chaining.v is configured to test thechaining DMA example Endpoint design. The BFM driver module configures theEndpoint Configuration Space registers and then tests the example Endpoint chainingDMA channel. This file is stored in the <working_dir>/testbench/<variation_name>/simulation/submodules directory.

The BFM test driver module performs the following steps in sequence:

1. Configures the Root Port and Endpoint Configuration Spaces, which the BFM testdriver module does by calling the procedure ebfm_cfg_rp_ep, which is part ofaltpcietb_bfm_configure.

2. Finds a suitable BAR to access the example Endpoint design Control Registerspace. Either BARs 2 or 3 must be at least a 256-byte memory BAR to perform theDMA channel test. The find_mem_bar procedure in thealtpcietb_bfm_driver_chaining does this.

3. If a suitable BAR is found in the previous step, the driver performs the followingtasks:

a. DMA read—The driver programs the chaining DMA to read data from the BFMshared memory into the Endpoint memory. The descriptor control fields arespecified so that the chaining DMA completes the following steps to indicatetransfer completion:

• The chaining DMA writes the EPLast bit of the Chaining DMA DescriptorTable after finishing the data transfer for the first and last descriptors.

• The chaining DMA issues an MSI when the last descriptor has completed.

a. DMA write—The driver programs the chaining DMA to write the data from itsEndpoint memory back to the BFM shared memory. The descriptor controlfields are specified so that the chaining DMA completes the following steps toindicate transfer completion:



• The chaining DMA writes the EPLast bit of the Chaining DMA DescriptorTable after completing the data transfer for the first and last descriptors.

• The chaining DMA issues an MSI when the last descriptor has completed.

• The data written back to BFM is checked against the data that was readfrom the BFM.

• The driver programs the chaining DMA to perform a test thatdemonstrates downstream access of the chaining DMA Endpoint memory.

Note: Edit this file if you want to add your own custom PCIe transactions. Insert your owncustom function after the find_mem_bar function. You can use the functions in theBFM Procedures and Functions section.

Related Links

• Chaining DMA Descriptor Tables on page 116

• BFM Procedures and Functions on page 130

9.4 DMA Write Cycles

The procedure dma_wr_test used for DMA writes uses the following steps:

1. Configures the BFM shared memory. Configuration is accomplished with threedescriptor tables described below.

Table 75. Write Descriptor 0

Offset in BFMin SharedMemory

Value Description

DW0 0x810 82 Transfer length in dwords and control bits as described in BitDefinitions for the Control Field in the DMA Write Control Register andDMA Read Control Register.

DW1 0x814 3 Endpoint address

DW2 0x818 0 BFM shared memory data buffer 0 upper address value

DW3 0x81c 0x1800 BFM shared memory data buffer 1 lower address value

Data Buffer0

0x1800 Increment by 1from0x1515_0001

Data content in the BFM shared memory from address: 0x01800–0x1840


Offset in BFMSharedMemory

Value Description

DW0 0x820 1,024 Transfer length in dwords and control bits as described in BitDefinitions for the Control Field in the DMA Write Control Register andDMA Read Control Register .


continued...




Value Description



Data Buffer1

0x02800 Increment by 1from0x2525_0001

Data content in the BFM shared memory from address: 0x02800



Value Description

DW0 0x830 644 Transfer length in dwords and control bits as described in BitDefinitions for the Control Field in the DMA Write Control Register andDMA Read Control Register.



DW3 0x83c 0x057A0 BFM shared memory data buffer 2 lower address value

Data Buffer2

0x057A0 Increment by 1from0x3535_0001

Data content in the BFM shared memory from address: 0x057A0

2. Sets up the chaining DMA descriptor header and starts the transfer data from theEndpoint memory to the BFM shared memory. The transfer calls the proceduredma_set_header which writes four dwords, DW0:DW3, into the DMA writeregister module.

Table 78. DMA Control Register Setup for DMA Write

Offset in DMAControlRegister(BAR2)

Value Description

DW0 0x0 3 Number of descriptors and control bits as described in Chaining DMAControl Register Definitions.

DW1 0x4 0 BFM shared memory descriptor table upper address value

DW2 0x8 0x800 BFM shared memory descriptor table lower address value

DW3 0xc 2 Last valid descriptor

After writing the last dword, DW3, of the descriptor header, the DMA write startsthe three subsequent data transfers.

3. Waits for the DMA write completion by polling the BFM share memory location0x80c, where the DMA write engine is updating the value of the number ofcompleted descriptor. Calls the procedures rcmem_poll and msi_poll todetermine when the DMA write transfers have completed.

Related Links

Chaining DMA Control and Status Registers on page 114



9.5 DMA Read Cycles

The procedure dma_rd_test used for DMA read uses the following three steps:

1. Configures the BFM shared memory with a call to the proceduredma_set_rd_desc_data which sets the following three descriptor tables. .

Table 79. Read Descriptor 0


Value Description

DW0 0x910 82 Transfer length in dwords and control bits as described in onpage 18–15

DW1 0x914 3 Endpoint address value


DW3 0x91c 0x8DF0 BFM shared memory data buffer 0 lower address value

Data Buffer0

0x8DF0 Increment by 1from0xAAA0_0001

Data content in the BFM shared memory from address: 0x89F0



Value Description

DW0 0x920 1,024 Transfer length in dwords and control bits as described in onpage 18–15




Data Buffer1

0x10900 Increment by 1from0xBBBB_0001

Data content in the BFM shared memory from address: 0x10900



Value Description

DW0 0x930 644 Transfer length in dwords and control bits as described in onpage 18–15


DW2 0x938 0 BFM shared memory upper address value

DW3 0x93c 0x20EF0 BFM shared memory lower address value

Data Buffer2

0x20EF0 Increment by 1from0xCCCC_0001

Data content in the BFM shared memory from address: 0x20EF0

2. Sets up the chaining DMA descriptor header and starts the transfer data from theBFM shared memory to the Endpoint memory by calling the proceduredma_set_header which writes four dwords, DW0:DW3 into the DMA readregister module.



Table 82. DMA Control Register Setup for DMA Read

Offset in DMA ControlRegisters (BAR2)

Value Description

DW0 0x0 3 Number of descriptors and control bits as described in Chaining DMAControl Register Definitions.

DW1 0x14 0 BFM shared memory upper address value

DW2 0x18 0x900 BFM shared memory lower address value

DW3 0x1c 2 Last descriptor written

After writing the last dword of the Descriptor header (DW3), the DMA read startsthe three subsequent data transfers.

3. Waits for the DMA read completion by polling the BFM shared memory location0x90c, where the DMA read engine is updating the value of the number ofcompleted descriptors. Calls the procedures rcmem_poll and msi_poll todetermine when the DMA read transfers have completed.

9.6 Root Port BFM

The basic Root Port BFM provides Verilog HDL task-based interface for requestingtransactions that are issued to the PCI Express link. The Root Port BFM also handlesrequests received from the PCI Express link. The following figure provides an overviewof the Root Port BFM.

Figure 74. Root Port BFM

BFM Shared Memory(altpcietb_bfm_shmem

_common)

BFM Log Interface(altpcietb_bfm_log

_common)

Root Port RTL Model (altpcietb_bfm_rp_top_x8_pipen1b)

IP Functional SimulationModel of the Root

Port Interface (altpcietb_bfm_driver_rp)

Avalon-ST Interface(altpcietb_bfm_vc_intf)

Root Port BFM

BFM Read/Write Shared Request Procedures

BFM Configuration Procedures

BFM Request Interface(altpcietb_bfm_req_intf_common)



The functionality of each of the modules included is explained below.

• BFM shared memory (altpcietb_bfm_shmem_common Verilog HDL include file)—The Root Port BFM is based on the BFM memory that is used for the followingpurposes:

• Storing data received with all completions from the PCI Express link.

• Storing data received with all write transactions received from the PCI Expresslink.

• Sourcing data for all completions in response to read transactions received fromthe PCI Express link.

• Sourcing data for most write transactions issued to the PCI Express link. The onlyexception is certain BFM write procedures that have a four-byte field of write datapassed in the call.

• Storing a data structure that contains the sizes of and the values programmed inthe BARs of the Endpoint.

A set of procedures is provided to read, write, fill, and check the shared memory fromthe BFM driver. For details on these procedures, see BFM Shared Memory AccessProcedures.

• BFM Read/Write Request Functions(altpcietb_bfm_driver_rp.v)—Thesefunctions provide the basic BFM calls for PCI Express read and write requests. Fordetails on these procedures, refer to BFM Read and Write Procedures.

• BFM Configuration Functions(altpcietb_bfm_driver_rp.v )—These functionsprovide the BFM calls to request configuration of the PCI Express link and theEndpoint Configuration Space registers. For details on these procedures andfunctions, refer to BFM Configuration Procedures.

• BFM Log Interface(altpcietb_bfm_driver_rp.v)—The BFM log functionsprovides routines for writing commonly formatted messages to the simulatorstandard output and optionally to a log file. It also provides controls that stopsimulation on errors. For details on these procedures, refer to BFM Log andMessage Procedures.

• BFM Request Interface(altpcietb_bfm_driver_rp.v)—This interface providesthe low-level interface between the altpcietb_bfm_rdwr andaltpcietb_bfm_configure procedures or functions and the Root Port RTLModel. This interface stores a write-protected data structure containing the sizesand the values programmed in the BAR registers of the Endpoint, as well as, othercritical data used for internal BFM management. You do not need to access thesefiles directly to adapt the testbench to test your Endpoint application.

• Avalon-ST Interfaces (altpcietb_bfm_vc_intf.v)—These interface moduleshandle the Root Port interface model. They take requests from the BFM requestinterface and generate the required PCI Express transactions. They handlecompletions received from the PCI Express link and notify the BFM requestinterface when requests are complete. Additionally, they handle any requestsreceived from the PCI Express link, and store or fetch data from the sharedmemory before generating the required completions.

Related Links

BFM Shared Memory Access Procedures on page 136



9.6.1 BFM Memory Map

The BFM shared memory is configured to be two MBs. The BFM shared memory ismapped into the first two MBs of I/O space and also the first two MBs of memoryspace. When the Endpoint application generates an I/O or memory transaction in thisrange, the BFM reads or writes the shared memory.

9.6.2 Configuration Space Bus and Device Numbering

The Root Port interface is assigned to be device number 0 on internal bus number 0.The Endpoint can be assigned to be any device number on any bus number (greaterthan 0) through the call to procedure ebfm_cfg_rp_ep. The specified bus number isassigned to be the secondary bus in the Root Port Configuration Space.

9.6.3 Configuration of Root Port and Endpoint

Before you issue transactions to the Endpoint, you must configure the Root Port andEndpoint Configuration Space registers. To configure these registers, call theprocedure ebfm_cfg_rp_ep, which is included in altpcietb_bfm_driver_rp.v.

The ebfm_cfg_rp_ep executes the following steps to initialize the ConfigurationSpace:

1. Sets the Root Port Configuration Space to enable the Root Port to sendtransactions on the PCI Express link.

2. Sets the Root Port and Endpoint PCI Express Capability Device Control registers asfollows:

a. Disables Error Reporting in both the Root Port and Endpoint. BFM does nothave error handling capability.

b. Enables Relaxed Ordering in both Root Port and Endpoint.

c. Enables Extended Tags for the Endpoint, if the Endpoint has that capability.

d. Disables Phantom Functions, Aux Power PM, and No Snoop in both theRoot Port and Endpoint.

e. Sets the Max Payload Size to what the Endpoint supports because the RootPort supports the maximum payload size.

f. Sets the Root Port Max Read Request Size to 4 KB because the exampleEndpoint design supports breaking the read into as many completions asnecessary.

g. Sets the Endpoint Max Read Request Size equal to the Max Payload Sizebecause the Root Port does not support breaking the read request intomultiple completions.

3. Assigns values to all the Endpoint BAR registers. The BAR addresses are assignedby the algorithm outlined below.



a. I/O BARs are assigned smallest to largest starting just above the endingaddress of BFM shared memory in I/O space and continuing as neededthroughout a full 32-bit I/O space.

b. The 32-bit non-prefetchable memory BARs are assigned smallest to largest,starting just above the ending address of BFM shared memory in memoryspace and continuing as needed throughout a full 32-bit memory space.

c. Assignment of the 32-bit prefetchable and 64-bit prefetchable memory BARSare based on the value of the addr_map_4GB_limit input to theebfm_cfg_rp_ep. The default value of the addr_map_4GB_limit is 0.

If the addr_map_4GB_limit input to the ebfm_cfg_rp_ep is set to 0, thenthe 32-bit prefetchable memory BARs are assigned largest to smallest,starting at the top of 32-bit memory space and continuing as needed down tothe ending address of the last 32-bit non-prefetchable BAR.

However, if the addr_map_4GB_limit input is set to 1, the address map islimited to 4 GB, the 32-bit and 64-bit prefetchable memory BARs are assignedlargest to smallest, starting at the top of the 32-bit memory space andcontinuing as needed down to the ending address of the last 32-bit non-prefetchable BAR.

d. If the addr_map_4GB_limit input to the ebfm_cfg_rp_ep is set to 0, thenthe 64-bit prefetchable memory BARs are assigned smallest to largest startingat the 4 GB address assigning memory ascending above the 4 GB limitthroughout the full 64-bit memory space.

If the addr_map_4 GB_limit input to the ebfm_cfg_rp_ep is set to 1, thenthe 32-bit and the 64-bit prefetchable memory BARs are assigned largest tosmallest starting at the 4 GB address and assigning memory by descendingbelow the 4 GB address to addresses memory as needed down to the endingaddress of the last 32-bit non-prefetchable BAR.

The above algorithm cannot always assign values to all BARs when there are afew very large (1 GB or greater) 32-bit BARs. Although assigning addresses toall BARs may be possible, a more complex algorithm would be required toeffectively assign these addresses. However, such a configuration is unlikely tobe useful in real systems. If the procedure is unable to assign the BARs, itdisplays an error message and stops the simulation.

4. Based on the above BAR assignments, the Root Port Configuration Space addresswindows are assigned to encompass the valid BAR address ranges.

5. The Endpoint PCI control register is set to enable master transactions, memoryaddress decoding, and I/O address decoding.

The ebfm_cfg_rp_ep procedure also sets up a bar_table data structure in BFMshared memory that lists the sizes and assigned addresses of all Endpoint BARs. Thisarea of BFM shared memory is write-protected, which means any user write accessesto this area cause a fatal simulation error. This data structure is then used bysubsequent BFM procedure calls to generate the full PCI Express addresses for readand write requests to particular offsets from a BAR. This procedure allows thetestbench code that accesses the Endpoint Application Layer to be written to useoffsets from a BAR and not have to keep track of the specific addresses assigned tothe BAR. The following table shows how those offsets are used.



Table 83. BAR Table Structure

Offset (Bytes) Description

+0 PCI Express address in BAR0






+24 PCI Express address in Expansion ROM BAR

+28 Reserved

+32 BAR0 read back value after being written with all 1’s (used to compute size)

+36 BAR1 read back value after being written with all 1’s





+56 Expansion ROM BAR read back value after being written with all 1’s

+60 Reserved

The configuration routine does not configure any advanced PCI Express capabilitiessuch as the AER capability.

Besides the ebfm_cfg_rp_ep procedure in altpcietb_bfm_driver_rp.v, routines toread and write Endpoint Configuration Space registers directly are available in theVerilog HDL include file. After the ebfm_cfg_rp_ep procedure is run the PCI ExpressI/O and Memory Spaces have the layout as described in the following three figures.The memory space layout is dependent on the value of the add r_map_4GB_limitinput parameter. If addr_map_4GB_limit is 1 the resulting memory space map isshown in the following figure.



Figure 75. Memory Space Layout—4 GB Limit

Root Complex Shared Memory

Unused

Configuration ScratchSpace Used by

BFM Routines - NotWriteable by UserCalls or Endpoint

BAR TableUsed by BFM

Routines - NotWriteable by UserCalls or End Point

Endpoint Non-Prefetchable Memory

Space BARsAssigned Smallest

to Largest

Endpoint MemorySpace BARs

Prefetchable 32-bit and 64-bit

Assigned Smallestto Largest

0xFFFF FFFF

0x0020 0000

0x0000 0000Address

0x001F FFC0

0x001F FF80

If addr_map_4GB_limit is 0, the resulting memory space map is shown in thefollowing figure.



Figure 76. Memory Space Layout—No Limit


Unused

Unused

Configuration ScratchSpace Used byRoutines - Not

Writeable by UserCalls or Endpoint


Routines - NotWriteable by UserCalls or Endpoint

Endpoint Non-Prefetchable Memory

Space BARsAssigned Smallest

to Largest


Prefetchable 64-bit Assigned Smallest

to Largest


Prefetchable 32-bit Assigned Smallest

to Largest

BAR-Size Dependent

BAR-Size Dependent

BAR-Size Dependent

0x0000 0001 0000 0000

0xFFFF FFFF FFFF FFFF

0x0020 0000

0x0000 0000Address

0x001F FF00

0x001F FF80

The following figure shows the I/O address space.



Figure 77. I/O Address Space


Unused

Configuration ScratchSpace Used by BFM




Endpoint I/O Space BARs

Assigned Smallestto Largest

BAR-Size Dependent

0xFFFF FFFF

0x0020 0000

0x0000 0000Address

0x001F FFC0

0x001F FF80



9.6.4 Issuing Read and Write Transactions to the Application Layer

Read and write transactions are issued to the Endpoint Application Layer by calling oneof the ebfm_bar procedures in altpcietb_bfm_driver_rp.v. The procedures andfunctions listed below are available in the Verilog HDL include filealtpcietb_bfm_driver_rp.v. The complete list of available procedures and functionsis as follows:

• ebfm_barwr—writes data from BFM shared memory to an offset from a specificEndpoint BAR. This procedure returns as soon as the request has been passed tothe VC interface module for transmission.

• ebfm_barwr_imm—writes a maximum of four bytes of immediate data (passed ina procedure call) to an offset from a specific Endpoint BAR. This procedure returnsas soon as the request has been passed to the VC interface module fortransmission.

• ebfm_barrd_wait—reads data from an offset of a specific Endpoint BAR andstores it in BFM shared memory. This procedure blocks waiting for the completiondata to be returned before returning control to the caller.

• ebfm_barrd_nowt—reads data from an offset of a specific Endpoint BAR andstores it in the BFM shared memory. This procedure returns as soon as the requesthas been passed to the VC interface module for transmission, allowing subsequentreads to be issued in the interim.

These routines take as parameters a BAR number to access the memory space andthe BFM shared memory address of the bar_table data structure that was set up bythe ebfm_cfg_rp_ep procedure. (Refer to Configuration of Root Port and Endpoint.)Using these parameters simplifies the BFM test driver routines that access an offsetfrom a specific BAR and eliminates calculating the addresses assigned to the specifiedBAR.

The Root Port BFM does not support accesses to Endpoint I/O space BARs.

Related Links

Configuration of Root Port and Endpoint on page 124

9.7 BFM Procedures and Functions

The BFM includes procedures, functions, and tasks to drive Endpoint applicationtesting. It also includes procedures to run the chaining DMA design example.

The BFM read and write procedures read and write data among BFM shared memory,Endpoint BARs, and specified configuration registers. The procedures and functions areavailable in the Verilog HDL. They are in the include file altpcietb_bfm_driver.v.These procedures and functions support issuing memory and configurationtransactions on the PCI Express link.

9.7.1 ebfm_barwr Procedure

The ebfm_barwr procedure writes a block of data from BFM shared memory to anoffset from the specified Endpoint BAR. The length can be longer than the configuredMAXIMUM_PAYLOAD_SIZE; the procedure breaks the request up into multipletransactions as needed. This routine returns as soon as the last transaction has beenaccepted by the VC interface module.



Location altpcietb_bfm_rdwr.v

Syntax ebfm_barwr(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.The bar_table structure stores the address assigned to each BAR sothat the driver code does not need to be aware of the actual assignedaddresses only the application specific offsets from the BAR.

bar_num Number of the BAR used with pcie_offset to determine PCI Expressaddress.

pcie_offset Address offset from the BAR base.

lcladdr BFM shared memory address of the data to be written.

byte_len Length, in bytes, of the data written. Can be 1 to the minimum of thebytes remaining in the BAR space or BFM shared memory.

tclass Traffic class used for the PCI Express transaction.

9.7.2 ebfm_barwr_imm Procedure

The ebfm_barwr_imm procedure writes up to four bytes of data to an offset from thespecified Endpoint BAR.

Location altpcietb_bfm_driver_rp.v

Syntax ebfm_barwr_imm(bar_table, bar_num, pcie_offset, imm_data, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.The bar_table structure stores the address assigned to each BAR sothat the driver code does not need to be aware of the actual assignedaddresses only the application specific offsets from the BAR.



imm_data Data to be written. In Verilog HDL, this argument is reg [31:0].Inboth languages, the bits written depend on the length as follows:Length Bits Written• 4: 31 downto 0• 3: 23 downto 0• 2: 15 downto 0• 1: 7 downto 0

byte_len Length of the data to be written in bytes. Maximum length is 4 bytes.

tclass Traffic class to be used for the PCI Express transaction.

9.7.3 ebfm_barrd_wait Procedure

The ebfm_barrd_wait procedure reads a block of data from the offset of thespecified Endpoint BAR and stores it in BFM shared memory. The length can be longerthan the configured maximum read request size; the procedure breaks the request upinto multiple transactions as needed. This procedure waits until all of the completiondata is returned and places it in shared memory.


Syntax ebfm_barrd_wait(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

continued...



Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.The bar_table structure stores the address assigned to each BAR so thatthe driver code does not need to be aware of the actual assignedaddresses only the application specific offsets from the BAR.



lcladdr BFM shared memory address where the read data is stored.

byte_len Length, in bytes, of the data to be read. Can be 1 to the minimum ofthe bytes remaining in the BAR space or BFM shared memory.

tclass Traffic class used for the PCI Express transaction.

9.7.4 ebfm_barrd_nowt Procedure

The ebfm_barrd_nowt procedure reads a block of data from the offset of thespecified Endpoint BAR and stores the data in BFM shared memory. The length can belonger than the configured maximum read request size; the procedure breaks therequest up into multiple transactions as needed. This routine returns as soon as thelast read transaction has been accepted by the VC interface module, allowingsubsequent reads to be issued immediately.

Location altpcietb_b fm_driver_rp.v

Syntax ebfm_barrd_nowt(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.



lcladdr BFM shared memory address where the read data is stored.

byte_len Length, in bytes, of the data to be read. Can be 1 to the minimum ofthe bytes remaining in the BAR space or BFM shared memory.

tclass Traffic Class to be used for the PCI Express transaction.

9.7.5 ebfm_cfgwr_imm_wait Procedure

The ebfm_cfgwr_imm_wait procedure writes up to four bytes of data to thespecified configuration register. This procedure waits until the write completion hasbeen returned.


Syntax ebfm_cfgwr_imm_wait(bus_num, dev_num, fnc_num, imm_regb_ad, regb_ln, imm_data,compl_status

Arguments bus_num PCI Express bus number of the target device.

dev_num PCI Express device number of the target device.

fnc_num Function number in the target device to be accessed.

regb_ad Byte-specific address of the register to be written.

continued...




regb_ln Length, in bytes, of the data written. Maximum length is four bytes. Theregb_ln and the regb_ad arguments cannot cross a DWORDboundary.

imm_data Data to be written.This argument is reg [31:0].The bits written depend on the length:• 4: 31 downto 0• 3: 23 downto 0• 2: 15 downto 0• 1: 7 downto 0

compl_status This argument is reg [2:0].This argument is the completion status as specified in the PCI Expressspecification. The following encodings are defined:• 3’b000: SC— Successful completion• 3’b001: UR— Unsupported Request• 3’b010: CRS — Configuration Request Retry Status• 3’b100: CA — Completer Abort

9.7.6 ebfm_cfgwr_imm_nowt Procedure

The ebfm_cfgwr_imm_nowt procedure writes up to four bytes of data to thespecified configuration register. This procedure returns as soon as the VC interfacemodule accepts the transaction, allowing other writes to be issued in the interim. Usethis procedure only when successful completion status is expected.


Syntax ebfm_cfgwr_imm_nowt(bus_num, dev_num, fnc_num, imm_regb_adr, regb_len,imm_data)





regb_ln Length, in bytes, of the data written. Maximum length is four bytes, Theregb_ln the regb_ad arguments cannot cross a DWORD boundary.

imm_data Data to be writtenThis argument is reg [31:0].In both languages, the bits written depend on the length. The followingencodes are defined.• 4: [31:0]• 3: [23:0]• 2: [15:0]• 1: [7:0]

9.7.7 ebfm_cfgrd_wait Procedure

The ebfm_cfgrd_wait procedure reads up to four bytes of data from the specifiedconfiguration register and stores the data in BFM shared memory. This procedurewaits until the read completion has been returned.




Syntax ebfm_cfgrd_wait(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr,compl_status)





regb_ln Length, in bytes, of the data read. Maximum length is four bytes. Theregb_ln and the regb_ad arguments cannot cross a DWORDboundary.

lcladdr BFM shared memory address of where the read data should be placed.

compl_status Completion status for the configuration transaction.This argument is reg [2:0].In both languages, this is the completion status as specified in the PCIExpress specification. The following encodings are defined.• 3’b000: SC— Successful completion• 3’b001: UR— Unsupported Request• 3’b010: CRS — Configuration Request Retry Status• 3’b100: CA — Completer Abort

9.7.8 ebfm_cfgrd_nowt Procedure

The ebfm_cfgrd_nowt procedure reads up to four bytes of data from the specifiedconfiguration register and stores the data in the BFM shared memory. This procedurereturns as soon as the VC interface module has accepted the transaction, allowingother reads to be issued in the interim. Use this procedure only when successfulcompletion status is expected and a subsequent read or write with a wait can be usedto guarantee the completion of this operation.


Syntax ebfm_cfgrd_nowt(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr)





regb_ln Length, in bytes, of the data written. Maximum length is four bytes. Theregb_ln and regb_ad arguments cannot cross a DWORD boundary.

lcladdr BFM shared memory address where the read data should be placed.

9.7.9 BFM Configuration Procedures

The BFM configuration procedures are available in altpcietb_bfm_driver_rp.v.These procedures support configuration of the Root Port and Endpoint ConfigurationSpace registers.

All Verilog HDL arguments are type integer and are input-only unless specifiedotherwise.



9.7.9.1 ebfm_cfg_rp_ep Procedure

The ebfm_cfg_rp_ep procedure configures the Root Port and Endpoint ConfigurationSpace registers for operation.


Syntax ebfm_cfg_rp_ep(bar_table, ep_bus_num, ep_dev_num, rp_max_rd_req_size,display_ep_config, addr_map_4GB_limit)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.This routine populates the bar_table structure. The bar_tablestructure stores the size of each BAR and the address values assigned toeach BAR. The address of the bar_table structure is passed to allsubsequent read and write procedure calls that access an offset from aparticular BAR.

ep_bus_num PCI Express bus number of the target device. This number can be anyvalue greater than 0. The Root Port uses this as its secondary busnumber.

ep_dev_num PCI Express device number of the target device. This number can beany value. The Endpoint is automatically assigned this value when itreceives its first configuration transaction.

rp_max_rd_req_size Maximum read request size in bytes for reads issued by the Root Port.This parameter must be set to the maximum value supported by theEndpoint Application Layer. If the Application Layer only supports readsof the MAXIMUM_PAYLOAD_SIZE, then this can be set to 0 and the readrequest size will be set to the maximum payload size. Valid values forthis argument are 0, 128, 256, 512, 1,024, 2,048 and 4,096.

display_ep_config When set to 1 many of the Endpoint Configuration Space registers aredisplayed after they have been initialized, causing some additional readsof registers that are not normally accessed during the configurationprocess such as the Device ID and Vendor ID.

addr_map_4GB_limit When set to 1 the address map of the simulation system will be limitedto 4 GB. Any 64-bit BARs will be assigned below the 4 GB limit.

9.7.9.2 ebfm_cfg_decode_bar Procedure

The ebfm_cfg_decode_bar procedure analyzes the information in the BAR table forthe specified BAR and returns details about the BAR attributes.


Syntax ebfm_cfg_decode_bar(bar_table, bar_num, log2_size, is_mem, is_pref, is_64b)


bar_num BAR number to analyze.

log2_size This argument is set by the procedure to the log base 2 of the size ofthe BAR. If the BAR is not enabled, this argument will be set to 0.

is_mem The procedure sets this argument to indicate if the BAR is a memoryspace BAR (1) or I/O Space BAR (0).

is_pref The procedure sets this argument to indicate if the BAR is a prefetchableBAR (1) or non-prefetchable BAR (0).

is_64b The procedure sets this argument to indicate if the BAR is a 64-bit BAR(1) or 32-bit BAR (0). This is set to 1 only for the lower numbered BARof the pair.



9.7.10 BFM Shared Memory Access Procedures

The BFM shared memory access procedures and functions are in the Verilog HDLinclude file altpcietb_bfm_driver.v. These procedures and functions supportaccessing the BFM shared memory.

9.7.10.1 Shared Memory Constants

The following constants are defined in altpcietb_bfm_driver.v. They select a datapattern in the shmem_fill and shmem_chk_ok routines. These shared memoryconstants are all Verilog HDL type integer.

Table 84. Constants: Verilog HDL Type INTEGER

Constant Description

SHMEM_FILL_ZEROS Specifies a data pattern of all zeros

SHMEM_FILL_BYTE_INC Specifies a data pattern of incrementing 8-bit bytes (0x00, 0x01, 0x02,etc.)

SHMEM_FILL_WORD_INC Specifies a data pattern of incrementing 16-bit words (0x0000, 0x0001,0x0002, etc.)

SHMEM_FILL_DWORD_INC Specifies a data pattern of incrementing 32-bit dwords (0x00000000,0x00000001, 0x00000002, etc.)

SHMEM_FILL_QWORD_INC Specifies a data pattern of incrementing 64-bit qwords(0x0000000000000000, 0x0000000000000001, 0x0000000000000002,etc.)

SHMEM_FILL_ONE Specifies a data pattern of all ones

9.7.10.2 shmem_write

The shmem_write procedure writes data to the BFM shared memory.


Syntax shmem_write(addr, data, leng)

Arguments addr BFM shared memory starting address for writing data

data Data to write to BFM shared memory.This parameter is implemented as a 64-bit vector. leng is 1–8 bytes.Bits 7 downto 0 are written to the location specified by addr; bits 15downto 8 are written to the addr+1 location, etc.

length Length, in bytes, of data written

9.7.10.3 shmem_read Function

The shmem_read function reads data to the BFM shared memory.


Syntax data:= shmem_read(addr, leng)

Arguments addr BFM shared memory starting address for reading data

leng Length, in bytes, of data read

Return data Data read from BFM shared memory.

continued...




This parameter is implemented as a 64-bit vector. leng is 1- 8 bytes. Ifleng is less than 8 bytes, only the corresponding least significant bits ofthe returned data are valid.Bits 7 downto 0 are read from the location specified by addr; bits 15downto 8 are read from the addr+1 location, etc.

9.7.10.4 shmem_display Verilog HDL Function

The shmem_display Verilog HDL function displays a block of data from the BFMshared memory.


Syntax Verilog HDL: dummy_return:=shmem_display(addr, leng, word_size, flag_addr,msg_type);

Arguments addr BFM shared memory starting address for displaying data.

leng Length, in bytes, of data to display.

word_size Size of the words to display. Groups individual bytes into words. Validvalues are 1, 2, 4, and 8.

flag_addr Adds a <== flag to the end of the display line containing this address.Useful for marking specific data. Set to a value greater than 2**21 (sizeof BFM shared memory) to suppress the flag.

msg_type Specifies the message type to be displayed at the beginning of eachline. See “BFM Log and Message Procedures” on page 18–37 for moreinformation about message types. Set to one of the constants defined inTable 18–36 on page 18–41.

9.7.10.5 shmem_fill Procedure

The shmem_fill procedure fills a block of BFM shared memory with a specified datapattern.


Syntax shmem_fill(addr, mode, leng, init)

Arguments addr BFM shared memory starting address for filling data.

mode Data pattern used for filling the data. Should be one of the constantsdefined in section Shared Memory Constants.

leng Length, in bytes, of data to fill. If the length is not a multiple of theincrementing data pattern width, then the last data pattern is truncatedto fit.

init Initial data value used for incrementing data pattern modes. Thisargument is reg [63:0].The necessary least significant bits are used for the data patterns thatare smaller than 64 bits.

Related Links

Shared Memory Constants on page 136

9.7.10.6 shmem_chk_ok Function

The shmem_chk_ok function checks a block of BFM shared memory against aspecified data pattern.



Location altpcietb_bfm_shmem.v

Syntax result:= shmem_chk_ok(addr, mode, leng, init, display_error)

Arguments addr BFM shared memory starting address for checking data.

mode Data pattern used for checking the data. Should be one of the constantsdefined in section “Shared Memory Constants” on page 18–35.

leng Length, in bytes, of data to check.

init This argument is reg [63:0].The necessary least significant bits areused for the data patterns that are smaller than 64-bits.

display_error When set to 1, this argument displays the miscomparing data on thesimulator standard output.

Return Result Result is 1-bit.• 1’b1 — Data patterns compared successfully• 1’b0 — Data patterns did not compare successfully

9.7.11 BFM Log and Message Procedures

The following procedures and functions are available in the Verilog HDL include filealtpcietb_bfm_driver_rp.v.

These procedures provide support for displaying messages in a common format,suppressing informational messages, and stopping simulation on specific messagetypes.

The following constants define the type of message and their values determinewhether a message is displayed or simulation is stopped after a specific message.Each displayed message has a specific prefix, based on the message type in thefollowing table.

You can suppress the display of certain message types. The default values determiningwhether a message type is displayed are defined in the following table. To change thedefault message display, modify the display default value with a procedure call toebfm_log_set_suppressed_msg_mask.

Certain message types also stop simulation after the message is displayed. Thefollowing table shows the default value determining whether a message type stopssimulation. You can specify whether simulation stops for particular messages with theprocedure ebfm_log_set_stop_on_msg_mask.

All of these log message constants type integer.

Table 85. Log Messages

Constant(Message

Type)

Description Mask BitNo

Displayby Default

SimulationStops byDefault

MessagePrefix

EBFM_MSG_DEBUG

Specifies debug messages. 0 No No DEBUG:

EBFM_MSG_INFO

Specifies informational messages,such as configuration registervalues, starting and ending of tests.

1 Yes No INFO:

continued...



Constant(Message

Type)

Description Mask BitNo

Displayby Default

SimulationStops byDefault

MessagePrefix

EBFM_MSG_WARNING

Specifies warning messages, suchas tests being skipped due to thespecific configuration.

2 Yes No WARNING:

EBFM_MSG_ERROR_INFO

Specifies additional information foran error. Use this message todisplay preliminary informationbefore an error message that stopssimulation.

3 Yes No ERROR:

EBFM_MSG_ERROR_CONTINUE

Specifies a recoverable error thatallows simulation to continue. Usethis error for data miscompares.

4 Yes No ERROR:

EBFM_MSG_ERROR_FATAL

Specifies an error that stopssimulation because the error leavesthe testbench in a state wherefurther simulation is not possible.

N/A YesCannotsuppress

YesCannot suppress

FATAL:

EBFM_MSG_ERROR_FATAL_TB_ERR

Used for BFM test driver or RootPort BFM fatal errors. Specifies anerror that stops simulation becausethe error leaves the testbench in astate where further simulation isnot possible. Use this errormessage for errors that occur dueto a problem in the BFM test drivermodule or the Root Port BFM, thatare not caused by the EndpointApplication Layer being tested.

N/A YCannotsuppress

YCannot suppress

FATAL:

9.7.11.1 ebfm_display Verilog HDL Function

The ebfm_display procedure or function displays a message of the specified type tothe simulation standard output and also the log file if ebfm_log_open is called.

A message can be suppressed, simulation can be stopped or both based on the defaultsettings of the message type and the value of the bit mask when each of theprocedures listed below is called. You can call one or both of these procedures basedon what messages you want displayed and whether or not you want simulation to stopfor specific messages.

• When ebfm_log_set_suppressed_msg_mask is called, the display of themessage might be suppressed based on the value of the bit mask.

• When ebfm_log_set_stop_on_msg_mask is called, the simulation can bestopped after the message is displayed, based on the value of the bit mask.


Syntax Verilog HDL: dummy_return:=ebfm_display(msg_type, message);

Argument msg_type Message type for the message. Should be one of the constants definedin Table 18–36 on page 18–41.

message The message string is limited to a maximum of 100 characters. Also,because Verilog HDL does not allow variable length strings, this routinestrips off leading characters of 8’h00 before displaying the message.

Return always 0 Applies only to the Verilog HDL routine.



9.7.11.2 ebfm_log_stop_sim Verilog HDL Function

The ebfm_log_stop_sim procedure stops the simulation.


Syntax Verilog HDL: return:=ebfm_log_stop_sim(success);

Argument success When set to a 1, this process stops the simulation with a messageindicating successful completion. The message is prefixed withSUCCESS:.Otherwise, this process stops the simulation with a message indicatingunsuccessful completion. The message is prefixed with FAILURE:.

Return Always 0 This value applies only to the Verilog HDL function.

9.7.11.3 ebfm_log_set_suppressed_msg_mask #Verilog HDL Function

The ebfm_log_set_suppressed_msg_mask procedure controls which messagetypes are suppressed.


Syntax bfm_log_set_suppressed_msg_mask (msg_mask)

Argument msg_mask This argument is reg [EBFM_MSG_ERROR_CONTINUE:EBFM_MSG_DEBUG].

A 1 in a specific bit position of the msg_mask causes messages of thetype corresponding to the bit position to be suppressed.

9.7.11.4 ebfm_log_set_stop_on_msg_mask Verilog HDL Function

The ebfm_log_set_stop_on_msg_mask procedure controls which message typesstop simulation. This procedure alters the default behavior of the simulation whenerrors occur as described in the BFM Log and Message Procedures.


Syntax ebfm_log_set_stop_on_msg_mask (msg_mask)

Argument msg_mask This argument is reg[EBFM_MSG_ERROR_CONTINUE:EBFM_MSG_DEBUG].A 1 in a specific bit position of the msg_mask causes messages of thetype corresponding to the bit position to stop the simulation after themessage is displayed.

Related Links

BFM Log and Message Procedures on page 138

9.7.11.5 ebfm_log_open Verilog HDL Function

The ebfm_log_open procedure opens a log file of the specified name. All displayedmessages are called by ebfm_display and are written to this log file as simulatorstandard output.


Syntax ebfm_log_open (fn)

Argument fn This argument is type string and provides the file name of log file tobe opened.



9.7.11.6 ebfm_log_close Verilog HDL Function

The ebfm_log_close procedure closes the log file opened by a previous call toebfm_log_open.


Syntax ebfm_log_close

Argument NONE

9.7.12 Verilog HDL Formatting Functions

The Verilog HDL Formatting procedures and functions are available in thealtpcietb_bfm_driver_rp.v. The formatting functions are only used by Verilog HDL.All these functions take one argument of a specified length and return a vector of aspecified length.

9.7.12.1 himage1

This function creates a one-digit hexadecimal string representation of the inputargument that can be concatenated into a larger message string and passed toebfm_display.


Syntax string:= himage(vec)

Argument vec Input data type reg with a range of 3:0.

Return range string Returns a 1-digit hexadecimal representation of the input argument.Return data is type reg with a range of 8:1

9.7.12.2 himage2

This function creates a two-digit hexadecimal string representation of the inputargument that can be concatenated into a larger message string and passed toebfm_display.



Argument range vec Input data type reg with a range of 7:0.

Return range string Returns a 2-digit hexadecimal presentation of the input argument,padded with leading 0s, if they are needed. Return data is type reg witha range of 16:1

9.7.12.3 himage4

This function creates a four-digit hexadecimal string representation of the inputargument can be concatenated into a larger message string and passed toebfm_display.






Return range Returns a four-digit hexadecimal representation of the input argument, padded with leading 0s, if theyare needed. Return data is type reg with a range of 32:1.

9.7.12.4 himage8

This function creates an 8-digit hexadecimal string representation of the inputargument that can be concatenated into a larger message string and passed toebfm_display.




Return range string Returns an 8-digit hexadecimal representation of the input argument,padded with leading 0s, if they are needed. Return data is type reg witha range of 64:1.

9.7.12.5 himage16

This function creates a 16-digit hexadecimal string representation of the inputargument that can be concatenated into a larger message string and passed toebfm_display.




Return range string Returns a 16-digit hexadecimal representation of the input argument,padded with leading 0s, if they are needed. Return data is type reg witha range of 128:1.

9.7.12.6 dimage1

This function creates a one-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.


Syntax string:= dimage(vec)


Return range string Returns a 1-digit decimal representation of the input argument that ispadded with leading 0s if necessary. Return data is type reg with arange of 8:1.Returns the letter U if the value cannot be represented.

9.7.12.7 dimage2

This function creates a two-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.







9.7.12.8 dimage3

This function creates a three-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.





9.7.12.9 dimage4

This function creates a four-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.





9.7.12.10 dimage5

This function creates a five-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.







9.7.12.11 dimage6

This function creates a six-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.

Location altpcietb_bfm_log.v




9.7.12.12 dimage7

This function creates a seven-digit decimal string representation of the input argumentthat can be concatenated into a larger message string and passed to ebfm_display.

Location altpcietb_bfm_log.v



Return range string Returns a 7-digit decimal representation of the input argument that ispadded with leading 0s if necessary. Return data is type reg with arange of 56:1.Returns the letter <U> if the value cannot be represented.

9.7.13 Procedures and Functions Specific to the Chaining DMA DesignExample

The procedures specific to the chaining DMA design example are in the Verilog HDLmodule file altpcietb_bfm_driver_rp.v.

9.7.13.1 chained_dma_test Procedure

The chained_dma_test procedure is the top-level procedure that runs the chainingDMA read and the chaining DMA write


Syntax chained_dma_test (bar_table, bar_num, direction, use_msi, use_eplast)



direction When 0 the direction is read.When 1 the direction is write.

Use_msi When set, the Root Port uses native PCI Express MSI to detect the DMAcompletion.

Use_eplast When set, the Root Port uses BFM shared memory polling to detect theDMA completion.



9.7.13.2 dma_rd_test Procedure

Use the dma_rd_test procedure for DMA reads from the Endpoint memory to theBFM shared memory.


Syntax dma_rd_test (bar_table, bar_num, use_msi, use_eplast)



Use_msi When set, the Root Port uses native PCI express MSI to detect the DMAcompletion.


9.7.13.3 dma_wr_test Procedure

Use the dma_wr_test procedure for DMA writes from the BFM shared memory to theEndpoint memory.


Syntax dma_wr_test (bar_table, bar_num, use_msi, use_eplast)





9.7.13.4 dma_set_rd_desc_data Procedure

Use the dma_set_rd_desc_data procedure to configure the BFM shared memory forthe DMA read.


Syntax dma_set_rd_desc_data (bar_table, bar_num)



9.7.13.5 dma_set_wr_desc_data Procedure

Use the dma_set_wr_desc_data procedure to configure the BFM shared memory forthe DMA write.


Syntax dma_set_wr_desc_data_header (bar_table, bar_num)





9.7.13.6 dma_set_header Procedure

Use the dma_set_header procedure to configure the DMA descriptor table for DMAread or DMA write.


Syntax dma_set_header (bar_table, bar_num, Descriptor_size, direction, Use_msi,Use_eplast, Bdt_msb, Bdt_lab, Msi_number, Msi_traffic_class,Multi_message_enable)



Descriptor_size Number of descriptor.

direction When 0 the direction is read.When 1 the direction is write.



Bdt_msb BFM shared memory upper address value.

Bdt_lsb BFM shared memory lower address value.

Msi_number When use_msi is set, specifies the number of the MSI which is set bythe dma_set_msi procedure.

Msi_traffic_class When use_msi is set, specifies the MSI traffic class which is set by thedma_set_msi procedure.

Multi_message_enable When use_msi is set, specifies the MSI traffic class which is set by thedma_set_msi procedure.

9.7.13.7 rc_mempoll Procedure

Use the rc_mempoll procedure to poll a given dword in a given BFM shared memorylocation.


Syntax rc_mempoll (rc_addr, rc_data, rc_mask)

Arguments rc_addr Address of the BFM shared memory that is being polled.

rc_data Expected data value of the that is being polled.

rc_mask Mask that is logically ANDed with the shared memory data before it iscompared with rc_data.

9.7.13.8 msi_poll Procedure

The msi_poll procedure tracks MSI completion from the Endpoint.


Syntax msi_poll(max_number_of_msi,msi_address,msi_expected_dmawr,msi_expected_dmard,dma_write,dma_read)

Arguments max_number_of_msi Specifies the number of MSI interrupts to wait for.

continued...




msi_address The shared memory location to which the MSI messages will be written.

msi_expected_dmawr When dma_write is set, this specifies the expected MSI data value forthe write DMA interrupts which is set by the dma_set_msi procedure.

msi_expected_dmard When the dma_read is set, this specifies the expected MSI data valuefor the read DMA interrupts which is set by the dma_set_msiprocedure.

Dma_write When set, poll for MSI from the DMA write module.

Dma_read When set, poll for MSI from the DMA read module.

9.7.13.9 dma_set_msi Procedure

The dma_set_msi procedure sets PCI Express native MSI for the DMA read or theDMA write.


Syntax dma_set_msi(bar_table, bar_num, bus_num, dev_num, fun_num, direction,msi_address, msi_data, msi_number, msi_traffic_class, multi_message_enable,msi_expected)



Bus_num Set configuration bus number.

dev_num Set configuration device number.

Fun_num Set configuration function number.

Direction When 0 the direction is read.When 1 the direction is write.

msi_address Specifies the location in shared memory where the MSI message datawill be stored.

msi_data The 16-bit message data that will be stored when an MSI message issent. The lower bits of the message data will be modified with themessage number as per the PCI specifications.

Msi_number Returns the MSI number to be used for these interrupts.

Msi_traffic_class Returns the MSI traffic class value.

Multi_message_enable Returns the MSI multi message enable status.

msi_expected Returns the expected MSI data value, which is msi_data modified bythe msi_number chosen.

9.7.13.10 find_mem_bar Procedure

The find_mem_bar procedure locates a BAR which satisfies a given memory spacerequirement.


Syntax Find_mem_bar(bar_table,allowed_bars,min_log2_size, sel_bar)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory

continued...




allowed_bars One hot 6 bits BAR selection

min_log2_size Number of bit required for the specified address space

sel_bar BAR number to use

9.7.13.11 dma_set_rclast Procedure

The dma_set_rclast procedure starts the DMA operation by writing to the EndpointDMA register the value of the last descriptor to process (RCLast).


Syntax Dma_set_rclast(bar_table, setup_bar, dt_direction, dt_rclast)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory

setup_bar BAR number to use

dt_direction When 0 read, When 1 write

dt_rclast Last descriptor number

9.7.13.12 ebfm_display_verb Procedure

The ebfm_display_verb procedure calls the procedure ebfm_display when theglobal variable DISPLAY_ALL is set to 1.

Location altpcietb_bfm_driver_chaining.v

Syntax ebfm_display_verb(msg_type, message)

Arguments msg_type Message type for the message. Should be one of the constants definedin BFM Log and Message Procedures.

message The message string is limited to a maximum of 100 characters. Also,because Verilog HDL does not allow variable length strings, this routinestrips off leading characters of 8'h00 before displaying the message.

Related Links

BFM Log and Message Procedures on page 138

9.8 Setting Up Simulation

Changing the simulation parameters reduces simulation time and provides greatervisibility.

9.8.1 Changing Between Serial and PIPE Simulation

By default, the Intel testbench runs a serial simulation. You can change between serialand PIPE simulation by editing the top-level testbench file. For Endpoint designs, thetop-level testbench file is

The serial_sim_hwtcl and enable_pipe32_sim_hwtcl parameters control serialmode or PIPE simulation mode. To change to PIPE mode, changeenable_pipe32_sim_hwtcl to 1'b1 and serial_sim_hwtcl to 1'b0.



Table 86. Controlling Serial and PIPE Simulations

Data Rates Parameter Settings

serial_sim_hwtcl enable_pipe32_sim_hwtcl

Serial simulation 1 0

PIPE simulation 0 1

9.8.2 Using the PIPE Interface for Gen1 and Gen2 Variants

Running the simulation in PIPE mode reduces simulation time and provides greatervisibility.

Complete the following steps to simulate using the PIPE interface:

1. Change to your simulation directory, <work_dir>/<variant>/testbench/<variant>_tb/simulation

2. Open <variant>_tb.v.

3. Search for the string, serial_sim_hwtcl. Set the value of this parameter to 0 ifit is 1.

4. Save <variant>_tb.v.

9.8.3 Viewing the Important PIPE Interface Signals

You can view the most important PIPE interface signals, txdata, txdatak, rxdata,and rxdatak at the following level of the design hierarchy:altpcie_<device>_hip_pipen1b|twentynm_hssi_<gen>_<lanes>_pcie_hip.

9.8.4 Disabling the Scrambler for Gen1 and Gen2 Simulations

The encoding scheme implemented by the scrambler applies a binary polynomial tothe data stream to ensure enough data transitions between 0 and 1 to prevent clockdrift. The data is decoded at the other end of the link by running the inversepolynomial.

Complete the following steps to disable the scrambler:

1. Open <work_dir>/<variant>/testbench/<variant>_tb/simulation/submodules/altpcie_tbed_<dev>_hwtcl.v.

2. Search for the string, test_in.

3. To disable the scrambler, set test_in[2] = 1.

4. Save altpcie_tbed_sv_hwtcl.v.

9.8.5 Disabling 8B/10B Encoding and Decoding for Gen1 and Gen2Simulations

You can disable 8B/10B encoding and decoding to facilitate debugging.

For Gen1 and Gen2 variants, you can disable 8B/10B encoding and decoding bysetting test_in[2] = 1 in altpcietb_bfm_top_rp.v.



A Troubleshooting and Observing the Link

A.1 Simulation Fails To Progress Beyond Polling.Active State

If your PIPE simulation cycles between the Detect.Quiet, Detect.Active, andPolling.Active LTSSM states, the PIPE interface width may be incorrect. The width ofthe DUT top-level PIPE interface is 32 bits for Stratix 10 devices.

A.2 Hardware Bring-Up Issues

Typically, PCI Express hardware bring-up involves the following steps:

1. System reset

2. Link training

3. BIOS enumeration

The following sections describe how to debug the hardware bring-up flow. Intelrecommends a systematic approach to diagnosing bring-up issues as illustrated in thefollowing figure.

Figure 78. Debugging Link Training Issues

No

System Reset Does Link Train

Correctly?

Check PIPEInterface

Use PCIe Protocol Analyzer

Soft Reset System to Force Enumeration

Check Configuration Space

Check LTSSMStatus

YesYes

No

SuccessfulOS/BIOS

Enumeration?

A.3 Link Training

The Physical Layer automatically performs link training and initialization withoutsoftware intervention. This is a well-defined process to configure and initialize thedevice's Physical Layer and link so that PCIe packets can be transmitted. If youencounter link training issues, viewing the actual data in hardware should help youdetermine the root cause. You can use the following tools to provide hardwarevisibility:

• SignalTap II Embedded Logic Analyzer

• Third-party PCIe protocol analyzer







You can use SignalTap II Embedded Logic Analyzer to diagnose the LTSSM statetransitions that are occurring on the PIPE interface. The ltssmstate bus encodes thestatus of LTSSM. The LTSSM state machine reflects the Physical Layer’s progressthrough the link training process. For a complete description of the states thesesignals encode, refer to Reset, Status, and Link Training Signals. When link trainingcompletes successfully and the link is up, the LTSSM should remain stable in the L0state. When link issues occur, you can monitor ltssmstate to determine the cause.

A.4 Link Hangs in L0 State

There are many reasons that link may stop transmitting data. The following table listssome possible causes.

Table 87. Link Hangs in L0

Possible Causes Symptoms and Root Causes Workarounds and Solutions

Avalon-ST signalingviolates Avalon-STprotocol

Avalon-ST protocol violationsinclude the following errors:• More than one tx_st_sop per

tx_st_eop.• Two or more tx_st_eop’s

without a correspondingtx_st_sop.

• rx_st_valid is not assertedwith tx_st_sop ortx_st_eop.

These errors are applicable to bothsimulation and hardware.

Add logic to detect situations where tx_st_readyremains deasserted for more than 100 cycles. Setpost-triggering conditions to check for the Avalon-STsignaling of last two TLPs to verify correct tx_st_sopand tx_st_eop signaling.

Incorrect payload size Determine if the length field of thelast TLP transmitted by End Pointis greater than the InitFC creditadvertised by the link partner. Forsimulation, refer to the log file andsimulation dump. For hardware,use a third-party logic analyzertrace to capture PCIe transactions.

If the payload is greater than the initFC creditadvertised, you must either increase the InitFC of theposted request to be greater than the max payloadsize or reduce the payload size of the requested TLP tobe less than the InitFC value.

Flow control creditoverflows

Determine if the credit fieldassociated with the current TLPtype in the tx_cred bus is lessthan the requested credit value.When insufficient credits areavailable, the core waits for thelink partner to release the correctcredit type. Sufficient credits maybe unavailable if the link partnerincrements credits more thanexpected, creating a situationwhere the Stratix 10 Hard IP forPCI Express IP Core creditcalculation is out-of-sync with itslink partner.

Add logic to detect conditions where the tx_st_readysignal remains deasserted for more than 100 cycles.Set post-triggering conditions to check the value of thetx_cred_* and tx_st_* interfaces. Add a FIFO statussignal to determine if the TXFIFO is full.

Malformed TLP istransmitted

Refer to the error log file to findthe last good packet transmittedon the link. Correlate this packetwith TLP sent on Avalon-ST

Revise the Application Layer logic to correct the errorcondition.

continued...



Possible Causes Symptoms and Root Causes Workarounds and Solutions

interface. Determine if the last TLPsent has any of the followingerrors:• The actual payload sent does

not match the length field.• The format and type fields are

incorrectly specified.• TD field is asserted, indicating

the presence of a TLP digest(ECRC), but the ECRC dword isnot present at the end of TLP.

• The payload crosses a 4KByteboundary.

Insufficient Posted creditsreleased by Root Port

If a Memory Write TLP istransmitted with a payload greaterthan the maximum payload size,the Root Port may release anincorrect posted data credit to theEndpoint in simulation. As a result,the Endpoint does not haveenough credits to send additionalMemory Write Requests.

Make sure Application Layer sends Memory WriteRequests with a payload less than or equal the valuespecified by the maximum payload size.

Missing completionpackets or droppedpackets

The RX Completion TLP mightcause the RX FIFO to overflow.Make sure that the totaloutstanding read data of allpending Memory Read Requests issmaller than the allocatedcompletion credits in RX buffer.

You must ensure that the data for all outstanding readrequests does not exceed the completion credits in theRX buffer.

Related Links

• Design Debugging with the SignalTap II Logic Analyzer


A.5 Use Third-Party PCIe Analyzer

A third-party logic analyzer for PCI Express records the traffic on the physical link anddecodes traffic, saving you the trouble of translating the symbols yourself. Athird-party logic analyzer can show the two-way traffic at different levels for differentrequirements. For high-level diagnostics, the analyzer shows the LTSSM flows fordevices on both side of the link side-by-side. This display can help you see the linktraining handshake behavior and identify where the traffic gets stuck. A trafficanalyzer can display the contents of packets so that you can verify the contents. Forcomplete details, refer to the third-party documentation.

A.6 BIOS Enumeration Issues

Both FPGA programming (configuration) and the initialization of a PCIe link requiretime. Potentially, an Intel FPGA including a Hard IP block for PCI Express may not beready when the OS/BIOS begins enumeration of the device tree. If the FPGA is notfully programmed when the OS/BIOS begins its enumeration, the OS does not includethe Hard IP for PCI Express in its device map.





You can use either of the following two methods to eliminate this issue:

• You can perform a soft reset of the system to retain the FPGA programming whileforcing the OS/BIOS to repeat its enumeration.

• You can use CvP to program the device. CvP is not available in the Quartus PrimePro Edition Software v17.0/17.1 Stratix 10 ES Version.



B PCI Express Core Architecture

B.1 Transaction Layer

The Transaction Layer is located between the Application Layer and the Data LinkLayer. It generates and receives Transaction Layer Packets. The following illustratesthe Transaction Layer. The Transaction Layer includes three sub-blocks: the TXdatapath, Configuration Space, and RX datapath.

Tracing a transaction through the RX datapath includes the following steps:

1. The Transaction Layer receives a TLP from the Data Link Layer.

2. The Configuration Space determines whether the TLP is well formed and directsthe packet based on traffic class (TC).

3. TLPs are stored in a specific part of the RX buffer depending on the type oftransaction (posted, non-posted, and completion).

4. The TLP FIFO block stores the address of the buffered TLP.

5. The receive reordering block reorders the queue of TLPs as needed, fetches theaddress of the highest priority TLP from the TLP FIFO block, and initiates thetransfer of the TLP to the Application Layer.

6. When ECRC generation and forwarding are enabled, the Transaction Layerforwards the ECRC dword to the Application Layer.

Tracing a transaction through the TX datapath involves the following steps:

1. The Transaction Layer informs the Application Layer that sufficient flow controlcredits exist for a particular type of transaction using the TX credit signals. TheApplication Layer may choose to ignore this information.

2. The Application Layer requests permission to transmit a TLP. The Application Layermust provide the transaction and must be prepared to provide the entire datapayload in consecutive cycles.

3. The Transaction Layer verifies that sufficient flow control credits exist andacknowledges or postpones the request.

4. The Transaction Layer forwards the TLP to the Data Link Layer.







Figure 79. Architecture of the Transaction Layer: Dedicated Receive Buffer

Transaction Layer TX Datapath

Transaction Layer RX Datapath

Avalon-STRX Control

Configuration Space

TLPs to theData Link Layer

TLPs from the Data Link Layer

Avalon-ST RX Data

Avalon-STTX Data

to Application Layer

ConfigurationRequests

Reordering

RX Buffer

Posted & Completion

Non-Posted

Flow Control Update

Transaction Layer Packet FIFO

WidthAdapter( <256

bits)

PacketAlignment TX

Control

RXControl

TX Flow Control

B.2 Data Link Layer

The Data Link Layer is located between the Transaction Layer and the Physical Layer.It maintains packet integrity and communicates (by DLL packet transmission) at thePCI Express link level.

The DLL implements the following functions:

• Link management through the reception and transmission of DLL packets (DLLP),which are used for the following functions:

— Power management of DLLP reception and transmission

— To transmit and receive ACK/NAK packets

— Data integrity through generation and checking of CRCs for TLPs and DLLPs

— TLP retransmission in case of NAK DLLP reception or replay timeout, using theretry (replay) buffer

— Management of the retry buffer

— Link retraining requests in case of error through the Link Training and StatusState Machine (LTSSM) of the Physical Layer



Figure 80. Data Link LayerTo Transaction Layer

Tx Transaction LayerPacket Description & Data Transaction Layer

Packet Generator

Retry Buffer

To Physical Layer

Tx Packets

Ack/NackPackets

RX Datapath

TX Datapath

Rx Packets

DLLPChecker

Transaction LayerPacket Checker

DLLPGenerator

Tx Arbitration

Data Link Controland Management

State Machine

Control& StatusConfiguration Space

Tx Flow Control Credit Information

Rx Flow Control Credit Information

Rx Transation LayerPacket Description & Data

PowerManagement

Function

Note:(1) The L0s (Standby) or L1 (Low Power Standby) states are not supported. The DLL has the following sub-blocks:

• Data Link Control and Management State Machine—This state machine connects toboth the Physical Layer’s LTSSM state machine and the Transaction Layer. Itinitializes the link and flow control credits and reports status to the TransactionLayer.

• Power Management—This function handles the handshake to enter low powermode. Such a transition is based on register values in the Configuration Space andreceived Power Management (PM) DLLPs. None of Stratix 10 Hard IP for PCIe IPcore variants support low power modes.

• Data Link Layer Packet Generator and Checker—This block is associated with theDLLP’s 16-bit CRC and maintains the integrity of transmitted packets.

• Transaction Layer Packet Generator—This block generates transmit packets,including a sequence number and a 32-bit Link CRC (LCRC). The packets are alsosent to the retry buffer for internal storage. In retry mode, the TLP generatorreceives the packets from the retry buffer and generates the CRC for the transmitpacket.

• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledgedpackets in the case of NAK DLLP reception. In case of ACK DLLP reception, theretry buffer discards all acknowledged packets.



• ACK/NAK Packets—The ACK/NAK block handles ACK/NAK DLLPs and generates thesequence number of transmitted packets.

• Transaction Layer Packet Checker—This block checks the integrity of the receivedTLP and generates a request for transmission of an ACK/NAK DLLP.

• TX Arbitration—This block arbitrates transactions, prioritizing in the followingorder:

— Initialize FC Data Link Layer packet

— ACK/NAK DLLP (high priority)

— Update FC DLLP (high priority)

— PM DLLP

— Retry buffer TLP

— TLP

— Update FC DLLP (low priority)

— ACK/NAK FC DLLP (low priority)

B.3 Physical Layer

The Physical Layer is the lowest level of the PCI Express protocol stack. It is the layerclosest to the serial link. It encodes and transmits packets across a link and acceptsand decodes received packets. The Physical Layer connects to the link through ahigh-speed SERDES interface running at 2.5 Gbps for Gen1 implementations, at 2.5 or5.0 Gbps for Gen2 implementations, and at 2.5, 5.0 or 8.0 Gbps for Gen3implementations.

The Physical Layer is responsible for the following actions:

• Training the link

• Scrambling/descrambling and 8B/10B encoding/decoding for 2.5 Gbps (Gen1),5.0 Gbps (Gen2), or 128b/130b encoding/decoding of 8.0 Gbps (Gen3) per lane

• Serializing and deserializing data

• Equalization (Gen3)

• Operating the PIPE 3.0 Interface

• Implementing auto speed negotiation (Gen2 and Gen3)

• Transmitting and decoding the training sequence

• Providing hardware autonomous speed control

• Implementing auto lane reversal

The Physical Layer is subdivided by the PIPE Interface Specification into two layers(bracketed horizontally in above figure):

• PHYMAC—The MAC layer includes the LTSSM and the scrambling/descrambling.byte reordering, and multilane deskew functions.

• PHY Layer—The PHY layer includes the 8B/10B encode and decode functions forGen1 and Gen2. It includes 128b/130b encode and decode functions for Gen3.The PHY also includes elastic buffering and serialization/deserialization functions.



The Physical Layer integrates both digital and analog elements. Intel designed thePIPE interface to separate the PHYMAC from the PHY. The Stratix 10 Hard IP for PCIExpress complies with the PIPE interface specification.

Note: The internal PIPE interface is visible for simulation. It is not available for debugging inhardware using a logic analyzer such as SignalTap® II. If you try to connect SignalTapII to this interface you will not be able to compile your design.



Figure 81. Physical Layer Architecture

TX+ / TX-

TX+ / TX-

Descrambler 8B/10BDecoder

128b/130bEncoder/Decoder

Lane n

RX+ / RX-ElasticBuffer

LTSSM State Machine

SKIPGeneration

Control & StatusRX MAC

Gen1, Gen2

Gen1, Gen2

Gen1, Gen2

Gen1, Gen2

Gen3

Gen3

TX MAC

Link S

erial

izer

for a

n x8 L

inkTX Packets

Dev

ice T

rans

ceive

r (pe

r Lan

e) w

ith 2.

5, 5.0

, or 8

Gbp

s SER

DES &

PLL

Descrambler

Lane 0RX+ / RX-

PIPEInterface

Byte

Reor

der &

Mult

ilane

Des

kew

Link S

erial

izer f

or an

x8 Li

nk

RX Packets

TransmitData Path

ReceiveData Path

PHYMAC Layer PHY Layer

To LinkTo Data Link Layer

8B10BDecoder

128b/130bEncoder/Decoder

ElasticBuffer

Lane 0(or) 128b/130b

Encoder/Decoder

Scrambler 8B/10BEncoder

Gen3

Lane n(or) 128b/130b

Encoder/Decoder

8B/10BEncoder

Gen3

Scrambler

The PHYMAC block comprises four main sub-blocks:

• MAC Lane—Both the RX and the TX path use this block.

— On the RX side, the block decodes the Physical Layer packet and reports to theLTSSM the type and number of TS1/TS2 ordered sets received.

— On the TX side, the block multiplexes data from the DLL and the Ordered Setand SKP sub-block (LTSTX). It also adds lane specific information, includingthe lane number and the force PAD value when the LTSSM disables the laneduring initialization.

• LTSSM—This block implements the LTSSM and logic that tracks TX and RX trainingsequences on each lane.

• For transmission, it interacts with each MAC lane sub-block and with the LTSTXsub-block by asserting both global and per-lane control bits to generate specificPhysical Layer packets.

— On the receive path, it receives the Physical Layer packets reported by eachMAC lane sub-block. It also enables the multilane deskew block. This blockreports the Physical Layer status to higher layers.

— LTSTX (Ordered Set and SKP Generation)—This sub-block generates thePhysical Layer packet. It receives control signals from the LTSSM block andgenerates Physical Layer packet for each lane. It generates the same PhysicalLayer Packet for all lanes and PAD symbols for the link or lane number in thecorresponding TS1/TS2 fields. The block also handles the receiver detectionoperation to the PCS sub-layer by asserting predefined PIPE signals andwaiting for the result. It also generates a SKP Ordered Set at every predefinedtimeslot and interacts with the TX alignment block to prevent the insertion of aSKP Ordered Set in the middle of packet.

— Deskew—This sub-block performs the multilane deskew function and the RXalignment between the initialized lanes and the datapath. The multilanedeskew implements an eight-word FIFO buffer for each lane to store symbols.Each symbol includes eight data bits, one disparity bit, and one control bit.The FIFO discards the FTS, COM, and SKP symbols and replaces PAD and IDLwith D0.0 data. When all eight FIFOs contain data, a read can occur. When themultilane lane deskew block is first enabled, each FIFO begins writing after thefirst COM is detected. If all lanes have not detected a COM symbol after sevenclock cycles, they are reset and the resynchronization process restarts, or elsethe RX alignment function recreates a 64-bit data word which is sent to theDLL.



C TX Credit Adjustment Sample CodeThis sample Verilog HDL code computes the available credits for non-posted TLPs. Itprovides the updated credit information from the remote device on the tx_nph_cdtsand tx_npd_cdts buses. The tx_nph_cdts and tx_npd_cdts buses drive theactual available credit space in the link partner's RX buffer. The actual credit space isdifficult to use due to the EMIB (Embedded Multi-die Interconnect Bridge) latency.

The following Verilog RTL restores the credit limit for non-posted TLPs that can beused by application logic before it sends a TLP.

module nph_credit_limit_gen ( input clk, input rst_n, input [7:0] tx_nph_cdts, input tx_hdr_cdts_consumed, input [1:0] tx_cdts_type, output [7:0] tx_nph_cdts_limit); reg tx_nph_credit_consume_1r; reg tx_nph_credit_consume_2r; reg [7:0] tx_nph_credit_consume_count_hip_3r; always @(posedge clk) begin if (!rst_n) begin tx_nph_credit_consume_1r <= 1'b0; tx_nph_credit_consume_2r <= 1'b0; tx_nph_credit_consume_count_hip <= 8'h0;

end else begin tx_nph_credit_consume_1r <= (tx_cdts_type == 2'b01) ? tx_hdr_cdts_consumed : 1'b0; tx_nph_credit_consume_2r <= tx_nph_credit_consume_1r; tx_nph_credit_consume_count_hip_3r <= tx_nph_credit_consume_count_hip_3r + tx_nph_credit_consume_2r; end end assign tx_nph_cdts_limit = tx_nph_cdts + tx_nph_credit_consume_count_hip_3r; endmodule

The following pseudo-code explains the Verilog RTL above.

// reset credit_consume_count initiallytx_nph_credit_consume_count = 0; tx_nph_credit_consume_count_hip = 0;

if (tx_nph_credit_limit_count – (tx_nph_credit_consume_count + tx_nph_credit_required) <= (2^8)/2) { send NPH packet tx_nph_credit_consume_count += tx_nph_credit_required;}

where

C TX Credit Adjustment Sample Code






tx_nph_credit_required: the number of credits required to send given NP TLP. For NP, tx_nph_credit_required is 1.tx_nph_credit_consume_count_hip += (tx_cdts_type == "01") ? tx_hdr_cdts_consumed : 1'b0;tx_nph_credit_consume_count_hip_delayed : three pld_clk cycle delayed tx_nph_credit_consume_count_hiptx_nph_credit_limit_count = tx_nph_cdts + tx_nph_credit_consume_count_hip_delayed;

C TX Credit Adjustment Sample Code


D Stratix 10 Avalon-ST Interface for PCIe Solutions UserGuide Archive

If an IP core version is not listed, the user guide for the previous IP core version applies.

IP Core Version User Guide

16.1 Stratix 10 EditionBeta Beta

Stratix 10 Avalon-ST Interface for PCIe Solutions User Guide

D Stratix 10 Avalon-ST Interface for PCIe Solutions User Guide Archive



https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/archives/ug-s10-pcie-avst-16.1eb.pdf




E Document Revision History

Date Version Changes

May 2017 Quartus PrimePro v17.1 Stratix10 ES Editions

Software

Made the following changes to the IP core:• Added support for the H-Tile transceiver including example designs

available in the installation directory.• Added support for a Gen3x16 simulation testbench model that you can

use in an Avery testbench.Made the following changes to the user guide:• Added definitions for Advance, Preliminary, and Final timing models.• Added Testbench and Design Example for the Avalon-ST Interface

chapter.• Added figures showing the connections between the Avalon-ST and

user application and between Stratix 10 Hard IP for PCI Express IPCore, system interfaces, and the user application.

• Revised Generation discussion to match theThe Quartus Prime Pro v17.1 Stratix 10 ES Editions Software designflow.

• Changed TX credit interface pseudo code.tx_nph_credit_consume_count_hip_delayed is 3 pld_clkcycle delayed, not 2.

October 2016 Quartus PrimePro – Stratix 10

Edition Beta

Initial release

E Document Revision History






Intel® Stratix® 10 Avalon-ST Interface for PCIe Solutions ...1 Introduction 1.1 Avalon-ST...

Documents

Transcript of Intel® Stratix® 10 Avalon-ST Interface for PCIe Solutions ...1 Introduction 1.1 Avalon-ST...