FELIX Design FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group Upgrades of detector readout meeting, 9 July 20141L. Levinson Guiding principles Goal: a GBT to LAN switch with as little awareness of detector specifics as possible No user code in FELIX (so far): configuration params for options Separate data movement from data processing Configurable data paths from E-link to network endpoint Scalable at Front end, Back end, and switch levels Do as much as possible in software COTS FPGA PCIe board But see talk on TTC clock recovery by Jinlong Zhang Readout, so latency is not important, but provide low latency GBT-to-GBT links Future proof by following COTS market, separately upgradeable components: GBT, FPGA PCIe board, serve PC, network cards Routing is for E-links; they are the logical connections L. LevinsonUpgrades of detector readout meeting, 9 July 20142 Virtex-7 PCIe board L. Levinson Upgrades of detector readout meeting, 9 July From Global HiTech, PCIe gen 3, 8 lanes Virtex X690T 2x12 bidir CXP connectors, FMC Off- detector architecture with FELIX Functions can now be separated, physically, or just logically. ROD = Event fragment builder/processor for ROS L. LevinsonUpgrades of detector readout meeting, 9 July ROD L. LevinsonUpgrades of detector readout meeting, 9 July 20145 Routing Configured via control port on main or separate network Simple/fixed: FELIX port # / E-link-range IPaddress / port or Infiniband local identifier / queue pair E-link packet protocol: Allows several logical streams to use the same E-link header with: stream-id + packet len (if not fixed length packets) FELIX port # / stream-id IPaddress / port Cloning, broadcasts / multicasts possible, in either direction Reliable transport from FELIX to end-point via TCPIP Configurable Quality of Service per logical connection Less-than-best-effort for event data for monitoring L. LevinsonUpgrades of detector readout meeting, 9 July 20146 Use cases 1 1.Receive data from an E-link or bonded group of E-links (a channel) and route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order. Optional 8b/10b or HDLC checking and/or decoding Optionally verify checksum Packet boundary identification either by SOP/EOP or length, optionally there can be no packet boundary, just a byte stream. If a maximum packet length is configured, data arriving after the maximum is discarded and this error is marked in the packet status word. If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing. Sampling can be requested L. LevinsonUpgrades of detector readout meeting, 9 July 20147 L. Levinson Upgrades of detector readout meeting, 9 July Use cases 2 2.Receive BUSY from an E-link command, internally from FELIX itself, or from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system. 3.Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links The BC clock is supplied to the GBT because the FELIX's GBT link clock is the BC clock supplied by the TTC system to FELIX. Three options: 2-bit, 4-bit, 8-bit as described in the datasheet. 4.Receive data from a TCP port and route it to one or more E-links Optional 8b/10b or HDLC encoding A specific multicast channel can broadcast packets to all out-going E- links with the "MC" attribute. L. LevinsonUpgrades of detector readout meeting, 9 July 20149 Use case 3 8.Scheduled data transfer lists: setting and activating (list is per GBT, but E- links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list. Activation by TTC user bit Activation by command received via TCP 9.Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed. L. LevinsonUpgrades of detector readout meeting, 9 July L. Levinson Upgrades of detector readout meeting, 9 July Low latency path is between predefined GBT pairs. Uses cut-through Routing. FELIX: FW/SW interface Data interface PCIe low level packet format (fixed length) PCIe high level packets, spans low level packets Push/pull? Hints from RobinNP architecture Configuration parameter interface Need description of parameters that can generate: XML for OKS, VHDL for FPGA See (but needs to be updated): https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf Monitoring and statistics interface FPGA configuration interface IP core from HitechGlobal ~4k$ L. LevinsonUpgrades of detector readout meeting, 9 July L. Levinson Upgrades of detector readout meeting, 9 July L. Levinson Upgrades of detector readout meeting, 9 July Backup L. Levinson Upgrades of detector readout meeting, 9 July Functionality of E-lane handlers From GBT: need to decode 8b/10 or HDLC data Collect 2, 4, 8, 16 bits per 40MHz clock Align using comma symbols Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet Add 1KB block headers and chunk trailers Optionally verify checksum To GBT: need to encode 8b/10b or HDLC data Read from FIFO Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO Chop into 2, 4, 8, 16 bits per 40MHz clock Probably run at 80MHz to get extra clocks for headers/trailers L. LevinsonUpgrades of detector readout meeting, 9 July Clock recovery The 2x40MHz clock comes from the TTC link, i.e. a recovered clock. Probably OK for GTHs in the same GTH block of 4 transceivers ??? Can it be distributed to the other blocks? Does the recovered clock have sufficiently low jitter for the other GTHs? We should probably consider a TTC receiver and jitter remover on an FMC card. rework the GLIB version so that the FMC is on top of the PCIe mother board instead of extending beyond the PCIe board boundary L. LevinsonUpgrades of detector readout meeting, 9 July Error handling Just beginning the considerable thought that needs to be done here. Two issues: Discovery of corrupted data Surviving errors without out loosing event boundaries This requires some effort on the front ends Corruption within a GBT 80-bit word is detected by the GBTs FEC Hard to believe E-links can be mixed into each other, so expect errors within E-link channels: repeated clocks, lost clocks If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected. FELIX can flag these errors in the header it sends with a streams data L. LevinsonUpgrades of detector readout meeting, 9 July