Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at Fermilab

04/19/23Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at Fermilab - S. Rapisarda

1

Real-Time Data Reorganizer for the Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at FermilabD0 Central Fiber Tracker System at Fermilab

S.Rapisarda - Computing Division/Electronic System Engineering DepartmentS.Rapisarda - Computing Division/Electronic System Engineering Department

Real-Time Data Reorganizer for the Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at FermilabD0 Central Fiber Tracker System at Fermilab

S.Rapisarda - Computing Division/Electronic System Engineering DepartmentS.Rapisarda - Computing Division/Electronic System Engineering Department


2

D0 ExperimentD0 Experiment

D0 ExperimentD0 Experiment


3

D0 DetectorD0 Detector

14 m46 ft

20 m66 ft

Central Fiber Tracker ElectronicsScintillating Fibers Detector(D0 Central Fiber Tracker)


4

Central Fiber Tracker (CFT)Central Fiber Tracker (CFT)

Inserting CFT into calorimeter

bore

FinishedCFT

Truck convoy carrying CFT

to D0


5

Analog Front-End BoardsAnalog to digital signal conversion

Optical to Digital ConversionOptical to Digital Conversion

Visible Light Photon CountersOptical to electrical signal conversion

D0 Central Fiber Tracker40960 scintillating fibersEvent frequency 7.6 MHz

40960 clear fibers

320 LVDS SERDES 371MHz links

Links Throughput: 347 Gigabit/secondData Throughput: 311Gigabit/second


6

D0 Trigger Framework

Digital Front-End BoardsTrack recognition

Track RecognitionTrack Recognition




List of found tracks

320 LVDS SERDES 371MHz links


7

Central Fiber Tracker80 4.5º wide sectors, 512 fibers => 512 bits @ 7.6 MHz5 72º wide Super-Sectors, 8192 fibers => 8192 bits @7.6 MHz

Fibers from detectors have cylindrical geometry (in blocks of 256)

Track recognition electronics (DFEs) requires data organized in azimuthal (trigger) sectors….

Trigger sectorsTrigger sectors


8



The Missing PieceThe Missing Piece




Optical box to reorganize fibers?

Digital box to reorganize data?

40960clear fibers

320 LVDS SERDES371MHz links

or


9

Data Mixer SystemData Mixer System



Data Mixer SystemReal-time data reorganization







10

Mixer System RequirementsMixer System Requirements

Limitations and challenges: Defined I/O interfaces High data throughput 311 Gigabit/second Marginal throughput available on I/O links Limited space available Minimal delay (200 nsec ?) One board design should fit all the system needs Limited budget & time

Advantages: Some degrees of freedom in specifying the 1st level of serialization in the AFEs Minimal limitations imposed by the DFEs on the 1st level of serialization Minimal flexibility required for future upgrades/changes System partitionable in five identical and independent subsytems of four boards each. Each subsystem has to handle the data from one supersector.

1st level of data serialization from 7.5 MHz to 53 MHz 2nd level of serialization (LVDS SERDES) 53MHz to 371MHz







11

Board#1

Mixer SubsystemMixer Subsystem

Digital Front-End Boards

Data Mixer Subsystem

64 LVDS SERDES links

64 LVDS SERDES links

Board#2

Board#3

Board#4

Analog Front-End Boards

LVDS SERIALIZERS

DES DES DES DES

SER SERSERSER

LVDS DESERIALIZERS

(1/5 of the detector data)

The fiber detector can be logically partitioned into 5 identical and independent angular sectors.

The mixer system is partitioned into 5 identical and independent subsystems.


12

MS Excel Based ToolMS Excel Based Tool

A MS Excel based tool has been implemented and used to:

Operate on data

Visualize data attributes

Compare different board architectures and choose one

Minimize number of chip-to-chip connections

Minimize number of board-to-board connections

Define/assign data mapping for the input links, chip-to-chip connections (FPGAs I/O), board-to-board connections (backplane), output links

After having defined a data multiplexing model, to automatically generate VHDL code for data mixing.


13

MS Excel Based ToolMS Excel Based ToolMixer board 1 Mixer board 2 Mixer board 3 Mixer board 4

Colors…

Link# 1 16 17 32 33 48 49 64Bit# 1

140


14

Mixer Board ArchitectureMixer Board Architecture16 LVDS SERDES 371MHz Links

Data Throughput 15.5 Gigabits per second

16 LVDS SERDES 371MHz LinksData Throughput 15.5 Gigabits per second

MIL-STD1553

SubrackControllerB

ACKPLANE

LVDS Receivers

Data Synchronization

Data reorganization

LVDS Transmitters

BackplaneData TXs/RXs

Timingcontrol

BoardControl

Diagnostic


15

Mixer Board LayoutMixer Board Layout

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240 XCV50-6CXILINX

PQ240 XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240

XCV50-6CXILINX

PQ240XILINXSPARTAN XCS40XL-5C PQ240

XILINXSPARTAN XCS40XL-5C PQ240XILINXSPARTAN XCS40XL-5C PQ240

XILINXSPARTAN XCS40XL-5C PQ240 XILINXSPARTAN XCS40XL-5C PQ240

XILINXSPARTAN XCS40XL-5C PQ240XILINXSPARTAN XCS40XL-5C PQ240


16

Configuring Mixer System Configuring Mixer System One of the 17 FPGAs on the board (board controller) is automatically configured at power-up by an on-board EEPROM. The EEPROM content is modifiable through a front panel JTAG interface port.

The board controller interact with the subrack controller and handles the configuration of the remaining 16 FPGAs.

The FPGAs configuration files are stored on a removable CompactFlash card on the subrack controller.

At power-up the configuration time for the full mixer system is 30 seconds

128MB

MEMORY CARD

CompactFlashTM

Subrackcontroller

Mixers


17

DiagnosticsDiagnostics

R E S E T

3

5

1

2

0

4

7

6

8

1 2

9

1 3

1 0

1 4

1 1

1 5

3 . 3 2 . 5

0 1

0 0

0 2

0 3

0 4

0 5

1 5

1 4

1 3

1 2

1 1

re s e t /m o d e

b u tto n

fr o n t p a n e l b i- c o lo r L E D s

Input links clock error Input links frame marker error Input links clock synchronization error Input links frame synchronization error Input links control bits detection Input links test pattern detection Board local bus Backplane serial bus Backplane general-purpose bus FPGAs configuration status Output links test pattern transmission Error latching Measurement of data frame misalignment Data frames re-synchronization Control bits masking Input and output link shutdown Timing reference information

Front panel accessto diagnostic

Remotely accessible throughthe subrack controller


18

TestingTesting Boundary Scan (individual board)

Test-rack (individual subsystems)

Mixer System Test RackFull system under testbefore commissioning

Subsystem under test


19

Data Mixer SystemData Mixer System










20

Mixer SystemMixer System










21

AcknowledgmentAcknowledgmentThanks to

N.G.Wilcer co-designer of the mixer system

J. Anderson, R.Angstadt, F. Borcherding, B. Haynes, D. Husby, J.Olsen, J. Smith, and M. Vaz for major contributions to the design of the Mixer system;

S. Bledsoe, J. Chramowicz, S. Grünendahl, G. Ginther, S. Lynn, P. Rubinov, and M. Tomoto for their invaluable help in the assembly, debugging, and commissioning of the Mixer system.

Contacts: S. Rapisarda, N. Wilcer (email [email protected], [email protected])Project website: http://www-ese.fnal.gov/D0_CTT_Mixer/

Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at Fermilab

Documents

Transcript of Real-Time Data Reorganizer for the D0 Central Fiber Tracker System at Fermilab