ADRIAN FIERGOLSKI1,2, MICHELE QUINTO1,3

1INFN-Bari, Italy 2 Warsaw University of Technology, Poland

3University of Bari, Italy

TOTEM READOUT USING THE SRS

RD51 E-School, 3rd of February 2014

Outline

• Introduction of the TOTEM experiment• Integration of SRS with the TOTEM DAQ• Firmware development• Firmware verification• Tests and results of SRS-based DAQ

3-Feb-2014 Adrian Fiergolski, Michele Quinto 2

TOTEM experiment

3-Feb-2014 Adrian Fiergolski, Michele Quinto 3

T1: Cathode Strip ChamberT2: triple GEM

Roman Pot: 10 planes of silicon edgeless detector

• TOTEM uses 3 tracking detectors (T1,T2,RP) located symmetrically with respect to the IP5

• For the luminosity independent measurement of the p-p cross section at low momentum transfer, TOTEM requires magnetic configuration of the accelerator optics with high β* (90m, 1000m, 1535m)

• TOTEM demands special LHC runs, which allow RP to approach the beam

TOTEM DAQ before Long Shutdown (LS1)

3-Feb-2014 Adrian Fiergolski, Michele Quinto 4

In the TOTEM standalone configuration, the VME bus bandwidth limitsthe trigger rate to 1 kHz.

VFAT2 chip

3-Feb-2014 Adrian Fiergolski, Michele Quinto 5

• Trigger and tracking capabilities• 128 channels• 0.25 µm CMOS process node• supports LHC clock frequency of 40 MHz• Radiation hard• Single Event Upset (SEU) protection

Optical transmission

3-Feb-2014 Adrian Fiergolski, Michele Quinto 6

Data

16

TX_en

TX_er

READY 800Mbps1310 nm

9/125 µm single-mode fiber

K28.5 D5.6 K28.5 D5.6 D0.6 D27.5 D13.6 D16.6 D11.7 K28.5 D5.6 K28.5 D5.6

Comma CommaData

Ethernet codding (IEEE-802.3):

8b10b Encoding

3-Feb-2014 Adrian Fiergolski, Michele Quinto 7

Assumptions:• line code• maps 8-bit symbols to 10-bit symbols

Properties:• DC-balance • bounded disparity• reasonable clock recovery

Code structure:• Difference between the count of 1s and 0s in a string of at least 20bits

is no more than 2• No more than five 1s or 0s in a row

All codes that represent the 256 data values: data (D) codes.The codes representing 12 special non-data characters: control (K) codes

8b

10b

TOTEM’s DAQ evolution

3-Feb-2014 Adrian Fiergolski, Michele Quinto 8

Scalable Readout System

3-Feb-2014 Adrian Fiergolski, Michele Quinto 9

Advantages:• Cost effective replacement for the currently used

VME-based solution offering higher bandwidth• TOTEM’s implementation will be compatible with the

CMS DAQ• Allow standalone runs of TOTEM• Enable hardware data filtration

TOTEM’s C-Card: Opto-FEC

The development board linking the OptoRx and the FEC.

32-bit parallel bus following S-Link protocol clocked at 40 MHz

2.5 Gbps SERDES 8 LVDS lines providing 5.36 Gbps Clock generator/jitter cleaner TTC interface I2C configuration lines TTS support JTAG support Independent power supply mode

3-Feb-2014 Adrian Fiergolski, Michele Quinto 10

Firmware development guidelines

• Hardware description and verification in System Verilog language– compactness, syntax structures

→ more re-usable, less error prone code– the language consequently gains attention of industry

→ increasing maturity of the EDA tools– Possibility to use legacy VHDL, VERILOG modules (eg.

open cores)• The communication between entities via standard interfaces

– AMBA AXI4-Stream, AHB• Automatic register generation from register map specification

– IDesignSpec

3-Feb-2014 Adrian Fiergolski, Michele Quinto 12

Firmware scheme

3-Feb-2014 Adrian Fiergolski, Michele Quinto 13

• System UnitProvides set of common interfaces and services. Development of mutual modules can by a shared effort of the SRS community.

• Application UnitApplication specific data processing part.

Firmware verification

3-Feb-2014 Adrian Fiergolski, Michele Quinto 14

The firmware is simulated using System Verilog combined with the Universal Verification Methodology (UVM):

• High level of abstraction (reusable)• Random test vector generation (guided by constraints)• Coverage indicating verification progress• EDA tools provide UVM libraries to test popular interfaces (eg. Ethernet, I2C)

The verification of the FEC defines two kind of simulations:• Partial simulation

→ to achieve faster simulation coverage of complex modules• Full design

SRS based DAQ in the LHC environment

3-Feb-2014 Adrian Fiergolski, Michele Quinto 15

Conditions:• data from 3 full RP detectors containing about 120 VFATs• FEC was read directly by a standard PC running DATE software• commercial SATA storage medium

Results:Without transmission error, the system acquired 10M events reaching maximum trigger rate of 10 kHz

Even faster SRS

3-Feb-2014 Adrian Fiergolski, Michele Quinto 16

• Trigger rate at flat top ~25kHz• Readout bandwidth close to the

link limit 118MB/s• System stability over more than

140M events• None of the event has been lost

Modifications:• new firmware following the presented guidelines• distributed data storage on up to 3 DAQ nodes• custom, combined hardware-software solution to achieve

lossless transmission via unreliable UDP protocol

Conclusion:In term of trigger rate, the new DAQ is more than one order of magnitude faster,

reaching 24.7kHz against 1 kHz of standard, VME based , system.