Frank LemkeDPG Frühjahrstagung 2010
Time synchronization and measurements of a Time synchronization and measurements of a hierarchical DAQ networkhierarchical DAQ network
DPG Conference Bonn 2010
Session: HK 70.3
University of Heidelberg
Frank Lemke, Sebastian Manz and Wenxue Gao
19.03.2010
2Frank LemkeDPG Frühjahrstagung 2010
OutlineOutline
• Data Acquisition Topology
• DAQ Features
• DAQ Components- Readout Controller- Data Combiner Board- Active Buffer Board
• Conclusion & Outlook
Frank LemkeDPG Frühjahrstagung 2010
Data Acquisition Topology Data Acquisition Topology
ABB
DCB
ROC
ROC
…
FEB
FEB
FEB
FEB
DCB
ROC
ROC
…FEB
FEB
FEB
FEB
Com
puting Cluster (F
LES
)……
Detector S
ystem
Optical connectionNon-Optical connection
FEB = Frontend BoardROC = Readout ControllerDCB = Data Combiner BoardABB = Active Buffer Board
Frank LemkeDPG Frühjahrstagung 2010
CBM Optical Link ProtocolCBM Optical Link Protocol
• CBM protocol- Three traffic classes- Delivers specific adapted
performance & features
• Generic link protocol- Minimized interface- Modularized- Efficient interface
synchronization- Transparent
16
16
ctrl2send_startctr l2send_end
ctrl2send
data2send_startdata2send_end
data2send
data2send_stop
ctrl2send_stop
4
4
16
16
dlm2send_vadlm2send_type
dlm_rec_typedlm_rec_va
data_recdata_rec_startdata_rec_end
ctr l_recctrl_rec_startctr l_rec_end 125 MHz ctrl_rec_stop
data_rec_stop
link_active
clk
CBMProtocolModule
CoreModule
link_clk
res_n
(2,5 GBit Link)
DLM - Receive
Control
DataMessage
DataMessage
ControlMessage
DLM - Send
MessageSend
Send
Receive
Receive
crc_error_send
crc_error_rec
User Network
Frank LemkeDPG Frühjahrstagung 2010
ROC ROCROCROC ROCROC
Clock & Synchronization NetClock & Synchronization Net
ROC ROC ROC ROCROC
Frontend to Detector System
…… …… ………
DCB DCB DCB
Synchronization and clock distribution over optical link
Backend to Computing Cluster (FLES)
… …
JC
JC = Jitter cleaning and clock recovery
ROC ROC ROCROCROCROC ROCROC
JC JC JC JC JC
JC JC JC
Clock &Sync. Gen.Tree structured connection
Frank LemkeDPG Frühjahrstagung 2010
Jitter Cleaner ModuleJitter Cleaner Module
• Requirements- Internal deterministic
implementation- Recovered clock usage- Peak-to-peak below 40ps
• Results- Extender connected with
mezzanine- National LMK03000 Family- MMCX to transmit clock- Bit precise synchronisation
V 1.0
V 1.1
Frank LemkeDPG Frühjahrstagung 2010
Readout Controller BoardReadout Controller Board
• Functions- Interfacing the front-end
electronics- Timestamp expansion- Data transport via optics
(or Ethernet)
• Board development- SysCore 1: Virtex 4 FX20- SysCore 2: better suited
connectors- SysCore 3 (planned):
mother- and daughter-board, Spartan6? V 1.0
V 2.0
(exemplary)
Frank LemkeDPG Frühjahrstagung 2010
ROC – Functional DiagramROC – Functional Diagram
• Virtex-4 FX20• Ethernet, MGTs
for communication• DDR SDRAM• Actel Flash-FPGA
for configuration• SD-Card and
Flash• Many User-I/O
pins
(picture from http://cbm-wiki.gsi.de)
Frank LemkeDPG Frühjahrstagung 2010
ROC – Modular Firmware DesignROC – Modular Firmware Design
??
ReadoutLogic Transport
Logic
nXYTERROC
nXYTERROC
FEETROC
FEETROC
EthernetEthernet
MGTsMGTs
Bus
FIFO AB
BA
BB
DriverDriver
DABC
GPIO
© N.Abel
Frank LemkeDPG Frühjahrstagung 2010
Data Combiner BoardData Combiner Board
• Functions- Data switching and concentration
multiple ROCs => ABB- 6 SFPs- 250MHz low jitter
programmable clock oscillator- Interface for jitter cleaner
mezzanine
• Board development- DCB V1.0: Virtex-4 FX100 - DCB V1.3: Improvements on
MGT reference clock insertion
V 1.0
V 1.3
Frank LemkeDPG Frühjahrstagung 2010
Receive clk
Transmit clk
Functional DiagramFunctional Diagram
DCB V1.3
XV6-fx100
JC
ABBSFP
SFP
SFPSFP
SFP
SFP
SFP DLMLP
LP
LP
LP
LP
• Special deterministic latency message handling• Erroneous detector control message retransmission• Efficient switching mechanism• Well defined clocking scheme
ROCs
Frank LemkeDPG Frühjahrstagung 2010
Hardware DevelopmentsHardware Developments
• Protocol development• Protocol test and verification • Clock distribution with jitter cleaning• Synchronization via deterministic latency messages• Crossbar and switching logic
Frank LemkeDPG Frühjahrstagung 2010
Active Buffer BoardActive Buffer Board
• Functions- Buffering the incoming hit packets- Hits reorganization- Transfer events to the host node- PCI Express 4-lane- DMA upstream: DAQ data packets
from buffer to the host node
• Board development- ABB-1: Virtex 4 FX60- ABB-2 (AVNET): Virtex 5 LX110T- ABB-3 (ML605): Virtex 6 LX240T
ABB-1
ABB-3
Frank LemkeDPG Frühjahrstagung 2010
ABB Diagram and Measurement
• Control messages invoke the data flow from ROC via DCB to ABB
• Hit packets are first buffered in the FIFO• Host software starts DMA to transfer the buffered hits to
the host memory• Measured bandwidth of 224 MB/s for one optical link
Frank LemkeDPG Frühjahrstagung 2010
Software Developments
• Driver under Linux 2.6• Performance emphasized• Well-layered architecture• ABBDaemon masks driver details to the users
- Enables access from multiple users- Uses UNIX IPC mechanism
• Provided test programs- RAM memory- One-hole memory / FIFO- CTL messages
Frank LemkeDPG Frühjahrstagung 2010
Test SetupTest Setup
Frank LemkeDPG Frühjahrstagung 2010
Conclusion & OutlookConclusion & Outlook
• Prototype systems working and in use
• Virtex6 adaption and development for all boards• ROC
- Modular hardware (mother/daughterboard)
• DCB- Protocol HDL optimizations
• ABB- Event Building - Flexible strategies for hit dispatching- Unified test program for more efficient debugging
18Frank LemkeDPG Frühjahrstagung 2010
Questions ?
Thank you for your attention !
Top Related