Atlas Wisconsin/LBNL Group March 21 st 2007
description
Transcript of Atlas Wisconsin/LBNL Group March 21 st 2007
Read-Out Driver for ATLAS Silicon Detectors
Atlas Wisconsin/LBNL Group
March 21st 2007
ATLAS Pixel B-Layer Upgrade Workshop
2
Control Path:
ROD Controller FPGAMaster DSP
Program Reset Manager FPGA
Operation, Commands, Triggers
96
Ba
ck
of
Cra
te C
ard
/ F
ron
t E
nd
Ele
ctr
on
ics S
Lin
kC
ard
Clo
ck
F
an
ou
t
Triggers and Event ID Data
Tim
ing
Inte
rfa
ce
Mo
du
le
BOCSetup Bus
Serial Input Links
Xon/Xoff
ROD Busy
DSP Farm (4)
Event Trapping, Histograms
Data Path:Formatter, Event Fragment Builder & Router FPGA,Diagnostic Memory, S-Link Interface & SDSP DMA
DMA 4
...
S-Link
48
System Clock
Serial Output Links
VME64x Signals
VM
E
Bu
s
40
DMA 1
ClocksHost Port
Silicon Read Out Driver (SiROD)
3
Silicon RevE(F) SiROD
MasterDSP
ControllerFPGA
FormatterFPGAs
Router FPGA
EventFragmentBuilderFPGA
SlaveDSPs
9U VME
VME Interface
Clock Distribution
4
• FE Module Interface:
• 96 FE Data input connections available
• 48 FE Command output connection available
• 2048x32 Data FIFO for each Module (32 total)– Pixel B-Layer: 6-7 modules
• 12-14 80MHz input data links
• 6- 7 40MHz output command links
– Pixel Barrel L2: 26 modules
• 26 40MHz input data links
• 26 40MHz output command links
– Pixel Disks: 13 modules (26 ROD channels)
• 13 80MHz input data links
• 13 40MHz output command links
ROD Overview (1)
5
• Diagnostic components included for stand alone testing
• Maintain data throughput at 100kHz trigger rate
• Flexible calibration processes• MDSP Control
• ROD Controller FPGA Control
• TIM Control
ROD Overview (2)
6
32 MB SDRAM
VME Slave FPGA
Configuration Controller
ROD BUSY Histogram
16Mb Flash Memory
Master DSP
TI 320C6201
160MHz
ROD CONTROLLER
FPGA
FORMATTER FPGA
8x
EFB FPGA
ROUTER FPGA
DSP FARM TI 320C6713
220MHz 256MB SDRAM
4x
DIAGNOSTIC FIFOs
MDSP EMIF
RODBus
VME Bus
Back Of Crate (BOC)
CNFG DATA to All FPGAs
RESET
ROD BUSY
BOOT ROM 512KB
VME Bandwidth:
~8MB/s MDSP~4MB/s SDSP
~22MB/s VME Slave
All DSPs:
16 bit Host Port Interface
D16
D32
A32/D32
D16
ROD Control/VME Path
7
RCF RODBus Interface
(FPGAs, DSP Farm)
MDSP EMIF Data (32)
Control & Status Registers
RODBus Data (16)
RODBus Address
RODBus Control
MDSP EMIF Address
MDSP EMIF Control
ROD CONTROLLER FPGA: Control Functions
Diagnostic FIFO Interface
DSP Interrupt Interface
Reset Command Interface
MDSP Interrupts SDSP Interrupts
MDSP Serial Port Interface
INPUT FIFO (32K x 96)
EVENTMEM FIFO (16K x 48)
To VME/PRM FPGA
To MDSP SP0/SP1 ports
8
MDSP: Primitive List State Machine• Primitive List execution: Host-to-MDSP and MDSP-to-SDSP
Primitive List Execution
IDLE
PAUSED EXECUTING(one primitive per
Iteration)PREPAREREPLY
ACKNOWLEDGE List Inserted?
YES
Pause?
Finished or Abort?
NO
YES
Abort?
Resume?
YES
inList Bit ClearedBy Host?
NONO
YES
NO
NOYES
NO
YES
9
Text Buffer State Machine
• Text Buffer reply: MDSP-to-Host and SDSP-to-MDSPError, Info, Diagnostic and Transfer (MDSP is a way-station for SDSP text buffers)
Text Buffer Transmission
Buffer Occupied?Buffer Read?
Read Request?
IDLE
WAITINGFROZEN
YES
YES
NO
NO
NO
YES
10
ROD CONTROLLER
FPGA
XC2S600-6FG676
FORMATTER FPGA
EFB FPGA
XC2S400-6FG676
ROUTER FPGA
XC2S400-6FG456
DSP FARM TI 320C6713
220MHz 256MB SDRAM
4x
Xoff
BOC
FORMATTER FPGA
FORMATTER FPGA
FORMATTER FPGA
FORMATTER FPGA
FORMATTER FPGA
FORMATTER FPGA
FORMATTER FPGA
INPUT FIFO 32K Deep
S-Link
EVENTFIFOs
216Kx48
A
B
Event Fragments
Xoff
Xoff
ROD BUSY
TTC
EVENT ID
READOUT
TIM
MDSP SP
BOCFE CMD (48)
XC2S600-6FG456
SiROD DATA PATH
Can sustain 100KHz Trigger Rate into S-Link
11
RCF RODBus Interface
MDSP EMIF Data (32)
Control & Status Registers
RODBus Data (16)
RODBus Address
RODBus Control
MDSP EMIF Address
MDSP EMIF Control
Trigger Processor &
Counter
MDSP Ser Port 0
MDSP Ser Port 1
TTC L1 Trigger
FE Command Processor &
Mask
Event Processor
Dynamic MasksMode Bits
Event ID DataTrigger Type
Internal Scan Processor
FE Event Counter
FE CMD (48)
TTC EVT
EFB EVT (16)
FMT MB (12)
Diagnostics Generator
FMT Trailer Detect
ROD CONTROLLER FPGA: Real Time Functions
12
RODBus Interface
Control & Status Registers (32)
RODBus Data (16)
RODBus Address
RODBus Control
Link Input MUX
&
Half Clock
Counters
IN0
ModeBits FIFO 512x24
Link FIFO 2048 x 32
Link Decoder: 40/80/160MHz
Read Out Controller
DATA OUT (40)
TOKEN
FMT MB (12)
Link FIFO 2048 x 32
Link Decoder: 40/80MHz
Link FIFO 2048 x 32
Link Decoder: 40MHz
Link FIFO 2048 x 32
Link Decoder: 40MHz
IN1
IN2
IN3
IN4
IN5
IN6
IN7
IN8
IN9
IN10
IN11
FMT Type
FMT ID
HT LIMIT
ROD BUSY
Trailer DetectTRAILER (12)
PIXEL FORMATTER FPGA (8x)
13
EVENT FRAGMENT BUILDER FPGA
EventMemA
FIFO16Kx48
Engine 0
Engine 1
EventMemC To
/Fro
m R
ou
terT
o/F
rom
Fo
rmat
ter
To
/Fro
m F
orm
atte
rT
o/F
rom
Co
ntr
olle
r
Output FIFO Data (43)
Header and Trailer Data (32)
Halt Output
Halt Output
Header/TrailerGenerator
Error Summary Word
Error Summary Word
Fo
rma
t &
Co
un
tF
orm
at
& C
ou
nt
L1
/BC
ID
Ch
ec
kL
1/B
C I
D C
he
ck
Err
or
Ch
ec
kE
rro
r C
he
ck
INC
/DE
C L
1ID
L1 & BC IDs
Event Data& Trigger Type
Dynamic Mask
Link Num. (6 bits)
Link Num. (6 bits)
Time-out Error
Time-out Error
L1ID Error
L1ID Error
BCID Error
BCID Error
Data (32 bits)
Data (32 bits)
Formatter Num. (2 bits)
Formatter Num. (2 bits)
Link Num. (4 bits)
Link Num. (4 bits)
Data (32 bits)
Data (32 bits)
Data Valid
Data Valid
Time Out
Time Out
INC
/DE
C L
1ID
Purpose: Collect Formatter output, check L1 and BC
IDs, count errors, generate Event Header & Trailer
FIFO Controller Xoff
EventMemB
FIFO16Kx48
14
ROD Router (FPGA) & Slave DSPs
SDSP 3SDSP 2SDSP 1SDSP 0
Read, formatand direct the
event data
2 TrapsEvent Type: ATLAS,
ROD or TIM
2 TrapsEvent Type: ATLAS,
ROD or TIM
2 TrapsEvent Type: ATLAS,
ROD or TIM
2 TrapsEvent Type: ATLAS,
ROD or TIM
DMA Transfer Engine1024 32-bit FIFO
DMA Transfer Engine DMA Transfer Engine DMA Transfer Engine
S-L
INK
To
/Fro
m R
OS
To
/Fro
m E
ven
t F
rag
men
t B
uil
der
XOn / XOff S-Link
S-Link DataDsp Halt Output
Halt Output
Error Format Data
Data Valid
Output FIFO (43b)
Event Header andTrailer Data (32b)
Texas Instruments 6713 floating point DSPs running at 220 MHz for monitoring and calibration histogramming
Purpose: Route formatted data to Level-2 and/or Slave DSPs (histogram)
1024 32-bit FIFO 1024 32-bit FIFO 1024 32-bit FIFO
15
Additional Slides
16
The Silicon Read-Out Driver (ROD)
MODULE BOC
SBCTIM
ROS ROD
Controller& Master DSP
Formatters Event FragmentBuilder
Router& Slave DSPs
(Read Out Subsystem)
(TIming Module)
(Back Of Crate)
(Single Board Computer)
To Event-Builderand Level-2
CalibrationHistograms
• Secondary purpose: Calibrations / MonitoringFPGAs for time-critical functions (Event Data Path)
DSPs for configuration, ROD control, calibrations and monitoring
• Primary purpose: Module configuration, Trigger propagation, Data formattingA hybrid of FPGAs and DSPs
• The Master DSP (MDSP) has ROD and BOC registers connected to one of its EMIFs (External Memory InterFace)
Event DataConfiguration & Triggers
Backpressure / Halt Output S
-Lin
k
17
Pixel Calibration Scans• A couple of common Pixel Scans:
Threshold (on chip charge-injection for each individual pixel; scan the number of hits for each injected charge to obtain the discriminator threshold)
Noise (a threshold scan without charge injection to measure noise)
Threshold Scan resultshown for a
single Pixel Module
Threshold for each pixel(note the 16 FE ICs)
18
SCT Calibration Scans• Some of the commonly run SCT Calibration Scans:
Rx Threshold Test (optimize the Rx threshold value in the BOC data-receiver chip)
NMask Test (demonstrate that the chip mask register functions properly)
Pipeline Test (scan the chip pipeline for defects; stuck on or off)
Full Bypass Test (does the chip bypass feature function properly)
Noise Occupancy Test (a threshold scan without charge injection to measure noise)
Synchronous Trigger Noise Test (triggers distributed synchronously across the system)
N-Mask Scan resultshown for a
single SCT Module
19
ROD use during detector production / assembly• Calibration scans have been routinely used during module production
Verify module functionality before and after mounting or transport
Classification of modules• Rank by the number of defects; only mount and install the very best
• This phase also gave feedback to ROD developers (and still continues to!)A few DSP software and FPGA Firmware problems have been uncovered this way
• Some unique bugs appeared when stressing the ROD with a full complement of modules
20
Cosmics Data-Taking• TRT/SCT combined run in May 2006
ROD performed well; successfully collected more than 400k events
Useful DAQ/Detector shake-down, data for efficiency and alignment studies
• On-ROD Histogramming used for timing• in the SCT during cosmics running• (i.e., delay the signal by the correct amount)
Transmission Delay(bunch-crossings)
Number of coincident hitson top and bottom strips
Noise
SCT
Transition Radiation Tracker (TRT)
A total of 504 SCT Modules into 12 RODs were used during the combined cosmics data-taking run
20 cm of concrete
Scintillator
Scintillator
Scintillator
21
SCT RODs in USA-15• Production RODs in their final location for ATLAS running