Apr.22,2005 Super B Factory Workshop

Detector (DAQ/Computing) Parallel

Takeo HiguchiInstitute of Particle and Nuclear Studies, KEK

on behalf of the COPPER working group

Pipelined ElectronicsPipelined Electronics

• Add-on modules

• Common pipelined-readout platform (motherboard)– Generalized DAQ software.

• Common pipelined-readout platform (motherboard)– Generalized DAQ software.

Design Concept: Low Cost

Sub-detector I/F User-defined module for

analog part only. Minimizing the

development cost.

Sub-detector I/F User-defined module for

analog part only. Minimizing the

development cost.

Readout CPU Commercially available

PMC module. No hardware

development cost.

Readout CPU Commercially available

PMC module. No hardware

development cost.

Common Readout Platform: COPPER

PMCProcessor

Trigger

GenericPMC slot

Sub-detector I/F

Sub-detector I/F

Sub-detector I/F

Sub-detector I/F

On-board Ether

Form factor = VME 9U

Sub-detector I/F slot 4Pipeline FIFO (1MB) 4PMC slot 332-bit local bus

32-bit 33MHz PCI bus

Local-PCI bridge

FastEthernet I/F

VME I/F

Sub-detector I/F slot 4Pipeline FIFO (1MB) 4PMC slot 332-bit local bus

32-bit 33MHz PCI bus

Local-PCI bridge

FastEthernet I/F

VME I/F

Data transfer on COPPERmeasured >65 MB/s.

>30 kHz L1 rate @ 1.6 kB/ev.Data size corresponds to typical SuberKEKB DC/COPPER.

Data transfer on COPPERmeasured >65 MB/s.

>30 kHz L1 rate @ 1.6 kB/ev.Data size corresponds to typical SuberKEKB DC/COPPER.

PMC / PMC Processor

• PCI Mezzanine Card

– 100% compliant with the PCI.– Suitable for high density applications.

– Many commercial products are available: Ethernet cards, GbE cards, memory modules, CPUs, etc.

• RadiSys EPC-6315– PentiumIII 800 MHz w/ 512 MB main memory.

– 32-bit 33/66 MHz PCI bus interface.

Contains Linux kernel image

PCI NIC PMC NIC

Sub-Detector I/F: FINESSE

• FINESSE: 18676 mm2 daughter card on the COPPER– Receives and digitizes sub-detector analog signals.– Holds the digitized signals for the level-1 trigger latency.– Outputs the data to the COPPER.

• Some FINESSE variations– TDC

– ADC

– TDC “tandem”

SuperBelle CDC Readout: QTC+TDC

• QTC: charge to time conversion

Qevent

0 tDRIFT

L1-trigger

drift chargeto sense wire

t

T(Q)

tSTOP

Q-to-T conversionQ-to-T conversion

t

t1

t2

TDC & ADC at the same time,simplifying the readout scheme.

• t1 = drift time

• t1 – t2 = drift charge

9U-VME sized QTC modulewith LeCroy MQT300 QTC chip

9U-VME sized QTC modulewith LeCroy MQT300 QTC chip

For tracking performance

Position resolution < 130 m

For dE/dx performance

Dynamic range 10 bit

Linearity < 0.5 – 1.0%

Other boundary conditions

Expected single rate per wire 100-200 kHz

Readout ch# ~ 15 k in total

For tracking performance

Position resolution < 130 m 27 m/bit (0.78 ns)

For dE/dx performance

Dynamic range 10 bit 17 bit

Linearity < 0.5 – 1.0% 0.49%

Other boundary conditions

Expected single rate per wire 100-200 kHz

Readout ch# ~ 15 k in total 24 ch/chip

SuperBelle CDC Readout: QTC+TDC

• TDC: time to digital conversion

Requirement

FINESSE with AMT-3 TDC chip (originally designed for ATLAS)FINESSE with AMT-3 TDC chip (originally designed for ATLAS)

AMT-3 TDC chip

AMT-3 TDC Chip

• Up to 17 bit timing measurement: coarse + fine counters

– Pulse edge timings: coarse + fine– Trigger timings: coarse

• Trigger matching– Only the edges associated to Level-1 trigger are output.

• Level-1 pipeline provided (256 edges)

Coarse counter: 13-bit bunch crossing counter (40MHz @ LHC)Fine counter: additional 4-bit time memory cellCoarse counter: 13-bit bunch crossing counter (40MHz @ LHC)Fine counter: additional 4-bit time memory cell

SuperBelle system clock• Bunch crossing rate: LHC = 40 MHz,

KEKB = 508 MHz.• We choose SuperKEKB system clock

= 42.33 MHz = 508/12.

SuperBelle system clock• Bunch crossing rate: LHC = 40 MHz,

KEKB = 508 MHz.• We choose SuperKEKB system clock

= 42.33 MHz = 508/12.

Level-1 pipeline FIFO(256 edges)

Readout FIFO(64 edges)

Channel buffer(4 edges)

Channel buffer(4 edges)

Channel buffer(4 edges)

Trigger matching

Trigger timing FIFO(8 events)

24

AMT-3

24-channel inputs

AMT-3 output

Level-1 trigger inputs

AMT-3 TDC Chip – Cont’d

collisiontime

L1 trigger

trigger latency (~5s?)

edge search

OK NG

Trigger matching

of AMT-3

No hardware output indicating FIFO full– Unexpected data loss is only notified by an “error flag” in the next

event output.

Readout “readout FIFO” of AMT-3 quickly

to COPPERDREADY

DATA

AMT-3AMT-3FPGA

w/ FIFOFPGA

w/ FIFO

SCLK: 42.3 MHz

BUSY

No further trigger

AMT-3 L1-Pipeline Depth

Less depth: 256 edges shared by 24-input channels– LeCroy TDC (we’re currently using): 16 edges / each channel.– Is it sufficiently large? MC simulation is needed.

• MC parameters – Assuming SuperBelle drift chamber– # of input channels: 24– Random wire hit: 100 kHz (Poisson distribution)– Level-1 trigger rate: 50 kHz (Poisson distribution)– # of gen. words per wire hit: 2 for signal edges + additional for noise– Level-1 trigger latency: 5 s

Extreme case

“Toy” MC Simulation: Buffer Usage

Level-1 pipeline usagesampled at every clock [edges]

En

trie

s / 1

ed

ge

13.7 s

1.37 s

137 ms

13.7 ms

FIFO full point

Extrapolate … = ~104

Extrapolation assuming the right side slope unchanged.

FIFO/AMT-3 won’t get fullbefore O(105) sec.

Very preliminary

TDC FINESSE with AMT-3

• Block diagram of TDC FINESSE (not finalized yet)A

MT

-3A

MT

-3 Dataformatter

Dataformatter

Triggerreply logic

Triggerreply logic

FIFO-fullmonitor

FIFO-fullmonitor

Busygenerator

Busygenerator

FIFO

42.3MHzclock Level-1 trigger

Trigger busy

Local-busI/F

Local-busI/F

Sub

-det

ecto

rS

ub-d

etec

tor

CO

PP

ER

eve

nt F

IFO

CO

PP

ER

eve

nt F

IFO

Local bus (A7D8)

COPPER FIFO full

Xilinx Spartan-3

[Readout data] [Readout data]

COPPERCOPPER

The First AMT-3 Signal on COPPER

Pulsegenerator

Pulsegenerator

NIMECLconverterNIMECLconverter

FINESSEFINESSE

AMT-3AMT-3250 ns 4.2 s

w#1: 00111010010111011110110110000110

w#2: 00111010010110011110111011011010

@ ch 0x0b

content tag(0x3 = edge data)

ch id.(ch = 0x0b) 1: leading edge

0: trailing edge 17-bit edge timing

pulse width

Measured pulse width dist.

peak = 251 nsrms = 0.93 ns

CPUCPU

w#1: 00111010010111011110110110000110

w#2: 00111010010110011110111011011010

Future Plan

• Short term (-Jul.): System test– TDC resolution study with test input pulses.– Investigation of FIFO full behavior of AMT-3.– Stability tests: stability against random trigger, long-run stability, etc.– Software development: device driver, readout software, etc.

• Middle term (Aug.-Nov.): First step integration to Belle DAQ (EFC)– EFC is the smallest sub-detector (total = 320 ch).– EFC has two alternative readout paths (CDAQ + EFC own): failsafe and easier cross

check.

• Long term (Dec.-): CDC readout with COPPER based system

First Step Integration into Belle DAQ

• Replace Belle EFC DAQ with COPPER EFC =Extreme Forward Calorimeter

EFC =Extreme Forward Calorimeter

ForwardEFC

ForwardEFC

BackwardEFC

BackwardEFC

160 ch

160 ch

COPPER COPPER TDCTDCTDCTDC

TDC3

TDC3

FASTBUS

VxWorksEB

VME crate

COPPER3

COPPER3

Eth

ern

et s

wit

ch

PC

TDC “Tandem”FINESSE

TDC “Tandem”FINESSE

LocalSTORLocalSTOR

Summary

• We have developed module-structured system for SuperBelle readout consisting of COPPER boards and FINESSE cards.

• The QTC and TDC will be employed to readout SuperBelle CDC data.

• We use AMT-3 TDC chip. The TDC FINESSE card with the AMT-3 has been developed.

• Level-1 pipeline depth in the AMT-3 is examined by MC simulation.

• We have observed the AMT-3 is correctly digitizing the input pulses.

• The first step integration of the COPPER system into the Belle DAQ is planned in this August using EFC.

16 ch

16 ch

8 ch

8 ch

TDC “Tandem” FINESSE

• LeCroy TDC: 6 connectors• COPPER: 4 FINESSE slots

• To use present analog cables…– We’ve made a ‘tandem-FINESSE’

module equipped with 3 connectors.

• Channel density– 48 ch / tandem-FINESSE.– Same as present LeCroy TDC.

Tandem-FINESSE

AMT-3AMT-3

AMT-3AMT-3

2

Readout PC and Event Builder

copper0copper0

copper1copper1

copper5copper5

…

Readout PC

Present event builder

Sw

itchi

ng h

ubS

witc

hing

hub

100 Base-T

GbE GbE GbE

• Readout PC:– CPU : Intel Dual Xeon 2.46 GHz– OS : RedHat9(2.4.18-20smp)

• Readout PC:– CPU : Intel Dual Xeon 2.46 GHz– OS : RedHat9(2.4.18-20smp)

Data transferperformance

Faster readout PC and/or software tuning are requested for SuperBelle use.

Calibration Run

• Calibration run– Daily detector calibration with test pulse and/or test trigger.

• Readout PCs should take responsibility to collect data– O(100) COPPERs / sub-detector 6 readout PCs Event building mechanism is needed for calibration run.

CPU Boot-up Mechanism

• Compact Flash– Boot up procedure is well established.

– Distribution of updated software to all CF cards is non- trivial.

– Write access to CF card shortens the card lifetime.

– CF card is getting out of date.

• Network boot– Distribution of updated software to all CF cards is easy.

– We have no established method yet to boot-up EPC-6315 from network.

– Boot-PC farm serving multiple boot requests must be developed. It is never trivial.

Run Control

copper0copper0

copper1copper1

copper5copper5

…

NSM Gateway

NSMNSM or other mechanism

To avoid increase of NSM nodes, we put NSM gateway (like SVD2).

fbdaqprivate network

Readout PC(l0efc)

NSM Master