1
Upgrades for the PHENIX Data Acquisition System
Martin L. Purschke, Brookhaven National Laboratoryfor the PHENIX Collaboration
RHIC from space
Long Island, NY
2
RHIC/PHENIX at a glance
RHIC:2 independent rings, one beam clockwise, the other counterclockwisesqrt(S)= 500GeV * Z/A~200 GeV for Heavy Ions~500 GeV for proton-proton (polarized)
PHENIX:
4 spectrometer arms
15 Detector subsystems
500,000 detector channels
Lots of readout electronics
Uncompressed Event size typically 280 -220 - 130 KB for AuAu, CuCu, pp
Data rate ~6KHz (Au+Au)
Front-end data rate 0.8 - 1.5 GB/s
Data Logging rate ~500MB/s, 700 MB/s max
3
TOF-W
RXNP MPC-N
Our youngest Detector Systems
HBD
4
Need for Speed: Where we are
ATLAS
CMS
LHCb
ALICE
CDF
~25 ~40
~100
~300All in MB/sall approximate~100
~150
600
~1250
Lvl1-Triggers in Heavy Ions have a notoriously low rejection factorthat's because so many events have something that's interesting (different from LHC)But hey, we could write out almost everything that RHIC gave us, so why bother... this approach has served us really well. It also opened up access to processes that you can't exactly trigger on, it “just” takes some more work offline.
5
600++ MB/sThis shows the aggregated data rate from the DAQ to disk in a RHIC fillWe are very proud of this performance...
Decay of RHIC Luminosity
Length of a DAQ run
It's not the best, it's one where I was there... the best RHIC fill best went up to 650MB/s
6
Run 10 Event statistics
3150 TB
950 TBPhysics
PHENIX Raw Data100 TB
7
Upgrade Programs
RHIC will give us several luminosity and beam livetime upgrades
The era where could mostly write out “everything” is coming to an end
The Future
we will add detectors in the central region which will significantly increase our data volume
8
Upgrades
3 main new detectors (that's in addition to the ones I showed before as “on board”):
• The Vertex/Forward Vertex detectors• A Muon trigger upgrade• RPC detectors
800-pound gorilla
23911Total
1.071002.8%6FVTX
1.92390.54%/0.16%3VTX pixel
1.25904.5%/2.5%2VTX strip
Data rate (Gbps)*
Event size (kbyte)
OccupancyDCM groups
Detector
Triples current event size 23911Total
1.071002.8%6FVTX
1.92390.54%/0.16%3VTX pixel
1.25904.5%/2.5%2VTX strip
Data rate (Gbps)*
Event size (kbyte)
OccupancyDCM groups
Detector
VTX pixel 3 0.54%/0.16% 39 1.92
VTX Pixels will be installed this summer for Run 11
9
Upgrades
Central Silicon Vertex Trackers
“VTX”
Pixel
Strippixel
10
Silicon Pixel in Run 11ALICE1LHCb readout chip:
Pixel: 50 µm (f) x 425 µm (Z). Channels: 256 x 32.Output: binary, read-out in [email protected] Hardness: ~ 30Mrad
Sensor module:
4 ALICE1LHCb readout chips.Bump-bonded (VTT) to silicon sensor.Thickness: 200 mThickness: r/o chips 150 µm
Half-ladder (2 sensor modules+bus)
1.36 cm x 10.9 cm.Thickness bus: < 240 µm.
SPIRO module • Control/read-out a half ladder
Send the data to FEM
FEM (interface to PHENIX DAQ)Read/control two SPIROs
• Interface to PHENIX DAQ
active arear
1.28 cm = 50mm x 256z
1.36 cm = 425mm x 32
Solder bump
~20m
All chips on Ladder #6 has good hit map by beta-ray source test
1st Complete Pixel Ladder on Dec 25, 2009
The hitmaps are a great success, but from where I stand, the fact that we are
reading out the ladder is most important
12
Buffer Box
Data Flow
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
Gigabit
Crossbar
Switch
To
HPSS
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Classic Event builder architecture
DCMDCMDCMDCM
DCMDCMDCMDCM
DCMDCMDCMDCM
Data Collection Modules (100's)
Sub Event Buffers
(~35)
Crossbar Switch
Assembly & Trigger
Processors (~60)
Buffer Boxes (7)
Data ConcentrationInteraction
Region Rack Room
13
The A”T”P ATP
• “Assembly and Trigger Processor” was deemed important at the time when
PHENIX was designed
• Meant to run what is today known as a HLT
• We soon learned that we can do without it
• Data logging capability has kept pace with the MB data rate, which is great
• Opens access to processes which you simply can't trigger on
With all the upgrades I'll show, we are determined to keep
it that way
14
Need for Speed
• Data Collection Module (DCM) was modern in its
days, DSP-based + some FPGA
• DCM II uses latest FPGA technology, FPGA is the
main component
• 10G networks are becoming a commodity
• Allows to make better use of multi-core machines
• Saves money and power and A/C in the end
• In the same spirit, replace PCI with PCI Express
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
SEB
Gigabit
10G
Crossbar
Switch
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
15
DCM-II and “jSEB”-II
• jSEB (“SubEvent Buffer”) is the PCI-Express card that reads a number of DCM's
• About a factor of 15 more bandwidth than the older generation
DC
M
II DC
M
II DC
M
II
Partitioner
II
INT
ER
FA
CE
BUSY
DATAL1
PC
JSE
B
II PCSEB
JSE
B
II
GTM
L1
Custom Backplane
FEM
SEBDCMDCMDCMDCM
16
Force-10 Switch
Standard Gigabit ports
“MRJ-20” cable bundles with 6 Gig ports each go to patch panels (or directly to the machines) May make our cable distribution easier, bring a few bundles to the racks
10GbE portsStandard fibers with “SFP+” connectors
17
Will add buffer boxes as needed
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Putting it together
DC
M
II DC
M
II DC
M
II
Partitioner
IIINT
ER
FA
CE
BUSYDATAL1
PC
JSE
B II PC
SEB
JSE
B II
GTM
L1
Custom Backplane
FEM
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
ATP
SEB
SEB
SEB
SEB
SEB
SEB
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
Buffer Box
DC
M
II DC
M
II DC
M
II
Partitioner
IIINT
ER
FA
CE
BUSYDATAL1
PC
JSE
B II PC
SEB
JSE
B II
GTM
L1
Custom Backplane
FEM
DC
M
II DC
M
II DC
M
II
Partitioner
IIINT
ER
FA
CE
BUSYDATAL1
PC
JSE
B II PC
SEB
JSE
B II
GTM
L1
Custom Backplane
FEM
The new jSEB-II's will exceed the limits of a Gigabit connection, need 10GbE
Will have 5 such jSEB-IIs in Run 11
20 for the full system in Run 13
The existing detectors will continue to use the current readout
The DAQ upgrades are geared towards maintaining the current event rate, not to increase it – for now. The existing detectors will keep their readout electronics and will limit us to the current rate
18
RPC (resistive plate counter)
RPC3
Test assembly of RPC-3 half octant support structure at UIUC
Adds timing resolution to the Muon detectors so we can distinguish muons from the vertex fron those traversing the IR
19
Outlook
PCI Express 10GB/s
Networks
DCM IIUpgrade
New Detectors on board or coming to us to search for dedicated signals
New hardware components to help maintain our current speed
The End
20
Backup
21
Pictures
22
Cabling “plan”
• Current Switch
23
Building up to record speed
• Over the previous runs we have been adding improvements
• Had lighter systems, d+Au, p-p, Cu-Cu in previous runs, less of a challenge than 200GeV Au+Au (most challenging)
• Distributed data compression (run 4)
• Multi-Event buffering (run 5)
• Mostly consolidating the achievements/tuning/etc in run 6, also lots of improvements in operations (increased uptime)
• 10G Network upgrade in run 7, added Lvl2 filtering
Ingredients:
24
Data Compression
LZO
algorithmNew buffer with the compressed one as payload
Add new
buffer hdr
buffer buffer buffer buffer buffer buffer
LZO
UnpackOriginal uncompressed buffer restored
This is what a file then looks like
On readback:
This is what a file normally looks like
All this is handled completely in the I/O layer, the higher-level routines just receive a buffer as before.
Found that the raw data are still gzip-compressible after zero-suppression and
other data reduction techniques
Introduced a compressed raw data format that supports a late-stage
compression
Run 11 Counts
strips pixel FVTX Total
Fibers 60 40 54
DCM2 module 8 5 7 21
Partitioner 3 5 3 4 12
JSEB 2 5 3 4 20 ** Include JSEB II for the crate controllers
The build for the final system running at full bandwidth.
The ratio between partitioner 3 to DCM is 2 to 1
16 fibers 1 fiber
26
Data Collection Module-II
• Front-end cards on the detector are getting read out by those (8 per card)
• DCM-II connects to a custom bus on the right
• A number of DCM-II's are read out via a PCI Express Card
48V in(Isolated)
5V in (control)
Download(readback)
L1
data
JTAG
(640MB/sec)
( 160 MB/sec)JTAG
clock320MB/sec
1.1V2.5V3.3V
27
Multi-Event Buffering: DAQ Evolution
PHENIX is a rare-event experiment, after all -- you don’t want to go down this path
Without MEB
Multi-Event buffering means to start the AMU sampling again while the current sample is still being digitized.
Trigger busy released much earlier
deadtime is greatly reduced
28
The Multi-Event Buffering Effect
29
Upgrades
• All new detectors have electronics with high rate capability• However, older detector readout limits the level 1 rate• no way to upgrade any time soon - $$$$$• We will need to focus on rare events more
Front end pipelines
Readout buffers
Processor farms
Switching network
Detectors
Lvl-1
HLT
40MHz
100KHz
100Hz
Remember, our Lvl1 is not the LHC Lvl1... ours is before digitizationHLT is no solution
CMS
Hence: FVTX has Lvl1 trigger “hookup” for displaced vertex triggersother upgrade is a trigger to begin with (W->muon)
30
u d W
d uW−
u u dd
ν μ /e
μ/eW
ALW
= σ−σ
σ +σ
Δ d x1 u x2 −Δu x 1 d x2u x 1 d x2d x1 u x2
Similar expression for W- to get Δῡ and Δd…
Since W is maximally parity violating large measured Δu and Δd require large asymmetries.
W Production Basics
No Fragmentation!
31
MuonTrigger and RPC upgrade
RPCs
μμ+/-+/-
uu
dd
ν μ /e
μ/eW
W physics with polarized
protons
• Trigger will allow to enhance the sample if high-momentum/straight-track muons
• RPC adds timing to reduce large background from non-collision muons (beam, cosmics)
FVTX
• Fitted track provides a DCA to the primary vertex (measured by central arm barrel VTX detector)
prompt
Pinpoint the decay vertexto eliminate backgrounds!
Endcap detects the following by displaced vertex (∆r, ∆z) of muons: D (charm) μ + X B (beauty) μ + X B J/ ψ + X μ+ μ-
21-October-2008 Jon S. Kapustinsky 2008 IEEE NSS-MIC Dresden 33
FVTX Section View
80 cmBarrel
• 4 discs of Si sensor in acceptance of each Muon Arm • Microstrips to accurately measure R coordinate of track
• Scheduled to be installed in FY11
Two endcap halves½ of one endcap
½ disks
Top Related