David Abbott - Jefferson Lab DAQ group Data Acquisition Development at JLAB.
-
Upload
lucinda-wilkerson -
Category
Documents
-
view
218 -
download
2
Transcript of David Abbott - Jefferson Lab DAQ group Data Acquisition Development at JLAB.
David Abbott - Jefferson Lab DAQ group
Data Acquisition Development at JLABData Acquisition Development at JLAB
DAQ Group now stands at 5 members. Recent Experiments have begun to test limits of the
current distribution of CODA (v 2.5). Aging technologies (software and hardware) are being
retired and replaced both for support of the 6 GeV program as well as development of 12 GeV.
Continue to use open standards and minimize the use of commercial software while maximizing use of commercial hardware.
We continue to focus on a “migration” plan from CODA2 to CODA3 (v 2.6).
Data Acquisition Status
General DAQ Issues…
Front-end hardware is evolving. Real-time intelligence is moving from the CPU to FPGAs. Old hardware technologies are no longer commercially supported (FASTBUS).
CPU-Based real-time readout on a per event basis limits the maximum accepted L1 trigger rate (~10 KHz).
32 crate limit on the trigger distribution system is nearly reached in Hall B. Event transport limitations in the current CODA architecture are being seen
for moderately complex systems. Computing platform and OS changes (Muli-core, more memory, 64 bit
systems etc…) are not taken advantage of. Aging software technologies and reliance on third party packages are
making code portability and upkeep difficult. Monitoring and control of large numbers of distributed objects are not
handled in a consistent way (too many protocols). “Slow” controls only minimally supported
CODA3 - Requirements/Goals Pipelined Electronics (FADC, TDC)
– Dead-timeless system– Replacement for obsolete electronics– Eliminate large numbers of delay cables
Integrated L1/L2 Trigger and Trigger Distribution System– Support up to 200 KHz L1 Trigger– Use FADC for L1 trigger input– Support 100+ crates
Parallel/Staged Event Building– Handle ~100 of input data streams– Scalable (>1 GByte/s) aggregate data throughput
L3 Online Farm– Online (up to x10) reduction in data to permanent storage
Integrated Experiment Control – DAQ RunControl + “Slow” control/monitoring– Distributed, scalable, and “intelligent”
Current DAQ Projects
Components:Components: CODA Objects CODA ROC CODA EMU (EB/ER/ANA) Run Control
Software Tools:Software Tools: cMsg ET EVIO Config and Display GUIs
Hardware:Hardware: FADC/F1TDC Trigger Interface (VME/PCI) Trigger/Clock DistributionCommercial Module Support
R&D:R&D: Embedded Linux Experiment Control Staged/Parallel Event Building 200KHz Trigger/readout Clock distribution L3 Farm
Front-End Systems
VME CPUVME CPU -(MV6100) PPC, GigE, vxWorks(GE V7865) Intel, GigE, Linux CODA ROC Readout ~160-200 MB/s
Trigger InterfaceTrigger Interface - (V3)
Pipeline TriggerEvent BlockingClock distributionEvent ID Bank Info
F1 TDCF1 TDCFlash ADCFlash ADC
R&D to support fully pipelined crates capable of 200 KHz trigger rates
VXS - L1 Trigger
VME CPU -???Intel, GigELinux CODA ROCVME Readout of Event Data
Switch Sum and Trigger Switch Sum and Trigger Distribution Modules (VXS)Distribution Modules (VXS)
Collect Sums/HitsPass Data to Master L1Clock distributionTrigger Distribution
Flash ADCFlash ADC
Use VXS High speed serial backplane (P0) to collect Energy sum and hit data from FADCs
Flash ADCFlash ADC
P0P0
Event Distribution
ET provides efficient transport of Data for building, and provides flexible User access
EMU provides easy configuration, and User specific processing options
Staged/Parallel Event Building
• Divide total throughput into N streams (1GB/sec -> N*xMB/sec).Divide total throughput into N streams (1GB/sec -> N*xMB/sec).• Two stages - Data Concentration -> Event Two stages - Data Concentration -> Event BuildingBuilding..• Each EMU is a software component running on a separate host.Each EMU is a software component running on a separate host.
AFECS - Integrated Experiment ControlAFECS - Integrated Experiment Control
FIPA Java-Based (v 1.5) “Intelligent” agents Extensions provide runtime “distributed” Containers (JVM). Agents provide a customizable intelligence (state machine)
and communication (cMsg, CA, SNMP etc…) with external processes.
Many independent “logical” control systems can operate within the platform.
System is scalable. Agents can migrate to JVM containers on different nodes at runtime. System tested: 3 Containers on different hosts with 1000
Agents controling 1000 physical components distributed over 20 other nodes.
~40% CPU and 200 MB usage for each JVM
CODA ROC
CODA EMU
EPICS IOC 1
EPICS CAG
Trigger soft
Trigger hard
Online ANA
A
A
A
A
A
A
A
S
S
S
S
S
NR NA NCPhysical Components
Normative Agents
Supervisor agent
Grand supervisor
Front-End
ACC
AFECS Platform(Java 1.5+)
IPC
IPC
WEB
IPC
Hierarchy of Control
IPC
CODA EvolvesCODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.
A typical system might look
Single Board Computer
FASTBUSVMECAMAC
ROC rcServer
EB
ROC
Network: Ethernet FDDI ATM
UNIX/LINUX
DISK/TAPEET
User Proc.
RunControlGUI
ER
cdev
dp_tcl
mSQL Database
msqld cmlogServer
cmlogDatabase
cMsg
AFECS
CODA3
CODA 2.6 Features
• Integrate new AFECS Run Control System• cMsg IPC replaces RC<->(ROC,EB,ER) communication as
well as CMLOG message logging.• Various Support for newer operating systems and compilers
– vxWorks 5.5, 6+– RHEL 4, Solaris 10, OS X
• New and Updated tools – ET upgraded, 64bit compliant– Db2cool– EVIO package
• Support for new CODA3 Objects and components• Integration of long time bug fixes, new driver libraries and
feature enhancements
Summary
The DAQ Group must support ALL experimental programs at JLAB. The current group must grow by at least 2 FTEs soon to manage current timelines.
CODA 2.6 is available now, and will provide an integration path for CODA 3 technologies.
Much DAQ software development is dependent on custom hardware development in order satisfy many 12GeV requirements.
Current DAQ projects reflect the philosophy that we can progress to support the physics of the 12 GeV program through an evolution of the existing proven system.