Towards a Homogeneous Software Environment for DAQ Applications Luciano Orsini Johannes Gutleber...
-
Upload
alvin-gibbs -
Category
Documents
-
view
216 -
download
0
Transcript of Towards a Homogeneous Software Environment for DAQ Applications Luciano Orsini Johannes Gutleber...
Towards a Homogeneous Software Environment for DAQ Applications
Luciano Orsini
Johannes Gutleber
CERN EP/CMD
2
Topics
• The CMS DAQ model– RU, BU, EVM, FU– Event building process
• Interoperability (DAQ Column, demonstrator)• Configuration Management• Use of the model in testbeams• Summary
3
Data Acquisition Baseline
Detector Frontend
Computing Services
ReadoutSystems
FilterSystems
Event Manager Builder Networks
Level 1Trigger
RunControl
4
DAQ Main Subsystems
FS
Interconnect
Interconnect
Interconnect
CSN
EVM
RCN
BCN BDN
Interconnect
FUFU
BUO BUS
BUMBUI
RUO RUS
RUMRUI
RU linkFMU
DDUDDUFED
RU
BU
LV1
5
Readout Unit
RU
FEDGTP
EVM BCN
RCN
DSNInterconnect
RUM
RUO RUS
RUI
RDL
BDN
RU Readout UnitRUI Readout Unit InputRUM Readout Unit MemoryRUO Readout Unit OutputRUS Readout Unit Supervisor
GTP Global Trigger ProcessorFED FrontEnd DriverRDL Readout Data LinkEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service Network
6
Filter Column
DSN
FS
BDN
CSN
BCNEVM
BU
RURCN
BU Builder UnitFS Filter Subfarm
RU Readout UnitEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service NetworkCSN Computing Service Network
7
Builder Unit
BU
FS
EVM BCN
RCN
DSN
BUM
BUO BUS
BUI
RU
BDN
InterconnectBU Builder UnitBUI Builder Unit InputBUM Builder Unit MemoryBUO Builder Unit OutputBUS Builder Unit SupervisorFS Filter Subfarm
EVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service NetworkRU Readout Unit
8
BU vs. FU
allocate
take (id, data)
FUBU
RU
EVM
9
BU vs. EVM
allocate
confirm (id)
FUBU
RU
EVM
10
EVM vs. RU
readout (id)
FUBU
RU
EVMtrigger
data
11
BU vs. RU
send (id)
FUBU
RU
EVM
cache (id, data)
12
BU to EVM
FUBU
RU
EVM
clear (id)
13
DAQ Interfaces
BU
allocate
RU
cache
send
allocate
take
readout
trigger
data
clear
confirm
EVM
FU
14
Test Beams
RUI RUI
RUO RUO
BUI BUI
BM
RMFLT
Generic reusable module
Test beam specific
Detector frontend
...
...
15
BM
RMFLT
Test Beam Scenarios
RUI
RUO
RUI
RUO
BU BU
BM
RMFLT
Detector frontend
...
...
RUI
RUO
BU
Detector frontend
...
...
CPU
FU
FU
16
Trigger and Readout
trigger()
busyLow()
readout(id)
readyToRead(id) readyToSend(id)
FLT RM
interrupt
BM
data
RUM
RUI
17
Interoperability
RUOBM
BUIInterfaces
18
Interoperability
Detector Frontend
Computing Services
ReadoutSystems
FilterSystems
Builder (Control & Data) Networks
Level 1Trigger
RunControl
ReadoutManager
FilterManager
RCS
BU
19
Interoperability
• Step 1 – Identify Interface requirements between components
(RU vs. BU, EVM vs. BU, etc.) = IRS
• Step2 – Definition of Interface as messages and their format
+ configuration = IDD
• Step3 – Implementation of API supporting interfaces = XDAQ
20
Interoperability
• Support integration of independently developed DAQ sub-systems in step 3
• Interoperability through – established application message formats
• (using I2O private message frames)
– transport (available communication media)– configuration (physical/logical addressing scheme)
21
Configuration Management
• Version Control through CVS– Controlled software repository – Organised multi-platform releases
• Release notes document (SRN) with every software baselines to track status and changes– Build and installation instructions– Description and capabilities– Changes from previous releases– Reference to Documentation
22
Summary
• Support development of DAQ applications according to a common model
• Implement compatible interfaces between subsystems (interoperability)
• Assemble event builders from basic software components
• Common approach to DAQ in testbeams and CMS DAQ – smooth transition from testbeam to CMS DAQ – reuse software written for offline and testbeams
DAQ Column Software Internals
24
Topics
• Design Issues• DAQ Column set-up and configuration• Software internals• Status and plans
25
Design Issues
• Reflect I2O core environment – Event driven model, peer operations, homogeneous
message passing model– Software download, predefined commands– Fill the blanks approach
• Every subsystem maps to a self contained software module– Encapsulation of a subsystem behaviour– Clean and narrow interface from/to other subsystems
• Location transparency of components– Automatic mapping of physical to logical addressing
26
DAQ Column Configuration
DAQ Executives tasks
RUIO ( iop480)
PPC(mvme2304)
PersonalComputer (pcPentium)
27
Idle
Waiting for I2O configuration messages
Zzz..zzz…zzz..
28
Received Configuration
• Where (e.g. IOP 34)• Who (e.g RUO 3)• How (e.g TCP, DLPI)• What
(e.g. BM, BUI 1, etc.)
Detector Frontend
Computing Services
ReadoutSystems
FilterSystems
Event Manager Builder Networks
Level 1Trigger
RunControl
29
Downloading
• What (e.g.libRU.so, libEVM.so)
BM
RMFLT RUI
RUO
30
RUOBUI
Operational
Send (eventId)
BM
31
Local Communication Efficiency
Through executive transparent local and remote use
higher overhead
no infinite recursion
Direct invocation fast
only local
limits the reusability in a distributed environment
danger of infinite recursion
RM
RUI
32
Message Delivery
I2O h
eade
rda
ta
Buffer pool
RUO
Peer T
rans
ports
TCP
Myri
DLPI
FIFO
Peer Transport Agent
33
Func() { f… do…}
Message Access
I2O h
eade
rda
ta
Buffer pool
RUO
Peer T
rans
ports
TCP
Myri
DLPI
FIFO
34
BU
FU
XDAQ Deployment
FU BU RUO BM
RUO BM
35
Buffer Policies
• Buffer pools of several sizes– 1 KB, 4KB, 8KB … 256 KB
• Long messages are built from scatter-gather lists– SGL header contains a list of
• memory locations and • data sizes
– Quality of Service parameters in receiving peer-transport allow
• Building of SGL in one contiguous memory block (must be provided, e.g. buffer pool 1MB or dynamically allocated)
• Building of a new SGL consisting of several blocks
36
Transport Configurations
TCP
Myri
DLPI
FIFOTCP
Myri
DLPI
FIFO
Thread per Peer Transport- higher OS service overhead+ no CPU monopolisation+ allows integration with other software
Polling Peer Transport Agent+ low OS service overhead- executive uses CPU continuously- no mixing of polling/blocking PTs
37
I2O Overview
• An architecture to develop hardware and OS independent device drivers– Uniform message passing protocol– Hardware access wrapper API– Event driven execution model– Predefined parameters and protocol for configuration– Dynamic configuration– Space for private extensions– Split of processing responsibilities to offload host CPU
• Provides interfaces (.h files and annotation)– e.g. message headers, scatter-gather lists
38
I2O Issues
• Use of documented messages– for configuration, control and data
• Provide means for configuration• I2O model maps well to distributed environments• Allows hardware evolution without the need to
change the application components• Use parts of the I2O architecture for modelling
distributed DAQ systems
XDAQ toolkit uses a subset of I2O features
39
I2O Private Message Format
MessageSize MessageFlags VersionOffset
TargetAddressInitiatorAddressFunction = 0FFh
InitiatorContext
TransactionContext
XFunctionCodeOrganizationID = 100h
PrivatePayload = Function Parameters
PrivatePayload
3 2 1 031 24 23 16 15 8 7 0
40
I2O and the XDAQ Toolkit
• Every DAQ subsystems maps into a set of I2O private device classes– (RU = RUI + RUO)
• Each class can be instantiated several times– (RU#1, RU#2, BU#1)
• Each class exposes a private and an I2O interface that can be accessed by other class instances– (private, configuration and control messages)
• Instances exchange messages• Toolkit takes care of send and delivery
41
The XDAQ Toolkit• Is a framework with C++ API
– Messages from network are received and delivered to registered applications (callback mechanism)
• Application class method is associated with a specific message (binding at compile time)
• Classes dynamically register with the framework when they are downloaded
• User provides implementations for the predefined interfaces (abstract functions)
• Access to remote classes through asynchronous method invocation – through various configurations (TCP, PCI, Myrinet, ...)
42
I2O Based Configuration
• ExecSysTabSet– Tell the executive about the
physical addresses of other machines
• ExecDeviceAssign– Tell the executive about remote software modules with which it
can communicate
• ExecSwDownload– Tell the executive which software module it hosts(and initialise)
• ExecSysEnable– Tell the executive to start the local software modules (if required)
• ExecIOPReset– Tell the executive to put the system into its initial state
43
I2O Based Configuration
• UtilParamSet– Set parameters for a DAQ software element– Set executive parameters
• UtilParamGet– Get parameters from a DAQ software element– Retrieve executive settings
44
External Interface Issues
• I2O provides messages for– Application control– Application configuration– Executive configuration
• Systems that generate I2O configuration and control messages can communicate with XDAQ
• Interface design documents exist– I2O specification: http://www.i2osig.com
– EVB data format: http://cmsdoc.cern.ch/cms/TRIDAS/html/wTDR/EB_pres_layer.pdfhttp://cmsdoc.cern.ch/cms/TRIDAS/html/wTDR/IRS.pdf
45
Status and Plans
• Interoperability test with DAQ demonstrator and benchmarks (ongoing)
• Availability– Solaris, VxWorks, Linux – LynxOS by Frederic Drouhin
• XDAQ in small DAQ setup provides well founded input to the DAQ TDR– True user and system requirements and design options
• XDAQ data acquisition toolkit– as vehicle for smooth transition from testbeam to CMS DAQ – tools for stand-alone data acquisition test software