DAQ Data Processing Chain

Vasilis Vlachoudis25-27 Feb 2015

Account: ntofdaq@lxplus• runDSTcreator.cc :

• acrontab every 5min• C++ program parses the INPUT.INP and submits an lsf batch run DSTcreator with

arguments the INPUT.INP parameters

• switch#.bat:• acrontab in between the runDSTcreator• shell script to overwrite the INPUT.INP with new parameters

• INPUT.INP:• input file with variable=value, parameters of runs to process

• DSTcreator:• C program checking all castor files (within the range defined in INPUT.INP) if they

are processed or not• if not then submit another lsf batch calling the “proc2014.bat” script

• proc2014.bat:• shell script running the raw2dst program• copy the output in castor and in /afs

Current Status [1/2]

Account: ntofuser@lxplus• raw2dst:

• C++ program reading the raw files with ntoflib• output:

• root: pulse fitting pulses routine (generic Petar’s UserInput.h, BaF2, ppac’s)

• dst files creation

• ntoflib:• C helper library to access the data structures of the raw format with the rfio castor

libraries

• UserInput.h:• Fitting parameters for all detector types for the generic Petar’s routine

• perl scripts• merging together the root/segment files in a single root file per run

Current Status [2/2]

UserInput.h

All detector type parametersare in a single file

• Complicated chain of shell/C++/C/perl programs, result of evolution over time (needs cleanup)

• Synchronization needed between users, to know which version/parameters are used for the pulse fitting Needs a key person to launch the process.

• Users cannot easily test new parameters. All detectors (even the ones they are not interested needs reprocessing)

• DST files are huge.• Are they useful?• Do we need to automatically generate and keep them all?

• raw2dst access the raw data as stream of bytes needs modification for 16bit or other type of data.

• Slow parameters not inside the root files. Needs correlation with the timestamp

• Primitive Versioning control: One subdirectory for every new version with all libraries copied inside. Multiple copies of libraries and programs.

• Verification of the process data. Who is checking them? Which version/parameters are the correct ones?

Issues

• Cleanup / rewrite all processing scripts chain up to raw2dst to provide the same/similar functionality

• SVN repositories • ntoflib responsibility moves to EN-STI

• provides us with an API to access the data with a common way, regardless of the format they are stored

• benefit: we are accessing our data with a set of routines and we don’t mess with the RAW format. All our processing programs are transparent in the format changes

Proposal: Phase I

nTOFLIB- getFile()- parseFile()Changes:- C C++- Update CASTOR

protocolrfio xroot

CASTOR

EN-STI nTOF

raw2dst- new libChanges:- C C++- use API to

access data

scriptsChanges:- scan /afs directory

for input parameters- flag them when done

• Rewrite the raw2dst raw2root• eliminate the dst from the automatic output (if not commonly used). Only on

demand• output one root file per run per segment per detector type

• End users, will provide (through some interface, web, script) the range of runs they want to process with which parameters ONLY for their detector

• Similar perl scripts will merge the data in a single root file if needed.

Proposal: Phase II

• Create a suite of validity checks of the data

• Correlations between the data for possible errors• Integral quantities between detectors, pulse intensity, etc…

• Automatic creation of standard plots• Available directly on the daq web site

Proposal: Phase III