A Universal Framework for Data
Validation
Giovanni OrgantiniINFN-Sez. di Roma
ACAT03-Giovanni Organtini 2
Data Validation
Data Validation (DV)– Fast analysis of data coming from a DAQ system to
evaluate data usability– Each atomic analysis program is called task
fast, light and single-objective
Data Validation must be– Able to evolve on a short timescale – Flexible to manage multiple and different validation tasks– Robust to allow many different users to cooperate without
interference
ACAT03-Giovanni Organtini 3
DV Framework Concepts
Sensors: elements that detect events such as an EOR or a Warning signal issued by analysis tasks
Events: any Sensor outcome Mediator: a process that collects Events
from Sensors and dispatches them to Solutions
Solutions: objects that recognize Events and act according to their content
ACAT03-Giovanni Organtini 4
Framework architecture
ACAT03-Giovanni Organtini 5
Framework architecture
Sensors
ACAT03-Giovanni Organtini 6
Framework architecture
Notify Event toMediator
ACAT03-Giovanni Organtini 7
The Sensors
AlarmSensor: parses HTML-formatted files and detects comments of the form <!–- SensorHTML
Creates events according to comment content
EORSensor: parses a text file (runlist) and detects updates. A line is added to runlist at each EOR
ACAT03-Giovanni Organtini 8
Framework architecture
Dispatch Event toSolutions
ACAT03-Giovanni Organtini 9
Framework architecture
Solutions
ACAT03-Giovanni Organtini 10
The Solutions
Program: launch one or more programs in a given directory (repair known problems)
E-mail: send an e-mail with arbitrary text to a list of recipient
Window: show a popup window on a given terminal with arbitrary HTML-formatted text
ACAT03-Giovanni Organtini 11
A more formal view
Implemented in Java– portable– distributed (RMI-server created to allow Sensors to run on
remote processors)
ACAT03-Giovanni Organtini 12
How it works
Sensors detect changes in files– an EOR-Sensor can detect an EOR looking at the content
of a given file containing the runlist– An AlarmSensor can parse output files produced by
analysis tasks to detect potential problems
The Mediator collects Events and notify them to all the existing Solutions
The Solutions choose to activate themselves according to Event content
ACAT03-Giovanni Organtini 13
EOR-Sensor
The EOR-Sensor generate an Event as soon as a new line appear in the runlist
runlist formatted as follows<commands/beam/> --date=20031111 –ebeam=50 –beam=e
ACAT03-Giovanni Organtini 14
EOR-Sensor
The EOR-Sensor generate an Event as soon as a new line appear in the runlist
runlist formatted as follows<commands/beam/> --date=20031111 –ebeam=50 –beam=e
directory containinganalysis tasks forthis application
ACAT03-Giovanni Organtini 15
EOR-Sensor
The EOR-Sensor generate an Event as soon as a new line appear in the runlist
runlist formatted as follows<commands/beam/> --date=20031111 –ebeam=50 –beam=e
directory containinganalysis tasks forthis application
parameters to be passed toanalysis tasks (arbitrary)
ACAT03-Giovanni Organtini 16
EOR-Sensor
The EOR-Sensor generate an Event as soon as a new line appear in the runlist
runlist formatted as follows<commands/beam/> --date=20031111 –ebeam=50 –beam=e
Program Solution activated
directory containinganalysis tasks forthis application
parameters to be passed toanalysis tasks (arbitrary)
ACAT03-Giovanni Organtini 17
Analysis tasks
Analysis tasks can be written in ANY language, provided they can accept arguments on the command line– Examples: C, C++, Perl, shell scripts, etc.
They must provide the output in HTML-formatted files to be located in a given directory– output can be either text and/or images– the result of any single task could be browsed from
everywhere using any browser– a process will automatically generate the list of output files
to be shown to the shifter
ACAT03-Giovanni Organtini 18
Alarm Sensors
They look for HTML-comments in output files and generate events according to the comment content– comments not shown by browsers– must be compliant with a set of rules – actions may be combined (i.e. can send e-mails and
notify a message to the shifter)
ACAT03-Giovanni Organtini 19
Example
<!– SensorHTML winFlag=false emailFlag=true commandFlag=false <emailBody> [email protected] senderName=Giovanni Organtini - ped analyzer [email protected],[email protected] smtpServer=localhost subject=Problem with pedestals in test beam <message> A problem has occurred during the analysis of the pedestal. May be you want to run to the detector to investigate… </message> </emailBody>..-->
ACAT03-Giovanni Organtini 20
Solution activation
The e-mail Solution is activated– sends an e-mail to Coordinator and Task responsible– the e-mail body is arbitrary and may contain links and
other HTML-tags
Other informative message can be written on the output file containing the comment– output file may contain
text images links
ACAT03-Giovanni Organtini 21
Applications
CMS Crystal Calorimeter Construction in Roma– Crystals measured by automatic machine– Results analyzed by validation tasks– Problems notified to construction supervisor via e-mail– Some known problems recovered via program Solution
CMS Crystal Calorimeter Test-Beam at CERN– Calorimeter modules exposed to e, beams– Data collected and analyzed by validation tasks– Problems notified to experts via e-mail and to shifters via
popup windows
ACAT03-Giovanni Organtini 22
Conclusion
A Data Validation Framework has been developed– simple to manage and use– universal– flexible– robust– configurable– no requirements on analysis tasks– no library to link or file to include
Package available soon (2004) for download on http://www.roma1.infn.it/people/organtini
Top Related