CWG9 and Event Display
B. von Haller
17.07.2014CERN
CWG 9 in a nutshell
▶Started in May 2013 along with O2
▶Group working on ▶the Data Quality Monitoring ▶the Quality Assurance▶the Visualization
▶For Run 2 and Run 3
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 2
CWG9 Members
▶ Renu Bala▶ Francesca Bellini▶ Mihaela Gheata▶ Lukasz Graczykowski▶ Malgorzata Janik▶ Andreas Morsch▶ Mihai Niculescu▶ Jeremi Niedziela▶ Ankita Sharma▶ Maciej Szymanski▶ Barthélémy Von Haller
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 3
Definitions of DQM and QAData Quality Monitoring & Quality Assurance
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 4
▶ Feedback on the quality of data ▶ Online (DQM)
▶ Make sure to record high quality data▶ Identify and solve problem(s) early
▶ Offline (QA)▶ Make sure to analyze high quality data▶ Identify high quality runs
▶ Involves▶ [Online gathering of data]▶ Analysis by user-defined algorithm
▶ Production of monitoring objects such as histograms▶ Assessment of the quality of the data based on the objects
▶ Storage of monitoring data▶ Visualization (+ human assessment of quality)
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
CWG9 Tasks▶ Group focusing on the data Quality Assurance, online and offline, and the visualization of its
results and the data itself▶ Run 3
▶ Study how to monitor data efficiently and in plenty without interfering with the data taking▶ Discuss QA output and results, incremental QA and procedures to formalize if the results are
acceptable or not▶ Determine the needs, and design the software, to access, visualize and interpret the results▶ Define and develop the software to visualize data, raw and reconstructed▶ Participate to the writing of the Technical Design Report and to the possible prototyping
▶ Run 2▶ Production data taking period -> coordinate maintenance and improvements of software▶ Opportunity to test concepts and software for Run 3
5
Data
flow Monitoring Object
GenerationAutomatic
Quality AssessmentStorage
Visualization
Event Display
Maintenance and development taken over by Warsaw group▶Jeremi Niedziela – PhD student at CERN▶Maciej Szymański – Service Task on ED▶Jakub Sala, Jakub Abelski, Adam Felis –
Summer students▶Warsaw group @ WUT for support▶Mihai – Offline, former ED developer
People currently working directly on ED
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 6
HLT HLTED
Offline recoEvent Display
Event Display
▶Run 2 requirements▶Split reconstruction and online ED▶Allow multiple sources and unify existing EDs▶Bookmarks
▶Under development by J. Niedziela and M. Szymanski
▶Work will continue over the summer ▶Ready for the commissioning and cosmics
Run 2 tasks
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 7
HLT HLTED
Off.Reco ED
HLT
Off.Reco ED
Future work
▶Event Display▶Alternative platforms (mostly web and/or mobile)
▶Summer students from Faculty of Mathematics and Computer Science started working on it
▶Longer term collaboration on this topic is possible in order to implement a full fledge event display
▶Possible use of early alfa components▶Prototyping needed here
▶DQM/QA▶Prototype of merging QA objects using map-reduce with
ZeroMQ▶Web based ROOT objects browser
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 8
Future work – CWG10
▶ Control and Configuration of hierarchical distributed systems▶ Development of a zeroMQ-based prototype for configuration
distribution and process control ▶ Evaluation of performance based on number of nodes, hierarchical
levels, processes and configuration size ▶ Process Management
▶ Evaluation of technologies for process execution in sandbox/container (Docker, etc.)
▶ Evaluation of performance based on number of nodes, hierarchical levels, processes
▶ Evaluation of ZooKeeper for possible usage in O2▶ Configuration management ▶ Synchronization ▶ Process control
Control & Configuration
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 9
Conclusion
▶CWG9 is an important and dynamic working group in the O2 project
▶WUT is a key player in CWG9 ▶In terms of people and responsibility ▶Crucial for the future of Visualization in ALICE▶The ED is already profiting greatly from its
involvement ▶There are many more tasks and opportunities
for collaboration in CWGs
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 10
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 11
Backups
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 12
▶ April 2014: First draft▶ August 2014: Submission▶ CWG9 participates to
▶ Chapter 4 System architecture : Quality control and assessment▶ Table 1: detectors needs▶ Explain DQM/QA architecture and the choices made▶ Figure 1: DQM/QA architecture▶ Explain Event Display architecture and the choices made▶ Figure 2: Event Display architecture
▶ Chapter 5 Technology survey, evaluations and prototypes1. Mergers architecture and feasibility tests with 0MQ2. Results of storage tests (e.g. DB technologies)3. [Web gui architecture (ROOT JS lib + DABC)]4. Event display design as tested (cf Run 2)
▶ Chapter 6 System Design
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
TDR and feasibility tests
13
Past and Current tasks
▶ Bring everyone aboard [done]▶ Make people aware of others’ work in the field▶ Give a picture of the current situation to everyone
▶ Run 2▶ Event Display review and meetings, Warsaw involvement
▶ Core refactoring▶ New features▶ Knowledge transfer Gain stability and support for Run 2
▶ DQM/QA review and preparation ▶ Proposal for the online reconstruction and calibration
▶ Run 3▶ System requirements and system functionalities document [done]▶ Detectors needs survey ▶ Definition of the future architecture and design▶ Prototypes and feasibility tests▶ Technical Design Report redaction
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 14
O2 Technical Design Report
▶ Schedule
▶ October ‘13: ▶ Define table of content▶ Establish editorial board
▶ December ‘13:▶ System Requirement Document▶ High-level dataflow model▶ Computing platforms benchmarks▶ Networking benchmark
▶ June ‘14▶ Software framework architecture
▶ Sep ‘14 ▶ TDR
15B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
CWG9 TDR Timeline
▶ January 14 ▶ Define list of tables and figures▶ Draft of the architecture of the system▶ Launch subsystems exhaustive survey Submit list of tables and figures to TDR EC
▶ February 14▶ Draft tables and figures▶ Skeleton of 4.2.4 and 5.6 Submit skeleton to TDR EC
▶ March 14▶ Finalize tables and figures, including subsystems input▶ Iterate on text using input of TDR EC Submit text and final tables and figures to TDR EC
▶ April 14▶ Finalize text Submit final text to TDR EC
▶ May 14▶ Iterate over our sections using CWGs input▶ Review work of other CWGs (especially what concerns us!)
Proposal
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 16
TDR – skeleton, tables & figures
▶4.2 Data processing and computing model▶4.2.4 DQM and QA
▶« Quality control and assessment »▶Table 1: detectors needs▶Explain architecture ▶Figure 1: architecture▶Explain the choices▶Figure 2: Event display arch.
Chapter 4 System architecture
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 17
TDR – skeleton, tables & figures
▶5.6 DQM and QA▶« Quality control and assessment »▶Technologies and design choices available concerning
key points of our system▶Storage▶Access to results worldwide▶Event display
▶Feasability tests & prototypes▶Table 1: results of storage tests (e.g. DB technologies)▶Figure 1: Web gui architecture (ROOT JS lib + DABC)▶Figure 2: Event display design as tested (cf Run 2)
Chapter 5 Technology survey, evaluations and prototypes
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 18
Subsystems survey
▶ What “tasks” (name it agents or algorithms if you prefer) will your subsystem need in Run 3 ?
▶ For each of these task or group of tasks, tell us▶ Whether it already exist today and if so what is its performance.▶ What is the expected performance of such a task in Run 3.▶ How many plots are expected to be produced (for the shifter and for the
experts).▶ Percentage of events needed to carry out the task online (minimum,
optimal).▶ What is the input ? i.e. at which stage will it run ?▶ How fast the response has to be taken into account in the data flow ?▶ Whether the DQM/QA results have to become persistent and for how
long ?▶ What does “Calibration QA” mean to you ?
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 19
Survey status
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 20
Subsys Ack Contact pers. Under disc. Answered ClearedACOCPVEMCFMD Absent from Run 3HMP During meetingMCHMTR See MIDPHOSPMDSDD See ITS2SPD See ITS2SSD See ITS2T0 See FITTOFTPCTRDV0 See FITZDCTriggerDAQMFTMIDFITITS 2
Current and short term work
▶ Run 2 : Prepare a proposal for the QA in relation with the online reco, calibration and monitoring (QA tasks, validation, bridge to DQM, …)▶ Online Calibration
▶ Mainly for TPC▶ Many open questions on
▶ Requirements▶ General architecture▶ Implementation▶ CDB
▶ Working on a proposal to meet requirements while minimizing work ▶ Use analysis QA train within HLT for reco monitoring▶ Use analysis QA train within HLT for calib monitoring▶ Use AMORE for raw data monitoring▶ Use AMORE infrastructure for storage and visualization
▶ Run 2 – Review and preparation▶ Detectors « interviews »▶ (DATE Monitoring update)
▶ Run 3 : Prepare requirements of the future system following CWG1 input▶ For the TDR (2014)
▶ Define requirements and general architecture and features of the QA-DQM-Viz for Run 3▶ Write it
▶ Event Display▶ Decentralized model under implementation
▶ Better stability▶ Split GUI and reconstruction▶ Possibility to switch between offline and HLT reco
▶ Bookmarks (for users and for PR)▶ Involvement of the Warsaw group▶ Implemented by the end of 2013
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 21
B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014 22
HCDBDCS, GRP
Reco Selection/filters Calib
OCDB
Data on Castor
Reading Writing ProducingDBs data procedures
ESDsRaw Calib param
PHYSICS Run
QC
QC
QC
(sort of)
Top Related