Xbox Software development – an alternative approachKai Papke

Current Workflows

● LabView

● LabView’s DIAdem

– Limited number of loaded data sets

– Very very slow● Matlab

– requires very memory intensive and time consuming conversion of files

● Python

– No conversion required, direct use of tdms files

TDMS file

Motivation● Open source

– No license issues– Avoid huge software packages e.g. Matlab, LabView

● Memory efficient data evaluation– Conversion between .tdms and .mat files very intense– Prefer formats designed for large data amounts (partial load)

● Fast data evaluation– High-level languages e.g. Matlab, LabView, Python may be very slow in

certain situations → Avoid bottle-necks e.g. loops● Maintainability and adaptability

– Modular but not too modular framework (e.g. avoid all-in-one-scripts)– Generic (e.g. avoid fixed size arrays)– Clear variable scope (e.g. no global variables)

So, lets try C++ and ROOT ...

Why C++ and ROOT?● ROOT files

– 20 years of experience with big data

– very user definable

– Various options to read and evaluate data in single and multithreading

ROOT HDF5 PB SQL Parquet Avro

Self-describing

x x - x x x

Nested types x ? x ? x x

Columnar layout

x ? - - x -

Compres-sion

x ? x - x x

x supported, - unsupported, ? Difficult/ unclear

https://indico.cern.ch/event/567550/contributions/2628878/attachments/1511966/2358123/hep-file-formats.pdfhttps://indico.cern.ch/event/613842/contributions/2585787/attachments/1463230/2260889/pivarski-data-formats.pdfhttps://indico.cern.ch/event/613842/contributions/2590628/attachments/1463381/2261193/hep-file-formats.pdf

Why C++ and ROOT?● Flexible framework

– Interactive environment as in Matlab or IPython available

– Bindings to Python opens the door to higher level scripting language

– Kernel for Jupyter

– Creation of user defined libraries and stand alone apps

Jupyter notebook

Why C++ and ROOT?● Scientific libraries

– Fitting, statistics, and minimiser tool boxes widely used features of ROOT

– Various other scientific packages available (linear algebra, fft, machine learning, image processing, ...)

Math libraries and packages of root

https://root.cern.ch/root/html534/guides/users-guide/ROOTUsersGuideA4.pdf

Drawbacks of using C++● Requires knowledge of data types, functions,

pointers, classes, ... ● Compiling and linking

– done by ROOT as long as within the ROOT framework

2nd point must be

as simple as possible

… but what if we want to go beyond ROOT?

Framework

InteractionInterpreter like Environment,

Gui apps

BuildingAutomatized process,

facilitates to extend the project, add packages,resolve dependencies,

Support of Linux, Windows, and MacOS

I/OReading TDMS files,Writing ROOT and

HDF5 files

AnalysisFiltering,

Breakdown localization,Pulse width, gradient, ...

CoreInternal data representation

Core ● Representation of

DAQ channels● Simple operations

– Interval selection– Max, Min, mean, ...– Calibration

(scaling)

Example of a file converter panel

I/O● Reading of TDMS files and mapping to common

format between Xboxes● Reading and writing of ROOT files● Writing of HDF5 files (for Matlab users)

command size Location of original

Time

Copy to local folder 500 Workspace 7 sec

python load tdms file 150 Local folder 2-4 sec

root load tdms file 150 Local folder 0.6 sec

python load tdms file 150 Workspace 33 sec

root load tdms file 500 Workspace 7.5 sec

Performace for loading tdms files

Conversion of TDMS to ROOT files ~60 times faster

than conversion of

TDMS to MAT files

Analyses● Signal interpolation and re-

sampling to improve accuracy

● Polynomial Noise filter to increase SNR

● Edge detection based on derivative, differences, and thresholds

● Signal alignment and localization of the divergence point

● Phase wrapping and unwrapping

Analyses● Breakdown localization still in progress ...

MATLAB ROOT

downmixer not locked to masterclock (1.4.2018)

Analyses● ROOT’s Draw option allows to plot filtered data directly

– E.g. power and pulse length of non-breakdown events:

● Famous new tool: RDataFrameshttps://root-forum.cern.ch/t/rdataframe-a-modern-tool-to-manipulate-and-analyze-root-datasets/29384

Interaction● Web based applications

– Future of ROOT graphics– Uses XML and Java– A bit too early stage

Example of a file converter panel

https://indico.cern.ch/event/697389/contributions/3062039/attachments/1712196/2760672/webwindow.pdf

Interaction● QT gui applications

– relatively easily implemented

– Potential for real time applications

– Very similar to PyQt

Example of a converter app

What next?● QT Data Viewer● Extending and simplifying analyser package● Adding TrendData● Getting users● Machine learning for BD detection

What next?● What about going beyond data analysis?

http://www.ni.com/white-paper/14623/it/

Thank you

● Misleading Dark current

reflectedamplitude

transmittedamplitude

transmittedphase

dark current

reflectedamplitude

transmittedamplitude

transmittedphase

dark current

● Breakdown before pulse not detected

reflectedamplitude

transmittedamplitude

transmittedphase

dark current

● Oscillations pulse disturb trigger

● Loading root files without root installation

`

● HDF5 Files

● Platform independent building tool: cmake ● facilitates to extend the project, add packages,

resolve dependencies

├── analyser├── cmake├── CMakeLists.txt├── converter├── core│   ├── CMakeLists.txt│   ├── xboxbase│   └── xboxchannel├── io│   ├── CMakeLists.txt│   ├── hdf5│   ├── root│   └── tdms├── macros├── README.md

Macros to find installed packages (Boost, GSL, HDF5, …)

Adaptation to Windows, Linux, MacOSAutomatic ROOT Dictionary generation

Automatic library and executable generation

Analysis of Xbox data (BD localisation, Bulse length, Gradient,

...)

Conversion from TDMS to root or HDF5 files

Basic xbox data representation during reading, analysing and writing

Library for providing fundamental io features:

Reading TDMS filesReading and writing ROOT filesWriting HDF5 files (for Matlab)Example macros to be used directly

within the ROOT frame work

● ROOT files

– Conversion from .tdms to .root files about 60 times faster than conversion from .tdms to .mat files

– Compression rate 30 - 50 % with respect to tdms files

– Comparing apples with beans: ten to twenty times faster evaluations with respect to pure python solution

0 10 20 30 40 50 60 70 80 90 100

ROOT using ROOT files

Python using ROOT files

Python using TDMS files

Run time [%]