Data management with Rucio for XENONnT

3rd Rucio WorkshopFermilab March 10-12 2020

University of ChicagoLuca Grandi Evan ShockleyRobert GardnerJudith StephenPascal PaschosLIncoln Bryant

University Wisconsin MadisonBenedikt Riedel

Stockholm UniversityBoris BauermeisterJan Conrad

USC-ISIMats Rynge

Rice University Chris Tunnell

paschos@uchicago.edu

XENON updates: nT● New phase for dark marker search experiment at LNGS Facility (Italy) ● Expands detector sensitivity with 8 tonnes of liquid Xe.● Notable changes:

○ TPC (time projection chamber)+neutron veto+muon veto○ Increased raw data volume - up 2 times from XENON1T○ Upgraded processing and analysis pipelines:

■ Base_environment: base computing framework for nT ■ Strax: Analysis framework for nT - replaces pax+hax■ Outsource: Job submission pipeline for nT (Pegasus) - replaces portions of CAX

(Mats Rynge)■ Admix: python wrapper for Rucio uploads - replaces Ruciax

● Updates in infrastructure at UChicago and LNGS

Lessons learned from XENON1T● XENON1T operated from 2016-2018 at the Gran Sasso National

Laboratory (LNGS)○ Collected over 800 TB of raw data ○ 10 peer-reviewed publications so far by the collaboration in the search for Dark Matter

● For 1T, Rucio and job submission to OSG was not in place at the start of the experiment. Collaboration transferred data from LNGS to Uchicago for processing on local HPC (Midway)○ UChicago team (Judith Stephen, Benedikt Riedel) in collaboration with Boris

Bauermeister (University of Stockholm) deployed, operationalized and integrated Rucio Services for the XENON data management

○ In collaboration with USC-ISI (Mats Rynge) integrated job submission to OSG for processing

● Lesson Learned: Be better prepared at the start of the experiment to meet scale of compute and storage requirements

Lessons learned (continued)● For 1T, there was no local storage for the event builder. If the latter

crashed, it could take down the DAQ and stop the flow of data○ Introduced the use of a local buffer to prevent this from happening for nT

● Processing pipeline for 1T (CAX) was too ‘monolithic’ as it was in charge for all pipeline components (updating DB, processing, creating minitrees, checking transfers etc.). Data re-processing was slow and inefficient ○ Resolved by modularizing the pipeline using existing tools (e.g. Strax) repurposed for

XENON (straxen)

Rucio XENON

● Data live on disk/tape on our various Rucio Storage Endpoints (RSEs) around the world - all managed by Rucio

● Supported distribution of data during previous XENON phases● Major Updates between 1T to nT

○ store raw, reduced and processed data in Rucio■ Optional: store MC simulations for each run

○ Updated Rucio to 1.20.6, migrated mariaDB to postgresql and updated new DB to the 1.20.6 schema

○ Added an RSE for “DaLI” at UChicago Research Computing Center to augment storage for analysis

Data Pipeline XENONnT

DAQ Event Builder xent-datamanager (LNGS_USERDISK)

NIKHEF_USERDISK

CCIN2P3_USERDISK

WEIZMANN_USERDISK

CNAF_USERDISK

SURFSARA_USERDISK

UC_OSG_USERDISK

CNAF_TAPE_USERDISK

UC_DALI_USERDISK

Midway Cluster (UChicago)

Open Science Grid (OSG)+

European Grid Infrastructure (EGI)

raw_records,records, peaks, ...

● Computing Scheme for XENONnT - presented during the 2nd Rucio Workshop in Stockholm by Boris Bauermeister et al

Rucio RSEs for 1T ~800 TB of data

Data processing-analysis pipeline

● Strax/straxen is the new, fast processing and analysis framework for XENONnT

● Replaces previous scheme (pax): raw data, processed data and minitrees● Strax allows for checkpointing the pipeline (state keeping)

○ Critical data objects are stored during processing stages ○ Other data objects can be recreated from the nearest stored one it depends on if a

parameter changes. This accelerates reprocessing campaigns.○ Stored data objects, e.g. raw_records, records (reduced raw) and peaks ( pulses in the

PMT - photomultiplier tubes) are ingested in and distributed by Rucio to RSE end points● Online: processing directly out of DAQ before Rucio ingestion ● Offline: Grid computing for processing ingested files in Rucio

Evan Shockley, Mats Rynge, Benedikt Riedel

Expanded scopes in catalogue for nT ● XENON1T →x1t_SR001_181129_1730_tpc:raw (both to Disk and Tape)

● XENONnT → ○ Xnt_170321_0520_tpc:corrected_areas○ Xnt_170321_0520_tpc:energy_estimates○ Xnt_170321_0520_tpc:event_basics○ xnt_170321_0520_tpc:event_positions○ Xnt_170321_0520_tpc:events○ xnt_170321_0520_tpc:peak_basics○ Xnt_170321_0520_tpc:peak_classification○ Xnt_170321_0520_tpc:peak_positions○ Xnt_170321_0520_tpc:peaks # first level of processed data of TPC○ Xnt_170321_0520_tpc:records #data-reduced raw data of TPC ○ Xnt_170321_0520_tpc:raw_records #raw data of TPC (tape)○ Xnt_170321_0520_mv:raw_records #raw data of muon veto○ Xnt_170321_0520_nv:raw_records #raw data of neutron veto

Aggressive ingestion into Rucio

Minimal ingestion into Rucio

Gateway to OSG for XENON● Grid Processing:

○ Likely needed during commissioning when gains, maps, etc. not ready

○ reprocess low-level data

○ Backup to online processing

○ Handled by Outsource, a wrapper around the workflow manager Pegasus

OSG login node

raw_records-000001

raw_records

raw_records-000000

raw_records-000002

raw_records-000003

Mats Rynge, Evan Shockley Grid Processing with Outsource

OSG login node

raw_records-000001

raw_records-000000

raw_records-000002

raw_records-000003

records-000001

records-000000

records-000002

records-000003

Rucio

raw_records records

Grid Processing with Outsource Mats Rynge, Evan Shockley

OSG login node

raw_records-000001

raw_records-000000

raw_records-000002

raw_records-000003

records-000001

records-000000

records-000002

records-000003

Rucio

raw_records records

recordspeaks

peaks Midway

Rucio

Grid Processing with Outsource Mats Rynge, Evan Shockley

aDMiX Boris Bauermeister https://advance-data-management-in-xenon.readthedocs.io/en/latest/usage.html

● advanced Data Management in XENON (aDMiX) handles Rucio API calls It allows: ○ Uploads/Downloads ○ Init Rucio transfer rules (runDB supported) ○ Update run database with RSE locations

● ADMIX will (mostly) run on xent-datamanager to upload data into Rucio and move to respective sites○ raw_records to tape storage (CNAF_TAPE_USERDISK), once we’re sure we don’t

need them anymore○ records to grid sites, e.g. UC_OSG_USERDISK, NIKHEF_USERDISK, etc. ○ High level data to analysis sites (UC_DALI_USERDISK, /dali/path/to/data)

● Middleware between Rucio and XENON ‘runs database’ that tracks information for each dataset.

Rucio Infrastructure Upgrades Judith Stephen UChicago

● Upgraded Rucio on the frontend server from 1.8.5 to 1.20.6○ Required the usual schema changes (easy via SQLAlchemy, no issues)○ Needed to move back to using the rucio-conveyor-submitter daemon instead of

the rucio-conveyor-transfer-submitter; otherwise the Rucio upgrade itself was smooth

● Migrated backend database from MariaDB 10.3 to PostgreSQL 11○ Used pgloader to perform the actual database migration

■ Due to the size of some of the tables, pgloader was very slow■ Ended up dropping some of the *_history tables in the original conversion

and eventually migrating later○ Problems with converting the various UUID columns, resolved by manually

pivoting the affected columns○ Mario helped verify the final schema (thank you!)

XENON1T data

● Raw data: keep at least a tape copy of the 1T catalogue for the extended future. Still have some time before disk space becomes an issue

● Processed data: Due to limits in project allocation on Midway, the start of nT operations will necessitate moving 1T processed data out

Summary

● Hardware and software setup is largely completed● XENON detector assembly ongoing, science data expected this year● Still to do

○ Build all services at LNGS (xent-datamanager) including the new RSE for that endpoint

○ Continue testing the full analysis software chain for any last minute bug fixes

● Questions?

Extra Slides

strax/straxen - Data Chunking● As the data moves through the chain and various

algorithms refines the “Data chunk”● What does this mean in practice?

○ raw_records - Portions of time separated but detector inactivity for X seconds

○ records - First refinement step○ peaks - Finding PMT (photomultiplier tubes)

pulses in data stream○ peak_* - Peak information○ events - Creating a grouping peaks that are

clustered

strax/straxen updated graph

strax/straxen - Keeping State● Various algorithms require various levels of input

data and data fidelity - Peaks can be processed on a “chunk” level, while “events” need to processed on a per-run level

● Keeps state○ e.g. when the energy_estimates algorithm

changes only produce new event_info objects○ Great for reprocessing - Less churn through

data● Comes at the cost of workflow complexity

and many tiny files

Admix code snippet

base_environment GitHub repo

CVMFS (Singularity image and source-able environment)CVMFS is a globally distributed read-only filesystem, available across Open Science Grid, EGI and Midway.

Interactive login and worker nodesSingularity images: /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-xenon:{version}Source-able environment: /cvmfs/xenon.opensciencegrid.org/releases/nT/{version}

Docker ImageDeployHQ first builds a new version of the Docker Image using the Dockerfile and create-env files.

CVMFS Tarball (source-able)The newly build Docker image and again the create-env scripts are used to also build a base_environment under /cvmfs - this can be used as a “source-able” environment just like in XENON1T

Checkins to the GitHub repo triggers DeployHQ builds

docker push ...

Container image is hosted in DockerHub (opensciencegrid/osgvo-xenon) The resulting tarball is deployed into the CVMFS server and published.

Synchronized automatically by OSGContainer image is hosted in Sylabs Cloud

(rynge/default/xenonnt)

Singularity ImageDeployHQ builds a Singularity image from the Docker image