The ATLAS EventIndex: An event catalogue for experiments ... fileThe ATLAS EventIndex: An event...
Transcript of The ATLAS EventIndex: An event catalogue for experiments ... fileThe ATLAS EventIndex: An event...
The ATLAS EventIndex: An event catalogue for
experiments collecting large amount of data
Andrea Favareto Università degli Studi di Genova & INFN Genova
18th Jun 2014 - Riunione PRIN, Bologna
Introduction• Modern scien,fic experiments collect very large amount of data
‣ ATLAS: e.g. 2 billion real and 4 billion simulated events in 2011 and 2012
• A catalog of data is needed according to different point of view
‣ mul,ple use cases and search criteria
• A database that contains the reference to the file that includes every event at every stage of processing is necessary to recall selected events from data storage systems
‣ In ATLAS an EventTag database already exists
-‐ designed in the late 1990’s
-‐ Oracle databases with separate tables for each reprocessing cycle. Implementa,on in Oracle par,cularly labor-‐intensive
-‐ each event is recorded several ,mes, once for each cycle of reconstruc,on
‣ EventTag is poten,ally very useful but very liLle used, at least in its DB format
-‐ main use: events skimming
2
EventIndex• GOAL: design a more agile system (“NoSQL” databases), compared to TagDB, that keeps
the func,onality of skimming by keeping only the pointers (currently GUIDs) to the files where the event in ques,on can be found at every stage of processing and elimina,ng other variables of lower importance
• EventIndex
‣ a complete catalog of ATLAS events
-‐ all events, real and simulated data
-‐ all processing stage
‣ contents
-‐ event iden,fiers (RunNumber, EventNumber, LumiBlockN, …)
-‐ online trigger paLern and hit counts
-‐ references (GUIDs) to the events at each processing stage (RAW, ESD, AOD, NTUP) in all permanent files on storage
3
Use Cases• old use cases (from TagDB)
‣ event picking
-‐ give me the reference (pointer) to “this” event in “that” format for a given processing cycle
‣ event skimming
-‐ give me the list of events passing “this” selec,on and their references
‣ produc,on consistency checks
-‐ technical checks that processing cycles are complete
• new use cases
‣ event service
-‐ give me the references for “this” range of events
4
Data Flow• Informa,on for the EventIndex is collected from all produc,on jobs running
at CERN and on the Grid and transferred to CERN using a messaging system
• This info is reformaLed, inserted into the Hadoop storage system and internally catalogued and indexed
• Query services (CLI and GUI) implemented also as web services query the EventIndex and retrieve the informa,on
5
EventIndex project breakdown
• Defini,on of 4 major work areas (or tasks)
‣ core architecture
-‐ design and prototyping, then development and deployment of the core infrastructure to implement the EventIndex using Hadoop technology; ,mescale: summer 2014
‣ data collec,on and storage
-‐ design and prototyping, then development, deployment and opera,on of the infrastructure to collect from Grid and non-‐Grid jobs the informa,on needed by the EventIndex, and transfer and upload it to the EventIndex DB; ,mescale: autumn 2014
‣ query services
-‐ adapta,on of the web services and APIs now used to query the Tag DB to the EventIndex using Hadoop technology; ,mescale: summer 2014
‣ func,onal tes,ng and opera,on
-‐ end-‐to-‐end tests of the completeness, func,onality and performance of the EventIndex and related services. Checks of the opera,on procedures and sebng up monitoring tools; ,mescale: end 2014
6
Data collection
7
• Data to be stored in the EventIndex are produced by all produc,on jobs that run at CERN or on the Grid
‣ snippet of informa,on of permanent output files (file unique iden,fier, event relevant aLributes) has to be sent to central server @CERN
• producers run within the “pilot” processon each worker node and transmit the datausing a messaging system to a queue in oneof the endpoints
• asynchronous consumers con,nuously get new data, check their validity with theproduc,on system and pass them into Hadoop
Core architecture• data are stored in Hadoop map files
‣ event aLributes stored in several groups (constants, variables, references,…)
‣ data can be indexed (e.g. using inverted indices for trigger info). Index files just have key + references to data files. Some index files created at upload, others added later. Results of queries can be cached to be used
• searches can be performed in two steps:
‣ get reference from index file
‣ using that ref., get data
• searches can be performed using keys (immediate results), or on data files
• full searches can be done with full scans or using MapReduce jobs
• access to the data is achieved by a single and simple interface for:
‣ upload: copy file into HDFS
‣ read: get & search
‣ update: add new (ver,cal) data
‣ index: create new indices
8
Core system - Performance tests• tests done with EI CLI and web service
• tests on data11_7TeV,physics_Muons,f415_m1025_m1024 (t0 processing, 16 runs, Sep-‐Nov 2011)
• some use cases I’m considering are:
1) COUNT/RETRIEVE events W/O FILTER (which simply means to count all the events in the dataset without any par,cular request)
2) COUNT/RETRIEVE events W/ trigger AND/OR W/runnumber
3) COUNT/RETRIEVE events (e.g. token) W/ GUID
• Queries that give the run number and event number (event picking use case) return their result in ~5 seconds
‣ data in Hadoop catalogue are indexed by RunNumber-‐EventNumber key
‣ currently we are thinking about to generate many indices for each event variable and to automa,se the search in order to use the proper index file depending on the variable on which the query is performed
9
Core system - Performance tests
10
#test selec,on #events frac,on (%) total ,me
RETRIEVE events W/O FILTER -‐ 20371303 100 08:50,4
RETRIEVE events W/ trigger EF_mu18 3435250 16,86 03:03,2
RETRIEVE events W/ runnumber 189822 6448550 31,66 03:05,9
RETRIEVE events W/ runnumber AND W/ trigger EF_mu18 AND 18922 1353032 6,64 02:29,1
RETRIEVE W/ GUID BE456704-‐BE08-‐E111-‐BC39-‐0030489470C6 8410 0,04 01:27,2
• #events cross-‐checked with previous tests and with iELLSI (EventTag web service)
• total ,me can fluctuate a bit (~ ±30 sec)
• output txt file created with all events informa,ons
Core system - Performance tests
11
#test selec,on #events frac,on (%) total ,me
COUNT events W/O FILTER -‐ 20371303 100 02:04,5
COUNT events W/ trigger EF_mu18 3435250 16,86 01:57,5
COUNT events W/ runnumber 189822 6448550 31,66 02:02,8
COUNT events W/ runnumber AND W/ trigger EF_mu18 AND 18922 1353032 6,64 02:03,7
COUNT W/ GUID BE456704-‐BE08-‐E111-‐BC39-‐0030489470C6 8410 0,04 01:23,7
• now simply count the events and output txt file NOT created
Hadoop cluster load tests
• The idea is to run some queries simultaneously on the hadoop cluster and measure it’s performances in this condi,ons
• in order to compare directly the results, the same query has been considered
• mul,ple queries on
‣ same dataset
‣ different datasets
• mul,ple queries from
‣ single user -‐ a single user can run up to 25 jobs simultaneously from the same machine
‣ two users
12
Hadoop cluster load tests: single user
13
#jobs SAME DATASET: total ,me
DIFFERENT DATASETs: total
1 01:32,3
2 01:43,5
5 02:42,8 02:24,0
10 03:19,7 02:31,5
15 04:31,1
20 05:35,2 02:55,7
26 (13x2 different machines) 06:32,5
30 (15x2 different machines) 07:49,7 03:49,2
36 (18x2 different machines) 08:49,4
40 (20x2 different machines) 09:28,3 04:07,2
00:00.0$
01:26.4$
02:52.8$
04:19.2$
05:45.6$
07:12.0$
08:38.4$
10:04.8$
0$ 10$ 20$ 30$ 40$ 50$
same$query$
different$queries$
same dataset different dataset
Hadoop cluster load tests: two users
14
SAME DATASET: total ,me
DIFFERENT DATASETs: total ,me
# total jobs USER 1 USER 2 USER 1 USER 2
6 (3+3) 02:35,5 02:36,2 01:39,7 01:32,9
10 (5+5) 02:58,2 03:11,3 02:32,7 02:00,1
16 (8+8) 04:32,3 04:34,2 02:27,7 01:59,7
20 (10+10) 05:11,5 05:05,6 02:32,6 02:03,4
26 (13+13) 06:07,3 05:47,6 02:46,2 02:05,2
30 (15+15) 07:44,6 07:42,1
00:00.0$
01:26.4$
02:52.8$
04:19.2$
05:45.6$
07:12.0$
08:38.4$
0$ 5$ 10$ 15$ 20$ 25$ 30$ 35$
Andrea$
Claudia$
dif_queries_Andrea$
dif_queries_Claudia$
sd: USER 1 sd: USER 2 dd: USER 1 dd: USER 2
Hadoop cluster load tests: comments
• tests show that
‣ the max number of jobs/user should be set to a large number to allow many jobs to run concurrently
‣ the response ,me depends linearly on the number of concurrent jobs. It doesn’t maLer who the concurrent jobs belong to
‣ the response ,me increase much slower for queries on different datasets
• not surprising, but good!
15
Conclusions ad outlook
16
• The EventIndex project is on track to provide ATLAS with a global event catalogue in advance of the resump,on of LHC opera,ons in 2015.
• The global architecture is defined and prototyping work is in progress; preliminary results are encouraging.
• Current work:
‣ full implementa,on of the final prototype, loaded with Run1 data
‣ extend the catalog also to MC
‣ comple,on of deployment and opera,on infrastructure
‣ sebng up of monitoring tool just started
‣ Tes,ng other ways to store EI info into HBASE only (YAEIS -‐ Yet Another Event Index Server)
Conference communications, posters and proceedings
17
• Poster at CHEP 2013, Amsterdam, Oct 2013
‣ The ATLAS EventIndex: an event catalogue for experiments collec>ng large amounts of data, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1606537;ATL-‐SOFT-‐SLIDE-‐2013-‐805
• Talk at CHEP 2013, Amsterdam, Oct 2013
‣ The future of event-‐level informa>on repositories, indexing, and selec>on in ATLAS, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1623340; ATL-‐SOFT-‐SLIDE-‐2013-‐864
• Proceedings for CHEP 2013, Amsterdam, Oct 2013
‣ The ATLAS EventIndex: an event catalogue for experiments collec>ng large amounts of data, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1606543; ATL-‐SOFT-‐PROC-‐2013-‐012 – published in J. Phys. Conf. Ser. 513 042002 doi:10.1088/1742-‐6596/513/4/042002
• Proceedings for CHEP 2013, Amsterdam, Oct 2013
‣ The future of event-‐level informa>on repositories, indexing, and selec>on in ATLAS, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1625848; ATL-‐SOFT-‐PROC-‐2013-‐046 – Published in J. Phys. Conf. Ser. 513 042009 doi:10.1088/1742-‐6596/513/4/042009
• Talk at ISGC 2014, Taipei, Mar 2014
‣ The ATLAS EventIndex: an event catalogue for experiments collec>ng large amounts of data, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1670054; ATL-‐SOFT-‐SLIDE-‐2014-‐118
• Proceedings for ISGC 2014, Taipei, Mar 2014
‣ The ATLAS EventIndex: an event catalogue for experiments collec>ng large amounts of data, D.Barberis, A.Favareto, et al. hLp://cds.cern.ch/record/1690609; ATL-‐SOFT-‐PROC-‐2014-‐002 – To be published in Proceedings of Science.
Backup
18
H/W of Hadoop cluster
19