David Adams

ATLAS

ADA, ARDA and PPDG

David AdamsBNL

June 28, 2004

PPDG Collaboration MeetingWilliams Bay, Wisconsin

ATLAS and ARDA ARDA Workshop June 21, 2004 2

David Adams

ATLAS

Contents

Definitions

ADA/DIAL model

Current status of ADA

Connections to PPDG

Conclusions

ATLAS and ARDA ARDA Workshop June 21, 2004 3

David Adams

ATLAS

DefinitionsADA – ATLAS Distributed Analysis

• ATLAS project to deliver an end-to-end analysis system

ARDA – A Realisation of Distributed Analysis for LHC• CERN project to coordinate the development of prototype

distributed analysis systems for the LHC experiments

• Prototypes are based on the EGEE middleware (gLite)

DIAL – Distributed Interactive Analysis of Large Datasets• BNL project to study interactive analysis of large amounts of data

• Delivered framework is also suitable for non-interactive jobs

AJDL – Abstract Job Definition Language• Generic and extensible definition of jobs intended for use with any

batch or workload management system

• Components: transformation (application and task), dataset and job

ATLAS and ARDA ARDA Workshop June 21, 2004 4

David Adams

ATLAS

ADA/DIAL modelUser interaction scenario:

• Select a dataset– AJDL, not HEPCAL

• Examine dataset– Display content (event ID’s, type-keys)– Location of data (logical files, physical files, DB, …)

> Including means to localize data

• Define transformation– Application that processes the data– Task carrying data to configure application– Select these from catalog and modify

• Create job to apply transformation to dataset– Submit app, task and dataset to scheduler (e.g. analysis service)– Result will be a new dataset

ATLAS and ARDA ARDA Workshop June 21, 2004 5

David Adams

ATLAS

ADA/DIAL model (cont)User interactions (cont)

• Monitor job– Status (running, done, failed, …)

– Status of sub-jobs

– Examine partial result

• Examine final result (output dataset)

• Repeat

Following block diagram shows major components• Client

• High-level services

• Middleware services

ATLAS and ARDA ARDA Workshop June 21, 2004 6

David Adams

ATLAS

ADA/DIAL model (cont)

M id d lew ar e s er v ic e in te r f ac es

C EW M S F ileC ata lo g

S E . . .e tc . M id d lew ar es er v ic es

H ig h lev el s erv ice in terfaces (A J D L )

In te ra c t iv eA n a ly s isS e rv ic e

Batc hAn aly s isS er v ic e

AT P R O DAn aly s isS er v ic e

R O O Tc m d lin e

C lien t

G AN G Ac m d lin e

C lien t

G AN G AT as k

M an ag em en t

C ap o n eAn aly s isS er v ic e

G r ap h ic a lJ o b

Bu ild er

G AN G AJ o b

M an ag em en t

Hig h - lev e ls er v ic es

C lien t to o ls

AR D AAn aly s isS er v ic eC ata lo g

s er v ic es

G AN G A G UI

D atas e tS p lit te r

D atas e tM er g er

J o bM an ag em en t

Release 1.0 Started

ATLAS and ARDA ARDA Workshop June 21, 2004 7

David Adams

ATLAS

Current status of ADAFramework based on DIAL 0.90

Clients• ROOT from DIAL (using rootcint)

• Python from GANGA (using lcgdict with DIAL classes)

• GUI being developed by GANGA

ATLAS and ARDA ARDA Workshop June 21, 2004 8

David Adams

ATLAS

Current status of ADA (cont)Analysis services

• DIAL provides the starting point for all the following

• Interactive service (i.e. low latency for short jobs)– Currently uses a fast LSF queue

– Condor COD being tested

– Running at BNL

– Will be deployed at other sites

– May also be deployed for long-running jobs

• Service to insert jobs in ATLAS production queue is being developed

– Jobs may be run on any of the ATLAS grids

• ARDA service based on gLite being developed– Jobs sent to gLite WMS

ATLAS and ARDA ARDA Workshop June 21, 2004 9

David Adams

ATLAS

Current status of ADA (cont)Applications

• DC1 CBNT histograms in place

• DC2 reconstruction for release 8.0.1 in place

• Working on DC2 event data histograms

• Capability to reuse production xforms being developed

Datasets• Generic classes in DIAL

• DC1: CbntDataset and HbookDataset

• DC2: AtlasPoolEventDataset and RootHistogramDataset

Catalogs• AMI adding required tables and user interface

– Repositories, selection catalogs, …

• Programmatic access via AMI web service

ATLAS and ARDA ARDA Workshop June 21, 2004 10

David Adams

ATLAS

Connections to PPDGDIAL

• Provides many parts of ADA– The overall framework

– The ROOT client

– Infrastructure for all analysis services delivered and under development

– Connection to LSF, Condor (including COD)

• Depends on PPDG for support and exchange of ideas

ATLAS and ARDA ARDA Workshop June 21, 2004 11

David Adams

ATLAS

Connections to PPDG (cont)AJDL

• Facilitates the development of clients or services without DIAL

– E.g. plan for GANGA python interface is to drop dependence on DIAL C++ libraries

> Lightweight client for any platform

• Goal to standardize within PPDG– Send non-ATLAS jobs to ATLAS analysis service

– And vice versa

ATLAS and ARDA ARDA Workshop June 21, 2004 12

David Adams

ATLAS

Connections to PPDG (cont)U.S. grid infrastructure

• U.S. sites important to ATLAS – And especially to USATLAS

• Assume continuing support from PPDG– Grid3, grid3+, OSG, …

• Most sites continue taking jobs from ATPROD queues

• A few sites to also host interactive analysis service

ATLAS and ARDA ARDA Workshop June 21, 2004 13

David Adams

ATLAS

Connections to PPDG (cont)Monitoring and modeling

• US monitoring data fed into ATLAS computing model– Job data from ADA catalogs

– Site data from grid monitoring

• ACM is interested in constructing models to understand analysis patterns

– Collaboration with CS-11?

ATLAS and ARDA ARDA Workshop June 21, 2004 14

David Adams

ATLAS

Connections to PPDG (cont)EGEE middleware

• Install gLite on one or more U.S. sites– At least services required to support WMS

> CE, SE, …

• Submit jobs from the ATLAS-ARDA prototype

• Common project– Also submit jobs from CMS and Alice prototypes

• Will U.S. sites deploy EGEE middleware?– Important issue for evolution of ADA

ATLAS and ARDA ARDA Workshop June 21, 2004 15

David Adams

ATLAS

Connections to PPDG (cont)File cataloging and movement

• SRM likely to become standard– Licensing issues?

• Many replica catalogs relevant to ATLAS– Magda (phased out for DC2)– EDG RLS (LCG catalog)– ISI RLS (separate catalogs for U.S. and NorduGrid)– EGEE/AliEn file catalog (required for ARDA prototype?)

• Also many options for high-level interfaces– ATLAS internal system Don Quijote– CMS file transfer system– EGEE file transfer system (same as CMS?)– LBL RMS/RRS

• PPDG can help sort this out

ATLAS and ARDA ARDA Workshop June 21, 2004 16

David Adams

ATLAS

ConclusionsATLAS has benefited in may ways from PPDG

• Magda, DIAL, AJDL, SRM, Grid3, …

• Ideas in HEPCAL and ARDA

Continue to look to PPDG• Forum to exchange ideas

• Provider or identifier of useful tools and services

• Support for U.S. grid infrastructure