DataGrid Applications
Federico Carminati
WP6 WorkShop
December 11, 2000
DataGrid WP6 Workshop
211 December 2000
The distributed The distributed computing modelcomputing model
AssumptionsRaw data will be kept at CERN – (backup if affordable)
Tier 1 will have ~10% of raw data
Reconstruction will be done at CERN (2 passes)
ESD (aka DST) and TAG will be shipped to Tier1-2
Simulation will all be done in the Tier1-2, data will be shipped to CERN (in which form? Raw, ESD, AOD)
The Tier1-2 will process ESD to produce AOD (formerly aka n-tuple) as many time as necessary, producing their own TAG
Users will access the ESD/AOD/TAG remotely
DataGrid WP6 Workshop
311 December 2000
The distributed The distributed computing modelcomputing model
Basic principleEvery physicist should have in principle equal access to the data and to the resources
The system will be extremely complexNumber of components in each site
Number of sites
Different tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis
DataGrid WP6 Workshop
411 December 2000
WP 8 philosophyWP 8 philosophy
Define a common upper middle-layer of GRID services common to the 4 experiments
Common API for common tasksFile replicaJob submission and monitoring….
Collaborate to define a common set of requirements and milestones, also with WP 9/10Share the same testbeds and facilities for data challengesIntroduce a user view in the project
DataGrid WP6 Workshop
511 December 2000
WP8 plans and WP8 plans and requirementsrequirements
Testbed Release 0 (1Q 2001)A working, standard, installation of Globus, at CERN and at other labs.Standard recipes for file transfer and job submission (at CERN, this installation will probably be interfaced to LSF).A contact point for GLOBUS and later DataGRID software in each lab incase of problemsA clear policy for "experiment-wide" authorisation to allow testing across national boundaries.
This will only work if enough support is provided to usersWP8 activities in general require to have a substantial part of the GRID services available since as soon as possible
DataGrid WP6 Workshop
611 December 2000
WP8 plans and WP8 plans and requirementsrequirements
Testbed release 1 (3Q 2001)Distributed user autentication and resource allocation or pre-allocation, as dynamic allocation can come later.
Distributed data dictionary (location of files on different servers).
Basic network configuration, monitoring and diagnostic tools.
Basic monitoring and diagnostic of a cluster of PC's.
Distributed scheduling for the jobs that are submitted in a coordinated way (this does not include the "chaotic job activity" coming from isolated users which should be addressed in the subsequent release).
Access to the basic information about job status and errors.
Guidelines for configurating farms of PC's with fast disk access.
DataGrid WP6 Workshop
711 December 2000
WP8 plans and WP8 plans and requirementsrequirements
Testbed release 2 (3Q 2002)Replica management and network optimised trasfer of data from different file systems.
Tools for configurating farms of PC's with fast disk access, monitoring their status and for automathized s/w installation and management.
A prototype of scheduling and load balancing for chaotic analysis jobs
Basic functionality for dynamic resource allocation
Basic functionality for job partitioning
DataGrid WP6 Workshop
811 December 2000
WP8 plans and WP8 plans and requirementsrequirements
Testbed release 3 (3Q 2003)Scheduling and balancing of chaotic analysis
Tools to ensure "robustness" and error recovery of the system
DataGrid WP6 Workshop
911 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
WP8 non-experiment-specific personpowerNeeded to implement WP8 common policies
CERN person identified and hired (I.Augustin)
Should come from the funded contribution of each partner (CNRS, CERN, PPARC, NIKHEF and INFN) as a share of the 60 funded person-months
A message sent to the PMB for each partner to identify this personpower
DataGrid WP6 Workshop
10
11 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
Two kinds of Test-bed participation.Formal participation defined by WP6
Informal commitment to install Datagrid provided tools, and participate in tests.
Kick-start and Update kitsExperiments to provide WP8 with installation and upgrade kits
WP8 personpower will install them in the WP6 test-beds locations
Kits have to coexist on the same machines without interference
Locations participating but not in WP6 will provide their own personpower for the installation
This activity will be coordinated by the CERN WP8 person (I.Augustin)
DataGrid WP6 Workshop
11
11 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
Collection of requirementsPresent WP8 requirements judged vague by other WP's (rightly!)Other WP's should have asked WP8/9/10 questions – they didn’tMeeting of December 1st of the ATF was rather inconclusive
New strategy: WP8-10 to produce a three tiered documentShort term use casesLong term use casesGeneral requirements
WP8-10 also to produce pilot applicationsDecember 15 ATF should consolidate user requirementsDataGRID Workshop on January 15
US experts invited to discuss user requirements and first proposition of architecture
DataGrid WP6 Workshop
12
11 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
Set up a technical WP8 technical WGone application software expert from each experiment and from ESA (WP9) and biology (WP10)
the WP8 experiment-neutral personpower in the different partners (5 people)
Chaired by the CERN WP8 person acting as WP8 architect)
DataGrid WP6 Workshop
13
11 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
Main tasks Help WP8 architect to collect the requirements from the experiment and ESA and Biology for the ATF Liaise with the Middleware WP's (1-5) regarding the services required by the applications, and the definition of the appropriate interfaces.Discuss WP8 architectural questions
Do WP8-10 require a common 'upper middleware' layer over middleware services?Do they want to interface to generic middleware services directly?
Definition of the 'sample' applicationsCMS and LHCb have existing running distributed applicationsALICE is following and ATLAS will see whether it can add its
Define requirements for the Test-bed and Networking
DataGrid WP6 Workshop
14
11 December 2000
WP8 Wkshop Nov 16WP8 Wkshop Nov 16ConclusionsConclusions
Main tasks (cont)Provide technical liaison with the Test-bed and Networking workpackages (e.g. attend their meetings)Special priority points are, for instance
1. Provision of Standard, Supported, Documented Globus installation kit 2. Definition of Test-bed sites and contact people 3. Definition of Certification system to be employed by the project
Compact sub-groupWill meet, virtually or in person, in its own right, preparing our point of view for the global Test-bed and Architecture meetingsExperiment TB reps, unless the same person, would normally not attend, unless required for a forthcoming TB meetingThe frequency of meetings decided by WP8 architect and coordinator.
DataGrid WP6 Workshop
15
11 December 2000
ConclusionsConclusions
Activity of WP8 well startedMain concerns are
Architecture design seems to start with difficulty
We do not have an architect yet
Coordination with WP9-10 not yet very effective
Need better communication with WP1-5
Need to start interacting more effectively with WP6-7
Top Related