05/23/2002Flavia Donno, INFN-Pisa1 DataTAG experiment grid integration plan Flavia Donno, INFN-Pisa...

05/23/2002 Flavia Donno, INFN-Pisa 1

DataTAG experiment grid integration planDataTAG experiment grid integration plan

Flavia Donno, INFN-PisaFlavia Donno, INFN-Pisa

DataTAG/iVDGL joint meeting, 23-24 May 2002DataTAG/iVDGL joint meeting, 23-24 May 2002

http://www.pparc.ac.uk/

http://www.infn.it/

http://www.cern.ch/


OutlineOutline

GoalsGoals

People and HW resourcesPeople and HW resources

Working plan with experiments (ATLAS, CDF, Working plan with experiments (ATLAS, CDF, CMS, ALICE,CMS, ALICE, D0 D0))


GoalsGoals

Goals:Goals:

Interoperability between EU and US Grids services fromInteroperability between EU and US Grids services from

DataGrid, GriPhyN, PPDG and in collaboration with iVDGL,DataGrid, GriPhyN, PPDG and in collaboration with iVDGL,

for the HEP applications.for the HEP applications.

Steps:Steps:

Produce an assessment of interoperability solutionsProduce an assessment of interoperability solutions

Provide test environment to LHC Applications to extendProvide test environment to LHC Applications to extend

existing use-cases to test interoperability of the gridexisting use-cases to test interoperability of the grid

componentscomponents

Provide input to a common Grid LHC architectureProvide input to a common Grid LHC architecture

Plan EU-US Integrated grid deploymentPlan EU-US Integrated grid deployment


General PlanGeneral Plan

Plan:Plan: Evaluation of existing packaging and deployment tools. Collection of a

set of requirements (VDT/PACMAN, EDG/LCFG, etc…) – first draft. DataTAG software distribution.

Evaluation/Deployment of existing services for job scheduling, submission and monitoring (DAGMAN, Condor-G, EDG/RB). Collection of requirements. First draft.

Evaluation/Deployment of existing DataManagement Tools (Globus/GridFTP, EDG/Replica Manager, MAGDA, SAM, etc). Collection of requirements. First draft.

Classifications of available services and identification of common requirements.

The evaluations will be done in the context of the VO infrastructure

provided by EDT-WP4 and in particular with the VO services for Authentication, and Authorization and the agreed common schema for Resource Specification/Discovery.


People and HW resourcesPeople and HW resourcesPEOPLE:PEOPLE:

DataTAG/WP4.3-4 people : INFN (7fte), PPARC (1fte)Coordinator: Flavia Donno - [email protected] Milano: David Rebatto - [email protected]

Luca Vaccarossa - [email protected] Padova: F. Fanzago - [email protected]

S. Fantinel - [email protected]. Smirnov - [email protected]

INFN CNAF :Cristina Vistoli - [email protected] Ciaschini - [email protected] Ghiselli - [email protected]

PPARC : Simone Ludwig - [email protected]

Human resources come also from partecipating experiments. Contact points established with experiments.

Contact points established with iVDGL.

HW Resources:HW Resources: Alice in progress (Italy, NiKhef, OCS, BNL) Atlas test layout in Milan, Argonne CDF test layout in Trieste, Bologna, FNAL CMS test layout in Italy, Ecole politecnique, Russia, … DataTAG test layout at CNAF for common services.

mailto:[email protected]

















Working plan with experimentsWorking plan with experiments

Plans established with CMS and CDF.Plans established with CMS and CDF.

Short term plan for ATLAS. A more detailed plan Short term plan for ATLAS. A more detailed plan needs to be agreed for future activities. Check with the needs to be agreed for future activities. Check with the US counterpart.US counterpart.

First meeting with Alice May 2First meeting with Alice May 2ndnd 2002. Preliminary 2002. Preliminary plan for Job submission and Data Catalogues.plan for Job submission and Data Catalogues.

D0 and Virgo/Ligo still to be defined.D0 and Virgo/Ligo still to be defined.For now mainly EDG tools are used (EDG v1.1.1, v1.1.2 and v1.1.4):For now mainly EDG tools are used (EDG v1.1.1, v1.1.2 and v1.1.4):machine configuration, WP6 distribution,machine configuration, WP6 distribution,VO management, grid-mapfile, Information Service,VO management, grid-mapfile, Information Service,EDG middleware software EDG middleware software (Globus Beta2.1,Condor-G 6.3.1, classads 0.9, GDMP2.1 and 3.0)(Globus Beta2.1,Condor-G 6.3.1, classads 0.9, GDMP2.1 and 3.0)


Working plan with experiments - ALICEWorking plan with experiments - ALICE Alice (P. Cerello):Alice (P. Cerello):

Goals: Goals: A.A. Complete the PPR (Physics Performance Report) by the end of this year exploiting the Complete the PPR (Physics Performance Report) by the end of this year exploiting the use of GRID tools on test layouts spanning in both EU and US. Officially, for the PPR Alice use of GRID tools on test layouts spanning in both EU and US. Officially, for the PPR Alice will use AliEn. will use AliEn. Alice would like to try to use AliEn managed resources and EDG resources Alice would like to try to use AliEn managed resources and EDG resources with a minimal version of AliEn so that steps not possible with EDG can be executed by with a minimal version of AliEn so that steps not possible with EDG can be executed by AliEn in a second time.AliEn in a second time.

B.B. I Interface AliEn with EDG/RB if possible so that the RB will act like an “AliEn client” or nterface AliEn with EDG/RB if possible so that the RB will act like an “AliEn client” or with iVDGL/Condor-G.with iVDGL/Condor-G.

C.C. I Interface or merge AliEn and EDG/RC data catalogues.nterface or merge AliEn and EDG/RC data catalogues.

D.D. E Exploit EDG/Spitfire and build with it a prototype of a “Grid” distributed data metadata xploit EDG/Spitfire and build with it a prototype of a “Grid” distributed data metadata catalogue to explore the problems related to this argumentcatalogue to explore the problems related to this argument..


Working plan with experiments - ALICEWorking plan with experiments - ALICEPlan:Plan:

AA..1 1 Define a DataTAG/iVDGL test layout for Alice. Define the hardware to be used by May Define a DataTAG/iVDGL test layout for Alice. Define the hardware to be used by May 31st 2002. 31st 2002.

B.1B.1 DataTAG people help setup the Alice test layout. First attempts for interfacing Alien DataTAG people help setup the Alice test layout. First attempts for interfacing Alien and the EDG/RB. Interface to iVDGL/Condor-G will come as a second step.and the EDG/RB. Interface to iVDGL/Condor-G will come as a second step.

B.2B.2 Alien installation on Alice test layout. Coordination with Pedrag Buncic at CERN. Try to Alien installation on Alice test layout. Coordination with Pedrag Buncic at CERN. Try to reach some results by September 2002 ?reach some results by September 2002 ?

C.1C.1 Interface of the AliEn and EDG catalogues A more detailed plan can be written down Interface of the AliEn and EDG catalogues A more detailed plan can be written down once we have more clear ideas on how to proceed and after talking to Pedrag and Fons.once we have more clear ideas on how to proceed and after talking to Pedrag and Fons.

D.1D.1 Domenico Vicinanza and Daniele Mura will try to install and configure Spitfire on Domenico Vicinanza and Daniele Mura will try to install and configure Spitfire on machines at Cagliari with the help of the spitfire support team in EDG and of DataTAG machines at Cagliari with the help of the spitfire support team in EDG and of DataTAG people. people.


Working plan with experiments - ATLASWorking plan with experiments - ATLAS Atlas (L. Perini):Atlas (L. Perini):

Goals: Goals: A.A. As a first step, Atlas is trying to use the EDG solution on its test layout for the Data As a first step, Atlas is trying to use the EDG solution on its test layout for the Data Challenge 1 (DC1) which should start by the 15Challenge 1 (DC1) which should start by the 15 thth of June. of June.

B.B. Atlas will need to distribute the EDG software to the American collaborators with VDT Atlas will need to distribute the EDG software to the American collaborators with VDT distribution tools.distribution tools.

C.C. The only development Atlas is carrying on is the integration of MAGDA (the Atlas data The only development Atlas is carrying on is the integration of MAGDA (the Atlas data manager) and EDG/GDMP.manager) and EDG/GDMP.

D.D. Further developments are ongoing in the areas of: Further developments are ongoing in the areas of:

Metadata catalogue for the data (they have one developed by Grenoble).Metadata catalogue for the data (they have one developed by Grenoble).

Griphyn and Virtual Data Language (GRAPPA). Griphyn and Virtual Data Language (GRAPPA).

““Gridified” Online notebook by Indiana University.Gridified” Online notebook by Indiana University.

Virtual Data (MOP per le produzioni di Atlas)Virtual Data (MOP per le produzioni di Atlas)


Working plan with experiments - ATLASWorking plan with experiments - ATLASPlan:Plan:

AA..1 1 DataTAG will help setup a test layout for MAGDA/GDMP integration.DataTAG will help setup a test layout for MAGDA/GDMP integration.

B.1B.1 DataTAG will try to produce a distribution of EDG/release 1.2 via Pacman by the 15 DataTAG will try to produce a distribution of EDG/release 1.2 via Pacman by the 15 thth of of June.June.

C.1C.1 Work is in progress with EDG and DataTAG people to realize the integration of GDMP Work is in progress with EDG and DataTAG people to realize the integration of GDMP and MAGDA. A first integration will just try to syncronize the MAGDA and GDMP internal and MAGDA. A first integration will just try to syncronize the MAGDA and GDMP internal catalogues for file mirroring. This should be achieved by the end of May.catalogues for file mirroring. This should be achieved by the end of May.


Working plan with experiments - CDFWorking plan with experiments - CDF CDF (S. Belforte, A. Sidoti, R. St. Denis):CDF (S. Belforte, A. Sidoti, R. St. Denis):

Goals: Goals:

A.A. CDF starts using European DataGRID toolkit for interoperability across European CDF starts using European DataGRID toolkit for interoperability across European

institution/laboratories, in conjunction with software configured as SAM stations to deal with institution/laboratories, in conjunction with software configured as SAM stations to deal with data handling and distribution.data handling and distribution.

B.B. Make it easier to connect from EU countries to FNAL Make it easier to connect from EU countries to FNAL

C.C. Share resources to provide common framework Share resources to provide common framework

D.D. Open the way to exploit coming high speed European networks to build a common Open the way to exploit coming high speed European networks to build a common (disk resident) data set with faster access than FNAL main repository(disk resident) data set with faster access than FNAL main repository


Working plan with experiments - CDFWorking plan with experiments - CDFPlan:Plan:

A/B/CA/B/C Create a DataTag testbed using EDG tools to allow CDF people to test job Create a DataTag testbed using EDG tools to allow CDF people to test job submission and scheduling tools. CDF VO in place, DOE CA accepted. Install EDG UI at submission and scheduling tools. CDF VO in place, DOE CA accepted. Install EDG UI at FNAL. Testbed open for CDF tests. (FNAL. Testbed open for CDF tests. (30 Apr 200230 Apr 2002) – DataTAG + CDF people) – DataTAG + CDF people

Study the SAM environment and exploit the possibility of integrating SAM in Study the SAM environment and exploit the possibility of integrating SAM in CDF/DataTAG EDG testbed (CDF/DataTAG EDG testbed (3131stst Jul 2002 Jul 2002) – DataTAG + CDF SAM support.) – DataTAG + CDF SAM support.

Test Glue MDS schema and check if it is enough to describe SAM resources. – Test Glue MDS schema and check if it is enough to describe SAM resources. – DataTAG + CDF SAM support.DataTAG + CDF SAM support.

Test current EDG/JDL and study possible extensions for SAM. – DataTAG + CDF Test current EDG/JDL and study possible extensions for SAM. – DataTAG + CDF SAM support.SAM support.

Work on-going with the SAM development/support team at FNAL to converge on a Work on-going with the SAM development/support team at FNAL to converge on a plan.plan.


Working plan with experiments - CMSWorking plan with experiments - CMS CMS (C. Grandi, P. Capiluppi):CMS (C. Grandi, P. Capiluppi):

Goals: Goals:

A.A. Have a complete production chain driven by IMPALA/BOSS on EDG User Have a complete production chain driven by IMPALA/BOSS on EDG User

Interface that uses the EDG Resource Broker to submit jobs to the DataGrid testbed by Interface that uses the EDG Resource Broker to submit jobs to the DataGrid testbed by June CMS week. First evaluation of EDG job scheduling and submission services . (June CMS week. First evaluation of EDG job scheduling and submission services . (10 Jun 10 Jun 20022002).).

B.B. Open DataTag testbed to US CMS partners and study interoperability issues (CA's, Open DataTag testbed to US CMS partners and study interoperability issues (CA's, II, RB, etc...). This implies the study of the packaging tools used in EDG and iVGDL. First II, RB, etc...). This implies the study of the packaging tools used in EDG and iVGDL. First set or requirements. (set or requirements. (3131 Jul 2002Jul 2002))

C.C. Install US grid software on DataTag resources. Study the possibility of submitting Install US grid software on DataTag resources. Study the possibility of submitting

jobs from EDG testbeds to iVDGL test layouts – Compatibility of common resources jobs from EDG testbeds to iVDGL test layouts – Compatibility of common resources schemas. First evaluation of iVDGL job scheduling, submission and monitoring services schemas. First evaluation of iVDGL job scheduling, submission and monitoring services ((31 Jul 200231 Jul 2002))

D.D. Request for some million events similar (all chain in a single job). Tony Wildish Request for some million events similar (all chain in a single job). Tony Wildish proposed to try to produce them using the grid test layout.proposed to try to produce them using the grid test layout.


Working plan with experiments - CMSWorking plan with experiments - CMSPlan:Plan:

AA Use DataTag testbed as development environment to define CE, SE, WNUse DataTag testbed as development environment to define CE, SE, WN.. Installation kits and UI configuration with IMPALA/BOSS to submit Pythia and cmsim jobs Installation kits and UI configuration with IMPALA/BOSS to submit Pythia and cmsim jobs ((30 Apr 200230 Apr 2002) - CMS people.) - CMS people.

Test Replica Catalogue registrationTest Replica Catalogue registration and file replication (and file replication (7 may 027 may 02) - CMS + ) - CMS + DataTAG peopleDataTAG people

Use DataTag testbed to develop what needed to submit ORCA jobs.Use DataTag testbed to develop what needed to submit ORCA jobs. ( (10 Jun 0210 Jun 02) - ) - CMS peopleCMS people

BB Develop DataGrid software packaging and distribution independent on LCFG. First Develop DataGrid software packaging and distribution independent on LCFG. First report (report (30 Apr 0230 Apr 02) - DataTag people) - DataTag people

Setup CMS VO and open it to iVGDL (Setup CMS VO and open it to iVGDL (30 Apr 0230 Apr 02) - DataTag people) - DataTag people

Define commonly recognized CA's for US (Define commonly recognized CA's for US (31 May 0231 May 02) - DataTag people) - DataTag people

EDG release 1.1.4-1.2 deployment on some US sites (EDG release 1.1.4-1.2 deployment on some US sites (31 May 0231 May 02) –) – DataTag people DataTag people

Test worldwide production with EDG software (Test worldwide production with EDG software (31 Jul 0231 Jul 02) - CMS + DataTAG people) - CMS + DataTAG people


Working plan with experiments - CMSWorking plan with experiments - CMSPlan:Plan:

CC Install US grid software on DataTag resources (it depends on B.1 -> B.3)(Install US grid software on DataTag resources (it depends on B.1 -> B.3)(31 May 31 May

0202) - DataTag people) - DataTag people

DD Start MB production with EDG software in EU only Start MB production with EDG software in EU only ((17 Jun 0217 Jun 02) - CMS people (it ) - CMS people (it depends on A)depends on A)

Start MB production with EDG software worldwide (Start MB production with EDG software worldwide (31 Jul 0231 Jul 02) - CMS people (it ) - CMS people (it depends on B)depends on B)

Start MB production with US software Start MB production with US software ((31 Jul 0231 Jul 02) - EU CMS people driven by ) - EU CMS people driven by CMS-US peopleCMS-US people (it depends on C)(it depends on C)


Organization and ConclusionsOrganization and Conclusions

Job organization internally in DataTAG.Job organization internally in DataTAG.

Establish commonalities among test layouts.Establish commonalities among test layouts.

Establish a working plan with iVDGL.Establish a working plan with iVDGL.

Interoperability tests with US.Interoperability tests with US.

SC2002 demo ?SC2002 demo ?

05/23/2002Flavia Donno, INFN-Pisa1 DataTAG experiment grid integration plan Flavia Donno, INFN-Pisa...

Documents

Transcript of 05/23/2002Flavia Donno, INFN-Pisa1 DataTAG experiment grid integration plan Flavia Donno, INFN-Pisa...