27-28 August 2002 Flavia Donno, INFN-Pisa 1
Report on GLUE activitiesReport on GLUE activities
Flavia Donno, INFN-PisaFlavia Donno, INFN-Pisa
EDG RetreatEDG Retreat
27-28 August 200227-28 August 2002
27-28 August 2002 Flavia Donno, INFN-Pisa 2
SummarySummary
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
Authorization services from VO LDAP to VOMSAuthorization services from VO LDAP to VOMS
Common software deployment proceduresCommon software deployment procedures
IST2002 and SC2002 joint EU-US demosIST2002 and SC2002 joint EU-US demos
Interoperability : between Grid domains for core grid services,coexistence of optimization/collective services
27-28 August 2002 Flavia Donno, INFN-Pisa 3
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
Computing Element
Cluster
Host
Computing ElementComputing Element
Sub-Cluster Representation Sub-Cluster Representation
Host
Host
Computing Resources Structure Description
Entry point into queueing system
Container grouping subcluster or nodes
Homogeneous collection of nodes
Physical computing nodes
27-28 August 2002 Flavia Donno, INFN-Pisa 4
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
CE Schema implementation statusCE Schema implementation status Logical structureLogical structure seems to have reached agreement and is stable seems to have reached agreement and is stable Recently added file system, versioningRecently added file system, versioning Interoperability: EDG needs one CE working on a cluster made Interoperability: EDG needs one CE working on a cluster made
of a single Subclusterof a single Subcluster Information about Hardware are in the subclusterInformation about Hardware are in the subcluster Information about Queue Status are in the CEInformation about Queue Status are in the CE Information providers have to provide Subcluster and CE infoInformation providers have to provide Subcluster and CE info LDAP implementation LDAP implementation dedefined, agreed and released!fined, agreed and released! Glue Schema will be included in Globus MDS 2.2 (September Glue Schema will be included in Globus MDS 2.2 (September
1515thth)) Globus Info Providers fill the host level infoGlobus Info Providers fill the host level info EDG Info providers filling Subcluster and CE are needed !! EDG Info providers filling Subcluster and CE are needed !!
When ? Is the 31When ? Is the 31stst of August confirmed ? of August confirmed ? It might be necessary to include the EDG info providers in VDTIt might be necessary to include the EDG info providers in VDT This implementation can be tested with MDS 2.1This implementation can be tested with MDS 2.1
27-28 August 2002 Flavia Donno, INFN-Pisa 5
SE Schema implementation statusSE Schema implementation status
Logical structureLogical structure almost agreed almost agreed Missing agreement on how to model connection Missing agreement on how to model connection
between storage services and file systembetween storage services and file system LDAP prototype implementation set upLDAP prototype implementation set up Questions to EDG for release 2:Questions to EDG for release 2:
Do we need to model multiple instances of Storage Service on an SE ? Multiple VO etc...
Could it be possible to remove the SE concept and introduce the 'storage services' insisting on the same hardware or 'Storage Device‘?
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
27-28 August 2002 Flavia Donno, INFN-Pisa 6
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
The US side set up the top-level GIIS The US side set up the top-level GIIS
mds-vo-name=DataTAGmds-vo-name=DataTAG mds-vo-mds-vo-name=iVDGLname=iVDGL
GIIS structure proposal for the GLUE testbedGIIS structure proposal for the GLUE testbed
mds-vo-name=gluemds-vo-name=glue
27-28 August 2002 Flavia Donno, INFN-Pisa 7
GIIS edt004.cnaf.infn.it Mds-vo-name=‘Datatag’ Resource Broker
Computing Element-1 PBS
WN1 edt001.cnaf.infn.it WN2 edt002.cnaf.infn.it
Gatekeeper grid006f.cnaf.infn.it
Computing Element -2 Fork/pbs
Gatekeeper edt004.cnaf.infn.it
GIIS giis.ivdgl.org mds-vo-name=ivdgl-glue
dc-user.isi.edurod.mcs.anl.gov
Job manager: Fork
hamachi.cs.uchicago.edu
iVDGL DataTAG
Condor
Gatekeeper: US-CMS
LSF
Gatekeeper: US-ATLAS
LSF
Gatekeeper: Padova-siteGIIS giis.ivdgl.org mds-vo-name=glue
Gatekeeper
27-28 August 2002 Flavia Donno, INFN-Pisa 8
Resource Discovery and GLUE schemaResource Discovery and GLUE schemaGLUE Schema testbed on-going activityGLUE Schema testbed on-going activity
Cnaf/Bologna site statusCnaf/Bologna site status- GIIS vo=Datatag edt004.cnaf.infn.itGIIS vo=Datatag edt004.cnaf.infn.it
- EDG 1.1.14, MDS 2.1 Glue Schema CE v1.0, Glue Schema SE v0.1SE v0.1 upgrade to edg1.2
- GRIS grid006f.cnaf.infn.itGRIS grid006f.cnaf.infn.it- EDG 1.1.14, MDS 2.1 Glue Schema CE v1.0- Upgrade to edg 1.2
- LDAP test (no EDG software, no Globus) gummo.cnaf.infn.itLDAP test (no EDG software, no Globus) gummo.cnaf.infn.it- Glue Schema CE v1.0, Glue Schema SE v0.1
- Configure RB EDG 1.2 in edt003.cnaf.infn.it Configure RB EDG 1.2 in edt003.cnaf.infn.it - Modify the RB to query the Glue Schema Modify the RB to query the Glue Schema - Install EDG/WP4 Information Provider Install EDG/WP4 Information Provider - Install SE EDG 1.2 and Glue Schema Install SE EDG 1.2 and Glue Schema SE v0.1SE v0.1
27-28 August 2002 Flavia Donno, INFN-Pisa 9
Resource Discovery and GLUE schemaResource Discovery and GLUE schemaGLUE Schema testbed on-going activityGLUE Schema testbed on-going activity
US side:US side: Top level GIIS with Glue schema CE v1.0 GRIS with Glue Schema CE v1.0 and MDS 2.1 and static info
provider to be upgraded to MDS 2.2 and Globus Info Provider Node with Glue Schema for SE ready for testing.
27-28 August 2002 Flavia Donno, INFN-Pisa 10
R-GMA implementation and interoperability R-GMA implementation and interoperability with MDSwith MDS
The Glue UML model is general and can have The Glue UML model is general and can have different implementations: LDAP, Relational different implementations: LDAP, Relational DB, XML etc.DB, XML etc.
Schema implementation in R-GMA :Schema implementation in R-GMA : implement the Glue UML model in R-GMA and test in the
Glue testbed, with EDG WP3 adapt the info providers to fill the the relational table, this
should be zero or very low impact Does MDS need to interoperate with R-GMA ?
Resource Discovery and GLUE schemaResource Discovery and GLUE schema
27-28 August 2002 Flavia Donno, INFN-Pisa 11
Resource Discovery and GLUE schemaResource Discovery and GLUE schemaPeople and HW resourcesPeople and HW resources
PEOPLE:PEOPLE:
• DataTAG/WP4.2 people : INFN (1.5 fte)
Coordinator: Cristina Vistoli (40%) - [email protected] CNAF: Sergio Andreozzi (100%) - [email protected]
Vicenzo Ciaschini (20%) - [email protected]
Hardware Resources:Hardware Resources: Local HW at CNAF/Padova Local HW at ISI
iVDGL Contacts:iVDGL Contacts: B. Tierney, J. Schopf, E. Deelman, C.
Kesselman, A. Shoshani
EDG Contacts:EDG Contacts: M. Sgaravatto, P. Kunstz, J. Gordon, O. Barring
27-28 August 2002 Flavia Donno, INFN-Pisa 12
Authorization services from VO Authorization services from VO LDAP to VOMSLDAP to VOMS
• EDG Tools proposed for use to iVDGL collaborators• VO LDAP server• VO GroupMAN• edg-mkgridmap
• VOMS under study by iVDGL collaborators.
•FNAL – Working on KCA/X509•Becoming aware of LCAS/LCMAPS
27-28 August 2002 Flavia Donno, INFN-Pisa 13
Authorization services Authorization services People and HW resourcesPeople and HW resources
PEOPLE:PEOPLE:
• DataTAG/WP4.2 people : INFN (2.5 fte)
Coordinator: Roberto Cecchini (30%) - [email protected] CNAF: Luca Dell’Agnello (30%) - [email protected]
Fabio Spataro (100%) - [email protected] Ciaschini (80%) - [email protected]
Hardware Resources:Hardware Resources: Local HW at CNAF
iVDGL Contacts:iVDGL Contacts: Doug Olson
EDG Contacts:EDG Contacts: Roberto Cecchini
27-28 August 2002 Flavia Donno, INFN-Pisa 14
Common software deployment Common software deployment proceduresprocedures
Goals:Goals: Contribute to a distribution that:Contribute to a distribution that:
Is EDG and VDT compliant (can be included in VDT/EDG)Includes main EDG and VDT componentsAllows for interoperation of testbedsEnforces common flexible configuration rules
Plan:Plan:Distribute EDG CADistribute EDG CA
Distribute EDG Replica Catalogue with GSIDistribute EDG Replica Catalogue with GSI
Distribute UI, RB, JSS, L&B as “standalone” componentsDistribute UI, RB, JSS, L&B as “standalone” components
(End of September, edg-release 1.2 + new schema) (End of September, edg-release 1.2 + new schema)
Distribute information providers if requestedDistribute information providers if requested
Propose a configuration strategy to both VDT and EDGPropose a configuration strategy to both VDT and EDG
Work on a common EDG-VDT Work on a common EDG-VDT versioning strategyversioning strategy
Helping experiments with software distributionHelping experiments with software distribution
(CMS/ATLAS PACMAN distribution)(CMS/ATLAS PACMAN distribution)
EDG/WP4
Done and installed on GLUE TB
Done
In progress
In progress
In progress
27-28 August 2002 Flavia Donno, INFN-Pisa 15
UI
EDG StructureEDG Structure
UI
UI
RB
JSS
RB
JSS
Many light UIs Few heavy RBs
SE
CE
SE
CE
SESE
CE
CE
Many Ces and SEs
LB IS RC
One LB, IS and RC
27-28 August 2002 Flavia Donno, INFN-Pisa 16
UI
VDT StructureVDT Structure
UI
RB
JSS
RB
JSS
Many VDT Clients
SE
CE
Many VDT Servers
LB IS RC
One LB, IS and RC
SE
CE
SE
CE
SE
CE
27-28 August 2002 Flavia Donno, INFN-Pisa 17
Common software deployment Common software deployment proceduresprocedures
People and HW resourcesPeople and HW resourcesPEOPLE:PEOPLE:
DataTAG/WP4.3 people : INFN (1.5 fte)
Coordinator: Flavia Donno (20%) - [email protected] Milano: David Rebatto (30%) - [email protected]
Luca Vaccarossa (50%) - [email protected] Padova: Sergio Fantinel (50%) - [email protected]
Hardware Resources:Hardware Resources: Local HW in Milan Local HW in Padova HW in US
iVDGL Contacts:iVDGL Contacts: Alain Roy, Saul Youseff, Jerry Gieraltowski
EDG Contacts:EDG Contacts: Enrico Ferro
27-28 August 2002 Flavia Donno, INFN-Pisa 18
Data Movement and ManagementData Movement and Management
It is part of GLUE phase II.It is part of GLUE phase II. Setting up of working groups in progress with Setting up of working groups in progress with
involvement with EDG/WP2.involvement with EDG/WP2. The existing collaboration between the Globus The existing collaboration between the Globus
Team and EDG/WP2 is a good starting point.Team and EDG/WP2 is a good starting point. Experiment requirements drive the effort.Experiment requirements drive the effort.
DataTag / iVDGL People:DataTag / iVDGL People: Nikolai Smirnov, David Rebatto, Mike Wilde
EDG Contacts:EDG Contacts: P. Kunstz
27-28 August 2002 Flavia Donno, INFN-Pisa 19
Joint EU-US grid demosJoint EU-US grid demosIST2002IST2002 4-6 November, Copenaghen4-6 November, Copenaghen
SC2002SC2002 16-22 November, Baltimore16-22 November, Baltimore
Goals:Goals: Basic collaboration between European and US grid projects Interoperability between grid domains for applications submitted by users from
different virtual organizations Controlled use of shared resources subject to agreed policy Integrated use of heterogeneous resources from iVDGL and EDG testbed
domains Infrastructure:Infrastructure:WEB site: http://www.ivdgl.org/demoWEB site: http://www.ivdgl.org/demo Hypernews: Hypernews: http://atlassw1.phy.bnl.gov/HyperNews/get/intergrid.htmlhttp://atlassw1.phy.bnl.gov/HyperNews/get/intergrid.html Mailing list:Mailing list: [email protected] [email protected] Archive:Archive: http://web.datagrid.cnr.it/hypermail/archivio/igdemohttp://web.datagrid.cnr.it/hypermail/archivio/igdemo GLUE testbed with common schemaGLUE testbed with common schema VO (DataTAG and iVDGL) LDAP Servers in EU and USVO (DataTAG and iVDGL) LDAP Servers in EU and US PACMAN cache with software distribution (DataTAG or iVDGL)PACMAN cache with software distribution (DataTAG or iVDGL) Planning document outlinePlanning document outline
27-28 August 2002 Flavia Donno, INFN-Pisa 20
Joint EU-US grid demosJoint EU-US grid demos
GLUE testbed with Common GLUE schema and authorization/authentication tools.GLUE testbed with Common GLUE schema and authorization/authentication tools. EDG 1.2 + extensions, VDT 1.1.3 + extensions “Old” authentication/authorization tools : VO LDAP servers, mkgridmap, etc. We rely on the availability of the new GLUE schema, RB and Information Providers
Concentrate on visualization: CMS/GENIUS, ATLAS/GRAPPA, EDG/MAPCENTER Concentrate on visualization: CMS/GENIUS, ATLAS/GRAPPA, EDG/MAPCENTER (EDG/WP7), Farm Monitoring tools (EDG/WP4), iVDGL/GANGLIA, DataTAG/NAGIOS(EDG/WP7), Farm Monitoring tools (EDG/WP4), iVDGL/GANGLIA, DataTAG/NAGIOS
Use WEB portals for job submission (CMS/Genius, ATLAS/Grappa) Provide a World Map of the sites involved (EDG-WP7/MapCenter, DataTAG/Nagios) Monitor job status and statistics (EDG-WP4, DataTAG/Nagios). In Nagios implemented “top users”, “top
applications”, “total resources”, “averages over time”, “per VO”,… Monitor farms (EDG/WP4, DataTAG/Nagios) Developing plugins/sensors for WP1/L&B info (only on EDG part) Services Monitoring (EDG/WP4, DataTAG/Nagios [using MDS info])
CMS and ATLAS demoCMS and ATLAS demo ATLAS simulation jobs, GRAPPA modified to use both RB or explicit resources Pythia & CMSIM simulation jobs submitted to intercontinental resources with IMPALA/BOSS interfaced to
VDT/MOP, EDG/JDL, Genius portal. Definition of the application demos in progress.
27-28 August 2002 Flavia Donno, INFN-Pisa 22
Joint EU-US grid demosJoint EU-US grid demos
Joint Task force in Chicago 23-30 September, 2002for first DataTAG/iVDGL and ATLAS rehearsal.
Work on infrastructure and application demos as much aspossible before the middle of September.
First simple sumission tests in progress.
Hardware definition in progress.
27-28 August 2002 Flavia Donno, INFN-Pisa 23
Suggestions to EDG for release 2Suggestions to EDG for release 2• Work on installation/configuration issues
• Relocatability• Configuration environemnt• Service configuration and startup
• Make sure “vanilla functionality” of core services is kept.• Security system with VO support• Information System• Submission services such as the Globus Gatekeeper, JobManager, Condor-G,...
• Schedule with time introduction of GLUE components in EDG. Active collaboration.
• Make service interfaces clear so to allow for easy software sharing and integration whenever possible.
• Discuss architecture design for Job scheduling, Data Management and Storage Services.
• ...
Top Related