OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure

7/30/2019 OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure

1/15

2006 Open Grid Forum

OGF-Europe Tutorial: How new

communities can get access to a GridinfrastructureNorduGrid - NDGF

Oxana Smirnova

NDGF / Lund UniversityOGF25

March 2 2009, Catania, Italy


2/15


OGF IPR Policies Apply

I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy. Intellectual Property Notices Note Well: All statements related to the activities of the OGF and

addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGFand its participants certain licenses and rights in such statements. Such statements include verbalstatements in OGF meetings, as well as written and electronic communications made at any time orplace, which are addressed to:

the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process

Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intendedto be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: Where the OGF knows of rights, or claimed rights, the OGF

secretariat shall attempt to obtain from the claimant of such rights, a written assurance that uponapproval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right toimplement, use and distribute the technology or works when implementing, using or distributingtechnology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology withrespect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. Theresults of this procedure shall not affect advancement of document, except that the GFSG may deferapproval where a delay may facilitate the obtaining of such assurances. The results will, however, be

recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of theresults be included in any GFD published containing the specification.

OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that supportthe Internet Standards Process.

2009-03-02 2www.ndgf.org www.nordugrid.org


3/15


Outlook

Basic production middleware overview:

Advanced Resource Connector (ARC)

NDGF and its users:the WLCG Tier1 and more

On the road towards standards-based

interoperability

www.ndgf.org www.nordugrid.org2009-03-02 3


4/15


NorduGrid, NDGF and others

NorduGrid an academic researchcollaboration that started in 2001 andspawned a number of projects

Develops and maintains the AdvancedResource Connector (ARC) Gridmiddleware

NDGF Nordic Grid Infrastructure

funded through the Nordic Council ofMinisters Provides Tier1 for LHC computing,

coordinates Tier2s

Makes use of ARC as an underlyingtechnology

EU KnowARC R&D project

providing new standard interfaces toARC

NGIn project funded by Nordunet3;focuses on ARC education andapplications



5/15


How to build a Grid:

ARC in a nutshell



6/15


Production ARC overview

Current ARC version in production: v0.6.5

Available freely at http://www.nordugrid.org

Open Source (GPL v2, next versions - Apache 2.0)

Binary packages for20+ Linux flavors/versions

(RH, Fedora, SuSE, Debian, Ubuntu)

Reliable implementation of fundamental Grid functionalities:

De-factostandard Grid security: GSI, VOMS, GACL (needs 3rd

party libraries) Job submission: by matchmaking/brokering or direct

Job monitoring, logging and life cycle management

Information services: resource aggregation, representation, discovery and

monitoring

Basic data management:

Automated seamless input/output data movement Interface to data indexing services (e.g. LFC), client-side data movement

Storage Elements (GridFTP, SRM own or 3rd party) NDGF uses dCache

Builds upon standard Open Source solutions and protocols:

Globus Toolkit pre-WS API and libraries (not services! )

OpenLDAP, OpenSSL, SASL, SOAP, GridFTP, GSI

http://www.nordugrid.org/http://www.nordugrid.org/


7/15


How new communities join

the Grid

No centralizedcontrol whatsoever No central

operations No databases of

resources or VOs

No availability orreliability tests

Typically, userscome via nationalGrid initiatives: NDGF, SweGrid,

SwiNG (Swiss),

UkrainianAcademic Grid etc


Educate users

Organize tutorials, workshops involving middleware experts

Deploy middleware over your computing facilities

Get additional hardware, ifneeded

Get additional system experts, ifneeded

If necessary, publish sites in theNorduGrid information system

Identify application projects

Can be on a competitive basis, via calls for proposals

Identify your Certification Authority

Must be IGTF-endorsed For tests, an Instant CA will do

Identify your resources

Computing facilities Storage facilities Manpower


8/15


Nordic DataGrid Facility

Provides a Grid infrastructure, incl.a unique distributed Tier1 centre

Involves 7 largest Nordic academicHPC centres

plus a handful of Universitycentres (Tier2 service)

Inter-Nordic shared 10Gbit networkfrom NORDUnet

Budget: staff only, 2 MEUR/year, byNordic research councils



9/15


NDGF Tier1 services overview

Level 1: central services,99% up-time

dCache: admin,doors: gridftp xrootd,srm

Data indexing: LFC

WLCG monitoring:site BDII

Level 2: distributed,98% up-time

Storage pools

ARC gatekeepers

VOBoxes (ALICE)

FTS: with GridFTPv2

WLCG 3D database

VOMS

Level 3: distributed,95% up-time

Worker nodes

ARC GIISes

Monitors



10/15


Example of a country

contribution: SweGrid

Investment Date Cost,KEUR

Six clusters (6x100 cores)including 12 TB FC disk Dec 2003 1 120Disk storage part 1, 60 TB SATA May 2004 320Disk storage part 2, 86.4 TB SATA May 2005 230

Centre Tapevolume, TB

Cost,

KEURHPC2N 120 100PDC 120 100NSC 120 100

SweGrid in 2003-2007Location ProfileHPC2N (Ume) IT

UPPMAX (Uppsala) IT, HEP

PDC (Stockholm) IT

C3SE (Gothenburg) IT

NSC (Linkping) IT

Lunarc (Lund) IT, HEP

Co-funded by the Swedish

Research Council and the Knutand Alice Wallenberg foundation One technician per centerMiddleware: ARC, dCache, gLite 1/3 allocated to LHC Computing



11/15


User communities

LHC ChemistryOther Physics Amanda/IcecubeGeo Science Fluid MechanicsBioinformatics MedicineComputer Science BiotechnologyMathematics StatisticsPharmacology Material Chemistry

Electronics



12/15


Enabling applications

Application-specific software is formalized as Run-TimeEnvironment (RTE)

Installed and configured at each site, typically by anauthorized user/manager

Advertised via information system for match-making

Jobs only need to specify the RTE name and version

Special authorization plugins are available in case of a

commercial application software Users with similar applications are grouped into Virtual

Organisations (VOs) A VO typically corresponds to a project

A VO contributes the hardware

Some VOs at NDGF: CERN VOs: ALICE, ATLAS, CMS

Gene databases for bio-informatics sciences Screening of CO2-Sequestration suitable reservoirs

Computational Chemistry



13/15


On interoperability

Strategy: interoperability via open standards BES, JSDL, GLUE2, SRM, GridFTP, X509, SAML etc Aggreed extensions are critical (profiles) Tip: attend the Production Grid Infrastructure (PGI-WG)

workshop on Wednesday Shorter term: transitional gateway-like solutions are

available (ARC-gLite) Currently in development:

ARC client library addresses the ARC other middlewaredirection

CLI will offer transparent access capability to 3rd partyservices

Primary target platforms: gLite, Unicore New ARC client can already now submit jobs to CREAM,

Unicore compute elements



14/15


Future perspectives

ARC evolves from a pre-WS solution to a WebService based one PGI-WG work ensures that it will be standards-

conformant and interoperable with other standards-conformant middlewares

ARC consortium (NorduGrid, NDGF, KnowARC etal), together with gLite and Unicore, contribute tocreation of the Universal Middleware Distribution(UMD)for the European Grid Initiative (EGI) Sites and VOs that use ARC will get an access to the

European e-Science infrastructure, just like those thatuse gLite or Unicore



15/15


Summary: how to proceed

Get documentation at http://www.nordugrid.org Useful tips also at http://www.ndgf.org, and new ideas at http://www.knowarc.eu

Get Grid certificates for you, your hardware and your users

Get ARC: http://download.nordugrid.org official source and binary packages, some 3rd

party middleware

https://svn.nordugrid.org code, arc0(production) and arc1 (WS) directories ofthe nordugridrepository

Submit support requests: [email protected] NDGF infrastructure and operations

[email protected] ARC middleware installation and usage

http://bugzilla.nordugrid.org feature requests, bug reports

Join the community:

Sign up for the [email protected] mailing list Attend workshops and conferences

Read ARC reference paper: "Advanced Resource Connector middleware for lightweight computational

Grids". M.Ellert et al., Future Generation Computer Systems 23 (2007) 219-240.

http://www.nordugrid.org/http://www.ndgf.org/http://www.knowarc.eu/http://download.nordugrid.org/https://svn.nordugrid.org/mailto:[email protected]:[email protected]://bugzilla.nordugrid.org/mailto:[email protected]:[email protected]:[email protected]:[email protected]://bugzilla.nordugrid.org/mailto:[email protected]:[email protected]:[email protected]:[email protected]://svn.nordugrid.org/http://download.nordugrid.org/http://www.knowarc.eu/http://www.ndgf.org/http://www.nordugrid.org/

OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure

Documents

Transcript of OGF-Europe Tutorial: How new communities can get access to a Grid infrastructure