Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18...
-
Upload
giles-little -
Category
Documents
-
view
215 -
download
0
Transcript of Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18...
![Page 1: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/1.jpg)
Porting applications to EU-IndiaGrid:
EGEE
Marco Verlato (@pd.infn.it)
EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India
![Page 2: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/2.jpg)
2
Outline
• Concepts• Job categories• Levels of Gridification• Software remote installation• Applications invocation
![Page 3: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/3.jpg)
3
Grid Applications Development
Where computer science meets the application communities!
VO-specific developments built on higher-level tools and core services
Makes Grid services useable by non-specialists
Grids provide the compute and data storage resources
Basic Grid services:AA, job submission, info, …
Higher-level grid services (brokering,…)
Application toolkits, …..
Application
Production grids provide these core services.
![Page 4: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/4.jpg)
4
Job concept• gLite middleware follows the job submission
concept for application execution and resource sharing
• A job is a self contained entity that packages and conveys all the required information and artifacts for the successful remote execution of an application−Executable files−Input/Output data−Parameters−Environment−Infrastructure Requirements−Workflows
![Page 5: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/5.jpg)
5
Type of applications
• The work done to port the application depends on its characteristics, complexity and the level of interactions with external grid services
• Size of input/output data• Metadata request• Access to external software
(database, libraries...)
![Page 6: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/6.jpg)
6
Applications categories
• Simulation• Bulk Processing• Responsive Apps.• Workflow• Parallel Jobs
![Page 7: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/7.jpg)
7
Simulation
• Examples– LHC Monte Carlo simulation– Fusion
• Characteristics– Jobs are CPU-intensive– Large number of independent jobs– Run by few (expert) users– Small input; large output
• Needs– Batch-system services– Minimal data management for storage of results
ATLAS
ITER
![Page 8: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/8.jpg)
8
Bulk Processing
• Examples– HEP processing of raw data, analysis– Earth observation data processing
• Characteristics– Widely-distributed input data– Significant amount of input and output data
• Needs– Job management tools (workload
management)– Meta-data services– More sophisticated data management
![Page 9: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/9.jpg)
9
Responsive Apps. (I)
• Examples–Prototyping new applications–Monitoring grid operations
• Characteristics–Small amounts of input and output data–Not CPU-intensive–Short response time (few minutes)
• Needs–Configuration which allows “immediate”
execution (QoS)–Services must treat jobs with minimum
latency
![Page 10: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/10.jpg)
10
Responsive Apps. (II)
• Grid as a backend infrastructure:– gPTM3D: interactive analysis of medical images– GPS@: bioinformatics via web portal– GATE: radiotherapy planning– DILIGENT: digital libraries
• Characteristics– Rapid response: a human waiting for the result!– Many small but CPU-intensive tasks– User is not aware of “grid”!
• Needs– Interfacing (data & computing) with non-grid application
or portal– User and rights management between front-end and grid
![Page 11: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/11.jpg)
11
Workflow
• Examples–“Bronze Standard”: medical images registration–Flood prediction
• Characteristics–Use of grid and non-grid services–Complex set of algorithms for the analysis–Complex dependencies between
individual tasks
• Needs–Tools for managing the workflow itself–Standard interfaces for services (I.e. web-
services)
![Page 12: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/12.jpg)
12
Parallel Jobs
• Examples– Climate modeling– Earthquake analysis– Computational chemistry
• Characteristics– Many interdependent, communicating tasks– Many CPUs needed simultaneously– Use of MPI libraries
• Needs– Configuration of resources for flexible use of
MPI– Pre-installation of optimized MPI libraries
![Page 13: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/13.jpg)
13
Universal Needs• Security
–Ability to control access to services and to data
•Fine-grained access control listsEncryption & logging for more demanding
disciplines•Access control consistently implemented over
all services
• VO Management–Management of users, groups, and roles–Changing the priority of jobs for different
users, groups, roles–Quota management for users, groups, roles–Definition and access to special resources
•Application-level services•Responsive queues (guaranteed, low-latency
execution)
![Page 14: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/14.jpg)
14
Levels of “Gridification”• No development: wrap existing applications as
jobs. No source code modification is required
• Minor modifications: the application exposes minimal interaction with the grid services (e.g. Data Managements)
• Major modifications: a wide portion of the code is rewritten to adopt to the new environment (e.g. parallelization, metadata, information)
• Pure grid applications: developed from scratch. Extensively exploit existing grid services to provide new capabilities customized for a specific domain (e.g. metadata, job management, credential management)
![Page 15: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/15.jpg)
15
No development
• It is the simplest case : the application has not external dependencies, just needs computational power
• Being sure that the application runs on the Worker Node, the executable, and the input files it may need are carried in the Input Sandbox and therefore executed
• Although the application can be directly run, it’s recommended to invoke it from a supplied script which prepare the environment before the execution
![Page 16: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/16.jpg)
16
Minor modifications• The application has minimal
interactions with grid service : it is self consistent but...– e.g. takes input files from an SE– output files are stored on a SE– Perform I/O from a metadata catalog– the application itself can be
downloaded from a SE • User provides an adapter script
which executes needed tasks, set the environment and run the application
![Page 17: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/17.jpg)
17
Major modifications
• A part of the application software is rewritten taking care of the new environment : e.g. because of parallelization, interaction at run time with grid services...– Use of API – Web Service invoked – CLI wrap
• Source code recompiled taking into account this
![Page 18: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/18.jpg)
18
Pure grid application
• Entirely developed from scratch taking into account grid capabilities
• Better exploits grid facilities• Needs a deep knowledge of grid
environment and possibilities in order to engineer entirely the application logic over a grid environment
• Very rare to meet...
![Page 19: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/19.jpg)
19
Additional software installed
• Independently from the integration level, applications may need external software (libraries) that, as application itself, can be provided in several ways– at run time
• carrying out it within InputSandbox• downloading it from a Storage Element
– permanent install on the CE• as tar.gz installed by the VO software
manager • as rpm being part of the gLite release
![Page 20: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/20.jpg)
20
• The Experiment Software Manager (ESM) is the member of the experiment VO entitled to install Application Software in the different sites.
• The ESM is generally acknowledged through a VOMS role (SoftwareManager)
• The ESM can manage (install, validate, remove...) Experiment Software on a site at any time through a normal Grid job, without previous communication to the site administrators.
• Such job has in general no scheduling priorities and will be treated as any other job of the same VO in the same queue.
• lcg-asis is a good generator for such installation jobs
Software installation on CE
![Page 21: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/21.jpg)
21
Software installation on CE
• The site provides a dedicated space where each supported VO can install or remove software. The amount of available space must be negotiated between the VO and the site
• An environmental variable holds the path for the location of a such space. Its format is the following:
VO_<name_of_VO>_SW_DIR
![Page 22: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/22.jpg)
22
Software validation
• Once the software is installed, the ESM can perform the validation
• The validation is a process verifying that the software was properly installed and works as expected
• It can be performed in the same job, after installation (validation on the fly) or later on, via a dedicated job.
![Page 23: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/23.jpg)
23
Software validation• If the ESM judges that the software was
properly installed he publishes in the Information System (IS) the tag which identifies unequivocally the software,
e.g.: VO-cms-CMSSW_1_2_2
• Such a tag is added to the GlueHostApplicationSoftwareRunTimeEnvironment attribute of the IS, using the GRIS running in the CE
• Jobs requesting for a particular piece of software can be directed to the appropriate CE just by setting special requirements on the JDL, e.g.: Requirements=Member("VO-cms-CMSSW_1_2_2",
other.GlueHostApplicationSoftwareRunTimeEnvironment);
![Page 24: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/24.jpg)
24
Invoke Application
• From the UI−Command Line Interfaces / Scripts−APIs−Higher level tools
• From portals−Accessible from any browser−Tailored to applications
![Page 25: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/25.jpg)
25
References (I)
https://grid.ct.infn.it/twiki/bin/view/GILDA/APIUsage
![Page 26: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/26.jpg)
26
References (II)
https://glite-demo.ct.infn.it/
![Page 27: Porting applications to EU-IndiaGrid: EGEE Marco Verlato (@pd.infn.it) EU-IndiaGrid Workshop 16-18 April 2007 Bangalore, India.](https://reader036.fdocuments.in/reader036/viewer/2022062500/56649e305503460f94b20cb6/html5/thumbnails/27.jpg)
27
Acknowledgements
• Thanks to Mike Mineter (NeSC), Valeria Ardizzone and Emidio Giorgio of the INFN GILDA team