Grid Application Support by the P-GRADE Portal
description
Transcript of Grid Application Support by the P-GRADE Portal
1CEOS WGISS Meeting, May 8-12, 2006, Budapest
MTA SZTAKIComputer and Automation Research Institute
Hungarian Academy of Sciences
www.lpds.sztaki.hu/[email protected]
Grid Application SupportGrid Application Support by the P-GRADE Portal by the P-GRADE Portal
Gergely SiposGergely Sipossipossipos@@sztaki.husztaki.hu
2CEOS WGISS Meeting, May 8-12, 2006, Budapest
OutlineOutline
• Motivation of our work
• P-GRADE Portal in a nutshell
• Grid application development
• Grid application execution and monitoring
• Support for collaborative work
• Conclusions
3CEOS WGISS Meeting, May 8-12, 2006, Budapest
Layered view of Grid systemsLayered view of Grid systems
Basic Grid services:security, job submission, info, …
Higher-level grid services: brokering, data replica, …
Application toolkits, portals
Applications
Command line grid tools
Graphical grid tools:P-GRADE Portal
Resources to be shared
Grid middleware:e.g. Globus, EGEE, …
E-Scientists
4CEOS WGISS Meeting, May 8-12, 2006, Budapest
Motivation to theMotivation to theP-GRADE PortalP-GRADE Portal
• Fast evolution of Grid middleware technologies and tools:– GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, …
• Many production Grids are now freely available for e-Scientists– EGEE (LCG-2 gLite), UK NGS (GT2),
US Open Science Grid (GT2 GT4), NorduGrid (ARC), …• The same set of services are available everywhere, but
implemented in different ways– Computation services, data services, security services, (brokers)
Let’s provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services!
5CEOS WGISS Meeting, May 8-12, 2006, Budapest
P-GRADE Portal in a nutshellP-GRADE Portal in a nutshell• General purpose, workflow-oriented computational Grid
portal. Supports the development and execution of workflow-based Grid applications.
• Based on standard portlet framework (Gridsphere)– Easy to expand with new portlets (e.g. application-specific portlets)– Easy to tailor to community needs
• Grid services supported by the portal:
Service EGEE grids Globus grids
Job execution Computing Element GRAM
File storage Storage Element GridFTP server
Certificate management MyProxy
Information system BDII MDS-2
Brokering Workload Management System ---
Job monitoring Mercury
Workflow & job visualization PROVE
The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level
6CEOS WGISS Meeting, May 8-12, 2006, Budapest
What is a P-GRADE Portal workflow?What is a P-GRADE Portal workflow?
• a directed acyclic graph where– Nodes represent jobs (batch
programs to be executed on a computing resource)
– Ports represent input/output files the jobs expect/produce
– Arcs represent file transfer operations
• semantics of the workflow:– A job can be executed if all
of its input files are available
7CEOS WGISS Meeting, May 8-12, 2006, Budapest
Two levels of parallelismTwo levels of parallelism
• The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems
• Semantics of the workflow enables two levels of parallelism:
Each job can be a parallel program
– Parallel execution inside a workflow node– Parallel execution among workflow nodes
Multiple jobs run parallel
8CEOS WGISS Meeting, May 8-12, 2006, Budapest
25 x
10 x25 x 5 x
Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property
Processed information:surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models
Requirements:•Execution time < 10 min•High resolution (1km)
Ultra-short range weather forecast Ultra-short range weather forecast (Hungarian Meteorology Service)(Hungarian Meteorology Service)
9CEOS WGISS Meeting, May 8-12, 2006, Budapest
Multi-Grid portal: Multi-Grid portal: Demonstration @ EGEE User ForumDemonstration @ EGEE User Forum
Job Repository Job Repository (GEMLCA)(GEMLCA)
ManchesterPortal Client in Geneva
Leeds
P-GRADE Portalserver
Portal Server in London
Executable
NGS GT2 (UK)
WestFocus GT4 Grid (UK)
UoW
Brunel
Poznan
Budapest
EGEE (CE)
Executable
Executable
ServiceInvocation
Executable
10CEOS WGISS Meeting, May 8-12, 2006, Budapest
The typical user scenarioThe typical user scenarioPart 1 - development phasePart 1 - development phase
Certificate servers
Portalserver
Gridservices
START EDITOR
OPEN & EDIT or DEVELOP WORKFLOW
SAVE WORKFLOW
11CEOS WGISS Meeting, May 8-12, 2006, Budapest
Certificate servers
Portalserver
Gridservices
TRANSFER FILES, SUBMIT JOBS
DOWNLOAD RESULTS
DOWNLOAD RESULTS
The typical user scenarioThe typical user scenarioPart 2 - execution phasePart 2 - execution phase
VISUALIZE JOBS and
WORKFLOW PROGRESS
MONITOR JOBS
DOWNLOAD PROXY CERTIFICATES
SUBMIT WORKFLOW
12CEOS WGISS Meeting, May 8-12, 2006, Budapest
Opening the workflow editorOpening the workflow editor
The editor is a Java Webstart applicationdownload and installation is only a click!
13CEOS WGISS Meeting, May 8-12, 2006, Budapest
Workflow Workflow EditorEditor
Define DAG of batch jobs:1. Drag & drop components:
jobs and ports
2. Connect ports by channels (no cycles, no loops)
3. Define job and port properties
14CEOS WGISS Meeting, May 8-12, 2006, Budapest
Job property windowJob property window
Properties of a job:• Binary executable• (Type of executable)• (Number of processors)• Command line
parameters• The resource to be used
for the execution:•Grid•Broker / resource
15CEOS WGISS Meeting, May 8-12, 2006, Budapest
Support for manual resource selection:Support for manual resource selection:information system browserinformation system browser
The information system portlet queries EGEE
and Globus information systems
16CEOS WGISS Meeting, May 8-12, 2006, Budapest
Support for broker-based Support for broker-based resource selectionresource selection
1. Select a broker Grid/VO for the job
2. (Specify extra ranks & requirements for the job in Job Description Language)
3. The broker will find the best resource for your job!
17CEOS WGISS Meeting, May 8-12, 2006, Budapest
Defining input/output dataDefining input/output data
File type: input: required by the job output: produced by the jobFile location: local: my desktop remote: grid storage siteFile name: unique name of the fileFile storage type: Permanent: final result of the WF Volatile: only used for inter-job data transfer
18CEOS WGISS Meeting, May 8-12, 2006, Budapest
The typical user scenarioThe typical user scenarioExecution phase – step 1:Execution phase – step 1:
Certificate servers
Portalserver
Gridservices
DOWNLOAD PROXY CERTIFICATES
19CEOS WGISS Meeting, May 8-12, 2006, Budapest
Prerequisite of workflow execution: Prerequisite of workflow execution: Download user certificatesDownload user certificates
1. Certificate server details:• Hostname• Port number• User name• Password
2. Certificate parameters:• Lifetime
20CEOS WGISS Meeting, May 8-12, 2006, Budapest
Result of certificate downloadResult of certificate download
Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time!
SEE-GRID resourcesHUNGRID resources
21CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
22CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
23CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
24CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
25CEOS WGISS Meeting, May 8-12, 2006, Budapest
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
White/Red/Green color means the job is initialised/running/finished
26CEOS WGISS Meeting, May 8-12, 2006, Budapest
On-Line application monitoringOn-Line application monitoring
- The portal monitors and visualizes parallel jobs
- The portal monitors and visualizes workflows:
-Input data transfer
-Job execution
-Output data transfer
27CEOS WGISS Meeting, May 8-12, 2006, Budapest
Downloading the results…Downloading the results…
- Download small files in a single ZIP file
- Keep large files in the Grid for future analysis!
28CEOS WGISS Meeting, May 8-12, 2006, Budapest
Collaborative work Collaborative work by the P-GRADE Portalby the P-GRADE Portal
• Export and import workflows and execution results from/into the portal
• Share legacy applications as services with other portal users• Share successfully executed jobs as services with other
portal users• Integrate other users’ legacy applications and jobs into your
workflow
29CEOS WGISS Meeting, May 8-12, 2006, Budapest
• P-GRADE Portal service is available for– SEE-GRID infrastructure– HUNGRID VO of EGEE– Central European VO of EGEE– GILDA: Training VO of EGEE– UK National Grid Service
• Under preparation for– US Open Science Grid, Economy-Grid,
Swiss BioGrid, BioInfoGrid, Turkish Grid
• P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO
Please contact us!
RReferenceseferences
30CEOS WGISS Meeting, May 8-12, 2006, Budapest
Main conclusion:Main conclusion:Easy-to-use, technology-neutral Easy-to-use, technology-neutral
Grid portal for e-ScientistsGrid portal for e-Scientists• The P-GRADE Portal hides differences of Grids
• Globus X – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies
• Graphical tools for application development, execution and monitoring
• Seq & parallel components can be integrated into large Grid applications
• Your code does not have to contain grid specific calls• Support for collaborative research
• Share workflows• Share legacy applications and jobs as application services
• Built by standard portlet API• customizable to specific application areas and user groups
(such as earth scientists…)
31CEOS WGISS Meeting, May 8-12, 2006, Budapest
Further information…Further information…
– Visit the www.lpds.sztaki.hu/pgportal site(manuals, slide shows, installation procedure, etc.)
– Visit or request a training event!(see list of events on P-GRADE Portal homepage)
• Lectures, demos, hands-on tutorials, application development support
– Get an account for a portal service:• VOCE portal - SZTAKI• SEEGRID portal – SZTAKI• HUNGrid portal – SZTAKI• NGS portal – University of Westminster
– Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community!
32CEOS WGISS Meeting, May 8-12, 2006, Budapest
• P-GRADE Portal tutorial in Paris (free event!)22-23, June – lectures, hands-on, application
supportSee details on P-GRADE Portal homepage!
• Joint Regional CE EGEE and SEE-GRIDSummer School on Grid Application Support
3-8 July, Budapest, SZTAKI
www.egee.hu/grid06
Thank you!Thank you!Questions?Questions?