B R E A D..:: part of the Polish culture ::... Our group Marcin Marcin Juliusz Artur Juliusz Artur.
Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, 21-22...
-
Upload
magdalen-dawson -
Category
Documents
-
view
214 -
download
0
Transcript of Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, 21-22...
Resource Brokering in the PROGRESS Project
Juliusz Pukacki
Grid Resource Management Workshop, 21-22 October 2003
Progress Project
Duration: December 2001 – May 2003Deployment: June 2003-December 2003Project Partners
SUN Microsystems PolandPSNC IBCh PoznańCyfronet AMM, KrakówTechnical University Łódź
Co-funded by The State Committee for Scientific Research (KBN) and SUN Microsystems Poland
PROGRESS Goals
Provide computational portal for scientistsProvide easy to use user interface
Provide Grid middleware infrastructure
Make use of distributed cluster of SUN SF6800 is a pilot testbed installation
Make use of PIONIER network infrastructure
PROGRESS Testbed
Cluster of 80 processors
Networked Storage of 1,3 TB
Software: ORACLE, HPC Cluster Tools, Sun ONE, Sun Grid Engine
Architecture - overview
PORTAL
HPCRESOURCES
GRIDMANAGEMENT
SYSTEM
GRID SERVICEPROVIDER
4-tier new4-tier newgrid-portal environmentgrid-portal environment
PORTAL
HPCRESOURCES
GRIDMANAGEMENT
SYSTEM
3-tier classical3-tier classicalgrid-portal grid-portal
environmentenvironment
Grid Broker - goals
Provide universal mechanism of resource management for Grid environment
Make use of common lower level Grid infrastructure (Globus)
Service oriented approach
Well-defined universal interface, implemented in common technology (Web Services)
Modular architecture
Grid Broker - functionality
Ability to choose the best resource for the job execution, according to Job Description and chosen mapping algorithm;Ability to submit the GRMS Task according to provided Job Description;Ability to submit a set of tasks with precedence constraints (sequence/parallel structure)Provides information about the Task status;Provides other information about Tasks (name of host where the Task is/was running, start time, finish time);
Grid Broker – functionality (2)
Provides list of resources that fullfills requirements provided in Job Description);Provides a list of Tasks submitted by given user;Ability to transfer input and output files (GridFTP, GASS, Data Management System);Ability to cancel the Task;Provides notification mechanism
Grid Broker - overview
Broker
Progress Grid Middleware
Data Management
System
•MDS•GRAM•GridFTP•GASS
Grid Infrastructure
PROGRESSPortal
User Access Layer
InformationService
Grid Broker - details
ResourceDiscovery
Broker
TaskManager
WebService
Interface
PROGRESS Portal
System LayerServices
PROGRESSServices
WorkflowMngmt.
TaskRegistry
DB
Grid Broker - modules
Web Service InterfaceProvides GSI enabled web service interface for Clients (PROGRESS Portal)Supports callback communication to Client (notification service)
Workflow Management ModuleCreates workflow graph of tasks from Job DescriptionPuts tasks to databaseControls of tasks execution according to precedence constraints
Grid Broker – modules (2)
Broker ModuleManages process of task submittion
Chooses the best resources for task execution
Transfers executable to chosen resource
Transfers input and output files
Resource Discovery ModuleFinds resources that fullfills requirements described in Job Description
Provides information about resources, required for task scheduling
Grid Broker – modules (3)
Task Manager ModuleAbility to check current status of task
Ability to cancel running task
Monitors for status changes of running task
Task RegistryStores information about Task execution (start time, finish time, machine where executed, current status, Job Description)
Job Description - XRSL
Provides ability to express precedence constraints of Task
Grid Broker’s Job contains one or more Tasks
Tasks are grouped into sequence or parallel execution
Job Description - schema
Parallel/sequence structure
Job1
single
Task1
Job3
parallel
Job2
sequence
Task2 Task3Task1 Task1
Task2
Task3
Task2a Task2b Task3c
Task2a
Task2b
Task2c
Job Description - Task
Task executable file location arguments file argument (files which have to be present in working directory of running executable) environment variables standard input standard output standard error
Job Description - Task
Resource requirements name of host for job execution operating system required local resource management system minimum memory required minimum number of cpus required minimum speed of cpu other parameter passed directly to Globus GRAM
XRSL - example<?xml version="1.0" encoding="UTF-8"?><job id="Application">
<user dn="dc=pl, dc=poznan, dc=man, uid=pukacki"/><task id="Task1">
<executable name="computation" dmsid="12345" type="single" count="1">
<arguments><value>4</value><file id="772" type="in">mydata.dat</file>
</arguments><environment>
<stdin dmsid ="772"></stdin><stdout dmsid="774"/><stderr dmsid="775"/><variable name="CONST">5</variable>
</environment></executable><resources>
<resource type="memory" value="20"/></resources>
</task></job>
XRSL – example (2)<?xml version="1.0" encoding="UTF-8"?><job id="Application">
<user dn="dc=pl, dc=poznan, dc=man, uid=pukacki"/><sequence id="seq1">
<task id="Task1"><executable name="computation" dmsid="12345">
<arguments><value>4</value>
</arguments><environment>
<stdout dmsid="774"/></environment>
</executable></task><task id="Task2">
<executable name="computation" dmsid="8888"><arguments>
<value>4</value></arguments><environment>
<stdout dmsid="774"/></environment>
</executable></task>
</sequence></job>
Thank you