Practical Mechanisms for Managing Parallel and Interactive Jobs on Grid Environments Enol Fernández...

Practical Mechanisms for Managing Parallel and Interactive Jobs on Grid

Environments

Enol FernándezUAB

INGRID 2008, 9th april 2008 2partner’s

logo

Introduction CrossBroker Glide In Parallel Job Support Interactive Job Support Conclusions


logo

REMOTE SITE

Internet

REMOTE SITE

Middleware Middleware

SERVICES

Middleware

Batch execution on Grids

F1 F2Job

O1 O2


logo

REMOTE SITE

Internet

REMOTE SITE

Middleware Middleware

SERVICES

Middleware

F1 F2Job

Parallel & Interactive Job Execution

Use of resources from different sitesResource-sets searchCo-allocation & synchronizationFast start-upExecution in high-occupancy situations

F1 F2Job

MPI

I/O forwarding


logo

CrossBroker

CrossBroker does automatic scheduling in Grid Environments

Resource discoveryResource SelectionJob Execution

Jobs not treated by gLite:parallel jobs (MPI)

Run in more than one resource, in a coordinated fashion.

Interactive jobsThe user interacts with the application during its execution


logo

CrossBroker

SchedulingAgent

ResourceSearcher

ApplicationLauncher

Condor-G DAGMan

MigratingDesktop

InformationIndex

ReplicaManager

CrossBroker

EGEE/Globus

LRMS

EGEE/Globus

LRMS

CE CE

WN WN

Outdated informationDynamic changes

LRMS (PBS, LSF, Condor): limited external control

Non cooperative LRMS

Local user jobs

INGRID 2008, 9th april 2008partner’s

logo

Glide In

The ideaEach batch job is encapsulated in an agent that takes control over the WN independently of its LRMS

Lightweight Virtual MachinesEach Worker Node is divided in 2 VMEach VM can execute jobs independently (e.g. batch and interactive)Fast startup of jobs (no need to go trough globus + LRMS)NOT a full virtual machine (Xen, VMWare,…)NO need for special priviledges in the WN


logo

Glide In

SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMSBatchJob


logo

Glide In

SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2

BatchJob


logo

Glide In

SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2

BatchJob

Available for other

jobs


logo

Parallel Job Support

Support for parallel jobs:Open MPIPACX-MPIMPICH-P4MPICH-G2Plain (just the machines)

Takes into account sites capabilites. Low level details of MPI implementations

and sites handled by starter scripts. mpi-start is configured automatically and used by default.


logo


Changes in JDLJOBTYPE:

Normal: sequential jobs, just one CPUParallel: more than one CPU

SUBJOBTYPE:openmpipacx-mpimpichmpich-g2Plain

Plain allows easy extension for supporting new parallel job types


logo


Type = "Job";VirtualOrganisation = "imain";JobType = "Parallel";SubJobType = "pacx-mpi";NodeNumber = 5;Executable = "test-app";Arguments = "-v";InputSandbox = {"test-app", "inputfile"};OutputSanbox = {"std.out", "std.err"};StdErr = "std.err“;StdOutput = "std.out";Rank = other.GlueHostBenchmarkSI00 ;Requirements = other.GlueCEStateStatus == "Production";


logo


[Groups with 1 CEs] [Rank=2000] aocegrid.uab.es:2119/jobmanager-pbs-workq freeCPUs = 10

[Groups with 2 CEs] [Rank=1500] zeus.cyf-kr.edu.pl:2119/jobmanager-pbs-workq freeCPUs = 2 bee001.ific.uv.es:2119/jobmanager-pbs-workq freeCPUs = 3 [Rank=1000] bee001.ific.uv.es:2119/jobmanager-pbs-workq freeCPUs = 3 lngrid02.lip.pt:2129/jobmanager-pbs-workq freeCPUs = 2

CE

CE4= xgrid.icm.edu.plFreeCPUs = 6Disk = 100AverageSI = 1000

CE

CE2=aocegrid.uab.esFreeCPUs = 10Disk = 100AverageSI = 4000

CE

CE3=bee001.ific.uv.esFreeCPUs = 3Disk = 100AverageSI = 1000

CE

CE1=zeus.cyf-kr.edu.plFreeCPUs = 2Disk = 100AverageSI = 2000

CrossBroker

MPI enabled CE

Non-MPI enabled CE

CE

CE5=lngrid02.lip.ptFreeCPUs = 2Disk = 100AverageSI = 1000

[Groups with 1 CEs] [Rank=2000] aocegrid.uab.es:2119/jobmanager-pbs-workq freeCPUs = 10

[Rank=1500] zeus.cyf-kr.edu.pl:2119/jobmanager-pbs-workq freeCPUs = 2 bee001.ific.uv.es:2119/jobmanager-pbs-workq freeCPUs = 3Rank=1000] lngrid02.lip.pt/jobmanager-pbs-workq freeCPUs = 2 bee001.ific.uv.es:2119/jobmanager-pbs-workq freeCPUs = 3


logo


CE

CE3=bee001.ific.uv.esFreeCPUs = 3Disk = 100AverageSI = 1000

CrossBroker

CE

CE5=lngrid02.lip.ptFreeCPUs = 2Disk = 100AverageSI = 1000

MPISubTask

MPISubTask

Startupserver

1. Launch a PACX Startup Server

2. Submit MPI Subtasks

3. MPI-START will start each of the Subtasks

4. Subtask notify the startup server and start running

5. CrossBroker monitors the application


logo


CrossBroker search and selects sets of resources for the jobs

There is no guarantee that all tasks of the same job will start at the same time

1st choice: select only sites with free resources. The job will run immediately. Unfortunately, free resources are not always available2nd choice: allocate a resource temporally and wait until all other tasks show up. Timeshare the resource with a backfilling policy to avoid resource iddleness


logo

Glide In for co-allocation

SchedulingAgent

Condor-G

CrossBroker Grid Resource

LRMSMPIJOB


logo


SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2

Waiting for the rest of

tasks

MPIJOB

MPITask


logo


SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2MPITASK

JOB

BackFillingWhile the MPI waits


logo


SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2MPI

TASK

All tasksReady!

JOB


logo

Interactive Job Support

Fast startup:Cache of resources: fast matchmakingScheduling priority: use free resources or glideinsFast notification of events

CrossBroker injects interactive agents that enable communication between user and job

Transparent to the userCondor Bypass & glogin agents


logo

Interactive Job Support

Changes in JDLINTERACTIVE: true/false. Indicates that the job is interactive and the broker should treat it with higher proirity

INTERACTIVEAGENTINTERACTIVEAGENTARGUMENTS

These attributes specify the command (and its arguments) used to communicate with the user.


logo

Interactive MPI application

Type = "Job";VirtualOrganisation = "imain";JobType = "Parallel";SubJobType = “openmpi";NodeNumber = 4;Interactive = TRUE;InteractiveAgent = “glogin“;InteractiveAgentArguments = “-r –p 195.168.105.65:23433“;Executable = "test-app";InputSandbox = {"test-app", "inputfile"};OutputSanbox = {"std.out", "std.err"};StdErr = "std.err“;StdOutput = "std.out";Rank = other.GlueHostBenchmarkSI00 ;Requirements = other.GlueCEStateStatus == "Production";


logo

Interactive MPI application

Worker

User’s Machine

Video Stream glogin Master

WorkerWorker

MPI

Started with mpi-start

Remote Resource Started by theCrossBroker


logo

Glide In for interactive jobs

SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2BATCH

INT.JOB


logo

Glide In for interactive jobs

SchedulingAgent

Condor-G

CrossBroker

ApplicationLauncher

Grid Resource

LRMS

Agent

VM1 VM2BATCH

INT.JOB

BATCH

Priorityadjustment

Startup-timeReductionOnly one

layer involved


logo

Conclusions & Future work

CrossBroker gives support to Parallel and Interactive jobs

Automatically

Interoperable with EGEE

Glide InFast startup of jobs

Co-allocation without reservation or wasting resources

Future work:Explore more complex multiprogramming (e.g. 3 or more VM)

Decentralization of the services

partner’slogo

Practical Mechanisms for Managing Parallel and Interactive Jobs on Grid

Environments

Enol FernándezUAB

Practical Mechanisms for Managing Parallel and Interactive Jobs on Grid Environments Enol Fernández...

Documents

Transcript of Practical Mechanisms for Managing Parallel and Interactive Jobs on Grid Environments Enol Fernández...