USC Viterbi School of Engineering Ewa Deelman Resource Management.

USC Viterbi School of Engineering

Ewa Deelman

Resource Management


Outline

• General resource management framework• Managing a set of resources

– Locally

– Across the network

• Managing distributed resources


Types of resources

LAN

O(100GB-1TB)

cluster

Myrinet, infiniband

ethernet

Local system

Loose set of resources

headnode


Resource managers/schedulers

RemoteResource

Loc

al S

ched

ule

r

Rem

ote

Con

nec

tivi

ty

Local Resource R

emot

e S

ub

mis

sion

Wor

klo

ad M

anag

emen

t

network “Su

per

” S

ched

ule

r

Wor

kfl

ow

Man

agem

ent


Basic Resource Management Functionality

• Accept requests• Start jobs on the resource• Log information about jobs• Reply to queries about request status• Prioritize among requests• Provide fault tolerance• Provide resource reservation• Management of dependent jobs


Schedulers/Resource Managers

• Local system– OS, scheduling across multiple cores

• Clusters– Basic Scheduler

• PBS, LSF, torque, Condor

– “Super” Scheduler• Maui

• Loose set of resources– Condor


Schedulers/Resource Managers

• Remote Connectivity– Globus GRAM

• Remote Submission to possibly many sites– Condor-G

• Workload management– Condor

• Workflow management– DAGMan

– Pegasus



RemoteResource

Loc

al S

ched

ule

r

Rem

ote

Con

nec

tivi

ty

Local Resource R

emot

e S

ub

mis

sion

Wor

klo

ad M

anag

emen

t

network “Su

per

” S

ched

ule

r

Wor

kfl

ow

Man

agem

ent

PBS

O(100GB-1TB)

cluster

headnode


Cluster Scheduling--PBS

– Aggregates all the computing resources (cluster nodes) into a single entity

– Schedules and distributes applications• On a single node, across multiple nodes

– Supports various types of authentication• User name based• X509 certificates

– Supports various types of authorization• User, group, system

– Provides job monitoring • Collects real time data on the state of the system and job status• Logs real time data and other info on queued jobs,

configuration information, etc


Cluster Scheduling PBS

• Performs data stage-in and out onto the nodes• Starts the computation on a node• Uses backfilling for job scheduling• Supports simple dependencies

– Run Y after X

– If X succeeds run Y otherwise run Z

– Run Y after X regardless of X success

– Run Y after specified time

– Run X anytime after specified time



• Advanced features– Resource reservation

• Specified time (start_t) and duration (delta)• Checks whether reservation conflicts with running jobs, and

other confirmed reservation• If OK—sets up a queue for the reservation with a user-level acl• Queue started at start_t• Jobs killed past the reservation time

– Cycle harvesting• Can include idle resources into the mix

– Peer scheduling• To other PBS systems• Pull-based



• Allocation properties– Administrator can revoke any allocation (running or

queued)

– Jobs can be preempted• Suspended

• Checkpointed

• Requeued

– Jobs can be restarted X times


Single Queue Backfilling

• A job is allowed to jump in the queue ahead of jobs that are delayed due to lack of resources

• Non-preemptive • Conservative backfilling

– A job is allowed to jump ahead provided it does not delay any previous job in the queue

• Aggressive backfilling– A job is allowed to jump ahead if the first job is not

affected

– Better performing then conservative backfilling


Aggressive backfilling

• Job (arrival time, number of procs, expected runtime)

• Any job that executes longer than expected runtime is killed

• Define pivot—first job in the queue• If enough resources for pivot, execute pivot and

define a new pivot• Otherwise sort all currently executing jobs in

order of their expected completion time– Determine pivot time—when sufficient resources for

pivot are available


Aggressive backfilling

• At pivot time, any idle processors not required for pivot job are defined as extra processors

• The scheduler searches for the first queued job that– Requires no more than the currently idle processors and

will finish by the pivot time, or

– Requires no more than the minimum currently idle processors and the extra processors

• Once a job becomes a pivot, it cannot be delayed• Possible that a job will end sooner and pivot starts

before pivot time


Aggressive backfilling example

Job A(10 procs,40mins)

8

time

QueueJob B (12 procs, 1 hour)Job C (20 procs, 2 hours)Job D (2 procs, 50 mins)Job E (6 procs, 1 hour)Job F (4 procs, 2 hours)

10

processors

Running Job A(10 procs, 40 mins)




8

time

queueJob B (12 procs, 1 hour)Job C (20 procs, 2 hours)Job E (6 procs, 1 hour)Job F (4 procs, 2 hours)

10

processorsPIVOTJob B

JOB D

Running Job A(10 procs, 40 mins)Job D (2 procs, 50 mins)



QueueJob B (12 procs, 1 hour)Job C (20 procs, 2 hours)Job E (6 procs, 1 hour)


8

time

10

processorsPIVOTJob B

Job D

Running Job A(10 procs, 40 mins)Job D (2 procs, 50 mins)Job F (4 procs, 2 hours)

Job F



QueueJob C (20 procs, 2 hours)Job E (6 procs, 1 hour)


8

time

10

processorsJob B

Job DJob F

Running Job D (2 procs, 50 mins)Job F (4 procs, 2 hours)Job B (12 procs, 1 hour)



QueueJob C (20 procs, 2 hours)Job E (6 procs, 1 hour)


8

time

10processors

Job B

Job DJob F

Running Job D (2 procs, 50 mins)Job F (4 procs, 2 hours)Job B (12 procs, 1 hour)

PivotJob C (20 procs, 2 hours)


Multiple-Queue Backfill

From “Self-adapting Backfilling Scheduling for Parallel Systems”, B.G.Lawson et al. Proceedings of the 2002 International Conference on Parallel Processing (ICPP'02), 2002



RemoteResource

Loc

al S

ched

ule

r

Rem

ote

Con

nec

tivi

ty

Local Resource R

emot

e S

ub

mis

sion

Wor

klo

ad M

anag

emen

t

network “Su

per

” S

ched

ule

r

Wor

kfl

ow

Man

agem

ent

Maui


“Super”/External Scheduler--Maui

• External scheduler– Does not mandate a specific resource manager– Fully controls the workload submitted through the local

management system– Enhances resource manager capabilities– Supports advance reservations

• <start-time, duration, acl>• Irrevocable allocations

– For time critical tasks such as weather modeling– The reservation is guaranteed to start at a given time and last for a

given duration regardless of future workload

• Revocable allocations– The reservation would be released if it precludes a higher priority

request from being granted


Maui

• Retry “X” times mechanism for finding resources and starting a job– If job does not start—can be put on hold

• Can be configured preemptive/nonpreemptive• Exclusive allocations

– No sharing of resources

• Malleable allocations– Changes can be made in time and space

• Increases if possible

• Decreases always


Maui

• Resource Querying– Request (account mapping, minimum duration,

processors)– Maui returns a list of tuples

• <feasible start_time, total available resources, duration, resource cost, quality of the information>

– Can also return a single value which incorporate• Anticipated, non-submitted jobs• Workload profiles

• Provides event notifications– Solitary and threshold events– Job/reservation: creation, start, preemption, completion,

various failures


Maui

• Courtesy allocation– Placeholder which is released if confirmation is not

received within a time limit

• Depending on the underlying scheduler Maui can– Suspend/resume

– Checkpoint/restart

– Requeue/restart



RemoteResource

Loc

al S

ched

ule

r

Rem

ote

Con

nec

tivi

ty

Local Resource R

emot

e S

ub

mis

sion

Wor

klo

ad M

anag

emen

t

network “Su

per

” S

ched

ule

r

Wor

kfl

ow

Man

agem

ent

Globus GRAM


GRAM - Basic Job Submission and Control Service

• A uniform service interface for remote job submission and control– Includes file staging and I/O management

– Includes reliability features

– Supports basic Grid security mechanisms

– Asynchronous monitoring

– Interfaces with local resource managers, simplifies the job of metaschedulers/brokers

• GRAM is not a scheduler.– No scheduling

– No metascheduling/brokering

Slide courtesy of Stuart Martin


Grid Job Management Goals

Provide a service to securely:• Create an environment for a job• Stage files to/from environment• Cause execution of job process(es)

– Via various local resource managers

• Monitor execution• Signal important state changes to client• Enable client access to output files

– Streaming access during execution



Job Submission Model

• Create and manage one job on a resource• Submit and wait• Not with an interactive TTY

– File based stdin/out/err

– Supported by all batch schedulers

• More complex than RPC– Optional steps before and after submission message

– Job has complex lifecycle• Staging, execution, and cleanup states

• But not as general as Condor DAG, etc.

– Asynchronous monitoring



Job Submission Options

• Optional file staging– Transfer files “in” before job execution

– Transfer files “out” after job execution

• Optional file streaming– Monitors files during job execution

• Optional credential delegation– Create, refresh, and terminate delegations

– For use by job process

– For use by GRAM to do optional file staging



Job Submission Monitoring

• Monitor job lifecycle– GRAM and scheduler states for job

• StageIn, Pending, Active, Suspended, StageOut, Cleanup, Done, Failed

– Job execution status• Return codes

• Multiple monitoring methods– Simple query for current state

– Asynchronous notifications to client



Secure Submission Model

• Secure submit protocol– PKI authentication

– Authorization and mapping• Based on Grid ID

– Further authorization by scheduler• Based on local user ID

• Secure control/cancel– Also PKI authenticated

– Owner has rights to his jobs and not others’



Secure Execution Model

• After authorization…• Execute job securely

– User account “sandboxing” of processes• According to mapping policy and request details

– Initialization of sandbox credentials• Client-delegated credentials

• Adapter scripts can be customized for site needs– AFS, Kerberos, etc



Secure Staging Model

• Before and after sandboxed execution…

• Perform secure file transfers– Create RFT request

• To local or remote RFT service

• PKI authentication and delegation

• In turn, RFT controls GridFTP– Using delegated client credentials

– GridFTP• PKI authentication

• Authorization and mapping by local policy files

• further authorization by FTP/unix perms



GRAM WSDLs+

Job Description Schema

(executable, args,env, …)

Users/Applications: Job Brokers, Portals, Command line tools, etc.

Resource Managers: PBS, Condor, LSF, SGE, Loadleveler, Fork

WS standard interfaces for subscription, notification,destruction

GRAM4



GRAM4 Approach

GridFTPRFT

Delegation

GridFTP

GRAMservices

local sched.

user job

compute element

compute element and service host(s)

remote storage element(s)

FTP data

FTP control

clie

nt

job submit

delegate

xfer

req

uest

local job control

delegateGRAMadaptersu

do



File staging retry policy

• If a file staging operation fails, it may be non-fatal and retry may be desired– Server defaults for all transfers can be configured

– Defaults can be overridden for a specific transfer

• Output can be streamed



Throttle staging work

• A GRAM submission that specifies file staging imposes load on the service node executing the GRAM service.– GRAM is configured for a

maximum number of “worker” threads and thus a maximum number of concurrent staging operations.

O(100GB-1TB)

cluster

GRAM/PBSHeadnode


Local Resource Manager Interface• The GRAM interface to the LRM to submit,

monitor, and cancel jobs.– Perl scripts + SEG

• Scheduler Event Generator (SEG) generated by local resource manager provides efficient monitoring between the GRAM service and the LRM for all jobs for all users



Fault tolerance• GRAM can recover from a container or host

crash. Upon restart, GRAM will resume processing of the users job submission– Processing resumes for all jobs once the service

container has been restarted

Job cancellation• Allow a job to be cancelled

– WSRF standard “Destroy” operation



State Access: Push

• Allow clients to request notifications for state changes– WS Notifications

• Clients can subscribe for notifications to the “job status” resource property

State Access: Pull

• Allow clients to get the state for a previously submitted job– The service defines a WSRF resource property that contains

the value of the job state. A client can then use the standard WSRF getResourceProperty operation.


USC Viterbi School of Engineering Ewa Deelman Resource Management.

Documents

Transcript of USC Viterbi School of Engineering Ewa Deelman Resource Management.