Using New Features in Condor 7.2
-
Upload
john-fletcher -
Category
Documents
-
view
19 -
download
3
description
Transcript of Using New Features in Condor 7.2
Condor ProjectComputer Sciences DepartmentUniversity of Wisconsin-Madison
Using New Features in Condor 7.2
www.cs.wisc.edu/Condor
Outline
› Startd Hooks› Job Router› Job Router Hooks› Power Management› Dynamic Slot Partitioning› Concurrency Limits› Variable Substitution› Preemption Attributes
2
www.cs.wisc.edu/Condor
Startd Job Hooks
› Users wanted to take advantage of Condor’s resource management daemon (condor_startd) to run jobs, but they had their own scheduling system.h Specialized scheduling needsh Jobs live in their own database or other
storage rather than a Condor job queue
3
www.cs.wisc.edu/Condor
Our solution
› Make a system of generic “hooks” that you can plug into:h A hook is a point during the life-cycle
of a job where the Condor daemons will invoke an external program
h Hook Condor to your existing job management system without modifying the Condor code
4
www.cs.wisc.edu/Condor
How does Condor communicate with
hooks?› Passing around ASCII ClassAds via standard input and standard output
› Some hooks get control data via a command-line argument (argv)
› Hooks can be written in any language (scripts, binaries, whatever you want) so long as you can read/write Stdin/out
5
www.cs.wisc.edu/Condor
What hooks are available?
› Hooks for fetching work (startd):h FETCH_JOBh REPLY_FETCHh EVICT_CLAIM
› Hooks for running jobs (starter):h PREPARE_JOBh UPDATE_JOB_INFOh JOB_EXIT
6
www.cs.wisc.edu/Condor
HOOK_FETCH_JOB
› Invoked by the startd whenever it wants to try to fetch new workh FetchWorkDelay expression
› Stdin: slot ClassAd
› Stdout: job ClassAd
› If Stdout is empty, there’s no work
7
www.cs.wisc.edu/Condor
HOOK_REPLY_FETCH
› Invoked by the startd once it decides what to do with the job ClassAd returned by HOOK_FETCH_WORK
› Gives your external system a chance to know what happened
› argv[1]: “accept” or “reject”› Stdin: slot and job ClassAds› Stdout: ignored
8
www.cs.wisc.edu/Condor
HOOK_EVICT_CLAIM
› Invoked if the startd has to evict a claim that’s running fetched work
› Informational only: you can’t stop or delay this train once it’s left the station
› Stdin: both slot and job ClassAds
› Stdout: ignored
9
www.cs.wisc.edu/Condor
HOOK_PREPARE_JOB
› Invoked by the condor_starter when it first starts up (only if defined)
› Opportunity to prepare the job execution environmenth Transfer input files, executables, etc.
› Stdin: both slot and job ClassAds› Stdout: ignored, but starter won’t
continue until this hook exits› Not specific to fetched work
10
www.cs.wisc.edu/Condor
HOOK_UPDATE_JOB_INFO
› Periodically invoked by the starter to let you know what’s happening with the job
› Stdin: slot and job ClassAdsh Job ClassAd is updated with additional
attributes computed by the starter:• ImageSize, JobState, RemoteUserCpu, etc.
› Stdout: ignored
11
www.cs.wisc.edu/Condor
HOOK_JOB_EXIT
› Invoked by the starter whenever the job exits for any reason
› Argv[1] indicates what happened:h “exit”: Died a natural deathh “evict”: Booted off prematurely by the
startd (PREEMPT == TRUE, condor_off, etc)
h “remove”: Removed by condor_rmh “hold”: Held by condor_hold
12
www.cs.wisc.edu/Condor
HOOK_JOB_EXIT …
› “HUH!?! condor_rm? What are you talking about?”h The starter hooks can be defined even for
regular Condor jobs, local universe, etc.
› Stdin: copy of the job ClassAd with extra attributes about what happened:h ExitCode, JobDuration, etc.
› Stdout: ignored
13
www.cs.wisc.edu/Condor
Defining hooks
› Each slot can have its own hook ”keyword”h Prefix for config file parametersh Can use different sets of hooks to talk to
different external systems on each sloth Global keyword used when the per-slot
keyword is not defined
› Keyword is inserted by the startd into its copy of the job ClassAd and given to the starter
14
www.cs.wisc.edu/Condor
Defining hooks: example # Most slots fetch work from the database system
STARTD_JOB_HOOK_KEYWORD = DATABASE# Slot4 fetches and runs work from a web serviceSLOT4_JOB_HOOK_KEYWORD = WEB
# The database system needs to both provide work and# know the reply for each attempted claimDB_DIR = /usr/local/condor/fetch/dbDATABASE_HOOK_FETCH_WORK = $(DB_DIR)/fetch_work.php DATABASE_HOOK_REPLY_FETCH = $(DB_DIR)/reply_fetch.php
# The web system only needs to fetch workWEB_DIR = /usr/local/condor/fetch/webWEB_HOOK_FETCH_WORK = $(WEB_DIR)/fetch_work.php
15
www.cs.wisc.edu/Condor
Semantics of fetched jobs
› Condor_startd treats them just like any other kind of job:h All the standard resource policy
expressions apply (START, SUSPEND, PREEMPT, RANK, etc).
h Fetched jobs can coexist in the same pool with jobs pushed by Condor, COD, etc.
h Fetched work != Backfill
16
www.cs.wisc.edu/Condor
Semantics continued
› If the startd is unclaimed and fetches a job, a claim is created
› If that job completes, the claim is reused and the startd fetches again
› Keep fetching until either:h The claim is evicted by Condorh The fetch hook returns no more work
17
www.cs.wisc.edu/Condor
Limitations of the hooks
› If the starter can’t run your fetched job because your ClassAd is bogus, no hook is invoked to tell you about ith We need a HOOK_STARTER_FAILURE
› No hook when the starter is about to evict you (so you can checkpoint)h Can implement this yourself with a
wrapper script and the SoftKillSig attribute
18
www.cs.wisc.edu/Condor
Job Router
› Automated way to let jobs run on a wider array of resourcesh Transform jobs into different formsh Reroute jobs to different destinations
19
www.cs.wisc.edu/Condor
What is “job routing”?
20
Universe = “vanilla”Executable = “sim”Arguments = “seed=345”Output = “stdout.345”Error = “stderr.345”ShouldTransferFiles = TrueWhenToTransferOutput = “ON_EXIT”
Universe = “grid”GridType = “gt2”GridResource = \ “cmsgrid01.hep.wisc.edu/jobmanager-condor”
Executable = “sim”Arguments = “seed=345”Output = “stdout”Error = “stderr”ShouldTransferFiles = TrueWhenToTransferOutput = “ON_EXIT”
JobRouter
Routing Table: Site 1 … Site 2 …
final status
routed (grid) joboriginal (vanilla) job
www.cs.wisc.edu/Condor
Routing is just site-level matchmaking
› With feedback from job queue• number of jobs currently routed to site X• number of idle jobs routed to site X• rate of recent success/failure at site X
› And with power to modify job ad• change attribute values (e.g. Universe)• insert new attributes (e.g. GridResource)• add a “portal” grid proxy if desired
21
www.cs.wisc.edu/Condor
Configuring the Routing Table
› JOB_ROUTER_ENTRIES• list site ClassAds in configuration file
› JOB_ROUTER_ENTRIES_FILE• read site ClassAds periodically from a file
› JOB_ROUTER_ENTRIES_CMD• read periodically from a script• example: query a collector such as Open
Science Grid Resource Selection Service
22
www.cs.wisc.edu/Condor
Syntax
› List of sites in new ClassAd format[ Name = “Grid Site 1”;…
][ Name = “Grid Site 2”; …][ Name = “Grid site 3”; …]…
23
www.cs.wisc.edu/Condor
Syntax
[ Name = “Site 1”; GridResource = “gt2 gk.foo.edu”; MaxIdleJobs = 10; MaxJobs = 200; FailureRateThreshold = 0.01; JobFailureTest = other.RemoteWallClockTime <
1800 Requirements = target.WantJobRouter is True; delete_WantJobRouter = true; set_PeriodicRemove = JobStatus == 5;]
24
www.cs.wisc.edu/Condor
What Types of Input Jobs?
› Vanilla Universe› Self Contained
(everything needed is in file transfer list)
› High Throughput(many more jobs than cpus)
25
www.cs.wisc.edu/Condor
Grid Gotchas
› Globus gt2h no exit status from job (reported as 0)
› Most grid universe typesh must explicitly list desired output files
26
www.cs.wisc.edu/Condor
JobRouter vs. Glidein
› Glidein - Condor overlays the gridh job never waits in remote queueh job runs in its normal universeh private networks doable, but add to complexityh need something to submit glideins on demand
› JobRouterh some jobs wait in remote queue (MaxIdleJobs)
h job must be compatible with target grid semanticsh simple to set up, fully automatic to run
27
www.cs.wisc.edu/Condor
Job Router Hooks
› Truly transform jobs, not just reroute themh E.g. stuff a job into a virtual machine
(either VM universe or Amazon EC2)
› Hooks invoked like startd ones
28
www.cs.wisc.edu/Condor
HOOK_TRANSLATE
› Invoked when a job is matched to a route
› Stdin: route name and job ad
› Stdout: transformed job ad
› Transformed job is submitted to Condor
29
www.cs.wisc.edu/Condor
HOOK_UPDATE_JOB_INFO
› Invoked periodically to obtain extra information about routed job
› Stdin: routed job ad
› Stdout: attributes to update in routed job ad
30
www.cs.wisc.edu/Condor
HOOK_JOB_FINALIZE
› Invoked when routed job has completed
› Stdin: ads of original and routed jobs
› Stdout: modified original job ad or nothing (no updates)
31
www.cs.wisc.edu/Condor
HOOK_JOB_CLEANUP
› Invoked when original job returned to schedd (both success and failure)
› Stdin: Original job ad
› Use for cleanup of external resources
32
www.cs.wisc.edu/Condor
Power Management
› Hibernate execute machines when not needed
› Condor doesn’t handle waking machines up yet
› Information to wake machines available in machine ads
33
www.cs.wisc.edu/Condor
Configuring Power Management
› HIBERNATEh Expression evaluated periodically by
all slots to decide when to hibernateh All slots must agree to hibernate
› HIBERNATE_CHECK_INTERVALh Number of seconds between
hibernation checks
34
www.cs.wisc.edu/Condor
Setting HIBERNATE
› HIBERNATE must evaluate to one of these strings:h “NONE”, “0”h “S1”, “1”, “STANDBY”, “SLEEP”h “S2”, “2”h “S3”, “3”, “RAM”, “MEM”h “S4”, “4”, “DISK”, “HIBERNATE”h “S5”, “5”, “SHUTDOWN”
› These numbers are ACPI power states
35
www.cs.wisc.edu/Condor
Power Management on Linux
› On linux, theses methods are tried in order for setting power level:h pm-UTIL toolsh /sys/powerh /proc/ACPI
› LINUX_HIBERNATION_METHOD can be set to pick a favored method
36
www.cs.wisc.edu/Condor
Sample Configuration
ShouldHibernate = \ ((KeyboardIdle > $(StartIdleTime)) \ && $(CPUIdle) \ && ($(StateTimer) > (2 * $(HOUR)))HIBERNATE = ifThenElse( \ $(ShouldHibernate), “RAM”, “NONE” )HIBERNATE_CHECK_INTERVAL = 300LINUX_HIBERNATION_METHOD = “/proc”
37
www.cs.wisc.edu/Condor
Dynamic Slot Partitioning
› Divide slots into chunks sized for matched jobs
› Readvertise remaining resources
› Partitionable resources are cpus, memory, and disk
38
www.cs.wisc.edu/Condor
How It Works
› When match is made…h New sub-slot is created for job and advertisedh Slot is readvertised with remaining resources
› Slot can be partitioned multiple times› Original slot ad never enters Claimed state
h But may eventually have too few resources to be matched
› When claim on sub-slot is released, resources are added back to original slot
39
www.cs.wisc.edu/Condor
Configuration
› Resources still statically partitioned between slots
› SLOT_TYPE_<N>_PARTITIONABLEh Set to True to enable dynamic
partition within indicated slot
40
www.cs.wisc.edu/Condor
New Machine Attributes
› In original slot machine adh PartitionableSlot = True
› In ad for dynamically-created slotsh DynamicSlot = True
› Can reference these in startd policy expressions
41
www.cs.wisc.edu/Condor
Job Submit File
› Jobs can request how much of partitionable resources they needh request_cpus = 3h request_memory = 1024h request_disk = 10240
42
www.cs.wisc.edu/Condor
Dynamic Partitioning Caveats
› Cannot preempt original slot or group of sub-slotsh Potential starvation of jobs with large
resource requirements
› Partitioning happens once per slot each negotiation cycleh Scheduling of large slots may be slow
43
www.cs.wisc.edu/Condor
Concurrency Limits
› Limit job execution based on admin-defined consumable resourcesh E.g. licenses
› Can have many different limits
› Jobs say what resources they need
› Negotiator enforces limits pool-wide
44
www.cs.wisc.edu/Condor
Concurrency Example
› Negotiator config fileh MATLAB_LIMIT = 5h NFS_LIMIT = 20
› Job submit fileh concurrency_limits = matlab,nfs:3
h This requests 1 Matlab token and 3 NFS tokens
45
www.cs.wisc.edu/Condor
New Variable Substitution
› $$(Foo) in submit fileh Existing featureh Attribute Foo from machine ad
substituted
› $$([Memory * 0.9]) in submit fileh New featureh Expression is evaluated and then
substituted
46
www.cs.wisc.edu/Condor
More Info For Preemption
› New attributes for these preemption expressions in the negotiator…h PREEMPTION_REQUIREMENTSh PREEMPTION_RANK
› Used for controlling preemption due to user priorities
47
www.cs.wisc.edu/Condor
Preemption Attributes
› Submitter/RemoteUserPrioh User priority of candidate and running jobs
› Submitter/RemoteUserResourcesInUseh Number of slots in use by user of each job
› Submitter/RemoteGroupResourcesInUseh Number of slots in use by each user’s group
› Submitter/RemoteGroupQuotah Slot quota for each user’s group
48