1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science...
-
Upload
kory-french -
Category
Documents
-
view
217 -
download
0
Transcript of 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science...
1
Condor BirdBathSOAP Interface to Condor
Charaka GoonatilakeDepartment of Computer Science
University College [email protected]
2
Background
3
Motivations Adding Web Service interfaces to Condor daemons
Facilitate the development of third-party applications capable of interacting with Condor (remotely).– E.g. build higher-level application specific scheduler that submits jobs to
multiple Condor pools based on application semantics
– These can be built using a wide range of languages/SOAP packages
– BirdBath has been tested on:• Java (Apache Axis, XSUL)• Python (ZSI)• C# (.Net)• C/C++ (gSOAP)
Condor accessible from platforms where its command-line tools are not supported/installed
4
The (Hypothetical) Big Picture
Schedd Scheddflocking
Schedd
SOAP
Globus/gridSAMCondor-G
Client
Site A Site B Site C
Site D
SOAP
5
Condor SOAP Interfaces
Central manager
Execution machine
Submission machine(s)
CollectorSchedd
Execution machine
Execution machine
Execution machine
Begin/commit/abort transaction Submit Job Obtain Job ClassAds Get/Send file Removed/hold/release job
Obtain resource ClassAds Obtain all daemon ClassAds
Shadow
Negotiator
Now available in Condor 6.7.5
6
Clients for BirdBath
WSDLs available that describe each daemon interface
SOAP libraries (stub code) can be generated automatically from WSDLs
Custom Client
holdJob()
SOAP library
Submission machine(s)
Schedd
7
Example: Query Collector
Obtain information (ClassAds) about all resources that have over 512MB RAM
collectorLocator = new CondorCollectorLocator();
collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”);
classAds = collector.queryStartdAds(“Memory>512”);
8
Example: Job Submission
1. Create New Transaction2. Create New JobId3. Send Files ({input files, binaries})4. Create Job Description5. Submit Job6. Commit Transaction
Transaction based process
Uses two phase commit to promote reliability and robustness
Files transferred in chunks – Base64 encoded SOAP attachments
DAGMan jobs supported
9
OMII Collaboration: gridSAM plugin Job Submission & Monitoring Web Service that submits to range of
resource managers (e.g. Condor, SGE, Globus) Uses JSDL (Job Submission Description Language) for
standardised job description
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
SOAP
JSDLJSDL
<?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://…"> <JobDescription> <Application> <Executable>/bin/echo</Executable> <Argument>hello world</Argument> </Application> </JobDescription></JobDefinition>
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
SOAP
JSDLJSDL
10
gridSAM plugin Existing Condor plugin
uses shell-based submission via command-line tools and monitoring of log files
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
Condor-WSCondor-WS Our plugin accesses Condor via the SOAP interfaces
Submissionmachine(s)
Schedd
Middle-tier
JSDLJSDLSOAP
SOAPClassAdClassAd
Building JSDL to ClassAd conversion libraries
11
Future Work Bundle plug-in into gridSAM
– Also provide separate JSDL to ClassAd conversion library
Extensions to other daemons (what functionality to expose?)– For example, allow checkpoint data to be retrieved through the shadow
in order to migrate jobs across pools
Roll out daemons on e-minerals mini-grid