GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, 24.10.2007 johannes...
-
Upload
nelson-ball -
Category
Documents
-
view
216 -
download
0
description
Transcript of GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga sw week, 24.10.2007 johannes...
GangaTasksGangaTasksAutomating of job Automating of job
submission wit submission wit Athena/AthenaMC in GangaAthena/AthenaMC in Ganga
sw weeksw week, 24.10.2007, 24.10.2007
johannes ebke, johannes ebke, tariq mahmoudtariq mahmoud
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC
outline
I. introduction, concept of tasksII. starting and submitting a taskIII. automating of job submissionIV. job interconnectionV. conclusion
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC I. motivation
aim: automate job submission.job chaining (interconnection)
the user gives few parameters …
and gets RESULTS
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC I. concept
taskfew parameters: task parameters: float, name, number of events etc.
process parameters: datasets, process name, geometry etc.
input/output specifications Athena version
AbstractJobgenerated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, …
AbstractJobAbstractJob
jobjob
job Ganga jobs
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC I. GangaTasks
MCTask
MCJobMCJobMCJob
AnaTask
AnaJob
Task
AnaJobAnaJob
repository/tools.data
TaskList(GangaObject)
GPI.tasks
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC II: starting and submitting a new
task
running numbering on all tasks created by the same user
if the task is 'new', make some settings permanent and set the status to 'running‘
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC III: automating
Task() AnaTask()AnaJob()
II: starting and submitting a new task
auto submitting info resubmitting
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC III: automating
s=AnaTask()s.name="PythiaZmumu"
s.float= 10s.application_option_file=‘$HOME/athena/…./run/AnalysisSkeleton_topOptions.py's.inputdata_dataset= "trig1_misal1_csc11.005145.PythiaZmumu.recon.AOD.v12000601_tid005999"s.requirements_sites=['LRZ', 'FZK', 'LYON‘]s.outputdata_outputdata=[“AnaPythiaZmumu.root"]s.submit()
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC III: info and more
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC III: resubmitting
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC IV: jobs interconnections
Task() MCTask()MCJob()MCJob()MCJob()
II: starting and submitting a new task III:
• auto submitting• info• resubmitting
chaining task steps interconnection,
prerequisites
automating of job submission in Athena/AthenaMC
IV: chaining task stepsmc.total_events = 200;mc.events_per_job={"evgen": 50, "simul": 5, "recon":10}
1 3 5 7 9
1 3 5 7 9
1 3 5 7 9
1 3 5 7 9
1 3 5 972 4 6 108
1 3 5 972 4 6 108
1 3 5 972 4 6 108
1 3 5 972 4 6 108
2 3 41
tariq mahmoud atlas sw week, cern, 24102007
evgen:i-0
simul:i-j
recon:i-k
mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10
AbstractJob.prerequisites()
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC IV: interconnections,
prerequisites (1)
Task.float: does NOT apply for subjobs
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC
AnaTask: optimise sites
CE= ‘lcg-lrz-ce.lrz-muenchen.de:2119/jobmanager-sge-atlas’requirements_sites= ['LRZ', 'FZK', 'LYON']optimize_sites=Falseinputdata_match_ce_all =Trueinputdata_min_num_files=100sites_optimized=[]
tariq mahmoud atlas sw week, cern, 24102007
automating of job submission in Athena/AthenaMC
V. conclusion
GangaTask package: automates job management (small scale production system)works with Ganga Athena and Ganga AthenaMC.wiki in progress: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasks
To do: 1. gather the largest number of files possible of an
in-complete dataset.2. test everything on Lxplus “and elsewhere”.3. soon available in one of the next in Ganga
releases
the user gives
few parameters
… and gets
RESULTS