Job Options forAlgorithms and Tools

JOE v2.0

Job Options forAlgorithms and Tools

JOE v2.0

GANGA Software Workshop - 20-22 April 2004 - CERNChun Lik Tan (Alvin) - Birmingham

clat@hep.ph.bham.ac.uk

AgendaAgenda

Format of extracted information

Persistency of job options

Format of extracted informationFormat of extracted information

Background:

Plex - This is the Python lexical analysis engine behind JOE’s job options parser.http://www.cosc.canterbury.ac.nz/~greg/python/Plex/

JOE is capable of parsing both text-based and Python script-based job options files.

Format of extracted informationFormat of extracted information

Parser produces tokens which are recognised as algorithms, tools, services, properties, values, comments or included files.

The overall structure is essentially a multi-depth Python dictionary. The keys used are the objects of the tokens extracted.

Format of extracted informationFormat of extracted information

A job options file object (i.e. internal JOE representation) has three constituent member data structures:

the job option ‘tree’ dictionary, the included file objects list, and

the comment objects list.

Format of extracted informationFormat of extracted information

Structure of job options ‘tree’ dictionary:{

Tool1:{Property1:{Property2:{’_value’:Value}}}, Tool2:{Property1:{’_value’:Value}}, Algorithm:{Property:{’_value’:Value}}, Service:{Property:{’_value’:Value}},}

The latter two member data structures are straight forward lists.

Persistency of job optionsPersistency of job options

JOE adopts a very pragmatic approach to job option persistency.... Save it in the same format it is capable of reading i.e. in text or Python.

Other formats e.g. Python pickle objects, XML representation can and will be supported when the need arises.

Persistency of job optionsPersistency of job options

To diff or not to diff...

Saving only changes imply having the original job options file at hand.

A particular job options file from different LHCb/ATLAS releases may/will differ slightly. Changes may be meaningless when applied to newer files.

Position sensitive changes?