LHCb Software Week 12-14 November 2003 Gennady Kuznetsov Production Manager Tools (New Architecture)
-
Upload
millicent-ward -
Category
Documents
-
view
212 -
download
0
Transcript of LHCb Software Week 12-14 November 2003 Gennady Kuznetsov Production Manager Tools (New Architecture)
LHCb Software Week12-14 November 2003 Gennady Kuznetsov
Production Manager Tools (New Architecture)
Data Challenges 2001
Gennady Kuznetsov
History: Controlling Monte-Carlo simulation.
In 2001, control system was based on a single Java Servlet which in fact was a single file with Java code. The sequence of execution was fixed and all site dependencies had to be programmed inside. Code had to be redistributed for each location with the Servlet runner.
Increased complexity of the tasks and demands for flexibility (requirement to run more then one type of job) required architectural changes.
As a result, the “Data Management” group produced a document with a new flexible architecture.
The Grand Design
Gennady Kuznetsov
Job Configuration, Data Production, Bookkeeping - Design Document, M. Frank (Editor), 04/12/2001
Data Challenges 2003
Gennady Kuznetsov
Agent
Agent
Agent
Agents interpret XML file with Workflow description
Advantages:This very flexible architecture gives us freedom to adjust any parameters in the workflow and also allows us to manipulate Steps known to the “Agent”.Disadvantages:• Agent had to exist on the site or had to be submitted.• For any change in execution, we had to update the agents everywhere.
Components - New System
Module
Module
Module
Module
Module
Module
Module
Step
Step
Step StepStep
Step Step
Step
WorkflowJob
Job Job Job
Job Job
JobJob
Job Job Job
Job Job
Job
Job
Job Job Job
Job Job
Job
ProductionLevels of usage:1. Module – Programmer2. Step – Production
manager3. Workflow –
User/Production manager
We add Module as additional
component of the architecture
Each step generates job as a Python
program.
This module structure allow Production Manager to construct any algorithm as a combination of modules.
Gennady Kuznetsov
Architecture - New System
Presenter Name:
Variable
VariableCollection
ModuleDefinition ModuleInstance
n
1
StepDefinition
n
1
n
StepInstance
WorkflowDefinition1
1
n n
1
Module Editor
Step Editor
Workflow Editor
edit
edit
edit
Module/Step exampleExecution Sequence
Gennady Kuznetsov
DistributionSoftware checkand installation
Example of the Workflow 370000-Sv260r2-Bv17r4
BrunelReconstruction
GaussSimulation
Mon. ServiceMonitoring
Mon. ServiceMonitoring
Data quality “rawh” validation
File ServiceFile transfer
Data quality “oodst” validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality “rawh” validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality “rawh” validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality “rawh” validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
Bookkeeping. ServiceBookkeeping
Another three lines (not enough space)
Module
Module/Step exampleData propagation
Gennady Kuznetsov
DistributionSoftware checkand installation
Example of the Workflow 370000-Sv260r2-Bv17r4
GaussSimulation
Mon. ServiceMonitoring
Data quality “rawh” validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
Module
Module Editor
Python code of single module. Can
be many classes.Module
variables.
Description
Module Name Stored as XML file
Gennady Kuznetsov
Step Editor
Step Name
Description
Definitions of Modules
Instances of Modules
Variables of currently selected instance
Selected instance
Stored as XML file, where all modules are embedded
Gennady Kuznetsov
Step variables.
Workflow Editor
Gennady Kuznetsov
Workflow Name
Description
Step Definitions
Step Instances
Variables of currently
selected Step Instance
Selected Step Instance
Workflow Variables.
Stored as XML file
Job Splitting
Gennady Kuznetsov
Step
Step StepStep
Step Step
Step
Workflow DefinitionJob
Job Job Job
Job Job
JobJob
Job Job Job
Job Job
Job
Job
Job Job Job
Job Job
Job
Production
Python List
The input value for the job splitting is a Python list object. Every single (top level) element of this list applies to the Workflow Definition and propagates through the code and generates single element of the production (one or several jobs).
Current Status
Gennady Kuznetsov
Done:• All three editors (Module, Step, Workflow) ready.• XML schema seems stable.
To do for the Data Challenge 2004:• Define variable types.• Extend Python code generator to cover Workflow.• Create appropriate module library.• Implement “list input” for the Workflow Editor (relates to the job splitting).
Future development:• Add graphical editing panel to Step and Workflow editors.• Finish embedded Python editor with highlighting feature.