Stephen Burke - WP8 Status - 9/5/2002

PartnerLogo

WP8 StatusStephen Burke, PPARC/RAL

Stephen Burke - WP8 Status - 09/05/2002 - 2/11

What is WP8?

HEP applications workpackage

Link between experiment software and EDG middleware

Works with WP9 (Earth Observation) and WP10 (Biomedical)

Applications Technical Working Group (ATWG) is a forum to define application requirements for the EDG middleware: “Loose Cannons” + experiment reps + WP9 + WP10

Stephen Burke - WP8 Status - 09/05/2002 - 3/11

Testing the Testbed

LCs have continued to be active in testing the functionality of the testbed

Automated test software under development, with publication of the results on the web

Ongoing feedback to middleware developers

Some application tests, but still limited because the job broker at CERN is unstable

Stability is our highest priority!

Stephen Burke - WP8 Status - 09/05/2002 - 4/11

Current Problems

Job submission works, but has been unstable, with jobs failing for various reasons much of the time

It isn’t yet fully defined how to set up an application environment (where to find compilers, experiment software etc) at each site, but this is improving

There are work-arounds for the initial problems with data storage, but the solutions are awkward and insecure

Documentation is improving but still not perfect!

Stephen Burke - WP8 Status - 09/05/2002 - 5/11

Near Future

Release 1.1.5 is still not stable, development has now stopped and applications will wait for 1.2

Move to 1.2 in ~ 2 weeks subject to testing

Should have new functionality, particularly from WP2 (Replica Manager, new GDMP) which should make data access easier

Should also resolve problems with access to Castor

Hope to have a stable “production” testbed soon

Hope to see a rollout to more sites soon

Stephen Burke - WP8 Status - 09/05/2002 - 6/11

Common Use Cases

ATWG is trying to define common needs of all applications, with the long-term aim of having a common software layer between the experiment software and the Grid middleware

Discussions held with the four LHC experiments to clarify their approaches and extract common themes

Various kinds of information: use cases, implementation/architecture ideas, requirements, scenarios, …

Stephen Burke - WP8 Status - 09/05/2002 - 7/11

Common Use Cases (2)

Preliminary “Common Use Case” document produced

Some overlap with LCG, RTAG will produce an updated use case document, work will continue in WP8

Working with computer scientists at Annecy to produce UML diagrams

Stephen Burke - WP8 Status - 09/05/2002 - 8/11

UML Diagram

Experiment-wide database selection

Output files Storage options preferences (SE, MSS, closest...)

Define execution criteria (CE, priority ...)

Submit Physic Appli

login

PRODUCTION: Simulation

else

If actor is proxy certified

Get LFNs for database access

Allocate output LFNs

Write submission job (JDL?) -Submit Job to Grid

VO metadata data description catalog

VO Job submission bookkeeping service

VO metadata configuration Catalog

Job resource match

VO replica catalog

Record job parameter

Allocate Job Id

Optimize CE choice /VO

Submit job to CE

Submit Job to Working Node

Prepare exec environment -associate PFN-LFN

Execute Physic Appli

Manage Output Files & update File catalog LFN-PFN

Record execution info

File management & PFN selection

Record job parameter (JDL, input, ...)

Register/update attributes (LFN)

Management of job-related information

Display available resources/JDL

Job execution accounting service

POSIX call -Open (LFN) Read/Write Close or grid wrapper to POSIX calls

VO Database access

Grid access via API

Application is never recompiled or rel inked to run on Grid - Access to data is done via standard POSIX calls (???????)

Register/Update attributes (LFN) in VO metadata Catalog

Publish job-related information

ex: automatic file replication or fi le transfer & fi le catalog update

PHYSIC APPLICATIONGRIDEXPERIMENT SPECIFIC MODULESPRODUCTION TEAM

Stephen Burke - WP8 Status - 09/05/2002 - 9/11

Administrivia

WP8 plenary on 14th June at CERN (3 per year)

Application Technical Working Group (ATWG) has been meeting once a month, in future will alternate WP8-only and WP8+9+10 meetings to separate HEP-specific issues from general ones

Architecture Task Force has been re-activated, Jeff Templon (NIKHEF) is the WP8 representative

Stephen Burke - WP8 Status - 09/05/2002 - 10/11

WP8 Technical Requirements

1. Realistic Large-Scale Tests Reliability! Need reliable

dg-job-* command suite

2. Data management Reliability! Need reliable

gdmp-* command suite, file-transfer commands

3. Mass Storage Support Working access to MSS

(CASTOR and HPSS at CERN, Lyon)

4. Lightweight User Interface Put on a laptop or std.

Desktop machine

5. Portability Demonstrable portability of

middleware: a) use other resources, b) debugging

6. Scratch Space Job requests X amount of

scratch space to be available during execution, system tells job where it is

7. Output File Support JDL support for output files:

specify where output should go in JDL, not in job script