Tony Doyle [email protected] GridPP – Making the Grid Work for the Science, ATSE e-Science...

Tony [email protected]

GridPP – Making the Grid Work for the Science, ATSE e-Science Visit, Edinburgh, 20 April 2004

http://www.warwick.ac.uk/

Tony Doyle - University of Glasgow

ContentsContents

• Context1. General

(yesterday)

2. Process (today)

3. Operations (tomorrow)

• Start where Steve left off yesterday..

• End up where Andrew begins tomorrow..

– How does the Grid Work?– Performance Indicators– Why was the “failure rate”

~20%?

– Software Process– External dependencies– Managing a distributed

project..– Is GridPP a Grid?

• What is the Grid anyway? (from PP perspective)

– Demo..


How Does theHow Does theGrid Work?Grid Work?

1. Authenticationgrid-proxy-init

2. Job submissioneedg-job-submit

3. Monitoring and controleedg-job-statusedg-job-canceledg-job-get-output

4. Data publication and replication

globus-url-copy, RLS

5. Resource scheduling – use of Mass Storage Systems

JDL, sandboxes, storage elements

0. Web User Interface… or CLI


Job SubmissionJob Submission(behind the scenes)(behind the scenes)

UIJDL

Logging &Logging &Book-keepingBook-keeping

ResourceResourceBrokerBroker

Job SubmissionJob SubmissionServiceService

StorageStorageElementElement

ComputeComputeElementElement

Information Information ServiceService

Job Status

ReplicaReplicaCatalogueCatalogue

DataSets info

Author.&Authen.

Job S

ub

mit

Even

t

Job

Qu

ery

Job

Stat

us

Input “sandbox”

Input “sandbox” + Broker InfoGlobus RSL

Output “sandbox”

Output “sandbox”

Job Status

Pu

blis

h

grid

-pro

xy-in

it

Exp

and

ed J

DL

SE & CE info


How do I Authorize?How do I Authorize?

o=testbed,dc=eu-datagrid, dc=org

CN=Steven Hawking

ou=People

CN=Tony Doyle

mkgridmap

grid-mapfile

VOVODirectoryDirectory

““AuthorizatioAuthorizationn

Directory”Directory”

CN=Homer Simpson

o=xyz,dc=eu-datagrid, dc=org

CN=Steven Hawking

CN=Tony Doyle

Authentication

Certificate

Authentication

Certificate

Authentication

Certificate

ou=People ou=Testbed1

ou=???

local users ban list


UK Certificate Authority andUK Certificate Authority andVirtual Organisation membership Virtual Organisation membership

1. UK e-Science Certificate Authority now used in application testbed

2. PP “users” engaged from many institutes

3. UK participating in 6 ex 9 EDG Virtual Organisations

0 20 40 60 80 100 120

BaBar

Eobs

Iteam

LHCb

Alice

BioMe

CMS

Atlas

WP6

Members

UK Members

1.

2. 3.


Performance indicatorsPerformance indicators(as measured by end users)(as measured by end users)

Number of jobs successfully completed

Total Number of jobs submittedcrudeE

Efficiency (Successful Jobs / Jobs submitted)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

dé

c-0

2

jan

v-0

3

févr

-03

ma

rs-0

3

avr

-03

ma

i-0

3

juin

-03

juil-

03

ao

ût-

03

sep

t-0

3

oct

-03

no

v-0

3

dé

c-0

3

jan

v-0

4

févr

-04

Su

cc

es

sfu

l Jo

bs

/ J

ob

s s

ub

mit

ed

CMS EDGv1.4 Altlas EDGv1.4 LHCb EDGv1.4 LCG1 (EDG v2.0) EDG appl. TB v2.x

Conclusion: prototype performance, but with quality assurance mechanisms built-in


Why was the Why was the “failure rate” “failure rate”

~20%?~20%?• Component Testing e.g. RB Stress Tests (LCG) • RB never crashed• ran without problems at load for several days in a row 20 streams

with 100 jobs each ( typical error rate ~ 2 % still present)• RB stress test in a job storm of 50 streams, 20 jobs each :

– 50% of the streams ran out of connections between UI and RB. (configuration parameter – but machine constraints)

– Remaining 50% streams finished normal (2% error rate)– Time between job-submit and return of the command (acceptance by

the RB) is 3.5 seconds (independent of number of streams) • PROBLEMS ARE END-TO-END:

e.g. Site advertisement communicated via class ads to all sites (inc. e.g. CNAF) results in RB sending application jobs (e.g. AliEn for ALICE) to “black hole” – these are recorded as “failures”

(application corrects for these via re-submission)• OTHER “PROBLEM” IS INCORPORATION OF ADDED

FUNCTIONALITY– ~Resolved by adherence to software process coupled to testbed

structure… improved significantly within LCG (leading to EGEE)

III. Grid Middleware

I. Experiment Layer

II. Application Middleware

IV. Facilities and Fabrics


DataGrid Release MilestonesDataGrid Release Milestones

EU Review (2.1.13)

Evaluations (2.0.12)

• Features (2.1.13) [0.5Mloc]– Reasonable stability, reliability

– VOMS incorporated

– Bug fixes for all services.

• Features (2.0.12)– R-GMA replaced MDS

– Refactored workload mgt.

– Interactive, MPI, chkpt. jobs

– Replica Location Service

– Web Service SE

• Stabilisation time on application testbed typically a few months


Software Process InfrastructureSoftware Process Infrastructure

– Adopt the same set of tools, standards and procedures

– Adopt commonly used open-source or commercial software when easily available

– Avoid “do it yourself solutions”

– Avoid commercial software, since it may give licensing problems

Common services and infrastructure

Tools, templates, training

General QA, tests, integration, release

Similar ways of working (process)

LCG Application Area

POOL, SEAL, PI, SIMU

LCG grid software applications

(LHC experiments, projects, etc)

SPI Infrastructure


SPI Services OverviewSPI Services Overview

Provide General Services needed by each project– CVS repository, Web Site, Software Library– Mailing Lists, Bug Reports, Task Management, Collaborative Facilities

Provide solutions specific to the Software Development phases– Tools, Templates, Policies, Support, Documentations, Examples

Coding Analysis and DesignDevelopment

Release

SpecificationsTesting

Build systemsDeployment and

InstallationDocumentation

Quality Assurance

Software Development

General Services

CVS service

Collaborative Facilities Task Management Mailing Lists

Web Portal External Software


External SoftwareExternal Software

• We install software needed by Particle Physics projects

• Open Source and Public Domain software (libraries and tools) like:– Compilers (icc, ecc)– HEP made packages – Scientific libraries (GSL)– General tools (python)– Test tools (cppunit,

qmtest)– Database software (mysql,

mysql++)– Documentation generators

(lxr, doxygen)– XML parsers (XercesC)

• There are currently 50 different packages, plus others under evaluation. For more than 300 installations

• The LCG projects propose what to install in agreement with LHC needs

• The platforms are decided by the Architect Forum– Linux RedHat 7.3 with the

compilers • gcc 3.2 (rh73_gcc32) • icc 7.1 (rh73_icc71) • ecc 7.1 (rh73_ecc71)

– Windows • Visual Studio .NET 7.1:

(win32_vc7).


Ta

gg

ed

re

lea

se s

ele

cte

d f

or

cert

ifica

tion

Ce

rtifi

ed

re

lea

se s

ele

cte

d f

or

de

plo

yme

nt

Ta

gg

ed

pa

cka

ge

Problem reports

add unittested code to

repository

Run nightly build

& auto. testsGrid certification

Fix problemsApplication Certification

BuildSystem

CertificationTestbed ~40CPU

ApplicationTestbed ~1000CPU

Certified publicrelease

for use by apps.

24x7

Build system

Test Group

WPs

Unit Test Build Certification Production

Users

DevelopmentTestbed ~15CPU

Individual WP tests

IntegrationTeam

Integration

Overall release

tests

Releases candidate

Tagged Releases

Releases candidate

Certified Releases

Apps. Representatives

How Is the process applied? How Is the process applied? Middleware Validation: From Testbed to ProductionMiddleware Validation: From Testbed to Production

Process to:Test frameworks

Test supportTest policiesTest documentationTest platforms/compilers


The UK TestbedThe UK Testbed


e.g. ScotGrid: Glasgow, e.g. ScotGrid: Glasgow, Edinburgh and DurhamEdinburgh and Durham

CE SE

EDG 1.4

ScotGRID

59xWN

• Glasgow farm: WNs on a private network with outbound NAT in place

• 100,000 jobs completed (900,000 CPU hours)

• 34 dual blade servers and 5TB FastT500 being integrated now (next door)

• Shared resources (LHC, CDF and Bioinformatics)

EDG 2.1

Data Management Testbed

CE SE MON

• Edinburgh: 24TB FastT700 and 8-way server: data storage focus

• Durham: 40 node farm

• All being integrated into LCG-2

CDF

LHC BIO


1. 1 2. 1 3. 1 4. 1 5. 1 6. 1 7. 1

1. 2 2. 2 3. 2 4. 2 5. 2 6. 2 7. 2

1. 3 2. 3 3. 3 4. 3 5. 3 6. 3 7. 3

1. 4 2. 4 3. 4 4. 4 5. 4

1. 5 2. 5 3. 5 4. 5

Navigate down

External link 2. 6 3. 6 4. 6 Link to goals

2. 7 3. 7

2. 8 3. 8

System

UK Grid Rollout

Data Challenges

Dissemination

Tier-1 Centre

Tier-A Centre

Applications

CERN DataGrid Applications

LCG Creation WP1

WP2

ResourcesInfrastructure

Presentation Deployment of

related areasGANGA/GaudiATLAS/LHCb

resources

Participation in

of GridPP

Monitoring of

UK e-Science

Computing Fabric

Grid Technology

Tier-2 Centres

CMSMonte Carlo

WP3

WP4

LHCb

Standards

Open SourceImplementation

Grid Deployment

resourcesEngagementof UK groups

Attract newParticle Physics

UK Testbed

resources

WP6

WP7

WP5

SAM Framework

UKQCD

BaBar

Integration

Integration

Applications

QCD Application

Other

Data Analysis

CDF/D0

GridPP Goal

ATLAS

in the UK for the use of the Particle Physics communityTo develop and deploy a large scale science Grid

5Interoperability

International

WP8

6 71 2 3 4

Worldwide

1 . 1 2 . 1 3 . 1 4 . 1 5 . 1 6 . 1 7 . 1

1 . 1 . 1 1 . 1 . 2 1 . 1 . 3 1 . 1 . 4 2 . 1 . 1 2 . 1 . 2 2 . 1 . 3 2 . 1 . 4 3 . 1 . 1 3 . 1 . 2 3 . 1 . 3 3 . 1 . 4 4 . 1 . 1 4 . 1 . 2 4 . 1 . 3 4 . 1 . 4 5 . 1 . 1 5 . 1 . 2 5 . 1 . 3 6 . 1 . 1 6 . 1 . 2 6 . 1 . 3 6 . 1 . 4 7 . 1 . 1 7 . 1 . 2 7 . 1 . 3 7 . 1 . 41 . 1 . 5 2 . 1 . 5 2 . 1 . 6 2 . 1 . 7 2 . 1 . 8 3 . 1 . 5 3 . 1 . 6 3 . 1 . 7 3 . 1 . 8 4 . 1 . 5 4 . 1 . 6 4 . 1 . 7 4 . 1 . 8 6 . 1 . 5

2 . 1 . 9 3 . 1 . 9 3 . 1 . 1 0 4 . 1 . 9

1 . 2 2 . 2 3 . 2 4 . 2 5 . 2 6 . 2 7 . 2 1 . 2 . 1 1 . 2 . 2 1 . 2 . 3 1 . 2 . 4 2 . 2 . 1 2 . 2 . 2 2 . 2 . 3 2 . 2 . 4 3 . 2 . 1 3 . 2 . 2 3 . 2 . 3 3 . 2 . 4 4 . 2 . 1 4 . 2 . 2 4 . 2 . 3 4 . 2 . 4 5 . 2 . 1 5 . 2 . 2 5 . 2 . 3 6 . 2 . 1 6 . 2 . 2 6 . 2 . 3 7 . 2 . 1 7 . 2 . 2 7 . 2 . 31 . 2 . 5 1 . 2 . 6 1 . 2 . 7 1 . 2 . 8 2 . 2 . 5 2 . 2 . 6 2 . 2 . 7 3 . 2 . 5 3 . 2 . 6 3 . 2 . 7 3 . 2 . 8 4 . 2 . 5 4 . 2 . 6 4 . 2 . 71 . 2 . 9 1 . 2 . 1 0 3 . 2 . 9

1 . 3 2 . 3 3 . 3 4 . 3 5 . 3 6 . 3 7 . 3

1 . 3 . 1 1 . 3 . 2 1 . 3 . 3 1 . 3 . 4 2 . 3 . 1 2 . 3 . 2 2 . 3 . 3 2 . 3 . 4 3 . 3 . 1 3 . 3 . 2 3 . 3 . 3 3 . 3 . 4 4 . 3 . 1 4 . 3 . 2 4 . 3 . 3 4 . 3 . 4 5 . 3 . 1 5 . 3 . 2 5 . 3 . 3 6 . 3 . 1 6 . 3 . 2 6 . 3 . 3 6 . 3 . 4 7 . 3 . 1 7 . 3 . 2 7 . 3 . 3 7 . 3 . 41 . 3 . 5 1 . 3 . 6 1 . 3 . 7 1 . 3 . 8 2 . 3 . 5 2 . 3 . 6 2 . 3 . 7 3 . 3 . 5 3 . 3 . 6 4 . 3 . 51 . 3 . 9 1 . 3 . 1 0 1 . 3 . 1 1

1 . 4 2 . 4 3 . 4 4 . 4 5 . 4 1 . 4 . 1 1 . 4 . 2 1 . 4 . 3 1 . 4 . 4 2 . 4 . 1 2 . 4 . 2 2 . 4 . 3 2 . 4 . 4 3 . 4 . 1 3 . 4 . 2 3 . 4 . 3 3 . 4 . 4 4 . 4 . 1 4 . 4 . 2 4 . 4 . 3 4 . 4 . 4 5 . 4 . 1 5 . 4 . 2 5 . 4 . 3 5 . 4 . 41 . 4 . 5 1 . 4 . 6 1 . 4 . 7 1 . 4 . 8 2 . 4 . 5 2 . 4 . 6 2 . 4 . 7 3 . 4 . 5 3 . 4 . 6 3 . 4 . 7 3 . 4 . 8 4 . 4 . 5 4 . 4 . 6 5 . 4 . 51 . 4 . 9 3 . 4 . 9 3 . 4 . 1 0 M e t r i c O K 1 . 1 . 1

M e t r i c n o t O K 1 . 1 . 1 1 . 5 2 . 5 3 . 5 4 . 5 T a s k c o m p l e t e 1 . 1 . 1

1 . 5 . 1 1 . 5 . 2 1 . 5 . 3 1 . 5 . 4 2 . 5 . 1 2 . 5 . 2 2 . 5 . 3 2 . 5 . 4 3 . 5 . 1 3 . 5 . 2 3 . 5 . 3 3 . 5 . 4 4 . 5 . 1 4 . 5 . 2 4 . 5 . 3 4 . 5 . 4 T a s k o v e r d u e 1 . 1 . 11 . 5 . 5 1 . 5 . 6 1 . 5 . 7 1 . 5 . 8 2 . 5 . 5 2 . 5 . 6 2 . 5 . 7 3 . 5 . 5 3 . 5 . 6 3 . 5 . 7 6 0 d a y s 1 . 1 . 11 . 5 . 9 1 . 5 . 1 0 T a s k n o t d u e s o o n 1 . 1 . 1

N o t A c t i v e 1 . 1 . 1 2 . 6 3 . 6 4 . 6 N o T a s k o r m e t r i c

2 . 6 . 1 2 . 6 . 2 2 . 6 . 3 2 . 6 . 4 3 . 6 . 1 3 . 6 . 2 3 . 6 . 3 3 . 6 . 4 4 . 6 . 1 4 . 6 . 2 4 . 6 . 32 . 6 . 5 2 . 6 . 6 2 . 6 . 7 2 . 6 . 8 3 . 6 . 5 3 . 6 . 6 3 . 6 . 7 3 . 6 . 8 N a v i g a t e u p 2 . 6 . 9 3 . 6 . 9 3 . 6 . 1 0 3 . 6 . 1 1 3 . 6 . 1 2 N a v i g a t e d o w n

E x t e r n a l l i n k 2 . 7 3 . 7 L i n k t o g o a l s

2 . 7 . 1 2 . 7 . 2 2 . 7 . 3 2 . 7 . 4 3 . 7 . 1 3 . 7 . 2 3 . 7 . 3 3 . 7 . 42 . 7 . 5 2 . 7 . 6 2 . 7 . 7 2 . 7 . 8 3 . 7 . 5 3 . 7 . 6

2 . 8 3 . 8 2 . 8 . 1 2 . 8 . 2 2 . 8 . 3 2 . 8 . 4 3 . 8 . 1 3 . 8 . 2 3 . 8 . 32 . 8 . 5

W P 8

1 2 3

D e p l o y m e n t

W P 4

W P 5

F a b r i c

T e c h n o l o g y

W P 6

D u e w i t h i n

A T L A S

G r i d P P G o a l

R e s o u r c e sI n t e r o p e r a b i l i t y D i s s e m i n a t i o n

T i e r - 1

T i e r - A

L H C b T i e r - 2

C E R N D a t a G r id A p p l i c a t io n s I n f r a s t r u c t u r e

W P 1

W P 2

W P 3

L C G C r e a t i o n

A p p l i c a t i o n s

W P 7

A T L A S / L H C b

C M S

B a B a r

C D F / D O

U K Q C D

O t h e r

D a t a C h a l l e n g e s

R o l l o u t

T e s t b e d

1 - J a n - 0 4S t a t u s D a t e

I n t . S t a n d a r d s

O p e n S o u r c e

W o r l d w i d e I n t e g r a t i o n

U K I n t e g r a t i o n

M o n i t o r i n g

D e v e l o p i n gE n g a g e m e n t

P a r t i c i p a t i o n

T o d e v e l o p a n d d e p l o y a l a r g e s c a l e s c i e n c e G r i di n t h e U K f o r t h e u s e o f t h e P a r t i c l e P h y s i c s c o m m u n i t y

P r e s e n t a t i o n D e p l o y m e n t

5 6 74

U p d a t e

C l e a r

1 . 1 2 . 1 3 . 1 4 . 1 5 . 1 6 . 1 7 . 1

1 . 1 . 1 1 . 1 . 2 1 . 1 . 3 1 . 1 . 4 2 . 1 . 1 2 . 1 . 2 2 . 1 . 3 2 . 1 . 4 3 . 1 . 1 3 . 1 . 2 3 . 1 . 3 3 . 1 . 4 4 . 1 . 1 4 . 1 . 2 4 . 1 . 3 4 . 1 . 4 5 . 1 . 1 5 . 1 . 2 5 . 1 . 3 6 . 1 . 1 6 . 1 . 2 6 . 1 . 3 6 . 1 . 4 7 . 1 . 1 7 . 1 . 2 7 . 1 . 3 7 . 1 . 41 . 1 . 5 2 . 1 . 5 2 . 1 . 6 2 . 1 . 7 2 . 1 . 8 3 . 1 . 5 3 . 1 . 6 3 . 1 . 7 3 . 1 . 8 4 . 1 . 5 4 . 1 . 6 4 . 1 . 7 4 . 1 . 8 6 . 1 . 5

2 . 1 . 9 3 . 1 . 9 3 . 1 . 1 0 4 . 1 . 9

1 . 2 2 . 2 3 . 2 4 . 2 5 . 2 6 . 2 7 . 2 1 . 2 . 1 1 . 2 . 2 1 . 2 . 3 1 . 2 . 4 2 . 2 . 1 2 . 2 . 2 2 . 2 . 3 2 . 2 . 4 3 . 2 . 1 3 . 2 . 2 3 . 2 . 3 3 . 2 . 4 4 . 2 . 1 4 . 2 . 2 4 . 2 . 3 4 . 2 . 4 5 . 2 . 1 5 . 2 . 2 5 . 2 . 3 6 . 2 . 1 6 . 2 . 2 6 . 2 . 3 7 . 2 . 1 7 . 2 . 2 7 . 2 . 31 . 2 . 5 1 . 2 . 6 1 . 2 . 7 1 . 2 . 8 2 . 2 . 5 2 . 2 . 6 2 . 2 . 7 3 . 2 . 5 3 . 2 . 6 3 . 2 . 7 3 . 2 . 8 4 . 2 . 5 4 . 2 . 6 4 . 2 . 71 . 2 . 9 1 . 2 . 1 0 3 . 2 . 9

1 . 3 2 . 3 3 . 3 4 . 3 5 . 3 6 . 3 7 . 3

1 . 3 . 1 1 . 3 . 2 1 . 3 . 3 1 . 3 . 4 2 . 3 . 1 2 . 3 . 2 2 . 3 . 3 2 . 3 . 4 3 . 3 . 1 3 . 3 . 2 3 . 3 . 3 3 . 3 . 4 4 . 3 . 1 4 . 3 . 2 4 . 3 . 3 4 . 3 . 4 5 . 3 . 1 5 . 3 . 2 5 . 3 . 3 6 . 3 . 1 6 . 3 . 2 6 . 3 . 3 6 . 3 . 4 7 . 3 . 1 7 . 3 . 2 7 . 3 . 3 7 . 3 . 41 . 3 . 5 1 . 3 . 6 1 . 3 . 7 1 . 3 . 8 2 . 3 . 5 2 . 3 . 6 2 . 3 . 7 3 . 3 . 5 3 . 3 . 6 4 . 3 . 51 . 3 . 9 1 . 3 . 1 0 1 . 3 . 1 1

1 . 4 2 . 4 3 . 4 4 . 4 5 . 4 1 . 4 . 1 1 . 4 . 2 1 . 4 . 3 1 . 4 . 4 2 . 4 . 1 2 . 4 . 2 2 . 4 . 3 2 . 4 . 4 3 . 4 . 1 3 . 4 . 2 3 . 4 . 3 3 . 4 . 4 4 . 4 . 1 4 . 4 . 2 4 . 4 . 3 4 . 4 . 4 5 . 4 . 1 5 . 4 . 2 5 . 4 . 3 5 . 4 . 41 . 4 . 5 1 . 4 . 6 1 . 4 . 7 1 . 4 . 8 2 . 4 . 5 2 . 4 . 6 2 . 4 . 7 3 . 4 . 5 3 . 4 . 6 3 . 4 . 7 3 . 4 . 8 4 . 4 . 5 4 . 4 . 6 5 . 4 . 51 . 4 . 9 3 . 4 . 9 3 . 4 . 1 0 M e t r i c O K 1 . 1 . 1

M e t r i c n o t O K 1 . 1 . 1 1 . 5 2 . 5 3 . 5 4 . 5 T a s k c o m p le t e 1 . 1 . 1

1 . 5 . 1 1 . 5 . 2 1 . 5 . 3 1 . 5 . 4 2 . 5 . 1 2 . 5 . 2 2 . 5 . 3 2 . 5 . 4 3 . 5 . 1 3 . 5 . 2 3 . 5 . 3 3 . 5 . 4 4 . 5 . 1 4 . 5 . 2 4 . 5 . 3 4 . 5 . 4 T a s k o v e r d u e 1 . 1 . 11 . 5 . 5 1 . 5 . 6 1 . 5 . 7 1 . 5 . 8 2 . 5 . 5 2 . 5 . 6 2 . 5 . 7 3 . 5 . 5 3 . 5 . 6 3 . 5 . 7 6 0 d a y s 1 . 1 . 11 . 5 . 9 1 . 5 . 1 0 T a s k n o t d u e s o o n 1 . 1 . 1

N o t A c t i v e 1 . 1 . 1 2 . 6 3 . 6 4 . 6 N o T a s k o r m e t r i c

2 . 6 . 1 2 . 6 . 2 2 . 6 . 3 2 . 6 . 4 3 . 6 . 1 3 . 6 . 2 3 . 6 . 3 3 . 6 . 4 4 . 6 . 1 4 . 6 . 2 4 . 6 . 32 . 6 . 5 2 . 6 . 6 2 . 6 . 7 2 . 6 . 8 3 . 6 . 5 3 . 6 . 6 3 . 6 . 7 3 . 6 . 8 N a v ig a t e u p 2 . 6 . 9 3 . 6 . 9 3 . 6 . 1 0 3 . 6 . 1 1 3 . 6 . 1 2 N a v ig a t e d o w n

E x t e r n a l l i n k 2 . 7 3 . 7 L in k t o g o a l s

2 . 7 . 1 2 . 7 . 2 2 . 7 . 3 2 . 7 . 4 3 . 7 . 1 3 . 7 . 2 3 . 7 . 3 3 . 7 . 42 . 7 . 5 2 . 7 . 6 2 . 7 . 7 2 . 7 . 8 3 . 7 . 5 3 . 7 . 6

2 . 8 3 . 8 2 . 8 . 1 2 . 8 . 2 2 . 8 . 3 2 . 8 . 4 3 . 8 . 1 3 . 8 . 2 3 . 8 . 32 . 8 . 5

W P 8

1 2 3

D e p l o y m e n t

W P 4

W P 5

F a b r i c

T e c h n o l o g y

W P 6

D u e w i t h i n

A T L A S

G r i d P P G o a l

R e s o u r c e sI n t e r o p e r a b i l i t y D i s s e m i n a t i o n

T i e r - 1

T i e r - A

L H C b T i e r - 2

C E R N D a t a G r id A p p l i c a t io n s I n f r a s t r u c t u r e

W P 1

W P 2

W P 3

L C G C r e a t i o n

A p p l i c a t i o n s

W P 7

A T L A S / L H C b

C M S

B a B a r

C D F / D O

U K Q C D

O t h e r

D a t a C h a l l e n g e s

R o l l o u t

T e s t b e d

1 - J a n - 0 4S t a t u s D a t e

I n t . S t a n d a r d s

O p e n S o u r c e

W o r l d w i d e I n t e g r a t i o n

U K I n t e g r a t i o n

M o n i t o r i n g

D e v e l o p i n gE n g a g e m e n t

P a r t i c i p a t i o n

T o d e v e l o p a n d d e p l o y a l a r g e s c a l e s c i e n c e G r i di n t h e U K f o r t h e u s e o f t h e P a r t i c l e P h y s i c s c o m m u n i t y

P r e s e n t a t i o n D e p l o y m e n t

5 6 74

U p d a t e

C l e a r

Managing a Distributed Project: Managing a Distributed Project: GridPP1 Project Status?GridPP1 Project Status?

Metric OK

Metric not OK

Tasks Complete

Tasks Overdue

Tasks due in next 60 days

Tasks Deleted

ChangeForms Enacted

Tasks not due

43 1 145 2 1 3 17 42 76% of the 190 GridPP1 tasks have been successfully completed


What is “The Grid”What is “The Grid” Is GridPP a Grid?Is GridPP a Grid?Anyway?Anyway?

1. Coordinates resources that are not subject to centralized control

2. … using standard, open, general-purpose protocols and interfaces

3. … to deliver nontrivial qualities of service

1. YES. This is why development and maintenance of a UK-EU-US testbed is important

2. YES... Globus/CondorG/EDG meet this requirement. Common experiment application layers are also important here.

3. NO(T YET)… Experiments define whether this is true - currently only ~100,000 jobs submitted via the testbed c.f. internal component tests of up 10,000 jobs per day. Next step: LCG-2 deployment outcome… this year

http://www-fp.mcs.anl.gov/~foster/Articles/WhatIsTheGrid.pdf


What is The Grid Anyway?

From Particle Physics PerspectiveThe Grid is:

not hype, but surrounded by it

a working prototype running on testbed(s)…

about seamless discovery of PC resources around the world

using evolving standards for interoperation

the basis for particle physics computing in the 21st Century

not (yet) as transparent as end-users want it to be


The Grid: DemonstrationsThe Grid: Demonstrationshttp://www.gridpp.ac.uk/demos/http://www.gridpp.ac.uk/demos/

Demos used to establish that e.g. the two LHC multi-purpose detector collaborations

• can run jobs on an International Grid

• Use common Grid infrastructure with secure Grid access

• But doesn’t mean that the Grid works in production mode

• (yet)

• This is however

significant

Tony Doyle [email protected] GridPP – Making the Grid Work for the Science, ATSE e-Science...

Documents

Transcript of Tony Doyle [email protected] GridPP – Making the Grid Work for the Science, ATSE e-Science...