Enabling e-Research over GridPP

Dan ToveyUniversity of Sheffield

28th March 2006 University of Sheffield2

ATLAS

• Large Hadron Collider (LHC) under construction at CERN in Geneva.

• When commences operation in 2007 will be the world’s highest energy collider.

• Sheffield key member of ATLAS collaboration building one of two General Purpose Detectors on LHC ring.

• Main motivations for building LHC and ATLAS:– Finding the Higgs boson– Finding evidence for

Supersymmetry – believed to be next great discovery / layer in our understanding of the universe.

28th March 2006 University of Sheffield3

ATLAS @ Sheffield

• Sheffield leads Supersymmetry (SUSY) searches at ATLAS

• Also coordinates all ATLAS physics activities in the UK including Higgs and SUSY searches.

• Sheffield responsible for building ATLAS Semiconductor Tracker (SCT) detector, and writing event reconstruction software.

SUSY (= Nobel Prize)SM

NB: This is a simulation!

28th March 2006 University of Sheffield4

ConstructionConstruction

28th March 2006 University of Sheffield5

Event Selection

LEVEL-1 Trigger Hardwired processors (ASIC, FPGA) Pipelined massive parallel

HIGH LEVEL Triggers Farms of

processors

10-9 10-6 10-3 10-0 103

25ns 3µs hour yearms

Reconstruction&ANALYSIS TIER0/1/2

Centers

ON-lineOFF-line

sec

Giga Tera Petabit

9 or

ders

of

mag

nitu

de

28th March 2006 University of Sheffield6

The Data Deluge

Understand/interpret data via numerically intensive simulations• e.g. 1 SUSY event (ATLAS Monte Carlo Simulation) =

20 mins/3.5 MB on 1 GHz PIII

16 Million channels

100 kHzLEVEL-1 TRIGGER

1 MegaByte EVENT DATA

200 GigaByte BUFFERS500 Readout memories

3 Gigacell buffers

500 Gigabit/s

Gigabit/s SERVICE LAN PetaByte ARCHIVE

Energy Tracks

Networks

1 Terabit/s(50000 DATA CHANNELS)

20 TeraIPS

EVENT BUILDER

EVENT FILTER

40 MHzCOLLISION RATE

Charge Time Pattern

Detectors

Grid Computing Service 300 TeraIPS

• Many events– ~109 events/experiment/year– >~1 MB/event raw data– several passes required

Worldwide LHC computing requirement (2007):– 100 Million SPECint2000

(=100,000 of today’s fastest processors)

– 12-14 PetaBytes of data per year (=100,000 of today’s highest capacity HDD).

28th March 2006 University of Sheffield7

LCG

• Aim to use Grid techniques to solve this problem• CERN LHC Computing Grid (LCG) project coordinating activities

in Europe.• Similar projects in US (Grid3/OSG) and Nordic countries

(NorduGrid).• LCG prototype went live in September 2003 in 12 countries

including UK. • Extensively tested by the LHC experiments

28th March 2006 University of Sheffield8

What is GridPP?

• 19 UK Universities, CCLRC (RAL & Daresbury) and CERN

• Funded by the Particle Physics and Astronomy Research Council (PPARC)

• GridPP1 - 2001-2004 £17m "From Web to Grid"

• GridPP2 - 2004-2007 £16m "From Prototype to Production"

• UK contribution to LCG.

28th March 2006 University of Sheffield9

UK Core e-Science

Programme

Institutes

Tier-2 Centres

CERNLCG

EGEE

GridPP

GridPP in Context

Tier-1/A

Middleware, Security,

Networking

Experiments

GridSupportCentre

Not to scale!

Apps Dev

AppsInt

GridPP

28th March 2006 University of Sheffield10

ARDA

Ex

pm

tsEG

EE LCG

Deployment Board

Tier1/Tier2,Testbeds,

Rollout

Servicespecification& provision

User Board

Requirements

ApplicationDevelopment

Userfeedback

Metadata

Workload

Network

Security

Info. Mon.

PMB

CB

Storage

28th March 2006 University of Sheffield11

Tier Structure

Tier-1Tier-1

Tier-0 (CERN)

Tier-1 (Lyon) Tier-1 (BNL)Tier-1 (RAL)

NorthGrid SouthGrid

ScotGrid ULGridTier-2

28th March 2006 University of Sheffield12

UK Tier-1/A Centre Rutherford Appleton

Laboratory• High quality data services• National and international

role• UK focus for international

Grid development1400 CPU80 TB Disk60 TB Tape

(Capacity 1PB)

Grid Resource Discovery Time = 8 Hours

2004 CPU Utilisation

28th March 2006 University of Sheffield13

UK Tier-2 Centres

ScotGridDurham, Edinburgh, Glasgow NorthGridDaresbury, Lancaster, Liverpool,Manchester, Sheffield (WRG)

SouthGridBirmingham, Bristol, Cambridge,Oxford, RAL PPD, Warwick

LondonGridBrunel, Imperial, QMUL, RHUL, UCL

28th March 2006 University of Sheffield14

NorthGrid

• Tier-2 collaboration between Sheffield (WRG), Lancaster, Liverpool, Manchester and Daresbury Lab.

28th March 2006 University of Sheffield15

WRG & NorthGrid

• White Rose Grid contributing to NorthGrid and GridPP with new SRIF2 funded machine at Sheffield (Iceberg).

• LCG component to Iceberg provides a base of 230kSI2k and on demand up to 340kSI2k, with state-of-the-art 2.4 GHz Opteron cpus.

• Delivered 2nd highest GridPP Tier-2 throughput for ATLAS in 2005.

http://lcg.shef.ac.uk/ganglia

28th March 2006 University of Sheffield16

GridPP Deployment Status

Three Grids on Global scale in HEP (similar functionality)

sites CPUs• LCG (GridPP) 228 (19) 17820 (3500)• Grid3 [USA] 29 2800• NorduGrid 30 3200

• GridPP deployment is part of LCG

• Currently the largest Grid in the world

28th March 2006 University of Sheffield17

ATLAS Data Challenges

ATLAS DC2 - LCG - September 71%

2%

0%

1%

2%

14%

3%

1%

3%

9%

8%

3%2%5%1%4%

1%

1%

3%

0%

1%

1%

4%1%

0%

12%

0%

1%

1%

2%

10%

1% 4%

at.uibk

ca.triumf

ca.ualberta

ca.umontreal

ca.utoronto

ch.cern

cz.golias

cz.skurut

de.fzk

es.ifae

es.ific

es.uam

fr.in2p3

it.infn.cnaf

it.infn.lnl

it.infn.mi

it.infn.na

it.infn.na

it.infn.roma

it.infn.to

it.infn.lnf

jp.icepp

nl.nikhef

pl.zeus

ru.msu

tw.sinica

uk.bham

uk.ic

uk.lancs

uk.man

uk.rl

ATLAS DC2 - CPU usage

LCG41%

NorduGrid30%

Grid329%

LCG

NorduGrid

Grid3

Total:

~ 1350 kSI2k.months~ 95000 jobs~ 7.7 Million events fully simulated (Geant4)~ 22 TB

• DC2 (2005): 7.7 M GEANT4 events and 22 TB• DC3/CSC (2006): > 20M G4 events• UK ~20% of LCG

• Ongoing..• (3) Grid

Production• Largest

total computing requirement

• Small fraction of what ATLAS needs..Now in Grid Production Phase

LCG now reliably used for production

28th March 2006 University of Sheffield18

Further Info

http://www.gridpp.ac.uk