May 2004Sverre Jarp1 Preparing the computing solutions for the Large Hadron Collider (LHC) at CERN...

May 2004 Sverre Jarp 1

Preparing the computing solutions

for the Large Hadron Collider (LHC) at CERN

Sverre Jarp, openlab CTOIT Department, CERN


Short overview of CERN


150m depth150m depth150m depth150m depth

Accelerators and detectors in underground tunnels and caverns


CERN in numbers Financed by 20 European countries

Special contributions also from other countries:

USA, Canada, China, Japan, Russia, etc.

1000 CHF (650 M€) budget to coveroperation + new accelerators

2,200 staff (and diminishing)

6,000 users (researchers) from all over the world

broad visitor and fellowship program


CERN User Community

Europe: 267 institutes, 4603 usersElsewhere: 208 institutes, 1632 users


Computing at CERN

May 2004 Sverre Jarp

reconstruction

simulation

analysis

interactivephysicsanalysis

batchphysicsanalysis

batchphysicsanalysis

detector

event summary data

rawdata

eventreprocessing

eventreprocessing

eventsimulation

eventsimulation

analysis objects(extracted by physics topic)

event filter(selection &

reconstruction)

event filter(selection &

reconstruction)

processeddata

Data Management and Computing for Physics Analysis


High Energy Physics Computing Characteristics

Independent events (collisions of particles) trivial (read: pleasant) parallel processing

Bulk of the data is read-only versions rather than updates

Meta-data in databases linking to “flat” files Compute power measured in SPECint (not SPECfp)

But good floating-point is important Very large aggregate requirements:

computation, data, input/output Chaotic workload –

research environment - physics extracted by iterative analysis, collaborating groups of physicists

Unpredictable unlimited demand


SHIFT architecture (Scalable Heterogeneous Integrated Facility)

Tapeserver

Diskserver

Batchserver

Interactiveserver

BatchanddiskSMP

Network- Ethernet

AFS

In 2001 SHIFT won the 21st Century Achievement Award issued by Computerworld

DiskserverDiskserver

BatchserverBatchserver

TapeserverTapeserver


CERN’s Computing Environment (today)

High-throughput computing (based on reliable “commodity” technology)

More than 1500 (dual Xeon processor) PCs with Red Hat Linux

About 3 Petabytes of data (on disk and tapes)


IDE Disk servers Cost-effective disk storage:

< 10 CHF/GB (mirrored)


The LHC Challenge


Large Hadron Collider A completely new particle

accelerator The largest superconductor installation in the

world Same tunnel as before; 27 km of magnets Super-fluid Helium cooled to 1.9°K Two counter-circulating proton beams in a

field of 8.4 Tesla

Collision energy: 7 + 7 TeV

Simulation tool:Sixtrack


CMSATLAS

LHCb

Accumulating data at 10 Petabytes/year (plus replicated copies)

Requirements:

Storage – Raw recording rate: up to 1 GB/s per experiment

2 Petabytes of disk/year

Total HEP processing needs – 50,000 (100,000) of today’s fastest processors

The Large Hadron Collider (LHC) has 4 Detectors:


The Large Hadron Collider (LHC) goal:

All charged tracks with pt > 2 GeV

Reconstructed tracks with pt > 25 GeV

(+30 minimum bias events)

Find new

physics,

such as the

Higgs

particle, and

get the

Nobel price !

selectivity: 1 in 1013 - 1 person in a thousand world populations


LHC Computing Grid Project

Goal of the project:To prepare, deploy and operate the computing environment for the experiments to analyse the data from the LHC detectors

Phase 1 – 2002-05development of common applications, libraries, frameworks, prototyping of the environment, operation of a pilot computing service

Phase 2 – 2006-08acquire, build and operate the LHC computing service

The Grid is

just a

tool to

wards achieving this goal


RAL

IN2P3

BNL

FZK

CNAF

PIC ICEPP

FNALComputing Model (simplified!!)

Tier-0 – the accelerator centre

Filter raw data Reconstruction summary data (ESD) Record raw data and ESD Distribute raw and ESD to Tier-1

Tier-1: Permanent storage and management of

raw, ESD, calibration data, meta-data, analysis data and databases grid-enabled data service

Data-heavy analysis Re-processing raw ESD National, regional support “online” to the data acquisition

process high availability, long-term

commitment managed mass storage

Tier-2: Well-managed disk storage –

grid-enabled Simulation End-user analysis – batch and

interactive High performance parallel

analysis (PROOF)

USCNIKHEFKrakow

CIEMAT

Rome

Taipei

TRIUMF

CSCS

Legnaro

UB

IFCA

IC

MSU

Prague

Budapest

Cambridge

Tier-1small

centres

Tier-2

desktopsportables


RAL

IN2P3

BNL

FZK

CNAF

USC

PIC ICEPP

FNAL

NIKHEFKrakow

Taipei

CIEMAT

TRIUMF

Rome

CSCS

Legnaro

UB

IFCA

IC

MSU

Prague

Budapest

Cambridge

Processing M SI2000**

Disk PetaBytes

Mass Storage

PetaBytes

CERN 20 5 20

Major data handling centres

(Tier 1)45 20 18

Other large centres (Tier 2)

40 12 5

Totals 105 37 43

** Current fast processor ~1K SI2000

Current estimates of Computing Resources needed at Major LHC Centres

First full year of data - 2008Data distribution~70 Gbits/sec


LCG Basics Getting the data from the detector to the grid

requires sustained data collection and distribution keeping up with the accelerator

To achieve the required levels of performance, reliability, resilience

at minimal cost (people, equipment) we also have to work on scalability and performance

of some of the basic computing technologies: cluster management mass storage management high performance networking

Workshop 3

Workshop 1

Workshop 2

Workshop 5


SW Rep

Fabric Automation at CERN

Node

CfgCache

SWCache

SPMA

SWRepCDB

CDBOraMon

NCMMSA

SMS

HMS

LEMON

LEAF

ConfigurationInstallation

Fault & hardware

Management

Monitoring

Includes technology developed by DataGrid


WAN connectivity

Itanium-2single stream:

5.44 Gbps1.1 TB in 30 mins

We now have to get from an R&D project (DATATAG) to a sustained, reliable service – GEANT, ESNET, ..

Microsoft enters the stage:

Multiple streams: 6.25 Gbps

20 April 2004


Preparing for 2007

The LCG installation is on a tight schedule

Due to the need for deployment and development in parallel

2004

2005

2006

2007

first data

Initial service in operation

Decisions on final core middleware

Demonstrate core data handling and batch analysis

Installation and commissioning


CERN openlab


openlab: The technology focus of CERN/IT

Industrial Collaboration:

Enterasys, HP, IBM, and Intel and Oracle are our partners

Voltaire (with Infiniband switches) just joined

Technology aimed at the LHC era: Network switches at 10 Gigabits ~ 100 rack-mounted HP servers 64-bit computing: Itanium-2

processors StorageTank storage system

w/28 TB ~1 GB/s throughput


64-bit porting status

Ported: Castor (data management subsystem)

GPL. Certified by authors. ROOT (C++ data analysis framework)

Own license. Binaries both via gcc and ecc. Certified by authors.

CLHEP (class library for HEP) GPL. Certified by maintainers.

GEANT4 (C++ Detector simulation toolkit) Own license. Certified by authors.

CERNLIB (all of CERN’s FORTRAN software) GPL. In test.

Zebra memory banks are I*4

ALIROOT (entire ALICE software framework) LCG-2 software from VDT/EDG

GPL-like license.

Being ported: CMS ORCA (part of CMS framework)


CERN“Where the Web was born…”®

CERN is busily preparing for the first arrival of LHC data in 2007 New and exciting technologies are

needed to manage the data Seamlessly, around the globe Together with our partners (EU, industry, other

Physics Labs, other sciences) we expect to come up with interesting proofs-of-concept and technological spin-off !

High Throughput Computing is “on the move” !

People Motivation

Technology Science

Innovation

May 2004Sverre Jarp1 Preparing the computing solutions for the Large Hadron Collider (LHC) at CERN...

Documents

Transcript of May 2004Sverre Jarp1 Preparing the computing solutions for the Large Hadron Collider (LHC) at CERN...