BNL Role in ATLAS Computing · PDF fileBNL Role in ATLAS Computing DOE Annual HEP Program...

BNL Role in ATLAS ComputingBNL Role in ATLAS Computing

DOE Annual HEP Program ReviewDOE Annual HEP Program Review

S. RajagopalanS. Rajagopalan

April 22, 2003April 22, 2003

DOE HEP Program Review, 22 April 2003DOE HEP Program Review, 22 April 2003 22

ATLAS ComputingATLAS Computingvv SoftwareSoftware§§ Core SoftwareCore Software§§ Reconstruction & SimulationReconstruction & Simulation

vv AnalysisAnalysis§§ Physics capabilities at the LHCPhysics capabilities at the LHC§§ Preparation for AnalysisPreparation for Analysis

vv Facility supportFacility support§§ Local (U.S.) software supportLocal (U.S.) software support§§ BNL Tier 1 CenterBNL Tier 1 Center§§ Production (Simulation & Reconstruction)Production (Simulation & Reconstruction)§§ Develop & deploy Grid Tools & ServicesDevelop & deploy Grid Tools & Services


BackgroundBackgroundvv PrePre--1999 era:1999 era:§§ Geant3 based simulationGeant3 based simulation§§ Fortran based reconstruction softwareFortran based reconstruction software§§ Physics Technical Design Report published in 1999Physics Technical Design Report published in 1999

qq Detector & Physics PerformanceDetector & Physics Performance

vv PostPost--1999 era:1999 era:§§ Work on Work on Geant Geant 4 based simulation4 based simulation§§ Object Oriented Reconstruction softwareObject Oriented Reconstruction software

qq Core Services, subCore Services, sub--system software & combined reconstructionsystem software & combined reconstructionqq Adopted the Adopted the LHCb LHCb framework as a starting pointframework as a starting pointqq The The LAr LAr Reconstruction was the first client of the new framework.Reconstruction was the first client of the new framework.

– Considerable influence in core design aspects§§ Data Challenges + Physics studies with the new softwareData Challenges + Physics studies with the new software

qq Data Challenge 1 in progressData Challenge 1 in progress


ATLAS Core Software BreakdownATLAS Core Software Breakdownvv FrameworkFrameworküü Event Data ModelEvent Data Modelüü Data ManagementData Managementvv Detector DescriptionDetector Descriptionvv GraphicsGraphicsvv Analysis ToolsAnalysis Toolsvv Grid IntegrationGrid Integration

EDM Coordinator: EDM Coordinator: S. RajagopalanS. RajagopalanRaw Data Model Coordinator: Raw Data Model Coordinator: H. MaH. MaLCG Applications Area Coordinator: LCG Applications Area Coordinator: T. T. WenausWenaus

ArchitectureArchitecture--Team (ATeam (A--Team): Team): S. Rajagopalan, T. S. Rajagopalan, T. WenausWenaus

Areas where BNL playsAreas where BNL playsa significant rolea significant role


Event Data ModelEvent Data ModelvvMajor contributions to EDM infrastructure from:Major contributions to EDM infrastructure from:§§ H. Ma, S. RajagopalanH. Ma, S. Rajagopalan

vv EDM Infrastructure (EDM Infrastructure (StoreGateStoreGate) provides tools for:) provides tools for:

§§ Memory management for Event Data ObjectsMemory management for Event Data Objects

§§ OnOn--demand access to objects from persistencydemand access to objects from persistency

§§ Persistable Persistable navigation between objects (Track navigation between objects (Track àà Hits)Hits)

§§ History (how objects are created)History (how objects are created)

§§ Efficient access to data in regions of interestEfficient access to data in regions of interest

vv ATLAS has ATLAS has baselined StoreGate baselined StoreGate as its choice of the EDM as its choice of the EDM infrastructure. infrastructure.

§§ High Level Trigger (HLT) will also use High Level Trigger (HLT) will also use StoreGateStoreGate..


Event Data Model Infrastructure Event Data Model Infrastructure ((StoreGateStoreGate))

Hit ReconstructionHit ReconstructionAlgorithmAlgorithm

Tracking AlgorithmTracking Algorithm

Analysis Analysis AlgorithmsAlgorithms

DDD

DD

DDD

D

HHH

HH T TT

“The In-Memory Event”

PersistencyPersistency

navigationnavigation


Event Data Model (2)Event Data Model (2)vv (H. Ma, S. Rajagopalan)(H. Ma, S. Rajagopalan)

vv BNL contribution to EDM software for physicists:BNL contribution to EDM software for physicists:§§ Design of the overall ATLAS Raw Data FlowDesign of the overall ATLAS Raw Data Flow

qq Implementation of the Calorimeter aspectsImplementation of the Calorimeter aspectsqq Emulation of Emulation of ByteStream ByteStream data flow and efficient unpacking and data flow and efficient unpacking and

access in regions of interestaccess in regions of interest–– Used in High Level TriggerUsed in High Level Trigger

qq Coordination of the Raw Data Flow across subCoordination of the Raw Data Flow across sub--sytemssytems

§§ Overall coordination of the ATLAS EDM effortOverall coordination of the ATLAS EDM effortqq Design & implementation of Calorimeter EDM Design & implementation of Calorimeter EDM qq Several aspects of the EDM for combined reconstructionSeveral aspects of the EDM for combined reconstruction


Persistency SupportPersistency Supportvv The baseline persistent technology for ATLAS changed The baseline persistent technology for ATLAS changed

from Objectivity to ROOT in 2001. from Objectivity to ROOT in 2001.

§§ All LHC experiments have a common baseline.All LHC experiments have a common baseline.

§§ BNL has significant expertise in this area (leveraging efforts fBNL has significant expertise in this area (leveraging efforts from rom RHIC)RHIC)

§§ We immediately provided an interim ROOT based persistency We immediately provided an interim ROOT based persistency mechanism for ATLAS that is being widely used in several applicamechanism for ATLAS that is being widely used in several applications.tions.

§§ Long term efforts are closely integrated with the LCG (LHC CompuLong term efforts are closely integrated with the LCG (LHC Computing ting & Grid project) to provide a common persistency solution.& Grid project) to provide a common persistency solution.


The LCG ProjectThe LCG Projectvv To help the LHC experiments prepare, build and operate the To help the LHC experiments prepare, build and operate the

computing environment needed to manage and analyze multicomputing environment needed to manage and analyze multi--PB PB scale data coming from each detector.scale data coming from each detector.

vv Main working body of the LCG is the Project Execution Board Main working body of the LCG is the Project Execution Board (PEB) chaired by Les Robertson. (PEB) chaired by Les Robertson.

vv PEB has four areas, each with a project manager:PEB has four areas, each with a project manager:

üüApplicationsApplications (T. (T. WenausWenaus))

Grid TechnologyGrid Technology

FabricsFabrics

Grid DeploymentGrid Deployment


LCG Applications AreaLCG Applications Areavv Identifies and provides solutions for common projects:Identifies and provides solutions for common projects:§§ Long term advantages in providing resources, support and maintenLong term advantages in providing resources, support and maintenanceance

vv Identified Common Projects are:Identified Common Projects are:

üü Persistency Framework Project (POOL)Persistency Framework Project (POOL)

§§ Physicist Interface Project (PI)Physicist Interface Project (PI)qq Interfaces and tools with which physicists will use the softwareInterfaces and tools with which physicists will use the software

§§ Core Libraries and Services Project (SEAL)Core Libraries and Services Project (SEAL)qq Core libraries, Object Dictionary, Scripting Services etc.Core libraries, Object Dictionary, Scripting Services etc.

§§ Software Process & Infrastructure Project (SPI)Software Process & Infrastructure Project (SPI)qq Provides basic environment and tools for software developmentProvides basic environment and tools for software development

§§ Simulation ProjectSimulation Projectqq Support for Geant3, Geant4, Detector DescriptionSupport for Geant3, Geant4, Detector Description


Persistency Support (2)Persistency Support (2)vv BNL current efforts are to integrate the ATLAS software BNL current efforts are to integrate the ATLAS software

with POOL to provide persistency support.with POOL to provide persistency support.§§ Its longer term include direct contributions to the POOL effort Its longer term include direct contributions to the POOL effort in the in the

areas of Persistency Support and Event Collectionsareas of Persistency Support and Event Collections§§ D. Adams, V. Fine, H. Ma, V. D. Adams, V. Fine, H. Ma, V. PerevotchikovPerevotchikov

vv BNL is also providing BNL is also providing MySQL MySQL based persistency for based persistency for conditions data. conditions data. (S. (S. KanadasamyKanadasamy, H. Ma, A. , H. Ma, A. UndrusUndrus) ) §§ Currently in use by ATLAS (especially by Calorimeter)Currently in use by ATLAS (especially by Calorimeter)§§ While we may switch to a new conditions database, the technologyWhile we may switch to a new conditions database, the technology

independent interface to the algorithms will be stable. independent interface to the algorithms will be stable.

vv Hong Ma: Overall Liquid Argon database coordinatorHong Ma: Overall Liquid Argon database coordinator§§ Oversees the needs of production, installation, online, offline Oversees the needs of production, installation, online, offline and and

testbeam testbeam efforts in the efforts in the LArLAr community.community.


MagdaMagdavv Distributed Data Manager for cataloging and data replicationDistributed Data Manager for cataloging and data replication§§ Developed by Developed by Wengsheng Wengsheng Deng (BNL).Deng (BNL).

vv Heavily used in ATLAS Data Challenges (DC0 & DC1)Heavily used in ATLAS Data Challenges (DC0 & DC1)§§ Catalog of ATLAS data at Alberta, CERN, Lyon, INFN (CNAF, Catalog of ATLAS data at Alberta, CERN, Lyon, INFN (CNAF,

Milan), FZK, IFIC, IHEP.Milan), FZK, IFIC, IHEP.susu..itepitep..ruru, , NorduGridNorduGrid, RAL and many US , RAL and many US institutions.institutions.§§ 288K files in catalog with total size of 77.5 TB as of 2003288K files in catalog with total size of 77.5 TB as of 2003--0404--1515§§ Main component in US Main component in US testbed testbed productionproduction§§ Data Replication task has transferred 10 TB between BNL HPSS andData Replication task has transferred 10 TB between BNL HPSS and

CERN CastorCERN Castor

vv Tested in EDG Tested in EDG testbed testbed and demonstrated to be useful.and demonstrated to be useful.

vv It will be implemented as a file catalog back end to the LCG It will be implemented as a file catalog back end to the LCG POOL persistency framework.POOL persistency framework.

Simulation, Reconstruction & AnalysisSimulation, Reconstruction & Analysis


Geant3 SimulationGeant3 Simulation

vv P. P. Nevski Nevski serves as the ATLAS Geant3 simulation coordinator.serves as the ATLAS Geant3 simulation coordinator.

§§ Data Challenge 0 & 1 utilize Geant3 based simulation.Data Challenge 0 & 1 utilize Geant3 based simulation.§§ Recent work geared toward implementing updated geometry.Recent work geared toward implementing updated geometry.

§§ Geant4 based simulation expected to be used for DC2 (April 2004)Geant4 based simulation expected to be used for DC2 (April 2004)qq Expect continued maintenance and support of Geant3 for ~ 2 yearsExpect continued maintenance and support of Geant3 for ~ 2 years..

– Until G4 has been validated.


Liquid Argon Software OrganizationLiquid Argon Software Organization

ØØ Current efforts Current efforts focussed focussed on delivering the needed software for Data on delivering the needed software for Data Challenge 2 & Combined Challenge 2 & Combined Testbeam Testbeam run: Spring 2004run: Spring 2004

CoordinatorCoordinatorS. RajagopalanS. Rajagopalan

Detector DescriptionDetector Description

ReconstructionReconstruction

SimulationSimulation

Database (H. Ma)Database (H. Ma)

Calibration/AlignmentCalibration/Alignment

TestbeamTestbeam

egammaegamma

Jet/Jet/MissingETMissingET

Trigger


Liquid Argon ReconstructionLiquid Argon Reconstructionvv Significant contributions in the Liquid Argon subSignificant contributions in the Liquid Argon sub--system:system:

§§ (H. Ma, F. Paige, S. Rajagopalan, K. Yip)(H. Ma, F. Paige, S. Rajagopalan, K. Yip)

§§ Responsible for deployment of the full sequence of steps, develoResponsible for deployment of the full sequence of steps, developing ping algorithms and associated EDM:algorithms and associated EDM:

qq Digitization & ROD EmulationDigitization & ROD Emulation

qq Cell, Tower and Cluster ReconstructionCell, Tower and Cluster Reconstruction

– Implementation and coordination of calibration/corrections

qq Combined Reconstruction to identify Combined Reconstruction to identify eeγγ, , ττ and Jet candidatesand Jet candidates

vv Used our success in influencing the overall architectural designUsed our success in influencing the overall architectural designand the design of reconstruction software in other suband the design of reconstruction software in other sub--systems.systems.


Other Reconstruction ActivitiesOther Reconstruction ActivitiesvvMuon Muon Reconstruction Reconstruction (K. (K. AssamaganAssamagan, Y. , Y. FisyakFisyak, D. Adams), D. Adams)

§§ EDM and Data Converters for EDM and Data Converters for Muon Muon ReconstructionReconstruction

vv Combined Reconstruction Activities: Combined Reconstruction Activities: (H. Ma, F. Paige, S. Rajagopalan)(H. Ma, F. Paige, S. Rajagopalan)

§§ ee--gamma Reconstruction gamma Reconstruction qq Develop algorithms to identify Develop algorithms to identify egamma egamma candidatescandidates

§§ Tau Tau reconstruction reconstruction §§ Jet Reconstruction and Jet Reconstruction and Hadronic Hadronic CalibrationCalibration§§ Missing EMissing ETT ReconstructionReconstruction

vv Physics capabilities in the SUSY sector Physics capabilities in the SUSY sector (F. Paige)(F. Paige)

§§ F. Paige serves as the coF. Paige serves as the co--coordinator of the ATLAS SUSY Groupcoordinator of the ATLAS SUSY Group

§§ GMSB & GMSB & mSUGRA mSUGRA models have been exploredmodels have been explored§§ Easy to discover if it exists, challenge is to understand underlEasy to discover if it exists, challenge is to understand underlying modelying model§§ Analysis being redone with full simulation + new softwareAnalysis being redone with full simulation + new software


Jet CalibrationJet Calibrationvv ATLAS calorimeters nonATLAS calorimeters non--compensating.compensating.§§ Hadronic Hadronic showers less dense than EM : so weight cells moreshowers less dense than EM : so weight cells more§§ H1 calibration (at cell level) produces improved linearity, jet H1 calibration (at cell level) produces improved linearity, jet and and

MissingETMissingET resolution compared to standard sampling calibrationresolution compared to standard sampling calibration


Tau Tau ReconstructionReconstructionvv Important ingredient in SUSY models.Important ingredient in SUSY models.§§ Must rely on Must rely on hadronic hadronic decays; hence large QCD decays; hence large QCD backgroudsbackgrouds..

Select narrow jets using many shapeSelect narrow jets using many shapevariables in likelihood function.variables in likelihood function.

A cut on this likelihood function A cut on this likelihood function + imposing track requirements:+ imposing track requirements:Have achieved S/B ~ 3 at 35% Have achieved S/B ~ 3 at 35% efficiency for Pefficiency for PTT > 35 > 35 GeVGeV

Data corresponds to ~ 2 Data corresponds to ~ 2 fbfb--11

for a specific for a specific mSugra mSugra pointpoint

opposite sign

same sign

Software Support, Facilities, Production & Software Support, Facilities, Production & GridGrid


Software SupportSoftware Supportvv A full time librarian (A. A full time librarian (A. UndrusUndrus) to manage the ATLAS and ) to manage the ATLAS and

associated software locally at BNL for U.S.associated software locally at BNL for U.S.§§ Evaluation of release tools and their subsequent deploymentEvaluation of release tools and their subsequent deployment

vv Alex is a member of the ATLAS software infrastructure team Alex is a member of the ATLAS software infrastructure team (SIT) representing U.S. ATLAS.(SIT) representing U.S. ATLAS.

vv BNL developed the nightlyBNL developed the nightly--build system that is now deployed build system that is now deployed at CERN.at CERN.§§ Central tool in day to day work for software developersCentral tool in day to day work for software developers§§ Testing procedures developed and integrated in automated buildsTesting procedures developed and integrated in automated builds§§ System likely to be deployed in the LCG Application area as wellSystem likely to be deployed in the LCG Application area as well..


LHC Computing Facilities ModelLHC Computing Facilities Model


Tier 1 CenterTier 1 Centervv BNL has been selected as the U.S. Tier 1 FacilityBNL has been selected as the U.S. Tier 1 Facility§§ B. B. Gibbard Gibbard (manager), R. Baker (deputy) + staff(manager), R. Baker (deputy) + staff§§ Currently operational at ~ 1% of the required 2008 capacityCurrently operational at ~ 1% of the required 2008 capacity§§ Total capacity ~20% of the Tier0 center at CERNTotal capacity ~20% of the Tier0 center at CERN§§ Five Tier 2 centers elsewhere in U.S., each with 20% of Tier1 caFive Tier 2 centers elsewhere in U.S., each with 20% of Tier1 capacitypacity

Tape Based 3 Center StandaloneModel Disk Model Disk Model

CPU (kSPECint95) 209 329 500Disk (TBytes) 365 483 1000Tape (PBytes) 2 2 2

Disk (GBytes/sec) 10 20 20Tape (MBytes/sec) 1000 200 200

WAN (Mbit/sec) 4610 9115 91151/3+1/6 of ESD on disk Add other 2/3 of ESD

ESD pass each month ESD pass each day


E450

Dual Intel

Dual Intel

RCFHPSSCOREServer

RCFBackupServer

USATLASHPSSServer

US ATLAS Regional Center (Tier 1) at BNL

62 Intel/LinuxDual 700/450 MHz256/512 MBytes

9/18 GBytes100 Mbit Ethernet(3,200 SPECint95)Processing Farm

9940BTapeDrive

NFS

AFSServers

AFS~150 GBytes

RAID Disk

US ATLAS Equipment

RCF Infrastructure

Web & EmailServers

November 2002

RCF TapeSilo

SSH & GridGateways

LANSwitch

NIS/DNS/DTSServers

...

...

Gigabit Connectionsto Data Servers &WAN

RCFSwitch/Firewall

WAN .at OC12 .

...

Dual Intel

Dual Intel

...

6 Intel/LinuxDual 1 GHz1 GBytes

forSftwr Devel &US ATLAS Lib

IntelInfrastructure

Servers

Cache

E480(NFS Server)11.4 TBytes

FibreChannelRAID Disk

NFS

... SANSwitch


ATLAS Data ChallengesATLAS Data Challengesvv A series of Data Challenges with increasing complexity is A series of Data Challenges with increasing complexity is

planned in ATLAS.planned in ATLAS.vv Goal is to help us prepare by exercising the computing Goal is to help us prepare by exercising the computing

infrastructure, the complete software and study the physicsinfrastructure, the complete software and study the physics

DC0: DC0: ~10~1055 events, events, December 2002December 2002DC1: DC1: ~10~1077 events, events, January January –– June 2003June 2003DC2: DC2: ~10~1088 events, events, Spring 2004Spring 2004DC3:DC3: ~5 x 10~5 x 1088 events, events, 20052005DC4: DC4: ~10~1099 events, events, 20062006

vv BNL has played a vital role in the DC0 & DC1 productionBNL has played a vital role in the DC0 & DC1 production§§ P. P. Nevski Nevski (simulation), Y. (simulation), Y. Fisyak Fisyak (Reconstruction) + …(Reconstruction) + …


vv BNL serves as one of the handful of primary data repositories.BNL serves as one of the handful of primary data repositories.§§ Consequently a primary analysis siteConsequently a primary analysis site

vv DC1 Phase 2 underway, which includes:DC1 Phase 2 underway, which includes:§§ PilePile--up production, Reconstruction productionup production, Reconstruction production

Installed CPU Capacity

1%2%

2%2%

4%

6%

7%

8%

8%

9%

10%

11%

12%

16%

USUSCERNCERN

Tier 1 UtilizationTier 1 Utilization

DC1 Phase 1 Contribution

0.0%

1.4%

2.4%

1.9%

9.6%

2.2%

28.6%

14.2%

10.9%

1.4%

3.2%

4.4%

10.7%

4.1%

USUS

CERNCERN

As of Feb. 2003


Grid Tools & ServicesGrid Tools & Servicesvv This effort is largely a PPDG funded project:This effort is largely a PPDG funded project:

§§ MAGDA: (W. Deng)MAGDA: (W. Deng)qq Distributed manager for cataloging and data replication (W. DengDistributed manager for cataloging and data replication (W. Deng))

§§ DIAL: (D. Adams)DIAL: (D. Adams)qq Distributed Interactive Analysis of Large datasets (D. Adams)Distributed Interactive Analysis of Large datasets (D. Adams)

§§ GUMS: (T. GUMS: (T. WlodekWlodek, D. Yu), D. Yu)qq Grid User Management SystemGrid User Management System

– Virtual Organization user management– Site authentication & authorization software

§§ Monitoring: (J. Smith, D. Yu)Monitoring: (J. Smith, D. Yu)qq Local Cluster Monitoring integrated into Grid MiddlewareLocal Cluster Monitoring integrated into Grid Middleware

§§ Grid oversight and liaison activities (R. Baker)Grid oversight and liaison activities (R. Baker)


ConclusionConclusionvv BNL is significantly contributing in several areas of ATLAS BNL is significantly contributing in several areas of ATLAS

computing:computing:§§ Major responsibilities both in ATLAS and U.S. ATLAS ComputingMajor responsibilities both in ATLAS and U.S. ATLAS Computing§§ As a Tier 1 Center, BNL is the main node in U.S. Grid As a Tier 1 Center, BNL is the main node in U.S. Grid testbedtestbed

vv BNL is developing significant expertise in many areas of BNL is developing significant expertise in many areas of software:software:§§ In turn, this expertise allows us to rapidly start looking at phIn turn, this expertise allows us to rapidly start looking at physicsysics§§ And play an important coordination role in U.S. ATLASAnd play an important coordination role in U.S. ATLAS

vv Our main problem: We are understaffed.Our main problem: We are understaffed.§§ Lost Kin Yip, who was making significant contributionsLost Kin Yip, who was making significant contributions§§ Significant cuts in project funding as wellSignificant cuts in project funding as well

BNL Role in ATLAS Computing · PDF fileBNL Role in ATLAS Computing DOE Annual HEP Program...

Documents

Transcript of BNL Role in ATLAS Computing · PDF fileBNL Role in ATLAS Computing DOE Annual HEP Program...