1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov...
-
Upload
basil-thornton -
Category
Documents
-
view
220 -
download
0
Transcript of 1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov...
1
RDMS CMS Computing:Russian CMS Tier-2 and Tier-1 for CMS at JINR
llya Gorbunov and Sergei ShmatovJoint Institute for Nuclear Research, Dubna
School on JINR/CERN GRID and Advanced Information Systems
“Joint Institute for Nuclear Research, Dubna
May 14-18, 2012
2
Compact Muon Solenoid goals and performance participation of Russian and Dubna Member State Institutions data taking, triggering and selection data flows
CMS Computing Model data model data flows data processing Tier-1/Tier-2 requirements and resources
RDMS Computing Model RDMS CMS Tier-2s (readiness, job status, data transfers) computing for remote operations (JINR ROC) CMS Tier-1 at Dubna (plans and prospects)
Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Brief Intro:What does physics require
from Computing?
3Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Why we need more accelerators and experiments ?
Being the remarkable event on the long way of searching of the true, Standard Model of interactions is powerless
to answer on very important questions:
• Origin of masses of fundamental blocks (quarks, leptons) • A reason for three generations of matter fermions• A reason of domination matter over antimatter and formation of observable Universe structure• A source of dark mater in the Universe • Possibility of high energy unification of three forces
We hope we will be able to answer on these questions at least partially with next generations of accelerators
And after all, what about gravity?
4Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Open Questions:
Mass: why do fundamental particle have mass and why these masses are such different?
Antimatter: why at the present days matter dominates antimatter?
Dark matter: where is concentrated the hidden mass of Universe?
World geometry: are there extra dimensions? bulk volume geometry?
New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)
QCD matter: what are the properties of QCD at extreme high densities, does QGP exist?
Standard model: validity bounds -> high precision tests of SM !
Open Questions:
Mass: why do fundamental particle have mass and why these masses are such different?
Antimatter: why at the present days matter dominates antimatter?
Dark matter: where is concentrated the hidden mass of Universe?
World geometry: are there extra dimensions? bulk volume geometry?
New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)
QCD matter: what are the properties of QCD at extreme high densities, does QGP exist?
Standard model: validity bounds -> high precision tests of SM !
LHC Physics Perspectives
5Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Compact Muon Solenoid
6Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
7
Compact Muon Solenoid (CMS)+ TOTEM (TOTal Elastic and diffractive cross section Measurement)
A Toroidal LHC ApparatuS (ATLAS)
A Large Ion Collider Experiment (ALICE)
LHC-B
LHC Experiments
LHCf
CMS at LHC
CMS
CERNBuilding 40
14 km15 min
1234
1233
1232
10)(
10)(
10)(
147)(
ccmhighL
ccmlowL
ccmupstartL
TeVppS
112810
/ 62.5)(
ccmL
nucleonTeVPbPbS
8Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Compact Muon Solenoid - CMS
Muon Chambers Tracker Electromagnetic calorimeter, ECAL
Hadron calorimeter HCAL
Forward Calorimeter, HF
Superconducting Coildiameter 6 m, length 13 m Return Yoke
Silicon Tracker
2.7% 0.150.5% E
E EE
PbWO4 ECAL
sampling brass HCAL
1205%E
E E
MUON Chambers
Detector subsystems are designed to measure: the energy and momentum of photons, electrons, muons, jets, missing ET up to a few TeV)%(15~ TeVpp
pT
T
T
GeVp
p
T
T 100@%5.10.1~
Weight 12 500 tDiameter 15.00 mLength 21.6 mMagnetic Field 3.9 Tesla
Weight 12 500 tDiameter 15.00 mLength 21.6 mMagnetic Field 3.9 Tesla 9Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Russian and Dubna Member State CMS Collaboration
CERN
France
Italy
UK
Switzerland
USA
Austria
Finland
Greece
Hungary
Belgium
Poland
Portugal
Spain
Pakistan
Georgia
Armenia
Ukraine
Uzbekistan
Cyprus
Croatia
China, PR
Turkey
Belarus
Estonia
India
Germany
Korea
Russia and JINR Member States
Bulgaria
China (Taiwan)
Iran
Serbia
New-Zealand
Brazil
Ireland
Mexico ColombiaLithuania
RDMS CMS:
2310 authors38 countries175 institutions
the RDMS CMS Collaboration was founded
in Dubna in September 1994
10Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
JINR Participation in CMS ConstructionJINR participates in the CMS in a framework of the RDMS CMS
Collaboration
RDMS bearsFull Responsibility
JINRParticipates
SE
HE
ME1/1
11Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Cross-sections of physics processes at LHC Energy
The fraction of “important” events is ~ 10-8 of total event flow !
The fraction of events from “new physics” is 10-10 - 10-15 (Higgs, SUSY, BSM)
• A very small part of events is a proper for physics analysis• No opportunity to store and to process the full data flow
Triggering and Selection12Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
OnlineOnline:: Trigger System Trigger System
• Multilevel trigger• rejection of background (filtration of events)• reduction of data volume• on-line reduction factor ~107
• Trigger menu• selection of required events
The data flow is reduced from 1000 TB/sec (1TB=1012 Bytes) at input of the first-level trigger down to 100-200 MB/sec on output of the third-level trigger (to record on disks or tapes for further analysis)
Level 1 - Special Hardware
Level 2 - Embedded Processors
40 MHz
40 MHz (1000 TB/sec)
(1000 TB/sec)
Level 3 – Farm of commodity CPUs
75 KHz 75 KHz (75 GB/sec)
(75 GB/sec)5 KHz5 KHz (5 GB/sec)
(5 GB/sec)100 Hz 100 Hz (100 MB/sec)
(100 MB/sec)
Data Recording &
Data Recording &
Offline Analysis
Offline AnalysisCMS event – 1-2 MB L1 trigger – 75 GB/с HLT trigger – 0.1 GB/с total flow ~ 3 PByear
13Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Level-1 Event Storage kHz MByte MByte/s ATLAS 100 1 100 CMS 100 1 100 LHCb 400 0.1 20 ALICE 1 25 1500
LHC Data Flows
~ 3 PB/year
~ 3 PB/year
14Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
The CMS Computing Requests
Event rate to tape: 100 Hz (109 events/year)
Days of running/year: 200
Data written to tape: ~ 3 PetaBytes/Year
Monte Carlo events: 9∙108
CPU (Tier-0 + Tier-1s + Tier-3s): ~ 26 000 kSI2k
Typical networks: 1 - 10 Gbps
Users: ~ 1000 physicists distributed all over the world
1 billion people surfing the Web
105
104
103
102
Level 1 Rate (Hz)
High Level-1 Trigger(1 MHz)
High No. ChannelsHigh Bandwidth(500 Gbit/s)
High Data Archive(PetaByte)
LHCB
KLOE
HERA-B
CDF II
CDFH1
ZEUS
UA1
LEP
NA49
ALICE
Event Size (bytes)
104 105 106
ATLASCMS
106
107
The full processing and analysis can not be performed by the standard tool kits
It is essential to use Distributed Computer Resources
The full processing and analysis can not be performed by the standard tool kits
It is essential to use Distributed Computer Resources
15Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Computing Model
16Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
17
Tier-0 (CERN):• accepts data from the CMS Online Data Acquisition and Trigger System (TriDAS)
• archives RAW data • the first pass of reconstruction and performs Prompt Calibration
• data distribution to Tier-1
Tier-1 (11 centers):• receives a data from the Tier-0 • data processing (re-reconstruction, skimming , calibration etc)• distributes data and MC to the other Tier-1 and Tier-2• secure storage and redistribution for data and MC
Tier-2 (>200 centers):• simulation• user physics analysis
Tier Structure of GRID Distributed Computing:
Tier-0/Tier-1/Tier-2/Tier-3
17Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS GRID Distributed Computing
Tier-0 at CERN
Tier-1 (7 centers): CERN , KIT (Germany), PIC (Spain), CCIN2P3 (France), CNAF (Italy), ASGC (Taipei), RAL (UK), FNAL (US) + two sites under construction at Dubna and India
Tier-2 (85 centers): 7 in Russian and DMS
18Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Data ModelEvent Format Contents Purpose Event Size (MB) DAQ-RAW Detector data from front end electronics + L1 trigger result. Primary record of physics event. Input to
online HLT 1-1.5
RAW Detector data after online formatting, the L1 trigger result, the result of the HLT selections (HLT trigger bits
Input to Tier-0 reconstruction. Primary archive of events at CERN.
1.5
RECO Reconstructed objects (tracks, vertices, jets, electrons, muons, etc.) and reconstructed hits/clusters
Output of Tier-0 reconstruction and subsequent rereconstruction passes..
0.25
AOD Subset of RECO. Reconstructed objects (tracks, vertices, jets, electrons, muons, etc
Physics analysis, limited refitting of tracks and clusters
0.05
TAG Run/event number, high-level physics objects, used to index Rapid identification of events 0.01
FEVT Full Event: Term used to refer to RAW+RECO together multiple 1.75
GEN Generated Monte Carlo event - - SIM Energy depositions of MC particles in detector (sim hits). - -
DIGI Sim hits converted into detector response (the same as the RAW) - 1.5
19Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
~ Pbyte/sec
10 Gbps
~ 100-1500 MBytes/sec
10 Gbps
0.1 - 10 Gbps
2-10 Gbps
CMS Data Flows (I)
20Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Data Flows (II)T0 to T1:
scheduled, time-critical, continuous during data-taking reliable transfer needed for fast aaccess to new data, and to ensure that data is stored safely
T1 to T1: redistributing data, generally after reprocessing (e.g. processing with improved algorithms)
T1 to T2: Data for analysis at Tier-2s
Detector data flow
Monte-Carlo data flow
T2 to T1: data transfer simulation events
T1 to T1/T2: data transfer re-processed simulation events (e.g. processing with improved algorithms)
21Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Data Processing (II)
Calibration Data Base
RAW Data Reconstruction Event Selection
Physics Analysis
Theory/ Monte Carlo Simulation
CMS-PAS-HIG-11-032CMS-PAS-HIG-11-032
Final Physics Results!
22Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Data Processing (II)
23Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Computing Resources
24Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
RDMS CMS Computing Model
25Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
RDMS CMS Grid Sector
CERNTier-0
CMS RDMS Tier-1
RDMS Grid Sector
Russia DMS
Tier-2
JINR
Dubna
Tier-2
ITEP
Moscow
Tier-2
SINP MSU
Moscow
Tier-2
INR
Troitsk
Tier-2
FIAN
Moscow
Tier-2
IHEP
Protivino
Tier-2
PNPI
St.Petersburg
Tier-2
KIPT
Kharkov
Tier-2
Minsk
Tier-2
Erevan
Tier-2
Sofia
Tier-2
Tbilisi
To provide data management and data transfer to RDMS Tier-2 and data processing in the framework of RDMS Physics Tasks the special СMS RDMS Tier-1 was created in CERN
Dubna CMS Tier-1
and
other CMS Tier-1 centres
26Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS T2 requirements
Basic requirements to CMS VO T2 sites for Physics group hosting:• info on contact persons responsible for site operation• site visibility (BDII) and CMS VO support• availability of CMSSW actual version• daily SAM availability > 80%• daily CMS job robot efficiency > 90% • regular file transfer test “OK”• Certified links with CMS T1: 2 up and 4 down• disk space ~ 150-200 TB for: - central space (~30 TB) - analysis space (~60-100 TB) - MC space (~20 TB) - local space (~30-60 TB)
- local CMS users space (~2 TB per user)• CPU resources ~ 3-4 KSI2K per 1 TB disk space, 2GB RAM memory per job
27Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
RDMS CMS Tier-2 Availability
28Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Status of RDMS Site Metrics
29Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Normalized CPU time (in 1K.SI2K.Hours) consumed by the CMS sites September, 2010 – August, 2011)
In total 283,729,684 K.SI2K.Hours
30Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Total Number of CMS Jobs at RDMS CMS sites
1 747 182 jobs
2 253 116 jobs
5.8%
8.7%
31Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Job Summary at RDMS CMS Sites as in the CMS Dashboard
Job Status per Sites
Site Activity Job Efficiency
32Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
PHEDEX Transfer Rate & Transfer Volumes for RDMS CMS sites
33Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
RDMS CMS T2 association
Now Future interest
Analysis Groups
Exotica: T2_RU_JINR Exotica: T2_RU_INR
HI: T2_RU_SINP QCD: T2_RU_PNPI
Electroweak: T2_UA_KIPT Top: T2_RU_SINP
FWD: T2_RU_IHEP
Object/Performance Groups
Muon: T2_RU_JINR e-gamma-ECAL: T2_RU_INR
JetMET-HCAL: T2_RU_ITEP
34Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
JINR Monitoring and Analysis Remote Centre
Monitoring of detector systems Data Monitoring / Express Analysis Shift Operations (except for run
control) Communications of JINR shifter
with personal at CMS Control Room (SX5) and CMS Meyrin centre
Communications between JINR experts and CMS shifters
Coordination of data processing and data management
Training and Information
Monitoring of detector systems Data Monitoring / Express Analysis Shift Operations (except for run
control) Communications of JINR shifter
with personal at CMS Control Room (SX5) and CMS Meyrin centre
Communications between JINR experts and CMS shifters
Coordination of data processing and data management
Training and Information
35Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
RDMS CMS Computing Model:Tier-1 at Dubna
36Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
WLCG Tier1 center in Russia
• Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011).
• The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011)
- for all four experiments ALICE, ATLAS, CMS and LHCb
- ~10% of the summary Tier1 (without CERN) resources
- increase by 30% each year
- draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of
next working LHC session.
• Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011).
• The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011)
- for all four experiments ALICE, ATLAS, CMS and LHCb
- ~10% of the summary Tier1 (without CERN) resources
- increase by 30% each year
- draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of
next working LHC session.
37Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Joint NRC "Kurchatov Institute" – JINR Tier1 Computing Centre
Project: «Creation of the automated system of data processing for experiments at the Large Hadron Collider (LHC) of Tier1 level and maintenance of Grid-services for a distributed analysis of these data»
Terms: 2011-2013
Type of project: R&D
Cost: federal budget - 280 million rubles (~8.5 MCHF), extrabudgetary sources - 50% of the total cost
Leading executor: RRC KI «Kurchatov institute» for ALICE, ATLAS, and LHC-B
Co-executor: LIT JINR (Dubna) for the CMS experiment
Project goal: creation in Russia of a computer-based system for processing experimental data received at the LHC and provision of Grid-services for a subsequent analysis of these data at the distributed centers of the LHC global Grid- system.
Core of the proposal: development and creation of a working prototype of the first-level center for data processing within the LHC experiments with a resource volume not less than 15% of the required one and a full set of grid-services for a subsequent distributed analysis of these data.
38Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
40Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
Tier-1 Services
Security (GSI)Computing Element (CE)Storage Element (SE)Monitoring and AccountingVirtual Organizations (VOMS)Workload management (WMS) Information service (BDII)File transfer service (FTS + PhEDEx)VOBOXSQUID / Frontier ServersCMS user services (Reconstruction Services, Analysis Services etc)
41Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
CMS Tier-1 Requirements
• input flow ~ 7 GB/s
• output flow ~ 3-4 GB/s
• CPU – 2.5 MSI2k
• disks – 1.2 PB
• tapes for archives and secure store – 2.8 PB
• data loss – 10 GB per 1 PB
• access to data ~ 800 MB/s
• I/O computing node bandwidth ~ GB
42Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
JINR Resources
JINR-LCG2Tier2 Sept. 2011 April 2012 Dec. 2012 Dec. 2013
CPU(HEP-SPEC06)
16 000 20 000 30 000 45 000
Disk(Tbytes)CMSATLASALICE
635240240275
995360 360275
1800660 660480
300011001100800
Tape (Tbytes) - - - 5000
JINR-CMSTier1
Dec. 2012Prototype
Sep. 2014Start
CPU(HEP-SPEC06)
4 000 45 000
Disk(Tbytes)CMS
500 8 000
Tape (Tbytes) - 12 000
43Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012
SUMMARY
RDMS Tier-2 sites are fully satisfied to CMS Computing Requirements are involved actively into data processing and analysis (~3.8% of all CMS resources)
New CMS Tier-1 site is under construction in Dubna (in 2012-2013 a prototype need to be designed and commmisined)
CMS Regional Operation Center is founded and operated in JINR for remote monitoring of detector systems and data express-analysis