Distributed Computing for CEPC

YAN Tian

On Behalf of Distributed Computing Group, CC, IHEP

for 4th CEPC Collaboration Meeting, Sep. 12-13, 2014

Draft Version 2014.9.3

Outline

Introduction Experience of BES-DIRAC Distributed Computing

Computing model Computing resources list Official MC production Data transfer system

Distributed Computing for CEPC – A test bed established – Sharing resources with BES – User job workflow – Physical validation – To do

INTRODUCTION Part I

About Distributed Computing

• Distributed computing plays an import role in discovery of Higgs – “Without the LHC Computing Grid, the discovery could not have occurred”

---- Foster

• Many HEP experiments have employed distributed computing to integrate resources contribued by collaboration members. – such as LHCb, Belle 2, CTA, ILC, BES 3, etc..

• Large HEP experiments need plenty of computing resources, which may not be afforded by only one institution or university.

DIRAC: a Interware • DIRAC (Distributed Infrastructure with Remote Agent Control) is a

interware for grid computing.

• It’s powfull, flexible and widely used as central component of grid solution.

• More info: • DIRAC Homepage: http://diracgrid.org/

• DIRAC Project @ Github: https://github.com/DIRACGrid/DIRAC

DIRAC Users: LHCb, Belle, CTA, ILC, etc…

ILC: ~ 3,000 CPU Cores

LHCb: ~ 40,000 CPU Cores

Belle2: ~ 12,000 CPU Cores

CTA: ~ 5,000 CPU Cores

EXPERIENCE OF BES-DIRAC DISTRIBUTED COMPUTING

Part II

BES-DIRAC: Computing Model Detector IHEP Data

Center DIRAC

Central SE

Remote Site

Raw dst &

ramdomtrg

Raw data

Remote Site Remote Site

MC dst

Remote Users Remote Users Remote Users IHEP Users

All dst

BES-DIRAC: Computing Resources List # Contributors CE Type CPU Cores SE Type SE Capacity Status

1 IHEP Cluster + Cloud 144 dCache 214 TB Active

2 Univ. of CAS Cluster 152 Active

3 USTC Cluster 200 ~ 1280 dCache 24 TB Active

4 Peking Univ. Cluster 100 Active

5 Wuhan Univ. Cluster 100 ~ 300 StoRM 39 TB Active

6 Univ. of Minnesota Cluster 768 BeStMan 50 TB Active

7 JINR gLite + Cloud 100 ~ 200 dCache 8 TB Active

8 INFN & Torino Univ. gLite + Cloud 264 20 TB Active

9 CERN Cloud 20 Active

10 Soochow Univ. Cloud 20 Active

Total 1868 ~ 3248 355 TB

11 Shandong Univ. Cluster 100 Preparing

12 BUAA Cluster 256 Preparing

13 SJTU Cluster 192 144 TB Preparing

Total 548 144 TB

BES-DIRAC: Official MC Production # Time Task BOSS Ver. Total Events Jobs Data Output

1 2013.9 J/psi inclusive (round 05) 6.6.4 900.0 M 32,533 5.679 TB

2 2013.11~2014.01 Psi3770 (round 03,04) 6.6.4.p01 1352.3 M 69,904 9.611 TB

Total 2253.3 M 102,437 15.290 TB

job running @ 2nd batch of 2nd production Physical Validation Check of 1st production

keep run ~1350 jobs for one week 2nd batch: Dec.7~15

BES-DIRAC: Simulation+Reconstruction

• Simulation + reconstruction jobs are supported.

• Randomtrg data has been distributed to remoted sites with SE.

• Job download randomtrg data from local SE, or directly read from SE mounted to nodes.

BES-DIRAC: Data Trasfer System • Data transfered from March to July 2014, total 85.9 TB

Data Source SE Destination SE Peak Speed Average Speed

randomtrg r04 USTC, WHU UMN 96 MB/S 76.6 MB/s (6.6 TB/day)

randomtrg r07 IHEP USTC, WHU 191 MB/s 115.9 MB/s (10.0 TB/day)

Data Type Data Data Size Source SE Destination SE

DST xyz 24.5 TB IHEP USTC

psippscan 2.5 TB IHEP UMN

Random trigger data

round 02 1.9 TB IHEP USTC, WHU, UMN, JINR

round 03 2.8 TB IHEP USTC, WHU, UMN

round 04 3.1 TB IHEP USTC, WHU, UMN

round 05 3.6 TB IHEP USTC, WHU, UMN

round 06 4.4 TB IHEP USTC, WHU, UMN, JINR

round 07 5.2 TB IHEP USTC, WHU

• high quality ( > 99% one-time success rate) • high transfer speed ( ~ 1 Gbps to USTC, WHU, UMN; 300Mbps to JINR):

IHEPUSTC, WHU

@ 10.0 TB/day

USTC, WHUUMN

@ 6.6 TB/day

one-time

success > 99%

DISTRIBUTED COMPUTING FOR CEPC

part III

A Test Bed Established

BES-DIRAC

Servers

Job flow

*.stdhep input data

*.slcio output data

BUAA Site

OS: SL 5.8

Remote WHU Site

OS: SL 6.4

Remote

IHEP PBS Site

OS: SL 5.5 IHEP-OpenStack Site

IHEP Lustre

WHU SE

IHEP Local Resources

IHEP DB

DB mirror

CVMFS

Server

Sharing Resources with BES

• Which resources can be shared? – Central DIRAC Servers & Mantainers. (hope CEPC coll. can contribute manpower)

– Computing & Storage resources contributed by sites who wish to support both BES and CEPC, such as IHEP, WHU, BUAA, Soochow Univ., etc …

• Multi-VO (Virutal Organization) support technology is under development – It’s a grid framework for managing resources for multi collaborations.

– VOMS server has been configured, tested, now is ready to use.

– multi-VO workload management system is under testing.

– StoRM SE with multi-VO support is under developing.

User Job Workflow

Submit a User Job Step by Step: (1) upload input data to SE

(2) prepare a JDL file: job.jdl

(3) prepare job.sh

(4) submit job to DIRAC

(5) monitoring job status in web portal

(6) Download output data to Lustre

Physical Validation Check

• Under going…

• will be finished before Sep.10

To Do List

• Add and test new sites;

• Deploy remote mirror MySQL database;

• Development a frontend module for massive job splitting, submission, monitoring & data management;

• Refine multi-VO suport to manage BES&CEPC sharing resources;

Thanks

• Thank you for your attention!

• Q & A

• Further questions and cooperations, please contact ZHANG Xiaomei (zhangxm@ihep.ac.cn) and YAN Tian (yant@ihep.ac.cn)