HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK.

HEP Data Grid in Japan

Takashi Sasaki

Computing Research Center

28/07/2004Takashi.Sasaki@kek.jp

Contents

• Japanese HENP projects

• Belle GRID

• ATLAS-Japan Regional Center

• Collaboration in Asia Pacific

Major HENP projects in Japan

• KEK Proton Synchrotron • K2K etc.

• KEKB– BELLE experiment

• 300 members from 54 institutes in10 countries • Super-B is planning

• J-PARC– New facility under construction at Tokai (~2008) – particle physics, material science and life science

• T2K

• International collaboration – LHC ATLAS, CERN– HERA ZEUS, DESY– RHIC, BNL– etc.

KEK-B Belle Experiment

Belle Data

• 15MB/sec data rate in maximum

• 200TB/year raw data recorded

• Equivalent number of Monte Carlo events

Belle PC farms

Super KEK-B• L = 10^35cm^-2s^-1 in 2007?• Data rate ~ 250MB/s

K2K now upgrading to T2K

T2K (Tokai to Kamioka)• J-PARC

– for particles physics, material and life science – joint project of JAERI and KEK– 100 times intense neutrino beam

• High trigger rate at the near detector– Operational in 2007

LHC-ATLAS

• Japanese contribution– Semiconductor Tracker– Endcap Muon Trigger System– Muon Readout Electronics– Superconducting Solenoid– DAQ system– Regional Analysis Facility

• ICEPP will the site of the regional center

HEP GRID related activities

• Networking – domestic/international

• BELLE– data distribution and remote MC production

• ALTAS Japan– collaboration between ICEPPand KEK

• Hadron therapy simulation – application of HEP tools into medical field

SuperSInetMajor HEP sites have a DWDM connectionto KEK with 1Gbps

Chicago

To Novosibirsk

5 G -> 10G

ASNET/TANET2 622 M

APII 1G

SuperSINET

TransPAC

CERNET155 M

Tsukuba WAN

Taiwan

Russia

SuperSINET

Current Network Connections around Japan

As of July 2004

SINET2MThailand

Hawaii 155M

BELLE GRID

• Distributed analysis and data sharing among institutes – Experimental data distribution to remote sites– Monte Carlo production at remote sites and send

back events to KEK– remote job submission – etc

• Testing SRB with GSI now– among Australia, Taiwan and KEK

• hopefully Korea, soon

Why SRB?• Easy to install, use and manage

– Most of Belle’s collaborating institutes are middle or small size universities

• poor in man power • Useful features

– file replication– parallel data transfer – Grid aware: GSI authentication– Command line interface, API, GUI – Fancy Web interface– etc

• Excellent user support– quick response – seminars

• Available today!– Solution is needed for Belle

SRB(Storage Resource Broker)

SRBserver

Internet

disktape DBnfsMCATRAID

SRBclient

SRB zone federation

SRBserver

Internet

disktape DBnfsMCATRAID

disk DB MCAT

SRBserver

SRB Command line interface

SRB test bed system

HPSS120TB

ZONE:glsrb01

Fire Wall

ZONE:anusf

store Internet

federation

MCATPostgreSQL

gtdmz01

ZONE:gtsrb13

ZONE:kekgt15

KEKAustralia

gl03gl01

MCATDB2

MCATPostgreSQL

gt15 RAID800GB

bcs20 NFS(Belle data)

BELLE Computer systemclient

BELLESecure net

Status

• Zone federation between ANU and KEK has been established – ANU, University of Melbourne and University of Sydn

ey are collaborating for BELLE/ATLAS Grid issue • One MCAT running at ANU

– Data can store/retrieve on Belle data system and also HPSS in KEK side

• Zone federation between Academia Sinica, Taiwan and KEK has been established – still need to solve some problems

BELLE GRID Future plan

• Participation of Korean sites• Mutual job submission using Globus• LCG2+SRB (if they wish)

– LCG2 is under testing at KEK also with help of ICEPP.

– Because many foreign institutes are working both for BELLE and one of LHC experiments, they want to use LCG rather than vanilla Globus

• baby “tier-0” at KEK and “tier-1” at ICEPP

– SRB and LCG-RLS synchronization will be tried based on Simon Matson’s (Bristol, CMS) work

Grid in ATLAS Japan

• Regional analysis center for ATLAS– ICEPP, the University of Tokyo

• Joint collaboration between ICEPP and KEK for Grid deployment

LCG MW Deployment in Japan

• RC Pilot Model System@ICEPP– Since 2002– LCG testbed. Now LCG2_1_1

• LCG2 test bed@KEK– baby tier-0– LCG2_1_1

• Regional Center Facility– Will be introduced in JFY2006– Aiming “Tier1” size resource

SuperSInet Performance Measurement“A” settingTCP 479Mbps -P 1 -t 1200 -w 128KBTCP 925Mbps -P 2 -t 1200 -w 128KBTCP 931Mbps -P 4 -t 1200 -w 128KBUDP 953Mbps -b 1000MB -t 1200 -w 128KB

"A" setting: 104.9MB/s "B" setting: 110.2MB/s

“B” setting (longer window size)TCP 922Mbps -P 1 -t 1200 -w 4096KBUDP 954Mbps -b 1000MB -t 1200 -w 4096KB

100Mbps

ICEPP KEK

100 CPUs6CPUs

HPSS 120TB

GRID testbed environmentwith HPSS through GbE-WAN

NorduGrid - grid-manager - gridftp-serverGlobus-mdsGlobus-replicaPBS server

NorduGrid - grid-manager - gridftp-serverGlobus-mdsPBS server

PBS clientsPBS clients

HPSS servers

~ 60km0.2TBSECE

User PCs

Client disk speed @ KEK = 48MB/sClient disk speed @ ICEPP=33MB/s

0 2 4 6 8 100

KEK client (LAN) ICEPP client(WAN)

# of file transfer in parallel

)Pftp→pftp HPSS mover disk Client disk

to /dev/null

to client disk Ftp buffer=64MB

Even 3ms latency affects on results

microscopic monitoring on network performance

•speed=increment of sum of TCP data size in every 10ms•window size grows slowly after packet losss

•longer windows is not perfect when you have packet loss

w/o packet loss w/ packet loss

• http://netlab.caltech.edu/FAST/

• Looks nice and we want to try, but we haven’t because of the patent and IP issue. – Some people at KEK afraid that they might

have a difficulty to work in the similar topic once they see the source code of it.

– Is this true?

• A key issue of GRID• ICEPP and KEK will run one CA jointly for

BELLE and ATLAS– only for active KEK users to simplify the procedure– current situation

• ICEPP depends on a foreign CA to join LCG • KEK is running a test CA locally

• CA management costs are not cheap. Any good idea?

Storage evaluation

• SAN solutions – IBM SANfile (aka. StorageTank)

• Linux+AIX in server side• tested AIX, Solaris and Linux clients

– Linux clients were fastest in our tests

– HP Lustre• waiting for beta product delivery

Distributed simulation in advanced radio therapy

• A model on hospitals and computer centers– Hospitals send CT, MRI or PET images (DICOM) with

treatment planning data to a computing center as input

• higher security is required to protect personal data

– Full simulation using Geant4 at computing centers • parallel simulation

– Analysis results and feedbacks in DICOM to hospitals

• Validation of treatment planning

Toward Asia-Pacific collaboration

• Taka an advantage on working together with people in neighboring time zones

• HEP population in Asia-Pacific– ATLAS

• Australia(7), China(15), Japan(45) and Taiwan(5)• 72/1446 = 5.0%

– CMS• China(31), Korea(17) and Taiwan(14)• 62/1676 = 3.7%

– Alice• China(20), Japan(3) and Korea(12)• 35/747 = 4.7%

– LHC-b• China(19)• 19/737 = 2.6%

– Belle • Australia(8),China(11),Japan(122),Korea(15) and Taiwan(13)• 169/246 = 67% (excluding Japan 20%)

Summary

• Japan is an unique country in Asia-Pacific region which have large accelerators for HENP – should have a “tier-0” center

• SRB is under testing for BELLE• ICEPP, U-Tokyo and KEK are collaborating

together to build the ATLAS regional center in Japan

• We seek a collaboration among Asia-Pacific countries– BELLE, LHC– More bandwidth is necessary among sites

HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK.

Documents

Transcript of HEP Data Grid in Japan Takashi Sasaki Computing Research Center KEK.

CAL POLY POMONA - Sasaki

Norita Kawanaka (KEK) Kunihito Ioka (KEK) Mihoko M. Nojiri (KEK/IPMU)

KEK on finite T & mu QCDT. Umeda (Hiroshima) QCD thermodynamics from shifted boundary conditions Takashi Umeda Lattice QCD at finite temperature and density,

Kozo Sasaki - pure.iiasa.ac.at

Takashi Miike

Computing Coordination in Japan Takashi Sasaki Computing Research Center KEK, Inter-University Research Institute Corporation High Energy Accelerator Research.

QCD vacuum in strong magnetic ﬁelds - 京都大学hhiqcd.ws/4th/Ozaki.pdf · QCD vacuum in strong magnetic ﬁelds Sho Ozaki (KEK) In collaboration with Takashi Arai (KEK) Koichi

Sasaki practical-linked-data

Latt05 Dublin1 Remarks on the Maximum Entropy Method applied to finite temperature lattice QCD Takashi Umeda (BNL) Hideo Matsufuru (KEK) Lattice 2005,

Thoughts on software development in Japan and KEK Takashi Sasaki KEK Computing Research Center.

Supercentenarian as a model of happy, healthy longevity · healthy longevity Nobuyoshi Hirose, Yasumichi Arai, Takashi Sasaki ... Many old people also forget and recall. I think that

GRID Deployment Status and Plan at KEK ISGC2007 Takashi Sasaki KEK Computing Research Center.

TAKASHI MIIKE'S

Takashi Menu

Superbeam long baseline experiments Takashi Kobayashi KEK 100830 Neutrino Summer School @Tokai.

Takashi Nagatsuka

Sali Sasaki - Illustration Works

Introduction to Geant4 Part I – General Introduction Makoto Asai (SLAC) Takashi Sasaki (KEK) March 22 nd, 2002 @ NIRS.

Mlw sasaki-20101027

Status of T2K Takashi Kobayashi KEK For T2K collaboration 1 June 14, 2010 Neutrino 2010 @Athens, Greece.