LIGO with GSDC - APCTP

LIGO with GSDC- status update

Beob Kyun KIM

2011. 7. 30

GSDC Workshop (2011.6.3) 1

Outline

2011 APCTP International School on Numerical Relativity and Gravitational Waves 2

Introduction to GSDC

2010 Activities

GSDC Resources and Status

LIGO


Introduction to GSDC

Organization

42011 APCTP International School on Numerical Relativity and Gravitational Waves

Introduction to GSDCMISSION

Physics 2009 ~ Next ~ALICESTARBelleCDF

Bio-informaticsEarth ScienceAstrophysics


Main ProjectsMinistry of Education,

Science and Technology(Korean Government)

GSDC

6The workshop for ALICE upgrades by Asian countries (2011) @ Yonsei University

http://english.mest.go.kr/index.jsp�

Our Mission

(CDF/FNAL)(Alice/CERN)

(Belle/KEK)

2PB/2,000CPU 10PB/10,000CPU

(LIGO/LLO)

(STAR/BNL)

Phase 1

(2009~2011)

Phase 2

(2012~2014)

Provide Global Science Data AnalysisEnvironmentNational Data CenterAsian-Pacific Hub Center

Expand Supporting Fields: Earth Environment, Biometrics, Nano-tech, etc.Global Computing Resources Assigning and Information SystemCyber Research and Training Environment


http://www.openclipart.org/detail/5159�





MembersName Role

Dr. Haengjin Jang Director of GSDC

Dr. Hyungwoo Park Deputy

Dr. Sungyun Yu Strategy

Mr. Heejun Yun System admin.

Dr. Christophe Bonnaud System admin.

Mr. Seunghee Lee System admin & GBRAIN

Dr. Beobkyun Kim Belle & LIGO, KiAF

Dr. Seo-young Noh CDF

Dr. Jonghu Lee ALICE Tier1 & STAR

Mr. GyeongRyoon Kim CDF

Ms. Tatyana Khan Belle & KiAF

Dr. Sulah Ahn ALICE Tier-1 (Physicist)

Dr. Seokmyun Kwon Cyber Lab.

Mr. Jin Kim Cyber Lab.

Mr. Jiwoong Kim Cyber Lab.

Ms. Gooyeun Baek General Affairs

Ms. Yongsuk Lee General Affairs

Network experts from KISTI

Power and air conditioningfrom KISTI


1 more staff will join us, in few months


2010 Activities

RAW Data Processing (Grid)144cpu/200TB 400cpu/600TB 1,000cpu/1,000TB

Data Analysis48cpu/800GB ~200core

ALICETier-1

KiAF

MC Production (Grid)120cpu/50TB

ALICETier-2

MC Production (Grid) Full Recon. Data Analysis256cpu/200TB 256cpu/300TB

MC Production (Grid) CDF Data Reprocessing (Grid)256cpu 400cpu/300TB

Belle/Belle2

CDF

2009 2010 2011 2012 2013

CERN

KEK

FNAL

LIGOData Analysis

16cpu/5TB 120+200cpu/100TB

5

History of GSDC Service


2010 ActivitiesPreliminaryALICE Tier-1

Now, in service with 144 core & 200TBKiAF (KISTI Analysis Facility for AP Region)

ALICE Tier-2 Site availability: 98% since Feb. 2009since 2008

Belle & Belle2Official Belle data service for Korean ResearchersBelle MC production on Grids (LCG)Contribution to Belle2 Software Development

CDFProviding computing resources under NAMCAFSupporting CDFSoft developmentCDF Data Reprocessing

LIGO Condor-based Testbed of LDG (LIGO Data Grid)

GBRAINComputing resource provision for Neuro SciencePlanning to extend CBRAIN to Grid version(mainly from McGill Univ. Canada)


Service Status – EGI


Service Status - OSG



GSDC Resources and Status

Current Computing Resources: CPUsExperiment Cluster Specification Memory Node Core kSI2k

ALICE Tier-2 ce-alice DellIntel Xeon E5405 2.0 GHz Quad 2 CPU

16GB 6 48 48

ALICE Tier-2 Ce01 HPIntel Xeon E5420 2.5GHz Quad 2 CPU

16GB 16 128 219

Belle & LIGO Ce02 IBMIntel Xeon E5450 3.0GHz Quad 2 CPU

16GB 16 128 324

CDF & etc. Ce03 IBMIntel Xeon E5450 3.0GHz Quad 2 CPU

16GB 16 128 324

IBMIntel Xeon X5650 2.66GHz 6 Core 2 CPU

24GB 20 240 744

ALICE Tier-1Prototype

ce12 IBMIntel Xeon X5650 2.66GHz 6 Core 2 CPU

24GB 12 144 446

KiAF afmaster01 IBMIntel Xeon X5650 2.66GHz 6 Core 2 CPU

24GB 4 48 180


Current Computing Resources: Storage

Experiment Model Type Usable Size

ALICE Tier-1 prototype

Hitachi USP-V Disk (SAN) 100TB, disk(+100TB, tape emul.)

ALICE Tier-2 NetApp FAS2050 Disk (SAN) 5oTB

CDF Hitachi USP-V Disk (SAN) 200TB

Belle Hitachi USP-V Disk (SAN) 200TB

STAR NetApp FAS6080 Disk (SAN) 100TB

LIGO, gBrain, etc NetApp FAS6080 Disk (SAN) 100TB

850TB


Internal network of resources


Storage architecture


Resource Plan


• CPU : 1,000 cores in procurement process

• Disk : 1,200 TBs usable (1,800 TBs physically) under construction

• Basically,

• Dedicated 2Gbps for each machine &

• 10Gbps for aggregated cluster of machines

1,000 cores & 1,200 TBs (usable)

• CPU

• Shared 120 core (PBS base)

• Dedicated 120 core (Condor base, ???)

• Disk

• 5TBs + 95TBs (???)

For LIGO

Now and tomorrow


LIGO in GSDC

MCMC


• Pusan National University & West Virginia University & KISTI

• Job run since 2011. May (discussed since March)

• Pbs based Shared resource (ce02.sdfarm.kr)

• OS : SL5.2 x86_64

• To avoid complexity in maintenance, SL is always our first choice

• 120 cores

Markov-Chain Monte Carlo Analysis of GW Signals

Resources for LIGO


ldg.sdfarm.kr

ldg.sdfarm.kr ce02.sdfarm.kr workers

workers

workers

moved

ldg.sdfarm.kr

Condor

Torque (PBS)

LIGO

LIGO, Belle, Nuero

Storage(/data/ligo)


MCMC (min) MCMC (ideal) KGWG (2011)

CPU 80 cores(3GHz)

1040 cores 120 core (dedicated) + several hundreds (shared)

RAM 2GB/core 2GB/core 2GB/core

Disk 1TB 10TB 100TB

• Competition is expected between LIGO, Belle, and Neuro (in soon)

Lack of CPUs & Disks

LIGO jobs at ce02


0

200

400

600

800

1000

1200

1400

1600

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

NEURO

LIGO

Belle

• Ce02 has 128 cores

• LIGO has occupied 30% of total run jobs in June.

• By MCMC group

Ce02 is a cluster which is shared by LIGO, Belle, and Neuro

LIGO with GSDC - APCTP

Documents

Transcript of LIGO with GSDC - APCTP