National Institute of Supercomputing and Networking

National Institute of

Supercomputing & Networking

Introduction KISTI ScienceDMZ

(Including PRP, NRP, Asia PRP)

Jeonghoon Moon8th Aug 2018

APAN 46th Meeting@Auckland, APRP Session

2 /592

Highlights of KREONET

1. 100G from Daejonto Chicago/StarLight

2. SDN-WAN Connection

to Chicago/StarLight,

Add Seattle and Hongkong (Aug)

3. 100G Ring linking major cities

4. 17 GigaPoPs with 1G, 10G or 40G

100G

KREONET2 & GLORIAD-Kr & SDN Deployment (KREONET-S)

Additional 100G from Daejeon to Seattle and Hongkong early 2019

100G

100G

• KISTI ScienceDMZ– Two types DTNs (SDN/L3 based)

• 100G : International & Giant Group Research & Study

• 10~40G : Domestic Scientific Research Group / ~1G : Normal User (Fionette)

– Participated in 1st PRP Workshop since 2015 & 1st NRP Workshop since 2017

• Deploy ScienceDMZ and Activites

– Domestic

• Focus on Giant Group Research & Study(Data-Intensive Science) areas

• Examples : K*GENOME, GENOME RDC, LSST, HEP(CMS, ALICE), SDO, HPC,

GSDC, KSTAR, KBSI TEM, eKVN etc.

• Especially, eKVN(Korea eVLBI) and LSST deploy 100G connection (KASI)

• Deploy KISTI New Supercomputer with a DTN Cluster on 100G

• Dark Matter research using astronomical/particle big data on ScienceDMZ

– International

• Participate in PRP, NRP, APRP

• Asia PRP : Collaboration with APAN members for Deploy and use ScienceDMZ

• Participate in ‘Peta Scale DTN Project’

KISTI ScienceDMZs

eKVN

KBSI TEM KSTAR

K*GENOME

SDOHEP (CMS, ALICE)

LSST

GSDC

Genome RDC

Supercomputer(5th)

KISTI ScienceDMZs

• SDN based DTN– Seoul & Daejeon on ONOS(SDN/VDN,

Currently, 10G)

– For SDN based applications of KREONET service

– Enabled SDN environment between Daejeon,

Chicago(now), Seattle, and Hongkong(in Aug

– Enabled SDN environments for 5 major places

in Domestic

• L3 based DTN– For International with 40G & 100G connection

for PRP and Peta Scale DTN

– From Aug, 2018, 100G test for Domestic

& International(SC18)

– Participated in PRP PS MaDDash (Prof. Tomas DeFanti & John Graham)

– Deployment for several areas

• Such as KISTI Supercomputer, eKVN, LSST, HEP, Climate Service and so on

• Expand 17 Giga Pops in KREONET(Domestic)

KISTI Supercomputer

KISTI SDN baseed DTN

SDN status of KREONET(Intl’ &Domestic)

Intl’ Exchange

(via StarLight)

Seoul-Core

Busan-Core

ChangWon-Core

GwangJu-Core

Chicago-Core

ScienceDMZ/DTN

ScienceDMZ/DTN

GwangJu-Edge

Deajeon-Core

Deajeon-Edge

ChangWon-Edge

Busan-Edge

Chicago-Edge

Seoul-Edge

KIMS

Tongmyong

Univ.

Ulsan Observatory

Tamna Observatory

Yonsei Observatory

PLSI

(via KISTI)

OverCloud-Box

(via GIST)

Cloud Gateway

(COREEN)

PLSI

(via GIST)

OverCloud-Box

(via GIST)

Seattle-Core

Intl’ Exchange

(via PacificWave)

Chicago-Edge

Hongkong-Core

Hongkong-EdgeIntl’ Exchange

(via HKLight)

Performance of KISTI ScienceDMZfor Domestic users 1/3

• Transfer performance of KISTI ScienceDMZ over SDN – Transfer performance between Seoul and Daejeon (over 160Km)

Iperf Verification(9.89Gb/s on 10GE)

GridFTP Verification(9.43Gb/s on 10GE)

SDN/ONOS/VDN Topology

Seoul-Daejeon transfer performance vi DTN

Transfer performance bwteen Seoul and Daejeon

- Iperf : 9.89Gb/s on 10GE- GridFTP : 9.43Gb/s on 10GE

VDN service on SDN/ONOSVDN service of TTA given connection and bandwidth

- supported

Performance of KISTI ScienceDMZ 2/3SC16@Salt Lake City(Nov.2016) Data Transfer Demonstration

• Long Distance Data Transfer Demo with Harvey Newman Team

• Between Daejeon(Korea) – Chicago – Salt Lake City : 10GE

• Approximately 10,000mi

• Result : FDT Transfer with normal TCP • AVG : 9.16Gb/s on 10GE

• Max : 9.48Gb/s on 10GE

• DTN – FIONA (Flash I/O Node Appliance) • Dedicated Data Transfer Node

FDT transfer with normal TCP

on 10GE between Daejeon-Salt Lake City

Performance of KISTI ScienceDMZ 3/3Expanding to the Global Research Platform

Via CENIC/Pacific Wave, Internet2, and International Links from Larry Smarr

PRP

PRP’s Current

International

Partners

Korea Shows Distance is Not the Barrier to Above 5Gb/s Disk-to-Disk Performance

Netherlands

Guam

Australia

Korea

Japan

Source: Larry Smarr(Calit2), 1st NRP Workshop@Bozeman 2017

LSSTLarge Synoptic Survey Telescope:

potential usage case of the DTN

⚫ LSST (Large Synoptic Survey Telescope; http://lsst.org)

− 10-year (2022 ~ 2032) survey of Southern Sky support

ed by DOE, NSF, international contributors, and private

donors.

− Production of raw data about at a rate of 15TB~30TB p

er night.

− 0.5Exabyte is expected around 2032.

− NCSA is in charge of processing, archiving, and serving

the data.

− Engineering “First Light: anticipated in 2019

− Science “First Light “ in 2021

⚫ KASI (Korea Astronomy and Space Science Institute)

is one of the international contributors on behalf of

the LSST Korea (http://lsst.kasi.re.kr).

⚫ KASI – Deployment of ScienceDMZ and PRP

ScienceDMZ for LSST in KASI

Data Volume of ~ 30TB per night• Data size : 0.5Exabyte for imaging (over 50PB for the catalog DB)

The LSST Network requirements• High demand end-to-end applications like LSST require that all networks in the path support

QoS and Programmability

• Each 12.7GB data set (6.4GB picture + 6.3GB metadata) must be transmitted to the U.S. in 5 seconds and Database synchronization, etc.

LSST data (part of data set) transfer from Chile and NCSA(US) to KASI• LSST data transfer demonstration at SC18 (Nov)

100G

100G

100G

ScienceDMZ Architecture for HPC(Collaboration with NERSC & Esnet)

• ScienceDMZ for KISTI 5th Supercomputer – Dedicated DTN connection with NERSC DTN

• Peta Scale DTN transfer Project– DTN Cluster on 100GE

• Globus Online– GridFTP

DTN

DTN

DTN

DTN

100G Switch

SDN(VPN/VDN)

5th Super-computer

File System

Normal User

…

DTN Cluster

F/W

슈퍼컴퓨터 5호기 100G/SDN

Peta Scale transfer environment over KISTI 5th Supercomputer

Collaboration with NERSC ScienceDMZ

DTN Cluster

ScienceDMZ architecture for HPC

4Gbps FabricInfiniband10GbE Network

Sun X4270 36EA / J4400 72EALustre OSS Servers

MPP Computer Cluster(3200 Nodes)

SUN X4600 4EASAM-QFS MDS Servers(Dual PCI-E(4) : Disks, PCI-E(1) : 10GbE, Dual PCI-X(2) : Tapes)

Brocade 48K

STK 6140 (24 Tray)(500GB SATA, ??TB Usable)

STK 8500 (LTO4 16 Drives)(1050 LTO3 Media)

IBMSMP

10GbE Network

6

6

(8 Port for Disk per Svr)(4 Port for Tape per Svr)

(1 Port for 10GbE per Svr)

2

22

22

22

22

66

66

6

66

6

Brocade 48K

SUN V215(ACSLS Server)

BACKUP2

[117TB]

2

4

Sun X4270 6EA / J4200 5EALustre MDS Servers

DTN 4 node(Dual PCI-E(1) : IB, PCI-E(1):10GbE)

66

6

6

6

Sun X4270 2EA [admin01.02]Provisioning Servers

HO

ME01/A

PPLIC

/SCRATCH

METAD

ATA

M9-648*8EA

M2-36 *6EA

KISTI Supercomputer architecture with DTN

DTN

SC18 Demonstration & LSST Project in Korea

• 100G connection with KISTI Booth@SC18

– 100G DTN demonstration between SC18 KISTI booth and Daejeon

– 100G switch and DTN will be set up at booth

– PRP/APRP collaboration

• Participate in Global 100G DTN demonstration with Caltech (Harvey Newman)

– Participate in several demonstrations with Global partners

• 100G LSST Demo (Working on a scenario)

– Focus on real LSST data from Chile to real user of Korea via US

– KASI(Korea Astronomy and Space Science Institute)• One of 100G users in KREONET

• One of the international contributors on behalf of the LSST Korea(http://lsst.kasi.re.kr)

– Two Endpoints will be available

• KASI and KISTI(Daejeon)

• 100G DTN will be set up before SC18

✓ Asia Pacific Research Platform(APRP) WG in APAN- Chair : Jeonghoon Moon, KISTI, Korea

- Co-Chair : Andrew Howard, NCI, Australia

- Secretary : Asif Khan, Perdana Univ. Malaysia

- Committee Member : Yves Poppe(NSCC, Singapore), Kenneth Ban (NUS, NSCC, Singapore),

Lin Gan(Tsinghua Univ. NSCC Wuxi, PRC), Garry Swan(CSIRO, Australia),

Suhaimi Napis(Putra Univ. Malaysia)

✓ APRP Project- 100G HK-Marseille link in June 2018 at TNC18 as first 100G Ring-Around-The-World

- Cooperation of Asia – USA – Europe to build a Global Research Platform with PRP, NRP,

ERP and APRP

- Asia-Pacific Bioinformatics Network (APBioNet) to propose genomics link with ELIXIR

- Ability for us to submit projects to TEIN*CC for Asia@Connect funding

- Related Research & Work for HPC in Asia

✓ APRP Roadmap- 2nd NRP Workshop at Bozeman, Aug, 2018

- APAN 46th Meeting at Auckland, Aug, 2018

- SC18 at Dallas, Nov, 2018

- APAN 47th at Daejeon Feb. 2018

- SCAsia19 at Singapore Mar, 2019 (APRP Conference will be held)

APRP - Asia Pacific Research Platform

National Institute of Supercomputing and Networking 15