Status and Perspective of INFN scientific Computing

A. ZoccoliINFN and University of Bologna

EU-T0 Meeting Bologna February 8 2018

OUTLINE

2

Organization and current status

Future challenges and requests

Guidelines and actions for future

developments

INFN Computing Organization

1 National Computing Center (CNAF in Bologna) hosting the WLCG Tier-1, where is concentrated the major part of the personnel

9 Tier-2 centers on the Italian territory sitting mainly in University Physics Departments

C3S “Comitato di Coordinamento attività Calcolo Scientifico INFN”. Mandate: make proposals for research and developments for scientific computing, including the development of a shared computing infrastructure, to support researchers primarily from INFN, but also from different research institutions.

3

Computing for experimental physics

4

Tier1:

•200 kHS06 CPU power equivalent to

20000 cores

•19 PB disk space

•57 PB tape library

10 Tier2:

•240 kHS06 CPU power equivalent to

24000 cores

•18 PB disk space

Network provided by GARR Consortium

(10-100 Gbps), under upgrading

Major part of the computing centers are multidisciplinary (HEP, astroparticle, neutrino, etc. )

Resources distribution

5

Italian contribution 8%

In the world WLCG

In Italy Half of the resources @ Tier1

Very effective infrastructure for LHC

data analysis !

Computing for theoretical physics

6

APE100

APEmille

apeNEXT

FERMI

GALILEO

MARCONI

TH-cluster@CNAF

clusters (BA-CT-MIB-PI)

HPC@INFN

(2017)

Agreement INFN – CINECA valid for 3 years:

use of 6% MARCONI + 15 Mcorehours

GALILEO (~1.4 Pflops)

#1

#500

Top500

24

Utilizzo risorse HPC dell’INFN al CINECA da

Settembre 2012 a oggi (*)

LATTICE

GENREL

NUCL-TH

FIELDTURBBIOPHYS

COND-MATOTHERS

FERMI- GALILEO- MARCONI

(*) in unità core BG/Q FERMI

Use of INFN resources @ CINECA

from 2012

76.6%

10.2%

Currently exploiting

CINECA, the national

HPC computing center.

Costs & manpower

7

Item Cost (M€)

CPU, disk and tape 4

HPC@CINECA 0.5

Electric power 2

Network(@GARR) 5

Total/year 11.5

Personnel

Infrastructure Middleware

and software

50 FTE 30 FTE

Budget

Future needs

LHC resources for High-Lumi

9

By assuming:

•Trigger rate 5-10 kHz, factor10 wrt today

•Expected Pileup ~140-200 ⇒ complexity of events increased by a factor 10

Back on the envelope evaluation of resources needs = 100x wrt

today !!

LHC resources for High-Lumi - 2

10

Expected mitigation effects:

Technological evolution for CPU and storage: +20%/y

performances 6x in 10 years

Computing Models Optimization (CPU: exploitation of parallel

architectures) and of the trigger scheme (raw data vs AOD)

CPU: 6x (CMS) e 4x (ATLAS)

Disk space: 4x (CMS) e 7x(ATLAS)

Current estimated gap:

Future Computing Resources

11

Googlesearches98PB

LHCSciencedata

~200PBSKAPhase1–

2023~300PB/yearsciencedata

HL-LHC– 2026~600PBRawdata

HL-LHC– 2026~1EBPhysicsdata

SKAPhase2–mid-2020’s~1EBsciencedata

LHC– 201650PBrawdata

Facebookuploads180PB

GoogleInternetarchive~15EB

Yearlydatavolumes

Future Astroparticle experiments (SKA, CTA, Euclid, …) will

produce unprecedented amount of data.

Future Computing Resources

12

The needs for ExaScale systems in science

HPC is mandatory to compare

observations with theoretical models

HPC infrastructure is the theoretical

laboratory to test the physical

processes.

Let ’s talk of Basic Science...High Energy & Nuclear Physics

LQCD (again...), Dark-energy and dark matter, Fission/ Fusion react ions

(ITER)

Facility and experiments designEffective design of accelerators (also for medical Physics, GEANT...)

Astrophysics: SKA, CTA

...

Life sciencePersonal medicine: individual or genomic medicine

Brain Simulation < – HBP (Human Brain Project) flagship project

P. V icini ( IN F N Rom e) E xaN eSt st at us L N GS, M ay 22-26, 2017 2 / 50

HEP and astroparticle

physics

Human brain

Biology

Personalized medicine &

genomics

Weather predictions

Climate change studies

• Material studies

• Smart manufacturing &

Industry 4.0

• IoT

• SmartC

Impact on private sector

Huge resources demand resulting in requests of HPC and Big Data

management will come from many different research fields in the

next years:

Future developments

Guidelines & Actions

14

• Evolution of the Middleware

• Evolution of the WLCG infrastructure

• Cooperation and synergy with the European context

(ESOC + EuroHPC)

• Optimization of the INFN e-infrastructure for HEP and

other disciplines

• Integration of the Italian HTC and HPC infrastructure

• Search a new location for the Tier-1

• Test beds for the use of the INFN e-infrastructure from

private companies

Middleware evolution

15

INFN is deeply involved in the transition to CLOUD for example with the

H2020 project:

INDIGO-DataCloud: INtegrating Distributed data Infrastructures for Global

ExplOitation (11,3 M€) 26 partners PI D. Salomoni (INFN)

Computing

&

data center

GRIDComputing

&

data center

Experiments

data

Network

Computing

&

data center

Computing

&

data centerExperiments

data

Network

Today

CLOUD

TomorrowYesterday

Downloads from INDIGO Repository

•ElectricIndigo

• ElectricIndigo6(https://www.indigo$datacloud.eu/service$component):

•

•

•

•

The Horizon2020 eXtreme DataCloud – XDC

project aims at developing scalable

technologies for federating storage resources

and managing data in highly distributed

computing environments, as required by the

most demanding, data intensive research

experiments in Europe and worldwide.

D. Salomoni

16

D. CEsini

Optimization of the INFN e-infrastructure for HEP and other disciplines

17

• INFN started a survey of its computing infrastructure in order to

evaluate the status, the sustainability and the evolution

• INFN signed a MOU started a collaboration with INAF to expand and

exploit a common infrastructure. First action is hosting data and provide

computing resources to the CTA experiment

• INFN started a collaboration with ASI to host data of the Copernicus

and CosmoSkyMed satellites and to provide computing resources to the

relevant interested communities

30 e-Infrastructures

Big

(T1+T2+CS+TS)

Small

10 20

95% Resources 5% Resources

Tests on flexible use of the INFN infrastructure

18

The goal: test a flexible use of the infrastructure by using remote

resources at Tier-1, by using in a transparent way CPU at Tier-2

Recas in Bari

T1 Bari600 km

It is working…

Performed tests also on small scale with private cloud providers:

ARUBA e AZURE (microsoft)

Integration of the Italian HTC and HPC infrastructure

19

• INFN started a project with CINECA in order to integrate them to

provide services to:

• Institutional basic and applied research

• Proof of concept and innovation for private organizations and

industries

DATA

Integrated Research Data Infrastructure

Network

Search for a new location for the Tier-1

20

ECMWF center will be hosted in Bologna from

2019 in the Tecnopolo area.

Possibility to host in the area also:

INFN Tier-1

Cineca computing center

Already allocated 40 M€ for the

Italian government to refurbish the

area. Looking for extra budget for IFN

& CINECA

The goal: provide a new location for the INFN Tier-1 to take intoaccount future expansion.

IPCEI project ICT-11 call

21

Goal: build an European Data Infrastructure (with HPC and Big Data resources) based on the existing e-infrastructures for research to provide services also for Public and Private Partnership

Partners: France, Spain, Luxemburg, Italy

Test beds with involvements of private companies:

Space, Smart Manifacturing, Materials, Personalized medicine

Modular e-infrastructure to be realized at European, national or regional level, which may serve different fields

Credits

22

The presentation is based on the work performed by the

C3S Committee.

Thanks to

all the people helped in the preparation of the slides: D.

Lucchesi, T. Boccali, S. Campana, L. Cosmai, D. Salomoni,

P. Vicini …….

Conclusions

23

In the next years we have to cope with un

unprecedented amount of data coming from

many different fields (not only HEP)

INFN e-infrastructure is in a transition phase

We are actively exploring the most suitable

solutions for our future

Backup slides

External funds

25

Anno Tipo Progetto Budget

Totale (M€)

Budget

INFN (M€)

2015 H2020 INDIGO-DataCloud 11.1 2.1

2015 FISR High performance data

network

12.5 12.5

2016 H2020 ExaNeSt 8 0.7

2017 H2020 EOSCpilot 10 0.3

2017 H2020 EOSC-HUB 30.0 1.8

2017 H2020 Extreme DataCloud 3.1 0.6

2017 H2020 Deep Hybrid DataCloud 3.0 0.4

2017 H2020 EuroEXA 20 0.7

Totale 97.7 19.1

2018 MIUR PON 15 ?

Toward a Cloud based system

26

EOSC-hub Data and sertvices for EOSC

PI: EGI INFN: L. Gaido

XDCeXtreme DataCloud

Development of technologies for storage resources

federation and for data management

PI: INFN D. Cesini

DEEP HybridDataCloudDesigning and Enabling

E-Infrastructures for intensive

Processing in a Hybrid

DataCloud

Services and support for intensive computing

for different disciplines

PI: CSIC (Spain) INFN: G. Donvito

Definition of the EOSC governance

Realization of demonstrators

PI:UK INFN: D. Salomoni

INFN projects toward HPC exascale

27

European Exascale System Interconnection Network

& Storage

PI: Foundation for Research & Technology, GR

INFN: P.Vicini

ExaNeSt: FETHPC 2014

ExaNeSt: European ExascaleSystem Interconnection Network &Storage

EU Funded projectH2020-FETHPC-1-2014

Duration: 3 years (2016-2018).Overall budget about 7 MEuro.

Coordination FORTH(Foundation for Research &Technology, GR)

12 Partners in Europe (6industrial partners)

P. V icin i ( IN F N Rom e) E xaN eSt st at us L N GS, M ay 22-26, 2017 13 / 50

ExaNeSt

Co-designed Innovation and System for Resilient Exascale

Computing in Europe: From Applications to Silicon

PI: Institute of communication and computer systems, GR

INFN: P. Vicini

Acquisition and exploitation of many core "next gen” for

INFN HPC e HTC infrastructures

P. Vicini et al.

Progetto CIPE

Theoretical physics HPC requests

28

45

Stima delle necessità per la comunità HPC

risorse di calcolo

adeguato supporto per postdoc grant e per

organizzazione di workshop e meeting

4

computat ional physicists. New agreements established with the Italian Nat ional Supercomput ing Cent re (CINECA)have made larger comput ing resources available while two special projects (SUMA first , and then the so-called CIPE-project ) have helped establish a more cohesive computat ional community, offering Post -Doc posit ions at the junct ionpoint between theoret ical physics, algorithm development and code opt imizat ion, as well as a number of venues toshare ideas and expert ise in HPC comput ing. These investments have allowed INFN teams to re-establish a leadingrole in many research areas and have produced virtuous side-effects e.g., allowing improved access of Italian theoret icalcomputat ional groups to the EU supported PRACE programme [2] and have also increased the possibility to influencemajor decisions within internat ional collaborat ions.

It is crucial that this posit ive t rend be maintained and possibly further enhanced in the foreseeable future. Theseissues have been discussed in a meet ing held in Rome on February 17th, 2017 [3]. The present document is a followup of that meet ing and also of a similar paper writ ten in late 2014; it describes in full details the likely evolut ion of thetheoret ical computat ional community in the next few years and quant ifies the investments (financial and otherwise)needed in the t ime frame 2018-2020 to ensure that the current level of excellence can be retained or improved.

Computat ional theoret ical physics covers several research areas in the INFN theoret ical group:

I High Energy Physics - Lat t ice;

I I High Energy Physics - Phenomenology;

I I I General Relat ivity, Cosmology and Ast ropart icle physics;

IV Nuclear Theory;

V Fluid Dynamics;

VI Quant itat ive Biology;

VI I Disordered Systems;

VI I I Condensed mat ter;

with many research projects and more than 100 researchers involved. Research areas I to IV, clearly of st rategicinterest for INFN, have used in the last 5 years approximately 90% of the overall available comput ing resources,and research areas V to VI I I, while using only 10% of the available resources, offer an invaluable contribut ion to amult i-faceted and culturally wide research environment . Moreover, computat ional theoret ical physics in the next fewyears has the potent iality to develop synergies and collaborat ions with experimental areas of interest to INFN, bothas a theoret ical support to experiments and in the sharing of comput ing techniques.

The following sect ions of this document i) describe in details the scient ific perspect ives of these research areas inthe t ime frame of the next three to five years ii) quant ify the computat ional resources that are needed to successfullypursue the envisaged research programmes in the t ime window 2018-2020 and iii) present a number of suggest ions onthe act ion that INFN should undertake to support the computat ional community in this t ime frame. The rest of thisExecut ive Summary briefly lists our requests and suggest ions.

2018 2019 2020

LGT: hadron physics 54 108 180LGT: QGP and BSM 207 432 648LGT: flavor physics 117 234 387

Colliders Phenomenology 1 2 3General Relat ivity 142 182 227Cosmology and Ast ropart icle Physics 3 4 6

Nuclear Theory 18 27 36

Fluid Dynamics 50 80 110Quant itat ive Biology 9 18 27

Disordered systems 4 6 8

Condensed mat ter 2 4 6

G r and Tot al (M cor e-h) 607 1097 1638

G r and Tot al (Eq. P fl ops) 4.6 8.4 12.5

TABLE I: Est imate of the yearly comput ing requirements of several areas of computat ional physics; units areMcorehours. In the last line, the grand total is converted in PFlops, using as reference the core of the Intel SkyLake

processor with a nominal peak performance of 67 GFlops.

HEP-LAT74%

HEP-PH0%

GENREL14%

ASTRO-PH0%

NUCL-TH2%

FLU IDS7%

BIOPHYS2%

DISORDERED1% COND-MAT

0%

Requested resources

Possible evolution of the WLCG infrastructure

29

Commercial Clouds and

HPC centers when

available (elastic model)

1-10 Tb/s

DC

DC DCCompute

Compute

Cloud

access

for

Users

Fast network (∼Tb/s)

Data Centers (DC): host the major part of

tape and disk space with small CPU (core

of the infrastructure)

Computing Centers, withCPU and

disk space

Cooperation and synergies with the European context

30

• In 2015 launch of the European Open Science Cloud (EOSC) initiative to

provide an e-infrastructure for all European research communities.

• In 2016 signature of the EuroHPC agreement to provide a 10 years European strategy toward HPC exascale machines

EOSC

EDI