Ian Bird CMS Computing & Software CERN, 15 th October 2015 15 Oct 2015 Ian Bird; CMS Offline &...

Ian Bird; CMS Offline & Computing 1

Science Cloud Initiatives

Ian Bird

CMS Computing & Software

CERN, 15th October 2015

15 Oct 2015


Cloud experiences Many experiences:

Through all LHC experiments With many labs and WLCG sites

Majority of usage has been with simulation use cases Not much experience with data intensive use

We know we can successfully use cloud resources, and make them useful

Even have had some reasonably large scale uses However, largely supported by grants, free access, and

etc. Amazon grants, Google offers, Helix Nebula project,

including a small scale procurement, etc.

15 Oct 2015


Why are clouds interesting? Must define what we mean

Private clouds• Technology – better/different implementation of

resource provisioning• Mechanism for connecting to external resources

Commercial clouds• Why are they relevant?

• Cost (actual cost, elasticity/flexibility, operations cost, …)

Bottom line is only how we can maximise the amount of computing we can buy

15 Oct 2015


Open issues What is the cost? How can we manage data in commercial clouds? How do we manage network policies? What would be a model that easily allows us to

use commercial clouds? What are the procurement issues?

Where can we procure? Is large-scale joint procurement possible? How to procure spot pricing, “backfill”, paying for what

is actually used, etc.• And how this is verified

15 Oct 2015


Cloud use CERN-specific

What is long term evolution of the CERN DC?• Wigner-style facility, essentially classic DC extension of CERN • Hosted dedicated facility?• Simply buy compute and storage services – paid by use?

How can commercial cloud services supplement or (partially) replace HEP-owned and managed facilities (When) will cost drive this?

What is the model? Cloud treated as a separate site like a Tier 1, or 2? Cloud treated as an extension of a site – as part of the pledge Jointly procured large resource pool? NB. Funding and procurement mechanisms will drive this

15 Oct 2015


Science Cloud – past, present Helix Nebula EC project in Europe (together with other sciences)

CERN is doing 2 procurements to understand market prices and usage models: 200 KCHF for compute (essentially simulation) 400 KCHF for data intensive use cases (needs market survey) To be procured as extensions of the CERN cloud Eventual use cases – investigate alternatives to “Wigner-style” extensions

for next round; dynamic expansion of CERN resources This capacity is in addition to pledges

PCP project has been approved and will start in January 2016 Derogation of CERN procurement rules to allow compliance with EC

procurement (EC member states) agreed by CERN Council last week Will allow a joint procurement across Tier 1s – understand if we can

obtain economies of scale see next few slides

15 Oct 2015


Science cloud … And parallel discussions with other

commercial providers AWS, Microsoft Azure Openlab project with Rackspace on federating

clouds and using federated ID’s (in openstack)

15 Oct 2015

The Helix Nebula Initiative

Strategic Plan

Establish multi-tenant, multi-provider cloud infrastructure

Identify and adopt policies for trust, security and privacy

Create governance structure

Define funding schemes

To support the computing capacity needs for the LHC

experiments

Setting up a new service to simplify analysis of large genomes, for a

deeper insight into evolution and biodiversity

To create an Earth Observation

platform, focusing on earthquake and volcano research

To improve the speed and quality

of research for finding surrogate biomarkers based on brain images

Adopters

Suppliers

Additional Users:

8Ian Bird; CMS Offline & Computing15 Oct 2015

http://www.awst.at/en/index.html













Example HNX procurementCERN price enquiry for a small fraction of the resources necessary to run the ATLAS experiment simulation software Several valid responses were received and Atos (data centre in

Tenerife, Spain) was awarded the contract accessed via the SlipStream broker and provisioning engine (from SixSq SME) and delivered over the GÉANT network

1.2 million CPU hours over 6 weeks producing 11.5 million GEANT4 tt� events with a high level of efficiency for CPU intensive workload that confirmed the use of commercial IaaS is technically feasible for simulation workloads variations in RESTful APIs can be overcome with reasonable technical effort independent monitoring of the quality of service and resources consumed is

essential

Ian Bird; CMS Offline & Computing 915 Oct 2015


Hybrid – link public research organisations, e-Infrastructures & commercial cloud services Use GEANT network to link Research Infrastructures, repositories

(EUDAT, OpenAIRE), EGI, PRACE etc. to commodity commercial cloud services (multiple providers)

A cornerstone of the Open Science Commons*

Trust - Researchers keep control of the cloud and their data Guarantee a copy of all the data is kept on public resources Ensure long-term preservation of the data Insulate users from changes of service supplier and technology

Economy - Must be cheaper than the ‘build our own’ approach Avoid separate ‘silos’ for each Research Infrastructure/Community Profit from the economies of scale in commercial data centres

* http://go.egi.eu/osc

Towards the EuropeanOpen Science Cloud

http://dx.doi.org/10.5281/zenodo.16140

15 Oct 2015

http://go.egi.eu/osc


11

Why PCP?Commercial IaaS exists but not certified, integrated with public e-infrastructures, offering std interfaces with suitable SLA and contractual terms & conditions.

PPIPotential follow-on project if this PCP project is successful15 Oct 2015 Ian Bird; CMS Offline & Computing

HNSciCloud H2020 PCP ProjectThe group of buyers have committed• ~1.6M€ of funds

(generating ~6M€ total funds)• Manpower• Applications & Data• In-house IT resources

To procure innovative IaaS cloud services integrated into a hybrid cloud model• Commercial cloud services• European e-Infrastructures

• GEANT, eduGain, EGI• In-house IT resources

Procured services will be made available to end-users from many research communities


Preparation

• Analysis of requirements, current market offers and relevant standards

• Build stakeholder group• Develop tender

material

Implementation Sharing

• Best practises• Recommendations• Training

Oct’16

PCP project phases

Launch of tender Pilots tested & assessed

9 months 15 months 6 months

Jan’16

Jan’18

Jun’18

All procuring organisations must sign letter to re-confirm commitment of funds

Design

Prototype

Pilot


User GroupsThe cloud resources procured will be made available to user groups during the pilot phase

LHC experiments via WLCG• Procured resources will count against the buyers’ pledges (during pilot phase)• CERN will provide the interface via Tier-0 (OpenStack in tender spec.)

ELIXIR• Managed by EMBL-EBI via the ELIXIR Compute platform

Other research communities via EGI Fed Cloud• Request OCCI interface in tender spec.

Local users at each buyers site• Each buyer is responsible for integration• Request web GUI interface in tender spec.

• Sites can also use OpenStack or OCCI interface as well

Each buyer decides what fraction of their procured resources is made available to each user group but cannot assign only to their local users

Collectively the users will form a user group with a role in the project to define requirements and provide feedback on pilot deployments

BBMRIDARIAHEISCAT_3DEPOSINSTRUCTLifeWatchLong-tail

15 Oct 2015


European Open Science Cloud

15 Oct 2015


Open Science Cloud and the EC

Questions posed by RJ Smits during visit to CERNwith Commissioner Moedas in January 2015: “The Commission will launch a European Cloud

initiative including cloud services certification, contracts, switching of cloud services providers and a research open science cloud.”

The draft H2020 Work Programme 2016 – 2017 (annex 4) includes funding call INFRADEV-04-2016 European Open Science Cloud for Research (10M€)

Commissioner Moedas announced in June 2015:

CERN produced a paper Towards the European Open Science Cloud that answered these questions http://dx.doi.org/10.5281/zenodo.16140

15 Oct 2015





Summary A lot of ongoing work exploring how to use

commercial (cloud) resources for various workloads And how to procure and provision such resources

This is needed in order to understand and plan for how to most cost-effectively procure compute and storage resources in the future

EC has a vision of a European Science Cloud This might develop out of some of the work we are

involved in We have to understand whether that vision matches how

we see the LHC computing models evolving

15 Oct 2015

Ian Bird CMS Computing & Software CERN, 15 th October 2015 15 Oct 2015 Ian Bird; CMS Offline &...

Documents

Transcript of Ian Bird CMS Computing & Software CERN, 15 th October 2015 15 Oct 2015 Ian Bird; CMS Offline &...