Bringing HPC to the Cloud - HPC-AI Advisory Council …HPC in the Cloud Wish Lists • End users:...

All rights reserved.

© CloudBroker GmbH

Making Technical Computing

Available to Everyone –

Bringing HPC to the Cloud

HPC Advisory Council Switzerland Conference

Lugano Convention Center, March 24, 2015

Dr. Wibke Sudholt

CloudBroker GmbH

Heinrichstrasse 267, CH-8005 Zürich, Switzerland

Phone: +41 44 515 21 70

Email: [email protected]

Web: http://www.cloudbroker.com

1 24.03.2015

HPCAC CH Conference, Lugano

mailto:[email protected]

http://www.cloudbroker.com


© CloudBroker GmbH

Overview

• Cloud computing for HPC

• CloudBroker solutions

• CloudSME use case

• Conclusions

24.03.2015 2



© CloudBroker GmbH

NIST Definition of Cloud Computing

• From the US National Institute of Standards and Technology (2011)

• Essential characteristics: – On-demand self-service

– Broad network access

– Resource pooling

– Rapid elasticity

– Measured service

• Service models: – Software as a

Service (SaaS)

– Platform as a Service (PaaS)

– Infrastructure as a Service (IaaS)

• Deployment models: – Private cloud

– Community cloud

– Public cloud

– Hybrid cloud

24.03.2015 3



© CloudBroker GmbH

Pros and Cons of Cloud Computing

+ On-demand and pay-per-use access highly increases flexibility

+ If using public clouds, no need for users to maintain own computer resources

+ New business possibilities for software vendors and computer centers

± If using public clouds, it is a form of outsourcing

± One has to consider security, privacy and legal issues

- For special requirements or steady high use, the overhead might be too big

24.03.2015 4



© CloudBroker GmbH

HPC in the Cloud Wish Lists

• End users: – Easy and cost-efficient access to the applications of interest

– Can focus on their applications, no need to worry about the infrastructure or license details and delivery

– Being able to select the best software and resources for their purposes

• Software vendors: – Being able to deploy, present and manage their software in the best

possible way to attract users

• Computer centers: – Optimal use of their resources to support users and to justify budget

– Easy access to their infrastructures for both users and applications to increase reach and coverage

– Being able to manage software and users and have transparency about usage in a straightforward way

– Business generation through commercial and pay-per-use offerings

– Adding value to create happy users and budget providers

24.03.2015 5



© CloudBroker GmbH

HPC Cloud Computing Stack

24.03.2015 6


HPC

IaaS

PaaS

SaaS

Gate-

ways

Users


© CloudBroker GmbH

HPC Cloud Solutions:

IaaS • On-demand access to compute resources in the cloud

• Example: Amazon EC2

+ Wide selection of usually virtual machine types at different prices, from 1 to 36 CPUs, incl. GPUs etc.

+ Resources available very quickly, within minutes

± Well suited for loosely-coupled parallel calculations, but usually not so suitable for applications requiring high-speed network interconnects or bare-metal servers

- Need to assemble a cluster yourself

- Previous deployment of software required

24.03.2015 7



© CloudBroker GmbH


HPC on Demand

• Pay-per-use access to clusters and super-computers

• Example: HLRS

+ Resources are classical HPC infrastructure

+ Cloud business model

± Need to be familiar with HPC

- Usually some preparation necessary

- Previous deployment of software required

24.03.2015 8



© CloudBroker GmbH


Individual Application Portals • Access to particular HPC applications as a service

• Example: SCI-BUS gateways

+ Easy access to specific HPC applications

+ Optimized software setup, often combined with preparation and analysis tools

± Not necessarily pay-per-use

- Often intransparency or inflexibility regarding the used infrastructure resources

- Usually fixed and heterogeneous application access with danger of vendor lock-in

24.03.2015 9



© CloudBroker GmbH


One-Stop-Shop • Marketplace for HPC applications

• Example: CloudBroker Platform and AppCenter

+ Easy, uniform and pay-per-use access to multiple HPC applications

+ Providers can offer optimized packages of infrastructure and software, and individual application interfaces can be build on top through APIs

+ Users can compare and select from different applications and/or infrastructures

± Interfaces are different from classical HPC usage

- Might not be suitable for applications with very special requirements

24.03.2015 10



© CloudBroker GmbH

CloudBroker Solutions

• Goal:

– High Performance Computing Software as a Service (HPC SaaS)

• Products:

– CloudBroker Platform: SaaS and PaaS for compute-intensive applications on different infrastructures

– CloudBroker AppCenter: Web one-stop-shop for modeling, simulation and analysis software

• Services:

– Consulting, training and support with focus on cloud computing for scientific and technical applications

24.03.2015 11



© CloudBroker GmbH

CloudBroker

Platform

CloudBroker

AppCenter


24.03.2015 12


HPC

IaaS

PaaS

SaaS

Gate-

ways

Users


© CloudBroker GmbH

CloudBroker Platform

• Web-based application store and middleware for the deployment and execution of scientific and technical software on different computer infrastructures

• Uses IaaS from resource providers, offers PaaS for software vendors and SaaS for end users

• Management of users, resources, software and jobs

• Pricing, accounting, billing and payments

• Automation of software deployment, utilization of compute and storage resources, data transfer, job queuing and execution, etc.

• Users can use resources and software provided by others or register and deploy their own

• Browser, programmatic and command line access, can be used as frontend or backend, applications exposed as web service

• SSL transport layer encryption between all involved tiers

• Offered as public, hosted or in-house service or licensed software

24.03.2015 13



© CloudBroker GmbH

App-

Center

CloudBroker Platform

…

Platform Architecture

24.03.2015 14


External Tools

Java Client Library

HPC

Domain A

Applications

Domain B

Applications

Domain C

Applications

Web

Browser

UI

…

Applications

REST Web Service API

Clo

udB

roker In

tegra

tion

End Users, Software Vendors, Resource Providers

CLI

Open-

Nebula

Open-

Stack

Cloud

Sigma Amazon


© CloudBroker GmbH

Platform Screenshot

24.03.2015 15



© CloudBroker GmbH

Infrastructure CloudBroker User

Interaction of Tiers

24.03.2015 16


Client Plat-

form

Master

Node

Worker

Node 1

Worker

Node 2

Storage

Deploy

ment

App-

Center


© CloudBroker GmbH

Implemented Cloud Adapters

• Compute:

– Amazon EC2

– CloudSigma

– OpenStack EC2

– OpenStack Nova

– OpenNebula

– Eucalyptus

– HLRS NEC cluster

– … more to be added

• Storage:

– Amazon S3

– OpenStack S3

– OpenStack Swift

– Ceph RADOS

– Eucalytpus Walrus

– SCP

– … more to be added

24.03.2015 17



© CloudBroker GmbH

Application Software

• Application requirements:

– Scientific and technical

applications from any

domain, commercial or free

/ open source

– Linux or Windows

– Usually compute-intensive,

batch and command line-

oriented, but others are

possible

– Single-core or multi-core,

single-node or multi-node

(MPI, NFS, …)

• Application areas:

– Biology, pharma and

chemistry (Rosetta, BLAST,

AutoDock, Gromacs, …)

– Engineering and

manufacturing (ASCOMP

TransAT, Elmer,

OpenFOAM, Simul8, …)

– Generic (shell scripts,

R, …)

– … and others

24.03.2015 18



© CloudBroker GmbH

Platform Clients and APIs

Tools

• Tools from external providers

• Integration with the CloudBroker Platform via its APIs

• Examples: WS-PGRADE/gUSE, KNIME, ASCOMP TransAT

CLI

• Command Line Interface

• Job submission

• Open source at https://github.com/CloudBroker/cbp-cli

Java

• Application Programming Interface

• Java client library

• Open source at https://github.com/CloudBroker/cbp-java-api

REST

• Representational State Transfer

• Web service interface

• Documentation freely available on the CloudBroker website

24.03.2015 19


https://github.com/CloudBroker/cbp-cli



https://github.com/CloudBroker/cbp-java-api






© CloudBroker GmbH

Example Use Cases

• Protein modeling: – With IBM and ETH Zurich (CH)

– Using Rosetta on IBM IaaS cloud resources

– 36’000 jobs on up to 1’008 CPUs in parallel

– Almost ¼ Mio hours processing time in 2 weeks

– 2.3 Mio 3D structures of 249 proteins

• Heat transfer: – UberCloud HPC Experiment

– With Biscarri (ES), CAELinux (CH) and CSC (FI)

– Using Elmer CFD software on Amazon IaaS cloud resources

• Science gateways: – SCI-BUS EU FP7 project

– 15 partners from 10 countries, incl. 5 SMEs

– 6 subcontractors from 6 countries, incl. 1 SME

– Building in total 28 gateways for different communities

24.03.2015 20



© CloudBroker GmbH

CloudBroker

Platform

CloudBroker

AppCenter


24.03.2015 21


HPC

IaaS

PaaS

SaaS

Gate-

ways

Users


© CloudBroker GmbH

CloudBroker AppCenter

• User-friendly web one-stop-shop and marketplace to provide software products and services from software vendors to end users

• Single point of entry to different applications and technologies

• Originally developed for simulations within the CloudSME project

• Different options for end users to buy, retrieve and execute applications

• Different options for software vendors to price and provide subscriptions of applications

• Adapters to deliver software as download, web service or CloudBroker Platform deployment

• Accounting, billing and payments

• Browser and programmatic access, can be used as frontend or backend

• Offered as public, hosted or in-house service or licensed software

24.03.2015 22



© CloudBroker GmbH

AppCenter Architecture

24.03.2015 23


CloudBroker AppCenter

Web Browser

UI

…

Applications

REST Web Service

API

End Users, Software Vendors, Service Providers

Domain A

Applications

Download …

Adapter

Domain B

Applications

Web Service CloudBroker

Platform

Software …

Service

IaaS,

HPC

External Tools

Domain C

Applications

PaaS SaaS


© CloudBroker GmbH

CloudSME Use Case

• Project: – Cloud based Simulation

platform for Manufacturing and Engineering

– Supported by the EU FP7 Public-Private Partnership Factories of the Future program under grant agreement no. 608886

– Part of the EU I4MS (ICT Innovation for Manufacturing SMEs) initiative

– 07/2013-12/2015

– http://www.cloudsme.eu

• Objectives: – Give European SMEs

access to simulations

– Development of a SaaS and PaaS platform for manufacturing and engineering simulations

– On top of different cloud and HPC infrastructures

– One-stop-shop for users

• Collaboration: – 29 partners from 8

countries

– 24 SMEs and 5 academic institutions

24.03.2015 25


http://www.cloudsme.eu


© CloudBroker GmbH

CloudSME Setup

• Platform technology: – CloudBroker Platform and

AppCenter: CloudBroker (CH) & ScaleTools (CH)

– WS-PGRADE/gUSE: MTA SZTAKI (HU)

• Infrastructure resources: – Commercial clouds:

CloudSigma (CH) & Amazon (external)

– OpenStack clouds: University of Westminster (UK) & University of Zaragoza (ES)

– OpenNebula cloud: MTA SZTAKI (HU)

– HPC: HLRS (external)

• Project coordination: – University of Westminster (UK)

• Simulation consulting: – Brunel University (UK) &

University of Westminster (UK)

• Marketing and dissemination: – Sander Werbung (DE) &

University of Zaragoza (ES)

• Application experiments: – Collaborations of software

vendors and end users

– 4 initial application experiments plus 7 additional experiments from Open Call

24.03.2015 26



© CloudBroker GmbH

CloudSME Experiments

• Fluid dynamics: ASCOMP (CH) & Eurobios (FR)

• Discrete event simulation: SIMUL8 (UK) & Saker (UK)

→ Freight transport intermodal terminals: PROYFE (ES)

→ Craft brewing: Hobsons (UK)

• Insole design: Ingecon (ES) & Podoactiva (ES)

→ Safety shoes: Base Protection (IT) & IOR (IT)

• Aircraft maintenance: 2MoRO (FR)

• Fluid dynamics and finite element analysis: DHCAE (DE) & ProcEng (CH)

• Emission reduction: CMCL (UK) & G-volution (UK)

• Business process optimization: Simsoft (TR) & Özdekan (TR) & Gökdogan (TR)

• Inventory analysis and forcasting: Outlandish (UK) & Tidy Books (UK)

24.03.2015 28



© CloudBroker GmbH

Example: ASCOMP and

Eurobios • ASCOMP: Swiss company for fluid-

flow modeling solutions and

simulation tools with emphasis on

multi-phase systems

• Eurobios: French company

providing business simulation and

optimization solutions, dealing with

complexity

• TransAT: ASCOMP’s suite of

computational multi-fluid dynamics

tools for different industry branches

⇒ TransAT is a CFD application and

has HPC requirements

⇒ In CloudSME, ASCOMP has

enabled TransAT to be offered from

the cloud using the CloudBroker

Platform and AppCenter

⇒ TransAT works reasonably well on

Amazon IaaS cloud resources

⇒ However, HPC resources are

expected to deliver superior

performance

24.03.2015 29



© CloudBroker GmbH

Integration of HLRS

• High Performance Computing Center Stuttgart, Germany

• Collaboration with CloudSME through the Fortissimo project

• New adapter in the CloudBroker Platform to the HLRS NEC cluster

• Features:

– Selection between different node types

– Pay-per-use for resource usage possible

– Each user required to have an own HLRS account

⇒ In addition to IaaS clouds, CloudSME partners will be able to use HPC resources at HLRS seamlessly through the same interface

24.03.2015 30



© CloudBroker GmbH

Conclusions

⇒ HPC resources can be made available in the cloud on the

infrastructure level, but also offering them combined with

applications as a service in a marketplace makes HPC

attractive to a wider range of users

⇒ CloudBroker Platform and AppCenter provide a generic

framework and one-stop-shop to enable application software

as a service on top of different cloud and HPC infrastructures

⇒ The CloudSME project is an interesting use case how

simulations for manufacturing and engineering can be made

available to SMEs on top of cloud and HPC infrastructures

24.03.2015 31



© CloudBroker GmbH

Thanks! – Questions?

For more information:

Dr. Wibke Sudholt, [email protected]

Nicola Fantini, [email protected]

Acknowledgements:

CloudBroker, ScaleTools and CloudSME teams, HLRS

24.03.2015 32




Bringing HPC to the Cloud - HPC-AI Advisory Council …HPC in the Cloud Wish Lists • End users:...

Documents

Transcript of Bringing HPC to the Cloud - HPC-AI Advisory Council …HPC in the Cloud Wish Lists • End users:...