OpenStack for academic use-cases - HPC Advisory Council · UZH/S3IT OpenStack for academic...

S3IT: Service and Support for Science IT

OpenStack for academicuse-cases ?Sergio MaffiolettiS3IT: Service and Support for ScienceIT,University of Zurich

22.03.2016

What is S3IT and who am I

S3IT - Service and Support for ScienceIT:http://www.s3it.uzh.ch.

Enabling research through Science IT competences.

S3IT is an academic solution provider (end-to-end).

Sergio MaffiolettiSenior Infrastructure and Application specialist atUniversity of Zurich.

UZH/S3IT OpenStack for academic use-cases 22.03.2016

http://www.s3it.uzh.ch

Overall message

Build and adaptinfrastructure to use-cases

and not the other way round.


Let’s start with someexamples


Research needs to do more than just dataanalysis.

Personalized server

Large-scale data analysis and simulations

Integrate solution



Personalized server→ RStudio/R Shiny→ Windows image analysis


Integrate solution



Personalized server


→ Hadoop/Spark→ Own SLURM cluster→ Matlab and toolboxes

Integrate solution



Personalized server


Integrate solution

→ from instrument to store, to processing to makedata accessible.


What is OpenStack ?


What is OpenStack and how we use it

→ Self-provisioning of Virtual Instances, Storage andNetwork→ Elastic allocation of resources on demand→ Multi-tenancy


OpenStack project

OpenSource projectask our sysadmin why this is important. . .

Foundation supported by more than 200 companies.https://www.openstack.org/foundation/companies/

Network APIs→ Amazon compute and storage cloud compliant→ Bindings for many languages


https://www.openstack.org/foundation/companies/

Let’s implement some of theexamples


What approach ?

→ Provision, Configure and Orchestrate basecomponents: network, storage, compute.→ Build on top of baseline infrastructure.→ Customize and tailor to use-cases needs.


Personalized server: R Shiny


Personalized server: R Shiny

1. Create own instance

2. Automated customization (e.g. ansible)

3. Application-specific settings

4. Save base image for further use


R Shiny: Create own instance

Instances are virtual machines that run inside the cloud.


R Shiny: Automated customization (e.g.ansible)

Ansible is a free-software configuration managementplatform for configuring and managing computers.


https://www.ansible.com/

R Shiny: Application-specific settings

Security groups are sets of IP filter rules that are applied toan instance’s networking.


R Shiny: Save base image for further use

The OpenStack snapshot mechanism allows to create newimages from running instance.


Create your own batch cluster: SLURM



1. Provision cluster instances (1 frontend + Xcompute)

2. Create own data volume and attach it to frontend

3. Install and configure frontend and compute

4. Export via NFS data volume to all compute nodes

5. Start NFS and slurm services



. . . or use elasticluster: http://gc3-uzh-ch.github.io/elasticluster/

https://youtu.be/gkXkcHDd588


http://gc3-uzh-ch.github.io/elasticluster/

https://youtu.be/gkXkcHDd588

Why flexibility: from our perspective

→ Reduce total time to get actual results→ Allow to run on multiple infrastructures(portability)→ Automate (more research, less development forend-user)


Why flexibility: from user perspective

Self-provisioning and Elasticity of resources

Customization and control of the environment

Network API



Self-provisioning and Elasticity of resources→ End-users can allocate and release resourceswhen needed.


Network API




Customization and control of the environment→ End-users can tailor the research infrastructureto their specific needs.

Network API





Network API→ To programmatically create and control an ownresearch infrastructure.→ Build services on top


Downsides

→ Resource allocation: policy is basically done bysupport team.→ Efficient use of resources: users need to takeresponsibility.→ Need to have strong support team to complementthe offer to end-users.→ Ops costs comparable as for regular batch clusterbut know-how has to be developed.→ Storage solution has to be planned separately andmostly depends on use-cases.


Conclusions

→ Cloud is flexible: build and adapt infrastructure touse-cases and not the other way round.→ Users have more possibilities but also moreresponsibilities.→ Easier for end-users to think and plan theirresearch infrastructure.


OpenStack for academic use-cases - HPC Advisory Council · UZH/S3IT OpenStack for academic...

Documents

Transcript of OpenStack for academic use-cases - HPC Advisory Council · UZH/S3IT OpenStack for academic...