15/04/2023

Cloud for Research and Innovation - UK USA HPC Workshop July 2015 1

Cloud for Research and InnovationMartin Hamilton

Photo CC BY Flickr user atosorigin

Cloud for research and innovation

1. Cloud adoption in enterprise IT– Trends in UK higher education– Bimodal IT versus bipolar institutions

2. Public clouds are starting to look more like HPC clusters– Orchestration, Infiniband, GPU and FPGA availability (at a price)

3. Code re-use and app portability– Will Docker and OpenStack be our Rosetta Stone(s)?– What does a unified computing model look like?

15/04/2023

Cloud for Research and Innovation - UK USA HPC Workshop July 2015 3

1. Cloud adoptionin enterprise IT

Cloud in UK higher education

Collaboration suites: Google Apps for Education» The University of Westminster estimates that it saved £1m by moving to

Google Apps for Education for email and other online collaboration facilities. They also state that “an additional benefit has been the reduced time spent in systems and user support with a minimal number of calls for support for such a significant system”, and that moving to the cloud has “liberated staff and students from the smaller storage limits of the previous in-house solution.”

http://services.google.com/fh/files/misc/univofwestminster.pdf

» Ealing, Hammersmith and West London College have also migrated to Google Apps. They observe that “each year, 20,000 new accounts are created providing 600 terabytes (around 150,000 DVDs worth) of storage in less than five minutes.” Subsequently to this article being written, Google moved to offering free unlimited storage.

http://www.wlc.ac.uk/college/news/article.asp?newsID=4228

Cloud in UK higher educationCollaboration suites: Microsoft Office365» The University of the West of London moved its 14,000 students to Office365, and

observes that once a student graduates “they can maintain that Office365 account through the University for life. We can continue to supply them with information about the University, help them find their second job, their third job and so on. This isn’t just about collaboration while studying, it’s about creating a life-long connection between the students and the University.”

http://www.slideshare.net/Microsofteduk/university-of-west-london-case-study-office-365

» The University of Dundee migrated all of its 22,000 student accounts to Microsoft’s Office365 for Education service over a week. They note that this was achieved with zero down time, and estimate that Office365 “will save us at least £500,000 in infrastructure and staffing over five years.”

http://www.slideshare.net/Microsofteduk/university-of-dundee-case-study-office-365

Cloud in UK higher education

Jisc Amazon cloud portal:› Monthly invoicing - credit cards are no

longer required for payment

› Itemized billing - consolidated across users/departments

› Billing in GBP, not dollars

› Setting of budget limits for individual user accounts or departments

› The retrieval of service usage information within own areas of responsibility

› Volume-discounts through aggregation across multiple educational institutions

Cloud in UK higher education

Leveraging the Janet network:

Cloud in UK higher education

Private peerings:

Cloud in UK higher education

Private peerings:

Cloud in UK higher education

From www.jasmin.ac.uk

Cloud in UK higher education

Dogfooding:› We moved

our own core infrastructure to AWS

› Estimate that this saved £100K p.a.

› Cloud first / mobile first for new stuff

› NB we are not an HPC provider

15/04/2023

Cloud for Research and Innovation - UK USA HPC Workshop July 2015 12

2. Public clouds arestarting to look like HPC clusters

Case study: Azure HPC

Case study: Azure HPC

Case study: Azure HPC

Case study: Azure HPC

http://azure.microsoft.com/blog/2015/07/09/azure-linux-rdma-hpc-available/

Case study: Azure HPC

Case study: Azure HPC

Case study: Azure HPC

Case study: Azure HPC

Case study: Google Compute Engine

Case study: AWS

Case study: AWS

Caveat Emptor: Magellan

http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/Magellan_Final_Report.pdf

Caveat Emptor: Magellan

http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/Magellan_Final_Report.pdf

15/04/2023

Cloud for Research and Innovation - UK USA HPC Workshop July 2015 26

3. Code re-use &application portability

Re-use & portability: OpenStack

http://www.datacenterknowledge.com/archives/2014/11/07/cerns-openstack-cloud-to-reach-150000-cores-by-2015/

Re-use & portability: OpenStack

http://www.datacenterknowledge.com/archives/2014/11/07/cerns-openstack-cloud-to-reach-150000-cores-by-2015/

http://www.datanami.com/2013/09/23/cern_turns_to_google_for_datacenter_direction/

Re-use & portability: OpenStack

Re-use & portability: OpenStack

Re-use & portability: Containers

Re-use & portability: Containers

https://wiki.openstack.org/wiki/Docker

Re-use & portability: Performance

http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2014/11/04/containers-docker-virtual-machines-and-hpc

Re-use & portability: Open Container Project

Re-use & app portability: Kubernetes & Heat

http://kubernetes.io http://techs.enovance.com/7104/multi-tenant-docker-with-openstack-heat

Re-use & app portability: A unified model?

This is one model. Other models are available › Run OpenStack on your in-house facility [Q: Still need one?]

– This creates your own private/community cloud– Machine images can be programmatically converted to/from QCOW2

› Package your app with Docker and you can run it anywhere– Embraced by public cloud providers, can run in-house too as open source– Could be bare metal or on top of e.g. OpenStack [Q: Performance?]

› Lots of options for orchestration from general purpose uses– Kubernetes scheduling particularly interesting from an HPC perspective

› Open questions from an e-Infrastructure perspective:– Integrating identity management, authentication, authorization, accounting– User driven (instant gratification) versus controlled environment– Funding model for use of public and community clouds, potential for national deals– Lots of new stuff to learn about! (for developer, sysadmin, …)– That’s all fine for the compute, but what about the data?

Photo CC-BY Flickruser codep8

15/04/2023

Cloud for Research and Innovation - UK USA HPC Workshop July 2015 37

That’s all folks…

Except where otherwise noted, this work is licensed under CC-BY

Martin HamiltonFuturist, Jisc, London

@martin_hamiltonmartin.hamilton@jisc.ac.uk