Enabling Automated HPC / Database Deployment via the

AppScale Hybrid Cloud Platform

Chris Bunch Chandra Krintz

HPCDB 2011 @ Seattle, WANovember 18, 2011

Cloud Computing

• Three tiers of abstraction:

• Infrastructure: Scalable hardware

• Platform: Scalable APIs

• Software: Scalable application

HPC in the Cloud

• Easy access to vast resources

• Hard to automatically configure and deploy libraries

• Requires in-depth knowledge of each technology involved

• Hard to get high performance on opaque cloud

• Wide range of APIs for similar services (e.g., compute, storage)

Solving these Problems

• Need a solution that is automated

• Minimize the amount of domain-specific knowledge a scientist has to learn

• Need to automate configuration and deployment

• Hard since the number of nodes used can be dynamic in the cloud

Cloud Computing

• Infrastructure: Completely self-service

• Platform: Automated access to certain APIs

• Software: Optimized for one piece of software

APIs to Support

• To support scientists, we need to support:

• Web services

• HPC codes

• Databases

• And it must be simple

Introducing AppScale

• An open source implementation of the Google App Engine APIs

• Can run over Amazon EC2 or Eucalyptus

• Configures and deploys automatically

• Scientist only needs to specify the number of nodes to run over

One Button Deployment

•appscale-run-instances

• --max 32

• --infrastructure ec2

AppScale + Neptune

• A domain-specific language for running HPC codes

• Supports codes written in:

• MPI, UPC, X10, and others

• Configures and deploys automatically

• Scientist only needs to specify the number of nodes to run over

One Command Deployment

•neptune :type => :mpi,

• :code => “/code/ring”,

• :nodes_to_use => 32,

• :output => “/output/ring”

Database Support

• Support for a dozen datastores, including:

• Neptune can also store to Amazon S3

• Or anything that uses the same APIs

• Completely automated

Hybrid Clouds

• The utilization of more than one cloud

• Fault tolerance: Run an app in multiple clouds

• Portability: Migrate an app from one cloud to another

• Adds another layer of complexity

Open Hybrid Clouds

• AppScale and Neptune both support automated hybrid cloud deployment

• Just provide your credentials

• Would like to expand both with capabilities for special use cases

• Quantify the cost of each use case

Government Use Case

• Compute anywhere, store only in a private, trusted cloud

App Server

App Server & Master

DB Node

DB Node

Cloud 1 - EC2 East Coast Cloud 2 - Private Eucalyptus Deployment

One DB in Each Cloud

• Compute anywhere, store in the same cloud

DB Node

App Server & Master

App Server

DB Node

Cloud 1 - EC2 East Coast Cloud 2 - Private Eucalyptus Deployment

Wrapping it Up

• Thanks to the AppScale team, especially co-lead Navraj Chohan and advisor Chandra Krintz

• AppScale 1.5 released in August

• Neptune 0.1.2 released last week

• Visit us at http://appscale.cs.ucsb.edu

• E-mail: cgb@cs.ucsb.edu