Google Compute Engine Google Cloud Storage

Evan Anderson, GCE Tech Lead (VM Environment) https://cloud.google.com/

RightScale : Compute 25 April 2013

The Plan

•  Overview of Google Cloud Platform

•  Google Compute Engine

•  Google Cloud Storage

•  Questions / ad hoc demos

Overview of Google Cloud Platform

Google Infrastructure

Performance

Redundancy

Disaster Recovery

Audits & Certifications

Security

Energy Efficient

Global Data Centers

99.95% Uptime SLA

Compute Storage Services

Compute Engine (IaaS) App Engine (PaaS)

Cloud Storage (Object) Cloud SQL (Relational)

BigQuery Cloud EndPoints Caching Queues and more...

Cloud Platform Google Services

Cloud Based Analytics Platform

Store

Backends + MapReduce

Extract & Transform

Hadoop

BigQuery

API SQL

Analyze interactively Serve

Logstore

Cloud Storage

Datastore

Log data

Unstructured data

Structured data Interactive Dashboards + apps

Application level code

Custom logic & 3rd party libraries

BI tools

Google Spreadsheets

Overview of Google Compute Engine

Google Compute Engine •  Run almost any application on Google's infrastructure

•  Offers virtual machines, storage, networking

•  Allows services to scale efficiently with little work

Google Compute Engine in pictures

•  tools, UI & API

•  project

•  instances

•  network &

firewalls

•  storage

Google Compute Engine in pictures

gcutil

APIs Console

Google Client libraries

Google Compute Engine in pictures

cloud.google.com/console

UI is built on AppEngine. •  Uses public GCE APIs! •  No special sauce!

Google Compute Engine in pictures

Linux VMs •  Modern CPU •  1, 2, 4, 8 CPUs •  3 memory sizes

(RAM per CPU): o  standard: 3.75GB o  highcpu: 0.9G o  highmem: 6.5G

•  Root access

Google Compute Engine in pictures

External Network •  External IPs •  Firewalls Private Network •  Isolated at the project level •  Internal DNS

Instance name ⇄ DNS name

Google Compute Engine in pictures

Storage options: •  Persistent •  Scratch •  Cloud Storage Or: •  App Engine •  Google Cloud SQL •  ... anything you can

connect to! (Cassandra, MongoDB...)

•  Consistent CPU, network, and persistent disk performance.

•  Dedicated scratch spindles for 4-cpu and 8-cpu VMs.

•  Private networks for each project that span zones and regions.

•  All data encrypted at rest.

•  Building a consistent foundation on Google's infrastructure:

o  So you don't have to worry about VM provisioning or location!

Strong Isolation of Virtual Machines

Frictionless Oauth2 with Service Accounts

•  Frictionless Access to Google APIs

•  Synthetic, managed identity for VMs and code – no passwords required!

•  Google Compute calling Google APIs

o  Examples: Cloud Storage, App Engine task queue API

•  App Engine calling Compute Engine API

o  Use App Engine as 'orchestrator'

o  Build your own customized dashboard and control logic

(since Google I/O)

What's new in Google Compute Engine

First, one thing that's not new...

•  Google Compute Engine is currently in Beta.

•  It's used by dozens of Google projects internally.

•  Recommended to build a multi-zone, multi-server architecture.

•  GCE scheduled maintenance:

o  Up to 2 weeks every 20 weeks, one zone at a time.

•  RightScale's management layer can migrate servers across zones.

•  RightScale simplifies building infrastructure on GCE!

•  For more memory relative to virtual cores, use high memory machine types.

•  For more virtual cores relative to memory, use high CPU machine types.

•  Machine types without scratch disk are discounted relative to scratch disks.

New: High memory and high CPU machine types

•  Add ability to hot-add and hot-remove external IPs from a running VM.

•  Even across zones!

•  Most useful with static external IPs.

New: Dynamically move external IPs

•  Add and remove persistent disks to a running VM.

•  Create a persistent disk from a stored disk image.

•  Launch a VM from a persistent root disk.

•  The same persistent disk can be attached read-only to multiple VMs.

New: More Flexibility with Persistent Disks

•  Create snapshots of existing persistent disks and apply them to new disks.

•  Works across zones and regions!

•  5GiB of data on 100GiB persistent disk == about 5GiB snapshot

•  https://developers.google.com/compute/docs/disks#snapshots

•  So, snapshots can be used to snapshot a read-write disk, create a new disk,

and mount the new disk on N instances in read-only mode!

New: Persistent Disk Snapshots

•  Added a --format flag for the list subcommands.

•  The flag accepts the following values: table, sparse, json, csv, and names.

•  --format=names allows gcutil to be used with Unix tool pipelines:

o  gcutil  listinstances  -­‐-­‐format=names  |  \  

   xargs  gcutil  deleteinstance  -­‐-­‐force  

New: Better command line scripting: gcutil --format

New: more new stuff!

•  Added new European zones.

o  europe-west1-a

o  europe-west1-b

•  Instance Cloning and Move Between Zones

•  Serial console output from a VM instance

•  Updated images & kernels a few times

Demos!

gce-demos.appspot.com/quick-start

Command line

$  gcutil  addinstance  INSTANCE_NAME  -­‐-­‐machine_type=n1-­‐standard-­‐1  -­‐-­‐zone=us-­‐central1-­‐a    $  gcutil  ssh  INSTANCE_NAME    <do  work  on  the  instance>  <...>  <later>    $  gcutil  deleteinstance  INSTANCE_NAME  

By the way... These demos are App Engine apps!

Stories from the field (We love our partners!)

MapR Terasort Record

MapR World Record Previous Record

Sort Time (s) 54 62

Number of Servers 1003 1460

Number of Cores 4012 11680

Number of Disks 1003 5840

Time to Build Cluster Minutes Months

•  Sorting depends on the speed of the slowest node.

•  4-CPU and larger nodes have a dedicated spindle.

•  High network bandwidth between nodes.

o  No need to play games for switch or rack placement.

•  CPU and memory isolation based on linux cgroups.

o  (Developed at Google to solve performance consistency!)

Consistency is the key to speed in distributed systems

RightScale and Google to-date: •  Complete integration with Google Compute Engine

•  Customers already running in beta

RightScale and Google are better together! •  Combine the speed of GCE with the flexibility of RightScale to provision entire environments in minutes

•  Reproduce existing deployments for development and testing with the click of a button

•  Generate rich media and push to outlets like YouTube, G+, and other Google Properties without being charged for bandwidth!

RightScale Seamless GCE Integration

High-end networking

Geo-Distributed

Very high Reliability

Strong performance

Governance

Multi-Cloud

One-click Deployment

Templatized Workload Low Cost

Automation

Better Together

Google Cloud Storage

It's an object store. AKA: blobstore, key-value store

•  Strong (read-after-write) data consistency

•  Streaming uploads and resumable transfers, range read support

•  Objects can be terabytes in size, with resumable uploads and downloads,

and range read support.

•  Serve static data via HTTP directly from Cloud Storage.

•  OAuth 2.0 or interoperable authentication

•  Signed URLs allow delegating access to non-authenticated users.

Overview

•  We replicate to data centers spanning multiple geographically diverse cities

o  reducing the risk of natural disasters

•  We can tolerate hardware failures to any part of the system

o  disk drives, servers, network routers, data centers, network links, etc

•  Data is delivered via Google's worldwide network

o  same network as YouTube, Gmail, etc

•  We exercise a deep set of system tests before new releases

•  Regular monitoring

•  99.9% Service Level Agreement

o  https://developers.google.com/storage/docs/sla

Reliable and available by design

•  Scalable...

o  requests, objects, GiB

o  scales down well - affordable, robust infrastructure

  Geographically redundant hosting for a single web page? Sure!

•  Cloud Storage is the glue between all Google Cloud Services

o  BigQuery, Compute Engine, App Engine, etc...

•  Reliable, available, durable...

Why should you care?

•  Durable Reduced Availability storage class (lower cost storage for backups)

•  Static website hosting

•  Cross-origin resource sharing for web and mobile clients

•  Integration with App Engine (new GCS Client Library)

•  Detailed permissions settings

•  Access logs, enabling audit of storage activity

•  Short-lived signed URLs for delegated download or upload

Continuous Innovation: recently added features...

•  Cloud Storage is an HTTP service

o  serve data directly over HTTP

o  APIs for all Cloud Storage features

•  Our general purpose tools are built on these APIs

•  The exact same APIs are used by all of our developers and partners

•  XML API is compatible with APIs from similar services: AWS S3, Eucalyptus

HTTP APIs (interoperable XML and Google standard JSON)

RightScale Google Offer

•  GCE access is available through RightScale today

•  RightScale usage charges are waived for the first year for new customers

•  Get access to the Google Compute for a low flat rate through RightScale

RightScale Offer