Google App EngineJonathan Blocksom

Stack Overflow DevDays DC, 10/26/09

Submit a Question: http://tinyurl.com/gae-dd09

Attention Conservation Notice• Jonathan Blocksom

jblocksom@google.com@jblocksom

Software EngineerGooglePublic Sector Projects Team

Submit a Question: http://tinyurl.com/gae-dd09

We’re lonely

Open positions at Google in DC

Submit a Question: http://tinyurl.com/gae-dd09

App Engine!• Launched April 2008

• Paid Use Feb 2009

• Java support May 2009

• Latest Release (1.2.7) October 2009

• Incoming Email

• Previous Release (1.2.5) September 2009

• XMPP Support

Submit a Question: http://tinyurl.com/gae-dd09

Your Scalable Website?• Brad Fitzpatrick describes LiveJournal in ’07:

http://danga.com/words/

LiveJournal Backend: Today(Roughly.)

User DB Cluster 1

uc1a uc1b

User DB Cluster 2

uc2a uc2b

User DB Cluster 3

uc3a uc3b

User DB Cluster N

ucNa ucNb

Job Queues (xN)

jqNa jqNb

Memcached

mc4

mc3

mc2

mcN

...

mc1

mod_perl

web4

web3

web2

webN

...

web1

BIG-IP

bigip2

bigip1perlbal (httpd/proxy)

proxy4

proxy3

proxy2

proxy5

proxy1

Global Database

slave1

master_a master_b

slave2 ... slave5

MogileFS Database

mog_a mog_b

Mogile Trackers

tracker3tracker1

Mogile Storage Nodes

...

sto2

sto8

sto1

net.

djabberd

djabberd

djabberd

gearmand

gearmand1

gearmandN

“workers”

gearwrkN

theschwkN

slave1 slaveN

3

2007

Submit a Question: http://tinyurl.com/gae-dd09

App Engine Scalable Website

App EngineRequest Response

Submit a Question: http://tinyurl.com/gae-dd09

What’s Scalable?• Google Moderator on App Engineused in

March 2009 for White House Town Hall meeting

Submit a Question: http://tinyurl.com/gae-dd09

Scalable

Submit a Question: http://tinyurl.com/gae-dd09

Google Moderator team didn’t...

• Provision a server

• Install and configure an OS

• Install and configure a web server

• Install and configure a web framework

• Install and configure a database

• Install and configure a load balancer

• Create user accounts

• Crash in the middle of any of the above

Submit a Question: http://tinyurl.com/gae-dd09

App Engine Architecture

Submit a Question: http://tinyurl.com/gae-dd09

Static Content• Served by Google

Static Content Servers

• Very Efficient

Submit a Question: http://tinyurl.com/gae-dd09

Dynamic Content• Java xor Python

• Sandboxed

• No file writes

• No sockets

• No spawning threads/processes

• Smart about caching

• Fairly standard CGI Framework

• Measured and Instrumented

Submit a Question: http://tinyurl.com/gae-dd09

Datastore 1 / 3• Based on BigTable,

transitioning to MegaStore

• ACID Compliant

• 2008 Talk on Implementation:http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore

Submit a Question: http://tinyurl.com/gae-dd09

Datastore 2 / 3• Typical types:

Int, String, DateTime, GeoPt, PhoneNumber, PostalAddress, Email, Link, Blob...

Submit a Question: http://tinyurl.com/gae-dd09

Datastore 3 / 3• GQL:

Google Query Language

• If it’s not SQL, it must be good!

SELECT [* | __key__] FROM <kind> [WHERE <condition> [AND <condition> ...]] [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]] [LIMIT [<offset>,]<count>] [OFFSET <offset>]

<condition> := <property> {< | <= | > | >= | = | != } <value> <condition> := <property> IN <list> <condition> := ANCESTOR IS <entity or key>

Submit a Question: http://tinyurl.com/gae-dd09

Memcache• Arbitrary key/value cache

• Similar API to memcached

• get, set

• get_multi, set_multi for speed

• Namespace Support

• 1MB limit per entry

• Free Quota 8.6M/day, 48K/min (800/sec)

Submit a Question: http://tinyurl.com/gae-dd09

Users & Authentication• @gmail.com addresses

• Apps for Domain

• Admin Privileges

Submit a Question: http://tinyurl.com/gae-dd09

URL Fetch• Load URL

• Asynchronous support

• HTTP / HTTPS

• Max 10 second response (default 5)

• Max 1MB data

Submit a Question: http://tinyurl.com/gae-dd09

Even More...• Scheduled Tasks

• Task Queue

• Email

• Image Manipulation

• XMPP

• ...

cron jobs

email

XMPP

Submit a Question: http://tinyurl.com/gae-dd09

Web Hooks Pattern• Incoming data / operations to perform are

URLs with POST data

• Handled same asweb request

Scheduled Tasks

Task Queue tasks

Incoming Email

XMPP Messages

Submit a Question: http://tinyurl.com/gae-dd09

Templates?• Treetop Wants to know

Submit a Question: http://tinyurl.com/gae-dd09

Templates!• Django Templates built in

Variable substitution, function calls, loops, etc

• Can add your own Python temlpate lib

Submit a Question: http://tinyurl.com/gae-dd09

Let’s Review!

Submit a Question: http://tinyurl.com/gae-dd09

Developing w/ App Engine• Local test server

• Write your code, html, etc on local machine

• Send code to App Engine

• Monitor in App Engine Admin Console

Submit a Question: http://tinyurl.com/gae-dd09

Dashboard Demo• http://appengine.google.com/

Submit a Question: http://tinyurl.com/gae-dd09

What have we done?• Served static web pages

• Served dynamic web pages

• Stored data (cached & persistent)

• Ran a cron Job

• Authenticated users

• We could have...

• Scheduled a task, sent or received email, loaded a URL, modified an image,rocked out with XMPP

Submit a Question: http://tinyurl.com/gae-dd09

Quotas• 34 global quotas

in Dashboard

• Individual quotas:request sizetimeamount of data

• Exceptions thrown when hit

• Free and Paid

Submit a Question: http://tinyurl.com/gae-dd09

Quota Rule of Thumb• Less than 1000 of everything

• The Zen of Quotas:Meet the quotas and you will be scalable

Submit a Question: http://tinyurl.com/gae-dd09

All For Good• Volunteering Opportunity Search Engine

• Started March 2009 in cooperation with White House, CNCS, Craigslist, Google...

• Released June 2009

• Open source, released under Apache Licensehttp://code.google.com/p/allforgood/

• Front end at http://allforgood.orgAPI powers search results on serve.gov

Submit a Question: http://tinyurl.com/gae-dd09

Tech Overview• Partners provide listings in XML or Google Spreadsheets

• AARP, United Way, AmeriCorp, HandsOnNetwork, United Way

• AFG indexes feeds, stores outside GAE

• Initially Google Base

• Now SOLR on private server

• GAE front end

• API for external apps, partners

• Integration with Google Maps, FriendConnect, Facebook

Submit a Question: http://tinyurl.com/gae-dd09

Tech Overview

App Engine

Search Backend(Base or SOLR)

memcache

Google Code SVN

Volunteering Opportunities

Volunteering Opportunities

Volunteering OpportunitiesCrawler

...Resu

lts

Queries

Listings

Static Content

User or API

Submit a Question: http://tinyurl.com/gae-dd09

Why A Different Back End?• No full text search in App Engine Datastore

Submit a Question: http://tinyurl.com/gae-dd09

On The Promiscuous Nature of GAE

• GAE plays well with others

• Do full text search with something else!

• Serve large files with S3!

• Serve videos with YouTube!

Submit a Question: http://tinyurl.com/gae-dd09

Challenges with AFG• Parsing questionable data

• Integration w/ Social Networks

• Google Base is an Odd Duck

• I Was New And Had No Clue What Was Going

• App Engine Outages

Submit a Question: http://tinyurl.com/gae-dd09

Q & A

Submit a Question: http://tinyurl.com/gae-dd09

More Information• App Engine Documentation

• Google I/O 2008 & 2009

• 2008 for intro, 2009 for deep dives / Java

• http://code.google.com/events/io/2009/sessions.html#appengine“App Engine Nitty-Gritty”“Spark Plug to Drive Train”

• Google Group

• Sampleshttp://code.google.com/p/google-app-engine-samples/