App Engine Dev Days DC 20091026

Google App EngineJonathan Blocksom

Stack Overflow DevDays DC, 10/26/09

Submit a Question: http://tinyurl.com/gae-dd09

Attention Conservation Notice• Jonathan Blocksom

[email protected]@jblocksom

Software EngineerGooglePublic Sector Projects Team


We’re lonely

Open positions at Google in DC


App Engine!• Launched April 2008

• Paid Use Feb 2009

• Java support May 2009

• Latest Release (1.2.7) October 2009

• Incoming Email

• Previous Release (1.2.5) September 2009

• XMPP Support


Your Scalable Website?• Brad Fitzpatrick describes LiveJournal in ’07:

http://danga.com/words/

LiveJournal Backend: Today(Roughly.)

User DB Cluster 1

uc1a uc1b

User DB Cluster 2

uc2a uc2b

User DB Cluster 3

uc3a uc3b

User DB Cluster N

ucNa ucNb

Job Queues (xN)

jqNa jqNb

Memcached

mc4

mc3

mc2

mcN

...

mc1

mod_perl

web4

web3

web2

webN

...

web1

BIG-IP

bigip2

bigip1perlbal (httpd/proxy)

proxy4

proxy3

proxy2

proxy5

proxy1

Global Database

slave1

master_a master_b

slave2 ... slave5

MogileFS Database

mog_a mog_b

Mogile Trackers

tracker3tracker1

Mogile Storage Nodes

...

sto2

sto8

sto1

net.

djabberd

djabberd

djabberd

gearmand

gearmand1

gearmandN

“workers”

gearwrkN

theschwkN

slave1 slaveN

3

2007


App Engine Scalable Website

App EngineRequest Response


What’s Scalable?• Google Moderator on App Engineused in

March 2009 for White House Town Hall meeting


Scalable


Google Moderator team didn’t...

• Provision a server

• Install and configure an OS

• Install and configure a web server

• Install and configure a web framework

• Install and configure a database

• Install and configure a load balancer

• Create user accounts

• Crash in the middle of any of the above


App Engine Architecture


Static Content• Served by Google

Static Content Servers

• Very Efficient


Dynamic Content• Java xor Python

• Sandboxed

• No file writes

• No sockets

• No spawning threads/processes

• Smart about caching

• Fairly standard CGI Framework

• Measured and Instrumented


Datastore 1 / 3• Based on BigTable,

transitioning to MegaStore

• ACID Compliant

• 2008 Talk on Implementation:http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore


Datastore 2 / 3• Typical types:

Int, String, DateTime, GeoPt, PhoneNumber, PostalAddress, Email, Link, Blob...


Datastore 3 / 3• GQL:

Google Query Language

• If it’s not SQL, it must be good!

SELECT [* | __key__] FROM <kind> [WHERE <condition> [AND <condition> ...]] [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]] [LIMIT [<offset>,]<count>] [OFFSET <offset>]

<condition> := <property> {< | <= | > | >= | = | != } <value> <condition> := <property> IN <list> <condition> := ANCESTOR IS <entity or key>


Memcache• Arbitrary key/value cache

• Similar API to memcached

• get, set

• get_multi, set_multi for speed

• Namespace Support

• 1MB limit per entry

• Free Quota 8.6M/day, 48K/min (800/sec)


Users & Authentication• @gmail.com addresses

• Apps for Domain

• Admin Privileges


URL Fetch• Load URL

• Asynchronous support

• HTTP / HTTPS

• Max 10 second response (default 5)

• Max 1MB data


Even More...• Scheduled Tasks

• Task Queue

• Email

• Image Manipulation

• XMPP

• ...

cron jobs

email

XMPP


Web Hooks Pattern• Incoming data / operations to perform are

URLs with POST data

• Handled same asweb request

Scheduled Tasks

Task Queue tasks

Incoming Email

XMPP Messages


Templates?• Treetop Wants to know


Templates!• Django Templates built in

Variable substitution, function calls, loops, etc

• Can add your own Python temlpate lib


Let’s Review!


Developing w/ App Engine• Local test server

• Write your code, html, etc on local machine

• Send code to App Engine

• Monitor in App Engine Admin Console


Dashboard Demo• http://appengine.google.com/


What have we done?• Served static web pages

• Served dynamic web pages

• Stored data (cached & persistent)

• Ran a cron Job

• Authenticated users

• We could have...

• Scheduled a task, sent or received email, loaded a URL, modified an image,rocked out with XMPP


Quotas• 34 global quotas

in Dashboard

• Individual quotas:request sizetimeamount of data

• Exceptions thrown when hit

• Free and Paid


Quota Rule of Thumb• Less than 1000 of everything

• The Zen of Quotas:Meet the quotas and you will be scalable


All For Good• Volunteering Opportunity Search Engine

• Started March 2009 in cooperation with White House, CNCS, Craigslist, Google...

• Released June 2009

• Open source, released under Apache Licensehttp://code.google.com/p/allforgood/

• Front end at http://allforgood.orgAPI powers search results on serve.gov


Tech Overview• Partners provide listings in XML or Google Spreadsheets

• AARP, United Way, AmeriCorp, HandsOnNetwork, United Way

• AFG indexes feeds, stores outside GAE

• Initially Google Base

• Now SOLR on private server

• GAE front end

• API for external apps, partners

• Integration with Google Maps, FriendConnect, Facebook


Tech Overview

App Engine

Search Backend(Base or SOLR)

memcache

Google Code SVN

Volunteering Opportunities

Volunteering Opportunities

Volunteering OpportunitiesCrawler

...Resu

lts

Queries

Listings

Static Content

User or API


Why A Different Back End?• No full text search in App Engine Datastore


On The Promiscuous Nature of GAE

• GAE plays well with others

• Do full text search with something else!

• Serve large files with S3!

• Serve videos with YouTube!


Challenges with AFG• Parsing questionable data

• Integration w/ Social Networks

• Google Base is an Odd Duck

• I Was New And Had No Clue What Was Going

• App Engine Outages


Q & A


More Information• App Engine Documentation

• Google I/O 2008 & 2009

• 2008 for intro, 2009 for deep dives / Java

• http://code.google.com/events/io/2009/sessions.html#appengine“App Engine Nitty-Gritty”“Spark Plug to Drive Train”

• Google Group

• Sampleshttp://code.google.com/p/google-app-engine-samples/

App Engine Dev Days DC 20091026

Technology

Transcript of App Engine Dev Days DC 20091026