Run corporate applications on Google App Engine? Yes we do
Transcript of Run corporate applications on Google App Engine? Yes we do
![Page 1: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/1.jpg)
![Page 2: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/2.jpg)
"Run corporate applications onGoogle App Engine? Yes we do."Ben Fried, Google CIOJustin McWilliamsMatt SimmonsIrwin Boutboul2010-05-19
![Page 3: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/3.jpg)
View live notes and ask questions aboutthis session on Google Wave http://bit.ly/appengine2
![Page 4: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/4.jpg)
Introductions
![Page 5: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/5.jpg)
![Page 6: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/6.jpg)
What?
![Page 7: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/7.jpg)
Why?
![Page 8: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/8.jpg)
![Page 9: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/9.jpg)
![Page 10: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/10.jpg)
![Page 11: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/11.jpg)
![Page 12: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/12.jpg)
There's a better way
![Page 13: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/13.jpg)
Have to do things differently
![Page 14: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/14.jpg)
![Page 15: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/15.jpg)
![Page 16: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/16.jpg)
![Page 17: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/17.jpg)
IntroducingAppReduce,
Tech Stop Online,and CloudCourse
![Page 18: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/18.jpg)
AppReduce
Justin McWilliams, CorpEng Developer
![Page 19: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/19.jpg)
Reduce License Spend
![Page 20: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/20.jpg)
Design Iterations - LAMP
![Page 21: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/21.jpg)
Framework Familiarities
PythonDjango
Google Closure JavaScript Library
Google Chart API
Google Analytics
![Page 22: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/22.jpg)
Challenges
Accessing corp APIs and services
![Page 23: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/23.jpg)
SDC BackgroundSecure Data Connector (aka SDC) allows for a secure tunnel to Corp HTTP services.
![Page 24: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/24.jpg)
SDC: Wrap non-HTTP services
![Page 25: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/25.jpg)
Challenges
Accessing corp APIs and services
Realtime user performance
![Page 26: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/26.jpg)
Cache corp data with BulkloaderBulkloader makes it easy to import large amounts of datafrom anywhere into AppEngine.
![Page 27: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/27.jpg)
Design Iterations - GAE
![Page 28: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/28.jpg)
Results and Success
Putting information into employees' hands can be powerful$450K+ worth of returns to date$150K+ in first 24 hours after launch
50 QPS spike; 20 QPS sustainedBuilt, released and supported by only 2 engineers
![Page 29: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/29.jpg)
TechStop Online
Matthew Simmons, CorpEng Developer
![Page 30: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/30.jpg)
Tech Stop Online: What's in a name?
Home of internal Google IT support
“Self-powered”
![Page 31: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/31.jpg)
Tech Stop Online
Documentation repository
Managed LAMP -> App Engine
Django + App Engine + Closure
SDC, Task Queues, Bulk Loader
![Page 32: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/32.jpg)
Motivations and Challenges
Scalability
Search
Long running tasks
![Page 33: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/33.jpg)
What is "Bursty" demand?
![Page 34: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/34.jpg)
Scalability
Old platform:Scalability == Expensive
Cycles from applicationdevelopment "stolen" byplatform concerns.
![Page 35: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/35.jpg)
AppEngine: Scalable by default
It just works
What did we do with our extra time?
![Page 36: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/36.jpg)
Tech Stop Online: Navigation is king.
Problem: No native TextProperty indexing
SolutionsDIY indexingOffload to Google Search Appliance
![Page 37: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/37.jpg)
Search - Solution 1: DIY Indexing
![Page 38: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/38.jpg)
Search - Solution 2: GSA Indexing
IndexingProblem: content behind authenticationPush feed to GSA index over SDC
SearchingPull from GSA API over SDCDisplay results inline
![Page 39: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/39.jpg)
Using SDC
Secure Data Connector - http://code.google.com/securedataconnector
from google.appengine.api import urlfetch
result = urlfetch.fetch( url='http://corp.example.com/people.csv', headers={'use_intranet': 'yes'})
if result.status_code == 200: parseCSV(result.content)
![Page 40: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/40.jpg)
Computing Similarity
Goal: Automatically generate “related articles”
Idea: Use document x document similarity
Feasible, but...New documents need to be addedUpdates too much for a single request
![Page 41: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/41.jpg)
Task Queues
Defer expensive jobs to Task Queue
Write a handler like usual, but run it asynchronously
Gets the technology out of the way of the task.
![Page 42: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/42.jpg)
Flatten Your Data
Kiss BCNF goodbye!
Datastore denormalization
Use Reference properties where necessary
Watch out for reference properties in loops
It's OK to create "cached" copies of data on multiple models
![Page 43: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/43.jpg)
Testing Instances: The Easy Way
application: techstop-onlineversion: 1runtime: pythonapi_version: 1
handlers:- url: /script: main.py
application: techstop-online-qaversion: 1runtime: pythonapi_version: 1
handlers:- url: /script: main.py
![Page 44: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/44.jpg)
Testing Instances: Another Easy Way
application: techstop-online-qaversion: mattruntime: pythonapi_version: 1
handlers:- url: /script: main.py
application: techstop-online-qaversion: justinruntime: pythonapi_version: 1
handlers:- url: /script: main.py
![Page 45: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/45.jpg)
Summary
Scalable by default!
Fetch corporate data with SDC
Defer expensive jobs to Task Queue
Datastore denormalization
Set up a -qa instance and iterate away
![Page 46: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/46.jpg)
CloudCourse
Irwin Boutboul, CorpEng Developer
![Page 47: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/47.jpg)
1
2
Create an Activity
Schedule a Session
![Page 48: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/48.jpg)
3 Find a session
![Page 49: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/49.jpg)
Technology
App Engine / PythonDjango (w/ patch)ClosureGData
![Page 50: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/50.jpg)
Challenges
No joinsLong running computationHigh performance transactional system
![Page 51: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/51.jpg)
Challenges
No joinsLong running computationHigh performance transactional system
![Page 52: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/52.jpg)
Schema - Relational
Python 101 June 5th - MTVJune 6th - SFOJune 6th - Webcast
![Page 53: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/53.jpg)
Schema - non relational
exploding indices1 MB limitperformance / quota
Python 101[session1 .... sessionN][June 5th:MTV, ..., June 6th:VC]
![Page 54: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/54.jpg)
Schema - non relational
exploding indexes1 MB limitperformance / quotatrade-offs
![Page 55: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/55.jpg)
Challenges
No joinsLong running computationHigh performance transactional system
![Page 56: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/56.jpg)
Long running computation
Push burden of work to backend processesDefer tasks / transactionalTasks chaining / cron jobs
![Page 57: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/57.jpg)
Challenges
No joinsLong running computationHigh performance transactional system
![Page 58: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/58.jpg)
Schema - App Engine
![Page 59: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/59.jpg)
Enterprise constraints
Each signup is a transaction on an activity1-10 writes/sec per entity groupToo slow !
![Page 60: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/60.jpg)
Schema - App Engine
![Page 61: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/61.jpg)
Schema - App Engine
![Page 62: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/62.jpg)
Solution
memcache.incr(max_people)Conservative responseHigh level locks offline -> correct response
![Page 63: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/63.jpg)
Scalability
Online ~ 600 ms per enrollment
100+ writes per second for an activity
![Page 64: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/64.jpg)
Scalability
![Page 65: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/65.jpg)
Summary
De-normalization trade-offsAsynchronous processingLeverage memcache atomic ops
![Page 66: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/66.jpg)
Releases and Sandbox DemosCloudCourse
http://code.google.com/p/cloudcourse AppReduce
http://code.google.com/p/appreducehttp://code.google.com/p/gord
Techstop Online: coming soon
Visit our sandbox booth for demos and questions!
![Page 67: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/67.jpg)
View live notes and ask questions aboutthis session on Google Wave http://bit.ly/appengine2
![Page 68: Run corporate applications on Google App Engine? Yes we do](https://reader035.fdocuments.in/reader035/viewer/2022070222/613d0aa2736caf36b7589e4b/html5/thumbnails/68.jpg)