Post on 05-Dec-2014
description
Andrew MackenzieDaniel RuizBCNTouchhttp://ta.bcntouch.com
Google App Engine & Google Web Toolkitour experience developing TouchActive
The purpose of this talk
● Share our experiences using GAE and GWT
● Help guide you if you are considering similar technologies
● Provide ideas if you are already developing with these technologies
● Provide feedback to Google on things to maintain, improve and renew/drop
Contents
● Solution● PaaS choice● GAE overview● GWT overview● GAE + GWT debugging● TouchActive
implementation● Addit● Optimization tools● Phonegap experiment● Avoiding lock-in● What would we do
differently● Feedback to Google
Our Project
GAE + GWT
Implementation
Summary
Feedback
Our Project
TouchActive Platform- what we built
A platform to put consumers and
brands in touch with each other
Brand
Consumer
Using tagged products that the consumers were
interested in
Mobile apps to read tagged products- NFC, QR, barcode
Cloud service and data storage
Mobile apps read products tagged
with various technologies
Scalable store in the cloud with product
data and synchronization of
user data
Get product information, help
remember, organize in lists, etc.
Focus on easy and intuitive interaction with no data entry. NFC the best.
Get information about productsfrom tagged objects
Management console for brand- multi-tenant, groups, roles
Cloud service and data storage
Access control on changes
Products findable in open queries from
mobile devices
Consumer Web Interface- Lists, Products, History, etc
Cloud service and data storage
Mobile web version for mobile platforms without a native app
User data synchronized with native mobile app
Addit "Web Clipper" to capture products from web pages
Social networks, Identity and Sharing
Leverage existing user identities for
easy sign-up (web and mobile)
- OpenId and OAuth
Sharing of Products, Lists via:
● Social Networks● EMail● NFC Tags● NFC peer-to-peer● QR codes● Bump
TouchActive Solution Overview- what we built
TouchActive Solution Overview- what we built
Mobile Apps
Cloud backend
Desktop and Mobile consumer web
Addit
Tagged products
Management console
Identity Providers
Sharing (web and mobile) over net and face-to-face
Context for our choices2+ years ago
● Start Small○ pre-start-up: build prototype○ uncertain financing (probably small)○ small team of engineers
● Think big, or the possibility of big!
● Uncertainties○ Mobile platforms (especially NFC)○ Cloud computing platforms
Cooling
Power / UPS
Security
Buildings
Cooling
Power / UPS
Security
Buildings
LAMP stack on owned or hosted HW
Amazon Web Services(AWS)
Google App Engine(GAE)
Newer options:● OpenStack (Open SW, multiple service
providers)● Windows Azure● Amazon features for auto scaling, etc.
small start-up, low capital investment.
Pay as you go, don't buy servers.
But we didn't want to get "locked-in" to a single service provider
don't spend people time and money on managing servers, runtimes, etc. Focus on the application.
unknown/unexpected demand
we chose GAE
Applications developed with GWT
And many more, see:http://gwtreferencelist.appspot.comhttps://developers.google.com/web-toolkit/developer_spotlighthttp://gwtgallery.appspot.com/http://www.quora.com/What-web-applications-use-Google-Web-Toolkit-(GWT)
AdWords AdSense Orkut Wave Health Blogger
Google Web ToolkitComplex UI
Complex UI with graphical Data representation
GAE + GWT
10
Google App Engine Features
● Free developer / Pay-as-you go costs● Platform not Infrastructure
○ No need to architect the SW stack for scalability○ No need to manage the SW stack○ No machine administration○ Reasonable set of platform services
● Scalability○ Instances created automatically based on demand○ Scalable, distributed, datastore○ Content Distribution Network (CDN)○ High-performance Image service
Google App EngineTools and Languages
Language choices at the time: Java and Python○ Now "GO" is also available now
We chose Java● Developing J2ME and Android mobile apps in Java
at the same time
Eclipse IDE○ Local development server○ Java debugging of server code
Appstats for server performance analysis (RPC)Management dashboardCommand line tools and ANT tasks for automation
Google App EngineDatastore
Advantages● Scalable● Time vary with result set
size, not search set size● Compatible with existing
frameworks○ Java Data Objects (JDO)○ Java Persistence API (JPA)
● Distributed, replicated across data centers
Drawbacks● Pay-per-use● NoSql & pre-built
indexes○ No JOIN operations
○ No inequality filters on more than one "field"
○ No filtering of data based on subqueries
○ No Grouping of results
● Write rate limited
Datastore
Megatable
BigTable
GFS2
Google App EnginePlatform Services
Standard Implementations● Mail (JMail)● Datastore (JPA/JDO)● MemCache (JSR107)● Logging (Java Logging)
Non-Standard● Cron Tasks● Task Queues● Back-ends● Users (Identity)● Blob Store● Image Service● Chat (XMPP/Jabber)
with Presence● Channels API (Push)
Web Client DevelopmentGoogle Web Toolkit
● Develop Javascript AJAX client with Java○ Strongly, statically, typed language. ○ Many errors caught at compile time○ Full IDE support (completion, etc)○ Internationalization (i18n) built-in○ Generates optimized JavaScript, client bundles, code splitting, etc.○ Hides browser differences
■ Compiles a "permutation" for each language and browser you support
● Integrated Server and Client development project○ Java across Server and Client
○ Shared classes between Server and Client
○ "Transparent" RPCs between server and client (takes care of serialization and exception transport)
Web Client DevelopmentGoogle Web Toolkit
● Deferred binding with "properties"○ Used for loading script variants based on browser, language, form
factor, etc
● Reasonable Widget set○ Has improved a lot
○ Other widget sets available on top (GXT from Sencha, SmartGWT, Vadim, etc)
● "DevMode"○ Java debug experience in IDE for client side code○ Combined Client/Server/RPC debug experience○ No-compile, fast turn-around, hot code replace○ Memory intensive! :-(
Web Client DevelopmentGoogle Web Toolkit
Encourages reusable modular code development● Custom widgets● Code in defined "modules"
(server/client/shared code, RPCs, html, css, images, etc)
● Bundle modules into GWT libraries (.jar)
image
Web Client DevelopmentGoogle Web Toolkit
Ease of implementing Design Patterns● Model-View-Presenter with
late-binding of views● Same Presenter, different
Views● Detect Form Factor from
UA and serve the most appropriate view at run-time
Google Web ToolkitLate-Binding of views with ClientFactory
TouchActive.java
TouchActive.gwt.xml
Google Web ToolkitLate-Binding of views with ClientFactory
TouchActive.java
Desktop Mobile
Google Web ToolkitMVP, Activities & Places
Event-Bus across client code for subscribe/broadcast event programming
Application Activity and Place management via browser history
History Token
Google Web ToolkitUiBinder (XML) for declarative layout
With standard or custom widgets
GAE/GWT Debug experience:An example
Click Event on Facebook sign in button
GAE/GWT Debug experience:Client debugging
GAE/GWT Debug experience:Server debugging
GAE/GWT Debug experience:Client debugging
Implementation
20
Our implementation of TouchActive with GAE + GWT
● Authentication / Identity management
● Architecture overview
● Data Persistence Layer
● Platform API
● Portability
● CMS Integration
● Addit
Authentication:Own, portable, Identity management
● Created own User Object and related to standard external accounts○ OAuth (FB, Twitter, LikedIn)○ OpenID (Google, Yahoo, MySpace, ...)○ Google Apps domains for company users
● Username + Password also● Avoid forcing users having to have a Google account● Use OAuth & OpenId on mobile apps transparently for
account creation and sign-up
Avoided lock-in, didn't use Google App Engine Users Service
Architecture Overview
Servlet Filters
Request● Headers, cookies, ...
RequestContext
Platform Specific Services
Platform API
Data Persistence Layer
App Logic
Servlet Filters
● Flexible, just map filters to servlets with a config file using paths○ /api○ /rpc○ /cms○ /static
● Keeps servlet code clean and modular● Used for:
○ Language selection○ Desktop/Mobile pages while maintaining constant URL○ Authentication + assignment user to their Multi-Tenancy namespace○ Query logging for analytics
Servlet Filters:EntryPoint Filter (The Special One)
● Different types of EntryPoint filters that set different permissions and tokens○ Initializes the Request Context used in your app and DataManager○ Use of Thread variables to make available to all code but multi-threaded
● Authenticates the user and sets the correct "Namespace" in the request context
● Defined the appropriate filter for each request based on Access Control rules in web.xml
Servlet Filters
Request● Headers, cookies, ...
RequestContext
EntryPoint Filter
Data Persistence Layer
DataManager I/F to rest of app
RequestContext
Data Manager
JDO portable library across multiple datastore implementations
Google Datastore MySQL
Google AppEngine AWS MySQL Server
Data Persistence Layer
DataManager Interface● Created an API to manage data
objects for reading and writing● DataManager layer is used by all
app, app is kept 100% portable
Data Objects● ACL, Protected, Global, etc...● Enforce visibility and
permissions according to filters used on request and object type
Multi-Tenancy● Create data areas in the same
datastore● Avoided using Google Datastore
Namespaces
DataManager I/F to rest of app
Data Manager
JDO
Google Datastore MySQL
Google AppEngine AWS MySQL Server
Platform Specific Services
Custom build process outside IDE● ANT tasks to build, deploy, test,
release● Run manually from Command
Line or scripts / cron● Significant effort
App Logic
Defined a Platform API for platform dependant services
Cron Tasks
Task Queues
Blobstore
Image Service
Chat (XMPP)
Channels (Push)
GAE Implementation
Cron Tasks
Task Queues
Blobstore
Image Service
Chat (XMPP)
Channels (Push)
Other Implementation
Additional Need:Web content beyond the appEditable by non-programmersNo need for application release on each changeLocalized to multiple languages
Use a CMS
CMS Content
GWT
Integrating a CMS
Chose to serve content, scripts and RPCs from same domain○ Deploy CMS to same GAE "appID" as our app○ This could maybe be avoided with subdomains...
Chose Vosao GAE/J based CMS○ A bit immature at the time....still < 1.0○ Uses Apache Velocity (Java based) scripting language○ Java plugin can integrate tightly with your app and call app code directly
from within CMS web pages○ Has locked us in to GAE at CMS level, but our code is still portable○ Manual integration of config files, with help from Vosao team it could be
automated
Internationalization (i18N)
● Client GWT UI○ Built-in i18N support in GWT, strings in property files○ Permutation of script compiled for each language and correct
language loaded at run time
● Server messages○ We have avoided as much as possible○ Java Message Bundle + Properties files
● CMS content○ Core feature of CMS. Individual strings or entire pages can be
localized with common template
Other Features
30
Other features implemented
● Chat functionality with server (XMPP)○ used for testing at the moment
● "Presence" detection of users (XMPP)
● Push messages to browser using the GAE Channel API○ GWT Client JSNI Wrapper and EventBus connection
● "Addit" Bookmarklet and "web-clipper"
Addit -Objectives
● Help users create products in TouchActive by capturing product from any web page
● Reduce need for data entry by capturing as much data as possible automatically○ Use eCommerce/Search meta data when available
● Relatively simple and easy to use● Broad usage
○ Support all main web browsers○ Support as many web sites as possible○ Desktop, Tablet and Mobile versions
Product meta-data
meta and micro-formats used by eCommerce sites to provide data to search engines or crawlers:
Various Standards Exist:
● Facebook○ OpenGraph "product"
● Schema.org● GoodRelations● GoogleProducts
○ Google Merchant Center / Google Product Search
● hProduct
Google Product Search
Addit Installation
Drag and drop bookmarklet from web-page to bookmarks bar or folder
Install extension directly● From Chrome Web Store● Chrome, Safari, Firefox from web page
Addit UseUser clicks either the
Bookmarklet or the Extension button (FF, Safari, Chrome)
AdditScript loading
Keeping script out of the bookmarklet or
extention allows us to update the script
continually, without any re-install or
redistribution to users
Load localized script from cloud server and injected
into the page
Script NOT from Same-Origin as page and
security concerns and barriers to overcome
Detect language from page contents
AdditFetch user lists and detect currency
Fetches users lists from their TouchActive
account using a Cross-Domain "JSONP" RPC
call
Select default currency based on page language, page
contents and user settings retrieved with
lists
Additproduct meta-data parsing
Parse DOM and detect micro-format product
markup elements.Prepares products for
user automatically
If no product detected, uses page-level markup (meta, OpenGraph) or
general page information (Title, etc)
If multiple products detected, a simple
product selector UI is shown initially
AdditPersonalize and save to account
Select one of their own lists they would like to add the
product to
Will be automatically synced with their mobile app
Users take the suggested texts, or customize them.Select image, price, etc.
... and "Addit" to their TouchActive account
Add a personal note
AdditDifferent layouts for different form factors
DesktopWider, no vertical scrolling, smaller mouse targets
MobileNarrower (adjust to Viewport),
vertical scrolling, larger touch targets
All Developed in GWT
● GWT XSI Cross-Site linker for CORS script loading● Wrote own JSONP library for Cross-Site RPC
○ in Java with some JSNI
● GWT "late run-time binding" of scripts for Localization and Form Factor permutations using custom "Property Providers"○ Detect "best guess" language from host page using multiple methods
(poor standards implementations)○ Detect form factor using a combination of User Agent and display size
● Avoid older Layout Widgets that compile down to HTML Tables (Box Model and CSS problems...)
● Fast product meta-data parsing from DOM using "querySelectorAll()" and CSS3 selectors
Difficulties - mainly due to injecting a script into a page you don't control
● Unknown, uncontrolled CSS / DOCTYPE / QuirksMode○ Can't just do a "CSS Reset"○ CSS properties influence complex layouts○ DOCTYPES affect browser behavior (a nightmare on IE 7/8/9)
● "Difficult" sites○ Don't follow W3C standard for language selection○ iFrames/Frames/Frameset based○ Flash sites!○ Broken image links
● Different implementations of product markup standards○ Especially prices "," vs "." EUR vs € before/after/middle (1€25)
● JavaScript image galleries and tricks○ Images loaded but not visible○ Images being pre-cached (not always in DOM as image)○ Images adding / removed from the DOM
IE and DOCTYPE and QuirksFor masochists only!http://blogs.msdn.com/b/askie/archive/2009/03/23/understanding-compatibility-modes-in-internet-explorer-8.aspx
Phonegap Experiment
Can you take a GWT compiled app (HTML, CSS, JS, Images etc) and create a "native" app for multiple platforms?
Kind-of...● GWT RPCs from Phonegap client to server!
○ Must be able to go cross-domain○ We wrapped the RPC class so that it remaps the RPC endpoint to
default server when running "offline"○ Version problems expected with serialization in the future!!
● Phonegap has problems with relative file paths○ Processed GWT comiled output to use absolute paths
● Prototype produced from existing GWT app○ Automated with ANT tasks
Q
A
Optimization
40
Optimization Tools Available
Chrome Developer Tools "Audit"
PageSpeed
AppStats for server GAE internal RPC
SpeedTracer
SpeedTracer with AppStats
Chrome Audit
Page Speed
GAE AppStatsunderstand AppEngine RPC usage
AppStatsdrill-down to calls, duration and latency
SpeedTracerUnderstanding client side performance
SpeedTracerNetwork resource usage from client
Resource Loads● HTML● CSS● JavaScript
SpeedTracer (client) integrated with AppStats (server)
RPC request to server
Server side trace and timing from AppStats on server
What's new in GWT2.5
● Super Dev Mode (experimental)○ Replaces DevMode, uses "SourceMaps" in Chrome
to enable debugging in language JS was written in (Java, Dart...)
● New compiler optimizations○ Smaller, faster JavaScript code○ With slightly longer compile times
● Elemental○ Lightweight JavaScript library
● Updated ARIA support● UIBinder enhancements● Validation enhancements
GWT Compiled codesize reductions
GWT 2.4
100%
GWT 2.5
93%
GWT 2.5Closure Compiler
87%
Summary
40
Did we avoid provider "Lock-in"?
Portable data layer built on JDO● Maybe not the most optimal when running on GAE● Have to work with limitations of JDO/GAE subset● Over time, actually moving the data becomes a problem
Platform API has kept rest of code portable● needs work to get running on additional platforms
GAE based CMS has increased it● outside of the code, but still important for the project● decouple by moving to multiple domain approach
Things we would do differently if we started again
New competitors to GAE
Analytics
Phonegap app
API and directory structure
CMS
● New AWS services bring it closer to PaaS
● OpenStack from multiple providers
● Analyze more closely and prototype if time permits
Things we would do differently if we started again
New competitors to GAE
Analytics
Phonegap app
API and directory structure
CMS
● Avoid programming a lot of that, and falling short
● Expose analytics data and use external tools
● Give business users top-class tools with flexibility of analysis, drill-down etc
Things we would do differently if we started again
New competitors to GAE
Analytics
Phonegap app
API and directory structure
CMS
● Design and structure web-app for that from day one
Things we would do differently if we started again
New competitors to GAE
Analytics
Phonegap app
API and directory structure
CMS
● Assume you will have to version your external API
● Design app structure early on○ statics, resources, API,
CMS, RPC,...
Things we would do differently if we started again
New competitors to GAE
Analytics
Phonegap app
API and directory structure
CMS
● Explore using other CMS outside of GAE
● Sub-domains for "Single-Origin" content, RPCs
Possible Future Work
● Optimize○ Using tools to focus our efforts○ More use of memcache for DS accesses
● Explore use JDO3 (Polymorphism)● Prototype alternative CMS
○ Run on sub-domain in parallel
● Write platform code for other platforms○ Test on AWS or OpenStack?
● Phonegap app for other platforms○ Blackberry, Windows Phone...
● Investigate Continuous Integration Support○ Jenkins on GAE
GAE + GWTThe Good
● Scalable, without knowing about or managing the stack● Free for developers, our costs have been pretty low● Java across mobile, backend and client● Programming large app "all together" as client-server
○ But avoid that permitting spaghetti, structure APIs and code○ Hierarchy of reusable modules of client, server and shared code○ Drop-in GTW libraries with client, server and shared code
● Excellent debug experience when it works!○ including hot code replacement (server + client)
● Model-View-Presenter architecture combined with late-binding○ extend your app to mobile, tablet views more easily
● UI Binder for declarative UI layout, code for behavior
GAE + GWT The Bad
DevMode problems● Lack of maintenance on browser plug-ins● Memory intensive
You have to put in effort to avoid "lock-in"
JDO● Some people reporting long start-up times (meta-data related)
○ Has worked pretty well for us● Some features missing: e.g. No Async writes
Feedback
44
Feedback for GoogleApp Engine
● Don't introduce features or ways to use features that increase lock-in. ○ If you do, we will go to "standard" stacks on AWS or OpenStack...○ e.g. threads, etc
● More transparency on Billing and scheduler behaviour
● Need better attention to developers in forums○ Paying customers!○ Especially around scheduling and billing - invest people hours in this
● Channels API○ Support multiple listeners per page and broadcast○ Google supported JSNI GWT library
Feedback for GoogleGWT Debugging
Don't break the debug experience!
● Combined server/RPC/client debug experience in a single IDE is fantastic and unique (?) in developing an AJAX client - cloud server application○ Make it better! More maintenance of plug-in○ Support at least on Chrome / WebKit browser○ Less of a memory hog
● GWT 2.5 SuperDevMode does not replace it IMHO○ list of limitations of seems to outweigh the advantages○ main advantages (plug-in maintenance) are for Google
Feedback for GoogleGWT - others
● Allow client-only code and server-only code in same file○ E.g. Via @annotations○ Avoid having to complicate the class hierarchy to
have shared/server-only/client-only code for a class
● Java Calendar emulation in GWT and allow for shared-code date handling
Thank You!Questions?
Andrew Mackenzieandrew@mackenzie-serres.netandrew.mackenzie@bcntouch.com
http://www.linkedin.com/in/andrewdmackenzie
Daniel Ruiz Giménezdruizgimenez@gmail.comdaniel.ruiz@bcntouch.comhttp://www.linkedin.com/in/danielruizgimenez
50
http://www.slideshare.net/andrewdmackenzie/developing-touchactive-with-gae-and-gwt-google-devfest-barcelona-2012