MongoDB and Web Scraping with the Gyes platform. MongoDB Atlanta 2013
Rapid web development using tornado web and mongodb
-
Upload
ikailan -
Category
Technology
-
view
10.902 -
download
4
description
Transcript of Rapid web development using tornado web and mongodb
![Page 1: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/1.jpg)
Rapid Web Development with Tornado Web and MongoDB
Ikai LanTwitter: @ikai
Friday, June 10, 2011
![Page 2: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/2.jpg)
About the speaker
• Developer Relations at Google
• Software Engineering background
• Lives in San Francisco, CA
• Writes Java, Python on a day to day basis
• Twitter: @ikai
Friday, June 10, 2011
![Page 3: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/3.jpg)
This talk
• Why Tornado Web and MongoDB?
• Whirlwind tour of Tornado (get it?!)
• Intro to MongoDB
• Brief look at a demo app
Friday, June 10, 2011
![Page 4: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/4.jpg)
What I won’t be talking about
• Scalability - loaded topic
• Reliability - another loaded topic
Friday, June 10, 2011
![Page 5: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/5.jpg)
So why tornado/mongo?
Friday, June 10, 2011
![Page 6: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/6.jpg)
Joy.Friday, June 10, 2011
![Page 7: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/7.jpg)
Where does this joy come from?
Friday, June 10, 2011
![Page 8: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/8.jpg)
Freedom.Friday, June 10, 2011
![Page 9: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/9.jpg)
So really, my talk should have been called “The
Joy of Tornado Web and Mongo DB”
Friday, June 10, 2011
![Page 10: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/10.jpg)
Intangibles
• Tornado - fast development, very few rules
• MongoDB - freedom from a predefined schema
• Python, of course. Dynamically typed variables, duck typing plus everything else that is cool about the language
Friday, June 10, 2011
![Page 11: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/11.jpg)
Tornado Web history
• Open Sourced by Facebook
• ... formerly of FriendFeed
• ... former Googlers (that’s why the framework looks like App Engine’s webapp - they wrote it)
• Tornado Web powers FriendFeed
Friday, June 10, 2011
![Page 12: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/12.jpg)
A whirlwind tourFriday, June 10, 2011
![Page 13: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/13.jpg)
Tornado features
• Out of the box auth with Twitter, Facebook, Google and FriendFeed
• Out of box support for localized strings
• Simple templating that allows arbitrary Python code
• Asynchronous requests
• Small codebase
Friday, June 10, 2011
![Page 14: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/14.jpg)
Simple handlersimport tornado.ioloopimport tornado.web
class HelloHandler(tornado.web.RequestHandler): def get(self): self.write("Hello, world :)")
class GoodbyeHandler(tornado.web.RequestHandler): def get(self): self.render("goodbye.html", message=”Goodbye, world”)
application = tornado.web.Application([ (r"/hello", HelloHandler), (r"/goodbye", GoodbyeHandler), ])
if __name__ == "__main__": application.listen(8888) tornado.ioloop.IOLoop.instance().start()
Friday, June 10, 2011
![Page 15: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/15.jpg)
Asynchronous natureclass MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous def get(self): http = tornado.httpclient.AsyncHTTPClient() http.fetch("http://friendfeed-api.com/v2/feed/bret", callback=self.on_response)
def on_response(self, response): if response.error: raise tornado.web.HTTPError(500) json = tornado.escape.json_decode(response.body) self.write("Fetched " + str(len(json["entries"])) + " entries " "from the FriendFeed API") self.finish()
Friday, June 10, 2011
![Page 16: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/16.jpg)
Unrestrictive templating {% for student in [p for p in people if p.student and p.age > 23] %} <li>{{ escape(student.name) }}</li> {% end %}
# Sample of arbitrary functions in templates def add(x, y): return x + y template.execute(add=add)
### The template {{ add(1, 2) }}
Friday, June 10, 2011
![Page 17: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/17.jpg)
Lies, damn lies and benchmarks
Friday, June 10, 2011
![Page 18: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/18.jpg)
Saving dataFriday, June 10, 2011
![Page 19: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/19.jpg)
MongoDB is an object database
• No schema - anything that can be a JSON object can be stored
• You can define new collections on the fly
Friday, June 10, 2011
![Page 20: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/20.jpg)
Getting started with Mongo in 4 steps
• Download a binary for your system
• Create a data directory (mkdir -p /data/db)
• Run mongod
• Install pymongo
• ... start writing code!
Friday, June 10, 2011
![Page 21: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/21.jpg)
Pymongo code sampleconnection = pymongo.Connection()database = connection["your_database"]
def get_or_create_location_by_id(location_id): """ Attempts to fetch location data from database. If it doesn't exist, create it. Note that this is NOT concurrency safe. """ location_data = database["locations"].find_one({ "_id" : location_id }) if location_data is None: location_data = { "_id" : location_id, "guards": [], "owner" : None, "history" : [], "last_extort_time" : None } database.location.save(location_data, safe=True) return location_data
Friday, June 10, 2011
![Page 22: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/22.jpg)
Pymongo queries# Add Tyson as a guard to every location owned by “Joe Smith”locations = database["locations"].find({ "owner" : "Joe Smith" })for location in locations: location["guards"].append("Tyson") database["locations"].save(location)
# Find everyone who has Tyson as a guardlocations = database["locations"].find({"guards" : "Tyson"})
# Find everyone who has *ONLY* Tyson as a guardlocations = database["locations"].find({"guards" : ["Tyson"]}) # Note []s
# Find everyone who has Tyson as a guard whose owner is “Ikai Lan”locations = database["locations"].find({"guards" : "Tyson", "owner" : "Ikai Lan" })
Friday, June 10, 2011
![Page 23: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/23.jpg)
Mold as you go alongFriday, June 10, 2011
![Page 24: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/24.jpg)
Freedom from
• ... waiting to get started
• ... predefining a schema
• ... worrying about complex data structures that don’t fit well into the SQL box. Anything that is JSON can be stored!
Friday, June 10, 2011
![Page 25: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/25.jpg)
More MongoDB features
• Simple Geospatial indexing
• GridFS - distributed filesystem running on MongoDB
• Out of the box mapreduce
• Out of the box replication, data partitioning
Friday, June 10, 2011
![Page 26: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/26.jpg)
Putting it all together
• MobSquare, a location based game that uses the Facebook API.
• Mashup: Mafia Wars + FourSquare
• https://github.com/ikai/mobsquare-demo
• “Check in” to locations, take them over, extort money and fight other gangs
Friday, June 10, 2011
![Page 27: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/27.jpg)
Why are Tornado/Mongo a fit?
• If I haven’t said it enough yet, they’re fun
• Facebook API performance varies - asynchronous so we can serve requests while waiting
• Highly structured player data
Friday, June 10, 2011
![Page 28: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/28.jpg)
OAuth 2 upgrade flowclass OnLoginHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous def get(self): # Store this somewhere code = self.get_argument("code") access_token_url = ACCESS_TOKEN_URL_TPL + code client = httpclient.AsyncHTTPClient() client.fetch(access_token_url, self.on_fetched_token) def on_fetched_token(self, response): """ Callback inokved when the auth_token is fetched """ matches = ACCESS_TOKEN_REGEX.search(response.body) if matches: access_token = matches.group(1) client = httpclient.AsyncHTTPClient() # lambda is effectively a function factory for us client.fetch(API["profile"] % access_token, lambda response: self.on_profile_fetch(response, access_token)) def on_profile_fetch(self, response, access_token): """ Callback invoked when we have fetched the user's profile """ profile = json.loads(response.body) profile["access_token"] = access_token profile_id = db.save_profile(profile) self.set_secure_cookie("user_id", str(profile_id)) self.redirect("/") # implictly calls self.finish()
Friday, June 10, 2011
![Page 29: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/29.jpg)
Known gotchas
• Immaturity of frameworks, tools
• Tornado templating errors result in somewhat useless stack traces
• MongoDB nuances
• Tornado community fairly small
Friday, June 10, 2011
![Page 30: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/30.jpg)
Questions?
• Ikai Lan - @ikai on Twitter
• Github: https://github.com/ikai
• Google Profile: https://profiles.google.com/ikai.lan/about
• Thank you for having me at Pycon!
Friday, June 10, 2011
![Page 31: Rapid web development using tornado web and mongodb](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582211d8b42a5e468b5043/html5/thumbnails/31.jpg)
Attributions
• Slide 6: Kirstin Jennings - “big smile!” http://www.flickr.com/photos/methyl_lives/2973265796/
• Slide 8: Tony Blay - “I am a bird now!” http://www.flickr.com/photos/toniblay/59415205/
• Slide 12: Antonio Sofi - “whirlwind” http://www.flickr.com/photos/webgol/36546734
• Slide 17: Tornado Web documentation - http://www.tornadoweb.org/documentation
• Slide 18: Christine of Robot Brainz - “Cassette” http://www.flickr.com/photos/robotbrainz/2786347158/
• Slide 23: Samuel Stroube - “Play-Doh on a Picnic Table” http://www.flickr.com/photos/samoube/203586664
Friday, June 10, 2011