RubyNation Visualizing Big Data on Small Devices
-
Upload
tomzeng -
Category
Technology
-
view
105 -
download
0
description
Transcript of RubyNation Visualizing Big Data on Small Devices
![Page 1: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/1.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Visualizing Big Data on Small Devices
Tom Zeng Director of Engineering
[email protected] @tomzeng
www.linkedin.com/in/tomzeng
![Page 2: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/2.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Agenda
Introduction
Front End - HTML5/Bootstrap, Backbone/CoffeeScript, D3, MapBox
Backend - Rails, MongoDB
Big Data Processing - Hadoop, Hive, Pig
Showcase - Mobile and Data Visualization Related Projects
Q & A
![Page 3: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/3.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Intridea - Rails, UX/Data Visualization, Mobile, Big Data, e-commerce
American Bible Society (ABS http://www.americanbible.org/) - partners with Bible publishers
· Provides API access to 539 Bible versions in 242 languages
· The usage of the APIs is tracked at the verse level, along with ip location, timestamp, and duration
· 530 million view logs/year(’12-’13 data), 1.5 mil/day, each view log has packed about 12 bible views
· Amounts to 5-6 billion Bible views each year
ABS asked Intridea to build the dashboard app Scripture Analytics (http://www.scriptureanalytics.com)
Introduction
![Page 4: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/4.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
BY WORKING REMOTELY
9,816 Hours Saved AnnuallyACROSS THE US & OVERSEAS
30+ EmployeesFOUNDED & STARTED IN 2007
Washington D.C.
We Make
! 🌎 #
ON GITHUB
Open Source Software$
![Page 5: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/5.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Major Open Source Contributions
OmniAuth is a flexible authentication system utilizing Rack middleware.
OmniAuthAn opinionated micro-framework for
creating REST-like APIs in Ruby.
Grape
Hashie is a simple collection of useful Hash extensions.
HashieA Ruby wrapper for the OAuth 2.0
protocol.
oauth2
A symbol font that makes it easy to create a map of the U.S. with HTML/CSS.
Stately
A generic swappable back-end for JSON handling.
Multi_JSON
Mission control dashboard for your distributed teams.
Houston
github.com/intridea
![Page 6: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/6.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
Simplified and modernized application experience for ADP
Application engineering for BusinessWeek.com
World’s first 100% web based Point of Sale system
![Page 7: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/7.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Where people go to make a difference with their investment capital.
Simplifying ADP’s core business: Payroll
Engineering for the most-trafficked wedding planning solution
![Page 8: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/8.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
![Page 9: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/9.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
ABS Scripture Analytics Query RequirementsVisualizations
![Page 10: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/10.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Public and private dashboards visualizing Bible reading across the InternetUp to the minute dashboards showing what Bible verses are being read when, and where, all over the globe.
![Page 11: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/11.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Mobile
Desktop
Tablet
Responsive Web App
www.scriptureanalytics.com
![Page 12: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/12.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Front End User Interface
Single Page Application using Backbone.js
CoffeeScript (Ruby like, Jasmine in CoffeeScript similar to RSpec)
D3 for Data Visualization
Twitter Bootstrap for Responsive UI
Packery for Responsive Layout - http://packery.metafizzy.co/
Mapbox for Map Rendering - https://www.mapbox.com/
![Page 13: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/13.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Backend Servers/Services
Rails on Ruby Application mostly as the API server
MongoDB as the data store/cache
Mongoid for Active Record like queries
MongoDB Aggregation Framework for complex queries
Pulling data periodically from S3 to populate the Mongo database
Local R&D Hadoop and Mongo clusters for data exploration
![Page 14: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/14.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB
Document oriented, schema free, JSON format
Very high data read and write throughput
Rich query capabilities (aggregation framework), flexible indexes
Scale with auto-sharded replica sets
Map/Reduce in JavaScript
![Page 15: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/15.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hadoop/Pig/Hive/Impala
Hadoop cluster (AWS Elastic Map/Reduce on-demand) to process and store data in S3
Pig to parse, transform, geo-code data
Hive to query data and generate aggregated JSON reports
Impala is similar to Hive (but much fast than the older version of Hive), used for ETL
!
![Page 16: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/16.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Elastic Map/Reduce Hadoop Cluster - On Demand Processing
![Page 17: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/17.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Elastic Map/Reduce Hadoop Cluster - Terminated when done
![Page 18: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/18.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Cloudera CDH4 - on local 10-node cluster
![Page 19: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/19.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Cloudera CDH4 - Streaming Data into Hive Table
![Page 20: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/20.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Pig Sample Query
![Page 21: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/21.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Sample Query
![Page 22: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/22.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Bible views by City
![Page 23: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/23.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse before Mother’s day
28 Her children show their appreciation, and her husband praises her.
![Page 24: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/24.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse on Mother’s day
28 Her children show their appreciation, and her husband praises her.
![Page 25: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/25.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse after Mother’s day
28 Her children show their appreciation, and her husband praises her.
![Page 26: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/26.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB Aggregation Framework Example
http://docs.mongodb.org/manual/core/aggregation-pipeline/
![Page 27: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/27.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB Aggregation Framework Example
![Page 28: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/28.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
ABS Data Processing using Hadoop and MongoDB
![Page 29: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/29.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Analyzing Twitter using Hadoop and MongoDB
![Page 30: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/30.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Mobile and Data Visualization Project Showcase
ADP
BLiNQ
PEW Templeton - Global Religious Futures
Cato Institute - HumanProgress
!
!
![Page 31: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/31.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Redefining ADP’s touch and desktop experiencesADP processes one out of every six paychecks in the United States. We’re bringing payroll into the decade of touch.
![Page 32: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/32.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
ADP TLM
![Page 33: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/33.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
ADP HCR
![Page 34: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/34.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
![Page 35: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/35.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Dashboard insights for 600 of the world’s largest advertisersAnalytics, planning, and flight management for social advertising campaigns and brand engagement.
![Page 36: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/36.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
![Page 37: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/37.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
![Page 38: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/38.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
![Page 39: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/39.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Analyzing religious change and its impact on societies around the worldInteractive website exploring the patterns and trends in religions across the globe
![Page 40: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/40.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
![Page 41: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/41.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
![Page 42: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/42.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
![Page 43: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/43.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
Human advancement to a higher stageHuman Progress seeks to document changes in living standards in the past and present while explaining and exploring the best ways to improve conditions for people.
![Page 44: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/44.jpg)
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
humanprogress.org
![Page 45: RubyNation Visualizing Big Data on Small Devices](https://reader034.fdocuments.in/reader034/viewer/2022052618/54c679854a79595e4a8b4569/html5/thumbnails/45.jpg)
Gracias
Merci ありがとう
Danke 谢谢
Thank You
Copyright © 2014 Intridea Inc. All rights reserved.
Tom Zeng Director of Engineering
[email protected] @tomzeng
www.linkedin.com/in/tomzeng