McGraw Hill Couchbase SF 2013
-
Upload
couchbase -
Category
Technology
-
view
1.837 -
download
0
Transcript of McGraw Hill Couchbase SF 2013
![Page 1: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/1.jpg)
ZiggridProcessing Data in Near Real-Time Using Couchbase
Christopher Tse (Head of R&D, McGraw-HIll Education) Gareth Powell, Ph. D. (Chief Scientist, Ziniki Network)CouchConf SF 2013 - Sep 13, 2013
![Page 2: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/2.jpg)
@ McGraw-Hill Education
+
Research & Development
![Page 3: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/3.jpg)
@ McGraw-Hill Education
+
Research & Development
![Page 4: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/4.jpg)
Leveraging EmberJS, a JavaScript MVC framework to rethink the teaching and learning experiences on the Web and on mobile devices
HTML
![Page 5: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/5.jpg)
Collecting and analyzing multiple streams of student engagement, performance, and demographics for dashboards.
Data
FACT
Dimension
DimensionDimension
Dimension Dimension
![Page 6: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/6.jpg)
Action
Collections
EdSense: Real-time Reactions
Learning Style
Engagement
User Intents
Recommendations ReactionActivity Log
Previous
ly
Achievements Efficacy
![Page 7: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/7.jpg)
Action
Collections
EdSense: Real-time Reactions
Learning Style
Engagement
User Intents
Recommendations
Reaction
Activity Log
Previous
ly
Achievements Efficacy
![Page 8: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/8.jpg)
Learning Portal• Designed and built as a
collaboration between MHE Labs and Couchbase
• Serves as proof-of-concept and testing harness for Couchbase + ElasticSearch integration
• Available for download and further development as open source code
http://github.com/couchbaselabs/learningportal
Unveiled during CouchConf SF 2012
![Page 9: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/9.jpg)
SQL
Building Content-Driven Apps
Couchbase and Elasticsearch for content and metadata stores
Interactive content-driven apps need to store and share di!erent kinds of content and metadata. Whether you’re designing an online publication, a product catalog, a website landing page, or other digital content, it’s likely you want to support rich, personalized content for millions of users worldwide.
To handle the needs of these types of applications, you’ll want to consider a NoSQL database. In this paper, you’ll learn what makes Couchbase Server a good "t for apps that store and serve content and metadata.
What content-driven apps need
Your database is critical for your app because it is a central store for data and metadata. That means your app needs a database that lets you:
Store unstructured content and metadata. The vast majority of data generated today is unstructured, with a variety of new and changing data types. So, if you’re building a content catalog, you may store tens of millions of di!erent objects in a variety of forms – unstructured content and metadata that may be hierarchical, sparse, free-form text, or varying length. Keeping your app fresh means being able to add new types of data quickly, without dealing with the complexity and time it takes to change a schema. Storing content in a RDBMS requires knowing ahead of time the exact structure of the data your app needs to store, a requirement that is hard to meet for content and metadata apps that must support new data types as they emerge.
Hi!
![Page 10: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/10.jpg)
SQL ETL
![Page 11: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/11.jpg)
SQLSome-sort-of query language
ETLTo extract, transformand load in steps
We mean:
So we can: Declaratively express the logic for the machine to calculate and process
But: Processing complex, multi-layered queries upon request can be slow
Store the results from the intermediate or final steps of our calculations
Stored data gets out-of-sync with reality. And refresh is often expensive
When we say:
![Page 12: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/12.jpg)
SQL ETL
![Page 13: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/13.jpg)
SQL ETLLogic
![Page 14: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/14.jpg)
SQL ETLLogic Steps
![Page 15: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/15.jpg)
SQL ETLLogic Steps
Fresh Data
![Page 16: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/16.jpg)
SQL ETLLogic Steps
Fresh Data Fast Access
![Page 17: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/17.jpg)
SQLETLLogic Stepsin
Fresh Data Fast Access&
![Page 18: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/18.jpg)
FRPLogic Stepsin
Fresh Data Fast Access&
![Page 19: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/19.jpg)
Introduc
ing
Functional Reactive ProgrammingFRP
![Page 20: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/20.jpg)
Functional reactive programming (FRP) is a programming paradigm for reactive programming using the building blocks of functional programming.
The key traits of FRP are:
• The concept of "behaviors" or "signals" which model values that vary over continuous time.
• The concept of "events" which have occurrences at finitely many points in time.
• A means to change the FRP system in response to events, generally termed "switching".• The separation of evaluation details such as sampling rate from the reactive model.
An additional common but contentious trait is a notion of consistency when ordering events (not just within one stream). Variants include synchrony and glitch freedom.
The semantic model of FRP in side-effect free languages is typically in terms of continuous functions, and typically over time. In contrast, integration with a host language that has side-effects is typically given in terms of data flow or dependency graphs by extending the typical operational semantics to manipulate and use them.
WTF is FRP?
![Page 21: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/21.jpg)
Functional reactive programming (FRP) is a programming paradigm for reactive programming using the building blocks of functional programming.
The key traits of FRP are:
• The concept of "behaviors" or "signals" which model values that vary over continuous time.
• The concept of "events" which have occurrences at finitely many points in time.
• A means to change the FRP system in response to events, generally termed "switching".• The separation of evaluation details such as sampling rate from the reactive model.
An additional common but contentious trait is a notion of consistency when ordering events (not just within one stream). Variants include synchrony and glitch freedom.
The semantic model of FRP in side-effect free languages is typically in terms of continuous functions, and typically over time. In contrast, integration with a host language that has side-effects is typically given in terms of data flow or dependency graphs by extending the typical operational semantics to manipulate and use them.
TL;DR
WTF is FRP?
![Page 22: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/22.jpg)
Hint
![Page 23: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/23.jpg)
Excel is FRP
![Page 24: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/24.jpg)
Excel is FRPFunctionalEvery cell is either is a value or a f(x) that generates a value
![Page 25: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/25.jpg)
Excel is FRPFunctional
Reactive
Every cell is either is a value or a f(x) that generates a value
If you change one cell, all the other cells that refer to it changes immediately
![Page 26: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/26.jpg)
Excel is FRPFunctional
Reactive
Every cell is either is a value or a f(x) that generates a value
If you change one cell, all the other cells that refer to it changes immediately
![Page 27: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/27.jpg)
Excel is FRPFunctional
Reactive
Programming
Every cell is either is a value or a f(x) that generates a value
If you change one cell, all the other cells that refer to it changes immediately
Yes, you are programming when you are create a model in an Excel spreadsheet
![Page 28: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/28.jpg)
Start with a simple sum()Adding numbers within one worksheet
Excel is FRP
![Page 29: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/29.jpg)
Start with a simple sum()
Add more tabs
Adding numbers within one worksheet
To reflect higher level aggregates
Excel is FRP
![Page 30: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/30.jpg)
Start with a simple sum()
Add more tabs
Draw fancy graphs
Adding numbers within one worksheet
To reflect higher level aggregates
That visualizes the valuable aggregates
Excel is FRP
![Page 31: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/31.jpg)
The world runs on Excel. :)
![Page 32: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/32.jpg)
The world runs on Excel. : )
![Page 33: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/33.jpg)
What if...
Cells inside Sheets Documents in JSONData Model:
Calculating: When you open the file
Visualization: Supported chart types
All the time in the cloud
Anything drawable in HTML5
Instead of... We have...
=SUM(A1:B10) function Sum() { ... }Language:
![Page 34: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/34.jpg)
What if...
Cells inside Sheets Documents in JSONData Model:
Calculating: When you open the file
Visualization: Supported chart types
All the time in the cloud
Anything drawable in HTML5
Instead of... We have...
=SUM(A1:B10) function Sum() { ... }Language:
![Page 35: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/35.jpg)
![Page 36: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/36.jpg)
Ziggrid is FRP
![Page 37: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/37.jpg)
f(x) f(x)
f(x)
Ziggrid is FRPStores values in JSON
Specifies f(x) in JSON
Inside a Couchbase cluster
Also builds a dependency graph
![Page 38: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/38.jpg)
f(x) f(x)
f(x)
Ziggrid is FRPStores values in JSON
Specifies f(x) in JSON
Inside a Couchbase cluster
Also builds a dependency graph
Push data out via JSONSo clients can render data in HTML5, etc
![Page 39: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/39.jpg)
Ziggrid is FRPStores values in JSON
Specifies f(x) in JSON
Push data out via JSON
Inside a Couchbase cluster
Also builds a dependency graph
So clients can render data in HTML5, etcf(x) f(x)
f(x)
“The Ziggurat”
![Page 40: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/40.jpg)
Ziggrid is FRPStores values in JSON
Specifies f(x) in JSON
Push data out via JSON
Inside a Couchbase cluster
Also builds a dependency graph
So clients can render data in HTML5, etcf(x) f(x)
f(x)
“The Ziggurat”
JS N
![Page 41: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/41.jpg)
Layers of the Ziggurat
Raw Events
Enhanced Events
Summaries
Rankings
Correlations
Snapshots
Composites
![Page 42: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/42.jpg)
Gareth Powell, Ph. D.Functional Programming ExpertWrote doctorate thesis on Haskell
![Page 43: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/43.jpg)
Gareth Powell, Ph. D.Functional Programming ExpertWrote doctorate thesis on HaskellBaseball Fanatic
![Page 44: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/44.jpg)
Example: Baseball Data Analysis Model
Raw Events
Enhanced Events
Summaries
Rankings
Correlations
Snapshots
Composites
Plate AppearancesPlayer
SituationOutcome
Player Totals
Correlate vs Situation
Snapshots of Player Totals
Player Profile
Snapshots of Correlation
Game Results
Leaderboards (HR, AVG, PROD)
Win / Loss Record
![Page 45: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/45.jpg)
LIVE DEMO
![Page 46: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/46.jpg)
![Page 47: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/47.jpg)
Beane Counter Architecture
HTML5 Data Tables and SVG VisualizationEmber.js + D3.js via WebSockets
Mid
dlew
are
Fron
t-end
Model Description, Calculation, and Event ChainingJava via Memcached Protocol
Back
end
Raw and Aggregated Data Storage and IndexingCouchbase JSON Store + Incremental MapReduce
![Page 48: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/48.jpg)
ZiggridModels
• Data model described in JSON structure
{ "name": "plateAppearance", "fields": [ { "name": "team", // The team identifier from the Retrosheet Event file "type": "string", "key": true }, { "name": "player", // The player identifier from the Retrosheet Event file "type": "string", "key": true }, { "name": "season", // Year represented as YYYY "type": "string", "key": true }, { "name": "dayOfYear", // 1-365, proxy for which game it was "type": "number", "key": true }, { "name": "inning", // 1-9 for regular innings "type": "number", "key": true },
...}
JS N
![Page 49: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/49.jpg)
{ "enhanced": "situation", "from": "plateAppearance", "enhance": { "player": "player", "season": "season", "dayOfYear": "dayOfYear", "atbat": { "op": "+", "args": [{ "op": "*", "args": [ 3, "inning" ] }, "outs", -3 ] }, "bases": "bases", "lead": { "op": "group", "value": { "op": "ifelse", "test": "home", "true": { "op": "-", "lhs": "homeScore", "rhs": "awayScore" }, "false": { "op": "-", "lhs": "awayScore", "rhs": "homeScore" } }, "dividers": [ -3, -1, 0, 2 ], // (-inf, -3], (-3, -1], (-1, 0], (0, 2], "moreThan": 3 // (2,inf) },
ZiggridAlgorithms
• Data model described in JSON structure
• Define all calculation via communative and associative operators
JS N
![Page 50: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/50.jpg)
{ "composeInto": "profile", "from": "correlate_on_situation_groupedBy_player_and_season", "key": [ "player/", { "field": "player" } ], "fields": { "clutchness": "correlation" } }, { "leaderboard": "hotness", "from": "snapshot_playerSeasonToDate", "groupby": [ [ "season", "dayOfYear" ] ], "sortby": [ "average" ], "order": "desc", "values": [ "player" ] }, { "composeInto": "profile", "from": "snapshot_playerSeasonToDate", "key": [ "player/", { "field": "player" } ], "fields": { "hotness": "average" } } ...]
ZiggridComposites
• Data model described in JSON structure
• Define all calculation via communative and associative operators
• Projecting data via composite definition
JS N
![Page 51: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/51.jpg)
https://github.com/Ziniki-Network/Ziggrid
Ziggrid is 100%Open SourceLet’s work together!
![Page 52: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/52.jpg)
Future Improvements
Using Couchbase View Engine to do more of the processing in the database via Incremental MapReduce. Currently, only the leaderboards are computed using views.
GREATER SCALABILITY
Expand the functions support by Ziggrid to perform transformation, statistical calculations typical of Big Data analysis, and even ones for machine learning.
Allow in-browser development of new models using a subset of data. We need to finish developing a pure JavaScript-based Ziggrid processing engine.
Using UPR protocol to be notified of changes in inside Couchbase to allow more immediate, and thus more real-time propagation of events up the Ziggurat.
EASIER MODELDEVELOPMENT
REDUCEDLATENCY
DEEPERANALYTICS
![Page 53: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/53.jpg)
Hadoop
![Page 54: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/54.jpg)
HadoopBig Data
![Page 55: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/55.jpg)
HadoopBig DataBut Slow
![Page 56: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/56.jpg)
![Page 57: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/57.jpg)
Zebrasfor
![Page 58: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/58.jpg)
Thanks to 2 members of the Ember.js Core Team Who helped us design and code the sexy Ember + D3.js + WebSockets front-end
@machty
@stefanpenner
![Page 59: McGraw Hill Couchbase SF 2013](https://reader038.fdocuments.in/reader038/viewer/2022103021/55d52bfcbb61ebfa778b45b8/html5/thumbnails/59.jpg)
Questions?
@christseFollow me on Twitter