Dataweek-Talk-2014

BUSINESS PROBLEM

Financial Apps has a lot of great data on users. The data can change and be enhanced on the fly.

For many companies this data sits there adding no real value.

When data is actionable it can have greater value.

• Drive Decisions, Workflows and Content• Change the user experience based on what we know, Now

• about the user• about the markets• about the world

• Monitoring and Alerts• Spending and Budgeting• Cash Flow• Fraud

• Deliver content and data• Offers and Deals• Advice• Aggregated Data Sets• Data Transformation (HTML / PDF)

BUSINESS PROBLEM

We need a way to make this data actionable in real-time without waiting for developers.

Home Depot Transaction Analysis

Description The Numbers

Percentage of transactions for Home Depot 3,723

Average single transaction amount $72.47

Highest single transaction $10,450

Percent of users with at least one transaction 13%

Average number of visits per month 1.5

Average spend per month $108.70

May 1st, 2014 – September 2nd, 2014

Having the ability to derive and act on data, when news breaks, is critical.

BUSINESS PROBLEM

There is knowledge in your data yet to be discovered.

• Dynamic Data Management• Add new user data, offer feeds and advice at real-time.• No build of the software required to add or modify data.• Flexibility to work with and aggregate any available data.• Solution: MongoDB

• Flexible And Scalable Computing• Leverage Linux PAAS technologies.• Grow computing/users at a reasonable cost.• Solution: Iron.io

• Rule Management API• Add, edit and execute rules on demand via an API.• Write rules against any collection of data in the platform.• Join collections of data to create complex rules and data sets.• Leverage MongoDB and Iron.IO to their fullest.• Solution: Go Programming Language

THE SYSTEM

If we had a system that could do the following, we could accomplish our goals.

http://docs.mongodb.org/manual/core/data-modeling-introduction/

Data is stored in “Collections” as individual documents.

Relationships can be created by using references. This is in step with how relational database systems store data.

WHY MONGODB – DYNAMIC DATA MANAGEMENT

MongoDB’s schemaless database provides great flexibility.

http://docs.mongodb.org/manual/core/data-modeling-introduction/


Embedding data allows all the data for an entity to be organized in a single document.

http://docs.mongodb.org/manual/core/aggregation-pipeline/


We can leverage the aggregation pipeline for writing rules.

Build single processes and use the computing you need, when you need it.

The System Is Driven By Data And Processes That Each Perform A Single Task.

WHY IRON.IO – FLEXIBLE AND SCALABLE COMPUTING

Iron.IO queues and runs worker tasks on their high performance computing platform. We get scalability out of the box and can realize all the computing we need, when we

need it.

• Comes with a robust standard library

• Concurrency and garbage collection

• Works on a multitude of platforms

• Code is statically compiled so deployment is trivial

• Comes with a large set of online documentation

• Tools to lint, vet, test, profile and benchmark your code

• mgo (Mango) driver for MongoDB by Gustavo Niemeyer

WHY GO – DO MORE WITH LESS

Go balances between being a low level systems language with all the features that modern languages have today. It allows you to be incredibly productive, performant and

fully in control.

Generate a budget for any given user, based on their transactions,a budget model and a set of categories.

{ "user_id" : "9f6b481b-e9fd-473b-5a62-14d3f54e892d", "account_id" : "5409fcbb6685720018000003", "account_name" : "Bank Visa Platinum1", "amount" : 150.50, "type" : "debit", "merchant_name" : "Sam's Club", "categories" : [ { "category_master_id" : 22200, "type" : 2, "amount" : 150.50 } ]}

DEMO – USER BUDGET

Transaction Data


{ "name" : "budget-model-pw", "data" : [ { "category_id" : 20900, "category" : "Entertainment", "percentage" : 0.03 }, { "category_id" : 20002, "category" : "Phone", "percentage" : 0.02 }

]}


Budget Model Data


{ "category_master_id" : 20900, "parent_id" : 0, "name" : "Entertainment", "type" : 2, "is_locked" : 0, "modified_date" : ISODate("2014-08-27T15:13:12.657Z"), "created_date" : ISODate("2014-08-27T15:13:12.657Z")}


Category Data


Budget Workflow

Find expenses from transactions and sum

by category.

Find income from transactions and sum.

Join the income to each category expense.

Calculate the percent of spend.

Load all the expense categories.

Save temp_db

Transactions transactions

Save temp_db

temp_db

Save temp_db

category_master

Save temp_db

Join the category name to the documents.

temp_db

Save temp_db

Load the“Financial Apps”

Budge Model.

relevance_models

Save temp_db

Join the budget percentage per category. Then calculate if the percent of spend is over or under the budget limit.

temp_db

• Go Language• Systems programming language• Compiles to binary code for target OS/Architectures• Cross compile on many operating systems• Access to scalable cloud computing environments• MGO driver for Go provides excellent MongoDB support

• MongoDB• Scalability and redundancy out of the box• Great MongoDB hosting providers• Schemaless database that provides great flexibility• Aggregation pipeline to build rules and datasets• Can search against text with good performance

• Iron.IO• Something• Something• Something


Query WorBy combining the data flexibility and aggregation capabilities of MongoDB with the Go language and template framework, we have a scalable, redundant and

feature rich solution.


What you have seen is the result of the data flexibility and aggregation capabilities of MongoDB, the power of the Go Programming language and the computing power of

Iron.io.

• Go Programming Language• Systems programming language• Compiles to binary code for target OS/Architectures• Cross compile on many operating systems• Access to scalable cloud computing environments• mgo driver for Go provides excellent MongoDB support


• Iron.io - IronWorker• High-Scale processing and scalability• Flexible task scheduling and on demand via API• Guaranteed reliability• Security, Monitoring and Administration• No maintenance or IT required

• Go Language• Systems programming language• Compiles to binary code for target OS/Architectures• Cross compile on many operating systems• Access to scalable cloud computing environments• MGO driver for Go provides excellent MongoDB support


• Iron.IO• Something• Something• Something


Query WorBy combining the data flexibility and aggregation capabilities of MongoDB with the Go language and template framework, we have a scalable, redundant and

feature rich solution.

LEARN MORE – GOINGGO.NET / GOINGGO TRAINING

How can you start building your own engines using MongoDB and Go?

goinggo.net/2014/06/actionable-data-monogdb-go.htmlHow to use MongoDB and Go to make your own data actionable.

How to use MongoDB to analyze data in a Go program.

goinggo.net/2013/07/analyze-data-with-mongodb-and-go.html

blog.mongodb.org/post/80579086742/running-mongodb-queries-concurrently-with-go

Getting Started With MongoDB and Go

GoingGoTraining.net / GoInActionBook.comGo and MongoDB Workshops and Training

Dataweek-Talk-2014

Documents

Transcript of Dataweek-Talk-2014