The Artful Business of Data Mining: Distributed Schema-less Document-Based Databases

The Artful Businessof Data Mining

Distributed Schema-lessDocument-Based Databases

Wednesday 27 March 13

David Coallier@davidcoallier


Data ScientistAt Engine Yard (.com)


RDBMs


StructureRestrictionsSafety


id name age address

1234567...

daviddivadfoobarjohnjackjill...

134142

331548...

315513198851166...


id name age address

1234567...


13

4142

331548...

315513198851166...


id name age address

1234567...


134142

331548...

315513198851166...


What If?


id name age address phone

1234567...


262742311712821...

IEUSIE

CANZDKIE...

3531

3531

131311353...


BeforeMoving on


JSON


What is JSON?


{ "firstName": "David", "lastName": "Coallier", "age": 26, "address": { "streetAddress": "Mansfield House", "city": "Crosshaven", }, "phoneNumbers": [ { "type": "mobile", "number": "0863299999" } ]}


What is HTTP?


What is a Schema?


Alternative


Schema-less


DoesNOTMeanStructure-less


DocumentsandK-V Buckets


CouchDBCluster of unreliable commodity hardware


Replication AttachmentsGenerated “random” idsDictionary Revisions?JSON ObjectsHTTP CRUD


Documents


{ "_id": "131dafsd1vasd", "_rev": "12-fva32asdf", "firstName": "David", "lastName": "Coallier", "age": 26, "address": { "streetAddress": "Mansfield House", "city": "Crosshaven", }, "phoneNumbers": [ { "type": "mobile", "number": "0863299999" } ]}


How do youfindAnything?


Map/Reduce


...


Riak


DynamoPaper


CAPTheorem


Key-ValueBuckets


Differences?


CouchDB Riak

Storage Model append-only bitcask

Access HTTP HTTP, PB

Retrieval Views(M/R) M/R, Indexes, Search

Versioning Eventual Consistency Vector Clocks

Concurrency No Locking Client Resolution

Replication master/master/slave replication, clustering

Scaling In/Out Big Couch Built-in

Management Futon/Fuxton Riak Controlhttp://downloads.basho.com/papers/bitcask-intro.pdfhttp://guide.couchdb.org


http://downloads.basho.com/papers/bitcask-intro.pdf




Map/Reduce


Mapper:

Reducer:Receives output from mappers

Executed on document


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}


{ "age": "32", "heads": "3",}


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}

Map: find-ages


function find_ages(doc) { if (typeof(doc.age) != undefined) { emit(doc._id, doc.age); }}

Map: find-ages


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}

Map: find-ages


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}

Map: find-ages

26 32 42 17


Map: find-ages

26 32 42

Reduce: sum

17


Reduce: sum

function sum(values) { return sum(values);}


Map: find-ages

26 32 42

Reduce: sum

17

117Wednesday 27 March 13

Mapper:




SoWhat?


The MachinesThey Lurn.


The Problem


Statistics Example


Mean,Std. DeviationAge


µ = 1n

xii=1

n

∑Wednesday 27 March 13

σ = 1n

(xi − µ)2i=1

n

∑


Mapper:




Mapper:

Reducer:Receive, process further.

Retrieve values, pre-process


{ "_id": "...", "_rev": "...", "age": "26"}

{ "_id": "...", "_rev": "...", "age": "32", "heads": "3",}

{ "_id": "...", "_rev": "...", "age": "42"}

{ "_id": "...", "_rev": "...", "age": "17"}


[ [ 26, 676], [ 32, 1024], [ 42, 1764], [ 17, 289 ]

]


/** * Our mapper function. */map: function(doc) { emit(null, [doc.age, doc.age * doc.age]);}

/** * Our reducer... */reduce: function(keys, values, rereduce) { var N = 0; var summed = 0; var summedSquare = 0;

for (var i in values) { N += 1; summed += values[i][0]; summedSquare += values[i][1]; }

var mean = summed / N; var standard_deviation = Math.sqrt( (summedSquare / N) - (mean* mean) )

return [mean, standard_deviation]}


/** * Our mapper function. */map: function(doc) { emit(null, [doc.age, doc.age * doc.age]);}

/** * Our reducer... */reduce: function(keys, values, rereduce) { var N = values.length; var summed = sum(values.map(function(v) { return v[0]; })); var summedSquares = sum(values.map(function(v) { return v[1];}));

var mean = summed / N; var standard_deviation = Math.sqrt( (summedSquares / N) - (mean*mean) )

return [mean, standard_deviation]}


NaiveBayes


Real LifeFraud


P(x j = k | y = fraudulent)P(x j = k | y = normal)P(y)


We need to:Sum , for each yto calculate P(x|y)

x j = k


We need:More than 1 mapper.


We need

4mappers


Mapper #1:1i P(x j = k | y = fraudulent)∑


Mapper #2:1i P(x j = k | y = normal)∑


Mapper #3:1i P(y = fraudulent)∑


Mapper #4:1i P(y = normal)∑


ReducerSums up results for parameters


ClusterAnalysis


k-means


Mapper:

Reducer:Sum up the sums, get new centroids.

Divide vectors into subgroups,Calculate d(p,q) between vectors, find centroids,sum them up.


The Artful Business of Data Mining: Distributed Schema-less Document-Based Databases

Technology

Transcript of The Artful Business of Data Mining: Distributed Schema-less Document-Based Databases