Collaborative Filtering in Map/Reduce

Collaborative Filteringin

Map/Reduce

Ole-Martin Mørk - Open AdExchange

tirsdag 14. september 2010

Vision

• Learn that Map/Reduce is simple

• Learn that Map/Reduce may be powerful

• Collaborative Filtering is fun!

Agenda

• Map/Reduce

• Collaborative Filtering

• Collaborative Filtering with Map/Reduce

• Amazon Elastic MapReduce

Map/Reduce

• Very scalable algorithm

• Inspirered by map and reduce from functional programming.

• Everything is based on key/value

6 phases

• Reader

• Map

• Partition

• Comparison

• Reduce

• Writer

6 phases

• Reader

•Map

• Partition

• Comparison

•Reduce

• Writer

List(“hello”,“dude”).map{x=>x.substring(0,1)}

functional map

Map/Reduce map

• Input is key/value

• Output is key/value

Simple Example, Map

• Count occurences of words in a document

• Input is: <linenumber>, <content of line>

• For each word on the line, the output is <word>, <count>

Reducetirsdag 14. september 2010

functional reduce

val sum=List(32,40,23).reduceLeft{_+_}

Map/Reduce reduce

• Input is key/list of values

• Output is key/value

Simple Example, Reduce

• Reduce input is <word, counts>

• For each value we increase the count

• Output is <word>, <sum of counts>

Reduce

CollaborativeFiltering

Amazon

Last.fm

Sceneami.com

User based

• Useful when we have

• Small number of users

• High correlation between users

• Data that changes often

Item based

• Useful for big sites like Amazon etc..

• Small overlap between users

• Mostly static data

Pattern Matching in Scala

Euclidean Distance

Rating

Euclidean Distance

• Alf‘s presentations:1,25,56,57,58,98 (6)

• Kari’s presentations: 2,25,98,99 (4)

• Equal presentations: 25 and 98 (2)

• Unmatched presentations: 6-2 + 4-2 = 6

• Distance score: 1/1+sqr(6)= 0.29

Recommended sessions

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62)

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62)

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62), 4 (0.41)

More Map/Reduce

Several iterations

Iteration 1

Iteration 2

Iteration 3

Several iterations

Iteration 3

Iteration 1 Iteration 2

Partitioning

Reducer Reducer

Paul Mary Kate Lea Jeff Ali

Comparison

Reducer Reducer

Pres 2

Pres 2 JeffPres 2

Pres 1

Pres 1 AliPres 1

Pres 1Pres 1Pres 1 Pres 2Pres 2Pres 2Paul Lea Ali Jeff Mary Kate

Guidelines

• Never access external sources during computation.

• Your functions should be small and fast

• You might not have all the data available

Hadoop

• Hadoop is reusing objects, so remember to clone if you plan to keep them.

• You can read and write all objects implementing hadoop.WritableComparable

• write(DataOutput)

• readFields(DataInput)

• compareTo(Object)

Collaborative Filtering, the Map/Reduce way

Overview

• Create an application that recommends JavaZone presentations.

• Overall goal: Scalable performance

• 4 iterations

• Reading input from text file

Iteration 1

• Map input: <user>, <presentations>

• Map output: <presentation>, <user>

• Reduce output: <presentation>, <userList>

Iteration 2

• Map input: <presentation>, <userList>

• Map output: <user>, <userList>

• Reduce input: <user>, <list of userList>

• Reduce output: <userTuplet>, <match count>

Iteration 3

• Map input: <userTuplet>, <match count>

• Map output: <userTuplet>, <diff>

• Map output: <userTuplet reversed>, <diff>

• Reduce output: <user>, <similaruser>

Iteration 4

• Map input: <user>, <similaruser>

• Map output: <user>, <presentation with score>

• Reduce output: <user>, <presentations>

Map/Reduce on EC2

Elastic Map/Reduce

• Same code

• Same input

• Different configuration

Upload files

s3cmd put oax-jz10:jar/oax-jz10.jar target/oax.jz10.jar

s3cmd.rb put oax-jz10:input/data.txt data.txt

Create job flow

elastic-mapreduce --create --alive --log-uri s3n://oax-jz10/log

Register iterations

elastic-mapreduce --jobflow j-1NLAIW45QUN4B --jar s3n://oax-jz10/jar/oax-jz10.jar --arg com.openadex.pres.iterations.Iteration1 --arg s3n://oax-jz10/input --arg s3n://oax-jz10/output1

Download output

s3cmd.rb get oax-jz10:output4/part-00000 out

Summary

• Map/Reduce may be simple

• Map/Reduce can be really powerful

• Collaborative filtering is fun :-)

Thank you

Ole-Martin Mørkolemartin@gmail.comtwitter.com/olemartin

del.icio.us/olemartin/jz10

All images are licensed with Creative Commons. See http://bit.ly/mr-photos for details,

Collaborative Filtering in Map/Reduce

Technology

Transcript of Collaborative Filtering in Map/Reduce

Socially Collaborative Filtering

CoBaFi : Collaborative Bayesian Filtering

Collaborative Filtering CAPTCHAs - Berkeley

Presented By :Ayesha Khan. Content Introduction Everyday Examples of Collaborative Filtering Traditional Collaborative Filtering Socially Collaborative.

DISTRIBUTED COLLABORATIVE FILTERING FOR ROBUST RECOMMENDATION AGAINST SHILLING ATTACKS DISTRIBUTED COLLABORATIVE FILTERING FOR ROBUST RECOMMENDATION AGAINST.

eecs349 collaborative filtering - GitHub Pages · Collaborative Filtering EECS 349 Machine Learning Bongjun Kim Fall, 2015. What is Collaborative Filtering? •Recommendation system

Case Study 4: Collaborative Filtering · Case Study 4: Collaborative Filtering Collaborative Filtering ! Goal: Find movies of interest to a user based on movies watched by the user

Boston.rb: Collaborative Filtering

Tutorial 14 (collaborative filtering)

Collaborative filtering intro - Full

Collaborative Filtering with Spark

Recommendations via Collaborative Filtering

Collaborative Filtering

Collaborative Filtering in iCAMP

Collaborative Filtering Recommendation

Collaborative Filtering - Practical Machine Learning, CS ... › ... › lectures › collaborative › slides.… · What is Collaborative Filtering? Insight: Personal preferences

Collaborative Filtering using KNN

1 Information Filtering Rong Jin. 2 Outline Brief introduction to information filtering Collaborative filtering Adaptive filtering.

Vertical Recommendation Using Collaborative Filtering

Collaborative Filtering Matrix Factorization Approachcourses.washington.edu/css581/lecture_slides/10...Collaborative Filtering Matrix Factorization Approach Jeff Howbert Introduction