Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

27
1 (A Few) Key Lessons Learned Building Recommender Systems For Large-scale Social Networks

TAGS:

description

Invited Talk at KDD 2012 (Industry Practice Expo) http://kdd2012.sigkdd.org/indexpo.shtml#posse Abstract: By helping members to connect, discover and share relevant content or find a new career opportunity, recommender systems have become a critical component of user growth and engagement for social networks. The multidimensional nature of engagement and diversity of members on large-scale social networks have generated new infrastructure and modeling challenges and opportunities in the development, deployment and operation of recommender systems. This presentation will address some of these issues, focusing on the modeling side for which new research is much needed while describing a recommendation platform that enables real-time recommendation updates at scale as well as batch computations, and cross-leverage between different product recommendations. Topics covered on the modeling side will include optimizing for multiple competing objectives, solving contradicting business goals, modeling user intent and interest to maximize placement and timeliness of the recommendations, utility metrics beyond CTR that leverage both real-time tracking of explicit and implicit user feedback, gathering training data for new product recommendations, virility preserving online testing and virtual profiling.

Transcript of Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Page 1: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

1

(A Few) Key Lessons Learned Building Recommender Systems For Large-scale Social Networks

Page 2: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

World’s Largest Professional Network

2

2 4 8

17

32

55

90

2004 2005 2006 2007 2008 2009 2010 2011 LinkedIn Members (Millions)

175M+

25th Most visit website worldwide (Comscore 6-12)

Company pages

>2M

62% non U.S.

2/sec

85% Fortune 500 Companies use LinkedIn to hire

Page 3: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

3

LinkedIn Homepage

Powered by Recommendations!

Page 4: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Similar Profiles

Events You May Be Interested In

News

The Recommendations Opportunity

4

Network updates

Connections

Page 5: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

What are they worth? Think 50%

!  > 50% of connections are from recommendations (PYMK)

!  > 50% of job applications are from recommendations (JYMBII)

!  > 50% of group joins are from recommendations (GYML)

5

Page 6: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

6

What is a Recommender System?

A Recommender selects a product that if acquired by the “buyer” maximizes value of both “buyer” and “seller” at a given point in time

Page 7: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

7

Lesson 1 : Recommendations must make strategic sense…

!  Conflicts of interest between ‘buyers’ !  E.g. job posters vs. job seekers

!  What is the (long-term) value of an action? !  How do we compare an invitation from a job application from a

group join or the reading of a news?

Page 8: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Ingredients of a Recommender System

A Recommender processes information and transforms it into actionable knowledge

Data (Feature

Engineering) Algorithms Business Logic

and Analytics User Experience

8

Page 9: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

9

Lesson 2: User Experience Matters Most

1.  Understand user intent " Define, model, leverage

2.  Be in the user flow 3.  Optimize location on the page 4.  Set right expectations (“You May…”) 5.  Explain recommendations 6.  Interact with the user 7.  Leverage Social Referral

Page 10: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

10

User Intent, User Flow, Location, Message

User experience

Optimization Impact on job application rate

User intent /location

Homepage personalization

2.5X

User flow / user intent

Before vs. after having applied to a job

7X

User flow LinkedIn homepage vs. Jobs homepage

10X

Location Center rail vs. right rail 5X Message Followers vs. leaders 2X

Item based collaborative filtering: " Follower audience Content based:

" Leader audience

Job recommendation use cases

Page 11: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

1.  Intent labeling •  Job seeker, recruiter, outbound professional, content

consumer, content creator, networker, profile curator

2.  Build normalized propensity model for each intent

11

User Intent Modeling

Page 12: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

12

Job Seeker Intent !  Features

!  Behavior: job searches, views & applications, job related email replies, has a job seeker sub, …

!  Social graphs: # of colleagues in network who recently left, …

!  Content: title, industry, anniversary date, …

!  Propensity Score !  Parametric accelerated failure

time survival model !  log Ti = !k"kxik+#$i

!  Score: P(switch job next month) !  Evaluation

!  Gold standard !  Directional validation

Page 13: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

13

Lesson 3: Recommendations often cater to multiple competing objectives

Suggest relevant groups … that one is more likely to participate in

Suggest skilled candidates … who will likely respond to hiring managers inquiries

TalentMatch (Top 24 matches of a posted job for sale)

Semantic + engagement objectives

Page 14: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

14

Multiple Objective Optimization

Constraint optimization problem 1.  Rank top K’ > K results wrt to primary objective (e.g.

relevance) 2.  Perturb the ranking with a parametric function which

leads to the inclusion of secondary objective(s) (e.g. engagement) !  Measure the perturbation using a distance

function wrt primary objective !  Create a framework to quantify the tradeoff

between the objectives

Page 15: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

15

TalentMatch Use Case I

•  Proxy for likelihood to answer hiring managers inquiries: •  Job seekers are 16X more likely to answer than non-seekers

" Increase % of job seekers in in TalentMatch results •  Control for TalentMatch booking rate and sent emails

rate

Distribution of min TalentMatch scores over 1 month of jobs posted on LinkedIn

Page 16: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

16

TalentMatch Use Case II

Page 17: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Data Sources for Feature Engineering

17

Content

Social Graphs

http://inmaps.linkedinlabs.com/

…

Behavior

PVs Actions (clicks)

Queries

Page 18: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Lesson 4: The Unreasonable Effectiveness of Big Data !  Data jujitsu: slice and dice (smartly), then count !  Rich features engineered from profiles jujitsu

!  Job tenure distributions !  Job transition probabilities !  Related titles, industries, companies !  Profile & job seniority

!  Impact on job recommendations: !  25% lift in views, viewers, applications, applicants !  90% drop in negative feedback

18

Page 19: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Lesson 4: The Unreasonable Effectiveness of Big Data (cont.)

!  Region stickiness !  Related regions

!  Impact on job recommendations: 20% lift in views, viewers, applications, applicants

!  Individuals propensity to migrate

19

Page 20: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

20

Lesson 5: Data Labeling Can Be Daunting !  Historical data

!  Similar objective !  Unrelated processes, e.g., same session search selection

!  reduce presentation bias, position bias !  What about intent bias?

!  Random suggestions !  Great with ads, company follows !  Not for products with high cost of bad recommendations

!  jobs, alumni groups, … !  Not for similar recommendations

!  Crowdsourced manual labeling !  Very challenging !  Pairwise comparison more suited than absolute rating " higher cost !  Expert-sourced manual labeling

Page 21: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

21

Lesson 6: Measure Everything

1.  Implicit user feedback !  E.g. impressions, immediate actions (clicks), secondary actions !  Understand and optimize flows, e.g.,

!  Impact of ‘See more’ link and landing page !  Conversion rate of a job view into a job application from various

channels !  E.g. Homepage vs email vs mobile

Page 22: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

22

Lesson 6: Measure Everything (cont.)

2.  Explicit user feedback !  Understand usefulness of the recommendation with unequal depth

!  Text based A/B test! !  Help prioritize future improvements !  Reveal unexpected complexities

!  E.g. ‘Local’ means different things for different members !  Prevent misinterpretation of implicit user feedback!

Page 23: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

23 23 23

1.  Novelty effect !  E.g., new job recommendation algorithms

have week-long novelty effect that shows lifts twice the stationary (real) one

2.  Cannibalization !  Zero-sum game or real lift?

3.  Random sampling destroys network effect

Lesson 7: Beware of some A/B testing pitfalls

!"

#$!!!"

%$!!!"

&$!!!"

'$!!!"

($!!!"

)$!!!"

*$!!!"

+$!!!"

,$!!!"

!" (" #!" #(" %!" %("

!"#$%&'()$*'+$,-$#./0'1$+234'$5$67,788$

-./"01234"526"(7"/89:2;"6<=>2"?")@(@##"

!"

#$!!!"

%$!!!"

&$!!!"

'$!!!"

($!!!"

)$!!!"

*$!!!"

!" (" #!" #(" %!" %("

!"#$%&'()$*+,-+,,$

+,-"./012")3#43##"

1 week lifts 2weeks lifts

Page 24: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

24

Lesson 8: One Unified Platform, One API

!  Scaling innovation !  Cross-leverage improvements between products !  Shared knowledgebase

!  Maintainability

!  Production serving and tracking !  Infrastructure for complete upgrades

!  Performance !  Billions of sub second computations

Page 25: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

Open Source Technologies

Zoie Bobo

Kafka Voldemort

25 http://sna-projects.com/sna/

Page 26: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

26

Conclusion

!  Social/professional networks are a new frontier for recommender systems

!  Still many open questions: !  How do we define and measure engagement? !  What is the utility of a recommendation for the member? !  What is the value of a recommendation for the network? !  How do we reconcile utility and value when they conflict? !  How do we network A/B test without tears? !  …

!  Learn as you go !  Track everything and invest in forensics analytics

!  Breadth – understand holistic impact !  Depth – understand flows

Page 27: Key Lessons Learned Building Recommender Systems for Large-Scale Social Networks (KDD 2012)

27

•  M. Rodriguez, C. Posse and E. Zhang. 2012. Multiple Objective Optimization in Recommendation Systems. To appear in Proceedings of the Sixth ACM conference on Recommender systems (RecSys '12)

•  M. Amin, B. Yan, S. Sriram, A. Bhasin and C. Posse. 2012. Social Referral : Using Network Connections to Deliver Recommendations. To appear in Proceedings of the Sixth ACM conference on Recommender systems (RecSys '12)

•  A. Reda, Y. Park, M. Tiwari, C. Posse and S. Shah. 2012. Metaphor: Related Search Recommendations on a Social Network. 2012. To appear in Proceedings of the 21st International Conference on Information and Knowledge Management (CIKM ‘12)

•  B. Yan, l. Bajaj and A. Bhasin. 2011. Entity Resolution Using Social Graphs for Business Applications. In Proceedings of the International Conference on Advances in Social Network Analysis and Mining (ASONAM ‘11)

References


Training and Testing of Recommender Systems on Data Missing Not at Random Harald Steck at KDD, July 2010 Bell Labs, Murray Hill.

Training and Testing of Recommender Systems on Data Missing Not at Random Harald Steck at KDD, July 2010 Bell Labs, Murray Hill.

Recsys 2016 tutorial: Lessons learned from building real-life recommender systems

Recsys 2016 tutorial: Lessons learned from building real-life recommender systems

KDD 2011 Doctoral Session

KDD 2011 Doctoral Session

KDD 2010 Opening Ceremony FINAL

KDD 2010 Opening Ceremony FINAL

KDD Cup Research Paper

KDD Cup Research Paper

INTRODUCTION TO RECOMMENDER - Leuphana … · SYSTEMS AND THEIR EVALUATION Olga ... recommender systems Introduction Properties of recommender ... 11 Tutorial: Recommender Problems

INTRODUCTION TO RECOMMENDER - Leuphana … · SYSTEMS AND THEIR EVALUATION Olga ... recommender systems Introduction Properties of recommender ... 11 Tutorial: Recommender Problems

Malstone KDD 2010

Malstone KDD 2010

Recommender Systems an Introduction Chapter07 Evaluating Recommender Systems

Recommender Systems an Introduction Chapter07 Evaluating Recommender Systems

KDD Cup 2009

KDD Cup 2009

Personalized Recommender by Exploiting Domain based Expert ... · Incremental Collaborative Filtering Recommender system. B. Content-Based Recommender System Content based recommender

Personalized Recommender by Exploiting Domain based Expert ... · Incremental Collaborative Filtering Recommender system. B. Content-Based Recommender System Content based recommender

TFR: A Tourist Food Recommender System based on Collaborative Filtering · 2018-08-14 · Artificial Intelligence, Recommender System Keywords Food Recommender System, Tourist Recommender

TFR: A Tourist Food Recommender System based on Collaborative Filtering · 2018-08-14 · Artificial Intelligence, Recommender System Keywords Food Recommender System, Tourist Recommender

d d D^ D > > KE Z > Z s/ Zd,hZ D^ DKZ ... Unclaimed Capital Credit List by Name.pdfd d &/ > Kdd Z D/ , >> Kdd KZKd,z ' Kdd '/E D Z/ Kdd

d d D^ D > > KE Z > Z s/ Zd,hZ D^ DKZ ... Unclaimed Capital Credit List by Name.pdfd d &/ > Kdd Z D/ , >> Kdd KZKd,z ' Kdd '/E D Z/ Kdd

Lessons learned from Large Scale Real World Recommender Systems

Lessons learned from Large Scale Real World Recommender Systems

Recommender Systems Recommender Systems

Recommender Systems Recommender Systems

KDD tutorial

KDD tutorial

The Yahoo! Music Dataset and KDD-Cup’11proceedings.mlr.press/v18/dror12a/dror12a.pdf · the challenge is related to the fact that learning a large scale recommender systems is a

The Yahoo! Music Dataset and KDD-Cup’11proceedings.mlr.press/v18/dror12a/dror12a.pdf · the challenge is related to the fact that learning a large scale recommender systems is a

Virtual KDD 2020 Prospectus Final 6.2.20€¦ · KDD. Supporting KDD through sponsorship is a way for you to support the international community of researchers that are working on

Virtual KDD 2020 Prospectus Final 6.2.20€¦ · KDD. Supporting KDD through sponsorship is a way for you to support the international community of researchers that are working on

Srivastava KDD 2010 Presentation

Srivastava KDD 2010 Presentation

Paper Trees Kdd

Paper Trees Kdd

Data Mining And KDD

Data Mining And KDD