Towards Complex User Feedback and Presentation Context in
Recommender Systems
Peter Vojtas and Ladislav Peška
Department of Software Engineering, Charles University in Prague,
Czech Republic
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
2
Recommender Systems Propose relevant items to the right persons at the right time Machine learning application Expose otherwise hard to find, uknown items Complementary to the catalogues, search engines etc. „Win-win strategy“
PPI 2017, Stuttgart, Germany
User Feedbackrating, clickstream,time on page, buys…
User, Object ProfilesObject attributes
(Context)Time, location,Possible choices…
RECOMMENDERSYSTEM
Top-K Recommended objects
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
3
Recommender Systems User feedback
Explicit feedback (user rating) Implicit feedback (user behavior)
User visited/bought object Usually binary feature
Recommending algorithms Collaborative filtering
(Users A and B were similar so far, the should like similar things in the future too)
Cold start problem Content-based filtering
(User A should like similar items to the ones he liked so far) Overspecialization, lack of diversity, obvious recommendations…
PPI 2017, Stuttgart, Germany
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
4
Challenge Recommending for small e-commerce websites
Tens of similar vendors, user can choose whichever she likes (Almost) no explicit feedback
(No incentives for users) Few visited pages
(Often usage of external search engines & landing on object details) Low user loyalty
(New vs. Returning visitors ratio 80:20)Þ Not enough data for collaborative filtering,
continuous cold-start problem
Focus on Implicit Feedback & Content-based recommendations Obtain precise implicit user feedback in enough quantity, as early as possible Gather external content to improve CB recommendations (other papers)
PPI 2017, Stuttgart, Germany
User FeedbackExplicit feedback Provided via website GUI
Rating an object via Likert Scale
Missing in small E-Commerces
Implicit feedback Often binary in the literature
User visited object User bought object
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
5
User FeedbackExplicit feedback Provided via website GUI
Rating an object via Likert Scale Comparing objects explicitly is
not so common Missing in small E-Commerces
Implicit feedback Often binary in the literature
User visited object User bought object
Virtually any event triggered by user could be feedback
Get better picture about user engagement / preference
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
6
User FeedbackExplicit feedback Provided via website GUI
Rating an object via Likert Scale Comparing objects explicitly is
not so common Missing in small E-Commerces
Implicit feedback Virtually any event could be
used as feedback Tracked via JavaScript
Dwell time Number of page views, Scrolling,
mouse events, copy text, printing Purchase process etc.
Purchases represents fully positive feedback
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
7
User Feedback
Software: Peska, IPIget: The Component for Collecting Implicit User Preference Indicators
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
8
User Feedback – Past Resarch Combine multiple implicit feedback features to estimate user rating
Standard CB / CF recommender systems can be used afterwards
Improvements over the usage of simple implicit feedback
Peska, Vojtas: How to Interpret Implicit User Feedback?Peska, Eckhardt, Vojtas: Preferential Interpretation of Fuzzy Sets in E-shop Recommendation with Real Data Experiments
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
9
Dwell time: 10sScrolling: 100pxMouse movement:250px
Dwell time: 100sScrolling: 200pxMouse movement:450px
Rating: 0.2
Rating: 0.8
User Feedback – Past Resarch Combine multiple implicit feedback features to estimate user rating
Standard CB / CF recommender systems can be used afterwards
Improvements over the usage of simple implicit feedback
Is that all we can do?PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
10
Dwell time: 10sScrolling: 100pxMouse movement:250px
Dwell time: 100sScrolling: 200pxMouse movement:450px
Rating: 0.2
Rating: 0.8
Context of User Feedback Combine multiple implicit feedback features to estimate user rating
Is that all we can do?
Pages may substantially vary in length, amount of content etc. This could affect perceived implicit feedback features Leveraging context could be important
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
11
Dwell time: 10sScrolling: 100pxMouse movement:250px
Dwell time: 100sScrolling: 200pxMouse movement:450px
Rating: 0.2
Rating: 0.8
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
12
Context of User Feedback
PPI 2017, Stuttgart, Germany
A B
Context of the user Location, Mood, Seasonality... Can affect user preference Out of scope of this paper
Context of device and page Page and browser dimensions Page complexity (amount of text, links, images,...) Device type Datetime Can affect percieved values of the user feedback
Outline of Our ApproachTraditional recommender User rates a sample of objects
Preference learning computes expected ratings of all objects
Top-k best rated objects are recommended
Our approach
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
13
𝑟𝑢 ,𝑜 :𝑜∈𝑺⊂𝑶 ;𝑟𝑢 ,𝑜∈[0,1]
�̂�𝑢={𝑜1 ,…,𝑜𝑘 }
¿
¿𝐹 𝑢 ,𝑜 ,𝐶𝑢 ,𝑜→𝑟𝑢 ,𝑜 :𝑜∈𝑺
Several imlicit feedback and contextual features are collected:
Learn estimated rating for visited objects based on feedback and context
„The more the better” heuristics (STD, CDF) Machine learning approach (J48)
Incorporate context As further feedback features (FB+C) As baseline predictors (AVGBP, CBP)
Learn rating on all objects as in traditional recommenders
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
14
Collecting User Behavior IPIget component for collecting user behavior
IPIget component download: http://ksi.mff.cuni.cz/~peska/ipiget.zip
PPI 2017, Stuttgart, Germany
Contextual featuresNumber of links
Number of images
Text size
Page dimensions
Visible area ratio
Hand-held device
Implicit Feedback FeaturesView Count
Dwell Time
Mouse Distance and Time
Scrolled Distance and Time
Clicks count
Hit bottom of the page
Purchase
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
15
Estimated Rating from Implicit Feedback
PPI 2017, Stuttgart, Germany
„The more the better” heuristics Various feedback features are not comparable in general
Dwell time (sec) vs. Distance travelled by mouse (pixels) Transform feedback features on comparable scale and average
Use standardization (STD) of feedback features (≈ N(0,1))
Use cummulative distribution (CDF) of each feedback feature
=>
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
16
Estimated Rating from Implicit Feedback
PPI 2017, Stuttgart, Germany
Machine learning approach J48 decision tree Purchases are golden standard
The only feedback which is a true indicator of positive preference Predict purchases based on other feedback features Use probability of purchase as estimated rating
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
17
Employ Context in Rating Estimation
PPI 2017, Stuttgart, Germany
Use context in the same way as feedback (FB+C) Leave the decision about usage of context on the underlined model Plausible strategy for e.g. decision trees or rule mining learning
approaches Use context as a baseline predictor of feedback
Calculate estimated value of feedback feature for particular context value
Substract the estimation from the actual value Use feedback with baseline estimators instead of the original one
Either employ average baseline predictor over all context features (AVGBP)
Or use carthesian product of feedback features and baseline predictors based on each context feature (CBP)
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
18
Preference Learning and Recommendations
PPI 2017, Stuttgart, Germany
Collaborative filtering not applicable Continuous cold-start problem
Use combination of content-based and non-personalized VSM content-based recommendation
Vector of object features (TF-IDF) User is represented as weighted sum of visited object’s features Resulting score is a cosine similarity of user and object vectors
Most popular non-personalized algorithm Based on estimated ratings
Final score is a multiplication of VSM score and most popular score
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
19
Evaluation
PPI 2017, Stuttgart, Germany
Czech travel agency dataset 3 variants of rating estimation (STD, CDF, J48) 3 variants of context incorporation (FB+C, AVGBP, CBP) 2 baselines (use raw feedback, use binary visits)
Leave-one-out on purchased objects Ranking prediction nDCG, recall@top-10
Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
20
Results
Results of nDCG, (*) = signifficant improvement of the best method J48 decision tree with both feedback and context on its input performs the
best Using „the more the better“ heuristics (CDF) with properly processed feedback
(AVGBP, CBP) also performs quite well
PPI 2017, Stuttgart, Germany
Processing method
Feedback and Context composition
Binary FB FB+C AVGBP CBP
STD + popVSM 0.255* 0.174* 0.197* 0.161* 0.158*CDF + popVSM 0.255* 0.257* 0.253* 0.258* 0.257J48 + popVSM 0.255* 0.256* 0.274 0.240* 0.247*J48 + objects popularity 0.180** 0.205* 0.211* 0.168* 0.186*
J48 + VSM 0.222* 0.224* 0.233 0.225* 0.224*
Conclusions, Future WorkKey outcomes Implicit feedback could be more than just a binary variable Observed feedback should be considered with respect to the context of
page and device Doing so could improve the quality of the recommended objects
Future work, Open Problems Better models of context employment and purchase prediction methods Further evaluation scenarios
Recommending on the beginning of a new session More refined feedback?
E.g. feedback on object’s attributes? On-line deployment and evaluation
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
21
PPI 2017, Stuttgart, Germany Peska, Vojtas: Towards Complex User Feedback and Presentation Context in
Recommender Systems
22
Thank you!
Questions, comments?
Supplementary materials: http://bit.ly/2g79VVOSlides:
Top Related