Karthik Raman, Pannaga Shivaswamy & Thorsten Joachims Cornell University 1.
Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims...
-
Upload
maggie-addy -
Category
Documents
-
view
216 -
download
1
Transcript of Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims...
![Page 1: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/1.jpg)
Evaluating the Robustness of Learning from Implicit FeedbackFilip Radlinski
Thorsten Joachims
Presentation by Dinesh Bhirud
![Page 2: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/2.jpg)
Introduction
• The paper evaluates the robustness of learning to rank documents based on Implicit feedback.
• What is implicit feedback?– Relevance feedback obtained from search engine
log files– Easier to collect large amount of such training data
as against explicitly collecting relevance feedback.
![Page 3: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/3.jpg)
Osmot
• Osmot – Search engine developed at Cornell University based on Implicit Feedback
• Name Osmot comes from the word “osmosis” – learning from the users by osmosis
• Query Chains – Sequence of reformulated queries.– Osmot learns ranked retrieval function by
observing query chains and monitoring user clicks.
![Page 4: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/4.jpg)
High Level Block Diagram
Data generation
User behavior
simulation (based on original
ranking
fucntion)
Preference generation
SVM Learning
User behavior
simulatoin (based on learn
ed ranki
ng function)
![Page 5: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/5.jpg)
Data Generation
• Set of W words are chosen, word frequencies obeying a Ziph’s law
• T topics are picked by picking N words/topic uniformly from W.
• Each document d is generated as– Pick kd binomially from [0,T]– Repeat kd times
• Pick topic t• Pick L/kd words from topic t.
![Page 6: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/6.jpg)
Relevance
• 3 kinds of relevance– Relevance with respect to topic
• Can be measured/known because document collection and topics are synthetic
• Used for evaluating the ranking function.
– Relevance with respect to query• Actual relevance score of a document with respect to a query• Used to rank documents
– Observed relevance• Relevance of a document as judged by the user seeing only the
abstract.• Used to simulate user behavior.
![Page 7: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/7.jpg)
User behavior parameters
• Noise – Accuracy of user’s relevance estimate– Affects observed relevance. (obsRel)– obsRel is drawn from an incomplete Beta distribution
where α gives noise level and β is selected so that mode is at rel(d,q)
• Threshold – User selectivity over results (rT)• Patience – Number of results user looks at before
giving up (rP)• Reformulation – How likely is the user to
reformulate query(Preform)
![Page 8: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/8.jpg)
User Behavior ModelWhile question T is unanswered
1.1 Generate query q (Let d1,d2..,dn be results for q)1.2 Start with document 1 ie i = 11.3 while patience (Rp) > 0
1.3.1 if obsRel(di,q) > rT1.3.1.1 if obsRel(di+1, q) > obsRel(di,q) + c then continue
looking further in the list 1.3.1.2 else
di is a good document, click on it.If rel(di,T) is 1, user is DONEDecrease patience Rp.
1.3.2 elseDecrease patience RpRp = Rp - (rT – obsRel(di,q))
1.3. 3 Set i = i + 11.4 With probability (1 – Preform) , user gives up.
![Page 9: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/9.jpg)
User Preference Model
• Based on the clickthrough log files, users’ preferences for documents given query q can be found.
• Clickthrough logs generated by simulating users.
• From preference, features values are calculated.
![Page 10: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/10.jpg)
Feedback Strategies Single Query Strategy
• Click >q Skip Above– For query q, if document di is clicked, di is
preferred over all dj, j < i.
• Click 1st >q No-Click 2nd
– For query q, if document 1 is clicked, it is preferred over the 2nd document in the list.
![Page 11: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/11.jpg)
Feedback Strategies 2-Query Strategy 1
• This strategy uses 2 queries in a query chain, but document rankings only for the later query.
• Given queries q' and q in a query chain• Click >q' Skip Above– For query q', if document di is clicked in query q, di
is preferred over all dj, j < i • Click 1st > q' No-Click 2nd
– For query q', if document 1 is clicked, it is preferred over the 2nd document in the list for q
![Page 12: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/12.jpg)
Feedback Strategies 2-Query Strategy 2
• This strategy uses 2 queries in a query chain, and document rankings for both used.
• Given queries q' and q in a query chain• Click >q' Skip Earlier Query– For query q', if document di is clicked in query q, di is
preferred over seen documents in query previous query.
• Click > q' Top two earlier Query– If no document clicked for query q', then di preferred
over top two in previous query.
![Page 13: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/13.jpg)
Example Q1 Q2
D1 D4
D2 D5
D3 D6
Preferences• D2 >q1 D1• D4 >q2 D5• D4 >q1 D5• D4 >q1 D1• D4 >q1 D3
![Page 14: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/14.jpg)
Features
• Document di would be mapped to feature vector with respect to query q.
• 2 types of features defined– Rank Features – Term/Document Features
q) , (diØ
q) , (diØ q) , Ø(di
term
rank
q) , Ø(di
q) , (diØrank
q) , (diØterm
![Page 15: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/15.jpg)
Rank Features
• Rank features allow representation of ranking given by the existing static retrieval function.
• Used a simple TFIDF weighted cosine similarity metric (rel0)
• 28 rank features used for ranks 1,2,..,10,15,20,…100.
• Set to 1 if clicked document is at or above specified rank.
![Page 16: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/16.jpg)
Term Features
• Allows representation of fine grained relationship between query terms and documents.
• If for query q, document d is clicked, then for each word ,
• Forms a sparse feature vector, as only very few words are included in query.
qw 1 w), (dØterm
![Page 17: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/17.jpg)
Learning
• Retrieval Function rel(di, q) defined as
where is the weight vector. • Intuitively, weight vector assigns weight to each feature
identified.• Task of learning a ranking function is reduced to the task
of learning an optimal weight vector.
q) , Ø(di q) rel(di, w
w
![Page 18: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/18.jpg)
How does affect ranking?
• Points are ordered by their projections onto
• For the ordering will be 1,2,3,4.
• For the ordering will be 2,3,1,4.
• Weight vector needs to be learnt that will minimize number of discordant rankings.
w
w
1w
2w
w
![Page 19: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/19.jpg)
Learning Problem
Learning problem can be formalized as follows• Find weight vector such that maximum of following inequalities fulfilled. such that then• Without using slack variables, this is NP-hard
problem.
w
1),( rdd ji ),(),( 11 qdrqdr ji q) , Ø(d q) , Ø(d ji ww
![Page 20: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/20.jpg)
SVM Learning
• Equivalent optimization problem would be
Minimize
Subject to rearranging which we get constraint
and and
ij
Cww ij2
1
ij - 1 q) , Ø(dw q) , Ø(d:),,( ji
wjiq
ij - 1 q)) , Ø(d - q) , Ø(d(:),,( ji wjiq
0: ijij 01.0:]28,1[ iwi
![Page 21: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/21.jpg)
Re-ranking using the learnt model
• SVM-Light package is used.• Model provides values for all support
vectors.• User behavior is again simulated, this time using
the learnt ranking function.• How does reranking work?– First, a ranked list of documents is obtained using the
original ranking function.– This list is re-ordered, using the weights of each feature
obtained from the learnt model.
y
![Page 22: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/22.jpg)
Experiments
• Experiments done to study the behavior of the search engine by varying parameters like– Noise in users’ relevance judgement– Ambiguity of words in topics and queries– Threshold value which user considers good
document– Users’ trust in ranking– Users’ probability of reformulation of query.
![Page 23: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/23.jpg)
Results - Noise
0 1 2 3 4 5 670
75
80
85
90
95
100
Ranking Function performance at various noise levels
Noise Low Noise Medium Noise High Maximum Noise
Learning Iterations
Expe
cted
Rel
evan
ce
![Page 24: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/24.jpg)
Noise – My experiment
• Did implementation for extracting preferences and encoding them in features.
0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.000070
72
74
76
78
80
82
84
86
Ranking function performance at various noise levels (My implementation)
Noise LowNoise MediumNoise High
Learning Iterations
Expe
cted
Rel
evan
ce
![Page 25: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/25.jpg)
Topic and Word Ambiguity
0 1 2 3 4 5 670
75
80
85
90
95
100
Ranking function performance at different levels of word ambiguity
No ambiguous wordsWords somewhat ambiguouosWords more ambiguous
Learning Iterations
Expe
cted
Rel
evan
ce
![Page 26: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/26.jpg)
Probability of user reformulating query
0 1 2 3 4 5 670
75
80
85
90
95
100
25% give up probability50% Give up probabilty75% Give up probability100% Give up probability
Learning Iterations
Expe
cted
Rel
evan
ce
![Page 27: Evaluating the Robustness of Learning from Implicit Feedback Filip Radlinski Thorsten Joachims Presentation by Dinesh Bhirud bhiru002@d.umn.edu.](https://reader033.fdocuments.in/reader033/viewer/2022051819/55173821550346fe558b6008/html5/thumbnails/27.jpg)
Thank You