“Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ......
Transcript of “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ......
![Page 1: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/1.jpg)
“Engagemoi”:Fromretrievaleffec4veness,usersa4sfac4ontouserengagement
Mounia Lalmas Yahoo London [email protected]
![Page 2: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/2.jpg)
This talk
§ Evaluation in search § Interpreting the signals
§ From intra-session to inter-session evaluation › Search › Others
happy users come back
User engagement
![Page 3: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/3.jpg)
Our case study: Evaluation in search
![Page 4: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/4.jpg)
How to evaluate in search
§ Coverage§ Speed§ Querylanguage§ Userinterface
§ UserhappinessUsersfindwhattheywantandreturntothesearchsystem
§ Butletusremember:Incarryingoutasearchtask,searchisameans,notanend
Sec. 8.6
(Manning, Raghavan & Schütze, 2008; Baeza-Yates & Ribeiro-Neto, 2011)
![Page 5: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/5.jpg)
Measuring user happiness Mostcommonproxy:relevanceofretrievedresults
Sec. 8.1
Relevant
Retrieved
all items
§ Userinforma(onneedtranslatedintoaquery
§ Relevanceassessedrela4vetoinforma(onneednotthequery
§ Example:› Informa4onneed:Iamlookingfortennis
holidayinacountrywithnorain› Query:tennisacademygoodweather
Evaluation measures: • precision, recall, R-precision; precision@n; average precision; F-measure; … • bpref; cumulative gains, rank-biased precision, expected reciprocal rank, Q-measure, …
precision
recall
![Page 6: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/6.jpg)
Measuring user happiness Mostcommonproxy:relevanceofretrievalresults
Sec. 8.1
Explicit signals Test collection methodology (TREC, CLEF, NCTIR, …) Human labeled corpora
Implicit signals User behavior in online settings (clicks, skips, …)
Explicit and implicit signals can be and are used together
![Page 7: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/7.jpg)
What is a happy user in search
1. The user information need is satisfied 2. The user has learned about a topic and even
about other topics 3. The system was inviting and even fun to use
In-the-moment engagement Users on a site Long-term engagement Users come back frequently
USER ENGAGEMENT
![Page 8: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/8.jpg)
How can we know whether a user is happy? Interpreting the signals
![Page 9: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/9.jpg)
Examples of implicit signals § Number of clicks
§ SAT click § Quick-back click
§ Click at given position § Time to first click
§ Skipping
§ Abandonment rate § Number of query reformulations
§ Dwell time § Hover
![Page 10: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/10.jpg)
Click-through rate
CTR
new ranking algorithm new design of search result page …
![Page 11: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/11.jpg)
Multimedia search activities often driven by entertainment needs, not by information needs
Relevance in multimedia search
(Slaney, 2011)
Clicks (I)
![Page 12: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/12.jpg)
(Miliaraki, Blanco & Lalmas, 2015)
Clicks (II) Explorative and serendipitous search
![Page 13: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/13.jpg)
I just wanted the phone number … I am totally happy J
No click: Abandonment (I) Information-rich snippet
![Page 14: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/14.jpg)
§ Abandonment is when there is no click on the search result page › User is dissatisfied (bad abandonment) › User found result(s) on the search result page (good abandonment)
§ 858 queries (21% good vs. 79% abandonment manually examined) § Cursor trail length
› Total distance (pixel) traveled by cursor on SERP › Shorter for good abandonment
§ Movement time › Total time (second) cursor moved on SERP › Longer when answers in snippet (good abandonment)
§ Cursor speed › Average cursor speed (pixel/second) › Slower when answers in snippet (good abandonment)
(Huang et al, 2011)
No click: Abandonment (II)
![Page 15: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/15.jpg)
“reading” cursor heatmap of relevant document vs “scanning” cursor heatmap of non-relevant document (both dwell time of 30s)
(Guo & Agichtein, 2012)
Dwell time (I)
![Page 16: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/16.jpg)
Dwell time (II)
“reading” a relevant long document vs “scanning” a long non-relevant document
(Guo & Agichtein, 2012)
![Page 17: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/17.jpg)
Mobile advertising: Dwell time (III)
DWELL TIME used a proxy of user experience
Publisher click on an ad on mobile device
Dwell time on non-optimized landing pages comparable and even higher than on mobile-optimized ones
… when mobile optimized, users realize quickly whether they “like” the ad or not?
(Lalmas etal, 2015)
non-mobile optimized mobile optimized
![Page 18: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/18.jpg)
User happiness User engagement
![Page 19: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/19.jpg)
“User engagement is a quality of the user experience that emphasizes the phenomena associated with wanting to use a technological resource longer and frequently” (Attfield et al, 2011)
19
happy users come back
![Page 20: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/20.jpg)
20
What you want to optimize for
each task, session, query
What you want to optimize long-
term Mi LTVj
happy users come back
![Page 21: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/21.jpg)
Happy users come back From intra- to inter-session evaluation
![Page 22: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/22.jpg)
From intra- to inter-session evaluation
What you want to optimize for
each task, session, query
search metrics
(signals)
absence time
(revisit the
site)
Mi LTVj
What you want to optimize long-
term Search system
Models
Features
![Page 23: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/23.jpg)
intra-session search metrics
• Dwell time • Number of clicks • Time to 1st lick • Skipping • Click through rate • Abandonment rate • Number of query
reformulations • …
Dwell time as a proxy of user interest Dwell time as a proxy of relevance
User engagement metrics for search (Proxy: relevance of search results)
intra-session
inter-session
![Page 24: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/24.jpg)
Search result page for “asparagus” (I)
![Page 25: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/25.jpg)
Search result page for “asparagus” (II)
![Page 26: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/26.jpg)
Absence time and survival analysis
story 1story 2story 3story 4story 5story 6story 7story 8story 9
0 5 10 15 20
0.0
0.2
0.4
0.6
0.8
1.0
Users (%) who did come back
Users (%) who read story 2 but did not come back after 10 hours
SURVIVE
DIE
DIE = RETURN TO SITE è ABSENCE TIME è Hazard Rate
hours
![Page 27: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/27.jpg)
Absence time applied to search Ranking functions on Yahoo Answer Japan
Two-weeks click data on Yahoo Answer Japan: search One millions users Six ranking functions
Session boundary: 30 minutes of inactivity
![Page 28: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/28.jpg)
survival analysis: high hazard rate (die quickly) = short absence
5 clicks
cont
rol =
no
clic
k
Absence time and number of clicks on search result page
3 clicks
§ No click means a bad user experience § Clicking between 3-5 results leads to same user experience § Clicking on more than 5 results reflects poorer user experience; users cannot
find what they are looking for (Dupret & Lalmas, 2013)
![Page 29: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/29.jpg)
Using DCG versus absence time to evaluate five ranking functions
DCG@1 Ranking Alg 1 Ranking Alg 2 Ranking Alg 3 Ranking Alg 4 Ranking Alg 5
DCG@5 Ranking Alg 1 Ranking Alg 3 Ranking Alg 2 Ranking Alg 4 Ranking Alg 5
Absence time Ranking Alg 1 Ranking Alg 2 Ranking Alg 5 Ranking Alg 3 Ranking Alg 4
(Dupret & Lalmas, 2013)
![Page 30: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/30.jpg)
Absence time and search experience
From 21 experiments carried out through A/B testing, using absence time agrees with 14 of them (which one is better)
(Chakraborty etal, 2014)
Positive signals • One more query in session • One more click in session • SAT clicks • Query reformulation
Negative signals • Abandoned session • Quick-back clicks
search session metrics à absence time
![Page 31: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/31.jpg)
The complete methodology?
![Page 32: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/32.jpg)
Large-scale online measurement
Decide the in-the-moment
metric(s)
Decide the long-term-value metric(s)
System
Models
Features
Which in-the-moment metric(s)
are good predictor of long-
term value metric(s)
Optimize for the identified in-the-
moment metric(s)
Lots of data required to
remove noise
![Page 33: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/33.jpg)
Happy users come back Other examples
![Page 34: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/34.jpg)
Dwell time and absence time for landing page quality
0%
200%
400%
600%
short ad clicks long ad clicks
ad c
lick
diffe
renc
e
Dwell time à ad click
Positive post-click experience (“long” clicks) has an effect on users clicking on ads again
(mobile)
(Lalmas etal, 2015)
Absence time: • return to publisher • click on an ad
![Page 35: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/35.jpg)
Relatedoff-sitecontent
Off-site links and absence time in focused news reading
Off-site link à absence time Providing links to related off-site content has a positive long-term effect (for 70% of news sites, probability that users return within 12 hours increases by 76%)
(Lehmann etal, 2016)
News provider
p(a
bse
nce
12
h)
Non-focused Focused Ext-focused
![Page 36: “Engage moi”: From retrieval effecveness, user sasfacon to ... · Information-rich snippet ... › Longer when answers in snippet (good abandonment) ... 2012) Dwell time (II)](https://reader033.fdocuments.in/reader033/viewer/2022043001/5f7a2c2993b5d05ab5748d23/html5/thumbnails/36.jpg)
Final words
happy users come back
we need to properly identify that a user is
happy
&