Personalizing Atypical Web Search Sessions (WSDM'13)

February 7, 2013

Challenge the future

Delft University of Technology

Personalizing Atypical Web Search Sessions

Carsten Eickhoff, Kevyn Collins-Thompson, Paul Bennett and Susan Dumais

2 Personalizing Atypical Web Search Sessions

Introduction

• Web search personalization is used to account for user

preferences based on historic observations

• This seems appropriate for typical search tasks

• Atypical search tasks in unfamiliar domains may not

benefit as much from personalization


Overview

1. Investigation of nature, frequency and cause of atypical search sessions

2. Automatic prediction of atypical search sessions

3. Personalization of atypical search sessions


Atypical Search Tasks

• Motivation • Atypical search tasks are often caused by external needs

• Topic domain

• They cover unfamiliar, previously unseen topics & genres

• Behavior

• Due to limited domain knowledge, the searcher may encounter problems during query formulation and result selection


Atypical Search Sessions

• Sessions tend to be task-centered • 200 most active Bing users in January 2012

• Navigational queries removed

• Based on a profile, each session is manually judged as

typical or atypical


Human-readable User Profiles

• 55% Football(“nfl”,”philadelphia eagles”,”mark sanchez”)

• 14% Boxing(“espn boxing”,”mickey garcia”,”hbo boxing”)

• 09% Television(“modern familiy”,”dexter 8”,”tv guide”)

• 06% Travel(“rome hotels”,“tripadvisor seattle”,“rome pasta”)

• 05% Hockey(“elmira pioneers”,”umass lax”,”necbl”)


Example 1:






• Boxing(“soto vs ortiz”)

• Boxing(“humberto soto”)


Example 2:






• Dentistry(“oral sores”)

• Dentistry(“aphthous sore”)

• Healthcare(“aphthous ulcer treatment”)


Frequency of Atypical Sessions

• 166 out of 2790 sessions (~6%) were judged atypical

• 74% of all users show atypical search sessions

• On average, 7.5% of a user's query volume is atypical


Typical vs. Atypical Sessions

• Atypical sessions have:




• longer queries (often natural language questions)





• more diverse query vocabulary






• higher SAT reading level






• higher SAT reading level

• Observed differences persist within profiles


Topic Spread

• Most frequent topics for typical sessions include (sports,

celebrities & gossip, entertainment)

• Often typicality describes the cut between what you choose to

do (typical) and what you have to do (atypical)


Predicting Atypical Sessions

• User profiling based on activity in Jan. 2012

• Post hoc classification of sessions in Apr. 2012

• Session features (25)

• Session length

• Query length

• Question words

• POS ratios

• Longest query position

• Reading level

• ...

• Profile features (27)

• δ session length

• δ query length

• …

• Query vocabulary

divergence

• Topic divergence


Classification Performance

• Logistic regression model

• 52-dimensional feature space

• CV performance F1 0.84 (P 0.82 / R 0.86) • The resulting performance is comparable to the

agreement among human judges


Strongest Typicality Indicators

1. Query length divergence from profile

2. Absolute query length

3. Question word ratio

4. Verb ratio divergence from profile

5. Topic divergence from profile


Robustness to Sparsity

• It takes 15-20 sessions to reliably characterize users

• Most users reach this point in 14 days of search activity


Retrieval Performance

• Moving from qualitative setting to web scale

• 155k users

• 10.4M sessions

• LambdaMART learning scheme

• Re-ranking the top 10 returned results


Profiling Scopes

• Session

• All previous activity in the same session

• Historic

• All previous activity before this session

• Aggregate

• All previous activity before current query


Personalization Performance


Hybrid Approach


Hybrid Performance


Conclusion

• Investigation of atypical web search sessions • Atypical sessions occur for most users • They account for a significant proportion of queries

• Typicality Prediction • Automatic prediction comparable to human accuracy

• Benefit for Retrieval Settings • Including typicality information into the personalization

process improved retrieval performance


Future Work

• Classification of ongoing sessions • Currently looking at last query, do it earlier for more impact • Weakly supervised approach based on pre-classified

complete sessions

• From atypical to typical • How do information needs change over time? • Life cycle of an information need / a profile

• Information value at risk • How important is a piece of information for the user? • How much effort will they invest?


Thank You!

[email protected]

Personalizing Atypical Web Search Sessions (WSDM'13)

Technology

Transcript of Personalizing Atypical Web Search Sessions (WSDM'13)