Diane Kelly, Filip Radlinski, Jaime Teevan

Observational Approaches to Information RetrievalSIGIR 2014 Tutorial: Choices and Constraints (Part II)

Diane Kelly, Filip Radlinski, Jaime Teevan

Slides available at: http://aka.ms/sigirtutorial

Diane Kelly, University of North Carolina, USA

Filip Radlinski, Microsoft, UK

Jaime Teevan, Microsoft Research, USA

Tutorial Goals 1. To help participants develop a broader perspective of

research goals and approaches in IR. Descriptive, predictive and explanatory

2. To improve participants’ understandings of research choices and constraints. Every research project requires the researcher to make a series of

choices about a range of factors and usually there are constraints that influence these choices.

By using some of our own research papers, we aim to expose you to the experiential aspects of the research process by giving you a behind the scenes view of how we make/made choices in our own research.

Research Goals & Approaches

Describe• Report a set of observations and provide benchmarks (e.g.,

average queries per user, problems a user experiences when engaging in search)

• Such studies might also present categorizations of the observations

Predict• Seek to establish predictable relationships• Take as input some set of features (click through rate, dwell

time) and use these to predict other variables (query abandonment, satisfaction)

Explain(why?)

• Propose a theoretical model that explains how select constructs interact and interrelate

• Devise procedures to measure those constructs (that is, translate the constructs into variables that can be controlled and measured)

• Devise protocol (usually experimental) to observe phenomenon of interest.

• Seek to demonstrate causality, not just show the variables are related.

Research Goals & Approaches

Describe Predict Explain

AfternoonField Observation

✔

Log Analysis ✔ ✔

Morning

Laboratory Experiment

✔ ✔ ✔

Field Experiment ✔ ✔ ✔

Example: Search Difficulties

Describe• A diary study might be used to gain insight about when and

how users’ experience and address search difficulties.• Log data might also be analyzed to identify how often these

events occur.

Predict• A model might be constructed using the signals available in

a log to predict when users will abandon search result pages without clicking. This model might then be evaluated with other log data.

Explain

• Results from these studies might then be used to create an explanatory/theoretical model of search difficulty, which can be used to generate testable hypotheses. The model can include constructs and variables beyond those which are available in the log data.

• An experiment might be designed to test the explanatory power of the theory indirectly by examining the predictive power of the hypotheses.

Overview Observational log analysis

What we can learn

Collecting log data

Cleaning log data (Filip)

Analyzing log data

Field observations (Diane)

Dumais, Jeffries, Russell, Tang & Teevan. “Understanding User Behavior through

Log Data and Analysis.”

What We Can Learn

Observational Approaches to Information Retrieval

David Foster Wallace

Mark Twain

Cowards die many times before their deaths.

Annotated by Nelson Mandela

I have discovered a truly marvelous proof ...which this margin is too narrow to contain.Pierre de Fermat

(1637)

Students prefer used textbooks that are annotated.

[Marshall 1998]

http://upload.wikimedia.org/wikipedia/commons/2/24/Diophantus-II-8.jpg

Digital Marginalia Do we lose marginalia with digital documents? Internet exposes information experiences

Meta-data, annotations, relationships Large-scale information usage data

Change in focus With marginalia, interest is in the individual Now we can look at experiences in the aggregate

Practical Uses for Behavioral Data Behavioral data to improve Web search

Offline log analysis Example: Re-finding common, so add history support

Online log-based experiments Example: Interleave different rankings to find best algorithm

Log-based functionality Example: Boost clicked results in a search result list

Behavioral data on the desktop Goal: Allocate editorial resources to create Help docs How to do so without knowing what people search for?

Value of Observational Log Analysis Focus of observational log analysis

Description: What do people currently do? Prediction: What will people do in similar situations?

Study real behavior in natural settings Understand how people search Identify real problems to study Improve ranking algorithms Influence system design Create realistic simulations and evaluations Build a picture of human interest

Societal Uses of Behavioral Data Understand people’s information needs Understand what people talk about Impact public policy? (E.g., DonorsChoose.org)

Baeza-Yates, Dupret, Velasco. A study of mobile search queries in Japan. WWW 2007

Personal Use of Behavioral Data Individuals now have a lot

of behavioral data Introspection of personal

data popular My Year in Status Status Statistics

Expect to see more As compared to others For a purpose

Defining Behavioral Log Data Behavioral log data are:

Traces of natural behavior, seen through a sensor Examples: Links clicked, queries issued, tweets posted

Real-world, large-scale, real-time

Behavioral log data are not: Non-behavioral sources of large-scale data Collected data (e.g., poll data, surveys, census data)

Not recalled behavior or subjective impression

Real-World, Large-Scale, Real-Time Private behavior is exposed

Example: Porn queries, medical queries

Rare behavior is common Example: Observe 500 million queries a day

Interested in behavior that occurs 0.002% of the time Still observe the behavior 10 thousand times a day!

New behavior appears immediately Example: Google Flu Trends

Drawbacks Not controlled

Can run controlled log studies Discussed in morning tutorial (Filip)

Adversarial Cleaning log data later today (Filip)

Lots of missing information Not annotated, no demographics, we don’t know why Observing richer information after break (Diane)

Privacy concerns Collect and store data thoughtfully Next section addresses privacy

Query Time User

sigir 2014 10:41 am 1/15/14 142039

goldcoast sofitel 10:44 am 1/15/14 142039

learning to rank 10:56 am 1/15/14 142039

sigir 2014 11:21 am 1/15/14 659327

ool transportation 11:59 am 1/15/14 318222

restaurants brisbane 12:01 pm 1/15/14 318222surf lessons 12:17 pm 1/15/14 318222

james allen 12:18 pm 1/15/14 142039

daytrips from brisbane 1:30 pm 1/15/14 554320

sigir 2014 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451

information retrieval 4:56 pm 1/15/14 142039

sigir 2014 5:02 pm 1/15/14 312055

xxx clubs on gold coast 10:14 pm 1/15/13 142039

sex videos 1:49 am 1/16/13 142039

Query Time User

sigir 2014 10:41 am 1/15/14 142039


teen sex 10:56 am 1/15/14 142039

sigir 2014 11:21 am 1/15/14 659327



james allen 12:18 pm 1/15/14 142039


sex with animals 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451

Information retrieval 4:56 pm 1/15/14 142039

sigir 2014 5:02 pm 1/15/14 312055

xxx clubs on gold coast 10:14 pm 1/15/14 142039

sex videos 1:49 am 1/16/14 142039

cheap digital camera 12:17 pm 1/15/14 554320



社会科学11:59 am 11/3/23

12:01 pm 11/3/23

Porn

Language

Spam

System

errors

Query Time User

sigir 2014 10:41 am 1/15/14 142039



sigir 2014 11:21 am 1/15/14 659327



james allen 12:18 pm 1/15/14 142039


sigir 2014 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451


sigir 2014 5:02 pm 1/15/14 312055

kangaroos 10:14 pm 1/15/14 142039

machine learning 1:49 am 1/16/14 142039

Query Time User

sigir 2014 10:41 am 1/15/14 142039



sigir 2014 11:21 am 1/15/14 659327



james allen 12:18 pm 1/15/14 142039


sigir 2014 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451


sigir 2014 5:02 pm 1/15/14 312055

kangaroos 10:14 pm 1/15/14 142039


Query typology

Query Time User

sigir 2014 10:41 am 1/15/14 142039



sigir 2014 11:21 am 1/15/14 659327



james allen 12:18 pm 1/15/14 142039


sigir 2014 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451


sigir 2014 5:02 pm 1/15/14 312055

kangaroos 10:14 pm 1/15/14 142039


Query typology

Query behavior

Query Time User

sigir 2014 10:41 am 1/15/14 142039



sigir 2014 11:21 am 1/15/14 659327



james allen 12:18 pm 1/15/14 142039


sigir 2014 1:30 pm 1/15/14 659327

sigir program 2:32 pm 1/15/14 435451

sigir2014.org 2:42 pm 1/15/14 435451


sigir 2014 5:02 pm 1/15/14 312055

kangaroos 10:14 pm 1/15/14 142039


Query typology

Query behavior

Long term trends

Uses of Analysis• Ranking

– E.g., precision• System design

– E.g., caching• User interface

– E.g., history• Test set

development• Complementary

research

Surprises About Query Log Data From early log analysis

Examples: Jansen et al. 2000, Broder 1998 Scale: Term common if it appeared 100 times!

Queries are not 7 or 8 words long Advanced operators not used or “misused” Nobody used relevance feedback Lots of people search for sex Navigation behavior common Prior experience was with library search

Surprises About Microblog Search?

Ordered by time

Ordered by relevance

8 new tweets


Ordered by time

Ordered by relevance

8 new tweets


• Time important• People important• Specialized syntax• Queries common• Repeated a lot• Change very little

• Often navigational• Time and people

less important• No syntax use• Queries longer• Queries develop


What we can learn Understand and predict user behavior

Collecting log data

Cleaning log data

Analyzing log data

Field observations

Collecting Log Data


How to Get Logs for Analysis Use existing logged data

Explore sources in your community (e.g., proxy logs) Work with a company (e.g., FTE, intern, visiting researcher)

Generate your own logs Focuses on questions of unique interest to you Examples: UFindIt, Wikispeedia

Construct community resources Shared software and tools

Client side logger (e.g., VIBE logger) Shared data sets Shared platform

Lemur Community Query Log Project

Web Service Logs

Government contractor

Recruiting

Academic field

Example sources Search engine Commercial site

Types of information Queries, clicks, edits Results, ads, products

Example analysis Click entropy Teevan, Dumais & Liebling. To

Personalize or Not to Personalize: Modeling Queries with Variation in User Intent. SIGIR 2008

Controlled Web Service Logs Example sources

Mechanical Turk Games with a purpose

Types of information Logged behavior Active feedback

Example analysis Search success Ageev, Guo, Lagun & Agichtein.

Find It If You Can: A Game for Modeling … Web Search Success Using Interaction Data. SIGIR 2011

Public Web Service Content Example sources

Social network sites Wiki change logs

Types of information Public content Dependent on service

Example analysis Twitter topic models Ramage, Dumais & Liebling.

Characterizing microblogging using latent topic models. ICWSM 2010 j

http://twahpic.cloudapp.net

Web Browser Logs Example sources

Proxy Logging tool

Types of information URL visits, paths followed Content shown, settings

Example analysis DiffIE Teevan, Dumais and Liebling. A

Longitudinal Study of How Highlighting Web Content Change Affects .. Interactions. CHI 2010

Web Browser Logs Example sources

Proxy Logging tool

Types of information URL visits, paths followed Content shown, settings

Example analysis Revisitation Adar, Teevan and Dumais. Large

Scale Analysis of Web Revisitation Patterns. CHI 2008

Rich Client-Side Logs Example sources

Client application Operating system

Types of information Web client interactions Other interactions – rich!

Example analysis Stuff I’ve Seen Dumais et al. Stuff I've Seen: A

system for personal information retrieval and re-use. SIGIR 2003

dumais

beijing

sigir 2014

vancouver

A Simple Example Logging search Queries and Clicked Results

Web Service

Web Service

Web Service

“SERP”

chi 2014

A Simple Example

Logging Queries Basic data: <query, userID, time>

Which time? timeClient.send, timeServer.receive, timeServer.send, timeClient.receive

Additional contextual data: Where did the query come from? What results were returned? What algorithm or presentation was used? Other metadata about the state of the system

A Simple Example

Logging Clicked Results (on the SERP) How can a Web service know which SERP links are clicked?

Proxy re-direct Script (e.g., JavaScript)

Dom and cross-browser challenges, but can instrument more than link clicks No download required; but adds complexity and latency, and may influence user

interaction What happened after the result was clicked?

What happens beyond the SERP is difficult to capture Browser actions (back, open in new tab, etc.) are difficult to capture To better interpret user behavior, need richer client instrumentation

http://www.chi2014.org vs. http://redir.service.com/?q=chi2014&url=http://www.chi2014.org/&pos=3&log=DiFVYj1tRQZtv6e1FF7kltj02Z30eatB2jr8tJUFR

<img border="0" id="imgC" src=“image.gif" width="198" height="202" onmouseover="changeImage()" onmouseout="backImage()"><script lang="text/javascript"> function changeImage(){ document.imgC.src="thank_you..gif “; } function backImage(){ document.imgC.src=“image.gif"; }</script>

A (Not-So-) Simple Example Logging: Queries, Clicked Results, and Beyond

What to Log Log as much as possible

Time keyed events, e.g.: <time, userID, action, value, context> Ideal log allows user experience to be fully reconstructed

But … make reasonable choices Richly instrumented client experiments can provide guidance Consider the amount of data, storage required

Challenges with scale Storage requirements

1k bytes/record x 10 records/query x 100 mil queries/day = 1000 Gb/day Network bandwidth

Client to server; Data center to data center

What to Do with the Data Keep as much raw data as possible

And allowable Must consider Terms of Service, IRB

Post-process data to put into a usable form Integrate across servers to organize the data

By time By userID

Normalize time, URLs, etc. Rich data cleaning

Practical Issues: Time Time

Client time is closer to the user, but can be wrong or reset Server time includes network latencies, but controllable In both cases, need to synchronize time across multiple machines

Data integration Ensure that joins of data are all using the same basis

(e.g., UTC vs. local time)

Accurate timing data is critical for understanding the sequence of user activities, daily temporal patterns, etc.

Practical Issues: Users Http cookies, IP address, temporary ID

Provides broad coverage and easy to use, but … Multiple people use same machine Same person uses multiple machines (and browsers)

How many cookies did you use today? Lots of churn in these IDs

Jupiter Res (39% delete cookies monthly); Comscore (2.5x inflation) Login or download client code (e.g., browser plug-in)

Better correspondence to people, but … Requires sign-in or download Results in a smaller and biased sample of people or data (who

remember to login, decided to download, etc.) Either way, loss of data

Using the Data Responsibly What data is collected and how it can be used?

User agreements (terms of service) Emerging industry standards and best practices

Trade-offs More data:

More intrusive and potential privacy concerns, but also more useful for understanding interaction and improving systems

Less data: Less intrusive, but less useful

Risk, benefit, and trust

August 4, 2006: Logs released to academic community 3 months, 650 thousand users, 20 million queries Logs contain anonymized User IDs

August 7, 2006: AOL pulled the files, but already mirrored August 9, 2006: New York Times identified Thelma Arnold

“A Face Is Exposed for AOL Searcher No. 4417749” Queries for businesses, services in Lilburn, GA (pop. 11k) Queries for Jarrett Arnold (and others of the Arnold clan) NYT contacted all 14 people in Lilburn with Arnold surname When contacted, Thelma Arnold acknowledged her queries

August 21, 2006: 2 AOL employees fired, CTO resigned September, 2006: Class action lawsuit filed against AOL

AnonID Query QueryTime ItemRank ClickURL---------- --------- --------------- ------------- ------------1234567 uw cse 2006-04-04 18:18:18 1 http://www.cs.washington.edu/1234567 uw admissions process 2006-04-04 18:18:18 3 http://admit.washington.edu/admission1234567 computer science hci 2006-04-24 09:19:321234567 computer science hci 2006-04-24 09:20:04 2 http://www.hcii.cmu.edu1234567 seattle restaurants 2006-04-24 09:25:50 2 http://seattletimes.nwsource.com/rests1234567 perlman montreal 2006-04-24 10:15:14 4 http://oldwww.acm.org/perlman/guide.html1234567 uw admissions notification 2006-05-20 13:13:13…

Example: AOL Search Dataset

Example: AOL Search Dataset Other well known AOL users

User 711391 i love alaska http://www.minimovies.org/documentaires/view/ilovealaska

User 17556639 how to kill your wife User 927

Anonymous IDs do not make logs anonymous Contain directly identifiable information

Names, phone numbers, credit cards, social security numbers Contain indirectly identifiable information

Example: Thelma’s queries Birthdate, gender, zip code identifies 87% of Americans

Example: Netflix Challenge October 2, 2006: Netflix announces contest

Predict people’s ratings for a $1 million dollar prize 100 million ratings, 480k users, 17k movies Very careful with anonymity post-AOL

May 18, 2008: Data de-anonymized Paper published by Narayanan & Shmatikov Uses background knowledge from IMDB Robust to perturbations in data

December 17, 2009: Doe v. Netflix March 12, 2010: Netflix cancels second competition

Ratings1: [Movie 1 of 17770]12, 3, 2006-04-18 [CustomerID, Rating, Date]1234, 5 , 2003-07-08 [CustomerID, Rating, Date]2468, 1, 2005-11-12 [CustomerID, Rating, Date]…

Movie Titles…10120, 1982, “Bladerunner”17690, 2007, “The Queen”…

All customer identifying information has been removed; all that remains are ratings and dates. This follows our privacy policy. . . Even if, for example, you knew all your own ratings and their dates you probably couldn’t identify them reliably in the data because only a small sample was included (less than one tenth of our complete dataset) and that data was subject to perturbation.

Using the Data Responsibly Control access to the data

Internally: Access control; data retention policy Externally: Risky (e.g., AOL, Netflix, Enron, Facebook public)

Protect user privacy Directly identifiable information

Social security, credit card, driver’s license numbers Indirectly identifiable information

Names, locations, phone numbers … you’re so vain (e.g., AOL) Putting together multiple sources indirectly (e.g., Netflix, hospital records)

Linking public and private data k-anonymity; Differential privacy; etc.

Transparency and user control Publicly available privacy policy Give users control to delete, opt-out, etc.



Collecting log data Not as simple as it seems

Cleaning log data – Filip!

Analyzing log data

Field observations

[Filip on data cleaning]

52

Observational Approaches to Information RetrievalSIGIR 2014 Tutorial: Choices and Constraints (Part II)

Diane Kelly, Filip Radlinski, Jaime Teevan




Cleaning log data Significant portion of log analysis about cleaning

Analyzing log data

Field observations

Analyzing Log Data


Develop Metrics to Capture Behavior

[Joachims 2002]

Sessions 2.20 queries long

[Silverstein et al. 1999]

[Lau and Horvitz, 1999]

Navigational, Informational, Transactional

[Broder 2002]

2.35 terms[Jansen et al. 1998]

Queries appear 3.97 times[Silverstein et al. 1999]

Summary measures Query frequency Query length

Analysis of query intent Query types and topics

Temporal features Session length Common re-formulations

Click behavior Relevant results for query Queries that lead to clicks

Develop Metrics to Capture Behavior

Lee, Teevan, de la Chica. Characterizing multi-click search behavior. SIGIR 2014

Partitioning the Data Language Location Time User activity Individual Entry point Device System variant

Baeza-Yates, Dupret, Velasco. A study of mobile search queries in Japan. WWW 2007

Partition by Time

Periodicities Spikes Real-time data

New behavior Immediate feedback

Individual Within session Across sessions

Beitzel, et al. Hourly analysis of a .. topically categorized web query log. SIGIR 2004

Partition by User

Temporary ID (e.g., cookie, IP address) High coverage but high churn Does not necessarily map directly to users

User account Only a subset of users

Teevan, Adar, Jones, Potts. Information re-retrieval: Repeat queries … SIGIR 2007

Partition by System Variant Also known as controlled experiments Some people see one variant, others another Example: What color for search result links?

Bing tested 40 colors Identified #0044CC Value: $80 million

Considerations When Partitioning Choose comparison groups carefully

From the same time period With comparable users, tasks, etc.

Log a lot because it can be hard to recreate state Which partition did a particular behavior fall into?

Confirm partitions with metrics that should be the same

White, Dumais, Teevan. Characterizing the influence of domain expertise... WSDM 2009

Interpreting Significant Metrics Often, everything is significant

Adar, Teevan, Dumais. Large scale analysis of web revisitation patterns. CHI 2008

Interpreting Significant Metrics Everything is significant, but not always meaningful

“All differences significant except when noted.” Choose the metrics you care about first Look for converging evidence

Look at the data

Beware: Typically very high variance Large variance by user, task, noise Calculate empirically

Confidence Intervals Confidence interval (C.I.):

Interval around the treatment mean that contains the true value of the mean x% (typically 95%) of the time

Gives useful information about the size of the effect and its practical significance

C.I.s that do not contain the control mean are statistically significant (statistically different from the control)

This is an independent test for each metric Thus you will get 1 in 20 results (for 95% C.I.s) that are

spurious Challenge: You don't know which ones are spurious

Confidence Intervals

Lee, Teevan, de la Chica. Characterizing multi-click search behavior. SIGIR 2014

Radlinski, Kurup, Joachims. How does clickthrough data reflect retrieval quality? CIKM 2008.

When Significance Is Wrong Sometimes there is spurious significance

Confidence interval only tells you there is a 95% chance that this difference is real; not 100%

If only a few things significant, chance a likely explanation Sometimes you will miss significance

Because the true difference is tiny/zero or because you don’t have enough power

If you did your sizing right, you have enough power to see all the differences of practical significance

Sometimes reason for change is unexpected Look at many metrics to get a big picture

Chilton, Teevan. Addressing Info. Needs Directly in the Search Result Page. WWW 2011

Be Thoughtful When Combining Metrics 1995 and 1996 performance != Combined performance

Simpsons Paradox Changes in mix (denominators) make combined metrics

(ratios) inconsistent with yearly metrics

Batting Average

1995 1996 Combined

Hits At Bat Hits At Bat Hits At Bat

Derek Jeter 12 48 183 582 195 630

.250 .314 .310David Justice 104 411 45 140 149 551

.253 .321 .270

Detailed Analysis Big Picture Not all effects will point the same direction

Take a closer look at the items going in the “wrong” direction Can you interpret them?

E.g., people are doing fewer next-pages because they are finding their answer on the first page

Could they be artifactual? What if they are real?

What should be the impact on your conclusions? on your decision?

Significance and impact are not the same thing Looking at % change vs. absolute change helps Effect size depends on what you want to do with the data

Beware of Tyranny of the Data Can provide insight into behavior

Example: What is search for, how needs are expressed Can be used to test hypotheses

Example: Compare ranking variants or link color Can only reveal what can be observed Cannot tell you what you cannot observe

Example: Nobody uses Twitter to re-find

People’s intent People’s success People’s experience People’s attention People’s beliefs

Behavior can mean many things 81% of search sequences ambiguous

[Viermetz et al. 2006]

<Back to results>

<Back to results>7:16 – Try new engine

What Logs Cannot Tell Us

<Open in new tab>

<Open in new tab>7:16 – Read Result 17:20 – Read Result 37:27 –Save links locally

7:12 – Query

7:14 – Click Result 1

7:15 – Click Result 3

HCI

Example: Click Entropy Question: How ambiguous

is a query? Approach: Look at

variation in clicks Measure: Click entropy

Low if no variation human computer …

High if lots of variation hci

Companies

Wikipedia disambiguation HCI

Teevan, Dumais, Liebling. To personalize or not to personalize... SIGIR 2008

Which Has Less Variation in Clicks? www.usajobs.gov v. federal government jobs

find phone number v. msn live search

singapore pools v. singaporepools.com

tiffany v. tiffany’s

nytimes v. connecticut newspapers

campbells soup recipes v. vegetable soup recipe

soccer rules v. hockey equipment

?

?

?

Results change

Result quality varies

Tasks impacts # of clicks

Clicks/user = 1.1 Clicks/user = 2.1

Click position = 2.6 Click position = 1.6

Result entropy = 5.7 Result entropy = 10.7

Supplementing Log Data Enhance log data

Collect associated information Example: For browser logs, crawl visited webpages

Instrumented panels Converging methods

Usability studies Eye tracking Surveys Field studies Diary studies

Large-scale log analysis of re-finding

Do people know they are re-finding? Do they mean to re-find the result they do? Why are they returning to the result?

Small-scale critical incident user study Browser plug-in that logs queries and clicks Pop up survey on repeat clicks and 1/8 new clicks

Insight into intent + Rich, real-world picture Re-finding often targeted towards a particular URL Not targeted when query changes or in same session

Example: Re-Finding Intent

Tyler, Teevan. Large scale query log analysis of re-finding. WSDM 2010

Example: Curious Browser Browser plug-in to examine relationship between implicit and explicit behavior

Capture many implicit actions (e.g., click, click position, dwell time, scroll) Probe for explicit user judgments of relevance of a page to the query

Deployed to ~4k people in US and Japan Learned models to predict explicit judgments from implicit indicators

45% accuracy w/ just click; 75% accuracy w/ click + dwell + session Used to identify important features; then apply model in open loop setting

Fox, et al. Evaluating implicit measures to improve the search experience. TOIS 2005


What we can learn Partition logs to observe behavior


Cleaning log data Clean and sanity check

Analyzing log data Big picture more important than individual metrics

Field observations – Diane!

[Diane on field observations]

78

Diane Kelly, Filip Radlinski, Jaime Teevan

Documents

Transcript of Diane Kelly, Filip Radlinski, Jaime Teevan