Coverage and Independence:Measuring Quality in Web Search Results

Panagiotis Takis MetaxasLilia Ivanova

Eni Mustafaraj

Department of Computer Science

Wellesley College,

USA

Precision and Recall in Traditional IR

Precision and Recall in Web IRHigh Precision is easy to achieve but does not convey useful information

Recall is uninteresting and cannot be computed accurately because of the enormous size of the web

85% of Web Searchers never look past top-10!

But what is Quality?

Quality when searching controversial issues?

Quality when searching Political Issues?

But Google is usually so good in finding info… Why does it do that?

Define Search Quality in a web-meaningful way

Comprehensive Coverage = Lack of bias towards some search results

For a controversial issue (at a minimum): cover the pro, con and balanced opinions

For k opinions, and top-N results:

expected # of results / opinion: N/k

Coverage Bias = total distance from N/k

Define Search Quality in a web-meaningful way

Comprehensive Coverage = Lack of bias towards some search results

(bad coverage) 0 ≤ C ≤ 1 (good coverage)

Now we can talk about, e.g., 60% coverage

Define Search Quality in a web-meaningful way

Independent search results = Results that are not dependent due to spamming

u: URL Dependency

r: Redirection Dependency

c: Content Dependency

l: Link Dependency

Example of Dependent Results:Google’s “HGH benefits”

Redirection dependency URL dependency

Table 1: Top-10 results of Google when given the query ”HGH benefits” for August, 2007 and September, 2007. For each entry we have calculated the size of the backGraph as (|V |, |E |) revealed by the Google API and the change between these two dates.

Example of Dependent Results:Yahoo’s “Is ADHD a real disease”

Link dependency Content dependency

Table 4: Top-10 results of the Yahoo search engine when given the query ”Is ADHD a real disease” (August and September, 2007).

Define Search Quality in a web-meaningful way

Independent search results = Results that are not dependent due to spamming

u: URL Dependency

r: Redirection Dependency

c: Content Dependency

l: Link Dependency

(total dependence) 0 ≤ ≤ 1 (total independence)

Evaluating Quality of 3 Search Results

Query with commercial interest: “Human Growth Hormone (HGH) benefits”

Query with medical interest: “Is ADHD a real disease?”

Query with political interest: “Morality of abortions”

Evaluating Quality of 3 Search Results

Coverage of Google Coverage of Yahoo Independence

Our result show low coverage for controversial questions that are not highly pursued and higher coverage for an issue that is highly pursued (“Abortion”).

They also show high independence of results that are not highly pursued andhigher independence for an issue that is highly pursued (“Abortion”).

There is significant overlap between the top-10 returns of both Yahoo and Google results!

Comparing visible neighborhoods

Google

Yahoo

Both

Coverage and Independence:Measuring Quality in Web Search Results

Panagiotis Takis Metaxas

Department of Computer Science

Wellesley College,

USA

Thank you!pmetaxas@wellesley.edu

Example of Dependent Results:Yahoo’s “HGH benefits”

Example of Dependent Results:Google’s “Is ADHD a real disease”

Example of Dependent Results:Google’s “Morality of Abortion”

Example of Dependent Results:Yahoo’s “Morality of Abortion”