Web search results clustering

• Web search results clustering is a version of document clustering, but…

• Billions of pages• Constantly changing• Data mainly unstructured and heterogeneous• Additional information to consider (i.e. links,

click-through data, etc.)

Some requirements

• Fast– Immediate response to query

• Flexible– Web content changes constantly– overlapping

• User-oriented– Main goal is to aid the user in finding information– Meaningful labels– Visualization: GUI

Architecture of Web clustering engine

Main issues

• Online or offline clustering? • What to use as input

– Entire documents– Snippets– Structure information (links)– Other data (i.e. click-through)– Use stop word lists, stemming, etc.

• How to define similarity?– Content (i.e. vector-space model)– Link analysis– Usage statistics

• How to group similar documents?• How to label the groups?

Clustering algorithms

• Flat or hierarchical? • Overlapping?• Hard or soft? (One object to one cluster or

multiple clusters)• Incremental?• Predefined cluster number?• Requiring explicit similarity measure? Distance

measure?

Clustering algorithms• Distance-based

– Hierarchical• Agglomerative Hierarchical Clustering (AHC)

– Flat• K-means (can be fuzzy)• Single-pass (incremental)

• Other– Suffix Tree Clustering (Grouper)– Query directed clustering– Self-organizing (Kohonen) maps (neural networks)– Latent Semantic Indexing (LSI) (reducing the dimensionality of the

vector-space)

Clustering algorithms

• Data Centric algorithms– E.g. Scatter/Gather– Vector space model, k-means or HAC– Cluster labels is not good enough

• Description-Aware algorithms– STC

• Description-Centric Algorithms– E.g Vivisimo, Lingo, SRC– Good labels.

Research Prototype systems

Commercial systems

Scatter/Gather from Lin and Pantel (2002) and Hearst and Pederson (1996)

KeySRC

KartOO

Others

• More at http://www.folden.info/searchengineclustertechnology.shtml

Scatter/Gather

• (Cutting et. al. 1992)• Designed for browsing• Based on two novel clustering algorithms– Buckshot – fast for online clustering– Fractionation – accurate for offline initial

clustering of the entire set

Grouper

• (Zamir and Etzioni 1997, 1999)• Online• Operates on query result snippets• Clusters together documents with large

common subphrases• Suffix Tree Clustering (STC): linear, incremental,

overlapping, can be extended to hierarchical• STC induces labeling

Iwona Białynicka-Birula - Clustering Web Search Results

STC algorithm• Step 1: Cleaning

– Stemming– Sentence boundary identification– Punctuation elimination

• Step 2: Suffix tree construction– Produces base clusters (internal nodes)– Base clusters are scored based on size and phrase score (which

depends on length and word „quality”) • Step 3: Merging base clusters

– Highly overlapping clusters are merged

Carrot2

• (Stefanowski and Weiss 2003)• http://www.cs.put.poznan.pl/dweiss/carrot/• Component framework• Allows substituting components for

– Input (i.e. snippets from other search engines)– Filter

• Stemming• Distance measure• Clustering

– Output

Vivísimo

• Commercial• http://www.vivisimo.com/• Online• Hierarchical• Conceptual

Iwona Białynicka-Birula - Clustering Web Search Results

Other• Mapuccino (IBM)

– (Maarek et. al. 2000)– http://www.alphaworks.ibm.com/tech/mapuccino– Relatively efficient AHC (O(n2))– Similarity based on vector-space model

• (Su et. al. 2001)– Only usage statistics used as input– Recursive Density Based Clustering

• SHOC– (Zhang and Dong 2004)– Grouper-like– Key phrase discovery

Problems

• The performance is far from being perfect– Incompleteness of clusters,

• Hard to tell why some cluster are generated, some are missing

– Different cluster granularity, • Some clusters are very specific, some are very broad

– Inconsistency: the contents and the label, • lack of intra and inter-cluster consistency.

– Label expressiveness– Lack of evaluation, data sets

Future research trends

To extract more powerful features: hyperlinks, external info, temporal attributes

To generate more expressive or effective descriptions of clusters

To improve the accuracy of the hierarchy structureTo consider user characteristics, web usage dataIntegration with ontologyBetter visualisation of the clustersTo apply to Mobile searchXML documents clustering