Post on 14-Feb-2019
Introduction to Information RetrievalIntroduction to Information Retrieval
Introduction to
Information Retrieval
TDT4215 Web‐intelligenceg
Based on slides by:
Hinrich Schütze and Christina LiomaHinrich Schütze and Christina Lioma
Chapter 16: Flat Clustering
1
Introduction to Information RetrievalIntroduction to Information Retrieval
Overview
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
2
Introduction to Information RetrievalIntroduction to Information Retrieval
Outline
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
3
Introduction to Information RetrievalIntroduction to Information Retrieval
Clustering: Definition
(Document) clustering is the process of grouping a set of documents into clusters of similar documents.
Documents within a cluster should be similar.
Documents from different clusters should be dissimilar.
Clustering is the most common form of unsupervisedlearning.
Unsupervised = there are no labeled or annotated data.
44
Introduction to Information RetrievalIntroduction to Information Retrieval
Data set with clear cluster structure
Propose algorithm for finding the gcluster structure in this example
55
Introduction to Information RetrievalIntroduction to Information Retrieval
Classification vs. Clustering
Classification: supervised learning
Clustering: unsupervised learningClustering: unsupervised learning
Classification: Classes are human‐defined and part of the input to the learning algorithm.p g g
Clustering: Clusters are inferred from the data without human input.p
However, there are many ways of influencing the outcome of clustering: number of clusters, similarity measure, representation of documents, . . .
66
Introduction to Information RetrievalIntroduction to Information Retrieval
Outline
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
7
Introduction to Information RetrievalIntroduction to Information Retrieval
The cluster hypothesis
Cluster hypothesis. Documents in the same cluster behavesimilarly with respect to relevance to information needs. Ally papplications of clustering in IR are based (directly or indirectly) onthe cluster hypothesis. Van Rijsbergen’s original wording: “closely associated documents tend to be relevant to the same requests”.
88
Introduction to Information RetrievalIntroduction to Information Retrieval
Applications of clustering in IR
Application What isclustered?
Benefit
Search result clustering search more effectiveSearch result clustering searchresults
more effectiveinformationpresentation to user
Scatter‐Gather (subsetsof) collection
alternative userinterface: “searchwithout typing”
Collection clustering collection effective informationt ti fpresentation for
exploratory browsing
Cluster based retrieval collection higher efficiency:
99
Cluster‐based retrieval collection higher efficiency:faster search
Introduction to Information RetrievalIntroduction to Information Retrieval
Search result clustering for better navigation (1)
1010
Introduction to Information RetrievalIntroduction to Information Retrieval
Search result clustering for better navigation (2)
1111
Introduction to Information RetrievalIntroduction to Information Retrieval
Scatter‐Gather
1212
Introduction to Information RetrievalIntroduction to Information Retrieval
Global navigation: Yahoo
1313
Introduction to Information RetrievalIntroduction to Information Retrieval
Global navigation: MESH (upper level)
1414
Introduction to Information RetrievalIntroduction to Information Retrieval
Navigational hierarchies: Manual vs. automatic creation
Note: Yahoo/MESH are not examples of clustering. But they are well known examples for using a global y p g ghierarchy for navigation.
Some examples for global navigation/exploration based on clustering: Cartia ThemescapesThemescapes Google News
1515
Introduction to Information RetrievalIntroduction to Information Retrieval
Global navigation combined with visualization (1)
1616
Introduction to Information RetrievalIntroduction to Information Retrieval
Global navigation combined with visualization (2)
1717
Introduction to Information RetrievalIntroduction to Information Retrieval
Clustering for improving recall
To improve search recall:
Cluster docs in collection a prioriCluster docs in collection a priori
When a query matches a doc d, also return other docs in the cluster containing d
Hope: if we do this: the query “car” will also return docs containing “automobile”
Because the clustering algorithm groups together docs containing “car” with those containing “automobile”.
Both types of documents contain words like “parts”, “dealer”, “mercedes”, “road trip”.
1818
Introduction to Information RetrievalIntroduction to Information Retrieval
Data set with clear cluster structure
Propose algorithm for finding the gcluster structure in this example
1919
Introduction to Information RetrievalIntroduction to Information Retrieval
Desiderata for clustering
General goal: put related docs in the same cluster, put unrelated docs in different clusters.
How do we formalize this?
The number of clusters should be appropriate for the data set l iwe are clustering.
Initially, we will assume the number of clusters K is given.
i i h d f d i i Later: Semiautomatic methods for determining K
Secondary goals in clustering
Avoid very small and very large clusters
Define clusters that are easy to explain to the user
20
Many others . . .
20
Introduction to Information RetrievalIntroduction to Information Retrieval
Flat vs. Hierarchical clustering
Flat algorithms
Usually start with a random (partial) partitioning of docs into groups
Refine iteratively
l h Main algorithm: K‐means
Hierarchical algorithms
Create a hierarchy
Bottom‐up, agglomerative
Top‐down, divisive
2121
Introduction to Information RetrievalIntroduction to Information Retrieval
Hard vs. Soft clustering
Hard clustering: Each document belongs to exactly one Hard clustering: Each document belongs to exactly one cluster. More common and easier to do
Soft clustering: A document can belong to more than one cluster.
k f l l k b bl Makes more sense for applications like creating browsablehierarchies You may want to put sneakers in two clusters:ou ay a t to put s ea e s t o c uste s
sports apparel shoes
You can only do that with a soft clustering approach You can only do that with a soft clustering approach. We will do flat, hard clustering only in this class. See IIR 16.5, IIR 17, IIR 18 for soft clustering and hierarchical
22
gclustering
22
Introduction to Information RetrievalIntroduction to Information Retrieval
Flat algorithms
Flat algorithms compute a partition of N documents into a set of K clusters.
Given: a set of documents and the number K
Find: a partition into K clusters that optimizes the chosen p ppartitioning criterion
Global optimization: exhaustively enumerate partitions, p y ppick optimal one
Not tractable
Effective heuristic method: K‐means algorithm
2323
Introduction to Information RetrievalIntroduction to Information Retrieval
Outline
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
24
Introduction to Information RetrievalIntroduction to Information Retrieval
K‐means
Perhaps the best known clustering algorithm
Simple, works well in many cases
Use as default / baseline for clustering documents
2525
Introduction to Information RetrievalIntroduction to Information Retrieval
Document representations in clustering
Vector space model
As in vector space classification, we measure relatedness between vectors by Euclidean distance . . .
. . .which is almost equivalent to cosine similarity.
Almost: centroids are not length‐normalized.
2626
Introduction to Information RetrievalIntroduction to Information Retrieval
K‐means
h l i i d fi d b id Each cluster in K‐means is defined by a centroid. Objective/partitioning criterion: minimize the average squared difference from the centroidsquared difference from the centroid
Recall definition of centroid:
h d lwhere we use ω to denote a cluster. We try to find the minimum average squared difference by iterating two steps:iterating two steps: reassignment: assign each vector to its closest centroid recomputation: recompute each centroid as the average of
27
the vectors that were assigned to it in reassignment
27
Introduction to Information RetrievalIntroduction to Information Retrieval
K‐means algorithm
28
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Set of to be clustered
29
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Random selection of initial centroids
Exercise: (i) Guess what theExercise: (i) Guess what the
optimal clustering into two clusters is in this case; (ii) compute the
centroids of the clusters
30
centroids of the clusters
30
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest center
31
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
32
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
33
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
34
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
35
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
36
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
37
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
38
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
39
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
40
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
41
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
42
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
43
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
44
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
45
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
46
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
47
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster centroids
48
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assign points to closest centroid
49
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Assignment
50
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Example: Recompute cluster caentroids
51
Introduction to Information RetrievalIntroduction to Information Retrieval
Worked Ex.: Centroids and assignments after convergence
52
Introduction to Information RetrievalIntroduction to Information Retrieval
K‐means is guaranteed to converge: Proof
RSS = sum of all squared distances between document vector and closest centroid
RSS decreases during each reassignment step.
because each vector is moved to a closer centroid
RSS decreases during each recomputation step.
see next slide
There is only a finite number of clusterings.
Thus: We must reach a fixed point.
Assumption: Ties are broken consistently.
5353
Introduction to Information RetrievalIntroduction to Information Retrieval
Recomputation decreases average distance
the residual sum of squares (the “goodness”– the residual sum of squares (the goodnessmeasure)
The last line is the componentwise definition of the centroid! Wepminimize RSSk when the old centroid is replaced with the newcentroid. RSS, the sum of the RSSk , must then also decrease
54
during recomputation.
54
Introduction to Information RetrievalIntroduction to Information Retrieval
K‐means is guaranteed to converge
But we don’t know how long convergence will take!
If we don’t care about a few docs switching back and forth, th i ll f t ( 10 20 it ti )then convergence is usually fast (< 10‐20 iterations).
However, complete convergence can take many more iterationsiterations.
5555
Introduction to Information RetrievalIntroduction to Information Retrieval
Optimality of K‐means
Convergence does not mean that we converge to the optimal clustering!
Thi i th t k f K This is the great weakness of K‐means.
If we start with a bad set of seeds, the resulting clustering can be horriblecan be horrible.
5656
Introduction to Information RetrievalIntroduction to Information Retrieval
Convergence Exercise: Suboptimal clustering
Wh t i th ti l l t i f K 2? What is the optimal clustering for K = 2?
Do we converge on this clustering for arbitrary seeds di , dj?
5757
Introduction to Information RetrievalIntroduction to Information Retrieval
Initialization of K‐means
Random seed selection is j st one of man a s K means Random seed selection is just one of many ways K‐means can be initialized.
Random seed selection is not very robust: It’s easy to get a Random seed selection is not very robust: It s easy to get a suboptimal clustering.
Better ways of computing initial centroids: Better ways of computing initial centroids:
Select seeds not randomly, but using some heuristic (e.g., filter out outliers or find a set of seeds that has “good coverage” of g gthe document space)
Use hierarchical clustering to find good seeds
Select i (e.g., i = 10) different random sets of seeds, do a K‐means clustering for each, select the clustering with lowest RSS
5858
Introduction to Information RetrievalIntroduction to Information Retrieval
Time complexity of K‐means
Computing one distance of two vectors is O(M).
Reassignment step: O(KNM) (we need to compute KNdocument‐centroid distances)
Recomputation step: O(NM) (we need to add each of the document’s < M values to one of the centroids)
Assume number of iterations bounded by I
Overall complexity: O(IKNM) – linear in all important dimensions
However: This is not a real worst‐case analysis.
In pathological cases, complexity can be worse than linear.
5959
Introduction to Information RetrievalIntroduction to Information Retrieval
Outline
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
60
Introduction to Information RetrievalIntroduction to Information Retrieval
What is a good clustering?
Internal criteria
Example of an internal criterion: RSS in K‐meansExample of an internal criterion: RSS in K means
But an internal criterion often does not evaluate the actual utility of a clustering in the application.y g pp
Alternative: External criteria
Evaluate with respect to a human‐defined classificationp
6161
Introduction to Information RetrievalIntroduction to Information Retrieval
External criteria for clustering quality
Based on a gold standard data set, e.g., the Reuters collection we also used for the evaluation of classification
Goal: Clustering should reproduce the classes in the gold standard
(But we only want to reproduce how documents are divided into groups, not the class labels.)
First measure for how well we were able to reproduce the classes: purity
6262
Introduction to Information RetrievalIntroduction to Information Retrieval
External criterion: Purity
Ω= ω1, ω2, . . . , ωK is the set of clusters andΩ ω1, ω2, . . . , ωK is the set of clusters and C = c1, c2, . . . , cJ is the set of classes.
For each cluster ωk : find class cj with most members nkj in ωkk j kj k
Sum all nkj and divide by total number of points
6363
Introduction to Information RetrievalIntroduction to Information Retrieval
Example for computing purity
To compute purity: 5 = maxj |ω1 ∩ cj | (class x, cluster 1);4 | | ( l l 2) d 3 | |4 = maxj |ω2 ∩ cj | (class o, cluster 2); and 3 = maxj |ω3 ∩ cj | (class ⋄, cluster 3). Purity is (1/17) × (5 + 4 + 3) ≈ 0.71.
6464
Introduction to Information RetrievalIntroduction to Information Retrieval
Rand index
Definition:
Based on 2x2 contingency table of all pairs of documents: Based on 2x2 contingency table of all pairs of documents:
TP+FN+FP+TN is the total number of pairs. There are pairs for N documents. Example: = 136 in o/⋄/x example Each pair is either positive or negative (the clustering puts Each pair is either positive or negative (the clustering puts the two documents in the same or in different clusters) . . .
. . . and either “true” (correct) or “false” (incorrect): the
65
( ) ( )clustering decision is correct or incorrect.
65
Introduction to Information RetrievalIntroduction to Information Retrieval
Rand Index: ExampleAs an example we compute RI for the o/⋄/x example We firstAs an example, we compute RI for the o/⋄/x example. We firstcompute TP + FP. The three clusters contain 6, 6, and 5 points,respectively, so the total number of “positives” or pairs ofdocuments that are in the same cluster is:
Of these the x pairs in cluster 1 the o pairs in cluster 2 the ⋄Of these, the x pairs in cluster 1, the o pairs in cluster 2, the ⋄pairs in cluster 3, and the x pair in cluster 3 are true positives:
Th FP 40 20 20 FN d TN t d i il l
66
Thus, FP = 40 − 20 = 20. FN and TN are computed similarly.
66
Introduction to Information RetrievalIntroduction to Information Retrieval
Rand measure for the o/ /x example
(20 + 72)/(20 + 20 + 24 + 72) ≈ 0.68.
6767
Introduction to Information RetrievalIntroduction to Information Retrieval
Two other external evaluation measures
Two other measures
Normalized mutual information (NMI)
How much information does the clustering contain about the gclassification?
Singleton clusters (number of clusters = number of docs) have maximumMI
Therefore: normalize by entropy of clusters and classes
F measure
Like Rand, but “precision” and “recall” can be weighted
6868
Introduction to Information RetrievalIntroduction to Information Retrieval
Evaluation results for the o/ /x example
All four measures range from 0 (really bad clustering) to 1 (perfectclustering).
6969
Introduction to Information RetrievalIntroduction to Information Retrieval
Outline
❶ Cl t i I t d ti❶ Clustering: Introduction
❸ Clustering in IR❸ Clustering in IR
❹ K‐means❹ K means
❺ Evaluation❺
❻ How many clusters?
70
Introduction to Information RetrievalIntroduction to Information Retrieval
How many clusters?
Number of clusters K is given in many applications.
E.g., there may be an external constraint on K. Example: In the case of Scatter‐Gather, it was hard to show more than 10–20 clusters on a monitor in the 90s.
Wh if h i l i ? I h “ i h ” What if there is no external constraint? Is there a “right” number of clusters?
O t d fi ti i ti it i One way to go: define an optimization criterion
Given docs, find K for which the optimum is reached.
Wh t ti i ti it i ? What optimiation criterion can we use?
We can’t use RSS or average squared distance from centroid as criterion: always chooses K = N clusters
71
criterion: always chooses K = N clusters.
71
Introduction to Information RetrievalIntroduction to Information Retrieval
Simple objective function for K
Basic idea: Basic idea:
Start with 1 cluster (K = 1)
Keep adding clusters (= keep increasing K) Keep adding clusters (= keep increasing K)
Add a penalty for each new cluster (= cost λ)
Trade off cluster penalties against average squared distance Trade off cluster penalties against average squared distance from centroid
Choose the value of K with the best tradeoff Choose the value of K with the best tradeoff
Still need to determine good value for λ . . .
7272
Introduction to Information RetrievalIntroduction to Information Retrieval
Finding the “knee” in the curve
Pick the number of
clusters where curve
“flattens”. Here: 4 or 9.
7373
Introduction to Information RetrievalIntroduction to Information Retrieval
Take‐away today
What is clustering? What is clustering?
Applications of clustering in information retrieval
K l ith K‐means algorithm
Evaluation of clustering
l ? How many clusters?
7474
Introduction to Information RetrievalIntroduction to Information Retrieval
Resources
Chapter 16 of IIRChapter 16 of IIR
Resources at http://ifnlp.org/ir
K‐means exampleK means example
Keith van Rijsbergen on the cluster hypothesis (he was one of
the originators)the originators)
Bing/Carrot2/Clusty: search result clustering
7575