Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x...
Transcript of Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x...
![Page 1: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/1.jpg)
CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University
Clustering Algorithms
![Page 2: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/2.jpg)
Given a set of data points, group them into a clusters so that: points within each cluster are similar to each other points from different clusters are dissimilar
Usually, points are in a high-‐dimensional space, and similarity is defined using a distance measure Euclidean, Cosine, Jaccard, edit distance, …
![Page 3: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/3.jpg)
Weight
Height Chihuahuas Dachshunds
Beagles
![Page 4: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/4.jpg)
A catalog of 2 billion “sky objects” represents objects by their radiaHon in 7 dimensions (frequency bands).
Problem: cluster into similar objects, e.g., galaxies, nearby stars, quasars, etc.
Sloan Sky Survey is a newer, beQer version.
![Page 5: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/5.jpg)
Cluster customers based on their purchase histories
Cluster products based on the sets of customers who purchased them
Cluster documents based on similar words or shingles
Cluster DNA sequences based on edit distance
![Page 6: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/6.jpg)
Hierarchical (AgglomeraHve): IniHally, each point in cluster by itself. Repeatedly combine the two “nearest” clusters into one.
Point Assignment: Maintain a set of clusters. Place points into their “nearest” cluster.
![Page 7: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/7.jpg)
Key OperaHon: repeatedly combine two nearest clusters
Three important quesHons: How do you represent a cluster of more than one point?
How do you determine the “nearness” of clusters? When to stop combining clusters?
![Page 8: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/8.jpg)
Each cluster has a well-‐defined centroid i.e., average across all the points in the cluster
Represent each cluster by its centroid Distance between clusters = distance between centroids
![Page 9: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/9.jpg)
(5,3) o (1,2) o
o (2,1) o (4,1)
o (0,0) o (5,0)
x (1.5,1.5)
x (4.5,0.5) x (1,1)
x (4.7,1.3)
![Page 10: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/10.jpg)
The only “locaHons” we can talk about are the points themselves. I.e., there is no “average” of two points.
Approach 1: clustroid = point “closest” to other points. Treat clustroid as if it were centroid, when compuHng intercluster distances.
![Page 11: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/11.jpg)
Possible meanings: 1. Smallest maximum distance to the other points.
2. Smallest average distance to other points. 3. Smallest sum of squares of distances to other
points.
4. Etc., etc.
![Page 12: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/12.jpg)
1 2
3 4
5
6
intercluster distance
clustroid
clustroid
![Page 13: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/13.jpg)
Approach 2: intercluster distance = minimum of the distances between any two points, one from each cluster.
Approach 3: Pick a noHon of “cohesion” of clusters, e.g., maximum distance from the clustroid. Merge clusters whose union is most cohesive.
![Page 14: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/14.jpg)
Approach 1: Use the diameter of the merged cluster = maximum distance between points in the cluster.
Approach 2: Use the average distance between points in the cluster.
![Page 15: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/15.jpg)
Approach 3: Use a density-‐based approach: take the diameter or average distance, e.g., and divide by the number of points in the cluster. Perhaps raise the number of points to a power first, e.g., square-‐root.
![Page 16: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/16.jpg)
Stop when we have k clusters Stop when the cohesion of the cluster resulHng from the best merger falls below a threshold
Stop when there is a sudden jump in the cohesion value
![Page 17: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/17.jpg)
Naïve implementaHon: At each step, compute pairwise distances between each pair of clusters
O(N3) Careful implementaHon using a priority queue can reduce Hme to O(N2 log N)
Too expensive for really big data sets that don’t fit in memory
![Page 18: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/18.jpg)
Assumes Euclidean space. Start by picking k, the number of clusters. IniHalize clusters by picking one point per cluster. Example: pick one point at random, then k -‐1 other points, each as far away as possible from the previous points.
![Page 19: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/19.jpg)
1. For each point, place it in the cluster whose current centroid it is nearest, and update the centroid of the cluster.
2. Aeer all points are assigned, fix the centroids of the k clusters.
3. OpHonal: reassign all points to their closest centroid. SomeHmes moves points between clusters.
![Page 20: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/20.jpg)
1
2
3
4
5
6
7 8 x
x
Clusters after first round
Reassigned points
![Page 21: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/21.jpg)
Try different k, looking at the change in the average distance to centroid, as k increases.
Average falls rapidly unHl right k, then changes liQle.
k
Average distance to centroid
Best value of k
![Page 22: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/22.jpg)
x x x x x x
x x x x x x x
x x
x xx x
x x x x x
x x x x
x
x x x x x x x x x
x
x
x
Too few; many long distances to centroid.
![Page 23: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/23.jpg)
x x x x x x
x x x x x x x
x x
x xx x
x x x x x
x x x x
x
x x x x x x x x x
x
x
x
Just right; distances rather short.
![Page 24: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/24.jpg)
x x x x x x
x x x x x x x
x x
x xx x
x x x x x
x x x x
x
x x x x x x x x x
x
x
x
Too many; little improvement in average distance.
![Page 25: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/25.jpg)
BFR (Bradley-‐Fayyad-‐Reina) is a variant of k -‐means designed to handle very large (disk-‐resident) data sets.
It assumes that clusters are normally distributed around a centroid in a Euclidean space. Standard deviaHons in different dimensions may vary.
![Page 26: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/26.jpg)
Points are read one main-‐memory-‐full at a Hme.
Most points from previous memory loads are summarized by simple staHsHcs.
To begin, from the iniHal load we select the iniHal k centroids by some sensible approach.
![Page 27: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/27.jpg)
PossibiliHes include: 1. Take a small random sample and cluster
opHmally. 2. Take a sample; pick a random point, and then k –
1 more points, each as far from the previously selected points as possible.
![Page 28: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/28.jpg)
1. The discard set: points close enough to a centroid to be summarized.
2. The compression set: groups of points that are close together but not close to any centroid. They are summarized, but not assigned to a cluster.
3. The retained set: isolated points.
![Page 29: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/29.jpg)
A cluster. Its points are in the DS.
The centroid
Compressed sets. Their points are in the CS.
Points in the RS
![Page 30: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/30.jpg)
For each cluster, the discard set is summarized by: 1. The number of points, N. 2. The vector SUM: i th component = sum of the
coordinates of the points in the i th dimension.
3. The vector SUMSQ: i th component = sum of squares of coordinates in i th dimension.
![Page 31: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/31.jpg)
2d + 1 values represent any number of points. d = number of dimensions.
Centroid (mean) in i th dimension = SUMi /N. SUMi = i th component of SUM.
Variance in dimension i can be computed by: (SUMSQi /N ) – (SUMi /N )2
QuesHon: Why use this representaHon rather than directly store centroid and standard deviaHon?
![Page 32: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/32.jpg)
1. Find those points that are “sufficiently close” to a cluster centroid; add those points to that cluster and the DS.
2. Use any main-‐memory clustering algorithm to cluster the remaining points and the old RS. Clusters go to the CS; outlying points to the
RS.
![Page 33: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/33.jpg)
3. Adjust staHsHcs of the clusters to account for the new points. Add N’s, SUM’s, SUMSQ’s.
4. Consider merging compressed sets in the CS. 5. If this is the last round, merge all compressed
sets in the CS and all RS points into their nearest cluster.
![Page 34: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/34.jpg)
How do we decide if a point is “close enough” to a cluster that we will add the point to that cluster?
How do we decide whether two compressed sets deserve to be combined into one?
![Page 35: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/35.jpg)
We need a way to decide whether to put a new point into a cluster.
BFR suggest two ways: 1. The Mahalanobis distance is less than a
threshold. 2. Low likelihood of the currently nearest centroid
changing.
![Page 36: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/36.jpg)
Normalized Euclidean distance from centroid.
For point (x1,…,xk) and centroid (c1,…,ck): 1. Normalize in each dimension: yi = (xi -‐ci)/σi
2. Take sum of the squares of the yi ’s. 3. Take the square root.
![Page 37: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/37.jpg)
If clusters are normally distributed in d dimensions, then aeer transformaHon, one standard deviaHon = √d. I.e., 70% of the points of the cluster will have a Mahalanobis distance < √d.
Accept a point for a cluster if its M.D. is < some threshold, e.g. 4 standard deviaHons.
![Page 38: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/38.jpg)
σ
2σ
![Page 39: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/39.jpg)
Compute the variance of the combined subcluster. N, SUM, and SUMSQ allow us to make that calculaHon quickly.
Combine if the variance is below some threshold.
Many alternaHves: treat dimensions differently, consider density.
![Page 40: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/40.jpg)
Problem with BFR/k -‐means: Assumes clusters are normally distributed in each dimension.
And axes are fixed – ellipses at an angle are not OK.
CURE: Assumes a Euclidean distance. Allows clusters to assume any shape.
![Page 41: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/41.jpg)
e e
e e
e e
e
e e e
e
h
h
h
h
h
h
h h
h
h
h
h h
salary
age
![Page 42: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/42.jpg)
1. Pick a random sample of points that fit in main memory.
2. Cluster these points hierarchically – group nearest points/clusters.
3. For each cluster, pick a sample of points, as dispersed as possible.
4. From the sample, pick representaHves by moving them (say) 20% toward the centroid of the cluster.
![Page 43: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/43.jpg)
e e
e e
e e
e
e e e
e
h
h
h
h
h
h
h h
h
h
h
h h
salary
age
![Page 44: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/44.jpg)
e e
e e
e e
e
e e e
e
h
h
h
h
h
h
h h
h
h
h
h h
salary
age
Pick (say) 4 remote points for each cluster.
![Page 45: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/45.jpg)
e e
e e
e e
e
e e e
e
h
h
h
h
h
h
h h
h
h
h
h h
salary
age
Move points (say) 20% toward the centroid.
![Page 46: Clustering Algorithms - Stanford University · Clustering Algorithms ... xx x x x x x x x x x x x x x x x x x x x x x x x Too few; many long distances to centroid. x x x x x x x x](https://reader033.fdocuments.in/reader033/viewer/2022052217/5acb09677f8b9a51678e58f2/html5/thumbnails/46.jpg)
Now, visit each point p in the data set. Place it in the “closest cluster.” Normal definiHon of “closest”: that cluster with the closest (to p ) among all the sample points of all the clusters.