Post on 28-Dec-2015
DATA MINING
CLUSTERINGK-Means
Clustering Definition
• Techniques that are used to divide data objects into groups– A form of classification in that it creates a labelling
object with class(cluster) labels. The labels are derived from the data
• Cluster analysis is categorized as unsupervised classification– When you have no idea how to define groups,
clustering method can be useful
2
Types of Clustering
• Hierarchical vs Partitional– Hierarchical nested cluster, organized as tree– Partitional fully non-overlapping
• Exclusive vs Overlapping vs Fuzzy– Exclusive each object is assigned to a single cluster– Overlapping an object can simultaneously belong to more than one
cluster– Fuzzy every object belongs to every cluster with a membership
weigth that is between 0 and 1
• Complete vs Partial– Complete assigns every object to cluster– Partial not all objects are assigned
3
4
Types of Clusters
• Well-separated• Prototype-based• Graph-based• Density-based• Shared-property(Conceptual Cluster)
K-Means
• Partitional clustering• Prototype-based• One level
5
Basic K-Means
• k, the number of clusters that are to be formed, must be decided before beginning
• Step 1– Select k data points to act as the seeds (or initial cluster
centroids)• Step 2– Each record is assigned to the centroid which is nearest,
thus forming a cluster• Step 3– The centroids of the new clusters are then calculated. Go
back to Step 26
Basic K-means -2-
7
Assign each record to the nearest centroid
Calculate new centroid
Determine cluster boundaries
Choosing Initial Centroids
• Random initial centroids– Poor– Can have empty cluster
• Limits of random initialization– Multiple runs with different set of randomly
choosen centroids then select the set of cluster with the minimum SSE
8
Similarity, Association, and Distance
• The method just described assumes that each record can be described as a point in a metric space– This is not easily done for many data sets (e.g., categorical and some
numeric variables)• Pre-processing is often necessary
• Records in a cluster should have a natural association. A measure of similarity is required.– Euclidean distance is often used, but it is not always suitable– Euclidean distance treats changes in each dimension equally, but
changes in one field may be more important than changes in another• and changes of the same “size” in different fields can have very different
significances• e.g. 1 metre difference in height vs. $1 difference in annual income
9
Measures of Similarity
• Euclidean distance between vectors X and Y
• Weighting
10
Redefine Cluster Centroids• Sum of the Squared Error for data in euclidean space. The
centroid(mean) of the ith cluster is defined:
• Other case:
11
Proximity Function Centroid Objective Function
Manhattan (L1) median Minimize sum of L1 distance of an object to its cluster centroid
Square Euclidean(L22) mean Minimize sum of the squared L2 distance of an object to its cluster
centroid
Cosine mean Maximize sum of the cosine similarity of an object to its cluster centroid
Bregman divergence mean Minimize sum of the Bregman divergence of an object to its cluster centroid
iCxi
i xm
c1
Bisecting K-means
• Basic idea:– Split the set of all points into two cluster– Select one of these clusters to split– so on, until K cluster have been produced
• Choose the cluster to split:– Cluster with largest SSE– Cluster with largest size– Both, or other criterion
• Bisecting is less susceptible to initialization problems
12
Strengths and Weaknesses
• Strengths– Simple and can be used for wide variety data
types– Efficient in computation
• Weaknesses– Not suitable for all types of data– Cannot contains outliers, should be remove– Restricted to data for which there is a notion of a
center(centroids)
13
14