Cluster Analysis - MyCourses...Cluster Analysis (CA) ~ method for organizing data (people, things,...
Transcript of Cluster Analysis - MyCourses...Cluster Analysis (CA) ~ method for organizing data (people, things,...
Cluster Analysis
Pekka Malo 30E00500 – Quantitative Empirical Research Spring 2016
What is cluster analysis?
18.01.16 Cluster Analysis
2
Cluster analysis is known by many names …
Segmentation Q-analysis
Classification
Unsupervised learning Taximetrics
Learning without a teacher
18.01.16 Cluster Analysis
3
Purpose: Find a way to group data in a meaningful manner
Cluster Analysis (CA) ~ method for organizing data (people, things, events, products, companies, etc.) into meaningful
groups or taxonomies based on a set of variables that describe the key features of the observations
Cluster ~ a group of observations, which are similar to each other and different from observations in other clusters
18.01.16 Cluster Analysis
4
http://obelia.jde.aca.mmu.ac.uk/multivar/other.htm
18.01.16 Cluster Analysis
5
Example 1
How many clusters and how do you cluster?
18.01.16 Cluster Analysis
6
Example 2
How many clusters and how do you cluster?
18.01.16 Cluster Analysis
7
Example 3
How many clusters and how do you cluster?
18.01.16 Cluster Analysis
8
Between-Cluster Variation = Maximize
Within-Cluster Variation = Minimize
Objectives in Cluster Analysis
Source: Hair et al. (2010)
18.01.16 Cluster Analysis
9
Within-groups vs. Between-groups • Within-groups property: Each group is homogenous with
respect to certain characteristics, i.e. observations in each group are similar to each other
• Between-groups property: Each group should be different from other groups with respect to the same characteristics, i.e. observations of one group should be different from the observations of other groups
18.01.16 Cluster Analysis
10
How many clusters and how do you cluster?
18.01.16 Cluster Analysis
11
How many clusters and how do you cluster?
18.01.16 Cluster Analysis
12
What can we do with cluster analysis?
• Detect groups which are statistically significant – Taxonomy description: Natural groups in data
– Simplification of data: Groups instead of individuals
• Identify meaning for the clusters – Which relationships can be identified?
• Explain and find ways how they can be used
18.01.16 Cluster Analysis
13
Classification vs. Clustering • Classification
– We know the “groups” for at least some of the observations
– Objective is to find a rule / function which correctly assigns observations into groups
– Supervised learning procedure
• Clustering – We don’t know the groups a priori
– Objective is to group together points “which are similar”
– Identify the underlying “hidden” structure in the data
– Unsupervised learning procedure (i.e. no labeled data for training)
18.01.16
Cluster Analysis
14
Clustering ~ “post hoc” segmentation
• Any discrete variable is a segmentation – E.g., gender, geographical area, etc.
• A priori segmentation – Use existing discrete variables to create segments
• Post hoc segmentation – Collect data on various attributes
– Apply statistical technique to find segments
18.01.16 Cluster Analysis
15
Sample size considerations • Representativeness: The sample used for obtaining the
cluster analysis should be representative of the population and its underlying structure (in particular the potential groups of interest)
• Minimum group sizes based on relevance to research question and confidence needed in characterization of the groups
18.01.16 Cluster Analysis
16
Phases of Clustering
Evaluation of significance
Decision regarding the number of clusters
Technique (Hierarchical / Nonhierarchical)
Similarity Measures
Choice of variables
18.01.16 Cluster Analysis
17
Step 1: Goals and choice of variables
• No theoretical guidelines
• Driven by the problem and practical significance – Do the variables help to characterize the objects?
– Are the variables clearly related to the objectives?
• Warning: – Avoid including variables “just because you can”
– Results are dramatically affected by inclusion of even one or two inappropriate or undifferentiated variables
18.01.16 Cluster Analysis
18
Choice of variablesExample: Gamers and their 4 screens
Source: gameindustryblog.com (Newzoo model)
18.01.16 Cluster Analysis
19
Choice of variables (cont.)Example: Gamers and their 4 screens
Source: gameindustryblog.com (Newzoo model)
18.01.16 Cluster Analysis
20
Step 2: Choice of similarity measure
How close or similar are two observations?
Interobject similarity is an empirical measure of correspondence, or resemblance, between objects to
be clustered.
18.01.16 Cluster Analysis
21
Types of similarity measures
• Distance (or dissimilarity) Measures – Euclidean Distance
– Minkowski Metric
– Euclidean Distance for Standardized Data
– Mahalanobis Distance
• Association Coefficient
• Correlation Coefficient
• Subjective Similarity
18.01.16 Cluster Analysis
22
Distance Measures • Minkowski metric:
– n = 2 : Euclidean Distance
– n = 1 : City-block Distance
18.01.16 Cluster Analysis
23
Euclidean Distance: Example
18.01.16 Cluster Analysis
24
Standardization of variables
• Standardization of variables is commonly preferred to avoid problems due to different scales
• Most commonly done using Z-scores
• If groups are to be formed based on respondents’ response styles, then within-case or row-centering standardization can be considered
18.01.16 Cluster Analysis
25
Distance Measures (cont.)
• Euclidean Distance for Standardized Data:
• Mahalanobis Distance:
18.01.16 Cluster Analysis
26
Standardized Euclidean Distance: Example
18.01.16 Cluster Analysis
27
Mahalanobis Distance: Example
18.01.16 Cluster Analysis
28
Association Coefficients
• Consider two objects described by p binary variables. Similarity between objects A and B can be presented as follows:
where
0 10 a b1 c d
Object B
Object A Example
A: 0 0 1 1 0 0 1 1 1 1 0
B: 1 1 0 0 1 0 0 1 1 0 0
18.01.16 Cluster Analysis
29
Correlation Coefficients
• Between Observations i and j
18.01.16 Cluster Analysis
30
Procedures and choice of measure
• Some procedures are sensitive to the similarity measure
• Multiple measures should be employed to compare results and gain confidence of the solution
18.01.16 Cluster Analysis
31
Outliers • Should be removed when
– Observation is aberrant and not representative of population
– Representative of small or insignificant segments which are not interest for research objectives
• Should be retained when – Under-sampling issues
– Poor representations of relevant groups in population
18.01.16 Cluster Analysis
32
Identification of outliers using similarity measures
• Consider observations with large distances from all other observations
• Use graphic profile diagrams to highlight aberrant observations
• Appearance in very small or single-member clusters
18.01.16 Cluster Analysis
33
Multicollinearity of variables • Substantial multicollinearity of input variables is
problematic
• If large collinearity is found, the following strategies could be considered
– Reduce variables to equal numbers in each set of correlated measures
– Use a distance measure that compensates for the correlation (e.g. Mahalanobis distance)
– Include only variables that are not highly correlated
18.01.16 Cluster Analysis
34
Step 3: Hierarchical Clustering
• Centroid method
• Nearest-neighbor or single-linkage method
• Farthest-neighbor or complete-linkage method
• Average linkage method
• Ward method
18.01.16 Cluster Analysis
35
Hierarchical Clustering with SPSS
18.01.16 Cluster Analysis
36
Types of hierarchical methods
• Agglomerative ~ Build-up
• Divisive ~ Break-down
18.01.16 Cluster Analysis
37
Agglomerative vs. Divisive
Source: Hair et al. (2010)
18.01.16 Cluster Analysis
38
How do agglomerative approaches work?
18.01.16 Cluster Analysis
39
Start with all observations as their own cluster
Use selected similarity measure to combine two most similar observations into a new cluster of two observations
Repeat the procedure using the similarity measure to group together the most similar observations or combinations of observations into another new cluster
Continue until all observations are in a single cluster
Example: Single-Linkage Method
• Principle – The distance between two clusters is represented by the
minimum of the distance between all possible pairs of subjects in the two groups
18.01.16 Cluster Analysis
40
Example: Single-Linkage Method Single Linkage
0
5
10
15
20
25
0 5 10 15 20 25 30 35
Points
18.01.16 Cluster Analysis
41
Example: Complete-Linkage Method
• Principle: – The distance between two-clusters is represented by the
maximum of the distance between all possible pairs of subjects in the two groups
18.01.16 Cluster Analysis
42
Example: Complete-Linkage Method
18.01.16 Cluster Analysis
43
Complete Linkage
0
5
10
15
20
25
0 5 10 15 20 25 30 35
Points
Source: Hair et al. (2010)
Source: Hair et al. (2010)
Ward’s Method • “Incremental sum of squares method” that uses the
within-cluster distances and between-cluster distances
• Joins groups A and B that minimize increase in SSE:
SSEA =nAX
i=1
(yi � yA)0(yi � yA)
SSEB =nBX
i=1
(yi � yB)0(yi � yB)
SSEAB =nABX
i=1
(yi � yAB)0(yi � yAB)
IAB = SSEAB � (SSEA + SSEB)
18.01.16 Cluster Analysis
46
Choice of hierarchical approachPros and cons • Single-linkage
– Most versatile, but poorly delineated cluster structures in a dataset may lead to snakelike cluster-chains
• Complete-linkage – No chaining, but impacted by outliers
• Average linkage – Considers average similarity of all individuals in a cluster
– Tends to generate clusters with small within-cluster variation
– Less affected by outliers
18.01.16 Cluster Analysis
47
Choice of hierarchical approach (cont.)Pros and cons
• Ward’s method – Uses total sum of squares within clusters
– Most appropriate when equally sized clusters are expected
– Easily distorted by outliers
• Centroid linkage – Considers difference between cluster centroids
– Less affected by outliers
18.01.16 Cluster Analysis
48
18.01.16 Cluster Analysis
49
Choosing the number of clusters • No single objective procedure
• Evaluation based on following considerations: – Occurrence of single-member of extremely small clusters is not
acceptable and should be eliminated
– Ad-hoc stopping rules in hierarchical methods based on the rate of change in total similarity measure as the number of clusters increases or decreases
– Clusters should be significantly different across the set of variables
– Solutions must have theoretical validity based on external validation
18.01.16 Cluster Analysis
50
Measures of heterogeneity change
• Percentage changes in heterogeneity – E.g. use of agglomeration coefficient in SPSS, which measures
heterogeneity as distance at which clusters are formed
– E.g. within-cluster sum of squares when Ward’s method is considered
• Measures of variance change – Root mean square standard deviation (RMSSTD) ~ square root of
the variance of the new cluster formed by joining two clusters, where the variance is computed across all clustering variables
– Large increase in RMSSTD indicates joining of two dissimilar clusters
18.01.16 Cluster Analysis
51
Visualization of the solution
Dendrogram is convenient when the number of observations is not very high
18.01.16 Cluster Analysis
52
Use agglomeration schedule to decide number of clusters
Seek for demarcation point
18.01.16 Cluster Analysis
54
Step 4: Refine the solution with Non-hierarchical Clustering Procedures
• Sometimes a combination of hierarchical and nonhierarchical methods is considered:
– Use hierarchical method (e.g., Ward’s) to choose number of clusters and profile cluster centers that serve as initial seeds
– Use nonhierarchical method (e.g., k-Means) to cluster all observations using the seed points to provide more accurate cluster membership
18.01.16 Cluster Analysis
55
Hierarchical vs. Non-hierarchical • Choose hierarchical method when
– Wide range (possibly all) cluster solutions are to be examined
– Sample size is moderate (under 300-400), no more than 1000
• Choose nonhierarchical method when – Number of clusters is known
– Initial seed points can be specified by practical, objective or theoretical basis
– Results are less susceptible to outliers, distance measure or inclusion of irrelevant variables
– Works on large datasets
18.01.16 Cluster Analysis
56
Hybrid approaches • Sometimes a combination of hierarchical and
nonhierarchical methods is considered
• Idea: – Use hierarchical method to choose number of clusters and
profile cluster centers that serve as initial seeds
– Use nonhierarchical method to cluster all observations using the seed points to provide more accurate cluster memberships
18.01.16 Cluster Analysis
57
Simple k-Means algorithm Given an initial seed, the algorithm alternates between the following steps:
1. Assignment step: – Add each observation to the cluster, whose mean leads to the
least within-group sum of squares (Squared Euclidean distance)
2. Update step: – Compute new cluster means and use them as centroids for
observations in the updated cluster
18.01.16 Cluster Analysis
58
K-Means in SPSS
Save solution and examine output
18.01.16 Cluster Analysis
60
Non-hierarchical Clustering
• Distance Measures
• Statistical Criteria – tr W “Within-group sums of squares”
(= Euclidean Distance)
– tr W-1T “Hotelling’s trace”
– |W| “Wilks’ lambda”
– Largest Eigenvalue of W-1T
–
18.01.16 Cluster Analysis
61
tr W
18.01.16 Cluster Analysis
62
tr W-1T
18.01.16 Cluster Analysis
63
|W|
18.01.16 Cluster Analysis
64
18.01.16 Cluster Analysis
65
Step 5: Evaluation of cluster solutions
Source: Marketing research (Winter 2010)
18.01.16 Cluster Analysis
66
How informative is your solution?
• Generalizability: Are the segments identifiable in a larger population?
• Substantiality: How sizeable are the segments when compared to each other?
• Accessibility and actionability: How easily can the segments be reached? Can we execute strategies using the solution?
• Stability: Is the solution repeatable (e.g. if new measurements are done)?
Segmentation is information compression. Good segmentation conveys key information about the important
variables or attributes.
18.01.16 Cluster Analysis
67
Statistical vs. Practical criteria • Statistical:
– Do the segment profiles differ in a statistically significant manner?
– What attributes contribute most to the group differences?
– Are the groups internally homogeneous and externally heterogeneous?
• Practical: – Are the segments substantial enough for making profit?
– Is the solution stable?
– Can we reach the segment in a cost-effective manner?
– Is it useful for decision making purposes?
– Do the segments respond consistently to stimulus?
18.01.16 Cluster Analysis
68
Dash of criticism • Conceptual vs. empirical support
• Descriptive, atheoretical, non-inferential?
• Clusters always produced regardless of empirical structure?
• Solution not generalizable due to dependence on variables used for defining similarity measure?
18.01.16 Cluster Analysis
69
Comparison of profiles
Source: Rencher: Methods of Multivariate Analysis
18.01.16 Cluster Analysis
70
Any differences between population means ? µi : a sample mean for population Pi and W n-1 : a common variance-covariance matrix (Estimator)
W = W1 + W2 + … + Wg
Λ = |W | / | T | = | W | / | W + B | = | T - B | / | T |,
where, – T = "total" sum of squares and products (SSP) matrix – W = "within-groups" SSP-matrix – Wk= ”total” SSP matrix in group k – B = "between-groups" SSP-matrix = T – W
18.01.16 Cluster Analysis
71
T: Total SSP-matrix
18.01.16 Cluster Analysis
72
W: Within Groups SSP-matrix
18.01.16 Cluster Analysis
73
One-Way MANOVA
(overall mean) (treatment effect) (error)
18.01.16 Cluster Analysis
74
Assumptions for MANOVA test
• Observations must be independent
• Variance-covariance matrices must be equal for all treatment groups
• The set of dependent variables must follow a multivariate normal distribution
18.01.16 Cluster Analysis
75
Interpretation of solutions
• Examine profiles for distinguishing characteristics – Where and how much do the cluster profiles differ?
• Solutions failing to show substantial differences in mean profiles indicate that other solutions should be examined
• Cluster centroids should be assessed with respect to prior expectations based on theory or practical experience
18.01.16 Cluster Analysis
76
Profiling the cluster solutions • Once clusters are identified, the objective is to describe
the characteristics of each cluster and how they differ on relevant dimensions
• Utilize data not included in the cluster procedure to profile the characteristics of each cluster
– Demographics, psychographics, consumption patterns, etc.
• Often done using Discriminant Analysis to compare average score profiles for the clusters
– Dependent variable (categorical) = cluster membership
– Independent variables = Demographics + Psychographics + …
18.01.16
Cluster Analysis
77
Validation • Cross-validation:
– Create sub-samples of the dataset (random splitting)
– Compare cluster solutions for consistency (number of clusters and profiles) ■ Very stable solution would be produced with less than 10% of
observations assigned differently ■ Stable solution is when 10-20% of observations are assigned to a
different group ■ Somewhat stable solution when 20-25% are assigned to a different
cluster
• Using relevant external variables: – Examine differences on variables not included in the cluster analysis
but for which there is a theoretical and relevant reason to expect variation across the clusters
18.01.16 Cluster Analysis
78
Thank you!
79
R – give it a spin!
18.01.16 Cluster Analysis
80