Unsupervised learning

Clustering using the k-means algorithm

Avi Libster

clustering

• Used when we have a very large data set with very high dimensionality and lots of complex structure.

• Basic assumption : attributes of the data are independent.

Cluster Analysis

• Given a collection of data points in space ,which might be high dimensional one, the goal is to find structure in the data: organize that data into sensible groups, so that each group will contain points that are near in some sense.

• We want points in the same cluster to have high intersimilarity and low outersimilarity compared to points from different clusters.

Taxonomy of Clustering

What is K-means

• An unsupervised learning algorithm

• Used for partitioning datasets.

• Simple to use

• Based on the minimization of the square error.

Basic K-means algorithm

Begin Initialize n , k ,m1 .. mk

do classify n samples according to nearest mi

recompute miuntil no change in mi

return m1 … mk

End

The goal is to minimize E = ∑i=1,K∑vЄCid(μiv)

And now for something completely different…

*Pictures

* Adopted from http://www.cs.ucr.edu/~eamonn/teaching/cs170materials/MachineLearning3.ppt

0

1

2

3

4

5

0 1 2 3 4 5

K-means Clustering: Step 1K-means Clustering: Step 1N – points , 3 centers randomly chosen

k1

k2

k3

0

1

2

3

4

5

0 1 2 3 4 5

K-means Clustering: Step 2K-means Clustering: Step 2Notice that the 3 centers divide the space into 3 parts

k1

k2

k3

0

1

2

3

4

5

0 1 2 3 4 5

K-means Clustering: Step 3K-means Clustering: Step 3New centers are calculated according to the instances of each K.

k1

k2

k3

0

1

2

3

4

5

0 1 2 3 4 5

K-means Clustering: Step 4K-means Clustering: Step 4Classifying each point to the new calculated K.

k1

k2

k3

0

1

2

3

4

5

0 1 2 3 4 5

expression in condition 1

ex

pre

ss

ion

in

co

nd

itio

n 2

K-means Clustering: Step 5K-means Clustering: Step 5After classifying the points to previous K vector , calculating new one

k1

k2 k3

Classic K-Means Strengths

First and furthermost

VERY easy to implement and understand. Nice results for a simple algorithm.

Classical K-Means - Strengths

K-means can be viewed as a stochastic hill climbing procedure. we are looking for local optimum and not global optimum (as opposed to genetic or deterministic annealing algorithms which look for global optimum).

Why Hill climbing ?

Actually Hill climbing can be a misleading term in this context. The hill climbing Is not done over the dataset points , but over the means values. When the k-means algorithm is running we actually change the values of the means (k of those). The changes of the means is somewhat dependent on each other.

Hill climbing continued…

• The algorithm is said to converge when we don’t change the means values.

• This happens when dmi/dt = 0. in the phase plane created from the means values we have reached the top of the hill (stable point or saddle point).

K-means Strengths continued…

K-means complexity is easily derived from the algorithm :

O(ndcT)n – number of samplesd – number of features (usually the dimension of the samples)k – number of centers checked T – number of iterations

When the datasets are not to large and of low dimensions the average time of running is not high.

Using K-means strengths

The following are real life situations in which K-means is used as the key clustering algorithm.

During the presentation of the samples, I would like to emphasize some important points considering the implementation of k-means.

Sample :Understanding Gene regulation using Expression array cluster analysis

Scheme of gene expression research

Gene Expression Data

Distance/Similarity Matrix

Gene Clusters

Regulatory Elements / Gene Functions

Pairwise Measures

Clustering

Motif Searching/Network Construction

Integrated Analysis(NMR/SNP/Clinic/….)

Normalized Expression Data

Gene Expression Clustering

Protein/protein complex

DNA regulatory elements

Semantics of clusters:From co-expressed to co-regulated

The meaning of gene clusters

• genes that are consistently either up- or downregulated in given set of conditions. Down or upregulation may shade light on the causes of biological processes

• patterns of gene expression and grouping genes into expression classes might provide much greater insight into their biological function and relevance.

Why should clusters emerge ?

Genes contained in a particular pathway, or that respond to some environmental change , should be co-regulated and consequently, should show similar patterns of expression.

from that fact one can see that the main goal is to identify genes which show similar patterns of expressions. Examples:

1. If gene a expression is rising, gene b expression is rising too (might be because gene a encodes a protein which regulates the expression of b).

2. Gene a is always expressed with gene b (it might happen because of the co-regulation by same protein).

Classic Microarray Experiment

Control

Treated

mRNA

RT andlabel with fluor dyes

cDNA

Spot (DNA probe):• known cDNA or• Oligo

Mix and hybridize target to microarray

Measured Microarray raw data

Measure amounts of green and red fluorescence. For each well 4 results are possible :

1. No color – the gene wasn’t expressed2. Red color – the gene was expressed only by the control

group3. Green color – the gene was expressed only by the

treated group4. Yellow color – the gene was expressed by both of the

control groups.

Important conclusion : don’t let color blind people perform the test .

Example of microarray image

Data extraction Process

• Adjust fluorescent intensities using standards (as necessary)

• Calculate ratio of red to green fluorescence

• Convert to log2 and round to integer

• Values may be : saturated green=-2 to black = 0 to saturated red = 2

Input data for clustering

• Genes in rows , conditions in columns. Condition can be seen as : exposure to specific environment, time , ect. Each column is one microarray test.

YORF NAME GWEIGHT Cell-cycle Alpha-Factor 1Cell-cycle Alpha-Factor 2Cell-cycle Alpha-Factor 3EWEIGHT 1 1 1YHR051W YHR051W COX6 oxidative phosphorylation cytochrome-c oxidase subunit VI S00010931 0.03 0.3 0.37YKL181W YKL181W PRS1 purine, pyrimidine, tryptophanphosphoribosylpyrophosphate synthetase S00016641 0.33 -0.2 -0.12YHR124W YHR124W NDT80 meiosis transcription factor S00011661 0.36 0.08 0.06YHL020C YHL020C OPI1 phospholipid metabolism negative regulator of phospholipid biosynthesS00010121 -0.01 -0.03 0.21YGR072W YGR072W UPF3 mRNA decay, nonsense-mediated unknown S00033041 0.2 -0.43 -0.22YGR145W YGR145W unknown unknown; similar to MESA gene of Plasmodium fS00033771 0.11 -1.15 -1.03YGR218W YGR218W CRM1 nuclear protein targeting nuclear export factor S00034501 0.24 -0.23 0.12YGL041C YGL041C unknown unknown S00030091 0.06 0.23 0.2YOR202W YOR202W HIS3 histidine biosynthesis imidazoleglycerol-phosphate dehydratase S00057281 0.1 0.48 0.86YCR005C YCR005C CIT2 glyoxylate cycle peroxisomal citrate synthase S00005981 0.34 1.46 1.23YER187W YER187W unknown unknown; similar to killer toxin Khs1p S00009891 0.71 0.03 0.11YBR026C YBR026C MRF1' mitochondrial respiration ARS-binding protein S00002301 -0.22 0.14 0.14YMR244W YMR244W unknown unknown; similar to Nca3p S00048581 0.16 -0.18 -0.38YAR047C YAR047C unknown unknown S00000831 -0.43 -0.56 -0.14YMR317W YMR317W unknown unknown S00049361 -0.43 -0.03 0.21

Why Data extraction process is relevant and important ?

• Creating an easy to work with scale (-2 < x < 2)

• more important k-means is sensitive to the measure units we chose, more correctly to linear transformations. Let’s demonstrate that :

What happens if we change the measurement unit to ft ?

Age Height

Person (yr) (cm)

A 35 190

B 40 190

C 35 160

D 40 160

160

170

180

190

20 30 40 5035

A B

C D

Height (cm)

Age (year)

when the measurement units were changed, a very different clustering

structure emerged.

Age Height

Person (yr) (ft)

A 35 6.2

B 40 6.2

C 35 5.2

D 40 5.2

4

5

6

7

35 37 3938

A B

C D

Height (ft)

Age (year)36 40

How to overcome measure unit problems ?

• It’s clear that if k-means algorithm is to be used the data should be normalized and standardized.

let’s just have a brief look on the dataset structure …

We are dealing with a Multivariate Dataset

composed of p variables (p microarrays tests done) for n independent observations (genes). We represent it using n x p matrix M consisting of vectors X1 through Xn each of length p.

Dataset structure

• Normalization:

– Calculates the mean value of variables

– a measure how well the of the ith variable is spread over the data

• Mean absolute deviation

(column) everyfor jxxxn

m njjjj 211

||||||1

21 jnjjjjjj mxmxmxn

s

222

21 )()()(

1

1jnjjjjjj mxmxmx

nstd

Note: if this is done we are becoming sensitive to outliers.More of that in pitfalls of k-means section

• z-scores: standardize measurements

j

jijij

s

mxz

variables

objects

p

zz

zz

n

npn

p

......

............

............

......

1

111

calculating covariance matrix will come handy later. The covariance matrix is PxP matrix.

cov( j,k) M(i, j) I( j ) M(i,k) I(k)

i1

n

n 1

before running k-means on the data it’s also a good idea to do mean centering. Mean centering reduces the effect of the variables with the largest values (column) , which obscure other important differences.

but When applying the previous steps to the data you should be cautious. When data is being standardized it may cause some damages to the clusters structure , because of the reduced effects ,from variables with a big contribution, being divided by large sj.

Back to microarrays world

a1

a2 b2

b1

Distance

Sample 2

Sample 1

Gene aGene b

Sample 1 sample 2

a1 a2b1 b2

Distance

The way Distance is measured is of the highest importance to k-means algorithm. Using the distance function we aim to classify points to different centers.

Distance should be a function with the following properties : 1. d(A,A) = 02. d(A,B) = d(B,A)3. d(A,B) > 0

Distances example 1

Below are distances which are good to use when we are looking for similarities

• Euclidean Distance

• Manhattan Distance

• Minkowski Distance

• Weighted Euclidean Distance

2222

211 )()()(),( jpipjiji xxxxxxjid

||||||),( 2211 jpipjiji xxxxxxjid

qqjpip

qji

qji xxxxxxjid

1

2211 )|||||(|),(

22222

2111 )()()(),( jpippjiji xxwxxwxxwjid

Distances example 2The following examples are a measure of

dissimilarity :• Mahalanobis distance

• Tanimoto distance

)()(),( 12jiji xxCxxjir

Covariance matrix C (calculated before)

jijT

jiT

i

ji

xxxxxx

xxjid

T

T

),(

ji xx Tdenotes the number of common attributes between i and j

Pearson correlation coefficient

the most common distance measurement used in microarrays.

Effects of choosing distance function

Euclidean distances from the mean, of points a and b are equal but point B is clearly “more different” from the population than point A (it lies on the border of the ellipse).

BA

The ellipse was created using a distance function using the covariance matrix. it shows the 50% contour of a hypothetical population.

Result from microarrays data analysis

Results clustering for patterns

Problems with K-means type algorithms

• Clusters are approximately spherical

• Local optimum may be incorrect and influenced by the choice of the first K means values

• High dimensionality is a problem

• The value of K is an input parameter

• Sensitive to outliers

Clusters are approximately spherical

• What happens if cluster is not spherical ?

Also when k-means assumes the data to be spherical , it becomes sensitive to coordinate changes (i.e. weighting changes).

• What happens with non conventional structures ?

Min

stance

Average

distance

Max

distance

Local optimum problem

• If we begin with b,e as the k values we end with {a,b,c} and {e,d,f} as clusters. If we begin with e,f as k values we end with {c,f} and {a,b,d,e} as clusters.

Outliers problem

• What happens in the next situation ?

High dimensionality

Poses a computational problem. The higher the dimension , the more resources are required. Also a linear connection exists between the dimension and running time.

Dimensions reduction

Using PCA (principle component analysis) we can guess the dimensions which are most important to us (most changes of data occur in them). After finding the proper dimensions we reduce the data set dimensions accordingly.

The value of k is an input parameter

Insufficient number of centers Too much centers

Variation K-means

• Instead of fixed number of centers , the number of centers change as the algorithm runs.

• G-means is an example of such an algorithm.

G-means algorithm

Ideas behind G-means

• Each cluster adheres to unimodal distribution , such as Gaussian.

• Doesn’t presume prior or domain specific knowledge

• Increment of the number of centers occurs only if we have a cluster without gaussian distribution.

• Statistical test is used to determine whether clusters have gaussian distributions or not.

Testing clusters for Gaussian fit

we need a test to detect whether the data assigned to a center is sampled from a gaussian. The alternative hypotheses are:

• H0 – the data around the center is sampled from a gaussian• H1 – the data around the center is not sampled from a

gaussian

Accepting the null H means we believe that one center is sufficient and we shouldn’t split the clusters.

Regecting the null H, then we want to split the cluster

Anderson – Darling Statistic

• One dimensional test.• Normality test based on the empirical cumulative distribution

function (ECDF)

Equations :

If the mean and variance are estimated from the data

Hypothesis test

Hypothesis test for subset data X in d dimensions that belongs to center c• Choose the significance level for the test.• Initialize two centers , called children of “C”• Run k-means on these two centers in X.• Let v be the vector that connects between c1 and c2.v is the direction

that k-means believes to be important for clustering project X onto v let it be X’. X’is a one dimensional representation of the data projected onto v. Transform X’ to mean 0 and variance 1.

• Zi = F(x’i), calculate according to that the anderson – darling statistic. Decide critical level according to significance level . If in the range of non critical accept null H hypothesis and discard the children. Otherwise keep the children

Two clusters composed from 1000 points each. Alpha = 0.0001. critical value for anderson-darling test is 1.8692 for this confidence level. Starting with on center.

After one iteration of G-means we have two centers. Calculation of the anderson-darling statistics is 38.103 .It’s much larger than the critical value so we reject the Null H and accept this spilit.

On the next iteration we split each new center and repeat the Statistical test. The values for two splits is 0.386 and 0.496Both are below the critical value. Null H is accepted for bothTests and the splits are discarded. Thus G-means final answer Is 2.

Another strength of the g-means algorithm it’s is ability to handle non-spherical data, even when the number of points for each cluster is small .

One should notice that experiments shows a tendency to type II error (not splitting when should) when the number of points in each cluster is small.