Download - Unsupervised Learning: Part II Self-Organizing Maps

8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

1/36

Klinkhachorn:CpE520

Unsupervised Learning: Part IISelf-Organizing Maps


2/36

Klinkhachorn:CpE520

Classification of ANN

Paradigms

Unsupervised


3/36

Klinkhachorn:CpE520

Self -Organizing Maps

Presented by Kohonen in 1988

Two layer Network

Input layer Competitive layer

Develops a topological map of

relationships between the input

patterns Orderly progression of inputs produce and

orderly progression of outputs


4/36

Klinkhachorn:CpE520

Teuvo KohonenDr. Eng., Emeritus Professor of the Academy of Finland;Academician

His research areas are the theory of self-organization,associative memories, neural networks, and patternrecognition, in which he has published over 300research papers and four monography books. His fifthbook is on digital computers. His more recent work isexpounded in the third, extended edition (2001) of hisbook Self-Organizing Maps.

Since the 1960s, Professor Kohonen has introduced several new concepts toneural computing: fundamental theories of distributed associative memory andoptimal associative mappings, the learning subspace method, the self-organizing feature maps (SOMs), the learning vector quantization (LVQ), novelalgorithms for symbol processing like the redundant hash addressing,dynamically expanding context and a special SOM for symbolic data, and a SOMcalled the Adaptive-Subspace SOM (ASSOM) in which invariant-feature filtersemergence. A new SOM architecture WEBSOM has been developed in hislaboratory for exploratory textual data mining. In the largest WEBSOMimplemented so far, about seven million documents have been organized in aone-million neuron network: for smaller WEBSOMs, see the demo athttp://websom.hut.fi/websom/.

http://www.cis.hut.fi/research/som-research/teuvo.html


5/36

Klinkhachorn:CpE520

Self -Organizing Maps


6/36

Klinkhachorn:CpE520

SOM - Training algorithm

Determine winner in competition by either of

two methods (both determine the weight vector

best matching the input vector)

1. Calculate difference between weight and

input vector for each neuron

Vector difference =

Winner = Dc = min (Dj)

jD = ix - jiw( )2

i

2.Normalize inputs (as presented to network) and

weight vectors (as updated) and calculate dot product

Winner = Dc = min (Dj)

DotProduct=jD = ix * jiw( )

i


7/36

Klinkhachorn:CpE520

Vector Normalization


8/36

Klinkhachorn:CpE520

Example - Preserve Direction

( ) 543 22 =+

Divide each element of vector by the magnitude of

the vector

If inputs = 6 and 8

Magnitude =

Normalized inputs = 0.6 and 0.8

If inputs = 3 and 4

Magnitude =

Normalized inputs = 0.6 and 0.8

( ) 108622 =+


9/36

Klinkhachorn:CpE520

Example - Preserve relationship between vectors

( ) 14.14515 22 =-

Add another element which places the vector on

the surface of a sphere and then divide by the

magnitude of the vector

If inputs = 6 and 8 , N = 15

d =

Normalized inputs = 0.4,0.53, .75

If inputs = 3 and 4

d =

Normalized inputs = 0.2, 0.27, 0.94

( ) 18.111015 22 =-


10/36

Klinkhachorn:CpE520

SOM - Training algorithm

Update weights of winner (Dc) and weights of

all others in the neighborhood of winner.

Wji(t+1) = Wji(t) + a(xi- Wji(t))

for all neurons in neighborhood of winner

Choice ofneighborhoods (size and

dimensionality) affect network performanceand function.

Neighborhood and gain should be decreases

as time progresses


11/36

Klinkhachorn:CpE520

SOM - Neighborhoods


12/36

Klinkhachorn:CpE520

SOM Parameter Adjustments

Gain parameter, a, should decrease as time

progresses

a = A*(1-t/T) where, A = initial gain

T = final interation time

Neighborhood adjustment, d, should also decrease

as time progresses

d = D*(1-t/T) where, D = initial neighborhood

T = final interation time


13/36

Klinkhachorn:CpE520

SOM Training Summary

Locate winner of Competitive Layer

Adapt weights of all units inneighborhood to increase similarity

between input and weight vectors

Decrease gain and neighborhood astime progresses


14/36

Klinkhachorn:CpE520

SOM Example 1

The weight vector distribution will

approximate the probability density

distribution of the input space


15/36

Klinkhachorn:CpE520

SOM Weight Maps

Iteration: 100 Iteration: 1200


16/36

Klinkhachorn:CpE520

SOM Example 2

Input Distribution Weight Map after 1700 iterations


17/36

Klinkhachorn:CpE520

SOM 1-D Neighborhood


18/36

Klinkhachorn:CpE520

SOM Example 3

The weight vector distribution will

approximate the probability density

distribution of the input space


19/36

Klinkhachorn:CpE520

SOM Weight Map

Iteration: 100 Iteration: 3400


20/36

Klinkhachorn:CpE520

SOM Characteristics

Dimension Mapping (the dimensions

of the input space are mapped to the

dimensions of the output space)

Variable discretion among input

patterns due to frequency of occurance

Weight distribution tends to

approximate the probability density of

the input vectors


21/36

Klinkhachorn:CpE520

SOM Example 4

Input distribution


22/36

Klinkhachorn:CpE520

SOM Example 4

Initial Weight Map


23/36

Klinkhachorn:CpE520

SOM Example 4


24/36

Klinkhachorn:CpE520

SOM Example 4


25/36

Klinkhachorn:CpE520

SOM Example 4


26/36

Klinkhachorn:CpE520

SOM Example 4


27/36


28/36

Klinkhachorn:CpE520

SOM Example 5

Traveling Salesman Problem


29/36

Klinkhachorn:CpE520

SOM Example 5

Traveling Salesman Problem


30/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map

24 bits RGB


31/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map

256 color Indexed


32/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map


33/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map

Typical color reduction map


34/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map

Statistical clustering method


35/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map


36/36

Klinkhachorn:CpE520

SOM Example 5

Color Reduction Map

Kohonen neural Network