Unsupervised Learning: Part II Self-Organizing Maps

download Unsupervised Learning: Part II Self-Organizing Maps

of 36

  • date post

    06-Apr-2018
  • Category

    Documents

  • view

    214
  • download

    0

Embed Size (px)

Transcript of Unsupervised Learning: Part II Self-Organizing Maps

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    1/36

    Klinkhachorn:CpE520

    Unsupervised Learning: Part IISelf-Organizing Maps

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    2/36

    Klinkhachorn:CpE520

    Classification of ANN

    Paradigms

    Unsupervised

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    3/36

    Klinkhachorn:CpE520

    Self -Organizing Maps

    Presented by Kohonen in 1988

    Two layer Network

    Input layer Competitive layer

    Develops a topological map of

    relationships between the input

    patterns Orderly progression of inputs produce and

    orderly progression of outputs

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    4/36

    Klinkhachorn:CpE520

    Teuvo KohonenDr. Eng., Emeritus Professor of the Academy of Finland;Academician

    His research areas are the theory of self-organization,associative memories, neural networks, and patternrecognition, in which he has published over 300research papers and four monography books. His fifthbook is on digital computers. His more recent work isexpounded in the third, extended edition (2001) of hisbook Self-Organizing Maps.

    Since the 1960s, Professor Kohonen has introduced several new concepts toneural computing: fundamental theories of distributed associative memory andoptimal associative mappings, the learning subspace method, the self-organizing feature maps (SOMs), the learning vector quantization (LVQ), novelalgorithms for symbol processing like the redundant hash addressing,dynamically expanding context and a special SOM for symbolic data, and a SOMcalled the Adaptive-Subspace SOM (ASSOM) in which invariant-feature filtersemergence. A new SOM architecture WEBSOM has been developed in hislaboratory for exploratory textual data mining. In the largest WEBSOMimplemented so far, about seven million documents have been organized in aone-million neuron network: for smaller WEBSOMs, see the demo athttp://websom.hut.fi/websom/.

    http://www.cis.hut.fi/research/som-research/teuvo.html

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    5/36

    Klinkhachorn:CpE520

    Self -Organizing Maps

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    6/36

    Klinkhachorn:CpE520

    SOM - Training algorithm

    Determine winner in competition by either of

    two methods (both determine the weight vector

    best matching the input vector)

    1. Calculate difference between weight and

    input vector for each neuron

    Vector difference =

    Winner = Dc = min (Dj)

    jD = ix - jiw( )2

    i

    2.Normalize inputs (as presented to network) and

    weight vectors (as updated) and calculate dot product

    Winner = Dc = min (Dj)

    DotProduct=jD = ix * jiw( )

    i

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    7/36

    Klinkhachorn:CpE520

    Vector Normalization

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    8/36

    Klinkhachorn:CpE520

    Example - Preserve Direction

    ( ) 543 22 =+

    Divide each element of vector by the magnitude of

    the vector

    If inputs = 6 and 8

    Magnitude =

    Normalized inputs = 0.6 and 0.8

    If inputs = 3 and 4

    Magnitude =

    Normalized inputs = 0.6 and 0.8

    ( ) 108622 =+

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    9/36

    Klinkhachorn:CpE520

    Example - Preserve relationship between vectors

    ( ) 14.14515 22 =-

    Add another element which places the vector on

    the surface of a sphere and then divide by the

    magnitude of the vector

    If inputs = 6 and 8 , N = 15

    d =

    Normalized inputs = 0.4,0.53, .75

    If inputs = 3 and 4

    d =

    Normalized inputs = 0.2, 0.27, 0.94

    ( ) 18.111015 22 =-

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    10/36

    Klinkhachorn:CpE520

    SOM - Training algorithm

    Update weights of winner (Dc) and weights of

    all others in the neighborhood of winner.

    Wji(t+1) = Wji(t) + a(xi- Wji(t))

    for all neurons in neighborhood of winner

    Choice ofneighborhoods (size and

    dimensionality) affect network performanceand function.

    Neighborhood and gain should be decreases

    as time progresses

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    11/36

    Klinkhachorn:CpE520

    SOM - Neighborhoods

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    12/36

    Klinkhachorn:CpE520

    SOM Parameter Adjustments

    Gain parameter, a, should decrease as time

    progresses

    a = A*(1-t/T) where, A = initial gain

    T = final interation time

    Neighborhood adjustment, d, should also decrease

    as time progresses

    d = D*(1-t/T) where, D = initial neighborhood

    T = final interation time

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    13/36

    Klinkhachorn:CpE520

    SOM Training Summary

    Locate winner of Competitive Layer

    Adapt weights of all units inneighborhood to increase similarity

    between input and weight vectors

    Decrease gain and neighborhood astime progresses

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    14/36

    Klinkhachorn:CpE520

    SOM Example 1

    The weight vector distribution will

    approximate the probability density

    distribution of the input space

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    15/36

    Klinkhachorn:CpE520

    SOM Weight Maps

    Iteration: 100 Iteration: 1200

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    16/36

    Klinkhachorn:CpE520

    SOM Example 2

    Input Distribution Weight Map after 1700 iterations

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    17/36

    Klinkhachorn:CpE520

    SOM 1-D Neighborhood

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    18/36

    Klinkhachorn:CpE520

    SOM Example 3

    The weight vector distribution will

    approximate the probability density

    distribution of the input space

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    19/36

    Klinkhachorn:CpE520

    SOM Weight Map

    Iteration: 100 Iteration: 3400

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    20/36

    Klinkhachorn:CpE520

    SOM Characteristics

    Dimension Mapping (the dimensions

    of the input space are mapped to the

    dimensions of the output space)

    Variable discretion among input

    patterns due to frequency of occurance

    Weight distribution tends to

    approximate the probability density of

    the input vectors

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    21/36

    Klinkhachorn:CpE520

    SOM Example 4

    Input distribution

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    22/36

    Klinkhachorn:CpE520

    SOM Example 4

    Initial Weight Map

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    23/36

    Klinkhachorn:CpE520

    SOM Example 4

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    24/36

    Klinkhachorn:CpE520

    SOM Example 4

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    25/36

    Klinkhachorn:CpE520

    SOM Example 4

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    26/36

    Klinkhachorn:CpE520

    SOM Example 4

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    27/36

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    28/36

    Klinkhachorn:CpE520

    SOM Example 5

    Traveling Salesman Problem

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    29/36

    Klinkhachorn:CpE520

    SOM Example 5

    Traveling Salesman Problem

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    30/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

    24 bits RGB

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    31/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

    256 color Indexed

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    32/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    33/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

    Typical color reduction map

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    34/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

    Statistical clustering method

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    35/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

  • 8/3/2019 Unsupervised Learning: Part II Self-Organizing Maps

    36/36

    Klinkhachorn:CpE520

    SOM Example 5

    Color Reduction Map

    Kohonen neural Network