Appetizer: Why use a SOM€¦ · Xpto: 36 Visualize multidimensional data Project a n-dimensional...

Self-Organizing MapsV 1.4 V.Lobo, EN 2010

Self-Organizing Maps for Exploratory Data Analysis and Multidimentional DataVisualization

Victor LoboComputation World 2010Lisbon

Appetizer: Why use a SOM ?(1/3)Large amounts of data

⇒ Need for powerful analysis tools

Visualize and “feel” multidimensional dataCluster the data to simplify itExplore the data structure

SOM (Kohonen networks) can be put to better use than they are by most researchers…


Why do I really like them ?

Visual insight into multidimensional data.

Easy to use.

Good results in many problems.

SumaryWhat can I do with a SOM ?

What is a SOM ?Historical perspective & basic principles

Mathematical formalization.

How can I see results ?Exemples

Available Software

Maritime Applications


What can I do with a SOM?Define and Detect clustersVisualize multidimensional dataExplore dataOther…

Define clusters (k-mean clustering)Market segmentation

Localization

Age

Aver

age

acco

unt(

€)

Clients of a Bank ⇒ Account managers Shop location ⇒ Warehouse location


FordC.B.: 23.4N.Emp:10 000Receitas: 23MRatio A/P: 0.4Xpto: 36

PepsiC.B.: 23.4N.Emp:10 000Receitas: 23MRatio A/P: 0.4Xpto: 36

Visualize multidimensional dataProject a n-dimensional space onto 1 or 2 dimensions

Maintain topological relations

Detect similarities

Detect outliers

-0.50

0.51

1.5

-0.50

0.51

1.5-0.5

0

0.5

1

1.5

65

124

3

VolvoC.B.: 23.4N.Emp:10 000Receitas: 23MRatio A/P: 0.4Xpto: 36

Coca-ColaS.V.: 23.4N.Emp:55 000Sales: 23.1BNet Rev.:4.9BXpto: 36

3-dimensional pointsSOM

(bi-dimensional)

Coca-cola

Pepsi

Volvo

Ford

Sun

MercedesHeinz

GM

Johnson

SOM

n-dimensional data

Detect clustersExplore dataIdentify the data structure

How many clusters ?, How big ?...

Socio-economic indicators invarious countries

3D points located on6 vertices of a cube

Income

NºP

eopl

e

Income

NºP

eopl

e

Country A

Country B

Distribution of income in 2 countries


Other types of problems…

TSP, Robot contol, data sorting, data interpolation, data classification, feature extraction, sampling, alarm detection, etc, etc, etc

Travelling salesman problemSorting colours

What is a SOM ?

Historical perspectiveBasic principles and overview The maths


Historical perspective

Historical PerspectiveProf. Tuevo Kohonen (Technical University of Helsinki)

1970s - Associative Memory1982 - First papers on SOM1988 - Book on SOM, SOM paper in IEEE1990s - Widespread use1995,1997,2001 – “Self Organizing Maps” BookWorkshop on SOM (WSOM) conferences (next in 2011)

Motivation and inspirationVector Quantization methodsAssociative MemoriesPreserve topology over the mapping: nearby patterns should be mapped to nearby neurons


Biological inspiration (Just interesting…)

Biological systems have self-organization

There is evidence of:Layered structure in the brainInformation is spatially organized in the BrainSimilar “Concepts” are stores in adjacent areasExperimental work with animals suggests the is na organization similar to SOM of patterns in the visual cortex

Overview of SOM


Some general ideas:Neural Network

Set of neurons, or UNITSUnsupervised learning (unlike most Neural Nets)

TRAINING the neural netThe net is TRAINED, i.e., its parameters are adjusted (incrementally) according to the available data

USING the neural netAfter training the network, we can do many things with it: make predictions, detect clusters, etc,etc

Basic SOMNeurons (units) are set on a 2-dimensional grid

It may be 1-dimensional (line) ou m-dimensional …

One single layerCompetitive learning (almost “winner-take all”)

Input pattern (n-dimensional)

SOM (output space)


Basic SOM (another vision...)

Inputspace

xi,1

xi,2

xi,3

yj,1

yj,2

yj,3

x

∑ − 2)(1

jlil yx

Input vs output space

INPUT space = n-dimensional space of the data

OUTPUT space = space defined by the grid of units (usually 2)

Each unit (neuron) is a point in the output space (defined by the grid position), and a point in the input space (defined by the weight vector), just as the data patterns


Training the network

Units are pulled to the positions of the data, dragging with them their grid neighbours

SOM ≈ rubber sheet, stretched and twisted so that it passes in (or near) the places where the data patterns are

BMU- Best Matching UnitData patterns are compared with all network units; the closest is considered its BMU.-Best Matching Unit.That the data pattern is thus mapped to the position of it’s BMU, or for short, is mapped to its BMU.The BMU is updated (so that it resembles even more the data pattern that it maps), and it’s neighbors in the grid are also updated.

There is always a slight difference between the data patterns and the BMUs that represent them. That difference is the quantization error.


Example 1: 3D to 2D mapping3D points around 4 corners of a cube

Input Space(data)

Output Space(grid)

140

150

160

170

180

190

200

40 50 60 70 80 90 100

Weight (kg)

Hei

ght (

cm)

Example 2:

Physical example

Neurons=bolts=units

Sheet = input space

Problem:

Analyze height vs weight of the people in this room


Example 3: 2D to 2D mappingData uniformly distributed in the square

Used in the Matlab Demo

[Kohonen 95]

Example 4: 2D to 1D mapping

Animation

Red dots=dataGreen dots=unitsBlue lines=grid


Example 5: 2D to 2D

(animation)

Different densities

Mathematical formalization


Data, network and initializationLet:

X = { x1, x2,..xn } m-dimensional training dataset.xi = [ xi1, xi2, … xim]T , where xij are real-valued scalars.

W be a grid with p×q units wiwj = [ wj1, wj2, … wjm]TInitial wj chosen randomly in the “data area”

h(wi,wj,r) a real-valued funcion (neighborhood function)When || wi-wj||(na grellha) →∞ , h(wi,wj,r) →0r sets the radius (area of influence)

α be the learning rate0≤α ≤ 1Initialized with a large value, and decreases to zero during training

xi

wj

h

x w

α

wBMU dist

Training AlgorithmFor all xi∈X :1) Calculate the distance between xi and all units w

(di,j = || xi - wj || )2) Choose the BMU

wbmu : di,bmu = min( di,j)3) Update each unit according to the learning rule

wj = wj + α h(wbmu,wj, r) || xi – wj ||

Repeat the process, reducing α and r, using all the training data several times, until a stopping crieteria is reached.

123

Go !

Stop !


Main decisions:

How many units ? What type of grid ? How big ? How wide ? How high ?

How many iterations?

What type of neighborhood function ? Which initial value for r ? Which final value ?

Which initial value for α ?

Learning rate α

0≤α ≤ 1

Defines the plasticity of the networkLarge values ⇒ The network moves fast, and adapts quicklySmall values ⇒ The network moves slowly, and stabilizes

Start with large values, and reduce them to zero during training

Nº of iterations

αLinear decrease

Exponencial decrease


Neighborhood function

ShapeGaussianRectangular (bubble)RampOthers

Responsible for topological orderingForces “lateral connections” between units that are neighbors in the grid

222 )()(

21

),,(

−+−−

=r

mqnp

mnpqg erwwh

>−+−⇐

≤−+−⇐=

rmqnprmqnpwwh mnpqs 22

22

)()(0)()(1),(

Raduis r of the neighborhood function

Function of time (or number of iterations) r(t)Large radius ⇒ Many units are updates ⇒ Allows unfoldingSmall radius ⇒ Only close neighbors are updated ⇒ Fine-tuning

Initial values for r1st phase – similar to the network size2nd phase –radius of the clusters we hope to obtain

Final value for r0 ⇒Good fit (k-means)1 ⇒Keeps ordering, but has border effect

h

r


Size of the SOMk-means SOM (KSOM)

Few units1 unit for each expected cluster

Emergent SOM (ESOM)Many units for each expected clusterAllows representations of “complicated” and “varied” clustersAllows us to understand the data structure, detect the number of clusters, etc

C1

C1

C2

C2

Dimension and type of grid

Unidimensional grid ( line )Subsitute for k-meansAllows ordering of the data

3D or n-dimensional gridHard to visualize

2-dimensional gridMost usedSquare grid

Easy to work withHexagonal or triangular grid

Induces less distortion

Square grid

Hexagonal grid


Number of iterations and unfoldingNumber of iterations

Epochs vs individual datumWhen in doubt… choose more!

Unfolding problemsThere are many local minimaTopological error metrics

SolutionsSeveral initializations and runs2 phases (unfolding+fine tuning)Look at the topological and quantization errors

[Ritter 92]

Theoretical aspectsEnergy function that is minimized: [Hertz 91]

Highly non-linear, not global due to the concept of BMU

Reasonable results for 1-dimensional net and data [Cottrell]

Reasonable approximations for 2D [Ritter]There are “well behaved” update rules [Heskes]There “well founded” alternatives [Bishop]

Magnification factorUnit density ∝ (data density)k, with k


How can I see results?

U-Matrices (U-MAT)

Calibration (or labeling)

Component planes

Others

U-Matrices (U-MAT) [Ultsch 93]Allows identification of clusters

Computes the distance, in the input space, of neighbors in the output spaceDistance is color coded

Low values ⇒ Units close ⇒ clusterHigh values ⇒ Units far ⇒ empty space

Ideal Real U-mat

0

5

10

15

010

20

0.050.1

0.15

U-MAT


Calibration (or labeling)

ObjectiveIdentify what the cluster arePerform supervised classification

LVQ can be better …

How ?If training data have an associated classe……their BMUs can inherit those classes

Setosa

Setosa

Setosa

Versicolor

Versicolor

Versicolor

Versicolor

Versicolor

Versicolor

Virginica

Virginica

Setosa

Setosa

Versicolor

Versicolor

Versicolor

Versicolor

Virginica

Virginica

Setosa

Setosa

Setosa

Versicolor

Versicolor

Versicolor

Versicolor

Versicolor

Virginica

Setosa

Setosa

Setosa

Versicolor

Versicolor

Versicolor

Virginica

Versicolor

Virginica

Virginica

Setosa

Setosa

Setosa

Versicolor

Virginica

Virginica

Virginica

Virginica

Virginica

Setosa

Setosa

Setosa

Versicolor

Versicolor

Virginica

Virginica

Virginica

Virginica

Component planesSee how a given variable (component) varies along the map

See what defines the clusters, that variable contribute to them, how are they correlated

Candidate grade High-school grade

Maths exam Age

U-MAT


Other visualizations

HitsIdentify how many data are mapped to each unit

TrajectoriesSee how the BMU changes along a data series

Example

“Artificial” data


DatasetPoints in a 3D space

Generated with some dispersion around 6 corners of a cube

MATLAB codeSOMTOOLBOX

“True”classes 0

0.51

00.5

1

0

0.5

1

00.5

1

00.5

1

0

0.5

1

xy

z1

23

45

3

6

Training a 3x2 map

ParametersSquare grid (3x2)Linear initializationInitial r = 2Final r = 0Initial α = 0.1Num.Iter.=900

(1,5 epochs)

00.5

10

0.51

0

0.5

1

00.5

1

0

0.5

1

0

0.5

1

Animation


Visualization in the output space

During training After labeling with training data

5

3

1

6

4

2

00.5

1

0

0.5

1

0

0.5

1

Training a 30x20 map

ParametersSquare grid (30x20)Linear initializationInitial r = 15Final r = 0Initial α = 0.1Num.Iter.=900

(1,5 epochs)

00.5

1

0

0.5

1

0

0.5

1

900/900 training steps

00.5

1

0

0.5

1

0

0.5

1

Animação


U-MAT Coordenada X

Coordenada Y Coordenada Z

U-MAT and component planes

After labeling with training data

00.5

1

0

0.5

1

0

0.5

1

900/900 training steps

0

5

10

15

010

20

0.050.1

0.15

U-MAT

With larger variance in the data…

-0.50

0.51

1.5

0

1

2-1

-0.5

0

0.5

1

1.5

xy

z

U-MAT


Available Software

Available SoftwareSOM-PAK

(http://www.cis.hut.fi/research/som_lvq_pak.shtml)C code, compilable in UNIX or MS-DOSFast, reliable, easy to use

Somtoolbox for MATLAB(www.cis.hut.fi/projects/somtoolbox)

Good visualization, easily changed, “ideal” for R&D

Many othersSAS Enterprise Miner, SPSS-Clementine, IBM Intelligent Miner, Weka, etc...


Our implementation of SOM

GeoSOM-SUITEBased on SOMToolboxUser-friendly GUIImports ArcGIS ShapefilesAllows dyanmically-linked windowsImplements SOM and GeoSOM algorithmsMany views (Som,c-planes,u-mat,parallelplots)www.isegi.unl.pt/labnt/geosom

GeoSOM SuiteInteractive exploration of data with SOM


Maritime and GIS Applications

Common applications

Clustering and classificationSatellite or remotely sensed images, and other data sets

Cluster pixels to reduce the number of pattersManually classify a few pixelsUse the manually classified pixels to automatically classify the others[Niang 2003][Leloup 2007][Liu 2007][Cavazos 2000][Liu 2002,2007,2008][Tozuka 2008][Solidoro 2007]…

Control of Underwater Autonomous Vehicles

Detecting anomalous behavior of ships

Predicting tides in estuaries

Studying the movements of fluids


SOMs for underwater acoustics

WHO is making that noise ?

Other sources of sound (wind, waves, marine animals,seismic effects, etc) are added to the ships noise

The submarine itself generatesnoise and interference

The sonar equipmentcaptures the sound

Ships generate noisefrom a number of sources

Ship noise is passed

into the water

The ocean is a multipath and dispersive sound conductorand will introduce distortion

Other sources of sound (wind, waves, marine animals,seismic effects, etc) are added to the ships noise

(1/5)

SOMs for underwater acousticsData Sources

Classified Submarine Squadron dataRecordings at the acoustic tank of the navy shipyardRecordings off the Portuguese coast

Recordings in the Shipyard Tank Recordings at sea (Academy Boats)

(2/5)



For PCw/ soundcard

Operational Software

SpectrogramPull-down menus

Frequency Pointer

Blinkingindicator

KohonenSOM

Legend

Pop-upmenus

(3/5)

SOMs for underwater acousticsOther windows

Training (over PVM)Off-line analysis

0 1000 2000 3000 4000 5000 600070

80

90

100

110

120

130

140

150

160

0 500 1000 1500 2000 2500 30000

5

10

15

20

25

(4/5)



Main advantages of SOM in this caseExplore the available data

Is it interesting ?Can we do what we want ?

Classify known soundsDetect new sounds

Give similarities to known sounds

(5/5)

SOMs for route planningRoute planning

2 to 1-dimensional mappingTraining patterns

2-dimensional Coordinates of points of interest1-dimensional grid of units Patrol routes for ships

Illegal activities,and patrol routes

(1/3)


SOMs for route planningRoutes for multiple sensors, with moving targets

1-Dimensional simple caseUAVs for monitoring motorways

Basic ideaKeep on training the network as data arrivesKeep historical/reference data with lower weight

0 100 200 300 400 500 600 700 800 900 1000

0.16

0.18

0.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

Number of iterations

% d

etec

ted

cars

SOM basedBenchmark

(2/3)

SOMs for route planning

2-Dimensinal caseFleet of UAV monitoring ships

(3/3)


SOMs for zoning and environmentDefine areas for environment control in a very sensitive river estuary

Cooperation with École Navale (France)

2km

ATLANTIC OCEAN

SADO ESTUARY

(1/4)

SOMs for zoning and environmentPrevious work

Experts decided based on experienceClassical k-means (too fragmented)

2km

ATLANTIC OCEAN

SADO ESTUARY

(2/4)


SOMs for zoning and environmentClassical SOM

Better

2km

ATLANTIC OCEAN

SADO ESTUARY

(3/4)

SOMs for zoning and environmentGeoSOM

Provided the best balance

2km

(4/4)


Future work and interesting topics

Clustering trajectories and dealing with time/space issues in SOMs

Mapping data onto 3D SOMs visualizing themSome progress with geographical data…

Using SOMs for PipingRouting pipes and cables in a ship

Theoretical and experimental studies of magnification, convergence, and energy functions

Bibliography and support“Self-Organizing Maps”, Prof.Tuevo Kohonen

Springer-Verlag 2001

Technical University of Helsinki (www.cis.hut.fi/projects/somtoolbox/links)

Public-domain software, manuals, guides, e documentation

SOM-PAK for DOS, SOM Toolbox for MATLABExtensive bibliography

“www.cis.hut.fi/research/som-bibl”7718 references in December de 2008

My group’s homepagewww.isegi.unl.pt/labnt/geosom.htmlwww.isegi.unl.pt/docentes/vlobo


That’s allFolks !

Appetizer: Why use a SOM€¦ · Xpto: 36 Visualize multidimensional data Project a n-dimensional...

Documents

Transcript of Appetizer: Why use a SOM€¦ · Xpto: 36 Visualize multidimensional data Project a n-dimensional...