Graph mining with kernel self-organizing map

MotivationsDissimilarities and distances between vertices

Kernel SOMApplication and comments

Graph mining with kernel self-organizing map

Nathalie Villa-Vialaneixhttp://www.nathalievilla.org

Joint work with Fabrice Rossi, INRIA, Rocquencourt, France

Institut de Mathématiques de Toulouse, - IUT de Carcassonne, Université dePerpignan

France

SanTouVal, February 1st, 2008

Nathalie Villa - [email protected] SanTouVal - Feb. 2008



Table of contents

1 Motivations

2 Dissimilarities and distances between vertices

3 Kernel SOM

4 Application and comments




Exploring a big historic database

Data1000 agrarian contracts,

from four seignories (about 10 villages) of South West ofFrance,

established between 1250 and 1350 (before the HundredYears’ war).

Historian’s questions:family or geographical social links ?central people having a main social role ?. . .

⇒ Data mining is required.




A graph clustering problem

From the database, building a weighted graph:

with 615 vertices x1, . . . , xn := peasants found in thecontracts;

with weights (wi,j)i,j=1,...,n := ]{contracts where xi and xj arementionned}.

Number of vertices: 615Number of edges: 4193Total of weights: 40 329Diameter: 10Density: 2,2%

Clustering the vertices into homogeneous social groups tounderstand the structure of the peasant community.




Other fields modelized by large graphs

Computer science: World Wide Web, P2P network. . .

Social networks

Biology: Protein interactions, Neuronal network,. . .

Business, management: Transportation networks, Industrypartnerships. . .

Question: Understanding the structure of these large graphs

Clustering: building relevant homogeneous groups;

Graph drawing: giving a global representation of the graph.

Here: Self-Organizing Map for nonvectorial data.




Table of contents

1 Motivations


3 Kernel SOM





Usual dissimilarities between vertices

The Dice (Jaccard) index:

D(xi , xj) =

∣∣∣Γ(xi) ∩ Γ(xj)∣∣∣

|Γ(xi)|+ |Γ(xj)|

(non weighted graphs);

Dissimilarities based on the shortest paths;

Dissimilarities or distances based on the Laplacian matrix:spectral clustering.




Laplacian

DefinitionsFor a graph with vertices V = {x1, . . . , xn} having positive weights(wi,j)i,j=1,...,n such that, for all i, j = 1, . . . , n, wi,j = wj,i and di =

∑nj=1 wi,j ,

Laplacian: L = (Li,j)i,j=1,...,n where

Li,j =

{−wi,j if i , jdi if i = j

;




Laplacian: property I [von Luxburg, 2007]

Connected subgraphs

KerL = Span{IA1 , . . . , IAk } where Ai indicates the positions of thevertices of the ith connected component of the graph.

1

4

5

2

3

KerL = Span

10011

;

01100




Laplacian: property II [Boulet et al., 2008]

Perfect community : Complete subgraph (clique) which verticesshare the same neighbors outside the clique.

Laplacian and perfect communitiesFor a non weighted graph,

The graph has a perfect community with m vertices⇔

L has m eigenvectors such that each eigenvector has the samen −m coordinates that vanish.

Application :

But: only 1/3 of the graph can be drawn this way.




Laplacian: property II [Boulet et al., 2008]

Perfect community : Complete subgraph (clique) which verticesshare the same neighbors outside the clique.Application :

But: only 1/3 of the graph can be drawn this way.




Laplacian: property III [von Luxburg, 2007]

Min Cut problem: Suppose that we have a connected graph.Find a classification of the vertices of the graph, A1, . . . ,Ak suchthat

12

k∑i=1

∑j∈Ai ,j′<Ai

wj,j′

is minimum , is equivalent to minimize

H = arg minh∈Rn×k

Tr(hT Lh

)subject to

hT h = Ihi = 1/

√|Ai |1Ai

can be approached by


Tr(hT Lh

)subject to hT h = I

Spectral clustering: Find the k smallest eigenvectors of L , H, andmake the classification on the rows of H.






12

k∑i=1

∑j∈Ai ,j′<Ai

wj,j′

is minimum , is equivalent to minimize


Tr(hT Lh

)subject to

hT h = Ihi = 1/

√|Ai |1Ai

⇒ NP-complete problem.

can be approached by


Tr(hT Lh








12

k∑i=1

∑j∈Ai ,j′<Ai

wj,j′

is minimum can be approached by


Tr(hT Lh






A regularized version of L

Regularization : the diffusion matrix : pour β > 0,Kβ = e−βL =

∑+∞k=1

(−βL)k

k ! .⇒

k β : V × V → R

(xi , xj) → Kβi,j

diffusion kernel (or heat kernel).




Diffusion process on the graph

If Z0 = (1 1 1 . . . 1 1)T is the “energy” of each vertex at time 0 andif a small fraction ε of this energy is propagated among the edgesof the graph at each time step, then after t steps, the energy of thevertices of the graph is:

Zt = (1 + εL)t Z0

Limits: Time step↘ ∆t by t ↪→ t/(∆t) and ε ↪→ ε∆t ; then(∆t)→ 0 (continuous process) gives

lim Zt = eεtL = K εt




Properties

1 Diffusion on the graph: k β(xi , xj) ' quantity of energyaccumulated in xj after a given time if energy 1 is injected in xi

at time 0 and if diffusion is done continuously along the edges.β ' intensity of diffusion;

2 Regularization operator: for u ∈ Rn ∼ V , uT Kβu is higher forvectors u that vary a lot over “close” vertices of the graph.β ' intensity of regularization (for small β, direct neighbors aremore important);

3 Reproducing kernel property: k β is symmetric and positive⇒ ∃ Hilbert space (H , 〈., .〉) and φ : V → H such that

k β(xi , xj) = 〈φ(xi), φ(xj)〉.




Table of contents

1 Motivations


3 Kernel SOM





Kohonen map

Mapping the data onto a 2 dimensional map

Each neuron of the map, i = 1, . . . ,M is associated to aprototype, pi ∈ H ;

Neurons are related to each others by a neighborhoodrelationship (“distance”: d) :

Classifying the vertices on the map

Each xi is associated to a neuron (cluster or class) of the map,f(xi).




Preserving the initial topology

Energy

The goal is to minimize the energy of the map:

E =

∫ M∑i=1

h(d(f(x), i))‖x − pi‖2H

dP(x)

where h is a decreasing function (ex: h(t) = αe−t/2σ2).

Energy is approached by its empirical version:

En =n∑

j=1

M∑i=1

h(d(f(xj), i))‖xj − pi‖2H.

and minimization is approached by SOM algorithm.




Preserving the initial topology

Energy

The goal is to minimize the energy of the map:

E =

∫ M∑i=1

h(d(f(x), i))‖x − pi‖2H

dP(x)

where h is a decreasing function (ex: h(t) = αe−t/2σ2).

Energy is approached by its empirical version:

En =n∑

j=1

M∑i=1

h(d(f(xj), i))‖xj − pi‖2H.

and minimization is approached by SOM algorithm.Nathalie Villa - [email protected] SanTouVal - Feb. 2008



Batch kernel SOM [Villa and Rossi, 2007]

Initialize randomly γ0ji ∈ R (i, j = 1, . . . , n) and p0

j =∑n

i=1 γ0jiφ(xi).

Then, for l = 1, . . . , n repeat

Initialize randomly γ0ji ∈ R

(i, j = 1, . . . , n) and p0j =

∑ni=1 γ

0jiφ(xi).


Assignment step

for all xi ,

f(xi) = arg minj=1,...,M

n∑u,u′=1

γjuγju′k β(xu, xu′) − 2n∑

u=1

γjuk β(xu, xi)

Representation step

γlji =

h(f l(xi), j))∑ni′=1 h(f l(xi′ , j))






j =∑n

i=1 γ0jiφ(xi).


Assignment step

for all xi ,

f l(xi) = arg minj=1,...,M

∥∥∥∥∥∥∥φ(xi) −n∑

i=1

γljiφ(xi)

∥∥∥∥∥∥∥H


j =∑n

i=1 γ0jiφ(xi).


Assignment step

for all xi ,


n∑u,u′=1


u=1

γjuk β(xu, xi)

Representation step

γlji =

h(f l(xi), j))∑ni′=1 h(f l(xi′ , j))






j =∑n

i=1 γ0jiφ(xi).


Assignment step

for all xi ,

f l(xi) = arg minj=1,...,M

∥∥∥∥∥∥∥φ(xi) −n∑

i=1

γljiφ(xi)

∥∥∥∥∥∥∥H

Representation step

γlj = arg min

γ∈Rn

n∑i=1

h(f l(xi), j)

∥∥∥∥∥∥∥φ(xi) −n∑

l′=1

γl′φ(xl′)

∥∥∥∥∥∥∥2

H


j =∑n

i=1 γ0jiφ(xi).


Assignment step

for all xi ,


n∑u,u′=1


u=1

γjuk β(xu, xi)

Representation step

γlji =

h(f l(xi), j))∑ni′=1 h(f l(xi′ , j))






j =∑n

i=1 γ0jiφ(xi).


Assignment step

for all xi ,


n∑u,u′=1


u=1

γjuk β(xu, xi)

Representation step

γlji =

h(f l(xi), j))∑ni′=1 h(f l(xi′ , j))




Table of contents

1 Motivations


3 Kernel SOM





Results on a 7 × 7 rectangular map




Expected developments

1 Hierarchical clustering;2 Achieve a classification based on density criterium (joint work

with S. Gadat);3 Adapting the algorithm to very large graphs (thousands of

vertices).




References

Boulet, R., Jouve, B., Rossi, F., and Villa, N. (2008).Batch kernel SOM and related laplacian methods for social networkanalysis.Neurocomputing.To appear.

Villa, N. and Rossi, F. (2007).A comparison between dissimilarity SOM and kernel SOM for clustering thevertices of a graph.In Proceedings of the 6th Workshop on Self-Organizing Maps (WSOM 07),Bielefield, Germany.

von Luxburg, U. (2007).A tutorial on spectral clustering.Technical Report TR-149, Max Planck Institut für biologische Kybernetik.Avaliable at http://www.kyb.mpg.de/publications/attachments/luxburg06_TR_v2_4139%5B1%5D.pdf.


http://www.kyb.mpg.de/publications/attachments/luxburg06_TR_v2_4139%5B1%5D.pdf

http://www.kyb.mpg.de/publications/attachments/luxburg06_TR_v2_4139%5B1%5D.pdf

Graph mining with kernel self-organizing map

Science

Transcript of Graph mining with kernel self-organizing map