Graph spectra,Information

Theory and Complex Neworks

Edwin Hancock

Department of Computer Science

University of York

Supported by a Royal Society

Wolfson Research Merit Award

Aims

• How to characterise graphs in a compact manner.

• Laplacian matrix and graph spectra.

• Spanning trees and prime cycles.

• Relationship with quantum and random walks, and with

information theory.

Structural Variations

Protein-Protein Interaction Networks

Graph data

• Problems based on graphs arise in areas such as

language processing, proteomics/chemoinformatics,

data mining, computer vision and complex systems.

• Relatively little methodology available, and vectorial

methods from statistical machine learning not easily

applied since there is no canonical ordering of the nodes

in a graph.

• Can make considerable progress if we develop

permutation invariant characterisations of variations in

graph structure.

Graph data

• Problems based on graphs arise in areas such as

language processing, proteomics/chemoinformatics,

data mining, computer vision and complex systems.

• Relatively little methodology available, and vectorial

methods from statistical machine learning not easily

applied since there is no canonical ordering of the nodes

in a graph.

• Can make considerable progress if we develop

permutation invariant characterisations of variations in

graph structure.

Graph data

• Problems based on graphs arise in areas such as

language processing, proteomics/chemoinformatics,

data mining, computer vision and complex systems.

• Relatively little methodology available, and vectorial

methods from statistical machine learning not easily

applied since there is no canonical ordering of the nodes

in a graph.

• Can make considerable progress if we develop

permutation invariant characterisations of variations in

graph structure.

Characterising graphs

• Topological: e.g. average degree, degree distribution, edge-density, diameter, cycle frequencies etc.

• Spectral or algebraic: use eigenvalues of adjacency matrix or Laplacian, or equivalently the co-efficients of characteristic polynomial.

• Complexity: use information theoretic measures of structure (e.g. Shannon entropy).

Complex systems

• Spatial and topological indices: node degree

stats; edge density;

• Communicability: communities, measures of

centrality, separation, etc. (Baribasi, Watts and

Strogatz, Estrada).

• Processes on graphs: Markov process, Ising

models, random walks, searchability (Kleinberg).

Aims

• Show how random walks can be used as

probes of graph structure.

• Explain links with spectral graph theory.

• Show links with complexity

characterisations, and information theory.

Our contributions

• IJCV 2007 (Torsello, Robles-Kelly, Hancock) –shape classes from edit distance using pairwise clustering.

• PAMI 06 and Pattern Recognition 05 (Wilson, Luo and Hancock) – graph clustering using spectral features and polynomials.

• PAMI 07 (Torsello and Hancock) – generative model for variations in tree structure using description length.

• CVIU09 (Xiao, Wilson and Hancock) – generative model from heat-kernel embedding of graphs.

• QIC09 (Emms, Wilson and Hancock) quantum version of commute time.

• PR09a,b,c,(Emms, Wilson and Hancock) graph matching using quantum walks, lifting cospectrality of graphs using quantum walks.

• PR109(Xiao, Wilson and Hancock) graph characteristics from heat kernel trace.

• TNN11 (Ren, Wilson and Hancock) Ihara zeta function as graph characterisation,

• Phy Rev E (Escolano, Hancock, Lozano) Heat Diffusion: The theromdynanmc depth approach to network complexity

Random walks

And links to graph spectra.

Problem studied • How can we find efficient means of characterising

graph structure which does not involve exhaustive

search? Enumerate properties of graph structure without

explicit search, e.g. count cycles, path length frequencies,

etc..

• Can we analyse the structure of sets of graphs

without solving the graph-matching problem? Inexact graph matching is computational bottleneck for

most problems involving graphs.

• Answer: let a random walker do the work.

Graph Spectral Methods Use eigenvalues and eigenvectors of adjacency graph (or

Laplacian matrix) - Biggs, Cvetokovic, Fan Chung

Singular value methods for exact graph-matching and

point-set alignment). (Umeyama, Scott and Longuet-Higgins,

Shapiro and Brady).

Use of eigenvectors for image segmentation (Shi and Malik)

and for perceptual grouping (Freeman and Perona, Sarkar

and Boyer).

Graph-spectral methods for indexing shock-trees (Dickinson

and Shakoufandeh)

Random walks on graphs

• Determined by the Laplacian spectrum

(and in continuous time case by heat-

kernel).

• Can be used to interpret, and analyse,

spectral methods since they can be

understood intuitively as path-based.

Graph spectra and

random walks

Use spectrum of Laplacian matrix

to compute hitting and commute

times for random walk on a graph

Laplacian Matrix

• Weighted adjacency matrix

• Degree matrix

• Laplacian matrix

otherwise

EvuvuwvuW

0

),(),(),(

Vv

vuWuuD ),(),(

WDL

Laplacian spectrum

• Spectral Decomposition of Laplacian

• Element-wise

)()(),( vuvuL kk

k

k

T

k

k

kk

TL

),....,( ||1 Vdiag )|.....|( ||1 V

||21 ....0 V

Properties of the Laplacian

• Eigenvalues are positive and smallest eigenvalue is zero

• Multiplicity of zero eigenvalue is number connected components of graph.

• Zero eigenvalue is associated with all-ones vector.

• Eigenvector associated with the second smallest eigenvector is Fiedler vector.

||21 .....0 V

Discrete-time random walk

• State-vector

• Transition matrix

• Steady state

• Determined by leading eigenvector of

adjacency matrix A, or equivalently the second

(Fiedler) eigenvector of the Laplacian.

01 pTTpp t

tt

ADT 1

ss Tpp

Continuous time random walk

Heat Kernels

• Solution of heat equation and measures

information flow across edges of graph

with time:

• Solution found by exponentiating

Laplacian eigensystem

tt Lht

h

TT

kk

k

kt tth ]exp[]exp[

TWDL

Heat kernel and random walk

• State vector of continuous time random

walk satisfies the differential equation

• Solution

tt Lp

t

p

00]exp[ phpLtp tt

Heat kernel and path lengths

• In terms of number of paths of

length k from node u to node v

!),(]exp[),(

|

1 k

tvuPtvuh

k

k

kt

),( vuPk

)()()1(),( vuvuP ii

k

Vi

ik

Example.

Graph shows spanning tree of heat-kernel. Here weights of graph are

elements of heat kernel. As t increases, then spanning tree evolves from

a tree rooted near centre of graph to a string (with ligatures).

Low t behaviour dominated by Laplacian, high t behaviour dominated by

Fiedler-vector.

Moments of the heat-kernel

trace

….can we characterise graph by

the shape of its heat-kernel trace

function?

Heat Kernel Trace

Time (t)->

Trace

]exp[][ thTri

it

Shape of heat-kernel

distinguishes

graphs…can we

characterise its shape

using moments

Heat Kernel Trace

Time (t)->

Trace

]exp[][ thTri

it

Shape of heat-kernel

distinguishes

graphs…can we

characterise its shape

using moments

dtthTrts s )]([)(0

1

Use moments of heat kernel

trace:

Rosenberg Zeta function

• Definition of zeta function

s

k

k

s

)()(0

Heat-kernel moments

• Mellin transform

• Trace and number of connected components

• Zeta function

dttts

i

ss

i ]exp[)(

1

0

1

dttts s ]exp[)(0

1

]exp[][0

tChTri

it

dtChTrts

s t

ss

i

i

][)(

1)(

0

1

0

C is multiplicity of zero

eigenvalue or number of

connected components in

graph.

Zeta-function is related

to moments of heat-

kernel trace.

Zeta-function behavior

Objects

72 views of each object taken in 5 degree intervals as camera moves

in circle around object.

Feature points extracted using corner detector.

Construct Voronoi tesselation image plane using corner points as

seeds.

Delaunay graph is region adjacency graph for Voronoi regions.

Heat kernel moments

(zeta(s), s=1,2,3,4)

PCA using zeta(s), s=1,2,3,4)

PCA on Laplace spectrum

Ox-Caltech database

Line-patterns

• Use Huet+Hancock representation (TPAMI-99).

• Extract straight line segments from Canny edge-map.

• Weight computed using continuity and proximity.

• Captures arrangement Gestalts.

Zeta function derivative

• Zeta function in terms of natural exponential

• Derivative

• Derivative at origin

]lnexp[)()(00

kk

k

s

k ss

]lnexp[ln)('0

k

kk ss

0

0

1lnln)0('

k

k k

k

Meaning

• Number of spanning trees in graph

)](exp[)( ' od

d

G

Vu

u

Vu

u

COIL

Ox-Cal

Eigenvalue polynomials (COIL)

Trace

Determinant

Eigenvalue Polynomials (Ox-

Cal)

Spectral polynomials (COIL)

Spectral Polynomials (Ox-Cal)

COIL: node and edge frequency

Ox-Cal: node+edge frequency

Performance

• Rand index=correct/(correct+wrong).

Zeta func. 0.92

Sym. Poly.

(evals)

0,90

Sym. Poly.

(matrix)

0.88

Laplacian

spectrum

0.78

Deeper probes of structure

Ihara zeta function

Zeta functions

• Used in number theory to characterise

distribution of prime numbers.

• Can be extended to graphs by replacing

notion of prime number with that of a

prime cycle.

Ihara Zeta function

• Determined by distribution of prime cycles.

• Transform graph to oriented line graph (OLG) with edges as nodes and edges indicating incidence at a common vertex.

• Zeta function is reciprocal of characteristic polynomial for OLG adjacency matrix.

• Coefficients of polynomial determined by eigenvalues of OLG adjacency matrix.

• Powers of OLG adjacency matrix give prime cycle frequencies.

Oriented Line Graph

Ihara Zeta Function

• Ihara Zeta Function for a graph G(V,E)

– Defined over prime cycles of graph

– Rational expression in terms of characteristic polynomial of oriented line-graph

A is adjacency matrix of line digraph

Q =D-I (degree matrix minus identity

Characteristic Polynomials from IZF

• Perron-Frobenius operator is the adjacency matrix TH

of the oriented line graph

• Determinant Expression of IZF

– Each coefficient,i.e. Ihara coefficient, can be derived from the

elementary symmetric polynomials of the eigenvalue set

• Pattern Vector in terms of

Analysis of determinant

• From matrix logs

• Tr[T^k] is symmetric polynomial of

eigenvalues of T

]][exp[]det[

1)(

1 k

sTTr

TsIs

k

k

k

N

N

N

N

TTr

TTr

TTr

.............][

.....

...][

........][

21

2

21

2

1

2

1

1

Distribution of prime cycles

• Frequency distribution for cycles of length l

• Cycle frequencies

l

l

lsNsds

ds )(ln

][)(ln)!1(

10

l

sl

l

l TTrsds

d

lN

Experiments: Edge-weighted Graphs

Feature Distance

& Edit Distance

Three Classes of Randomly

Generated Graphs

Experiments: Hypergraphs

Complexity

Information theory, graphs and

kernels.

Protein-Protein Interaction Networks

Characterising graphs

• Topological: e.g. average degree, degree distribution, edge-density, diameter, cycle frequencies etc.

• Spectral or algebraic: use eigenvalues of adjacency matrix or Laplacian, or equivalently the co-efficients of characteristic polynomial.

• Complexity: use information theoretic measures of structure (e.g. Shannon entropy).

Complexity characterisation

• Information theory: entropy measures

• Structural pattern recognition: graph

spectral indices of structure and topology.

• Complex systems: measures of centrality,

separation, searchability.

Information theory

• Entropic measures of complexity:

Shannon , Erdos-Renyi, Von-Neumann.

• Description length: fitting of models to

data, entropy (model cost) tensioned

against log-likelihood (goodness of fit).

• Kernels: Use entropy to computeJensen-

Shannon divergence

Von-Neumann Entropy

• Derived from normalised Laplacian

spectrum

• Comes from quantum mechanics and is

entropy associated with density matrix.

2

ˆln

2

ˆ||

1

i

V

i

iVNH

TDADDL ˆˆˆ)(ˆ 2/12/1

Approximation

• Quadratic entropy

• In terms of matrix traces

||

1

2||

1

||

1

ˆ4

1ˆ2

1

2

ˆ1

2

ˆ V

i

i

V

i

ii

V

i

iVNH

]ˆ[4

1]ˆ[

2

1 2LTrLTrHVN

Computing Traces

• Normalised Laplacian

• Normalised Laplacian squared

||]ˆ[ VLTr

Evu vudd

VLTr),(

2

4

1||]ˆ[

Simplified entropy

Evu vu

VNdd

VH),( 4

1||

4

1

Collect terms together, von Neumann

entropy reduces to

Homogeneity index

Evuvuvu

Evu

vu

ddddVVG

ddG

),(

2

),(

2/12/1

211

1||2||

1)(

)()(

Based on degree

statistics

Homogeneity meaning

Evu

vuAvuCTG),(

),(2),(~)(

Limit of large degree

Largest when commute time differs from 2

due to large number of alternative

connecting paths.

Uses

• Complexity-based clustering (especially

protein-protein interaction networks).

• Defining information theoretic (Jensen-

Shannon) kernels.

• Controlling complexity of generative

models of graphs.

Protein-Protein Interaction Networks

Experiment

Quantum walks

….have richer structure due to interference and complex representation.

What can a quantum walker discover about a graph that a classical walker can not?

Quantum Walks

• Have both discrete time (DQW) and continuous time (CQW) variants.

• Use qubits to represent state of walk.

• State-vector is composed of complex numbers rather than an probabilities. Governed by unitary matrices rather than stochastic matrices.

• Admit interference of different feasible walks.

• Reversible, non ergodic, no limiting distribution.

Quantum Walks

• Governed by a wave-function (complex-number) rather than a probability vector.

• Collapse wave function to give observable probability state-vector

• Different quantum walks on a graph can interfere, and this gives them interesting additional structure.

*

tttp

New possibilities

Interference in quantum walks used to develop alternatives to classical random walks algorithm.

Consequences of interference

• Quantum walks hit exponentially faster than classical

ones.

• Local symmetry reflected in cancellation of complex

amplitude.

• Can be exploited to lift co-spectrality of strongly regular

graphs.

• Quantum version of commute time embedding can be

used to detect symmetries.

Literature

• Fahri and Gutmann: CQW can penetrate family of trees in polynomial time, whereas classical walk can not.

• Childs: CQW hits exponentially faster than classical walk.

• Kempe: exploits polynomial hitting time to solve 2-SAT problem and suggest solution to routing problem.

• Shenvi, Kempe and Whaley: search of unordered database.

Discrete Quantum walk

Discrete-time Quantum Walk

• State of walk described by a complex wave function which is updated according to

• Update operator U=TC is a unitary rather than stochastic matrix.

• The coin operator C gives the amplitudes for each path.

• The transition operator T determines transitions between nodes.

ttt UTC 1

The unitary operator governing the evolution of the walk can be written in matrix form as

where the basis states are the set of all ordered pairs (i,j) such that

• Based on same graph transformation as Ihara zeta function.

• However, the eigenvalues of U are nnn where the i are the n eigenvalues of the transition matrix for the lassical random walk (the adjacency matrix normalised) and

• Thus the spectrum of U is of no use in distinguishing strongly regular graphs.

Unitary matrix for DQW

Quantum Walk

Ordered pairs of

vertices

Unitary

Classical Walk

Vertices

Stochastic

•States:

•Transition

Matrix

•Probability

of state:

•Evolution

Discrete Walks Compared

Example: Walk on the line

• Probability distribution for the quantum walk on the line (T=200 steps, only even values plotted).

• Very intricate

• Highly peaked around position

• Almost uniform in centre.

Walks compared

Classical walk Quantum walk

Matrix interpretation

• For a classical walk entries of r-th power of

A is number of paths of length r between a

pair of nodes.

• Motivates the study of r-th power of U

which is total amplitude associated with all

quantum walks of length r between a pair

of nodes.

Strongly Regular Graphs

• Two SRGs with parameters (16,6,2,2)

A B

Is the sp+(U3) able to distinguish all

SRGs?

• There is no method

proven to be able to

decide whether two

SRGs are isomorphic in

polynomial time.

• There are large families

of strongly regular

graphs that we can test

the method on

MDS embeddings of the SRGs with parameters

(25,12,5,6)-red, (26,10,3,4)-blue, (29,14,6,7)-black,

(40,12,2,4)-green using the adjacency spectrum

(top) and the spectrum of S+(U^3) (bottom).

Testing

• The algorithm takes time

O(|E|^3).

• We have tested it for all

pairs of graphs from the

table opposite and found

that it successfully

determines whether the

graphs are isomorphic in

all cases.

Conclusions

i) Spectral graph theory and links to random walk.

ii) Characterisations based on graph spectra and random

walks.

iii) Zeta functions provide compact representation, linked to

spanning tree and prime cycle structure of graphs.

iv) Ihara zeta function constructed from oriented line graph

and is linked to discrete quantum walk on a graph.