Applications of Information Geometry

Clustering based on Divergence: Cluster CenterStochastic Reasoning: Belief Propagation in Graphical ModelSupport Vector MachineBayesian Framework and Restricted Boltzmann MachineNatural Gradient in Multilayer Perceptron LearningIndependent Component AnalysisSparse Signal Analysis and Minkovskian GradientConvex Optimization

Clustering of Points1 2 3{ , , ,..., }ND

Clustering : center of a cluster of points

1, , mC

arg min , ii

D

Center of C

C1

23

m

is simply the arithmetic average in the dual coordinate system

k‐means:  clustering algorithm 

*

[ : ] ( ) ( )[ : ]

( ) 01*

i i i

i i

i

i

DD

m

k‐means:  clustering algorithm

1. Choose k cluster centers2. Classify patterns of D  into clusters3. Calculate new cluster centers

Voronoi Diagram:  Boundaries of Clusters

1 2[ : *] [ : *]D D

Total Bregman divergence (Vemuri)

2

( ) ( )tBD( : )

1 | ( ) |p qp q

p qq

t-center of C

2

*

1

1

i i

i

i

i

ww

w

Conformal change of divergence

: :D p q q D p q

ij ijg p g

( )

logijk ijk k ij j ik i jk

i i

T T s g s g s g

s

t-center is robust

1, , ;

1; ,

nE

n

y

z y

influence fun ;ction z y

robust as : c z y

1, yGw

z y

y

21

1

y y

yw

w

y

y

Robust: is boundedz

21Euclidean case 2

f x

1

2,

1

,

G

yz yy

z y y

1

1 [ : ] [ : y]minimize ( )1

i

i N

D DN w w

MPEG7 database• Great intraclass variability, and small interclass dissimilarity.

Shape representation

First clustering then retrieval

Other TBD applications

Diffusion tensor imaging (DTI) analysis [Vemuri]• Interpolation• Segmentation

Baba C. Vemuri, Meizhu Liu, Shun-ichi Amari and Frank Nielsen, Total Bregman Divergence and its Applications to DTI Analysis, IEEE TMI, to appear

Information Geometryof Stochastic Reasing

Belief Propagation in Graphical Model

• Shun‐ichi Amari (RIKEN BSI)• Shiro Ikeda (Inst. Statist. Math.)• Toshiyuki Tanaka (Kyoto U.)

Stochastic Reasoning:  Graphical Model

( , , , , )p x y z r s

( , , | , ), , ,... 1, 1p x y z r sx y z

Stochastic Reasoningq(x1,x2,x3,…| observation)

X= (x1 x2 x3 …..) x = 1, ‐1

Ｘ= argmax q(x1, x2 ,x3 ,…..)     maximum likelihood

Ｘi = sgn E[xi]     least bit error rate estimator

Mean Value Marginalization: projection to independent distributions

0 1 1 2 2 0( ) ( ) ( )... ( ) ( )n nq q x q x q x q x x

1, 1( ) ( ..., ) .. ..i i n i nq x q x x dx dx dx

0[ ] [ ]q q E x E x

1

1

1

1 2

exp

,

1, 1

e p

,

x

s

ij

L

i j i

i i r qr

r r i i s

i

i

q k x c

c c x x r

q w x x h x

i i

x r i i

x

x x

x

Boltzmann machine, spin glass, neural networksTurbo Codes, LDPC Codes

cliques

Computationally DifficultComputationally Difficult

exp r q

q E

q c

x x

x x

mean‐field approximation

belief propagation

tree propagation, CCCP (convex‐concave)

Information Geometry ofMean Field Approximation

• m‐projection• e‐projection

q = argmin D[q:p]q = argmin D[p:q]

( )[ : ] ( )log( )x

q xD q p q xp x

0e

0m 0 { ( )}i i iM p x

0( )p x M

0

*

*

(x, *)

(x, *) *

( ) *( )( )*( )

( ), * ( *) ( ) *( ) 0

[ ] [ ]

px

q p

p q M

p x

q x p xt xp x

t x x x q x p x

E x E x

m‐projection keeps the expectation of x

Information GeometryInformation Geometry

0 0 0 0 0, exp

, expr r r r r r

M p

M p c

x x

x x x

1, ,r L

q x

rM

'rM

0M

( ) exp{ ( )rq x c x

Belief Prop Algorithm

0M

rM

'rM

r

'r

r

'r

0 0

1 t0

1 1'

'11

0

( , ) ( , )

( , ) : belief for ( )

t tr r r

t tr r r r r

t tr r

r rtt

r

p x p x

p x c x

Belief PropagationBelief Propagation( , ) exp{ ( ) }r r r rp x c x x

Equilibrium of BPEquilibrium of BP , rξ

1)  m‐condition

*0 ,r rp x

-flat submanifold m M

rM

'rM

0M

rM

'rM

0M

q2)  e‐condition 

*r

θ ξ

-flat submanifold q ex

ξ1( ')ξ θ

Belief Propagation  e‐condition OK

CCCP m‐condition OK

1 2

1 2 1 2

( ; , , , ' '( , , ) ( ' , ' , ' )

L r

L L

θ ξ ξ ξ θ ξ

0 0

0

: ( , ) ( , )

'

r r r r

r

p x p x

θ θ

θ

Convex‐Concave ComputationalProcedure (CCCP) :   A. Yuille

1 21

1 2

( ) ( ) ( )

( ) ( )

t t

F F F

F FElimination of double loops

Geometry of Support Vector Machinemodification of kernel

Simple perceptron: decision surface

( )| |

| |

f x w x bw x bd

w

2minimize | |constraint ( ) 1i i

wy w x b

21( , , ) | | ( )2

0 : support vector : 0

( , )

i i

i i i i

i i i i i i

L w b w w x b

y x

w y x f x w y x b

Embedding in higher‐dimensional space

( )( ) ( )

x z xf x w x b

z: infinite‐dimensional

Kernel trick ( , ') ( ) ( ')

( , ') ( ') ' ( ')

1( ) ( )

( , ) ( , )

i i i

ii

i i i

K x x x x

K x x x dx x

x x

f x w y K x x

2 2

2

| '

| ( ) ( ) | z( ) ( )

( ) ( ) ( )

( ) ( , x')

i ji j

iji j

ij x xi j

ds z x dx z x x z x dx dxx x

g x x xx x

g x K xx x

2

2

2

( ) exp{ { ( )}

( , ') ( ) ( ') ( , ')( ) exp{ | x x * | }

g ( ) { ( )} g ( ) ( ) ( )i i

ij ij i j

x f x

K x x x x K x xx

x x x x x

Conformal Transformation of Kernel

Basic Principles ofUnsupervised and SupervisedLearning Toward Deep Learning

Shun‐ichi Amari  (RIKEN Brain Science Institute)collaborators:    R. Karakida, M. Okada (U. Tokyo)

Deep LearningSelf‐Organization  +  Supervised Learning

RBM: Restricted Boltzmann MachineAuto‐Encoder,  Recurrent Net

DropoutContrastive divergence

Boltzmann Machine

RBM:  Restricted Boltzmann Machine

RBM

Simple Hebbian Self‐Organization

self‐organization of  

Self‐Organization

Interaction of Hidden Neurons

Bayesian Duality in Exponential Family

Data x          Parameter (higher‐order concepts) 

Curved exponential family

RBM= h,     x = Wvx = v        = hW

Gaussian Boltzmann Machine

Equilibrium Solution

Equilibrium Solution

General Solution 

•

You can choose m(≤ k) eigen values  form

• diagonalized by 

Stable Solution  the case of m = k

‐ No connections within layers  

Contrastive Divergence

50

• How to train RBM  Maximum Likelihood (ML) learning is hard 

• RBM  ‐ 2‐layered probabilistic neural network

Sampling

EquilibriumInput 

Many iterations of Gibbs Sampling demand too muchcomputational time 

W

Hidden hVisible v

Contrastive Divergence Solution

Two Manifolds

Geometry of CDn (contrastive divergence)

Bernoulli‐Gaussian RBM

ICAR. Karakida

Simulation

InputMixing

Sources p (s) ICA Solution

OutputUniformDistribution

The number of Neurons: N = M = 2,  σ = 1/2

Independent sources are extracted in G‐B RBM

CD

55

Information Geometry of Neuromanifolds

Natural Gradient and Singularities

Shun-ichi AmariRIKEN Brain Science Institute

Multilayer Perceptron

Mathematical Neurons

i iy w x h w x

x y( )u

u

Multilayer Perceptrons

i iy v n w x

21; exp ,2

, i i

p y c y f

f v

x x

x w x

x ｙ

1 2( , ,..., )nx x x x

1 1( ,..., ; ,..., )m mw w v v

Multilayer Perceptron: Neuromanifold

1 1,

,

, ; ,i i

m m

y f

v

v v

x θ

w x

θ w w

neuromanifold ( )x

space of functions

Universal Function Approximator

xwx

xx

xx

ii

i

m

ii

i

N

i

v

a

v

1

1i

Learning from examples

ˆ, xfx

1 1examples { , , , , }n nD y y x x

learning ; estimation

Backpropagation ‐‐‐gradient learningBackpropagation ‐‐‐gradient learning

1 1

2

examples : , , , training set1( , ; ) ,2

log , ;

t ty y

E y x y f

p y

x x

x

x

,

t t

i i

E

f v

x w x

( , )y f x n

Multilayer Perceptron: NeuromanifoldMetric and Topology

1 1,

,

, ; ,i i

m m

y f

v

v v

x θ

w x

θ w w

neuromanifold ( )x

space of functions

Metric: Riemannian manifold

22

ij i j

T

ds d

g d d

d G d

j

i

d

log ( | ; ) log ( | ; )( ) [ ]iji j

p y x p y xg E

Topology: Neuromanifold

• Metrical structure

• Topological structure

singularities

Geometry of singular model

y v n w x

Ｗ

v| | 0v w

Parameter Space S

S

Equivalence

1) 02)

/

i i

i j i j

vv v

M S

ww w

i iy v n w x

Topological Singularities

: parameter spaceS

: behavioral spaceM

/M S

2  hidden‐units

1 1 2

1 2 1 1

1 2

: 0

1

y v v n

S v v

v x w v x w

2

2 2

w x w x

w w w w

Gaussian mixtures

21( ) exp2i ip x v x w

Gaussian mixture

1 2 1 2; , , 1p x v w w v x w v x w

21 1exp22

x x

1 2 s ingular: , 1 0w w v v

1w2w

v

Learning, Estimation, and Model Selection

gen 0

train emp

: ;

;

E D p y p y

E D p y

x x

x

gen

gen train

: dimension2dE dn

dE En

Problem of Backprop

• slow convergence‐‐‐‐plateau‐‐‐saddle

• local minima 

( , ; )t t t t tl x y

Flaws of MLPFlaws of MLPslow convergence : plateau

local minima

Boosting, Bagging, SVM

error

Steepest Direction --- Natural GradientSteepest Direction --- Natural Gradient

1

1

2

, ,

=

n

i jij

l ll

l G l

d d Gd G d d

d

( )l

( , ; )t t t t tl x y

Natural Gradient

2 2

1

max

under ij i j

dl l d l l d

d g d d

d l G l

( , ; )t t t t tl x y

Information Geometry of MLPInformation Geometry of MLPNatural Gradient Learning :

S. Amari ; H.Y. Park

1

1 1 1 11 1 T

t t t t

lG

G G G f f G

Adaptive natural gradient learning

Regular statistical model ,

: Fisher information

M p x

G

11TE Gn

01ˆ, : ,2

2

E KL p x p x G En

dn

AIC,   MDL

Landscape of error at singularityMilner attractor

Dynamics of Learning

1,

( , ), ( , )

( , ) , ( , )

d dl G ldt dt

du dzf u z k u zdt dt

du f u zdz k u z

2 2 1 log

2u z z c

Coordinate Transformation

2 1

1 1 2 2

1 2

2 2

: 0

1

v vv

v v v v vv vz z

v

u w w uw ww w w

Dynamics of Learning:  Natural gradient works!!

1,

( , ), ( , )

( , ) , ( , )

d dl G ldt dt

du dzf u z k u zdt dt

du f u zdz k u z

2 2 1 log

2u z z c

Dynamic vector fields: General case (|z|<1 part stable)

Dynamic vector fields: General case (|z|>1 part stable )

Adaptive Natural Gradient works well

Signal ProcessingICA : Independent Component Analysis

t t t tA x s x s

sparse component analysis

positive matrix factorization

mixture and unmixtureof independent signals

2x

1s

ns2smx

1x1

n

i ij jj

x A s

x As

Independent Component Analysis

1

i ij jA x A s

W W A

x s

y x

observations: x(1), x(2), …, x(t)recover: s(1), s(2), …, s(t)

s A W y

x

Semiparametric Statistical Model

( ; , ) | | ( )p r rx W W Wx

unknown

x(1), x(2), …, x(t)

ir r , ( ) :r s 1W A

Natural Gradient

, Tl

y W

W W WW

Space of Matrices : Lie group

-1d dX WW

2 1tr trT T T

T

d d d d d

ll

W X X WW W W

W WW

:dX

I I d X

WdW W

non-holonomic basis

1W

Information Geometry of ICA

natural gradientestimating functionstability, efficiency

S ={p(y)}

1 1 2 2{ ( ) ( )... ( )}n nI q y q y q y

{ ( )}p Wx

r q

( ) [ ( ; ) : ( )] ( )

l KL p qr

W y W yy

Estimating Functions

,

)0, '

, ' 0, 'rE

W

W F(y,WW W

F y WW W

estimating equation

0 t tt

tF y ,W y Wx

learning

t t t W F W x

Admissible class

on-line learnin

, { }

,

0: estimating equation

canonical estimating functio

g :

n:

T

t

t t t t

W

F IW

F y W I y y

F R W F y W

R W F Wx

W R W F W x

{ }i j j iy y y y W

Basis Given: overcomplete caseSparse Solution

many solutionsmany 0

ˆ

i i

i

t t

A s

s

A

x s a

x s

ˆ ˆ ˆ: A x Assparse

generalized inverse

ˆmin 2is

sparse solution

ˆmin ii s

ˆ ˆ ˆ: x A Assparse

2 :-normL

1 :-normL

Minkovskian Gradient

min : convex functionconstraint F c

typical case:

21 1 ( *) ( *)2 21 ;

T

pi

X G

Fp

y

2, 1, 1/ 2p p p

Optimization under Sparsity Condition:

(X y n)

: -sparsek

2 logm k n

1

, 1,2,...k

tt i i t

i

y x n t N

1

2

xN

xx

X X

y n

Linear regression

Overcomplete:  N < kunder‐determined  infinitely many solutions

Sparsity constraint solves the problem

21ˆ arg min , ( )2t t

t

y u u

x

' 0t t tt

y x x

ˆ, T TG X X G X y

1 †ˆ T TX X X

y = X y

Maximum likelihood estimator

Euclidean case (Gaussian noise):

Not sparse

Sparse Solution

*min min [ ] ( ) ( *) ( *)( *) ( *) 0

penalty = pp i

D

F c

0

1 1

22

#1[ 0] :

:

: 0 1

:

i

i

p

i

F

F L

F p

F

sparsest solution

solution

generalized inverse solution

Sparse solution: overcomplete case

( ) : *min β

orthogonal projection, dual

D β : β , F β = c dual geodesic

projec

projec

tion

tion

( )c cF

dual

a) : 2, 1cR n p b) : 2, 1cR n p

c) : 2, 1cR n p

Fig. 1

non-convex

L1-constrained optimizationLASSO

LARS

min under

solution : 0

0c

F c

c c

min

solution 0

0

F

, ,

: ,

* *c λsolutions β and β : coincide λ = λ c p 1

p < 1 λ = λ c multiple noncontinuous stability different

　λP Problem

　cP Problem

*

*

Projection from to F = c (information geometry)*β

n

F n

Fig. 5 subgradient

( )c cF

LASSO path and LARS path(stagewise solution)

min :

min

F c

F

,c c λ correspondence

Active set and gradient

1

0

sgn ,, ,

1,1

i

pi i

p

A i

i AF i A

Solution path

1

0,

;

c c

A c c A c c

A A c c A A c c A c

c cA

c

cK

F

F F

ddc

F

c c cK G F

1 1 10; (sgn ) : iF F L

Solution pathin the subspace of the active set

1

0 : active direction

A A A

A A

F

K F

turning point A A

Gradient Descent Method

{ ( )}: covariant

{ ( )}: contravariant

i

ji

i

L L

L g L

2min L( +a): i jijg a a

1 ( )t t tc L

ij i jG g a aa

0pG t t G a a

. 1piG a p a

0L G

a aa

Riemannian metric

Minkovskian metric

Lp-norm

Steepest direction of L

1sign pi iG p a a a

ii

f F

1

1sgn pi i ia c f f

1, ij i jG g a a F G f a Natural gradient

F f

1

1sgn pi iF c f f

0

1sgn

0

0

iF c f

Euclidean case

1

arg max ii f

max i i jf f f

1, for and ,0 otherwise.i

i i jF

1t t F LASSOTry for various p, p->1Try for various noise functionLASSO and flat geometry

Extended LARS (p = 1) and Minkovskian gradient

11

norm

max under 1

1

sgn , max , ,

0, otherwise

pip

p

p

i i NA

a

p

1

a

a a

a a

arg max ii f

max i i jf f f

1, for and ,0 otherwise.i

i i jF

1t t F LARS

c-trajectory and-trajectory

Ex. 1-dim 21 *2

21 2 22 f F

L1/2 constraint: non-convex optimization

2: min , cP c

c c

: 0P f 0

ˆ : Xu Zongben's operator R

c c c

0 c

R

c

An Example of the greedy path 

β１

β２

LASSO and LARS :

1 : is non-sparse1 : sparse

cpp

Minkovskian gradient

2

1

:

0

c

c c c

c c c

D F

F

F

c

1

c c c c c c c

c c c

F F

G H

G H F

p

Solution Path :

not continuous, not-monotonejump

c

c

Linear Programming

inner met

max

lo

ho

g

d

ij j i

i i

ij j ii

A x b

c x

A x b

x

Convex Cone Programming

P : positive semi-definite matrix

convex potential function

dual geodesic approach

, minA x b c x

Support vector machine

Convex Programming━ Inner Method

:LP A x b

min c x

log

ij j iA x b x

i x

Simplex method ; inner method

Barrier function

Polynomial-Time Algorithm

curvature : step-size 2mH

min : geodesict t c x x x

Integration of evidences:

arithmetic meangeometric meanharmonic mean‐mean

1 2, ,... mx x x

Generalized mean: f‐meanf(u): monotone; f‐representation of u

1 ( ) ( )( , ) { }2f

f a f bm a b f

12( ) , 1

log ,

( , )

1

( ,

)

=

f fm ca cb cm a

f u u

b

u

scale free

α-representation

‐mean : 

2

1 :

1: 2

10 : ( )4 2

min( , ) max( , )

aba b

a ba b ab

m a bm a b

1 2( ( ), ( ))m p s p s

Various Means

2a b

ab2

1 1a b:

arithmetic geometric

Any other mean?

:

harmonic

Family of Distributions 1( ), , ( )kp s p s

1( ) ( ), 1

k

mix i i ii

p s t p s t

explog ( ) log ( )i ip s t p s

mixture family :

exponential family :

1( ; ) { ( ( ))}i ip x f f p x

α‐geodesic projection

robust estimator