Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation...

Physics Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics7th~10th Belief propagation

Appendix

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.

ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.


Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for

Ferromagnetic Materials

p p

p p

)1,1()1,1()1.1()1.1( PPPP

pPP )1.1()1,1(

11 a

1

12 a

1

11

1 1

p

PP

2

1

)1.1()1,1(


Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for

Ferromagnetic Materials

Prior probability prefers to the configuration with the least number of red lines.

> >=

Lines Red of #Lines Blue of # )2

1()( ppaP

p p

11 a 112 a 111 1 1


More is different in Probabilistic Model for Ferromagnetic Materials

Disordered State

Ordered State

Sampling by Markov Chain Monte Carlo method

p p

Small p Large p

p p

More is different.

p2

1p

2

1

Critical Point(Large fluctuation)


Fundamental Probabilistic Models for Magnetic Materials

Since h is positive, the probablity of up spin is larger than the one of down spin ．

1

)exp(

)exp()(

a

ha

haaP

1a

+1 -1

he he

)tanh()(1

haaPma

h ： External Field

)(tanh1)()( 2

1

2 haPmaaVa

Variance

Average

0h



Since J is positive, (a1,a2)=(+1,+1) and (-1,-1) have the largest probability ．

1 121

2121

1 2

)exp(

)exp(),(

a a

aJa

aJaaaP

11 a

0),(1 1

2111

1 2

a a

aaPam

J ： Interaction

1),()(1 1

212

111

1 2

a a

aaPmaaVVariance

Average

0J

Je Je

+1 +1 -1 -1

+1 +1 -1-1

12 aJe Je



a

aEZ

))(exp(

Eji

jiVi

i aaJahaE},{

)(

Translational Symmetry

),( EVJ

J

h h

)(exp1

)( aEZ

aP

),,,( 21 Naaaa

E ： Set of All the neighbouring Pairs of Nodes

1ia 1ia

N

i ai aPa

Nm

1

)(1

Problem: Compute

)'()()'()( aPaPaEaE



Eji

jiVi

i aaJahaE},{

)(

N

i ai

NhaPa

Nm

10)(

1limlim

)(exp1

)( aEZ

aP

),,,( 21 Naaaa

1ia

Problem: Compute


),( EV

J

J

h h

Spontaneous Magnetization


Mean Field Approximation for Ising Model

)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying

Vi

iaJmhaE )4()(

2mmamaaa ijji

Eji

jiVi

i aaJahaE},{

iJm

Jm

JmJm

h

are large.


Mean Field Approximation for Ising Model

)4tanh()(1

1

JmhaPaN

mN

i ai

Vi

ii aPaEZ

aP )())(exp(1

)(

Fixed Point Equation of m)(mm

We assume that all random variables ai are independent of each other, approximately.

Vi

iaJmhaE )4()(


Fixed Point Equation and Iterative Method

• Fixed Point Equation ** MM



• Fixed Point Equation ** MM • Iterative Method

0

xy

)(xy

y

x*M




0M0

xy

)(xy

y

x*M




01 MM

0M

1M

0

xy

)(xy

y

x*M




12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M




12

01

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M




23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M


Marginal Probability Distribution in Mean Field Approximation

))4exp((1

)()(

1 2 1 1

ii

a a a a aii

aJmhZ

aPaP

i i N

i

JmJm

JmJm

h

1

)(

iaiii aPam

))4tanh(( mJhm Jm： Mean Field


Advanced Mean Field Method

))4exp((1

)( ii

ii ahZ

aP

)))(3exp((1

),( jijii

jiij aJaaahZ

aaP

l

h

ll

h

h

1

),()(

jajiijii aaPaP

))3tanh()(tanh(arctanh hJ

Bethe Approximation

Kikuchi Method (Cluster Variation Meth)

l ： Effective Field

Fixed Point Equation for l

J


Average of Ising Model on Square Grid Graph

(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution （ L. Onsager ， C.N.Yang ）

J/1

a

iNh

aPa

)(limlim

0

Ejiji

Vii aaJah

ZaP

},{

exp1 ),( EVJ

J

h h


Model Representation in Statistical Physics

),,,(},,,Pr{ 212211 NNN aaaPaAaAaA

a

aEZ

))(exp(

)(}Pr{ aPaA

))(exp(1

)( aEZ

aP

),,,( 21 NAAAA

Gibbs Distribution Partition Function

)))(exp(ln(ln a

aEZF

Free Energy

Energy Function


Gibbs Distribution and Free Energy

Gibbs Distribution

ZPFaQQFaQ

ln][}1)(|][{min

))(exp(1

)( aEZ

aP

)(ln)()()(][ aQaQaQaEQFaa

Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a)

Free Energy Functional of Trial Probability Distribution Q(a)

a

aEZ

))(exp(lnlnFree Energy


Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional

ZPFaQQFaQ

ln][}1)(|][{min

)(

)(exp

)(exp)(ˆ aP

aE

aEaQ

a

1)()())(ln)((1)(

aaa

aQaQaQaEaQQFQL

01)(ln)(

)(

aQaEaQ

QL

1)(exp)(ˆ aEaQ

Normalization Condition


Kullback-Leibler Divergence and Free Energy

0)(

)(ln)(

aP

aQaQPQD

a

a

aQaQ

1)( ,0)(

ZQF

ZaQaQaEaQPQD

QF

aa

ln][

ln)(ln)()()(]|[

][

0)()( PQDaPaQ

))(exp(1

)( aEZ

aP

}1)(|]|[{minarg}1)(|][{minarg aQaQ

aQPQDaQQF


Interpretation of Mean Field Approximation as Information Theory

Vi

ii aQaQ )()(

)(

)(ln)(

aP

aQaQPQD

a

))(exp(1

aEZ

aP

and

Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aQaQaQ

Minimization of Kullback-Leibler Divergence between


Interpretation of Mean Field Approximation as Information Theory

Eji

jiVi

i aaJahaE},{

)(

Vi a

iiVi a

i

i

aPaV

aPaV

m1

)(||

1)(

||

1

)(exp1

)( aEZ

aP

),,,( ||21 Vaaaa

1ia

Problem: Compute


),( EV

J

J

h h

Magnetization

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aPaPaP


Kullback-Leibler Divergence in Mean Field Approximation for Ising Model

Vi

ii aQaQ )()(

ZViQFPQD i ln|MF

)(

)(ln)(

aP

aQaQPQD

a

Viii

Ejiji

Viii

QQQQJ

QhViQF

1},{ 11

1MF

ln))()()((

)(}]|[{

1 2 1 1 2

)(

)()(\

a a a a a a

aaii

i i i N

i

aQ

aQaQ

Eji

jiVi

i aaJahaE},{

)( )(exp

1)( aEZ

aP


Minimization of Kullback-Leibler Divergence and Mean Field Equation

)( ))(ˆ(exp1ˆ

1

ViQJhZ

Qij

ji

i

} ,1)(|]|[{minarg)}(ˆ{}{

ViQPQDQ iQ

ii

Fixed Point Equations for {Qi|iV}

Variation

1 1

))(ˆ(exp

ij

ji QJhZ

i

Set of all the neighbouring nodes of the node i

Ejiji },{


Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model

iiii amaQ2

1

2

1)(

1

)(

iaiii

aii aQaaQam

),,,( 21 Naaaa 1ia

ia

iiia

iia

iii

aii

ai

aii

iiii

maQadddacaaQa

aQccdacaQ

adacaQ

iii

iii

2

1)(

2

12)()(

2

1)(

2

12)()(

1)( )(

111

111

2

1 2 1 1

)()()(\ a a a a aaa

ii

i i Ni

aQaQaQ


Conventional Mean Field Equation in Ising Model

)4tanh( Jmhm

iiiiiaa

ii maamaQaPaPi

2

1

2

1

2

1

2

1)(ˆ)()(

\

maPaN

N

i ai

1

)(1

Fixed Point Equation

mmmm N 21

Eji

jiVi

i aaJahaE},{

)(

))4exp((1

))(ˆ(exp1

)(ˆ1

ii

iij

ji

ii aJmhZ

aQJhZ

aQ

VJ

J


h h

)( 4|| Vii


Interpretation of Bethe Approximation (1)

Eji

jiVi

i aaJahaE},{

)(

)(exp1

)( aEZ

aP

),,,( ||21 Vaaaa

1ia


),( EV

J

J

h h

1 2 1 1 2

)()()(\ a a a a a aaa

ii

i i i Ni

aPaPaP

1 2 1 1 2 1 1 2

)()(),(},\{ a a a a a a a a aaaa

jiij

i i i j j j Nji

aPaPaaP

Eji

jiij aaZ

aP},{

),(1

)(

jijijiij aJaha

jha

iaa

||

1

||

1exp),(

a Eji

jiij aaZ

},{

),(

Compute

and


ZQFPQD ln

aQaQaaWaaQ

aQaQaaWaQ

aQaQaaWaQQF

aEji a ajiijjiij

aEji a ajiij

aaa

aEjijiij

a

i j

i j ji

ln)(,ln,

ln)(,ln)(

ln)(,ln)(

},{

},{ ,\

},{

0ln)(

aP

aQaQPQD

a

Free EnergyKL Divergence

Eji

jiij aaWZ

P},{

,1

x

ji aaa

jiij

aQ

aaQ

,

)(

),(

\

33Physics Fluctuomatics (Tohoku

University)


ZQFPQD ln

Ejijjiiijij

Viii

Ejiijij

a

Ejiijij

QQQQQQ

QQ

WQ

aQaQ

WQQF

},{

},{

},{

lnln,ln,

ln

,ln,

ln)(

,ln,

Bethe Free

Energy

Free EnergyKL Divergence

Eji

jiij aaWZ

aP},{

,1

ji aaajiij aQaaQ

,

)(),(\

iaaii aQaQ

\

)()(


University)


FPQDQQ

minargminarg

,iji QQ

ZQQFPQD iji ln,Bethe

ijiQQQ

QQFPQDiji

,minargminarg Bethe,

1,

iji QQ

Ejijjiiijij

Viii

Ejiijijiji

QQQQQQ

QQWQQQF

},{

},{Bethe

lnln,ln,

ln,ln,,


University)


Ejiijij

Viii

Vi ijijijii

ijiiji

QQ

QQ

QQFQQL

},{

},{,

BetheBethe

1,1

,

,,

1, ,,,minarg Bethe,

ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints


University)


Ejiijij

Viii

Vi ijijijji

Ejijjiiijij

Viii

Ejiijij

Ejiijij

Viii

Vi ijijijiiijiiji

QQQQ

QQQQQQ

QQQ

QQ

QQQQFQQL

},{},{,

},{

},{

},{

},{,BetheBethe

1,1,

lnln,ln,

ln,ln,

1,1

,,,

0,Bethe

iji

ii

QQLxQ

• Extremum Condition

0,, Bethe

ijijiij

QQLxxQ


University)


FGfP yxyxyx g ,,,

ik

ikiiii ai

aQ )(1||

1exp },{, )()(exp,, 2}2,1{,21}2,1{,121122112 aaaaaaQ

Extremum

Condition 0,Bethe

iji

ii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ


University)

115114

1131121

11

1

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaaW

aMaMaMZ

aaQ

)()(exp\

},{, ijik

ikijii aMa


FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

115114

1131121

111

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaaW

aMaMaMZ

aaQ

Extremum

Condition 0,Bethe

iji

ii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ


University)

26M

14

5

13M

14M

15M

12W3

2

6

27M

8

7

28M


FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

412W

1

5

13M

14M

15M

3

26M

2

6

27M

8

7

28M

,121 QQ

115114

1131121

11

1

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaa

aMaMaMZ

aaQ

1514

1312

21

,

MM

M

M

Message Update Rule


University)


15141312

15141312

21 ,

,

MMMW

MMMW

M

1

3

4 2

5

13M

14M

15M

21M

14

5

3

2

6

8

7

2a

14 2

5

3

=

Message Passing Rule of Belief Propagation

It corresponds to Bethe approximation in the statistical mechanics.


University)


1 1 \

1 \

,

,

jikikij

jikikij

ji MW

MW

M


University)

))tanh()(tanh(arctanh\

jik

ikji hJ

jijiM exp

))3tanh()(tanh(arctanh hJ

ji Translational Symmetry


Summary

Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation...

Documents

Transcript of Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation...