Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 12th Bayesian network and belief...

Physics Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics12th Bayesian network and belief propagation in

statistical inference

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese).


Joint Probability and Conditional Probability

aAaAbBbBaA

aA

bBaAaAbB

PrPr,Pr

Pr

,PrPr

A

B

Conditional Probability of Event B=b when Event A=a has happened.

Probability of Event A=a }Pr{ aAJoint Probability of Events A=a and B=b

)()(Pr,Pr bBaAbBaA

Fundamental Probabilistic Theory for Image Processing



a c d

dDcCbBaAbB ,,,Pr}Pr{

Marginal Probability of Event B

A B

C D

Marginalization



},Pr{

},,Pr{},|Pr{

bBaA

cCbBaAbBaAcC

Causal Independence

}|Pr{},|Pr{ bBcCbBaAcC

C

A B

C

A B

)F,F(),T,F(),F,T(),T,T(),(

}F|Pr{}T|Pr{

cb

b,BAcCb,BAcC



},,Pr{

},,,Pr{

},,|Pr{

cCbBaA

dDcCbBaA

cCbBaAdD

Causal Independence

)F,F(),T,F(),F,T(),T,T(),(

}FF|Pr{}TF|Pr{

}FT|Pr{},TT|Pr{

dc

c,C,BAdDc,C,BAdD

c,C,BAdDcC,BAdD

}|Pr{},,|Pr{ cCdDcCbBaAdD

D

A B C

D

A B C



aAaAbBbBcC

aAaAbBbBaAcC

bBaAcBbAcCcCbBaA

PrPrPr

PrPr,Pr

,Pr,Pr,,Pr

a b

b

cCbBaA

cCbBaA

cC

cCaA

cCaA

,,Pr

,,Pr

Pr

,Pr

PrA

B

C

}|Pr{ aAbB

bBcC

bBaAcC

Pr

,Pr

Causal Independence



),(),(

PrPrPr,,Pr

},{},{ cbfbaf

aAaAbBbBcCcCbBaA

CBBA

A

B

C

aAaAbBbaf

bBcCcbf

BA

CB

PrPr),(

Pr),(

},{

},{

A

B

C

Directed Graph Undirected Graph


Simple Example of Bayesian Networks

? TTPrTTPr WSWR

AAAA



CC

CCSS

CCRR

RRSSWW

CCCCSS

SSCCRR

RRSSCCWW

SSCCSSCCRR

RRSSCCWW

RRSSCC

RRSSCCWW

WWRRSSCC

Pr

Pr

Pr

,Pr

PrPr

,Pr

,,Pr

,Pr,Pr

,,Pr

,,Pr

,,Pr

,,,Pr

aA

aAaA

aAaA

aAaAaA

aAaAaA

aAaAaA

aAaAaAaA

aAaAaAaAaA

aAaAaAaA

aAaAaA

aAaAaAaA

aAaAaAaA



TPr

TT,PrTTPr

W

WRWR

A

AAAA

),(),(

PrPrPr,,Pr

},{},{ cbfbaf

aAaAbBbBcCcCbBaA

CBBA



2781.0T,,T,PrT,TPrFT, FT,

WRRSCCWS

C R

a a

AaAAaAAA

6471.0T,,,PrTPrFT, FT, FT,

WRRSSCCW

C S R

a a a

AaAaAaAA

4581.0T,T,,PrT,TPrFT, FT,

WRCSCCWR

C S

a a

AAaAaAAA



7079.0

6471.0

4581.0

TPr

TT,PrTTPr

W

WRWR

A

AAAA

4298.0

6471.0

2781.0

TPr

TT,PrTTPr

W

WSWS

A

AAAA



),(),(),,(

PrPr

Pr,Pr

,,,Pr

CS}CS,{CR}CR,{RSW}RS,,{

CCCCSS

CCRRRRSSWW

WWRRSSCC

aafaafaaaf

aAaAaA

aAaAaAaAaA

aAaAaAaA

W

C

S R

W

CCCCSSCS}CS,{

CCRRCR}CR,{

RRSSWWRSW}RS,,{

PrPr),(

Pr),(

,Pr),,(

aAaAaAaaf

aAaAaaf

aAaAaAaaaf W

C

S R

W

Directed Graph

Undirected Graph


Belief Propagation for Bayesian Networks

Belief propagation cannot give us exact computations in Bayesian networks on cycle graphs.Applications of belief propagation to Bayesian networks on cycle graphs provide us many powerful approximate computational models and practical algorithms for probabilistic information processing.



22111133

22442255

443366

6677

665588

882211

PrPrPr

PrPr

,Pr

XPr

,Pr

,,,Pr

xXxXxXxX

xXxXxXxX

xXxXxX

xXx

xXxXxX

xXxXxX

),(),(),(

),(),,(),,(x

,,,Pr

311342245225

7667543346865568

882211

xxWxxWxxW

xxWxxxWxxW

xXxXxX


Joint Probability of Probabilistic Model with Graphical Representation including Cycles

),(),(),(

),(),,(),,(x

,,,Pr

311342245225

7667543346865568

882211

xxWxxWxxW

xxWxxxWxxW

xXxXxX

Directed Graph

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87 Undirected Hypergraph


Marginal Probability Distributions

ix

iiii

xXxXxX

xPxX

\882211 ,,,Pr

)(}Pr{

x

kji xxx

kjiijkkkjjii

xXxXxX

xxxPxXxXxX

,,\882211 ,,,Pr

),,(},,Pr{

x

ji xx

jiijjjii

xXxXxX

xxPxXxX

,\882211 ,,,Pr

),(},Pr{

x

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Approximate Representations of Marginal Probability Distributions in terms of Messages

3 4 6)()(

)()(),,(

)()(

)()(),,(

),,(

656866676

42443133643346

656866676

42443133643346

643346

x x x xMxM

xMxMxxxW

xMxM

xMxMxxxW

xxxP

1

3

2

4

6 5

87

346W

)( 3133 xM )( 4244 xM

)( 6676 xM )( 65686 xM

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Approximate Representations of Marginal Probability Distributions in terms of Messages

6

66766568663466

6676656866346666

x

xMxMxM

xMxMxMxP

3 4

6 5

87

)( 6676 xM

)( 65686 xM

)( 63466 xM 1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87

3 4

64334666 ,,x x

xxxPxP


Basic Strategies of Belief Propagations in Probabilistic Model with Graphical Representation including Cycles

1

3

2

4

6 5

87

346W

)( 3133 xM )( 4244 xM

)( 6676 xM

)( 65686 xM

3 4

6 5

87

)( 6676 xM

)( 6586 xM

)( 6346 xM

667665686

42443133

643346346

643346

,,1

,,

xMxM

xMxM

xxxWZ

xxxP

667665686

634666

66

1

xMxM

xMZ

xP

Approximate Expressions of Marginal Probabilities

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Simultaneous Fixed Pint Equations for Belief Propagations in

Hypergraph Representations

3 4 6

3 4

)()(),,(

)()(),,(

)(42443133643346

42443133643346

63466

x x x

x x

xMxMxxxW

xMxMxxxW

xM

Belief Propagation Algorithm

6

1

3

2

4)( 3133 xM )( 4244 xM

)( 63466 xM

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Fixed Point Equation and Iterative Method

Fixed Point Equation ** MM

Iterative Method

23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M


Belief Propagation for Bayesian Networks

Belief propagation can be applied to Bayesian networks also on hypergraphs as powerful approximate algorithms.


Numerical Experiments

4393.0)0(5607.0)1( 88 PP

4360.0)0(5640.0)1( 88 PP

Belief Propagation

Exact

8\88221188 },,,Pr{)(

x

xXxXxXxPx

3629.0)0,0( 0871.0)1,0(

0764.0)0,1( 4736.0)1,1(

5858

5858

PP

PP

3636.0)0,0( 0864.0)1,0(

0724.0)0,1( 4776.0)1,1(

5858

5858

PP

PP

85 ,\8822118558 },,,Pr{),(

xx

xXxXxXxxPx

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87



8261.04393.0

3629.0

PresentPr

Present,PresentPr

PresentPresentPr

Dyspnea

DyspneaBronchitis

DyspneaBronchitis

X

XX

XX

Belief PropagationBelief Propagation

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Linear Response Theory

}|{ Ωx ixi

ij

PPnPnPnPjj xnxnjjj

i

Nodeat Field External respect to with Nodeat Average ofDeviation

)()(~

)()(~

)( ,,)(

xx

xx

i

ji

h

jiij

h

nP

nPmPnmP

i

)(lim

)()(),(

)(

0

mxi ihP

ZP ,exp)(~

1)(

~ xx

3

1 2

4

6 5

87



0187.0

4393.0

0082.0

PresentPr

Present,PresentPr

PresentPresentPr

Dyspnea

DyspneaisTuberculos

DyspneaisTuberculos

X

XX

XX

1

3

2

4

6 5

13W

67W

24W

25W346W

568W

87


Summary

Bayesian Network for Probabilistic InferenceBelief Propagation for Bayesian Networks


Practice 11-1

Compute the exact values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8), numerically, in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:

Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in

Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.

21

132425

43667658

821

PrPr

PrPrPr

,PrXPr,Pr

,,,Pr

XX

XXXXXX

XXXXXXX

XXX


Practice 11-2

Each conditional probability table and probability table is given in Figure 3.12 and Table 3.11 in

Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009.The algorithm has appeared explicitly in the above textbook.

Make a program to compute the approximate values of the marginal probability Pr{Xi} for every nodes i(=1,2,…,8) by using the belief propagation method in the Bayesian network defined by the joint probability distribution Pr{X1,X2,…,X8} as follows:

21132425

43667658

821

PrPrPrPrPr

,PrXPr,Pr

,,,Pr

XXXXXXXX

XXXXXXX

XXX

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 12th Bayesian network and belief...

Documents

Transcript of Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 12th Bayesian network and belief...