Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing
description
Transcript of Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 11
Mathematical Structures of Mathematical Structures of Belief Propagation Algorithms in Belief Propagation Algorithms in
Probabilistic Information ProcessingProbabilistic Information Processing
Kazuyuki TanakaKazuyuki Tanaka
Graduate School of Information Sciences, Graduate School of Information Sciences, Tohoku University, Sendai, JapanTohoku University, Sendai, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 22
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
National Tsin Hua University, Taiwan 3
Computational model for information processing in data with uncertainty
Probabilistic Inference
Probabilistic model with graphical structure ( Bayesian network )
Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness.
Inference system for data with uncertainty
modeling
Node is random variable.Arrow is conditional
probability.
Mathematical expression of uncertainty=>Probability and Statistics
Graph with cycles
Important aspect24 November, 2011
National Tsin Hua University, Taiwan 4
Computational Model for Probabilistic Information Processing
Probabilistic Information Processing Probabilistic Model
Bayes Formula
Algorithm
Monte Carlo MethodMarkov Chain Monte Carlo
MethodRandomized Algorithm
Approximate MethodBelief PropagationVariational Bayes MethodExpectation Propagation
Randomness and Approximation
24 November, 2011
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 55
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
24 November, 2011 National Tsin Hua University, Taiwan 6
Joint Probability and Conditional Probability
)()|(),( s.t.
PrPr,Pr
)|(Pr
,PrPr
bPabPbaP
aAaAbBbBaA
abPaA
bBaAaAbB
a
b
Conditional Probability of Event A=a when Event B=b has happened.
Probability of Event A=a )(}Pr{ aPaA Joint Probability of Events A=a and B=b
),()()(Pr,Pr baPbBaAbBaA
Random Variable
State Variable
Probability Distribution
Joint Probability Distribution
24 November, 2011 National Tsin Hua University, Taiwan 7
Joint Probability and Independency of Events
PrPr bBaAbB a
b
In this case, the conditional probability can be expressed as
Events A and B are independent of each other
bBaAbBaA PrPr,Pr
a
b
1Pr
1Pr0Pr
MAbB
AbBAbB
24 November, 2011 National Tsin Hua University, Taiwan 8
Marginal Probability
1
0
,PrPrM
a
bBaAbB
Let us suppose that the sample space is expressed by Ω= (A=0) (∪ A=1) … (∪ ∪ A=M1) where every pair of events is exclusive of each other.
Marginal Probability of Event B=b in Joint Probability Pr{A=a,B=b}
Marginalize
aa b
a
bBaAbB ,PrPrSimplified Notation
Summation over all the possible events in which every pair of events are exclusive of each other.
aa ba
=
Message
Graph with Two Nodes and One Edge
24 November, 2011 9
Marginal Probabiilty of High-Dimentional Joint Probabilty
Marginalization with respect to c and d
aa bb
cc dd
Hyperedge
a b
c da c d
Hypergraph
Message
c d
dDcCbBaAbBaA ,,,Pr,Pr
aa bb
cc dd
Hyperedge
aa bb
cc ddc d
Hypergraph
Message
National Tsin Hua University, Taiwan
a c d
dDcCbBaAbB ,,,PrPr
Marginalization with respect to a, c and d
24 November, 2011 National Tsin Hua University, Taiwan 10
Bayes Formulas
aA
aAbBbBaA
Pr
Pr,Pr
bB
aAaAbB
bB
bBaAbBaA
Pr
PrPr
Pr
,PrPr
a
b
bBbBaAbBaA PrPr,Pr
a
a
aAaAbB
bBaAbB
Pr|Pr
,PrPrPrior Probability
Posterior Probability
Marginal Likelihood
Bayesian Network
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 1111
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
National Tsin Hua University, Taiwan 12
Image Restoration by Probabilistic Model
Original Image
Degraded Image
Transmission
Noise
Likelihood Marginal
PriorLikelihood
Posterior
}ageDegradedImPr{
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.
Bayes Formula24 November, 2011
24 November, 2011 National Tsin Hua University, Taiwan 13
Prior Probability in Probabilistic Image Processing
EjiEjiji xxxX
},{},{
2
2
1expPr
xi xj
x1 x2 x3 x4
x5 x6 x7 x8
x9 x10 x11 x12
xi xj
2
2
1exp ji xx
}12,11,10,9,8,7,6,5,4,3,2,1{V
}12,8{},11,7{},10,6{},9,5{},8,4{},7,3{},6,2{},5,1{
}12,11{},11,10{},10,9{},8,7{},7,6{},6,5{},4,3{},3,2{},2,1{E
State Variable of Light Intensity at i-th Pixel in Original Image
xi
24 November, 2011 National Tsin Hua University, Taiwan 14
Additive White Gaussian Noise
ViVi
ii yxxXyY 222
1expPr
Conditional Probability of Degradation Process
X1 X2 X3 X4
X5 X6 X7 X8
X9 X10 X11 X12
y1 y2 y3 y4
y5 y6 y7 y8
y9 y10 y11 y12
xi
222
1exp ii yx
State Variable of Light Intensity at i-th Pixel in Original Imagexi
yi
State Variable of Light Intensity at i-th Pixel in
Original Imageyi
xi yi
24 November, 2011 National Tsin Hua University, Taiwan 15
Bayesian Image Analysis
Eji
jiVi
ii xxyx
yY
xXxXyYyYxX
},{
222 2
1
2
1exp
Pr
PrPrPr
x
g
xX Pr xXyY
Pr y
Original Image
Degraded Image
Prior Probability
Posterior Probability
Degradation Process
Image Processing is reduced to computations of avereages, variance at each pixel and covariances of each pair of neghbouring pixels
24 November, 2011 National Tsin Hua University, Taiwan 16
Statistical Estimation of Hyperparameters
z
z
zXzXyY
yYzXyY
}|Pr{},|Pr{
},|,Pr{},|Pr{
},|Pr{max arg)ˆ,ˆ(
,
yY
x
g
Marginalized with respect to X
}|Pr{ xX
},|Pr{ xXyY
yOriginal Image
Marginal Likelihood
Degraded ImageV
},|Pr{ yY
Hyperparameters are determinedso as to maximize the marginal likelihood Pr{Y=y|,} with respect to ,
EM (Expectation Maximization) Algorithm
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 1717
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
24 November, 2011 National Tsin Hua University, Taiwan 18
Gaussian Graphical Model(Gauss Markov Random Fields)
xxyx
xxyx
yxP
Ejiji
Viii
CT22
},{
222
2
1
2
1exp
2
1
2
1exp
,,|
yyyP
V
CI
C
CI
C2
T2 2
1exp
det2
det,
yxdyxPxx 12)|(ˆ
CI
Multidimensional Gauss Integral Formulas
,max argˆ,ˆ,
gP
Maximum Likelihood Estimation EM Algorithm
),( ix
otherwise,0
},{,1
,4
Eji
Vji
ji C
24 November, 2011 National Tsin Hua University, Taiwan 19
One-Dimensional Signal Processing
EM Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
ix
iy
ix̂
Original Signal
Degraded Signal
Estimated Signal
40
.,,maxarg
1,1
,ttQ
tt
24 November, 2011 National Tsin Hua University, Taiwan 20
Bayesian Image Analysis by Gaussian Graphical Model
0
0.0002
0.0004
0.0006
0.0008
0.001
0 20 40 60 80 100 t
ty
f̂
ytttx 12 ))()(()(ˆ CI
Iteration Procedure of EM algorithm in Gaussian Graphical Model
EM
f̂
y
),|(max arg)ˆ,ˆ(
,
gP
40
.,,,maxarg1,1,
yttQtt
0007130ˆ
624.37ˆ
.
24 November, 2011 National Tsin Hua University, Taiwan 21
Image Restoration by Gaussian Graphical Model and Conventional Filters
2ˆ||
1MSE
Viii ff
V
MSE
Gaussian Graphical Model 315
Lowpass Filter(3x3) 388
(5x5) 413
Median Filter(3x3) 486
(5x5) 445
(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianGaussian Gaussian Graphical Graphical
ModelModel
Original ImageOriginal ImageDegraded Degraded Image (Image (=40)=40)
V:Set of all the pixels
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 2222
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
National Tsin Hua University, Taiwan 23
What is an important point in What is an important point in computational complexity? computational complexity?
How should we treat the calculation of the summation over 2N configuration?
1
0
1
0
1
021
1 2
,,,x x x
N
N
xxxf
}
}
}
;,,,
1){or 0for(
1){or 0for(
1){or 0for(
;0
21
2
1
N
N
xxxfaa
x
x
x
a
N fold loops
If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.
Markov Chain Monte Carlo MethodBelief Propagation Method
This Talk
24 November, 2011
National Tsin Hua University, Taiwan 24
Strategy of Approximate Algorithm in Probabilistic Information Processing
It is very hard to compute marginal probabilities exactly except some tractable cases.
What is the tractable cases in which marginal probabilities can be computed exactly?
Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?
2 3 4
,,,,, 432111x x x x
N
N
xxxxxPxP
3 4
,,,,,, 43212112x x x
NN
xxxxxPxxP
24 November, 2011
24 November, 2011 National Tsin Hua University, Taiwan 25
Graphical Representations of Tractable Models
24 November, 2011 National Tsin Hua University, Taiwan 26
Graphical Representations of Tractable Models
a b c d ea b c d e
a ba b c d e
b c d eX
a b
b c d e a
b c d e
a b
a b c d eb c d e
24 November, 2011 National Tsin Hua University, Taiwan 27
Graphical Representations of Tractable Models
b c d e
c d eX
c d e b
c d e
b c
a b c d eb c d e
a b c
a b c
b c d ec d e
X
24 November, 2011 National Tsin Hua University, Taiwan 28
Graphical Representations of Tractable Models
a b c d eb c d e
b c d ec d e
a b c d ea b c d e
c d ed e
d ee
24 November, 2011 National Tsin Hua University, Taiwan 29
Graphical Representations of Tractable Models
a
b
eca b c d e f
d
f
a
b
ec
c d e f
d
fa
b
ec
b c d e f
d
f
ecd e f
d
f
ece f
d
f
ef f
24 November, 2011 National Tsin Hua University, Taiwan 30
Belief Propagation for Probabilistic Model on Square Grid Graph
E: Set of all the links
Eji
jijiL xxxxxPxP},{
},{21 ,,,,
24 November, 2011 National Tsin Hua University, Taiwan 31
Marginal Probability
1 3 4x x x xN
2 2
1 3 4
,,,,, 432122x x x x
NN
xxxxxPxP
24 November, 2011 National Tsin Hua University, Taiwan 32
Marginal Probability
3 4x x xN
3 4
,,,,,, 43212112x x x
NN
xxxxxPxxP
1 2 1 2
24 November, 2011 National Tsin Hua University, Taiwan 33
Belief Propagation
1
211222 ,x
xxPxP
Message Update Rule
3
21
5
41x
1 2
33 88
144
5
2
6
77
88
1 2
6
77
24 November, 2011 National Tsin Hua University, Taiwan 34
Belief Propagation on Graph with Cycles
MM
Simultaneous Fixed Point Equations of Messages
3
4 1 2
5
Average, variances and covariances can be expressed Average, variances and covariances can be expressed in terms of messages.in terms of messages.
13M
14M15M
21M3
21
5
41x
1 2
24 November, 2011 National Tsin Hua University, Taiwan 35
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
24 November, 2011 National Tsin Hua University, Taiwan 36
Fundamental Structures of Belief Propagation in Probabilistic Image Processing
Three Inputs and One Output Message Passing Rules
MM
24 November, 2011 National Tsin Hua University, Taiwan 37
Belief Propagation and EM Algorithm
Input
Output
BP EM
Update Rule of BP
21
3
4
5
24 November, 2011 National Tsin Hua University, Taiwan 38
Maximization of Marginal Likelihood by EM Algorithm
.,,,maxarg1,1,
gttQtt
y
ymx
,ˆ,ˆˆ
0
0.0002
0.0004
0.0006
0.0008
0.001
0 20 40 60 80 100
Loopy Belief Propagation
Exact
0006000ˆ
335.36ˆ
.
LBP
LBP
0007130ˆ
624.37ˆ
.
Exact
Exact
24 November, 2011 39National Tsin Hua University, Taiwan
Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model
Original ImageOriginal Image
MSE:315MSE:315
MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411
MSE: 1512MSE: 1512
Degraded ImageDegraded Image
Lowpass FilterLowpass Filter Median FilterMedian Filter
Exact
Wiener Filter
2ˆ|V|
1MSE
Viii xx
Belief PropagationBelief Propagation
MSE:325MSE:325
24 November, 2011 National Tsin Hua University, Taiwan 40
Digital Images Inpainting based on MRF
Inpu
t
Ou
tpu
t
MarkovRandomFields
M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings ofCIMCA&IAWTIC2005.
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 4141
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing7.7. SummarySummary
24 November, 2011 National Tsin Hua University, Taiwan 42
Belief Propagation for Bayesian Networks
24 November, 2011 National Tsin Hua University, Taiwan 43
Factor Graph Representations of Bayesian Networks and Belief Propagations
National Tsin Hua University, Taiwan 44
Error Correcting Code
Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004.
High Performance Decoding Algorithm
010 000001111100000 001001011100001
0 1 0
code
010
error
decode
Parity Check Code
Turbo Code, Low Density Parity Check (LDPC) Code
majority ruleError Correcting Codes
24 November, 2011
24 November, 2011 National Tsin Hua University, Taiwan 45
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
6439
5328
3217
XXXX
XXXX
XXXX
1
1
0
0
1
1
6
5
4
3
2
1
x
x
x
x
x
x
1
0
0
1
1
0
0
1
1
9
8
7
6
5
4
3
2
1
x
x
x
x
x
x
x
x
x
1
0
1
1
1
0
1
1
1
9
8
7
6
5
4
3
2
1
y
y
y
y
y
y
y
y
y
1)2 (mod 100
0)2 (mod 101
0)2 (mod 011
9
8
7
X
X
X
1 1
0
p1
1
0
0
p1
p
p
Received Word
Code Word
Binary Symmetric Channel
24 November, 2011 National Tsin Hua University, Taiwan 46
Error Correcting Codes and Belief Propagation
)2 (mod
)2 (mod
)2 (mod
5329
6438
4217
XXXX
XXXX
XXXX
Fundamental Concept for Turbo Codes and LDPC CodesFundamental Concept for Turbo Codes and LDPC Codes
24 November, 2011 National Tsin Hua University, Taiwan 47
Satisfactory Problem (3-SAT)
985872
642653431
XXXXXX
XXXXXXXXX
),,(),,(
),,(),,(),,(1
},,Pr{
985}9,8,5{872}8,7,2{
642}6,4,2{653}6,5,3{431}4,3,1{6611
xxxfxxxf
xxxfxxxfxxxfZ
xXxX
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 4848
ContentsContents
1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Statistical Performance AnalysisStatistical Performance Analysis7.7. Various Applications of Probabilistic Various Applications of Probabilistic
Information ProcessingInformation Processing8.8. SummarySummary
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 4949
SummarySummary
Fundamental Structures of Bayesian modeling Fundamental Structures of Bayesian modeling have been introduced.have been introduced.Formulation of probabilistic image processing Formulation of probabilistic image processing algorithms by means of loopy belief propagation algorithms by means of loopy belief propagation has been summarized.has been summarized.Various applications of Bayesian Network Systems Various applications of Bayesian Network Systems have been reviewed.have been reviewed.
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 5050
ReferenceReferencess1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM
Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.
2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.
3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008
4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009
5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.
6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.
24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 5151
TextbooksTextbooks
Kazuyuki Tanaka: Introduction of Image Processing Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese).Ltd., 2006 (in Japanese).
Kazuyuki Tanaka: Mathematics of Statistical Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).Co., Ltd., 2009 (in Japanese).