Statistical Learning Procedure in Loopy Belief Propagation for Probabilistic Image Processing

30 November, 200530 November, 2005 CIMCA2005, ViennaCIMCA2005, Vienna 11

Statistical Learning ProcedureStatistical Learning Procedurein Loopy Belief Propagationin Loopy Belief Propagation

for Probabilistic Image Processingfor Probabilistic Image Processing

Kazuyuki TanakaKazuyuki TanakaGraduate School of Information Sciences,Graduate School of Information Sciences,

Tohoku UniversityTohoku [email protected]@smapip.is.tohoku.ac.jp

http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/

ReferencesReferencesK. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).


Bayesian Network and Bayesian Network and Belief PropagationBelief Propagation

Probabilistic Information Processing

Probabilistic Model

Bayes Formula

Belief Propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).

Bayesian Network

Graphical Model


How should we treat the calculation of the How should we treat the calculation of the summation over 256summation over 256NN configurations? configurations?

255

0

255

0

255

021

1 2

,,,x x x

N

N

xxxW

It is very hard to calculate it exactly except soIt is very hard to calculate it exactly except some special cases.me special cases.

Formulation for approximate algorithmAccuracy of the approximate algorithm

Belief PropagationBelief Propagation

30 November, 2005 CIMCA2005, Vienna 4

ContentsContents

1. Introduction

2. Bayesian Image Analysis and Gaussian Graphical Model

3. Belief Propagation

4. Concluding Remarks

30 November, 2005

CIMCA2005, Vienna 5

Bayesian Image Analysis

Original Image Degraded Image

Transmission

Noise

Likelihood Marginal

yProbabilit PrioriA Processn Degradatio

yProbabilit PosterioriA Image Degraded

Image OriginalImage OriginalImage DegradedImage DegradedImage Original

Pr

PrPr Pr

30 November, 2005

CIMCA2005, Vienna 6

Bayesian Image AnalysisDegradation Process ,, ii gf

iii gffgP 2

22

1exp

2

1,

2,0~ Nn

nfg

i

iii


Transmission

Additive White Gaussian Noise

30 November, 2005

CIMCA2005, Vienna 7


A Priori Probability ,if

Nijji ff

ZfP 2

PR 2

1exp

1

Standard Images

Generate

Similar?

30 November, 2005

CIMCA2005, Vienna 8


,, ji gf

Nij

jii

ii ffgf

gZ

gP

fPfgPgfP

22

2

POS

2

1

2

1exp

,,

1

,

,,,

A Posteriori Probability

Gaussian Graphical Model

30 November, 2005

CIMCA2005, Vienna 9


y

x

Lffff ,,, 21

Lgggg ,,, 21

f

g

fP ,fgP

g


,

,,,

gP

fPfgPgfP

iiiii dfgfPffdgfPff

,,,,ˆ

A Priori Probability

A Posteriori Probability

Degraded Image

Pixels

30 November, 2005

CIMCA2005, Vienna 10

Hyperparameter Determination by Maximization of Marginal Likelihood

fdfPfgPfdg,fPgP

,,,

,max argˆ,ˆ,

gP

MarginalizationMarginalization

g ,gP

Original Image

Marginal Likelihood

Degraded Image

y

x

fdσ,α,gfPff ii

ˆˆˆ

f

g fP ,fgP

g

fPfgPgfP

,,,

Lffff ,,, 21

Lgggg ,,, 21

30 November, 2005


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fdαfP,σfgPα,σgP

Marginal Likelihood

Incomplete Data

y

x

fdg,fPgfPgQ

','ln,,,,','

0,,',''

,0,,','' ','','

gQgQ

0, ,0,

gPgP

Equivalent

Q-Function

Lgggg ,,, 21

30 November, 2005


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fdαfP,σfgPα,σgP

Marginal Likelihood

y

x

fdα',σ'g,fP,αα,gfPgQ

ln,,',' Q-Function

.,','maxarg1,1 :Step-M

.','ln,,,',' :Step-E

','ttQtt

fdg,fPttgfPttQ

Iterate the following EM-steps until convergence:

EM Algorithm

A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data

via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).


ContentsContents

1. Introduction




30 November, 2005


Probabilistic Model on a Graph with Loops

Nij

jiijL xxWZ

xxxP ,1

,,, 21

2 3 4

,,,,, 432111x x x x

L

L

xxxxxPxP

Marginal Probability

1 2

,x x x Nij

jiij

L

xxWZ

3 4

,,,,,, 43212112x x x

L

L

xxxxxPxxP

30 November, 2005


Belief Propagation

27M

144 2

5

13M

14M

15M

12M

33

26M

144

5

13M

14M

15M

12W33

2

6

27M

88

77

28M

,121 PP

115114

1131121

11

1

xMxM

xMxMZ

xP

2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxP

1514

1312

21

,

MM

MW

M

Message Update Rule

30 November, 2005


Message Passing Rule of Belief Propagation

Fixed Point Equations for Massage MM

15141312

15141312

21 ,

,

MMMW

MMMW

M

1

33

44 2

5

13M

14M

15M

21M

30 November, 2005


Fixed Point Equation and Iterative Method

Fixed Point Equation ** MM

Iterative Method

23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M


Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

Original ImageOriginal Image Degraded ImageDegraded Image

MSE: 1529MSE: 1529

MSE: 1512MSE: 1512

EM Algorithm with Belief Propagation


Comparison of Belief Propagation with Comparison of Belief Propagation with Exact Results in Gaussian Graphical ModelExact Results in Gaussian Graphical Model

2ˆ||

1MSE

i

ii ff

MSEMSE

Belief Belief PropagationPropagation

327327 0.0006110.000611 36.30236.302 -5.19201-5.19201

ExactExact 315315 0.0007590.000759 37.91937.919 -5.21444-5.21444

ˆ,ˆln gP

MSEMSE

Belief Belief PropagationPropagation

260260 0.0005740.000574 33.99833.998 -5.15241-5.15241

ExactExact 236236 0.0006520.000652 34.97534.975 -5.17528-5.17528

ˆ,ˆln gP

40

40

,max argˆ,ˆ,

gP


Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:315MSE:315MSE: 325MSE: 325

MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411

MSE: 1512MSE: 1512

Degraded ImageDegraded Image Belief PropagationBelief Propagation

Lowpass FilterLowpass Filter Median FilterMedian Filter

Exact

Wiener Filter

2ˆ||

1MSE

i

ii ff


Original ImageOriginal Image

MSE236MSE236MSE: 260MSE: 260

MSE: 372MSE: 372 MSE: 244MSE: 244MSE: 224MSE: 224

MSE: 1529MSE: 1529

Degraded ImageDegraded Image Belief PropagationBelief Propagation

Lowpass FilterLowpass Filter Median FilterMedian Filter

Exact

Wiener Filter

2ˆ||

1MSE

i

ii ff

Image Restoration by Gaussian Image Restoration by Gaussian Graphical ModelGraphical Model


ContentsContents

1. Introduction





SummarySummary

Formulation of belief propagationFormulation of belief propagation

Accuracy of belief propagation in Bayesian Accuracy of belief propagation in Bayesian image analysis by means of Gaussian image analysis by means of Gaussian graphical model (Comparison between the graphical model (Comparison between the belief propagation and exact calculation)belief propagation and exact calculation)

Statistical Learning Procedure in Loopy Belief Propagation for Probabilistic Image Processing

Documents

Transcript of Statistical Learning Procedure in Loopy Belief Propagation for Probabilistic Image Processing