Statistical performance analysis by loopy belief propagation in probabilistic image processing

14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 11

Statistical performance analysis by Statistical performance analysis by loopy belief propagation in loopy belief propagation in

probabilistic image processing probabilistic image processing Kazuyuki TanakaKazuyuki Tanaka

Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku University, JapanTohoku University, Japan

http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/

CollaboratorsD. M. Titterington (University of Glasgow, UK)

M. Yasuda (Tohoku University, Japan)S. Kataoka (Tohoku University, Japan)

22

IntroductionIntroductionBayesian network is originally one of the methods for probabilistic inferences in artificial intelligence. Some probabilistic models for information processing are also regarded as Bayesian networks. Bayesian networks are expressed in terms of products of functions with a couple of random variables and can be associated with graphical representations. Such probabilistic models for Bayesian networks are referred to as Graphical Model.

14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud)

33

Probabilistic Image Processing by Bayesian NetworkProbabilistic Image Processing by Bayesian Network

Probabilistic image processing systems are formulated on square grid graphs. Averages, variances and covariances of the Bayesian network are approximately computed by using the belief propagation on the square grid graph.

14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud)

10 March, 201010 March, 2010 IW-SMI2010 (Kyoto)IW-SMI2010 (Kyoto) 44

MRF, Belief Propagation and MRF, Belief Propagation and Statistical PerformanceStatistical Performance

Nishimori and Wong (1999): Physical Review ENishimori and Wong (1999): Physical Review EStatistical Performance Estimation for MRFStatistical Performance Estimation for MRF

(Infinite Range Ising Model and Replica Theory)(Infinite Range Ising Model and Replica Theory)

Is it possible to estimate the performance of belief Is it possible to estimate the performance of belief propagation statistically?propagation statistically?

Tanaka and Morita (1995): Physics Letters ATanaka and Morita (1995): Physics Letters ACluster Variation Method for MRF in Image ProcessingCluster Variation Method for MRF in Image Processing

CVM= Generalized Belief Propagation (GBP)CVM= Generalized Belief Propagation (GBP)

Geman and Geman (1986): IEEE Transactions on PAMIGeman and Geman (1986): IEEE Transactions on PAMIImage Processing by Image Processing by Markov Random Fields (MRF)Markov Random Fields (MRF)


OutlineOutline

1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov

Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss

Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary

Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation

5.5. Concluding RemarksConcluding Remarks


OutlineOutline






Image Restoration by Bayesian Statistics

Original Image

14 October, 2010 7LRI Seminar 2010 (Univ. Paris-Sud)


Original Image

Degraded Image

Transmission

Noise



Original Image

Degraded Image

Transmission

Noise

Estimate


10


PriorProcessn Degradatio

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

Bayes Formula

Original Image

Degraded Image

Transmission

Noise

Estimate

Posterior


11



Posterior



Assumption 1: Original images are randomly generated by according to a prior probability.

Bayes Formula

Original Image

Degraded Image

Transmission

Noise

Estimate

Posterior


7 July, 2010 12



Posterior



Bayes Formula

Assumption 2: Degraded images are randomly generated from the original image by according to a conditional probability of degradation process.

Original Image

Degraded Image

Transmission

Noise

Estimate

Posterior

12LRI Seminar 2010 (Univ. Paris-Sud)14 October, 2010

14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 13

Bayesian Image Analysis

Prior Probability

Ejiji

Ejiji fffffF

},{

2

},{

2 )(21exp)(

21exp}|Pr{

Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.

Prior

Likelihood

Posterior

}Image OriginalPr{

}Image Original|Image DegradedPr{


0

14


Prior Probability

Ejiji

Ejiji fffffF

},{

2

},{

2 )(21exp)(

21exp}|Pr{


Prior

Likelihood

Posterior

}Image OriginalPr{



0


15


0005.0 0030.00001.0

Patterns by MCMC.

Prior Probability

Ejiji

Ejiji fffffF

},{

2

},{

2 )(21exp)(

21exp}|Pr{


Prior

Likelihood

Posterior

}Image OriginalPr{



0




Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.

Prior

Likelihood

Posterior

}Image OriginalPr{



Viii fg

fFgG

22

)(2

1exp

},|Pr{

0V:Set of all the pixels

17


Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.

Prior

Likelihood

Posterior

}Image OriginalPr{




Viii fg

fFgG

22

)(2

1exp

},|Pr{

18

Bayesian Image AnalysisAssumption: Degraded image is generated from the original image by Additive White Gaussian Noise.

Prior

Likelihood

Posterior

}Image OriginalPr{



Viii fg

fFgG

22

)(2

1exp

},|Pr{

0




f

g

}|Pr{ fF

},|Pr{ fFgG

gOriginal

ImageDegraded

Image

Prior ProbabilityPosterior Probability

Degradation Process

Eji

jiVi

ii ffgf

gGfFfFgG

gGfF

},{

222

)(21)(

21exp

},|Pr{},Pr{},|Pr{

},,|Pr{

Estimate

20


f

g

}|Pr{ fF

},|Pr{ fFgG

gOriginal

ImageDegraded

Image


Degradation Process

Eji

jiVi

ii ffgf

gGfFfFgG

gGfF

},{

222

)(21)(

21exp

},|Pr{},Pr{},|Pr{

},,|Pr{

SmoothingData Dominant

Estimate


11 March, 2010 IW-SMI2010 (Kyoto) 21


f

g

}|Pr{ fF

},|Pr{ fFgG

gOriginal

ImageDegraded

Image


Degradation Process

Eji

jiVi

ii ffgf

gGfFfFgG

gGfF

},{

222

)(21)(

21exp

},|Pr{},Pr{},|Pr{

},,|Pr{


Bayesian Network Estimate


gzdgGzFz

ghf

�

12

},,|Pr{

),,(ˆ

CI

22


f

g

}|Pr{ fF

},|Pr{ fFgG

gOriginal

ImageDegraded

Image


Degradation Process

Eji

jiVi

ii ffgf

gGfFfFgG

gGfF

},{

222

)(21)(

21exp

},|Pr{},Pr{},|Pr{

},,|Pr{

Field Random Markov Gauss),(

iF


Bayesian Network Estimate


otherwise,0

},{,1|,|

EjiVjii

ji C


Image Restorations by Image Restorations by Gaussian Markov Random Fields Gaussian Markov Random Fields

and Conventional Filtersand Conventional Filters

2||ˆ||

||1MSE

Viff

V

MSEGauss Markov Random Field 315

Lowpass Filter

(3x3) 388

(5x5) 413

Median Filter

(3x3) 486

(5x5) 445

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianGauss Markov Gauss Markov Random FieldRandom Field

Original ImageOriginal Image Degraded ImageDegraded Image

RestoredRestoredImageImage VV: Set of all : Set of all

the pixelsthe pixels


iF


OutlineOutline







Statistical Performance by Sample Average of Numerical Experiments

f

Original Images

26


1g

f

2g

3g

4g

5g

Original Images Noi

se P

r{G

|F=f

,}

ObservedData


11 March, 2010 IW-SMI2010 (Kyoto) 27


1g

f

2g

3g

4g

5g

1h

2h

3h

4h

4h

Post

erio

r Pr

obab

ility

Pr{F

|G=g

,,

}

Original Images Noi

se P

r{G

|F=f

,}

Estimated ResultsObserved

Data


28


1g

f

2g

3g

4g

5g

1h

2h

3h

4h

4h

Post

erio

r Pr

obab

ility

Pr{F

|G=g

,,

}

Sample Average of Mean Square Error

Original Images Noi

se P

r{G

|F=f

,}

5

1

2||||51),|,(n

nhffD


Data



Statistical Performance Estimation

gdfFgGfghV

fD

},|Pr{),,(1),|,(2

ggh

12

),,(

CI

gg

Additive White Gaussian Noise

},|Pr{ fFgG

f

},,|Pr{ gGfF

Posterior Probability

RestoredImage

Original Image Degraded Image

},|Pr{ fFgG

Additive White Gaussian Noise

otherwise,0

},{,1|,|

EjiVjii

ji C


Statistical Performance Estimation for Gauss Markov Random Fields

T22

242

22

2

22

||2

2

2

)(||1

)(Tr1

21exp

211

},|Pr{),,(1),|,(

fI

fVV

gdfggfV

gdfFgGghfV

fD

V

CC

CII

CII

otherwise

EjiVjii

ji,0

},{,1|,|

C

0

200

400

600

0 0.001 0.002 0.003

=40

0

200

400

600

0 0.001 0.002 0.003

=40

),|,( fD

),|,( fD

OutlineOutline







32

Marginal Probability in Belief Propagation

1 3 4

,,,,,PrPr 43212F F F F

NN

FFFFFF gGgG

14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)

In order to compute the marginal probability Pr{F2|G=g},we take summations over all the pixels except the pixel 2.

33


1 3 4F F F FN

2


1 3 4

,,,,,PrPr 43212F F F F

NN

FFFFFF gGgG

34


1 3 4F F F FN

2 2


1 3 4

,,,,,PrPr 43212F F F F

NN

FFFFFF gGgG

In the belief propagation, the marginal probability Pr{F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixel 2.

35


3 4F F FN

1 2


In order to compute the marginal probability Pr{F1,F2|G=g},we take summations over all the pixels except the pixels 1 and2.

3 4

,,,,,Pr,Pr 432121F F F

NN

FFFFFFF gGgG

36


3 4F F FN

1 2 1 2


3 4

,,,,,Pr,Pr 432121F F F

NN

FFFFFFF gGgG

In the belief propagation, the marginal probability Pr{F1,F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixels 1 and 2.

Belief Propagation in Probabilistic Image Processing



Image Restorations by Image Restorations by Gaussian Markov Random Fields Gaussian Markov Random Fields and Conventional Filtersand Conventional Filters

MSE

Gauss MRF (Exact) 315

Gauss MRF (Belief Propagation) 327

Lowpass Filter

(3x3) 388

(5x5) 413

Median Filter

(3x3) 486

(5x5) 445

Belief PropagationExact

Original Image Degraded Image

RestoredRestoredImageImage

VV: Set of all : Set of all the pixelsthe pixels


iF

2ˆ||

1MSE

Vi

ii ffV

40

Gray-Level Image Restoration(Spike Noise)

Original Image

MSE:135MSE: 217

MSE: 371 MSE: 523

MSE: 244

MSE: 3469

MSE: 2075

Degraded Image

Belief Propagation Lowpass Filter Median Filter

MSE: 39514 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)


Binary Image Restoration byLoopy Belief Propagation

42


1g

f

2g

3g

4g

5g

1h

2h

3h

4h

4h

Post

erio

r Pr

obab

ility

Pr{F

|G=g

,,

}

Sample Average of Mean Square Error

Original Images Noi

se P

r{G

|F=f

,}

5

1

2||||51),|,(n

nhffD


Data



Statistical Performance Estimation for Binary Markov Random Fields

gdfFgGfghV

fD },|Pr{),,(1),|,(

2

||211 1 1 },{

222

22

1 2 ||

)(21)(

21explog

)(2

1exp

Vz z z Eji

jiVi

ii

Viii

dgdgdgzzgz

fg

V

It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields g1, g2,…,g|V|.

Free Energy of Ising Model with Random External Fields

1ifLight intensities of the original image can be regarded as spin states of ferromagnetic system.

== >

== >

Eji },{

Eji

jiVi

ii ffgf

gGfF

},{

222

)(21)(

21exp

},,|Pr{

Statistical Performance Estimation for Binary Markov Random Fields


},|Pr{)(),,(||

1

},|Pr{),,(1),|,(

2

2

iiiiij

ijijVi

iiij

iji fFgGghfddgV

gdfFgGfghV

fD

},|Pr{)(

tanh)tanh(tanh

)(

*

}/{

}/{2

1

}{\

jjjjijk

jkjk

ijljl

iij

Vi ijkjkiijij

fFgG

g

ddg

ijij

ii

ggh

2sgn),,(

gdfFgGfghV

fD },|Pr{),,(1),|,(

2


0

0.1

0.2

0.3

0.4

0 0.2 0.4 0.6 0.8 1

Statistical Performance Estimation for Markov Random Fields

0

200

400

600

0 0.001 0.002 0.003

=40=1

),|,( fD

Spin Glass Theory in Statistical MechanicsLoopy Belief Propagation

0.8

0.6

0.40.2

),|,( fD

Multi-dimensional Gauss Integral Formulas


Statistical Performance Estimation for Markov Random Fields

gdfFgGghfV

fD },|Pr{),,(1),|,(

2

OutlineOutline








SummarySummary

Formulation of probabilistic model for image Formulation of probabilistic model for image processing by means of conventional statistical processing by means of conventional statistical schemes has been summarized.schemes has been summarized.Statistical performance analysis of probabilistic Statistical performance analysis of probabilistic image processing by using Gauss Markov Random image processing by using Gauss Markov Random Fields has been shown.Fields has been shown.One of extensions of statistical performance One of extensions of statistical performance estimation to probabilistic image processing with estimation to probabilistic image processing with discretediscrete states has been demonstrated.states has been demonstrated.

Image Impainting by Gauss MRF and LBP


Gauss MRFand LBP

Our framework can be extended to erase a scribbling.


ReferencesReferences1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM

Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.

2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.

3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008

4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009

5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.

6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.

Statistical performance analysis by loopy belief propagation in probabilistic image processing

Documents

Transcript of Statistical performance analysis by loopy belief propagation in probabilistic image processing