Introduction to Probabilistic Image Processing and ... › ~kazu › tutorial... · Bayes Formula...

29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 11

Introduction to Introduction to Probabilistic Image Processing and Probabilistic Image Processing and

Bayesian NetworksBayesian NetworksKazuyuki TanakaKazuyuki Tanaka

Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku University, Sendai, JapanTohoku University, Sendai, Japan

http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/

CollaboratorsProf. Mike Titterington (University of Glasgow, UK)

Dr. Koji Tsuda (MPI for Biological Cybernetics, Germany)Dr. Muneki Yasuda (Tohoku University, Japan)


ContentsContents

1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks


ContentsContents



More is differentMore is different

Probabilistic information processing can give Probabilistic information processing can give us unexpected capacity in a system constructed us unexpected capacity in a system constructed from many cooperating elements with from many cooperating elements with randomness. randomness. The circumstances are similar to everything in The circumstances are similar to everything in the natural world being made up of a lot of the natural world being made up of a lot of molecules with interactions and fluctuations. molecules with interactions and fluctuations. The collection of molecules can give rise to The collection of molecules can give rise to unpredictable phenomena. unpredictable phenomena. This is often called This is often called ““More is differentMore is different”” in in physics. physics.

29 December, 2008 National Tsing Hua University, Taiwan 5

Main InterestsMain InterestsInformation Processing: Information Processing:

DataDataPhysics: Physics:

Material, Material, Natural PhenomenaNatural Phenomena

System of a lot of elements with mutual relationSystem of a lot of elements with mutual relationCommon Concept between Computer Sciences and PhysicsCommon Concept between Computer Sciences and Physics

MaterialMolecule

Materials are constructed from a lot of molecules.

Molecules have interactions of each other.

０,1 101101110001

010011101110101000111110000110000101000000111010101110101010Bit

Data

Data is constructed from many bits

A sequence is formed by deciding the arrangement of bits.

A lot of elements have mutual relation of each otherA lot of elements have mutual relation of each otherSome physical concepts Some physical concepts in Physical models are in Physical models are useful for the design of useful for the design of computational models in computational models in probabilistic probabilistic information processing.information processing.


More is different in informatics as well.More is different in informatics as well.Our goal is to establish theoretical paradigms Our goal is to establish theoretical paradigms for probabilistic information processing by for probabilistic information processing by means of statistical science and statistical means of statistical science and statistical physics. physics. The probabilistic information processing is The probabilistic information processing is based on both modeling of problems and based on both modeling of problems and design of algorithms, which is often realized design of algorithms, which is often realized as graphical models including Bayesian as graphical models including Bayesian network.network.


Purpose of My TalkPurpose of My Talk

Review of formulation of probabilistic model for Review of formulation of probabilistic model for image processing by means of conventional image processing by means of conventional statistical schemes.statistical schemes.Review of probabilistic image processing by using Review of probabilistic image processing by using Gaussian graphical model (Gauss Markov Gaussian graphical model (Gauss Markov Random Fields) as the most basic example.Random Fields) as the most basic example.Review of how to construct a belief propagation Review of how to construct a belief propagation algorithm for image processing.algorithm for image processing.


ContentsContents

1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical Model Gaussian Graphical Model 4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks


Bayes Formula and Bayesian Network

Posterior Probability

}Pr{}Pr{}|Pr{}|Pr{

BAABBA =

Bayes Rule

Prior Probability

Event B is given as the observed data.Event A corresponds to the original information to estimate. Thus the Bayes formula can be applied to the estimation of the original information from the given data.

A

BBayesian Network

Data-Generating Process


Image Restoration by Probabilistic Model

Original Image

Degraded Image

Transmission

Noise

444 3444 21

444 8444 764444444 84444444 76

4444444 84444444 76

Likelihood Marginal

PriorProcess nDegradatio

Posterior

}Image DegradedPr{}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

=

Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.

Bayes Formula


Image Restoration by Probabilistic Model

Degraded

Image

i

fi: Light Intensity of Pixel iin Original Image

),( iii yxr =r

Position Vector of Pixel i

gi: Light Intensity of Pixel iin Degraded Image

i

Original

Image

The original images and degraded images are represented by f = (f1,f2,…,f|V|)T and g = (g1,g2,…,g|V|)T , respectively.


Probabilistic Modeling of Image Restoration

444 8444 764444444 84444444 76

4444444 84444444 76

PriorLikelihood

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{


∝

∏∈Ψ=

ViiiVV fgfffgggP )|(),,,|,,,( ||21||21 LL

Random Fieldsfi

gi

fi

gi

or

Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.


Probabilistic Modeling of Image Restoration

444 8444 764444444 84444444 76

4444444 84444444 76

PriorLikelihood

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{


∝

∏∈Φ=Eji

jiV fffffP},{

||21 ),(),,,( L

Random Fields

Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels.

i j

Product over All the Nearest Neighbour Pairs of Pixels


Prior Probability for Binary Image

== >p p p−

21 p−

21i j Probability of

NeigbouringPixel

∏∈Φ=Eji

ji fffP},{

),()( i j

It is important how we should assume the function Φ(fi,fj) in the prior probability.

)0,1()1,0()0,0()1,1( Φ=Φ>Φ=Φ

We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.

1,0=if


Prior Probability for Binary Image

Prior probability prefers to the configuration with the least number of red lines.

Which state should the center pixel be taken when the states of neighbouring pixels are fixed to the white states?

？

>

== >p pi j Probability of

Nearest NeigbourPair of Pixels


Prior Probability for Binary ImagePrior Probability for Binary Image

Which state should the center pixel be taken when the states of neighbouring pixels are fixed as this figure?

？-？== >

p p

> >=

Prior probability prefers to the configuration with the least number of red lines.


What happens for the case of large number of pixels?

p 0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0lnp

Disordered State Critical Point(Large fluctuation)

small p large p

Covariance between the nearest neghbourpairs of pixels

Sampling by Marko chain Monte Carlo

Ordered State

Patterns with both ordered statesand disordered states are often generated near the critical point.


Physical model of ferromagnetism and Probabilistic model of image processingPhysical model of ferromagnetism and Probabilistic model of image processing

p p

p p

>

=

=

x

Ising Model

Markov Random Field (MRF) Model

>

=

=

Up Spin State

Down Spin State

Black State

White State

Probabilistic models for image processing has the similar structure as physical models of ferromagnetism

Regular graph

x

y


Pattern near Critical Point of Prior Probability

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0ln p

similar

small p large p

Covariance between the nearest neghbourpairs of pixels

We regard that patterns generated near the critical point are similar to the local patterns in real world images.


Bayesian Image Analysis

fg

)( fP )|( fgP gOriginal Image Degraded Image

Prior Probability

Bayes Formulas=>Posterior Probability

Degradation Process

Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.

E：Set of all the nearest neighbourpairs of pixels

V：Set of All the pixels

)()()|()|(

gPfPfgPgfP =

∫ ∫ ∫+∞∞−

+∞∞−

+∞∞−

= ||21||21||21 ),,,|,,,(ˆVVVii dfdfdfgggfffPff LLLL


ContentsContents



Bayesian Image Analysis by Gaussian Graphical Model

0005.0=α 0030.0=α0001.0=α

Patterns are generated by MCMC.

Markov Chain Monte Carlo Method

Prior Probability ( )+∞∞−∈ ,if

E:Set of all the nearest-neghbourpairs of pixels

V:Set of all the pixels

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−∝ ∑

∈EjijiV fffffP

},{

2||21 2

1exp),,,( αL



( )2,0~ σNfg ii −Histogram of Gaussian Random Numbers

n Noise GaussianfImage Original gImage Degraded

( )+∞∞−∈ ,, ii gf

Degraded image is obtained by adding a white Gaussian noise to the original image.

Degradation Process is assumed to be the additive white Gaussian noise.


( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

ViiiNN gffffgggP 2

222121 21exp

2

1),,,|,,,(σπσ

LL


( ) gCIdfgffPf 12 )( ˆ −+== ∫ ασ


( )+∞∞−∈ ,, ii gf

( )( )

( )⎪⎩

⎪⎨

⎧∈−=

=otherwise0

14

Eijji

jCi

Multi-Dimensional Gaussian Integral Formula

Posterior Probability

Average of the posterior probability can be calculated by using the multi-dimensional Gauss integral Formula

|V|x|V| matrix

E:Set of all the nearest-neghbourpairs of pixels


( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−∝

=

∑∑},{

222 2

12

1exp

)()()|()|(

jiji

iii ffgf

gPfPfgPgfP

ασ



0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100( )tσ

( )tα

g f̂

gCttItf 12 ))()(()(ˆ −+= σα

Iteration Procedure of EM algorithm in Gaussian Graphical Model

EM

f̂

g( )

),|(max arg)ˆ,ˆ(,

σασασα

gP=

( ) ( )( ) ( ) ( )( )gttQtt ,,1,1 σασα ←++


Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

( )2ˆ||

1MSE ∑∈

−=Vi

ii ffV

315315Statistical MethodStatistical Method

445445(5x5)(5x5)486486(3x3)(3x3)Median Median

FilterFilter

413413(5x5)(5x5)388388(3x3)(3x3)LowpassLowpass

FilterFilter

MSEMSE

(3x3) (3x3) LowpassLowpass (5x5) Median(5x5) MedianMRFMRF

Original ImageOriginal Image Degraded ImageDegraded Image

RestoredRestoredImageImage

VV:Set:Set of all of all the pixelsthe pixels


ContentsContents



Graphical Representation for Tractable Models

Tractable Model

⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∑∑∑

∑ ∑ ∑

===

= = =

1,01,01,0

1,0 1,0 1,0

),(),(),(

),(),(),(

CBA

A B C

DChDBgDAf

DChDBgDAf A

B CD

∑ ∑ ∑= = =1,0 1,0 0,1A B C

Intractable Model∑ ∑ ∑= = =1,0 1,0 1,0

),(),(),(A B C

AChCBgBAf

A

B C

Tree Graph

Cycle Graph

It is possible to calculate each summation independently.

It is hard to calculate each summation independently.

∑ ∑ ∑= = =1,0 1,0 0,1A B C


Loopy Belief Propagation for Graphical Model in Image Processing

Graphical model for image processing is represented in terms of the square lattice.Square lattice includes a lot of cycles.Belief propagation are applied to the calculation of statistical quantities as an approximate algorithm.

Every graph consisting of a pixel and its four neighbouring pixels can be regarded as a tree graph.

Loopy Belief PropagationLoopy Belief Propagation

1 2 4

5

31 2 4

5

3

3

1 2

5

4∫+∞∞−

← 2df21


Loopy Belief Propagation in Image Processing

We have four kinds of message passing rules for each pixel.

Each massage passing rule includes 3 incoming messages and 1 outgoing message

Visualizations of Passing Messages


EM algorithm by means of Belief Propagation

Input

Output

LoopyBP EM

Update Rule of Loopy Belief Propagation

EM Algorithm for Hyperparameter Estimation

( ) ( )( ) ( ) ( )( )gttQtt ,,1,1 σασα ←++

3

1 2

5

4∫+∞∞−

← 2df21


Probabilistic Image Processing by Probabilistic Image Processing by EMEM Algorithm and Loopy BP for Algorithm and Loopy BP for

Gaussian Graphical ModelGaussian Graphical Model( ) ( )( ) ( ) ( )( ).,,1,1 gttQtt σασα ←++

gf̂

Loopy Belief PropagationLoopy Belief Propagation

ExactExact

0006000ˆ335.36ˆ

LBP

LBP

.=

=

ασ

0007130ˆ624.37ˆ

Exact

Exact

.=

=

ασ

MSE:327

MSE:315

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100( )tσ

( )tα


ContentsContents

1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical Model Gaussian Graphical Model 4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks


Digital Images Digital Images InpaintingInpaintingbased on MRFbased on MRF

Inpu

t

Out

put

MarkovRandomFields

M. Yasuda, J. Ohkubo M. Yasuda, J. Ohkubo and K. Tanaka: and K. Tanaka: Proceedings ofProceedings ofCIMCA&IAWTIC2005. CIMCA&IAWTIC2005.


ContentsContents



SummarySummary

Formulation of probabilistic model for image Formulation of probabilistic model for image processing by means of conventional statistical processing by means of conventional statistical schemes has been summarized.schemes has been summarized.Probabilistic image processing by using Gaussian Probabilistic image processing by using Gaussian graphical model has been shown as the most basic graphical model has been shown as the most basic example.example.It has been explained how to construct a belief It has been explained how to construct a belief propagation algorithm for image processing.propagation algorithm for image processing.


ReferencesReferences1. K. Tanaka: “Statistical-Mechanical Approach to Image Processing” (Topical

Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002.

2. K. Tanaka: “Probabilistic Inference by Means of Cluster Variation Method andLinear Response Theory,” IEICE Transactions on Information and Systems, vol.E86-D, no.7, pp.1228-1242, 2003.

3. K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: “Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing,”Journal of Physics A: Mathematical and General, vol.37, no.36, pp.8675-8696, 2004.

4. J. Ohkubo, M. Yasuda and K. Tanaka: “Statistical-mechanical Iterative Algorithms on Complex Networks,” Physical Review E, vol.72, no.4, Article No.046135, 2005.

5. K. Tanaka and D. M. Titterington: “Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.

6. M. Yasuda and K. Tanaka: “The Mathematical Structure of the Approximate Linear Response Relation,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.


SMAPIP ProjectSMAPIP ProjectSMAPIP Project

MEXT Grant-in Aid for Scientific Research on Priority AreasMEXT Grant-in Aid for Scientific Research on Priority Areas

Period: 2002 –2005Head Investigator: Kazuyuki TanakaPeriod: 2002 –2005Head Investigator: Kazuyuki Tanaka

Member:K. Tanaka, Y. Kabashima,H. Nishimori, T. Tanaka, M. Okada, O. Watanabe, N. Murata, ......

Statistical Mechanical Approach to Statistical Mechanical Approach to Probabilistic Information ProcessingProbabilistic Information Processing


DEX-SMI ProjectDEXDEX--SMI ProjectSMI Project

http://dex-smi.sp.dis.titech.ac.jp/DEX-SMI/http://dexhttp://dex--smi.sp.dis.titech.ac.jp/DEXsmi.sp.dis.titech.ac.jp/DEX--SMISMI//DEX-SMI GOGO

MEXT GrantMEXT Grant--in Aid for Scientific in Aid for Scientific Research on Priority AreasResearch on Priority Areas

Period: 2006 –2009Head Investigator: Prof. Yoshiyuki Kabashima

(Tokyo Institute of Technology)

Period: 2006 –2009Head Investigator: Prof. Yoshiyuki Kabashima

(Tokyo Institute of Technology)

Deepening and Expansion of Deepening and Expansion of Statistical Mechanical InformaticsStatistical Mechanical Informatics

Member:Y. Kabashima, K. Tanaka, H. Nishimori, T. Tanaka, M. Okada, S. Ishii, M. Hayashi,…


CERIES ProjectCERIES ProjectCERIES Project

http://www.ecei.tohoku.ac.jp/gcoe/http://http://www.ecei.tohoku.ac.jp/gcoewww.ecei.tohoku.ac.jp/gcoe//

CERIES GOGO

GCOE Program

Period: 2007 –2011Head Investigator:

Prof. Fumiyuki Adachi (Tohoku University)

Period: 2007 –2011Head Investigator:

Prof. Fumiyuki Adachi (Tohoku University)

Center of Education and Research for Center of Education and Research for Information Electronics SystemsInformation Electronics Systems

Introduction to Probabilistic Image Processing and ... › ~kazu › tutorial... · Bayes Formula...

Documents

Transcript of Introduction to Probabilistic Image Processing and ... › ~kazu › tutorial... · Bayes Formula...