Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
description
Transcript of Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
Phisical Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics4th Maximum likelihood estimation and EM algorithm
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Phisical Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.
Phisical Fluctuomatics (Tohoku University) 3
Statistical Machine Learning and Model SelectionInference of Probabilistic Model by using Data
Model Selection
Statistical Machine Learning Maximum Likelihood
EM algorithmImplement by Belief Propagation and Markov Chain Monte Carlo method
Akaike Information Criteria, Akaike Bayes Information Criteria, etc.
Extension
Complete Data and Incomplete data
Phisical Fluctuomatics (Tohoku University) 4
Maximum Likelihood Estimation
,
Parameter
Phisical Fluctuomatics (Tohoku University) 5
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Phisical Fluctuomatics (Tohoku University) 6
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data
Phisical Fluctuomatics (Tohoku University) 7
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
Phisical Fluctuomatics (Tohoku University) 8
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
Phisical Fluctuomatics (Tohoku University) 9
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
,maxargˆ,ˆ,
gP
Phisical Fluctuomatics (Tohoku University) 10
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 11
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gPExtremum Condition
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 12
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 13
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Histogram
Phisical Fluctuomatics (Tohoku University) 14
1
1
0
Nf
ff
f
Maximum Likelihood
1
0
222 2
1exp2
1,N
iii fgfgP
gP maxargˆ Data
0
,1
ˆ
gP
1
1
22 11ˆN
iig
N
Extremum Condition
1
0
2
21exp
21N
iiffP
ff
fPfgPgfPgP
,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
fdgfPff
,ˆ
gP
fPfgPgfP
,,
Bayes Formula
f
is unknown
Marginal Likelihood
Phisical Fluctuomatics (Tohoku University) 15
Probabilistic Model for Signal Processing
Source Signal Observable Data
Transmission
Noise
Likelihood Marginal
yProbabilitPrior LikelihoodyProbabilitPosterior
Signal Observable
Signal SourceData ObservableData ObservableSignal Source
PrPrSignal Source|PrPr
NoiseGaussian WhiteSignal SourceData Observable
i
fi
i
gi
Bayes Formula
Phisical Fluctuomatics (Tohoku University) 16
Prior Probability for Source Signal
Image DataOne dimensional Data
1 2 3 4 5
1 2 2 3X
3 4 4 5XX
=
Ejiji
EjiEjiji
ffZ
ffZ
fP
},{
2
Prior
},{},{
2
Prior
21exp1
21exp1
E: Set of all the links
i j
1 2 3 4
6 7 8 9
21 22 23 24
5
10
25
11 12 13 14
16 17 18 19
15
20
Phisical Fluctuomatics (Tohoku University) 17
Data Generating Process
Additive White Gaussian Noise
2,0~ Nfg ii
Viii gffgP 2
22 21exp
2
1,
V : Set of all the nodes
Phisical Fluctuomatics (Tohoku University) 18
Probabilistic Model for Signal Processing
Viii fgfgP 2
22 21exp
2
1,
Ejiji ff
ZfP
},{
2
prior 21exp1
1
1
0
Nf
ff
f
データ
1
1
0
Ng
gg
g
Hyperparameter
i
fi
i
gi
Parameter
fdfPfgP
fPfgPgfP
,
,,,
fdgfPff ii
,,ˆ
Posterior Probability
Phisical Fluctuomatics (Tohoku University) 19
1
1
0
Nf
ff
f
Maximum Likelihood in Signal Processing
,maxargˆ,ˆ,
gP
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgPExtremum Condition
fdfPfgPgP ,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
Incomplete Data
Marginal Likelihood
Phisical Fluctuomatics (Tohoku University) 20
1
1
0
Nf
ff
f
Maximum Likelihood and EM algorithm
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgP
Extemum Condition
fdfPfgPgP
,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
fdgfPgfP
Q
,,ln,,
,,
)(),(,maxarg
)1()1( Update:Step M
)(),(, Calculate :Step E
),(ttQ
t,σtα
ttQ
0
,,
0,,
,
,
Q
Q
Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.
Q function
Marginal Likelihood
Incomplete Data
Phisical Fluctuomatics (Tohoku University) 21
Model Selection in One Dimensional Signal
Expectation Maximization (EM) Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
if
ig
if
Original Signal
Degraded Signal
Estimated Signal
40
0.04
0.03
0.02
0.01
α(t)
0
α(0)=0.0001, σ(0)=100
Phisical Fluctuomatics (Tohoku University) 22
Model Selection in Noise Reduction
Original Image
Degraded Image EM algorithm and
Belief Propagationα(0)=0.0001σ(0)=100
Estimate of Original Image
MSE327 0.000611 36.30
MSE260 0.000574 34.00
2ˆ
||1MSE
i
ii ff
40
Phisical Fluctuomatics (Tohoku University) 23
Summary
Maximum Likelihood and EM algorithmStatistical Inference by Gaussian Graphical Model
Phisical Fluctuomatics (Tohoku University) 24
Let us suppose that data {gi |i=0,1,...,N1} are generated by according to the following probability density function:
,
Prove that estimates for average and variance 2 of the maximum likelihood
are given as
,
Practice 3-1
1
0
222 2
1exp2
1,N
iiggP
1
1
0
Ng
gg
g
,
,maxargˆ,ˆ,
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
,
, where