Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
description
Transcript of Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
Phisical Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics4th Maximum likelihood estimation and EM algorithm
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Phisical Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.
Phisical Fluctuomatics (Tohoku University) 3
Statistical Machine Learning and Model Selection
Inference of Probabilistic Model by using Data
Model Selection
Statistical Machine Learning
Maximum Likelihood
EM algorithmImplement by Belief Propagation and Markov Chain Monte Carlo method
Akaike Information Criteria, Akaike Bayes Information Criteria, etc.
Extension
Complete Data and Incomplete data
Phisical Fluctuomatics (Tohoku University) 4
Maximum Likelihood Estimation
,
Parameter
Phisical Fluctuomatics (Tohoku University) 5
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
1
1
0
Ng
g
g
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Phisical Fluctuomatics (Tohoku University) 6
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
Data
1
1
0
Ng
g
g
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data
Phisical Fluctuomatics (Tohoku University) 7
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
Data
1
1
0
Ng
g
g
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
Phisical Fluctuomatics (Tohoku University) 8
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
Data
1
1
0
Ng
g
g
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
Phisical Fluctuomatics (Tohoku University) 9
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
Data
1
1
0
Ng
g
g
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
,maxargˆ,ˆ,
gP
Phisical Fluctuomatics (Tohoku University) 10
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
,maxargˆ,ˆ,
gP
Data
1
1
0
Ng
g
g
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 11
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gPExtremum Condition
1
1
0
Ng
g
g
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 12
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆ
N
iig
N
1
0
22 ˆ1
ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
g
g
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Phisical Fluctuomatics (Tohoku University) 13
Maximum Likelihood Estimation
1
0
222 2
1exp
2
1,
N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆ
N
iig
N
1
0
22 ˆ1
ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
g
g
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Histogram
Phisical Fluctuomatics (Tohoku University) 14
1
1
0
Nf
f
f
f
Maximum Likelihood
1
0
2
22 2
1exp
2
1,
N
iii fgfgP
gP
maxargˆ Data
0
,1
ˆ
gP
1
1
22 11ˆ
N
iig
N
Extremum Condition
1
0
2
2
1exp
2
1N
iiffP
ff
fPfgPgfPgP
,,
1
1
0
Ng
g
g
g
Hyperparameter
Parameter
fdgfPff
,ˆ
gP
fPfgPgfP
,,
Bayes Formula
f
is unknown
Marginal Likelihood
Phisical Fluctuomatics (Tohoku University) 15
Probabilistic Model for Signal Processing
Source Signal Observable Data
Transmission
Noise
Likelihood Marginal
yProbabilitPrior LikelihoodyProbabilitPosterior
Signal Observable
Signal SourceData ObservableData ObservableSignal Source
Pr
PrSignal Source|PrPr
NoiseGaussian WhiteSignal SourceData Observable
i
fi
i
gi
Bayes Formula
Phisical Fluctuomatics (Tohoku University) 16
Prior Probability for Source Signal
Image DataOne dimensional Data
1 2 3 4 5
1 2 2 3X
3 4 4 5XX
=
Ejiji
EjiEjiji
ffZ
ffZ
fP
},{
2
Prior
},{},{
2
Prior
2
1exp
1
2
1exp
1
E: Set of all the links
i j
1 2 3 4
6 7 8 9
21 22 23 24
5
10
25
11 12 13 14
16 17 18 19
15
20
Phisical Fluctuomatics (Tohoku University) 17
Data Generating Process
Additive White Gaussian Noise
2,0~ Nfg ii
Viii gffgP 2
22 2
1exp
2
1,
V : Set of all the nodes
Phisical Fluctuomatics (Tohoku University) 18
Probabilistic Model for Signal Processing
Viii fgfgP 2
22 2
1exp
2
1,
Ejiji ff
ZfP
},{
2
prior 2
1exp
1
1
1
0
Nf
f
f
f
データ
1
1
0
Ng
g
g
g
Hyperparameter
i
fi
i
gi
Parameter
fdfPfgP
fPfgPgfP
,
,,,
fdgfPff ii
,,ˆ
Posterior Probability
Phisical Fluctuomatics (Tohoku University) 19
1
1
0
Nf
f
f
f
Maximum Likelihood in Signal Processing
,maxargˆ,ˆ,
gP
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgP
Extremum Condition
fdfPfgPgP ,,
1
1
0
Ng
g
g
g
Hyperparameter
Parameter
Incomplete Data
Marginal Likelihood
Phisical Fluctuomatics (Tohoku University) 20
1
1
0
Nf
f
f
f
Maximum Likelihood and EM algorithm
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgP
Extemum Condition
fdfPfgPgP ,,
1
1
0
Ng
g
g
g
Hyperparameter
Parameter
fdgfPgfP
Q
,,ln,,
,,
)(),(,maxarg
)1()1(
Update:Step M
)(),(, Calculate :Step E
),(ttQ
t,σtα
ttQ
0
,,
0,,
,
,
Q
Q
Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.
Q function
Marginal Likelihood
Incomplete Data
Phisical Fluctuomatics (Tohoku University) 21
Model Selection in One Dimensional Signal
Expectation Maximization (EM) Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
if
ig
if
Original Signal
Degraded Signal
Estimated Signal
40
0.04
0.03
0.02
0.01
α(t)
0
α(0)=0.0001, σ(0)=100
Phisical Fluctuomatics (Tohoku University) 22
Model Selection in Noise Reduction
Original Image
Degraded Image EM algorithm and
Belief Propagation
α(0)=0.0001σ(0)=100
Estimate of Original Image
MSE
327 0.000611 36.30
MSE
260 0.000574 34.00
2ˆ
||
1MSE
i
ii ff
40
Phisical Fluctuomatics (Tohoku University) 23
Summary
Maximum Likelihood and EM algorithm
Statistical Inference by Gaussian Graphical Model
Phisical Fluctuomatics (Tohoku University) 24
Let us suppose that data {gi |i=0,1,...,N-1} are
generated by according to the following probability density function:
,
Prove that estimates for average m and variance s2 of the maximum likelihood
are given as
,
Practice 4-1
1
0
222 2
1exp
2
1,
N
iiggP
1
1
0
Ng
g
g
g
,
,maxargˆ,ˆ,
gP
1
0
1ˆ
N
iig
N
1
0
22 ˆ1
ˆN
iig
N
,
, where