Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
description
Transcript of Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm
![Page 1: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/1.jpg)
Phisical Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics4th Maximum likelihood estimation and EM algorithm
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
![Page 2: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/2.jpg)
Phisical Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.
![Page 3: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/3.jpg)
Phisical Fluctuomatics (Tohoku University) 3
Statistical Machine Learning and Model SelectionInference of Probabilistic Model by using Data
Model Selection
Statistical Machine Learning Maximum Likelihood
EM algorithmImplement by Belief Propagation and Markov Chain Monte Carlo method
Akaike Information Criteria, Akaike Bayes Information Criteria, etc.
Extension
Complete Data and Incomplete data
![Page 4: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/4.jpg)
Phisical Fluctuomatics (Tohoku University) 4
Maximum Likelihood Estimation
,
Parameter
![Page 5: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/5.jpg)
Phisical Fluctuomatics (Tohoku University) 5
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
![Page 6: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/6.jpg)
Phisical Fluctuomatics (Tohoku University) 6
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data
![Page 7: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/7.jpg)
Phisical Fluctuomatics (Tohoku University) 7
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
![Page 8: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/8.jpg)
Phisical Fluctuomatics (Tohoku University) 8
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
![Page 9: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/9.jpg)
Phisical Fluctuomatics (Tohoku University) 9
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
Data
1
1
0
Ng
gg
g
,
Parameter
1,,1,0 NV
0 1 2
3 4 5
6 7 8
Data Histogram
,maxargˆ,ˆ,
gP
![Page 10: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/10.jpg)
Phisical Fluctuomatics (Tohoku University) 10
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
![Page 11: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/11.jpg)
Phisical Fluctuomatics (Tohoku University) 11
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gPExtremum Condition
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
![Page 12: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/12.jpg)
Phisical Fluctuomatics (Tohoku University) 12
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
![Page 13: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/13.jpg)
Phisical Fluctuomatics (Tohoku University) 13
Maximum Likelihood Estimation
1
0
222 2
1exp2
1,N
iiggP
,maxargˆ,ˆ,
gP
Data
0
,
0,
ˆ,ˆ
ˆ,ˆ
gP
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
Extremum Condition
Sample AverageSample Deviation
1
1
0
Ng
gg
g
,
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.g
Histogram
![Page 14: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/14.jpg)
Phisical Fluctuomatics (Tohoku University) 14
1
1
0
Nf
ff
f
Maximum Likelihood
1
0
222 2
1exp2
1,N
iii fgfgP
gP maxargˆ Data
0
,1
ˆ
gP
1
1
22 11ˆN
iig
N
Extremum Condition
1
0
2
21exp
21N
iiffP
ff
fPfgPgfPgP
,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
fdgfPff
,ˆ
gP
fPfgPgfP
,,
Bayes Formula
f
is unknown
Marginal Likelihood
![Page 15: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/15.jpg)
Phisical Fluctuomatics (Tohoku University) 15
Probabilistic Model for Signal Processing
Source Signal Observable Data
Transmission
Noise
Likelihood Marginal
yProbabilitPrior LikelihoodyProbabilitPosterior
Signal Observable
Signal SourceData ObservableData ObservableSignal Source
PrPrSignal Source|PrPr
NoiseGaussian WhiteSignal SourceData Observable
i
fi
i
gi
Bayes Formula
![Page 16: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/16.jpg)
Phisical Fluctuomatics (Tohoku University) 16
Prior Probability for Source Signal
Image DataOne dimensional Data
1 2 3 4 5
1 2 2 3X
3 4 4 5XX
=
Ejiji
EjiEjiji
ffZ
ffZ
fP
},{
2
Prior
},{},{
2
Prior
21exp1
21exp1
E: Set of all the links
i j
1 2 3 4
6 7 8 9
21 22 23 24
5
10
25
11 12 13 14
16 17 18 19
15
20
![Page 17: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/17.jpg)
Phisical Fluctuomatics (Tohoku University) 17
Data Generating Process
Additive White Gaussian Noise
2,0~ Nfg ii
Viii gffgP 2
22 21exp
2
1,
V : Set of all the nodes
![Page 18: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/18.jpg)
Phisical Fluctuomatics (Tohoku University) 18
Probabilistic Model for Signal Processing
Viii fgfgP 2
22 21exp
2
1,
Ejiji ff
ZfP
},{
2
prior 21exp1
1
1
0
Nf
ff
f
データ
1
1
0
Ng
gg
g
Hyperparameter
i
fi
i
gi
Parameter
fdfPfgP
fPfgPgfP
,
,,,
fdgfPff ii
,,ˆ
Posterior Probability
![Page 19: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/19.jpg)
Phisical Fluctuomatics (Tohoku University) 19
1
1
0
Nf
ff
f
Maximum Likelihood in Signal Processing
,maxargˆ,ˆ,
gP
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgPExtremum Condition
fdfPfgPgP ,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
Incomplete Data
Marginal Likelihood
![Page 20: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/20.jpg)
Phisical Fluctuomatics (Tohoku University) 20
1
1
0
Nf
ff
f
Maximum Likelihood and EM algorithm
Data
0
,,0
,
ˆ,ˆˆ,ˆ
gPgP
Extemum Condition
fdfPfgPgP
,,
1
1
0
Ng
gg
g
Hyperparameter
Parameter
fdgfPgfP
Q
,,ln,,
,,
)(),(,maxarg
)1()1( Update:Step M
)(),(, Calculate :Step E
),(ttQ
t,σtα
ttQ
0
,,
0,,
,
,
Q
Q
Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.
Q function
Marginal Likelihood
Incomplete Data
![Page 21: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/21.jpg)
Phisical Fluctuomatics (Tohoku University) 21
Model Selection in One Dimensional Signal
Expectation Maximization (EM) Algorithm
i
i
i
0 127 255
0 127 255
0 127 255
100
0
200
100
0
200
100
0
200
if
ig
if
Original Signal
Degraded Signal
Estimated Signal
40
0.04
0.03
0.02
0.01
α(t)
0
α(0)=0.0001, σ(0)=100
![Page 22: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/22.jpg)
Phisical Fluctuomatics (Tohoku University) 22
Model Selection in Noise Reduction
Original Image
Degraded Image EM algorithm and
Belief Propagationα(0)=0.0001σ(0)=100
Estimate of Original Image
MSE327 0.000611 36.30
MSE260 0.000574 34.00
2ˆ
||1MSE
i
ii ff
40
![Page 23: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/23.jpg)
Phisical Fluctuomatics (Tohoku University) 23
Summary
Maximum Likelihood and EM algorithmStatistical Inference by Gaussian Graphical Model
![Page 24: Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm](https://reader036.fdocuments.in/reader036/viewer/2022062410/568162b3550346895dd33c1b/html5/thumbnails/24.jpg)
Phisical Fluctuomatics (Tohoku University) 24
Let us suppose that data {gi |i=0,1,...,N1} are generated by according to the following probability density function:
,
Prove that estimates for average and variance 2 of the maximum likelihood
are given as
,
Practice 3-1
1
0
222 2
1exp2
1,N
iiggP
1
1
0
Ng
gg
g
,
,maxargˆ,ˆ,
gP
1
0
1ˆN
iig
N
1
0
22 ˆ1ˆN
iig
N
,
, where