Probability Theory and Parameter Estimation I
Transcript of Probability Theory and Parameter Estimation I
![Page 1: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/1.jpg)
Probability Theory and
Parameter Estimation I
![Page 2: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/2.jpg)
Least Squares and Gauss
● How to solve the least square problem?● Carl Friedrich Gauss solution in 1794 (age 18)
● Why solving the least square problem?● Carl Friedrich Gauss solution in 1822 (age 46)● Least square solution is optimal in the sense that it is the best linear unbiased estimator of the coefficients of the polynomials ● Assumptions: errors have zero mean and equal variances
http://en.wikipedia.org/wiki/Least_squares
![Page 3: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/3.jpg)
Probability Theory
Apples and Oranges
![Page 4: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/4.jpg)
Probability Theory
Marginal Probability
Conditional ProbabilityJoint Probability
![Page 5: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/5.jpg)
Probability Theory
Sum Rule
Product Rule
![Page 6: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/6.jpg)
The Rules of Probability
Sum Rule
Product Rule
![Page 7: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/7.jpg)
Bayes’ Theorem
posterior ∝ likelihood × prior
![Page 8: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/8.jpg)
Probability Densities
![Page 9: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/9.jpg)
Transformed Densities
![Page 10: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/10.jpg)
Expectations
Conditional Expectation(discrete)
Approximate Expectation(discrete and continuous)
![Page 11: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/11.jpg)
Variances and Covariances
![Page 12: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/12.jpg)
The Gaussian Distributionmean variance
![Page 13: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/13.jpg)
Gaussian Mean and Variance
![Page 14: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/14.jpg)
The Multivariate Gaussian
determinant of covariance matrix
![Page 15: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/15.jpg)
Gaussian Parameter Estimation
Likelihood function
N observations of variable x
i.i.d. samples =independentand identicallydistributedsamples of a randomvariable
Goal: find parameters(mean and variance)that maximize the product of the bluepoints
![Page 16: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/16.jpg)
Maximum (Log) Likelihood
Maximizing with respect to µ
Maximizing with respect to variance
![Page 17: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/17.jpg)
Properties of and
Biased estimator
![Page 18: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/18.jpg)
Maximum Likelihood Estimation
Bias is at the core of the problem of MLE
![Page 19: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/19.jpg)
Curve Fitting Re-visited
![Page 20: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/20.jpg)
Maximum Likelihood I
Maximize log likelihood
Surprise! Maximizing log likelihood is equivalent to minimizing sum of square error function!
![Page 21: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/21.jpg)
Maximum Likelihood II
Determine by minimizing sum-of-squares error, .
Maximize log likelihood
![Page 22: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/22.jpg)
Predictive Distribution
![Page 23: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/23.jpg)
MAP: A Step towards Bayes
Determine by minimizing regularized sum-of-squares error.
Maximum posterior
prior over parameters
hyper-parameter
likelihood
![Page 24: Probability Theory and Parameter Estimation I](https://reader033.fdocuments.in/reader033/viewer/2022042107/6256cf510e85872296388820/html5/thumbnails/24.jpg)
MAP: A Step towards Bayes
Determine by minimizing regularized sum-of-squares error.
Surprise! Maximizing posterior is equivalent to minimizing regularized sum of square error function!