Presented by: Mingyuan Zhou Duke University January 20, 2012
description
Transcript of Presented by: Mingyuan Zhou Duke University January 20, 2012
![Page 1: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/1.jpg)
Characterizing the Function Space for Bayesian Kernel Models
Natesh S. Pillai, Qiang Wu, Feng LiangSayan Mukherjee and Robert L. Wolpert
JMLR 2007
Presented by: Mingyuan ZhouDuke UniversityJanuary 20, 2012
![Page 2: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/2.jpg)
Outline
• Reproducing kernel Hilbert space (RKHS)• Bayesian kernel model
– Gaussian processes– Levy processes
• Gamma process• Dirichlet process• Stable process
– Computational and modeling considerations• Posterior inference• Discussion
![Page 3: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/3.jpg)
RKHS
In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space is a Hilbert space of functions in which pointwise evaluation is a continuous linear functional. Equivalently, they are spaces that can be defined by reproducing kernels.
http://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space
![Page 4: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/4.jpg)
A finite kernel based solution
The direct adoption of the finite representation is not a fully Bayesian model since it depends on the (arbitrary) training data sample size . In addition, this prior distribution is supported on a finite-dimensional subspace of the RKHS. Our coherent fully Bayesian approach requires the specification of a prior distribution over the entire space H.
![Page 5: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/5.jpg)
Mercer kernel
![Page 6: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/6.jpg)
Bayesian kernel model
![Page 7: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/7.jpg)
![Page 8: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/8.jpg)
Properties of the RKHS
![Page 9: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/9.jpg)
Properties of the RKHS
![Page 10: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/10.jpg)
Bayesian kernel models and integral operators
![Page 11: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/11.jpg)
![Page 12: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/12.jpg)
Two concrete examples
![Page 13: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/13.jpg)
Two concrete examples
![Page 14: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/14.jpg)
Bayesian kernel models
![Page 15: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/15.jpg)
Gaussian processes
![Page 16: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/16.jpg)
Levy processes
![Page 17: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/17.jpg)
Levy processes
![Page 18: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/18.jpg)
Poisson random fields
![Page 19: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/19.jpg)
Poisson random fields
![Page 20: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/20.jpg)
Dirichlet Process
![Page 21: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/21.jpg)
Symmetric alpha-stable processes
![Page 22: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/22.jpg)
Symmetric alpha-stable processes
![Page 23: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/23.jpg)
Computational and modeling considerations
• Finite approximation for Gaussian processes
• Discretization for pure jump processes
![Page 24: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/24.jpg)
Posterior inference
• Levy process model
– Transition probability proposal– The MCMC algorithm
![Page 25: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/25.jpg)
![Page 26: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/26.jpg)
Classification of gene expression data
![Page 27: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/27.jpg)
Classification of gene expression data
![Page 28: Presented by: Mingyuan Zhou Duke University January 20, 2012](https://reader035.fdocuments.in/reader035/viewer/2022062801/56814400550346895db094d7/html5/thumbnails/28.jpg)
Discussion• This paper formulates a coherent Bayesian perspective for
regression using a RHKS model.• The paper stated an equivalence under certain conditions of
the function class G and the RKHS induced by the kernel. This implies: – (a) a theoretical foundation for the use of Gaussian processes, Dirichlet
processes, and other jump processes for non-parametric Bayesian kernel models.
– (b) an equivalence between regularization approaches and the Bayesian kernel approach.
– (c) an illustration of why placing a prior on the distribution is natural approach in Bayesian non-parametric modelling.
• A better understanding of this interface may lead to a better understanding of the following research problems:– Posterior consistency– Priors on function spaces– Comparison of process priors for modeling– Numerical stability and robust estimation