Gaussian Process Regression for Dummies
description
Transcript of Gaussian Process Regression for Dummies
Gaussian Process Regression for Dummies
Greg CoxRichard Shiffrin
Continuous response measures
The problem
What do we do if we do not know the functional form?
Rasmussen & Williams, Gaussian Processes for Machine Learninghttp://www.gaussianprocesses.org/
Linear regression
baxy
bxy
Bayesian linear regression
Gaussian processes
A Gaussian process is a collection of random variables, any subset of which is jointly normally distributed.
Normal regression:assume functional form mean and covariance among data
Gaussian process regression:assume form of mean and covariance among data functional form
*** ,~| yNyybaxy
*** ,~|,~ yy NyyKNy
Covariance kernel
How much does knowledge of one point tell us about another point?
Returning to linear regression
Mean = Function of parametersCovariance = Uncertainty about parameters + Observation noise
Takeaways from linear regression
• Rather than work in “parameter space”, we can bypass it by just working in “data space”
• This allows us to worry only about how different data points relate to one another without needing to specify the parameters of the data generating process
• The posterior predictive distribution encapsulates our uncertainty about the data generating process
• The choice of covariance kernel—which says how different observations inform one another—implies certain properties of the data generating process
Posterior predictive distribution
So far, we have computed the posterior predictive via the parameters (e.g., b) of the data generating process. But, a Gaussian process may have an infinite number of parameters (q). How can we compute the posterior predictive in this case?
The covariance kernel to the rescue! Let’s say we don’t know the data generating process, but we assume all observations are drawn from the same Gaussian process (i.e., are multivariate normal) and have an idea about how observations can mutually inform one another, the covariance kernel k(x, x’). Then...
New data values f*(x*), given observed data f(x):
But these are all multivariate normal!
dpp xfxxfxfxxxf *** ,||,,
),(),(),(),(,),(),(~|
),(),(
),(),(,~
*1***1*
***
*
XXKXXKXXKXXKXXKXXKN
XXKXXK
XXKXXKN
xfxfxf
0xf
xf
*
*
Building a function
A hierarchical Bayesian approach
Spivey, Grosjean, & Knoblich, 2005
The GP model
Model structure
The GP model
Results
Results
Inflection points can indicate important changes in cognitive processing
Summary
• Gaussian process models offer a useful and extensible way of dealing with behavioral trajectories
• Able to model entire spectrum of dynamics
• Can be embedded in a generative model to infer attractors and inflection points
• Allow for deeper inferences about underlying cognitive processes