Bayes=(Prior,Posterior,evidence,probability) &curve fitting

8/9/2019 Bayes=(Prior,Posterior,evidence,probability) &curve fitting

1/63

Using Probability Theory

Curve Fitting &Multisensory Integration

Hannah Chen, Rebecca Roseman and Adam Rule | COGS 202 | 4.14.2


2/63

Overheard at Porters

“You’re optimizing for the wrong thing!”


3/63

What would you say?

What makes for a good model?

Best performance?Fewest assumptions?Most elegant?Coolest name?


4/63

Motivation Finding Patterns

Tools Probability Theory

Applications Curve FittingMultimodal Sensory Integra

Agenda


5/63

FINDING PATTERNSThe Motivation


6/63

You have data... now find patterns


7/63


Unsupervisedx = data (training)

y(x) = model

ClusteringDensity estimation


8/63



y(x) = model


Supervisedx = data (trainint = (target vectoy(x) = model

ClassificationRegression


9/63



y(x) = model


Supervisedx = data (trainint = (target vectoy(x) = model

ClassificationRegression


10/63

Important Questions

What kind of model is appropriate?What makes a model accurate?Can a model be too accurate?

What are our prior beliefs about the model?


11/63

PROBABILITY THEORYThe Tools


12/63

Properties of a distribution

x = eventp(x) = prob. of event

1.2.


13/63

Rules

Sum

Product


14/63

Example (Discrete)


15/63

Bayes Rule (review)

prior

evidence

likelihoodposterior


16/63

Probability Density vs. Mass Function

PDF PMF

continuous discrete

Intuition: How much probability f(x)concentrated near x per length dx, how

dense is probability near x

Intuition: Probability mass is sameinterpretation but from discrete point of

view: f(x) is probability for each point,whereas in PDF f(x) is probability for ainterval dx

Notation: p(x) Notation: P(x)

p(x) can be greater than 1, as long as

integral over entire interval is 1


17/63

Expectation & Covariance


18/63

CURVE FITTING Application


19/63

Observed Data: Given n points (x,y)

Curve Fitting


20/63

Observed Data: Given n points (x,t)Textbook Example): generate x uniformly from ra[0,1] and calculate target data t with sin(2πx) funcnoise with Gaussian distribution

Why?

Curve Fitting


21/63

Observed Data: Given n points (x,t)Textbook Example): generate x uniformly from ra[0,1] and calculate target d ata t with sin(2πx) funnoise with Gaussian distribution

Why?Real data sets typically have underlying regularity thatwe are trying to learn.

Curve Fitting


22/63

Observed Data: Given n points (x,y)

Goal: use observed data to predict new targetvalues t’ for new values of x’

Curve Fitting


23/63

Curve Fitting

Observed Data: Given n points (x,y)Can we fit a polynomial function with this data

What values for w and M fits this data well?


24/63

How to measure goodness of fit?

Minimize an error function

Sum of squares

of error


25/63

Overfitting


26/63

Combatting Overfitting

1. Increase data points


27/63



Data observations may have just been noisy

With more data, can see if data variation is due to noise or if is part ofunderlying relationship between observations


28/63



Data observations may have just been noisy

With more data, can see if data variation is due to noise or if is part ofunderlying relationship between observations

2. Regularization

Introduce penalty term

Trade off between good fit and penalty

Hyperparameter, , is input to model. Hyperparam will reduce overfittinin turn reducing variance and increasing bias (difference between

estimated and true target)


29/63

How to check for overfitting?

Training and validation subsetheuristic: data points > (multiple)(# parameters)

Training vs Testingdon’t touch test set until you are actuallyevaluating experiment!!


30/63

Cross-validation

1. Use portion, (S-1)/S, for training (white)

2. Assess performance (red)

3. Repeat for each run4. Average performance scores

4-fold cross-validation


31/63

Cross-validation

When to use?


32/63

Cross-validation

When to use?Validation set is small. If very little data, use S=number ofobserved data points


33/63

Cross-validation

When to use?Validation set is small. If very little data, use S=number ofobserved data points

Limitations:● Computationally expensive - # of training runs increase

by factor of S● Might have multiple complexity parameters - may lead

to training runs that is exponential in # of parameters


34/63

Alternative Approach:

Want an approach that depends only on size oftraining data rather than # of training runs

e.g.) Akaike information criterion (AIC),Bayesian information criterion (BIC, Sec 4.4)


35/63

Akaike Information Criterion (AIC)

Choose model with largest value for

best-fit loglikelihood

# of adjustablemodel parameters


36/63

Gaussian Distribution

This satisfies the two properties for a probabilitdensity! (what are they?)


37/63


38/63

Curve Fitting (ver. 1)

Assumption:1. Given value of x, corresponding target value t has aGaussian distribution with mean y(x, w)

2. Data { x,t } drawn independently from distribution:

likelihood =


39/63

Maximum Likelihood Estimation (MLE

Log likelihood:

What does maximizing the log likelihood looksimilar to?


40/63

Log likelihood:

What does maximizing the log likelihood looksimilar to?

wrt w:

Maximum Likelihood Estimation (MLE


41/63

Simpler example: use Gaussian of form

With Bayes’ can calculate posterior:

Maximum Posterior (MAP)


42/63


Determine w by maximizing posteriordistributionEquivalent to taking negative log of:

and combining the Gaussian & log likelihoodfunction from earlier...


43/63


Minimum of

What does this look like?


44/63


Minimum of


What is regularization parameter?


45/63


Minimum of


What is regularization parameter?


46/63

Bayesian Curve Fitting

However…

MLE and MAP are not fully Bayesian becausethey involve using point estimates for w


47/63

Curve Fitting (ver. 2)Given training data { x, t} and new point x,

predict the target value t

Assume parameters are fixed

Evaluate predictive distribution:


48/63


Fully Bayesian approach requires integratingover all values of w, by applying sum andproduct rules of probability

B i C Fi i


49/63

Bayesian Curve Fittingmarginalization

posterior

This posterior is Gaussian and can beevaluated analytically (Sec 3.3)

B i C Fi i


50/63


Predictive is Gaussian of form

with mean and variance and matrix

B i C Fi i


51/63


Need to define

Mean and variance depend on x as a result ofmarginalization

(not the case in MLE/MAP


52/63

MULTIMODAL SENSORYINTEGRATION

Application

T M d l


53/63

Two ModelsVisual Capture

p r o

b a b i l i t y

location

v

a

T M d l


54/63

Two ModelsVisual Capture MLE

location

p r o

b a b i l i t y v

a

Proced re


55/63

Procedure

Final Result


56/63

Final Result

location

p r o b a b

i l i t y v

a

The Math (MLE Model)


57/63


Likelihood



58/63


Likelihood

The Math (“Bayesian” Model)


59/63

The Math ( Bayesian Model)

Likelihood * Prior

Uniform Inverse Gamma

The Math (Empirical)


60/63

The Math (Empirical)

logistic function

likelihood

location estimates

weight constraint

Final Result


61/63

Final Result


62/63

QUESTIONS?

Two Models (Prediction)


63/63

Two Models (Prediction)

Visual Capture MLE

Visual Noise

V i s u a l

W e i g h

t

Visual Noise

V i s u a l

W e i g h

t

Bayes=(Prior,Posterior,evidence,probability) &curve fitting

Documents

Transcript of Bayes=(Prior,Posterior,evidence,probability) &curve fitting