OLS Assumptions and Goodness of Fit

A little warm-up

• Assume I am a poor free-throw shooter. To win a contest I can choose to attempt one of the two following challenges:

• A. Make three out of four free throws• B. Make six out of eight

• Which should I choose? Why?

Gauss-Markov Assumptions

• These are the full ideal conditions• If these are met, OLS is BLUE — i.e. efficient

and unbiased.• Your data will rarely meet these conditions–This class helps you understand what to do

about this.

Pop Quiz

• Take out a sheet of paper and write down all the Gauss-Markov assumptions.

Assumptions of Classical Linear Regression

• A1: The regression model is linear in parameters– It may not be linear in variables

Y=B1+B2Xi

Assumptions of Classical Linear Regression

• A1: The regression model is linear in parameters– It may not be linear in variables

Y=B1+B2X+B3X2

Assumptions of Classical Linear Regression

• A2: X values are fixed in repeated sampling • Think about an experiment with different

dosages assigned to different groups

• We can also do this if X values vary in repeated sampling, as long as cov(Xi, ui) = 0– See chapter 13 if you’re curious about the

details

What if we violate linearity?

• If you have a non-linear relationship between X and Y and you don’t include an X-squared or X-cubed term, what is the problem?

• A true relationship may exist between X and Y that you fail to detect.

• A2: X values are fixed in repeated sampling • Think about an experiment with different

dosages assigned to different groups

• We can also do this if X values vary in repeated sampling, as long as cov(Xi, ui) = 0

• Think about this as requiring random sampling

Expected Value of Errors is Zero

• A3: Mean value of ui = 0.– E[ui|Xi] = 0– E[ui] = 0 if X is fixed (non-stochastic)

• Its ok to have big errors, but we can’t be wrong systematically– We call that bias

What if the expected value of the errors is not zero?

• This would indicate specification error –Omitted variable bias, for example

Assumptions of Classical Linear Regression

• Homoskedasticity or Constant Variance of ui

What happens if we violate homoskedasticity?

• This is called heteroskedasticity. –Model uncertainty varies from observation to

observation.•Often true in cross-sectional data due to

omitted variable bias.

– See chapter 13 if you’re curious about the details of heteroskedasticity

No Autocorrelation

• A5: No autocorrelation between disturbances– cov(ui,uj |Xi, Xj) = 0

• The observations are sampled independently

What if we have autocorrelation?

• More or less always the case in panel data.– So we have panel-corrected standard errors,

etc. • Also sometimes the case if we sample multiple

children from the same family, or multiple regions from the same country, etc. – Clustered standard errors

Degrees of Freedom

• A6: Number of observations n must be greater than the number of parameters to be estimated – n > number of explanatory variables– AKA degrees of freedom

Not Enough Degrees of Freedom

• If you don’t have enough degrees of freedom, you can’t estimate your parameters– The smaller your sample size, the less precise

your estimates (i.e. large standard errors).– Unable to reject the null hypothesis of no

difference even if the true effect is large.

Variation but no Outliers

• A7: X must vary, but there must not be any outliers

What if there are outliers?

• Our model works too hard to fit these values, giving them effectively too much weight• This is the squared errors problem.

Correct specification• A8: Regression model is correctly

specified.– The correct variables are included– We have the correct functional form– Correct assumptions about the probability

distributions of Yi, Xi and ui.

No perfect multicollinearity• A9: With multiple regression, we add the

assumption of no perfect multicollinearity– The correlation between any two x variables <

1

No perfect multicollinearity• With perfect collinearity, we have to drop

one x variable to even estimate our betas.• With near-perfect collinearity, variance is

inflated–But estimates are not biased

Gauss-Markov Theorem

• When all those assumptions hold, OLS is BLUE– Best Linear Unbiased Estimator– “Best” means least variance (most efficient)– Unbiased means: E[�̂2] = �2

How “good” does it fit?• To measure “reduction in errors” we need a

benchmark for comparison.• The mean of the dependent variable is a

relevant and tractable benchmark for comparing predictions.

• The mean of Y represents our “best guess” at the value of Yi absent other information.

Sums of Squares

• This gives us the following 'sum-of-squares' measures:

• Total Variation = Explained Variation + Unexplained Variation

How well does our model perform?

• R squared statistic– = TSS-USS/TSS– =ESS/TSS• Bounded between 0 and 1• Higher values indicate a better fit

Questions• How do the fitted values of Y change if we

multiply X by a constant?• What if we add a constant to X?

Why do we have an error term• The error term includes the effect of all X

variables not in our model that still effect Y.– Parsimony, intrinsic randomness of humans, – Vague theory, measurement error, wrong

functional form

What does an error term imply?• If we run our project multiple times, we will

estimate a slightly different regression line every time.

How do we know if our test statistic is any good?

• OLS is an estimator– It calculates the slope of the sample

regression line (i.e. the SRF)• It gives us a test statistic (i.e. a p-value)– What does that mean?

• IF AND ONLY IF the assumptions of OLS are met, and the true slope of the population regression line is 0, there is an x percent chance we would estimate a slope this large in our sample regression.

Can we test that?• YES!• First, estimate our regression line, and

calculate the critical value (p=.05)• 2nd, lets make there be no relationship.– “shuffle” the data

• 3rd, re-estimate the regression line. Is the slope steeper than our critical value?

• Repeat steps 2 and 3 10,000 times. How often should the slope of the regression line be greater than the critical value I’ve

What does that tell us?• It tells us our type 1 error rate– How often would we reject the null when we

shouldn’t (i.e. when the null is true)• What about Type 2 errors?– How often would we fail to reject the null when

the true value of beta is actually B1? – To calculate that, we need the sample size,

the variance of x and y, all the OLS assumptions

– Or we can simulate it• Validity and Power