1 Topic 2 – Simple Linear Regression KKNR Chapters 4 – 7.
-
Upload
aubrie-harrington -
Category
Documents
-
view
218 -
download
0
description
Transcript of 1 Topic 2 – Simple Linear Regression KKNR Chapters 4 – 7.
1
Topic 2 – Simple Linear Regression
KKNR Chapters 4 – 7
2
Overview
Regression Models; Scatter Plots SAS GPLOT Procedure
Estimation and Inference in SLR SAS REG Procedure
ANOVA Table & Coefficient of Determination (R2)
3
Simple Linear Regression Model
We take n pairs of observations
The goal is to find a model that best fits with the data.
Model will be linear in terms of the parameters (betas). These won’t appear in exponents or anything unusual.
Allowed to be nonlinear in terms of predictor variables (we may transform these somewhat freely). We may also transform the response.
( ) ( ) ( )1 1 2 2, , , ,..., ,n nX Y X Y X Y
4
Simple Linear Regression Model (2)
Some sample models
Notice the betas always function in the same way, and the analysis will always proceed in the same way too (after we make whatever transformations we might need).
0 1
0 1
0 1
log
log
i i i
i i i
i i i
Y X
Y X
Y X
5
Simple Linear Regression Model (3)
Key question: How do you decide on the “best” form for the model?
Always view a scatter plot (use PROC GPLOT in SAS). Curvature in the plot will help you determine the need for a transformation on either X or Y.
Always consider residual plots. Some patterns in these plots will also indicate the need for transformation (more on this later).
6
Scatter Plot Approach
If you can look at a scatter plot and the data “look linear”, then likely no transformation is necessary. Try not to look for things that are not there.
If you see curvature, then some transformation may be appropriate:
Use scientific theory & experience Try transformations you think may work – look
at scatter plots of the transformed data to assess whether they do work.
7
Finding the “Best” Model
There is no “absolute” strategy. Some common mistakes (why are these bad?):
Try several different methods and simply take the one for which you get the best results (e.g. highest R2)
Over-fit the model by including lots of extra terms (e.g. squares, cubes, etc.) in hopes to get the curve to go through all of the data points (note that this would be MLR)
8
Collaborative Learning Activity
CLG #2.1-2.3First, make sure you read enough to understand the dataset we will be considering. Then, please try to answer these questions related to scatter plots.
9
Scatter Plot Examples (1)
Wi t h Est i mat ed Regr ess i on Li ne
800
900
1000
1100
1200
St at ewi de Expendi t ur es
3 4 5 6 7 8 9 10
10
Scatter Plot Examples (2)
Wi t h Es t i mat ed Regr ess i on Li ne
800
900
1000
1100
1200
Per cent age of El i gi bl e St udent s Taki ng SAT
0 10 20 30 40 50 60 70 80 90
11
Scatter Plot Examples (3)Wi t h Nonpar amet er i c Smoot h
800
900
1000
1100
1200
Per cent age of El i gi bl e St udent s Taki ng SAT
0 10 20 30 40 50 60 70 80 90
12
Scatter Plot Examples (4)Log Tr ans f or med Pr edi ct or Wi t h Nonpar amet er i c Smoot h
800
900
1000
1100
1200
Log- Tr ans f or med Per cent age of El i gi bl e St udent s Taki ng SAT
1 2 3 4 5
13
Comments on GPLOT
Utilize SYMBOL, AXIS, and TITLE statements to make your plots look nice.
ORFONT provides a good symbol-set. You can also manipulate the COLOR of symbols in order help the viewer differentiate groups.
Be careful to remember that SAS reuses these statements, so you will need to redefine them as necessary.
14
SAS ORFONT
15
Fitting the SLR Model
• Once we decide on the form of our model, we need to estimate the parameters that yield the “best” fit.
• The arithmetic involved is accomplished with a computer, but it is useful to have some understanding of the how the estimates are calculated.
16
The SLR Model
Whatever the transformations may be, our model is in the form of a straight line:
Epsilon represents the inherent variation (or error) in the model.
Model involves two other parameters (unknown, but fixed in value): slope (change in y for a one unit change in x) intercept (value of y for x = 0; usually not particularly
interested in this)
0 1Y X
17
Observations
An observation Y at a particular X is a random variable. So you can think of each observation as having been drawn from a normal distribution centered at and having standard deviation .
Be careful to remember that these parameters (represented by greek letters) are fixed – but can never be known exactly. We can only estimate them.
0 1X
18
Graphical Representation
19
Model Assumptions
We make three assumptions on the error term in our model. A simple statement of these assumptions is that
The assumptions on the errors apply to both regression and ANOVA and we will be assuming these throughout the course.
For regression, we also make a 4th assumption that our model (in this case linear relationship between X and Y) is appropriate.
2~ 0,IIDi N
20
Assumptions on Errors
Constance Variance (Homoscedasticity) – the variance associated to the error is the same for ANY value of X.
Normality – the errors follow a normal distribution with a mean of zero.
Independence – the errors (and hence also the responses) are statistically independent of each other.
21
Checking Your Assumptions
Reminder: We can never know the exact values of the errors because we can never know the true regression equation.
We can (and will) estimate the errors by the residuals. The residuals can then be used (mostly in graphical analyses) to assess the assumptions – giving us some idea of whether the assumptions of our model are satisfied. More on this later...
22
Estimation of the “Best” Line
We want to obtain estimates of the parameters and (remember, we can never know them exactly).
Notation: Generally, I will use lower case English letters to represent estimates for parameters. You may also see hat-notation. For example, if is a parameter, would be its estimated value from data.
Our estimates will be denoted . The residuals will be denoted by .
0 1
0 1,b bie
23
Parameter Estimates
Key Point: Parameters are fixed, but their estimates are random variables. If we take a different sample, we’ll get a different estimate.
Thus all of the estimates we compute will have associated standard errors that we may also estimate.
The method of least squares is used to obtain both parameter estimates and standard errors. This method is desirable because the estimates are unbiased, minimum variance estimates.
0 1, , ib b e
24
Least-squares Method
The least squares method obtains the estimated regression line that minimizes the sum of the squared residuals (also called the SSE or sum of squares error).
Another way to think of this is that the least squares estimates allow us to explain as much of the variation in Y as we possibly can using X as a predictor. The SSR (sums of squares due to our model) is maximized.
22 ˆi i iSSE e Y Y
25
Least Squares Estimates
The estimates have formulas in terms of the data:
11
2
1
0 1
0 1
2 2
ˆ
ˆ 2
n
i ii
n
ii
i i i i i
X X Y Yb
X X
b Y b X
e Y Y Y b b X
s SSE n
26
Least Squares Estimates (2)
It is not important to memorize these formulas. I won’t ask you to calculate a parameter estimate by hand from the data. We have computers for this.
What is important will be to understand that, because the Yi are random variables, and because all of these estimates depend on the Yi, the estimates themselves will also be random variables. Thus we may estimate their standard errors, develop confidence intervals, and draw statistical inferences.
27
Inference about the Slope
• A non-zero slope implies a linear association between the predictor and response.
• In some experimental cases the relationship may be causal as well.
• Thus statistical inference for the slope is quite important.
28
Inference About the Slope
The first thing to remember is that b1 is a random variable. In order to do inference, we must first consider that...
is normally distributed (why?)
The standard error associated to b1 is
2
1
1n
ii
MSE
X XMSEs bSSX
1
12
1
n
i ii
n
ii
X X Y Yb
X X
29
Slope Inference (2)
For testing , the statistic
has a t-distribution with n – 2 degrees of freedom when the null hypothesis is true.
A two-sided confidence interval for the slope will be:
0 1:H k
1
1
b kTs b
1 2,1 2 1nb t s b
30
Slope Inference (3) Want to determine: Does X help explain Y
through a linear model???? If we reject the null hypothesis ,
then we may conclude that there is a linear association between X and Y Must have assumptions satisfied.
Key point: Failing to reject does not necessarily allow us to conclude that X is unimportant Maybe we need a bigger sample to give better power
0 1: 0H
31
Slope Inference (4)
Another Key Point: Violations of the model assumptions may invalidate the significance test.
In particular, if there is a nonlinear association or some type of dependence issue, the SLR model should not be used.
See pages 65 for some pictures illustrating this.
32
Experimental Control
In some situations you have experimental control over your predictor variable. Thus you have some control over the SE for the slope:
Making SSX large will decrease the SE of your estimate for the slope. Do this by spreading you chosen X values further apart.
Increasing n (and hence increasing degrees of freedom) may also help to decrease the SE.
1MSEs bSSX
33
Inference About the Intercept
Hypothesis tests and confidence intervals may be constructed similar to inference for the slope. See page 63.
Key point: Unless the observed predictor is often in the neighborhood of zero, we have no reason to be interested in the intercept.
In fact, the intercept will usually just be an artifact of the model. And if the scope of the model does not include zero, there is no reason to even worry whether the value of b0 makes sense.
34
Further Inference
Confidence Intervals for the Mean Response Prediction Intervals
35
The Predicted Value
The line describes the mean population response for each value of the predictor. If an association exists, then the mean response depends on the value of X.
The predicted value of Y at a given X = x0 is
Reminder: Notation may differ some from the text – I try to keep our notation as simple as possible.
0 0 1 0xY b b x
36
C.I. for the Mean Response or Prediction Interval?
If you are trying to predict for a group C.I. for the Mean Response
Example: Trying to predict the average blood pressure for all 40 year olds
Interval is usually narrower
If you are trying to predict for a single observation Prediction Interval
Example: Trying to predict the blood pressure for a single 40 year old
Interval is usually much wider because of individual variation Some 40 year olds will have much higher or lower B.P.s
37
*Calculations of SE for Mean Response
First step is to write in terms of estimates. We also need to use a small trick to avoid worrying about a covariance between the two parameter estimates.
0 0 1 0
1 1 0
0 1
xVar Y Var b b x
Var Y b x b x
Var Y x x b
38
*Calculations of SE for Mean Response (2)
We know the variances for Y-bar and b1. And it turns out that, even though Y-bar is used in the calculation of b1, the two are still independent. So the variance of the sum is the sum of the variances:
0
20 1
2 220
202
ˆ
1
xVar Y Var Y x x Var b
x xn SSX
x xn SSX
39
SE for the Mean Response
The mean response is a random variable since b0 and b1 are random. Hence it will have a standard error.
It is good to have some understanding of how this works, so we will look very briefly at the calculations (you should just try to follow them, but not worry about memorizing them)
0
201
x
x xs Y MSE
n SSX
40
Confidence Intervals for Mean Response
You sometimes want to get confidence intervals for the mean response. Because we have estimated both the mean and variance, the t-distribution applies (n – 2 degrees of freedom). The CI for a given value of X is:
0 02,1 / 2ˆ ˆx n xY t s Y
41
SE for the Prediction Interval
So the prediction variance is the sum of these two components:
0 0
0
2
2
0 2 2
2
0
ˆ
1
11
x x
x
Var pred Var Y
X Xn SSX
X Xs pred MSE
n SSX
42
Prediction Intervals (1)
Now consider predicting a new observation at X = x0. Our point estimate for this would just be the point on the regression line, .
Our prediction interval will be of the same form as for the mean response, but with a different standard error:
0xY
0 02,1 / 2x n xY t s pred
43
Why the difference for the Prediction Interval?
The key to understanding the standard error for prediction is to understand the random components involved. (1) The regular variance associated to getting a
predicted value (same as the CI mean response error)
(2) The individual error for a single observation It basically gives us an extra σ2 piece Think of the NORMAL DISTRIBUTION centered
around the regression line (see slide 18)
44
Multiple Confidence Intervals
We did intervals for 40 year olds, both group and individual, what about 30? 35?
Getting multiple CI’s presents a similar problem to multiple hypothesis tests. We would expect one errant CI for every 20 CI’s that we obtain at 95% confidence. Thus some adjustment may need to be made.
Bonferroni is too conservative here, because these CI’s are actually dependent and it is possible to take advantage of this.
45
Confidence Bands The solution is to change our critical value. Instead of
using T, we use a critical value related to the F distribution:
This allows us to produce CI’s for the mean response at any and all possible values for the predictor variable. Hence we may also use this to draw confidence bands around the regression line.
Useful trick: For significance level 0.05, the value of W is approximately 0.6 more than the value of T. This is slightly conservative, but simplifies computation.
2, 2,12 nW F
46
Interpolation vs. Extrapolation
Interpolation (x0 within the domain of the observed X’s) is generally ok if the assumptions are satisfied.
Extrapolation (x0 outside the domain of the observed X’s) is usually a bad idea.
No assurance that linearity continues outside the observed domain.
Example – Height regressed on age in children.
47
Key Concepts
The standard error formulas for the slope, the regression line, and prediction are related – your goal should be to understand these relationships.
You should also be able to construct CI’s and do hypothesis tests.
Point Estimates Critical Values (know how to look these up) Standard Errors (generally would not be
asked to compute these, just use them)
48
SAS Review
proc reg data=sat; model score=expend / clb clm cli; id state expend; output out=fit r=res p=pred;
49
Collaborative Learning Activity
Please complete problems 2.4 (constructing CI’s) and 2.5 (interpreting regression output) on the handout.
50
Output: ANOVA Table
Sum of Mean Source DF Squares Square F Value Pr > F Model 1 39722 39722 8.13 0.0064 Error 48 234586 4887.2 Total 49 274308 Root MSE 69.90851 R-Square 0.1448 Dependent Mean 965.92000 Adj R-Sq 0.1270 Coeff Var 7.23751
51
Output: Parameter Estimates Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > |t| Intercept 1 1089.29372 44.38995 24.54 <.0001 expend 1 -20.89217 7.32821 -2.85 0.0064 Variable DF 95% Confidence Limits Intercept 1 1000.04174 1178.54569 expend 1 -35.62652 -6.15782
52
Output: Output Statistics Output Statistics Dependent Predicted Std Error Obs state expend Variable Value Mean Predict 95% CL Mean 9 Louisian 4.761 1021 989.8 12.9637 963.8 1016 10 Minnesot 6 1085 963.9 9.9109 944.0 983.9 11 Missouri 5.383 1045 976.8 10.6015 955.5 998.1 12 Nebraska 5.935 1050 965.3 9.8890 945.4 985.2 Obs state expend 95% CL Predict Residual 9 Louisian 4.761 846.9 1133 31.1739 10 Minnesot 6 822.0 1106 121.0593 11 Missouri 5.383 834.7 1119 68.1689 12 Nebraska 5.935 823.3 1107 84.7013
53
ANOVA Table
ANOVA stands for analysis of variance. We use an ANOVA table in regression to organize our estimates of different components of variation. It is important to understand how this works for SLR since we will use ANOVA tables for MLR and ANOVA procedures as well.
54
ANOVA Table
Table consists of variance estimates used to assess the following two questions:
Is there an linear association between the response and predictor(s)?
How “strong” is that linear association?
We need to start by understanding the different components of variation for a single data point.
55
Components of Variation
56
Combining Over All Data
We might look at the total deviation as follows:
But we cannot simply add deviations across data points. Why? Options?
ˆ ˆi i i iY Y Y Y Y Y
57
Combining Over All Data (2)
Squared deviations are chosen because it turns out that they can be used to estimate variances. It also (conveniently) turns out that:
2 22
1 1 1
ˆ ˆ
n n n
i i i ii i i
TOT R E
Y Y Y Y Y Y
SS SS SS
58
Sums of Squares
The total sums of squares (SST) represents the total available variation that could be explained by the predictor (that not already explained by Ybar).
We break this into the two components: Model/Regression sums of squares (SSR) is the
part that is explained by the predictor. Error sums of squares (SSE) is the part that is
still left unexplained.
59
Degrees of Freedom
Each SS has an associated degrees of freedom. For simple linear regression,
DFT = n – 1
DFR = 1
DFE = n – 2
Always have In general, you lose one degree of freedom for each
parameter you estimate. Since we estimate Ybar before we start, dfTOT is n – 1.
T R EDF DF DF
60
Degrees of Freedom (2)
It is important to understand how DF are assigned with the models that we will be discussing. Some key principles:
DF Total is always 1 less than the number of observations.
You should next determine DF for the model. For regression, each continuous variable requires a slope estimate and takes 1 DF.
Lastly, the error DF is determined by subtraction (avoid memorization of formulas).
61
Mean Squares
A mean square is a SS divided by its associated degrees of freedom.
These are the actual variance estimates:2
2
20 1
(population variance)1
(error variance)2
under null hyp. H : 01
Y
R
SSTs MSTnSSEs MSEnSSs MSR
62
F-tests
Because both MSR and MSE estimate under the null hypothesis, we may utilize their ratio in order to test whether there is a linear association.
If the null hypothesis is true, the statistic
will have an F distribution with 1 and n – 2 DF. Note: To get your DF for the F-test, simply use
the DF for the associated mean squares.
2
MSRFMSE
63
Relationship of F to T
For SLR the F test is identical to the t-test for the slope as in fact:
Additionally you will find that in terms of critical values, .
1
22
ˆ1F S T
21,v vF t
64
Example Sum of Mean Source DF Squares Square F Value Pr > F Model 1 39722 39722 8.13 0.0064 Error 48 234586 4887.2 Total 49 274308 Root MSE 69.90851 R-Square 0.1448 Dependent Mean 965.92000 Adj R-Sq 0.1270 Coeff Var 7.23751 Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > |t| Intercept 1 1089.29372 44.38995 24.54 <.0001 expend 1 -20.89217 7.32821 -2.85 0.0064 Variable DF 95% Confidence Limits Intercept 1 1000.04174 1178.54569 expend 1 -35.62652 -6.15782
65
Example (2)
For SLR, the F-test in ANOVA Table is exactly the same as the test for zero slope.
Note 8.13 = (-2.85)2.
Caution: When we get into multiple regression, if the F-test has a small p-value this is a good start, but not the end! In multiple regression, the F-test may be thought of as a test for “model significance”. But it doesn’t tell us which variable(s) are important and which are not.
66
Other Statistics from REG
R-square and Adjusted R-square help us to assess the “strength” of the linear relationship.
The coefficient of variation is calculated using
It measures the variation as a percentage of the mean.
100CV MSE Y
67
Coefficient of Determination
The coefficient of determination (R2) gives us some idea as to the strength of the regression relationship.
68
Coefficient of Determination (R2)
Reflects the variation in Y that is explained by the regression relationship as a percentage of the total:
With perfect linear association, SSE will be zero and R2 will be 1.
If no linear association, SSE will be the same as SST and R2 will be 0.
2 1R E
TOT TOT
SS SSRSS SS
69
Common Misconceptions
Steeper slope means bigger R2. This is not true. In fact R2 has nothing to do with the magnitude of the slope for our regression line.
The larger the value of R2, the better the model. This is also not true. R2 says nothing about appropriateness of model (see page 98).
R2 could be 0 but there could be a non-linear association between X & Y
R2 could be near 1 while a curvilinear model would be more appropriate (scatterplots will generally reveal this)
70
The Correlation Coefficient (r)
Takes the sign of the slope and, for SLR, is simply the square root of R2.
Dimensionless – ranges between -1 and 1. Symmetric – interchanging X & Y will not
change the correlation between them.
72
SAT Example: Interpret R2
Simple interpretation: 14.8% of the variation in SAT scores is explained by the expenditures.
Reality Check: (1) though significant, this is not a very strong relationship and (2) the slope parameter is negative, suggesting that increasing expenditures is associated with a decrease in the average score!
73
Adjusted R2
Uses the mean squares to adjust (penalize) for the number of parameters in the model
We’ll discuss this more in multiple regression as it really isn’t important for SLR.
2 /1 1
/ 1E
aTOT
SSE n p MSRSST n MS
74
Regression Diagnostics
Check Your Assumptions!!!
75
Regression Diagnostics
Assumptions Correct Model (linearity) Independent Observations Normally Distributed Errors Constant Variance
Checking these generally involves PLOTS of the residuals and predicted values.
Residual = observed – predictedˆ
i i ie Y Y
76
Regression Diagnostics (2)
Key Point: Most assumption checks may be done visually by looking at various plots. They may also be done using statistical tests. Looking at plots is generally easier!
So the general formula is to check the plots, and if you still have questions then perhaps consider the statistical tests.
77
Checking Normality
Histogram or Box-plot of Residuals Is the histogram bell-shaped? Is the box-plot symmetric?
Normal Probability / QQ plot This is the method we would normally use.
Ordered residuals are plotted against cumulative normal probabilities and the result should be approximately linear.
PROC UNIVARIATE: QQPLOT statement
Shapiro-Wilks or Kolmogorov-Smirnov Test
78
SAS Code: QQPlot
proc univariate data=fit noprint; var res; title 'Normal Probability Plot'; qqplot res / normal(l=1 mu=est sigma=est);
79
Constancy of Variance
Plot the residuals against fitted (predicted) values
Check to see if size of residual is somehow associated with predicted value.
Megaphone shapes are indicative of a violation.
Bartlett’s or Levene’s Test Statistical tests are generally sensitive to
violations of normality and cannot be used if the normality assumption is not met.
80
Plot: Residuals vs. Fitted Values
proc gplot data=fit; plot res*pred /vref=0;
81
Checking Independence
This is the hardest assumption to check. One check on this assumption is to simply think of
how the data are collected. Ask the question: Is there anything in the collection of data that could lead to dependent responses?
Plot the residuals over time (if applicable). Is there a “drift” or other pattern as trials proceed?
Durbin-Watson Test
82
Other Issues
Linearity Assumption: A nonlinear pattern in the residuals vs. predicted values plot suggests that we need to revise our assumption of a linear parametric relationship between X and Y.
Outliers: These will show up in rather obvious ways on the various plots.
83
When the assumptions are violated...
Discarding data is almost always the wrong thing to do. Some things you can do are... Consider transformations of the data.
Transformations of the response variable [e.g. Log(Y)] often help with normality and/or constancy of variance issues.
Transformations of the predictor variable(s) may solve nonlinearity issues
Lastly, we may consider other more complex models.
84
When outliers are present...
Some formal tests exist to classify outliers (we’ll talk about them later)
Investigate – don’t eliminate Lacking a very good reason (e.g. experimenter
made error in recording the data) you should never be throwing an outlier away.
One good thing to do is to try to figure out how much effect the outlier has on your various estimates (we’ll also learn how to do this later)
85
Collaborative Learning Activity
Please discuss problem #2.6 from the handout.
86
Questions?
87
Upcoming in Topic 3...
Multiple Regression Analysis
Related Reading: Chapter 8