Simple and multiple regression analysis in matrix form
-
Upload
jerome-grimes -
Category
Documents
-
view
105 -
download
1
description
Transcript of Simple and multiple regression analysis in matrix form
![Page 1: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/1.jpg)
Simple and multiple regression analysis in matrix form Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R2
Test on parameters Covariance matrix of the Standard error of the
![Page 2: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/2.jpg)
Simple and multiple regression analysis in matrix form Tests on individual predictors Variance of individual predictors Correlation between predictors Standardized matrices Correlation matrices Sum of squares in Z R2 in Z
R2 between independent variables Standard error of in Z
![Page 3: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/3.jpg)
Least squareStarting from the general:
The method of least squares estimate of the beta parameter minimizing the sum of squares due to error.
In fact, if:
![Page 4: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/4.jpg)
You can estimate:
Least square
![Page 5: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/5.jpg)
Simple linear regression
![Page 6: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/6.jpg)
Simple linear regression
![Page 7: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/7.jpg)
Simple linear regression
![Page 8: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/8.jpg)
Simple linear regression
intercepts
slope
![Page 9: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/9.jpg)
Multiple regression Similar to the simple A single dependent variable (Y) Two or more independent
variables (X) Multiple correlation (rather than
simple) Estimation by least squares
![Page 10: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/10.jpg)
Simple linear regression (var.: 1 dep., 1 indep.)
Multiple linear regression (Var.:1 dep., 2 indep.)
intercepts errorIndependent variables
slope
Multiple regression
![Page 11: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/11.jpg)
Multiple regression matrix form
![Page 12: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/12.jpg)
Multiple regression matrix form
![Page 13: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/13.jpg)
Multiple regression matrix form
![Page 14: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/14.jpg)
X’X
inversa
Multiple regression matrix form
![Page 15: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/15.jpg)
Multiple regression matrix form
![Page 16: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/16.jpg)
![Page 17: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/17.jpg)
In matrix notation is briefly expressed :
Multiple regression with three predictors
![Page 18: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/18.jpg)
Multiple regression with three predictors
![Page 19: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/19.jpg)
Matrix form
![Page 20: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/20.jpg)
Matrix form
![Page 21: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/21.jpg)
Matrix form
![Page 22: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/22.jpg)
Matrix form
![Page 23: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/23.jpg)
General scheme
![Page 24: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/24.jpg)
General scheme
![Page 25: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/25.jpg)
The least squares method allows to check the following equality:
Sum squares
![Page 26: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/26.jpg)
Since in general:it's possible to derive that the sum of the squares of the distances of y from its average can be decomposed into the sum of squares due to regression and the sum of squares due to error, according to:
Sum squares
![Page 27: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/27.jpg)
Sum squares
It should be noted the equivalence of :
![Page 28: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/28.jpg)
Sum squares
![Page 29: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/29.jpg)
Sum squares
![Page 30: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/30.jpg)
In summary :
Sum squares
![Page 31: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/31.jpg)
R2
![Page 32: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/32.jpg)
Adjusted R2YY’
Because the coefficient of determination depends on both the number of observations (n) that the number of independent variables (k) it is convenient to correct by the degrees of freedom.
Adjusted R2YY’
In our example :
![Page 33: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/33.jpg)
Once a regression model has been constructed, it may be important to confirm the goodness of fit (R-squared )of the model and the statistical significance of the estimated parameters. Statistical significance can be checked by an F-test of the overall fit, followed by t-tests of individual parameters
Test on parameters
![Page 34: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/34.jpg)
You can test the hypothesis of differences with 0 of the parameters i taken together :
Test on parameters
![Page 35: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/35.jpg)
k= Number of columns of the matrix X excluding X0
n= Number of observations in y
Test on parameters
![Page 36: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/36.jpg)
Test on parameters
k= Number of columns of the matrix X excluding X0
n= Number of observations in y
![Page 37: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/37.jpg)
Covariance matrix of the
An estimate of the covariance matrix of the beta values result by:
We denote:
![Page 38: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/38.jpg)
Covariance matrix of the
Where the diagonal elements are an estimate of the variance of the single i
![Page 39: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/39.jpg)
Standard error of the
The standard error of the parameters can be calculated with the following formula:
where cii is the diagonal element inside the matrix(X’X)-1 corresponding to the parameter i .
![Page 40: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/40.jpg)
Standard error of the
Nota: quando il valore di cii è elevato il valore di sebi
cresce, indicando che la variabile Xi ha un alto coefficiente di correlazione multipla con le altre variabili X.
![Page 41: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/41.jpg)
Standard error of the
the increase in R2i led to a decreases of the
denominator of the ratio and, consequently, increases the value of the standard error of the parameter i.
The standard error of the i can also be calculated in the following way:
where
![Page 42: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/42.jpg)
With the standard error of measurement associated with each i you can make a t-test to verify:
Tests on individual predictors
![Page 43: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/43.jpg)
Tests on individual predictors
With the standard error of measurement associated with each i is also possible to estimate the confidence interval for each parameter:
![Page 44: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/44.jpg)
Tests on individual predictors
1. Calculate the SSreg for the model containing all the independent variables.
2. Calculate the SSreg for the model excluding the variable for which you want to test the significance (SS-i).
3. Perform an F-test with the numerator equal to the difference SSreg-SSi weighted for the difference between the degrees of freedom of the two models, and with denominator SSREs / (nk-1).
In order to conduct a statistical test on the regression coefficients is necessary:
![Page 45: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/45.jpg)
Tests on individual predictors
To test, for example, only the weight of the first predictor compared to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to the first predictor. From this follows immediately the calculation of SSi.
![Page 46: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/46.jpg)
Tests on individual predictors
![Page 47: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/47.jpg)
Tests on individual predictors
Same procedure is followed to test any subset of predictors.
Similarly we have:
![Page 48: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/48.jpg)
Tests on individual predictors
It is interesting to note that this test on a single predictor is equivalent to the t-test b1 = 0. When the numerator there is only one degree of freedom, that is in fact the equivalence:
![Page 49: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/49.jpg)
Summary table
On this occasion, none of the estimated parameters obtained statistical significance on the hypothesis i 0
![Page 50: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/50.jpg)
Variance of individual predictors Xi
Using the matrix X'X we can calculate the variance of each variable Xi .
![Page 51: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/51.jpg)
Variance of individual predictors Xi
![Page 52: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/52.jpg)
Covariance between predictors and the dependent variable
It is possible to calculate the covariance between the independent variables and the dependent variable according to:
![Page 53: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/53.jpg)
Covariance between predictors and the dependent variable
The correlation between the independent variables and the dependent variable is given by:
As we will see later the use of standardized matrices simplifies the calculation immediately.
![Page 54: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/54.jpg)
Test on multiple predictor You can perform a statistical test on a group
of predictors in order to verify the significance.
To do this, you use the formula specified above :
To test, for example, the weight of only the first and second predictors with respect to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to these predictors. From this follows immediately the calculation of SSi.
![Page 55: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/55.jpg)
Test on multiple predictor
![Page 56: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/56.jpg)
Correlation between predictors
Standard condition of independence between the variables Xi
![Page 57: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/57.jpg)
Correlation between predictors
Condition of dependence between variables Xi
Completely standardized solution.
![Page 58: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/58.jpg)
We denote by Ri. the multiple correlation of the variable Xi with the remaining variables, denoted by Xj
Correlation between predictors
The element cii represents the value of the diagonal of the matrix (X'X) -1 while S2
i is the variance of the variable Xi.
![Page 59: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/59.jpg)
In case you do not have the X'X matrix but you have the MSres and the standard error of the parameter i, the correlation between one X and the other one can be calculated in the following manner:
Correlation between predictors
![Page 60: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/60.jpg)
Correlation between predictors
![Page 61: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/61.jpg)
The X matrix and the y matrix can be converted into standardized scores by dividing the deviation of each element from the average for the appropriate standard deviation.
Standardized matrices
![Page 62: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/62.jpg)
In our example we have:
Standardized matrices
![Page 63: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/63.jpg)
Standardized matrices
With standardized variables is not necessary to include in the matrix Z the component 1 as the parameter 0 is equal to 0.
![Page 64: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/64.jpg)
The standardized coefficients can be obtained from those non-standardized using the formula:
The equation of the regression line becomes:
Standardized matrices
![Page 65: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/65.jpg)
Standardized matrices
In our example we have:
![Page 66: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/66.jpg)
Use standardized matrices allows to set the parameter 0 = 0. In fact, if the variables are standardized the intercept value for Y is 0, since all the means are equal to 0;Inoltre, essendo
the correlation between any two standardized variables is:
with i, j between 1 and k.
Standardized matrices
![Page 67: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/67.jpg)
Correlation matrices
If we multiply the matrix (Z'Z) for the scalar [1 / (n-1)] we obtain the correlation matrix R between the independent variables
![Page 68: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/68.jpg)
In our example we have:
Correlation matrices
![Page 69: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/69.jpg)
Correlation of Y with individual predictors
Similarly if the variable Y is also standardized and multiply the product by the scalar Z'Yz [1 / (n-1)] we obtain the correlation matrix ryi of the variable Y with its predictors Xi.
![Page 70: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/70.jpg)
Correlation of Y with individual predictors
![Page 71: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/71.jpg)
The solution of the system of normal equations of the line leads to the following equation:
The estimated values can be obtained using the equation:
Correlation of Y with individual predictors
![Page 72: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/72.jpg)
With standardized variables we have:
Starting from the general formulas it's possible to have the following simplified formulas:
Sum of squares in Z
![Page 73: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/73.jpg)
Sum of squares in Z
![Page 74: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/74.jpg)
Calculation of R2y.123
Having decomposed the variance component due to the regression and the component due to the residuals, it is immediate to calculate:
![Page 75: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/75.jpg)
Multiple correlation between the Xi.yz
If in general the squared multiple correlation of a variable with the other independent Xi is:
in the presence of standardized variables, it becomes:
where the element aii belongs to the diagonal of the matrix R-1.
![Page 76: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/76.jpg)
If you want to calculate the other two coefficients now you will have to do the following:
For example, the squared multiple correlation between the first variable X1 and the other two can be calculated in the following way:
Multiple correlation between the Xi.yz
![Page 77: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/77.jpg)
Standard error of z
The standard error of the standardized parameters is obtainable by the general formula:
![Page 78: Simple and multiple regression analysis in matrix form](https://reader038.fdocuments.in/reader038/viewer/2022102909/56812d94550346895d92afb2/html5/thumbnails/78.jpg)
Standard error of z
You now have all the elements to test the differences of individual predictors from 0, obtaining the same results obtained with the non-standardized variables.