MedLDA: Maximum Margin Supervised Topic Models for Regression and Classification
Topic 11: Matrix Approach to Linear Regression
-
Upload
zelenia-melton -
Category
Documents
-
view
231 -
download
17
description
Transcript of Topic 11: Matrix Approach to Linear Regression
![Page 1: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/1.jpg)
Topic 11: Matrix Approach to Linear
Regression
![Page 2: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/2.jpg)
Outline
• Linear Regression in Matrix Form
![Page 3: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/3.jpg)
The Model in Scalar Form• Yi = β0 + β1Xi + ei
• The ei are independent Normally distributed random variables with mean 0 and variance σ2
• Consider writing the observations: Y1= β0 + β1X1 + e1
Y2= β0 + β1X2 + e2
:
Yn= β0 + β1Xn + en
![Page 4: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/4.jpg)
The Model in Matrix Form
n
2
1
n10
210
110
n
2
1
X
X
X
Y
Y
Y
![Page 5: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/5.jpg)
The Model in Matrix Form II
n
2
1
1
0
n
2
1
n
2
1
X 1
X 1
X 1
Y
Y
Y
![Page 6: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/6.jpg)
The Design Matrix
n
2
1
2n
X1
X1
X1
X
![Page 7: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/7.jpg)
Vector of Parameters
1
0
12
![Page 8: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/8.jpg)
Vector of error terms
n
2
1
1n
ε
![Page 9: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/9.jpg)
Vector of responses
n
2
1
1n
Y
Y
Y
Y
![Page 10: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/10.jpg)
Simple Linear Regression in Matrix Form
1n121nY
XY
2n
![Page 11: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/11.jpg)
Variance-Covariance Matrix
}{Y}Y,{Y}Y,{Y
}Y,Y{
}{Y }Y,{Y
}Y,{Y}Y,{Y}Y{
)(
n2
1-nn 1n
n1n
22
12
n12 112
2
Y
Main diagonal values are the variances and off-diagonal values are the covariances.
![Page 12: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/12.jpg)
Covariance Matrix of e
nn
2
n
2
1
nn
2
Cov}{
Iε
Independent errors means that the covariance of any two residuals is zero. Common variance implies the
main diagonal values are equal.
![Page 13: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/13.jpg)
Covariance Matrix of Y
nn
2
n
2
1
nn
2 I
Y
Y
Y
Cov{Y}
![Page 14: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/14.jpg)
Distributional Assumptions in Matrix Form
• e ~ N(0, σ2I)• I is an n x n identity matrix• Ones in the diagonal elements specify that the
variance of each ei is 1 times σ2
• Zeros in the off-diagonal elements specify that the covariance between different ei is zero
• This implies that the correlations are zero
![Page 15: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/15.jpg)
Least Squares
• We want to minimize (Y-Xβ)(Y-Xβ)
• We take the derivative with respect to the (vector) β
• This is like a quadratic function
• Recall the function we minimized using the scalar form
![Page 16: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/16.jpg)
Least Squares
• The derivative is 2 times the derivative of (Y-Xβ) with respect to β
• In other words, –2X(Y-Xβ) • We set this equal to 0 (a vector of zeros)
• So, –2X(Y-Xβ) = 0• Or, XY = XXβ (the normal equations)
![Page 17: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/17.jpg)
Normal Equations
• XY = (XX)β
• Solving for β gives the least squares solution b = (b0, b1)
• b = (XX)–1(XY)
• See KNNL p 199 for details
• This same matrix approach works for multiple regression!!!!!!
![Page 18: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/18.jpg)
Fitted Values
Xb
X1
X1
X1
X
X
X
Y
Y
Y
Y1
0
n
2
1
n10
210
110
n
2
1
b
b
bb
bb
bb
![Page 19: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/19.jpg)
Hat Matrix
')'(
ˆ
')'(ˆ
ˆ
XXXXH
HYY
YXXXXY
XbY
1
1
We’ll use this matrix when assessing diagnostics in multiple regression
![Page 20: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/20.jpg)
Estimated Covariance Matrix of b
• This matrix, b, is a linear combination of the elements of Y
• These estimates are Normal if Y is Normal
• These estimates will be approximately Normal in general
![Page 21: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/21.jpg)
A Useful MultivariateTheorem
• U ~ N(μ, Σ), a multivariate Normal vector
• V = c + DU, a linear transformation of U
c is a vector and D is a matrix
• Then V ~ N(c+Dμ, DΣD)
![Page 22: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/22.jpg)
Application to b
• b = (XX)–1(XY) = ((XX)–1X)Y
• Since Y ~ N(Xβ, σ2I) this means the vector b is Normally distributed with mean (XX)–1XXβ = β and covariance
σ2 ((XX)–1X) I((XX)–1X) = σ2 (XX)–1
![Page 23: Topic 11: Matrix Approach to Linear Regression](https://reader033.fdocuments.in/reader033/viewer/2022061315/5681329c550346895d9938ce/html5/thumbnails/23.jpg)
Background Reading
• We will use this framework to do multiple regression where we have more than one explanatory variable
• Another explanatory variable is comparable to adding another column in the design matrix
• See Chapter 6