Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1...
Transcript of Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1...
![Page 1: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/1.jpg)
Econometrics
1
Lecture 3: Properties of the OLS Estimator For the OLS estimator yXXXb ')'( 1−= define:
• Vector of OLS residuals
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
−
−
=−=
∑
∑
=
=
K
kknkn
K
kkk
bxy
bxy
Xbye
1
111
![Page 2: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/2.jpg)
Econometrics
2
• Vector of OLS fitted or predicted values
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
==
∑
∑
=
=
K
kknk
K
kkk
bx
bx
Xby
1
11
ˆ
![Page 3: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/3.jpg)
Econometrics
3
Residuals and fitted values have properties
a. 0' =eX
b. If the regression has an intercept
bxy '= with
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡=
=
∑
∑
=
=
n
iiK
n
ii
xn
xn
x
1
11
1
)1(1
c. If the regression has an intercept
yy ˆ=
![Page 4: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/4.jpg)
Econometrics
4
d. If the regression has an intercept
∑∑ ∑== =
+−=−n
ii
n
i
n
iii eyyyy
1
22
1 1
2 )ˆ()(
![Page 5: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/5.jpg)
Econometrics
5
Proofs and remarks: Ad a. 0' =eX Proof: 0'')('' =−=−= XbXyXXbyXeX This implies that each column of X is orthogonal to vector e , i.e. for
Kk ,,1…=
∑=
=n
iiik ex
10
If regression has intercept (and hence X has column equal to ι , a vector of 1-s), then
∑=
===n
ii eee
10'ι
![Page 6: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/6.jpg)
Econometrics
6
The fundamental assumption in CLR model 0)|( =XE ε implies that 0)'( =εXE Hence OLS makes sample analog of )'( εXE equal to 0.
![Page 7: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/7.jpg)
Econometrics
7
Note MyyXXXXIXbye =−=−= − )')'(( 1 with ')'( 1 XXXXIM −−= The matrix M has a number of properties 0=MX
MM =' M is symmetric MM =2 M is idempotent
![Page 8: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/8.jpg)
Econometrics
8
Also PyyXXXXXby === − ')'(ˆ 1 with ')'( 1 XXXXP −= Note
PP =' P is symmetric PP =2 P is idempotent Because Xby =ˆ the matrix P projects yon the space spanned by the columns of X . For this reason P is called the least squares projection matrix. Projection matrices are idempotent, because if we project again nothing changes.
![Page 9: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/9.jpg)
Econometrics
9
Because 0' =eX and M is also a projection matrix, e is the projection of y on space orthogonal to the columns of X . Note eyy += ˆ Also 0'''ˆ == eXbey i.e. y and e are orthogonal and y is the sum of two orthogonal vectors.
![Page 10: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/10.jpg)
Econometrics
10
In graph (one observation with 2xxyb = )
y
bxy =ˆ
bxye −=
![Page 11: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/11.jpg)
Econometrics
11
Ad b: bxy '= Proof: If we write [ ]KxxX 1ι= with kx the −k th column of X , then the normal equations can be written as
y
x
xXb
x
x
KK⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
′
′=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
′
′ 22
'' ιι
![Page 12: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/12.jpg)
Econometrics
12
The first equation is yXb '' ιι = or
yn
Xbn
'1'1 ιι =
or ybx ='
![Page 13: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/13.jpg)
Econometrics
13
Ad c: yy ˆ= Proof: Because eyy += ˆ we have
en
yn
yn
'1ˆ'1'1 ιιι +=
or
yy =ˆ In words: sample average of y is predicted exactly.
![Page 14: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/14.jpg)
Econometrics
14
Ad d: ∑∑ ∑== =
+−=−n
ii
n
i
n
iii eyyyy
1
22
1 1
2 )ˆ()(
Proof: Because iii eyyyy +−=− ˆ , we have
∑ ∑∑∑= ===
−++−=−n
i
n
iiii
n
ii
n
ii yyeeyyyy
1 1
2
1
2
1
2 )ˆ(2)ˆ()(
The last term is 0.
![Page 15: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/15.jpg)
Econometrics
15
Terminology
∑=
−n
ii yy
1
2)( Total Sum of Squares (TSS)
∑=
−n
ii yy
1
2)ˆ( Explained Sum of Squares (ESS)
∑=
n
iie
1
2 Residual Sum of Squares (RSS)
Hence RSSESSTSS +=
![Page 16: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/16.jpg)
Econometrics
16
If we divide by TSS
TSSRSS
TSSESS
−= 1
Coefficient of determination
∑
∑
=
=
−
−== n
ii
n
ii
yy
yyR
1
2
1
2
2
)(
)ˆ(
TSSESS
This is a measure of goodness-of-fit.
![Page 17: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/17.jpg)
Econometrics
17
Note 10 2 ≤≤ R For the extreme values
XbynieeR in
ii =⇔==⇔=⇔= ∑
=,,1,001
1
22 …
i.e. a perfect fit.
0,
ˆ0)ˆ(0
21
1
222
====⇔
⇔=⇔=−⇔= ∑=
K
in
ii
bbyb
yyyyR
i.e. varying regressors Kxx ,,2 … do not help to explain y .
![Page 18: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/18.jpg)
Econometrics
18
Because ∑∑==
−−=−n
iii
n
ii yyyyyy
11
2 ))(ˆ()ˆ(
2
1
2
1
2
1
1
2
1
2
2
1
2
1
2
1
2
2
)ˆ()(
))(ˆ(
)ˆ()(
)ˆ(
)(
)ˆ(
∑∑
∑
∑∑
∑
∑
∑
==
=
==
=
=
=
−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
=
=−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=−
−=
n
ii
n
ii
n
iii
n
ii
n
ii
n
ii
n
ii
n
ii
yyyy
yyyy
yyyy
yy
yy
yyR
The final expression is the sample correlation of y and y .
![Page 19: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/19.jpg)
Econometrics
19
Note in 2R we use deviations from sample average, because
∑
∑
=
=n
ii
n
ii
y
y
1
2
1
2ˆ
can be made close to 1 by adding a constant to y .
![Page 20: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/20.jpg)
Econometrics
20
Partitioned regression Consider uXy += β Partition X as
[ ]
21
21
KnKnKnXXX
×××=
2
1
2
1
KK
⎥⎦
⎤⎢⎣
⎡=
ββ
β
with KKK =+ 21 . Hence we can write εββ ++= 2211 XXy
![Page 21: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/21.jpg)
Econometrics
21
What is the formula for the OLS estimator of 1β ? Define (compare with M defined previously) 2
12222 )( XXXXIM ′′−= −
2M is symmetric and idempotent ( 2
22 MM = ).
Further define **
222 ybXyyM =−= with yXXXb 2
122
*2 )( ′′= −
In words: *y is vector of OLS residuals of regression of y on 2X .
![Page 22: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/22.jpg)
Econometrics
22
In the same way define 12
*1 XMX =
In words: the −k th column of *
1X is the vectors of OLS residuals of the −k th column of 1X on 2X .
Hence
**
11*
1*11 ')'( yXXXb −=
In words: the OLS estimator in a regression on 1X and 2X , 1b , is the OLS estimator of the OLS residuals of y (in regression on 2X ) on the OLS residuals of 1X (in regression on 2X ).
![Page 23: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/23.jpg)
Econometrics
23
Note *1
*, Xy are purged of the effect of 2X , because OLS residuals are orthogonal to /uncorrelated with 2X . This is how least squares implements the ceteris paribus condition, i.e. 1b is the effect of 1X ‘holding 2X constant’. More like randomized assignment of 1X : *
1X is uncorrelated with the ‘omitted’ 2X , which is relegated to random error if only 1X is included.
![Page 24: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/24.jpg)
Econometrics
24
Special cases
• Columns of 1X and those of 2X are orthogonal/uncorrelated 012 =′ XX , so that 112 XXM = and
yXXXb 1
1111 )( ′′= −
Conclusion: if we omit 2X we get the same estimate.
• ι=2X with ι an −n vector of 1’s. Then
nIIM ')( 1
2ιιιιιι −=′′−= −
yynyyyM ιιι
−=′
−=2
In words: 2M takes y in deviation from its sample average.
![Page 25: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/25.jpg)
Econometrics
25
Conclusion: if we take both y and X in deviation from the sample means we obtain OLS estimators of y on varying regressors.
![Page 26: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/26.jpg)
Econometrics
26
Estimation of 2σ The other parameter in CLR model is the variance of the random error 2σ . Obvious idea: use sample variance of the OLS residuals e . Instead we use (in model with K regressors including constant)
eeKn
eKn
sn
ii ′
−=
−= ∑
=1
22 11
![Page 27: Lecture 3: Properties of the OLS Estimatorridder/Lnotes/PracEconometrics... · Econometrics 1 Lecture 3: Properties of the OLS Estimator For the OLS estimator b =(X'X)−1 X' y define:](https://reader031.fdocuments.in/reader031/viewer/2022020108/5ae6d6f87f8b9a8b2b8dc9aa/html5/thumbnails/27.jpg)
Econometrics
27
The sampling distribution of 2s can be derived from εεβ MXMMye =+== )( so that
εεMKn
s−
=12
It can be shown that 22 )|( σ=XsE so that ( ) 222 )|()( σ== XsEEsE X In words: 2s is an unbiased estimator of 2σ .