Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.
-
Upload
alban-johnson -
Category
Documents
-
view
217 -
download
0
description
Transcript of Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.
![Page 1: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/1.jpg)
Stats 443.3 & 851.3
Summary
![Page 2: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/2.jpg)
The Woodbury Theorem
11 1 1 1 1 1A BCD A A B C DA B DA
where the inverses11 1 1 1, and exist.A C C DA B
![Page 3: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/3.jpg)
11 12
21 22
q
n m n qp m p
A AA
A A
Block Matrices
Let the n × m matrix
be partitioned into sub-matrices A11, A12, A21, A22,
11 12
21 22
p
m k m pl k l
B BB
B B
Similarly partition the m × k matrix
![Page 4: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/4.jpg)
11 12 11 12
21 22 21 22
A A B BA B
A A B B
Product of Blocked Matrices
Then
11 11 12 21 11 12 12 22
21 11 22 21 21 12 22 22
A B A B A B A BA B A B A B A B
![Page 5: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/5.jpg)
11 12
21 22
p
n n n pp n p
A AA
A A
The Inverse of Blocked Matrices
Let the n × n matrix
be partitioned into sub-matrices A11, A12, A21, A22,
11 12
21 22
p
n n n pp n p
B BB
B B
Similarly partition the n × n matrix
Suppose that B = A-1
![Page 6: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/6.jpg)
11 12
21 22
p
n n n pp n p
A AA
A A
Summarizing
Let
11 12
21 22
p
n pp n p
B BB B
Suppose that A-1 = B
then
11 1 121 22 21 11 22 21 11 12 22 21B A A B A A A A A A
11 1 112 11 12 22 11 12 22 21 11 12B A A B A A A A A A
1 11 1 1 1 111 11 12 22 21 11 11 12 22 21 11 12 21 11B A A A A A A A A A A A A A
1 11 1 1 1 1
22 22 21 11 12 22 22 21 11 12 22 21 12 22B A A A A A A A A A A A A A
![Page 7: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/7.jpg)
Symmetric Matrices
• An n × n matrix, A, is said to be symmetric if
Note:AA
11
111
AA
ABAB
ABAB
![Page 8: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/8.jpg)
The trace and the determinant of a square matrix
11 12 1
21 22 2
1 2
n
nij
n n nn
a a aa a a
A a
a a a
Let A denote then n × n matrix
Then
1
n
iii
tr A a
![Page 9: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/9.jpg)
11 12 1
21 22 2
1 2
det the determinant of
n
n
n n nn
a a aa a a
A A
a a a
also
where1
n
ij ijj
a A
cofactor of ij ijA a
the determinant of the matrix
after deleting row and col.th thi j
11 1211 22 12 21
21 22
deta a
a a a aa a
ji1
![Page 10: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/10.jpg)
1. 1, I tr I n
Some properties
2. , AB A B tr AB tr BA
1 13. AA
122 11 12 22 2111 12
121 22 11 22 21 11 12
4. A A A A AA A
AA A A A A A A
22 11 12 21 if 0 or 0A A A A
![Page 11: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/11.jpg)
Special Types of Matrices
1. Orthogonal matrices– A matrix is orthogonal if PˊP = PPˊ = I– In this cases P-1=Pˊ .– Also the rows (columns) of P have length 1 and
are orthogonal to each other
![Page 12: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/12.jpg)
Special Types of Matrices(continued)
2. Positive definite matrices– A symmetric matrix, A, is called positive definite
if:
– A symmetric matrix, A, is called positive semi definite if:
022 112211222
111 nnnnn xxaxxaxaxaxAx
0 allfor
x
0 xAx
0 allfor
x
![Page 13: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/13.jpg)
Theorem The matrix A is positive definite if0,,0,0,0 321 nAAAA
nnnn
n
n
n
aaa
aaaaaa
AA
aaaaaaaaa
Aaaaa
AaA
21
22212
11211
332313
232212
131211
32212
12112111
and
,,,
where
![Page 14: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/14.jpg)
Special Types of Matrices(continued)
3. Idempotent matrices– A symmetric matrix, E, is called idempotent if:
– Idempotent matrices project vectors onto a linear subspace
EEE
xExEE
xE
x
![Page 15: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/15.jpg)
Eigenvectors, Eigenvalues of a matrix
![Page 16: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/16.jpg)
DefinitionLet A be an n × n matrixLet and be such thatx
with 0Ax x x
then is called an eigenvalue of A andand is called an eigenvector of A andx
![Page 17: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/17.jpg)
Note:
0A I x
1If 0 then 0 0A I x A I
thus 0 A I
is the condition for an eigenvalue.
![Page 18: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/18.jpg)
11 1
1
det = 0n
n nn
a aA I
a a
= polynomial of degree n in .
Hence there are n possible eigenvalues 1, … , n
![Page 19: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/19.jpg)
Thereom If the matrix A is symmetric with distinct eigenvalues, 1, … , n, with corresponding eigenvectors
1 1 1then n n nA x x x x
1, , nx x
Assume 1 i ix x
1 1
1
0, ,
0n
n n
xx x
x
PDP
![Page 20: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/20.jpg)
The Generalized Inverse of a matrix
![Page 21: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/21.jpg)
DefinitionB (denoted by A-) is called the generalized inverse (Moore – Penrose inverse) of A if
1. ABA = A2. BAB = B3. (AB)' = AB4. (BA)' = BA
Note: A- is unique
![Page 22: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/22.jpg)
Hence B1 = B1AB1 = B1AB2AB1 = B1 (AB2)'(AB1) '
= B1B2'A'B1
'A'= B1B2'A' = B1AB2 = B1AB2AB2
= (B1A)(B2A)B2 = (B1A)'(B2A)'B2 = A'B1'A'B2
'B2
= A'B2'B2= (B2A)'B2
= B2AB2 = B2
The general solution of a system of Equations
Ax b
x A b I A A z
The general solution
x A b I A A z
where is arbitrary
![Page 23: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/23.jpg)
1 1then C B BB A A A
Let C be a p×q matrix of rank k < min(p,q),
then C = AB where A is a p×k matrix of rank k and B is a k×q matrix of rank k
![Page 24: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/24.jpg)
The General Linear Model
npnn
p
p
pn xxx
xxxxxx
y
yy
21
22221
11211
2
1
2
1
,,Let Xβy
ondistributi , a has where 2I0εεβXy N
![Page 25: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/25.jpg)
Geometrical interpretation of the General Linear Model
p
npnn
p
p
pn xxx
xxxxxx
y
yy
xxXβy
1
21
22221
11211
2
1
2
1
,,Let
ondistributi , a has where 2I0εεβXy N
X
xxxβXyμ
of columns by the spanned spacelinear in the lies
221 ppE
![Page 26: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/26.jpg)
Estimation
The General Linear Model
![Page 27: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/27.jpg)
n
iiip
n
iiip
n
iii
n
ii yxxxxxx
11
112
1121
1
21
n
iiip
n
iiip
n
ii
n
iii yxxxxxx
12
122
1
221
121
n
iiipp
n
iip
n
iipi
n
iipi yxxxxxx
11
22
121
11
the Normal Equations
![Page 28: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/28.jpg)
yXβXX
The Normal Equations
npnn
p
p
n xxx
xxxxxx
y
yy
21
22221
11211
2
1
, where Xy
![Page 29: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/29.jpg)
Solution to the normal equations
yXβXX ˆ
. of is matrix theIf rank fullX
yXXXβ 1ˆ
![Page 30: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/30.jpg)
Estimate of 2
βXyβXy ˆˆ1ˆ 2
n
yXXXXIy 1
n1
βXyyy ˆ1
n
![Page 31: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/31.jpg)
Properties of The Maximum Likelihood Estimates
Unbiasedness, Minimum Variance
![Page 32: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/32.jpg)
yXXXβ
1ˆ EE
ββXXXXyXXX
11 E
βcβcβc
ˆˆ EE
22ˆ n
pnE
βXyβXy ˆˆ1ˆ 22
pnpn
ns
s2 is an unbiased estimator of 2.
Unbiasedness
![Page 33: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/33.jpg)
Distributional Properties
Least square Estimates (Maximum Likelidood estimates)
![Page 34: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/34.jpg)
The General Linear Model
and yXXXXIy 1
pns 1 2. 2
XXXAyAyXXXβ 11 whereˆ 1.
yBy
yXXXXIy 1
1 Now 22
2
spnU
XXXXIB 1 2
1 where
IβXy 2, ~
nN
The Estimates
![Page 35: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/35.jpg)
Theorem
.0 with ,~ 2. 22
2
pnspnU
12, ~ ˆ 1. XXββ
pN
tindependen are and ˆ 3. 2sβ
![Page 36: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/36.jpg)
The General Linear Model
with an intercept
![Page 37: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/37.jpg)
npnn
p
p
pn xxx
xxxxxx
y
yy
21
22221
11211
2
1
0
2
1
1
11
,,Let Xβy
ondistributi , a has i.e. 2IβXy
N
ondistributi , a has where 2I0εεβXy
N
The matrix formulation (intercept included)
Then the model becomes
Thus to include an intercept add an extra column of 1’s in the design matrix X and include the intercept in the parameter vector
![Page 38: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/38.jpg)
The Gauss-Markov Theorem
An important result in the theory of Linear models
Proves optimality of Least squares estimates in a more general setting
![Page 39: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/39.jpg)
The Gauss-Markov TheoremAssume
IyβXy 2var and E
Consider the least squares estimate of
ˆ 1 yXXXβ
β
nn yayaya
2211
1
ˆ
yayXXXcβc
, an unbiased linear estimator of βc
and
Let nn ybybyb
2211ybdenote any other unbiased linear estimator of βc
βcyb ˆvarvarthen
![Page 40: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/40.jpg)
Hypothesis testing for the GLM
The General Linear Hypothesis
![Page 41: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/41.jpg)
Testing the General Linear Hypotheses
The General Linear Hypothesis H0: h111 + h122 + h133 +... + h1pp = h1
h211 + h222 + h233 +... + h2pp = h2
...hq11 + hq22 + hq33 +... + hqpp = hq
where h11h12, h13, ... , hqp and h1h2, h3, ... , hq are known coefficients. In matrix notation
11
qppqhβH
![Page 42: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/42.jpg)
Testing hβH
:0H
βXyβXy
hβHHXXHhβH
ˆˆ
ˆˆ
statistictest 1
111
pn
qF
pnqFFH , if Reject 0
![Page 43: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/43.jpg)
An Alternative form of the F statistic
pnRSS
qRSSRSSF H
0
0 assuming Squares of Sum Residual 0
HRSSH
0 assuming Squares of Sum Residual HRSS not
![Page 44: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/44.jpg)
Confidence intervals, Prediction intervals, Confidence Regions
General Linear Model
![Page 45: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/45.jpg)
cXXcβc 12
ˆ st pn
One at a time (1 – )100 % confidence interval for βc
(1 – )100 % confidence interval for 2 and .
2
2/1
2
22/
2
to
spnspn
22/1
22/
to
pnspns
![Page 46: Stats 443.3 & 851.3 Summary. The Woodbury Theorem where the inverses.](https://reader036.fdocuments.in/reader036/viewer/2022081514/5a4d1b507f8b9ab0599a7435/html5/thumbnails/46.jpg)
Multiple Confidence Intervals associated with the test hβH
:0H
Theorem: Let H be a q × p matrix of rank q.
then
ccHXXHcβHc
allfor ,ˆ 1spnqqF
form a set of (1 – )100 % simultaneous confidence interval for cβHc
allfor
cβHcc
allfor Consider