Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the...
-
Upload
kendall-crumb -
Category
Documents
-
view
224 -
download
3
Transcript of Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the...
![Page 1: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/1.jpg)
Ordinary Least-Squares
Ordinary Least-Squares
![Page 2: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/2.jpg)
Ordinary Least-Squares
Outline
• Linear regression• Geometry of least-squares• Discussion of the Gauss-Markov
theorem
![Page 3: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/3.jpg)
Ordinary Least-Squares
One-dimensional regression
a
b
![Page 4: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/4.jpg)
Ordinary Least-Squares
One-dimensional regression
axb
Find a line that represent the
”best” linear relationship:
a
b
![Page 5: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/5.jpg)
Ordinary Least-Squares
One-dimensional regression
xabe iii
• Problem: the data does not go through a line
xab ii
a
b
![Page 6: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/6.jpg)
Ordinary Least-Squares
One-dimensional regression
xabe iii
• Problem: the data does not go through a line
• Find the line that minimizes the sum:
2)( i
ii xab
xab ii
a
b
![Page 7: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/7.jpg)
Ordinary Least-Squares
One-dimensional regression
x̂
xabe iii
2)()( i
ii xabxe
• Problem: the data does not go through a line
• Find the line that minimizes the sum:
• We are looking for that minimizes
2)( i
ii xab
xab ii
a
b
![Page 8: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/8.jpg)
Ordinary Least-Squares
Matrix notation
Using the following notations
and
na
a
:1
a
nb
b
:1
b
![Page 9: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/9.jpg)
Ordinary Least-Squares
Matrix notation
Using the following notations
and
we can rewrite the error function using linear algebra as:
na
a
:1
a
nb
b
:1
b
2
2
)(
)()(
)()(
ab
abab
xxe
xx
xabxe
T
iii
![Page 10: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/10.jpg)
Ordinary Least-Squares
Matrix notation
Using the following notations
and
we can rewrite the error function using linear algebra as:
na
a
:1
a
nb
b
:1
b
2
2
)(
)()(
)()(
ab
abab
xxe
xx
xabxe
T
iii
![Page 11: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/11.jpg)
Ordinary Least-Squares
Multidimentional linear regression
Using a model with m parameters
j
jjmm xaxaxab ...11
![Page 12: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/12.jpg)
Ordinary Least-Squares
Multidimentional linear regression
Using a model with m parameters
2a
j
jjmm xaxaxab ...11
1a
b
![Page 13: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/13.jpg)
Ordinary Least-Squares
Multidimentional linear regression
Using a model with m parameters
2a
j
jjmm xaxaxab ...11
1a
b
![Page 14: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/14.jpg)
Ordinary Least-Squares
Multidimentional linear regression
Using a model with m parameters
and n measurements
j
jjmm xaxaxab ...11
2
2
1,
2
1 1, )()(
Axb
b
x
m
jjji
n
i
m
jjjii
xa
xabe
![Page 15: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/15.jpg)
Ordinary Least-Squares
Multidimentional linear regression
Using a model with m parameters
and n measurements
j
jjmm xaxaxab ...11
2
2
1,
2
1 1,
)(
)()(
Axbx
b
x
e
xa
xabe
m
jjji
n
i
m
jjjii
![Page 16: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/16.jpg)
Ordinary Least-Squares
Axb
mmnn
m
n x
x
aa
aa
b
b
:.
..
::
..
:1
,1,
,11,11
Axb
![Page 17: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/17.jpg)
Ordinary Least-Squares
Axb
)...(
:
)...(
:.
..
::
..
:
,11,
,111,11
1
,1,
,11,11
mmnnn
mm
mmnn
m
n
xaxab
xaxab
x
x
aa
aa
b
b
Axb
![Page 18: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/18.jpg)
Ordinary Least-Squares
Axb
)...(
:
)...(
:.
..
::
..
:
,11,
,111,11
1
,1,
,11,11
mmnnn
mm
nmnn
m
n
xaxab
xaxab
x
x
aa
aa
b
b
Axb
parameter 1
![Page 19: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/19.jpg)
Ordinary Least-Squares
Axb
)...(
:
)...(
:.
..
::
..
:
,11,
,111,11
1
,1,
,11,11
mmnnn
mm
nmnn
m
n
xaxab
xaxab
x
x
aa
aa
b
b
Axbmeasurement n
parameter 1
![Page 20: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/20.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
![Page 21: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/21.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
minx
)(xe
![Page 22: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/22.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
is flat at
minx
)(xe
)e(x minx
![Page 23: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/23.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
0x )( mineis flat at
minx
)(xe
)e(x minx
![Page 24: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/24.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
0x )( mineis flat at
does not go downaround)e(x
minx
minx
)(xe
)e(x minx
![Page 25: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/25.jpg)
Ordinary Least-Squares
Minimizing )(xe
x x ifminimizes )e( min
0x )( mine
definite-semi
positive is )( minxeH
is flat at
does not go downaround)e(x
minx
minx
)(xe
)e(x minx
![Page 26: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/26.jpg)
Ordinary Least-Squares
Positive semi-definite
x0Axx
A
allfor ,
T
definite-semi positive is
In 1-D In 2-D
![Page 27: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/27.jpg)
Ordinary Least-Squares
Minimizing )(xe
x̂
xxHx )ˆ(2
1e
T)(xe
![Page 28: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/28.jpg)
Ordinary Least-Squares
Minimizing 2
)( Axbx e
x̂
xxHxx )ˆ(2
1)( e
Te
![Page 29: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/29.jpg)
Ordinary Least-Squares
Minimizing
x x ifminimizes )e( ˆ
bAxAA TT ˆ
definite-semi
positive is AAT2
2)( Axbx e
![Page 30: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/30.jpg)
Ordinary Least-Squares
Minimizing 2
)( Axbx e
x x ifminimizes )e( ˆ
bAxAA TT ˆ
definite-semi
positive is AAT2
Always true
![Page 31: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/31.jpg)
Ordinary Least-Squares
Minimizing 2
)( Axbx e
x x ifminimizes )e( ˆ
bAxAA TT ˆ
definite-semi
positive is AAT2
Always true
The normal equation
![Page 32: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/32.jpg)
Ordinary Least-Squares
Geometric interpretation
![Page 33: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/33.jpg)
Ordinary Least-Squares
Geometric interpretation• b is a vector in Rn
b
![Page 34: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/34.jpg)
Ordinary Least-Squares
Geometric interpretation• b is a vector in Rn
• The columns of A define a vector space range(A)
b
2a
1a
![Page 35: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/35.jpg)
Ordinary Least-Squares
Geometric interpretation• b is a vector in Rn
• The columns of A define a vector space range(A)• Ax is an arbitrary vector in range(A)
b
2a
1a
Ax 2211 aa xx
![Page 36: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/36.jpg)
Ordinary Least-Squares
Geometric interpretation• b is a vector in Rn
• The columns of A define a vector space range(A)• Ax is an arbitrary vector in range(A)
b
2a
1a
Axb
Ax 2211 aa xx
![Page 37: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/37.jpg)
Ordinary Least-Squares
Geometric interpretation• is the orthogonal projection of b onto range(A)
b
2a
1a
xA ˆˆˆ 2211 aa xx
xAb ˆ
bAxAAxAbA TTT ˆˆ 0
xA ˆ
![Page 38: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/38.jpg)
Ordinary Least-Squares
The normal equation: bAxAA TT ˆ
![Page 39: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/39.jpg)
Ordinary Least-Squares
The normal equation:
• Existence: has always a solution
bAxAA TT ˆ
bAxAA TT ˆ
![Page 40: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/40.jpg)
Ordinary Least-Squares
The normal equation: bAxAA TT ˆ
• Existence: has always a solution
• Uniqueness: the solution is unique if the columns
of A are linearly independent
bAxAA TT ˆ
![Page 41: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/41.jpg)
Ordinary Least-Squares
The normal equation:
b
2a1a
xA ˆ
• Existence: has always a solution
• Uniqueness: the solution is unique if the columns
of A are linearly independent
bAxAA TT ˆ
bAxAA TT ˆ
![Page 42: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/42.jpg)
Ordinary Least-Squares
Under-constrained problem
2a
1a
b
![Page 43: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/43.jpg)
Ordinary Least-Squares
Under-constrained problem
2a
1a
b
![Page 44: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/44.jpg)
Ordinary Least-Squares
Under-constrained problem
2a
1a
b
![Page 45: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/45.jpg)
Ordinary Least-Squares
Under-constrained problem
• Poorly selected data• One or more of the
parameters are redundant
2a
1a
b
![Page 46: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/46.jpg)
Ordinary Least-Squares
Under-constrained problem
• Poorly selected data• One or more of the
parameters are redundant
Add constraints
x bAAxA xmin with TT 2a
1a
b
![Page 47: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/47.jpg)
Ordinary Least-Squares
How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem
![Page 48: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/48.jpg)
Ordinary Least-Squares
How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem
Let and be two sets of random variables and define:
mmiiii xaxabe ,11, ...
ib jx
![Page 49: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/49.jpg)
Ordinary Least-Squares
How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem
Let and be two sets of random variables and define:
If
mmiiii xaxabe ,11, ...
ib jx
, ,0cov
, ,var
, ,0
:
ji),e(e
i)(e
i)E(e
a
ji
i
i
i,j
and allfor :A4
allfor :A3
allfor :A2
, variablesrandomnot areA1
![Page 50: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/50.jpg)
Ordinary Least-Squares
How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem
Let and be two sets of random variables and define:
If
Then is the best unbiased linear estimator
mmiiii xaxabe ,11, ...
ib jx
, ,0),cov(
, ,var
, ,0
:
jiee
i)(e
i)E(e
a
ji
i
i
i,j
and allfor :A4
allfor :A3
allfor :A2
, variablesrandomnot areA1
2minargˆiexx
![Page 51: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/51.jpg)
Ordinary Least-Squares
a
b
ei
no errors in ai
![Page 52: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/52.jpg)
Ordinary Least-Squares
a
b
ei
a
b
ei
no errors in ai errors in ai
![Page 53: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/53.jpg)
Ordinary Least-Squares
a
b
homogeneous errors
![Page 54: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/54.jpg)
Ordinary Least-Squares
a
b
a
b
homogeneous errors non-homogeneous errors
![Page 55: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/55.jpg)
Ordinary Least-Squares
a
b
no outliers
![Page 56: Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.](https://reader036.fdocuments.in/reader036/viewer/2022062307/551a9d8b55034643688b629e/html5/thumbnails/56.jpg)
Ordinary Least-Squares
a
b
a
b
no outliers outliers
outliers