Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a...
-
Upload
magnus-johnston -
Category
Documents
-
view
218 -
download
0
description
Transcript of Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a...
Least Squares Problems
From Wikipedia, the free encyclopedia
The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., more equations than unknowns.
The most important application is in data fitting
The least-squares method was first described by Carl Friedrich Gauss around 1794
Legendre was the first to publish the method, however.
The Problem:
nmnmAbAx , is ,
residual theis
,min
:such that Find
22
m
n
Axbr
yb-AyAxb
x
The Problem
Range of A
xAx
b
If we have 21 data points we can find a unique polynomial interpolant to these points by solving:
Data-Fitting
20,,0 ),()( ixfxP ii
20
0
20,,0 ),(j
ijii ixfxc
)(
)(
1
1
20
0
20
0
202020
2000
xf
xf
c
c
xx
xx
Without changing the data points we can do better by reducing the degree of the polynomial
In the previous example: Polynomial of degree 8:
Polynomial Least Squares Fitting
)(
)(
1
1
20
0
8
0
82020
800
xf
xf
c
c
xx
xx
Orthogonal Projection and the Normal Equations
Theorem:
ly equivalentor ly equivalent
)range(
min satisfies A vector
given. be and )(Let
22
AxPbbAAxA
Ar
AwbAxbx
bnmA
TT
w
n
mnm
n
Pseudoinverse
mnTT AAAA 1)(
)( 1AAT exists If A has full rank then
Is called the Pseudoinverse, and
bAx Is the least squares solution
Four Algorithms
1. Find the Pseudoinverse
2. Solve the Normal Equation (A full rank):
TT AAAA 1)(
Then calculate bAx
Requires A to have full rank
AAT Is positive definite and we use the Cholesky factorization
Four Algorithms
3- QR Factorization:^^RQA Reduced QR
^^TQQP Orthogonal projector onto range(A)
)(Range ^^
AbQQPby T
hence and such that then yAxx
bQxR
Q
bQQxR
T
T
T
^^
^
^^^^
by multipl-left
Q
Four Algorithms
4- SVD TVUA^^ Reduced SVD
^^TUUP Orthogonal projector onto range(A)
)(Range ^^
AbUUPby T
hence and such that then yAxx
bUxV
U
bUUxVU
TT
T
TT
^^
^
^^^^
by multipl-left