A LEAST-SQUARES APPROXIMATION METHOD FOR THE TIME-HARMONIC ...
Method of Least Squares
-
Upload
oliver-nichols -
Category
Documents
-
view
82 -
download
3
description
Transcript of Method of Least Squares
ELE 774 - Adaptive Signal Processing 2Least Squares
Least Squares Method of Least Squares:
Deterministic approach
The inputs u(1), u(2), ..., u(N) are applied to the system The outputs y(1), y(2), ..., y(N) are observed
Find a model which fits the input-output relation to a (linear?) curve, f(n,u(n))
‘best’ fit by minimising the sum of the squres of the difference f - y
0 5 10 15 20 25 30 35 40 45 500
5
10
15
20
25
30
35
40
45
50
ELE 774 - Adaptive Signal Processing 3Least Squares
Least Squares
The curve fitting problem can be formulated as
Error: Sum-of-error-squares:
Minimum (least-squares of error) is achieved when the gradient is zero
model observationsvariable
ELE 774 - Adaptive Signal Processing 4Least Squares
Problem Statement For the inputs to the system, u(i) The observed desired response
is, d(i)
Relation is assumed to be linear
Unobservable measurement error Zero mean
White
Then
deterministic
ELE 774 - Adaptive Signal Processing 5Least Squares
Problem Statement Design a transversal filter which finds the least squares solution
Then, sum of error squares is
ELE 774 - Adaptive Signal Processing 6Least Squares
Data Windowing We will express the input in matrix form Depending on the limits i1 and i2 this matrix changes
Covariance Methodi1=M, i2=N
Prewindowing Methodi1=1, i2=N
Postwindowing Methodi1=M, i2=N+M1
Autocorr. Methodi1=1, i2=N+M1
ELE 774 - Adaptive Signal Processing 7Least Squares
Error signal
Least squares (minimum of sum of squares) is achieved when
i.e., when
The minimum-error time series emin(i) is orthogonal to the time series of the input u(i-k) applied to tap k of a transversal filter of length M for k=0,1,...,M-1 when the filter is operating in its least-squares condition.
Principle of Orthogonality
!Time averaging!(For Wiener filtering)
(this was ensemble average)
ELE 774 - Adaptive Signal Processing 8Least Squares
Corollary of Principle of Orthogonality
LS estimate of the desired response is
Multiply principle of orthogonality by wk* and take summation over k
Then
When a transversal filter operates in its least-squares condition, the least-squares estimate of the desired response -produced at the output of the filter- and the minimum estimation error time series are orthogonal to each other over time i.
ELE 774 - Adaptive Signal Processing 9Least Squares
Energy of Minimum Error
Due to the principle of orthogonality, second and third terms are orthogonal, hence
where
, when eo(i)= 0 for all i, impossible
, when the problem is underdetermined fewer data points than parameters infinitely many solutions (no unique soln.)!
ELE 774 - Adaptive Signal Processing 10Least Squares
Normal Equations
Hence,
Expanded system of the normal equations for linear least-squares filters.
Minimum error: Principle of Orthogonality→
(t,k), 0≤(t,k) ≤M-1time-average
autocorrelation functionof the input
z(-k), 0 ≤k ≤M-1time-average
cross-correlation bwthe desired response
and the input
ELE 774 - Adaptive Signal Processing 11Least Squares
Normal Equations (Matrix Formulation)
Matrix form of the normal equations for linear least-squares filters:
Linear least-squares counterpart of the Wiener-Hopf eqn.s. Here and z are time averages, whereas in Wiener-Hopf eqn.s
they were ensemble averages.
(if -1 exists!)
ELE 774 - Adaptive Signal Processing 12Least Squares
Minimum Sum of Error Squares
Energy contained in the time series is
Or,
Then the minimum sum of error squares is
ELE 774 - Adaptive Signal Processing 13Least Squares
Properties of the Time-Average Correlation Matrix
Property I: The correlation matrix is Hermitian symmetric,
Property II: The correlation matrix is nonnegative definite,
Property III: The correlation matrix is nonsingular iff det() is nonzero
Property IV: The eigenvalues of the correlation matrix are real and non-negative.
ELE 774 - Adaptive Signal Processing 14Least Squares
Properties of the Time-Average Correlation Matrix
Property V: The correlation matrix is the product of two rectangular Toeplitz matrices that are Hermitian transpose of each other.
ELE 774 - Adaptive Signal Processing 15Least Squares
Normal Equations (Reformulation)
But we know that
which yields
Substituting into the minimum sum of error squares expression gives
then
! Pseudo-inverse !
ELE 774 - Adaptive Signal Processing 16Least Squares
Projection
The LS estimate of d is given by
The matrix
is a projection operator onto the linear space spanned by the columns of data matrix A i.e. the space Ui.
The orthogonal complement projector is
ELE 774 - Adaptive Signal Processing 17Least Squares
Projection - Example
M=2 tap filter, N=4 → N-M+1=3 Let
Then
And
orthogonal
ELE 774 - Adaptive Signal Processing 19Least Squares
Uniqueness of the LS Solution
LS always has a solution, is that solution unique?
The least-squares estimate is unique if and only if the nullity (the dimension of the null space) of the data matrix A equals zero.
AKxM, (K=N-M+1)
Solution is unique when A is of full column rank, K≥M All columns of A are linearly independent Overdetermined system (more eqns. than variables (taps)) (AHA)-1 nonsingular → exists and unique
Infinitely many solutions when A has linearly dependent columns, K<M
(AHA)-1 is singular
ELE 774 - Adaptive Signal Processing 20Least Squares
Properties of the LS Estimates
Property I: The least-squares estimate is unbiased, provided that the measurement error process eo(i) has zero mean.
Property II: When the measurement error process eo(i) is white with zero mean and variance 2, the covariance matrix of the least-squares estimate equals 2-1.
Property III: When the measurement error process eo(i) is white with zero mean, the least squares estimate is the best linear unbiased estimate.
Property IV: When the measurement error process eo(i) is white and Gaussian with zero mean, the least-squares estimate achieves the Cramer-Rao lower bound for unbiased estimates.
ELE 774 - Adaptive Signal Processing 21Least Squares
Computation of the LS Estimates The rank (W) of an KxN (K≥N or K<N) matrix A gives
The number of linearly independent columns/rows The number of non-zero eigenvalues/singular values
The matrix is said to be full rank (full column or row rank) if
Otherwise, it is said to be rank-deficient
Rank is an important parameter for matrix inversion If K=N (square matrix) and the matrix is full rank (W=K=N) (non-
singular) inverse of the matrix can be calculated, A-1=adj(A)/det(A)
If the matrix is not square (K≠N), and/or it is rank-deficient (singular), A-1 does not exist, instead we can use the pseudo-inverse (a projection of the inverse), A+
ELE 774 - Adaptive Signal Processing 22Least Squares
SVD We can calculate the pseudo-inverse using SVD.
Any KxN matrix (K≥N or K<N) can be decomposed using the Singular Value Decomposition (SVD) as follows:
ELE 774 - Adaptive Signal Processing 23Least Squares
SVD
The system of eqn.s, is overdetermined if K>N, more eqn.s than unknowns,
Unique solution (if A is full-rank) Non-unique, infinitely many solutions (if A is rank-deficient)
is underdetermined if K<N, more unknowns than eqn.s, Non-unique, infinitely many solutions
In either case the solution(s) is(are)
where
ELE 774 - Adaptive Signal Processing 24Least Squares
Computation of the LS Estimates
Find the solution of (A: KxM)
If K>M and rank(A)=M, ( ) the unique solution is
Otherwise , infinitely many solutions, but pseudo-inverse gives the minimum-norm solution to the least squares problem.
Shortest length possible in the Euclidean norm sense.
ELE 774 - Adaptive Signal Processing 25Least Squares
Minimum-Norm Solution
We know that
Then
min is achieved when
where min is determined by c2 (desired response, uncontrollable)
min is independent of b2 !
ELE 774 - Adaptive Signal Processing 26Least Squares
Minimum-Norm Solution Then the optimum filter coefficients become
Norm of filter coeff.s is (VHV=I)
which is minimum when
then
Even when , the vector is unique in the sense that
it is the only tap-weight vector that simultaneously satisfy Minimum sum-of-error-squares (LS solution) The smallest Euclidean norm possible.
Hence, is called the minimum-norm LS solution.
≥0 ≥0