EEL 6502: Adaptive Signal Processing Homework #4 (LMS)

Name: Jo, Youngho Cyhio@ufl.edu) WID: 58434260

The purpose of this homework i s to compare the performance between Prediction Error Filter and LMS

(Least Mean Square Emor) Algorithm in terms of complexity of calculation, accuracy of solution (stirnation

maladjustment), speed of adaptation and stationary versus non-stat ionary.

In forward linear pmhctor, the tap inputs u(n - I), u(n - 21,. , , , u(n - M) define the M-by- 1 tapinput

vector

Hence, the correlation matrix of the tap input equals

wherer(k) is the autocorrelation function ofinput process for lag k, where k = O,l, ..., M - 1 . The cross-correlation vector between the tap inputs u(n - I), u (n - 21,. . ., u(n - M) and the desired

response u(n)

The variance of u En) equals r (0) , since u(n) has zero mean.

Thus, the optimum tapweight vector of the predictor is w = [w,, w, ,..., w, p and on be solved by

Wiener-Hopf Equation and forward predictor error power is:

R W = P C S W = R - [ P

At linear predictor, weight of prediction error filter is computed by autocomEation of input data and

cross-correlation of input data and desired response using Wiener-Hopf Equation. As a result, performpnce of

linear predictor depends on filter order and window size of input data in stationary and non-stationary input data.

In LMS algorithm which is one of special form of steepest descent method, procedures of computation are as follow:

Table 1 Summary of LMS Algorithm

Parameters: M = number of taps (i.e. filter order or length) p = step-size

1 O<p<-

A- where 2- is maximum ualue of eigenvalue of autocwrelation of tap input data u(n) .

Initlallzation: if prior knowledge of the tap-weight vector q n ) is available, use it to select an appropriate value for %O) . Ohenvise, set q O ) = 0 . Data

- Given u(n) = M - by - 1 tap - input vector at time n

= [u(n),u(n - l),u(n - 21, ..., u(n - M + 111' d(n) = desired response at time n

- To be computed +(n + I) =estimate of tap -input vector at time n + 1

Computation: for n=0,1,2,3 ... compute - Estimation Error or Error signal 4n) = d(n) - iirT (n)u(n) = estimate of tap - input vector at time n + 1 +(n + 1) = +(n) + pu(n)e(n)

1. Complexity of Calculation Given number of taps (i.e. filter order) M

- Linear Predictor: Wiener-Hopf Equation, w* = R-'P, uses matrix Inversion of autocornlation of input

data and multiplication by cross-correlation matrix. Therefore, wmprexity of calculation is O ( M 3 ) . - LMS: To update filter tap weight as shown using i ( n + 1) = *(n)+ pu(n)e(tn) in Table 1, LMS

algorithm uses matrix multiplication and addition. Therefore, complexity of cdculation is O(M) .

2. Accuracy of the solution (Estimation Muadjustment) To compare accuracy of solution of linear predictor and LMS, normalized error power (MSE e m r

divided by input power) is used.

Figure 1 Comparison of Normalized Error Power of MG data

Figure 2 Comparison of Normalized Error Power of Speech Signal

We can see that normalized error power of linear predictor (filter order 15 and window size 100) has

better performance than LMS algorithm (filter order 1 5 and stepsize 0.08) in both stationary and non-stationary

data as shown in Figure 1 and Figure 2.

To see eEwt of stepsize on eomalized error power of LMS for stationary and non-stationary data,

averaged normalized error power of MG data and speech data is summarized in following Table 2 and Table 3.

As you can see in Table 2 and 3 , higher order filter for the same stepsize has better performance in both MG and speech data. However, smaller stepsize for the same filter order has much averaged normalized error power than

bigger stepsize which satisfies 0 < ,u < 1 / A- . This means that bigger stepsize has lugher speed of adaptation

than smaller stepsize, has less averaged normalizsd m r power. Thmfore the more computation and data are

required to reach optimal weights for smaller stepsize. Tbe effect of data size and stepsize on the speed of

adaptation will be discussed in the later section.

Table 2 Normalized Error Powers of LMS for MG Data

Table 3 Normatized Error Powers of LMS for Speech Data

In LMS algorithm, excess MSE and misadjustment M are: M

excess MSE = A,, &-I) = km, t r [ ~ ] u=O

exccss MSE M = F: = ~ [ R I

where p is stepsize, tr[~] is trace matrix of autocorrelation matrix, and Am is eigenvaluc of autocorrclation of

input data. Therefore, misadjusbnent in the LMS algorithm is proportional to stepsize p. For fdter order of 15,

misadjustment of given stepsize is summarized in following Table 4. We can see that misadjustment of LMS

algorithm is proportion to stepsize from Table 4.

Table 4 Misadjustment of LMS for different stepsize and fiiter order

3. Speed of adaptation Time constant of LMS is:

The ration of largest eigenvalue and smaIlest eigenvalue also called eigenspread, and this varies from I to infinite. Therefore, smallest eigenvalue control adaptation time in given steps& and larger stepsize decrease

time constant of LMS algorithm.

Stepsi w h) M=2

M=6

M= 15

M=30

Figure 3 Comparison of speed of adaptation

0.02

0.03520006767262 89

0.10560020301 7887

0.264000507544717

0.52800101 5089433

0.01

0.0 176000338363 144

0.0528001015089433

0.132000253772358

0.26400050754471 7

For convergence of the LMS algorithm, we have to choose the stepsize as mentioned in Table 1. For stationary MG data, the maximum eigenvalue for filter of order 15 is 12.0373 1 6. Therefore, maximum stepsize

0,W

0.070400 1353452578

0.21 1200406035773

0.52800 101 5089434

1.05600203017887

0.08

0.1408002706905 16

0.422400812071547

1.056002030 17887

2,11200406035773

is 0.083075 and eigenspread is 23862.437292. In top of Figure 3 wiib fixed filter order IS, we can see that

learning curve of stepsize 0.08 converges faster than any other stepsize. In bottom of Figure 3 with f w d stepsize

0.04, n o d z e d e m s power is affected by fiI ter order.

The cost function which is expressed by

t = ~[e ' (n ) ] = ~ [ d ' (n)] - 2pTw + W'RW is quadratic function of weights. In this case. MSE space is called performance surface. Figure 4 shows the 3-

dimemoan1 prforrnance surface with 2 weights for MG data.

reaches certain area of optimal weights with increasing in raffling of weights. Stepsize 0.2, which greater than 11 A,, , has much raffling and may have ovenhoots in adaptation of weight.

Figure 5 Weight track of fllter order 2 with step size 0.01,0.04,0.057 and 0.2

To study the effect of data size in small stepsix, we use the same MG data several times and campared

weigbt track in Figure 6. As we can that smaller stepsize incmues the number of computation to reach optimal

weight In Left topmost of Figure 6, overlapped weight track for stepsize of 0.01 and 0.057 is given for

comparison. The weight tracks with small stepsize curves are smoother than large stepsize. This mans that finslr

weight vectur values are more close to the o p W so the excess MSE is small, but need more computation. or

mare data to rcach optimal weight vector vahes. Therefore, small stepsize decrease the excess MSE error and

ruffling of weight but slower speed of adaptation in LMS algorithm. The * mark in data size 10,000 and 15,000

in Figure 6 is the last weight vector value hm data size 25,000 for the comparison. As you can see that small

stepsize has little raffling but has slow speed of adaptation.

In Tablc I , the initial condition for weights is set to zeros for the iteration. In following Figure 7, we have shown the effect of initial condition on the weight track for h e comparison of Figure 5 and 6. We can see that

changing in initial condition makes different path for weight track, however updated weights of LSM track the

direction of larger eigenvalue at first, and then follow tho direction of fast mode as mentioned before.

Figure 6 Comparison weight track for data she fmm 5,000 to 25,900

Figure 7 Effect of initial condition on weight tmck

4. Stationary versus Non-stattonary : Figures of merit of the LMS have been derived assuming stationarity conditions. Non-stationary may

a&e at Iast in 2 important cases.

- The desired signal is time varying (system identification). Here the autocorrelation function is time

invariant, but the crass~t~elation function is time varying. - The input is non-stationary (equalization, noise cancellation). Here both autmorreiation and cross-

correlation arc non-stat ionary.

In non-stationary environments, the optimum set of weights change with time. Therefore, the LMS must not only find the minimum but also track the ~hange of minimum of the performance surface. In this cast, we

have two error sources for the actual weights. One is related to the e m s of the estimate of the &mt. The

other is related to the lag in the adaptive process. The first contribution is direct1 y proportional to stepsize and

the second contribution is inversely proportional to stepsize. We can assume them additive so the total

misadjustment has s single minimum.

For comparison between stationary and non-stationary data, weight track of non-stationary speech signal

is given in following Figure 8. As we can see in figure with respect to weight track of stationary MG data,

weight track of non-stationary data has much raHe thzln stationary data due to nw-s tationarity of speech signal.

Moreover path of weight track of non-stationary data is less straight than stationary data and has more overshoots and meandering to search optimum weight.

Figure 8 Weight tracks of speech dab

Red PM R d Put

Rgure 9 Polezero plots of Isverse filter LMS and h e a r predictor

-1 0 1 Real Part

Figure 10 Pole-zero plots of inverse filter for speech signal.

To consult the property of minimum-phase system of LMS algorithm for MG data and speech data, the

pole-zen, plot are given in following Figwe 9 and Figure 10, respectively.

As shown in Figure 9, we can see that some poles of LMS are outside of unit circle for both filtm of

order 15 and 30, then unstable and non-minimum phase. However, all poles of inverse fr l ter of linear predictor

all inside unit circle. Therefore, linear predictor has stable and casual inverse filter and minimum phase system.

Xn Fj gure 10, effect of stepsize on the stability of inverse filter for order 15 studied using speech signal,

As shown in figure, some poles of stepsize 0.8 are outside of unit circle. However all poles of small step art

inside unit circle. This means that raffling of weight with larger stepsize contributes stability of inverse filter.

This is dso true for stationary MG data. Therefore, we need to choose adequate stepsize to guarantee propay of minimum phase of inverse filter and stability of prediction error filter.

With statistically stationary inputs, the quadratic performance surface is fixed (deterministic) and Wiener

solution is fixed. With non-stationary input, the performance surface is varying in time and Wiener Solution is not fixed but time varying. In linear prediction problem, we assume the laally stationarity of speech data by

taking window. However longer window degraded performance of non-stationary data by Wiener-Hopf Equation.

The performance of LMS algorithms is equivalent when we use the same size for both stationary and

non-stationary data. This means that the LMS have no assumptions or knowledge about the statistics of input or desired response. The algorithm does not use any averaging, squaring, perturbation, etc. Each components of

d e n t is obtained from a single data input. However, Due to noisy estimation of LMS algorithm, excess MSE

increase with stepsize and greater than the minimum value sf MSE error power by the linear predictor.