ADAPTIVE FILTERS

ADAPTIVE FILTERSTHEORY AND APPLICATIONS

Second Edition

Behrouz Farhang-BoroujenyUniversity of Utah

USA

A John Wiley & Sons, Ltd., Publication

This edition first published 2013© 2013, John Wiley & Sons, Ltd

First Edition published in 1998© 1998, John Wiley & Sons, Ltd.

Registered officeJohn Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply forpermission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the author to be identified as the author of this work has been asserted in accordance with theCopyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, inany form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted bythe UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not beavailable in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand namesand product names used in this book are trade names, service marks, trademarks or registered trademarks of theirrespective owners. The publisher is not associated with any product or vendor mentioned in this book.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts inpreparing this book, they make no representations or warranties with respect to the accuracy or completeness ofthe contents of this book and specifically disclaim any implied warranties of merchantability or fitness for aparticular purpose. It is sold on the understanding that the publisher is not engaged in rendering professionalservices and neither the publisher nor the author shall be liable for damages arising herefrom. If professionaladvice or other expert assistance is required, the services of a competent professional should be sought.

MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does notwarrant the accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® softwareor related products does not constitute endorsement or sponsorship by The MathWorks of a particularpedagogical approach or particular use of the MATLAB® software.

Library of Congress Cataloging-in-Publication Data

Farhang-Boroujeny, B.Adaptive filters : theory and applications / Behrouz Farhang-Boroujeny. – Second edition.

pages cmIncludes bibliographical references and index.ISBN 978-1-119-97954-8 (cloth)1. Adaptive filters. 2. Adaptive signal processing. I. Title.TK7872.F5F37 2013621.3815′324 – dc23

2012050968

A catalogue record for this book is available from the British Library.

ISBN: 978-1-119-97954-8

Set in 10/12 Times by Laserwords Private Limited, Chennai, India

To Diana for her continuous support, understandingand love throughout my career

Contents

Preface xvii

Acknowledgments xxi

1 Introduction 11.1 Linear Filters 11.2 Adaptive Filters 21.3 Adaptive Filter Structures 31.4 Adaptation Approaches 7

1.4.1 Approach Based on Wiener Filter Theory 71.4.2 Method of Least-Squares 8

1.5 Real and Complex Forms of Adaptive Filters 91.6 Applications 9

1.6.1 Modeling 91.6.2 Inverse Modeling 111.6.3 Linear Prediction 151.6.4 Interference Cancellation 20

2 Discrete-Time Signals and Systems 282.1 Sequences and z-Transform 282.2 Parseval’s Relation 322.3 System Function 332.4 Stochastic Processes 35

2.4.1 Stochastic Averages 352.4.2 z-Transform Representations 372.4.3 The Power Spectral Density 382.4.4 Response of Linear Systems to Stochastic Processes 412.4.5 Ergodicity and Time Averages 44Problems 44

3 Wiener Filters 483.1 Mean-Squared Error Criterion 483.2 Wiener Filter – Transversal, Real-Valued Case 50

viii Contents

3.3 Principle of Orthogonality 553.4 Normalized Performance Function 573.5 Extension to Complex-Valued Case 583.6 Unconstrained Wiener Filters 61

3.6.1 Performance Function 613.6.2 Optimum Transfer Function 643.6.3 Modeling 663.6.4 Inverse Modeling 693.6.5 Noise Cancellation 74

3.7 Summary and Discussion 79Problems 80

4 Eigenanalysis and Performance Surface 904.1 Eigenvalues and Eigenvectors 904.2 Properties of Eigenvalues and Eigenvectors 914.3 Performance Surface 104

Problems 112

5 Search Methods 1195.1 Method of Steepest Descent 1205.2 Learning Curve 1265.3 Effect of Eigenvalue Spread 1305.4 Newton’s Method 1315.5 An Alternative Interpretation of Newton’s Algorithm 133

Problems 135

6 LMS Algorithm 1396.1 Derivation of LMS Algorithm 1396.2 Average Tap-Weight Behavior of the LMS Algorithm 1416.3 MSE Behavior of the LMS Algorithm 144

6.3.1 Learning Curve 1466.3.2 Weight-Error Correlation Matrix 1496.3.3 Excess MSE and Misadjustment 1516.3.4 Stability 1536.3.5 The Effect of Initial Values of Tap Weights

on the Transient Behavior of theLMS Algorithm 155

6.4 Computer Simulations 1566.4.1 System Modeling 1566.4.2 Channel Equalization 1586.4.3 Adaptive Line Enhancement 1636.4.4 Beamforming 165

6.5 Simplified LMS Algorithms 1676.6 Normalized LMS Algorithm 1706.7 Affine Projection LMS Algorithm 1736.8 Variable Step-Size LMS Algorithm 177

Contents ix

6.9 LMS Algorithm for Complex-Valued Signals 1796.10 Beamforming (Revisited) 1826.11 Linearly Constrained LMS Algorithm 186

6.11.1 Statement of the Problem and Its Optimal Solution 1866.11.2 Update Equations 1876.11.3 Extension to the Complex-Valued Case 188Problems 190Appendix 6A: Derivation of Eq. (6.39) 205

7 Transform Domain Adaptive Filters 2077.1 Overview of Transform Domain Adaptive Filters 2087.2 Band-Partitioning Property of Orthogonal Transforms 2107.3 Orthogonalization Property of Orthogonal Transforms 2117.4 Transform Domain LMS Algorithm 2137.5 Ideal LMS-Newton Algorithm and Its Relationship with TDLMS 2157.6 Selection of the Transform T 216

7.6.1 A Geometrical Interpretation 2167.6.2 A Useful Performance Index 2207.6.3 Improvement Factor and Comparisons 2217.6.4 Filtering View 224

7.7 Transforms 2297.8 Sliding Transforms 230

7.8.1 Frequency Sampling Filters 2307.8.2 Recursive Realization of Sliding Transforms 2317.8.3 Nonrecursive Realization of Sliding Transforms 2347.8.4 Comparison of Recursive and Nonrecursive Sliding Transforms 238

7.9 Summary and Discussion 242Problems 243

8 Block Implementation of Adaptive Filters 2518.1 Block LMS Algorithm 2528.2 Mathematical Background 255

8.2.1 Linear Convolution Using the Discrete Fourier Transform 2558.2.2 Circular Matrices 2578.2.3 Window Matrices and Matrix Formulation of the

Overlap-Save Method 2588.3 The FBLMS Algorithm 260

8.3.1 Constrained and Unconstrained FBLMS Algorithms 2618.3.2 Convergence Behavior of the FBLMS Algorithm 2628.3.3 Step-Normalization 2638.3.4 Summary of the FBLMS Algorithm 2648.3.5 FBLMS Misadjustment Equations 2668.3.6 Selection of the Block Length 266

8.4 The Partitioned FBLMS Algorithm 2678.4.1 Analysis of the PFBLMS Algorithm 2698.4.2 PFBLMS Algorithm with M > L 271

x Contents

8.4.3 PFBLMS Misadjustment Equations 2748.4.4 Computational Complexity and Memory Requirement 2748.4.5 Modified Constrained PFBLMS Algorithm 276

8.5 Computer Simulations 276Problems 279Appendix 8A: Derivation of a Misadjustment Equation for the BLMS

Algorithm 285Appendix 8B: Derivation of Misadjustment Equations for the FBLMS

Algorithms 288

9 Subband Adaptive Filters 2949.1 DFT Filter Banks 295

9.1.1 Weighted Overlap–Add Method for Realizationof DFT Analysis Filter Banks 296

9.1.2 Weighted Overlap–Add Method for Realizationof DFT Synthesis Filter Banks 297

9.2 Complementary Filter Banks 2999.3 Subband Adaptive Filter Structures 3039.4 Selection of Analysis and Synthesis Filters 3049.5 Computational Complexity 3079.6 Decimation Factor and Aliasing 3089.7 Low-Delay Analysis and Synthesis Filter Banks 310

9.7.1 Design Method 3109.7.2 Filters Properties 312

9.8 A Design Procedure for Subband Adaptive Filters 3139.9 An Example 3169.10 Comparison with FBLMS Algorithm 318

Problems 319

10 IIR Adaptive Filters 32210.1 Output Error Method 32310.2 Equation Error Method 32710.3 Case Study I: IIR Adaptive Line Enhancement 332

10.3.1 IIR ALE Filter, W(z) 33310.3.2 Performance Functions 33410.3.3 Simultaneous Adaptation of s and w 33510.3.4 Robust Adaptation of w 33710.3.5 Simulation Results 337

10.4 Case Study II: Equalizer Design for Magnetic Recording Channels 34310.4.1 Channel Discretization 34410.4.2 Design Steps 34510.4.3 FIR Equalizer Design 34510.4.4 Conversion from FIR into IIR Equalizer 34710.4.5 Conversion from z Domain into s Domain 348

Contents xi

10.4.6 Numerical Results 34810.5 Concluding Remarks 349

Problems 352

11 Lattice Filters 35511.1 Forward Linear Prediction 35511.2 Backward Linear Prediction 35711.3 Relationship Between Forward and Backward Predictors 35911.4 Prediction-Error Filters 35911.5 Properties of Prediction Errors 36011.6 Derivation of Lattice Structure 36211.7 Lattice as an Orthogonalization Transform 36711.8 Lattice Joint Process Estimator 36911.9 System Functions 37011.10 Conversions 370

11.10.1 Conversion Between Lattice and Transversal Predictors 37111.10.2 Levinson–Durbin Algorithm 37211.10.3 Extension of Levinson–Durbin Algorithm 374

11.11 All-Pole Lattice Structure 37611.12 Pole-Zero Lattice Structure 37611.13 Adaptive Lattice Filter 378

11.13.1 Discussion and Simulations 38011.14 Autoregressive Modeling of Random Processes 38311.15 Adaptive Algorithms Based on Autoregressive Modeling 385

11.15.1 Algorithms 38611.15.2 Performance Analysis 39011.15.3 Simulation Results and Discussion 394Problems 400Appendix 11A: Evaluation of E[ua(n)xT(n)K(n)x(n)uT

a (n)] 407Appendix 11B: Evaluation of the parameter γ 408

12 Method of Least-Squares 41012.1 Formulation of Least-Squares Estimation for a

Linear Combiner 41112.2 Principle of Orthogonality 41212.3 Projection Operator 41512.4 Standard Recursive Least-Squares Algorithm 416

12.4.1 RLS Recursions 41612.4.2 Initialization of the RLS Algorithm 41812.4.3 Summary of the Standard RLS Algorithm 419

12.5 Convergence Behavior of the RLS Algorithm 42112.5.1 Average Tap-Weight Behavior of the RLS Algorithm 42212.5.2 Weight-Error Correlation Matrix 42212.5.3 Learning Curve 423

xii Contents

12.5.4 Excess MSE and Misadjustment 42612.5.5 Initial Transient Behavior of the RLS Algorithm 427Problems 430

13 Fast RLS Algorithms 43313.1 Least-Squares Forward Prediction 43413.2 Least-Squares Backward Prediction 43513.3 Least-Squares Lattice 43713.4 RLSL Algorithm 440

13.4.1 Notations and Preliminaries 44013.4.2 Update Recursion for the Least-Squares Error Sums 44313.4.3 Conversion Factor 44413.4.4 Update Equation for Conversion Factor 44613.4.5 Update Equation for Cross-Correlations 44713.4.6 RLSL Algorithm Using A Posteriori Errors 45013.4.7 RLSL Algorithm with Error Feedback 450

13.5 FTRLS Algorithm 45313.5.1 Derivation of the FTRLS Algorithm 45413.5.2 Summary of the FTRLS Algorithm 45813.5.3 Stabilized FTRLS Algorithm 458Problems 460

14 Tracking 46314.1 Formulation of the Tracking Problem 46314.2 Generalized Formulation of LMS Algorithm 46414.3 MSE Analysis of the Generalized LMS Algorithm 46514.4 Optimum Step-Size Parameters 46914.5 Comparisons of Conventional Algorithms 47114.6 Comparisons Based on Optimum Step-Size Parameters 47514.7 VSLMS: An Algorithm with Optimum Tracking Behavior 477

14.7.1 Derivation of VSLMS Algorithm 47714.7.2 Variations and Extensions 47814.7.3 Normalization of the Parameter ρ 48014.7.4 Computer Simulations 480

14.8 RLS Algorithm with Variable Forgetting Factor 48514.9 Summary 486

Problems 488

15 Echo Cancellation 49215.1 The Problem Statement 49215.2 Structures and Adaptive Algorithms 495

15.2.1 Normalized LMS (NLMS) Algorithm 49615.2.2 Affine Projection LMS (APLMS) Algorithm 49915.2.3 Frequency Domain Block LMS Algorithm 50115.2.4 Subband LMS Algorithm 502

Contents xiii

15.2.5 LMS-Newton Algorithm 50215.2.6 Numerical Results 505

15.3 Double-Talk Detection 51215.3.1 Coherence Function 51215.3.2 Double-Talk Detection Using the Coherence Function 51315.3.3 Numerical Evaluation of the Coherence Function 51315.3.4 Power-Based Double-Talk Detectors 51715.3.5 Numerical Results 518

15.4 Howling Suppression 52115.4.1 Howling Suppression Through Notch Filtering 52115.4.2 Howling Suppression by Spectral Shift 521

15.5 Stereophonic Acoustic Echo Cancellation 52415.5.1 The Fundamental Problem 52615.5.2 Reducing Coherence Between x1(n) and x2(n) 52815.5.3 The LMS-Newton Algorithm for Stereophonic Systems 532Appendix 15A: Multitaper method 542Appendix 15B: Derivation of the Two-Channel Levinson–Durbin

Algorithm 549

16 Active Noise Control 55116.1 Broadband Feedforward Single-Channel ANC 553

16.1.1 System Block Diagram in the Absenceof the Secondary Path S1(z) 554

16.1.2 Filtered-X LMS Algorithm 55516.1.3 Convergence Analysis 55516.1.4 Adding the Secondary Path S1(z) 557

16.2 Narrowband Feedforward Single-Channel ANC 55916.2.1 Waveform Synthesis Method 56016.2.2 Adaptive Notch Filters 569

16.3 Feedback Single-Channel ANC 57316.4 Multichannel ANC Systems 577

16.4.1 MIMO Blocks/Transfer Functions 57816.4.2 Derivation of the LMS Algorithm for MIMO Adaptive Filters 579Appendix 16A: Derivation of Eq. (16.46) 582Appendix 16B: Derivation of Eq. (16.53) 583

17 Synchronization and Equalization in Data Transmission Systems 58417.1 Continuous Time Channel Model 58517.2 Discrete Time Channel Model and Equalizer Structures 589

17.2.1 Symbol-Spaced Equalizer 59017.2.2 Fractionally Spaced Equalizer 59117.2.3 Decision Feedback Equalizer 592

17.3 Timing Recovery 59317.3.1 Cost Function 59317.3.2 The Optimum Timing Phase 595

xiv Contents

17.3.3 Improving the Cost Function 59817.3.4 Algorithms 60017.3.5 Early-Late Gate Timing Recovery 60017.3.6 Gradient-Based Algorithm 604

17.4 Equalizers Design and Performance Analysis 60617.4.1 Wiener–Hopf Equation for Symbol-Spaced Equalizers 60617.4.2 Numerical Examples 613

17.5 Adaptation Algorithms 61717.6 Cyclic Equalization 618

17.6.1 Symbol-Spaced Cyclic Equalizer 61817.6.2 Fractionally Spaced Cyclic Equalizer 62517.6.3 Alignment of s(n) and x(n) 62717.6.4 Carrier and Timing Phase Acquisition and Tracking 627

17.7 Joint Timing Recovery, Carrier Recovery, and Channel Equalization 62817.8 Maximum Likelihood Detection 62917.9 Soft Equalization 631

17.9.1 Soft MMSE Equalizer 63317.9.2 Statistical Soft Equalizer 63517.9.3 Iterative Channel Estimation and Data Detection 641

17.10 Single-Input Multiple-Output Equalization 64317.11 Frequency Domain Equalization 645

17.11.1 Packet Structure 64617.11.2 Frequency Domain Equalizer 64717.11.3 Packet Structure for Fast Tracking 64817.11.4 Summary 649

17.12 Blind Equalization 64917.12.1 Examples of Kurtosis 65117.12.2 Cost Function 65217.12.3 Blind Adaptation Algorithm 654Problems 654

18 Sensor Array Processing 65918.1 Narrowband Sensor Arrays 660

18.1.1 Array Topology and Parameters 66018.1.2 Signal subspace, noise subspace, and spectral factorization 66218.1.3 Direction of Arrival Estimation 66518.1.4 Beamforming Methods 670

18.2 Broadband Sensor Arrays 67818.2.1 Steering 67918.2.2 Beamforming Methods 680

18.3 Robust Beamforming 68318.3.1 Soft-Constraint Minimization 68618.3.2 Diagonal Loading Method 68818.3.3 Methods Based on Sample Matrix Inversion 690Problems 692

Contents xv

19 Code Division Multiple Access Systems 69519.1 CDMA Signal Model 695

19.1.1 Chip-Spaced Users-Synchronous Model 69619.1.2 Chip-Spaced Users-Asynchronous Model 69819.1.3 Fractionally Spaced Model 699

19.2 Linear Detectors 69919.2.1 Conventional Detector: The Matched Filter Detector 70019.2.2 Decorrelator Detector 70019.2.3 Minimum Mean-Squared Error (Optimal) Detector 70119.2.4 Minimum Output Energy (Blind) Detector 70319.2.5 Soft Detectors 707

19.3 Adaptation Methods 70719.3.1 Conventional Detector 70719.3.2 Decorrelator Detector 70719.3.3 MMSE Detector 70819.3.4 MOE Detector 70819.3.5 Soft Detectors 709Problems 709

20 OFDM and MIMO Communications 71120.1 OFDM Communication Systems 711

20.1.1 The Principle of OFDM 71120.1.2 Packet Structure 71420.1.3 Carrier Acquisition 71620.1.4 Timing Acquisition 71720.1.5 Channel Estimation and Frequency Domain Equalization 71720.1.6 Estimation of Rhh and Rνν 72020.1.7 Carrier-Tracking Methods 72120.1.8 Channel-Tracking Methods 730

20.2 MIMO Communication Systems 73020.2.1 MIMO Channel Model 73220.2.2 Transmission Techniques for Space-Diversity Gain 73220.2.3 Transmission Techniques and MIMO Detectors

for Space-Multiplexing Gain 73720.2.4 Channel Estimation Methods 741

20.3 MIMO–OFDM 743Problems 743

References 746

Index 761

Preface

This book has grown out of the author’s research work and teaching experience in thefield of adaptive signal processing as well as signal processing applications to a variety ofcommunication systems. The second edition of this book, while preserving the presentationof the basic theory of adaptive filters as in the first edition, expands significantly on abroad range of applications of adaptive filters. Six new chapters are added that look intovarious applications of adaptive filters.

This book is designed to be used as a text to teach graduate-level courses in adaptivefilters at different levels. It is also intended to serve as a technical reference for practicingengineers.

A typical one-semester introductory course on adaptive filters may cover Chapters 1,3–6, and 12, and the first half of Chapter 11, in depth. Chapter 2, which contains a shortreview of the basic concepts of the discrete-time signals and systems, and some relatedconcepts from random signal analyses, may be left as self-study material for students.Selected parts of the rest of this book may also be taught in the same semester, or,broader range of chapters may be used for a second semester course on advanced topicsand applications.

In the study of adaptive filters, computer simulations constitute an important supple-mental component to theoretical analyses and deductions. Often, theoretical developmentsand analyses involve a number of approximations and/or assumptions. Hence, computersimulations become necessary to confirm the theoretical results. Apart from this, com-puter simulation turns out to be a necessity in the study of adaptive filters for gaining anin-depth understanding of the behavior and properties of the various adaptive algorithms.MATLAB® from MathWorks Inc. appears to be the most commonly used software simula-tion package. Throughout this book, MATLAB® is used to present a number of simulationresults to clarify and/or confirm the theoretical developments. The programs as well asdata files used for generating these results can be downloaded from the accompanyingwebsite of this book at www.wiley.com/go/adaptive_filters

Another integral part of this text is exercise problems at the end of chapters. With theexception of the first few chapters, two kinds of exercise problems are provided in eachchapter:

1. The usual problem exercises. These problems are designed to sharpen the readers’skill in theoretical development. They are designed to extend results developedin the text and illustrate applications to practical problems. Solutions to these

xviii Preface

problems are available to instructors on the companion website to the book:www.wiley.com/go/adaptive_filters

2. Simulation-oriented problems. These involve computer simulations and are designedto enhance the readers’ understanding on the behavior of the different adaptive algo-rithms that are introduced in the text. Most of these problems are based on theMATLAB® programs, which are provided on the accompanying website. In addition,there are also other (open-ended) simulation-oriented problems, which are designed tohelp the readers to develop their own programs and prepare them to experiment withpractical problems.

The book assumes that the reader has some background of discrete-time signals andsystems (including an introduction to linear system theory and random signal analysis),complex variable theory, and matrix algebra. However, a review of these topics is providedin Chapters 2 and 4.

This book starts with a general overview of adaptive filters in Chapter 1. Many examplesof applications such as system modeling, channel equalization, echo cancellation, andantenna arrays are reviewed in this chapter. This follows with a brief review of discrete-time signals and systems in Chapter 2, which puts the related concepts in a frameworkappropriate for the rest of this book.

In Chapter 3, we introduce a class of optimum linear systems collectively known asWiener filters. Wiener filters are fundamental to the implementation of adaptive filters.We note that the cost function used to formulate the Wiener filters is an elegant choice,leading to a mathematically tractable problem. We also discuss the unconstrained Wienerfilters with respect to causality and duration of the filter impulse response. This studyreveals many interesting aspects of Wiener filters and establishes a good foundation forthe study of adaptive filters for the rest of this book. In particular, we find that, in thelimit, when the filter length tends to infinity, a Wiener filter treats different frequencycomponents of underlying processes separately. Numerical examples reveal that when thefilter length is limited, separation of frequency components may be replaced by separationof frequency bands within a good approximation. This treatment of adaptive filters, whichis pursued throughout this book, turns out to be an enlightening engineering approach forthe study of adaptive filters.

Eigenanalysis is an essential mathematical tool for the study of adaptive filters. Athorough treatment of this topic is covered in the first half of Chapter 4. The secondhalf of this chapter gives an analysis of the performance surface of transversal Wienerfilters. This is followed by search methods, which are introduced in Chapter 5. The searchmethods discussed in this chapter are idealized versions of the statistical search methodsthat are used in practice for actual implementation of adaptive filters. They are idealized inthe sense that the statistics of the underlying processes are assumed to be known a priori.

The celebrated least-mean-square (LMS) algorithm is introduced in Chapter 6 andextensively studied in Chapters 7–11. The LMS algorithm, which was first proposedby Widrow and Hoff in 1960’s, is the most widely used adaptive filtering algorithm, inpractice, owing to its simplicity and robustness to signal statistics.

Chapters 12 and 13 are devoted to the method of least-squares. This discussion, althoughbrief, gives the basic concept of the method of least-squares and highlights its advantagesand disadvantages compared to the LMS-based algorithms. In Chapter 13, the reader is

Preface xix

introduced to the fast versions of least-squares algorithms. Overall, these two chapters laya good foundation for the reader to continue his/her study of this subject with referenceto more advanced books and/or papers.

The problem of tracking is discussed in Chapter 14. In the context of a system modelingproblem, we present a generalized formulation of the LMS algorithm, which covers mostof the algorithms that are discussed in the previous chapters of this book, thus bringing acommon platform for comparison of different algorithms. We also discuss how the step-size parameter(s) of the LMS algorithm and the forgetting factor of the RLS algorithmmay be optimized for achieving good tracking behavior.

Chapters 15–20 cover a range of applications where the theoretical results of theprevious chapters are applied to a wide range of practical problems. Chapter 15 presentsa number of practical problems related to echo cancelers. The chapter emphasis is onacoustic echo cancellation that is encountered in teleconferencing applications. In suchapplications, one has to deal with specific problems that do not fall into the domain ofthe traditional theory of adaptive filters. For instance, when both parties at the two sidesof the conferencing line talk simultaneously, their respective signals interfere with oneanother and hence, the adaptation of both echo cancelers on the two sides of the line maybe disrupted. Therefore, specific double-talk detection methods should be designed. Thestereophonic acoustic echo cancelers that have gained some momentums in recent yearsare also discussed in detail.

Chapter 16 presents and discusses the underlying problems related to active noisecancellation control, which are also somewhat different from the traditional adaptivefiltering problems.

Chapters 17 is devoted to the issues related to synchronization and channel equalizationin communication systems. Although many fundamentals of the classical adaptive filterstheory have been developed in the context of channel equalization, there are a numberof specific issues in the domain of communication systems that can be only presented asnew concepts, which may be thought of as extensions of the classical theory of adaptivefilters. Many such extensions are presented in this chapter.

Sensor array processing and code division multiple access (CDMA) are two areas whereadaptive filters have been used extensively. Although these seem to be two very differentapplications, there are a number of similarities that if understood allows one to use resultsof one application for the other as well. As sensor array processing has been developedwell ahead of CDMA, we follow this historical development and present sensor arrayprocessing techniques in Chapter 18, followed by a presentation of CDMA theory andthe relevant algorithms in Chapter 19.

The recent advancement in adaptive filters as applied to the design and implementationmulticarrier systems (respectively, orthogonal frequency division multiplexing–OFDM)and communication systems with multiple antennas at both transmitter and receiver sides(known as multi-input multi-output–MIMO) are discussed in Chapter 20. This chapterexplains some practical issues related to these modern signal processing techniques andpresents a few solutions that have been adopted in the current standards, e.g., WiFi,WiMax, and LTE.

The following notations are adopted in this book. We use nonbold lowercase letters forscalar quantities, bold lowercase for vectors, and bold uppercase for matrices. Nonbolduppercase letters are used for functions of variables, such as H(z), and lengths/dimensions

xx Preface

of vectors/matrices. The lowercase letter “n” is used for the time index. In the case ofblock processing algorithms, such as those discussed in Chapters 8 and 9, we reserve thelowercase letter “k” as the block index. The time and block indices are put in brackets,while subscripts are used to refer to elements of vectors and matrices. For example, theith element of the time-varying tap-weight vector w(n) is denoted as ωi(n). The super-scripts “T” and “H” denote vector or matrix transposition and Hermitian transposition,respectively. We keep all vectors in column form. More specific notations are explainedin the text as and when found necessary.

Behrouz Farhang-Boroujeny

Acknowledgments

Development of the first edition of this book as well as the present edition has relied ona variety of contributions from my students in the past 20 years. I wish to thank all ofthem for their support, enthusiasm, and encouragements.

I am grateful to Professor Stephen Elliott, University of Southampton, UK, for review-ing Chapter 16 and making many valuable suggestions. I am also thankful to ProfessorVellenki Umapathi Reddy for reviewing Chapters 2–7, providing valuable suggestions,and giving me moral support. My special thanks go to Dr. George Mathew of HitachiGlobal Storage Technologies for critically reviewing the entire manuscript of the firstedition of this book.

Many colleagues who adopted the first edition of this book for their classes have beenthe source of encouragement for me to take the additional task of preparing the secondedition of this book. I hope they find this edition more useful in broadening the knowledgeof our graduate students by giving them many examples of advanced applications ofadaptive filters.

1Introduction

As we begin our study of “adaptive filters,” it may be worth trying to understand themeaning of the terms adaptive and filters in a very general sense. The adjective “adaptive”can be understood by considering a system that is trying to adjust itself so as to respondto some phenomenon that is taking place in its surroundings. In other words, the systemtries to adjust its parameters with the aim of meeting some well-defined goal or target thatdepends on the state of the system as well as its surrounding. This is what “adaptation”means. Moreover, there is a need to have a set of steps or certain procedure by whichthis process of “adaptation” is carried out. And finally, the “system” that carries out andundergoes the process of “adaptation” is called by the more technical, yet general enough,name “filter” – a term that is very familiar to and a favorite of any engineer. Clearly,depending on the time required to meet the final target of the adaptation process, whichwe call convergence time, and the complexity/resources that are available to carry out theadaptation, we can have a variety of adaptation algorithms and filter structures. From thispoint of view, we may summarize the contents/contribution of this book as “the study ofsome selected adaptive algorithms and their implementations along with the associatedfilter structures from the points of view of their convergence and complexity performance.”

1.1 Linear Filters

The term filter is commonly used to refer to any device or system that takes a mixture ofparticles/elements from its input and processes them according to some specific rules togenerate a corresponding set of particles/elements at its output. In the context of signalsand systems, particles/elements are the frequency components of the underlying signalsand, traditionally, filters are used to retain all the frequency components that belong to aparticular band of frequencies, while rejecting the rest of them, as much as possible. In amore general sense, the term filter may be used to refer to a system that reshapes the fre-quency components of the input to generate an output signal with some desirable features,and this is how we view the concept of filtering throughout the chapters which follow.

Filters (or systems, in general) may be either linear or nonlinear. In this book, weconsider only linear filters and our emphasis will also be on discrete-time signals and sys-tems. Thus, all the signals will be represented by sequences, such as x(n). The most basicfeature of linear systems is that their behavior is governed by the principle of superposi-tion. This means that if the responses of a linear discrete-time system to input sequences

Adaptive Filters: Theory and Applications, Second Edition. Behrouz Farhang-Boroujeny.© 2013 John Wiley & Sons, Ltd. Published 2013 by John Wiley & Sons, Ltd.

2 Adaptive Filters

⊕input signal output signal error signal

desiredsignal+

−filter

Figure 1.1 Schematic diagram of a filter emphasizing its role of reshaping the input signal tomatch the desired signal.

x1(n) and x2(n) are y1(n) and y2(n), respectively, the response of the same system to theinput sequence x(n) = ax1(n) + bx2(n), where a and b are arbitrary constants, will bey(n) = ay1(n) + by2(n). This property leads to many interesting results in “linear systemtheory.” In particular, a linear system is completely characterized by its impulse responseor the Fourier transform of its impulse response known as transfer function. The transferfunction of a system at any frequency is equal to its gain at that frequency. In other words,in the context of our discussion above, we may say that the transfer function of a systemdetermines how the various frequency components of its input are reshaped by the system.

Figure 1.1 depicts a general schematic diagram of a filter emphasizing the purpose forwhich it is used in different problems addressed/discussed in this book. In particular, thefilter is used to reshape a certain input signal in such a way that its output is a good estimateof the given desired signal. The process of selecting the filter parameters (coefficients) soas to achieve the best match between the desired signal and the filter output is often doneby optimizing an appropriately defined performance function. The performance functioncan be defined in a statistical or deterministic framework. In the statistical approach,the most commonly used performance function is the mean-squared value of the errorsignal, that is, difference between the desired signal and the filter output. For stationaryinput and desired signals, minimizing the mean squared error (MSE) results in the well-known Wiener filter, which is said to be optimum in the mean-square sense. The subjectof Wiener filters is extensively covered in Chapter 3. Most of the adaptive algorithmsthat are studied in this book are practical solutions to Wiener filters. In the deterministicapproach, the usual choice of performance function is a weighted sum of the squared errorsignal. Minimizing this function results in a filter that is optimum for the given set ofdata. However, under some assumptions on certain statistical properties of the data, thedeterministic solution will approach the statistical solution, that is, the Wiener filter, forlarge data lengths. Chapters 12 and 13 deal with the deterministic approach in detail. Werefer the reader to Section 1.4 for a brief overview of the adaptive formulations under thestochastic (i.e., statistical) and deterministic frameworks.

1.2 Adaptive Filters

As we mentioned in the previous section, the filter required for estimating the given desiredsignal can be designed using either the stochastic or the deterministic formulations. Inthe deterministic formulation, the filter design requires the computation of certain averagequantities using the given set of data that the filter should process. On the other hand, thedesign of Wiener filter (i.e., in the stochastic approach) requires a priori knowledge of

Introduction 3

the statistics of the underlying signals. Strictly speaking, a large number of realizations ofthe underlying signal sequences are required for reliably estimating these statistics. Thisprocedure is practically not feasible because we usually have only one realization for eachof the signal sequences. To resolve this problem, it is assumed that the underlying signalsequences are ergodic, which means that they are stationary and their statistical and timeaverages are identical. Thus, using the time averages, Wiener filters can be designed, eventhough there is only one realization for each of the signal sequences.

Although, direct measurement of the signal averages to obtain the necessary informationfor the design of Wiener or other optimum filters is possible, in most of the applications,the signal averages (statistics) are used in an indirect manner. All the algorithms that arecovered in this book take the output error of the filter, correlate that with the samplesof filter input in some way, and use the result in a recursive equation to adjust the filtercoefficients iteratively. The reasons for solving the problem of adaptive filtering in aniterative manner are as follows:

1. Direct computation of the necessary averages and their application for computing thefilter coefficients requires accumulation of a large amount of signal samples. Iterativesolutions, on the other hand, do not require accumulation of signal samples, therebyresulting in a significant amount of saving in memory.

2. Accumulation of signal samples and their postprocessing to generate the filter output,as required in noniterative solutions, introduces a large delay in the filter output. This isunacceptable in many applications. Iterative solutions, on the contrary, do not introduceany significant delay in the filter output.

3. The use of iterations results in adaptive solutions with some tracking capability. Thatis, if the signal statistics are changing with time, the solution provided by an iterativeadjustment of the filter coefficients will be able to adapt to the new statistics.

4. Iterative solutions, in general, are much simpler to code in software or implement inhardware than their noniterative counterparts.

1.3 Adaptive Filter Structures

The most commonly used structure in the implementation of adaptive filters is thetransversal structure, depicted in Figure 1.2. Here, the adaptive filter has a single input,x(n), and an output, y(n). The sequence d(n) is the desired signal. The output, y(n), isgenerated as a linear combination of the delayed samples of the input sequence, x(n),according to Equation (1.1)

y(n) =N−1∑i=0

wi(n)x(n − i) (1.1)

where wi(n)’s are the filter tap weights (coefficients) and N is the filter length. We referto the input samples, x(n − i), for i = 0, 1, . . . , N − 1, as the filter tap inputs. The tapweights, wi(n)’s, which may vary with time, are controlled by the adaptation algorithm.

In some applications, such as beamforming (Section 1.6.4), the filter tap inputs are notthe delayed samples of a single input. In such cases, the structure of the adaptive filter

4 Adaptive Filters

⊕

z−1 z−1 z−1

⊕adaptationalgorithm

x(n) x(n − 1) x(n − N + 1)

w0(n) w1(n) wN−1(n)

y(n)

d(n)e(n)

• • •

+

−

Figure 1.2 Adaptive transversal filter.

⊕

⊕adaptationalgorithm

w0(n) w1(n) wN−1(n)

y(n)

d(n)e(n)

• • •

+

−

...

x0(n) x1(n) xN−1(n)

Figure 1.3 Adaptive linear combiner.

assumes the form shown in Figure 1.3. This is called a linear combiner as its output is alinear combination of the different signals received at its tap inputs:

y(n) =N−1∑i=0

wi(n)xi(n) (1.2)

Note that the linear combiner structure is more general than the transversal. The latter, asa special case of the former, can be obtained by choosing xi(n) = x(n − i).

Introduction 5

The structures of Figures 1.2 and 1.3 are those of the nonrecursive filters, that is,computation of filter output does not involve any feedback mechanism. We also refer toFigure 1.2 as a finite-impulse response (FIR) filter as its impulse response is of finite dura-tion in time. An infinite-impulse response (IIR) filter is governed by recursive equationssuch as (Figure 1.4)

y(n) =N−1∑i=0

ai(n)x(n − i) +M−1∑i=1

bi(n)y(n − i) (1.3)

where ai(n) and bi(n) are the forward and feedback tap weights, respectively. IIR filtershave been used in many applications. However, as we shall see in the later chapters,because of the many difficulties involved in the adaptation of IIR filters, their applicationin the area of adaptive filters is rather limited. In particular, they can easily becomeunstable because their poles may get shifted out of the unit circle (i.e., |z| = 1, in thez-plane, Chapter 2) by the adaptation process. Moreover, the performance function (e.g.,MSE as a function of filter coefficients) of an IIR filter usually has many local minimapoints. This may result in convergence of the filter to one of the local minima and not tothe desired global minimum point of the performance function. On the contrary, the MSEfunctions of FIR filter and linear combiner are well-behaved quadratic functions with asingle minimum point, which can easily be found through various adaptive algorithms.Because of these points, the nonrecursive filters are the sole candidates in most of theapplications of adaptive filters. Hence, most of our discussions in the subsequent chaptersare limited to the nonrecursive filters. The IIR-adaptive filters with two specific examplesof their applications are discussed in Chapter 10.

The FIR and IIR structures shown in Figures 1.2 and 1.4 are obtained by direct realiza-tion of the respective difference equations (1.1) and (1.3). These filters may alternativelybe implemented using the lattice structures. The lattice structures, in general, are morecomplicated than the direct implementations. However, in certain applications, they havesome advantages which make them better candidates than the direct forms. For instance, inthe application of linear prediction for speech processing where we need to realize all-pole(IIR) filters, the lattice structure can be more easily controlled to prevent possible instabil-ity of the filter. Derivation of lattice structures for both FIR and IIR filters are presented inChapter 11. Also, in the implementation of the method of least-squares (Section 1.4.2), theuse of lattice structure leads to a computationally efficient algorithm known as recursiveleast-squares (RLS) lattice. A derivation of this algorithm is presented in Chapter 13.

The FIR and IIR filters which were discussed above are classified as linear filtersbecause their outputs are obtained as linear combinations of the present and past samplesof input and, in the case of IIR filter, the past samples of the output also. Although mostapplications are restricted to the use of linear filters, nonlinear adaptive filters becomenecessary in some applications where the underlying physical phenomena to be modeledare far from being linear. A typical example is magnetic recording where the recordingchannel becomes nonlinear at high densities because of the interaction among the magne-tization transitions written on the medium. The Volterra series representation of systemsis usually used in such applications. The output, y(n), of a Volterra system is related to

6 Adaptive Filters

⊕

⊗z−1

z−1

z−1

⊗

⊗

⊗

⊗z−1

z−1

z−1

⊗

⊗

x(n) y(n)

x(n − 1)

x(n − 2)

x(n − N + 1) y(n − M + 1)

y(n − 1)

y(n − 2)

a0(n)

a1(n)

a2(n)

aN−1(n) bN−1(n)

b2(n)

b1(n)

Figure 1.4 The structure of an IIR filter.

its input, x(n), according to the equation

y(n) = w0,0(n) +∑

i

w1,i (n)x(n − i)

+∑i,j

w2,i,j (n)x(n − i)x(n − j)

+∑i,j,k

w3,i,j,k(n)x(n − i)x(n − j)x(n − k) + . . . (1.4)

where w0,0(n), w1,i (n)’s, w2,i,j (n)’s, w1,i,j,k(n)’s, . . . are filter coefficients. In this book,we do not discuss the Volterra filters any further. However, we note that all the summa-tions in Eq. (1.4) may be put together and the Volterra filter may be thought of as a linearcombiner whose inputs are determined by the delayed samples of x(n) and their cross-multiplications. Noting this, we find that the extension of most of the adaptive filteringalgorithms to the Volterra filters is straightforward.