Quantization Noise

Quantization Noise

If you are working in digital signal processing, control, or numerical analysis, you willfind this authoritative treatment of quantization noise (roundoff error) to be aninvaluable resource.

Do you know where the theory of quantization noise comes from, and under whatcircumstances it is true? Expert authors, including the founder of the field andformulator of the theory of quantization noise, Bernard Widrow, answer these andother important practical questions. They describe and analyze uniform quantization,floating-point quantization, and their applications in detail.

Key features include:

– heuristic explanations along with rigorous proofs;– worked examples, so that theory is understood through examples;– focus on practical cases, engineering approach;– analysis of floating-point roundoff;– dither techniques and implementation issues analyzed;– program package for Matlab� available on the web, for simulation and analysis

of fixed-point and floating-point roundoff;– homework problems and solutions manual; and– actively maintained website with additional text on special topics on quantization

noise.

The additional resources are available online throughwww.cambridge.org/9780521886710

Bernard Widrow, an internationally recognized authority in the field ofquantization, is a Professor of Electrical Engineering at Stanford University,California. He pioneered the field and one of his papers on the topic is the standardreference. He is a Fellow of the IEEE and the AAAS, a member of the US NationalAcademy of Engineering, and the winner of numerous prestigious awards.

István Kollár is a Professor of Electrical Engineering at the Budapest Universityof Technology and Economics. A Fellow of the IEEE, he has been researching thetheory and practice of quantization and roundoff for the last 30 years. He is the authorof about 135 scientific publications and has been involved in several industrialdevelopment projects.

www.cambridge.org© Cambridge University Press

Cambridge University Press978-0-521-88671-0 - Quantization Noise: Roundoff Error in Digital Computation, Signal Processing,Control, and CommunicationsBernard Widrow and Istvan KollarFrontmatterMore information

http://www.cambridge.org/9780521886710

http://www.cambridge.org


Quantization NoiseRoundoff Error in Digital Computation, Signal Processing,Control, and Communications

Bernard Widrow andIstván Kollár






cambridge university pressCambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi

Cambridge University PressThe Edinburgh Building, Cambridge CB2 8RU, UK

Published in the United States of America by Cambridge University Press, New York

www.cambridge.orgInformation on this title: www.cambridge.org/9780521886710

© Cambridge University Press 2008

This publication is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place withoutthe written permission of Cambridge University Press.

First published 2008

Printed in the United Kingdom at the University Press, Cambridge

A catalog record for this publication is available from the British Library

ISBN 978-0-521-88671-0 hardback

Cambridge University Press has no responsibility for the persistence oraccuracy of urls for external or third-party internet websites referred toin this publication, and does not guarantee that any content on suchwebsites is, or will remain, accurate or appropriate.






We dedicate this work to our fathers and ourteachers. They influenced our lives and our think-ing in a very positive way.

I would like to dedicate this book to ProfessorsDavid Middleton and John G. Linvill, and to thememory of Professor William K. Linvill and myfather, Moses Widrow.

Bernard Widrow

I would like to dedicate this book to ProfessorsAndras Prekopa and Peter Osvath, and to thememory of Professor Laszlo Schnell and my fa-ther, Lajos Kollar.

Istvan Kollar






Contents

Preface XIX

Acknowledgments XXI

Glossary of Symbols XXIII

Acronyms and Abbreviations XXVII

Part I Background

1 Introduction 31.1 Definition of the Quantizer 31.2 Sampling and Quantization (Analog-to-Digital Conversion) 91.3 Exercises 10

2 Sampling Theory 132.1 Linvill’s Frequency Domain Description of Sampling 142.2 The Sampling Theorem; Recovery of the Time Function from its

Samples 182.3 Anti-Alias Filtering 222.4 A Statistical Description of Quantization, Based on Sampling

Theory 252.5 Exercises 28

3 Probability Density Functions, Characteristic Functions, Moments 313.1 Probability Density Function 313.2 Characteristic Function and Moments 333.3 Joint Probability Density Functions 353.4 Joint Characteristic Functions, Moments, and Correlation

Functions 403.5 First-Order Statistical Description of the Effects of Memoryless

Operations on Signals 43

VII






VIII Contents

3.6 Addition of Random Variables and Other Functions of RandomVariables 46

3.7 The Binomial Probability Density Function 473.8 The Central Limit Theorem 493.9 Exercises 53

Part II Uniform Quantization

4 Statistical Analysis of the Quantizer Output 614.1 PDF and CF of the Quantizer Output 614.2 Comparison of Quantization with the Addition of Independent

Uniformly Distributed Noise, the PQN Model 664.3 Quantizing Theorems I and II 694.4 Recovery of the PDF of the Input Variable x from the PDF of the

Output Variable x ′ 704.5 Recovery of Moments of the Input Variable x from Moments of

the Output Variable x ′ when QT II is Satisfied; Sheppard’sCorrections and the PQN Model 80

4.6 General Expressions of the Moments of the Quantizer Output, andof the Errors of Sheppard’s Corrections: Deviations from the PQNModel 84

4.7 Sheppard’s Corrections with a Gaussian Input 844.8 Summary 854.9 Exercises 87

5 Statistical Analysis of the Quantization Noise 935.1 Analysis of the Quantization Noise and the PQN Model 935.2 Satisfaction of Quantizing Theorems I and II 995.3 Quantizing Theorem III/A 995.4 General Expressions of the First- and Higher-Order Moments of

the Quantization Noise: Deviations from the PQN Model 1025.5 Quantization Noise with Gaussian Inputs 1065.6 Summary 1075.7 Exercises 108

6 Crosscorrelations between Quantization Noise, Quantizer Input,and Quantizer Output 1136.1 Crosscorrelations when Quantizing Theorem II is Satisfied 113

6.1.1 Crosscorrelation between Quantization Noise and theQuantizer Input 113

6.1.2 Crosscorrelation between Quantization Noise and theQuantizer Output 115






Contents IX

6.1.3 Crosscorrelation between the Quantizer Input and theQuantizer Output 116

6.2 General Expressions of Crosscorrelations 1166.2.1 Crosscorrelation between Quantization Noise and the

Quantizer Input 1166.2.2 Crosscorrelation between Quantization Noise and the

Quantizer Output Signal 1196.2.3 Crosscorrelation between the Quantizer Input and Output

Signals 1226.3 Correlation and Covariance between Gaussian Quantizer Input and

Its Quantization Noise 1236.4 Conditions of Orthogonality of Input x and Noise ν: Quantizing

Theorem III/B 1266.5 Conditions of Uncorrelatedness between x and ν: Quantizing

Theorem IV/B 1276.6 Summary 1286.7 Exercises 129

7 General Statistical Relations among the Quantization Noise, theQuantizer Input, and the Quantizer Output 1317.1 Joint PDF and CF of the Quantizer Input and Output 1317.2 Quantizing Theorems for the Joint CF of the Quantizer Input and

Output 1387.3 Joint PDF and CF of the Quantizer Input and the Quantization

Noise: Application of the PQN Model 1407.4 Quantizing Theorems for the Joint CF of the Quantizer Input and

the Quantization Noise: Application of the PQN Model 1467.5 Joint Moments of the Quantizer Input and the Quantization Noise:

Quantizing Theorem III 1497.5.1 General Expressions of Joint Moments when Quantizing

Theorem III is not satisfied 1517.6 Joint Moments of the Centralized Quantizer Input and the

Quantization Noise: Quantizing Theorem IV 1527.6.1 General Expressions of Joint Moments 153

7.7 Joint PDF and CF of the Quantization Noise and the QuantizerOutput 154

7.8 Three-Dimensional Probability Density Function andCharacteristic Function 1587.8.1 Three-Dimensional Probability Density Function 1587.8.2 Three-Dimensional Characteristic Function 159

7.9 General Relationship between Quantization and the PQN Model 1607.10 Overview of the Quantizing Theorems 162






X Contents

7.11 Examples of Probability Density Functions Satisfying QuantizingTheorems III/B or QT IV/B 165

7.12 Summary 1707.13 Exercises 171

8 Quantization of Two or More Variables: Statistical Analysis ofthe Quantizer Output 1738.1 Two-Dimensional Sampling Theory 1748.2 Statistical Analysis of the Quantizer Output for Two-Variable

Quantization 1798.3 A Comparison of Multivariable Quantization with the Addition of

Independent Quantization Noise (PQN) 1848.4 Quantizing Theorem I for Two and More Variables 1868.5 Quantizing Theorem II for Two and More Variables 1878.6 Recovery of the Joint PDF of the Inputs x1, x2 from the Joint PDF

of the Outputs x ′1, x ′

2 1878.7 Recovery of the Joint Moments of the Inputs x1, x2 from the Joint

Moments of the Outputs x ′1, x ′

2: Sheppard’s Corrections 1908.8 Summary 1928.9 Exercises 193

9 Quantization of Two or More Variables: Statistical Analysis ofQuantization Noise 1979.1 Analysis of Quantization Noise, Validity of the PQN Model 1979.2 Joint Moments of the Quantization Noise 2009.3 Satisfaction of Quantizing Theorems I and II 2039.4 Quantizing Theorem III/A for N Variables 2049.5 Quantization Noise with Multiple Gaussian Inputs 2069.6 Summary 2079.7 Exercises 207

10 Quantization of Two or More Variables: General StatisticalRelations between the Quantization Noises, and the QuantizerInputs and Outputs 20910.1 Joint PDF and CF of the Quantizer Inputs and Outputs 20910.2 Joint PDF and CF of the Quantizer Inputs and the Quantization

Noises 21010.3 Joint PDF, CF, and Moments of the Quantizer Inputs and Noises

when Quantizing Theorem I or II is Satisfied 21110.4 General Expressions for the Covariances between Quantizer

Inputs and Noises 21310.5 Joint PDF, CF, and Moments of the Quantizer Inputs and Noises

when Quantizing Theorem IV/B is Satisfied 214






Contents XI

10.6 Joint Moments of Quantizer Inputs and Noises with QuantizingTheorem III Satisfied 216

10.7 Joint Moments of the Quantizer Inputs and Noises withQuantizing Theorem IV Satisfied 217

10.8 Some Thoughts about the Quantizing Theorems 21810.9 Joint PDF and CF of Quantization Noises and Quantizer Outputs

under General Conditions 21810.10 Joint PDF and CF of Quantizer Inputs, Quantization Noises, and

Quantizer Outputs 21910.11 Summary 22110.12 Exercises 222

11 Calculation of the Moments and Correlation Functions of QuantizedGaussian Variables 22511.1 The Moments of the Quantizer Output 22511.2 Moments of the Quantization Noise, Validity of the PQN Model 23311.3 Covariance of the Input x and Noise ν 23711.4 Joint Moments of Centralized Input x and Noise ν 24011.5 Quantization of Two Gaussian Variables 24211.6 Quantization of Samples of a Gaussian Time Series 24911.7 Summary 25211.8 Exercises 253

Part III Floating-Point Quantization

12 Basics of Floating-Point Quantization 25712.1 The Floating-Point Quantizer 25712.2 Floating-Point Quantization Noise 26012.3 An Exact Model of the Floating-Point Quantizer 26112.4 How Good is the PQN Model for the Hidden Quantizer? 26612.5 Analysis of Floating-Point Quantization Noise 27212.6 How Good is the PQN Model for the Exponent Quantizer? 280

12.6.1 Gaussian Input 28012.6.2 Input with Triangular Distribution 28512.6.3 Input with Uniform Distribution 28612.6.4 Sinusoidal Input 290

12.7 A Floating-Point PQN Model 30212.8 Summary 30312.9 Exercises 304

13 More on Floating-Point Quantization 30713.1 Small Deviations from the Floating-Point PQN Model 307






XII Contents

13.2 Quantization of Small Input Signals with High Bias 31113.3 Floating-Point Quantization of Two or More Variables 313

13.3.1 Relationship between Correlation Coefficients ρν1,ν2 andρνFL1 ,νFL2

for Floating-Point Quantization 32413.4 A Simplified Model of the Floating-Point Quantizer 32513.5 A Comparison of Exact and Simplified Models of the Floating-

Point Quantizer 33113.6 Digital Communication with Signal Compression and Expansion:

“µ-law” and “A-law” 33213.7 Testing for PQN 33313.8 Practical Number Systems: The IEEE Standard 343

13.8.1 Representation of Very Small Numbers 34313.8.2 Binary Point 34413.8.3 Underflow, Overflow, Dynamic Range, and SNR 34513.8.4 The IEEE Standard 346


14 Cascades of Fixed-Point and Floating-Point Quantizers 35514.1 A Floating-Point Compact Disc 35514.2 A Cascade of Fixed-Point and Floating-Point Quantizers 35614.3 More on the Cascade of Fixed-Point and Floating-Point Quantizers 36014.4 Connecting an Analog-to-Digital Converter to a Floating-Point

Computer: Another Cascade of Fixed- and Floating-PointQuantization 367

14.5 Connecting the Output of a Floating-Point Computer to a Digital-to-Analog Converter: a Cascade of Floating-Point and Fixed-PointQuantization 368


Part IV Quantization in Signal Processing, FeedbackControl, and Computations

15 Roundoff Noise in FIR Digital Filters and in FFT Calculations 37315.1 The FIR Digital Filter 37315.2 Calculation of the Output Signal of an FIR Filter 37415.3 PQN Analysis of Roundoff Noise at the Output of an FIR Filter 37615.4 Roundoff Noise with Fixed-Point Quantization 37715.5 Roundoff Noise with Floating-Point Quantization 38115.6 Roundoff Noise in DFT and FFT Calculations 383

15.6.1 Multiplication of Complex Numbers 385






Contents XIII

15.6.2 Number Representations in Digital Signal ProcessingAlgorithms, and Roundoff 386

15.6.3 Growing of the Maximum Value in a Sequence Resultingfrom the DFT 387

15.7 A Fixed-Point FFT Error Analysis 38815.7.1 Quantization Noise with Direct Calculation of the DFT 38815.7.2 Sources of Quantization Noise in the FFT 38915.7.3 FFT with Fixed-Point Number Representation 392

15.8 Some Noise Analysis Results for Block Floating-Point andFloating-Point FFT 39415.8.1 FFT with Block Floating-Point Number Representation 39415.8.2 FFT with Floating-Point Number Representation 394


16 Roundoff Noise in IIR Digital Filters 40316.1 A One-Pole Digital Filter 40316.2 Quantization in a One-Pole Digital Filter 40416.3 PQN Modeling and Moments with FIR and IIR Systems 40616.4 Roundoff in a One-Pole Digital Filter with Fixed-Point

Computation 40716.5 Roundoff in a One-Pole Digital Filter with Floating-Point

Computation 41416.6 Simulation of Floating-point IIR Digital Filters 41616.7 Strange Cases: Exceptions to PQN Behavior in Digital Filters with

Floating-Point Computation 41816.8 Testing the PQN Model for Quantization Within Feedback Loops 41916.9 Summary 42516.10 Exercises 427

17 Roundoff Noise in Digital Feedback Control Systems 43117.1 The Analog-to-Digital Converter 43217.2 The Digital-to-Analog Converter 43217.3 A Control System Example 43417.4 Signal Scaling Within the Feedback Loop 44217.5 Mean Square of the Total Quantization Noise at the Plant Output 44717.6 Satisfaction of QT II at the Quantizer Inputs 44917.7 The Bertram Bound 45517.8 Summary 46017.9 Exercises 461

18 Roundoff Errors in Nonlinear Dynamic Systems – A Chaotic Example 46518.1 Roundoff Noise 465






XIV Contents

18.2 Experiments with a Linear System 46718.3 Experiments with a Chaotic System 470

18.3.1 Study of the Logistic Map 47018.3.2 Logistic Map with External Driving Function 478


Part V Applications of Quantization Noise Theory

19 Dither 48519.1 Dither: Anti-alias Filtering of the Quantizer Input CF 48519.2 Moment Relations when QT II is Satisfied 48819.3 Conditions for Statistical Independence of x and ν, and d and ν 48919.4 Moment Relations and Quantization Noise PDF when QT III or

QT IV is Satisfied 49219.5 Statistical Analysis of the Total Quantization Error ξ = d + ν 49319.6 Important Dither Types 497

19.6.1 Uniform Dither 49719.6.2 Triangular Dither 50019.6.3 Triangular plus Uniform Dither 50119.6.4 Triangular plus Triangular Dither 50219.6.5 Gaussian Dither 50219.6.6 Sinusoidal Dither 50319.6.7 The Use of Dither in the Arithmetic Processor 503

19.7 The Use of Dither for Quantization of Two or More Variables 50419.8 Subtractive Dither 506

19.8.1 Analog-to-Digital Conversion with Subtractive Dither 50819.9 Dither with Floating-Point 512

19.9.1 Dither with Floating-Point Analog-to-Digital Conversion 51219.9.2 Floating-Point Quantization with Subtractive Dither 51519.9.3 Dithered Roundoff with Floating-Point Computation 516

19.10 The Use of Dither in Nonlinear Control Systems 52019.11 Summary 52019.12 Exercises 522

20 Spectrum of Quantization Noise and Conditions of Whiteness 52920.1 Quantization of Gaussian and Sine-Wave Signals 53020.2 Calculation of Continuous-Time Correlation Functions and Spectra 532

20.2.1 General Considerations 53220.2.2 Direct Numerical Evaluation of the Expectations 53520.2.3 Approximation Methods 536






Contents XV

20.2.4 Correlation Function and Spectrum of Quantized GaussianSignals 538

20.2.5 Spectrum of the Quantization Noise of a Quantized SineWave 544

20.3 Conditions of Whiteness for the Sampled Quantization Noise 54820.3.1 Bandlimited Gaussian Noise 55020.3.2 Sine Wave 55420.3.3 A Uniform Condition for White Noise Spectrum 556


Part VI Quantization of System Parameters

21 Coefficient Quantization 56521.1 Coefficient Quantization in Linear Digital Filters 56621.2 An Example of Coefficient Quantization 56921.3 Floating-Point Coefficient Quantization 57221.4 Analysis of Coefficient Quantization Effects by Computer

Simulation 57421.5 Coefficient Quantization in Nonlinear Systems 57621.6 Summary 57821.7 Exercises 579

APPENDICES

A Perfectly Bandlimited Characteristic Functions 589A.1 Examples of Bandlimited Characteristic Functions 589A.2 A Bandlimited Characteristic Function Cannot Be Analytic 594

A.2.1 Characteristic Functions that Satisfy QT I or QT II 595A.2.2 Impossibility of Reconstruction of the Input PDF when

QT II is Satisfied but QT I is not 595

B General Expressions of the Moments of the Quantizer Output,and of the Errors of Sheppard’s Corrections 597B.1 General Expressions of the Moments of the Quantizer Output 597B.2 General Expressions of the Errors of Sheppard’s Corrections 602B.3 General Expressions for the Quantizer Output Joint Moments 607

C Derivatives of the Sinc Function 613






XVI Contents

D Proofs of Quantizing Theorems III and IV 617D.1 Proof of QT III 617D.2 Proof of QT IV 618

E Limits of Applicability of the Theory – Caveat Reader 621E.1 Long-time vs. Short-time Properties of Quantization 621

E.1.1 Mathematical Analysis 624E.2 Saturation effects 626E.3 Analog-to-Digital Conversion: Non-ideal Realization of Uniform

Quantization 628

F Some Properties of the Gaussian PDF and CF 633F.1 Approximate Expressions for the Gaussian Characteristic Function 634F.2 Derivatives of the CF with E{x} �= 0 635F.3 Two-Dimensional CF 636

G Quantization of a Sinusoidal Input 637G.1 Study of the Residual Error of Sheppard’s First Correction 638G.2 Approximate Upper Bounds for the Residual Errors of Higher

Moments 640G.2.1 Examples 642

G.3 Correlation between Quantizer Input and Quantization Noise 643G.4 Time Series Analysis of a Sine Wave 645G.5 Exact Finite-sum Expressions for Moments of the Quantization

Noise 648G.6 Joint PDF and CF of Two Quantized Samples of a Sine Wave 653

G.6.1 The Signal Model 653G.6.2 Derivation of the Joint PDF 654G.6.3 Derivation of the Joint CF 657

G.7 Some Properties of the Bessel Functions of the First Kind 660G.7.1 Derivatives 660G.7.2 Approximations and Limits 661

H Application of the Methods of Appendix G to Distributions other thanSinusoidal 663

I A Few Properties of Selected Distributions 667I.1 Chi-Square Distribution 667I.2 Exponential Distribution 670I.3 Gamma Distribution 672I.4 Laplacian Distribution 674I.5 Rayleigh Distribution 676I.6 Sinusoidal Distribution 677






Contents XVII

I.7 Uniform Distribution 679I.8 Triangular Distribution 680I.9 “House” Distribution 682

J Digital Dither 685J.1 Quantization of Representable Samples 686

J.1.1 Dirac Delta Functions at q/2 + kq 688J.2 Digital Dither with Approximately Normal Distribution 689J.3 Generation of Digital Dither 689

J.3.1 Uniformly Distributed Digital Dither 690J.3.2 Triangularly Distributed Digital Dither 693

K Roundoff Noise in Scientific Computations 697K.1 Comparison to Reference Values 697

K.1.1 Comparison to Manually Calculable Results 697K.1.2 Increased Precision 698K.1.3 Ambiguities of IEEE Double-Precision Calculations 698K.1.4 Decreased-Precision Calculations 700K.1.5 Different Ways of Computation 700K.1.6 The Use of the Inverse of the Algorithm 702

K.2 The Condition Number 703K.3 Upper Limits of Errors 705K.4 The Effect of Nonlinearities 707

L Simulating Arbitrary-Precision Fixed-Point and Floating-PointRoundoff in Matlab 711L.1 Straightforward Programming 712

L.1.1 Fixed-point roundoff 712L.1.2 Floating-Point Roundoff 712

L.2 The Use of More Advanced Quantizers 713L.3 Quantized DSP Simulation Toolbox (QDSP) 716L.4 Fixed-Point Toolbox 718

M The First Paper on Sampling-Related Quantization Theory 721

Bibliography 733

Index 742






Preface

For many years, rumors have been circulating in the realm of digital signal processingabout quantization noise:

(a) the noise is additive and white and uncorrelated with the signal being quantized,and

(b) the noise is uniformly distributed between plus and minus half a quanta, givingit zero mean and a mean square of one-twelfth the square of a quanta.

Many successful systems incorporating uniform quantization have been built andplaced into service worldwide whose designs are based on these rumors, therebyreinforcing their veracity. Yet simple reasoning leads one to conclude that:

(a) quantization noise is deterministically related to the signal being quantized andis certainly not independent of it,

(b) the probability density of the noise certainly depends on the probability densityof the signal being quantized, and

(c) if the signal being quantized is correlated over time, the noise will certainlyhave some correlation over time.

In spite of the “simple reasoning,” the rumors are true under most circumstances, orat least true to a very good approximation. When the rumors are true, wonderfulthings happen:

(a) digital signal processing systems are easy to design, and

(b) systems with quantization that are truly nonlinear behave like linear systems.

In order for the rumors to be true, it is necessary that the signal being quantizedobeys a quantizing condition. There actually are several quantizing conditions, allpertaining to the probability density function (PDF) and the characteristic function(CF) of the signal being quantized. These conditions come from a “quantizing theo-rem” developed by B. Widrow in his MIT doctoral thesis (1956) and in subsequentwork done in 1960.

Quantization works something like sampling, only the sampling applies in thiscase to probability densities rather than to signals. The quantizing theorem is related

XIX






XX Preface

to the “sampling theorem,” which states that if one samples a signal at a rate at leasttwice as high as the highest frequency component of the signal, then the signal isrecoverable from its samples. The sampling theorem in its various forms traces backto Cauchy, Lagrange, and Borel, with significant contributions over the years comingfrom E. T. Whittaker, J. M. Whittaker, Nyquist, Shannon, and Linvill.

Although uniform quantization is a nonlinear process, the “flow of probability”through the quantizer is linear. By working with the probability densities of thesignals rather than with the signals themselves, one is able to use linear samplingtheory to analyze quantization, a highly nonlinear process.

This book focuses on uniform quantization. Treatment of quantization noise,recovery of statistics from quantized data, analysis of quantization embedded in feed-back systems, the use of “dither” signals and analysis of dither as “anti-alias filtering”for probability densities are some of the subjects discussed herein. This book alsofocuses on floating-point quantization which is described and analyzed in detail.

As a textbook, this book could be used as part of a mid-level course in digitalsignal processing, digital control, and numerical analysis. The mathematics involvedis the same as that used in digital signal processing and control. Knowledge of sam-pling theory and Fourier transforms as well as elementary knowledge of statisticsand random signals would be very helpful. Homework problems help instructorsand students to use the book as a textbook.

Additional information is available from the following website:

http://www.mit.bme.hu/books/quantization/

where one can find data sets, some simulation software, generator programs for se-lected figures, etc. For instructors, the solutions of selected problems are also avail-able for download in the form of a solutions manual, through the web pages above.It is desirable, however, that instructors also formulate specific problems based ontheir own experiences.

We hope that this book will be useful to statisticians, physicists, and engineersworking in digital signal processing and control. We also hope that we have rescuedfrom near oblivion some ideas about quantization that are far more useful in today’sdigital world than they were when developed between 1955–60, when the numberof computers that existed was very small. May the rumors circulate, with propercaution.






Acknowledgments

A large part of this book was written while Istvan Kollar was a Fulbright scholar vis-iting with Bernard Widrow at Stanford University. His stay and work was supportedby the Fulbright Commission, by Stanford University, by the US-Hungarian JointResearch Fund, and by the Budapest University of Technology and Economics. Wegratefully acknowledge all their support.

The authors are very much indebted to many people who helped the creationof this book. Ideas described were discussed in different details with Tadeusz Do-browiecki, Janos Sztipanovits, Ming-Chang Liu, Nelson Blachman, Michael God-frey, Laszlo Gyorfi, Johan Schoukens, Rik Pintelon, Yves Rolain, and Tom Bryan.The ideas for some real-life problems came from Andras Vetier, Laszlo Kollar andBernd Girod. Many exercises were taken from (Kollar, 1989). Valuable discussionswere continued with the members of TC10 of IEEE’s Instrumentation and Measure-ment Society, furthermore with the members of EUPAS (European Project for ADC-based devices Standardisation). Students of the reading classes EE390/391 of schoolyears 2005/2006 and 2006/2007 at Stanford (Ekine Akuiyibo, Paul Gregory Baum-starck, Sudeepto Chakraborty, Xiaowei Ding, Altamash Janjua, Abhishek PrasadKamath, Koushik Krishnan, Chien-An Lai, Sang-Min Lee, Sufeng Li, Evan StephenMillar, Fernando Gomez Pancorbo, Robert Prakash, Paul Daniel Reynolds, AdamRowell, Michael Shimasaki, Oscar Trejo-Huerta, Timothy Jwoyen Tsai, Gabriel Ve-larde, Cecylia Wati, Rohit Surendra Watve) pointed out numerous places to corrector improve.

The book could not have come to life without the continuous encouragementand help of Professor George Springer of Stanford University, and of ProfessorGabor Peceli of the Budapest University of Technology and Economics.

A large fraction of the figures were plotted by Ming-Chang Liu, Janos Mar-kus, Attila Sarhegyi, Miklos Wagner, Gyorgy Kalman, and Gergely Turbucz. Vari-ous parts of the manuscript were typed by Mieko Parker, Joice DeBolt, and PatriciaHalloran-Krokel. The LATEX style used for typesetting was created by Gregory Plett.Very useful advice was given by Ferenc Wettl, Peter Szabo, Laszlo Balogh, andZsuzsa Megyeri when making the final form of the book pages.

Last but not least, we would like to thank our families for their not ceasingsupport and their patience while enduring the endless sessions we had together oneach chapter.

XXI






Glossary of Symbols

Throughout this book, a few formulas are repeated for easier reference during read-ing. In such cases, the repeated earlier equation number is typeset in italics, like in(4.11).

ak , bk Fourier coefficientsA signal amplitudeApp signal peak-to-peak amplitudeAT transpose of AA∗ complex conjugate transpose of AA complex conjugate of AB bandwidth, or the number of bits in a fixed-point number

(including the sign bit)cov{x, y} covariance, page 42C(τ ) covariance functiond dither, page 485dxdt derivative

exp(·) exponential function, also e(·)E( f ) energy density spectrumE{x} expected value (mean value)f frequencyfs sampling frequency, sampling ratef0 center frequency of a bandpass filterf1 fundamental frequency, or first harmonicf x (x) probability density function (PDF), page 31Fx (x) probability distribution function, Fx (x0) = P(x < x0)

�x (u) characteristic function (CF): �x (u) = ∫ ∞−∞ fx (x)e jux dx = E{e jux }

Eq. (2.17), page 27F{·} Fourier transform: F{x(t)} = ∫ ∞

−∞ x(t)e− j2π f t dtfor the PDF–CF pair, the Fourier transform is defined as∫ ∞−∞ f (x)e jux dx

XXIII






XXIV Glossary of Symbols

F−1{·} inverse Fourier transform: F−1{X ( f )} = ∫ ∞−∞ X ( f )e j2π f t d f

for the PDF–CF pair, the inverse Fourier transform is1

2π

∫ ∞−∞ �(u)e− jux du

h(t) impulse responseH( f ) transfer functionIm{·} imaginary partj

√−1k running index in time domain serieslg(·) base-10 logarithmln(·) natural logarithm (base e)Mr r th moment difference with PQN: E{(x ′)r } − E{xr }

Eq. (4.27), page 81Mr r th centralized moment difference with PQN: E{(x ′)r } − E{xr }n pseudo quantization noise (PQN), page 69n frequency index (or: summation index in certain sums)N number of samplesNr small (usually negligible) terms in the r th moment:

E{(x ′)r } = E{xr } + Mr + Nr , Eq. (B.1) of Appendix B, page 597Nr small (usually negligible) terms in the r th centralized moment:

E{(x ′)r } = E{xr } + Mr + NrN (µ, σ ) normal distribution, page 49O(x) decrease as quickly as x for x → 0p precision in floating-pointpi probabilityP{·} probability of an eventq quantum size in quantization, page 25qd quantum size of a digital dither, page 686qh step size of the hidden quantizer, page 357Q quality factor or weighting coefficientR(τ ) correlation function, Eq. (3.40), page 42Rxy(τ ) crosscorrelation function, Rxy(τ ) = E{x(t)y(t + τ)}

Eq. (3.41), page 42Rr residual error of Sheppard’s r th correction

Eq. (B.7) of Appendix B, page 602Rr residual error of the r th Kind correctionRe{·} real partrect(z) rectangular pulse function, 1 if |z| ≤ 0.5, zero elsewhererectw(z) rectangular wave, 1 if −0.25 ≤ z < 0.25; −1 if 0.25 ≤ z < 0.75;

repeated with period 1s Laplace variable, or empirical standard deviations∗ corrected empirical standard deviationSr Sheppard’s r th correction, Eq. (4.29), page 82






Glossary of Symbols XXV

Sr r th Kind correctionS( f ) power spectral densitySc( f ) covariance power spectral densitysign(x) sign functionsinc(x) sin(x)/xT sampling intervalTm measurement timeTp period lengthTr record lengthtr(z) triangular pulse function, 1 − |z| if |z| ≤ 1, zero elsewheretrw(z) triangular wave, 1 − 4|z| if |z| ≤ 0.5, repeated with period 1u standard normal random variableu(t) time function of voltageU effective value of voltageUp peak valueUpp peak-to-peak valuevar{x} variance, same as square of standard deviation: var{x} = σ 2

xw(t) window function in the time domainW ( f ) window function in the frequency domainx random variablex ′ quantized variablex ′ − x quantization noise, ν

x centralized random variable, x − µx , Eq. (3.13), page 34x(t) input time functionX ( f ) Fourier transform of x(t)X ( f , T ) finite Fourier transform of x(t)z−1 delay operator, e− j2π f T

δ angle error f frequency increment, fs/N in DFT or FFTε errorεc width of confidence intervalεr relative errorϕ phase angleγ ( f ) coherence function: γ ( f ) = Sxy( f )√

Sxx ( f )Syy( f )

µ mean value (expected value)ν quantization error, ν = x ′ − x quantization fineness, = 2π/qω radian frequency, 2π f� sampling radian frequency, page 17ρ correlation coefficient (normalized covariance, cov{x,y}

σxσy)

Eq. (3.39), page 42






XXVI Glossary of Symbols

ρ(t) normalized covariance functionσ standard deviation� covariance matrixτ lag variable (in correlation functions)ξ ξ = d + ν, total quantization error (in nonsubtractive dithering)

Eq. (19.16), page 491∈ element of set, value within given interval� convolution:

∫ ∞−∞ f (z)g(x − z) dz = ∫ ∞

−∞ f (x − z)g(z) dz= definition� first derivative, e. g. �x (l ) = d �(u)

d(u)

∣∣∣u=l

� second derivative, e. g. �x (l ) = d2 �(u)

d(u)2

∣∣∣u=l

x ′ quantized version of variable xx centralized version of variable x : x = x − µx , Eq. (3.13), page 34x estimated value of random variable xx� nearest integer smaller than or equal to x (floor(x))x deviation from a given value or variable






Acronyms and Abbreviations

AC alternating currentACF autocorrelation functionA/D analog-to-digitalADC analog-to-digital converterAF audio frequency (20 Hz–20 kHz)AFC automatic frequency controlAGC automatic gain controlALU arithmetic and logic unitAM amplitude modulation

BW bandwidth

CDF cumulative distribution functionCF characteristic functionCRT cathode ray oscilloscope

D/A digital-to-analogDAC digital-to-analog converterdBV decibels relative to 1 VdBm decibels relative to 1 mWDC direct currentDFT discrete Fourier transformDIF decimation in frequency (a form of FFT algorithm)DNL differential nonlinearityDIT decimation in time (a form of FFT algorithm)DSP digital signal processing or digital signal processorDUT device under testDVM digital voltmeter

FIR finite impulse responseFFT fast Fourier transformFM frequency modulationFRF frequency response function (nonparametric)

HP highpass (sometimes: Hewlett-Packard)HV high voltage

XXVII






XXVIII Glossary of Symbols

IC integrated circuitIF intermediate frequencyIIR infinite impulse responseINL integral nonlinearityI/O input/output

LMS least mean squaresLP lowpass (not long play in this book)LS least squaresLSB least significant bitLSI large scale integration

MAC multiply and accumulate operation (A=A+B∗C)MSB most significant bitMUX multiplexer

NFPQNP normalized floating-point quantization noise powerNSR noise-to-signal ratio, 10 log10(Pnoise/Psignal)

PDF probability density functionPF power factorPLL phase-locked loopPQN pseudo quantization noisePSD power spectral density (function)PWM pulse-width modulation

QT n Quantizing Theorem n

RAM random access memoryRC resistance-capacitance (circuit)RF radio frequencyRMS root mean squareROM read only memory

SEC stochastic-ergodic converterS/H sample-holdSI Systeme International (d’Unites): International System of UnitsSNR signal-to-noise ratio, 10 · log10(Psignal/Pnoise)

SOS sum of squares

TF transfer function (parametric)

U/D up/downULP unit in the last place






Quantization Noise

Documents

Transcript of Quantization Noise