INFORMATION THEORYAND

CODING

Nitin MittalHead of Department

Electronics and Communication EngineeringModern Institute of Engineering & Technology

Mohri, Kurukshetra

BHARAT PUBLICATIONS135-A, Santpura Road, Yamuna Nagar - 135001

© Reserved with the publisher

All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem or transmitted, in any form or by any means, electronic, mechanical, photocopying, recordingor otherwise, without the prior permision of the publisher.

Dedicated to our Beloved Parents

First Edition: Aug. 2009Second Edition: June 2010

Price: Rs. 200.00

ISBN-13: 978-81-909129-3-8

Publishing by: Bharat Publications,135 -A, Santpura Road, Yamuna Nagar - 135001Phone: 01732-227178, 232178, 94162-27140

Laser setting by: Chanda Computers, 9896471565

Legal Warning: The publisher has taken all possible precautions in publishing this book, in spite of all this efforts some errors might have crept in. Any mistake, erroror discrepancy noted may be brought to our knowledge which shall be taken care of in the next edition. It is notified that neither the publisher nor the book seller or the author willbe responsible for any damage or loss of action to anyone, of any kind, in any manner, therefrom. For missing pages or misprints the publisher takes responsibility to exchangewithin 15 days of purchase of the same edition. All costs in this connection are to be borne by the purchaser.

(v)

PREFACE

The present volume is the outcome of my experience in teaching of the informationtheory to the undergraduate classes for the last few years and the exposure to the problemsfaced by the students in grasping the abstract nature of the subject. This experience is thefoundation and, I hope, the strength of the text. Earnest efforts have been exerted to presentthe subject matter in a well–knit manner so as not only to stimulate the interest of the studentsbut also to provide them with an insight into the complexities of a subject of great intrinsicbeauty.

The book is intended to serve as a text for undergraduate students especially thoseopting for a course in Electronics and Communication Engineering. However, post graduatestudents will find it equally useful.

This book offers a comprehensive review of the Information Theory and Coding. Themain text can be divided into four sections on Probability Theory, Information Theory, SourceCoding and Error Control Coding. Fairly sufficient ground has been covered in all the foursections. Information theory is the study of achievable bounds for communication and is largelyprobabilistic and analytic in nature. Coding theory then attempts to realize the promise of thesebounds by models which are constructed through mainly algebraic means. Different conceptshave been explained with the help of examples. A large number of problems with solutions havebeen provided to assist one to get a firm grip on the ideas developed. There is a plenty of scopefor the reader to try and solve problems at his own in the form of exercises.

I am deeply indebted to all those authors whose research paper on InformationTheory and Coding influenced my learning of the subject and take this opportunity to expressmy sincere gratitude to them. I am thankful to Dr. Rajesh Goel (Principal, MIET, Mohri);Mr. R.S. Chauhan (Assistant Prof., JMIT, Radaur); Mr. Vikas Mittal (Sr. Lect. in HEC, Jagadhri)and Mr. Amanjeet Panghal (Lect. in MIET, Mohri) for motivation and continuous encouragementduring the preparation of this manuscript. I also wish to thank my collegues and friends whohave given many valuable suggestions on the scope and contents of the book. I am also indebtedto M/s Bharat Publications, Yamuna Nagar for bringing out the book in the short time.

It is my earnest belief that no work is ever complete till it has had its share ofcriticism and hence I'll be too glad to receive comments and suggestions for the betterment ofthe book.

Author

(vi)

FORWARD

It is great honour and immense pleasure for me to write a foreward of a book onInformation Theory and Coding by one of my esteemed Colleagues, Mr. Nitin Mittal.

Considering the needs of engineering students and the fact that they hardly get anyexposure to translate technology into practical applications, a basic knowledge in InformationTheory and Coding is essential and to be considered as a main subject in Electronics andCommunication Engineering. To cover the course material for such a vast and wide field, acomprehensive and easy to understand approach to the subject is required. In this book, theauthor has tried to put maximum efforts in this direction. The matter has been presented in wellstructured manner, an easy to understand language which can be grasped easily by students ofdifferent disciplines.

Attention has also been paid to the fact that the text as well as diagrams could bereproduced by the students in theory examinations. A number of review questions given at theend of each chapter will further enhance the understanding of basic concepts.

I am sure that this book would be quite useful to the students at undergraduate level invarious institutions, along with post graduate aspirants as well.

With my best wishes to the author.

Dr. RAJESH GOELPrincipal,MIET, MohriKurukshetra

(vii)

Chapter 1. Probability Theory And Random Variables 1–521.1 Introduction 11.2 Probability Theory 2

1.2.1 Experiment 21.2.2 Sample Space And Events 21.2.3 Algebra of Events 3

1.3 Probability of Events 41.3.1 Properties of Probability 4

1.4 Conditional Probability 61.4.1 Conditional Probability of Independent Events 71.4.2 Bayes’ Formula 7

1.5 Random Variables 131.5.1. Discrete Random Variables 131.5.2. Continuous Random Variables 14

1.6 Probability Distribution of A Discrete Random Variable 141.7 Cumulative Distribution Function (CDF) 15

1.7.1 Properties of Cumulative Distribution Function 161.8 Probability Density Function (PDF) 17

1.8.1 Properties of Probability Density Function 171.9 Two – Dimensional Random Variables 20

1.9.1 Joint Distribution Function 201.9.2 Marginal Distribution Function 211.9.3 Independent Random Variables 211.9.4 Joint Probability Density Function 211.9.5 Marginal Probability Density Function 221.9.6 Conditional Probability Density Function 22

1.10 Functions of Random Variables 241.11 Statistical Averages of Random Variables 26

1.11.1 Expectation 261.11.2 Moments And Variance 271.11.3 Covariance And Correlation Coefficient 28

1.12 Some Important Distributions 281.12.1 The Uniform or Rectangular Distribution 281.12.2 The Exponential Distribution 291.12.3 Gaussian or Normal Distribution 301.12.4 Rayleigh Distribution 32

CONTENTS

(viii)

1.13. Characteristic Transformation Functions of Random Variables 341.13.1 Properties of Moment Generating Function 351.13.2 Determination of Statistical Averages Using MGF 36

1.14 Convergence of A Sequence of Random Variables 371.14.1 Law of Large Numbers 371.14.2 Central Limit Theorem 38

Chapter 2. Random Processes 53–862.1 Introduction 532.2. Random Processes 542.3 Statistical Averages of Random Process 55

2.3.1 Ensemble Averages 552.3.2 Time Averages 56

2.4 Stationary Random Process 572.4.1 Strictly Stationary Process 572.4.2 Wide Sense Stationary Process 58

2.5 Ergodic Process 582.5.1 Properties of Ergodic Random Process 59

2.6 Correlation Function 602.6.1 Auto-Correlation Function 612.6.2 Cross-Correlation Function 622.6.3 Auto Covariance Function 632.6.4 Cross Covariance Function 63

2.7 Spectral Densities 642.7.1 Power Spectral Density 652.7.2 Cross Power Spectral Density 672.7.3 Energy Spectral Density 67

2.8 Response of Linear Systems To The Input Random Processes 692.9 Special Classes of Random Processes 73

2.9.1 Gaussian Process 732.9.2 Markov Process 742.9.3 Poisson Process 752.9.4 White Noise or White Process 762.9.5. Band - Limited White Noise or Process 77

Chapter 3. Elements of Information Theory 87–1343.1. Introduction 873.2. Information Sources 883.3. Information: A Measure of Uncertainty 883.4 Average Information or Entropy 89

3.4.1. Properties of Entropy 913.5 Binary Sources 943.6 Extension of A Discrete Memoryless Source 95

(ix)

3.7 Measure of Information For Two - Dimensional Discrete Finite 96Probability Scheme3.7.1 Discrete Memoryless Channels 98

3.8 Noise Characteristics of A Channel 1013.9 Measure of Mutual Information 102

3.9.1 Relationship Among Various Entropies 1033.9.2 Mutual Information 1033.9.3 Properties of Mutual Information 104

3.10 Channel Capacity 1073.11 Channel Capacity of Binary Noise Structures 107

3.11.1 Channel Capacity of A BSC (Binary Symmetric Channel) 1083.11.2 Channel Capacity of A BEC (Binary Erasure Channel) 109

3.12 Differential Entropy And Mutual Information For Continuous Signals 1103.13 Shannon’s Theorem On Coding For Memoryless Noisy Channel 113

Chapter 4. Source Encoding 135–1844.1 Introduction 1354.2 Source Encoding 1364.3 Basic Properties of Codes 1374.4 Separable Binary Codes 1394.5 Shannon – Fano Encoding 1414.6 Noiseless Coding Theorem 1444.7 Theorem of Decodability 1494.8 Average Length of Encoded Messages 1504.9 Shannon’s Binary Encoding 1524.10 Fundamental Theorem of Discrete Noiseless Coding 1544.11 Huffman’s Minimum – Redundancy Code 156

Chapter 5. Error Control Coding For Digital 185–206 Communication System

5.1 Introduction 1855.2 Elements of Digital Communication System 1865.3 Mathematical Models For Communication Channels 1925.4 Channel Codes 1945.5 Modulation And Coding 1965.6 Maximum Likelihood Decoding 2005.7 Types of Errors 2025.8 Error Control Strategies 203

Chapter 6. Error Detection And Correction 207–2246.1 Introduction 2076.2 Types of Errors 208

(x)

6.3 Error Detection 2096.3.1 Parity Check 2106.3.2 Cyclic Redundancy Check (CRC) 2116.3.3 Checksum 213

6.4 Error Correction 2156.4.1 Single – Bit Error Correction 2156.4.2. Burst Error Correction 219

Chapter 7. Field Algebra 225–2547.1 Introduction 2257.2 Binary Operations 2257.3 Groups 227

7.3.1. Commutative Group 2277.3.2. Semi – Group 2287.3.3. Order of A Group 2287.3.4. Addition Modulo M 2287.3.5. Multiplication Modulo M 2287.3.6. General Properties of Groups 230

7.4 Fields 2307.4.1 Characteristics of The Field 234

7.5 Binary Field Arithmetic 2347.5.1 Irreducible Polynomial Over GF (2) 2367.5.2 Primitive Polynomial Over GF (2) 237

7.6 Construction of Galois Field GF (2m) 2397.7 Basic Properties of Galois Field GF (2m) 2437.8 Vector Spaces 2467.9 Matrices 249

Chapter 8. Linear Block Codes 255–2828.1 Introduction 2558.2 Repetition Code 256

8.2.1 Majority Voting Decoder 2568.2.2 Single Error Correcting Repetition Code 2568.2.3 Advantages And Disadvantages of Repetition Codes 257

8.3 Binary Block Codes 2578.4 Linear Block Code 258

8.4.1 Systematic Linear Block Code 2598.4.2 Encoder For Linear Block Code 2628.4.3 Parity – Check Matrix 263

8.5 Syndrome Calculation For Linear Block Code 2648.5.1 Properties of The Syndrome (S) 268

8.6 The Hamming Distance of A Block Code 2708.7 Error – Detecting And Correcting Capabilities 271

(xi)

8.8 Syndrome Decoding of Linear Block Code 2738.9 Burst Error Correcting Block Codes 2758.10 Other Important Block Codes 277

8.10.1 Hamming Codes 2778.10.2 Extended Codes 278

Chapter 9. Cyclic Codes 283–3089.1 Introduction 2839.2 Cyclic Codes 2849.3 Generator Polynomial of Cyclic Codes 2859.4 Parity – Check Polynomial of Cyclic Codes 2869.5 Systematic Cyclic Codes 2889.6 Generator And Parity – Check Matrices of Cyclic Codes 2909.7 Encoder For Cyclic Codes 2929.8 Syndrome Polynomial Computation 2959.9 Decoding of Cyclic Codes 2979.10 Error – Trapping Decoding 2999.11 Advantages And Disadvantages of Cyclic Codes 3019.12 Important Classes of Cyclic Codes 301

Chapter 10. BCH Codes 309–33810.1 Introduction 30910.2 Binary BCH Codes 31010.3 Generator – Polynomial of Binary BCH Codes 31010.4 Parity – Check Matrix of BCH Code 31410.5 Encoding of BCH Codes 31610.6 Properties of BCH Codes 31810.7 Decoding of BCH Codes 318

10.7.1 Syndrome Computation 31810.7.2 Error Location Polynomial 320

10.8 BCH Decoder Architecture 32110.8.1 Peterson’s Direct Algorithm 32210.8.2 Berlekamp’s Iterative Algorithm 32610.8.3. Chien Search Algorithm 332

10.9 Non - Primitive BCH Code 33310.10 Non – Binary BCH Codes And RS Codes 334

Chapter 11. Convolutional Codes 339–37611.1 Introduction 33911.2 Convolutional Codes 34011.3 Convolutional Encoder 341

11.3.1 Encoding Using Discrete Convolution 34211.3.2 Encoding Using Generator Matrix 344

(xii)

11.4. Structural Properties of Convolutional Codes 34611.4.1. Code – Tree Diagram 34611.4.2 Trellis Diagram 34811.4.3 State Diagram Representation 349

11.5 Decoding of Convolutional Code 35011.5.1 Maximum - Likelihood Decoding 35011.5.2 The Viterbi Decoding Algorithm 35211.5.3 Sequential Decoding of Convolutional Codes 356

11.6 Distance Properties of Convolutional Codes 35711.7 Burst Error Correcting Convolutional Codes 359

Chapter 12. Basic ARQ Strategies 377–38812.1 Introduction 37712.2 Automatic Repeat Request (ARQ) 37812.3 Stop-And-Wait ARQ 37912.4 Continuous ARQ 381

12.4.1 Go-Back-N ARQ 38112.4.2. Selective Repeat ARQ 383

12.5 Hybrid ARQ 384

Chapter 13. Cryptography 389–40413.1 Introduction 38913.2 Cryptography 390

13.2.1 Need of Cryptography 39013.2.2 Cryptographic Goals 39013.2.3 Evaluation of Information Security 391

13.3 Cryptography Components 39213.4 Symmetric Key Cryptography 393

13.4.1 Symmetric Key encryption / decryption 39413.4.2 Techniques for coding plain text to chiper text 39413.4.3 Advantages and disadvantages of symmetric key cryptography 396

13.5 Asymmetric Key Cryptography 39613.5.1 Public Key encryption / decryption 39713.5.2 Conversion of plain text to chiper text algorithms 39813.5.3 Advantages and disadvantages of public-key cryptography 400

13.6 Comparison between symmetric and public-key cryptography 40113.7 Cryptography Applications 401

Sample Model Papers 405–407References 408Index 409–412

1. A. Bruce Carlson, Janet C. Rutledge and Crilly, “Communication Systems”, 3rd Ed., McGraw Hill, 2002.

2. A. Papoulis, “Probability, Random Variables and Stochastic Processes”, Mc Graw Hill,1991.

3. Andrew S Tanenbaum, “Computer Networks”, 3rd ed., Upper Saddle River, NJ: PrenticeHall, 1996.

4. B. P. Lathi, “Modern Digital and Analog Communication Systems”, Third Edition, OxfordPress, 2007.

5. B. R. Bhat, “Modern Probability Theory”, New Age International Ltd, 1998.6. Behrouz A Forouzan, “Data Communication and Networking”, Tata McGraw-Hill, 2003.7. D. Stinson, “Cryptography: Theory and Practice”, CRC Press, Second edition, 2000.8. Das, Mullick and Chatterjee, “Digital Communication”, Wiley Eastern Ltd., 1998.9. Fazlollah M. Reza, “An Introduction to Information Theory”, Dover Publications, Inc.,

New York, 1994.10. Gregory Karpilovsky, “Field Theory: Classical Foundations and multiplicative groups”,

Tata McGraw-Hill, 1988.11. Herbert Taub and Donald L Schilling, “Principles of Communication Systems”, 3rd Edition,

Tata McGraw Hill, 2008.12. J. E. Pearson, “Basic communication theory”, Upper Saddle River, NJ: Prentice Hall,

1992.13. John G. Proakis, Masoud Salehi, “Fundamentals of Communication Systems”, Pearson

Education, 2006.14. R. E. Blahut, “Principles and Practice of Information Theory”, Addison-Wesley, Reading,

Mass, 1987.15. R. P Singh and S. D. Sapre, “Communication Systems – Analog and Digital”, Tata McGraw

Hill, 2nd Edition, 2007.16. R. M.Gray, L. D Davission, “Introduction to Statistical Signal Processing”, Web Edition,

1999.17. Robert G. Gallanger, “Information Theory and Reliable Communication”, Mc Graw Hill,

1992.18. Shu Lin and D. J. Costello, “Error Control Coding: Fundamentals and Applications”,

Prentice Hall, 1983.19. Simon Haykin, “Communication Systems”, John Wiley & sons, New York, 4th Edition,

2001.20. W. Stallings, “Cryptography and Network Security: Principles and Practice”, Second

edition, Prentice Hall, 1999.21. William Stallings, “Data and Computer Communications”, 5th ed., Upper Saddle River,

NJ: Prentice Hall, 1997.

(408)

REFERENCES

AAcknowledgement,

–negative, 379, 382–positive, 379

Addition,–modulo–2, 247, 265–modulo–m, 228–vector, 247

Additive White Gaussian Noise, 192, 197Analog-to-digital (A/D) Converter, 185Arithmetic;

–binary field, 234–Galois field, 252

ARQ,–Continuous, 204, 378, 381–No-back-N, 204, 381–Hybrid, 204, 384–Selective-repeat, 204, 381–Stop-and-Wait, 204, 378–Type-I hybrid, 204, 386–Type-II hybrid, 204, 387

Auto Correlation, 60

BBandwidth, 115, 192BCH Codes, 303, 309

–binary, 310–decoding, 318–encoding, 316–generator polynomial, 310–non-binary, 334–non primitive, 333–parity-check matrix, 315–primitive, 311–properties, 318–syndrome, 318–syndrome computation, 318

BCH Decoder, 321Berlekamp Iterative Algorithm, 326Binary Erasure Channel (BEC), 87Binary Operation, 225Binary Symmetric Channel (BSC), 87, 202Block Codes,

–binary, 257–burst-error-correcting, 275–hamming, 277–interleaved, 276–linear, 258

(409)

INDEX

–random-error-correcting, 275–systematic, 259

Burst Error, 202, 211Burst Length, 208, 360

CCentral Limit Theorem, 38Channel,

–additive White Gaussiannoise, 115, 192

–binary erasure, 109–binary symmetric, 108–burst - error, 203–deterministic, 101–discrete memoryless, 98, 198–lossless, 101–noiseless, 102

Channel Capacity, 107Channel encoder, 189Characteristic of field, 234Checksum, 213Chien search, 330Cipher-text, 392Code efficiency, 137Code length, 136Code vector, 194Codeword, 137, 188Correlation functions,

–Auto Correlation, 60–Cross Correlation, 62

Complementary error function, 199Constraint length, 340Convolutional Codes, 195, 339

–burst-error-correcting, 360–constraint length, 340–distance properties, 358–generator matrix, 345–structural properties, 346–transfer function, 358

Coset, 274Coset leader, 274Cryptography, 389

– applications, 401– asymmetric key, 396— symmetric key, 393

Cyclic Codes,–decoding, 297–encoder, 292

4 1 0 INFORMATION THEORY AND CODING

–generator matrix, 290–generator polynomial, 285–Meggitt decoder, 297, 299–parity-check matrix, 291–parity-check polynomial, 286–syndrome, 295–systematic, 288

Cyclic Shifts, 284

DDecoders,

–channel, 191–maximum-likelihood, 201–Meggitt, 297, 299–Syndrome, 295

Decoding,–BCH codes, 318–cyclic codes, 297–error-trapping, 299–maximum likelihood, 200–syndrome, 273–viterbi, 196, 352

Decryption, 393Demodulator, 197Determinist Signals, 1Digital-to-Analog (D/A) Conversion, 187,191Digital Communication System, 186

–channel decoder, 191–channel encoder, 189–decryption, 191–demodulator, 191–destination, 186–encryption, 188–information source, 186–modulator, 190–source decoder, 191–source encoder, 188–synchronisation, 191

Digital signature, 397, 399Discrete Memoryless Channel

(DMC), 98,198–channel representation, 98–channel transition probability, 99–channel matrix, 99

Distance,–Hamming, 270–minimum, 270

Distance Properties of Convolutional Codes, 357Distributions,

–exponential, 29–Gaussian, 30–Rayleigh, 32–uniform, 28

Distribution Function,–cummutative distribution function, 15–joint distribution function, 20–marginal distribution function, 21

DSA algorithm, 399

EEncoders,

–channel, 189–convolutional code, 196, 339–cyclic code, 292–linear block code, 262–source, 136

Encoding, 190Encryption, 392Entropy, 89Ergodicity, 58Error control strategies, 203

–automatic-repeat request(ARQ), 204, 377

–for digital storage system, 204–forward error control, 203, 377

Error Correction Capability, 271Error Detection Capability, 271Error Location Numbers, 321Error Location Polynomial, 318, 320Error Patterns,

–correctable, 379, 385–detectable, 379–uncorrectable, 385–undetectable, 265, 379

Errors,–burst, 202, 277–random, 202, 277–transmission, 265–undetected, 265, 378

Event, 3, 94Expectation, 26Experiment, 2, 94

–head, 2–trial, 2–tail, 2

FField,

–binary, 233–finite, 233–Galois, 233, 239–prime, 233

Finite Field, 233Forward Error Correction, 203Fundamental Theorem of Discrete Noiseless

Coding, 154

I N D E X 4 1 1

GGalois Field, 233, 239Galois Field Arithmetic, 252Gaussian Process, 73Generator Matrix,

–block codes, 260–convolutional codes, 345–cyclic codes, 291

Generator Polynomial,–BCH codes, 310–cyclic codes, 285–Galoy codes, 302

GF(2), 233GF(2m), 234GF(P), 233GF(q), 234Galoy Codes, 302Group,

–cummutative, 227

HHamming Bound, 273Hamming Codes, 216, 277Hamming Distance, 270Hamming Weight, 270Huffman's Minimum Redundancy Codes, 156Hybrid ARQ Schemes, 204, 384, 385

IIdentity Element, 227Identity Matrix, 263Information Source, 186

–source alphabet, 188–symbol, 188

Information Theory,–average information, 89–DMS, 88, 95–information rate, 92–Information sources, 88–Information, 88–source alphabet, 88

Interleaving, 277Inverse,

–additive, 232–multiplicative, 231

Irreducible Polynomial, 236Iterative Algorithm, 326, 328

LLaw of Large Numbers, 37Linear Block Codes,

–block code, 257–generator matrix, 259

–linear code, 258–parity-check matrix, 263–systematic codes, 259

Linearity Property, 284Linearly Dependent Vectors, 248Linearly Independent Vectors, 248Location Numbers, 321

MMarkov Process, 74Matrix,

–channel, 99–generator, 259–identity, 263–parity-check, 263–transpose of, 249

Maximum Length Codes, 302Maximum Likelihood Decoding, 200, 350Mean, 26Meggitt Decoder, 297, 299Minimal Polynomial, 311Minimum Distance, 270Modulator, 190Modulo-2 addition, 265Modulo-m addition, 228Modulo-m multiplication, 228Moments, 27Multiplication,

–modulo-m, 228–scalar, 246

Mutual Information, 102

NNegative Acknowledgement, 379, 382Newton's Identities, 323(n,k) Block Code, 196, 257(n,k, K) Convolutional Code, 196, 341Noiseless Coding Theorem, 144Non-binary BCH Codes, 334Non-primitive BCH Codes, 333n–tuple, 195Null Space, 249

OOptimal Codes, 139Order,

–of a field, 231–of a group, 228

PParity Check Matrix,

–BCH codes, 315–block codes, 263

4 1 2 INFORMATION THEORY AND CODING

–cyclic codes, 291Plain text, 392Polynomials,

–irreducible, 236–minimal, 311–primitive, 312

Poisson Process, 75Positive Acknoweldgement, 379Power Spectral Density, 65Prime Numbers, 233Primitive BCH Codes, 311Primitive Elements, 311Primitive Polynomials, 237, 312Probability,

–conditional, 6–properties of, 4–transition, 98, 199

Probability Density Function, 17–conditional, 22–joint, 21–marginal, 22

RRandom Error Correcting Codes, 277Random Errors, 277Random Signals, 2Random Process, 54Random Variables, 13

–continuous, 14–discrete, 13

Rayleigh Distribution, 32Registers,

–buffer, 383–message, 262–shift, 341–syndrome, 298

Repetition Code, 256Representation of Galois Fields, 241RSA Algorithm, 398Retransmission, 378Round–trip Delay, 382Response of Linear Systems, 69

SSample Space, 2Selective-Repeat ARQ, 204, 381, 383Separable Binary Codes, 139Sequence, 351Shannons' Binary Code, 152Shannon–Fano Coding, 141Single Error Correcting Codes, 310Source,

–decoder, 191

–digital, 187–encoder, 188–information, 186

Source Encoding,–code efficiency, 137–code length, 136–code redundancy, 137–distinct codes, 137–instantaneous codes, 138–Kraft– McMillan inequality, 140–optimal codes, 139–prefix codes, 139–uniquely decodable codes, 138–variable length codes, 137

Space, 246Span, 248Spectral Densities,

–power spectral densities, 65–energy spectral densities, 67

State Diagram, 349Stationary Process,

–strict sense, 57–wide sense, 58

Substitution techniques, 394–Mono alphabetic, 394–Poly alphabetic, 395

Syndrome,–BCH codes, 318–block codes, 264–cyclic codes, 295–decoding, 273

Syndrome register, 298Systematic codes, 259

TTheorem of Decodability, 149Throughput Efficiency,

–go-back-N ARQ, 204, 383–selective-repeat ARQ, 204, 384–stop-and-wait ARQ, 204, 380

Transfer Function of Convolutional Codes, 358Transition Probabilities, 99, 199Transmission Errors, 265Transposition techniques, 395Transpose of a Matrix, 249Trellis Diagram, 348

VVectors, 246Vector Addition, 247Vector Space, 246Veterbi Algorithm, 352