nts.uni-due.dents.uni-due.de/downloads/cod/CT_SS2011.pdfnts.uni-due.de

Prof. Dr.-Ing. Andreas Czylwik Coding TheorySS 2011

p. 1Fachgebiet Nachrichtentechnische Systeme

N T SUNIVERSITÄT

D U I S B U R GE S S E N

Coding Theory

Prof. Dr.-Ing. Andreas Czylwik

N T SUNIVERSITÄT

Coding TheoryOrganization

Lecture: 2 hours per week Transparencies will be available at:

http://nts.uni-duisburg.de Exercise: 1 hour per week, M.Sc. Bo Zhao New research areas at the Chair of Communication

Systems (Nachrichtentechnische Systeme) Subjects for master theses

N T SUNIVERSITÄT

Coding Theory Textbooks Textbooks for the lecture:

M. Bossert: Channel coding for telecommunications, John Wiley Blahut: Theory and practice of error control codes, Addison-

Wesley J. H. van Lint: Introduction to coding theory B. Friedrichs: Kanalcodierung, Springer-Verlag H. Schneider-Obermann: Kanalcodierung, Vieweg-Verlag

Textbooks in the field of digital communications incl. coding theory: S. Benedetto, E. Biglieri, V. Castellani: Digital transmission

theory, Prentice-Hall J.G. Proakis: Digital communications, McGraw-Hill S. Haykin: Communication systems, John Wiley

N T SUNIVERSITÄT

Coding Theory Contents

Introduction Information theory Channel coding in digital communication systems Algebraic foundations for coding Block codes Convolutional codes Coding techniques Outlook

N T SUNIVERSITÄT

Block diagram of a digital communication system

Coding Theory Introduction

Source Sourceencoder

Channelencoder Modulator

Transmissionchannel Noise

DemodulatorChanneldecoder

SourcedecoderSink

Digital source

Digital sinkDiscrete channel

N T SUNIVERSITÄT

Source coding Compression of the communication signal to a minimum number

of symbols without the loss of information (reduction of redundancy)

Further compression when loss of information is tolerated (e.g. video or audio transmission)

Coding for encryption Security against unwanted listening Information recovery only with a secret key

N T SUNIVERSITÄT

Error control coding: defined addition of redundancy to improve transmission quality FEC (forward error correction) Simplified model:

Channel quality determines residual error rate after decoding Data rate does not depend on channel quality

Datasource

Channelencoder Channel Channel

decoderDatasink

N T SUNIVERSITÄT

Channel coding for error detection (CRC – cyclic redundancy check, used for automatic repeat request methods (ARQ))

Reverse channel necessary Adaptive insertion of redundancy (addition redundancy only in

case of errors) Residual error probability does not depend on channel quality Net data rate (throughput) depends on channel quality

ARQcontrol

Channelencoder Channel Channel decoder

(error detection)ARQ

control

Error informationBackward channel

N T SUNIVERSITÄT

Application: secure data transmission via wave guides or radio channels (especially mobile radio channels), secure data storage

Father of information theory − Claude E. Shannon: Using channel coding the error probability can be reduced to an arbitrarily small value, if the data rate is smaller than the channel capacity. (Shannon does not present a construction method.)

N T SUNIVERSITÄT

Basic idea of channel coding: Insertion of redundancy Goal: error detection or error correction Block encoding process

Input vector: u = (u1, ... , uk) Output vector: x = (x1, ... , xn) Code rate:

nkR =C (1)

Length k Length n

Input vector u Output vector x

N T SUNIVERSITÄT

Code cube with n = 3, k = 3

Uncoded transmission: RC = 1 Smallest distance between code words: dmin = 1 Error detection or error correction not possible

N T SUNIVERSITÄT

Coded transmission: RC = 2/3 Smallest distance between code words: dmin = 2 Detection of a single error is possible – error correction is not

possible

N T SUNIVERSITÄT

Coded transmission: RC = 1/3 Smallest distance between code words: dmin = 3 Detection of two errors and correction of a single error is possible

N T SUNIVERSITÄT

Coding Theory Information theory

Information theory: mathematical description of information and of its transmission

Central questions: Quantitative calculation of the information content of messages Calculation of the capacity of transmission channels Analysis and optimization of source and channel coding

N T SUNIVERSITÄT

Messages from the point of view of information theory

irrelevant relevant

errorinformation

redundant non redundant

N T SUNIVERSITÄT

Information content of a message: entropy Qualitative ordering of messages: a message is more

important if it is more difficult to predict it Example:

Tomorrow, the sun rises. Tomorrow, there will be bad weather. Tomorrow, there will be a heavy

thunderstorm so that the electric power network will break down.

Messages from a digital source: sequence of symbols

Importance

Probability

N T SUNIVERSITÄT

Number of binary decisions that is needed to select a message (symbol) from a source: H0 Total number of symbols: N

H0 = ld(N) bit/symbol

with ld(x) = logarithm to base 2 (logarithmus dualis) Unit: bit = binary digit Example: English text as a source

26 ⋅ 2 characters, 14 special characters including „space“: ‘ ’ “ ” ( ) - . , ; : ! ?

In total: 66 characters H0 = ld(66) = ln(66) / ln(2) = 6.044 bit/character

N T SUNIVERSITÄT

Example: one page of English text with 40 lines and 70 characters per line Number of different pages: N = 6640⋅70

Number of binary decisions to select a page: H0 = ld(6640⋅70) = 40 ⋅ 70 ⋅ ld (66) = 16.92 kbit/page

The number of binary decisions H0 does not take into account the probability of symbols!

N T SUNIVERSITÄT

Source alphabet: X ∈ {x1, ... , xN} Probabilities of the symbols: p(x1), ... , p(xN) Desired properties of the information content I = f(p):

I(xi) ≥ 0 for 0 ≤ p(xi) ≤ 1 I(xi) → 0 for p(xi) → 1 I(xi) > I(xj) for p(xi) < p(xj) Two subsequent statistically independent symbols xi and xj

with p(xi,xj) = p(xi) ⋅ p(xj) :I(xi,xj) = I(xi) + I(xj)

General solution: I(xi) = −k ⋅ logb(p(xi)) (3)

N T SUNIVERSITÄT

Definition of the information content:

Entropy H(X) = average information content of a source:

bit/symbol)(

)()()()(

xIxpxIXH

N T SUNIVERSITÄT

Example: binary source with X ∈ {x1, x2} Probabilities: p(x1) = p , p(x2) = 1 − p Entropy: H(X) = −p ld(p) − (1 − p) ld(1 − p) Shannon function: H(X) = S(p)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

S(p) / bit

N T SUNIVERSITÄT

The entropy becomes maximum for equiprobable symbols: ⇒ H(X) ≤ H0 = ld N. Proof:

Using the relation:

∑∑

⋅⋅=

⋅−⋅=−

)(1ld)(

ld)()(

1ld)(ld)(

1ln −≤ xx

(8) -1.0

1.0 2.0 x

x − 1

N T SUNIVERSITÄT

Redundancy of a source: RS = H0 − H(X)

0112ln

1)(12ln

1)(ld)(

−⋅=

−≤

⋅⋅≤−

NxpxpNXH

)1(2ln

1ld1ln −≤⇒−≤ xxxx

N T SUNIVERSITÄT

Transmission via a binary channel:design of binary coding schemes for a discrete source

Tasks of source coding: Assignment of binary code words of length L(xi) to all symbols xi

Minimizing the average length of the code words

⋅==N

iiii xLxpxLL

1)()()(

L x1 1001x2 011

xN 010111L(xN)

N T SUNIVERSITÄT

Examples for binary coding of symbols: ASCII-Code: fixed code word length L(xi) = 8 (block code)Morse code (alphabet with dots, dashes and pauses for the

separation of code words): more frequently occuring characters are assigned to shorter code words

Prefix condition for codes with variable length: no codeword equals the prefix of any other longer codeword Example for a code without prefix condition:

x1 → 0x2 → 01x3 → 10x4 → 100

Unique decoding of a bit sequence is not possible!

Possible decoding results for the sequence 010010: x1x3x2x1, x2x1x1x3, x1x3x1x3, x2x1x2x1,

x1x4x3

N T SUNIVERSITÄT

Example for a code with prefix condition:

The code is uniquely and instantaneously decodable! Decoding of the sequence 010010110111100:

x1x2x1x2x3x4x2x1

Synchronization: begin of the sequence

x1 → 0x2 → 10x3 → 110x4 → 111

N T SUNIVERSITÄT

Decoding with a binary tree:

x1 → 0x2 → 10x3 → 110x4 → 111

Level 1: L(x1) = 1

0 1Level 2: L(x2) = 2

Level 3: L(x3) = L(x4) = 3

N T SUNIVERSITÄT

Kraft inequality: a binary code with the code word lengths L(x1), L(x2), ... , L(xN) fulfilling the prefix condition exists only, if:

Proof: Length of the tree structure = maximum code word length

Lmax = max(L(x1), L(x2), ... , L(xN)) A code word in level L(xi) eliminates of all

possible code words in level Lmax . Sum over all eliminated code words ≤ maximum number in the

level Lmax

)( ≤∑=

)(max2 ixLL −

maxmax 221

xLL i ≤∑=

N T SUNIVERSITÄT

The equality holds, if all ends of the code tree are used by code words.

General relation: Example for the special case that all probabilities are given by

powers of 2:

Assignment of code words corresponding to the relation:L(xi) = Ki

Special case:

( ) iKixp 2

)(XHL ≥

815)( == XHL

xi p(xi) code words x1 1/2 1 x2 1/4 00 x3 1/8 010 x4 1/16 0110 x5 1/16 0111

N T SUNIVERSITÄT

Shannons coding theorem: For any source a binary coding can be found with :

Proof: left-hand side: average length of code words ≥ average information content (entropy)right-hand side: selection of code words with

I(xi) ≤ L(xi) ≤ I(xi) + 1Multiplication with p(xi) and summing up over all i ⇒ (14)

1)()( +≤≤ XHLXH (14)

N T SUNIVERSITÄT

Proof, that a code with the prefix condition (11) exists:left-hand side of (15):

Inserting in (11):

Shannon Coding Code word length corresponding (15): I(xi) ≤ L(xi) ≤ I(xi) + 1 Accumulated probabilities:

1)(211

)( ∑∑==

− =≤N

xL xpi

1 2)()(ld)( ii

xLiixpi xpxLxI −≥⇒≤

∑−

jji xpP

N T SUNIVERSITÄT

Code words are decimals of Pi in binary notation:

Example:0.90 = 1⋅2−1 + 1⋅2−2 + 1⋅2−3 + 0⋅2−4 + 0⋅2−5 + 1⋅2−6 + 1⋅2−7 + 0⋅2−8 +0⋅2−9 + 1⋅2−10 + ...

i p(xi) I(xi) L(xi) Pi code 1 0.22 2.18 3 0.00 000 2 0.19 2.40 3 0.22 001 3 0.15 2.74 3 0.41 011 4 0.12 3.06 4 0.56 1000 5 0.08 3.64 4 0.68 1010 6 0.07 3.84 4 0.76 1100 7 0.07 3.84 4 0.83 1101 8 0.06 4.06 5 0.90 11100 9 0.04 4.64 5 0.96 11110

bit54.3=Lbit97.2)( =XH

N T SUNIVERSITÄT

Tree representation of a Shannon code:

Disadvantage: not all ends of the tree are used for code words

N T SUNIVERSITÄT

Lengths of code words can be reduced ⇒ the code is not optimum Redundancy of a code: Redundancy of a source:

Huffman Coding Recursive procedure Starting point: symbols with smallest probabilities Same code word lengths for the two symbols with smallest

probabilities Huffman coding minimizes the average code word length

)(C XHLR −=)(0Q XHHR −=

N T SUNIVERSITÄT

Algorithm:

Step 1: Ordering of symbols corresponding to their probability

Step 2: Assignment of 0 and 1 to the two symbols with lowest probability

Step 3: Combination of the two symbols with lowest probability xN−1 and xN to a new symbol with probability p(xN−1 ) + p( xN)

Step 4: Repetition of steps 1 - 3, until only one symbol is left

Example

N T SUNIVERSITÄT

x1 x2 x3 x4 x5 x6 x7 x8 x9 0.22 0.19 0.15 0.12 0.08 0.07 0.07 0.06 0.04

x1 x2 x3 x4 x8 x9 x5 x6 x7 0.22 0.19 0.15 0.12 0.10 0.08 0.07 0.07

0 1 0 1

x1 x2 x3 x6 x7 x4 x8 x9 x5 0.22 0.19 0.15 0.14 0.12 0.10 0.08

0 1 00 01 1

x1 x2 x8 x9 x5 x3 x6 x7 x4 0.22 0.19 0.18 0.15 0.14 0.12

00 01 1 00 01 1

N T SUNIVERSITÄT

x6 x7 x4 x1 x2 x8 x9 x5 x3 0.26 0.22 0.19 0.18 0.15

00 01 1 000 001 01 1

x8 x9 x5 x3 x6 x7 x4 x1 x2 0.33 0.26 0.22 0.19

000 001 01 1 00 01 1 0 1

x1 x2 x8 x9 x5 x3 x6 x7 x4 0.41 0.33 0.26

0 1 0000 0001 001 01 100 101 11

x8 x9 x5 x3 x6 x7 x4 x1 x2 0.59 0.41

00000 00001 0001 001 0100 0101 011 10 11

N T SUNIVERSITÄT

Tree structure of the Huffman code example

bit01.3=L

N T SUNIVERSITÄT

Discrete source without memory Joint entropy of sequences of symbols

Two independent symbols: p(xi,yk) = p(xi) ⋅ p(yk)

)()(),(

))((ld)()())((ld)()(

))((ld))((ld)()(

)),((ld),(),(

1´111

YHXHYXH

ypypxpxpxpyp

ypxpypxp

yxpyxpYXH

⋅⋅−⋅⋅−=

+⋅⋅−=

∑∑∑∑

∑ ∑

N T SUNIVERSITÄT

H(X1,...,XM) ≤ ≤ H(X1,...,XM) + 1

M⋅H(X) ≤ ≤ M⋅H(X) + 1

H(X) ≤ ≤ H(X) + 1/M

M independent symbols from the same source:

H(X1, X2, ... , XM) = M ⋅ H(X)

More efficient coding by coding of symbol sequences Shannon coding theorem:

Disadvantage of coding of symbol sequences: rapidly increasing computational effort (exponential increase of number of combined symbols)

),,( 1 MM XXL

LM ⋅

N T SUNIVERSITÄT

Example: Coding of symbol sequences Binary source with X = {x1, x2} Probabilities: p(x1) = 0.2, p(x2) = 0.8 Entropy: H(X) = 0.7219 bits/symbol Coding of single symbols:

Average code word length:

symbol p(xi) code L(xi) p(xi)⋅L(xi) x1 0.2 0 1 0.2 x2 0.8 1 1 0.8

Σ = 1

bit/symbol 1=L

N T SUNIVERSITÄT

Coding of pairs of symbols:

Average code word length:

pair of symbols

p(xi) code L(xi) p(xi)⋅L(xi)

x1x1 0.04 101 3 0.12 x1x2 0.16 11 2 0.32 x2x1 0.16 100 3 0.48 x2x2 0.64 0 1 0.64

Σ = 1.56

bit/symbol 78.0=L

N T SUNIVERSITÄT

Coding of triplets of symbols:

Average Code word length:

triplet of symbols

p(xi) code L(xi) p(xi)⋅L(xi)

x1x1x1 0.008 11111 5 0.040 x1x1x2 0.032 11100 5 0.160 x1x2x1 0.032 11101 5 0.160 x1x2x2 0.128 100 3 0.384 x2x1x1 0.032 11110 5 0.160 x2x1x2 0.128 101 3 0.384 x2x2x1 0.128 110 3 0.384 x2x2x2 0.512 0 1 0.512

Σ = 2.184

bit/symbol 728.0=L

N T SUNIVERSITÄT

Discrete source with memory Real sources: correlation between individual symbols Two correlated symbols: p(xi,yk) = p(xi) ⋅ p(yk|xi) = p(yk) ⋅ p(xi|yk)

)|()(),(

))|((ld),())((ld)()|(

))|((ld))((ld)|()(

)),((ld),(),(

1 11 1

XYHXHYXH

xypyxpxpxpxyp

xypxpxypxp

yxpyxpYXH

kikiiki

⋅−⋅⋅−=

+⋅⋅−=

∑ ∑∑ ∑

∑ ∑

= == =

N T SUNIVERSITÄT

H(Y | X) = conditional entropy

Entropy of a symbol ≥ conditional entropy

H(Y) ≥ H(Y | X )

Proof:

(21)∑ ∑= =

⋅−=N

kikki xypyxpXYH

1 1))|((ld),()|(

∑ ∑

∑ ∑∑

⋅=−

⋅−=⋅−=

xypypyxpYHXYH

ypyxpypypYH

)|()(ld),()()|(

))((ld),())((ld)()(

N T SUNIVERSITÄT

Using the inequality:

with p(xi,yk) = p(xi) ⋅ p(yk|xi)

∑ ∑= =

−⋅⋅≤−

kki xyp

ypyxpYHXYH1 1

1),()()|(

)1(2ln

1ld1ln −≤⇒−≤ xxxx

0),()()(2ln

1)()|(1 11 1

−≤− ∑ ∑∑ ∑

= == =

kki yxpypxpYHXYH

= 1 = 1

N T SUNIVERSITÄT

Entropy of a source with memory < entropy of a source without memory

Very efficient source coding for sequences of symbols with memory

General description of a discrete source with memory as a Markov source Markov processes:

Sequence of random variables z0, z1, z2, ... ,zn, (n = time axis) zi and zj are statistically independent:

)(),...,,|( 021,...,,| 021 nznnnzzzz zfzzzzfnnnn

=−−−− (23)

N T SUNIVERSITÄT

zi and zj are stat. dependent (Markov process of mth order):

Often: first order Markov process (m = 1):

zi are from a limited number of discrete values: zi ∈ {x1, ... , xN}

⇒ Markov chain Complete description of a Markov chain by transition

probabilities:

)|(),...,,|( 1|021,...,,| 1021 −−− −−−= nnzznnnzzzz zzfzzzzf

),...,|(),...,|(101 101 mnnn imninjniinjn xzxzxzpxzxzxzp

−−−======= −−−

),...,|(),...,,|( 1,...,|021,...,,| 1021 mnnnzzznnnzzzz zzzfzzzzfmnnnnnn −−−− −−−−

N T SUNIVERSITÄT

Homogeneous Markov chain: transition probabilities do not depend on time:

stationary Markov chain: steady-state does not depend on initial probabilities

Homogeneous and stationary Markov chain of first order (m = 1):

jjjnkn

ikjnkn

wxpxzpxzxzp ====== +∞→

+∞→

)()(lim)|(lim

ijinjn pxzxzp === − )|( 1

),...,|(),...,|(11 11 mm imkikjkimninjn xzxzxzpxzxzxzp ======= −−−−

N T SUNIVERSITÄT

Transition matrix:

Properties of the transition matrix:

Probability vector:

Calculation of w utilizing the steady state condition:

))()()(()( 2121 NN xpxpxpwww ==w

pppppp

P (30)

N T SUNIVERSITÄT

Stationary Markov source: first order Markov chain Entropy of a stationary Markov source = steady state entropy:

∑ ∑

= =−−

=−−

−−−∞→

==⋅==⋅−=

==⋅==−=

=⋅===

jinjninjni

jinjninjn

iinniiinn

nnnnnn

xzxzpxzxzpw

xzxzpxzxzp

xzzHwxzzH

zzHzzzzHZH

))|((ld)|(

))|((ld),(

)|()|(

)|(),,,|(lim)(

N T SUNIVERSITÄT

⇒ Entropy H∞(Z) = conditional entropy, since symbols from the past are already known

Coding of a Markov source: Taking into account the memory e. g. Huffman coding which takes into account the

instantaneous state of the source

Fundamental problem for variable length source codes: catastrophic error propagation

)()()|()( 01 nnnn zHzHzzHZH ≤≤= −∞ (38)

N T SUNIVERSITÄT

Example of a Markov source: States = transmitted symbols

zn ∈ {x1, x2, x3}

Transition probabilities:

( ) ( )

===== −

3.01.06.02.05.03.04.04.02.0

)|( 1 Pijinjn pxzxzp

zn zn−1

x1 x2 x3

x1 0.2 0.4 0.4 x2 0.3 0.5 0.2 x3 0.6 0.1 0.3

0.20.3

N T SUNIVERSITÄT

Calculation of steady-state probabilities with:

w1 = 0.2 w1 + 0.3 w2 + 0.6 w3

w2 = 0.4 w1 + 0.5 w2 + 0.1 w3

w3 = 0.4 w1 + 0.2 w2 + 0.3 w3

1 = w1 + w2 + w3

Linear dependence ! Solution:

1 and 1

== ∑=

iiwPww

3011.03441.03548.0 9328

1 ≈=≈=≈= www

N T SUNIVERSITÄT

Calculation of the entropy with (35):

Inserting numbers:

∑ ∑

=−∞

⋅⋅−=

jijiji

xzzHwZH

symbolbit / 2955.1ld3.0ld1.0ld6.0)|(

≅⋅+⋅+⋅==

N T SUNIVERSITÄT

Entropy:

For comparison:

symbol/bit441.1)|()|()|()( 313212111

≅=+=+== −−−∞ xzzHwxzzHwxzzHwZH nnnnnn

symbol/bit5814.11ld)(lim)(1

≅⋅== ∑=∞→

n wwzHZH

symbol/bit5850.13ld0 ≅=H

N T SUNIVERSITÄT

State-dependent Huffman coding for the example: different codings for each state zn−1

pij code words

average code word length:

zn zn−1

x1 x2 x3 ⟨L⟩|zn−1

x1 11 10 0 1.6 x2 10 0 11 1.5 x3 0 11 10 1.4

symbol/bit5054.1

)|()|(1 1

==⋅⋅==== ∑ ∑= =

−−N

jinjnijiinjn xzxzLpwxzxzLL

zn zn−1

x1 x2 x3

x1 0.2 0.4 0.4 x2 0.3 0.5 0.2 x3 0.6 0.1 0.3

N T SUNIVERSITÄT

Source coding without knowledge of statistical parameters Run-length coding

Replacing sequences of repeated symbols with the count and only one single symbol

Example: Source sequence:

aaaabbbccccccccdddddeeeeeaaaaaaabddddd..... Encoded sequence:

4a3b8c5d5e7a1b5d..... Encoding with dictionaries

Idea: repetitions within the data sequence are replaced by (shorter) references to the dictionary

N T SUNIVERSITÄT

Static dictionary: little compression for most data sources poor matching of the dictionary to particular data

Semi-adaptive dictionary dictionary tailored for the message to be encoded dictionary has to transmitted via the channel two passes over the data:

» to build up the dictionary» to encode the data

Adaptive dictionary single pass for building up the dictionary and encoding Lempel Ziv algorithm: *.zip *.gzip files

N T SUNIVERSITÄT

Lempel Ziv algorithm (LZ77)

Search for the longest match between the symbols of the look-ahead buffer and a sequence within the search buffer

Fixed-length codewords contain: position of match (counting from 0) length of match next symbol in the look-ahead buffer

search buffer look-ahead buffer

..... a b c b a a b c b d f e e a a b a a c c d d d c d .....

sliding window

N T SUNIVERSITÄT

Parameters: symbol alphabet: {x0,x1,x2, .... , xα−1} input sequence: S = {S1,S2,S3,S4, ....} length of look-ahead buffer: LS

window size: n

Codewords: Ci = {pi,li,Si} position of match: pi

length of match: li

next symbol: Si

length of codewords: same alphabet for data as well as codewords

1)(log)(log SSC ++−= LLnL αα

N T SUNIVERSITÄT

Example: symbol alphabet: {0,1,2} input sequence: S = {0010102102102120210212001120.....} length of look-ahead buffer: LS = 9 window size: n = 18 Step 1:

Step 2:

0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 2 1 0 2 1 0 2 . . . .

C1 = {22 02 1}

0 0 0 0 0 0 0 0 1 0 1 0 2 1 0 2 1 0 2 1 2 0 . . . .

C2 = {21 10 2}

N T SUNIVERSITÄT

Step 3:

Step 4:

Number of encoded source symbols after 4 steps: 3 + 4 + 8 + 9 = 24

Number of code word symbols after 4 steps: 4 × 5 = 20

0 0 0 0 1 0 1 0 2 1 0 2 1 0 2 1 2 0 2 1 0 2 . . . .

C3 = {20 21 2}

2 1 0 2 1 0 2 1 2 0 2 1 0 2 1 2 0 0 1 1 2 0 . . . .

C4 = {02 22 0}

N T SUNIVERSITÄT

Decoding: Step 1: C1 = {22 02 1}

Step 2: C2 = {21 10 2}

Step 3: C3 = {20 21 2}

Step 4: C4 = {02 22 0}

0 0 0 0 0 0 0 0 0 0 0 1

0 0 0 0 0 0 0 0 1 0 1 0 2

0 0 0 0 1 0 1 0 2 1 0 2 1 0 2 1 2

2 1 0 2 1 0 2 1 2 0 2 1 0 2 1 2 0 0

N T SUNIVERSITÄT

Transmission of information via a discrete memoryless channel Noiseless channel:

information at the output = information at the input ⇒ transmitted information = entropy of the source

Noisy channel: information at the output < information at the input ⇒ transmitted information < entropy of the source

Definition: average mutual information = actually transmitted information

N T SUNIVERSITÄT

Discrete memoryless channel (DMC)

Input signal: Output signal:

},...,,{ 21 XNxxxX ∈

},...,,{ 21 YNyyyY ∈

discretememoryless

channel

yNYpNXNY

N T SUNIVERSITÄT

Transition matrix:

Input probabilities:

Output probabilities:

Relation between input and output probabilities:

( ) ( )

pppppp

pxXyYp

Ppp ⋅= XY

( ))(),...,(),( 21 XNX xpxpxp=p

( ))(),...,(),( 21 YNY ypypyp=p

N T SUNIVERSITÄT

Chain of two channels:

Output probabilities:

Resulting transition matrix:

(45))()( 2121 PPpPPpp ⋅⋅=⋅⋅= XXZ

X Y P2Z

21 PPP ⋅=

N T SUNIVERSITÄT

Binary channel:

Probabilities at input and output:

Error probability of a binary channel:

212121

212121)()(

)|()()|()()error(pxppxp

xypxpxypxpp⋅+⋅=

⋅+⋅=

1211pppp

22211211

2121 ))(),(())(),(( ppppxpxpypyp

N T SUNIVERSITÄT

Binary symmetric channel (BSC)

Error probability:

errerr21

212121)]()([

)()()error(ppxpxp

pxppxpp=⋅+=

⋅+⋅=

errerr

errerr1

ppP (49)

y11−perr

x21−perr

N T SUNIVERSITÄT

Example for mutual information – qualitative consideration: Transmission of 1000 binary statistically independent and

equiprobable symbols (p(0) = p(1) = 0.5) Binary symmetric channel with perr = 0.01 Average number of correctly received symbols: 990 But: T(X,Y) < 0.99 bit/symbol Reason: exact position of errors is not known

N T SUNIVERSITÄT

Definitions of entropy at a discrete memoryless channel: Input entropy = average information content of the input

symbols:

Output entropy = average information content of the output symbols:

Joint entropy = average information content (uncertainty) of the whole transmission system:

)(1ld)()(

∑ ∑= =

⋅=X YN

j jiji yxp

yxpYXH1 1 ),(

1ld),(),(

N T SUNIVERSITÄT

Conditional entropy H(Y|X) = average information content at the output for known input symbols = entropy of the irrelevance

Conditional entropy H(X|Y) = average information content at the input for known output symbols = entropy of the information that is lost in the channel = entropy of the equivocation

∑ ∑= =

⋅=X YN

j ijji xyp

yxpXYH1 1 )|(

1ld),()|(

∑ ∑= =

⋅=X YN

j jiji yxp

yxpYXH1 1 )|(

1ld),()|(

N T SUNIVERSITÄT

Relations between different types of entropy:

Average information flow (average mutual information):

)|()()|()(),(),( YXHYHXYHXHXYHYXH +=+==

)()|( XHYXH ≤

)()|( YHXYH ≤

),()()(

)|()(),(

YXHYHXH

YXHXHYXT

N T SUNIVERSITÄT

Information flow

source sourceencoder

sourcedecoder sink

H(U)H(X)

H(X|Y)

H(Y|X)

equivocation

irrelevance

information flow

)ˆ(UH

transmission channel

T(X,Y)

N T SUNIVERSITÄT

Examples – information flow: Ideal noiseless channel:

Entropies: H(X|Y) = 0, H(Y|X) = 0, H(X,Y) = H(X) = H(Y)T(X,Y) = H(X) = H(Y)

Useless, very noisy channel: p(xi,yj) = p(xi) ⋅ p(yj) = p(yj|xi) ⋅ p(xi) ⇒ p(yj|xi) = p(yj) ⇒ pij = pkj

Entropies: H(X|Y) = H(X), H(Y|X) = H(Y), H(X,Y) = H(X) + H(Y)T(X,Y) = 0

pij for 0for 1

N T SUNIVERSITÄT

Example for mutual information – quantitative consideration: Transmission of 1000 binary statistically independent and

equiprobable symbols (p(0) = p(1) = 0.5) Binary symmetric channel with perr = 0.01

bit/symbol9192.0

)(11ld1

1ld)1())1()0((1

)|(1ld)|()(

)(1ld)(

)|()(),(

errerr

−−+−=

⋅⋅−⋅=

∑ ∑∑= ==

xypxypxp

XYHYHYXTX YY N

j ijiji

N T SUNIVERSITÄT

Channel capacity The average mutual information (average information flow) depends on

the probability of the source symbols. Definition of channel capacity:

Channel capacity = maximum average mutual information flow ∆T = symbol period Unit for channel capacity: bit/s C depends on properties of the channel – it does not depend on the

source!

),(max1

ixp∆= (62)

N T SUNIVERSITÄT

Definition of the average information flow ... Average information flow = entropy / time:

H′(X) = H(X) / ∆T

Average mutual information flow = average mutual information /

time:T′(X,Y) = T(X,Y) / ∆T

Decisions for selecting symbols / time: H0′(X) = H0(X) /∆T

N T SUNIVERSITÄT

Example: binary symmetric channel – BSC:

−+−

+−−+−−+

−+−+=

errerr

err1err1err1err1

11ld)1(1ld

211ld)21(

21ld)2(

)|()(),(

pppppppp

XYHYHYXT

N T SUNIVERSITÄT

Average mutual information

Maximum for p1 = p(x1) = 0.5

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 p(x1)

T(X,Y)

lperr = 0

perr = 0.1

perr = 0.2

perr = 0.3

perr = 0.5

N T SUNIVERSITÄT

Channel capacity

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 perr

C⋅∆T / bit

1ld)1(1ld1),(max errerr

errerr

pYXTTCixp

−+−==∆⋅

N T SUNIVERSITÄT

Example: binary erasure channel – BEC

P err1 pTC −=∆⋅

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 perr

C⋅∆T / bit

(69)(68)

N T SUNIVERSITÄT

Theorem of channel capacity (Shannon 1948) For any ε > 0 and any information flow of a source R smaller than

channel capacity C (R < C ), a binary block code of length n (nsufficiently large) can be found, so that the residual error probability after decoding in the receiver is smaller than ε .

Reverse statement: even with largest coding effort the residual error probability cannot decrease below a certain limit for R > C.

Proof of the theorem uses random block codes (random coding argument): Proof for an average over all codesMost known codes are bad codes

No rule for construction of codes

N T SUNIVERSITÄT

Channel capacity: optimum for infinite code word length ⇒ infinite time delay, infinite complexity

More practical theorem related to the theorem of channel capacity by definition of Gallager‘s error exponent for a DMC with NX input symbols:

There exists always an (n,k) block code with RC = k / n ld NX < C ∆T, so that the word error probability is given by:

⋅−⋅−= ∑ ∑

+≤≤

sijixps

xypxpRsRE1

CG )|()(ldmaxmax)(

)(w CG2 REnP ⋅−<

N T SUNIVERSITÄT

Gallager‘s error exponent Properties:

EG(RC) > 0 for RC < C·∆TEG(RC) = 0 for RC ≥ C·∆T

Definition of R0(computational cut-off rate) :

R0 = EG(RC = 0) 0.0

0.0 0.2 0.4 0.6 0.8 1.0

EG(RC)

R0 C ·∆T

N T SUNIVERSITÄT

Computational cut-off rate R0:Maximum of EG(RC = 0) lies at s = 1

Comparison for s = 1:

⋅−=== ∑ ∑

xpxypxpRER

1)(CG0 )|()(ldmax)0(

)(CG )|()(ldmax)( RRxypxpRRE

xp−=

⋅−−≥ ∑ ∑

N T SUNIVERSITÄT

R0-theorem There exists always an (n,k) block code with RC = k / n ld NX < C ∆T,

so that the word error probability (for maximum likelihood decoding) is given by:

No rules for construction of good codes Range of values for the code rate:

0 ≤ RC ≤ R0 PW is bounded by (75)R0 ≤ RC ≤ C ∆T an upper bound for PW is difficult to calculateRC > C ∆T PW cannot become arbitrarily small

)(w C02 RRnP −−< (75)

N T SUNIVERSITÄT

Comparison between channel capacity and computational cut-off rate R0 :

0.0 0.1 0.2 0.3 0.4 0.5 perr

C ∆T

N T SUNIVERSITÄT

Coding Theory Channel coding in digital communication systems

Goals: principles and examples for block codes Design of code words

Binary code words Redundant codes Code C = set of all code words

Code word c = (c0, c1, ... , cn-1) with c ∈ C Encoding is a memoryless assignment:

information word code word u = (u0, u1, ... , uk-1) → c = (c0, c1, ... , cn-1)

k information bits n code bits n ≥ k

N T SUNIVERSITÄT

Identical codes: codes with the same code words

Equivalent codes: Codes, which become identical after a permutation of bits

General notation: (n,k,dmin)q block code

q = number of symbols (size of alphabet)

Code rate:

Number of code words: N = qk = 2k

1C ≤=nkR (76)

N T SUNIVERSITÄT

Systematic codes: Code word c = (u, p)

m = n-kparity-check bits

Non-systematic codes: information and parity-check bits cannot be separated

Linear block codes can always be converted into equivalent systematic codes

u0 u1 u2 u3 ... ... uk-1

↓ ↓ ↓ ↓ ↓ ↓ ↓

c0 c1 c2 c3 ... ... ck-1 ck ck+1 ... ... cn-1

N T SUNIVERSITÄT

Addition and multiplication in a binary number system (modulo 2):

Two binary vectors x and y with the same length Hamming distance:

dH(x,y) = number of different bits for x and y

Example: dH( 0 1 1 1 0 1 0 1,1 0 1 0 0 1 0 1 ) = 3

⊕ 0 10 0 11 1 0

⊗ 0 10 0 01 0 1

N T SUNIVERSITÄT

(Hamming) weight of a vector x:

Example: wH(0 1 1 1 0 1 0 1) = 5

Hamming distance: dH(x,y) = wH(x + y)

Example: x = ( 0 1 1 1 0 1 0 1 )y = ( 1 0 1 0 0 1 0 1 )

x + y = ( 1 1 0 1 0 0 0 0 )wH(x + y) = 3

0 fromdifferent bits ofnumber )(1

0H == ∑

iixw x (79)

N T SUNIVERSITÄT

Transmission via a binary channel:

y = x + e

e = error vector

Repetition code k = 1 information bit → n − 1 repetitions

2k = 2 code words: c1 = (0 0 0 ... 0) and c2 = (1 1 1 ... 1) Repetition code is a systematic code Simple decoding by majority vote (especially, if n is odd) (n − 1)/2 errors can be corrected, n − 1 errors can be detected

(81)x y

N T SUNIVERSITÄT

Example for a repetition code: n = 5 ⇒ RC = 1/5

u1 = (0) → c1 = (0 0 0 0 0) and u2 = (1) → c2 = (1 1 1 1 1)

Transmission via a noisy channel:e1 = (0 1 0 0 1) and x1 = (1 1 1 1 1) y1 = x1 + e1 = (1 0 1 1 0) →

e2 = (1 1 0 1 0) and x2 = (0 0 0 0 0) y2 = x2 + e2 = (1 1 0 1 0) →

Two errors can be corrected, four detected

)1(ˆ1 =u

)1(ˆ 2 =u

N T SUNIVERSITÄT

Parity check code k information bits, a single parity check bit (m = 1) → n = k + 1

2k code words

c = (u , p) with p = u0 + u1 + u2 + ... + uk−1

(number of ones in code words c is even)

The parity check code is a systematic code.

No error can be corrected, an odd number of errors can be detected. Parity check:

s0 = y0 + y1 + y2 + ... + yn−1 = 0 → no errors0 = y0 + y1 + y2 + ... + yn−1 = 1 → error

N T SUNIVERSITÄT

Example: k = 3

y1 = ( 0 1 1 0) → no error y2 = ( 1 1 1 0) → error

code word information bits

Parity check bit

c0 000 0 c1 001 1 c2 010 1 c3 011 0 c4 100 1 c5 101 0 c6 110 0 c7 111 1

N T SUNIVERSITÄT

Hamming code A single error in a codeword can be corrected. Example: (7,4) Hamming code

Parity check bits:c4 = c0 + c1 + c2

c5 = c0 + c1 + c3

c6 = c0 + c2 + c3

u0 u1 u2 u3

↓ ↓ ↓ ↓

c0 c1 c2 c3 c4 c5 c6

(83a)(83b)(83c)

N T SUNIVERSITÄT

Matrix representation of block codesx = u G

G = generator matrix (k × n matrix)

Systematic block codes:

G = [Ik P]

Ik = identity matrix k × k, P = parity bit matrix k × (n − k)

Example: (7,4) Hamming code

110|101|011|111|

1000010000100001

G (86)

N T SUNIVERSITÄT

(7,4)-Hamming code continued

Calculation of code words by matrix multiplication

Number of code words: 2k = 16

Number of all possible receiver input words: 2n = 128

( ) ( )

110|101|011|111|

1000010000100001

001|10011001

N T SUNIVERSITÄT

Table of code words

parity check bits

c8 1000 111 c9 1001 100 c10 1010 010 c11 1011 001 c12 1100 001 c13 1101 010 c14 1110 100 c15 1111 111

parity check bits

c0 0000 000 c1 0001 011 c2 0010 101 c3 0011 110 c4 0100 110 c5 0101 101 c6 0110 011 c7 0111 000

N T SUNIVERSITÄT

Properties of linear block codes Each code word is a linear combination of rows of G.

The code is given by all linear combinations of rows of G. The sum of code words is again a code word. Any linear block code contains the all-zeros vector (0 0 0 ... 0).

⇒ A code is linear, if it can be constructed by matrix multiplication x = u G (u = arbitrary information vector).

Generator matrix: row vectors must be linearly independent.

N T SUNIVERSITÄT

Elementary row operations do not change a code: Interchanging two rowsMultiplication of a row with a scalar (different from 0) Addition of a row to a second

The minimum distance between two code words equals the minimum weight:

Proof:

( )( ) minH

212121Hmin;|)(min

;,|),(minww

dd=≠∈=

≠∈=

0ccccccccc

( )( )( )( )0ccc

0ccc0cccccc0

cccccc

≠∈=

≠∈+=

≠∈=

;|)(min;|),(min

;,|),(min;,|),(min

212121H

212121Hmin

N T SUNIVERSITÄT

Error correction for a (7,4) Hamming code: Evaluation of parity check equations:

s0 = y0 + y1 + y2 + y4

s1 = y0 + y1 + y3 + y5

s2 = y0 + y2 + y3 + y6

Syndrome : s = (s0 s1 s2)

The syndrome does not depend on the code word, only on the error.

Table for the assignment of error positions

(88a)(88b)(88c)

syndrome error position s0 s1 s2

no error 0 0 0 error at bit No. 0 1 1 1 error at bit No. 1 1 1 0 error at bit No. 2 1 0 1 error at bit No. 3 0 1 1 error at bit No. 4 1 0 0 error at bit No. 5 0 1 0 error at bit No. 6 0 0 1

N T SUNIVERSITÄT

Decoding of Hamming codes: Evaluation of parity check equations → syndrome Error position read from the syndrome table Error correction: add a „1“ at the error position

m = n − k parity check bits assign 2m − 1 error positions and the error-free transmission

Code word length: n = 2m − 1 Possible parameters for Hamming codes:

(2m − 1,2m − 1 − m) block code

m 2 3 4 5 6 7 8n 3 7 15 31 63 127 255k 1 4 11 26 57 120 247

N T SUNIVERSITÄT

Parity check matrix Definition: c HT = 0 for all c ∈ C

and x HT ≠ 0 for all x ∉ C

Properties of the parity check matrix:

0 = c HT = (u G) HT = u (G HT) → G HT = 0

→ Generator matrix and parity check matrix are orthogonal.

Elementary row operations for H are allowed.

Parity check matrix: H = (PT In − k) with G = [Ik P]

Dimensions of the parity check matrix H: (n − k) × n

N T SUNIVERSITÄT

Proof of orthogonality:

Dual code Code C : generator matrix G, parity check matrix H

Dual code Cd : generator matrix Gd = H, parity check matrix Hd = G

Code words of both codes are orthogonal:

with c = u G ∈ C and cd = v H ∈ Cd it follows:

(93)( ) 0PPIPPII

PPIHG =+=+=

−knk

( ) ( ) 0TTTTTd ==== v0uvHGuHvGucc

N T SUNIVERSITÄT

Example for dual codes:

Repetition code

Parity check code:

( )111|1 =G

010001

( )111|1 =H

010001

N T SUNIVERSITÄT

Calculation of the syndrome using a matrix representation:s = y HT

Properties of the syndrome:

The syndrome is the all-zeros vector only, if y is a code word.

All error vectors e are detected which are not code words.

The syndrome does not depend on the code word:

s = y HT = (x + e) HT = e HT

N T SUNIVERSITÄT

110|101|011|111|

1000010000100001

== −

)I( TknPH

N T SUNIVERSITÄT

s = y HT

( ) ⇔

−−−=

100010001

110101011111

610 yyy ss0 = y0 + y1 + y2 + y4s1 = y0 + y1 + y3 + y5s2 = y0 + y2 + y3 + y6

N T SUNIVERSITÄT

Design of Hamming codes The syndrome depends only on the error vector:

s = e HT

A single error at the position i (ei = 1) leads to a syndrome, which equals the corresponding row of HT .

⇒ All rows of HT have to be different, so that the error position can be determined uniquely.

No row of HT must be the all-zeros vector. ⇒ The rows of HT / columns of H are built up by all possible

sequences except the all-zeros vector.

N T SUNIVERSITÄT

Parity check matrix of a systematic Hamming code:

== −

010001

all possible column vectors with more than one „1“

1000010000100001

|110|101|011|000

1001010100111110

1101101101111111

N T SUNIVERSITÄT

Modifications of linear codes Extending of codes: additional parity check bits

n' > n, k' = k, m' > m, RC' < RC, dmin' ≥ dmin

Puncturing of codes: reducing the number of parity check bitsn' < n, k' = k, m' < m, RC' > RC, dmin' ≤ dmin

Lengthening of codes: additional information bitsn' > n, k' > k, m' = m, RC' > RC, dmin' ≤ dmin

Shortening of codes: reducing the number of information bits n' < n, k' < k, m' = m, RC' < RC, dmin' ≥ dmin

N T SUNIVERSITÄT

Example: Hamming code: A single error can be corrected by syndrome decoding. Two errors lead to: s ≠ 0

Extended Hamming code: Detection of a double error situation by an additional parity

check bit (2m,2m − 1 − m) block code Generator matrix of the extended Hamming code:

HextH,

GG (101)

HextH,

N T SUNIVERSITÄT

σi are the sums of the rows of G: σi = gi,0 + gi,1 + ... + gi,n−1

Error events: No error: s = 0 A single error: s ≠ 0, sm+1 = 1 Two errors: s ≠ 0, sm+1 = 0

Decoding: s = 0 : receiver input vector = code word s ≠ 0, sm+1 = 1 : odd number of errors ⇒ a single error is

assumed and is corrected by evaluation of the syndrome s ≠ 0, sm+1 = 0 : even number of errors ⇒ errors cannot be

corrected

N T SUNIVERSITÄT

Error correction and error detection Minimum distance between code words:

dmin = 1 : a single error can neither be corrected nor detected dmin = 2 : at least a single error can be detected dmin = 3 : at least a single error can be corrected and at least two

errors can be detected

( )C∈= 2121Hmin ,|),(min ccccdd

N T SUNIVERSITÄT

Error correction and error detection

Number of errors that can be detected: te = dmin − 1

Number of errors that can be corrected: If dmin is even: t = (dmin − 2) / 2 If dmin is odd: t = (dmin − 1) / 2

dmin = 3 dmin = 4

(103)(104)

N T SUNIVERSITÄT

Singleton bound: minimum distance and minimum weight of a linear code are limited by:

Proof: systematic code word with a single information bit unequal 0 weight / distance in information bits: dH,information = 1 weight / distance in parity check bits: dH,parity ≤ m = n − k

A code, for which equality in (105) holds, is called maximum distance separable (MDS)

mknwd +=−+≤= 11minmin (105)

N T SUNIVERSITÄT

Error detection (102): te + 1 = dmin ≤ 1 + m ⇒ m ≥ te

At least one parity check bit is needed per error to be detected.

Error correction (103,104): (dmin − 1) / 2 ≥ t ≥ (dmin − 2) / 2

2 t + 1 ≤ dmin ≤ 1 + m ⇒ m ≥ 2 t

At least two parity check bits are needed per error to be corrected.

N T SUNIVERSITÄT

Decoding sphere n-dimensional sphere around a code word with radius t – all vectors

within the decoding spheres are decoded as the respective code word Total number of vectors within decoding spheres ≤ total number of

all possible vectors ⇒

Hamming bound For a binary (n,k) block code with error correction capability t the

following relation holds:

(108)nt

N T SUNIVERSITÄT

Proof: Number of vectors around a code word with dH = 1:

Number of vectors around a code word with dH = 2:

Number of vectors around a code word with dH = t:

Total number of vectors within decoding spheres:

1)1())1(()1(

⋅⋅−⋅−−⋅⋅−⋅

tttnnn

nktnnnn

N T SUNIVERSITÄT

Perfect codes In (108) equality holds. All received words fall into decoding spheres. Only a few perfect codes are known.

Example: (7,4) Hamming code dmin = wmin = 3 ⇒ t = 1

In general: Hamming codes are perfect.

2128)71(16

==+⋅

N T SUNIVERSITÄT

Plotkin bound For a binary (n,k) block code with minimum distance dmin the

following relation holds:

Approximation:

Proof: minimum weight ≤ average weight Average weight of any bit of a code word: 1/2 Average weight of a code word of length n: n/2 Average weight without all-zeros vector:

(110)12

2 1min

−⋅≤

12for2min >>≤ knd

2 −⋅ k

N T SUNIVERSITÄT

Gilbert-Varshamov bound There is a binary (n,k) block code with a minimum distance dmin , if

the parameters n, k, dmin are related by:

Bound for the existance of a code, which does not give a construction rule

(111)kn

i in −

−∑ 2

N T SUNIVERSITÄT

Proof: parity check matrix

c is a code word:

For each code word, there are at least wmin = dmin ones⇒ dmin columns of H depend linearly on each other⇒ (dmin − 1) columns of H are linearly independent

nknknkn

hhhhhh

,22,21,2

,12,11,1

−−−

==⇒=n

TT 0hcH0Hc

N T SUNIVERSITÄT

A column hSj and (dmin − 2) additional columns have to be linearly independent.

This is fulfilled, if the column hSj can create more different column vectors than can be created by linear combinations of the (dmin − 2) additional columns.

Number of different columns of hSj : 2n−k

Number of linear combinations with exactly i columns in (n − 1) columns:

Total number of linear combinations for 1 ... (dmin − 2) columns:

i in −

−∑ 2

N T SUNIVERSITÄT

Bounds of large code word lengths (n → ∞): Relation between code rate RC and normalized distance dmin/n

Singleton bound:

Hamming bound:

Plotkin bound:

Elias bound:

Gilbert-Varshamov bound:

(112)ndR min

C 1−≤

⋅−≤

ndSR min

ndR min

C 21 ⋅−≤

−−−≤

ndSR min

C 211211

−≥

ndSR min

N T SUNIVERSITÄT

Asymptotic bounds (n → ∞) between code rate RC and normalized distance dmin / n

0.0 0.2 0.4 0.6 0.8 1.0

RC=k/n

SingletonPlotkinEliasHamming

dmin / n

Gilbert-Varshamov

upperbounds

N T SUNIVERSITÄT

Weight distribution Number of code words with weight i: Ai

Weight function: two representations

The weight function can be calculated analytically only for a few codes. Weight distribution of linear codes:

A0 = 1; An ≤ 1Ai = 0 for 0 < i < dmin

Some codes: symmetric weight distribution: Ai = An−i

= ∑=

yxAyxW

= ∑=

−(117) (118)

N T SUNIVERSITÄT

Example: (4,3) parity check code

code word information word

parity check bit

weight

c0 000 0 0 c1 001 1 2 c2 010 1 2 c3 011 0 2 c4 100 1 2 c5 101 0 2 c6 110 0 2 c7 111 1 4

4261)( zzzA ++=

N T SUNIVERSITÄT

Weight distributions allows exact calculation of residual error probability

MacWilliams identity Given: (n,k) block code with weight function A(z) Weight function of the dual code:

respectively:

⋅+⋅= −zzAzzA nk

11)1(2)(d

),(2),(d yxyxWyxW k −+⋅= −

⋅+⋅= −−zzAzzA nkn

11)1(2)( d

),(2),( d)( yxyxWyxW kn −+⋅= −−

N T SUNIVERSITÄT

Example: Code C = {000,011,101,110} (parity check code)

generator matrix:

weight function: A(z) = 1 + 3 z2

Dual code Cd = {000,111} (repetition code)

generator matrix: Gd = ( 1 1 1 )

weight function:

110101

1131)1(2)( z

zzzzA +=

+⋅+⋅= −

N T SUNIVERSITÄT

Channel coding and decoding

Strategies for decoding Goal: minimum word error probability

)ˆ()ˆ(W xxuu ≠=≠= ppp (121)

channelencoder

code wordestimator

assignmentof information

wordschannel

ex y xu u

channel decoder

N T SUNIVERSITÄT

Possible transmission results: y = x error-free transmission

decoder result: y ≠ x, y ∈ C error into a different code word – cannot be

corrected decoder result:

y ≠ x, y ∉ C erroneous transmission – error can be detected and may be correcteddecoder result:

uuxx == ˆ,ˆ

uuxx ≠≠ ˆ,ˆ

N T SUNIVERSITÄT

Possible decoding results

Correct decoding:

Error-free channel

Errors are corrected in the right way

Erroneous decoding:

Error was corrected in a wrong way

Decoder failure:

An error cannot be corrected, since the decoder does not find a solution.

uuxx == ˆ,ˆ

uuxx ≠≠ ˆ,ˆ

N T SUNIVERSITÄT

Word error probability for a DMC:

Probability of an error:

Assumption: transmitted code words are equiprobable

∑∈

⋅≠=

Cxxxxx

xx)sent ()sent |ˆ(

)ˆ(Wpp

∑∑≠

=≠xx

xyxxxyy ˆ|

)|(ˆ|

)sent |received ()sent |ˆ( ppp

N T SUNIVERSITÄT

Word error probability

has to be chosen such, that becomes maximum

∑ ∑∑ ∑

∑ ∑

⋅−=

−⋅=

∈ =∈

x xxyx y

)ˆ|(21

])|()|([2

x )ˆ|( xyp

N T SUNIVERSITÄT

Optimum decision strategies Knowledge of the receiver: y

Strategy of the receiver: maximize the a-posteriori probability p(x|y)

MAP(maximum a posteriori probability) decoder:

Search for a code word which maximizes p(x|y) xx ˆ=

C∈≥= xyxyxx allfor )|()|ˆ( pp (125)

N T SUNIVERSITÄT

Bayes theorem:

Assumption: equiprobable information / code words: (identical a-priori probabilities): p(xi) = 2−k

p(y|x) = likelihood function

A-posteriori probability and likelihood function exhibit their maximum at the same position

Maximum likelihood decoder:Search for the code word which maximizes p(y|x)

)()()|()|(

yxxyyx

pppp ⋅

)|()()|( 0 xyyyx pkp ⋅=

xx ˆ=

C∈≥= xxyxxy allfor )|()ˆ|( pp

N T SUNIVERSITÄT

Maximum likelihood decoder minimizes the word error probability

Maximum likelihood decoder for a BSC:

The estimation result is the code word with the minimum Hamming distance to the received vector

errerr

),(err

)1()1(

forfor1

)|()|(

xyxyxy

xypxyp

⋅−=⋅−=

≠=−

=∏∏

C∈≤ xxyxy allfor ),()ˆ,( HH dd

N T SUNIVERSITÄT

Example: Code C = {c1,c2,c3,c4} = {00000,11100,00111,11011} dmin = 3 Example for maximum likelihood decoding:

y dH(y,c1) dH(y,c2) dH(y,c3) dH(y,c4) 00111 3 4 0 3 c3 10000 1 2 4 3 c1 11000 2 1 5 2 c2 10001 2 3 3 2 c1 or c4

N T SUNIVERSITÄT

Maximum likelihood decoding (MLD): optimum method Large compuational effort Number of comparisons of vectors: 2k

More than t errors may be corrected

Bounded distance decoding (BDD) Sphere around each code word with radius t0, Decoding only those received vectors y which lie within a single

sphere Received vectors y which do not lie in a sphere or lie in more

than one sphere, are not decoded

N T SUNIVERSITÄT

Bounded minimum distance decoding (BMD) Sphere around each code word with radius t, Decoding only those received vectors y which lie within spheres Decoding spheres are disjunct

MLD BDD BMD

N T SUNIVERSITÄT

More suboptimum decoding schemes: Syndrome decoding

simple realization suboptimum, since not all available information is utilized

Majority vote

N T SUNIVERSITÄT

Probability of errors in code words Transmission channel: binary symmetric (memoryless) channel with

error probability perr

Average number of errors in a code word:

Probability of no error:

Probability of a single error at a certain position (e = e1, wH(e1) = 1):

errerror pnn ⋅=

nppppp )1()1()1()1()( errerrerrerr −=−⋅⋅−⋅−== 0e

n factors

1errerr1 )1()( −−⋅== nppp ee

N T SUNIVERSITÄT

Probability of a single error at any arbitrary position (e = e1, wH(e1) = 1):

Probability of ne errors at arbitrary positions (e = ene, wH(ene

) = ne):

Binomialcoefficients:

Binomial theorem:

1errerr1 )1()( −−⋅⋅== nppnp ee

)1()( errerre

nnnn pp

p −−⋅⋅

=−⋅

babakn

2;)(00

∑∑=

N T SUNIVERSITÄT

Binomial distribution of bit errors per code word for n = 7

0.0 1.0 2.0 3.0 4.0 5.0

p(wH(e) = ne)

perr = 0,01

perr = 0,1perr = 0,3

N T SUNIVERSITÄT

Error detection: undetected errors

linear block code C = {c0,c1,c2, ... ,c2k-1} with cν = (cν,0,cν,1, ... ,cν,n-1) and c0 = 0

An error remains undetected if e ∈ C and e ≠ 0

Probability of undetected errors pue:

Undetected error: e = cν

⇒ ej = cν,j = 0 at n − wH(cν) bit positions with p(ej = 0) = 1 − perr

⇒ ej = cν,j = 1 at wH(cν) bit positions with p(ej = 1) = perr

∑−

===≠∩∈=

1ue )()(

νce0ee C (136)

N T SUNIVERSITÄT

Probability of undetected errors pue:

Summation versus the weight of the code words:

with it results:

∑∑−

−−

=⋅−===

1ue HH)1()(

kkwwn pppp

ννν νν ccce

− ⋅−⋅=n

iini ppAp

minerrerrue )1(

−−⋅⋅+=

iii ppA

)1(11 errerr

−= 11

)1(err

errerrue p

pApp n

N T SUNIVERSITÄT

Approximation for small error probabilities:

Weight function: A(z) = 1 + 7 z3 + 7 z4 + z7

dmin = 3

pue = 7 (1 − perr)4 perr3 + 7 (1 − perr)3 perr

4 + perr7

= 7 perr3 − 21 perr

4 + 21 perr5 − 7 perr

6 + perr7 ≈ 7 perr

minmin

minerrerrerrue 1)( d

ii pApApAp ⋅≈−=⋅≈ ∑

=(140)

N T SUNIVERSITÄT

Residual error probability

Linear block code C = {c0,c1,c2, ... ,c2k-1} with error correction capability t

Assumption: bounded minimum distance decoding (BMD)

Word error probability:

∑∑+==

===−=

+≥=≤−=

iiwpiwp

twptwp

))(())((1

)1)(())((1

)decodingcorrect (1

N T SUNIVERSITÄT

Probability of error vectors with weight wH(e) = i(Eq. (133)):

Word error probability with BMD:

Estimate for small error probabilities:

Word error probability for MLD:

∑∑+=

− −

ini ppinppi

errerr0

errerrBMDW, )1()1(1

(142)ini pp

iwp −−⋅⋅

== )1())(( errerrH e

BMDW,MLDW, pp ≤

1errBMDW, 1+

N T SUNIVERSITÄT

Estimate for the bit error probability

The number of bit errors is limited to the number of information bits k and to i + t , since:

( )∑=

=⋅==

iiwpiw

))(()(| worddecodedper errorsbit ofNumber 1

worddecodedper errorsbit ofNumber 1

)ˆ,(),()ˆ,()ˆ,( HHHH xyyxxxuu dddd +≤≤

= i ≤ t (BMD)

=⋅+≤n

tiiwptik

1Hbit ))((),min(1 e

N T SUNIVERSITÄT

Estimate for the bit error probability:

Estimate for small error probabilities:

(149)ini

ktip −

+=−⋅⋅

≤ ∑ )1(,1min errerr1

(150)1

errmin

bit 1,1min +⋅

≤ tp

N T SUNIVERSITÄT

Example: (7,4) Hamming code:

Equality in (145) holds since the code is perfect.

2errerrerr

2errerr

6errerr

)61(7)2171(1

)1(7)1(1

pppppp

+−−−+−−=

−−−−=

−−

N T SUNIVERSITÄT

Word error probablity for Hamming codes:

-6.0 -5.0 -4.0 -3.0 -2.0 -1.0 0.0 lg perr

n = 3n = 7n = 15n = 31n = 63n = 127

pW = perr

N T SUNIVERSITÄT

Coding Theory Channel coding in digital communication systems Word error

probability and probability of undetected errors for Hamming codes:

-6 -5 -4 -3 -2 -1 0-12

lg perr

lg pue

pW = perr

pW = perr2

pW = perr3

N T SUNIVERSITÄT

Minimum distance of Hamming codes

All columns of the parity check matrix are different

⇒ two columns are linear independent

Three columns are linear dependent, e.g. 100000...., 010000.... and 110000....

⇒ dmin = 3, t = 1

Weight function:

−⋅+⋅++

)1()1()1(1

n zznzn

zA (152)

N T SUNIVERSITÄT

Products of two vectors

Scalar product

Elementwise vector product

),,,,(,),,,,( 12101210 −− == nn yyyyxxxx yx

11221100T

−− ⋅++⋅+⋅+⋅=⋅ nn yxyxyxyx yx

),,,,( 11221100 −−=× nn yxyxyxyx yx

N T SUNIVERSITÄT

Sum construction of codes

Given: linear (n,k1,dmin,1) block code C1 and linear (n,k2,dmin,2) block code C2

Sum construction: linear (2n,k1+k2,dmin) block code C

Generator matrix:

Distance:

⇒ dmin = min(2dmin,1,dmin,2)

{ } 221121121 andwith),(& CCCCC ∈∈+== ccccc

11G0GG

G (155)

N T SUNIVERSITÄT

Proof:

⇒ dmin ≤ min(2dmin,1,dmin,2)

for c2 = 0 and c1 ≠ 0 : wH((c1,c1)) = 2 wH(c1) ≥ 2 dmin,1

for c2 ≠ 0 and c1 ≠ 0 :

wH((c1,c1 + c2)) = wH(c1) + wH(c1 + c2)

≥ wH(c1 + (c1 + c2)) = wH(c2) ≥ dmin,2

⇒ dmin ≥ min(2dmin,1,dmin,2)

CC ∈∈ ),0(and),( 211 ccc

N T SUNIVERSITÄT

Reed Muller codes Sum construction

Parameters: r, m with 0 ≤ r ≤ m

(n,k,dmin) Reed Muller code RM(r,m) with

Recursion procedure:

RM(r + 1, m + 1) = RM(r + 1, m) & RM(r, m)

Start values: RM(0, m) = (2m, 1, 2m) repetition code

RM(m, m) = (2m, 2m, 1) uncoded

kn −

== ∑ 2,,2 min

0(157)

N T SUNIVERSITÄT

Example for the construction of the generator matrix of the Reed Muller codes RM(r,m) :

RM(1, 2) = RM(1, 1) & RM(0, 1)

Alternative construction:

( )1101 =G

110010100101

N T SUNIVERSITÄT

G0 = all-ones vector of length n = 2m: G0 = (1 1 1 1 ... 1)

G1 = m×2m matrix: columns contain all possible binary words of length m

Gl : vector products of all combinations of l row vectors of G1

Each row of G corresponds to an information bit

1 (161)

N T SUNIVERSITÄT

Example: m = 4, r = 3 ⇒ n = 16

( )11111111111111110 =G

1010101010101010110011001100110011110000111100001111111100000000

N T SUNIVERSITÄT

××××××

100010001000100010100000101000001100000011000000101010100000000011001100000000001111000000000000

gggggggggggg

××××××××

4Z3Z2Z

4Z3Z1Z

4Z2Z1Z

3Z2Z1Z

1000000010000000100010000000000010100000000000001100000000000000

gggggggggggg

N T SUNIVERSITÄT

Alternative construction of the example: m = 2, r = 1 ⇒ n = 4

101011001111

110010100101

00111111100101010110101011000000

111011101001110010100000

10010011010111110110110010100000

111011101001110010100000

N T SUNIVERSITÄT

Minimum distance: calculation using the results of sum constructions

dmin = min(2dmin,1,dmin,2)

dmin (RM(r + 1, m + 1)) = min(2 dmin(RM(r + 1, m)), dmin,2(RM(r, m)))

Table: dmin (RM(r, m)) = 2m−r(166)(167)

N T SUNIVERSITÄT

m r n 0 1 2 3 4 5

0 1 k = 1 dmin = 1

1 2 k = 1 dmin = 2

k = 2 dmin = 1

2 4 k = 1 dmin = 4

k = 3 dmin = 2

k = 4 dmin = 1

3 8 k = 1 dmin = 8

k = 4 dmin = 4

k = 7 dmin = 2

k = 8 dmin = 1

4 16 k = 1 dmin = 16

k = 5 dmin = 8

k = 11 dmin = 4

k = 15 dmin = 2

k = 16 dmin = 1

5 32 k = 1 dmin = 32

k = 6 dmin = 16

k = 16 dmin = 8

k = 26 dmin = 4

k = 31 dmin = 2

k = 32 dmin = 1

N T SUNIVERSITÄT

Decoding by majority votes: example m = 4, r = 1 ⇒ n = 16, dmin = 8 ⇒ t = 3

10101010101010101100110011001100111100001111000011111111000000001111111111111111

N T SUNIVERSITÄT

x = u G ⇒

x0 = u0x1 = u0 + u4

x2 = u0 + u3x3 = u0 + u3 + u4

x4 = u0 + u2x5 = u0 + u2 + u4

x14 = u0 + u1 + u2 + u3x15 = u0 + u1 + u2 + u3 + u4

u4 = x0 + x1

u4 = x2 + x3

u4 = x4 + x5

u4 = x14 + x15

N T SUNIVERSITÄT

Equations for other information bits:

u3 = x0 + x2 = x1 + x3 = x4 + x6 = x5 + x7 = x8 + x10 = x9 + x11 = x12 + x14 = x13 + x15

u0: majority votes for the elements of the vector:v1 = x + (0, u1, u2, u3, u4) G

N T SUNIVERSITÄT

Cosets

Given: (n,k) block code C with parity check matrix H

Syndrome: s = y HT = (x + e) HT = e HT

Number of different error vectors e: 2n − 2k

Number of possible syndromes: 2n − k

Numbering of syndromes: sµ with 0 ≤ µ ≤ 2n−k − 1

Definition:

Coset = set Mµ of all error vectors e, which yield the same syndrome sµ : Mµ = {e | e HT = sµ} (169)

N T SUNIVERSITÄT

coset leader = error vector eµ with the smallest Hamming weight

Coset leaders need not to be unique.

Property:

for e1, e2 ∈ Mµ it follows e1 + e2 ∈ C , since s = y HT = (e1 + e2) HT = e1 HT + e2 HT = sµ + sµ = 0

Mµ can be generated from:

{ } C∈+= cec with)( µµM

N T SUNIVERSITÄT

Example: (5,2) code C = {00000,10110,01011,11101}

µ eµ M µ sµ 0 00000 00000 10110 01011 11101 000 1 00001 00001 10111 01010 11100 001 2 00010 00010 10100 01001 11111 010 3 00100 00100 10010 01111 11001 100 4 01000 01000 11110 00011 10101 011 5 10000 10000 00110 11011 01101 110 6 11000 11000 01110 10011 00101 101 7 01100 01100 11010 00111 10001 111

1101001101

100100101100101

N T SUNIVERSITÄT

Cyclic codes Subset of linear block codes

Definition: a linear (n,k) block code is called cyclic, if any cyclic shift of a codeword yields again a codeword:

Example:

CC ∈⇒∈ −−− ),,,,(),,,,( 21011210 nnn cccccccc

1011000010110000101100001011

N T SUNIVERSITÄT

Code words:

For all cyclic codes at least one generator matrix with band structure can be found.

u x0000 00000001000 11010000100 01101000010 00110100001 00011011110 10001100111 01000111101 1010001

u x1011 11111111010 11100100101 01110011100 10111000110 01011100011 00101111111 10010111001 1100101

N T SUNIVERSITÄT

Generator matrices with band structure do not necessarily describe cyclic codes:

C = {00000, 11010, 01101, 10111}

A single row describes the generator matrix completely.

Description of a vector by a polynomial:

1011001011

( ) ∑−

=− =⇔=

0110 )(

iin xcxcccc c (171)

N T SUNIVERSITÄT

Examples for polynomials of code words of length n:

Summation of vectors corresponds to summation of polynomials:

c1 + c2 ⇔ c1(x) + c2(x)

c c(x)0000 ... 0 01000 ... 0 10100 ... 0 x0010 ... 0 x2

000 ... 01 xn−1

1111 ... 1 1 + x + x2 + ... + xn−1

1010 ... 0 1 + x2

N T SUNIVERSITÄT

In general, cyclic codes are non-systematic codes.

Generator matrix of a cyclic code:

n − k + 1 non-zero entries in each row: g0, g1, g2, ... , gn−k

gggggggg

00000000

000000000000

N T SUNIVERSITÄT

Information word: u = (u0, u1, u2, ... , uk−1)

Code word: c = (c0, c1, c2, ... , cn−1)

Calculation of code words using the generator matrix:

c = u G

c0 = u0 g0

c1 = u0 g1 + u1 g0

c2 = u0 g2 + u1 g1 + u2 g0

N T SUNIVERSITÄT

Simplified description information and code word vectors as well as generator matrix by polynomials:

u(x) = u0 + u1 x + u2 x2 + ... + uk−1 xk−1

c(x) = c0 + c1 x + c2 x2 + ... + cn−1 xn−1

g(x) = g0 + g1 x + g2 x2 + ... + gn−k xn−k

Coding rule: c(x) = u(x) ⋅ g(x)

c0 = u0 g0

c1 = u0 g1 + u1 g0

c2 = u0 g2 + u1 g1 + u2 g0

N T SUNIVERSITÄT

Theorem: The generator polynomial g(x) is a divisor of any code word c(x).

Example:

g(x) = 1 + x + x3, u(x) = 1 + x2 + x3

⇒ c(x) = u(x) ⋅ g(x) = 1 + x + x2 + x3 + x4 + x5 + x6 ⇔ c = (1111111)

Theorem: The generator polynomial g(x) of a (n,k) cyclic code is a divisor of the polynomial xn + 1 :

xn + 1 = 0 mod g(x) (185)

N T SUNIVERSITÄT

Proof: The following relation shall hold:

c = (c0, c1, c2, ... , cn−1) ∈ C ⇒ d = (cn−1, c0, c1, ... , cn−2) ∈ C

c(x) = c0 + c1 x + c2 x2 + ... + cn−1 xn−1

g(x) is divisor of c(x) :

c(x) = 0 mod g(x) ⇒ x ⋅ c(x) = 0 mod g(x)

d(x) = cn−1 + c0 x + c1 x2 + ... + cn−2 xn−1

= cn−1 + x (c0 + c1 x + c2 x2 + ... + cn−1 xn−1) + cn−1 xn

= c(x)

= x ⋅ c(x) + cn−1 (1 + xn)

N T SUNIVERSITÄT

d is code word only if g(x) is divisor of d(x):

⇒ 1 + xn = 0 mod g(x)

A cyclic code is obtained only, if relation (189) holds for the generator polynomial

Properties of the generator polynomial:

Condition for the coefficients of the generator polynomial:

g0 = gn−k = 1

⇒ g(x) = 1 + g1 x + g2 x2 + ... + xn−k

N T SUNIVERSITÄT

Calculation of a generator polynomial:

Division of 1 + xn into factors – each factor may be used as generator polynomial

Example: x7 + 1 = (1 + x) ⋅ (1 + x + x3) ⋅ (1 + x2 + x3)

Calculation of the code word length for a given generator polynomial: backward Gaussian division algorithm: Example: (x3 + x + 1) ⋅ p(x) = xn + 1

x3 + 0 + x + 1x4 + 0 + x2 + x

x5 + 0 + x3 + x2

x7 + 0 + x5 + x4

x7 + 0 + 0 + 0 + 0 + 0 + 0 + 1 ⇒ n = 7

N T SUNIVERSITÄT

General representation of a code word polynomial:

c(x) = (u0 + u1 x + u2 x2 + ... + uk−1 xk−1) ⋅ g(x)

⇒ g(x) is the code word polynomial with lowest degree

Systematic coding

Multiplication of the information polynomial u(x) = u0 + u1 x + u2 x2 + ... + uk−1 xk−1 with xn−k :

u(x) ⋅ xn−k = u0 xn−k + u1 xn−k+1 + u2 xn−k+2 + ... + uk−1 xn−1

(corresponds to a shift of information bits into the direction of highest exponents)

N T SUNIVERSITÄT

Division of the polynomial u(x) ⋅ xn−k by g(x) :

u(x) ⋅ xn−k = q(x) ⋅ g(x) + r(x)

with q(x) = integer quotient

r(x) = remainder of the division, in general: r(x) ≠ 0

r(x) = r0 + r1 x + r2 x2 + ... + rn−k−1 xn−k−1

Forcing g(x) being a divisor of the right-hand side:u(x) ⋅ xn−k + r(x) = q(x) ⋅ g(x) = c(x)

Code word: c = (r0, r1, r2, ... , rn−k−1, u0, u1, u2, ... , uk−1)

(193))()()(

xgxrxq

xgxxu kn

+=⋅ −

N T SUNIVERSITÄT

Example for systematic coding: g(x) = x3 + x + 1 , u = (1 0 0 1) , u(x) = x3 + 1 n = 7, k = 4, n − k = 3

u(x) ⋅ xn−k = x6 + x3

Division of the polynomial u(x) ⋅ xn−k by g(x) :

(x6 + 0 + 0 + x3) : (x3 + 0 + x + 1) = x3 + x ++ (x6 + 0 + x4 + x3)

(0 + 0 + x4 + 0 + 0 + 0)+ (x4 + 0 + x2 + x)

(0 + 0 + x2 + x) = r(x)

c(x) = u(x) ⋅ xn−k + r(x) = x6 + x3 + x2 + x ⇒ c = (0 1 1 1 0 0 1)

N T SUNIVERSITÄT

Parity-check matrix and parity-check polynomial h(x)

Relation for the generator polynomial: xn + 1 = g(x) ⋅ h(x)

h(x) is of degree k:

h(x) = h0 + h1 x + h2 x2 + ... + hk xk

with h0 = hk = 1

h(x) is the parity-check polynomial

Parity-check matrix H is of dimensions (n − k) × n

N T SUNIVERSITÄT

Parity-check matrix

−−

00000000

000000000000

hhhhhhhh

N T SUNIVERSITÄT

Decoding of the received vector y = c + e ⇔ y(x) = c(x) + e(x)

The generator polynomial is a divisor of any code word.

Division by the generator polynomial = calculation of the information word and test whether the received word is a code word:

s(x) = syndrome: y(x) = q(x) ⋅g(x) + s(x)

For s(x) ≠ 0, error correction is carried out using a syndrome table.

Realization of the division of polynomials by shift register circuits

No matrix multiplications for encoding and decoding necessary

(200))()()(

xgxsxq

N T SUNIVERSITÄT

Implementation of encoding and decoding by shift register circuits Basic circuits

Shift register: each register cell delays by one clock period

Notation: Sj(i) = contents of register cell j at time i

Sj+1(i+1) = Sj(i)

S0 . . . .S1 Sm−1

a(x) x ⋅ a(x)

N T SUNIVERSITÄT

Adding device

Multiplication device

Scaling device

a a + b

a a ⋅ b

a a ⋅ bb

N T SUNIVERSITÄT

Cyclic shift of a polynomial

Circuit calculates: xi ⋅ a(x) mod (xn + 1)

Most simple example of a feedback shift register

. . . .

x1 x2 xn−1xn−2xn−3

N T SUNIVERSITÄT

Multiplication of two polynomials

c(x) = a(x) ⋅ [b0 + b1 x + b2 x2 + ... + bn xn]

. . . .a(x)

. . . .

b1 b2 bn

N T SUNIVERSITÄT

Multiplication of two polynomials: alternative structure

c(x) = a(x) ⋅ [b0 + b1 x + b2 x2 + ... + bn xn]

. . . .a(x)

. . . .

b1bn−1bn

N T SUNIVERSITÄT

Example or a multiplication

c(x) = a(x) ⋅ b(x) = (x4 + x2 + x +1 )⋅(x3 + x2 + 1)

= x7 + x6 + x5 + x4 + 0 + 0 + x + 1

b0 = 1 b1 = 0 b2 = 1 b3 = 1

11110011

s0 s1 s2 s3

i s0 s1 s2 s3 ci

−1 0 0 0 0 00 1 0 0 0 11 1 1 0 0 12 1 1 1 0 03 0 1 1 1 04 1 0 1 1 15 0 1 0 1 16 0 0 1 0 17 0 0 0 1 18 0 0 0 0 0

N T SUNIVERSITÄT

Division of two polynomials

q(x−1) = a(x−1) xm + q(x−1) [b0 xm + b1 xm−1 + ... +bm−1 x]

⇒ a(x−1) = q(x−1) [b0 + b1 x−1 + ... + bm−1 x−(m−1) + x−m] = q(x−1) b(x−1)

Substitution x−1 → x ⇒ a(x) = q(x) b(x) ⇒

. . . .

a(x−1) . . . . q(x−1)

b1 bm−1

)()()(

xbxaxq =

N T SUNIVERSITÄT

Division of two polynomials with remainder:

The division process is stopped, when the last bit of a(x−1) has been input into the shift register.

⇒ The integer division is finished.

q(x−1) = output word

s(x−1) = contents of register cells

)()()(

xbxsxq

N T SUNIVERSITÄT

Example for the division of two polynomials with remainder:

(x8 + x3) : (x4 + x3 + x + 1) = x4 + x3 + x2 +

x0 x1 x2 x3 x4

000100001 001110000s0 s1 s2 s3

q(x−1)a(x−1)

N T SUNIVERSITÄT

Example for the division of two polynomials (cont'd)

i ai s0 s1 s2 s3 qi

9 0 0 0 0 0 08 1 1+0=1 0+0=0 0 0+0=0 07 0 0+0=0 1+0=1 0 0+0=0 06 0 0+0=0 0+0=0 1 0+0=0 05 0 0+0=0 0+0=0 0 1+0=1 04 0 0+1=1 0+1=1 0 0+1=1 13 1 1+1=0 1+1=0 1 0+1=1 12 0 0+1=1 0+1=1 0 1+1=0 11 0 0+0=0 1+0=1 1 0+0=0 00 0 0+0=0 0+0=0 1 1+0=1 0

N T SUNIVERSITÄT

Encoding circuit for a systematic cyclic code

N T SUNIVERSITÄT

Principle

1st step: inputting information bits u0, u1, u2, ... , uk−1, starting with uk−1

Input from the right hand side corresponds to a multiplication with xn−k

After inputting k information bits, the shift register contains the remainder from the division r(x)

2nd step: moving switches S1 and S2 into the dashed positions (disconnecting the feedback loop)

3rd step: output of parity-check bits

N T SUNIVERSITÄT

Example: u(x) = x3 + x2 + 1 and g(x) = x3 + x + 1

N T SUNIVERSITÄT

Example: u(x) = x3 + x2 + 1 and g(x) = x3 + x + 1

t s1 b0 b1 b2 s2 s3

0 0 0 0 0 0 01 1 1 1 0 1 12 1 1 0 1 1 13 0 1 0 0 1 04 1 1 0 0 1 15 0 1 0 0 06 0 0 1 0 07 0 0 0 0 1

N T SUNIVERSITÄT

Circuit for decoding: dividing the received word polynomial by the generator polynomial and calculating the syndrome

N T SUNIVERSITÄT

Circuit for decoding: inputting the received word from the right hand side

N T SUNIVERSITÄT

Decoding and error correction with the Meggitt decoder

N T SUNIVERSITÄT

Coding Theory Algebraic foundations for coding

Groups Definition: A set G and an operation ° are called Group (G,°), if the

following conditions are fulfilled:

The operation is associative: a ° (b ° c) = (a ° b) ° cwith a,b,c ∈ G

The set G is closed for the operation ° : if a,b ∈ G also a ° b ∈ G

The set G contains a neutral element e with the property: a ° e = e ° a = a for a, e ∈ G

For any element a of G there is an inverse element a′, so that: a ° a′ = a′ ° a = e for a, e ∈ G

N T SUNIVERSITÄT

The group is called commutative, if for any a,b ∈ G : a ° b = b ° a

The neutral element of a group is unique:

Proof with the assumption that there are two neutral elements e1 and e2: e1 = e1 ° e2 = e2 ° e1 = e2

Examples:

The set of integer numbers Z = {... −2, −1, 0, 1, 2, ...} is a commutative group with respect to addition:

neutral element: 0, inverse element for a: −a

N T SUNIVERSITÄT

The set of integer numbers Z = {... −2, −1, 0, 1, 2, ...} is no group with respect to multiplication:

There are no inverse elements.

Definition: If the group contains only a finite number of elements, it is called finite group.

Example: finite set of integer numbers G = {0, 1, 2, ... , m−1} and modulo-m addition „⊕“:

neutral element: 0, inverse element for a: a′ = m − a

rmqbamrq

+⋅=+⇔+=+

N T SUNIVERSITÄT

Example: binary set B = {0, 1} and modulo-2 addition „⊕“neutral element: 0, inverse elements: a = 1 ⇒ a′ = 1

a = 0 ⇒ a′ = 0

Example: G = {0, 1, 2, 3, 4, 5, 6} and modulo-7 addition „⊕“

neutral element: 0, inverse element for a:

a′ = m − a

⊕ 0 1 2 3 4 5 60 0 1 2 3 4 5 61 1 2 3 4 5 6 02 2 3 4 5 6 0 13 3 4 5 6 0 1 24 4 5 6 0 1 2 35 5 6 0 1 2 3 46 6 0 1 2 3 4 5

⊕ 0 10 0 11 1 0

N T SUNIVERSITÄT

Example: finite set of integer numbers G = {1, 2, 3, ... , p−1} with p = prime number and modulo-p multiplication „⊗“:

closed set: the remainder of a ⋅ b / p is non-zero, since p is a prime numberneutral element: 1, inverse elements exist

rpqbaprq

+⋅=⋅⇔+=⋅

N T SUNIVERSITÄT

Example: p = 7 ⇒ G = {1, 2, 3, 4, 5, 6} and modulo-7 multiplication „⊗“

⊗ 1 2 3 4 5 61 1 2 3 4 5 62 2 4 6 1 3 53 3 6 2 5 1 44 4 1 5 2 6 35 5 3 1 6 4 26 6 5 4 3 2 1

N T SUNIVERSITÄT

Cyclic groups

Definition: If all elements of a multiplicative group G = {1, g1, g2, ... , gm−1} can be represented as powers of at least one element gi , the group is called cyclic.

G = {1, g1, g2, ... , gm−1} consists of m elements gij :

G = {gi0, gi

1, gi2, ... , gi

m−1}

gi is called primitive element of the group (with order m)

„One“-element: 1 = gi0

Order (gk) = number of elements that can be created by gkl

N T SUNIVERSITÄT

Coding Theory Algebraic foundations for coding Examples:

multiplicative modulo-5 group multiplicative modulo-7 group

G = {1, 2, 3, 4} G = {1, 2, 3, 4, 5, 6}

primitive elements

z z1 z2 z3 z4 order 1 1 1 1 1 1 2 2 4 3 1 4 3 3 4 2 1 4 4 4 1 4 1 2

z z1 z2 z3 z4 z5 z6 order 1 1 1 1 1 1 1 1 2 2 4 1 2 4 1 3 3 3 2 6 4 5 1 6 4 4 2 1 4 2 1 3 5 5 4 6 2 3 1 6 6 6 1 6 1 6 1 2

N T SUNIVERSITÄT

Rings Definition: For the set R the operations ⊕ and ⊗ are defined and

the following properties:

R is a commutative group with restect to ⊕

R is closed with respect to ⊗ : a,b ∈ R → a ⊗ b ∈ R

R is associative with respect to ⊗ : a ⊗ (b ⊗ c) = (a ⊗ b) ⊗ cwith a,b,c ∈ R

The distributive property holds: a ⊗ (b ⊕ c) = (a ⊗ b) ⊕ (a ⊗ c) with a,b,c ∈ R

Commutative ring, if ⊗ is commutative

Ring with neutral element: a ⊗ 1 = 1 ⊗ a = a with a ∈ R

N T SUNIVERSITÄT

Example of a commutative ring with a neutral element: integer numbers Z with addition and multiplication

Example of a commutative ring with a neutral element: integer numbers Zm = {0,1, 2, ... , m−1} with modulo-m addition and modulo-m multiplication

Unique inverse element only, if m = prime number

N T SUNIVERSITÄT

Field Definition: For the set K the operations ⊕ and ⊗ are defined and the

following properties hold:

K is a commutative group with respect to ⊕ , 0 is neutral element

K without 0 is a commutative group with respect to ⊗, 1 is neutral element

Distributive property: a ⊗ (b ⊕ c) = (a ⊗ b) ⊕ (a ⊗ c) with a,b,c ∈ K

Field with a finite number of elements = Galois field

Integer numbers Zm = {0,1, 2, ... , m−1} form a Galois-Field GF(m) = Zm, if m is a prime number: m = p, or if m = pk (extended Galois field)

N T SUNIVERSITÄT

Some properties of Galois fields without proof:

a ⊗ 0 = 0 ⊗ a = 0

a ≠ 0 and b ≠ 0 → a ⊗ b ≠ 0 (no divisor of 0 – follows from closed multiplication)

a ⊗ b = 0 and a ≠ 0 → b = 0

− (a ⊗ b) = (−a) ⊗ b = a ⊗ (−b)

a ⊗ b = a ⊗ c and a ≠ 0 → b = c

1 ⊕ 1 ⊕ 1 ⊕ ... ⊕ 1 = 0

m terms

For each Galois field, at least one primitive element exists.

N T SUNIVERSITÄT

Example: GF(2)

Example: GF(5)

⊕ 0 10 0 11 1 0

⊗ 0 10 0 01 0 1

⊕ 0 1 2 3 40 0 1 2 3 41 1 2 3 4 02 2 3 4 0 13 3 4 0 1 24 4 0 1 2 3

⊗ 0 1 2 3 40 0 0 0 0 01 0 1 2 3 42 0 2 4 1 33 0 3 1 4 24 0 4 3 2 1

N T SUNIVERSITÄT

Examples in GF(5)

Addition by table: 1 + 2 = 3, 2 + 4 = 1, 4 + 4 = 3

Inverse elements of the addition: a + (−a) = 0

Subtraction using inverse elements of the addition:

3 − 2 = 3 + (−2) = 3 + 3 = 1

1 − 4 = 1 + (−4) = 1 + 1 = 2

a 0 1 2 3 4−a 0 4 3 2 1

N T SUNIVERSITÄT

Examples in GF(5)

Multiplication by table: 1 ⋅ 2 = 2, 2 ⋅ 4 = 3, 4 ⋅ 4 = 1

Inverse elements of the multiplication: a ⋅ (a−1) = 1

Division using inverse elements of the multiplication:

3 ÷ 2 = 3 ⋅ (2−1) = 3 ⋅ 3 = 4

1 ÷ 3 = 1 ⋅ (3−1) = 1 ⋅ 2 = 2

a 1 2 3 4a−1 1 3 2 4

N T SUNIVERSITÄT

Example in GF(5): system of linear equationsI: 2 x0 + x1 = 2II: 3 x1 + x2 = 3III: x0 + x1 + 2 x2 = 3

I: 2 x0 = 2 − x1II: x2 = 3 − 3 x12 ⋅ III: (2 − x1) + 2 x1 + 4 (3 − 3 x1) = 2 ⋅ 3

⇒ 2 + 4 ⋅ 3 − 2 ⋅ 3 = x1(1 − 2 + 4 ⋅ 3) ⇒ 3 = x1⇒ x2 = 3 − 3 x1 = 4⇒ x0 = 2−1 (2 − x1) = 2

N T SUNIVERSITÄT

Extension fields G = {0, 1, 2, 3} with modulo-4 addition and modulo-4 multiplication

is not a Galois field !

⊕ 0 1 2 30 0 1 2 31 1 2 3 02 2 3 0 13 3 0 1 2

⊗ 0 1 2 30 0 0 0 01 0 1 2 32 0 2 0 23 0 3 2 1

N T SUNIVERSITÄT

Extension fields Extension of Galois fields corresponding to the extension of real

numbers to complex numbers

Definition of complex numbers: R → C x2 + 1 = 0 does not have a solution for x ∈ R Extension of the field: Definition of complex numbers: c = c0 + c1α with c0, c1 ∈ R

Definition of an extended Galois field: GF (2) → GF(22) x2 + x + 1 = 0 does not have a solution for x ∈ GF (2) Extension of the field:

Def. of elements of GF(22): c = c0 + c1α with c0, c1 ∈ GF (2)

1 22 −=⇔−=⇔=+ ααα

1 22 +=⇔=++ αααα

N T SUNIVERSITÄT

Definition of an extended Galois field: GF (p) → GF(pm) p(x) = xm + pm−1xm−1 + ... + p1x + p0 = 0 does not have a

solution for x ∈ GF (p) and x ∈ GF (pn) with n < m Extension of the field:

p(α) = 0 ⇔ αm = −pm−1αm−1 − ... − p1α − p0

Def. of elements of GF(pm): c = c0 + c1α + ... + cm−1αm−1

with ci ∈ GF (p) GF(pm) contains pm elements

Very important for coding in digital communication systems: GF(pm) mit p = 2

N T SUNIVERSITÄT

Addition in C: Multiplication in C:

Addition in GF(22): Multiplication in GF(22):

ααα

)()()()(

1010baba

bbaabac

+++=+++=

)()1()(

10011100

11100100

babababababababa

bbaabac

++−=−+++=

+⋅+=⋅=

ααα

)()()()(

1010baba

bbaabac

+++=+++=

ααα

)()1()(

1110011100

11100100

bababababababababa

bbaabac

++++=++++=

+⋅+=⋅=

N T SUNIVERSITÄT

Addition and multiplication tables for GF(22) = {0, 1, α, 1+α} in component representation:

⊕ 0 1 α 1+α0 0 1 α 1+α1 1 0 1+α αα α 1+α 0 1

1+α 1+α α 1 0

⊗ 0 1 α 1+α0 0 0 0 01 0 1 α 1+αα 0 α 1+α 1

1+α 0 1+α 1 α

N T SUNIVERSITÄT

Interpretation as the remainder of polynomial divisions:

The condition x2 + x + 1 = 0 corresponds to a calculation modulo x2 + x + 1

Example in GF(22): (1 + α) ⋅ (1 + α) = 1 + α2

(α2 + 1) : (α2 +α + 1) = 1 ++ (α2 + α + 1)

α = r(α)

GF(22) is defined by all possible results of p(α) modulo α2 + α + 1

12 ++ααα

N T SUNIVERSITÄT

Addition and multiplication tables for GF(22) in vector representation:

⊕ 00 10 01 1100 00 10 01 1110 10 00 11 0101 01 11 00 1011 11 01 10 00

⊗ 00 10 01 1100 00 00 00 0010 00 10 01 1101 00 01 11 1011 00 11 10 01

N T SUNIVERSITÄT

Exponential representation

αk = αk mod 3

Addition and multiplication tables for GF(22) in exponential representation:

⊕ 0 1 α α2

0 0 1 α α2

1 1 0 α2 αα α α2 0 1α2 α2 α 1 0

⊗ 0 1 α α2

0 0 0 0 01 0 1 α α2

α 0 α α2 1α2 0 α2 1 α

0 = 01 = α0

α = α1

1+α = α2

1 = α3 = α0

α = α4 = α1

1+α = α5 = α2

... ...

)1(mod −=mpkk αα (205)

N T SUNIVERSITÄT

Definition: irreducible polynomial p(x) = polynomial of degree m, which cannot be written as a product of two polynomials pa(x), pb(x) with degree ≥ 1 and < m

Example: p(x) = x2 + x + 1

test polynomials of degree 1: pa(x) = x, pb(x) = x + 1

(x2 + x + 1) ÷ x = x + 1 +

(x2 + x + 1) ÷ (x + 1) = x +

Irreducible polynomials have similar properties as prime numbers

Theorem: irreducible polynomials do not exhibit zeros in GF (p)

N T SUNIVERSITÄT

Definition: primitive polynomial = irreducible polynomial p(x) of degree m, which exhibits a zero α (i.e. p(α) = 0) with the following property:

The terms αi mod p(α) create all possible (pm − 1) non-zero elements of the Galois field GF(pm)

α ∈ GF(pm) is called primitive element

with α0 = αn = 1 for n = pm − 1

For each Galois field GF(pm), there is at least one primitive polynomial p(x)

Primitive polynomials are irreducible.

In general, irreducible polynomials are not primitive polynomials.

N T SUNIVERSITÄT

Representation of primitive polynomials by their zeros:

Factorization by all (n = pm − 1) non-zero elements ai ∈ GF(pm):

∏−

)( )()(m

pixxp α

∏−

=−=−

N T SUNIVERSITÄT

Table of primitive polynomials

m primitive polynomial m primitive polynomial 1 x + 1 9 x9 + x4 + 1 2 x2 + x + 1 10 x10 + x3 + 1 3 x3 + x + 1 11 x11 + x2 + 1 4 x4 + x + 1 12 x12 + x7 + x4 + x3 + 1 5 x5 + x2 + 1 13 x13 + x4 + x3 + x + 1 6 x6 + x + 1 14 x14 + x8 + x6 + x + 1 7 x7 + x + 1 15 x15 + x + 1 8 x8 + x6 + x5 + x4 + 1 16 x16 + x12 + x3 + x + 1

N T SUNIVERSITÄT

Discrete Fourier transform (DFT)

Transform of polynomials:A(x) = DFT(a(x)) ⇔ a(x) = IDFT(A(x))

given: polynomials of degree ≤ n − 1 = pm − 2 with coefficients from GF(pm), z = primitive element of GF(pm):

∑−

=− =⇔=

0110 )(),...,,(

iin xaxaaaaa

∑−

=− =⇔=

0110 )(),...,,(

jjn xAxAAAAA

N T SUNIVERSITÄT

Representations of the DFT:

a(x) A(x)

time domain frequency domain

Transformation rules:

∑−

⋅−− −=−=1

jj zazaA

∑−

⋅==1

ii zAzAa (211)

N T SUNIVERSITÄT

Properties of the DFT

Uniqueness

IDFT(DFT(a(x))) = a(x)

DFT(DFT(a(x))) = −a(x)

DFT(DFT(DFT(DFT(a(x))))) = a(x)

z−j is a zero of a(x) ⇔ Aj = 0 ⇔

zi is a zero of A(x) ⇔ ai = 0 ⇔

),...,,0,,...,,(0)( 11110 −+−− =⇔= njj

j AAAAAza A

),...,,0,,...,,(0)( 11110 −+−=⇔= niii aaaaazA a

N T SUNIVERSITÄT

Matrix representation:

−−−−

−−

)1()1(21

)1(242121

zzzzzz

−−−−−−−

−−−−−−−−

)1()1(2)1(

)1(242)1(21

zzzzzz

N T SUNIVERSITÄT

Example: GF(7), z = 5, A(x) = 4 + 5x

543210

1234524024303034204254321

543210

111111

AAAAAA

zzzzzzzzzzzzzzzzzzzzzzzzz

aaaaaa

506312

326451

111111

000054

546231421421616161241241326451111111

543210

aaaaaa

N T SUNIVERSITÄT

Inverse transform: GF(7), z = 5, a(x) = 2 + x + 3x2 + 6x3 + 5x5

−−−−−−−−−−−−−−−−−−−−−−−−−

543210

1234524024303034204254321

543210

111111

aaaaaa

AAAAAA

543210

5432142042303032402412345

543210

111111

aaaaaa

AAAAAA

N T SUNIVERSITÄT

Inverse transform: GF(7), z = 5, a(x) = 2 + x + 3x2 + 6x3 + 5x5

A = (4,5,0,0,0,0) a = (2,1,3,6,0,5)

000054

000023

506312

326451241241616161421421546231111111

543210

AAAAAA

N T SUNIVERSITÄT

Shift register circuit for the DFT: ∑−

⋅−−=1

jiij zaA

a0z−j⋅0

a1z−j⋅1

an−1z−j⋅(n−1)

z−(n−1)

an−1

−1start values

Memory contents after j cycles

N T SUNIVERSITÄT

Shift register circuit for the IDFT:

A0zi⋅0

A1zi⋅1

An−1zi⋅(n−1)

zn−1

An−1

∑−

jiji zAa

start values

Memory contents after i cycles

N T SUNIVERSITÄT

Convolution theorem of the DFT

given: polynomials a(x), b(x), c(x) of a degree ≤ n − 1 = pm − 2 with coefficients from GF(pm), z = primitive element of GF(pm):

x is element of GF(pm) ⇒ xn = x0 = 1 ⇒ calculation modulo xn − 1

N T SUNIVERSITÄT

Product of two polynomials: a(x) ⋅ b(x) = c(x)

The product of two polynomials corresponds to the cyclic convolution of coefficients:

)...()...()...( 1110

−−

−− +++=+++⋅+++ n

n xcxccxbxbbxaxaa

∑−

−−

−−−

++++⋅=++++⋅=

211201101

112211000...

jnijji

ababababcababababc

)1(mod)()( −⋅⇔∗ nxxbxaba

N T SUNIVERSITÄT

Convolution theorem:

a(x) ⋅ b(x) mod (xn−1) − Aj ⋅ Bj

ai ⋅ bi A(x) ⋅ B(x) mod (xn−1)

with a(x) A(x), b(x) B(x)

Proof of (219):

c(x) = a(x) ⋅ b(x) + γ(x) ⋅ (xn−1)

Cj = − c(z−j) = − a(z−j) ⋅ b(z−j) − γ(z−j) ⋅ (z−jn−1)

−Aj −Bj =1

N T SUNIVERSITÄT

Theorem for cyclic shift in time and frequency domain

x ⋅ a(x) mod (xn−1) z−j ⋅ Aj

zi ⋅ ai x ⋅ A(x) mod (xn−1)

with a(x) A(x)

Proof by inserting b(x) = x and B(x) = x into the convolution theorem

N T SUNIVERSITÄT

Coding Theory Block codes

Reed-Solomon and Bose-Chaudhuri-Hocquenghem codes RS and BCH codes were developed about 1960

Analytical construction

Minimum distance can be preset (weight distribution is known)

The Singleton bound (dmin ≤ 1 + m) is fulfilled for RS codes with equality.

Powerful and of great practical relevance

Communication with space crafts

CD (compact disk), mobile radio communications

N T SUNIVERSITÄT

Reed-Solomon codes Fundamental theorem of linear algebra:

A polynomial A(x) = A0 + A1 x + A2 x2 + ... + Ak−1 xk−1 of degree k−1 with coefficients Ai ∈ GF(pm) and Ak−1 ≠ 0 exhibits maximally k−1 different zeros.

Proof: representation of a polynomial by linear factors:

∏−

=− −⋅=

11 )()(

iik xxAxA (223)

N T SUNIVERSITÄT

Theorem of minimum weight of vectors:

Given: A(x) = A0 + A1 x + A2 x2 + ... + Ak−1 xk−1 with coefficients Ai ∈ GF(pm)

degree {A(x)} = k − 1 ≤ n − d

A(x) a(x) ⇔ a = (a0, a1, a2, .... , an−1)

⇒ wH(a) ≥ d

Proof:

(213): zeros of A(zi) correspond to elements ai = 0

⇒ number of non-zero elements: n − (k − 1) ≥ d

N T SUNIVERSITÄT

Definition of Reed-Solomon codes:

Given: z is primitive element of GF(pm).

a = (a0, a1, a2, .... , an−1) is code word of length n = pm − 1 ofthe RS code C with dimension k = pm − d and minimumdistance d = n − k + 1, if:

C = {a | ai = A(zi), degree{A(x)} ≤ k − 1 = n − d}

a = (a0, a1, a2, .... , an−1) A = (A0, A1, A2, .... , Ak−1,0,0, ... ,0)

d − 1 parity check frequencies

N T SUNIVERSITÄT

For RS codes the Singleton bound

holds with equality: n − k = dmin − 1

optimum case: dmin is odd ⇒ t = (dmin − 1)/2

n − k = 2 t

⇒ number of errors that can be detected or corrected can be adjusted

Example:

Choice of the Galois field GF(pm): p = 2, m = 4 ⇒ GF(24)

Code word length: n = pm − 1 = 16 − 1 = 15

Number of correctable symbols: t = 3 ⇒ n − k = dmin − 1 = 6

)(1min knd −+≤

N T SUNIVERSITÄT

Code word in the frequency domain:

k = n − (dmin − 1) = 9 dmin − 1

Aj ∈ {0, 1, z, z+1, z2, z2+1, z2+z, z2+z+1, z3, z3+1, z3+z, z3+z+1, z3+z2, z3+z2+1, z3+z2+z, z3+z2+z+1}

Number of information symbols: k = 9

Number of information bits: k ⋅ m = 9 ⋅ 4 = 36

Number of code word symbols: n = 15

Number of code word bits: n ⋅ m = 15 ⋅ 4 = 60

A0 A1 A2 A3 A4 A5 A6 A7 A8 0 0 0 0 0 0

N T SUNIVERSITÄT

⇒ Hexadecimal (15,9) RS code = binary (60,36) code

Capability to correct errors: t = 3 hexadecimal symbols

⇒ maximum number of correctable bits: tb,max = 3 ⋅ 4 = 12

⇒ minimum number of correctable bits: tb,min = 3 ⋅ 1 = 3

Examples for error vectors in binary representation

e = (0000 0000 0000 1111 1111 1111 0000 0000 0000 0000 0000 0000 0000 0000 0000) ⇒ correctable

e = (0000 0000 0001 1111 1111 1110 0000 0000 0000 0000 0000 0000 0000 0000 0000) ⇒ not correctable

e = (0100 0000 0000 0000 0001 0000 0000 0100 0000 0000 0000 0000 0000 0010 0000) ⇒ not correctable

N T SUNIVERSITÄT

Any 3 single bit errors and any 9 directly adjacent bit errors correctable

Error bursts correctable up to a length of

tb,burst = m (t − 1) + 1 = m (d − 3) / 2 + 1 bits

Code rate:

Example: k / n = 9 / 15 = 60 %

nkR 11)1(

−=−−

N T SUNIVERSITÄT

Generator polynomial

Code word in the frequency domain: Aj = 0 for j = k ... n − 1

DFT: Aj = − a(z−j)

Each code word polynomial exhibits zeros:

a(z−j) = 0 for j = k ... n − 1

Product of all linear factors which are obtained from the zeros:

Each code word polynomial can be represented as: a(x) = u(x) ⋅ g(x) ⇒ g(x) is the generator polynomial

∏∏−

− −=−=kn

j zxzxxg1

1)()()( (231)

N T SUNIVERSITÄT

Parity check polynomial

The parity check polynomial contains all other non-zero zeros:

Proof:

degree{g(x)} = n − k , degree{u(x)} = k − 1 , degree{g(x) ⋅ u(x)} = n − 1, degree{h(x)} = k

∏∏+−=

− −=−=n

j zxzxxh1

0)()()(

)1(mod0)1()()()(

)()()()()(1

0−=−⋅=−⋅=

⋅⋅=⋅

∏−

i xxxuzxxu

xhxgxuxhxa

N T SUNIVERSITÄT

Example: (6,2) RS code for GF(7) with z = 5

4652)65()56(

)2)(6()4)(5(

)5()5()(

xxxxxxxx

++++=++⋅++=

−−⋅−−=

−=−= ∏∏==

)3)(1(334652

6−−=++=

xxxxxxxx

n(233)

N T SUNIVERSITÄT

Generator matrix (coding of cyclic codes): a = u ⋅ G

Example:

−−

gggggggg

210210

00000000

000000000000

= 146520

014652G

N T SUNIVERSITÄT

Transformation into a systematic code

substitute the first row by the sum of both rows:

multiply all elements by 4:

= 146520

1534021G

= 423610

465201systG

N T SUNIVERSITÄT

Parity check matrix, 1st version

331000033100003310000331

−−−−

−−

021021

00000000

000000000000

hhhhhhhh

kkkkkk

N T SUNIVERSITÄT

Parity check matrix, 2nd version: G = [Ik P] ⇒ H2 = [−PT In-k]

Parity check matrix, 3rd Version: direct evaluation of parity check equations

−−−−−−−−

=100033010051001042000115

100044010026001035000162

1for0)(1

0−=⋅−=

−==−=

∑−

nkjzaAn

)1)(1(

+−−−

+−+−+

−−−

−−

−−−−⋅−=−−−−⋅−=

zazazazaAzazazazaA

N T SUNIVERSITÄT

Example:

−−−−−−−

+−−+−+−

−−−−

)1()1(2)1(

)2)(1()2(2)2(

)1)(1()1(2)1(

zzzzzz

−−−−−−−−−−−−−−−−

326451241241616161421421

12345240243030342042

zzzzzzzzzzzzzzzzzzzz

N T SUNIVERSITÄT

Calculation of the syndrome

Received signal: r = a + e R = A + E

r(x) = a(x) + e(x) R(x) = A(x) + E(x)

Syndrome = received vector in the frequency domain at the parity check frequencies

A(x) = A0x0 + A1x1 + ... + Ak−1xk−1

E(x) = E0x0 + E1x1 + ... + Ek−1xk−1 + Ekxk + ... + En−1xn−1

R(x) = R0x0 + R1x1 + ... + Rk−1xk−1 + Ekxk + ... + En−1xn−1

with Rj = Aj + Ej syndrome coefficients

S(x) = S0x0 + ... + Sn−k−1xn−k−1 = Ekx0 + ... + En−1xn−k−1

N T SUNIVERSITÄT

Syndrome as a vector: S = (S0, S1, ... , Sn−k−1)

Syndrome as a multiplication with the parity check matrix:

S = r ⋅ H3T ⇔ ST = H3 ⋅ rT (242)

−−−−−−−

+−−+−+−+−−+−+−

−−−−

−− 1

)1()1(2)1(

)2)(1()2(2)2()1)(1()1(2)1(

knkkknkkknkk

zzzzzz

N T SUNIVERSITÄT

Encoding of RS codes

Extending the definition (226):

C = {a | ai = A(zi) with Aj = 0 for n − k cyclically adjacent symbols j}

Proof: cyclic shift theorem of the DFT

Method 1: encoding in the frequency domain

Transformation of information symbols with the IDFT

A = (u0, u1, u2, .... , uk−1,0,0, ... ,0) a = (a0, a1, a2, .... , an−

N T SUNIVERSITÄT

u0 A0 a0 c0

u1 A1 a1 c1

u2 A2 a2 c2

uk−1 Ak−1

0 Ak+1

0 An−1 an−1 cn−1

................

∑−

ijji zAa

IDFT....

Infor-mationvector

Codeword

N T SUNIVERSITÄT

n = p − 1 = 7 − 1 = 6, k = 2 ⇒ degree{A(x)} ≤ k − 1 = 1

A(x) = 3 + 3x ai = A(x = zi)

a0 = A(x = z0 = 50 = 1) = A0 + A1z0 = 3 + 3 ⋅ 1 = 6 a1 = A(x = z1 = 51 = 5) = A0 + A1z1 = 3 + 3 ⋅ 5 = 4 a2 = A(x = z2 = 52 = 4) = A0 + A1z2 = 3 + 3 ⋅ 4 = 1 a3 = A(x = z3 = 53 = 6) = A0 + A1z3 = 3 + 3 ⋅ 6 = 0 a4 = A(x = z4 = 54 = 2) = A0 + A1z4 = 3 + 3 ⋅ 2 = 2 a5 = A(x = z5 = 55 = 3) = A0 + A1z5 = 3 + 3 ⋅ 3 = 5

A = (3,3,0,0,0,0) a = (6,4,1,0,2,5)

N T SUNIVERSITÄT

B(x) = 1 + 1x

B = (1,1,0,0,0,0) b = (2,6,5,0,3,4)

dH(a,b) = dH((6,4,1,0,2,5),(2,6,5,0,3,4)) = 5

for comparison: dmin = n − k + 1 = 6 − 2 + 1 = 5

Method 2: non-systematic encoding in the time domain

Multiplication of information and generator polynomial

a(x) = u(x) ⋅ g(x) (246)

N T SUNIVERSITÄT

Method 3: Systematic encoding in the time domain

Multiplication of the information polynomial with xn−k :

u(x) ⋅ xn−k = u0xn−k + u1xn−k+1 + ... + uk−1xn−1

Division of u(x) ⋅ xn−k by g(x) :

u(x) ⋅ xn−k = q(x) ⋅ g(x) + r(x)

result = remainder: r(x) = r0 + r1x + ... + rn−k−1xn−k−1

Code word:

− r(x) + u(x) ⋅ xn−k = q(x) ⋅ g(x) = a(x)

a = (−r0, −r1, −r2, ... , −rn−k−1, u0, u1, u2, ... , uk−1)

N T SUNIVERSITÄT

u = (3,4) ⇒ u(x) = 3 + 4x ⇒ u(x) ⋅ xn−k = 3x4 + 4x5

Division by g(x) = 2 + 5x + 6x2 + 4x3 + x4 :

(4x5 + 3x4) ÷ (1x4 + 4x3 + 6x2 + 5x + 2) = 4x + 1 +− (4x5 + 2x4 + 3x3 + 6x2 + 1x)

1x4 + 4x3 + 1x2 + 6x − (1x4 + 4x3 + 6x2 + 5x + 2)

2x2 + 1x + 5 = r(x)

−r(x) + u(x) xn−k = 2 + 6x + 5x2 + 3x4 + 4x5 ⇒ a = (2, 6, 5, 0, 3, 4)−r u

N T SUNIVERSITÄT

Alternative: Encoding with the generator matrix of a systematic code:

u = (3,4)

Example: (7,3) RS code for GF(23)

» code word length: n = pm − 1 = 8 − 1 = 7

» number of correctable symbols: t = 2 ⇒ n − k = 4, k = 3

» primitive polynomial: p(x) = 1 + x + x3

» primitive element: 1 + z + z3 = 0

( ) ( )0,5,6,2,4,34236104652014,3syst =

⋅=⋅= Gub

N T SUNIVERSITÄT

» extended Galois field:

» generator polynomial:g(x) = (x − z) (x − z2) (x − z3) (x − z4)

= (x2 + xz4 + z3) (x2 + xz6 + 1)= x4 + x3z3 + x2 + xz + z3

» parity check polynomial:h(x) = (x − z5) (x − z6) (x − z7)

= x3z3 + x2z3 + xz2 + z4

0 = 0 z1 = z z3 = 1+z z5 = 1+z+z2

z0 = 1 z2 = z2 z4 = z+z2 z6 = 1+z2

N T SUNIVERSITÄT

» encoding of: u = (1, 1, 1) ⇒ u(x) = 1 + x + x2

⇒ u(x) ⋅ xn−k = x4 + x5 + x6

division by g(x) = x4 + x3z3 + x2 + xz + z3 :

(x6 + x5 + x4) ÷ (x4 + x3z3 + x2 + xz + z3) = x2 + xz + z4 + + (x6 + x5z3 + x4 + x3z + x2z3)

x5z + x3z + x2z3

+ (x5z + x4z4 + x3z + x2z2 + xz4)x4z4 + x2z5 + xz4

+ (x4z4 + x3 + x2z4 + xz5 + 1)x3 + x2 + x + 1 = r(x)

−r(x) + u(x) xn−k = 1+x+x2+x3+x4+x5+x6 ⇒ a = (1,1,1,1,1,1,1)

N T SUNIVERSITÄT

» generator matrix:

Weight function of RS codes (holds also for maximum distance separable codes in general, q = pm):

110001100011

zzzzzz

iAi 1for 0

0for 1

ii zAzA

jdiji ≥

−−−

−− for 1

)1()1(0

N T SUNIVERSITÄT

Example for a weight distribution: (7,3) RS code for GF(23)

Minimum distance: d = (n − k) + 1 = 5

A0 = 1

A1 = A2 = A3 = A4 = 0

1477217)!57(!5

!71757

5 =⋅=⋅−⋅

=⋅⋅

147)58()!67(!6

!7)1158(76

76 =−⋅

−⋅=⋅

−⋅⋅

217)154864(7)12681

68(777 2

7 =+−⋅=⋅

−⋅⋅

N T SUNIVERSITÄT

Weight function:

Correctness check:

RS codes are especially suited for correction of burst errors

RS codes may be inefficient when correcting single errors, since always whole symbols have to be corrected.

The capability of correcting symbols requires more redundancy than that of correcting single bits.

===+++=n

33 85122171471471

765 2171471471)( zzzzA +++=

N T SUNIVERSITÄT

Decoding RS codes

Algebraic decoding

Decoding by tables: sufficient memory

Algebraic decoding

Procedure

Calculation of the syndrome

Calculation of error positions (key equation)

Calculation of the values of the error vector (in general, RS codes are non-binary codes)

Correction of errors

N T SUNIVERSITÄT

Calculation of error positions

Error position polynomial: zeros indicate error positions

c(x) C(x)

definition: ci = 0 for ei ≠ 0

⇒ ci ⋅ ei = 0 for i = 0, 1, ... , n − 1

each error position ci = 0 generates a zero / a linear factor in C(x):

degree{C(x)} = number of error positions e

∏≠

−=0,

)()(iei

izxxC (257)

N T SUNIVERSITÄT

DFT of the product ci ⋅ ei :

ci ⋅ ei = 0 C(x) ⋅ E(x) = 0 mod (xn − 1)

C(x) ⋅ E(x) exhibits all possible zeros

assumption:

error position polynomial:

C(x) = C0 + C1x + ... + Cexe

selecting C0 : C0 = 1

2knte −

(260)e

i xCxCxzxCi

+++=⋅−= ∏≠

− ...1)1()( 10,

N T SUNIVERSITÄT

Error position polynomial

a(x) xxxxxxxxxxxxxxxxxxxx A(x) xxxxxxxxxxxxxx000000

e(x) 00000x000x000000x000 E(x) xxxxxxxxxxxxxxxxxxxx

r(x) xxxxxxxxxxxxxxxxxxxx R(x) xxxxxxxxxxxxxxxxxxxx S(x)

c(x) xxxxx0xxx0xxxxxx0xxx C(x) xxxx0000000000000000

ci⋅ei = 0 C(x)⋅E(x) = 0 mod (xn−1)

N T SUNIVERSITÄT

Calculation of coefficients Ci : assumption: e = t Representation of C(x) ⋅ E(x) = 0 mod (xn − 1) as an equation

system:

1322110

2423120

1212110

222110

12322110

1120110

221100

=+++=++++

=++++=++++=++++

=++++=++++

−−−−−

+−−−−+−

−−−−−−

−−−−−−−−

+−−

−−−

tntnnn

tnttntntn

ECECECECECECECEC

ECECECECECECECECECECECEC

ECECECECECECECEC

N T SUNIVERSITÄT

Frame in (261):

t equations and t variables

All required coefficients for calculation of Ci are known:

S0 = En−2t , S1 = En−2t+1 , .... , S2t−1 = En−1

= cyclic convolution:

1322221120

112110022110

=++++=++++

−−−−

−+−−

tttttttt

SCSCSCSC

SCSCSCSCSCSCSCSC

− −==⋅t

iiji ttjSC

012,,for0 (263)

N T SUNIVERSITÄT

Alternative representation by multiplication of polynomials C(x)⋅S(x) :

02221212

132222112012

1121101

022110

0121101

−−−

−−

−−−−−

−−

−−−−

=+++=++++

=++++=++++

tttttt

TSCSCSCxSCSCSCSCx

SCSCSCSCxSCSCSCSCx

SCSCSCx

SCSCxSCx

N T SUNIVERSITÄT

Definition of the polynomial T(x), which can be used for calculation of error values:

T(x) = T0 + T1x + ... + Tt−1xt−1

Comparison of coefficients ⇒ key equation:

C(x) ⋅ E(x) = 0 mod (xn − 1)

= T(x) ⋅ (xn − 1) = T(x) ⋅ xn − T(x)

In general, the number of errors is unknown: e ≠ t

Errors with small weight are more likely than errors with large weight ⇒ assumption: e ≤ t

N T SUNIVERSITÄT

Searched coefficients: (C0,) C1, C2, ... , Ce

Number of variables: e, number of equations: t Overdetermined equation system ⇒ different approaches are

required for C(x):

SSSSSSSS

etettt

,,2,1for

1222212

22123222

−−−−−

−−−−−−

tetejSCe

iiji ,,2,1for12,,for0

0 =−==⋅∑

N T SUNIVERSITÄT

Starting the search for coefficients of the error position polynomial with the most likely error event: e = 1, e = 2, ...

Decoder failure is possible, if no error position polynomial with e zeros is found.

Solution of the key equation with low computational complexity:

Berlekamp-Massey algorithm

Euklid’s divisions algorithm

N T SUNIVERSITÄT

Example: (6,2)-RS-Code for GF(7) with z = 5

Parameters: n = 6, k = 2, d = dH,min = 5, t = 2

Received vector and syndrome:

r = (r0,r1,r2,r3,r4,r5) = (1,2,3,1,1,1)

R = (R0,R1,R2,R3,R4,R5) = (R0,R1,E2,E3,E4,E5) = (5,0,4,6,6,1)

S = (E2,E3,E4,E5) =(S0,S1,S2,S3) = (4,6,6,1)

Assumption e = 1: C(x) = C0 + C1x normalization: C1 = 1

1616646

231201 C

SSSSSS

N T SUNIVERSITÄT

6 C0 + 4 = 0 ⇒ C0 = 46 C0 + 6 = 0 ⇒ C0 = 61 C0 + 6 = 0 ⇒ C0 = 1

Contradiction ⇒ e = 2: C(x) = C0 + C1x + C2x2 normalization: C2 = 1

I: 6 C0 + 6 C1 + 4 = 0II: 1 C0 + 6 C1 + 6 = 0I−II: 5 C0 − 2 = 0 ⇒ C0 = 2⋅5−1 = 6 II: 6 C1 + 5 = 0 ⇒ C1 = −5⋅6−1 = 5

1661466

123012 C

SSSSSS

N T SUNIVERSITÄT

Error position polynomial: C(x) = 6 + 5 x + x2

Search for zeros (Chien search):C(x = z0 = 1) = 6 + 5 + 1 = 5 C(x = z1 = 5) = 6 + 4 + 4 = 0 ⇒ zero at z1

C(x = z2 = 4) = 6 + 6 + 2 = 0 ⇒ zero at z2

C(x = z3 = 6) = 6 + 2 + 1 = 2 C(x = z4 = 2) = 6 + 3 + 4 = 6 C(x = z5 = 3) = 6 + 1 + 2 = 2

Errors at the 1st and 2nd symbol position

Test: C(x) = (x − z1)⋅(x − z2) = 6 + 5 x + x2

N T SUNIVERSITÄT

Recursive calculation of symbol error values

Eq. (258) corresponds to a cyclic convolution of coefficients:

ci ⋅ ei = 0 C(x) ⋅ E(x) = 0 mod (xn − 1)

with C(x) = C0 + C1x + ... + Cexe

and E(x) = E0 + E1x + ... + Ek−1xk−1 + Ekxk + Ek+1xk+1 + ... + En−1xn−1

unknown part known part (S)

) mod(index 1,,1,0for00

nnjECe

iiji −==⋅∑

=− (269)

0 nnjECECe

iijij −==⋅+ ∑

=− (270)

N T SUNIVERSITÄT

Solving for Ej and recursive calculation of error values:

Example (contd.): (6,2) RS code for GF(7) with z = 5

R = (R0,R1,E2,E3,E4,E5) = (5,0,4,6,6,1) C(x) = 6 + 5 x + x2

E0 = −C0−1 ⋅ (C1 ⋅ E−1 + C2 ⋅ E−2)

= −C0−1 ⋅ (C1 ⋅ E5 + C2 ⋅ E4)

= −6−1 ⋅ (5 ⋅ 1 + 1 ⋅ 6) = 1 ⋅ (5 + 6) = 4

10 nnjECCE

iijij −=⋅⋅−= ∑

− (271)

N T SUNIVERSITÄT

E1 = −C0−1 ⋅ (C1 ⋅ E0 + C2 ⋅ E−1)

= −C0−1 ⋅ (C1 ⋅ E0 + C2 ⋅ E5)

= −6−1 ⋅ (5 ⋅ 4 + 1 ⋅ 1) = 1 ⋅ (6 + 1) = 0

Complete error vector in the frequency domain:

E = (E0,E1,E2,E3,E4,E5) = (4,0,4,6,6,1)

Transformation back into the time domain:

e = (e0,e1,e2,e3,e4,e5) = (0,1,2,0,0,0)

Result:

= r − e = (1,2,3,1,1,1) − (0,1,2,0,0,0) = (1,1,1,1,1,1) a

N T SUNIVERSITÄT

Correcting errors and erasures for RS codes

Discrete memoryless channel with erasures for transmission of symbols from GF(pm)

N T SUNIVERSITÄT

Description of the errors-and-erasures channel:

r = a + e + b

Input vector: a =(a0,a1,a2,...,an−1) with ai ∈ GF(pm)

Output vector: r =(r0,r1,r2,...,rn−1) with ri ∈ {GF(pm),?}

Error vector: e =(e0,e1,e2,...,en−1) with ei ∈ GF(pm)

Erasure vector: b =(b0,b1,b2,...,bn−1) with bi ∈ {0,?}

Calculation rules for erasures (with ai ∈ GF(pm)):

0for?0for0

N T SUNIVERSITÄT

Error position polynomial:

Erasure position polynomial:

degree{Ce(x)} = number of errors

degree{Cb(x)} = number of erasures

Numbers of errors e and erasures b that can be corrected:

b + 2 ⋅ e ≤ 2 ⋅ t = d − 1 = n − k

∏≠

−=0,

)()(iei

ie zxxC

−=?,

)()(ibi

ib zxxC

N T SUNIVERSITÄT

Procedure

Modification of the received word: multiplication with the erasure position polynomial

Calculation of error positions

Calculation of error and erasure values

1. Modification of the received word

Erasure position polynomial in the time domain Cb(x) cb(x) :

cb,i = Cb(x = zi) = 0 für {i|bi = ?}

⇒ bi ⋅ cb,i = 0

N T SUNIVERSITÄT

Definition:

Transformation into the frequency domain:

bcrr ×=~

ibiibiibiiiibii

cecacbeacrr~~

)(~,,,,

⋅+⋅=⋅++=⋅=

)1mod()(~)(~)1mod()()()()(

)1mod()())()(()()()(~

−⋅+⋅=

−⋅+==

xxCxExCxA

xxCxExAxCxRxR

N T SUNIVERSITÄT

Effect of the multiplication in the frequency domain:

(n−2t) symbols 2t symbols

(n−2t) symbols b symbols (2t−b) symbols (syndrome )

ibii crr ,~ ⋅=)0,,0,0,,,,( 110 −= kAAAA

)0,,0,0,~,,~,~,~,,~,~(~11110 −++−= bkkkk AAAAAAA

)~,,~,~,~,,~,~,~,,~,~(~1111110 −+++−++−= nbkbkbkkkk RRRRRRRRR R

)~,,~,~,~,,~,~,~,,~,~(~1111110 −+++−++−= nbkbkbkkkk EEEEEEEEE E

)~,,~,~,~,,~,~,~,,~,~( 1111110 −+++−++−= nbkbkbkkkk EEERRRRRR

N T SUNIVERSITÄT

Length of the remainder of the syndrome: (2t−b) symbols

⇒ number of correctable errors

2. Calculation of error positions

Restriction of the key equation on the remainder of the polynomial

Key equation:

⇒ error positions and erasure positions are known

−=≤2maxbtee (283)

)~,,~,~()~,,~,~(~111210 max −+++− == nbkbke EEESSS S (284)

(285)max

0max, ,,2,1for12,,for0~ eeeejSC

iijif =−==⋅∑

N T SUNIVERSITÄT

3. Calculation of error and erasure values

Errors-and-erasures polynomial

Modification of the received vector

Substitution of erasures by an arbitrary value (0):

Error vector with

?für0ˆ)(fürˆ

==∈=

miiirr

prrr GF

∏=∩≠

−=⋅=?)()0(,

)()()()(ii bei

ibe zxxCxCxC

rr ˆ→

)0()0(for0ˆ?)()0(for0ˆ

=∩===∪≠≠

iiibeebee

N T SUNIVERSITÄT

Modified received vector:

estimate for the transmitted code word

Condition for error-and-erasures position polynomial:

Recursive calculation of error values corresponding Eq. (271):

EARear ˆˆˆˆˆˆ +=+= (289)

) mod(index 1,,1,0forˆˆ1

10 nnjECCE

iijij −=⋅⋅−= ∑

)1(mod0)(ˆ)(0ˆ −=⋅=⋅ nii xxExCec

N T SUNIVERSITÄT

with C(x) = C0 + C1x + ... + Ce+bxe+b

unknown part known part ( )

)ˆ,,,ˆ,ˆ,ˆ,,ˆ,ˆ(ˆ 11110 −+−= nkkk EEEEEE E (293)

N T SUNIVERSITÄT

Parameters: n = 6, k = 2, d = dH,min = 5, t = 2

Code word vector: A = (A0,A1,A2,A3,A4,A5) = (3,3,0,0,0,0)

Code word vector: a = (a0,a1,a2,a3,a4,a5) = (6,4,1,0,2,5)

Error vector: e = (e0,e1,e2,e3,e4,e5) = (0,5,0,0,0,0)

Erasure vector: b = (b0,b1,b2,b3,b4,b5) = (0,0,0,?,?,0)

Received vector: r = (r0,r1,r2,r3,r4,r5) = (6,2,1,?,?,5)

Erasure position polynomial: Cb(x) = (x−z3)⋅(x−z4) = x2 + 6x + 5

Number of erasures: b = 2

Number of correctable errors: 1222

2max =−=

−=≤btee

N T SUNIVERSITÄT

Transform of the erasure position polynomial into time domain:

Cb = (5,6,1,0,0,0) cb = (5,4,3,0,0,4)

1. Modification of the received vector:

Transform into the frequency domain:

Syndrome of the modified received vector:

)6,0,0,3,1,2()4,0,0,3,4,5()5?,?,,1,2,6(~ =×=×= bcrr

)5,4,2,2,1,2(~)6,0,0,3,1,2(~ == Rr

)5,4()~,~(~10 == SSS

N T SUNIVERSITÄT

Assumption for the error position polynomial: e = 1, emax = 1

Ce(x) = Ce,0 + Ce,1x

Definition: Ce,1 = 1

Key equation:

5Ce,0 + 4 = 0 ⇒ Ce,0 = −4 ⋅ 5−1 = 3 ⋅ 3 = 2

Error position polynomial: Ce(x) = 2 + x

Error position search: Ce(zi) = 0

Ce(z0=1) = 2 + 1 = 3, Ce(z1=5) = 2 + 5 = 0 ⇒ i = 1

( ) ( ) 01

45~~ 0,

0,01 =

N T SUNIVERSITÄT

Errors-and-erasures polynomial:

C(x) = Ce(x) ⋅ Cb(x) = (x + 2)⋅(x + 1)⋅(x + 5) = x3 + x2+ 3x + 3

(C0,C1,C2,C3) = (3,3,1,1)

2. modification of the received vector:

Recursive calculation of the error values:

)0,2,0,1,3,0(ˆ)5,0,0,1,2,6(ˆ == Rr

S)0,2,0,1,ˆ,ˆ()ˆ,ˆ,ˆ,ˆ,ˆ,ˆ(ˆ 10543210 EEEEEEEE ==E

n) mod(index 1,,1,0forˆˆ1

10 −=⋅⋅−= ∑

− njECCEbe

N T SUNIVERSITÄT

Estimate of the error vector in the frequency domain:

3032021011

ˆˆˆ)ˆ~ˆ(ˆ

ECECECCE −−−− ++⋅−=

422)012103(3ˆ 10 =⋅=⋅+⋅+⋅⋅−= −E

3132121111

ˆˆˆ)ˆˆˆ(ˆ

ECECECCE −−−− ++⋅−=

002)210143(3ˆ 11 =⋅=⋅+⋅+⋅⋅−= −E

)0,2,0,1,0,4()ˆ,ˆ,ˆ,ˆ,ˆ,ˆ(ˆ 543210 == EEEEEEE

N T SUNIVERSITÄT

Error correction:

==−=−=

)5,2,0,1,4,6()0,5,0,0,5,0()5,0,0,1,2,6(ˆˆˆ

)0,0,0,0,3,3()0,2,0,1,0,4()0,2,0,1,3,0(ˆˆˆ

N T SUNIVERSITÄT

BCH codes (Bose-Chaudhuri-Hocquenghem codes) Extension of Hamming codes

Optimized for correction of several statistically independent single errors

Special case of RS codes

Binary Code, code word in the time domain: ai ∈ GF(2)

Code word in the frequency domain: Ai ∈ GF(2m)

Definition similar as RS codes in the frequency domain by zeros at parity check frequencies in the time domain by a generator polynomial

N T SUNIVERSITÄT

Conjugate elements

Two elements a, b ∈ GF(pm) are called conjugate with each other a ∼ b, if a can be written as a power of b:

Properties:

for all a ∈ GF(pm) : a ∼ a

for all a, b ∈ GF(pm) : a ∼ b ⇒ b ∼ a

for all a, b, c ∈ GF(pm) : a ∼ b and b ∼ c ⇒ a ∼ c

(294)rpba =

N T SUNIVERSITÄT

Equivalence class: subsets composed from all conjugate elements [zi] = {zi, zj, zk, zl, ...} with zi ∼ zj ∼ zk ∼ zl ...

Equivalence classes build up a disjunct partition of GF(pm)

Circular classes

Origin: complex numbers:

nth root of 1:

Extended Galois fields can be viewed as circular fields:

Circular classes Kj contain all exponents of elements which are conjugate with each other:

12j == πezn

101 ===− zzz npm

1with}1,,1,0|mod{ −=−=⋅= mij pnminpjK

N T SUNIVERSITÄT

Example: GF(24) with primitive polynomial p(x) = x4 + x + 1 and primitivem Element z mit z4 + z + 1 = 0

Elements in GF(24): GF(24) = {0, z0, z1, z2, ... , zn−1}

with n = pm − 1 = 24 − 1 = 15

Calculation modulo n in the exponents: z0 = zn

Equivalence class: {z1, z2, z4, z8}, circular class: {1, 2, 4, 8}

( )( )

( )( ) 11628181

11624141

11622121

zzzzzz

===⇒

N T SUNIVERSITÄT

Minimal polynomial, definition and properties

For each a ∈ GF(pm) there is a uniquely defined normalized polynomial m[a](x) of minimum degree with coefficients from GF(p) for which a is a zero. ⇒ minimal polynomial

m[a](x) is irreducible

Two conjugate elements have the same minimal polynomial: a ∼ b ⇒ m[a](x) = m[b](x)

Degree of the minimal polynomial = number of elements of the equivalence class

N T SUNIVERSITÄT

Calculation of the minimal polynomial:

Product of all minimal polynomials from GF(pm)

One of the minimal polynomials corresponds to the primitive polynomial.

∏∏∈∈

−=−=][,][,

][ )()()(a

abba axbxxm (296)

1)(][ −=∏ n

ia xxm

i (297)

N T SUNIVERSITÄT

Example (continued)

m0(x) ⋅ m1(x) ⋅ m3(x) ⋅ m5(x) ⋅ m7(x) = x15 − 1

Kreisteilungsklasse MinimalpolynomK0 = {0} m0(x) = (x − z0) = x + 1K1 = {1,2,4,8} m1(x) = (x − z1) (x − z2) (x − z4) (x − z8) = x4 + x + 1K3 = {3,6,9,12} m3(x) = (x − z3) (x − z6) (x − z9) (x − z12) = x4 + x3 + x2 + x + 1K5 = {5,10} m5(x) = (x − z5) (x − z10) = x2 + x + 1K7 = {7,11,13,14} m7(x) = (x − z7) (x − z11) (x − z13) (x − z14) = x4 + x3 + 1

N T SUNIVERSITÄT

Derivation of BCH codes in the frequency domain

Binary code: ai ∈ GF(2), Ai ∈ GF(2m)

Code word length: n = 2m − 1

DFT for odd n: n mod 2 = 1 ⇒ n−1 = 1

Ai = a(x = z−i) , aj = A(x = zj)

Zeros in the frequency domain: Ai = 0 = a(x = z−i)

Theorem of polynomials f(x) in GF(2): [f(x)]2 = f(x2)

A2i = a(x = z−2i) = a(x = (z−i)2) = [a(x = z−i)]2 = (Ai)2

A4i = a(x = z−4i) = a(x = (z−2i)2) = [a(x = z−2i)]2 = (A2i)2 = (Ai)4

A8i = a(x = z−8i) = a(x = (z−4i)2) = [a(x = z−4i)]2 = (A4i)2 = (Ai)8

(300)(301)

N T SUNIVERSITÄT

General condition for which ai ∈ GF(2):

The (n,k,dmin) BCH code is defined by the parameters p, m, d which can be selected:code word length: n = pm − 1minimum distance: dmin ≥ ddimension: k ≤ n + 1 − dmin ≤ n + 1 − dnumber of parity check symbols: n − k ≥ 2 t

C = {a | ai = A(zi), degree{A(x)} ≤ n − d, ai ∈ GF(p), Ai ∈ GF(pm)}

with z = primitive element from GF(pm).

(303)( ) )2()2(

jj ii AA =⋅

N T SUNIVERSITÄT

binary BCH code: p = 2

C = {a | ai = A(zi), degree A(x) ≤ n − d, ai ∈ GF(2), Ai ∈ GF(2m)}

C = {a | ai = A(zi), degree A(x) ≤ n − d, A(2 j⋅i) = (Ai)(2 j), Ai ∈ GF(pm)}

Example: BCH code for GF(23) with n = 23 − 1 = 7, t = 1, d = 3

Code word vector: A = (A0,A1,A2,A3,A4,A5,A6) = (A0,A1,A2,A3,A4,0,0)

A0: A(2 j⋅ 0) = A0 = (A0)(2 j) ⇒ A0 ∈ GF(2)

A1: A(2 j⋅ 1) = A(2 j) = (A1)(2 j)

⇒ A2 = (A1)2 , A4 = (A1)4 , A8 mod 7 = A1 = (A1)8 ⇒ A1 ∈ GF(23)

N T SUNIVERSITÄT

A3: A(2 j⋅ 3) = (A3)(2 j)

⇒ A6 = (A3)2 , A12 = A5 = (A3)4, A24 = A3 = (A3)8 ⇒ A3 ∈ GF(23)

but: A5 = A6 = 0 ⇒ A3 = 0

Encoding example with p(x) = 1 + x + x3, 1 + z + z3 = 0, prim. el.: z

A0 = 1, A1 = z ⇒ A = (1,z,z2,0,z4,0,0), A(x) = 1 + z x + z2 x2 + z4 x4

a0 = A(z0) = 1+z1+z212+z414 = 1+z+z2+z4 = 1+z+z2+z+z2 = 1

a1 = A(z1) = 1+zz+z2z2+z4z4 = 1+z2+z4+z8 = 1+z2+z+z2+z = 1 ...

⇒ a = (1,1,0,1,0,0,0)

0 = 0 z1 = z z3 = 1+z z5 = 1+z+z2

z0 = 1 z2 = z2 z4 = z+z2 z6 = 1+z2

N T SUNIVERSITÄT

Generating a BCH code by DFT

N T SUNIVERSITÄT

Constructing BCH codes in the time domain with generator polynomial

Kj are circular classes with respect to n = 2m − 1

z is primitive element in GF(pm)

M is the union of the circular classes: M = {Kj1∪Kj2

∪...}

A BCH code of length n = 2m − 1 is determined by the generator polynomial:

If there are d − 1 subsequent numbers in M, there designed minimum distance is d .Dimension : k = n − grad (g(x))

(306)⋅⋅=−= ∏∈

)()()()(21

xmxmzxxg jjMi

N T SUNIVERSITÄT

Example in GF(24) with n = 24 − 1 = 15

designed minimum distance: d = 3, since M = {K1,K2} = K1 = {1,2,4,8} contains d − 1 = 2 subsequent numbers ( M = K1 = {K1∪K2} !)Dimension: k = n − degree {g(x)} = 15 − 4 = 11

designed minimum distance: d = 5, since M = {K1∪K3} = {1,2,3,4,6,8,9,12} contains d − 1 = 4 subsequent numbers Dimension: k = n − degree {g(x)} = 15 − 8 = 7

1)()()( 41

},{ 21

++==−= ∏∈

xxxmzxxgKKi

1)1()1(

)()()()(

46782344

31},,,{ 4321

++++=++++⋅++=

⋅=−= ∏∈

xxxxxxxxxx

xmxmzxxgKKKKi

N T SUNIVERSITÄT

designed minimum distance: d = 7, since M = {K1∪K3∪K5} = {1,2,3,4,5,6,8,9,10,12} contains d − 1 = 6 subsequent numbersDimension: k = n − degree {g(x)} = 15 − 10 = 5

)1()1()1(

)()()()()(

245810

531},,,{ 621

++++++=

++⋅++++⋅++=

⋅⋅=−= ∏∈

xxxxxx

xxxxxxxx

xmxmxmzxxgKKKi

N T SUNIVERSITÄT

Encoding of BCH codes as any other linear cyclic codes

non-systematic encoding by multiplication with a generator polynomial

systematic by division by the generator polynomial

Difference with respect to RS codes: shift of parity check frequencies has an influence on the performance

Decoding similar to RS codes

advantage: for a binary code, calculation of error values is not necessary

N T SUNIVERSITÄT

Coding Theory Convolutional codes

Properties and definition no blockwise generation of code words, but convolution of a whole

sequence with a set of generator coefficients

no analytical methods for construction of code words ⇒ computer search

simple processing of reliability information of the demodulators (soft decision input)

sensitive with respect to burst errors

usually binary codes

ML decoding with the Viterbi algorithm

N T SUNIVERSITÄT

Definition of convolutional codes:

Information blocks ur = (ur,1,ur,2, ... ,ur,k)

Code blocks ar = (ar,1,ar,2, ... ,ar,n)

Binary code: ai,j , ui,j ∈ GF(2) = {0,1}

Encoding is an assignment with memory: The actual code block is determined by the actual and m previous information blocks:

ar = ar(ur, ur−1, ... , ur−m)

Convolutional codes are linear ⇒ linear combination of information bits:

∑ ∑= =

−⋅=m k

rr uga0 1

,,,,µ κ

κµνµκν

N T SUNIVERSITÄT

Meaning of indices:

r = block number

k = length of information blocks

n = length of code blocks

m = memory length

µ = memory index

κ = index for the position within an information block

ν = index for the position within a code block

N T SUNIVERSITÄT

Binary Coefficients: gκ,µ,ν ∈ GF(2) = {0,1}

with 1 ≤ κ ≤ k , 0 ≤ µ ≤ m , 1 ≤ ν ≤ n

L = m + 1 = constraint length

Size of memory (in bits): k ⋅ (m + 1), (minimum k ⋅ m)

Code rate:

Block structure is not of great importance:

typical values: k = 1, 2, n = 2, 3, 4, m ≤ 8

Block codes can be seen as a special case of convolutional codes with memory length m = 0

nkR =C (309)

N T SUNIVERSITÄT

Block diagram of a general (n,k,m) convolutional encoder

N T SUNIVERSITÄT

Example 1:

n = 2, k = 1, m = 2

(2,1,2) convolutional encoder

RC = 1/2

ar,1 = ur + ur−1 + ur−2

ar,2 = ur + ur−2

u = (1,1,0,1,0,0,...)

a = (11,01,01,00,10,11,...)

N T SUNIVERSITÄT

Example 2: n = 3, k = 2, m = 2

(3,2,2) convolutional encoder

RC = 2/3

ar,1 = ur,2 + ur−1,1 + ur−2,2

ar,2 = ur,1 + ur−1,1 + ur−1,2

ar,3 = ur,2

u = (11,10,00,11,01,...)

a = (111,010,010,111,001,...)

N T SUNIVERSITÄT

Restriction for the following: k = 1

Description by polynomials

Generator polynomial:

Sequence of information bits:

Sequence of code blocks:

µνµν

∑∞

rr xuxu

( ) ∑∞

0,21 )(with)(,),(),()(

rrn xaxaxaxaxax ννa

N T SUNIVERSITÄT

Convolutional encoding corresponds to multiplication of polynomials:

⇔ a(x) = u(x) ⋅ G(x)with the generator matrix:

Defining convolutional codes by multiplication of polynomials:

Memory length:

nxgxuxa ,,1for)()()( =⋅= ννν

( ) ( ))(,),(),()()(,),(),( 2121 xgxgxgxuxaxaxa nn ⋅=

( ))(,),(),()( 21 xgxgxgx n=G (316)

∈=⋅= ∑∞

=0}1,0{,)(|)()(

rr uxuxuxxu GC

)}({degreemax1

νν ≤≤

= (318)

N T SUNIVERSITÄT

Example 1:

G(x) = (g1(x),g2(x)) = (1 + x + x2, 1 + x2)

u = (1,1,0,1,0,0,...) ⇔ u(x) = 1 + x + x3

a(x) = (a1(x),a2(x)) = u(x) G(x)

= ((1 + x + x3)(1 + x + x2),(1 + x + x3)(1 + x2))

= (1 + x4 + x5,1 + x + x2 + x5)

⇔ a = (11,01,01,00,10,11,00,00,...)

N T SUNIVERSITÄT

Alternative description of encoding by convolution of vectors

gν = (g0,ν , g1,ν , g2,ν , ... , gm,ν )

gν corresponds to the impulse response of a transversal filter

Code sequence for each bit of an output block:

aν = u ∗ gν ⇔ for ν = 1, 2, ... n

with u = (u0, u1, u2, ... )

Ordering of bits:

a = (a1,0, a2,0, ... , an,0, a1,1, a2,1, ... , an,1, a1,2, a2,2, ... , an,2, ... )

− ⋅=m

ii gua0

νµµν

N T SUNIVERSITÄT

Example 1:

g1(x) = 1 + x + x2, g2(x) = 1 + x2

g1 = (111), g2 = (101)

information sequence: u = (1,1,0,1,0,0,...)

result of graphical convolution:

a1 = (1,0,0,0,1,1,0,0,...)

a2 = (1,1,1,0,0,1,0,0,...)

⇒ a = (11,01,01,00,10,11,00,00,...)

N T SUNIVERSITÄT

Graphical convolution

N T SUNIVERSITÄT

Special classes of convolutional codes Systematic convolutional codes

Information bits are copied to coded bits

Usually, non-systematic convolutional codes show better properties

N T SUNIVERSITÄT

Block diagram of a general systematic convolutional encoder

N T SUNIVERSITÄT

Terminated convolutional codes

Inserting m known blocks (tail bits – usually zeros) after Linformation blocks

Termination ⇒ block structure ⇒ block code

Coderate: nmL

kLR⋅+

)(terminatedC, (321)

N T SUNIVERSITÄT

Punctured convolutional codes (usually k = 1)

Procedure

Convolutional encoding with a mother code with rate RC = 1/n

Combination of P code blocks (P = puncturing length)

Erase l code bits corresponding to a puncturing scheme (puncturing matrix P)

Code rate:

RC,punctured < 1 ⇒ l < P ⋅ (n − k)

lnPkPR−⋅

⋅=puncturedC, (322)

N T SUNIVERSITÄT

High practical importance, especially RCPC codes (rate compatible punctured convolutional codes)

High rate codes are included in low rate codes.

Simple switching of code rates

Almost the same performance as non-punctured convolutional codes (k > 1) with the same code rate

Less computational effort for decoding if compared with non-punctured convolutional codes with k > 1

N T SUNIVERSITÄT

• Example:

u = (1,1,0,1,0,0,...)

a = (11,01,01,00,10,11,...)

apunctured = (11,0 ,01,0 ,10,1 ,...)

puncturedC, =−⋅

N T SUNIVERSITÄT

Representation of convolutional codes Shift register circuit

Generator polynomials: multiplication of polynomials, convolution

Code tree

Node = state of the memory

2k branches starting at each node

Exponential increase of nodes by each encoding step

Number of possible paths:

with Nblock = number of processed blocks

( ) blockblock 22pathsNkNkN ⋅==

N T SUNIVERSITÄT

Example for a code tree

N T SUNIVERSITÄT

State diagram

Representation without repetition (no temporal information)

Idea: output block depends only on input block and contents of the memory

New contents of memory depends on the input block and old memory content

The minimum number of encoding steps to go from state A to state B can be read from a state diagram

Number of states:( ) mkmkN ⋅== 22states (324)

N T SUNIVERSITÄT

Notation:

= ur−1 ur−2

input bits (output bits)

= ur (ar,1 ar,2)

Example 1

n = 2, k = 1, m = 2

RC = 1/2

memory contents =

N T SUNIVERSITÄT

Coding Theory Convolutional codes Example 2:

n = 3, k = 2, m = 2RC = 2/3

N T SUNIVERSITÄT

Trellis diagram

Corresponds state diagram with an additional temporal component

Basis for decoding

Trellis segment with all possible transitions of example 1

N T SUNIVERSITÄT

Trellis diagram for example 1 (encoding example)

N T SUNIVERSITÄT

Properties of convolutional codes Linearity: convolution corresponding to Eq. (319) is a linear

operation

The minimum distance is not suitable for convolutional codes −code words of convolutional codes do not exhibit a certain length

Definition of the free distance df : minimum Hamming distance between code sequences of arbitrary different information sequences

df = min {dH(a1,a2)| u1 ≠ u2} (325)

N T SUNIVERSITÄT

Linear code ⇒ distance with respect to the all-zeros sequence is sufficient for the calculation

Free distance df = minimum weight of all possible code sequences:

Optimum convolutional code: code with the maximum free distance for a given code rate and memory length

Return-to-zero path − definition: path, that leaves the all-zeros path and goes back to the all-zeros path without touching the all-zeros path in between

≠=⋅= ∑∞

=0Hf 0)(,)(|))()((min

ii xuxuxuxxuwd G (326)

N T SUNIVERSITÄT

Table of optimum convolutional codes

m df g1(x) g2(x) g3(x)2 5 1 + x + x2 1 + x2

3 6 1 + x + x3 1 + x + x2 + x3

4 7 1 + x3 + x4 1 + x + x2 + x4

5 8 1 + x2 + x4 + x5 1 + x + x2 + x3 + x5

6 10 1 + x2 + x3 + x5 + x6 1 + x + x2 + x3 + x6

2 8 1 + x + x2 1 + x2 1 + x + x2

3 10 1 + x + x3 1 + x + x2 + x3 1 + x2 + x3

4 12 1 + x2 + x4 1 + x + x3 + x4 1 + x + x2 + x3 + x4

5 13 1 + x2 + x4 + x5 1 + x + x2 + x3 + x5 1 + x3 + x4 + x5

6 15 1 + x2 + x3 + x5 + x6 1 + x + x4 + x6 1 + x + x2 + x3 + x4 + x6

N T SUNIVERSITÄT

Search for a path that leaves the all-zeros path and returns to the all-zeros path with minimum weight

= return-to-zero path with minimum weight

Procedure:

Cut the state diagram at the all-zeros state

Describing the distance with the parameter D

Shortest path from 00 to 00 :

00 → 10 → 01 → 00 ⇒ wH = 5(11) (10) (11)

N T SUNIVERSITÄT

Cutting the state diagram at the all-zeros state

N T SUNIVERSITÄT

Two paths with wH = 6

00 → 10 → 11 → 01 → 00 ⇒ wH = 6

00 → 10 → 01 → 10 → 01 → 00 ⇒ wH = 6

Generating function, transfer function:

(11) (01) (01) (11)

(11) (10) (00) (10) (11)

DDDDTkk

)2421(

+++++⋅=

N T SUNIVERSITÄT

Description of states by formal variables Zi

Equation system for the cut state diagram:

Zb = D2Za + D0Zc

Zc = D1Zb + D1Zd

Zd = D1Zb + D1Zd

Ze = D2Zc

Solving the equation system yields the generating function:

N T SUNIVERSITÄT

Development into a power series:

Result:

⇒ df = 5

1for11

1 432 <+++++=−

xxxxxx

+++++=−

⋅= 987655 16842211)( DDDDD

N T SUNIVERSITÄT

Generating function (extended version)

Introducing further formal variables into the state diagram

Variable for the distance (Hamming weight of the code word sequence): D

Variable for input ones (Hamming weight of the information word sequence): N

Variable for numbering of the encoding steps: L

Most important properties of the code can be read directly from the generating function.

N T SUNIVERSITÄT

Example for a generating function:

++⋅++++=

+⋅−==

37736735725624635

)1(1),,(

NLDNLDNLDNLDNLDNLD

LDLNNLD

ZZNLDT

N T SUNIVERSITÄT

Relation between minimum distance dmin and the number of encoding steps i: distance profile

N T SUNIVERSITÄT

Catastrophic error propagation: a limited number of transmission errors leads to an infinite number of errors in the decoder output

Criterion to detect a catastrophic encoder:

There are at least two different states that are not left if the information word does not change.

Code sequences for these states are identical.

Requirement to avoid catastrophic encoders k = 1:

Generator polynomials must not have a common divisor:

gcd(g1(x), g2(x), ... , gn(x)) = 1

N T SUNIVERSITÄT

Example for a catastrophic encoder:

g1(x) = 1 + x

g2(x) = 1 + x2 = (1 + x) ⋅ (1 + x)

N T SUNIVERSITÄT

Coding Theory Convolutional codes Decoding

Viterbi decoding

Low computational effort (can be realized up to m ≈ 10)

Optimum (maximum likelihood decoding)

Sequential decoding

Suboptimum, low memory requirement (can be realized up to m ≈10 … 20)

Maximum likelihood decoding, Viterbi metric

Terminated code sequence a of length N , received sequence r

Goal: estimation of a code sequence , which coincides as good as possible with the transmitted sequence a

N T SUNIVERSITÄT

Search for a code sequence , for which the likelihood function p(r|a) becomes maximum

Corresponds to maximize the log-likelihood function log p(r|a)

Memoryless channel:

Maximum-likelihood detection corresponds to maximize the metric

= metric increment

aa ˆ=

∑∏−

0)|(log)|(logand)|()|(

iii arpparpp arar

( ) ∑∑

=+⋅=

)|()|(log

)|(log)|(

µβα

βαar

)|( ii arµ

N T SUNIVERSITÄT

Binary symmetric channel (BSC)

Metric increment with

{ } { }1,1, 21 +−=∈ xxai

{ } { }1,1, 21 +−=∈ yyri

−=+=−

≠=−

1for1for1

forfor1

raprap

errerr

err log1and1log/2 pp

pi αβα −−=

iiii ra

−=−+=+

=1for11for1

)|(µ (329)

1−perr

x21−perr

N T SUNIVERSITÄT

Maximizing the Viterbi metric corresponds to:

minimizing the Hamming distance or

maximizing the number of coincident bits.

The product metric is also optimum for the AWGN channel.

Viterbi algorithm: decoding scheme using the trellis diagram

assigning a metric to each of the 2k ⋅m states (start value = 0)

comparing the 2k possible code sequences with the received sequence = calculation of the Viterbi metric

metric of the subsequent state = metric of the old state + metric increment

iiii raar =)|(µ

N T SUNIVERSITÄT

from all 2k arriving paths, only the path with the highest metric will be maintained – it is called survivor

all other 2k − 1 paths are erased, since they can never reach a higher metric

paths with the same metric are chosen randomly

Decoding result = path with highest metric

Practical systems: restriction to paths of length 5 ⋅ k ⋅ m bit

N T SUNIVERSITÄT

Memory requirement ≈ number of states × path length

= 2k ⋅ m ⋅ 5 ⋅ k ⋅ m bit

hard decision decoding: Viterbi decoding after taking hard decisions

soft decision decoding: Viterbi decoding directly with analog samples of the received signal (using reliability information)

Calculation of residual error probability

» simulation

» using the weight function

residual errors: error bursts

N T SUNIVERSITÄT

Example 1: (2,1,2) convolutional code

Information vector: u = (1,1,0,1,0,0,0,1,0,...)

Code sequence vector: a = (11,01,01,00,10,11,00,11,10,...)

Error vector: f = (00,10,00,00,01,00,00,00,00,...)

Received vector: r = (11,11,01,00,11,11,00,11,10,...)

Hard decision decoding

Used metric: number of coincident bits

N T SUNIVERSITÄT

Example: 1. step

N T SUNIVERSITÄT

Example: 2. step

N T SUNIVERSITÄT

Example: 3. step

N T SUNIVERSITÄT

Example: 4. step

N T SUNIVERSITÄT

Example: 5. step

N T SUNIVERSITÄT

Example: 6. step

N T SUNIVERSITÄT

Example: 7. step

N T SUNIVERSITÄT

Example: 8. step

N T SUNIVERSITÄT

Example: 9. step

N T SUNIVERSITÄT

Coding Theory Coding techniques

Concatenation of codes

Important: requirements to the outer code are very much reduced if an adapted interleaver is used

outerchannelencoder

inter-leaver

innerchannelencoder

channel

outerchanneldecoder

deinter-leaver

innerchanneldecoder

N T SUNIVERSITÄT

Interleaver: Reordering of bits so that at the output of the deinterleaver quasi-

single errors are obtained Transparent transmission Problem: delay Interleaving for bits or blocks

Delay in coded transmission systems Delay because of the block structure of encoder and decoder Delay because of interleavers Delay from the decoding process (fundamental problem for

convolutional codes)

N T SUNIVERSITÄT

interleaver

N T SUNIVERSITÄT

Coding Theory Outlook

Capacity of an AWGN channel with continuous

signal alphabet and discrete

signal alphabet

N T SUNIVERSITÄT

Coding Theory

Thanks for your attention and

much success for the examination!

nts.uni-due.dents.uni-due.de/downloads/cod/CT_SS2011.pdfnts.uni-due.de

Documents

Transcript of nts.uni-due.dents.uni-due.de/downloads/cod/CT_SS2011.pdfnts.uni-due.de

´18 - uni-due.de

Radio Propagation Channels - nts.uni-due.dents.uni-due.de/downloads/rpc/radio_pro_ch_1_122.pdf · Andreas Czylwik Radio Propagation Channels WS ... WCDMA for UMTS, John Wiley Fachgebiet

Applied Analytical Chemistry - uni-due.de

GLOBAL TRENDS ANALYSIS - uni-due.de

University of Duisburg-Essen - uni-due.de

Conference Program - uni-due.de

Radio Propagation Channels - uni-due.dents.uni-due.de/downloads/rpc/RPC_WS_2011.pdf · Andreas Czylwik Radio Propagation Channels WS 2009/2010 S. 1 Radio Propagation Channels Prof.

Sound Body, Sound Mind? - uni-due.de

Manley Absorptin Enhancement - uni-due.de

Universit at Duisburg-Essen - uni-due.de

Classroom Management - uni-due.de

Bidirektionale OFDM- bertragungsstrecke mit USRP2 …nts.uni-due.de/downloads/nt-prakt12/dokum/1408013410_Vortrag_MA.… · Bidirektionale OFDM-Übertragungsstrecke mit USRP2-Modulen

Cancellation Challenges of Networking in Technical ...nts.uni-due.de › news › EUSAS-Flyer-Grasbrunn-2013.pdf · Challenges of Networking in Technical Security Systems with a focus

„Sustainable Urban - uni-due.de

New Words New Phrases - uni-due.de

New Note - uni-due.de

Seminar III - uni-due.de

Welding of Metals beuke - uni-due.de

M TECHNIK Zahnheilkunde - uni-due.de

sensors - uni-due.de