þþþ ßßß - ee.nchu.edu.tw

3C III***��ßßßþþþ

óóó��;;;GGG (Digital Communication)

Source Coding and Channel Coding

WWW¡¡¡FFF

Communication and Coding Laboratory

Dept. of Electrical Engineering,

National Chung Hsing University

Error Correcting Codes 1

ÏÏÏÞÞÞ��---: ;;;GGG (Communication)

ÏÞ�-�b"I5�5½

• ó�;G

• é\ç�Gr�§

• 4G��;G

• Pa ½ç(WiFi)�PaKºç(WiMax)

• ;�t&ÝPa�?

CC Lab, EE, NCHU


óóó��;;;GGG (Digital Communication)

ó�;G��!�ë�I5(�P��+Û):

• ó�;G�+�

• GÙD (source coding): Huffman codes�

• ;¼D (channel coding): Hamming codes�

• x�¢�hê :

1. Simon Haykin, Communication Systems (2001)

2. John Proakis, Communication Systems Engineering (2002)

CC Lab, EE, NCHU


PART I: óóó��;;;GGG��+++

CC Lab, EE, NCHU


M-Taiwan

CC Lab, EE, NCHU


Wireless Communication

CC Lab, EE, NCHU


An overview

'&

$%

Source coding

GÙD

Information Theory

�>§¡

'&

$%

Channel coding

;¼D

'&

$%

Cryptography

ÛD.

'&

$%

Digital Communication

ó�;G

� -

?

?

CC Lab, EE, NCHU


CODING

• Coding for efficient communication over bandlimited channels

– source coding

– data compression (£]D¹)

• Coding for secure communication over wiretap channels

– cryptography

– cryptology

– cryptoanalysis

• Coding for reliable communication over noisy channels

– channel coding

– error correcting code (ý0?ÑD)

– forward error control code (ý0�×D)

CC Lab, EE, NCHU


óóó��;;;GGG��ÄÄÄ��PPP

• �½;GI*Ý��¡ÎÎ¯(volice)�£](data)TÅ

(multimedia)ÝGrFí�Km�Bãó�;Ý��¼�§�

• ó�;Ý��x�Î Ý1¹GrFLT\ã�ºb0-�

• Ý¾Õ±0-Ý8T�FíÐm¯�_D�á;¼�#[Ð

�;¼�#åGr¡�m¯¡�D

• &Æº�EGr�ãø(sampling)��;(quantization)�

• #½�&Æ¸àGÙ_D�D�;¼_D�DEó�;Gr�©

�Ýó.�§�

• GÙ_D�D: �´b§�;¼¾Õ£]D¹�

• ;¼_D�D: bÓG�;¼¾Õ£]1��

CC Lab, EE, NCHU


ãããøøø��;;;

Figure 1: (a) analog signal (b) digital signal after sampling and quan-

tization

CC Lab, EE, NCHU


• r(t) = s(t) ⊗ h(τ ; t) + n(t) (linear time-variant channel aP`�;¼)

• r(t) = s(t)⊗ h(t) + n(t) (linear time-invariant channel aP&`�;¼)

• r(t) = s(t) + n(t) (additive noise channel �WPÓG;¼)

CC Lab, EE, NCHU


• ó�;GÝÃÍÌF�ôµÎGrÝ®ß�FX�õ#[�KÎ

BãÞ×Ý�-0�1XàW�

• �Ä�3;G;¼�GrÝFX`�I5ÝGrÞº.åÕÓG

(noise) �®ß´Ë�

• �X]°: 5WÞÍI5¼�´Ë;��

1. GÙ_D (source coding)� x�Îb[2D¹¼Ù£]Wó

�Gr�|¦�FXGrÝ>�� ê�Ì £]D¹ (data

compression)�

2. ;¼_D (channel coding)� �¢ã;¼_D¼¦�(ÓG�

æ� |¾Wb[2GrFí� Í��;¼_Dê�Ì ý0

?ÑDTG'ý0�×D (forward error control code)�

CC Lab, EE, NCHU


Example: ;;;¼¼¼___DDD

• Discrete-time

– u = (u1, . . . , uk) (information vector)

– c = u · G = (c1, . . . , cn) (codeword)

– r = c + e = (r1, . . . , rn) (received vector)

– ci ∈ X = {0, 1} (input alphabet)

– ri ∈ Y = {0, 1} or Y = R (output alphabet)

• Memoryless

– p(r|c) = p(r1|c1) · · · p(rn|cn), need p(yj |xi), 1 ≤ i ≤ q, 1 ≤ j ≤ Q

CC Lab, EE, NCHU


;;;¼¼¼555vvv×××: BSC

input output

0

1

0

1

p

p

1-p

1-p

Figure 2: Binary Symmetric Channel, Þ-EÌ;¼

• Channel probability

p(1|0) = p(0|1) = p and p(0|0) = p(1|1) = 1 − p

• BSC capacity

C = 1 − H(p) = 1 + p log2p + (1 − p) log

2(1 − p)

• Decoding: Hamming distance

CC Lab, EE, NCHU


• Encoding: u ∈ F k2 = {0, 1}k → c = u · G ∈ C ⊂ Fn

2 = {0, 1}n

• Modulation in transmitter:

binary sequence =⇒ complex sequence =⇒ waveform

c = (c1, . . . , cn) ∈ C =⇒ x = (x1, . . . , xn) ∈ Cn

=⇒ s(t) = (s1(t), . . . , sn(t))

• Channel: r(t) = s(t) + e(t)

• Demodulation in receiver:

waveform =⇒ complex sequence =⇒ binary sequence

r(t) =⇒ r = (r1, . . . , rn) ∈ Cn =⇒ y = (y1, . . . , yn) ∈ Fn

2

• Hard decoding: y ∈ Fn2 → c ∈ C → u ∈ F k

2

CC Lab, EE, NCHU


;;;¼¼¼555vvvÞÞÞ: AWGN

+ c e c r + =

e

Figure 3: Additive White Gaussian Noise Channel, �Wç�{úÓG;¼

• Channel probability (e ∼ N(0, N0/2))

p(ri|ci) =1√πN0

e−(ri−ci)

2

N0

• AWGN capacity

C = W log2(1 +

Eb

N0

C

W)

• Decoding: Euclidean distance

CC Lab, EE, NCHU


• Encoding: u ∈ F k2 → c = u · G ∈ C ⊂ Fn

2

• Modulation in transmitter:

binary sequence =⇒ complex sequence =⇒ waveform

c = (c1, . . . , cn) ∈ C =⇒ x = (x1, . . . , xn) ∈ Cn

=⇒ s(t) = (s1(t), . . . , sn(t))

• Channel: r(t) = s(t) + e(t)

• Demodulation in receiver:

waveform =⇒ complex sequence

r(t) =⇒ r = (r1, . . . , rn) ∈ Cn

• Soft decoding: r ∈ Cn → c ∈ C → u ∈ F k2

• Iterative decoding: r1 ∈ Cn → r2 ∈ Cn · · · → rl ∈ Cn → u ∈ F k2

CC Lab, EE, NCHU


PART II: GGGÙÙÙ___DDD

1. Introduction to source coding

2. Huffman codes

CC Lab, EE, NCHU


• ��Ý£GA�|æ]P£];DÝ��£ÄQªà��B7

�� A��|¸à×°��">Ý]°¯£]Í�m�ÝD;

�ª±�&ÆÌ £]D¹TGrÙ_D(data compression,

source coding)�

• £]D¹�þày^ðßþ×°;D'n��A�ó�8

^�DVD(MPEG-4)�|C;G �Ù8nÝFíGH_D®¼�

9ª�FíK�àÕD¹*��

• GÙ_D��¼1�µÎ.£]��Ä�3õT3�#åÝ´Ë

�ì�t�Ä�Ý£G��¡��_D�®�

• A¢´0£]�Ý3õ(redundency)TÎ3�#å´Ë�Ý�µ

ì�tGrÝ3õÎ�µ£]Í�Ý©P��

CC Lab, EE, NCHU


(Data compression)

(Audio Compression)(Vudio Compression)

(Image Compression)

ITU-T G.7XX

IS-XXXX

GSM-XXX

MPEG-4

H.26X

JPEG-2000JPEG

(statistical)

(quantization)

(prediction

coding)

(dictionary)

(transform

coding)

(subband

coding)

(others)

Figure 4: GÙ_DT£]D¹Ý*��Tà

CC Lab, EE, NCHU


• A�×ùÅBÄó�8^mï¡�X;DÝïn�-óÎ

512 · 512 · 3 = 3 · 218 bytes.

• A�|56kbpsÝ>�FXm� 786488 ∗ 8/56K =112JÝ` �

A�BÄ¹D*�¡�ÍD¹£Î34¹�.h&Æ©�3.2JÇ�

F�%n£]�

W hy Data Com pression?

786488bytes 23116bytes,Cr=34.0

CC Lab, EE, NCHU


• Example: ¨3�'kFX×à£]ÐrGrÙAì

{A, B, A, B, A, C, A, D}

• &Æ��ËËÞ-_DAì

symbol ^£ code I code II

A 4/8 00 0

B 2/8 01 10

C 1/8 10 110

D 1/8 11 111

• code I FLC (Fix length coding), code II VLC (variable

length coding)�

CC Lab, EE, NCHU


• code I NÍsymbol¿íXmÝ�-ó

2 · 2/8 + 2 · 4/8 + 2 · 1/8 + 2 · 1/8 = 2bits

• code II NÍsymbol¿íXmÝ�-ó

1 · 4/8 + 2 · 2/8 + 3 · 1/8 + 3 · 1/8 = 1.75bits

• 3GrÙ�A��Ù��Ø°Ðr�¨^£Á{�Ø°Ð

rù��¨^£Á±�

• A��ÑGrÙÐrÝ^£5µ`� Jð�¨ÝÐràÝ_D�

-��ð�¨ÝÐr×ø�J3G>îÝ¸à£-CW�

ð�9Ì _D3õ(coding redundancy)�

CC Lab, EE, NCHU


• �ª±9Ë_Dî�Ä�Ýª´�J�¿àGrÙÝ�¨^£

�GrÙÝ_D�®: ^£�¸àKbits�^£�¸à9bits�

• code I or code IIb×Í¥�ÝP²�ôÎ�D`Xm�ÝÇÎÇ

`PÝ(instantaneous)�

• h©P3#[ÕÐr`�|ñÇ�D��m��Í�f��Ç`

PDb×Í©Ç�ÇÎ3_DÐr��^bÐrÎ¨×ÍÐrÝ

C´(prefix)�

CC Lab, EE, NCHU


0

1

0

1

0

1

S1

S2

S3

S4

Figure 5: code II WÏ%

CC Lab, EE, NCHU


��GGG£££___DDD(Huffman coding)

• �G£_D´�ã�G£31952OXè��G£_Dt ßX

áÝµÎJPEGÅ_D�

• Huffman_D�Õ°

1. 0��¨^£t�ÝËÍ;F�

2. |(1)�ñ×Íl;F�Í�J Ë�;Fõ�

3. ��(2)�Ë�;F0�1_D�

4. ¥±4��¨^£¡�(1), (2), and (3)�®�

5. ¥«M»à�_D�W�

CC Lab, EE, NCHU


• Example: ¨3�'×à£]Ðr{A, B, C, D, E}�¨Íó5½

{15, 7, 6, 6, 5}�

• ¸àFLC��m�3 · (15 + 7 + 6 + 6 + 5) = 117Íbits�

• àVLC�G£_D�_D

{A = 0, B = 100, C = 101, D = 110, E = 111}

�m�1 · 15 + 3 · (7 + 6 + 6 + 5) = 87Íbits�

CC Lab, EE, NCHU


A

E

D

C

B

0

1

0

1

0

1

0

1

Figure 6: WÏ%

CC Lab, EE, NCHU


PART III: ;;;¼¼¼___DDD

1. Introduction: Coding history and coding type

2. Coding method: Hamming codes

3. Coding research: application, practice, theory, and connection

CC Lab, EE, NCHU


Summary: Error Correcting Codes

• Origin: 1948 paper, ”A Mathematical Theory of Communication”

It is possible to transmit information over a channel of capacity C

with arbitrarily small error at a rate R if and only if R < C.

• Idea: Add redundant bits to information bits

Block codes( ðD) vs. Convolutional codes(]ID)

• Goal: Trade-off between Eb

N0(GÓf) and Pb(ý0£) with the price of

reducing R

W(�´[£) and adding encoder/decoder complexity (_D

��D �Ó�).

CC Lab, EE, NCHU


ýýý000???ÑÑÑDDDÝÝÝaaaªªª

• 1948: Information theory is born, Hamming and Golay codes

• 1948-58: {block codes, convolutional codes} ⊂ codes on algebra

• 1958-68: BCH, LDPC, concatenated codes, and algebraic decoding

• 1968-78: BCJR, Chase and Viterbi decoding

• 1978-88: {Goppa code, AG code} ⊂ codes on geometry

{TCM and BCM} ⊂ coded modulation

• 1988-98: {Turbo codes, LDPC codes} ⊂ codes on graph

{trellis, tailbiting trellis} ⊂ factor graph

• 1998-2008: Multilevel codes, space time codes, and iterative decoding

• 2008-: Network coding, Dirty paper codes, and coding application

CC Lab, EE, NCHU


��

��

�

channel coding

source coding

cryptography

��

��

�

⊂ Informatin Theory ⊂

��

�

Mathematics

Communication

��

�

CC Lab, EE, NCHU


• Golay codes (1949), Hamming codes (1950), RM codes (1954)

• linear block codes (1956), convolutional codes (1955)

CC Lab, EE, NCHU


• BCH (1959), RS (1960), LDPC (1963), concatenated codes (1966)

• BCH algebraic decoding (1960) and Viterbi decoding (1967)

CC Lab, EE, NCHU


• Goppa codes (1970), JPL bound (1977)

• Chase (1972), BCJR (1974) decoding, multilevel codes (1977)

CC Lab, EE, NCHU


• A-G codes (1982), Tanner Graph (1981), coset codes (1988)

• TCM (1982), V.32 (1984), SOVA (1989)

CC Lab, EE, NCHU


• Trellis of block codes (1993), Factor Graph (1995)

• Turbo codes (1993), iterative decoding (1993)

CC Lab, EE, NCHU


ýýý000???ÑÑÑDDDÝÝÝvvvlll

• _D �ÌbB7��ùÇNÍCD�-'� c(i) �¢ãGr

�-'� u(i) �ö�ßWÎp G(D) = G ¼®ßAì�

c(i) = u(i)G (1)

CC Lab, EE, NCHU


• ]ID�_D ÌbB7��íá£]3õD �t�;

ô` M�CD�- c(i)�¬��îAì�

c(i) = u(i)G0 + u(i − 1)G1 + · · · + u(i − M)GM (2)

CC Lab, EE, NCHU


Decoding

CC Lab, EE, NCHU


ýýý000???ÑÑÑDDDÝÝÝààà��]]]PPP

ý0?ÑD�|5 �óD(Codes on algebra)�%�D(Codes on

graph)�õ¿¢D(Codes on geometry)ë�v�

• �óD�x�ÎÚxyb§� (finite field)�N§¡ (group

theory)�8n§¡�

• %�D�x�ÎÚxy5�%�(factor graph)�õ��Õ°(sum

product algorithm)�8n§¡�

• ¿¢D�x�ÎÚxy�ó¿¢(algebraic geometry)�b§¿

¢(finite geometry) �8n§¡�

CC Lab, EE, NCHU


codes on algebra, ��óóóDDD

x7x6 x5

x3x2

x4

x1

Figure 7: A [7, 7, 1] code

• No constraint on xi and there are 27 vectors.

CC Lab, EE, NCHU


x7x6 x5

x3x2

x4

x1

Figure 8: The [7, 6, 2] parity check code

• The number of 1 in these 7 positions are even.

CC Lab, EE, NCHU


I.e., the code is the set satisfying the following:

C = {x ∈ F 72 : Hx = 0}

Hx =[

1 1 1 1 1 1 1]

x1

x2

x3

x4

x5

x6

x7

= 0

CC Lab, EE, NCHU


x7x6 x5

x3x2

x4

x1

Figure 9: A [7, 1, 7] repetition code

• The value in these 7 positions are the same.

CC Lab, EE, NCHU



C = {x ∈ F 72 : Hx = 0}

Hx =

1 1 0 0 0 0 0

1 0 1 0 0 0 0

1 0 0 1 0 0 0

1 0 0 0 1 0 0

1 0 0 0 0 1 0

1 0 0 0 0 0 1

x1

x2

x3

x4

x5

x6

x7

=

0

0

0

0

0

0

CC Lab, EE, NCHU


x7x6 x5

x3x2

x4

x1

Figure 10: A [7, 4, 3] Hamming code

• The number of 1 in each circle is even.

CC Lab, EE, NCHU



C = {x ∈ F 72 : Hx = 0}

Hx =

0 0 0 1 1 1 1

0 1 1 0 0 1 1

1 0 1 0 1 0 1

x1

x2

x3

x4

x5

x6

x7

=

0

0

0

CC Lab, EE, NCHU


Codes on graph, %%%��DDD

Example: g(x1, x2, x3, x4) = fA(x1)fB(x1, x2, x3)fC(x3, x4)fD(x3)

1

g

3 4 2

2

3 1

4

f D f C f B f A

S = {x1, x2, x3, x4} and Q = {fA, fB , fC , fD}

CC Lab, EE, NCHU


• A factor graph of a linear code C[n, k] with parity check matrix

H = 〈h1, h2, . . . , hn−k〉

• [(x1, x2, . . . , x6) ∈ C] (a complicated function)

= [x1 ⊕ x3 ⊕ x4 = 0] · [x1 ⊕ x2 ⊕ x5 = 0] · [x2 ⊕ x3 ⊕ x6 = 0]

CC Lab, EE, NCHU


• A factor graph of a trellis of a linear code C[n, k]

T (C) = [(s0, c1, s1) ∈ T1][(s1, c2, s2) ∈ T2] · · · [(sn−1, cn−1, sn) ∈ Tn]

CC Lab, EE, NCHU


• A factor graph of the turbo code and its decoding

CC Lab, EE, NCHU


Coding Research

CC Lab, EE, NCHU


ÞÞÞ000

1. ¶��>§¡X®ß�ëËDCÍ¸à�êÝ�

2. BSC;¼��à��

3. AWGN;¼��à��

4. ¸àHuffman_D�æ§�

5. [7, 4, 3] HammingD�_D��D�

6. ;¼D�ÞË5v ¢�

7. ;¼D�ëËà�]° ¢�

CC Lab, EE, NCHU

þþþ ßßß - ee.nchu.edu.tw

Documents

Transcript of þþþ ßßß - ee.nchu.edu.tw