An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory...
Transcript of An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory...
![Page 1: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/1.jpg)
An Introduction toAlgorithmic Coding Theory
M. Amin ShokrollahiBell Laboratories
1
![Page 2: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/2.jpg)
Part 1: Codes
1-1
![Page 3: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/3.jpg)
A puzzle
What do the following problems have in common?
2
![Page 4: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/4.jpg)
Problem 1: Information Transmission
MASSAGE ??MESSAGE
A
HB
I
J
C
KL
D
M
E
N
F
G
X
YZ
UV
W
T
R
S
3
![Page 5: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/5.jpg)
Problem 2: Football Pool Problem
Bayern Munchen : 1. FC Kaiserslautern 0
Homburg : Hannover 96 1
1860 Munchen : FC Karlsruhe 2
1. FC Koln : Hertha BSC 0
Wolfsburg : Stuttgart 0
Bremen : Unterhaching 2
Frankfurt : Rostock 0
Bielefeld : Duisburg 1
Feiburg : Schalke 04 2
4
![Page 6: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/6.jpg)
Problem 3: In-Memory Database Systems
malloc
malloc
malloc
malloc
User memory
Code memory
Database
Corrupted!
5
![Page 7: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/7.jpg)
Problem 4: Bulk Data Distribution
������������������������
������������������������
������������������
������������������
����������������������������������������
����������������������������������������
���������������������
���������������������
���������������������
���������������������
6
![Page 8: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/8.jpg)
What do they have in common?
They all can be solved using
(algorithmic) coding theory
7
![Page 9: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/9.jpg)
Basic Idea of Coding
Adding redundancy to be able to correct!
MESSAGEMASSAGEMESSAGEMESSAGE
Objectives:
• Add as little redundancy as possible;
• correct as many errors as possible.
8
![Page 10: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/10.jpg)
Codes
A code of block-length n over the alphabet GF (q) is a set of
vectors in GF (q)n.
If the set forms a vector space over GF (q), then the code is called
linear.
[n, k]q-code
9
![Page 11: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/11.jpg)
Encoding Problem
source
??
?
?
?
?
Codewords
Efficiency!
10
![Page 12: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/12.jpg)
Linear Codes: Generator Matrix
k nx
k n
source codeword
Generator matrix
O(n2)
11
![Page 13: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/13.jpg)
Linear Codes: Parity Check Matrix
Parity check matrix
codeword
0
O(n2) after O(n3) preprocessing.
12
![Page 14: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/14.jpg)
The Decoding Problem: Maximum LikelihoodDecoding
a b c d 7→ a b c d | a b c d | a b c d
Received: a x c d | a b z d | a b c d.
a x c d a b z d a b c d
a x c d a x c d a x c d 3
a b z d a b z d a b z d 3
a b c d a b c d a b c d 2
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ≥ 3
13
![Page 15: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/15.jpg)
Hamming Distance
Hamming distance between two vectors of equal dimension is
number of positions at which they differ.
[n, k, d]q-code
14
![Page 16: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/16.jpg)
Maximum Likelihood Decoding
Given received word, find a codeword that has least Hamming
distance to it.
Intractable in general.
15
![Page 17: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/17.jpg)
Worst Case Error Correction
[n, k, d]q-code is capable of correcting up to e = (d− 1)/2 errors in
whatever locations!
Hamming balls of
radius e
e
d
16
![Page 18: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/18.jpg)
Errors in Known Locations: Erasures
[n, k, d]q-code is capable of correcting up to d− 1 errors in whatever
locations if the locations are known.
17
![Page 19: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/19.jpg)
Projective plane over GF (2)
1001001
1100010
1010100
0110001
0011010
0101100
0000111
0123456
0
1 2
3
4 5
6
Minimum distance = 4
18
![Page 20: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/20.jpg)
Hamming Code
0
1 2
3
4 5
6
7,4,3][ -code2
0110001
0011010
0101100
1010100
1100010
1001001
1
010101
0110110
0110110
0101011
00
1101
1
1010011
1100101
0000111
0000000
1111111
19
![Page 21: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/21.jpg)
A Solution to the Football Match Problem
[4, 2, 3]3-Hamming Code with generator matrix:
0 1
1 0
1 1
1 2
Codewords:
0 0 1 2 1 2 2 1 0
2 1 0 0 1 2 1 2 0
2 1 1 2 2 1 0 0 0
2 1 2 1 0 0 2 1 0
20
![Page 22: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/22.jpg)
A Solution to the Football Match Problem
This Hamming code is perfect: Hamming balls of radius 1 fill the
space GF (3)4:
32 ·1
∑
i=0
(
4
i
)
2i = 9 · (1 + 4 · 2) = 81.
Any vector in GF (3)4 has Hamming distance at most 1 to a
codeword.
Fix
ed g
ame
Hamming code Hamming code
21
![Page 23: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/23.jpg)
Bounds
How can we prove optimality of codes? (Fix any two of the three
parameters n, k, d and maximize the third.)
1. Hamming bound:∑e
i=1
(
ni
)
(q − 1)i ≤ qn−k.
Equality: perfect codes
2. Plotkin bound: d+ k ≤ n+ 1.
Equality: MDS codes.
3. Other more refined bounds....
22
![Page 24: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/24.jpg)
Perfect Codes
Have been completely classified by van Lint and Tieatvainen.
Essentially: Hamming codes, Golay codes.
Not desirable in communication scenarios.
23
![Page 25: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/25.jpg)
MDS Codes
Not classified completely.
Open problem: Given dimension k and field size q, determine
maximum block length of an MDS-code. (Conjecturally q + 1 or
q + 2.)
MDS codes are desirable in practice in worst case scenarios if
efficient encoding and decoding available.
Prototype: Reed-Solomon codes.
24
![Page 26: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/26.jpg)
Reed-Solomon Codes: Applications
1. Satellite Communication,
2. Hard disks,
3. Compact Discs, Digital Versatile Disks, Digital Audio Tapes,
4. Wireless Communication,
5. Secret sharing,
6. ...
25
![Page 27: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/27.jpg)
Reed-Solomon Codes: Definitions
Choose n different elements x1, . . . , xn in GF (q).
Reed-Solomon code is image of the morphism
GF (q)[x]<k 7→ GF (q)n
f 7→ (f(x1), . . . , f(xn))
Block length = n.
Dimension? Minimum distance?
26
![Page 28: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/28.jpg)
Reed-Solomon Codes: Parameters
Theorem. Nonzero polynomial of degree m over a field can have
at most n zeros over that field.
27
![Page 29: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/29.jpg)
Reed-Solomon Codes: Dimension
GF (q)[x]<k 7→ GF (q)n
f 7→ (f(x1), . . . , f(xn))
Kernel: 0 if k ≤ n since nonzero polynomial of degree ≤ k − 1 has
at most k − 1 zeros.
Dimension: k (if k ≤ n).
28
![Page 30: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/30.jpg)
Reed-Solomon Codes: Minimum Distance
GF (q)[x]<k 7→ GF (q)n
f 7→ (f(x1), . . . , f(xn))
Minimum distance: Maximal number of zeros in a nonzero
codeword is k − 1, since evaluating polynomial of degree ≤ k − 1.
Minimum distance is ≥ n− k + 1, hence equal, hence MDS code!.
29
![Page 31: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/31.jpg)
Reed-Solomon Codes: Encoding
f 7→ (f(x1), . . . , f(xn)) is easy to compute!
O(n2) using naive algorithm.
O(n log2(n) log log(n) using fast algorithms.
30
![Page 32: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/32.jpg)
Reed-Solomon Codes: Decoding
No efficient maximum likelihood decoding known.
Concentrate on bounded distance decoding.
Transmitted word
Received word
Number of agreements ≥ (n+ k)/2.
Number of disagreements ≤ (n− k)/2.
31
![Page 33: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/33.jpg)
Welch-Berlekamp Algorithm
Transmitted word: (f(x1), . . . , f(xn)).
Received word: (y1, . . . , yn).
Number of agreements ≥ (n+ k)/2.
Find f !
32
![Page 34: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/34.jpg)
Welch-Berlekamp Algorithm
Step 1: Find g(x) ∈ GF (q)[x]<(n+k)/2 and
h(x) ∈ GF (q)[x]≤(n−k)/2, not both zero, such that
∀ i = 1, . . . , n: g(xi) + yih(xi) = 0.
(Solving a system of equations!)
Step 2: Then f = g/h!
33
![Page 35: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/35.jpg)
Welch-Berlekamp Algorithm: Proof
H(x) := g(x)− f(x)h(x).
Degree of H(x) < (n+ k)/2.
If yi = f(xi), then H(xi) = 0.
H(x) has at least (n+ k)/2 zeros.
H(x) is zero.
f(x) = g(x)/h(x).
34
![Page 36: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/36.jpg)
Welch-Berlekamp Algorithm: Running time
Step 1: Solving a homogeneous n× (n+ 1) system of equations;
O(n3).
Can be reduced to O(n2) (Welch-Berlekamp, 1983; displacement
method (Olshevsky-Shokrollahi, 1999)).
Step 2: Polynomial division; O(n2).
35
![Page 37: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/37.jpg)
Welch-Berlekamp Algorithm: Generalization
Has been generalized to more than (n− k)/2 errors (list-decoding,
Sudan, 1997, Guruswami-Sudan, 1999).
Step 2 requires factorization of bivariate polynomials.
Can be done more efficiently (Gao-Shokrollahi 1999,
Olshevsky-Shokrollahi 1999).
36
![Page 38: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/38.jpg)
A Solution to the In-Memory Database Problem
malloc
malloc
malloc
malloc
User memory
Code memory
Database Redundant
37
![Page 39: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/39.jpg)
Reed-Solomon Codes: Generalization
Disadvantage of RS-codes: GF (q) must be large to accommodate
many points, so long codes impossible.
Interpret GF (q) as affine line over itself, and generalize to more
complicated algebraic curves.
Lead to best known codes in terms of minimum distance,
dimension, block-length.
Above algorithms can be generalized to these Algebraic-geometric
codes.
38
![Page 40: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/40.jpg)
Probabilistic Methods
38-1
![Page 41: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/41.jpg)
Channels
0.2
0.30.5
0.90.1
Inpu
t alp
habe
t Output alphabet
Transition probabilities
0.5
0.5
39
![Page 42: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/42.jpg)
Entropy and Mutual Information
X and Y discrete random variables on alphabets X and Y and
distributions p(x) and q(x). p(x, y) their joint distribution.
Entropy H(X) of X
H(X) = −∑
x∈X
p(x)log p(x).
Mutual information I(X ;Y )
I(X ;Y ) = −∑
y∈Y
∑
x∈X
p(x, y)logp(x, y)
p(x)p(y).
40
![Page 43: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/43.jpg)
Entropy and Mutual Information
• H(X) is the amount of uncertainty of random variable X .
• I(X ;Y ) is the reduction in the uncertainty of X due to the
knowledge of Y .
41
![Page 44: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/44.jpg)
Capacity
Capacity of a channel with input alphabet X and output alphabet
Y and probability transition matrix p(y | x) is
C = maxp(x)
I(X ;Y ),
where maximum is over all possible input distributions p(x).
42
![Page 45: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/45.jpg)
Examples of Capacity: BEC
Binary Erasure Channel:
p
1-p
1-p0 0
11
E
p
Capacity = 1− p
43
![Page 46: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/46.jpg)
Examples of Capacity: BSC
Binary Symmetric Channel:
1-p0
1 1
0
p
p
1-p
Capacity = 1 + p log2(p) + (1− p) log2(1− p)
44
![Page 47: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/47.jpg)
Capacity and Communication
Shannon’s Coding Theorem, 1948:
C channel with capacity C. For any rate R ≤ C there exists a
sequence of codes of rate R such that the probability of error of the
Maximum Likelihood Decoding for these codes approaches zero as
the block-length approaches infinity.
The condition R ≤ C is necessary and sufficient.
45
![Page 48: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/48.jpg)
Problems
• How to find the sequences of codes?
(Random codes, Concatenated codes, · · ·)
• How to decode efficiently?
Has been open for almost 50 years.
Low-Density Parity-Check Codes
46
![Page 49: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/49.jpg)
Part 2:
Low-Density Parity-Check Codes
46-1
![Page 50: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/50.jpg)
Low-Density Parity Check Codes
Gallager 1963
Zyablov 1971
Zyablov-Pinsker 1976
Tanner 1981
Turbo Codes 1993
Berroux-Glavieux-Thitimajshima
47
![Page 51: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/51.jpg)
Sipser-Spielman, Spielman 1995
MacKay-Neal, MacKay 1995
Luby-Mitzenmacher-S-Spielman-Stemann 1997
Luby-Mitzenmacher-S-Spielman 1998
Richardson-Urbanke 1999
Richardson-Shokrollahi-Urbanke 1999
48
![Page 52: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/52.jpg)
Code Construction
Codes are constructed from sparse bipartite graphs.
49
![Page 53: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/53.jpg)
Code Construction
Any binary linear code has a graphical representation.
f
c d e
a c e
c
a
b
c
d
b
a
f
e
0!=
=! 0
=! 0
Not any code can be represented by a sparse graph.
50
![Page 54: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/54.jpg)
Parameters
Rate n-rn
average left degreeaverage right degree
1-Raten
r
51
![Page 55: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/55.jpg)
Dual Construction
a
b
c
d
e
f
g
h
a b f
a b c g
c e g h
b d e f g h
Sour
ce b
its
Redundant bits
Encoding time is proportional to number of edges.
52
![Page 56: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/56.jpg)
Algorithmic Issues
• Encoding?
– Is linear time for the dual construction
– Is quadratic time (after preprocessing) for the Gallager
construction. More later!
• Decoding?
– Depends on the channel,
– Depends on the fraction of errors.
53
![Page 57: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/57.jpg)
Decoding on a BSC: Flipping
1
1
unsatisfied check
satisfied check
0
1
1
1
0
54
![Page 58: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/58.jpg)
Decoding on a BSC:Gallager Algorithm A(Message passing)
x=y=z=ux
b else
ifm =
b
xy
z
u
m
m=x y z u
x
y
z
u
m
MESSAGE CHECK
55
![Page 59: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/59.jpg)
Decoding on a BSC:Belief Propagation
x+y+z+u+bm = m=x * y z u* *
a,b( )*(c,d):=(a+c, b+d mod 2)
x
y
z
u
m
MESSAGE CHECK
b
xy
z
u
m
hyperbolic
transform
Messages in log-likelihood ratios.
56
![Page 60: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/60.jpg)
Optimality of Belief Propagation
Belief propagation is bit-optimal if graph has no loops.
Maximizes the probability
P (cm = b | y) =∑
c∈C
P (c | y).
57
![Page 61: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/61.jpg)
Performance on a (3,6)-graph
Shannon limit: 11%
Flipping algorithm: 1%?
Gallager A: 4%
Gallager B: 4% (6.27%)
Erasure decoder: 7%
Belief propagation: 8.7% (10.8%)
58
![Page 62: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/62.jpg)
The Binary Erasure Channel (BEC)
p
1-p
1-p0 0
11
E
p
59
![Page 63: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/63.jpg)
Decoding on a BEC:Luby-Mitzenmacher-Shokrollahi-Spielman-
Stemann
elsem =
if x1 y z u =1
0
x
y
z
u
m
MESSAGE
y
z
u
m
x
else
if x y z um =
= = = =11
0
CHECK
60
![Page 64: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/64.jpg)
Decoding on a BEC
Phase 1: Direct recovery
db
f
g
c
a
h
b d e
b
e
?
?
?
61
![Page 65: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/65.jpg)
Decoding on a BEC
Phase 2: Substitution
d ef
g
c
a
h
b
e
?
?
b
d
62
![Page 66: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/66.jpg)
Example
(a) (b) (c)
(d) (e) (f)
Complete Recovery
63
![Page 67: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/67.jpg)
The (inverse) problem
Have: fast decoding algorithms.
Want: design codes that can correct many errors using these
algorithms.
Focus on the BEC in the following.
64
![Page 68: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/68.jpg)
Experiments
Choose regular graphs.
An (d, k)-regular graph has rate at least 1− d/k. Can correct at
most an d/k-fraction of erasures.
Choose a random (d, k)-graph.
p0 := maximum fraction of erasures the algorithm can correct.
d k d/k p0
3 6 0.5 0.429
4 8 0.5 0.383
5 10 0.5 0.341
3 9 0.33 0.282
4 12 0.33 0.2572
What are these numbers?
65
![Page 69: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/69.jpg)
A Theorem
Luby, Mitzenmacher, Shokrollahi, Spielman, Stemann, 1997:
A randomly chosen (d, k)-graph can correct a p0-fraction of
erasures with high probability if and only if
p0 · (1− (1− x)k−1)d−1 < x for x ∈ (0, p0).
66
![Page 70: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/70.jpg)
67
![Page 71: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/71.jpg)
Analysis: (3, 6)-graphs
Expand neighborhoods of message nodes.
68
![Page 72: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/72.jpg)
Analysis: (3, 6)-graphs
pi probability that message node is still erased after ith iteration.
pi+1
pi
Message
Check
Message
pi+1 = p0(1− (1− pi)5)2.
69
![Page 73: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/73.jpg)
Successful Decoding
Condition:
p0(1− (1− pi)5)2<pi
70
![Page 74: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/74.jpg)
Analysis: (3, 6)-graphs
Making arguments exact:
• Neighborhood is tree-like: high probability, standard argument.
• Above argument works for expected fraction of erasures at ℓth
round.
Real value is sharply concentrated around expected value pℓ:
Edge exposure martingale, Azuma’s inequality.
71
![Page 75: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/75.jpg)
The General Case
Let λi and ρi be the fraction of edges of degree i on the left and the
right hand side, respectively.
Let λ(x) :=∑
i λixi−1 and ρ(x) :=
∑
i ρixi−1.
Condition for successful decoding for erasure probability p0 is then
p0λ (1− ρ(1− x)) < x
for all x ∈ (0, p0).
72
![Page 76: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/76.jpg)
Belief propagation
Richardson-Urbanke, 1999:
fℓ: density of the probability distribution of the messages passed
from the check nodes to the message nodes at round ℓ of the
algorithm.
P0: density of the error distribution (in log-likelihood
representation).
Consider (d, k) regular graph.
Γ (fℓ+1) =(
Γ(
P0 ⊗ fℓ⊗(k−1)
))⊗(d−1)
,
where Γ is a hyperbolic change of measure function,
Γ(f)(y) := f(ln coth y/2)/sinh(y),
and ⊗ denotes convolution.
73
![Page 77: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/77.jpg)
We want fℓ to converge to a Delta function at ∞.
Gives rise to high-dimensional optimization algorithms.
74
![Page 78: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/78.jpg)
Achieving capacity
Want to design codes that can recover from a fraction of 1−R of
erasures (asymptotically).
Want to have λ and ρ so that
p0λ(1− ρ(1− x)) < x
for all x ∈ (0, p0), and p0 arbitrarily close to
1−R =
∫ 1
0
ρ(x)dx
∫ 1
0
λ(x)dx
.
75
![Page 79: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/79.jpg)
Tornado codes
Extremely irregular graphs provide for any rate R sequences of
codes which come arbitrarily close to the capacity of the erasure
channel!
Degree structure?
Choose design parameter D.
λ(x) :=1
H(D)
(
x+x2
2+ · · ·+
xD
D
)
ρ(x) := exp (µ(x− 1)),
where H(D) = 1+1/2+ · · ·+1/D and µ = H(D)/ (1− 1/(D + 1)).
76
![Page 80: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/80.jpg)
Tornado Codes: Left Degree Distribution
77
![Page 81: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/81.jpg)
Right regular codes
Shokrollahi, 1999:
Graphs that are regular on the right.
Degrees on the left are related to the Taylor expansion of
(1− x)1/m.
These are the only known examples of LDPC codes that achieve
capacity on a nontrivial channel using a linear time decoding
algorithm.
78
![Page 82: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/82.jpg)
Other channels?
f density function.
λ(f) :=∑
i λif⊗(i−1).
ρ(f) :=∑
i ρif⊗(i−1).
Γ (fℓ+1) = ρ (Γ (P0 ⊗ λ(fℓ))).
Want P0 such that fℓ → ∆∞.
79
![Page 83: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/83.jpg)
Conditions on the density functions
Richardson-Shokrollahi-Urbanke, 1999:
• Consistency: if the channel is ”symmetric”, then the density
functions fℓ satisfy f(x) = f(−x)ex.
• Fixed point theorem: If Perr(fi) = Perr(fj) for i < j, then
fi = fj is a fixed point of the iteration.
80
![Page 84: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/84.jpg)
Conditions on the density functions
• Stability: let
r := − limn→∞
1
nlogPerr(P
⊗n0 ).
Then for λ2ρ′(1) > er we have Perr(fℓ) > ǫ for some fixed ǫ and
all ℓ.
If λ2ρ′(1) < er, then the fixed point ∆∞ is stable.
Perr(f) :=
∫ 0
−∞
f(x)dx
is the error probability.
81
![Page 85: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/85.jpg)
Stability
• Erasure channel with erasure probability p0:
λ2ρ′(1) ≤
1
p0.
• BSC channel: with probability p:
λ2ρ′(1) ≤
1
2√
p(1− p).
• AWGN channel: with variance σ2:
λ2ρ′(1) ≤ e−
1
2σ2 .
82
![Page 86: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/86.jpg)
Stability for the Erasure Channel
Shokrollahi, 1999:
λ(1−ρ(1− ))p0 x - x
λ(1−ρ(1− ))p0
x - x
not stablestable
83
![Page 87: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/87.jpg)
Flatness: Higher Stability Conditions
Shokrollahi, 2000:
(λm(x), ρm(x)) capacity achieving sequence of degree distributions.
Then:
(1− R)λm(1− ρm(1− x))− x
converges uniformly to the zero-function on the interval [0, 1−R].
No equivalent known for other channels.
84
![Page 88: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/88.jpg)
Flatness: Higher Stability Conditions
λ(1−ρ(1− )) − p0
x x
85
![Page 89: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/89.jpg)
Capacity achieving
No sequences of c.a. degree distributions for channels other than
the erasure channel known.
Conjecture: They exist!
0.0 0.2 0.4 0.6 0.8 1.0 1.210-6
10-5
10-4
10-3
10-2
Eb/N0 [dB]
0.159 0.153 0.147 0.142 0.136 0.131 0.125 0.12 Pb
1.0 0.977 0.955 0.933 0.912 0.891 0.871 0.851 σ
Shannon Lim
it
Turbo C
ode; n=10
6
(3, 6)-regular LDP
CC
; n=10
6
Threshold
irregular LDP
CC
; n=10
6
Threshold
86
![Page 90: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/90.jpg)
Applications to computer networks
Distribution of bulk data to a large number of clients.
Want
• fully reliable,
• low network overhead,
• support vast number of receivers with heterogeneous
characteristics
• users want to access data at times of their choosing and these
access times overlap.
87
![Page 91: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/91.jpg)
A Solution
Broadcast
Server
88
![Page 92: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/92.jpg)
A Solution
Client joins multicast group until enough of the encoding has been
received, and then decodes to obtain original data.
Time
Am
ount
of e
ncod
ing
rece
ived
Digital Fountain, http://www.dfountain.com.
89
![Page 93: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/93.jpg)
Open problems
Asymptotic theory
1. Classification of capacity achieving sequences for the erasure
channel.
2. Capacity achieving sequences for other channels.
3. Exponentially small error probabilities for the decoder (instead
of polynomially small).
Explicit constructions
1. Constructions using finite geometries.
2. Construction using Reed-Solomon-Codes.
3. Algebraic constructions.
Short codes
Graphs with loops.
90
![Page 94: An Introduction to Algorithmic Coding Theory · An Introduction to Algorithmic Coding Theory M.AminShokrollahi Bell Laboratories 1. Part 1: Codes 1-1. A puzzle What do the following](https://reader030.fdocuments.in/reader030/viewer/2022040215/5edc9124ad6a402d666748ce/html5/thumbnails/94.jpg)
Algorithmic issues
1. Design and analysis of new decoding algorithms.
2. Design of new encoders.
Applications
Packet based wireless networks.
Randomness
Use of randomness in other areas: random convolutional codes?.
91