7/29/2019 EC 515 Introduction
1/37
Introduction to Channel Coding
Introduction to Algebra
EC 515: Coding Theory
7/29/2019 EC 515 Introduction
2/37
Channel coding theorem
Let a DMS with an alphabetXhave entropy H(X) and
produce symbols once every Ts sec. Let a DMC have
capacity Cand be used once every Tcsec. Then, if
there exists a coding scheme for which the source
output can be transmitted over the channel and bereconstructed with an arbitrarily small prob. of error.
csTC
TXH )(
7/29/2019 EC 515 Introduction
3/37
Code rate
The parameterC/ Tc is called the critical rate. When
the above expression is satisfied with the equality
sign, the system is said to be signaling at critical rate.
Code rate: a block ofkbits mapped to n bits byadding (n k) redundant bits. Then code rate is given
as r= k/ n
Example: The source generates bits 0 or 1 with
equal probability, one bit in Ts sec. H(X) = 1
The channel is binary symmetric channel with bit-
error probabilityp, channel carrying one bit in Tcsec
7/29/2019 EC 515 Introduction
4/37
Code rate
Channel capacity
C= 1 +plog2p+ (1 p) log2(1 p) = 1 H(p)
For every 1/Ts bits generated per sec, 1/Tcbits are
transmitted over the channel per sec
Code rate r= Tc / Ts .
Then for reliable transmission we require r C.
7/29/2019 EC 515 Introduction
5/37
Channel coding basics
While source coding reduces redundancy, channelcoding introduces redundancy.
Idea is to increase Hamming distance betweenevery pair of codes; If min. dist. among all pairs of
codes is dmin and the max. possible bit-error is t, then
Error can be detected if
Error can also be corrected if
One very trivial form of channel coding is repetitivecoding.
1min dt
2
1mindt
7/29/2019 EC 515 Introduction
6/37
Groups
A set of objects G, along with a binary operation * onthe elements of the set, is called a group if thefollowing properties are satisfied:
Closure: G is closed under *
Associativity
Guaranteed existence of an identity element.
Existence of an inverse element for every
element.
The group is denoted by {G, * }.
7/29/2019 EC 515 Introduction
7/37
Groups
Theorem: The identity element in a group G is
unique.
Theorem: The inverse of a group element is unique.
Orderof a group = number of elements
Finite group = finite order group
Abelian group = if the operation on the set elements
is commutative.
7/29/2019 EC 515 Introduction
8/37
Groupsexamples
Set of all integers under real addition (abelian)
Set of all rational numbers excluding 0 under real
multiplication (abelian)
Set of binary numbers 0 and 1 under binary operationXOR (finite, abelian)
Additive group: finite and abelian
(a + b) mod m, where G = {0, 1, 2, ., m1)
Multiplicative group: finite and abelian
(a b) mod p, where G = {1, 2, ., p1), p is prime
7/29/2019 EC 515 Introduction
9/37
Subgroups and cosets
Subgroup: subset H of G if closed under the groupoperation * of G and satisfies all conditions of agroup.
Left coset: a * H = {a * h}, where a is an element of
G and h is an element of H
Right coset: H * a
For commutative group, left coset = right coset
Theorem: For any element h in H, its inverse hisalso in H.
7/29/2019 EC 515 Introduction
10/37
Subgroups and cosets
Theorem: No two elements in a coset of H are
identical.
Theorem: No two elements in two different (distinct)
cosets of H are identical. Every element in G appears in one and only one
coset of H.
All the distinct cosets of H are disjoint.
The union of all the distinct cosets of H forms the
group G.
7/29/2019 EC 515 Introduction
11/37
Fields
A set F together with two binary operations addition(denoted by +) and multiplication (denoted by ) isa field if
The set F is a commutative group under addition.
Identity element w.r.t. addition is called additiveidentity or zero element (denoted by 0)
The set F minus the zero element is acommutative group under multiplication.
Identity element w.r.t. multiplication is calledmultiplicative identity or unit element (denoted by1)
Multiplication is distributive over addition.
7/29/2019 EC 515 Introduction
12/37
Fields
Subtracting one field element b from another field
element a = adding additive inverse of b to a
Similarly, dividing a by b = multiplying a with
multiplicative inverse of b Note, addition, multiplication, subtraction and division
here are field operations and need not be same as
ordinary addition, multiplication, subtraction and
division. Order of field = number of elements in F.
Finite order field is known as Galois Field.
7/29/2019 EC 515 Introduction
13/37
Field properties
a 0 = 0 a = 0
a b 0, for any two non-zero elements a, b
a b = 0 and a 0 implies b = 0
(a b) = (a) b = a (b)
a b = a c implies b = c, if a 0
7/29/2019 EC 515 Introduction
14/37
Examples of Galois field
Set {0, 1} under mod-2 addition (XOR) and
multiplication (AND)
Called binary field denoted by GF(2)
{0, 1, .., p1} under mod-p addition and mod-pmultiplication, where p is prime.
Called prime field denoted by GF(p)
Extension field: prime field GF(p) extended to a fieldof pm elements, denoted as GF(pm), m is pos. integer
The order of any finite field is a power of a prime.
7/29/2019 EC 515 Introduction
15/37
Binary field arithmetic
We are concerned with binary field GF(2) or its
extension in digital system using binary coding.
Binary mod-2 addition mod-2 subtraction.
Solving equations in binary arithmetic follows thesame rule as in ordinary arithmetic; Cramers rule is
applicable.
Polynomialover GF(2) with one variable and
coefficients 0 or 1:
f(X) = a0 + a1X + ..+ anXn
7/29/2019 EC 515 Introduction
16/37
Binary field arithmetic
Degree of polynomial = largest power of X with non-
zero coefficeint, i.e. degree of f(X) is n if an 0.
We can have 2n polynomials with degree n.
Polynomials over GF(2) can be added, subtracted,multiplied and divided in the usual way where
multiplication and addition of coefficients are mod-2.
Addition and multiplication of polynomials are
commutative and associative.
Multiplication is distributive over addition.
Euclids division algorithm: f(X) = q(X).g(X) + r(X)
7/29/2019 EC 515 Introduction
17/37
Binary field arithmetic
f(X) divisible by (Xa) (X+a) if a is a root of f(X), i.e.f(a) = 0.
It follows, f(X) divisible by (X+1) if it has an evennumber of terms.
Irreducible polynomial = polynomial p(X) of degreem not divisible by any polynomial of degree less thanm but greater than 0.
For any m 1, there exists at least one irreduciblepolynomial of degree m.
Theorem: Any irreducible polynomial p(X) ofdegree m always divides Xn + 1, where n = 2m 1.
7/29/2019 EC 515 Introduction
18/37
Binary field arithmetic
Primitive polynomial: when p(X) does not divide
any other polynomial of the form XK + 1, where K < n
= 2m 1.
All irreducible polynomials are not primitivepolynomials.
Example: X4 + X + 1 and X4 + X3 + X2 + X + 1 are
both irreducible polynomial of degree 4, but only
the first polynomial is also primitive polynomial(the second polynomial can divide X5 + 1)
f2(X) = f(X2) [f(X)]2k = f(X2k), k 0.
7/29/2019 EC 515 Introduction
19/37
Construction of GF(2m)
Choose a primitive polynomial p(X) over GF(2) oforder m > 1. (Why only a primitive polynomial?)
Determine the root of p(X).
Let, be the root of p(X) p() = 0 and 0,1since p(X) not divisible by X or (X+1).
Then, F = {0, 1, , 2, , n1} forms a Galois fieldof order 2m= n + 1 under binary field addition + andmultiplication
Multiplication of two non-zero elements of F givesi j = i+j (for i, j 0, where 0 = 1), which iscontained in F (in case i + j n, check that n = 1)
7/29/2019 EC 515 Introduction
20/37
Construction of GF(2m)
Divide every element of F by p() and obtain theremainder.
Dividing 0 gives remainder 0.
Dividing i
(0 i < m) gives non-zero remainderri() =
i since 0.
Dividing i (m i < n) gives remainder ri() whichis a non-zero polynomial of degree less than m.
Check that
All ri(), for 0 i < n, are distinct.
i = ri() for 0 i < n
7/29/2019 EC 515 Introduction
21/37
Construction of GF(2m)
Thus, F = {0, 1, , 2, , m1, rm(),, rn1()} is
the Galois field of order 2m.
Check that
Elements of F are distinct polynomials (including0) of degree less than m.
Number of non-zero polynomials of degree less
than m is n = 2m 1.
Therefore, adding any two elements of F gives a
polynomial (including 0) of degree less than m
which is also contained in F.
7/29/2019 EC 515 Introduction
22/37
Construction of GF(2m)
Thus, GF(2m) may be represented in following ways:
Power representation
F = {0, 1, , 2, , n1}
Polynomial representation
F = {0, 1, , 2, , m1, rm(),, rn1()}
m-tuple representation: (ai,0 ai,1.. ai,m1) where
ai,k are the coefficients of the polynomial ri()
Example: Construct GF(24) from the generator
polynomial p(X) = 1 + X + X4
7/29/2019 EC 515 Introduction
23/37
Basic properties of GF(2m)
If an element in GF(2m) is root of f(X), then P, P =2l, l 0 is also root of f(X).
All n = 2m 1 non-zero elements of GF(2m) form allthe roots of Xn + 1.
All elements of GF(2m), including 0, form all theroots of Xn+1 + X.
Minimal polynomial(X) of an element in GF(2m)is the polynomial of smallest degree whose root is ,i.e. () = 0.
Min. polynomial for 0 and 1 are X and X + 1,respectively.
7/29/2019 EC 515 Introduction
24/37
Basic properties of GF(2m)
Minimal polynomial (X) of an element in GF(2m) isirreducible.
If element in GF(2m) is root of f(X), then (X)divides f(X).
It follows, (X) divides Xn+1 + X, n = 2m 1.
If element in GF(2m) is root of an irreduciblepolynomial f(X), then the minimal polynomial of is(X) = f(X).
Let, e be the smallest non-negative integer for anelement in GF(2m), so that P = , P = 2e then thepolynomial f(X) given below is irreducible.
7/29/2019 EC 515 Introduction
25/37
Basic properties of GF(2m)
The min. polynomial (X) for the element is givenby the polynomial f(X) above.
If the degree of the minimal polynomial (X) for anelement in GF(2m) is e then e m, and P = ,where P = 2e
The element P, P = 2l is called a conjugate of.
2, , P, P = 2e1 form distinct conjugates of.
All conjugates of are all the roots of(X).
Check that n = 1n+1 = e m
1
0
2)(
e
i
i
XXf
7/29/2019 EC 515 Introduction
26/37
Basic properties of GF(2m)
Cyclic group: If there exists an element whose
powers constitute the whole group.
Order of a field element a is the smallest positive
integer N such that aN
= 1. Primitive element: If the order of an element in
GF(q) is q 1.
It may be proved that aq1 = 1 for any element a in
GF(q).
Powers of a primitive element gives all the non-
zero elements of GF(q).
7/29/2019 EC 515 Introduction
27/37
Basic properties of GF(2m)
Example: 3 is a primitive element in GF(7), while
2 is not.
If is a primitive element in GF(2m) then its
conjugates are also primitives. Example: For GF(24) = {0, 1, , 2, , 14},
check that = 7 is a primitive. Accordingly, we
may check that its conjugates 14, 13, 11 are also
primitives. If an element in GF(2m) has order N, then all its
conjugates also have order N.
7/29/2019 EC 515 Introduction
28/37
Vector spaces
F is a field whose elements are called scalars.
V is a set of elements on which a binary operation +is defined.
A multiplication operation between elements of Fand V is defined.
V is called a vector space over the field F if
V is a commutative group under +
a v is in V where a is in F and v in V
Distributive laws for any two elements a, b in Fand any two elements u, v in V.
7/29/2019 EC 515 Introduction
29/37
Vector spaces
Associative laws for any two elements a, b in F
and any element v in V.
1 v = v, where 1 is the unit element in F.
Elements of V are called vectors.
Addition + on V is called vector addition.
Multiplication combining a scalar and a vector is
called scalar multiplication or scalar product. Additive identity in V is denoted by 0.
7/29/2019 EC 515 Introduction
30/37
Properties of vector spaces
0 v = 0, where 0 is the zero element in F.
a 0 = 0
(a) v= a (v) = (a v)
Therefore, (a) vor a (v) is the additive inverseof a v
Consider vector space Vn over GF(2) that haselements given by n-tuple over GF(2) total 2ndistinct n-tuples.
Vector0 is all 0 n-tuple (additive identity)
Vector addition is mod-2 addition.
Additive inverse of an n-tuple is itself.
7/29/2019 EC 515 Introduction
31/37
Properties of vector spaces
Scalar multiplication is mod-2 multiplication.
Subspace: Subset S of V that is also a vector space
over F is called subspace of V.
Let S be a nonempty subset of a vector space V overa field F. Then, S is a subspace of V if
For any two vectors u, v in S, u + v is also a
vector in S.
For any element a in F and any vectorvin S, a v
is also a vector in S.
7/29/2019 EC 515 Introduction
32/37
Properties of vector spaces
Linear combination of vectors v1, v2, .., vN in V:
a1v1 + a2v2+ + aNvN
Sum of two linear combinations ofv1, v2, .., vN is
also a linear combination ofv1, v2, .., vN Product of a scalar in F and a linear combination
ofv1, v2, .., vN is also a linear combination ofv1,
v2, .., vN
Linearly dependent vectors: a1v1+ + aNvN = 0
Linearly independent vectors: a1v1+ + aNvN0
except when a1 = a2= = aN = 0
7/29/2019 EC 515 Introduction
33/37
Basis of vector space
B = A set of linearly independent vectors so that anyvector in V can be expressed as a linear combinationof the vectors in B:
B spans the vector space V and is called the
basis or base of the vector space V. At least one basis can be found.
Number of basis vectors is the dimension of thevector space.
Dimension of Vn = n. A set of k (where k < n) linearly independent
vectors from Vn form a k-dimensional subspace ofVn consisting of 2
k vectors.
7/29/2019 EC 515 Introduction
34/37
Dot product
Inner product ordot product: gives scalar result, u
v = u0 v0 + u1 v1+ .. + un1 vn 1 (mod-2
addition)
Orthogonal vectors: u v = 0 Properties of dot product:
u v = v u
u (v + w)= u v + u w (au) v =a(u v)
7/29/2019 EC 515 Introduction
35/37
Null space
S is a subspace of Vn.
Define Sd as another subspace of Vn so that u v = 0,
where u and v are any vectors from S and Sd,
respectively.
Check that Sd must contain vector0, and so is non-
empty subset of Vn.
For any element a in GF(2), a v = 0 orv which is
also contained in Sd.
Also, u (v + w) = 0 where v and w are any two
vectors from Sd
7/29/2019 EC 515 Introduction
36/37
Null space
This means, v and w, and also v + w are
orthogonal to u
Hence, v + w is also in Sd
Sd is the null or dual space of S
Sd is the subset of all vectors that are orthogonal to
every vector in S.
Check that, vice-versa is also true.
Hence, conversely, S is the dual space of Sd
Dimension of Sd is n k.
7/29/2019 EC 515 Introduction
37/37
Example
Let n = 3, k = 2; Vn is the vector space consisting of
all the 8 number of 3-tuples: 000, 001, 010, 011, 100,
101, 110, 111.
S = {000, 011, 101, 110} Check that S forms a subspace.
We can find a set of 2 linearly independent vectors
that spans S: {011, 101} or {101, 110} or {011, 110}
2-diemnsional vector space.
Null-space for S is given by Sd = {000, 111}; vector
111 only spans the vector space Sd 1-dimensional
vector space
Top Related