EE5139R Communication Systems, Problem Set 1 Page 1

EE5139R, Problem Set 1

Q1. X and Y are two discrete random variables with joint distribution P (X = 0, Y = 0) = P (X =0, Y = 1) = P (X = 1, Y = 0) = 13 . Compute H(X), H(Y ), H(X|Y ), H(Y |X) and H(X,Y ).

Q2. A useful inequality - Prove that ln(x) x 1. Also plot ln(x) and x 1 on the same graph.Q3. The output of a discrete source consists of the possible letters x1, x2, , xn, which occur with

probability p1, p2, , pn, respectively. Show that the entropy of the source is at most log(n).Hint: Use the useful inequality you derived above.

Q4. Let X be a random variable with entropy H(X) of 8 bits. Let Y (X) be a deterministic functionof X and suppose that Y (X) is an invertible function.a. Compute H(Y ), H(Y |X), H(X|Y ), and H(X,Y ).b. If Y (X) is not invertible, what can you say about H(Y ) and H(X|Y )?

Q5. X and Y are two discrete random variables with joint distribution P (x, y). Show that I(X;Y ) 0with equality if and only if X and Y are statistically independent. Hint: Use the useful inequalityyou derived above.