ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy,...
Transcript of ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy,...
![Page 1: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/1.jpg)
ECE 515Information Theory
Joint Entropy, Equivocation and Mutual Information
1
![Page 2: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/2.jpg)
Entropy
2
![Page 3: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/3.jpg)
Joint Entropy
3
![Page 4: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/4.jpg)
Conditional Entropy
4
![Page 5: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/5.jpg)
Chain Rule
5
![Page 6: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/6.jpg)
6
![Page 7: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/7.jpg)
Example
• Random vector [X,Y,Z]• Equally likely to take any of the 4 values
[0,0,0][0,1,0][1,0,0][1,0,1]
• Find H(XYZ) usingH(XYZ) = H(X) + H(Y|X) + H(Z|XY)
7
![Page 8: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/8.jpg)
Example
𝐻𝐻 𝑋𝑋 = −�𝑖𝑖=1
2
𝑝𝑝 𝑥𝑥𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙2 𝑝𝑝 𝑥𝑥𝑖𝑖
𝐻𝐻 𝑌𝑌 𝑋𝑋 = −�𝑖𝑖=1
2
�𝑗𝑗=1
2
𝑝𝑝 𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗 𝑙𝑙𝑙𝑙𝑙𝑙2 𝑝𝑝 𝑦𝑦𝑗𝑗|𝑥𝑥𝑖𝑖
𝐻𝐻 𝑍𝑍 𝑋𝑋𝑌𝑌
= −�𝑖𝑖=1
2
�𝑗𝑗=1
2
�𝑘𝑘=1
2
𝑝𝑝 𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗 , 𝑧𝑧𝑘𝑘 𝑙𝑙𝑙𝑙𝑙𝑙2 𝑝𝑝 𝑧𝑧𝑘𝑘|𝑥𝑥𝑖𝑖 ,𝑦𝑦𝑗𝑗
8
![Page 9: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/9.jpg)
Examplep(x1) = p(x2) = ½
H(X) = - ½log2½ - ½log2½ = 1 bit
p(xi,yj) p(yj|xj) p(0,0) = ¼ p(0|0) = ½p(1,1) = 0 p(1|1) = 0p(0,1) = ¼ p(1|0) = ½p(1,0) = ½ p(0|1) = 1
H(Y|X) = - ¼log2½ - ¼log2½ - ½log21= ½log22 = ½ bit
9
![Page 10: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/10.jpg)
Example
p(xi,yj,zk) p(zk|xj,yj) p(0,0,0) = ¼ p(0|0,0) = 1p(0,1,0) = ¼ p(0|0,1) = 1p(1,0,0) = ¼ p(0|1,0) = ½p(1,0,1) = ¼ p(1|1,0) = ½
H(Z|XY) = - ¼log21 - ¼log21 - ¼log2½ - ¼log2½= ½log22 = ½ bit
10
![Page 11: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/11.jpg)
Example
• H(XYZ) = H(X) + H(Y|X) + H(Z|XY)• H(X) = 1 bit• H(Y|X) = ½ bit• H(Z|XY) = ½ bit• H(XYZ) = 1 + ½ + ½ = 2 bits
• p(y1) = ¾ p(y2) = ¼• H(Y) = - ¾log2¾ - ¼log2¼ = .811 bit > H(Y|X)
11
![Page 12: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/12.jpg)
Information Channels
• An information channel is described by an• Input random variable X• Output random variable Y• Set of conditional probabilities p(yj|xi)
12
ChannelX Y
![Page 13: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/13.jpg)
Binary Symmetric Channel
13
x1
x2
y1
y2
X Y
![Page 14: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/14.jpg)
Binary Symmetric Channel
14
x1
x2
y1
y2
pY|X(0|1) = pY|X(1|0) = ppY|X(0|0) = pY|X(1|1) = 1-p
− = −
1 1 2 1
1 2 2 2
p( | ) p( | ) 1p( | ) p( | ) 1
y x y x p py x y x p p
![Page 15: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/15.jpg)
• The probabilities p(yj|xi) are called the forward transition probabilities
• Using Bayes’ Theorem
• The probabilities p(xi|yj) are called the backward transition probabilities
15
=p( | )p( )
p( | )( )
j i ii j
j
y x xx y
p y
![Page 16: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/16.jpg)
Non-symmetric Binary Channel
16
channel matrix
=
1 1 2 1
1 2 2 2
2 1p( | ) p( | ) 3 3p( | ) p( | ) 1 9
10 10
y x y xy x y x
34
14
0 0
1 1
23
910
13
110
x1
x2
y1
y2
![Page 17: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/17.jpg)
Backward Transition Probabilities
• H(X|y=0) = - p(1|0)log2p(1|0) - p(0|0)log2p(0|0)= - (1/21)log2(1/21) - (20/21)log2(20/21)= .209 + .067 = .276 bit
• H(X|y=1) = - p(1|1)log2p(1|1) - p(0|1)log2p(0|1)= - (9/19)log2(9/19) - (10/19)log2(10/19)= .511 + .487 = .998 bit
• H(X|Y) = p(y=0) H(X|y=0) + p(y=0) H(X|y=0)= (21/40)×(.276) + (19/40)×(.998) = .619 bit
17
= =
X|Y X|Y1 1 2 1
X|Y X|Y1 2 2 2
20 1p (0|0) p (1|0)p( | ) p( | ) 21 21p (0|1) p (1|1) 10 9p( | ) p( | )
19 19
x y x yx y x y
![Page 18: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/18.jpg)
Two Questions
• Given two random variables X and Y
– How much information does Y give about X?
– How much information does X give about Y?
18
![Page 19: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/19.jpg)
Mutual Information
19
Channelxi yj
![Page 20: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/20.jpg)
Mutual Information
20
![Page 21: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/21.jpg)
21
![Page 22: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/22.jpg)
Average Mutual Information
22
![Page 23: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/23.jpg)
Average Mutual Information
23
![Page 24: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/24.jpg)
24
![Page 25: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/25.jpg)
25
![Page 26: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/26.jpg)
Four Vectors Example
• [0,0,0], [0,1,0], [1,0,0], [1,0,1]• p(y=1) = .250 → H(Y) = .811 bit• I(X;Y) = H(Y) - H(Y|X)
= .811 - .500 = .311 bit• p(x=1) = .50 → H(X) = 1 bit• I(X;Y) = H(X) - H(X|Y)
= 1.0 - .689 = .311 bit• H(X;Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) = 1.5 bits
26
![Page 27: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/27.jpg)
Non-symmetric Binary Channel
27
channel matrix
=
1 1 2 1
1 2 2 2
2 1p( | ) p( | ) 3 3p( | ) p( | ) 1 9
10 10
y x y xy x y x
34
14
0 0
1 1
23
910
13
110
x1
x2
y1
y2
![Page 28: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/28.jpg)
Non-symmetric Channel Example
I(X;Y) = .192 bitH(X) = .811 bit
• H(X|Y) = H(X) - I(X;Y) = .619 bit
H(Y) = .998 bit• H(Y|X) = H(Y) - I(X;Y) = .806 bit
• H(XY) = H(X) + H(Y|X)= H(Y) + H(X|Y) = 1.617 bits
28
![Page 29: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/29.jpg)
Mutual Information for the BSC
29
BSCX Ycrossover probability
1p p= −p
channel matrix
p pp p
0wX Y
1w
p( 0)x w= =p( 1) 1x w w= = − =
0
1
pp
p
p
I(X;Y) = H(Y) - H(Y|X)I(X;Y) = H(X) - H(X|Y)
![Page 30: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/30.jpg)
Mutual Information for the BSC
30
![Page 31: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/31.jpg)
Binary Entropy Functionh(p) = - plog2p - (1-p)log2(1-p) 0≤p≤1
31p
h(p)
![Page 32: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/32.jpg)
32
![Page 33: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/33.jpg)
33
![Page 34: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/34.jpg)
34
![Page 35: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/35.jpg)
Conditional Mutual Information
35
Channelxiyj
zk
p(xi) p(xi|zk)
p(xi|yj,zk)
![Page 36: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/36.jpg)
Conditional Mutual Information
36
![Page 37: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/37.jpg)
37
Conditional Mutual Information
![Page 38: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/38.jpg)
Conditional Mutual Information
38
![Page 39: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/39.jpg)
39
I(X;Y) = I(Y;X)I(X;Y) ≥ 0I(X;Y) ≤ min[H(X),H(Y)]I(X;Y) = H(X) – H(X|Y)
![Page 40: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/40.jpg)
40
I(X;Y|Z) = I(Y;X|Z)I(X;Y|Z) ≥ 0I(X;Y|Z) ≤ min[H(X|Z),H(Y|Z)]I(X;Y|Z) = H(X|Z) – H(X|YZ)
H(Z)
H(Y)H(X)
![Page 41: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/41.jpg)
Joint Mutual Information
41
I(X;YZ) = I(X;Y) + I(X;Z|Y)I(X;YZ) = I(X;Z) + I(X;Y|Z)
![Page 42: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/42.jpg)
Joint Mutual Information
Y
X
Z
42
Example: Broadcast Network• Source X• Receivers Y, Z• Transmissions can be
encrypted or unencrypted• Separate encryption for Y and Z
I(X;Y) information received at Y (encrypted and unencrypted)I(X;Z|Y) information received at Z (encrypted)I(X;Z) information received at Z (encrypted and unencrypted)I(X;Y|Z) information received at Y (encrypted)I(X;YZ) = I(X;Y) + I(X;Z|Y) = I(X;Z) + I(X;Y|Z)
![Page 43: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/43.jpg)
Mutual Information
• For two random variablesI(X;Y) = H(X) – H(X|Y)
• For three random variablesI(X;Y;Z) = I(X;Y) – I(X;Y|Z)
43
![Page 44: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/44.jpg)
Joint Mutual Information
Y
X
Z
44
Example: Broadcast Network• Source X• Receivers Y, Z• Transmissions can be
encrypted or unencrypted
I(X;YZ)= I(X;Z|Y) information received at Z (encrypted)+ I(X;Y|Z) information received at Y (encrypted)+ I(X;Y;Z) information received at Y and Z (unencrypted)
![Page 45: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/45.jpg)
45
H(Z)
H(Y)H(X)
Three Random Variables X, Y and Z
I(X;Y;Z) = H(X) + H(Y) + H(Z) – H(XY) – H(XZ) – H(YZ) + H(XYZ)
H(XYZ)
![Page 46: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/46.jpg)
46
Three Random Variables X, Y and Z
H(XYZ)
![Page 47: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/47.jpg)
47
H(XYZ) H(XYZ)
I(X;Y|Z)H(Y|XZ)
H(Z|XY)
I(X;Y;Z)
H(X|YZ)
I(X;Z|Y) I(Y;Z|X)
H(X)
H(YZ|X)
H(YZ)
H(X|YZ) H(Y|X)
H(X|YZ)
H(Z|XY)
I(X;YZ)
![Page 48: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/48.jpg)
XOR Gate
48
= ⊕
== ≠
01
k i j
i jk
i j
z x y
x yz
x y
XY Z
x y z0 0 00 1 11 0 11 1 0
p(xi)=p(yj)=0.5X and Y are SI
I(X;Y;Z) = I(X;Y) – I(X;Y|Z)
![Page 49: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/49.jpg)
Probabilities for Three RVs
49
![Page 50: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/50.jpg)
XOR Gate
50
0
00X Y
Z
1
1
1-1
![Page 51: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/51.jpg)
XOR Gate
• I(X;Y;Z) = I(X;Y) – I(X;Y|Z)• X and Y are independent so I(X;Y) = 0• I(X;Y|Z) = 1 bit• I(X;Y;Z) = 0 – 1 = -1 bit
• I(X;Y;Z) is called the Interaction Information
51
![Page 52: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/52.jpg)
I(X;Y;Z)
• X – rain• Y – dark• Z – cloudy
• Which is larger?I(rain;dark) or I(rain;dark|cloudy)
52
![Page 53: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/53.jpg)
Additivity of Mutual Information
53
ChannelX
Y1
Y2
YN…
![Page 54: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/54.jpg)
Additivity of Mutual Information
I(X;Y1Y2…YN) = I(X;Y1) + I(X;Y2|Y1) + I(X;Y3|Y1Y2) + … + I(X;YN|Y1Y2…YN-1)
I(X;Y1Y2…YN) ≤ H(X)
All terms on the RHS ≥ 0
54
![Page 55: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/55.jpg)
Binary Symmetric Channel
55
channel matrix
p pp p
BSCX Y
X Y
crossover probability 1p p= −
p
![Page 56: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/56.jpg)
56
![Page 57: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/57.jpg)
Additivity of Mutual Information
57
BSCXYZ
![Page 58: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/58.jpg)
Probabilities for a Repetitive BSC
58
![Page 59: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/59.jpg)
Additivity of Mutual Information
59
![Page 60: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/60.jpg)
60
N=1N=2N=3
![Page 61: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/61.jpg)
Cascaded Channels
61
X YChannel 1 Channel 2 Channel N
![Page 62: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/62.jpg)
Cascaded Channels
The RVs from X to Y form aMarkov chain
f the conditional distributions of the channel outputs depend only on the immediate inputs and are conditionally independent of the previous RVs
62
![Page 63: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/63.jpg)
Two Cascaded Channels
63
p(yj|xi,zk) = p(yj|zk)p(xi|zk,yj) = p(xi|zk)
![Page 64: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/64.jpg)
Two Cascaded Channels
[Cover and Thomas p. 34]Three random variables X,Y,Z form a Markov chain, denoted by X → Z → Y, if their joint probability
p(xi,yj,zk) = p(xi)p(zk|xi)p(yj|xi,zk)can be factored as
p(xi,yj,zk) = p(xi)p(zk|xi)p(yj|zk)
64
![Page 65: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/65.jpg)
Two Cascaded Channels
• If X,Y,Z form a Markov chain, thenI(X;Y) ≤ I(X;Z)
• To prove this, note that I(X;Y|Z) = 0 andI(X;YZ) = I(X;Z) + I(X;Y|Z) = I(X;Y) + I(X;Z|Y)so thatI(X;Z) = I(X;Y) + I(X;Z|Y)orI(X;Y) = I(X;Z) - I(X;Z|Y)
65
![Page 66: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/66.jpg)
Two Cascaded Channels
• I(X;Y|Z) = 0• I(X;Y) ≤ I(X;Z)
H(X) – H(X|Y) ≤ H(X) – H(X|Z)H(X|Y) ≥ H(X|Z)
• I(Y;X) ≤ I(Y;Z)H(Y) – H(Y|X) ≤ H(Y) – H(Y|Z)H(Y|X) ≥ H(Y|Z)
66
![Page 67: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/67.jpg)
Data Processing Inequality
• The mutual information between the input and output can never exceed the mutual information between the input and an intermediate point
I(X;Y) ≤ I(X;Z)• The mutual information between the output and
input can never exceed the mutual information between the output and an intermediate point
I(Y;X) ≤ I(Y;Z)• Data processing cannot increase the amount of
information
67
![Page 68: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/68.jpg)
Cascaded BSCs
68
X YBSC 1 BSC 2 BSC N
BSC channel matrix
p pp p
![Page 69: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/69.jpg)
Cascaded BSCs p=.01
69
Number ofChannels N
Equivalent CrossoverProbability
I(X;Y)
1 .01 .919
2 .0198 .860
3 .0294 .809
4 .0388 .763
5 .0480 .722
10 .0915 .559
20 .166 .352
30 .227 .227
40 .277 .149
50 .318 .0978
64 .363 .0549
256 .497 .0000260
![Page 70: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/70.jpg)
70
I(X;Y
)
N=1
N=2
N=3
![Page 71: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/71.jpg)
A Mathematical Theory of Communications, BSTJ July, 1948
``The fundamental problem of communication isthat of reproducing at one point exactly orapproximately a message selected at anotherpoint. … If the channel is noisy it is not in generalpossible to reconstruct the original message orthe transmitted signal with certainty by anyoperation on the received signal.’’
71
![Page 72: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/72.jpg)
A Mathematical Theory of Communications, BSTJ July, 1948
通信的基本问题是,在一个点上的再现准确或约在另一点选择的消息。如果通道是噪声是不一般未能重建原始消息,或确定所传输的信号,由接收到的信号上的任何操作。Communication is a basic problem in accurate reproduction of a point or another point about the selected message. If the channel is the noise is generally possible to reconstruct the original message, or to determine the transmitted signal from the received signal to any operation.
72
![Page 73: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/73.jpg)
A Mathematical Theory of Communications, BSTJ July, 1948و أالتواصل ھو المشكلة األساسیة في االستنساخ الدقیق للنقطة
إذا القناة ھو الضجیج عموما . نقطة أخرى حول الرسالة المحددةممكن إلعادة بناء الرسالة األصلیة، أو لتحدید اإلشارة المرسلة .من إشارة وردت إلى أي عملیةNetworking is a fundamental problem in the exact reproduction of one point or another about the selected message. If the channel noise is generally possible to reconstruct the original message, or to determine the transmitted signal from the received signal to any process.
73
![Page 74: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/74.jpg)
A Mathematical Theory of Communications, BSTJ July, 1948
Networking ist ein grundsätzliches Problem in der exakten Wiedergabe der einen oder anderen Punkt über die ausgewählte Nachricht. Wenn der Kanal Rauschen ist in der Regel möglich, die ursprüngliche Nachricht zu rekonstruieren, um die übertragenen Signals aus dem empfangenen Signal für jeden Prozess zu bestimmen.Networking is a fundamental problem in the exact reproduction of one point or another over the selected message. If the channel noise is normally possible to reconstruct the original message in order to determine the transmitted signal from the received signal for each process.
74
![Page 75: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/75.jpg)
A Mathematical Theory of Communications, BSTJ July, 1948
नेटविक� ग चुने गए संदेश पर एक िबंदु या िकसी अ� कीसटीक प्रजनन म� एक मूलभूत सम�ा है. चैनल शोर प्र�ेकप्रिक्रया के िलए प्रा� संकेत से संकेत संचा�रत िनधा��रतकरने के िलए मूल संदेश को िफर से संगिठत करने के िलएसामा� �प से संभव है.Networking at one point or another of the selected message is a fundamental problem in accurate reproduction. Channel noise for each process receives the signal from the transmit signal to determine the message again to organize normally possible.
75
![Page 76: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/76.jpg)
Entropy
• Let X be a random variable with probability distribution
p(X) = {p(xi)}
• H(X) = Ep[-log(p(X)]
76
![Page 77: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/77.jpg)
Relative Entropy
• Let X be a random variable with two different probability distributions
p(X) = {p(xi)}q(X) = {q(xi)}
77
![Page 78: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/78.jpg)
Relative Entropy
• The relative entropy between two probability distributions p(X) and q(X) is defined as the expectation of the logarithm of the ratio of the distributions
D[p(X)||q(X)] = Ep[log(p(X)/q(X))]
78
![Page 79: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/79.jpg)
Relative Entropy
79
![Page 80: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/80.jpg)
Relative Entropy
• The relative entropy is a measure of how different two probability distributions p and q (over the same sample space X) are.
80
![Page 81: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/81.jpg)
Relative Entropy
• Divergence InequalityD[p(X)||q(X)] ≥ 0
with equality iff p(X)=q(X)
81
![Page 82: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/82.jpg)
Relative Entropy
• If nothing is known about the source, the best approximation is a uniform distribution
q(xi) = 1/N• In this case
D[p(X)||q(X)] = log2N - H(p(X))
82
InformationSource X
![Page 83: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/83.jpg)
Example 1: Four Symbol Source
• p(x1) = 1/2 p(x2) = 1/4 p(x3) = p(x4) = 1/8• q(x1) = q(x2) = q(x3) = q(x4) = 1/4 (equiprobable)
• H(p(X)) = 1.75 bits• H(q(X)) = log2N = 2.00 bits• D[p(X)||q(X)] = log2N - H(p(X)) = 0.25 bit
83
InformationSource
![Page 84: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/84.jpg)
Example 2: Two Symbol Source
• p(x1) = p(x2) = 1/2• q(x1) = 1/4 q(x2) = 3/4
• D[p(X)||q(X)] = .208 bits• D[q(X)||p(X)] = .188 bits
84
![Page 85: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/85.jpg)
D[p(X)||q(X)] versus D[q(X)||p(X)]
• p(xi) = 1/N• q(x1) = ε q(xi) = (1-ε)/(N-1) i≠1• as ε→0
D[p(X)||q(X)] → ∞D[q(X)||p(X)] → log(N/(N-1))
85
![Page 86: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/86.jpg)
Cross Entropy
• The cross entropy between the probability distributions p(X) and q(X) is defined as
H(p,q) = H(p(X))+D(p(X)||q(X))H(p,q) = Ep[-log(q(X)]
86
![Page 87: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/87.jpg)
Example 3: Four Symbol Source
• p(x1) = 1/2 p(x2) = 1/4 p(x3) = p(x4) = 1/8• q(x1) = 1/2 q(x2) = q(x3) = q(x4) = 1/6
• H(p(X)) = 1.75 bits• H(p,q) = 1.7925 bits• D[p(X)||q(X)] = 0.0425 bits
87
![Page 88: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/88.jpg)
88
![Page 89: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/89.jpg)
89
![Page 90: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/90.jpg)
Cross Entropy in ANNs
90
![Page 91: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/91.jpg)
Mutual Information
91
![Page 92: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/92.jpg)
Conditional Relative Entropy
• For joint probability density functionsp(XY) and q(XY)
the conditional relative entropy isD[p(Y|X)||q(Y|X)]
92
![Page 93: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/93.jpg)
Chain Rule for Relative Entropy
D[p(XY)||q(XY)] = D[p(X)||q(X)] + D[p(Y|X)||q(Y|X)]
93
![Page 94: ECE 515 Information Theoryagullive/joint515.pdf · ECE 515 Information Theory Joint Entropy, Equivocation and Mutual Information 1](https://reader034.fdocuments.in/reader034/viewer/2022050718/5e16d48ec457b14f4409f921/html5/thumbnails/94.jpg)
Three Random Variables X, Y and Z
94
X
Y
Z
H(X)
H(X|YZ)
I(X;Y|Z)I(X;Y;Z)
H(Y|X) =H(Y|XZ)+I(Y;Z|X)
I(X;Z) = I(X;Y;Z) + I(X;Z|Y)