chap2_solution manual _ elements of Information thoery,pdf

7/30/2019 chap2_solution manual _ elements of Information thoery,pdf

1/32

C h a p t e r 2

E n t r o p y , R e l a t i v e E n t r o p y a n d

M u t u a l I n f o r m a t i o n

1 . C o i n i p s . A f a i r c o i n i s i p p e d u n t i l t h e r s t h e a d o c c u r s . L e t X d e n o t e t h e n u m b e r

o f i p s r e q u i r e d .

( a ) F i n d t h e e n t r o p y H ( X ) i n b i t s . T h e f o l l o w i n g e x p r e s s i o n s m a y b e u s e f u l :

1

X

n = 1

r

n

=

r

1 ? r

;

1

X

n = 1

n r

n

=

r

( 1 ? r )

2

:

( b ) A r a n d o m v a r i a b l e X i s d r a w n a c c o r d i n g t o t h i s d i s t r i b u t i o n . F i n d a n \ e c i e n t "

s e q u e n c e o f y e s - n o q u e s t i o n s o f t h e f o r m , \ I s X c o n t a i n e d i n t h e s e t S ? " C o m p a r e

H ( X ) t o t h e e x p e c t e d n u m b e r o f q u e s t i o n s r e q u i r e d t o d e t e r m i n e X .

S o l u t i o n :

( a ) T h e n u m b e r X o f t o s s e s t i l l t h e r s t h e a d a p p e a r s h a s t h e g e o m e t r i c d i s t r i b u t i o n

w i t h p a r a m e t e r p = 1 = 2 , w h e r e P ( X = n ) = p q

n ? 1

, n 2 f 1 ; 2 ; : : : g . H e n c e t h e

e n t r o p y o f X i s

H ( X ) = ?

1

X

n = 1

p q

n ? 1

l o g ( p q

n ? 1

)

= ?

"

1

X

n = 0

p q

n

l o g p +

1

X

n = 0

n p q

n

l o g q

#

=

? p l o g p

1 ? q

?

p q l o g q

p

2

=

? p l o g p ? q l o g q

p

= H ( p ) = p b i t s .

I f p = 1 = 2 , t h e n H ( X ) = 2 b i t s .

7


2/32

8

E n t r o p y , R e l a t i v e E n t r o p y a n d M u t u a l I n f o r m a t i o n

( b ) I n t u i t i v e l y , i t s e e m s c l e a r t h a t t h e b e s t q u e s t i o n s a r e t h o s e t h a t h a v e e q u a l l y l i k e l y

c h a n c e s o f r e c e i v i n g a y e s o r a n o a n s w e r . C o n s e q u e n t l y , o n e p o s s i b l e g u e s s i s

t h a t t h e m o s t \ e c i e n t " s e r i e s o f q u e s t i o n s i s : I s X = 1 ? I f n o t , i s X = 2 ?

I f n o t , i s X = 3 ? : : : w i t h a r e s u l t i n g e x p e c t e d n u m b e r o f q u e s t i o n s e q u a l t o

P

1

n = 1

n ( 1 = 2

n

) = 2 : T h i s s h o u l d r e i n f o r c e t h e i n t u i t i o n t h a t H ( X ) i s a m e a -

s u r e o f t h e u n c e r t a i n t y o f X . I n d e e d i n t h i s c a s e , t h e e n t r o p y i s e x a c t l y t h e

s a m e a s t h e a v e r a g e n u m b e r o f q u e s t i o n s n e e d e d t o d e n e X , a n d i n g e n e r a l

E ( # o f q u e s t i o n s ) H ( X ) . T h i s p r o b l e m h a s a n i n t e r p r e t a t i o n a s a s o u r c e c o d -

i n g p r o b l e m . L e t 0 = n o , 1 = y e s , X = S o u r c e , a n d Y = E n c o d e d S o u r c e . T h e n

t h e s e t o f q u e s t i o n s i n t h e a b o v e p r o c e d u r e c a n b e w r i t t e n a s a c o l l e c t i o n o f ( X ; Y )

p a i r s : ( 1 ; 1 ) , ( 2 ; 0 1 ) , ( 3 ; 0 0 1 ) , e t c . . I n f a c t , t h i s i n t u i t i v e l y d e r i v e d c o d e i s t h e

o p t i m a l ( H u m a n ) c o d e m i n i m i z i n g t h e e x p e c t e d n u m b e r o f q u e s t i o n s .

2 . E n t r o p y o f f u n c t i o n s . L e t X b e a r a n d o m v a r i a b l e t a k i n g o n a n i t e n u m b e r o f v a l u e s .

W h a t i s t h e ( g e n e r a l ) i n e q u a l i t y r e l a t i o n s h i p o f H ( X ) a n d H ( Y ) i f

( a ) Y = 2

X

?

( b ) Y = c o s X ?

S o l u t i o n : L e t y = g ( x ) . T h e n

p ( y ) =

X

x : y = g ( x )

p ( x ) :

C o n s i d e r a n y s e t o f x ' s t h a t m a p o n t o a s i n g l e y . F o r t h i s s e t

X

x : y = g ( x )

p ( x ) l o g p ( x )

X

x : y = g ( x )

p ( x ) l o g p ( y ) = p ( y ) l o g p ( y ) ;

s i n c e l o g i s a m o n o t o n e i n c r e a s i n g f u n c t i o n a n d p ( x )

P

x : y = g ( x )

p ( x ) = p ( y ) . E x -

t e n d i n g t h i s a r g u m e n t t o t h e e n t i r e r a n g e o f X ( a n d Y ) , w e o b t a i n

H ( X ) = ?

X

x

p ( x ) l o g p ( x )

= ?

X

y

X

x : y = g ( x )

p ( x ) l o g p ( x )

?

X

y

p ( y ) l o g p ( y )

= H ( Y ) ;

w i t h e q u a l i t y i g i s o n e - t o - o n e w i t h p r o b a b i l i t y o n e .

( a ) Y = 2

X

i s o n e - t o - o n e a n d h e n c e t h e e n t r o p y , w h i c h i s j u s t a f u n c t i o n o f t h e

p r o b a b i l i t i e s ( a n d n o t t h e v a l u e s o f a r a n d o m v a r i a b l e ) d o e s n o t c h a n g e , i . e . ,

H ( X ) = H ( Y ) .

( b ) Y = c o s ( X ) i s n o t n e c e s s a r i l y o n e - t o - o n e . H e n c e a l l t h a t w e c a n s a y i s t h a t

H ( X ) H ( Y ) , w i t h e q u a l i t y i f c o s i n e i s o n e - t o - o n e o n t h e r a n g e o f X .


3/32


9

3 . M i n i m u m e n t r o p y . W h a t i s t h e m i n i m u m v a l u e o f H ( p

1

; : : : ; p

n

) = H ( p ) a s p r a n g e s

o v e r t h e s e t o f n - d i m e n s i o n a l p r o b a b i l i t y v e c t o r s ? F i n d a l l p ' s w h i c h a c h i e v e t h i s

m i n i m u m .

S o l u t i o n : W e w i s h t o n d a l l p r o b a b i l i t y v e c t o r s p = ( p

1

; p

2

; : : : ; p

n

) w h i c h m i n i m i z e

H ( p ) = ?

X

i

p

i

l o g p

i

:

N o w ? p

i

l o g p

i

0 , w i t h e q u a l i t y i p

i

= 0 o r 1 . H e n c e t h e o n l y p o s s i b l e p r o b a b i l i t y

v e c t o r s w h i c h m i n i m i z e H ( p ) a r e t h o s e w i t h p

i

= 1 f o r s o m e i a n d p

j

= 0 ; j 6= i .

T h e r e a r e n s u c h v e c t o r s , i . e . , ( 1 ; 0 ; : : : ; 0 ) , ( 0 ; 1 ; 0 ; : : : ; 0 ) , : : : , ( 0 ; : : : ; 0 ; 1 ) , a n d t h e

m i n i m u m v a l u e o f H ( p ) i s 0 .

4 . A x i o m a t i c d e n i t i o n o f e n t r o p y . I f w e a s s u m e c e r t a i n a x i o m s f o r o u r m e a s u r e o f i n f o r -

m a t i o n , t h e n w e w i l l b e f o r c e d t o u s e a l o g a r i t h m i c m e a s u r e l i k e e n t r o p y . S h a n n o n u s e d

t h i s t o j u s t i f y h i s i n i t i a l d e n i t i o n o f e n t r o p y . I n t h i s b o o k , w e w i l l r e l y m o r e o n t h e

o t h e r p r o p e r t i e s o f e n t r o p y r a t h e r t h a n i t s a x i o m a t i c d e r i v a t i o n t o j u s t i f y i t s u s e . T h e

f o l l o w i n g p r o b l e m i s c o n s i d e r a b l y m o r e d i c u l t t h a n t h e o t h e r p r o b l e m s i n t h i s s e c t i o n .

I f a s e q u e n c e o f s y m m e t r i c f u n c t i o n s H

m

( p

1

; p

2

; : : : ; p

m

) s a t i s e s t h e f o l l o w i n g p r o p e r -

t i e s ,

N o r m a l i z a t i o n : H

2

1

2

;

1

2

= 1 ;

C o n t i n u i t y : H

2

( p ; 1 ? p ) i s a c o n t i n u o u s f u n c t i o n o f p ,

G r o u p i n g : H

m

( p

1

; p

2

; : : : ; p

m

) = H

m ? 1

( p

1

+ p

2

; p

3

; : : : ; p

m

) + ( p

1

+ p

2

) H

2

p

1

p

1

+ p

2

;

p

2

p

1

+ p

2

,

p r o v e t h a t H

m

m u s t b e o f t h e f o r m

H

m

( p

1

; p

2

; : : : ; p

m

) = ?

m

X

i = 1

p

i

l o g p

i

; m = 2 ; 3 ; : : : : ( 2 . 1 )

T h e r e a r e v a r i o u s o t h e r a x i o m a t i c f o r m u l a t i o n s w h i c h a l s o r e s u l t i n t h e s a m e d e n i t i o n

o f e n t r o p y . S e e , f o r e x a m p l e , t h e b o o k b y C s i s z a r a n d K o r n e r 3 ] .

S o l u t i o n : A x i o m a t i c d e n i t i o n o f e n t r o p y . T h i s i s a l o n g s o l u t i o n , s o w e w i l l r s t

o u t l i n e w h a t w e p l a n t o d o . F i r s t w e w i l l e x t e n d t h e g r o u p i n g a x i o m b y i n d u c t i o n a n d

p r o v e t h a t

H

m

( p

1

; p

2

; : : : ; p

m

) = H

m ? k

( p

1

+ p

2

+ + p

k

; p

k + 1

; : : : ; p

m

)

+ ( p

1

+ p

2

+ + p

k

) H

k

p

1

p

1

+ p

2

+ + p

k

; : : : ;

p

k

p

1

+ p

2

+ + p

k

: ( 2 . 2 )

L e t f ( m ) b e t h e e n t r o p y o f a u n i f o r m d i s t r i b u t i o n o n m s y m b o l s , i . e . ,

f ( m ) = H

m

1

m

;

1

m

; : : : ;

1

m

: ( 2 . 3 )

W e w i l l t h e n s h o w t h a t f o r a n y t w o i n t e g e r s r a n d s , t h a t f ( r s ) = f ( r ) + f ( s ) .

W e u s e t h i s t o s h o w t h a t f ( m ) = l o g m . W e t h e n s h o w f o r r a t i o n a l p = r = s , t h a t


4/32

1 0


H

2

( p ; 1 ? p ) = ? p l o g p ? ( 1 ? p ) l o g ( 1 ? p ) . B y c o n t i n u i t y , w e w i l l e x t e n d i t t o i r r a t i o n a l

p a n d n a l l y b y i n d u c t i o n a n d g r o u p i n g , w e w i l l e x t e n d t h e r e s u l t t o H

m

f o r m 2 .

T o b e g i n , w e e x t e n d t h e g r o u p i n g a x i o m . F o r c o n v e n i e n c e i n n o t a t i o n , w e w i l l l e t

S

k

=

k

X

i = 1

p

i

( 2 . 4 )

a n d w e w i l l d e n o t e H

2

( q ; 1 ? q ) a s h ( q ) . T h e n w e c a n w r i t e t h e g r o u p i n g a x i o m a s

H

m

( p

1

; : : : ; p

m

) = H

m ? 1

( S

2

; p

3

; : : : ; p

m

) + S

2

h

p

2

S

2

: ( 2 . 5 )

A p p l y i n g t h e g r o u p i n g a x i o m a g a i n , w e h a v e

H

m

( p

1

; : : : ; p

m

) = H

m ? 1

( S

2

; p

3

; : : : ; p

m

) + S

2

h

p

2

S

2

( 2 . 6 )

= H

m ? 2

( S

3

; p

4

; : : : ; p

m

) + S

3

h

p

3

S

3

+ S

2

h

p

2

S

2

( 2 . 7 )

.

.

. ( 2 . 8 )

= H

m ? ( k ? 1 )

( S

k

; p

k + 1

; : : : ; p

m

) +

k

X

i = 2

S

i

h

p

i

S

i

: ( 2 . 9 )

N o w , w e a p p l y t h e s a m e g r o u p i n g a x i o m r e p e a t e d l y t o H

k

( p

1

= S

k

; : : : ; p

k

= S

k

) , t o o b t a i n

H

k

p

1

S

k

; : : : ;

p

k

S

k

= H

2

S

k ? 1

S

k

;

p

k

S

k

+

k ? 1

X

i = 2

S

i

S

k

h

p

i

= S

k

S

i

= S

k

( 2 . 1 0 )

=

1

S

k

k

X

i = 2

S

i

h

p

i

S

i

: ( 2 . 1 1 )

F r o m ( 2 . 9 ) a n d ( 2 . 1 1 ) , i t f o l l o w s t h a t

H

m

( p

1

; : : : ; p

m

) = H

m ? k

( S

k

; p

k + 1

; : : : ; p

m

) + S

k

H

k

p

1

S

k

; : : : ;

p

k

S

k

; ( 2 . 1 2 )

w h i c h i s t h e e x t e n d e d g r o u p i n g a x i o m .

N o w w e n e e d t o u s e a n a x i o m t h a t i s n o t e x p l i c i t l y s t a t e d i n t h e t e x t , n a m e l y t h a t t h e

f u n c t i o n H

m

i s s y m m e t r i c w i t h r e s p e c t t o i t s a r g u m e n t s . U s i n g t h i s , w e c a n c o m b i n e

a n y s e t o f a r g u m e n t s o f H

m

u s i n g t h e e x t e n d e d g r o u p i n g a x i o m .

L e t f ( m ) d e n o t e H

m

(

1

m

;

1

m

; : : : ;

1

m

) .

C o n s i d e r

f ( m n ) = H

m n

(

1

m n

;

1

m n

; : : : ;

1

m n

) : ( 2 . 1 3 )


5/32


1 1

B y r e p e a t e d l y a p p l y i n g t h e e x t e n d e d g r o u p i n g a x i o m , w e h a v e

f ( m n ) = H

m n

(

1

m n

;

1

m n

; : : : ;

1

m n

) ( 2 . 1 4 )

= H

m n ? n

(

1

m

;

1

m n

; : : : ;

1

m n

) +

1

m

H

n

(

1

n

; : : : ;

1

n

) ( 2 . 1 5 )

= H

m n ? 2 n

(

1

m

;

1

m

;

1

m n

; : : : ;

1

m n

) +

2

m

H

n

(

1

n

; : : : ;

1

n

) ( 2 . 1 6 )

.

.

. ( 2 . 1 7 )

= H

m

(

1

m

; : : : :

1

m

) + H (

1

n

; : : : ;

1

n

) ( 2 . 1 8 )

= f ( m ) + f ( n ) : ( 2 . 1 9 )

W e c a n i m m e d i a t e l y u s e t h i s t o c o n c l u d e t h a t f ( m

k

) = k f ( m ) .

N o w , w e w i l l a r g u e t h a t H

2

( 1 ; 0 ) = h ( 1 ) = 0 . W e d o t h i s b y e x p a n d i n g H

3

( p

1

; p

2

; 0 )

( p

1

+ p

2

= 1 ) i n t w o d i e r e n t w a y s u s i n g t h e g r o u p i n g a x i o m

H

3

( p

1

; p

2

; 0 ) = H

2

( p

1

; p

2

) + p

2

H

2

( 1 ; 0 ) ( 2 . 2 0 )

= H

2

( 1 ; 0 ) + ( p

1

+ p

2

) H

2

( p

1

; p

2

) ( 2 . 2 1 )

T h u s p

2

H

2

( 1 ; 0 ) = H

2

( 1 ; 0 ) f o r a l l p

2

, a n d t h e r e f o r e H ( 1 ; 0 ) = 0 .

W e w i l l a l s o n e e d t o s h o w t h a t f ( m + 1 ) ? f ( m ) ! 0 a s m ! 1 . T o p r o v e t h i s , w e

u s e t h e e x t e n d e d g r o u p i n g a x i o m a n d w r i t e

f ( m + 1 ) = H

m + 1

(

1

m + 1

; : : : ;

1

m + 1

) ( 2 . 2 2 )

= h (

1

m + 1

) +

m

m + 1

H

m

(

1

m

; : : : ;

1

m

) ( 2 . 2 3 )

= h (

1

m + 1

) +

m

m + 1

f ( m ) ( 2 . 2 4 )

a n d t h e r e f o r e

f ( m + 1 ) ?

m

m + 1

f ( m ) = h (

1

m + 1

) : ( 2 . 2 5 )

T h u s l i m f ( m + 1 ) ?

m

m + 1

f ( m ) = l i m h (

1

m + 1

) : B u t b y t h e c o n t i n u i t y o f H

2

, i t f o l l o w s

t h a t t h e l i m i t o n t h e r i g h t i s h ( 0 ) = 0 . T h u s l i m h (

1

m + 1

) = 0 .

L e t u s d e n e

a

n + 1

= f ( n + 1 ) ? f ( n ) ( 2 . 2 6 )

a n d

b

n

= h (

1

n

) : ( 2 . 2 7 )

T h e n

a

n + 1

= ?

1

n + 1

f ( n ) + b

n + 1

( 2 . 2 8 )

= ?

1

n + 1

n

X

i = 2

a

i

+ b

n + 1

( 2 . 2 9 )


6/32

1 2



( n + 1 ) b

n + 1

= ( n + 1 ) a

n + 1

+

n

X

i = 2

a

i

: ( 2 . 3 0 )

T h e r e f o r e s u m m i n g o v e r n , w e h a v e

N

X

n = 2

n b

n

=

N

X

n = 2

( n a

n

+ a

n ? 1

+ : : : + a

2

) = N

N

X

n = 2

a

i

: ( 2 . 3 1 )

D i v i d i n g b o t h s i d e s b y

P

N

n = 1

n = N ( N + 1 ) = 2 , w e o b t a i n

2

N + 1

N

X

n = 2

a

n

=

P

N

n = 2

n b

n

P

N

n = 2

n

( 2 . 3 2 )

N o w b y c o n t i n u i t y o f H

2

a n d t h e d e n i t i o n o f b

n

, i t f o l l o w s t h a t b

n

! 0 a s n ! 1 .

S i n c e t h e r i g h t h a n d s i d e i s e s s e n t i a l l y a n a v e r a g e o f t h e b

n

' s , i t a l s o g o e s t o 0 ( T h i s

c a n b e p r o v e d m o r e p r e c i s e l y u s i n g ' s a n d ' s ) . T h u s t h e l e f t h a n d s i d e g o e s t o 0 . W e

c a n t h e n s e e t h a t

a

N + 1

= b

N + 1

?

1

N + 1

N

X

n = 2

a

n

( 2 . 3 3 )

a l s o g o e s t o 0 a s N ! 1 . T h u s

f ( n + 1 ) ? f ( n ) ! 0 a s n ! 1 : ( 2 . 3 4 )

W e w i l l n o w p r o v e t h e f o l l o w i n g l e m m a

L e m m a 2 . 0 . 1 L e t t h e f u n c t i o n f ( m ) s a t i s f y t h e f o l l o w i n g a s s u m p t i o n s :

f ( m n ) = f ( m ) + f ( n ) f o r a l l i n t e g e r s m , n .

l i m

n ! 1

( f ( n + 1 ) ? f ( n ) ) = 0

f ( 2 ) = 1 ,

t h e n t h e f u n c t i o n f ( m ) = l o g

2

m .

P r o o f o f t h e l e m m a : L e t P b e a n a r b i t r a r y p r i m e n u m b e r a n d l e t

g ( n ) = f ( n ) ?

f ( P ) l o g

2

n

l o g

2

P

( 2 . 3 5 )

T h e n g ( n ) s a t i s e s t h e r s t a s s u m p t i o n o f t h e l e m m a . A l s o g ( P ) = 0 .

A l s o i f w e l e t

n

= g ( n + 1 ) ? g ( n ) = f ( n + 1 ) ? f ( n ) +

f ( P )

l o g

2

P

l o g

2

n

n + 1

( 2 . 3 6 )

t h e n t h e s e c o n d a s s u m p t i o n i n t h e l e m m a i m p l i e s t h a t l i m

n

= 0 .


7/32


1 3

F o r a n i n t e g e r n , d e n e

n

( 1 )

=

n

P

: ( 2 . 3 7 )

T h e n i t f o l l o w s t h a t n

( 1 )

< n = P , a n d

n = n

( 1 )

P + l ( 2 . 3 8 )

w h e r e 0 l < P . F r o m t h e f a c t t h a t g ( P ) = 0 , i t f o l l o w s t h a t g ( P n

( 1 )

) = g ( n

( 1 )

) ,

a n d

g ( n ) = g ( n

( 1 )

) + g ( n ) ? g ( P n

( 1 )

) = g ( n

( 1 )

) +

n ? 1

X

i = P n

( 1 )

i

( 2 . 3 9 )

J u s t a s w e h a v e d e n e d n

( 1 )

f r o m n , w e c a n d e n e n

( 2 )

f r o m n

( 1 )

. C o n t i n u i n g t h i s

p r o c e s s , w e c a n t h e n w r i t e

g ( n ) = g ( n

( k )

+

k

X

j = 1

0

@

n

( i ? 1 )

X

i = P n

( i )

i

1

A

: ( 2 . 4 0 )

S i n c e n

( k )

n = P

k

, a f t e r

k =

l o g n

l o g P

+ 1 ( 2 . 4 1 )

t e r m s , w e h a v e n

( k )

= 0 , a n d g ( 0 ) = 0 ( t h i s f o l l o w s d i r e c t l y f r o m t h e a d d i t i v e p r o p e r t y

o f g ) . T h u s w e c a n w r i t e

g ( n ) =

t

n

X

i = 1

i

( 2 . 4 2 )

t h e s u m o f b

n

t e r m s , w h e r e

b

n

P

l o g n

l o g P

+ 1

: ( 2 . 4 3 )

S i n c e

n

! 0 , i t f o l l o w s t h a t

g ( n )

l o g

2

n

! 0 , s i n c e g ( n ) h a s a t m o s t o ( l o g

2

n ) t e r m s

i

.

T h u s i t f o l l o w s t h a t

l i m

n ! 1

f ( n )

l o g

2

n

=

f ( P )

l o g

2

P

( 2 . 4 4 )

S i n c e P w a s a r b i t r a r y , i t f o l l o w s t h a t f ( P ) = l o g

2

P = c f o r e v e r y p r i m e n u m b e r P .

A p p l y i n g t h e t h i r d a x i o m i n t h e l e m m a , i t f o l l o w s t h a t t h e c o n s t a n t i s 1 , a n d f ( P ) =

l o g

2

P .

F o r c o m p o s i t e n u m b e r s N = P

1

P

2

: : : P

l

, w e c a n a p p l y t h e r s t p r o p e r t y o f f a n d t h e

p r i m e n u m b e r f a c t o r i z a t i o n o f N t o s h o w t h a t

f ( N ) =

X

f ( P

i

) =

X

l o g

2

P

i

= l o g

2

N : ( 2 . 4 5 )

T h u s t h e l e m m a i s p r o v e d .

T h e l e m m a c a n b e s i m p l i e d c o n s i d e r a b l y , i f i n s t e a d o f t h e s e c o n d a s s u m p t i o n , w e

r e p l a c e i t b y t h e a s s u m p t i o n t h a t f ( n ) i s m o n o t o n e i n n . W e w i l l n o w a r g u e t h a t t h e


8/32

1 4


o n l y f u n c t i o n f ( m ) s u c h t h a t f ( m n ) = f ( m ) + f ( n ) f o r a l l i n t e g e r s m ; n i s o f t h e f o r m

f ( m ) = l o g

a

m f o r s o m e b a s e a .

L e t c = f ( 2 ) . N o w f ( 4 ) = f ( 2 2 ) = f ( 2 ) + f ( 2 ) = 2 c . S i m i l a r l y , i t i s e a s y t o s e e

t h a t f ( 2

k

) = k c = c l o g

2

2

k

. W e w i l l e x t e n d t h i s t o i n t e g e r s t h a t a r e n o t p o w e r s o f 2 .

F o r a n y i n t e g e r m , l e t r > 0 , b e a n o t h e r i n t e g e r a n d l e t 2

k

m

r

< 2

k + 1

. T h e n b y

t h e m o n o t o n i c i t y a s s u m p t i o n o n f , w e h a v e

k c r f ( m ) < ( k + 1 ) c ( 2 . 4 6 )

o r

c

k

r

f ( m ) < c

k + 1

r

( 2 . 4 7 )

N o w b y t h e m o n o t o n i c i t y o f l o g , w e h a v e

k

r

l o g

2

m <

k + 1

r

( 2 . 4 8 )

C o m b i n i n g t h e s e t w o e q u a t i o n s , w e o b t a i n

f ( m ) ?

l o g

2

m

c

<

1

r

( 2 . 4 9 )

S i n c e r w a s a r b i t r a r y , w e m u s t h a v e

f ( m ) =

l o g

2

m

c

( 2 . 5 0 )

a n d w e c a n i d e n t i f y c = 1 f r o m t h e l a s t a s s u m p t i o n o f t h e l e m m a .

N o w w e a r e a l m o s t d o n e . W e h a v e s h o w n t h a t f o r a n y u n i f o r m d i s t r i b u t i o n o n m

o u t c o m e s , f ( m ) = H

m

( 1 = m ; : : : ; 1 = m ) = l o g

2

m .

W e w i l l n o w s h o w t h a t

H

2

( p ; 1 ? p ) = ? p l o g p ? ( 1 ? p ) l o g ( 1 ? p ) : ( 2 . 5 1 )

T o b e g i n , l e t p b e a r a t i o n a l n u m b e r , r = s , s a y . C o n s i d e r t h e e x t e n d e d g r o u p i n g a x i o m

f o r H

s

f ( s ) = H

s

(

1

s

; : : : ;

1

s

) = H (

1

s

; : : : ;

1

s

| { z }

r

;

s ? r

s

) +

s ? r

s

f ( s ? r ) ( 2 . 5 2 )

= H

2

(

r

s

;

s ? r

s

) +

s

r

f ( s ) +

s ? r

s

f ( s ? r ) ( 2 . 5 3 )

S u b s t i t u t i n g f ( s ) = l o g

2

s , e t c , w e o b t a i n

H

2

(

r

s

;

s ? r

s

) = ?

r

s

l o g

2

r

s

?

1 ?

s ? r

s

l o g

2

1 ?

s ? r

s

: ( 2 . 5 4 )

T h u s ( 2 . 5 1 ) i s t r u e f o r r a t i o n a l p . B y t h e c o n t i n u i t y a s s u m p t i o n , ( 2 . 5 1 ) i s a l s o t r u e a t

i r r a t i o n a l p .


9/32


1 5

T o c o m p l e t e t h e p r o o f , w e h a v e t o e x t e n d t h e d e n i t i o n f r o m H

2

t o H

m

, i . e . , w e h a v e

t o s h o w t h a t

H

m

( p

1

; : : : ; p

m

) = ?

X

p

i

l o g p

i

( 2 . 5 5 )

f o r a l l m . T h i s i s a s t r a i g h t f o r w a r d i n d u c t i o n . W e h a v e j u s t s h o w n t h a t t h i s i s t r u e f o r

m = 2 . N o w a s s u m e t h a t i t i s t r u e f o r m = n ? 1 . B y t h e g r o u p i n g a x i o m ,

H

n

( p

1

; : : : ; p

n

) = H

n ? 1

( p

1

+ p

2

; p

3

; : : : ; p

n

) ( 2 . 5 6 )

+ ( p

1

+ p

2

) H

2

p

1

p

1

+ p

2

;

p

2

p

1

+ p

2

( 2 . 5 7 )

= ? ( p

1

+ p

2

) l o g ( p

1

+ p

2

) ?

n

X

i = 3

p

i

l o g p

i

( 2 . 5 8 )

?

p

1

p

1

+ p

2

l o g

p

1

p

1

+ p

2

?

p

2

p

1

+ p

2

l o g

p

2

p

1

+ p

2

( 2 . 5 9 )

= ?

n

X

i = 1

p

i

l o g p

i

: ( 2 . 6 0 )

T h u s t h e s t a t e m e n t i s t r u e f o r m = n , a n d b y i n d u c t i o n , i t i s t r u e f o r a l l m . T h u s w e

h a v e n a l l y p r o v e d t h a t t h e o n l y s y m m e t r i c f u n c t i o n t h a t s a t i s e s t h e a x i o m s i s

H

m

( p

1

; : : : ; p

m

) = ?

m

X

i = 1

p

i

l o g p

i

: ( 2 . 6 1 )

T h e p r o o f a b o v e i s d u e t o R e n y i 1 0 ]

5 . E n t r o p y o f f u n c t i o n s o f a r a n d o m v a r i a b l e . L e t X b e a d i s c r e t e r a n d o m v a r i a b l e .

S h o w t h a t t h e e n t r o p y o f a f u n c t i o n o f X i s l e s s t h a n o r e q u a l t o t h e e n t r o p y o f X b y

j u s t i f y i n g t h e f o l l o w i n g s t e p s :

H ( X ; g ( X ) )

( a )

= H ( X ) + H ( g ( X ) j X ) ( 2 . 6 2 )

( b )

= H ( X ) ; ( 2 . 6 3 )

H ( X ; g ( X ) )

( c )

= H ( g ( X ) ) + H ( X j g ( X ) ) ( 2 . 6 4 )

( d )

H ( g ( X ) ) : ( 2 . 6 5 )

T h u s H ( g ( X ) ) H ( X ) :

S o l u t i o n : E n t r o p y o f f u n c t i o n s o f a r a n d o m v a r i a b l e .

( a ) H ( X ; g ( X ) ) = H ( X ) + H ( g ( X ) j X ) b y t h e c h a i n r u l e f o r e n t r o p i e s .

( b ) H ( g ( X ) j X ) = 0 s i n c e f o r a n y p a r t i c u l a r v a l u e o f X , g ( X ) i s x e d , a n d h e n c e

H ( g ( X ) j X ) =

P

x

p ( x ) H ( g ( X ) j X = x ) =

P

x

0 = 0 .

( c ) H ( X ; g ( X ) ) = H ( g ( X ) ) + H ( X j g ( X ) ) a g a i n b y t h e c h a i n r u l e .

( d ) H ( X j g ( X ) ) 0 , w i t h e q u a l i t y i X i s a f u n c t i o n o f g ( X ) , i . e . , g ( : ) i s o n e - t o - o n e .

H e n c e H ( X ; g ( X ) ) H ( g ( X ) ) .


10/32

1 6


C o m b i n i n g p a r t s ( b ) a n d ( d ) , w e o b t a i n H ( X ) H ( g ( X ) ) .

6 . Z e r o c o n d i t i o n a l e n t r o p y . S h o w t h a t i f H ( Y j X ) = 0 , t h e n Y i s a f u n c t i o n o f X , i . e . ,

f o r a l l x w i t h p ( x ) > 0 , t h e r e i s o n l y o n e p o s s i b l e v a l u e o f y w i t h p ( x ; y ) > 0 .

S o l u t i o n : Z e r o C o n d i t i o n a l E n t r o p y . A s s u m e t h a t t h e r e e x i s t s a n x , s a y x

0

a n d t w o

d i e r e n t v a l u e s o f y , s a y y

1

a n d y

2

s u c h t h a t p ( x

0

; y

1

) > 0 a n d p ( x

0

; y

2

) > 0 . T h e n

p ( x

0

) p ( x

0

; y

1

) + p ( x

0

; y

2

) > 0 , a n d p ( y

1

j x

0

) a n d p ( y

2

j x

0

) a r e n o t e q u a l t o 0 o r 1 .

T h u s

H ( Y j X ) = ?

X

x

p ( x )

X

y

p ( y j x ) l o g p ( y j x ) ( 2 . 6 6 )

p ( x

0

) ( ? p ( y

1

j x

0

) l o g p ( y

1

j x

0

) ? p ( y

2

j x

0

) l o g p ( y

2

j x

0

) ) ( 2 . 6 7 )

> > 0 ; ( 2 . 6 8 )

s i n c e ? t l o g t 0 f o r 0 t 1 , a n d i s s t r i c t l y p o s i t i v e f o r t n o t e q u a l t o 0 o r 1 .

T h e r e f o r e t h e c o n d i t i o n a l e n t r o p y H ( Y j X ) i s 0 i f a n d o n l y i f Y i s a f u n c t i o n o f X .

7 . P u r e r a n d o m n e s s a n d b e n t c o i n s . L e t X

1

; X

2

; : : : ; X

n

d e n o t e t h e o u t c o m e s o f i n d e -

p e n d e n t i p s o f a b e n t c o i n . T h u s P r f X

i

= : 1 g = p ; P r f X

i

= 0 g = 1 ? p ,

w h e r e p i s u n k n o w n . W e w i s h t o o b t a i n a s e q u e n c e Z

1

; Z

2

; : : : ; Z

K

o f f a i r c o i n

i p s f r o m X

1

; X

2

; : : : ; X

n

. T o w a r d t h i s e n d l e t f : X

n

! f 0 ; 1 g , w h e r e f 0 ; 1 g =

f ; 0 ; 1 ; 0 0 ; 0 1 ; : : : g i s t h e s e t o f a l l n i t e l e n g t h b i n a r y s e q u e n c e s , b e a m a p p i n g

f ( X

1

; X

2

; : : : ; X

n

) = ( Z

1

; Z

2

; : : : ; Z

K

) , w h e r e Z

i

B e r n o u l l i (

1

2

) , a n d K m a y d e p e n d

o n ( X

1

; : : : ; X

n

) . I n o r d e r t h a t t h e s e q u e n c e Z

1

; Z

2

; : : : a p p e a r t o b e f a i r c o i n i p s , t h e

m a p f f r o m b e n t c o i n i p s t o f a i r i p s m u s t h a v e t h e p r o p e r t y t h a t a l l 2

k

s e q u e n c e s

( Z

1

; Z

2

; : : : ; Z

k

) o f a g i v e n l e n g t h k h a v e e q u a l p r o b a b i l i t y ( p o s s i b l y 0 ) , f o r k = 1 ; 2 ; : : : .

F o r e x a m p l e , f o r n = 2 , t h e m a p f ( 0 1 ) = 0 , f ( 1 0 ) = 1 , f ( 0 0 ) = f ( 1 1 ) = ( t h e n u l l

s t r i n g ) , h a s t h e p r o p e r t y t h a t P r f Z

1

= 1 j K = 1 g = P r f Z

1

= 0 j K = 1 g =

1

2

.

G i v e r e a s o n s f o r t h e f o l l o w i n g i n e q u a l i t i e s :

n H ( p )

( a )

= H ( X

1

; : : : ; X

n

)

( b )

H ( Z

1

; Z

2

; : : : ; Z

K

; K )

( c )

= H ( K ) + H ( Z

1

; : : : ; Z

K

j K )

( d )

= H ( K ) + E ( K )

( e )

E K :

T h u s n o m o r e t h a n n H ( p ) f a i r c o i n t o s s e s c a n b e d e r i v e d f r o m ( X

1

; : : : ; X

n

) , o n t h e

a v e r a g e .

( f ) E x h i b i t a g o o d m a p f o n s e q u e n c e s o f l e n g t h 4 .

S o l u t i o n : P u r e r a n d o m n e s s a n d b e n t c o i n s .


11/32


1 7

n H ( p )

( a )

= H ( X

1

; : : : ; X

n

)

( b )

H ( Z

1

; Z

2

; : : : ; Z

K

)

( c )

= H ( Z

1

; Z

2

; : : : ; Z

K

; K )

( d )

= H ( K ) + H ( Z

1

; : : : ; Z

K

j K )

( e )

= H ( K ) + E ( K )

( f )

E K :

( a ) S i n c e X

1

; X

2

; : : : ; X

n

a r e i . i . d . w i t h p r o b a b i l i t y o f X

i

= 1 b e i n g p , t h e e n t r o p y

H ( X

1

; X

2

; : : : ; X

n

) i s n H ( p ) .

( b ) Z

1

; : : : ; Z

K

i s a f u n c t i o n o f X

1

; X

2

; : : : ; X

n

, a n d s i n c e t h e e n t r o p y o f a f u n c t i o n o f a

r a n d o m v a r i a b l e i s l e s s t h a n t h e e n t r o p y o f t h e r a n d o m v a r i a b l e , H ( Z

1

; : : : ; Z

K

)

H ( X

1

; X

2

; : : : ; X

n

) .

( c ) K i s a f u n c t i o n o f Z

1

; Z

2

; : : : ; Z

K

, s o i t s c o n d i t i o n a l e n t r o p y g i v e n Z

1

; Z

2

; : : : ; Z

K

i s 0 . H e n c e H ( Z

1

; Z

2

; : : : ; Z

K

; K ) = H ( Z

1

; : : : ; Z

K

) + H ( K j Z

1

; Z

2

; : : : ; Z

K

) =

H ( Z

1

; Z

2

; : : : ; Z

K

) :

( d ) F o l l o w s f r o m t h e c h a i n r u l e f o r e n t r o p y .

( e ) B y a s s u m p t i o n , Z

1

; Z

2

; : : : ; Z

K

a r e p u r e r a n d o m b i t s ( g i v e n K ) , w i t h e n t r o p y 1

b i t p e r s y m b o l . H e n c e

H ( Z

1

; Z

2

; : : : ; Z

K

j K ) =

X

k

p ( K = k ) H ( Z

1

; Z

2

; : : : ; Z

k

j K = k ) ( 2 . 6 9 )

=

X

k

p ( k ) k ( 2 . 7 0 )

= E K : ( 2 . 7 1 )

( f ) F o l l o w s f r o m t h e n o n - n e g a t i v i t y o f d i s c r e t e e n t r o p y .

( g ) S i n c e w e d o n o t k n o w p , t h e o n l y w a y t o g e n e r a t e p u r e r a n d o m b i t s i s t o u s e

t h e f a c t t h a t a l l s e q u e n c e s w i t h t h e s a m e n u m b e r o f o n e s a r e e q u a l l y l i k e l y . F o r

e x a m p l e , t h e s e q u e n c e s 0 0 0 1 , 0 0 1 0 , 0 1 0 0 a n d 1 0 0 0 a r e e q u a l l y l i k e l y a n d c a n b e u s e d

t o g e n e r a t e 2 p u r e r a n d o m b i t s . A n e x a m p l e o f a m a p p i n g t o g e n e r a t e r a n d o m

b i t s i s

0 0 0 0 !

0 0 0 1 ! 0 0 0 0 1 0 ! 0 1 0 1 0 0 ! 1 0 1 0 0 0 ! 1 1

0 0 1 1 ! 0 0 0 1 1 0 ! 0 1 1 1 0 0 ! 1 0 1 0 0 1 ! 1 1

1 0 1 0 ! 0 0 1 0 1 ! 1

1 1 1 0 ! 1 1 1 1 0 1 ! 1 0 1 0 1 1 ! 0 1 0 1 1 1 ! 0 0

1 1 1 1 !

( 2 . 7 2 )


12/32

1 8


T h e r e s u l t i n g e x p e c t e d n u m b e r o f b i t s i s

E K = 4 p q

3

2 + 4 p

2

q

2

2 + 2 p

2

q

2

1 + 4 p

3

q 2 ( 2 . 7 3 )

= 8 p q

3

+ 1 0 p

2

q

2

+ 8 p

3

q : ( 2 . 7 4 )

F o r e x a m p l e , f o r p

1

2

, t h e e x p e c t e d n u m b e r o f p u r e r a n d o m b i t s i s c l o s e t o 1 . 6 2 5 .

T h i s i s s u b s t a n t i a l l y l e s s t h e n t h e 4 p u r e r a n d o m b i t s t h a t c o u l d b e g e n e r a t e d i f

p w e r e e x a c t l y

1

2

.

W e w i l l n o w a n a l y z e t h e e c i e n c y o f t h i s s c h e m e o f g e n e r a t i n g r a n d o m b i t s f o r l o n g

s e q u e n c e s o f b e n t c o i n i p s . L e t n b e t h e n u m b e r o f b e n t c o i n i p s . T h e a l g o r i t h m

t h a t w e w i l l u s e i s t h e o b v i o u s e x t e n s i o n o f t h e a b o v e m e t h o d o f g e n e r a t i n g p u r e

b i t s u s i n g t h e f a c t t h a t a l l s e q u e n c e s w i t h t h e s a m e n u m b e r o f o n e s a r e e q u a l l y

l i k e l y .

C o n s i d e r a l l s e q u e n c e s w i t h k o n e s . T h e r e a r e

?

n

k

s u c h s e q u e n c e s , w h i c h a r e

a l l e q u a l l y l i k e l y . I f

?

n

k

w e r e a p o w e r o f 2 , t h e n w e c o u l d g e n e r a t e l o g

?

n

k

p u r e

r a n d o m b i t s f r o m s u c h a s e t . H o w e v e r , i n t h e g e n e r a l c a s e ,

?

n

k

i s n o t a p o w e r o f

2 a n d t h e b e s t w e c a n t o i s t h e d i v i d e t h e s e t o f

?

n

k

e l e m e n t s i n t o s u b s e t o f s i z e s

w h i c h a r e p o w e r s o f 2 . T h e l a r g e s t s e t w o u l d h a v e a s i z e 2

b l o g

(

n

k

)

c

a n d c o u l d b e

u s e d t o g e n e r a t e b l o g

?

n

k

c r a n d o m b i t s . W e c o u l d d i v i d e t h e r e m a i n i n g e l e m e n t s

i n t o t h e l a r g e s t s e t w h i c h i s a p o w e r o f 2 , e t c . T h e w o r s t c a s e w o u l d o c c u r w h e n

?

n

k

= 2

l + 1

? 1 , i n w h i c h c a s e t h e s u b s e t s w o u l d b e o f s i z e s 2

l

; 2

l ? 1

; 2

l ? 2

; : : : ; 1 .

I n s t e a d o f a n a l y z i n g t h e s c h e m e e x a c t l y , w e w i l l j u s t n d a l o w e r b o u n d o n n u m b e r

o f r a n d o m b i t s g e n e r a t e d f r o m a s e t o f s i z e

?

n

k

. L e t l = b l o g

?

n

k

c . T h e n a t l e a s t

h a l f o f t h e e l e m e n t s b e l o n g t o a s e t o f s i z e 2

l

a n d w o u l d g e n e r a t e l r a n d o m b i t s ,

a t l e a s t

1

4

t h b e l o n g t o a s e t o f s i z e 2

l ? 1

a n d g e n e r a t e l ? 1 r a n d o m b i t s , e t c . O n

t h e a v e r a g e , t h e n u m b e r o f b i t s g e n e r a t e d i s

E K j k 1 ' s i n s e q u e n c e ]

1

2

l +

1

4

( l ? 1 ) + +

1

2

l

1 ( 2 . 7 5 )

= l ?

1

4

1 +

1

2

+

2

4

+

3

8

+ +

l ? 1

2

l ? 2

( 2 . 7 6 )

l ? 1 ; ( 2 . 7 7 )

s i n c e t h e i n n i t e s e r i e s s u m s t o 1 .

H e n c e t h e f a c t t h a t

?

n

k

i s n o t a p o w e r o f 2 w i l l c o s t a t m o s t 1 b i t o n t h e a v e r a g e

i n t h e n u m b e r o f r a n d o m b i t s t h a t a r e p r o d u c e d .

H e n c e , t h e e x p e c t e d n u m b e r o f p u r e r a n d o m b i t s p r o d u c e d b y t h i s a l g o r i t h m i s

E K

n

X

k = 0

n

k

!

p

k

q

n ? k

b l o g

n

k

!

? 1 c ( 2 . 7 8 )

n

X

k = 0

n

k

!

p

k

q

n ? k

l o g

n

k

!

? 2

!

( 2 . 7 9 )

=

n

X

k = 0

n

k

!

p

k

q

n ? k

l o g

n

k

!

? 2 ( 2 . 8 0 )


13/32


1 9

X

n ( p ? ) k n ( p + )

n

k

!

p

k

q

n ? k

l o g

n

k

!

? 2 : ( 2 . 8 1 )

N o w f o r s u c i e n t l y l a r g e n , t h e p r o b a b i l i t y t h a t t h e n u m b e r o f 1 ' s i n t h e s e q u e n c e

i s c l o s e t o n p i s n e a r 1 ( b y t h e w e a k l a w o f l a r g e n u m b e r s ) . F o r s u c h s e q u e n c e s ,

k

n

i s c l o s e t o p a n d h e n c e t h e r e e x i s t s a s u c h t h a t

n

k

!

2

n ( H (

k

n

) ? )

2

n ( H ( p ) ? 2 )

( 2 . 8 2 )

u s i n g S t i r l i n g ' s a p p r o x i m a t i o n f o r t h e b i n o m i a l c o e c i e n t s a n d t h e c o n t i n u i t y o f

t h e e n t r o p y f u n c t i o n . I f w e a s s u m e t h a t n i s l a r g e e n o u g h s o t h a t t h e p r o b a b i l i t y

t h a t n ( p ? ) k n ( p + ) i s g r e a t e r t h a n 1 ? , t h e n w e s e e t h a t E K

( 1 ? ) n ( H ( p ) ? 2 ) ? 2 , w h i c h i s v e r y g o o d s i n c e n H ( p ) i s a n u p p e r b o u n d o n t h e

n u m b e r o f p u r e r a n d o m b i t s t h a t c a n b e p r o d u c e d f r o m t h e b e n t c o i n s e q u e n c e .

8 . W o r l d S e r i e s . T h e W o r l d S e r i e s i s a s e v e n - g a m e s e r i e s t h a t t e r m i n a t e s a s s o o n a s e i t h e r

t e a m w i n s f o u r g a m e s . L e t X b e t h e r a n d o m v a r i a b l e t h a t r e p r e s e n t s t h e o u t c o m e o f

a W o r l d S e r i e s b e t w e e n t e a m s A a n d B ; p o s s i b l e v a l u e s o f X a r e A A A A , B A B A B A B ,

a n d B B B A A A A . L e t Y b e t h e n u m b e r o f g a m e s p l a y e d , w h i c h r a n g e s f r o m 4 t o 7 .

A s s u m i n g t h a t A a n d B a r e e q u a l l y m a t c h e d a n d t h a t t h e g a m e s a r e i n d e p e n d e n t ,

c a l c u l a t e H ( X ) , H ( Y ) , H ( Y j X ) , a n d H ( X j Y ) .

S o l u t i o n :

W o r l d S e r i e s . T w o t e a m s p l a y u n t i l o n e o f t h e m h a s w o n 4 g a m e s .

T h e r e a r e 2 ( A A A A , B B B B ) W o r l d S e r i e s w i t h 4 g a m e s . E a c h h a p p e n s w i t h p r o b a b i l i t y

( 1 = 2 )

4

.

T h e r e a r e 8 = 2

?

4

3

W o r l d S e r i e s w i t h 5 g a m e s . E a c h h a p p e n s w i t h p r o b a b i l i t y ( 1 = 2 )

5

.

T h e r e a r e 2 0 = 2

?

5

3


6

.

T h e r e a r e 4 0 = 2

?

6

3


7

.

T h e p r o b a b i l i t y o f a 4 g a m e s e r i e s ( Y = 4 ) i s 2 ( 1 = 2 )

4

= 1 = 8 .

T h e p r o b a b i l i t y o f a 5 g a m e s e r i e s ( Y = 5 ) i s 8 ( 1 = 2 )

5

= 1 = 4 .

T h e p r o b a b i l i t y o f a 6 g a m e s e r i e s ( Y = 6 ) i s 2 0 ( 1 = 2 )

6

= 5 = 1 6 .

T h e p r o b a b i l i t y o f a 7 g a m e s e r i e s ( Y = 7 ) i s 4 0 ( 1 = 2 )

7

= 5 = 1 6 .

H ( X ) =

X

p ( x ) l o g

1

p ( x )

= 2 ( 1 = 1 6 ) l o g 1 6 + 8 ( 1 = 3 2 ) l o g 3 2 + 2 0 ( 1 = 6 4 ) l o g 6 4 + 4 0 ( 1 = 1 2 8 ) l o g 1 2 8

= 5 : 8 1 2 5


14/32

2 0


H ( Y ) =

X

p ( y ) l o g

1

p ( y )

= 1 = 8 l o g 8 + 1 = 4 l o g 4 + 5 = 1 6 l o g ( 1 6 = 5 ) + 5 = 1 6 l o g ( 1 6 = 5 )

= 1 : 9 2 4

Y i s a d e t e r m i n i s t i c f u n c t i o n o f X , s o i f y o u k n o w X t h e r e i s n o r a n d o m n e s s i n Y . O r ,

H ( Y j X ) = 0 .

S i n c e H ( X ) + H ( Y j X ) = H ( X ; Y ) = H ( Y ) + H ( X j Y ) , i t i s e a s y t o d e t e r m i n e

H ( X j Y ) = H ( X ) + H ( Y j X ) ? H ( Y ) = 3 : 8 8 9

9 . I n n i t e e n t r o p y . T h i s p r o b l e m s h o w s t h a t t h e e n t r o p y o f a d i s c r e t e r a n d o m v a r i a b l e c a n

b e i n n i t e . L e t A =

P

1

n = 2

( n l o g

2

n )

? 1

. ( I t i s e a s y t o s h o w t h a t A i s n i t e b y b o u n d i n g

t h e i n n i t e s u m b y t h e i n t e g r a l o f ( x l o g

2

x )

? 1

. ) S h o w t h a t t h e i n t e g e r - v a l u e d r a n d o m

v a r i a b l e X d e n e d b y P r ( X = n ) = ( A n l o g

2

n )

? 1

f o r n = 2 ; 3 ; : : : h a s H ( X ) = + 1 .

S o l u t i o n : I n n i t e e n t r o p y . B y d e n i t i o n , p

n

= P r ( X = n ) = 1 = A n l o g

2

n f o r n 2 .

T h e r e f o r e

H ( X ) = ?

1

X

n = 2

p ( n ) l o g p ( n )

= ?

1

X

n = 2

1 = A n l o g

2

n

l o g

1 = A n l o g

2

n

=

1

X

n = 2

l o g ( A n l o g

2

n )

A n l o g

2

n

=

1

X

n = 2

l o g A + l o g n + 2 l o g l o g n

A n l o g

2

n

= l o g A +

1

X

n = 2

1

A n l o g n

+

1

X

n = 2

2 l o g l o g n

A n l o g

2

n

:

T h e r s t t e r m i s n i t e . F o r b a s e 2 l o g a r i t h m s , a l l t h e e l e m e n t s i n t h e s u m i n t h e l a s t

t e r m a r e n o n n e g a t i v e . ( F o r a n y o t h e r b a s e , t h e t e r m s o f t h e l a s t s u m e v e n t u a l l y a l l

b e c o m e p o s i t i v e . ) S o a l l w e h a v e t o d o i s b o u n d t h e m i d d l e s u m , w h i c h w e d o b y

c o m p a r i n g w i t h a n i n t e g r a l .

1

X

n = 2

1

A n l o g n

>

Z

1

2

1

A x l o g x

d x = K l n l n x

1

2

= + 1 :

W e c o n c l u d e t h a t H ( X ) = + 1 .

1 0 . C o n d i t i o n a l m u t u a l i n f o r m a t i o n v s . u n c o n d i t i o n a l m u t u a l i n f o r m a t i o n . G i v e e x a m p l e s

o f j o i n t r a n d o m v a r i a b l e s X , Y a n d Z s u c h t h a t

( a ) I ( X ; Y j Z ) < I ( X ; Y ) ,


15/32


2 1

( b ) I ( X ; Y j Z ) > I ( X ; Y ) .

S o l u t i o n : C o n d i t i o n a l m u t u a l i n f o r m a t i o n v s . u n c o n d i t i o n a l m u t u a l i n f o r m a t i o n .

( a ) T h e l a s t c o r o l l a r y t o T h e o r e m 2 . 8 . 1 i n t h e t e x t s t a t e s t h a t i f X ! Y ! Z t h a t

i s , i f p ( x ; y j z ) = p ( x j z ) p ( y j z ) t h e n , I ( X ; Y ) I ( X ; Y j Z ) . E q u a l i t y h o l d s i f

a n d o n l y i f I ( X ; Z ) = 0 o r X a n d Z a r e i n d e p e n d e n t .

A s i m p l e e x a m p l e o f r a n d o m v a r i a b l e s s a t i s f y i n g t h e i n e q u a l i t y c o n d i t i o n s a b o v e

i s , X i s a f a i r b i n a r y r a n d o m v a r i a b l e a n d Y = X a n d Z = Y . I n t h i s c a s e ,

I ( X ; Y ) = H ( X ) ? H ( X j Y ) = H ( X ) = 1

a n d ,

I ( X ; Y j Z ) = H ( X j Z ) ? H ( X j Y ; Z ) = 0 :

S o t h a t I ( X ; Y ) > I ( X ; Y j Z ) .

( b ) T h i s e x a m p l e i s a l s o g i v e n i n t h e t e x t . L e t X ; Y b e i n d e p e n d e n t f a i r b i n a r y

r a n d o m v a r i a b l e s a n d l e t Z = X + Y . I n t h i s c a s e w e h a v e t h a t ,

I ( X ; Y ) = 0

a n d ,

I ( X ; Y j Z ) = H ( X j Z ) = 1 = 2 :

S o I ( X ; Y ) < I ( X ; Y j Z ) . N o t e t h a t i n t h i s c a s e X ; Y ; Z a r e n o t m a r k o v .

1 1 . A v e r a g e e n t r o p y . L e t H ( p ) = ? p l o g

2

p ? ( 1 ? p ) l o g

2

( 1 ? p ) b e t h e b i n a r y e n t r o p y

f u n c t i o n .

( a ) E v a l u a t e H ( 1 = 4 ) u s i n g t h e f a c t t h a t l o g

2

3 1 : 5 8 4 . H i n t : C o n s i d e r a n e x p e r i -

m e n t w i t h f o u r e q u a l l y l i k e l y o u t c o m e s , o n e o f w h i c h i s m o r e i n t e r e s t i n g t h a n t h e

o t h e r s .

( b ) C a l c u l a t e t h e a v e r a g e e n t r o p y H ( p ) w h e n t h e p r o b a b i l i t y p i s c h o s e n u n i f o r m l y

i n t h e r a n g e 0 p 1 .

( c ) ( O p t i o n a l ) C a l c u l a t e t h e a v e r a g e e n t r o p y H ( p

1

; p

2

; p

3

) w h e r e ( p

1

; p

2

; p

3

) i s a u n i -

f o r m l y d i s t r i b u t e d p r o b a b i l i t y v e c t o r . G e n e r a l i z e t o d i m e n s i o n n .

S o l u t i o n : A v e r a g e E n t r o p y .

( a ) W e c a n g e n e r a t e t w o b i t s o f i n f o r m a t i o n b y p i c k i n g o n e o f f o u r e q u a l l y l i k e l y

a l t e r n a t i v e s . T h i s s e l e c t i o n c a n b e m a d e i n t w o s t e p s . F i r s t w e d e c i d e w h e t h e r t h e

r s t o u t c o m e o c c u r s . S i n c e t h i s h a s p r o b a b i l i t y 1 = 4 , t h e i n f o r m a t i o n g e n e r a t e d

i s H ( 1 = 4 ) . I f n o t t h e r s t o u t c o m e , t h e n w e s e l e c t o n e o f t h e t h r e e r e m a i n i n g

o u t c o m e s ; w i t h p r o b a b i l i t y 3 = 4 , t h i s p r o d u c e s l o g

2

3 b i t s o f i n f o r m a t i o n . T h u s

H ( 1 = 4 ) + ( 3 = 4 ) l o g

2

3 = 2

a n d s o H ( 1 = 4 ) = 2 ? ( 3 = 4 ) l o g

2

3 = 2 ? ( : 7 5 ) ( 1 : 5 8 5 ) = 0 : 8 1 1 b i t s .


16/32

2 2


( b ) I f p i s c h o s e n u n i f o r m l y i n t h e r a n g e 0 p 1 , t h e n t h e a v e r a g e e n t r o p y ( i n

n a t s ) i s

?

Z

1

0

p l n p + ( 1 ? p ) l n ( 1 ? p ) d p = ? 2

Z

1

0

x l n x d x = ? 2

x

2

2

l n x +

x

2

4

!

1

0

=

1

2

:

T h e r e f o r e t h e a v e r a g e e n t r o p y i s

1

2

l o g

2

e = 1 = ( 2 l n 2 ) = : 7 2 1 b i t s .

( c ) C h o o s i n g a u n i f o r m l y d i s t r i b u t e d p r o b a b i l i t y v e c t o r ( p

1

; p

2

; p

3

) i s e q u i v a l e n t t o

c h o o s i n g a p o i n t ( p

1

; p

2

) u n i f o r m l y f r o m t h e t r i a n g l e 0 p

1

1 , p

1

p

2

1 .

T h e p r o b a b i l i t y d e n s i t y f u n c t i o n h a s t h e c o n s t a n t v a l u e 2 b e c a u s e t h e a r e a o f t h e

t r i a n g l e i s 1 / 2 . S o t h e a v e r a g e e n t r o p y H ( p

1

; p

2

; p

3

) i s

? 2

Z

1

0

Z

1

p

1

p

1

l n p

1

+ p

2

l n p

2

+ ( 1 ? p

1

? p

2

) l n ( 1 ? p

1

? p

2

) d p

2

d p

1

:

A f t e r s o m e e n j o y a b l e c a l c u l u s , w e o b t a i n t h e n a l r e s u l t 5 = ( 6 l n 2 ) = 1 : 2 0 2 b i t s .

1 2 . V e n n d i a g r a m s . U s i n g V e n n d i a g r a m s , w e c a n s e e t h a t t h e m u t u a l i n f o r m a t i o n c o m m o n

t o t h r e e r a n d o m v a r i a b l e s X , Y a n d Z s h o u l d b e d e n e d b y

I ( X ; Y ; Z ) = I ( X ; Y ) ? I ( X ; Y j Z ) :

T h i s q u a n t i t y i s s y m m e t r i c i n X , Y a n d Z , d e s p i t e t h e p r e c e d i n g a s y m m e t r i c d e -

n i t i o n . U n f o r t u n a t e l y , I ( X ; Y ; Z ) i s n o t n e c e s s a r i l y n o n n e g a t i v e . F i n d X , Y a n d Z

s u c h t h a t I ( X ; Y ; Z ) < 0 , a n d p r o v e t h e f o l l o w i n g t w o i d e n t i t i e s :

I ( X ; Y ; Z ) = H ( X ; Y ; Z ) ? H ( X ) ? H ( Y ) ? H ( Z ) + I ( X ; Y ) + I ( Y ; Z ) + I ( Z ; X )

I ( X ; Y ; Z ) = H ( X ; Y ; Z ) ? H ( X ; Y ) ? H ( Y ; Z ) ? H ( Z ; X ) + H ( X ) + H ( Y ) + H ( Z )

T h e r s t i d e n t i t y c a n b e u n d e r s t o o d u s i n g t h e V e n n d i a g r a m a n a l o g y f o r e n t r o p y a n d

m u t u a l i n f o r m a t i o n . T h e s e c o n d i d e n t i t y f o l l o w s e a s i l y f r o m t h e r s t .

S o l u t i o n : V e n n D i a g r a m s . T o s h o w t h e r s t i d e n t i t y ,

I ( X ; Y ; Z ) = I ( X ; Y ) ? I ( X ; Y j Z ) b y d e n i t i o n

= I ( X ; Y ) ? ( I ( X ; Y ; Z ) ? I ( X ; Z ) ) b y c h a i n r u l e

= I ( X ; Y ) + I ( X ; Z ) ? I ( X ; Y ; Z )

= I ( X ; Y ) + I ( X ; Z ) ? ( H ( X ) + H ( Y ; Z ) ? H ( X ; Y ; Z ) )

= I ( X ; Y ) + I ( X ; Z ) ? H ( X ) + H ( X ; Y ; Z ) ? H ( Y ; Z )

= I ( X ; Y ) + I ( X ; Z ) ? H ( X ) + H ( X ; Y ; Z ) ? ( H ( Y ) + H ( Z ) ? I ( Y ; Z ) )

= I ( X ; Y ) + I ( X ; Z ) + I ( Y ; Z ) + H ( X ; Y ; Z ) ? H ( X ) ? H ( Y ) ? H ( Z ) :

T o s h o w t h e s e c o n d i d e n t i t y , s i m p l y s u b s t i t u t e f o r I ( X ; Y ) , I ( X ; Z ) , a n d I ( Y ; Z )

u s i n g e q u a t i o n s l i k e

I ( X ; Y ) = H ( X ) + H ( Y ) ? H ( X ; Y ) :

T h e s e t w o i d e n t i t i e s s h o w t h a t I ( X ; Y ; Z ) i s a s y m m e t r i c ( b u t n o t n e c e s s a r i l y n o n n e g -

a t i v e ) f u n c t i o n o f t h r e e r a n d o m v a r i a b l e s .


17/32


2 3

1 3 . C o i n w e i g h i n g . S u p p o s e o n e h a s n c o i n s , a m o n g w h i c h t h e r e m a y o r m a y n o t b e o n e

c o u n t e r f e i t c o i n . I f t h e r e i s a c o u n t e r f e i t c o i n , i t m a y b e e i t h e r h e a v i e r o r l i g h t e r t h a n

t h e o t h e r c o i n s . T h e c o i n s a r e t o b e w e i g h e d b y a b a l a n c e .

( a ) F i n d a n u p p e r b o u n d o n t h e n u m b e r o f c o i n s n s o t h a t k w e i g h i n g s w i l l n d t h e

c o u n t e r f e i t c o i n ( i f a n y ) a n d c o r r e c t l y d e c l a r e i t t o b e h e a v i e r o r l i g h t e r .

( b ) ( D i c u l t ) W h a t i s t h e c o i n w e i g h i n g s t r a t e g y f o r k = 3 w e i g h i n g s a n d 1 2 c o i n s ?

S o l u t i o n : C o i n w e i g h i n g .

( a ) F o r n c o i n s , t h e r e a r e 2 n + 1 p o s s i b l e s i t u a t i o n s o r \ s t a t e s " .

O n e o f t h e n c o i n s i s h e a v i e r .

O n e o f t h e n c o i n s i s l i g h t e r .

T h e y a r e a l l o f e q u a l w e i g h t .

E a c h w e i g h i n g h a s t h r e e p o s s i b l e o u t c o m e s - e q u a l , l e f t p a n h e a v i e r o r r i g h t p a n

h e a v i e r . H e n c e w i t h k w e i g h i n g s , t h e r e a r e 3

k

p o s s i b l e o u t c o m e s a n d h e n c e w e

c a n d i s t i n g u i s h b e t w e e n a t m o s t 3

k

d i e r e n t \ s t a t e s " . H e n c e 2 n + 1 3

k

o r

n ( 3

k

? 1 ) = 2 .

L o o k i n g a t i t f r o m a n i n f o r m a t i o n t h e o r e t i c v i e w p o i n t , e a c h w e i g h i n g g i v e s a t m o s t

l o g

2

3 b i t s o f i n f o r m a t i o n . T h e r e a r e 2 n + 1 p o s s i b l e \ s t a t e s " , w i t h a m a x i m u m

e n t r o p y o f l o g

2

( 2 n + 1 ) b i t s . H e n c e i n t h i s s i t u a t i o n , o n e w o u l d r e q u i r e a t l e a s t

l o g

2

( 2 n + 1 ) = l o g

2

3 w e i g h i n g s t o e x t r a c t e n o u g h i n f o r m a t i o n f o r d e t e r m i n a t i o n o f

t h e o d d c o i n , w h i c h g i v e s t h e s a m e r e s u l t a s a b o v e .

( b ) T h e r e a r e m a n y s o l u t i o n s t o t h i s p r o b l e m . W e w i l l g i v e o n e w h i c h i s b a s e d o n t h e

t e r n a r y n u m b e r s y s t e m .

W e m a y e x p r e s s t h e n u m b e r s f ? 1 2 ; ? 1 1 ; : : : ; ? 1 ; 0 ; 1 ; : : : ; 1 2 g i n a t e r n a r y n u m b e r

s y s t e m w i t h a l p h a b e t f ? 1 ; 0 ; 1 g . F o r e x a m p l e , t h e n u m b e r 8 i s ( - 1 , 0 , 1 ) w h e r e

? 1 3

0

+ 0 3

1

+ 1 3

2

= 8 . W e f o r m t h e m a t r i x w i t h t h e r e p r e s e n t a t i o n o f t h e

p o s i t i v e n u m b e r s a s i t s c o l u m n s .

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

3

0

1 - 1 0 1 - 1 0 1 - 1 0 1 - 1 0

1

= 0

3

1

0 1 1 1 - 1 - 1 - 1 0 0 0 1 1

2

= 2

3

2

0 0 0 0 1 1 1 1 1 1 1 1

3

= 8

N o t e t h a t t h e r o w s u m s a r e n o t a l l z e r o . W e c a n n e g a t e s o m e c o l u m n s t o m a k e

t h e r o w s u m s z e r o . F o r e x a m p l e , n e g a t i n g c o l u m n s 7 , 9 , 1 1 a n d 1 2 , w e o b t a i n

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

3

0

1 - 1 0 1 - 1 0 - 1 - 1 0 1 1 0

1

= 0

3

1

0 1 1 1 - 1 - 1 1 0 0 0 - 1 - 1

2

= 0

3

2

0 0 0 0 1 1 - 1 1 - 1 1 - 1 - 1

3

= 0

N o w p l a c e t h e c o i n s o n t h e b a l a n c e a c c o r d i n g t o t h e f o l l o w i n g r u l e : F o r w e i g h i n g

# i , p l a c e c o i n n

O n l e f t p a n , i f n

i

= ? 1 .

A s i d e , i f n

i

= 0 .


18/32

2 4


O n r i g h t p a n , i f n

i

= 1 .

T h e o u t c o m e o f t h e t h r e e w e i g h i n g s w i l l n d t h e o d d c o i n i f a n y a n d t e l l w h e t h e r

i t i s h e a v y o r l i g h t . T h e r e s u l t o f e a c h w e i g h i n g i s 0 i f b o t h p a n s a r e e q u a l , - 1 i f

t h e l e f t p a n i s h e a v i e r , a n d 1 i f t h e r i g h t p a n i s h e a v i e r . T h e n t h e t h r e e w e i g h i n g s

g i v e t h e t e r n a r y e x p a n s i o n o f t h e i n d e x o f t h e o d d c o i n . I f t h e e x p a n s i o n i s t h e

s a m e a s t h e e x p a n s i o n i n t h e m a t r i x , i t i n d i c a t e s t h a t t h e c o i n i s h e a v i e r . I f

t h e e x p a n s i o n i s o f t h e o p p o s i t e s i g n , t h e c o i n i s l i g h t e r . F o r e x a m p l e , ( 0 , - 1 , - 1 )

i n d i c a t e s ( 0 ) 3

0

+ ( ? 1 ) 3 + ( ? 1 ) 3

2

= ? 1 2 , h e n c e c o i n # 1 2 i s h e a v y , ( 1 , 0 , - 1 ) i n d i c a t e s

# 8 i s l i g h t , ( 0 , 0 , 0 ) i n d i c a t e s n o o d d c o i n .

W h y d o e s t h i s s c h e m e w o r k ? I t i s a s i n g l e e r r o r c o r r e c t i n g H a m m i n g c o d e f o r t h e

t e r n a r y a l p h a b e t ( d i s c u s s e d i n S e c t i o n 8 . 1 1 i n t h e b o o k ) . H e r e a r e s o m e d e t a i l s .

F i r s t n o t e a f e w p r o p e r t i e s o f t h e m a t r i x a b o v e t h a t w a s u s e d f o r t h e s c h e m e .

A l l t h e c o l u m n s a r e d i s t i n c t a n d n o t w o c o l u m n s a d d t o ( 0 , 0 , 0 ) . A l s o i f a n y c o i n

i s h e a v i e r , i t w i l l p r o d u c e t h e s e q u e n c e o f w e i g h i n g s t h a t m a t c h e s i t s c o l u m n i n

t h e m a t r i x . I f i t i s l i g h t e r , i t p r o d u c e s t h e n e g a t i v e o f i t s c o l u m n a s a s e q u e n c e

o f w e i g h i n g s . C o m b i n i n g a l l t h e s e f a c t s , w e c a n s e e t h a t a n y s i n g l e o d d c o i n w i l l

p r o d u c e a u n i q u e s e q u e n c e o f w e i g h i n g s , a n d t h a t t h e c o i n c a n b e d e t e r m i n e d f r o m

t h e s e q u e n c e .

O n e o f t h e q u e s t i o n s t h a t m a n y o f y o u h a d w h e t h e r t h e b o u n d d e r i v e d i n p a r t ( a )

w a s a c t u a l l y a c h i e v a b l e . F o r e x a m p l e , c a n o n e d i s t i n g u i s h 1 3 c o i n s i n 3 w e i g h i n g s ?

N o , n o t w i t h a s c h e m e l i k e t h e o n e a b o v e . Y e s , u n d e r t h e a s s u m p t i o n s u n d e r

w h i c h t h e b o u n d w a s d e r i v e d . T h e b o u n d d i d n o t p r o h i b i t t h e d i v i s i o n o f c o i n s

i n t o h a l v e s , n e i t h e r d i d i t d i s a l l o w t h e e x i s t e n c e o f a n o t h e r c o i n k n o w n t o b e

n o r m a l . U n d e r b o t h t h e s e c o n d i t i o n s , i t i s p o s s i b l e t o n d t h e o d d c o i n o f 1 3 c o i n s

i n 3 w e i g h i n g s . Y o u c o u l d t r y m o d i f y i n g t h e a b o v e s c h e m e t o t h e s e c a s e s .

1 4 . D r a w i n g w i t h a n d w i t h o u t r e p l a c e m e n t . A n u r n c o n t a i n s r r e d , w w h i t e , a n d b b l a c k

b a l l s . W h i c h h a s h i g h e r e n t r o p y , d r a w i n g k 2 b a l l s f r o m t h e u r n w i t h r e p l a c e m e n t

o r w i t h o u t r e p l a c e m e n t ? S e t i t u p a n d s h o w w h y . ( T h e r e i s b o t h a h a r d w a y a n d a

r e l a t i v e l y s i m p l e w a y t o d o t h i s . )

S o l u t i o n : D r a w i n g w i t h a n d w i t h o u t r e p l a c e m e n t . I n t u i t i v e l y , i t i s c l e a r t h a t i f t h e

b a l l s a r e d r a w n w i t h r e p l a c e m e n t , t h e n u m b e r o f p o s s i b l e c h o i c e s f o r t h e i - t h b a l l i s

l a r g e r , a n d t h e r e f o r e t h e c o n d i t i o n a l e n t r o p y i s l a r g e r . B u t c o m p u t i n g t h e c o n d i t i o n a l

d i s t r i b u t i o n s i s s l i g h t l y i n v o l v e d . I t i s e a s i e r t o c o m p u t e t h e u n c o n d i t i o n a l e n t r o p y .

W i t h r e p l a c e m e n t . I n t h i s c a s e t h e c o n d i t i o n a l d i s t r i b u t i o n o f e a c h d r a w i s t h e

s a m e f o r e v e r y d r a w . T h u s

X

i

=

8

>

<

>

:

r e d w i t h p r o b .

r

r + w + b

w h i t e w i t h p r o b .

w

r + w + b

b l a c k w i t h p r o b .

b

r + w + b

( 2 . 8 3 )


H ( X

i

j X

i ? 1

; : : : ; X

1

) = H ( X

i

) ( 2 . 8 4 )


19/32


2 5

= l o g ( r + w + b ) ?

r

r + w + b

l o g r ?

w

r + w + b

l o g w ?

b

r + w + b

l o g b :( 2 . 8 5 )

W i t h o u t r e p l a c e m e n t . T h e u n c o n d i t i o n a l p r o b a b i l i t y o f t h e i - t h b a l l b e i n g r e d i s

s t i l l r = ( r + w + b ) , e t c . T h u s t h e u n c o n d i t i o n a l e n t r o p y H ( X

i

) i s s t i l l t h e s a m e a s

w i t h r e p l a c e m e n t . T h e c o n d i t i o n a l e n t r o p y H ( X

i

j X

i ? 1

; : : : ; X

1

) i s l e s s t h a n t h e

u n c o n d i t i o n a l e n t r o p y , a n d t h e r e f o r e t h e e n t r o p y o f d r a w i n g w i t h o u t r e p l a c e m e n t

i s l o w e r .

1 5 . A m e t r i c . A f u n c t i o n ( x ; y ) i s a m e t r i c i f f o r a l l x ; y ,

( x ; y ) 0

( x ; y ) = ( y ; x )

( x ; y ) = 0 i f a n d o n l y i f x = y

( x ; y ) + ( y ; z ) ( x ; z ) .

( a ) S h o w t h a t ( X ; Y ) = H ( X j Y ) + H ( Y j X ) h a s t h e a b o v e p r o p e r t i e s , a n d i s t h e r e f o r e

a m e t r i c . N o t e t h a t ( X ; Y ) i s t h e n u m b e r o f b i t s n e e d e d f o r X a n d Y t o

c o m m u n i c a t e t h e i r v a l u e s t o e a c h o t h e r .

( b ) V e r i f y t h a t ( X ; Y ) c a n a l s o b e e x p r e s s e d a s

( X ; Y ) = H ( X ) + H ( Y ) ? 2 I ( X ; Y ) ( 2 . 8 6 )

= H ( X ; Y ) ? I ( X ; Y ) ( 2 . 8 7 )

= 2 H ( X ; Y ) ? H ( X ) ? H ( Y ) : ( 2 . 8 8 )

S o l u t i o n : A m e t r i c

( a ) L e t

( X ; Y ) = H ( X j Y ) + H ( Y j X ) : ( 2 . 8 9 )

T h e n

S i n c e c o n d i t i o n a l e n t r o p y i s a l w a y s 0 , ( X ; Y ) 0 .

T h e s y m m e t r y o f t h e d e n i t i o n i m p l i e s t h a t ( X ; Y ) = ( Y ; X ) .

B y p r o b l e m 2 . 6 , i t f o l l o w s t h a t H ( Y j X ) i s 0 i Y i s a f u n c t i o n o f X a n d

H ( X j Y ) i s 0 i X i s a f u n c t i o n o f Y . T h u s ( X ; Y ) i s 0 i X a n d Y

a r e f u n c t i o n s o f e a c h o t h e r - a n d t h e r e f o r e a r e e q u i v a l e n t u p t o a r e v e r s i b l e

t r a n s f o r m a t i o n .

C o n s i d e r t h r e e r a n d o m v a r i a b l e s X , Y a n d Z . T h e n

H ( X j Y ) + H ( Y j Z ) H ( X j Y ; Z ) + H ( Y j Z ) ( 2 . 9 0 )

= H ( X ; Y j Z ) ( 2 . 9 1 )

= H ( X j Z ) + H ( Y j X ; Z ) ( 2 . 9 2 )

H ( X j Z ) ; ( 2 . 9 3 )


20/32

2 6


f r o m w h i c h i t f o l l o w s t h a t

( X ; Y ) + ( Y ; Z ) ( X ; Z ) : ( 2 . 9 4 )

N o t e t h a t t h e i n e q u a l i t y i s s t r i c t u n l e s s X ! Y ! Z f o r m s a M a r k o v C h a i n

a n d Y i s a f u n c t i o n o f X a n d Z .

( b ) S i n c e H ( X j Y ) = H ( X ) ? I ( X ; Y ) , t h e r s t e q u a t i o n f o l l o w s . T h e s e c o n d r e l a t i o n

f o l l o w s f r o m t h e r s t e q u a t i o n a n d t h e f a c t t h a t H ( X ; Y ) = H ( X ) + H ( Y ) ?

I ( X ; Y ) . T h e t h i r d f o l l o w s o n s u b s t i t u t i n g I ( X ; Y ) = H ( X ) + H ( Y ) ? H ( X ; Y ) .

1 6 . E x a m p l e o f j o i n t e n t r o p y . L e t p ( x ; y ) b e g i v e n b y

@

@

@

X

Y

0 1

0

1

3

1

3

1 0

1

3

F i n d

( a ) H ( X ) ; H ( Y ) :

( b ) H ( X j Y ) ; H ( Y j X ) :

( c ) H ( X ; Y ) :

( d ) H ( Y ) ? H ( Y j X ) :

( e ) I ( X ; Y ) .

( f ) D r a w a V e n n d i a g r a m f o r t h e q u a n t i t i e s i n ( a ) t h r o u g h ( e ) .

S o l u t i o n : E x a m p l e o f j o i n t e n t r o p y

( a ) H ( X ) =

2

3

l o g

3

2

+

1

3

l o g 3 = 0 : 9 1 8 b i t s = H ( Y ) .

( b ) H ( X j Y ) =

1

3

H ( X j Y = 0 ) +

2

3

H ( X j Y = 1 ) = 0 : 6 6 7 b i t s = H ( Y j X ) .

( c ) H ( X ; Y ) = 3

1

3

l o g 3 = 1 : 5 8 5 b i t s .

( d ) H ( Y ) ? H ( Y j X ) = 0 : 2 5 1 b i t s .

( e ) I ( X ; Y ) = H ( Y ) ? H ( Y j X ) = 0 : 2 5 1 b i t s .

( f ) S e e F i g u r e 1 .

1 7 . I n e q u a l i t y . S h o w l n x 1 ?

1

x

f o r x > 0 :

S o l u t i o n : I n e q u a l i t y . U s i n g t h e R e m a i n d e r f o r m o f t h e T a y l o r e x p a n s i o n o f l n ( x )

a b o u t x = 1 , w e h a v e f o r s o m e c b e t w e e n 1 a n d x

l n ( x ) = l n ( 1 ) +

1

t

t = 1

( x ? 1 ) +

? 1

t

2

t = c

( x ? 1 )

2

2

x ? 1


21/32


2 7

F i g u r e 2 . 1 : V e n n d i a g r a m t o i l l u s t r a t e t h e r e l a t i o n s h i p s o f e n t r o p y a n d r e l a t i v e e n t r o p y

H(X|Y) I(X;Y)H(Y|X)

H(Y)

H(X)

s i n c e t h e s e c o n d t e r m i s a l w a y s n e g a t i v e . H e n c e l e t t i n g y = 1 = x , w e o b t a i n

? l n y

1

y

? 1

o r

l n y 1 ?

1

y

w i t h e q u a l i t y i y = 1 .

1 8 . E n t r o p y o f a s u m . L e t X a n d Y b e r a n d o m v a r i a b l e s t h a t t a k e o n v a l u e s x

1

; x

2

; : : : ; x

r

a n d y

1

; y

2

; : : : ; y

s

, r e s p e c t i v e l y . L e t Z = X + Y :

( a ) S h o w t h a t H ( Z j X ) = H ( Y j X ) : A r g u e t h a t i f X ; Y a r e i n d e p e n d e n t , t h e n H ( Y )

H ( Z ) a n d H ( X ) H ( Z ) : T h u s t h e a d d i t i o n o f i n d e p e n d e n t r a n d o m v a r i a b l e s

a d d s u n c e r t a i n t y .

( b ) G i v e a n e x a m p l e ( o f n e c e s s a r i l y d e p e n d e n t r a n d o m v a r i a b l e s ) i n w h i c h H ( X ) >

H ( Z ) a n d H ( Y ) > H ( Z ) :

( c ) U n d e r w h a t c o n d i t i o n s d o e s H ( Z ) = H ( X ) + H ( Y ) ?

S o l u t i o n : E n t r o p y o f a s u m .

( a ) Z = X + Y . H e n c e p ( Z = z j X = x ) = p ( Y = z ? x j X = x ) .

H ( Z j X ) =

X

p ( x ) H ( Z j X = x )

= ?

X

x

p ( x )

X

z

p ( Z = z j X = x ) l o g p ( Z = z j X = x )

=

X

x

p ( x )

X

y

p ( Y = z ? x j X = x ) l o g p ( Y = z ? x j X = x )

=

X

p ( x ) H ( Y j X = x )

= H ( Y j X ) :

I f X a n d Y a r e i n d e p e n d e n t , t h e n H ( Y j X ) = H ( Y ) . S i n c e I ( X ; Z ) 0 ,

w e h a v e H ( Z ) H ( Z j X ) = H ( Y j X ) = H ( Y ) . S i m i l a r l y w e c a n s h o w t h a t

H ( Z ) H ( X ) .


22/32

2 8


( b ) C o n s i d e r t h e f o l l o w i n g j o i n t d i s t r i b u t i o n f o r X a n d Y L e t

X = ? Y =

(

1 w i t h p r o b a b i l i t y 1 = 2

0 w i t h p r o b a b i l i t y 1 = 2

T h e n H ( X ) = H ( Y ) = 1 , b u t Z = 0 w i t h p r o b . 1 a n d h e n c e H ( Z ) = 0 .

( c ) W e h a v e

H ( Z ) H ( X ; Y ) H ( X ) + H ( Y )

b e c a u s e Z i s a f u n c t i o n o f ( X ; Y ) a n d H ( X ; Y ) = H ( X ) + H ( Y j X ) H ( X ) +

H ( Y ) . W e h a v e e q u a l i t y i ( X ; Y ) i s a f u n c t i o n o f Z a n d H ( Y ) = H ( Y j X ) , i . e . ,

X a n d Y a r e i n d e p e n d e n t .

1 9 . E n t r o p y o f a d i s j o i n t m i x t u r e . L e t X

1

a n d X

2

b e d i s c r e t e r a n d o m v a r i a b l e s d r a w n

a c c o r d i n g t o p r o b a b i l i t y m a s s f u n c t i o n s p

1

( ) a n d p

2

( ) o v e r t h e r e s p e c t i v e a l p h a b e t s

X

1

= f 1 ; 2 ; : : : ; m g a n d X

2

= f m + 1 ; : : : ; n g : L e t

X =

(

X

1

; w i t h p r o b a b i l i t y ;

X

2

; w i t h p r o b a b i l i t y 1 ? :

( a ) F i n d H ( X ) i n t e r m s o f H ( X

1

) a n d H ( X

2

) a n d :

( b ) M a x i m i z e o v e r t o s h o w t h a t 2

H ( X )

2

H ( X

1

)

+ 2

H ( X

2

)

a n d i n t e r p r e t u s i n g t h e

n o t i o n t h a t 2

H ( X )

i s t h e e e c t i v e a l p h a b e t s i z e .

S o l u t i o n : E n t r o p y . W e c a n d o t h i s p r o b l e m b y w r i t i n g d o w n t h e d e n i t i o n o f e n t r o p y

a n d e x p a n d i n g t h e v a r i o u s t e r m s . I n s t e a d , w e w i l l u s e t h e a l g e b r a o f e n t r o p i e s f o r a

s i m p l e r p r o o f .

S i n c e X

1

a n d X

2

h a v e d i s j o i n t s u p p o r t s e t s , w e c a n w r i t e

X =

(

X

1

w i t h p r o b a b i l i t y

X

2

w i t h p r o b a b i l i t y 1 ?

D e n e a f u n c t i o n o f X ,

= f ( X ) =

(

1 w h e n X = X

1

2 w h e n X = X

2

T h e n a s i n p r o b l e m 1 , w e h a v e

H ( X ) = H ( X ; f ( X ) ) = H ( ) + H ( X j )

= H ( ) + p ( = 1 ) H ( X j = 1 ) + p ( = 2 ) H ( X j = 2 )

= H ( ) + H ( X

1

) + ( 1 ? ) H ( X

2

)

w h e r e H ( ) = ? l o g ? ( 1 ? ) l o g ( 1 ? ) .


23/32


2 9

2 0 . A m e a s u r e o f c o r r e l a t i o n . L e t X

1

a n d X

2

b e i d e n t i c a l l y d i s t r i b u t e d , b u t n o t n e c e s s a r i l y

i n d e p e n d e n t . L e t

= 1 ?

H ( X

2

j X

1

)

H ( X

1

)

:

( a ) S h o w =

I ( X

1

; X

2

)

H ( X

1

)

:

( b ) S h o w 0 1 :

( c ) W h e n i s = 0 ?

( d ) W h e n i s = 1 ?

S o l u t i o n : A m e a s u r e o f c o r r e l a t i o n . X

1

a n d X

2

a r e i d e n t i c a l l y d i s t r i b u t e d a n d

= 1 ?

H ( X

2

j X

1

)

H ( X

1

)

( a )

=

H ( X

1

) ? H ( X

2

j X

1

)

H ( X

1

)

=

H ( X

2

) ? H ( X

2

j X

1

)

H ( X

1

)

( s i n c e H ( X

1

) = H ( X

2

) )

=

I ( X

1

; X

2

)

H ( X

1

)

:

( b ) S i n c e 0 H ( X

2

j X

1

) H ( X

2

) = H ( X

1

) , w e h a v e

0

H ( X

2

j X

1

)

H ( X

1

)

1

0 1 :

( c ) = 0 i I ( X

1

; X

2

) = 0 i X

1

a n d X

2

a r e i n d e p e n d e n t .

( d ) = 1 i H ( X

2

j X

1

) = 0 i X

2

i s a f u n c t i o n o f X

1

. B y s y m m e t r y , X

1

i s a

f u n c t i o n o f X

2

, i . e . , X

1

a n d X

2

h a v e a o n e - t o - o n e r e l a t i o n s h i p .

2 1 . D a t a p r o c e s s i n g . L e t X

1

! X

2

! X

3

! ! X

n

f o r m a M a r k o v c h a i n i n t h i s o r d e r ;

i . e . , l e t

p ( x

1

; x

2

; : : : ; x

n

) = p ( x

1

) p ( x

2

j x

1

) p ( x

n

j x

n ? 1

) :

R e d u c e I ( X

1

; X

2

; : : : ; X

n

) t o i t s s i m p l e s t f o r m .

S o l u t i o n : D a t a P r o c e s s i n g . B y t h e c h a i n r u l e f o r m u t u a l i n f o r m a t i o n ,

I ( X

1

; X

2

; : : : ; X

n

) = I ( X

1

; X

2

) + I ( X

1

; X

3

j X

2

) + + I ( X

1

; X

n

j X

2

; : : : ; X

n ? 2

) : ( 2 . 9 5 )

B y t h e M a r k o v p r o p e r t y , t h e p a s t a n d t h e f u t u r e a r e c o n d i t i o n a l l y i n d e p e n d e n t g i v e n

t h e p r e s e n t a n d h e n c e a l l t e r m s e x c e p t t h e r s t a r e z e r o . T h e r e f o r e

I ( X

1

; X

2

; : : : ; X

n

) = I ( X

1

; X

2

) : ( 2 . 9 6 )


24/32

3 0


2 2 . B o t t l e n e c k . S u p p o s e a ( n o n - s t a t i o n a r y ) M a r k o v c h a i n s t a r t s i n o n e o f n s t a t e s , n e c k s

d o w n t o k < n s t a t e s , a n d t h e n f a n s b a c k t o m > k s t a t e s . T h u s X

1

! X

2

! X

3

,

X

1

2 f 1 ; 2 ; : : : ; n g , X

2

2 f 1 ; 2 ; : : : ; k g , X

3

2 f 1 ; 2 ; : : : ; m g .

( a ) S h o w t h a t t h e d e p e n d e n c e o f X

1

a n d X

3

i s l i m i t e d b y t h e b o t t l e n e c k b y p r o v i n g

t h a t I ( X

1

; X

3

) l o g k :

( b ) E v a l u a t e I ( X

1

; X

3

) f o r k = 1 , a n d c o n c l u d e t h a t n o d e p e n d e n c e c a n s u r v i v e s u c h

a b o t t l e n e c k .

S o l u t i o n :

B o t t l e n e c k .

( a ) F r o m t h e d a t a p r o c e s s i n g i n e q u a l i t y , a n d t h e f a c t t h a t e n t r o p y i s m a x i m u m f o r a

u n i f o r m d i s t r i b u t i o n , w e g e t

I ( X

1

; X

3

) I ( X

1

; X

2

)

= H ( X

2

) ? H ( X

2

j X

1

)

H ( X

2

)

l o g k :

T h u s , t h e d e p e n d e n c e b e t w e e n X

1

a n d X

3

i s l i m i t e d b y t h e s i z e o f t h e b o t t l e n e c k .

T h a t i s I ( X

1

; X

3

) l o g k .

( b ) F o r k = 1 , I ( X

1

; X

3

) l o g 1 = 0 a n d s i n c e I ( X

1

; X

3

) 0 , I ( X

1

; X

3

) = 0 .

T h u s , f o r k = 1 , X

1

a n d X

3

a r e i n d e p e n d e n t .

2 3 . R u n l e n g t h c o d i n g . L e t X

1

; X

2

; : : : ; X

n

b e ( p o s s i b l y d e p e n d e n t ) b i n a r y r a n d o m v a r i -

a b l e s . S u p p o s e o n e c a l c u l a t e s t h e r u n l e n g t h s R = ( R

1

; R

2

; : : : ) o f t h i s s e q u e n c e ( i n

o r d e r a s t h e y o c c u r ) . F o r e x a m p l e , t h e s e q u e n c e X = 0 0 0 1 1 0 0 1 0 0 y i e l d s r u n l e n g t h s

R = ( 3 ; 2 ; 2 ; 1 ; 2 ) . C o m p a r e H ( X

1

; X

2

; : : : ; X

n

) , H ( R ) a n d H ( X

n

; R ) . S h o w a l l

e q u a l i t i e s a n d i n e q u a l i t i e s , a n d b o u n d a l l t h e d i e r e n c e s .

S o l u t i o n : R u n l e n g t h c o d i n g . S i n c e t h e r u n l e n g t h s a r e a f u n c t i o n o f X

1

; X

2

; : : : ; X

n

,

H ( R ) H ( X ) . A n y X

i

t o g e t h e r w i t h t h e r u n l e n g t h s d e t e r m i n e t h e e n t i r e s e q u e n c e

X

1

; X

2

; : : : ; X

n

. H e n c e

H ( X

1

; X

2

; : : : ; X

n

) = H ( X

i

; R ) ( 2 . 9 7 )

= H ( R ) + H ( X

i

j R ) ( 2 . 9 8 )

H ( R ) + H ( X

i

) ( 2 . 9 9 )

H ( R ) + 1 : ( 2 . 1 0 0 )

2 4 . M a r k o v ' s i n e q u a l i t y f o r p r o b a b i l i t i e s . L e t p ( x ) b e a p r o b a b i l i t y m a s s f u n c t i o n . P r o v e ,

f o r a l l d 0 ,

P r f p ( X ) d g l o g

1

d

H ( X ) : ( 2 . 1 0 1 )


25/32


3 1

S o l u t i o n : M a r k o v i n e q u a l i t y a p p l i e d t o e n t r o p y .

P ( p ( X ) < d ) l o g

1

d

=

X

x : p ( x ) < d

p ( x ) l o g

1

d

( 2 . 1 0 2 )

X

x : p ( x ) < d

p ( x ) l o g

1

p ( x )

( 2 . 1 0 3 )

X

x

p ( x ) l o g

1

p ( x )

( 2 . 1 0 4 )

= H ( X ) ( 2 . 1 0 5 )

2 5 . L o g i c a l o r d e r o f i d e a s . I d e a s h a v e b e e n d e v e l o p e d i n o r d e r o f n e e d , a n d t h e n g e n e r a l i z e d

i f n e c e s s a r y . R e o r d e r t h e f o l l o w i n g i d e a s , s t r o n g e s t r s t , i m p l i c a t i o n s f o l l o w i n g :

( a ) C h a i n r u l e f o r I ( X

1

; : : : ; X

n

; Y ) , c h a i n r u l e f o r D ( p ( x

1

; : : : ; x

n

) j j q ( x

1

; x

2

; : : : ; x

n

) ) ,

a n d c h a i n r u l e f o r H ( X

1

; X

2

; : : : ; X

n

) .

( b ) D ( f j j g ) 0 , J e n s e n ' s i n e q u a l i t y , I ( X ; Y ) 0 .

S o l u t i o n : L o g i c a l o r d e r i n g o f i d e a s .

( a ) T h e f o l l o w i n g o r d e r i n g s a r e s u b j e c t i v e . S i n c e I ( X ; Y ) = D ( p ( x ; y ) j j p ( x ) p ( y ) ) i s a

s p e c i a l c a s e o f r e l a t i v e e n t r o p y , i t i s p o s s i b l e t o d e r i v e t h e c h a i n r u l e f o r I f r o m

t h e c h a i n r u l e f o r D .

S i n c e H ( X ) = I ( X ; X ) , i t i s p o s s i b l e t o d e r i v e t h e c h a i n r u l e f o r H f r o m t h e

c h a i n r u l e f o r I .

I t i s a l s o p o s s i b l e t o d e r i v e t h e c h a i n r u l e f o r I f r o m t h e c h a i n r u l e f o r H a s w a s

d o n e i n t h e n o t e s .

( b ) I n c l a s s , J e n s e n ' s i n e q u a l i t y w a s u s e d t o p r o v e t h e n o n - n e g a t i v i t y o f D . T h e

i n e q u a l i t y I ( X ; Y ) 0 f o l l o w e d a s a s p e c i a l c a s e o f t h e n o n - n e g a t i v i t y o f D .

2 6 . S e c o n d l a w o f t h e r m o d y n a m i c s . L e t X

1

; X

2

; X

3

: : : b e a s t a t i o n a r y r s t - o r d e r M a r k o v

c h a i n . I n S e c t i o n 2 . 9 , i t w a s s h o w n t h a t H ( X

n

j X

1

) H ( X

n ? 1

j X

1

) f o r n = 2 ; 3 : : : .

T h u s c o n d i t i o n a l u n c e r t a i n t y a b o u t t h e f u t u r e g r o w s w i t h t i m e . T h i s i s t r u e a l t h o u g h

t h e u n c o n d i t i o n a l u n c e r t a i n t y H ( X

n

) r e m a i n s c o n s t a n t . H o w e v e r , s h o w b y e x a m p l e

t h a t H ( X

n

j X

1

= x

1

) d o e s n o t n e c e s s a r i l y g r o w w i t h n f o r e v e r y x

1

.

S o l u t i o n : S e c o n d l a w o f t h e r m o d y n a m i c s .

H ( X

n

j X

1

) H ( X

n

j X

1

; X

2

) ( C o n d i t i o n i n g r e d u c e s e n t r o p y ) ( 2 . 1 0 6 )

= H ( X

n

j X

2

) ( b y M a r k o v i t y ) ( 2 . 1 0 7 )

= H ( X

n ? 1

j X

1

) ( b y s t a t i o n a r i t y ) ( 2 . 1 0 8 )

A l t e r n a t i v e l y , b y a n a p p l i c a t i o n o f t h e d a t a p r o c e s s i n g i n e q u a l i t y t o t h e M a r k o v c h a i n

X

1

! X

n ? 1

! X

n

, w e h a v e

I ( X

1

; X

n ? 1

) I ( X

1

; X

n

) : ( 2 . 1 0 9 )


26/32

3 2


E x p a n d i n g t h e m u t u a l i n f o r m a

chap2_solution manual _ elements of Information thoery,pdf

Documents

Transcript of chap2_solution manual _ elements of Information thoery,pdf