Sep 12 - 15 - Shannon Information

8/11/2019 Sep 12 - 15 - Shannon Information

1/27

Shannon InformationThe best way to say as little as possible


2/27

Information is the resolution of uncertainty.

Claude E. Shannon, 19


3/27

Learning Outcomes

1. Ability to explain qualitatively the amount of information experiment.

2. Ability to quantitatively calculate the Shannon informatioentropy given a probability distribution.

3. Describe three questions related to information that impacompression.


4/27

Anatomy of the Course


5/27

Three questions of compression

1.How much information is contained in a data file?

2.How can we compress this file?

3.What is the smallest file size possible (limit)?

Proposal:

These questions can only be answered if we define what isinformation

Claim:

You already have an intuition about what information is.


6/27

Your inner information detector (P)Youre an editor and three

stories come across yourdesk. Which one do youprint.

AClock strikes 12 at

noon!

BFlash flood in the

Sahara desert.

CUBC B-Line full filled to

capacity.

Least likely of the

three events.

Is information some how

to the probability of an ev


7/27

Bent Coin Ensemble

Experiment: Coin toss

Sample space:

Random Variable:

Probability:

{Heads, Tails}X

S

{ (Heads) 1, (Tails) 0}x X X

( 1) , ( 0) (1 )P x p P x p


8/27

0000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

You observe the following bent coin res

1 What is p? Probably zero

2

What is the information in this set of

outcomes? Probably zero

3 What is the compression algorithm? Dont send an

4 How much information is contained? Zero


9/27

0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0

0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0

How much informationdo we have for p=0.1?

What is a compressionstrategy?

A) Little B) Some C) Lots


10/27

Aside:Log / Exponential functions

log ( )b

x

x y

b y

Changing the base of a log functionlog

log

log1

loglog

log

ba

b

b

b

b

xx

a

xa

C x


11/27

Proposition: Shannon Information

21log s)

)( bit

(P xh x

Is this a reasonable

proposition and how would

you check?


12/27

Properties of Shannon Information Cont

Information is additive for independent random variables

( , ) ( ) ( ), iff ( , ) ( ) ( ) ,h x y h x h y p x y p x p y x y


13/27

What is the information of an experime

AKA: What is the information content of an Ensemble?

2

1log

(( ) ( ) (bits)

)Xx S

Px

xH X P


14/27

The weighing babies problem

1. Design a strategy to determine which is the odd baby and

is heavier or lighter in as few uses of the balance as possib

2. What is your first weighing distribution (# babies on left/ri

From lecture note

You are given 12 babi

equal in weight excepone that is either hea

or lighter.


15/27

How many weighings meetthe requirements? (P)

A 3-4

B 5-6

C 7-8

D 9-10E 11-12


16/27

What is your ensemble for baby weighin

Set of outcomes SX Random Variable

1

-1

0

Proba

How you

the babi


17/27

What is your first weighingdistribution? (P)

A 6v6

B 5v5

C 4v4

D 3v3E 2v2


18/27

What does the entropy term look like fo

X

2

1log( ) ( ) (bit

(s)

)p xH X x p x


19/27

What probability distribution of outcommaximizes the amount of information?

+

Under the constraint that ( ) 1Xx S

p x


20/27

Maximum amount of information for a c

( ( ))

( ( ))

p X Heads

p X Tails


21/27

Entropy:Average amount of information of a Random Var

Entropy is maximized if P(x) is uniform:

2log (| ) with equality iff ( ) 1/ |( ) | |X XS P x H X S

Choose distributions of babies that equalize the probability o


22/27

Modulo Operator I

What is the answer for 32 mod 5?

A 6

B 2

C 0.4


23/27

Modulo Operator II

The modulo operator work on

which numbers sets?

A Imaginary

B Integer

C Real

D All of above

E Dont know


24/27

The game of 63

There is a set of 64 numbers:

{0,1,2,...,63}x

I secretly pick one number from the set and you have to guesmay ask me any yes/no question you like.

1. What strategy would you use in asking questions?

2. What is the minimum number of questions you need to aThe answer is:

6


25/27

Significance of game of 63

The game 63 shows:

1. Numbers can be represented by a code c(x) of 0s and 1s2. The code c(x) has the maximum amount of Shannon infor

content.

Together with baby weighing, the game of 63 is another (wea

example to support the proposition that the Shannon informadefinition is how we should represent information.


26/27

Summary I

Proposed two definitions:

a) (Shannon) information defined by the probability of an ou

b) (Shannon) entropy to quantify the average information in outcomes:

2

1log s)

)( bit

(P xh x

2

1log

(( ) ( ) (bits)

)Xx S

Px

xH X P


27/27

Summary II

Proposed the following properties about Shannon informatio

a) The Shannon information of a joint ensemble of independequal to the sum of the individual r.v. information

b) The Shannon entropy is maximized when the probabilitiesoutcomes are equal

Carried out some examples to convince you of the validity of Shannon information / entropy definitions.

2log (| ) with equality iff ( )( ) 1/ || |X XS P x SH X

( , ) ( ) ( ), iff ( , ) ( ) ( ) ,h x y h x h y p x y p x p y x y

Sep 12 - 15 - Shannon Information

Documents

Transcript of Sep 12 - 15 - Shannon Information