Lecture 5 Information Theory

8/3/2019 Lecture 5 Information Theory

http://slidepdf.com/reader/full/lecture-5-information-theory 1/51

Information Theory



- Bit rate is the number of bits per second.

- Baud rate is the number of signal units per second.

Baud rate is less than or equal to

the bit rate.



Example

An analog signal carries 4 bits in each

signal unit. If 1000 signal units are sent

per second.Find the baud rate and the bit rate.



Solution

Baud rate =Baud rate =

10001000 bauds per second (baud/s) bauds per second (baud/s) Bit rate = 1000 x 4 = 4000 bpsBit rate = 1000 x 4 = 4000 bps



Example

The bit rate of a signal is 3000. If

each signal unit carries 6 bits, what is

the baud rate?



Example

Baud rate = 3000 / 6 = 500 baud/sBaud rate = 3000 / 6 = 500 baud/s



Example

Find the minimum bandwidth for an ASK

system has a signal transmitting at 2000

bps. The transmission mode is half-duplex.



Solution

In ASK system the baud rate and bit rate

are the same. The baud rate is therefore

2000. An ASK signal requires a minimum bandwidth equal to its baud rate.

Therefore, the minimum bandwidth is

2000 Hz.



Solution

In ASK the baud rate is the same as the

bandwidth, which means the baud rate is

5000. But because the baud rate and the bit rate are also the same for ASK, the bit

rate is 5000 bps.



Quadrature amplitude modulation

( QAM) is a combination of ASK and

P SK so that a maximum contrast between each signal unit (bit, dibit,

tribit, and so on ) is achieved.



Bit and baud rate comparison

ModulationModulation UnitsUnits Bits/BaudBits/Baud Baud rateBaud rate Bit Rate

ASK, FSK, 2ASK, FSK, 2--PSK PSK Bit 1 N N

44--PSK, 4PSK, 4--QAMQAM Dibit 2 N 2N

88--PSK, 8PSK, 8--QAMQAM Tribit 3 N 3N

1616--QAMQAM Quadbit 4 N 4N

3232--QAMQAM Pentabit 5 N 5N

6464--QAMQAM Hexabit 6 N 6N

128128--QAMQAM Septabit 7 N 7N

256256--QAMQAM Octabit 8 N 8N



Information

What does a word information mean?

There is no some exact definition, however:

Information carries new specific knowledge,

which is definitely new for its recipient; Information is always carried by some specific

carrier in different forms (letters, digits, differentspecific symbols, sequences of digits, letters, and

symbols , etc.); Information is meaningful only if the recipient is

able to interpret it.



The information materialized is a message.

Information is always about something (size of aparameter, occurrence of an event, etc).

Viewed in this manner, information does not have

to be accurate; it may be a truth or a lie. Even a disruptive noise used to inhibit the flow of

communication and create misunderstandingwould in this view be a form of information.

However, generally speaking, if the amount of information in the received message increases,the message is more accurate.



Information

- More information means less

predications

- Less information means morepredications



Example

1. The sun will rise tomorrow

2. The FEU basketball team will win

against the Ateneo Basketball team

the next time they will play.

Question: Which sentence contains the most

information?



Information Theory

How we can measure the amount of

information?

How we can ensure the correctness of information?

What to do if information gets corrupted by

errors?

How much memory does it require to store

information?



Basic answers to these questions that formed

a solid background of the modern information

theory were given by the great American

mathematician, electrical engineer, and

computer scientist Claude E. Shannon in his

paper A Mathematical Theory of

Communication published in The BellSystem Technical Journal in October, 1948.



Claude Elwood Shannon (1916-2001)

The father of information theory

The father of practical digital circuit design theory

Bell Laboratories (1941-1972), MIT(1956-2001)

22



Information Content

What is the information content of any

message?

Shannons answer is: The information contentof a message consists simply of the number of

1s and 0s it takes to transmit it.

23



Information Content

Hence, the elementary unit of information is a

binary unit: a bit, which can be either 1 or 0;

true or false; yes or no, black and

white, etc.

One of the basic postulates of information

theory is that information can be treated like a

measurable physical quantity, such as densityor mass.

24



Information Content

Suppose you flip a coin one million times andwrite down the sequence of results. If you wantto communicate this sequence to another person,

how many bits will it take? If it's a fair coin, the two possible outcomes,

heads and tails, occur with equal probability.Therefore each flip requires 1 bit of information

to transmit. To send the entire sequence willrequire one million bits.

25



Information Content

Suppose the coin is biased so that headsoccur only 1/4 of the time, and tailsoccur 3/4. Then the entire sequence can

be sent in 811,300 bits, on average Thiswould seem to imply that each flip of the coin requires just 0.8113 bits totransmit.

How can you transmit a coin flip in lessthan one bit, when the only languageavailable is that of zeros and ones?

26



Obviously, you can't. But if the goal is totransmit an entire sequence of flips, and thedistribution is biased in some way, then you

can use your knowledge of the distribution toselect a more efficient code.

Another way to look at it is: a sequence of biased coin flips contains less "information"

than a sequence of unbiased flips, so it shouldtake fewer bits to transmit.



Information Content

Information Theory regards information as onlythose symbols that are uncertain to the receiver.

For years, people have sent telegraph messages,

leaving out non-essential words such as "a" and"the."

In the same vein, predictable symbols can be leftout, like in the sentence, "only information

essentially to understand must be tranmitted.Shannon made clear that uncertainty is the verycommodity of communication.

28



Information Content

Suppose we transmit a long sequence of onemillion bits corresponding to the first example.What should we do if some errors occur

during this transmission? If the length of the sequence to be

transmitted or stored is even larger that 1million bits, then 1 billion bits what shouldwe do?

29



Two main questions of

Information Theory

What to do if information gets corrupted by

errors?

How much memory does it require to storedata?

Both questions were asked and to a large

degree answered by Claude Shannon in his

1948 seminal article:

use error correction and data compression

30



Shannons basic principles of Information

Theory

Shannons theory told engineers how muchinformation could be transmitted over thechannels of an ideal system.

He also spelled out mathematically the principlesof data compression, which recognize what theend of this sentence demonstrates, that onlyinformation essentially to understand must betransmitted.

He also showed how we could transmitinformation over noisy channels at error rates wecould control.

31



32

Why is the Information Theory Important?

Thanks in large measure to Shannon's insights,

digital systems have come to dominate the world

of communications and information processing.

± Modems

± satellite communications

± Data storage

± Deep space communications

± Wireless technology



Information Theory

- Quantification of information



Channels

A channel is used to get informationacross:

Source

binary channel

0,1,1,0,0,1,1 Receiver

Many systems act like channels.

Some obvious ones: phone lines, Ethernet cables.

Less obvious ones: the air when speaking, TV screenwhen watching, paper when writing an article, etc.

All these are physical devices and hence prone to errors.

34



Noisy Channels

A noiseless binary channeltransmits bits without error:

0 0

1 1

A noisy, symmetric binary

channel applies a bit-flip

0m1 with probability p:

0 0

1 1

1p

1p

p

p

What to do if we have a noisy channel and

you want to send information across reliably?

35



Error Correction pre-Shannon

Primitive error correctionInstead of sending 0 and 1, send00 and 11.

The receiver takes the majority of the bit valuesas the intended value of the sender.

36



Channel Rate

When correcting errors, we have to be mindful of

the rate of the bits that you use to encode one bit

(in the previous example we had rate 1/3).

If we want to send data with arbitrarily small

errors,

then this requires arbitrarily low rates r, which iscostly.

37



Error Correction by Shannon

Shannons basic observations:

Correcting single bits is very wasteful andinefficient;

Instead we should correct blocks of bits.

38



A model for a Communication System

The communications systems are of a statistical

nature.

That is, the performance of the system can never

be described in a deterministic sense; it is always

given in statistical terms.

A source is a device that selects and transmits

sequences of symbols from a given alphabet. Each selection is made at random, although this

selection may be based on some statistical rule.

39




The channel transmits the incoming symbols

to the receiver. The performance of the

channel is also based on laws of chance.

If the source transmits a symbol A, with a

probability of P{A} and the channel lets

through the letter A with a probability

denoted by P{A|A}, then the probability of transmitting A and receiving A is P{A}·P{A|A}

40




The channel is generally lossy: a part of the

transmitted content does not reach its

destination or it reaches the destination in a

distorted form.

41




A very important task is the minimization of

the loss and the optimum recovery of the

original content when it is corrupted by the

effect of noise.

A method that is used to improve the

efficiency of the channel is called encoding.

An encoded message is less sensitive to noise.

42




Decoding is employed to transform the

encoded messages into the original form,

which is acceptable to the receiver.

Encoding: F : I F ( I )

Decoding: F -1: F ( I ) I

43



A Quantitative Measure of Information

Suppose we have to select some equipment

from a catalog which indicates n distinct

models:

The desired amount of information

associated with the selection of a particular

model must be a function of the

probability of choosing :

_ a1 2, ,..., n x x x

( )k I x

k x

k x

_ a ( )k k I x f P x!

44




If, for simplicity, we assume that each one of

these models is selected with an equal

probability, then the desired amount of

information is only a function of n:

1 1/k I x f n!

45




If each piece of equipment can be ordered in

one of m different colors and the selection of

colors is also equiprobable, then the amount

of information associated with the selection of

a color is :

_ a 2

1/ j j

I c f P c f m! !

jc

46




The selection may be done in two ways:

Select the equipment and then the color

independently of each other

Select the equipment and its color at the same

time as one selection from mn possible

choices: & 1 /k j

I x c f mn!

1 1& ( ) ( ) 1 / 1 /k j k j I x c I x I c f n f m! !

47




Since these amounts of information are identical,we obtain:

Among several solutions of this functionalequation, the most important for us is:

Thus, when a statistical experiment has neqiuprobable outcomes, the average amount of

information associated with an outcome is log n

log f x x!

1 / 1 / 1 / f n f m f mn !

48




The logarithmic information measure has thedesirable property of additivity forindependent statistical experiments.

The simplest case to consider is a selectionbetween two eqiuprobable events. Theamount of information associated with theselection of one out of two equiprobable

events is and provides aunit of information known as a bit.

2 2log 1/ 2 log 2 1 ! !

49



Information Entropy

- A key measure of measure

- Expressed by the average number of

bits needed for storage orcommunication



Entropy

- Quantifies the uncertainty involved in a

random variables.

Ex. Coin flip less entropy

Roll of die more entropy

Entropy units bits/second

- bits/symbol- unit less

Lecture 5 Information Theory

Documents

Transcript of Lecture 5 Information Theory