Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s...
Transcript of Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s...
![Page 1: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/1.jpg)
Introduction to Information Theory
1
![Page 2: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/2.jpg)
Claude E. Shannon 2
![Page 3: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/3.jpg)
3
![Page 4: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/4.jpg)
A General Communication SystemA General Communication System
4
![Page 5: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/5.jpg)
Shannon’s Information TheoryShannon s Information Theory
• Conceptualization of information & modeling• Conceptualization of information & modeling of information sourcesS di f i f ti th h l• Sending of information across the channel:What are the limits on the amount of information th t b t?that can be sent?What is the effect of noise on this communication. 5
![Page 6: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/6.jpg)
A General Communication SystemA General Communication System
6
![Page 7: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/7.jpg)
A General Communication SystemA General Communication System
CHANNEL
• Information Source• Transmitter• Channel• Receiver• Destination
7
![Page 8: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/8.jpg)
“Th f d l bl f i i“The fundamental problem of communication is that of reproducing at one point either
l i l l dexactly or approximately a message selected at another point.”
From: A Mathematical Theory of Communication, Shannon, 1948.
8
![Page 9: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/9.jpg)
Examples of Communication SystemsExamples of Communication Systems
9
![Page 10: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/10.jpg)
Motivating Noise…Motivating Noise…
XTransmittedSymbol
YReceivedSymbol
0 0
Symbol Symbol1 1
10
![Page 11: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/11.jpg)
Motivating Noise…Motivating Noise…
XTransmittedSymbol
YReceivedSymbol
0 0
Symbol Symbol1 1
P(Y=1|X=0) = fP(Y=0|X=1) = f
P(Y=0|X=0) = 1 ‐ fP(Y=1|X=1) = 1 ‐ f
11
![Page 12: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/12.jpg)
Motivating Noise…Motivating Noise…
f 0 1 10 000f = 0.1, n = ~10,000
1 f0 0
1 ‐ f
f
1 11 ‐ f
12
![Page 13: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/13.jpg)
QuestionQuestion
How can we achieve perfect communication over anHow can we achieve perfect communication over an imperfect, noisy communication channel?
Use more reliable components; Stabilize the environment; Use larger areas; Use power/cooling to reduce thermal noise.
These are all costly solutions.
13
![Page 14: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/14.jpg)
Alternately…Alternately…
How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?
Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER
Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems
Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems
14
![Page 15: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/15.jpg)
Alternately…Alternately…
How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?
Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER
REDUNDANCY IS KEY!
Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems
Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems
15
![Page 16: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/16.jpg)
Shannon’s Insight:Shannon s Insight:
High Reliability → Low Transmission Rateg e ab ty→ o a s ss o ate
I.e. Perfect reliability → Zero Transmission Ratey
For a given level of noise there is an associated rate gof transmission that can be achieved with arbitrarily good reliability.
e.g. Sending a lone T versus THIS…
16
![Page 17: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/17.jpg)
Meaning? What meaning?
“Frequently the messages havemeaning; that is
g g
Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual y p y pentities.
These semantic aspects of communication are irrelevant to the engineering problem.”g g p
From: A Mathematical Theory of Communication Shannon 1948From: A Mathematical Theory of Communication, Shannon, 1948.
17
![Page 18: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/18.jpg)
What is information?What is information?
• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.
GDI) σ is an instance of information, understood as semantic content, if and only if:
GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;
) h ll f d d f lGDI.3) the well‐formed data are meaningful.
18
![Page 19: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/19.jpg)
What is information?What is information?
• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.
GDI) σ is an instance of information, understood as semantic content, if and only if:
GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;
) h ll f d d f lGDI.3) the well‐formed data are meaningful.
19
![Page 20: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/20.jpg)
Shannon Information is…Shannon Information is…
• UncertaintyCan be measured by counting the number of possible messages.
• SurpriseSome messages are more likely than others.
• DifficultWhat is significant is the difficulty of transmitting the message from one point to the other.
E t• EntropyA fundamental measure of information.
20
![Page 21: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/21.jpg)
Understanding EntropyUnderstanding Entropy
l f d !It is sunny in California today!21
![Page 22: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/22.jpg)
Understanding EntropyUnderstanding Entropy Information is quantified using probabilities. Given a finite set of possible messages, associate a probability with
each message. A message with low probability represents more information than g p y p
one with high probability.
Definition of Information:Definition of Information:
I = log(1/p) ≡ ‐log(p)
Where p is the probability of the messageBase 2 is used for the logarithm so I is measured in bits
22
![Page 23: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/23.jpg)
Example: Information in a coin flipExample: Information in a coin flip
P(HEADS) = ½
I = ‐log(½) = 1 bit
23
![Page 24: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/24.jpg)
Example: Text AnalysisExample: Text Analysis
a 0.06428b 0.01147c 0.02413d 0.03188e 0.10210f 0.01842
0 01543
ab
c
dSPC
g 0.01543h 0.04313i 0.05767j 0.00082k 0.00514l 0.03338m 0.01959
e
fu
vw
x yz
n 0.05761o 0.06179p 0.01571q 0.00084r 0.04973s 0.05199t 0 07327
g
ht
u
t 0.07327u 0.02201v 0.00800w 0.01439x 0.00162y 0.01387z 0.00077
i
j
k
lm
nop
r
s
24
SPC 0.20096o
q
![Page 25: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/25.jpg)
Example Text AnalysisExample Text Analysis
25
![Page 26: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/26.jpg)
Example Text AnalysisExample Text AnalysisLetter Freq. I, h(pi)
a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0 05199 4 26552s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697
26
y 0.01387 6.17152z 0.00077 10.34877
SPC 0.20096 2.31502
![Page 27: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/27.jpg)
Definition of EntropyDefinition of Entropy
Information (I) is associated with knownInformation (I) is associated with known events/messagesEntropy (H) is the average information w r toEntropy (H) is the average information w.r.to all possible outcomes
27
![Page 28: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/28.jpg)
Example Text AnalysisExample Text AnalysisLetter Freq. I, h(pi)
a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697y 0.01387 6.17152
28
y 0.01387 6.17152z 0.00077 10.34877
SPC 0.20096 2.31502
![Page 29: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/29.jpg)
Entropy (2 outcomes)Entropy (2 outcomes)
29
![Page 30: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/30.jpg)
Entropy: PropertiesEntropy: Properties
Entropy is maximized if p is uniform. 30
![Page 31: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/31.jpg)
ReferencesReferences• Eugene Chiu, Jocelyn Lin, Brok Mcferron, Noshirwan Petigara, Satwiksai
Seshasai: Mathematical Theory of Claude Shannon: A study of the style and t t f hi k t th i f i f ti th MIT 6 933J /context of his work up to the genesis of information theory. MIT 6.933J /
STS.420J The Structure of Engineering Revolutions• Luciano Floridi, 2010: Information: A Very Short Introduction, Oxford
University Press, 2011.• Luciano Floridi, 2011: The Philosophy of Information, Oxford University Press,
2011.• James Gleick, 2011: The Information: A History, A Theory, A Flood, Pantheon
Books 2011Books, 2011.• David Luenberger, 2006: Information Science, Princeton University Press,
2006.• David J.C. MacKay, 2003: Information Theory, Inference, and Learning
Algorithms, Cambridge University Press, 2003.• Claude Shannon & Warren Weaver, 1949: The Mathematical Theory of
Communication, University of Illinois Press, 1949.• W N Francis and H Kucera: Brown University Standard Corpus of Present‐DayW. N. Francis and H. Kucera: Brown University Standard Corpus of Present Day
American English, Brown University, 1967.
31
![Page 32: Introduction to Information Theorydkumar/SoI/IntroToMTC-Part1.pdf · ShannonShannons’s Information Theory • Conceptualization of information & modeling ... • James Gleick, 2011:](https://reader030.fdocuments.in/reader030/viewer/2022041118/5f2f116f8e60ab788c2f9dfe/html5/thumbnails/32.jpg)
32