DIGITAL SYSTEMS Number systems & Arithmetic Rudolf Tracht and A.J. Han Vinck.
Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen...
-
Upload
helena-gray -
Category
Documents
-
view
216 -
download
2
Transcript of Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen...
![Page 1: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/1.jpg)
Information theory Information theory in the in the
Modern Information SocietyModern Information Society
A.J. Han VinckUniversity of Duisburg/Essen
January [email protected]
![Page 2: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/2.jpg)
contentcontent
What is Information theory ?
Why is it important ?
Where do we find it ?
![Page 3: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/3.jpg)
Claude Elwood Shannon: April 30, 1916 - February 24, 2001
Shannon (1948) , Information theory, The Mathematical theory of CommunicationShannon (1948) , Information theory, The Mathematical theory of Communication
![Page 4: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/4.jpg)
What is Information theory What is Information theory about ?about ?
Information:Information: knowledge that can be usedknowledge that can be used
Communication:Communication: exchange of Information exchange of Information
Our goal:Our goal: efficient; reliable; secureefficient; reliable; secure
![Page 5: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/5.jpg)
Express everything in 0 and 1
Discrete ensemble:
a,b,c,d 00, 01, 10, 11
in general: k binary digits specify 2k
messages
Analogue signal:
1) sample and 2) represent sample value binary
11
10
01
00t
v
Output
00, 10, 01, 01, 11
![Page 6: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/6.jpg)
Shannon‘s contributionsShannon‘s contributions
Modeling:Modeling: how to go from analogue to digital• fundamental communication models
Bounds:Bounds: how far can we go?• achievability• impossibility
Constructions:Constructions: constructive communication methods
• with optimum performance
and many more!!!and many more!!!
1011
R
P
![Page 7: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/7.jpg)
efficient:efficient: general problem statement
remove redundancy exact, no errors !!
remove irrelevance distortion !!
Topics: how ? how good ?how fast ? how
complex ?+ + +
![Page 8: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/8.jpg)
efficient: efficient: text
represent every symbol with 8 bit
1 book: 8 * (500 pages) * 1000 symbols = 4 Mbit 1 book
compression possible to 1 Mbit (1:4)
![Page 9: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/9.jpg)
efficient: efficient: speech
sampling speed 8000 samples/sec; accuracy 8 bits/sample;
speed 64 kBit/s;
45 minutes lecture = 45*60*64k =180Mbit 45 books
compression possible to 4.8 kBit/s (1:10)
![Page 10: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/10.jpg)
efficient: efficient: CD music
sampling speed 44.1 k samples/sec; accuracy 16 bits/sample
storage capacity for one hour stereo: 5 Gbit 1250 books
compression possible to 4 bits/sample ( 1:4 )
![Page 11: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/11.jpg)
efficient: efficient: digital pictures
300 x 400 pixels x 3 colors x 8 bit/sample
2.9 Mbit/picture; for 25 images/second we need 75 Mb/s
2 hour pictures need 540 Gbit 130.000 books
compression needed (1:100)
![Page 12: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/12.jpg)
efficient: summaryefficient: summarytext:
1 book storage: = 4 Mbit 1 book
speech:
45 minutes lecture = 45*60*64k =180Mbit 45 books
CD music:
storage capacity for one hour stereo: 5 Gbit 1250 books
digital pictures:
2 hour pictures need 540 Gbit 130.000 books
![Page 13: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/13.jpg)
efficient: general idea• represent likely symbols with short length binary words
where likely is derived from
- prediction of next symbol in source output
- context between the source symbols
words sounds context in pictures
q qu
q-ue, q-ua, q-ui, q-uo
![Page 14: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/14.jpg)
Morse
![Page 15: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/15.jpg)
efficient: applications
Text: Zip; GIF etc.
Music: MP3
Pictures: JPEG, MPEG
Contributors in data reduction/compression:
Information theorists: A. Lempel and Jacob Ziv
: Huffman a.m.m.
![Page 16: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/16.jpg)
efficient: example JPEG
15.817MB 4.566MB 3.267MB 2.351MB
![Page 17: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/17.jpg)
Secure:Secure: example 1 A wants to send a present to B in a safe way.
USER A USER B send box with open lock to A send locked box to B B has private key
Lock closes without key!
Problem: Is B the owner of the open lock?
![Page 18: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/18.jpg)
Secure:Secure: classical A wants to send a present to B in a safe way.
USER A USER B send key to B send locked box to B B has key
Lock closed
Problem: Is the key present at B?
![Page 19: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/19.jpg)
Secure:Secure: example 2Diffie Hellman key exchange. common paint + + + secret color A + secret color B = = assume that mixture separation is expensive public transport + + + secret color A + secret color B = =
common secret
![Page 20: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/20.jpg)
Reliable:Reliable:
Transmit
0 or 1
Receive
0 or 1
0 0 correct
0 1 in - correct
1 1 correct
1 0 in - correct
What What can can we we do do about about it ?it ?
![Page 21: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/21.jpg)
Reliable:Reliable: 2 examples2 examples
Transmit
A: = 0 0
B: = 1 1
Receive
0 0 or 1 1 OK
0 1 or 1 0 NOK1 error detected!A: = 0 0 0
B: = 1 1 1
000, 001, 010, 100 A
111, 110, 101, 011 B
1 error corrected!
![Page 22: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/22.jpg)
Error Sensitivity: Illustration
Error sensitivity: 0.0001=0.01% Error sensitivity: 0.0005=0.05%
![Page 23: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/23.jpg)
Optical StorageOptical Storage
•DVD's seven-fold increase in data capacity over the CD has been largely achieved by tightening up the tolerances throughout the predecessor system
•The data structure was made more efficient by using a better, more efficient error correction code system.
![Page 24: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/24.jpg)
Errors in networkingErrors in networking
![Page 25: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/25.jpg)
no- commentno- comment
![Page 26: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/26.jpg)
a meshed structure
3 links down
partial
FundamentalFundamental problems to consider
fast re-routing of information
how to include redundancy ?
how much redundancy?
Cost versus reliabilityCost versus reliability
![Page 27: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/27.jpg)
The success story Qualcomm CDMAThe success story Qualcomm CDMA
Founding information theorists:
Irwin Jacobs and Andrew Viterbi
![Page 28: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/28.jpg)
narrow-band and broad-band noise
SOLUTIONSOLUTION: FREQUENCY and TIME DIVISION
1
2
3
time
frequency
![Page 29: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/29.jpg)
PPM Code Example
• 6 code words: 123 231 312 132 213 321
• distance: = 2
1
2
3
![Page 30: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/30.jpg)
Code Division Multiple Access (CDMA)
• 6 users: 123 231 312 132 213 321– 1 2 3 4 5 6
• 2 transmit at the same time
1
2
32/3 2/3 2/3
5/6 5
1/2 2/4 2/4
4/5
2/5 2 2
![Page 31: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/31.jpg)
Why IT at this Why IT at this university?university?
• It is fundamental. The theory is well established
• Based on– Discrete Mathematics; algorithms– Physics
• Applications:– Communications; networking; Computer science– Multi-media; medical imaging; biology; languages– Information retrieval; information control
![Page 32: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/32.jpg)
Other application:
powerline communications
![Page 33: Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de.](https://reader030.fdocuments.in/reader030/viewer/2022032313/56649e735503460f94b734b0/html5/thumbnails/33.jpg)
Information TheoryInformation Theory
In 1948, Bell Labs scientist Claude Shannon developed Information Theory, and the world of communications
technology has never been the same.