Towards a theory of semantic communication
-
date post
12-Sep-2014 -
Category
Technology
-
view
3.980 -
download
9
description
Transcript of Towards a theory of semantic communication
![Page 1: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/1.jpg)
Towards a Theory of Semantic Communication
Jie Bao, RPI
Joint work with Prithwish Basu, Mike Dean, Craig Partridge, Ananthram
Swami, Will Leland and Jim Hendler 1
![Page 2: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/2.jpg)
Outline
• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
2
![Page 3: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/3.jpg)
Shannon, 1948
“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;... These semantic aspects of communication are irrelevant to the engineering problem.”
3Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 625-56, 1948.
message
message
Signal
Signal
![Page 4: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/4.jpg)
But, are these just sequences of bits?
• Movie streams• Software codes• DNA sequences• Emails• Tweets• ……
4
“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;..”“These semantic aspects of communication are irrelevant to the engineering problem”?
![Page 5: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/5.jpg)
Between a Talent Manager & Me
“Are you open to discuss greener pastures”?
5
“Thanks for contacting me. However, I'm not sure if my research is related to "greener pastures". I'm a computer scientist.”
![Page 6: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/6.jpg)
Misunderstanding can be costly
6
Mars Climate Orbiter (1998-1999), $125 million
Expressed
Pound (lbF)
Interpreted
Newton (N)
Image Source: Wikipedia, http://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Communications_loss
![Page 7: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/7.jpg)
Misunderstanding can be deadly
Afghan National Army (ANA) to ISAF• “Launch flares over the left side of the village”
Received and Understood as• “fire on the left side of the village”
Alternative semantic coding (e.g., illuminating shell) may save lives!
7Scenario based on report from http://www.closeprotectionworld.co.uk/security-news-asia/37466-afghanistan-war-what-happens-when-war-interpreter-doesnt-know-language.html
(Noisy) Battlefield Communication (Noisy) Battlefield Communication
![Page 8: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/8.jpg)
Our Contributions
• We develop a generic model of semantic communication, extending the classic model-theoretical work of (Carnap and Bar-Hillel 1952) ;
• We discuss the role of semantics in reducing source redundancy, and potential approaches for lossless and lossy semantic data compression;
• We define the notions of semantic noise, semantic channel, and obtain the semantic capacity of a channel.
![Page 9: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/9.jpg)
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
9
![Page 10: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/10.jpg)
(Classical) Information Theory Semantic Information Theory
Shannon, 1948
message
message
Shannon ModelShannon Model
Signal
Signal
ExpressedMessage(e.g., commands and reports)
Expressed Message
Semantic Channel
From IT to SIT
![Page 11: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/11.jpg)
A Three-level Model (Weaver)
Transmitter Receiver
Destination Destination Source Source
Physical Channel
Technical message
Technical Noise
Intended message
Expressed message
Semantic Transmitter
Semantic Transmitter
Semantic ReceiverSemantic Receiver
Semantic Noise
Semantic Noise
Shared knowledge
Shared knowledge
Local knowledge
Local knowledge
Local knowledge
Local knowledge
(effectiveness factors)
C: Effectiveness
B: Semantic
A: Technical
![Page 12: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/12.jpg)
A Semantic Communication Model
12
Message generator
World model
Background Knowledge
Inference Procedure
Messages
Sender
Message interpreter
World model
Background Knowledge
Inference Procedure
Receiver
Ws Wr
Ks KrIs Ir
{m}
World
M: Message Syntax
Feedback (?)
observations
Ms Mr
![Page 13: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/13.jpg)
Semantic Sources
• Key: A semantic source tells something that is “true”– Engineering bits are neither true or false!
• Goal: 1) more soundness (sent as “true”->received as “true”); 2) less ambiguity
13
![Page 14: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/14.jpg)
Outline
• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
14
![Page 15: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/15.jpg)
Measuring Semantic Information
• Basic Problem: What is the amount of “semantics” carried by a source and its messages?
15
![Page 16: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/16.jpg)
Measuring Semantic Information
• Statistical approach: Inference may change the distribution of symbols, hence the entropy of the source.
• Model-theoretical approach: The less “likely” a message is to be true, the more information it contains.
• Algorithmic approach: What’s the minimal program needed to describe messages and their deductions?
• Situation-theoretical approach: measuring the divergence of messages to “truth”.
16
Our ApproachOur Approach
![Page 17: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/17.jpg)
Shannon: Information = “surpriseness”
17
H(tyrannosaurus) > H(dog)H(tyrannosaurus) > H(dog)
Captured from: http://www.wordcount.org/main.php
![Page 18: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/18.jpg)
Which sentence is more “surprising”?
18
``Rex is not a tyrannosaurus''
``Rex is not a dog''
![Page 19: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/19.jpg)
????
Model Semantics
• tyrannosaurus • dog
19
??
![Page 20: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/20.jpg)
“Semantics” of DNA
20Image courtesy: http://www.yourdictionary.com/dna http://www.pnl.gov/biology/images/protein_molecule.jpg
“Syntax” Model (“Semantics”)
Gene expression
![Page 21: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/21.jpg)
Stone-age Semantic Communication
• Semantic communication predates symbolic communications
21Altamira Cave Painting http://mandyking.files.wordpress.com/2011/02/altamira-cave.jpg
![Page 22: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/22.jpg)
Semantics of Messages
• Messages are expressions, not just sequences of symbols– E.g., Saturday->Weekend, Sunny & Cold
• If an expression is more commonly true, it contains less semantic information– inf (Sunny & Cold) > inf (Cold)– inf (Cold) > inf (Cold or Warm)
22
![Page 23: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/23.jpg)
Semantics of Messages
• Carnap & Bar-Hillel (1952) - “An outline of a theory of semantic information”
m(exp) = |mod(exp)| / |all models|
inf(exp) = - log m(exp)
• Example– m(A v B) = ¾, m(A ^ B)=1/4– Inf(A v B)=0.415, inf(A^B )= 2
23
![Page 24: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/24.jpg)
Knowledge Entropy
• Extending Carnap & Bar-Hillel (1952) – Models have a distribution– Background knowledge may present
Weekend=2/7, Saturday=1/7
![Page 25: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/25.jpg)
Knowledge Entropy
• Logical prob. and knowledge entropy of Messages
• Model entropy of an information source
25
model distribution
logical probability
![Page 26: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/26.jpg)
Semantic Information Calculator (Demo)
• http://www.cs.rpi.edu/~baojie/sit/index.php
![Page 27: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/27.jpg)
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
27
![Page 28: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/28.jpg)
Conditional Knowledge Entropy
• When there is background knowledge, the set of possible worlds decreases.
28
![Page 29: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/29.jpg)
Model Compression with Shared Knlg
• Background knowledge (A->B), when shared, help compress the source– Side information in the form of entailment
29
![Page 30: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/30.jpg)
Lossless Message Compression
• Theorem : There is a semantically lossless code for source X, with message entropy H >= H(Xeq); no such code exists for H < H(Xeq)
– Xeq are equivalent classes of X
• Example: no need for coding both “pig” and “swine”, using one of them is sufficient.
• Example 2: a->(a^b)v(b^c) = a->b• Sometime, the loss is intentional compression
– Textual description of an image– Abstract of a paper
![Page 31: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/31.jpg)
Other Source Coding Strategies
• Lossless model compression– E.g., using minimal models
• Lossy message compression– E.g., compressing based on semantic similarity
• Leave as future work
31
![Page 32: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/32.jpg)
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
32
![Page 33: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/33.jpg)
Semantic Noise
Examples
• The meaning of a message is changed due to technical noises, e.g., from ``flare'' to ``fire'‘;
• Semantic mismatch: The source / receiver use different background knowledge or inference (e.g., during the loss of the Mars Climate Orbiter);
• Lost in translation: “Uncle” in English has no exact correspondence in Chinese.
33
![Page 34: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/34.jpg)
Semantic Noise and Channel Coding
34
“coffee machine”“copy machine”
“Xerox” “Xerox”
“copy machine”
p->ff
?
?
0.9
0.1
1.0
W X Y W’
Scenario developed based on reports in http://english.visitkorea.or.kr/enu/AK/AK_EN_1_6_8_5.jsp and http://blog.cleveland.com/metro/2011/03/identifying_photocopy_machine.html
![Page 35: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/35.jpg)
Semantic Channel Coding Theorem
• In the simplified model, assume no semantic mismatch (Ks=Kr, Is=Ir)
• Theorem 3: If transmission rate is smaller than Cs (semantic channel capacity), error-free coding exists
• Semantic channel capacity may be higher or lower than the engineering channel capacity (sup I(X;Y)) !– H(W|X) stands for encoder’s semantic ambiguity – avg(inf(Y)) is receiver’s “smartness”
35
![Page 36: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/36.jpg)
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
36
![Page 37: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/37.jpg)
Application in Coding & Validation
• Hypothesis 1: using semantics we can achieve better data compression
• Hypothesis 2: using semantics we can achieve more reliable communication
• Validation with comparison to non-semantic algorithms
![Page 38: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/38.jpg)
Extensions
• Extensions & connections to other fields
– First-order languages [probabilistic logics]– Inconsistent KBs (misinformation) [paraconsistent
logics]– Lossy source coding [clustering and similarity
measurement]– Semantic mismatches [extending Juba & Sudan
2011]– … …
![Page 39: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/39.jpg)
Path ahead – Broad Impact
– Communications (e.g., coding)– Linguistics (e.g., entropy of English)– Biology (e.g., semantics of genes)– Economics – ….– Areas wherever Shannon’s theory applies – And beyond (e.g., Semantic Web, ontology
engineering)
![Page 40: Towards a theory of semantic communication](https://reader033.fdocuments.in/reader033/viewer/2022061104/541279ae8d7f720a4e8b45bc/html5/thumbnails/40.jpg)
Questions?
40Image courtesy: http://www.addletters.com/pictures/bart-simpson-generator/900788.htm