of 2411/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 1
(Deterministic) Communication amid Uncertainty
Madhu SudanMicrosoft, New England
Based on joint works with:(1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba
(Harvard)and (2) Elad Haramaty (Technion)
of 24
Classical Communication
The Shannon setting Alice gets chosen from distribution Sends some compression to Bob. Bob computes
(with knowledge of ). Hope .
Classical Uncertainty:
Today’s talk: Bob knows .
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 2
of 24
Outline
Part 1: Motivation Part 2: Formalism Part 3: Randomized Solution Part 4: Issues with Randomized Solution Part 5: Deterministic Issues.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 3
of 24
Motivation: Human Communication
Human communication vs. Designed communication: Human comm. dictated by languages,
grammars … Grammar: Rules, often violated. Dictionary: multiple meanings to word. Redundant: But error-correcting code.
Theory for human communication? Information theory? Linguistics? (Universal grammars etc.)?
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 4
of 24
Behavioral aspects of natural communication
(Vast) Implicit context. Sender sends increasingly long messages to
receiver till receiver “gets” (the meaning of) the message. Where do the options come from? Comes from language/dictionary – but
how/why? Sender may use feedback from receiver if
available; or estimates receiver’s knowledge if not. How does estimation influence message.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 5
of 24
Model:
Reason to choose short messages: Compression. Channel is still a scarce resource; still want to
use optimally.
Reason to choose long messages (when short ones are available): Reducing ambiguity. Sender unsure of receiver’s prior (context). Sender wishes to ensure receiver gets the
message, no matter what its prior (within reason).
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 6
of 24
Back to Problem
Design encoding/decoding schemes () so that Sender has distribution on Receiver has distribution on Sender gets Sends to receiver. Receiver receives Decodes to
Want: (provided close), While minimizing
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 7
of 24
Contrast with some previous models
Universal compression? Doesn’t apply: P,Q are not finitely specified. Don’t have a sequence of samples from P; just
one! K-L divergence?
Measures inefficiency of compressing for Q if real distribution is P.
But assumes encoding/decoding according to same distribution Q.
Semantic Communication: Uncertainty of sender/receiver; but no special
goal.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 8
of 24
Closeness of distributions:
is -close to if for all , -close to (symmetrized, “worst-case” KL-divergence)
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 9
of 24
Dictionary = Shared Randomness?
Modelling the dictionary: What should it be?
Simplifying assumption – it is shared randomness, so …
Assume sender and receiver have some shared randomness and are independent of .
Want
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 10
of 24
Solution (variant of Arith. Coding)
Use to define sequences
…
where chosen s.t. Either Or
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 11
of 24
Performance
Obviously decoding always correct.
Easy exercise:
( “binary entropy”)
Limits: No scheme can achieve Can reduce randomness needed.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 12
of 24
Implications
Reflects the tension between ambiguity resolution and compression. Larger the ((estimated) gap in context), larger
the encoding length. Coding scheme reflects the nature of human
process (extend messages till they feel unambiguous).
The “shared randomness’’ is a convenient starting point for discussion Dictionaries do have more structure. But have plenty of entropy too. Still … should try to do without it.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 13
of 24
Deterministic Compression?
Randomness fundamental to solution. Needs independent of to work.
Can there be a deterministic solution? Technically: Hard to come up with single
scheme that compresses consistently for all (). Conceptually: Nicer to know “dictionary” and
context can be interdependent.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 14
of 24
Challenging special case
Alice has permutation on i.e., 1-1 function mapping
Bob has permutation Know both are close:
(say ) Alice and Bob know (say ).
Alice wishes to communicate to Bob. Can we do this with few bits?
Say bits if .
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 15
of 24
Consider family of graphs Vertices = permutations on Edges = close permutations with distinct
messages. (two potential Alices).
Central question: What is ?
Model as a graph coloring problem
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 16
𝜋
𝜎X
of 24
Main Results [w. Elad Haramaty]
Claim: Compression length for toy problem Thm 1:
( times) .
Thm 2: uncertain comm. schemes with
(-error).1. -error).
Rest of the talk: Graph coloring
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 17
of 24
Restricted Uncertainty Graphs
Will look at Vertices: restrictions of permutations to first
coordinates. Edges:
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 18
𝜋
𝜎X
𝜋 ′
𝜎 ′
of 24
Homomorphisms
homomorphic to () if Homorphisms?
is -colorable
Homomorphisms and Uncertainty graphs.
Suffices to upper bound
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 19
of 24
Chromatic number of
For , Let
Claim: is an independent set of
Claim:
Corollary:
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 20
of 24
𝐻𝐺
Better upper bounds:
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 21
Say
Lemma:
For
of 24
Lemma: For Corollary:
Aside: Can show: Implies can’t expect simple derandomization of
the randomized compression scheme.
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 22
Better upper bounds:
of 24
Future work?
Open Questions: Is ? Can we compress arbitrary distributions to ? or
even On conceptual side:
Better mathematical understanding of forces on language.
Information-theoretic Computational Evolutionary
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 23
of 24
Thank You
11/26/2012 MSR-I: Deterministic Communication Amid Uncertainty 24
Top Related