( Deterministic ) Communication amid Uncertainty

of 2411/20/2012 TIFR: Deterministic Communication Amid Uncertainty 1

(Deterministic) Communication amid Uncertainty

Madhu SudanMicrosoft, New England

Based on joint works with:(1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba

(Harvard)and (2) Elad Haramaty (Technion)

of 24

Classical Communication

The Shannon setting Alice gets chosen from distribution Sends some compression to Bob. Bob computes

(with knowledge of ). Hope .

Classical Uncertainty:

Today’s talk: Bob knows .

11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 2

of 24

Outline

Part 1: Motivation Part 2: Formalism Part 3: Randomized Solution Part 4: Issues with Randomized Solution Part 5: Deterministic Issues.


of 24

Motivation: Human Communication

Human communication (dictated by languages, grammars) very different from Shannon setting. Grammar: Rules, often violated. Dictionary: Often multiple meanings to a word. Redundant: But not as in any predefined way

(not an error-correcting code).

Theory? Information theory? Linguistics? (Universal grammars etc.)?


of 24

Behavioral aspects of natural communication

(Vast) Implicit context. Sender sends increasingly long messages to

receiver till receiver “gets” (the meaning of) the message. Where do the options come from?

Sender may use feedback from receiver if available; or estimates receiver’s knowledge if not. How does estimation influence message.

Language provides sequence of (increasingly) long ways to represent a message. How? Why? What features are good/bad.


of 24

Model:

Reason to choose short messages: Compression. Channel is still a scarce resource; still want to

use optimally.

Reason to choose long messages (when short ones are available): Reducing ambiguity. Sender unsure of receiver’s prior (context). Sender wishes to ensure receiver gets the

message, no matter what its prior (within reason).


of 24

Back to Problem

Design encoding/decoding schemes () so that Sender has distribution on Receiver has distribution on Sender gets Sends to receiver. Receiver receives Decodes to

Want: (provided close), While minimizing


of 24

Contrast with some previous models

Universal compression? Doesn’t apply: P,Q are not finitely specified. Don’t have a sequence of samples from P; just

one! K-L divergence?

Measures inefficiency of compressing for Q if real distribution is P.

But assumes encoding/decoding according to same distribution Q.

Semantic Communication: Uncertainty of sender/receiver; but no special

goal.


of 24

Closeness of distributions:

is -close to if for all , -close to (symmetrized, “worst-case” KL-divergence)


of 24

Dictionary = Shared Randomness?

Modelling the dictionary: What should it be?

Simplifying assumption – it is shared randomness, so …

Assume sender and receiver have some shared randomness and are independent of .

Want


of 24

Solution (variant of Arith. Coding)

Use to define sequences

…

where chosen s.t. Either Or


of 24

Performance

Obviously decoding always correct.

Easy exercise:

( “binary entropy”)

Limits: No scheme can achieve Can reduce randomness needed.


of 24

Implications

Reflects the tension between ambiguity resolution and compression. Larger the ((estimated) gap in context), larger

the encoding length. Coding scheme reflects the nature of human

process (extend messages till they feel unambiguous).

The “shared randomness’’ is a convenient starting point for discussion Dictionaries do have more structure. But have plenty of entropy too. Still … should try to do without it.


of 24

Deterministic Compression?

Randomness fundamental to solution. Needs independent of to work.

Can there be a deterministic solution? Technically: Hard to come up with single

scheme that compresses consistently for all (). Conceptually: Nicer to know “dictionary” and

context can be interdependent.


of 24

Challenging special case

Alice has permutation on i.e., 1-1 function mapping

Bob has permutation Know both are close:

(say ) Alice and Bob know (say ).

Alice wishes to communicate to Bob. Can we do this with few bits?

Say bits if .


of 24

Consider family of graphs Vertices = permutations on Edges = close permutations with distinct

messages. (two potential Alices).

Central question: What is ?

Model as a graph coloring problem


𝜋

𝜎X

of 24

Main Results [w. Elad Haramaty]

Claim: Compression length for toy problem Thm 1:

( times) .

Thm 2: uncertain comm. schemes with1. (-error).2. -error).

Rest of the talk: Graph coloring


of 24

Restricted Uncertainty Graphs

Will look at Vertices: restrictions of permutations to first

coordinates. Edges:


𝜋

𝜎X

𝜋 ′

𝜎 ′

of 24

Homomorphisms

homomorphic to () if Homorphisms?

is -colorable

Homomorphisms and Uncertainty graphs.

Suffices to upper bound


of 24

Chromatic number of

For , Let

Claim: is an independent set of

Claim:

Corollary:


of 24

𝐻𝐺

Better upper bounds:


Say

Lemma:

For

of 24

Lemma: For Corollary:

Aside: Can show: Implies can’t expect simple derandomization of

the randomized compression scheme.


Better upper bounds:

of 24

Future work?

Open Questions: Is ? Can we compress arbitrary distributions to ? or

even On conceptual side:

Better mathematical understanding of forces on language.

Information-theoretic Computational Evolutionary


of 24

Thank You


( Deterministic ) Communication amid Uncertainty

Documents

Transcript of ( Deterministic ) Communication amid Uncertainty