Decision Trees and Information: A Question of Bits
description
Transcript of Decision Trees and Information: A Question of Bits
![Page 1: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/1.jpg)
Decision Trees and Decision Trees and Information:Information:
A Question of BitsA Question of Bits
Great Theoretical Ideas In Computer Science
Steven Rudich, Anupam Gupta CS 15-251 Spring 2004
Lecture 20 March 25, 2004 Carnegie Mellon University
![Page 2: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/2.jpg)
A choice tree is a rooted, directed tree with an object called a “choice” associated with each edge and
a label on each leaf.
Choice TreeChoice Tree
![Page 3: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/3.jpg)
We satisfy these two conditions:• Each leaf label is in S• Each element from S on exactly one leaf.
Choice Tree Representation of SChoice Tree Representation of S
![Page 4: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/4.jpg)
I am thinking of an outfit. Ask me questions until you know which one.
What color is the beanie?What color is the tie?
Question Tree Representation of Question Tree Representation of SS
![Page 5: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/5.jpg)
When a question tree has at most 2 choices at each
node, we will call it a decision tree, or a decision
strategy.
Note: Nodes with one choices represent stupid questions, but we allow
stupid questions.
![Page 6: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/6.jpg)
20 Questions20 Questions
S = set of all English nouns
Game: I am thinking of an element of S. You may ask up to 20 YES/NO questions.
What is a question strategy for this game?
![Page 7: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/7.jpg)
20 Questions20 Questions
Suppose S = {a0, a1, a2, …, ak}
Binary search on S.
First question will be:“Is the word in {a0, a1, a2, …,
a(k-1)/2} ?”
![Page 8: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/8.jpg)
20 Questions20 QuestionsDecision Tree RepresentationDecision Tree Representation
A decision tree with depth at most 20, which has the elements of S on the leaves.
Decision tree for{a0, a1, a2, …, a(k-1)/2}
Decision tree for{a(k+1)/2, …, ak-1, ak}
![Page 9: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/9.jpg)
Decision Tree RepresentationDecision Tree Representation
Theorem: The binary-search decision tree for S with k+1 elements { a0, a1, a2, …, ak } has depth
d log (k+1) e
= log k + 1= |k|
![Page 10: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/10.jpg)
Another way to look at itAnother way to look at it
If you are thinking of the noun am in SWe are asking about each bit of index m
Is the leftmost bit of m 0?Is the next bit of m 0?
…
Theorem: The binary-search decision-tree for S = { a0, a1, a2, …, ak } has depth
|k| = log k + 1
![Page 11: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/11.jpg)
A lower boundA lower bound
Theorem: No decision tree for S can have depth d < log k + 1 .
Proof:Any depth d tree can have at most 2d leaves.
But d < log k + 1 number of leaves 2d < (k+1)Hence some element of S is not a leaf.
![Page 12: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/12.jpg)
Tight bounds!Tight bounds!
The optimal-depth decision tree for any set S with (k+1) elements has
depth
log k + 1 = |k|
![Page 13: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/13.jpg)
Prefix-free Set Prefix-free Set
Let T be a subset of {0,1}*.
Definition: T is prefix-free if for any distinct x,y 2 T,
if |x| < |y|, then x is not a prefix of y
Example: {000, 001, 1, 01} is prefix-free {0, 01, 10, 11, 101} is not.
![Page 14: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/14.jpg)
Prefix-free Code for S Prefix-free Code for S
Let S be any set.
Definition: A prefix-free code for S is a prefix-free set T and a 1-1 “encoding” function f: S -> T.
The inverse function f-1 is called the “decoding function”.
Example: S = {apple, orange, mango}. T = {0, 110, 1111}. f(apple) = 0, f(orange) = 1111, f(mango) = 110.
![Page 15: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/15.jpg)
What is so cool about prefix-free
codes?
![Page 16: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/16.jpg)
Sending sequences of elements of S over
a communications channel
Let T be prefix-free and f be an encoding function. Wish to send <x1, x2, x3, …>
Sender: sends f(x1) f(x2) f(x3)…
Receiver: breaks bit stream into elements of T and decodes using f-1
![Page 17: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/17.jpg)
Sending info on a channelSending info on a channel
Example: S = {apple, orange, mango}. T = {0, 110, 1111}. f(apple) = 0, f(orange) = 1111, f(mango) = 110.
If we see00011011111100…
we know it must be0 0 0 110 1111 110 0 …
and henceapple apple apple mango orange mango
apple …
![Page 18: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/18.jpg)
Morse Code is not Prefix-free!Morse Code is not Prefix-free!
SOS encodes as …---…
A .- F ..-. K -.- P .--. U ..- Z --..
B -... G --. L .-.. Q --.- V ...-
C -.-. H .... M -- R .-. W .--
D -.. I .. N -. S ... X -..-
E . J .--- O --- T - Y -.--
![Page 19: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/19.jpg)
Morse Code is not Prefix-free!Morse Code is not Prefix-free!
SOS encodes as …---…
Could decode as: IAMIE
A .- F ..-. K -.- P .--. U ..- Z --..
B -... G --. L .-.. Q --.- V ...-
C -.-. H .... M -- R .-. W .--
D -.. I .. N -. S ... X -..-
E . J .--- O --- T - Y -.--
![Page 20: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/20.jpg)
Unless you use pauses Unless you use pauses
SOS encodes as … --- …
A .- F ..-. K -.- P .--. U ..- Z --..
B -... G --. L .-.. Q --.- V ...-
C -.-. H .... M -- R .-. W .--
D -.. I .. N -. S ... X -..-
E . J .--- O --- T - Y -.--
![Page 21: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/21.jpg)
Prefix-free codes
are also called “self-delimiting”
codes.
![Page 22: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/22.jpg)
Representing prefix-free codesRepresenting prefix-free codes
A = 100
B = 010
C = 101
D = 011
É = 00
F = 11“CAFÉ” would encode as 1011001100
How do we decode 1011001100 (fast)?
CAB D
0 1
0
0
01
1
1
0 1 FÉ
![Page 23: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/23.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as:
![Page 24: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/24.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: A
![Page 25: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/25.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: AB
![Page 26: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/26.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABA
![Page 27: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/27.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABAD
![Page 28: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/28.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABADC
![Page 29: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/29.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABADCA
![Page 30: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/30.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABADCAF
![Page 31: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/31.jpg)
CAB D
0 1
0
0
01
1
1
0 1 FÉ
If you see: 1000101000111011001100
can decode as: ABADCAFÉ
![Page 32: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/32.jpg)
Prefix-free codes are yet another
representation of a decision tree.
Theorem:
S has a decision tree of depth d
if and only if S has a prefix-free code with all codewords bounded by length d
![Page 33: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/33.jpg)
Theorem:
S has a decision tree of depth d
if and only if S has a prefix-free code with all codewords bounded by length d
CAB D
0 1
0
0
01
1
1
0 1 FÉ
CAB D
FÉ
![Page 34: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/34.jpg)
Let S is a subset of
Theorem:
S has a decision tree where all length n elements of S have depth · D(n)
if and only if S has a prefix-free code where all length n strings in S have encodingsof length · D(n)
Extends to infinite setsExtends to infinite sets
![Page 35: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/35.jpg)
I am thinking of some natural number k -
ask me YES/NO questions in order to determine k.
Let d(k) be the number of questions that you ask when I am thinking of k.
Let D(n) = max { d(j) over n-bit numbers j}.
![Page 36: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/36.jpg)
Lousy strategy: Is it 0? 1? 2? 3? …
d(k) = k+1
D(n) = 2n+1 since 2n+1 -1 uses only n bits.
Effort is exponential in length of k !!!
I am thinking of some natural number k -
ask me YES/NO questions in order to determine k.
![Page 37: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/37.jpg)
What is an efficient question strategy?
I am thinking of some natural number k -
ask me YES/NO questions in order to determine k.
![Page 38: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/38.jpg)
I am thinking of some natural number k…
Does k have length 1? NODoes k have length 2? NODoes k have length 3? NO
… Does k have length n? YES
Do binary search on strings of length n.
![Page 39: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/39.jpg)
Does k have length 1? NODoes k have length 2? NODoes k have length 3? NO
… Does k have length n? YES
Do binary search on strings of length n.
Size First/ Binary Search
d(k) = |k| + |k| = 2 ( b log k c + 1 )
D(n) = 2n
![Page 40: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/40.jpg)
What prefix-free code corresponds to the
Size First / Binary Search decision strategy?
f(k) = (|k| - 1) zeros, followed by 1, and then by the
binary representation of k
|f(k)| = 2 |k|
![Page 41: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/41.jpg)
What prefix-free code corresponds to the
Size First / Binary Search decision strategy?
Or,
length of k in unary |k| bitsk in binary |k| bits
![Page 42: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/42.jpg)
Another way to look at fAnother way to look at f
k = 27 = 11011, and hence |k| = 5
f(k) = 00001 11011
![Page 43: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/43.jpg)
k = 27 = 11011, and hence |k| = 5
f(k) = 00001 11011
g(k) = 0101000111
Another way to look at the function g:
g(final 0) -> 10 g(all other 0’s) -> 00 g(final 1) -> 11 g(all other 1’s) -> 01
“Fat Binary” Size First/Binary Search strategy
11011
0101000111
Another way to look at fAnother way to look at f
![Page 44: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/44.jpg)
Is it possible to beat 2n questions to find a number of length n?
![Page 45: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/45.jpg)
Look at the prefix-free code…
Any obvious improvement suggest itself here?
the fat-binary map f concatenates
length of k in unary |k| bitsk in binary |k| bits
fat binary!
![Page 46: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/46.jpg)
In fat-binary, D(n) ≤ 2n
Now D(n) ≤ n + 2 (b log n c + 1)
better-than-Fat-Binary-code(k)concatenates
length of k in fat binary 2||k|| bitsk in binary |k| bits
Can you do better?
![Page 47: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/47.jpg)
better-than-Fat-Binary code
|k| in fat binary 2||k|| bitsk in binary |k| bits
Hey, wait!
In a better prefix-free code
RecursiveCode(k) concatenates
RecursiveCode(|k|) & k in binary
better-t-FB
better-t-better-thanFB
||k|| + 2|||k|||
![Page 48: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/48.jpg)
Oh, I need to remember how many levels of recursion r(k)
In the final code F(k) = F(r(k)) . RecursiveCode(k)
r(k) = log* k
Hence, length of F(k) = |k| + ||k|| + |||k||| + … + 1
+ | log*k | + …
![Page 49: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/49.jpg)
Good, Bonzo! I had thought you had fallen asleep.
Your code is sometimes called the Ladder code!!
![Page 50: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/50.jpg)
Maybe I can do better…
Can I get a prefix code for k with length log k ?
![Page 51: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/51.jpg)
No!
Let me tell you why length log k is not possible
![Page 52: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/52.jpg)
Decision trees have a natural Decision trees have a natural probabilistic interpretation. probabilistic interpretation.
Let T be a decision tree for S.Let T be a decision tree for S.
Start at the root, flip a fair Start at the root, flip a fair coin at each decision, and coin at each decision, and
stop when you get to a leaf. stop when you get to a leaf.
Each sequence Each sequence ww in S will be in S will be hit with probability hit with probability 1/21/2|w||w|
![Page 53: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/53.jpg)
Random walk down the treeRandom walk down the tree
É
CAB D
F
0 1
0
0
01
1
1
0 1
Hence, Pr(F) = ¼, Pr(A) = 1/8, Pr(C) = 1/8, …
Each sequence Each sequence ww in S will in S will
be hit with probability be hit with probability 1/21/2|w||w|
![Page 54: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/54.jpg)
Let T be a decision tree for S Let T be a decision tree for S (possibly countably infinite (possibly countably infinite
set)set)
The probability that some The probability that some element in S is hit by a element in S is hit by a
random walk down from the random walk down from the root isroot is
ww22 S S 1/2 1/2|w| |w|
Kraft Inequality
·· 1 1
![Page 55: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/55.jpg)
Kraft?mmm,
food…
![Page 56: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/56.jpg)
Let S be any prefix-free code. Let S be any prefix-free code.
Kraft Inequality:Kraft Inequality:
ww22 S S 1/2 1/2|w| |w| ·· 1 1
Fat BinaryFat Binary: : f(k) has 2|k| f(k) has 2|k| 2 log k bits 2 log k bits
kk22 1/k 1/k22 ·· kk22 ½ ½|f(k)| |f(k)| ·· 1 1
![Page 57: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/57.jpg)
Let S be any prefix-free code. Let S be any prefix-free code.
Kraft Inequality:Kraft Inequality:
ww22 S S 1/2 1/2|w| |w| ·· 1 1
Better CodeBetter Code: : f(k) has |k| + 2||k|| bitsf(k) has |k| + 2||k|| bits
kk22 1/(k (log k) 1/(k (log k)22) ) ·· 1 1
![Page 58: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/58.jpg)
Let S be any prefix-free code. Let S be any prefix-free code.
Kraft Inequality:Kraft Inequality:
ww22 S S 1/2 1/2|w| |w| ·· 1 1
Ladder CodeLadder Code: k is represented : k is represented by by
|k| + ||k|| + |||k||| + … bits|k| + ||k|| + |||k||| + … bits
kk22 1/(k logk loglogk …) 1/(k logk loglogk …) ·· 1 1
![Page 59: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/59.jpg)
Let S be any prefix-free code. Let S be any prefix-free code.
Kraft Inequality:Kraft Inequality:
ww22 S S 1/2 1/2|w| |w| ·· 1 1
Can a code that represents k byCan a code that represents k by |k| = logk|k| = logk bits exist? bits exist?
No, since No, since kk22 1/k 1/k diverges !!diverges !!
So you can’t get So you can’t get log nlog n, Bonzo…, Bonzo…
![Page 60: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/60.jpg)
Mmm…
![Page 61: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/61.jpg)
Back to compressing wordsBack to compressing words
The optimal-depth decision tree for any set S with (k+1) elements has
depth log k + 1
The optimal prefix-free codefor A-Z + “space” has length
log 26 + 1 = 5
![Page 62: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/62.jpg)
English Letter FrequenciesEnglish Letter Frequencies
But in English, different letters occur with different frequencies.
A 8.1% F 2.3% K .79% P 1.6% U 2.8% Z .04%
B 1.4% G 2.1% L 3.7% Q .11% V .86%
C 2.3% H 6.6% M 2.6% R 6.2% W 2.4%
D 4.7% I 6.8% N 7.1% S 6.3% X .11%
E 12% J .11% O 7.7% T 9.0% Y 2.0%
ETAONIHSRDLUMWCFGYPBVKQXJZ
![Page 63: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/63.jpg)
short encodings!short encodings!
Why should we try to minimize the maximum length of a codeword?
If encoding A-Z, we will be happy if the “average codeword” is short.
![Page 64: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/64.jpg)
Morse CodeMorse Code
A .- F ..-. K -.- P .--. U ..- Z --..
B -... G --. L .-.. Q --.- V ...-
C -.-. H .... M -- R .-. W .--
D -.. I .. N -. S ... X -..-
E . J .--- O --- T - Y -.--
ETAONIHSRDLUMWCFGYPBVKQXJZ
![Page 65: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/65.jpg)
Given frequencies for A-Z, what is the optimal
prefix-free encoding of the alphabet?
I.e., one that minimizes the average code length
![Page 66: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/66.jpg)
Huffman Codes: Optimal Prefix-free Huffman Codes: Optimal Prefix-free Codes Relative to a Given Codes Relative to a Given
DistributionDistribution
Here is a Huffman code based on the English letter frequencies given earlier:
A 1011 F 101001 K 10101000 P 111000 U 00100
B 111001 G 101000 L 11101 Q 1010100100 V 1010101
C 01010 H 1100 M 00101 R 0011 W 01011
D 0100 I 1111 N 1000 S 1101 X 1010100101
E 000 J 1010100110 O 1001 T 011 Y 101011
Z 1010100111
![Page 67: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/67.jpg)
But we are talking English!But we are talking English!
Huffman coding uses only letter frequencies.
But we can use correlations!E.g., Q is almost always followed by U…Can we use this?
Yes… Let us see how these correlations help
![Page 68: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/68.jpg)
Random wordsRandom words
Randomly generated letters from A-Z, spacenot using the frequencies at all:
XFOML RXKHRJFFJUJ ALPWXFWJXYJ FFJEYVJCQSGHYD QPAAMKBZAACIBZLKJQD
![Page 69: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/69.jpg)
Random wordsRandom words
Using only single character frequencies:
OCRO HLO RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL
![Page 70: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/70.jpg)
Random wordsRandom words
Each letter depends on the previous letter:
ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE
![Page 71: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/71.jpg)
Random wordsRandom words
Each letter depends on 2 previous letters:
IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE
![Page 72: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/72.jpg)
Random wordsRandom words
Each letter depends on 3 previous letters:
THE GENERATED JOB PROVIDUAL BETTER TRAND THE DISPLAYED CODE, ABOVERY UPONDULTS WELL THE CODERST IN THESTICAL IT DO HOCK BOTHEMERG. (INSTATES CONS ERATION. NEVER ANY OF PUBLE AND TO THEORY. EVENTIAL CALLEGAND TO ELAST BENERATED IN WITH PIES AS IS WITH THE)
![Page 73: Decision Trees and Information: A Question of Bits](https://reader035.fdocuments.in/reader035/viewer/2022070413/56814c2d550346895db93454/html5/thumbnails/73.jpg)
ReferencesReferences
The Mathematical Theory of Communication, by C. Shannon and W. Weaver
Elements of Information Theory, by T. Cover and J. Thomas