Data Compression Huffman Codes

8/7/2019 Data Compression Huffman Codes

1/60

Huffman Codes

Prof. Ja-Ling Wu

Department of Computer Scienceand Information EngineeringNational Taiwan University


2/60

2

Huffman Encoding

1. Order the symbols according to their probabilities

alphabet set : S1, S2, , SNprob. of occurrence : P1, P2, , PN

the symbols are rearranged so that

P1 P2 PN

2. Apply a contraction process to the two symbols with thesmallest probabilities

replace symbols SN-1 and SN by a hypothetical symbol, sayHN-1, that has a prob. of occurrence PN-1+PNthe new set of symbols has N-1 members:

S1, S2, , SN-2, HN-13. Repeat the step 2 until the final set has only one member.


3/60

3

The recursive procedure in step 2 can be viewed as

the construction of a binary tree, since at each step weare merging two symbols.

At the end of the recursion, all the symbols S1, S2, ,

SN will be leaf nodes of the tree.

The codeword for each symbol Si is obtained by

traversing the binary tree from its root to the leaf

node corresponding to Si


4/60

4

root

k e

f

f

b

b

1 e

ec

c

g

g

a

a

d

d

h

1 i

j 0

0

0

0

0

0

1

1

1

1

C(w)

a 10101

b 01

c 100

d 10100

e 11

f 00

g 1011

0.05d1.0

0.1h0.05a2.0

0.1c0.1g0.1g3.0

0.2f0.2i0.1c0.1c05.0

0.3j0.2b0.2f0.2f0.2f1.0

0.4k0.3e0.3j0.2b0.2b0.2b2.0

0.60.4k0.3e0.3e0.3e0.3e05.0

222221 54321

g

f

e

d

c

b

la

sps

StepStepStepStepStepStep

ii

0.1

h

0.2

i

0.3

j

0.4

k

0.6

e

1.0

root


5/60

5

Average codeword length

lave is a measure of the compression ratio.

In the above example,

7 symbols 3 bits fixed length code representation

lave(Huffman) = 2.6 bits

Compression ration = 3/2.6 = 1.15

i

iipllave


6/60

6

Properties of Huffman codes

Fixed-length symbols variable-length codewords :

error propagation

H(s) lave < H(s)+1

H(s) lave < P+0.086

where P is the prob. of the most frequently occurringsymbol. The equality is achieved when all symbolprobs. are inverse powers of two.

The Huffman code-tree can be constructed both by bottom-up method in the above example

top-down method


7/60


8/60

8

Huffman Decoding

Bit-Serial Decoding : fixed input bit rate variableoutput symbol rate

(Assume the binary coding tree is available to the decoder)In practice, Huffman coding tree can be reconstructed from thesymbol-to-codeword mapping table that is known to both theencoder and the decoder

Step 1:

Read the input compressed stream bit-by-bit and traverse thetree until a leaf node is reached.

Step 2:

As each bit in the input stream is used, it is discarded. When theleaf node is reached, the Huffman decoder outputs the symbol at

the leaf node. This completes the decoding for this symbol.Step 3:

Repeat steps 1 and 2 until all the input is consumed.

Since the codeword is variable in length, the decoding bit rate isnot the same for all symbols.


9/60

9

Lookup-table-Based Decoding : constant decodingrate for all symbols variable input rate

The look-up table is constructed at the decoder from the symbol-

to-codeword mapping table. If the longest codeword in this tableis L bits, then a 2L entry lookup table is needed. : space constraintsimage/video longest L = 16 to 20.

Look up table construction: Let Ci be the codeword that corresponds to symbol Si.Assume

Ci has li bits. We form an L-bit address in which the first li bitsare Ci and the remaining L- li bits take on all possiblecombinations of 0 and 1. Thus, for the symbol si, there will

be 2L- li addresses. At each entry we form the two-tuple (si, li).


10/60

10

Decoding Processes:

1. From the compressed input bit stream, we read in L bits intoa buffer.

2. We use the L-bit word in the buffer as an address into thelookup table and obtain the corresponding symbol, say sk.Let the codeword length be lk. We have now decode onesymbol.

3. We discard the first lk bits from the buffer and we append tothe buffer, the next lk bits from the input, so that the bufferhas again L bits.

4. Repeat steps 2 and 3 until all of the symbols have been

decoded.


11/60

11

Memory Efficient and High-SpeedSearch Huffman Coding, by R. Hashemian IEEEtrans. Communications, Oct. 1995, pp. 2576-2581

Due to variable-length coding, the Huffman tree getsprogressively sparse as it grows from the root

Waste of memory space

A lengthy search procedure for locating a symbol

Ex: if K-bit is the longest Huffman code assigned to a setof symbols, the memory size for the symbols mayeasily reach 2K words in size.

It is desirable to reduce the memory size from typical

value of 2K, to a size proportional to the number of theactual symbols.

Reduce memory size

Quicker access


12/60

12

Ordering and clustering based Huffman Coding

groups the codewords (tree nodes) within specified codeword lengths Characteristics of the proposed coding scheme:

1. The search time for more frequent symbols (shorter codes) issubstantially reduced compare to less frequent symbols,resulting in an overall faster response.

2. For long codewords the search for the symbol is also speedup. This is achieved through a specific partitioning techniquethat groups the code bits in a codeword, and the search for asymbol is conducted by jumping over the groups of bitsrather than going through the bit individually.

3. The growth of the Huffman tree is directed toward one side ofthe tree.

Single side growing Huffman tree (SGH-tree)


13/60

13

Ex: H=(S, P) S={S1, S2,, Sn}P={P1, P2,, Pn}

No. of occurrence

For a given source listing H, the table of codeword lengthuniquely groups the symbols into blocks, where each block isspecified by its codeword length (CL).

TABLE IReduction Process In The Source List

s1 48 s1 48 s1 48 s1 48 s1 48 a5 52

s2 31 s2 31 s2 31 s2 31 s2 31 s1 48

s3 7 s3 7 a2 8 a3 13 a4 21

s4 6 s4 6 s3 7 a2 8

s5 5 s5 5 s4 6

s6 2 a1 3

s7 1

Merge

Insert (in descending order)


14/60

14

Each block of symbols, so defined, occupies one level in theassociated Huffman tree.

CL: codeword length


15/60

15

Algorithm 1: Creating a Table of Codeword Lengths

1. The symbols are listed with the probabilities in decendingorder (the ordering of the symbols with equal probabilities isassumed indifferent).Next, the pair of symbols at the bottom of the ordered list aremerged and as a result a new symbol a1 is created. Thesymbol a1, with probability equal to the sum of the

probabilities of the pair, is then inserted at the proper locationin the ordered list.

To record this operation a codeword length recording (CLR)table is created which consists of three columns:

: Columns 1 and 2 hold the last pair of symbols before beingmerged, and column 3, initially empty, is identified as thecodeword length (CL) column (Table II).


16/60

16

In order to make the size of the CLR table small and the hardware

design simpler, the new symbol a1 (in general aj) is selected such

that its inverse represents the associated table address.

e.g. For an 8-bit address word,

a1 The first row in the CLR table, is given the value of 1111 1110

a1 0000 0001

a2 1111 11101 ( a2 0000 0010)

The sign of a composite symbol is different from an original symbol.

A dedicated sign (MSB) detector is all that is needed to distinguishbetween an original symbol and a composite symbol.

)(or1 j

aa

Composite symbol


17/60

17

2. Continue applying the same procedure, developed for asingle row in step 1, and construct the entire CLR table. Note

that table II contains both the original symbol si and thecomposite ones aj (carrying opposite signs).

3. The third column in Table II, designated by CL, is assigned tohold the codeword lengths. To fill up this column we startfrom the last row in the CLR and enter 1. This designates the

codeword length for both s1 and a5.Next, we check for the signs of each si and a5; if positive(MSB = 0) we skip, otherwise, the symbol is a composite one(a5)and its binary inverse (a5=0000101)is a row address for table II. We now increment the number

in the CL column, and assign a new value (2 in this example)to the CL column in row aj (5 in this example), and proceedapplying the same operation to other rows in the CLR table,as we move to the top, until the CL column is completelyfilled.


18/60

18

1

2

4

5

6

S7 S6

S5

S3S4

a2

a4

S1

a1

a3

a5

S2

3

2

1

4

4

5

CLSi Si-1Row No

1+1

3+13+1

2+1

4+1

3


19/60

19

role:

(i) Si

, Si-1

composite, composite CL new address

(ii)Si, Si-1 original, skip this row and CL

4. Table II indicates that each original symbol in the table has

its codeword length (CL) specified.Ordering the symbols according the their CL values givesTable III.

Associated with the so-obtained TOCL one can

actually generate a number of Huffman tables (orHuffman trees), each of which being different incodewords but identical in the codeword lengths.


20/60

20

Single-Side growing Huffman table (SGHT)

Table

Single-Side growing Huffman tree (SGH-Tree)

Fig. 1

are adopted.


21/60

21

a5

a4

a3 a2

a1

For a uniformlydistributed source

SGH-Tree becomes

full.


22/60


23/60

23

In general we can write :

where p and q are the codeword lengths for si andsi+1, respectively, and Sn (associated with Cn)denotes the terminal symbol.

C1 and Cn have unique forms easy to verify

111

and

2*1000

1

1

n

qp

ii

C

CCC


24/60

24

Super-tree (S-tree)


25/60

25

x

x

y

y

z

z


26/60

26


27/60

27

0 1 2 3 4 5 6 7 8 9 a b c d e f

0 00 01 02 03 04 05 06 07

1 08 09 0a 0b 0c 0d 0e 0f 10 11

2 12 13 14 15 16 17 18 19 1a

3 1b 1c 1d 1e 1f

TABLE VIIMemory (RAM) Space Associated with Table VI and Figs. 3 and 4


28/60

28

Storage Allocation

For non-uniform source, SGH-Tree becomes sparse.

How toi) optimize the storage space

ii) provide quick access to the symbol (data)

key Idea:

Break down the SGH-Tree into smaller clusters(subtrees) such that the memory efficiencyincreases.

The memory efficiency B for a K-level binary Huffmantree

%1002

nodesleafeffectiveofNo.

KKB


29/60

29

Ex:1

2 3

6 3

14 15

28 29 30 31

62 63

S1

S2

S3 S4 S5

S6 S7

a1

a2a3

a4

a5

2/21 : 100%

3/22 : 75%

4/23 : 50%

6/24 : 37%

7/25 : 22%


30/60

30

Remarks:

1. The efficiency changes only when we switch to a new level(or equivalently to a new CL), and it decreases as weproceed to the higher levels.

2. Memory efficiency can be interpreted as a measure of theperformance of the system in terms of memory spacerequirement; and it is directly related to the sparsity of the

Huffman tree.

3. Higher memory efficiency for the top levels (with smaller CL)is a clear indication that partitioning the tree into smaller andless sparse clusters will reduce the memory size. In addition,clustering also helps to reduce the search time for a symbol.

Definition:

A cluster (subtree) Ti with minimum memory efficiency (MME) Bi,if there is no level in Ti with memory efficiency less than Bi.


31/60

31

SGH-Tree Clustering

Given a SGH-tree, as shown in Fig. 2, depending onthe MME (or CL) assigned, the tree is partitioned by a

cut line, x-x, at the Lth level (L=4 for the choice ofMME=50%, in this example).

The first cluster (subtree), as shown in Fig.3(a), isformed by removing the remainder of the tree beyondthe cut-line x-x.

The cluster length is defined to be the maximum pathlength from the root to a node within the cluster the cluster length for the first cluster is 4.

Associated with each cluster a look up table (LUT) is

assigned, as shown at the bottom of Fig.3(a), toprovide the addressing information for thecorresponding terminal node (symbol) within thecluster, or beyond.


32/60

32

To identify other clusters, in the tree, we draw more cut lines y-y,and z-z, each L levels apart.

More clusters are generated, each of which starting from a singlenode (root of the cluster) and expanded until it is terminatedeither by terminal nodes, or nodes being intercepted by the nextcute line.

Next, we construct a super-tree (s-tree) corresponding to a

SGH-Tree. In a s-tree each cluster is represented by a node,and the links connecting these nodes, representing the branchingnodes in the SGH-tree, shared between two clusters.

The super-table (ST) associated with the s-tree is shown at thebottom of the tree.

Note that the s-tree has 7 nodes, one for each cluster, while itsST has 6 entries. This is because the root cluster a is left out andthe table starts from cluster b.


33/60

33

Entries in the ST and the LUTs

There are two numbers in each location in the STthe first number identifies the cluster length

the 2nd

number is the offset memory address for that clusterEx:

cluster length : 11+1 = 100, or 42aH : the starting address of the corresponding LUT, in thememory (see table)

the cluster f start at address 2aH in the memory table. (i.e.symbol 18)

Each entry in a LUT is an integer in sign/magnitudeformat.Positive integer, with 0 sign, correspond to the nodes existed inthe cluster, while negative numbers, with 1 sign, represent thenodes being cut by the next cut line.

11 2aHf

binary Hexa


34/60

34

The magnitude of a negative integer specifies a location in the ST,for further search.For example, consider the cluster-C

4 4 4 4 10 10 11 11 24 25 26 27 28 3 4 5

0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Sign/magnitude f28 ed27262524

0e 0f 10 11 12yy

14 15

76

12 13

4

6

C

5

2

10 110c 0d

0b


35/60

35

In the 15th entry of the above LUT we find as sign/magnitudeat that location.

the negative sign (1) indicates that we have to move to othercluster, and 4 refers to the 4th entry in the ST, whichcorresponds to cluster e, and contains 01 and 26H numbers.

The first number, 01, indicates the cluster length is 01+1=10(or

2), and the 2nd number, shows the starting address of cluster e inthe memory.

A positive entry in a LUT, indicates that the symbol isalready found, and the magnitude comprises three

pieces of informationi) location of the symbol in the memory

ii) the pertinent codeword

iii) the codeword length


36/60

36

Read the positive integer in binary form, andidentify the most significant 1 bit (MS1B).The position of the MS1B in the binarynumber specifies the codeword length, therest of the bits on the right side of the MS1B

gives the codeword, cj, and the magnitude ofcj is, in fact, the relative address of thesymbol in the memory (Table)

F l


37/60

37

For example

1 0 4

2 0 4

3 0 44 0 4

5 0 5

6 0 5

7 0 5

8 0 5

9 0 24

10 0 25

11 0 26

12 0 27

13 0 28

14 0 29

15 1 1

16 1 2

07 symbol

0 the symbol is found

29 = 0 0 0 1 1 1 0 1

Symbol 07 is located at 1101=dH

7 6 5 4 3 2 1 0

Codeword length = 4

(See Table )0 1 2 3 4 5 6 7 8 9 a b c d e f

00 01 02 03 04 05 06 07

The 1st row of the table

29)H= f)H+ d)H

indication of the tree-depth of the node

memory location (address) if the

memory table is offset by f

Cluster length

(codeword length)

LUT for Fig.3(a)


38/60

38

Huffman Decoding

The decoding procedure basically starts by receivingan L-bit code cj, where L is the length of the top

cluster in the associated SGH-Tree (or the SGHT).This L-bit code cj is then used as the address to theassociated look-up table [fig.3(a)]

Example:1. Received codeword : 01100 1011

first L=4 bit 0110=6 as an address to the LUT given inFig.3(a).The content of the table at this location is 0/5, as thesign/magnitude.

0the symbol is located in this cluster

5 = 0000101, the MS1B is at location 2 CL=2. Next to the MS1B we have 01, whichrepresents the codeword

the symbol (01H) is found at the address 01 in thememory (Table)

MSB

MS1B


39/60

39

2. Rd codeword : 110110011the first 4 bit 1101 = d)H at the d-th location of the LUT

(Fig.3(a)) we get 0/29

3. Rd codeword = 1111 1111 1111 00101111 the symbol is not in cluster a and refer to

location 2 in the ST, assigned to cluster C.

MSB

The symbol is at this cluster

29 = 00011101MS1B

The address to the memory is 1101=d)HAnd the symbol is found to be 07.

The offset for the symbol

memory designate for cluster C

: the memory addressing (table

) of cluster C started at 14)H

11)2|14)H : the cluster length is 11+1=100 or 4


40/60

40

Our next move is to select another slice of (with length 4) from the

bit stream, which 1111 again. From the 15th location of the LUT

for cluster C we get 1/5

we move to the 15th location in the LUT for cluster f. Here we find

1/6 and refer to the 6th item in the ST.The data here is read 00)2 and 3a)H

00 CL of cluster g is 00+1=01 ; or 13a)H the memory location offset of cluster g.

The symbol is not in cluster C

Location 5 of the ST cluster f

The content is 11)2 2a)H11+1 = 100 : or 4 : cluster length2a)H : the offset of memory location of cluster f

The symbol is not located yet, we have to choose

another 4 bit slice from the bit stream : 1111


41/60


42/60

42

Remarks:

1.For high probable symbols with short codewords (4bits or less) the search for the symbol is very fast,and is completed in the first try. For longercodewords, however, the search time grows almostproportional to the codeword length.

If CL is the codeword lengthL is the maximum level selected for each cluster

( L = 4 in the above example) then the search time isclosely proportional to 1+CL/L

2. Increasing L:

i. Decreasing search time speed up decoding

ii. Growing the memory space requirement

Trade-off


43/60


44/60

44

3. Augment S1 by Q to form a new set W. Construct an optimalprefix code for W using the design procedure for

unconstrained length Huffman codewords. codewords cs, for symbols in the set S1

codeword cq for the symbols Q.cq is the shortened prefix-code for symbols in S2

If li is the length of the i-th codeword of S1, then

x)toequalorlargerintegersmallestthedenotesx(

1logmaxmax 211 Lp

li

SsiSs ii

E di i t t i


45/60

45

Encoding : input message string m1,m2,,mk

For all miS1, output the corresponding codewordfrom .

For all miS2, output the codeword cq followed by anL-bit fixed-length binary representation for mi.(Actually, one can use fewer than L bits, since if thereare symbols in S2 and , then the fixed-

length binary representation for each mi isbits).

1sC

2sN NNs 2

log22 s

N


46/60

46

Let lsh be the average codeword length for the shortened codes.lw be the average codeword length for W.

The worst-case increase in the average codeword length for theshortened code is bits per symbol (this function attains a

maximum value of ).

5.1

11log)(

11

log1

log1

log11

log

1log

2

1log

1

1

1log

1log

1log

1log

2

2222

22

22

22

212

222

21

1

sHlavesH

qqsH

qq

pp

pp

ppwHlsH

pppLpqL

qLwHlsH

wHlwH

qLll

wHp

pp

psH

q

q

p

pwH

Si i

i

Si i

i

Si i

ish

Si i

i

SiLi

Si

i

sh

w

wsh

Si i

i

Si i

i

Si i

i

qq

1log2


47/60

47

Example:Symbol Si pi li Codeword

0 0.2820 2 11

1 0.2786 2 10

2 0.1419 3 011

3 0.1389 3 010

4 0.0514 4 0011

5 0.0513 4 0010

6 0.0153 5 00011

7 0.0153 5 00010

8 0.0072 6 000011

9 0.0068 6 000010

10 0.0038 7 000001111 0.0032 7 0000010

12 0.0019 7 0000001

13 0.0013 8 00000001

14 0.0007 9 000000001

15 0.0004 9 000000000

bits694.215

0

i iipllave

The longest codeword is 9 bits

29 = 512 - entry table is needed

for lookup-table-based decoding

Now suppose only a 128-entrylookup table can be permitted.

7-bit shortened Huffman code

Code construction


48/60

48

Code construction

1.

0253.0

15.to8iforsince,,,

,,

15

8

1281

15982

7101

i

i

i

Pq

PsssS

sssS

S0 S1 S2 S3 S4 S5 S6 S7 Q

11 01 101 100 0011 0010 00011 00010 0000

0.2820 0.2786 0.1419 0.1389 0.0514 0.0513 0.0153 0.0153 0.0253


49/60

49

Symbol i pi li Codeword Additional

0 0.2820 2 11

1 0.2786 2 01

2 0.1419 3 101

3 0.1389 3 100

4 0.0514 4 0011

5 0.0513 4 0010

6 0.0153 5 000117 0.0153 5 00010

8 0.0072 11 0000 0001000

9 0.0068 11 0000 0001001

10 0.0038 11 0000 0001010

11 0.0032 11 0000 0001011

12 0.0019 11 0000 0001100

13 0.0013 11 0000 0001101

14 0.0007 11 0000 0001110

15 0.0004 11 0000 0001111


50/60

50

2. For all symbols in S2 we need a prefix code of 4 bitsfollowed by a 7-bit representation for the specific

symbol in S2

bits8057.21115

8

7

0

i

i

i

iish ppll


51/60

51

Decoding:

1. We first construct a lookup table as described above.

2. From the input bit stream, we fetch bits into a buffer until the bufferhas 7 bits. We access the lookup table location, using the 7 bits asan address.This lookup table location contains (mk, lk)

3. The first lk bits in the buffer are discarded by shifting the buffercontents to the left by l

k

bits positions.

If mk Q, mk{S0, S1,S7}, and thus we have correctly decoded thissymbol

If mk = Q, additional bits from the input bit stream are needed fordecoding. We fetch lk bits from the bit stream to fill up the buffer. Thebuffer now contains the binary representation for one of the symbols S8,S9,S15, and thus we have correctly decoded a symbol from S2.

4. Repeat steps 2 and 3 until the complete message has beendecoded.


52/60

52

The key disadvantage of two-level decoding is that wecannot guarantee a constant symbol rate at the

Huffman decoder output.

Lookup table size (entries) Worst case lsh

-lave

(bits/symbol)

16 0.4213

32 0.2326

64 0.2326

128 0.1342

256 0.0731512 0.0338

Constrained-length Huffman codes: prefix-free constant


53/60

53

g poutput decoding rate with a table-lookup decoder

For a maximum codeword length of L bits, we define a thresholdT = 2-L

Sort si, i = 1, 2, , N so that pk pk+1 For each pi, if pi T, set pi = T

Design the codebook using the modified pi values and theunconstrained-length Huffman code table design approach

Since pi is restricted to at most 2-L, no codeword length will

exceed L bits.

Codeword length 1/ pi : not guaranteeThis is due to the fact that some of the properties were set to thethreshold T and hence the ordering among the properties isobscured.

Rearranging is done by simply sorting the codeword lengths inascending order of magnitude and associating this sorted list tothe corresponding list of codewords.

Reordered


54/60

54

Symbol i pi l CodewordReordered

l codeword

0 0.2820 2 11 2 11

1 0.2786 2 01 2 01

2 0.1419 3 101 3 1013 0.1389 3 100 3 100

4 0.0514 4 0010 4 0010

5 0.0513 4 0001 4 0001

6 0.0153 6 001100 6 001100

7 0.0153 6 001101 6 001101

8 0.0072 7 0011110 6 000010

9 0.0068 7 0011111 6 000011

10 0.0038 7 0011100 6 000000

11 0.0032 7 0011101 6 000001

12 0.0019 6 000010 7 0011110

13 0.0013 6 000011 7 0011111

14 0.0007 6 000000 7 0011100

15 0.0004 6 000001 7 0011101

lave 2.7308 2.7141

: single-layer decoding

Constrained-length Huffman codes


55/60

55

Constrained length Huffman codes: The Voorhis method [1974, IEEE Trans. IT]: near optimum codeword length

Determine code lengths l1, l2,, lN that minimize subject tothe constraints 1li L. For unique decodable codes, we alsorequire that . The resulting codeword lengths will be suchthat 1 l1 l2 lN L

The I-th codeword is the first li bits of the fraction computed by

.

For an N-symbol alphabet, if L=log2N+d, then this method has acomplexity of O(dN2).

N

N

NPPP

PPP

SSS

21

21

21and,

,,,

,,,

N

i

iipl

12 N

i

li

1

1

2

i

k

lk


56/60

56

Symbol Si Voorhis code

0 11

1 10

2 0113 010

4 0011

5 0010

6 00011

7 00010

8 0000111

9 0000110

10 0000101

11 0000100

12 0000011

13 0000010

14 0000001

15 0000000

lave 2.7045


57/60

57

Home Work:

1. Consider codes that satisfy the suffix condition, which

says that no codeword is a suffix of any othercodeword. Show that a suffix condition code isuniquely decodable, and show that the minimumaverage length over all codes satisfying the suffixcondition is the same as the average length of theHuffman code for that random variable.

Suffix code

2. Suppose that X=i with probability Pi, i=1, 2, m. Let libe the number of binary symbols in the codeword

associated with X=i, and let ci denote the cost perletter of the codeword when X=i. Thus the averagecost C of the description of X is

m

i

iii lcpC1


58/60

58

a) Minimize C over all l1,l2,,lm such that 2-li1. Ignore any

implied integer constraints on li. Exhibit the minimizing

l1*

,l2*

,,lm*

and the associated minimum value C*

.b) How would you use the Huffman code procedure to minimize

C over all uniquely decodable codes? Let CHuffman denote thisminimum. Show that

Huffman codes with costs.

m

iiiHuffman cpCCC 1

**


59/60

59

3. A computer generates a number X according to aknown prob. mass function p(x), x{1,2,,100}. The

player asks arbitrary Yes-No questions sequentiallyuntil X is determined. If he is right (i.e., X isdetermined), he receives a prize of value v(x).

a) How should the player proceed to maximize his expected

winnings? What is his expected return?b) Continuing (a), what if v(x) is fixed, but p(x) can be chosen by

the computer (and then announced to the player)? Thecomputer wishes to minimize the players expected return.What should p(x) be? What is the expected return to the

player? The game of Hi-Lo.


60/60

60

4. Although the codeword lengths of an optimal variablelength code are complicated functions of the

message probabilities {p1,p2,,pm}, it can be said thatless probable symbols are encoded into longercodewords. Suppose that the message probabilitiesare given in decreasing order p1p2 pm.

a) Prove that for any binary Huffman code, if the most probablemessage symbol has probability p1>2/5, then that symbolmust be assigned a codeword of length 1.

b) Prove that for any binary Huffman code, if the most probablemessage symbol has probability p1

Data Compression Huffman Codes

Documents

Transcript of Data Compression Huffman Codes