Lecture 3

Mei-Chen Yeh2010/03/16

Announcements (1)

• Assignment formats:– A word or a pdf file– Subject ( 主旨 ):

Multimedia System Design-Assignment #X-your student id-your nameMultimedia System Design-Assgnment #2-697470731- 游宗毅

– File name ( 檔名 )Assignment #X-your student id-your nameAssignment #2-697470731- 游宗毅 .doc

Announcements (2)

• For the assignment#2…• If you did not use either a pdf or a doc file, please re-send

your report to TA using the format.• Due 03/16 (today)

…and based on TA’s clock

Announcements (3)

• The reading list is finally released!• Sources:

– Proceedings of ACM MM 2008– Proceedings of ACM MM 2009– The best paper in MM 2006 and MM 2007

• Interesting papers not on the list? Let me know!

Announcements (4)

• So you need to…– Browse the papers– Discuss with your partners about the paper choice– …and do it as soon as possible! I love that paper!

That paper sucks…

How to access the papers?

• The ACM digital library– http://portal.acm.org/– Should be able to download the papers if you connect to the site on

campus

• MM08 paper on the web– http://mpac.ee.ntu.edu.tw/multimediaconf/acmmm2008.html

• Google search

Next week in class

• Bidding papers! ( 論文競標 )• Each team will get a ticket, where you put

your points.

Ticket # 7Team name: 夜遊隊Team members: 葉梅珍 游宗毅-------------------------------------------------------------------paper1 paper2 paper3 paper4 … paper 25

50 10 15 20 5

Total : 100 points

Bidding rules

• The team with the most points gets the paper.• Every team gets one paper.• When a tie happens…

…and the winner takes the paper.

More about the bidding process

• Just, fair, and open!

• I will assign a paper to teams in which no one show up for the bid.

公平 公正 公開

Questions

Multimedia Compression (1)

Outline

• Introduction• Information theory• Entropy ( 熵 ) coding

– Huffman coding– Arithmetic coding

Why data compression?

• Transmission and storage

– For uncompressed video• CD-ROM (650MB) could store 650MB x 8 / 221Mbps ≈ 23.5

seconds• DVD-5 (4.7GB) could store about 3 minutes

Approximate Bit Rates for Uncompressed Sources

What is data compression?

• To represent information in a compact form (as few bits as possible)

• Technique

Original

Reconstructed data

Compressed data

Compression Reconstruction

Codec = encoder + decoder

Technique (cont.)

• Lossless– The reconstruction is identical to the

original.

• Lossy– Involves loss of information Lossy!

Lossless? Not necessarily true!

Do not send money!Do now send money!

Encoder Decoder

source

Example: image codec

Performance MeasuresHow do we say a method is good or bad?

• The amount of compression• How closely the reconstruction is• How fast the algorithm performs• The memory required to implement

the algorithm• The complexity of the algorithm• …

Two phases: modeling and coding

• Modeling– Discover the structure in the data– Extract information about any redundancy

• Coding– Describe the model and the residual (how the

data differ from the model)

Original

Compressed data

Fewer bits!

Encoder

Example (1)

• 5 bits * 12 samples = 60 bits• Representation using fewer bits?

Example: Modeling

,...2,18ˆ nnxn

n

Example: CodingnxOriginal data

8ˆ nxnModel 10 11 12 13 14 15 16 17 18 19 20

Residual nnn xxe ˆ 1 0 -1 1 -1 0 1 -1 -1 1 10

• {-1, 0, 1} • 2 bits * 12 samples = 24 bits (compared with

60 bits before compression)We use the model to predict the value, then encode the residual!

Another Example

• Morse Code (1838)

Shorter codes are assigned to letters that occur more frequently!

A Brief Introduction to Information Theory

Information Theory (1)

• A quantitative ( 量化的 ) measure of information– You will win the lottery tomorrow.– The sun will rise in the east tomorrow.

• Self-information [Shannon 1948]P(A): the probability that the event A will happen)(log

)(

1log)( AP

APAi bb

b determines the unit of information

The amount of surprise or uncertainty in the message

Information Theory (2)

• Example: flipping a coin– If the coin is fairP(H) = P(T) = ½i(H) = i(T) = -log2(½) = 1 bit

– If the coin is not fairP(H) = 1/8, P(T)=7/8i(H) = 3 bits, i(T) = 0.193 bitsThe occurrence of a HEAD conveys more information!

Information Theory (3)

• For a set of independent events Ai

– Entropy (the average self-information)

– The coin example• Fair coin (1/2, 1/2): H=P(H)i(H) + P(T)i(T) = 1• Unfair coin (1/8, 7/8): H=0.544

SAi

)(log)()()()( ibiii APAPAiAPSH

Information Theory (4)

• Entropy– The best a lossless compression scheme can do– Not possible to know for a physical source– Estimate (guess)!

• Depends on our assumptions about the structure of the data

Estimation of Entropy (1)

• 1 2 3 2 3 4 5 4 5 6 7 8 9 8 9 10– Assume the sequence is i.i.d.

• P(1)=P(6)=P(7)=P(10)=1/16P(2) =P(3)=P(4)=P(5)=P(8)=P(9)=2/16

• H = 3.25 bits

– Assume sample-to-sample correlation exists• Model: xn = xn-1 + rn

• Residuals: 1 1 1 -1 1 1 1 -1 1 1 1 1 1 -1 1 1• P(1)=13/16, P(-1)=3/16• H = 0.7 bits

Estimation of the Entropy (2)

• 1 2 1 2 3 3 3 3 1 2 3 3 3 3 1 2 3 3 1 2– One symbol at a time

• P(1) = P(2) = ¼, P(3) = ½• H = 1.5 bits/symbol• 30 (1.5*20) bits are required in total

– In blocks of two• P(1 2) = ½, P(3 3)=½• H = 1 bit/block• 10 (1*10) bits are required in total

Coding

Coding (1)• The assignment of binary sequences to elements of an

alphabet

• Rate of the code: average number of bits per symbol• Fixed-length code and variable-length code

letter_

alphabet

codeword

code

Ambiguous Not uniquely decodable

Prefix codeinstantaneous

DecodableWith one-

symbol delay

Coding (3)

• Example of not uniquely decodable code

Letters Codea1 0a2 1a3 00a4 11

100a2 a3 =>a2 a1 a1 =>

back

100

Coding (4)

• Not instantaneous, but uniquely decodable code

Oops!

a2

a2 a3 a3 a3 a3 a3 a3 a3 a3

Prefix Codes

• No codeword is a prefix to another codeword• Uniquely decodable

Huffman Coding

• Basic algorithm• Extended form• Adaptive coding

Huffman Coding

• Observations of prefix codes

– Frequent symbols have shorter codewords– The two symbols that occur least frequently have

the same lengthHuffman procedure:Two least frequent symbols differ only in the last bit. Ex: m0, m1

Algorithm

• A = {a1, a2, a3, a4, a5}

• P(a1) = 0.2, P(a2) = 0.4, P(a3) = 0.2, P(a4) = 0.1, P(a5) = 0.1

a2 (0.4)a1 (0.2)a3(0.2) a4(0.1) a5(0.1)

a’4(0.2)

a’3(0.4)

a’1(0.6)

101000 0010 0011

(1)

111

100

0

0

Algorithm

• Entropy: 2.122 bits/symbol• Average length: 2.2 bits/symbol

101000 0010 0011a1 (0.2)a3(0.2) a4(0.1) a5(0.1)

a’4(0.2)

a’3(0.4)

a’1(0.6)(1)

111

100

0

0

a2 (0.4) 00

a2 (0.4)

a’2(0.6)

(1)

1110a3(0.2)a1(0.2)

a’1(0.4)0 1

1

1

011010a5(0.1)a4(0.1)

a’4(0.2)0 1

0

0

Which code is preferred? The one with minimum variance!

Exercisee h l o p t w

30.5% 13.4% 9.8% 16.1% 5% 22.8% 2.4%

e (30.5) h (13.4) o (16.1) p (5.0) w (2.4) l (9.8) t (22.8)

Length of Huffman Codes

• A source S with A = {a1,…,ak} and {P(a1),…,P(ak)}– Average codeword length:

• Lower and upper bounds

K

iii laPl

1

)(

1)()( SHlSH

entropy of the source

average code length

Extended Huffman Codes (1)

• Consider small alphabet and skewed probabilities– Example:

• Block multiple symbols together

symbol Prob. codeword

a 0.9 0

b 0.1 11 bit / letterNo compression!

symbol Prob. codeword

aa 0.81 0

ab 0.09 10

ba 0.09 110

bb 0.01 111 0.645 bit / letter

Extended Huffman Codes (2)

Another example:

H = 0.816 bits/symbol = 1.2 bits/symbol

H = 1.6315 bits/block = 0.816 bits/symbol = 1.7228 / 2 = 0.8614 bits/symbol

sizeblkSHlSH

1)()(

Adaptive Huffman Coding (1)

• No initial knowledge of source distribution• One-pass procedure• Based on statistics of encountered symbols

Maintain a Huffman Tree!

Adaptive Huffman Coding (2)

• Huffman tree• Node (id, weight)• Sibling property

– w(parent) = sum of w(children)– ids are ordered with non-

decreasing weights

weight: # of occurrence

Id: 1 2 3 4 5 6 7 8 9 10 11w: 2 3 5 5 5 6 10 11 11 21 32

Non-decreasing!

Adaptive Huffman Coding (3)• NYT (not yet transmitted) code

– w(NYT) = 0– Transmitted when seeing

a new letter– Smallest id in the tree

• Uncompressed code (ex: m letters)

Adaptive Huffman Coding: Encode (1)

Input: [a a r d v a r k] (Alphabet: 26 lowercase letters)Initial tree

a00000

Output: 00000

a1

1

NYT0

r10001

010001

NYT000

v1011

1

2

3

5

001011

Adaptive Huffman Coding: Encode (2)

Input: [a a r d v a r k] (Alphabet: 26 lowercase letters)

NYT00

d00011

Output: 0000010100010000011

Adaptive Huffman Coding: Encode (3)

1

2

3

5

Swap 47 and 48

3 2

5

Tree update

Adaptive Huffman Coding: Encode (4)

Swap 49 and 50

3 2

5

49 50

4748

45

43 44

46

Tree update

Adaptive Huffman Coding: Decode (1)

Input: 0000010100010000011001011…Initial tree

0000Not in the uncompressed codeGet one more bit

Output:

00000a

a

1a

a

Adaptive Huffman Coding: Decode (2)

Input: 0000010100010000011001011…

0NYT

1000Not in the uncompressed codeGet one more bit

00NYT

Output: aa

10001r

r ……

Arithmetic Coding

• Basic algorithm• Implementation

Small alphabetSkewed probability

Cases where Huffman Coding doesn’t work well

Letter Probability Codeword

a1 0.95

a2 0.02

a3 0.03

01110

H = -0.95*log(0.95)-0.02*log(0.02)-0.03*log(0.03)=0.335 bits/symbolAverage length = 0.95*1+0.02*2+0.03*2 = 1.05 bits/symbol

Average length = 1.222 bits/block = 0.611 bits / symbol

Huffman codes for large blocks

• # {codeword} grows exponential with block size– N symbols, group m symbols for a block =>

Nm codewords

• Generate codes for all sequences given a length m

• Not efficient!

Arithmetic Coding: Generate a tag

• View the entire sequence as a big block– Step 1: Map the sequence into a unique tagEx: A = {a1, a2, a3}, P(a1) = 0.7, P(a2) = 0.1, P(a3) = 0.2

Encode a1, a2, a3, …

– Step 2: Generate a binary code for the tag

0.0

1.0

0.70.8

a1

a2

a3

0.0

0.7

0.490.56

a1

a2

a3

0.49

0.56

0.5390.546

a1

a2

a3

0.546

0.56

0.55580.5572

a1

a2

a30.553

Arithmetic Coding: Interpret the tag

Ex: 0.553

0.0

1.0

0.70.8

a1

a2

a3

0.0

1.0

0.70.8

a1

a2

a3

Update the number(0.553-0.49)*(0.56-0.49)= 0.9

0.0

1.0

0.70.8

a1

a2

a3

a1

l(1) = 0.0u(1) = 0.7

a3

l(3) = 0.7+(0.8-0.7)*0.8u(3) = 0.7 +(0.8-0.7)*1.0

Update the number:(0.553-0.0)/(0.7-0.0)= 0.79

a2

l(2) = 0.0+(0.7-0.0)*0.7=0.49u(2) = 0.0+(0.7-0.0)*0.8=0.56

One more example

• A = {a1, a2, a3}, P(a1) = 0.8, P(a2) = 0.02, P(a3) = 0.18Encode a1, a3, a2, a1

0.0

1.0

0.80.82

a1

a2

a3

0.0

0.8

0.640.656

a1

a2

a3

0.656

0.8

0.77120.77408

a1

a2

a3

0.7712

0.77408

0.7735040.77356

a1

a2

a3

(0.7712+0.773504)/2= 0.772352

Ex: 0.772352

0.0

1.0

0.80.82

a1

a2

a3

(0.772352-0.0)/(0.8-0.0)= 0.96544

0.0

1.0

0.80.82

a1

a2

a3

(0.772352-0.656)*(0.8-0.656)= 0.808

0.0

1.0

0.80.82

a1

a2

a3

l(2) = 0.0+(0.8-0.0)*0.82=0.656u(2) = 0.0+(0.8-0.0)*1.0=0.8

l(1) = 0.0u(1) = 0.8

l(3) = 0.656+(0.8-0.656)*0.8=0.7712u(3) = 0.656+(0.8-0.656)*0.82=0.77408

(0.772352-0.7712)*(0.77408-0.0.7712)= 0.4

0.0

1.0

0.80.82

a1

a2

a3

Generating a binary code

• Use the binary representation of the tag– Truncate to bits– probability↗, interval↗, required bits↘Ex:

– Bounds:

Symbol Prob. Cdf Tag In binary Code

a1 0.5 0.5

a2 0.25 0.75

a3 0.125 0.875

a4 0.125 1.0

0.0

1.0

0.5

0.75

a1

a2

a40.875

a3

0.25

0.625

0.8125

0.9375

.0100

.1010

.1101

.1111

2

3

4

4

01

101

1101

1111An extreme case where the sequence has only one letter

lengthsequenceSHlSH

2)()(

Adaptive Arithmetic Coding

• A = {a1 , a2 , a3}

• Input sequence: a2 a3 a3 a2

0.0

1.0

0.33

0.67

a1

a2

a3

1/3

1/3

1/3

1/4

2/4

1/4

0.33

0.67

0.42

0.58

a1

a2

a3

0.60

0.63

1/5

2/5

2/5

0.58

0.67

a1

a2

a3

0.64

0.65

1/6

2/6

3/6

0.63

0.67

a1

a2

a3

1/7

3/7

3/7

0.64

0.65

a1

a2

a3

Implementation

• Two problems– Finite precision– Transmit the first bit after seeing the entire

sequence

Synchronized rescaling!

Incremental encoding!

Implementation: Encode (1)

• Incremental coding– Send the MSB when l(k) and u(k) have a common

prefix

• Rescaling– Map the half interval containing the code to [0, 1)

E1: [0, 0.5)E1(x) = 2xSend 0Left shift 1 bit

E2: [0.5, 1)E2(x) = 2(x-0.5)Send 1Left shift 1 bit

Implementation: Encode (2)

Example: a1, a3, a2, a1 (tag: 0.7734375)

0.0

1.0

0.80.82

a1

a2

a3

a1 a3 send 1

E1: [0, 0.5) -> [0, 1); E1(x) = 2xE2: [0.5, 1) -> [0, 1); E2(x) = 2(x-0.5)

a2

send 1

send 0

send 0

Implementation: Encode (3)

send 0

send 1

a1

Use 0.5send 10…0

0.7734375=(.1100011)2

How to stop? 1.Send the stream size2.Use EOF (00…0)

Implementation: Decode (1)Find the smallest interval (0.82-0.8=0.02) => 6 bits (2-6 < 0.02)

11000110…0.110001=0.765625decode a1

update code:(0.765625-0)/(0.8-0)=0.957decode a3

Input: 11000110…0

11000110…0

.100011=0.546875update code:(0.546875-0.312)/0.6-0.312=0.8155decode a2

11000110…0

11000110…0

11000110…0

11000110…0

11000110…0

Implementation: Decode (2)11000110…0

update code:(0.5-0.3568)/(0.54112-0.3568)=0.7769decode a1

.10=0.5

Enhancement (optional)

• One more mapping:– E3: [0.25, 0.75) -> [0, 1); E3(x) = 2(x-0.25)

How do we transfer information about an E3 mapping to the decoder?

E3 mapping

• E3E1

• E3E2

0.0

1.0

0.5

0.75

0.25

0.25

0.75

0.5

0.625

0.375

[¼, ½): send 0 1

0.0

1.0

0.5

0.75

0.25

0.25

0.75

0.5

0.625

0.375

[½, ¾): send 1 0

E3 mapping

• E3…E3E1

• E3…E3E2

m

m

0.0

1.0

0.5

0.75

0.25

0.25

0.75

0.5

0.625

0.375

0.375

0.625

0.5

0.5625

0.4375

…

[¼, ½): 01 [¼+⅛, ½): 011 [¼+⅛+…, ½): 011…1m

0.5

Send 1 0 0 … 0

Send 0 1 1 … 1

E3 mapping: EncodeExample: a1, a3, a2, a1 (tag: 0.7734375)

0.0

1.0

0.80.82

a1

a2

a3

a1 a3 send 1

E1: [0, 0.5) -> [0, 1); E1(x) = 2xE2: [0.5, 1) -> [0, 1); E2(x) = 2(x-0.5)E3: [0.25, 0.75) -> [0, 1); E3(x) = 2(x-0.25)

m = 1

a2

send 0

send 0

send 1 0 m = 0

E3 mapping: Encode

send 1

m = 1

a1

Use 0.5send 10…0

Output:11000110…0

E3 mapping: DecodeFind the smallest interval (0.82-0.8=0.02) => 6 bits (2-6 < 0.02)

11000110…0.110001=0.765625decode a1

update code:(0.765625-0)/(0.8-0)=0.957decode a3

Input: 11000110…0

11000110…0

.100011=0.5468752*(0.546875-0.25)=0.5938

m = 1

update code:(0.5938-0.124)/(0.7-0.124)=0.8155 decode a2

11000110…0 m = 0

11000110…0

11000110…0

11000110…0

m = 1

E3 mapping: Decode

m = 1

. 10…0 = 0.52*(0.5-0.25)=0.5update code:(0.5-0.2136)/(0.58224-0.2136)=0.7769 decode a1

Output:a1 a3 a2 a1

Next week

• In-class paper bidding• Decide how you distribute your points with

your partners before coming to the class!