No Slide Title · Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite...

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 1

Arithmetic coding

Elias coding

Arithmetic coding as finite-precision Elias

coding

Redundancy due to finite precision

Multiplication-free arithmetic coding

Context-adaptive coding


Elias coding

Entropy coding algorithm for sequences of

symbols x with general (conditional) probabilities

Representation of x by a subinterval of the unit interval [0,1)

Width of the subinterval is approximately equal to the probability fX(x)

Subinterval for x can be determined by recursive subdivision algorithm

Represent x by shortest binary fraction in the subinterval

Subinterval of width fX(x) is guaranteed to contain

one number that can be represented by L binary digits, with

fX x

0.1111

0.111

0.1101

0.11

0.1011

0.101

0.1001

0.1

0.0111

0.011

0.0101

0.01

0.0011

0.001

0.0001

0 0

1

2logL f X x


Example: Elias coding of memoryless binary source

1 3

0 14 4

X Xf f

0

1

1

4

nc

n nc a

0

1

4

0

1

1

16

4

16

1

7

64

16

64

0

37

256

28

256

1

121

1024

148

1024

1 1


Example: Elias coding of memoryless ternary source

Courtesy: T. Wiegand

f

Xa f

Xb

1

4 f

Xc

1

2


Elias coding: choose binary fraction in subinterval

Uniquely identify interval [cn,cn+an) by a binary fraction

Length of bit-string

Bit-string

bits

0. [ , )

n

n n n

L

bbbbbb b c c a

Ln

log2

1

an

1 h

X0:n1

x0:n1

1

ˆ 2 2 1n nL L

n n nc c c


Successive decoding

Elias bit string can be decoded symbol by symbol, starting with the

first symbol

For each symbol n=0,1,2, . . .

• Calculate the intervals [cn,cn+an) corresponding to all possible xn

• Determine the interval for which

• Emit the corresponding xn

ˆ ,n n n nc c c a


Elias coding for memoryless source

Elias coding algorithm

0 0

1

1

Initialize 0 and 1.

For each 0,1,

Update

Update

n n X n

n n n X n

c a

n

a a f x

c c a F x

Interval

width

Interval

lower limit

Start with unit

interval

n-th symbol

to be

encoded

Cumulative distribution

(excluding current symbol)

1

0

0 1 2 1

where { , , ,..., }

i

X i X j

j

X K

F f


Elias coding for changing probabilities

Elias coding algorithm

0 0

1

1

Initialize 0 and 1.

For each 0,1,

Update

Update

n

n

n n X n

n n n X n

c a

n

a a f x

c c a F x

Interval

width

Interval

lower limit

Start with unit

interval

n-th symbol

to be

encoded

Cumulative distribution

(excluding current symbol)

1

0

0 1 2 1

where { , , ,..., }

n n

n

i

X i X j

j

X K

F f


Elias coding for Markov random processes

Elias coding algorithm for Markov-p sources

: 1

: 1

0 0

1 | : 1

1 | : 1

Initialize 0 and 1.

For each 0,1,...

Set min ,

Update ,

Update ,

n n p n

n n p n

n n X n n p n

n n n X n n p n

c a

n

p p n

a a f x

c c a F x

X

X

x

x

Interval

width

Interval

lower limit

Start with unit

interval

n-th symbol

to be

encoded

Cumulative conditional

distribution (excluding

current symbol)

Up to p

previous

symbols


Arithmetic coding

Elias coding not practical for long symbol strings:

required arithmetic precision grows with string length

Finite precision implementations: “arithmetic coding”

Widely used in modern image and video compression algorithms:

JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

nxArithmetic

coder

Arithmetic

decoder

nx

PMF nX nf x PMF

nX nf x

bit stream c


Finite precision for arithmetic coding

0 0

1

1

Initialize 0 and 1.

For each 0,1,

Update

Update

n n X n

n n n X n

c a

n

a a f x

c c a F x

N-bit precision is

permissible, if

rounding down

P-bit approximation of

probabilities

At most N+P LSBs change,

except for a possible carry

affecting MSBs.

Each bit can be affected by

at most one carry.


Represent by

state variable Cn

Represent by

state variable rn

Send

Finite precision for arithmetic coding (cont.)

Output code string

Maximum value of rn can be limited by bit stuffing

MSBs that will 1 bits LSBs thatno longer change represent lowerthat might be

interval boundary foraffected by a further computationcarry

. 01111111 1

n

n

r N P

c xxxxxxxxx xxx cccccccc c


Inefficiency due to interval rounding

Recall: Subinterval of width fX(x) is guaranteed to contain one

number that can be represented by Ln binary digits, with

Hence, rounding one interval value an+1 increases bit string by

L

n log

2f

X0:n1

x0:n1

log2

an

fX

xn

an1

log2

2N 1 1

2N 1 log

21 21 N

1

ln221 N


2 2 2log 1 log log 1

( )1

ln 2

0

X X

X X

X X

X

X

X X

X X

x xX X

f x f xE f x E f x E

f x f xH X

f xE

f x

f x f xE f x f x

f x f x

Limited precision probabilities

Efficiency loss

provided that rounded

probabilities still add to 1


Example: inefficiency for binary source

due to representing smaller probability by 2-P

log

2p

rela

tive

bit

-rat

e in

crea

se


Interval width is normalized to by binary shift after

each symbol

Approximation, with :

Corresponds to approximating of p(L) within a factor of 2:

Factor can be lumped into external probabilities

Optimum depends somewhat on data, typically

JBIG

Multiplication-free binary arithmetic coding

2 3 or 0.708 or

3 4

( ) ( )

( ) (1 ( )) ( )

Ap L p L p L

Ap M A p L A p L A p L

1/ 2 1A

p(L) p(M)

p L p LA


MPS-LPS switch

Problem: interval width for symbol M can become negative

Solution: switch the role of M and L, if

Problem: interval for symbol M can be smaller than interval for L if

Solution: “Conditional exchange” of symbols, when problem occurs

Improves compression efficiency, implemented in JPEG and JBIG

1

0 if ,12

A p L p L A

12

p L

1 14 2

1 ,

2p L A p L p L A


Sufficiency of binary coders

Consider r.v. equivalent to a r.v.

No loss by conditionally encoding bits (in any order)

Supply arithmetic coder with symbol/probability pairs

Total number of (conditional) probabilities to be estimated per symbol x . . . same as with K-ary encoding of x

0,1, ,2 1KX

0

1

1K

B

B

B

B

MSB

LSB

0 1 0 1 0 1 2| | , ,K K

H X H

H B H B B H B B B B

B

0: 10 0 1 1 1 1 | 0: 1, , , , , , with 0,

k kK K k B kb p b p b p p f B b

11 2 2 2 1K K


Consider stationary binary Markov-k process with probabilities

Context labeling function, enumerating all possible context vectors

Backward probability estimation separately for each context

Adaptive probability estimation

nX

0: 1 ( ): 1| 0: 1 | ( ): 10, 0,

k k n n k nX k X n k nf f X Xx x “Context vector,”

2k possible

combinations

: 0,1 0,1, ,2 1k k x x

0 ( ): 1

0

0 ( ): 1 1 ( ): 1

,

, ,

n k n

n k n n k n

C np n

C n C n

x

x x

Count of zeroes that

have previously

occured in that context

Probability

of symbol

x[n]=0

Bias for low counts,

typically 1


Adaptive probability estimation (cont.)

Tracking changing statistics, compromise between

• Rapid adaptation (small past sample counts)

• Accurate estimates (large past sample counts)

Scaled count estimation algorithm

0 1

1 1

0 0

0 1 min 0 1 max

0 10 1

00

0 1

Initialize 0For 0,1, If 1 C 1 else C 1

If min C , or max C ,

C ; C2 2

Estimate 2

n

C Cn

xC

C

C C C C

C C

Cp n

C C


Context adaptive coding

2-d extension of Markov model

Markov conditional independence property

n“causal half-plane”

x n

“Coding

context”

n

fX n X

n

X n ,X n

fX n X

n

X n ,Xn


Context adaptive coding (cont.)

Coding context vector can be mapped into context label directly

without loss

• Feasible for binary images

• Example: JBIG uses 10 binary pixels as context label

For 8-bit images, number of different contexts

• Context might have to be clustered

• Combine with prediction

256


Reading

W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B.

Arps, “An overview of the basic principles of the Q-Coder

adaptive binary arithmetic coder,” IBM J. Res. Develop., vol.

32, no. 6, November 1988.

Witten, Radford, Neal, Cleary, “Arithmetic Coding for Data

Compression” Communications of the ACM, vol. 30, no. 6,

pp. 520-540, June 1987.

Moffat, Neal, Witten, “Arithmetic Coding Revisited,” ACM

Transactions on Information Systems, vol. 16, vo. 3, pp.

256–294, July 1998.

No Slide Title · Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite...

Documents

Transcript of No Slide Title · Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11 Finite...