Lossy compression - Stanford University

Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 1

Goal: Lower the bit-rate R by introducing some (acceptable)

distortion D of the signal X.

Lossy compression

Distortion D

Rate R Lossless coding

D=0

R H X


Topics in lossy compression

Information theoretical bounds for lossy compression:

rate distortion theory

R-D function for memoryless Gaussian sources with mean-squared

error distortion criterion

R-D function for Gaussian sources with memory

R-D function for images

Practical lossy compression techniques: quantization

Scalar quantization and vector quantization

Quantizer design for fixed length codes

Quantizer design for variable length codes

Embedded quantizers


Rate distortion theory calculates the minimum transmission

bit-rate R for a required picture quality

Results of rate distortion theory are obtained without

consideration of a specific coding method

Rate distortion theory

Decoder Coder Image

Source Distortion

d

x x̂

Bitrate at least

for distortion

R

d D


Distortion

Symbol (signal, image . . . ) x sent, received

Single-letter distortion measure:

Average distortion:

Distortion criterion:

ˆ( , ) 0

ˆ ˆ( , ) 0 for

x x

x x x x

ˆ,ˆˆ ˆ ˆ ˆ( , ) ( , ) , ( , )

X Xx xd X X E X X f x x x x

ˆ( , )d X X D

Maximum permissible

average distortion

x̂


Mutual information

"Mutual information" is the average information that random variables X and Y convey about each other

Reduction in uncertainty about X, if Y is observed

Reduction in uncertainty about Y, if X is observed

Properties

0 ( ; ) ( ; )

( ; ) ( )

( ; ) ( )

I X Y I Y X

I X Y H X

I X Y H Y

,

, 2

( ; ) ( ) ( | ) ( ) ( | )

( , )( , ) log

( ) ( )

X Y

X Yx yX Y

I X Y H X H X Y H Y H Y X

f x yf x y

f x f y


Rate distortion function

Definition:

Shannon’s Source Coding Theorem (and converse):

For a given maximum average distortion D, the rate

distortion function R(D) is the (achievable) lower bound for

the transmission bit-rate.

R(D) is continuous, monotonically decreasing for R>0 and

convex

Equivalently use distortion-rate function D(R)

ˆˆ: ( , )

ˆ( ) inf { ( ; )}X X

f d X X DR D I X X


Extension to continuous random variables

Differential entropy

Differential conditional entropy

Mutual information

Rate distortion function

2 2log logX X X

x

h X E f X f x f x dx

2 , 2

,

log , , log ,X YX Y X Y

x y

h X Y E f X Y f x y f x y dxdy

;I X Y h X h X Y h Y h Y X

ˆˆ: ( , )

ˆ( ) inf { ( ; )}X X

f d X X DR D I X X


Shannon lower bound

It can be shown that

Thus

Ideally, the source coder would introduce i.i.d. errors

that are statistically independent from the reconstructed

signal (not always possible!).

Shannon lower bound:

ˆ ˆ ˆ( | ) ( | )h X X X h X X

ˆ( ) inf{ ( ) ( | )}

ˆ( ) sup{ ( | )}

ˆ ˆ( ) sup{ ( | )}

d D

d D

d D

R D h X h X X

h X h X X

h X h X X X

ˆ( ) ( ) sup ( )d D

R D h X h X X

ˆX X

X̂


Shannon lower bound (cont.)

Mean squared error distortion measure:

Gaussian PDF possesses largest entropy for given variance

Equivalently

Distortion reduction by 6 dB requires 1 bit/sample

122

ˆ( ) ( ) sup ( )

log 2

d D

R D h X h X X

h X eD

2 212 2

2

h X RD Re


R(D) function for a memoryless Gaussian source and MSE distortion

Gaussian source, variance

Mean squared error

Rule of thumb: 6 dB 1 bit

R(D) for non-Gaussian

sources with the same

variance is always below

this Gaussian R(D) curve. 1 2

3 4

6 12 18 24

2

2ˆ{( ) }d E X X D

2

2

1010log [dB]SNRD

2

2

2 2

1( ) log

2

2 R

R DD

D R

SNR [dB]

R [bit]


Stationary, band-limited, jointly Gaussian source with power spectral density

Mean squared error distortion

R(D) function in parametric form

R(D) for non-Gaussian sources with the same power spectral density is always lower.

R(D) for Gaussian source with memory

( )xx

D() 1

2min ,

xx( )

d

R() 1

2max 0,

1

2log

xx

( )

d

2ˆ{( ) }d E X X D


R(D) for Gaussian source with memory

white noise

no signal transmitted

preserved spectrum

reconstruction error spectrum

( )xx

ˆˆ ( )xx


Rate distortion function for images

Signal model: Gaussian source with acf

Power spectral density

(neglecting aliasing):

0 0

6 dB

R

ssn

x,n

y

exp

0n

x

2 ny

2

ss

(x,

y)

2

0

21

x

2 y

2

0

2

3

2

0 ln(0.93)

correlation between

adjacent pixels

log ( , )ss x y

y x


Mean squared error criterion:

After numerical integration:

Rate distortion function for images (cont.)

2ˆ{( ) }D E S S

memoryless

Gaussian

Gaussian

with memory

0

increasing decreasing

SNR [dB]

Rat

e [b

its/

sam

ple

]

~2.3 bits

0 5 10 15 20 25 30 35 40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0


Gaussian scale mixture modeling

Images are not Gaussian

Local pixel differences or prediction errors possess Laplacian pdf

Transform coefficients (DCT, DWT, . . . ) possess Laplacian pdf

Gaussian scale mixture model

Exponentially distributed variance V, mean E[V]=2

,

Gaussian source

ss x y

V

2

2

1; 0

0

v

V

e vp v

else

,ss x yv


Gaussian scale mixture modeling

Infinite mixture of Gaussians with zero mean and

exponential variance distribution yields a Laplacian!

Elegant explanation of ubiquitous Laplacian pdfs in images

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.50

1

2

3

4

5

6

7

8

2 2

2

2

2

2

0

1

1

2

2

1 v

x

x v

Xp x e dvv

e

e

x


Rate reduction due to source splitting

Same distortion for each sub-source

(implies high-rate assumption)

Rate-distortion function, if 2 is known

Rate reduction relative to Gaussian with variance 2

2

2

22

0

2

1

1 1log

2ln 2 2

1lim ln 0.57721

1log

2

6

v

n

nk

R D e dvv

D D

nk

Euler's constant

0.4163732ln 2

bits

D v


Rate distortion function for images

memoryless

Gaussian

Gaussian

with

memory

SNR [dB]

Rat

e [b

its/

sam

ple

]

~2.3 bit

0 5 10 15 20 25 30 35 40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Gaussian scale mixture

with memory

~0.4 bits


Summary: rate distortion theory

Rate-distortion theory: minimum transmission bit-rate for

given distortion

Shannon Lower Bound assumes statistical independence

between distortion and reconstructed signal

R(D) for memoryless Gaussian source and MSE: 6 dB/bit

R(D) for Gaussian source with memory and MSE: encode

spectral components independently, introduce white noise,

suppress small spectral components

Theoretical gain ~2.3 bits/sample by exploiting spatial

redundancy in the video signal

Additional theoretical source splitting gain of

~0.4 bits/sample with Gaussian mixture model


Reading

Taubman, Marcellin, Chapter 3.1, “Rate-Distortion Theory”

T. M. Cover, J. A. Thomas, “Elements of Information

Theory,” Wiley 1991, Chapter 13: Rate Distortion Theory”

Lossy compression - Stanford University

Documents

Transcript of Lossy compression - Stanford University