Post on 03-Feb-2022
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 1
Goal: Lower the bit-rate R by introducing some (acceptable)
distortion D of the signal X.
Lossy compression
Distortion D
Rate R Lossless coding
D=0
R H X
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 2
Topics in lossy compression
Information theoretical bounds for lossy compression:
rate distortion theory
R-D function for memoryless Gaussian sources with mean-squared
error distortion criterion
R-D function for Gaussian sources with memory
R-D function for images
Practical lossy compression techniques: quantization
Scalar quantization and vector quantization
Quantizer design for fixed length codes
Quantizer design for variable length codes
Embedded quantizers
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 3
Rate distortion theory calculates the minimum transmission
bit-rate R for a required picture quality
Results of rate distortion theory are obtained without
consideration of a specific coding method
Rate distortion theory
Decoder Coder Image
Source Distortion
d
x x̂
Bitrate at least
for distortion
R
d D
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 4
Distortion
Symbol (signal, image . . . ) x sent, received
Single-letter distortion measure:
Average distortion:
Distortion criterion:
ˆ( , ) 0
ˆ ˆ( , ) 0 for
x x
x x x x
ˆ,ˆˆ ˆ ˆ ˆ( , ) ( , ) , ( , )
X Xx xd X X E X X f x x x x
ˆ( , )d X X D
Maximum permissible
average distortion
x̂
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 5
Mutual information
"Mutual information" is the average information that random variables X and Y convey about each other
Reduction in uncertainty about X, if Y is observed
Reduction in uncertainty about Y, if X is observed
Properties
0 ( ; ) ( ; )
( ; ) ( )
( ; ) ( )
I X Y I Y X
I X Y H X
I X Y H Y
,
, 2
( ; ) ( ) ( | ) ( ) ( | )
( , )( , ) log
( ) ( )
X Y
X Yx yX Y
I X Y H X H X Y H Y H Y X
f x yf x y
f x f y
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 6
Rate distortion function
Definition:
Shannon’s Source Coding Theorem (and converse):
For a given maximum average distortion D, the rate
distortion function R(D) is the (achievable) lower bound for
the transmission bit-rate.
R(D) is continuous, monotonically decreasing for R>0 and
convex
Equivalently use distortion-rate function D(R)
ˆˆ: ( , )
ˆ( ) inf { ( ; )}X X
f d X X DR D I X X
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 7
Extension to continuous random variables
Differential entropy
Differential conditional entropy
Mutual information
Rate distortion function
2 2log logX X X
x
h X E f X f x f x dx
2 , 2
,
log , , log ,X YX Y X Y
x y
h X Y E f X Y f x y f x y dxdy
;I X Y h X h X Y h Y h Y X
ˆˆ: ( , )
ˆ( ) inf { ( ; )}X X
f d X X DR D I X X
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 8
Shannon lower bound
It can be shown that
Thus
Ideally, the source coder would introduce i.i.d. errors
that are statistically independent from the reconstructed
signal (not always possible!).
Shannon lower bound:
ˆ ˆ ˆ( | ) ( | )h X X X h X X
ˆ( ) inf{ ( ) ( | )}
ˆ( ) sup{ ( | )}
ˆ ˆ( ) sup{ ( | )}
d D
d D
d D
R D h X h X X
h X h X X
h X h X X X
ˆ( ) ( ) sup ( )d D
R D h X h X X
ˆX X
X̂
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 9
Shannon lower bound (cont.)
Mean squared error distortion measure:
Gaussian PDF possesses largest entropy for given variance
Equivalently
Distortion reduction by 6 dB requires 1 bit/sample
122
ˆ( ) ( ) sup ( )
log 2
d D
R D h X h X X
h X eD
2 212 2
2
h X RD Re
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 10
R(D) function for a memoryless Gaussian source and MSE distortion
Gaussian source, variance
Mean squared error
Rule of thumb: 6 dB 1 bit
R(D) for non-Gaussian
sources with the same
variance is always below
this Gaussian R(D) curve. 1 2
3 4
6 12 18 24
2
2ˆ{( ) }d E X X D
2
2
1010log [dB]SNRD
2
2
2 2
1( ) log
2
2 R
R DD
D R
SNR [dB]
R [bit]
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 11
Stationary, band-limited, jointly Gaussian source with power spectral density
Mean squared error distortion
R(D) function in parametric form
R(D) for non-Gaussian sources with the same power spectral density is always lower.
R(D) for Gaussian source with memory
( )xx
D() 1
2min ,
xx( )
d
R() 1
2max 0,
1
2log
xx
( )
d
2ˆ{( ) }d E X X D
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 12
R(D) for Gaussian source with memory
white noise
no signal transmitted
preserved spectrum
reconstruction error spectrum
( )xx
ˆˆ ( )xx
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 13
Rate distortion function for images
Signal model: Gaussian source with acf
Power spectral density
(neglecting aliasing):
0 0
6 dB
R
ssn
x,n
y
exp
0n
x
2 ny
2
ss
(x,
y)
2
0
21
x
2 y
2
0
2
3
2
0 ln(0.93)
correlation between
adjacent pixels
log ( , )ss x y
y x
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 14
Mean squared error criterion:
After numerical integration:
Rate distortion function for images (cont.)
2ˆ{( ) }D E S S
memoryless
Gaussian
Gaussian
with memory
0
increasing decreasing
SNR [dB]
Rat
e [b
its/
sam
ple
]
~2.3 bits
0 5 10 15 20 25 30 35 40
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 15
Gaussian scale mixture modeling
Images are not Gaussian
Local pixel differences or prediction errors possess Laplacian pdf
Transform coefficients (DCT, DWT, . . . ) possess Laplacian pdf
Gaussian scale mixture model
Exponentially distributed variance V, mean E[V]=2
,
Gaussian source
ss x y
V
2
2
1; 0
0
v
V
e vp v
else
,ss x yv
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 16
Gaussian scale mixture modeling
Infinite mixture of Gaussians with zero mean and
exponential variance distribution yields a Laplacian!
Elegant explanation of ubiquitous Laplacian pdfs in images
-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.50
1
2
3
4
5
6
7
8
2 2
2
2
2
2
0
1
1
2
2
1 v
x
x v
Xp x e dvv
e
e
x
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 17
Rate reduction due to source splitting
Same distortion for each sub-source
(implies high-rate assumption)
Rate-distortion function, if 2 is known
Rate reduction relative to Gaussian with variance 2
2
2
22
0
2
1
1 1log
2ln 2 2
1lim ln 0.57721
1log
2
6
v
n
nk
R D e dvv
D D
nk
Euler's constant
0.4163732ln 2
bits
D v
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 18
Rate distortion function for images
memoryless
Gaussian
Gaussian
with
memory
SNR [dB]
Rat
e [b
its/
sam
ple
]
~2.3 bit
0 5 10 15 20 25 30 35 40
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Gaussian scale mixture
with memory
~0.4 bits
Bernd Girod: EE398A Image and Video Compression Rate Distortion Theory no. 19
Summary: rate distortion theory
Rate-distortion theory: minimum transmission bit-rate for
given distortion
Shannon Lower Bound assumes statistical independence
between distortion and reconstructed signal
R(D) for memoryless Gaussian source and MSE: 6 dB/bit
R(D) for Gaussian source with memory and MSE: encode
spectral components independently, introduce white noise,
suppress small spectral components
Theoretical gain ~2.3 bits/sample by exploiting spatial
redundancy in the video signal
Additional theoretical source splitting gain of
~0.4 bits/sample with Gaussian mixture model