[CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf ·...
-
Upload
truongkhuong -
Category
Documents
-
view
214 -
download
2
Transcript of [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf ·...
![Page 1: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/1.jpg)
Quantization
C.M. LiuPerceptual Signal Processing Lab College of Computer ScienceNational Chiao-Tung University
Office: EC538(03)5731877
http://www.csie.nctu.edu.tw/~cmliu/Courses/Compression/
![Page 2: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/2.jpg)
Contents
Quantization Problem
Uniform Quantization
Adaptive Quantization
Nonuniform Quantization
Entropy-Coded Quantization
Vector Quantization
Rate-Distortion Function
2
![Page 3: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/3.jpg)
Quantization
Definition:
The process of representing a large—possibly infinite—set of values with a smaller set.
Example:Source:
Real numbers in the [-10.0, 10.0]
Quantization
Q(x) = ⎣x+0.5⎦[-10.0, -10.0] {-10, -9, …, -1, 0, 1, 2, …, 9, 10}
Scalar vs. vector quantizationScalar: applied to scalars
Vector: applied to vectors
3
![Page 4: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/4.jpg)
The Quantization Process
Two aspectsEncoder mapping
Map a range of values to a codewordIf source is analog A/D converterKnowledge of the source can help pick more appropriate ranges
Decoder mappingMap the codeword to a value in the rangeIf output is analog D/A converterKnowledge of the source distribution can help pick better approximations
Quantizer = encoder + decoder
4
![Page 5: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/5.jpg)
Quantization Example 5
![Page 6: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/6.jpg)
Quantizer Input-Output Map6
![Page 7: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/7.jpg)
Quantization Problem Formulation
Input:X – random variablefX(x) – probability density function (pdf)
Output:{bi}i=0..M decision boundaries
{yi}i=1..M reconstruction levels
Discrete processes are often approximated by continuous distributions
E.g.: Laplacian model of pixel differenceIf source is unbounded, then first/last decision boundaries = ±∞
7
![Page 8: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/8.jpg)
Quantization Error
Mean squared quantization error
8
( )
( ) dxfyx
dxfxQx
X
M
i
b
b i
Xq
i
i∑∫
∫
=
∞
∞−
−
−=
−=
1
2
22
1
)(σ
iii bxbyxQ ≤<= −1iff)(
Quantization error is a.k.a.Quantization noiseQuantizer distortion
![Page 9: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/9.jpg)
Quantization Problem Formulation (2)
Bit rates w/ fixed-length codewordsR = ⎡log2M⎤E.g.: M = 8 R = 3
Quantizer design problemGiven:
input pdf fX(x) & MFind:
decision boundaries {bi} and Reconstruction levels {yi}
Such that: MSQE is minimized
9
![Page 10: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/10.jpg)
Quantization Problem Formulation (3)
Bit rates w/ variable-length codewordsR depends on boundary selectionExample
10
∑ ==
M
i ii yPlR1
)(
∫−
=i
i
b
b Xi dxxfyP1
)()(
∑ ∫=−
=M
i
b
b Xii
i
dxxflR1
1
)(
![Page 11: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/11.jpg)
Quantization Problem Formulation (4)
Rate-optimization formulation
Given:
Distortion constraint σq2 ≤ D*
Find:{bi}, {yi}, binary codes
Such that:R is minimized
Distortion-optimization formulation
Given:Rate constraint R ≤ R*
Find:{bi}, {yi}, binary codes
Such that:
σq2 is minimized
11
![Page 12: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/12.jpg)
Uniform Quantizer
All intervals of the same sizei.e., boundaries are evenly spaced (Δ)Outer intervals may be an exception
ReconstructionUsually the midpoint is selected
Midrise quantizerZero is not an output level
Midtread quantizerZero is an output level
12
![Page 13: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/13.jpg)
Midrise vs. Midtread Quantizer13
Midrise Midtread
![Page 14: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/14.jpg)
Uniform Quantization of Uniform Source
InputUniform [-Xmax, Xmax]
OutputM-level uniform quantizer
14
MX max2
=Δ
1221
2122
22
1)1(
max
22 Δ
=⎟⎠⎞
⎜⎝⎛ Δ
−−= ∑∫
=
Δ
Δ−
M
i
i
iq dxX
ixσ
![Page 15: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/15.jpg)
Alternative MSQE Derivation
Consider quantization error instead:q = x – Q(x)
q ∈ [-Δ/2, Δ/2]
15
∫Δ
Δ−
Δ=
Δ=
2
2
222
121 dqqqσ
( ) ( )
dB02.6)2(log20)(log10
212
122log1012
122log10log10)dB(SNR
102
10
2max
2max
102
2max
102
2
10
nM
MX
XX
n
q
s
===
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎟⎠⎞
⎜⎝⎛
=⎟⎟⎠
⎞⎜⎜⎝
⎛Δ
=⎟⎟⎠
⎞⎜⎜⎝
⎛=
σσ
![Page 16: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/16.jpg)
Examples (8 1, 2, 3 bits/pixel)16
Darkening, contouring & dithering
![Page 17: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/17.jpg)
Uniform Quantization of Nonuniform Sources
Example nonuniform source:x ∈ [-100, 100], P(x ∈ [-1, 1]) = 0.95
ProblemDesign an 8-level quantizer
The naïve approach leads to95% of samples values represented by two numbers:
-12.5 and 12.5
Max quantization error (QE)= 12.5
Min QE = 11.5 (!)
Consider an alternativeStep = 0.3
Max QE = 98.5, however 95% of the time QE < 0.15
17
![Page 18: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/18.jpg)
Optimizing MSQE18
Numerically solvable for specific PDF
![Page 19: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/19.jpg)
Example Optimum Step Sizes19
![Page 20: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/20.jpg)
QE for 3-bit Midrise Quantizer20
![Page 21: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/21.jpg)
Overload/Granular Regions21
"4"4,stdev
valuegranularmax11 loadingff σ→==
The step selectionTradeoff between overload noise and granular noise.
![Page 22: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/22.jpg)
Variance Mismatch Effects22
![Page 23: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/23.jpg)
Variance Mismatch Effects (2)23
![Page 24: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/24.jpg)
Distribution Mismatch Effects
8-level quantizers, SNR
24
![Page 25: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/25.jpg)
Adaptive Quantization
IdeaInstead of a static scheme, adapt to the actual data:
Mean, variance, pdf
Forward adaptive (off-line)Divide source in blocksAnalyze block statisticsSet quantization scheme
Side channel
Backward adaptive (on-line)Adaptation based on quantizer outputNo side channel necessary
25
![Page 26: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/26.jpg)
Forward Adaptive Quantization (FAQ)
Choosing block sizeToo large
Not enough resolutionIncreased latency
Too smallMore side channel information
Assuming a mean of zeroVariance estimate:
26
∑ −
= +=1
022 1ˆ N
i inq xN
σ
![Page 27: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/27.jpg)
Speech Quantization Example
16-bit speech 3-bit fixed
27
![Page 28: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/28.jpg)
Speech Quantization Example (2)
16-bit speech 3-bit FAQBlock = 128 samples
8-bit variance quantization
28
![Page 29: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/29.jpg)
FAQ Refinement
So far we assumed uniform pdf
RefinementAssume uniform pdf butRecord min/max values for each block
Example:Sena image8x8 blocks3-bit quantizationOverhead = 16/8x8 = 0.25 bits/pixel
29
![Page 30: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/30.jpg)
FAQ Refinement Example30
Original: 8 bits/pixel Quantized: 3.25 bits/pixel
![Page 31: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/31.jpg)
Backward Adaptive Quantization (BAQ)
ObservationOnly encoder sees inputAdaptation can only be based on quantized output
ProblemHow do we deduce mismatch information from output only?It is possible, if we know the pdf …… and we are very patient
31
![Page 32: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/32.jpg)
Jayant Quantizer
IdeaIf input falls in the outer levels
Expand step size
If input falls in the inner levelsContract step size
The product of expansions & contraction should be 1
Multipliers: Mk
If input Sn-1 falls in the kth interval, thenstep is multiplied by Mk
Inner Mk < 1, outer Mk > 1
32
1)1( −− Δ=Δ nnln M
![Page 33: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/33.jpg)
3-bit Jayant Quantizer Output Levels33
![Page 34: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/34.jpg)
Jayant Example
M0 = M4 = 0.8, M1 = M5 = 0.9
M2 = M6 = 1.0, M3 = M7 = 1.2, Δ0 = 0.5
Input: 0.1 -0.2 0.2 0.1 -0.3 0.1 0.2 0.5 0.9 1.5
34
![Page 35: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/35.jpg)
Picking Jayant Multipliers
Δmin / Δmax to prevent under/overflow.
Adaption speed affected by γ.
35
⎣ ⎦
∑∏
∏
∏
∏
==
=
=
=
=⇒=
=>=
==
=
=
M
kkk
M
k
Pl
kkl
k
M
k
kk
P
M
k
Nn
M
k
n
Pl
llMLetNnPM
M
M
kk
k
k
k
k
k
k
k
00
0
0
0
01
,1where,
where,1
1
1
γ
γγ
![Page 36: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/36.jpg)
Jayant Example36
“Ringing”
![Page 37: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/37.jpg)
Jayant Performance37
Expands more rapidly than contracts to avoid overload errors.Robustness over changing input statistics.
![Page 38: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/38.jpg)
Non-uniform Quantization
Idea: Pick the boundaries such that error is minimized
i.e., smaller/bigger step for smaller/bigger values
e.g.:
38
![Page 39: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/39.jpg)
Non-uniform Quantization-- pdf-optimized Quantization
Problem: Given fX, minimize MSQE:
39
( ) dxfyx X
M
i
b
b iqi
i∑∫
= −
−=1
22
1
σ
Set derivative w.r.t. yj to zero and solve for yj:
( )
( )dxxf
dxxfxy
j
j
j
j
b
b X
b
b X
j
∫∫
−
−=
1
1
Set derivative w.r.t. bj to zero and solve for bj:( ) 21 jjj yyb += +
![Page 40: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/40.jpg)
Non-uniform Quantization-- Lloyd-Max Algorithm
Observation:Circular dependency b/w bj and yj
Lloyd/Max/Lukaszewics/Steinhaus approach:Solve the two iteratively until an acceptable solution is found
Example:
40
![Page 41: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/41.jpg)
Non-uniform Quantization-- Lloyd-Max Algorithm (2)
Boundaries: { b1, b2, …, bM/2-1}b0 = 0, bM/2-1 = MAX_INPUT
Reconstruction levels: { y1, y2, …, yM/2-1}
41
( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=1
0
1
01
One equation, two unknowns: b1, y1Pick a value for b1 (e.g. b1 = 1), solve for y1 and continue:
112 2 yby += ( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=2
1
2
12
… and so on until all { bn} and { ym} are found
![Page 42: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/42.jpg)
Non-uniform Quantization-- Lloyd-Max Algorithm (3)Terminating condition:
42
12122 2ˆ −− += MMM yby
( ) ( )dxxfdxxfxy M
M
M
M
b
b X
b
b XM ∫∫−−
=2/
12/
2/
12/2
ε≤− 22 ˆMM yy
Else: pick a different b1 & repeat
![Page 43: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/43.jpg)
Non-uniform Quantization— Example: pdf-Optimized Quantizers
43
Significant improvement over the uniform quantizer.
![Page 44: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/44.jpg)
Lloyd-Max Quantizer Properties
1. MeanOUTPUT = MeanINPUT
2. VarianceOUTPUT ≤ VarianceINPUT
3. MSQE:
44
[ ]jj
M
jjxq bxbPy ≤≤−= −
=∑ 1
1
222 σσ
4. If N is a random variable representing QE
[ ] 2qXNE σ−=
5. Quantizer output and QE are orthogonal (uncorrelated)
[ ] 0,,,|)( 10 =MbbbNXQE K
![Page 45: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/45.jpg)
Mismatch Effects45
4-bit Laplacian pdf-optimized quantizer
![Page 46: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/46.jpg)
Companded Quantization (CQ)46
Compressor Uniform quantizer Expander
![Page 47: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/47.jpg)
CQ Example: Compressor47
![Page 48: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/48.jpg)
CQ Example: Uniform Quantizer48
![Page 49: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/49.jpg)
CQ Example: Expander49
![Page 50: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/50.jpg)
CQ Example: Equivalent Non-uniform Quantizer50
![Page 51: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/51.jpg)
Vector Quantization
Definition
in alues vector v) stic(determini L of one only takemay it in that
on distributi special a has y vector e vector thrandom ldimensiona -N : ,
variablesrandom real : 1 , )( , )(
)]( .......... )2( )1([ )]( .......... )2( )1([
NR
yx
Niiyix
NyyyyNxxxx
≤≤
==
![Page 52: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/52.jpg)
Vector Quantization (c.1)
Vector quantization
the vector quantization of x may be viewed as a pattern recognition problem involving the classification of the outcomes of the random variable x into a discrete number of categories or cell in N-space in a way that optimizes some fidelity criterion, such as mean square distortion.
y Q x= ( )
![Page 53: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/53.jpg)
Vector Quantization (c.2)
VQ Distortion
VQ Optimizationminimize the average distortion D.
D P x C E d x y x C
d x yR l l l
k k kk
L
kN
= ∈ ∈=
∞
∑ ( ) { ( , )| }
( , )1
1 2
are typically the distance measures in , including , , norm
![Page 54: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/54.jpg)
Vector Quantization (c.3)
Two conditions for optimalityNearest Neighbor Selection
minimize average distortion
=> applied to partition the N-dimensional space into cell when the joint pdf is known.
.1,for ),( ),( , )(
LjjkyxdyxdiffCxyxQ
jk
kk
≤≤≠≤∈=
NnxCxy
ky
ky
k
ddfyxd
CxyxdEDy
kξξξξ .....)....(),(....=
}|),({
11minarg
minargminarg
∈∫∫
∈===
{ , }C k Lk 1≤ ≤ fx( )•
![Page 55: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/55.jpg)
55
![Page 56: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/56.jpg)
Appendix C: Rate-Distortion Functions
Introduction
Rate-Distortion Function for a Gaussian Source
Rate-Distortion Bounds
Distortion Measure Methods
![Page 57: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/57.jpg)
1. Introduction
Considering questionGiven a source-user pair and a channel, under what conditions is it possible to design a communication system that reproduces the source output for the user with an average distortion that does not exceed some specified upper limit D?
The capacity (C) of a communication channel. The rate distortion function ( R(D) )of a source-user pair.
Rate-distortion function R(D)A communication system can be designed that achieves fidelity D if and only if the capacity of the channel that connects the source to user exceeds R(D).
The lower limit for data compression to achieve a certain fidelity subject to a predetermined distortion measure D.
![Page 58: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/58.jpg)
1. Introduction (cont.)
Equations representations :D is to r t io n D :
M u tu a l in fo rm a tio n :
R a te d is to r t io n fu n c t io n R (D ) :
d is to r t io n m e a su re fo r th e s o u rc e w o rd
= (
D d q p x q y x x y d x d y
I q p x q y xq y x
q yd x d y
R D I q Q q y x d q D
x y
x x
q Qd
n
D
= =
=
= = =
∫ ∫
∫ ∫
∈
( ) ( ) ( | ) ( , )
( ) ( ) ( | ) lo g( | )
( )
( ) in f ( ) , { ( | ) : ( ) }
( , ) :
, . . . , )
ρ
ρx 1 r e p ro d u c e d a s = (y
T h e fa m ily is c a l le d th e
s in g le - le t te r f id e li ty c r i te r io n g e n e ra te d b y .
1
n
y
x ,y
,. . . , )
( ) ( , )
{ , }
y
n x y
F n
n
t t
t
n
n
ρ ρ
ρ
ρρ
=
= ≤ < ∞
−
=∑1
1
1
![Page 59: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/59.jpg)
2. Rate-Distortion Bounds
Introduction
Rate-Distortion Function for A Gaussian Source R(D) for a memoryless Gaussian sourceSource coding with a distortion measure
Rate-Distortion Bounds
Conclusions
![Page 60: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/60.jpg)
3. Rate-Distortion Function for A Gaussian Source
Rate-Distortion for a memoryless Gaussian source
The minimum information rate (bpn) necessary to represent the output of a discrete-time, continuous-amplitude, memorylessstationary Gaussian source based on an MSE distortion measure per symbol.
R DD D
Dg
x x
x
( )log ( ),
,=
≤ ≤
≥
⎧⎨⎪
⎩⎪
12 2
2 2
20
σ σ
σ
0
![Page 61: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/61.jpg)
3. Rate-Distortion Function for A Gaussian Source (c.1)
Source coding with a distortion measure (Shannon, 1959)
There exists a coding scheme that maps the source output into codewords such that for any given distortion D, the minimum rate R(D) bpn is sufficient to reconstruct the source output with an average distortion that is arbitrarily close to D.
Transform the R(D) to distortion-rate function D(R)
21010
22
log106)(log10dBin Express
2)(
xg
xR
g
RRD
RD
σ
σ
+−=
= −
![Page 62: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/62.jpg)
Comparison between different quantizations
3. Rate-Distortion Function for A Gaussian Source (c.2)
![Page 63: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/63.jpg)
4. Rate-Distortion Bounds
Source:Memoryless, continuous-amplitude source with zero mean and finite variance with respect to the MSE distortion measure.
Upper boundAccording to the theorem of Berger (1971), it implies that the Gaussian source requires the maximum rate among all other sources for a specified level of mean square distortion.
R DD
R D D
D R D R
xg x
gR
x
( ) log ( ),
( ) ( )
≤ = ≤ ≤
≤ = −
12
0
2
2
22
2 2
σσ
σ
σx2
![Page 64: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/64.jpg)
Lower bound (Shannon lower bound)
where
4. Rate-Distortion Bounds (c.1)
)(2*
2*
22
1)(
2log21)()(
xHR
eRD
eDxHDR
−−=
−=
π
π
DeDeDR
exH
exf
xx
xg
x
xx
x
2
222
2*
22
2
log212log
212log
21)(
2log21)(
21)(
:sourceGaussian For 22
σπσπ
σπ
σπσ
=−=⇒
=
= −
∫∞
∞−−= ξξξ dffxH
xH
nxnx
def)(log)()(
entropy aldifferenti :)(
)(2)(
)()()(
)()()(*
*
RDRDRD
DRDRDR
g
g
≤≤
≤≤
![Page 65: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/65.jpg)
For Gaussian source, the rate-distortion, upper bound and lower bound are all identical to each other.The bound of differential entropy
4. Rate-Distortion Bounds (c.2)
)(by boundedupper isentropy aldifferenti The
)]()([6
)]()([6)()(
log10
)]()([66)(log10
*
*10
*10
xH
DRDR
xHxHRDRD
xHxHRRD
g
g
gg
g
⇒
−=
−=
−−−=
![Page 66: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/66.jpg)
Rate-distortion R(D) to channel capacity CFor C > Rg(D)
The fidelity (D) can be achieved.
For R(D)<C< Rg(D) Achieve fidelity for stationary sourceMay not achieve fidelity for random source
For C<R(D)Can not be sure to achieve fidelity
4. Rate-Distortion Bounds (c.3)
![Page 67: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/67.jpg)
4. Rate-Distortion Bounds (c.4)
![Page 68: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/68.jpg)
5. Distortion Measure Methods
{ }GG m r k c x m
k rcr
c rr
rrrrr
r( , , ) exp ( )
( / )( / )
( / ):
σ
σ
= − −
= =
= →= →= →= ∞ →
and
For different Laplacian pdf Gaussian pdf constant pdf uniform pdf
2 13
1
120
2ΓΓ
Γ
![Page 69: [CSCI 6990-DC] 09: Scalar Quantizationpeople.cs.nctu.edu.tw/~cmliu/Courses/Compression/chap9.pdf · Map a range of values to a codeword ... 0 1 K E Q X N b b b M] =( ) | , , , 0.](https://reader031.fdocuments.in/reader031/viewer/2022030502/5aaef27f7f8b9a25088ce327/html5/thumbnails/69.jpg)
Problems
Homeworks (Sayood 3rd, pp. 270-271)
3, 6.
Referenceshttp://en.wikipedia.org/wiki/Rate_distortion_theory.J.R. Deller, J.G. Proakis, and J.H.L. Hansen, “Discrete-Time Processing of Speech Signals,” IEEE Press
69