7.Transform coding 651 - eecs.umich.edu · Ui's than it increases for the small Ui's. We will show:...
27
Tr-1 T RANSFORM C ODING orthogonal transformation T T -1 Q X U 2 U 1 V V Y 2 U 1 V 2 Q 1 Q k U k Scalar Quantizers . . . V k R bits 2 R bits 1 R bits k A way to use scalar quantizers to efficiently encode dependent sources. Assume, as usual, the X 1 , X 2 , … is a stationary or WSS random process. Tr-2 Encoding: Instead of quantizing each X i with a scalar quantizer 1) Parse data sequence into blocks of some length k, X 1 , X 2 , … = X 1 , X 2 ... where X i = (X (i-1)k+1 ,…,X ik ); encode each X i as follows. 2) Find and scalar quantize the coefficients of an orthonormal expansion of X (= X i ) with respect to some orthonormal basis t 1 ,…, t k . ( t i = (t i1 ,…,t ik ) t ) a) Find coefficients U 1 ,...,U k such that X = U 1 t 1 + ... + U k t k b) Scalar quantize each U i with a quantizer that is optimized for it. The quantized values are V 1 = Q 1 (U 1 ), …, V k = Q k (U k ) The encoder output is B = e 1 (U 1 ), …, e k (U k ) Decoding: The decoder receives e 1 (U 1 ), …, e k (U k ) and produces Y = V 1 t 1 + ... + V k t k
Transcript of 7.Transform coding 651 - eecs.umich.edu · Ui's than it increases for the small Ui's. We will show:...
Tr-1
TR
AN
SFO
RM
CO
DIN
G
orth
ogon
altr
ansf
orm
atio
n T
T-1
QX
U2
U
1V
VY
2
U1
V2
Q1
Qk
Uk
Sca
lar
Qua
ntiz
ers
.. .V
k
R
bits
2
R
bits
1
R
bits
k
A w
ay t
o us
e sc
alar
qua
ntiz
ers
to e
ffici
ently
enc
ode
depe
nden
t so
urce
s.
Ass
ume,
as
usua
l, th
e X
1, X
2, …
is
a s
tatio
nary
or
WS
S r
ando
mp
roce
ss.
Tr-2
En
cod
ing
: I
nste
ad o
f qu
antiz
ing
each
X
i w
ith a
sca
lar
quan
tizer
1) P
arse
dat
a se
quen
ce in
to b
lock
s of
som
e le
ngth
k,
X1,
X2,
… =
X1,
X2
...
whe
re
Xi
=
(X(i-
1)k+
1,…
,Xik
);
enco
de e
ach
Xi
as f
ollo
ws.
2) F
ind
and
scal
ar q
uant
ize
the
coef
ficie
nts
of a
n or
thon
orm
al e
xpan
sion
of
X
(= X
i) w
ith r
espe
ct t
o so
me
orth
onor
mal
bas
is
t 1,…
,t k.
(t
i =(t
i1,…
,t ik)
t )
a) F
ind
coef
ficie
nts
U1,
...,U
k s
uch
that
X =
U1t
1 +
... +
Uk
t k
b) S
cala
r qu
antiz
e ea
ch
Ui
with
a q
uant
izer
tha
t is
opt
imiz
ed f
or it
.
The
qua
ntiz
ed v
alue
s ar
e
V1
= Q
1(U
1), …
, V
k =
Qk(
Uk)
The
enc
oder
out
put
is
B =
e1(
U1)
, …, e
k(U
k)
Dec
od
ing
: T
he d
ecod
er r
ecei
ves
e1(
U1)
, …
, e k
(Uk)
an
d pr
oduc
es
Y
=
V1t
1 +
... +
Vk
t k
Tr-3
Key
Id
eas:
A:
Cho
ose
t1,
…,t k
so
tha
t U
1,...
,Uk
are
unc
orre
late
d.
(If
they
wer
e co
rrel
ated
, th
en d
ecor
rela
ting
wou
ld s
eem
to
do b
ette
r.)
B:
Cho
ose
t1,
…,t k
so
tha
t re
lativ
ely
few
U
i's
are
larg
e.
(In
this
cas
e,th
e tr
ansf
orm
is s
aid
to d
o "e
nerg
y co
mpa
ctio
n.)
T
his
mea
ns t
hat
the
sour
ce v
ecto
r X
is
des
crib
ed p
retty
wel
l with
just
a f
ew o
f th
eba
sis
func
tions
.
Use
hig
her
rate
sca
lar
quan
tizer
s on
the
U
i's
that
are
larg
er o
n th
eav
erag
e, t
han
on t
he
Ui's
th
at a
re s
mal
ler
on t
he a
vera
ge.
It re
mai
ns t
o be
see
n th
at d
isto
rtio
n is
red
uced
mor
e fo
r th
e la
rge
Ui's
th
an it
incr
ease
s fo
r th
e sm
all U
i's.
We
will
sho
w:
A
⇔ B
.
Tr-4
Co
mp
lexi
ty:
sto
rag
eo
ps/
sam
ple
Enc
odin
gk2 +
∑ i=1k M
i n
um
be
rs 2
k +
1 k ∑ i=
1k lo
g 2 M
i = 2
k+R
op
s/sa
mpl
e
Dec
odin
gk2
+ ∑ i=
1k M
i nu
mbe
rs
2k
op
s/sa
mp
le
(
tran
sfor
m +
qua
ntiz
ers)
(t
rans
form
+ q
uant
izer
s)
(Ass
umin
g tr
ansf
orm
is d
one
by m
atrix
mul
tiplic
atio
n in
k2
op
s, a
ndqu
antiz
er p
artit
ioni
ng d
one
in
log 2
Mi
op's
“bi
nary
sea
rch”
Tr-5
MA
TH
EM
AT
ICA
L D
ET
AIL
S (F
RO
M L
INE
AR
AL
GE
BR
A)
Def
init
ion
s:
•Rk
=
set
of r
eal-v
alue
d k-
dim
ensi
onal
vec
tors
of
the
form
x
= (
x 1,…
,xk)
.
We
ofte
n th
ink
of v
ecto
rs a
s co
lum
n ve
ctor
s.
•In
ner
or d
ot p
rodu
ct:
(x,
y)
= (
y, x
) =
xt y
= y
t x =
∑ i=1k x
i yi
•Le
ngth
of
x:
||x||
= √
∑ i=1k x
2 i =
√
(x,x
) =
√x
t x
•F
act:
||x
+ y|
|2 =
(x+
y)t (
x+y)
= |
|x||2 +
2(x
, y)
+ ||y
||2
•O
rtho
gona
lity:
x a
nd y
ar
e or
thog
onal
if
xty
= 0
.
•Li
near
ind
epen
denc
e:
x 1,…
,xm
ar
e lin
early
inde
pend
ent
if th
ere
exis
t no
coef
ficie
nts
u1,
…,u
m s
uch
that
u1
x 1 +
… +
uk x k
= 0
, ex
cept
u1=
u 2=
…=
uk=
0.
•B
asis
: A
bas
is f
or R
k is
a li
near
ly in
depe
nden
t se
t of
vec
tors
{ t
1,…
,t k}
in R
k th
at s
pan
Rk .
T
hat
is,
ever
y ve
ctor
x
∈ R
k is
a li
near
com
bina
tion
of t
he m
em-
bers
of
the
set;
i.e.
the
re a
re c
oeff’
s u
1,…
,um
su
ch th
at x
= u
1t1
+ …
+ u
k t k
•O
rtho
norm
al b
asis
for
R
k :
{t1,
…,t k
} s
uch
that
tha
t m
embe
rs o
f th
e ba
sis
are:
orth
ogon
al:
tt i tj =
0 w
hen
i ≠
j,
and
nor
mal
ized
: ||t
i|| =
1 f
or e
ach
i Tr-6
Fa
cts
:
•E
very
bas
is f
or R
k h
as e
xact
ly
k e
lem
ents
.
•T
here
exi
sts
an o
rtho
norm
al b
asis
for
R
k fo
r ev
ery
k,
e.g.
{(1
,0,…
,0),
(0,
1,0,
…,0
), …
, (0,
…,0
,1)}
•G
iven
a b
asis
{t 1
,…,t k
} a
nd a
vec
tor
x
ther
e is
exi
sts
one
and
only
one
set
of c
oeffi
cien
ts
u 1,…
,uk
suc
h th
at
x =
u1t
1 +
… u
k t k
•G
iven
an
orth
onor
mal
bas
is
t 1,…
,t k
and
a ve
ctor
x,
th
e un
ique
coe
ffici
ents
u 1,…
,uk
suc
h th
at x
= u
1t1
+ …
+ u
k t k
are
u i =
tt i x,
i = 1
,...,k
Pro
of:
If x
= u
1t1
+ ..
. + u
k t k
, th
en
tt i x =
tt i (
u 1 t 1
+...
+ u
k t k
) =
u1 tt i t
1 +
... +
uk
tt i tk
=
ui
sin
ce t
i's o
rtho
norm
al
•If
u =
(u 1
,…,u
k)
is t
he v
ecto
r of
coe
ffici
ents
of
the
expa
nsio
n of
x
with
resp
ect
to o
rtho
norm
al b
asis
{t
1,…
,t k},
th
en
||u||2
= ||x
||2
(T
his
is a
ver
sion
of
Par
seva
l’s T
heor
em)
Pro
of:
||x||2
= ||
u 1 t 1
+…+u
k t k
||2 =
(u 1
t 1+…
+uk t k
)t (u1 t 1
+…
+u k
t k)
= u
2 1 +
… +
u2 k
beca
use
tt i t
j = 1
, i=
j0
, i ≠
j
= |
|u||2
Tr-7
Rec
all
Ste
p 2
a o
f tr
ansf
orm
en
cod
ing
2) G
iven
an
orth
onor
mal
bas
is
t 1,…
,t k.
(t i
= (
t i1,…
,t ik)
t )
(a)
Fin
d co
effic
ient
s U
1,...
,Uk
suc
h th
at X
= U
1t1
+ ..
. + U
k t k
.
By
a pr
evio
us f
act,
Ui
= tt i X
=
∑ j=1k t i
j Xj ,
i =
1,..
.,k
Equ
ival
ently
, w
e ca
n co
mpu
te t
he c
oeffi
cien
ts v
ia m
atrix
mul
tiplic
atio
n
U =
U
1… U
k =
t 1
1 t 1
2 …
t 1k
t 21
t 22
… t 2
k... t k
1 t k
2 …
t kk
X
1X
2 ... Xk
= T
X
whe
re
T
is t
he
k×k
mat
rix
T =
--tt 1-
-
--tt 2-
-... --
tt k--
=
t 1
1 t 1
2 …
t 1k
t 21
t 22
… t 2
k... t k
1 t k
2 …
t kk
Mo
re D
efin
itio
ns:
•In
thi
s co
ntex
t th
e m
atrix
T
is
cal
led
a tr
ansf
orm
, an
d T
X
is c
alle
d a
tran
sfor
mat
ion
of
X.
U
is t
he v
ecto
r of
tra
nsfo
rm c
oeffi
cien
ts.
•A
mat
rix w
ith o
rtho
norm
al r
ows,
suc
h as
the
one
s w
e ar
e in
tere
sted
in,
is c
alle
d "o
rtho
gona
l".
("O
thon
orm
al"
wou
ld b
e a
bette
r na
me,
but
"ort
hogo
nal"
is
wha
t th
e m
athe
mat
icia
ns u
se.)
Tr-8
Fac
ts:
The
fol
low
ing
are
equi
vale
nt s
tate
men
ts
(a)
T
is a
n or
thog
onal
mat
rix,
(b)
Row
s of
T
ar
e or
thor
mal
(c)
Col
umns
of
T
are
orth
onor
mal
, (
d)
||Tx|
| = |
|x||
for
all x
(e)
T-1
= T
t , (
f) T
-1
is o
rtho
gona
l.
Pro
ofs
:(a
) ⇔
(b)
: B
y de
finiti
on o
f an
ort
hogo
nal m
atrix
(b)
⇔ (
e):
Not
ice
that
T T
t =
--tt 1-
-
--tt 2-
-... --
tt k--
| t 1 |
| t 2 | - - - | t k |
=
1
0 ...
.. 0
0 1
0 ..
0 0
.....
0 1
= I
= id
entit
ym
atr
ix
if an
d on
ly if
the
row
s of
T
are
ort
hono
rmal
. T
hus
Tt
is th
e in
vers
e of
T
iff
its
row
s ar
e or
thon
orm
al.
(N
ote:
by
defin
ition
of
the
inve
rse
T-1
T =
I =
T T
-1 .)
(c)
⇔ (
e):
By
a si
mila
r ar
gum
ent
Tt T
= I
iff
th
e co
lum
ns o
f T
ar
e or
thon
orm
al.
(b)
⇒ (
d):
Pro
ved
earli
er.
(e)
⇒ (d
):
(e)
⇒ |
|Tx|
|2 =
(T
x,T
x) =
(T
x)t T
x =
xt T
t Tx
= x
t x =
||x
||2
(d)
⇒ (
e):
Ass
umin
g ||
Tx|
| =
||x||
for
all x
, w
e ha
ve
0 =
||Tx|
|2 -
||x||2
= (
Tx,
Tx)
- (
x,x)
= (
Tx)
t Tx
- x
t x =
xt T
t Tx
- xt I
x =
xt (T
t T-I
) x
sinc
e th
is h
olds
for
all
x
it m
ust
be t
hat
Tt T
=I,
i.e.
Tt =
T-1
.
(e)
⇔ (
f):
E
lem
enta
ry.
Tr-9
Fac
t:
For
an
orth
ogon
al t
rans
form
atio
n ||
Tx-
Ty|
| = ||
x-y
||
Pro
of:
||T
x-T
y|| =
||T
(x-y
)||
= |
|x-y
||
by (
d) o
f the
pre
viou
s fa
ct.
Bec
ause
an
orth
ogon
al t
rans
form
atio
n pr
eser
ves
the
leng
th o
f ve
ctor
s an
dal
so t
he d
ista
nce
betw
een
vect
ors,
we
may
vie
w i
t as
ess
entia
lly a
rota
tion.
(T
here
mig
ht a
lso
be a
xis
flips
.)
Exa
mpl
es o
f or
thog
onal
tra
nsfo
rmat
ions
:
DF
Tdi
scre
te F
ourie
r tr
ansf
orm
DC
Tdi
scre
te c
osin
e tr
ansf
orm
DW
Tdi
scre
te w
avel
et t
rans
form
WH
TW
alsh
-Had
amar
d
Tw
o-di
men
sion
al v
ersi
ons
of t
he a
bove
KLT
K
arhu
nen-
Loev
e tr
ansf
orm
Tr-
10
EX
AM
PLE
: D
CT
BA
SIS
VE
CT
OR
S (O
NE
-DIM
EN
SIO
NA
L)
k =
8k
= 16
Tr-
11
Exa
mp
le:
(R
eal)
FF
T B
asis
Vec
tors
(o
ne-
dim
ensi
on
al)
k =
8k
= 16
Tr-
12
PE
RFO
RM
AN
CE
OF
TR
AN
SFO
RM
CO
DIN
G
orth
ogon
altr
ansf
orm
atio
n T
T-1
QX
U2
U
1V
VY
2
U1
V2
Q1
Qk
Uk
Sca
lar
Qua
ntiz
ers
.. .V
k
R
bits
2
R
bits
1
R
bits
k
Rat
e:
R =
1 k ∑ i=
1k R
(Qi)
=
aver
age
rate
of
the
scal
ar q
uant
izer
s Q
1,...
,Qk
=
rate
of
the
the
prod
uct
quan
tizer
Q
1×...
×Qk
Tra
nsfo
rm c
ode
is f
ixed
rat
e if
scal
ar q
uant
izer
s ar
e fix
ed-r
ate;
var
iabl
era
te if
sca
lar
quan
t'rs
are
varia
ble-
rate
.
Tr-
13
Dis
tort
ion
:
D=
1 k ∑ i=
1k E
(Xi-Y
i)2 =
1 k
E
∑ i=
1k (
Xi-Y
i)2 =
1 k E
||X
-Y||2
=
1 k E
||U
-V||2
sin
ce fo
r or
thog
onal
tran
sfor
m,
||X-Y
|| =
||U
-V||
=
1 k E
∑ i=
1k (
Ui-V
i)2
=
1 k ∑ i=1k E
(Ui-V
i)2
=
1 k ∑ i=
1k D
Ui(Q
i)
=
avg
dis
tort
ion
of t
he s
cala
r qu
antiz
ers
Q1,
...,Q
k
=
D U
1,...
,Uk(Q
1×...
×Qk)
=
di
stor
tion
of p
rodu
ct q
uant
izer
Q
1×...
×Qk
Thi
s is
why
we
rest
rict
to o
rtho
gona
l tr
ansf
orm
s.
Tr-
14
OPT
IMA
L D
ESI
GN
AN
D T
HE
OPT
A F
UN
CT
ION
•G
iven
a d
imen
sion
k
and
a r
ate
R,
we
wis
h to
opt
imiz
e k-
dim
ensi
onal
tran
sfor
m c
odin
g.
Tha
t is
, w
e w
ish
to f
ind
the
k×k
tr
ansf
orm
T
an
d th
esc
alar
qua
ntiz
ers
Q1,
...,
Qk
tha
t m
inim
ize
dist
ortio
n su
bjec
t to
the
rat
eco
nstr
aint
. W
e al
so w
ish
to f
ind
the
dist
ortio
n th
at r
esul
ts w
hen
the
code
par
amet
ers
are
optim
ized
. T
hat
is,
we
wis
h to
fin
d th
e O
PT
Afu
nctio
n
δ tr
(k,R
) =∆
le
ast
dist
ort’n
of
tran
sfor
m c
odes
with
dim
'n k
and
rat
e R
or
less
=
min T
m
in
Q1,
...,Q
k: 1 k
∑ i-1k R
(Qi)≤
R
1 k
∑ i=1k D
Ui(Q
i),
whe
re
U =
TX
=
min T
δU
,pr(R
),
whe
re
δ U,p
r(R
) =
O
PT
A f
unct
ion
of p
rodu
ct q
uant
izer
s fo
r U
=T
X
•W
e fir
st in
vest
igat
e δ
U,p
r(R
) t
he p
rodu
ct q
uant
izer
OP
TA
fun
ctio
n, a
ndw
e'll
lear
n ho
w t
o ch
oose
the
Q
i's.
If
all t
he
Ui's
ar
e id
entic
al r
ando
mva
riabl
es,
then
the
opt
imal
Q
i's
turn
out
to
be id
entic
al.
How
ever
, w
ew
ill s
ee t
hat
a "g
ood"
tra
nsfo
rm m
akes
the
U
i's
have
rat
her
diffe
rent
prob
abili
ty d
ensi
ties.
S
o fin
ding
the
Q
i's
is n
ot t
rivia
l.
•W
e w
ill t
hen
lear
n ho
w t
o ch
oose
T
to
min
imiz
e δ
U,p
r(R
).
•F
inal
ly,
we'
ll ex
plor
e th
e ef
fect
of
vary
ing
the
dim
ensi
on
k.
Tr-
15
•T
he O
PT
A f
unct
ion
of a
k-d
imen
sion
al p
rodu
ct q
uant
izer
δ U,p
r(R)
=
min
Q1,
...,Q
k: 1 k
∑ i-1k R
(Qi)
≤ R
1 k ∑ i=
1k D
Ui(Q
i),
w
here
U
= T
X
=
min
R1≥
0,...
,Rk≥
0: 1 k
∑ i-1k R
i ≤ R
min
Q1,
...,Q
k: R
(Q1)
≤ R
1,...
,R(Q
k) ≤
Rk 1 k
∑ i=1k D
Ui(Q
i)
=
min
R1≥
0,...
,Rk≥
0: 1 k
∑ i-1k R
i ≤ R
1 k
∑ i=1k
min
Qi:
R(Q
i) ≤
Ri D
Ui(Q
i)
=
min
R1≥
0,...
,Rk≥
0: 1 k
∑ i-1k R
i ≤ R
1 k
∑ i=1k δ
Ui,s
q(R
i)
•W
e ca
n no
w s
ee t
hat
to f
ind
the
OP
TA
fun
ctio
n, w
e m
ust
find
R1,
...,
Rk
that
ave
rage
to
at m
ost
R
and
that
min
imiz
e th
e av
erag
e of
the
indi
vidu
al s
cala
r qu
antiz
er O
PT
A f
unct
ions
of
U1,
...,U
k.
•T
he c
hoic
e of
R
1,...
,Rk
is c
alle
d a
bit
or r
ate
allo
catio
n,
i.e.
an a
lloca
-tio
n of
kR
bi
ts a
mon
g th
e in
divi
dual
sca
lar
quan
tizer
s.
If so
me
are
give
n ra
te la
rger
tha
n R
, t
hen
othe
rs m
ust
be g
iven
rat
e le
ss t
han
R.
Tr-
16
Rec
all o
ne o
f th
e or
igin
al id
eas
of t
rans
form
cod
ing:
so
me
coef
ficie
nts
will
be
smal
ler
on t
he a
vera
ge t
han
othe
rs,
and
thes
e w
ill b
e en
code
dat
low
er r
ates
, i.e
. al
loca
ted
few
er b
its,
than
tho
se t
hat
are
larg
er o
nth
e av
erag
e.
If th
ere
is t
o be
a n
et g
ain,
the
n it
will
dep
end
on a
succ
essf
ul b
it al
loca
tion.
•W
e w
ish
to f
ind
the
rate
allo
catio
n th
at m
inim
izes
:
δ U,p
r(R)
=
min
R1>
0,...
,Rk>
0: 1 k
∑ i-1k R
i ≤ R
1 k
∑ i=1k δ
Ui,s
q(R
i)
•F
or f
ixed
-rat
e sc
alar
qua
ntiz
atio
n, e
ach
δ U
i,sq(
Ri)
has
a s
tairc
ase
form
.F
indi
ng
δ U,p
r(R)
is a
n in
tege
r pr
ogra
mm
ing
prob
lem
. T
here
are
var
ious
itera
tive
algo
rithm
s.
•F
or v
aria
ble-
rate
sca
lar
quan
tizat
ion,
eac
h δ
Ui,s
q(R
i) is
con
tinuo
us.
We
can
find
δ U,p
r(R)
usi
ng v
aria
tiona
l, e.
g. d
eriv
ativ
e, m
etho
ds.
•F
or f
ixed
-rat
e sc
alar
qua
ntiz
atio
n, w
hen
R
is la
rge,
mos
t R
i's
are
larg
ean
d δ
Ui,s
q(R
i) c
an b
e ap
prox
imat
ed a
s co
ntin
uous
fun
ctio
n w
ith Z
ador
'sfo
rmul
a.
•F
or n
ow le
t us
ass
ume
δi(R
) is
con
tinuo
us.
Thi
s is
the
cas
e fo
rva
riabl
e-le
ngth
cod
ing
and
appr
oxim
atel
y th
e ca
se f
or f
ixed
-rat
e co
ding
whe
n R
is
larg
e.
In t
his
case
, w
e ca
n us
e va
riatio
nal m
etho
ds.
Tr-
17
A U
SEFU
L O
PTIM
IZA
TIO
N L
EM
MA
Lem
ma:
Let
g 1(R
), …
, g k
(R)
be
con
tinuo
us,
posi
tive-
valu
ed,
stric
tlyde
crea
sing
, co
ntin
uous
fun
ctio
ns d
efin
ed o
n [
0,∞
).
If R
1,…
,Rk
min
imiz
e ∑ i=1k g
i(Ri)
sub
ject
to
∑ i=1k R
i ≤ R
, R
i≥0,
i =
1,..
.,k.
then
g 'i(R
i) =
g'j(R
j) fo
r al
l i,j
s.t.
Ri >
0, R
j > 0
and
|g'i(R
i)| ≤
|g'j(R
j)| f
or a
ll i,j
s.t.
Ri =
0, R
j > 0
Tha
t is
, fo
r th
e op
timum
cho
ice
of
R1,
…,R
k,
the
slop
es o
f th
e g
ifu
nctio
ns a
re t
he s
ame.
Not
e:
The
der
ivat
ives
of
gi(R
) a
re n
egat
ive.
Tr-
18
Ri
Rj
R +
εi
R -ε j
|g'(R
)|ε
i
|g'(R
)|ε
j
Pro
of
of
Lem
ma:
B
y co
ntra
dict
ion.
Cas
e 1:
S
uppo
se
R1,
…,R
k m
inim
ize
Σk i=
1 g i
(Ri)
with
Σk i=
1 R
i ≤ R
, R
i ≥ 0
,
and
for
som
e i,
j, R
i > 0
, Rj >
0,
g 'i(R
i) ≠
g 'j(R
j).
With
out
loss
of
gene
ralit
y su
ppos
e
g 'i(R
i) <
g'j(R
j) <
0
The
n fo
r so
me
smal
l ε,
re
plac
e
Ri
by
Ri+
ε a
nd
Rj
by R
j-ε .
Thi
s ha
s no
effe
ct o
n Σ
k i=1
Ri .
Ho
we
ver,
g i(R
i) c
hang
es t
o ap
prox
imat
ely
gi(R
i+ε)
≅ g
i(Ri)+
ε g'i(R
i).
g j(R
j) c
hang
es t
o ap
prox
imat
ely
g
j(Rj-ε
) ≅
gj(R
j) -
ε-g'j(R
j).
Sin
ce g
'i(Ri)
< g
'j(Rj),
th
is r
educ
es
Σk i=
1 g i
(Ri),
bec
ause
g i(R
i+ε)
+ g
j(Rj-ε
) ≅
gi(R
i)+g j
(Rj)+
ε( g
'i(Ri)-
g 'j(R
j)) <
gi(R
i)+g j
(Rj).
(Des
pite
the
"≅"
, th
e ov
eral
l ine
qual
ity w
ill b
e st
rict
if ε
is
suf
ficie
ntly
sm
all.)
Thi
s co
ntra
dict
s th
e or
igin
al a
ssum
ptio
n th
at t
he
R1,
..,R
k m
inim
ize
Σk i=
1 g i
(Ri)
.T
here
fore
, it
is n
ot p
ossi
ble
that
R
i > 0
, R
j > 0
, g 'i
(Ri)
≠ g
'j(Rj).
Tr-
19
Ri
Rj
R +
εi
R -
εj
|g'(R
)|ε
i
|g'(R
)|ε
j
Cas
e 2:
S
uppo
se
R1,
…,R
k m
inim
ize
Σk i=
1 g i
(Ri)
with
Σk i=
1 R
i ≤ R
, R
i ≥ 0
,
and
for
som
e i,
j, R
i = 0
, R
j > 0
, an
d |g
'i(Ri)|
> |
g 'j(R
j)|.
Sin
ce t
he s
lope
s ar
ene
gativ
e,
g 'i
(Ri)|
< |g
'j(Rj)|
.
The
n fo
r so
me
smal
l ε,
re
plac
e
Ri
by
Ri+
ε a
nd
Rj
by R
j-ε .
Thi
s ha
s no
effe
ct o
n Σ
k i=1
Ri .
Ho
we
ver,
g i(R
i) c
hang
es t
o ap
prox
imat
ely
g i(R
i+ε)
≅ g
i(Ri)+
ε g'i(R
i).
g j(R
j) c
hang
es t
o ap
prox
imat
ely
g j(R
j-ε)
≅ g
j(Rj)
- ε-g
'j(Rj).
Sin
ce g
'i(Ri)
< g
'j(Rj),
th
is r
educ
es
Σk i=
1 g i
(Ri),
bec
ause
g i(R
i+ε)
+ g
j(Rj-ε
) ≅
gi(R
i)+g j
(Rj)+
ε( g
'i(Ri)-
g 'j(R
j)) <
gi(R
i)+g j
(Rj).
(Des
pite
the
"≅"
, th
e ov
eral
l ine
qual
ity w
ill b
e st
rict
if ε
is
suf
ficie
ntly
sm
all.)
Thi
s co
ntra
dict
s th
e or
igin
al a
ssum
ptio
n th
at t
he
R1,
..,R
k m
inim
ize
Σk i=
1 g i
(Ri)
.T
here
fore
, it
is n
ot p
ossi
ble
that
R
i = 0
, Rj >
0,
|g'i(R
i)| >
|g'j(R
j)|.
Tr-
20
TH
E O
PTA
FU
NC
TIO
N O
F K
-DIM
EN
SIO
NA
L P
RO
DU
CT
QU
AN
TIZ
ER
:
HIG
H-R
ESO
LU
TIO
N A
NA
LY
SIS
Rec
all:
δ U,p
r(R)
=
min
R1>
0,...
,Rk>
0: 1 k
∑ i-1k R
i ≤ R
1 k
∑ i=1k δ
Ui,s
q(R
i)
Ass
ume
R
is s
o la
rge
that
the
opt
imal
R
i's
are
so la
rge
that
we
may
use
the
appr
oxim
atio
ns
δ sq,U
i(Ri)
≅≅≅≅
1 12
σ2 iα i
2-2
Ri
,
whe
re
σ2 i =
var
ianc
e of
Ui
, an
d fo
r fix
ed-r
ate
codi
ng
α i =
β U
i =
1 σ2 i
∫ f1/
3U
i(u
) du
3
and
for
varia
ble-
rate
cod
ing
α i =
η U
i =
1 σ2 i 2
2h(U
i)
,
whe
re
h(U
i) =
-
∫ -∞∞ f Ui(u
) lo
g 2 f U
i(u)
du
We
now
hav
e
δ U,p
r(R)
=
min
R1>
0,...
,Rk>
0: 1 k
∑ i-1k R
i ≤ R
1 k
∑ i=1k 1 1
2 σ2 i
α i 2
-2R
i
Tr-
21
Ap
ply
ing
th
e O
pti
miz
atio
n L
emm
a
The
lem
ma
show
s th
at if
R
1,…
,Rk
> 0
m
inim
ize
1 k ∑ i=
1k
1 12
σ2 iα i
2-2
Ri
su
bjec
t to
1 k
∑ i=1k R
i = R
,
then
the
re is
a c
onst
ant
c
suc
h th
at f
or e
ach
i,
c =
d dR
i
1 12
σ2 iα i
2-2
Ri
=
1 12
σ2 iα i
2-2
Ri
(
-2 ln
2)
Sol
ving
the
abo
ve y
ield
s
Ri
= 1 2
log 2
σ2 iα i
- 1 2
log 2
(-6
cln
2)
.
Sin
ce t
he
Ri's
av
erag
e to
R
, e
quat
ing
the
avg.
of
the
RH
S t
erm
s t
o R
gi
ves
- 1 2
log 2
(-6c
ln 2
) =
R -
1 k ∑ j=1k 1 2 lo
g 2 σ
2 jα j
=
R -
1 2 log 2
∏ j=
1k σ
2 jα j
1/k
The
refo
re t
he o
ptim
al r
ate
allo
catio
n is
Ri
=
1 2 log 2
σ2 iα i
+ R
- 1 2 lo
g 2
∏ j=
1k σ
2 jα j
1/k =
R +
1 2 log
2 σ2 iα
i
∏ j=1k σ
2 jαj1/
k
Sub
stitu
ting
the
optim
al a
lloca
tion
show
n ab
ove
into
the
exp
ress
ion
for
dist
ortio
n yi
elds
δ U,p
r(R)
≅ 1 1
2
∏ j=
1k σ
2 jα j
1/k 2
-2R
Tr-
22
Not
ice
also
tha
t su
bstit
utin
g th
e op
timal
R
i in
to
δ sq,U
i(Ri)
≅≅≅≅ 1 1
2 σ
2 iα i
2-2
Ri
give
s
δ sq,U
i(Ri)
≅≅≅≅
1 12
∏ j=
1k σ
2 jα j
1/k 2
-2R
≅≅≅≅
δ U,p
r(R
)
Thu
s, w
ith t
he o
ptim
al b
it al
loca
tion,
eac
h U
i is
qua
ntiz
ed w
ith t
he s
ame
dist
or-
tion.
Is
thi
s w
hat
you
expe
cted
? I
s th
is w
hat
JPE
G d
oes?
If
not,
why
not
?
Sum
mar
y:
Giv
en a
k-d
imen
sion
al r
ando
m v
ecto
r U
a
nd la
rge
R,
the
optim
al r
ate
allo
catio
n is
Ri
= R
+ 1 2 l
og2
σ2 iαi
Γ U ,
( *)
the
min
imal
dis
tort
ion,
i.e
the
OP
TA
fun
ctio
n, is
δ U,p
r(R)
≅≅≅≅
1 12
Γ U 2
-2R
,
( **)
and
with
the
opt
imal
rat
e al
loca
tion
all c
oeffi
cien
ts a
re q
uant
ized
with
app
roxi
mat
ely
the
sam
e di
stor
tion,
w
here
Γ U =
∏ j=
1k σ
2 jα j
1/k
=
geom
etric
ave
rage
of
σ2 jα
j's
Tr-
23
No
tes
:
•T
he U
i's
with
larg
er
σ2 iαi
are
quan
tized
with
hig
her
rate
tha
n th
ose
with
smal
ler
σ2 iα
i. T
hose
with
σ2 i
α i >
ΓU
are
quan
tized
with
rat
e R
i > R
,an
d th
ose
with
σ2 i
α i <
ΓU
are
quan
tized
with
rat
e R
i < R
. B
ut a
ll ar
equ
antiz
ed w
ith a
ppro
xim
atel
y th
e sa
me
dist
ortio
n.
•C
oeffi
cien
t co
rrel
atio
ns d
o no
t en
ter
into
for
mul
a fo
r δ
U,p
r(R).
T
hus,
ther
e is
no
dire
ct n
eed
for
the
coef
ficie
nts
to b
e un
corr
elat
ed.
•S
uppo
se in
stea
d of
the
opt
imal
bit
allo
catio
n, w
e as
sign
equ
al n
umbe
rsof
bits
to
each
U
i. T
hat
is,
supp
ose
Ri =
R,
all
i. T
hen
D ≅
1 k
∑ i=1k δ
Ui,s
q(R
i) =
1 k
∑ i=1k δ
Ui,s
q(R
) ≅
1 k
∑ i=1k
1 12
σ2 iα i
2-2
R
=
1 12
1 k
∑ i=1k σ2 i
α i 2
-2R
Sin
ce a
n ar
ithm
etic
ave
rage
is la
rger
tha
n a
geom
etric
ave
rage
(un
less
the
term
s be
ing
aver
aged
are
iden
tical
),
1 k ∑ i=
1k σ
2 iαi
≥ Γ
U.
The
refo
re,
the
SN
R g
ain
of t
he o
ptim
al b
it al
loca
tion
over
a n
aive
all-
equa
l bit
allo
catio
n is
10 lo
g 10
1 k Σk i=
1sσ2 i
α iΓ U
Tr-
24
•If
σ2 iα i
is
the
sam
e fo
r al
l i,
the
n th
e op
timal
bit
allo
catio
n is
the
all-
equa
l bit
allo
catio
n, a
nd
1 k
Σk i=1σ
2 iα i
=
ΓU.
•W
hat
to d
o if
(*)
yie
lds
a sm
all o
r ne
gativ
e R
i fo
r so
me
i?
Thi
sha
ppen
s w
hen
som
e σ
2 iαi's
ar
e ve
ry s
mal
l.
(Typ
ical
ly,
we
need
eac
hR
i ≥~ 3
fo
r th
is h
igh-
reso
lutio
n an
alys
is t
o be
acc
urat
e.)
If
only
a s
mal
lfr
actio
n of
the
coe
ffici
ents
hav
e sm
all o
r ne
gativ
e R
i, t
hen
the
anal
ysis
will
be
fairl
y ac
cura
te.
If
not,
we
may
hav
e to
res
ort
to a
n in
tege
rpr
ogra
mm
ing
appr
oach
to
findi
ng a
goo
d bi
t al
loca
tion.
Tr-
25
TH
E O
PTA
FU
NC
TIO
N O
F T
RA
NSF
OR
M C
OD
ING
WIT
H T
RA
NSF
OR
M T
•A
pply
ing
wha
t w
e ju
st le
arne
d ab
out
prod
uct
code
s, w
e fin
d th
at f
ortr
ansf
orm
cod
ing
with
a s
peci
fic t
rans
form
T
,
-th
e op
timal
bit
allo
catio
n is
Ri
= R
+ 1 2 l
og2
σ2 iαi
Γ TX
( *
)
-th
e O
PT
A f
unct
ion
of t
rans
form
cod
ing
with
k-d
imen
sion
al t
rans
form
T
is
δ tr,
T(k
,R)
= δ
TX
,pr(R
) ≅≅≅≅
1 1
2 Γ T
X 2
-2R
( *
*)
-w
ith t
he o
ptim
al r
ate
allo
catio
n, a
ll co
effic
ient
s ar
e qu
antiz
ed w
ith
appr
oxim
atel
y th
e sa
me
dist
ortio
n,
-Γ T
X =
∏ j=
1k σ
2 jα j
1/k
=
geom
etric
ave
rage
of
σ2 jα
j's
•W
e se
e th
at a
goo
d tr
ansf
orm
is o
ne t
hat
mak
es
Γ TX =
()
∏k j=
1 σ2 j
α j1/
k
smal
l.
•C
ompa
re
( **)
to
the
hig
h-re
solu
tion
appr
oxim
atio
n to
the
OP
TA
of
dire
ct s
cala
r qu
antiz
atio
n ap
plie
d to
X
δ sq,
X(R
) ≅≅≅≅
1 12
σ2 X α
X 2
-2R
Tr-
26
The
SN
R g
ain
of t
rans
form
cod
ing
with
tra
nsfo
rm
T
over
dire
ct s
cala
rqu
antiz
atio
n of
X
is
10 lo
g 10
δ sq,
X(R
)δ T
X,p
r(R)
≅≅≅≅ 1
0 lo
g 10
σ2 Xα X
∏ j=
1k σ
2 jα j
1/k
= 1
0 lo
g 10
σ2 Xα X
Γ TX
•W
e w
on't
com
pare
to
k-di
men
sion
al V
Q u
ntil
we
optim
ize
T.
Tr-
27
•If
X
is
Gau
ssia
n, t
hen
each
U
i is
Gau
ssia
n,
and
α 1=
... =
αk
= α
X =
αG =
32.
6 fo
r F
RC
and
17.
08 fo
r V
RC
.
The
refo
re,
the
α's
ca
ncel
out
of
the
rate
allo
catio
n fo
rmul
a, w
hich
be
com
es
Ri
= R
+ 1 2 l
og2
σ2 i
Σ2 TX
wh
ere
Σ2 TX =
∏ j=
1k σ
2 j1/
k =
ge
omet
ric m
ean
of t
he c
oeffi
cien
t va
rianc
es
The
α'
s a
lso
canc
el o
ut o
f th
e th
e ga
in o
ver
scal
ar q
uant
izat
ion
form
ula,
whi
ch b
ecom
es
10 lo
g 10
σ2 X
Σ2 TX
=
10 lo
g 10
ratio
of v
aria
nce
to g
eom
etric
mea
n of
coe
f va
rian
ces
Sin
ce
σ2 X =
1 k E
||X||2
= 1 k
E||U
||2 =
1 k ∑ j=
1k σ
2 j ,
th
e ga
in is
the
rat
io o
f th
e
arith
met
ic a
vera
ge t
o th
e ge
omet
ric a
vera
ge o
f th
e co
effic
ient
var
ianc
es.
And
the
OP
TA
fun
ctio
n fo
r tr
ansf
orm
cod
ing
with
tra
nsfo
rm
T
bec
omes
δ X,tr
,T(R
) ≅
1 12
α G Σ
2 TX 2
-2R
Tr-
28
•C
oncl
usio
ns f
or t
he G
auss
ian
case
:
-A
goo
d tr
ansf
orm
is
one
that
min
imiz
es t
he g
eom
etric
ave
rage
of
the
coef
ficie
nt v
aria
nces
Σ2 TX =
∏ j=
1k σ
2 j1/
k ,
Sin
ce t
he a
rithm
etic
ave
rage
1 k
∑ j=1k σ
2 j
is u
naffe
cted
by
the
tran
sfor
m,
this
is d
one
by m
akin
g so
me
σ2 j 's
as
sm
all
as p
ossi
ble
(thi
s is
wha
t m
akes
the
aver
age
smal
l),
but
sinc
e th
e ar
ithm
etic
ave
rage
is u
ncha
nged
, s
ome
σ2 j's
mus
t al
so b
e la
rge.
In
som
e se
nse,
a g
ood
tran
sfor
m m
akes
the
σ2 j 's
as d
iffer
ent
as p
ossi
ble.
Tr-
29
•M
ore
conc
lusi
ons
for
the
Gau
ssia
n ca
se
-If
inst
ead
of t
he o
ptim
al b
it al
loca
tion,
one
use
d th
e eq
ual-b
it-al
loca
tion,
the
n th
e tr
ansf
orm
cod
e w
ould
per
form
no
bette
r th
andi
rect
sca
lar
quan
tizat
ion
appl
ied
to
X.
Tha
t is
, th
e di
stor
tion
wou
ldb
e
D ≅
1 k
∑ i=1k δ
Ui,s
q(R
i) =
1 k
∑ i=1k δ
Ui,s
q(R
) ≅
1 k
∑ i=1k
1 12
σ2 iα G
2-2
R
=
1 12
αG
1 k
∑ i=1k σ2 i
2-2
R
=
1 12
αG σ
2 X 2
-2R
≅ δ
X,s
q(R
) .
Thi
s co
nclu
sion
app
lies
to a
ll tr
ansf
orm
s.
We
conc
lude
tha
t th
e pr
oper
bit
allo
catio
n is
cru
cial
.
-If
the
tran
sfor
m w
ere
to p
rodu
ce c
oeffi
cien
ts a
ll of
whi
ch h
ad t
he s
ame
varia
nce
σ2 i,
th
en t
he o
ptim
al r
ate
allo
catio
n w
ould
be
the
equa
l-bit
allo
catio
n, a
nd t
he t
rans
form
cod
e w
ould
per
form
no
bette
r th
an d
irect
scal
ar q
uant
izat
ion
appl
ied
to
X.
•If
X
is n
ot G
auss
ian,
the
re's
no
sim
ple
rela
tion
amon
g th
e α
i's.
Nev
erth
eles
s, m
inim
izin
g Σ
2 TX
is u
sual
ly a
goo
d st
rate
gy t
o m
akin
g Γ
TX
sma
ll.
Tr-
30
TH
E O
PTIM
AL
TR
AN
SFO
RM
We
now
dis
cuss
how
to
choo
se t
he o
rtho
gona
l tr
ansf
orm
T
to
min
imiz
e
Σ2 =∆
∏ j=
1k σ
2 j1/
k
Thi
s w
ill b
e th
e op
timal
tra
nsfo
rm f
or t
he G
auss
ian
case
, an
d a
"goo
d"tr
ansf
orm
mor
e ge
nera
lly.
Mai
n F
act:
A k
-dim
ensi
onal
tra
nsfo
rm
T
min
imiz
es
Σ2 if
and
onl
y if
it is
aK
arhu
nen-
Loev
e T
rans
form
(K
LT)
for
the
k×k
co
varia
nce
mat
rix
KX o
fX
.
A K
LT i
s an
y tr
ansf
orm
who
se r
ows
are
an o
rtho
norm
al s
et o
fei
genv
ecto
rs o
f K
X.
For
a K
LT,
the
coef
ficie
nt v
aria
nces
σ
2 i a
re t
he e
igen
valu
es o
f K
X,
deno
ted
λ1,
…,λ
k a
nd
Σ2 =
∏ j=
1k λ
j1/
k
= |
KX|1/
k
whe
re
|KX|
den
otes
the
det
erm
inan
t of
K
X.
Tr-
31
MO
RE
FA
CT
S FR
OM
LIN
EA
R A
LG
EB
RA
The
pro
of o
f th
e M
ain
Fac
t is
bas
ed o
n m
ore
fact
s fr
om li
near
alg
ebra
.
Def
init
ion
: S
uppo
se
K
is a
k×k
m
atrix
, λ
is
a r
eal o
r co
mpl
ex n
umbe
r an
dv
is a
k-d
imen
sion
al v
ecto
r su
ch t
hat
Kv
= λ
v.
The
n λ
is
sai
d to
be
an e
igen
valu
e of
K
, v
is s
aid
to b
e an
eig
enve
ctor
of
K,
and
(λ,
v) a
re a
n ei
genp
air
for
K.
Fac
t 1:
E
very
k×k
mat
rix h
as
k e
igen
valu
es,
thou
gh t
hey
need
not
be
dist
inct
.
Fac
t 2:
T
he e
igen
valu
es o
f a
k×k
diag
onal
mat
rix
K
are
the
diag
onal
ele
me
nts
.
Pro
of:
One
may
dire
ctly
ver
ify t
hat
the
(K
(i,i),
v)
is a
n ei
genp
air,
whe
re
v =
(0
... 0
1 0
... 0
) w
ith a
1 in
the
ith p
lace
.
Fac
t 3:
R
eal
sym
met
ric m
atric
ies
have
rea
l (n
ot c
ompl
ex)
eige
nval
ues.
The
eig
enve
ctor
s as
soci
ated
with
dis
tinct
eig
enva
lues
are
ort
hogo
nal.
The
re i
s an
ort
hono
rmal
set
of
eige
nvec
tors
.
If th
e m
atrix
is a
lso
posi
tive
(res
pect
ivel
y, n
onne
gativ
e) d
efin
ite,
i.e.
if x
t K x
> 0
fo
r al
l x,
(
resp
ectiv
ely,
xt K
x ≥
0),
th
en t
heei
genv
alue
s ar
e po
sitiv
e (r
espe
ctiv
ely,
non
nega
tive)
.
Tr-
32
Fac
t 4:
T
he d
eter
min
ant
of a
squ
are
mat
rix is
the
pro
duct
of
itsei
genv
alue
s. i.
e.
if a
k×k
m
atrix
has
eig
enva
lues
λ 1
,…,λ
k,
then
|K|
= ∏ j=
1k λ
j
Fac
t 5:
The
det
erm
inan
t of
an
orth
ogon
al m
atrix
T
is
|T
| = ±
1.
Pro
of:
Sup
pose
λ
is a
n ei
genv
alue
of
T
and
v
is a
cor
resp
ondi
ngei
genv
ecto
r, i.
e.
Tv
= λ
v.
The
n
|| v||
= ||T
v|| =
||λv
|| =
|λ|
||v|
|
whi
ch im
plie
s |
λ| =
1.
Sin
ce a
ll th
e ei
genv
alue
s ha
ve m
agni
tude
one
,F
act 4
impl
ies
|T| =
±1.
Tr-
33
FA
CT
S A
BO
UT
CO
VA
RIA
NC
E M
AT
RIC
ES
Fac
t 6:
F
or a
ny k
×k c
ovar
ianc
e m
atrix
K
, ∏ i=
1k K
i,i ≥
|K
|, e
qual
ity if
f K
is
diag
'l.
Pro
of:
See
Ger
sho
and
Gra
y, p
p. 2
41-2
42
Fac
t 7:
If U
= T
X,
then
K
U =
T K
X T
t
Pro
of:
KU
= E
U U
t =
E T
X (
TX
)t = E
TX
Xt T
t = T
EX
Xt T
t = T
KX T
t .
Fac
t 8:
If T
is
ort
hogo
nal a
nd
U =
TX
, th
en
(a)
KX
and
KU
have
the
sam
e ei
genv
alue
s.
(b)
|KU|
= |K
X|
Pro
of:(
a) L
et (
λ,v)
be
an e
igen
pair
for
KX.
The
n K
UT
v =
TK
XT
t Tv
= T
λv =
λT
v.H
ence
, (
λ,T
v)
is a
n ei
genp
air
for
KU.
Thu
s ev
ery
eige
nval
ue o
f K
X
is a
lso
an e
igen
valu
e of
K
U.
The
sam
e ar
gum
ent
appl
ied
to K
U a
nd T
-1 s
how
s th
at e
very
eig
enva
l. of
K
U
is a
lso
an e
igen
valu
e of
K
X.
Thu
s K
X
and
KU
have
the
sam
e ei
genv
alue
s.
(b)
Sin
ce
KX
and
KU
have
the
sam
e ei
genv
alue
s, F
act
4 im
plie
s th
ey h
ave
the
sam
e de
term
inan
ts.
Fac
t 9:
For
any
cov
aria
nce
mat
rix t
here
is a
set
of
k o
rtho
norm
al e
igen
vect
ors.
Pro
of:
B
ecau
se c
ovar
ianc
e m
atric
es a
re r
eal a
nd s
ymm
etric
, an
d F
act
3.
Tr-
34
PR
OO
F O
F O
PTIM
AL
ITY
OF
KL
T
Lem
ma:
If
T
is
ort
hogo
nal a
nd m
akes
K
U
is d
iago
nal,
then
for
any
oth
eror
thog
onal
mat
rix
~ T,
~ Σ2 ≥
Σ2 ,
with
equ
ality
if a
nd o
nly
if K
~ U
is d
iago
nal.
Pro
of:
For
sim
plic
ity c
onsi
der
the
kth
pow
er o
f ~ Σ2 :
~ Σ2k =
∏ i=
1k ~ σ2 i
by
def
initi
on o
f ~ Σ2 ,
w
here
~ σ2 i
= E
~ U2 i
&
~ U=~ T
X.
≥ |
K ~ U|
F
act 6
fact
that
~ σ2 i's
are
dia
g el
emen
ts o
f K
~ U
= |
KX|
F
act 8
(b)
= |
KU|
Fac
t 8(b
)
=
∏ i=1k σ
2 i
Fac
t 6 &
fact
that
σ2 i '
s ar
e di
ag e
lem
ents
of
KU
=
Σ2k
for
T
By
Fac
t 6, e
qual
ity h
olds
iff
K ~ U
is d
iago
nal.
We
see
from
thi
s le
mm
a th
at o
ne c
an d
o no
bet
ter
than
to
mak
e th
e tr
ansf
orm
coef
ficie
nts
have
a d
iago
nal c
ovar
ianc
e m
atrix
.
Whe
n th
is is
don
e, t
he c
oeff.
var
ianc
es a
re t
he d
iago
nal e
lem
ents
, w
hich
by
Fac
t 2,
ar
e th
e ei
genv
alue
s of
K
U.
By
Fac
t 8a
, th
ese
are
also
the
eig
enva
lues
of
KX
as c
laim
ed in
the
Mai
n F
act.
Tr-
35
It re
mai
ns o
nly
to s
how
tha
t th
ere
is a
cho
ice
of T
tha
t m
akes
KU d
iago
nal.
Acc
ordi
ngly
, le
t T
be
the
typ
e of
mat
rix m
entio
ned
in t
he M
ain
Fac
t, i.e
. its
row
s ar
e an
ort
hono
rmal
set
of
eige
nvec
tors
t 1
,...,
t k
The
n, b
y F
act
5, a
nd t
he f
act
that
t i
is a
n ei
genv
ecto
r w
ith e
igen
valu
e λ
i
KU =
T K
X T
t =
--tt 1-
-
--tt 2-
-... --
tt k--
KX
| t 1 |
| t 2 | - - - | t k |
=
--tt 1-
-
--tt 2-
-... --
tt k--
| λ 1
t 1|
| λ 2t 2
| - - - | λ k
t k|
=
λ 1
0 ..
... 0
0 λ 2
0 ..
0 0
.....
0 λ k
.
So
as w
e ho
ped,
K
U
is d
iago
nal.
Thi
s fin
ishe
s th
e pr
oof
of t
he m
ain
fact
, na
mel
y, t
hat
Σ2
is m
inim
ized
by
a K
LTan
d th
at t
he r
esul
ting
min
imum
val
ue is
Σ2 =
∏ j=
1k λ
j1/
k
= |
KX|1/
k
.
Tr-
36
We
can
now
fin
d th
e O
PT
A f
unct
ion
for
tran
sfor
m c
odin
g a
Gau
ssia
n so
urce
.R
ecal
l tha
t fo
r a
Gau
ssia
n so
urce
δ X,tr
,T(R
) ≅
1 12
α G
Σ2 T
X 2
-2R
With
the
KLT
, σ
2 1,...
,σ2 k
= λ
1,...
,λk
and
Σ2 T
X =
∏ j=
1k σ
2 k1/
k
=
∏ j=
1k λ
j1/
k
= |
KX|1/
k
.
The
refo
re.
Th
e O
PT
A F
un
ctio
n f
or
k-d
imen
sio
nal
Tra
nsf
orm
Co
din
g a
pp
lied
to
aS
tati
on
ary,
Gau
ssia
n S
ou
rce:
For
larg
e R
,
δ tr(
k,R
) ≅≅≅≅
1 12
|KX|1/
k
αG
2-2
R
,
wh
ere α G
= 2
π 3
3/2
= 3
2.6
fo
r FR
C2
π e
= 1
7.0
8 fo
r VR
C
Tr-
37
Mo
reo
ver
•T
he b
est
tran
sfor
m is
the
KLT
, i.
e. r
ows
are
orth
onor
mal
eig
enve
ctor
s fo
rK
X.
•T
he r
esul
ting
coef
ficie
nts
U1,
…,U
k a
re u
ncor
rela
ted
(inde
ed,
inde
pend
ent)
.
•T
heir
varia
nces
σ2 1,
...,σ
2 k e
qual
the
eig
enva
lues
λ 1
,…,λ
k o
f K
X.
•T
he r
ate
allo
cate
d to
the
ith
coef
ficie
nt is
:
Ri
=
R +
1 2 log 2
λ i
|KX|1/
k
•T
he r
esul
ting
coef
ficie
nt d
isto
rtio
ns a
re a
ll th
e sa
me
and
equa
l to
δtr(k
,R).
Tr-
38
CO
MPA
RIS
ON
S FO
R T
HE
GA
USS
IAN
CA
SE
Opt
imal
k-D
imen
sion
al T
rans
form
cod
ing
δ tr(
k,R
) ≅≅≅≅
1 12
|KX|1/
k
αG
2-2
R
,
α G
= 2
π 3
3/2
= 3
2.6
fo
r FR
C2
π e
= 1
7.0
8 fo
r VR
C
Opt
imal
Sca
lar
Qua
ntiz
atio
n
δ sq(
R)
≅≅≅≅
1 12
σ2
α G 2
-2R
SN
R G
ain
over
sca
lar
quan
tizat
ion.
10 lo
g 10
δ sq(
R)
δ tr(
k,R
) ≅≅≅≅
10
log 1
0 σ2 X
|KX|1/
k
Opt
imal
k-d
imen
sion
al V
Q:
δ vq(
k,R
) ≅
m
* k σ2 X
αG
,k 2
-2R
,
α G
,k =
2π
(k+2 k)(k
+2)
/2 |K
|1/k
1 σ2 X
for
FR
C
2π e
|K|1
/k 1 σ2 X
fo
r VR
C
SN
R G
ain
of O
ptim
al k
-dim
VQ
ove
r O
pt k
-dim
'l T
rans
form
Cod
ing
10 lo
g 10
δ tr(
k,R
)δ v
q(k,
R)
≅≅≅≅ 1
0 lo
g 10
1/12 m* k
|K
X|1/
k
αG
σ2 X α
G,k
=
10
log 1
0 1/
12 m* k
×
33/2
(k+2 k)(k
+2)
/2 f
or F
RC
1 f
or V
RC
Tr-
39
SN
R G
ain
of O
ptim
al h
igh
dim
'l V
Q o
ver
high
dim
'l T
rans
form
Cod
ing
in d
B
10 lo
g 10
δ tr(
∞,R
)δ v
q(∞
,R)
≅≅≅≅ 1
0 lo
g 10
1/12
1/2π
e +
10
log 1
0 3
3/2 e
for
FR
C1 f
or V
RC
=
1.53
+ 2
.81
fo
r FR
C0 f
or V
RC
=
4.35
fo
r FR
C1
.53 f
or V
RC
The
se a
re t
he s
ame
gain
s as
opt
imal
VQ
ove
r op
timal
SQ
for
IID
Gau
ssia
nso
urc
e.
Why
?
Whe
n op
timiz
ed,
tran
sfor
m c
odin
g su
ffers
no
mem
ory
loss
,bu
t it
suffe
rs t
he s
ame
cubi
c, o
blon
gitis
and
poi
nt d
ensi
ty lo
sses
as
optim
ized
sca
lar/
prod
uct
quan
tizat
ion
for
the
IID c
ase.
Fix
ed-r
ate
codi
ng:
One
can
sho
w t
hat
tran
sfor
m c
odin
g co
uld
be d
esig
ned
(a)
To
have
opt
im'a
l poi
nt d
ensi
ty.
In t
his
case
it w
ould
hav
e hi
ghob
long
itis
loss
.
(b)
To
have
cub
ic c
ells
. I
n th
is c
ase
it su
ffers
larg
e po
int
dens
ity lo
ss
The
opt
imal
is
a co
mpr
omis
e th
at c
ause
s sa
me
loss
es a
s in
the
IID
cas
e.
Var
iabl
e-ra
te c
odin
g:
One
can
des
ign
the
tran
sfor
m c
ode
to h
ave
the
optim
al p
oint
den
sity
(w
hich
is u
nifo
rm)
and
cubi
c ce
lls.
So
it su
ffers
onl
yth
e cu
bic
loss
.
Tr-
40
TH
E E
FFE
CT
OF
DIM
EN
SIO
N k
ON
|K
(k)
|
Let
K(k
)
be
the
k×k
co
varia
nce
mat
rix o
f X
w
ith e
igen
valu
es
λ(k)
1,…
,λ(k
)k
.
Fac
t 10:
|K
(k)
|
= M
k-1
|K(k
-1)
|
= σ
2 X ∏ i=
1
k-1 M
i
whe
re
Mk
is t
he M
SE
of
the
best
line
ar p
redi
ctor
for
X
i fr
om
Xi-k
, …,X
i-1.
Pro
of:
G
iven
in t
he D
PC
M n
otes
.
Fac
t 11:
|K
(k+1
)
|1/(k
+1)
≤
|K
(k)
|1/
k
Thi
s im
plie
s th
at i
ncre
asin
g di
men
sion
will
not
dec
reas
e pe
rfor
man
ce.
Pro
of:
By
Fac
t 10,
|K(k
)
|1/k
=
geo
met
ric a
vera
ge o
f σ
2 X, M
1, …
, Mk
|K(k
+1)
|1/
(k+1
)
=
g
eom
etric
ave
rage
of
σ2 X, M
1, …
, Mk,
Mk+
1.
Obs
erva
tion:
M
k+1
≤ M
k,
beca
use
the
best
(k+
1)th
ord
er p
redi
ctor
mus
t be
at
leas
t as
goo
d as
the
bes
t kt
h or
der
pred
icto
r.
Obs
erva
tion:
th
e se
cond
geo
met
ric a
vera
ge is
like
the
firs
t ex
cept
it h
ason
e ad
ditio
nal t
erm
tha
t is
no
larg
er t
han
all t
he o
ther
s.
The
refo
re,
the
seco
nd g
eom
etric
ave
rage
is n
o la
rger
tha
n th
e fir
st.
Tr-
41
Not
e:
(Opt
iona
l Rea
ding
)
Fac
ts 1
0 an
d 11
are
rem
inis
cent
of
h(X
1,…
,Xk)
= h
(X1,
…,X
k-1)
+ h
(Xk|
X1,
…,X
k-1)
and
1 k h
(X1,
…,X
k) ≤
1 k-1
h(X
1,…
,Xk-
1)
Inde
ed,
they
can
be
prov
ed b
y us
ing
the
abov
e re
latio
ns f
or d
iffer
entia
len
trop
y.
Tr-
42
Fac
t 12:
lim k→∞ |K
(k)
|1/
k
= Q
(sta
ted
earli
er w
ithou
t pro
of)
wh
ere Q
=
exp
{ 1 2π
∫ -ππ ln S
(ω)
dω }
=
"one
-ste
p pr
edic
tion
erro
r"
=
MS
E o
f opt
imum
line
ar p
redi
ctor
for
Xi
base
d on
X
i-1, X
i-2, …
S(ω
) =
∑ n=-∞∞
RX(n
) e-jn
ω
=
pow
er s
pect
ral d
ensi
ty o
f ra
ndom
pro
cess
X
Pro
of:
lim k→
∞ |K
(k)
|1/
k
= l
im k→∞
(∏ i=
1k λ
(k)
i)1/
k
=
l
im k→∞
exp
{ ln
(∏ i=
1k λ
(k)
i)1/
k
}
= l
im k→∞
exp
{ 1 k
∑ i=1k ln
λ(k
)i
}
=
exp{
lim k→∞
1 k ∑ i=
1k ln
λ(k
)i
}
To
com
plet
e th
e pr
oof
we
need
to
show
lim k→∞
1 k ∑ i=
1k ln
λ(k
)i
=
1 2π
∫ -ππ ln S
(ω)
dω
Tr-
43
Fac
t 13
: S
zeg
o's
Eig
enva
lue
Dis
trib
uti
on
Th
eore
m
Let
{X
i} b
e w
ide-
sens
e st
atio
nary
ran
dom
pro
cess
with
pow
er
spec
tral
den
sity
S
(ω)
and
with
k-d
imen
sion
al c
ovar
ianc
e m
atrix
K
(k)
ha
ving
eig
enva
lues
λ(k
)1
,…,λ
(k)
.
The
n fo
r an
y pi
ecew
ise
cont
inuo
us
func
tion
g
lim k→∞ 1 k
∑ i=1k g
(λ(k
)i
) =
1 2π
∫ -ππ g(S
(ω))
dω
Ref
eren
ces:
U.
Gre
nand
er a
nd G
. S
zego
, T
oepl
itz F
orm
s an
d T
heir
App
licat
ions
(bo
ok)
R.M
. G
ray,
"T
oepl
itz a
nd C
ircul
ant
Mat
rices
: A
Rev
iew
" ,
(pap
er)
http
://w
ww
-ee.
stan
ford
.edu
/~gr
ay/t
oepl
itz.h
tml
Inte
rpre
tati
on
: T
his
theo
rem
det
erm
ines
the
asy
mpt
otic
"dis
trib
utio
n" o
f th
e ei
genv
alue
s of
the
cov
aria
nce
mat
rices
of
{X
i}.F
or e
xam
ple,
whe
n k
is
larg
e,
it de
term
ines
wha
t fr
actio
n of
the
eige
nval
ues
lie b
etw
een
a
and
b
for
any
a,b
.
See
nex
t pa
ge.
Tr-
44
Con
side
r th
e fu
nctio
n
g λ(s)
= 1
, s
≤ λ
0 ,
els
e
F(λ
) =
lim k→
∞ 1 k
∑ i=1k g
λ(λ(k
)i
) =
a
sym
pt fr
ac o
f e.v
.'s ≤
λ
=
dis
trib
utio
n fu
nctio
n fo
r th
e ei
genv
alue
s
The
G-S
The
orem
im
plie
s
F(λ
) =
1 2π
∫ -ππ g λ(
S(ω
)) d
ω
=
leng
th o
f {ω
: S(ω
) ≤ λ
}2π
λ
ω1
ω2
ω3
ω4
ω5
ω6
S(ω
)
ω
π-π
{ ω
: S
(ω)
≤ λ}
The
refo
re t
he "
dens
ity"
of e
igen
valu
es w
ith v
alue
s ne
ar
λ is
f(λ)
=
d dλ F
(λ)
≅≅≅≅
1 2π
1
|S'(ω
1)| +
1
|S'(ω
2)| +
…
whe
re
ω1,
ω
2, …
ar
e th
e fr
eque
ncie
s su
ch t
hat
S(ω
1) =
λ.
We
see
ther
ear
e m
any
eige
nval
ues
whe
re s
lope
is
flat
and
few
whe
re s
lope
is
stee
p.
Tr-
45
Com
plet
ion
of P
roof
of
Fac
t 12
:
Let
g(s)
= ln
(s).
The
n by
Fac
t 13
(th
e ei
genv
alue
dis
trib
utio
n th
eore
m)
lim k→∞ |K
(k)
|1/
k
=
lim k→
∞
∏ i=1k λ
(k)
i1/
k
=
ex
p{ lim k→
∞
1 k ∑ i=
1k ln
λ(k
)i
}
=
exp
{ 1 2π
∫ -ππ ln S
(ω) d
ω}
=
Q
whi
ch i
s th
e de
sire
d re
sult.
Tr-
46
Th
e O
PT
A F
un
ctio
n f
or
k-d
imen
sio
nal
Tra
nsf
orm
Co
din
g a
pp
lied
to
aS
tati
on
ary,
Gau
ssia
n S
ou
rce:
For
larg
e R
,
δ tr(
k,R
) ≅≅≅≅
1 12
|KX|1/
k
αG
2-2
R
→
δ tr(
R)
≅≅≅≅
1 12
Q α
G 2
-2R
as
k →
∞
wh
ere α G
= 2
π 3
3/2
= 3
2.6
fo
r FR
C2
π e
= 1
7.0
8 fo
r VR
C
Q =
e
xp{
1 2π
∫ -ππ ln S
(ω)
dω }
In c
ompa
rison
, fo
r op
timal
VQ
and
Gau
ssia
n
δ vq(
k,R
) ≅
m
* k |K
X|1/
k
αG 2
-2R
→
δ tr(
R)
≅≅≅≅
1 2πe
Q α
G 2
-2R
a
s k
→ ∞
Tr-
47
EX
AM
PLE
: FI
RST
-OR
DE
R A
R, G
AU
SSIA
N S
OU
RC
E:
•X
i =
ρ X
i-1 +
Zi
whe
re
Zi's
ar
e IID
Gau
ssia
n, z
ero
mea
n w
ith v
aria
nces
σ2 Z
, an
d Z
i is
inde
pend
ent
of p
ast
X's
. I
n th
is c
ase,
•B
est
linea
r pr
edic
tor
for
Xi
from
X
i-1,…
,Xi-k
is
~ Xi
= ρ
Xi-1
. Its
MS
E is
σ2 Z.
⇒
M1
= M
2 =
M3
= …
= σ
2 Z =
σ2 X (
1-ρ2
) =
Q =
exp
{ 1 2π
∫ -ππ ln S
(ω) d
ω }
•F
act 1
0 ⇒
|K
(k)
|
= M
k-1
|K(k
-1)
|
= σ
2 X ∏ i=
1
k-1 M
i =
σ2k X
(1-
ρ2)k-
1
⇒ |
K(k
)
|1/k
=
σ2 X (
1-ρ2 )
(k-1
)/k
(
stat
ed e
arlie
r w
/o p
roof
in Z
ador
not
es)
→
σ2 X (
1-ρ2 )
as
k →
∞.
•O
PT
A,
k-di
m.
Tra
nsf.
Cod
ing
of S
tat'r
y, G
auss
, A
R S
ourc
e w
ith c
orr.
coe
f. ρ:
For
larg
e R
,
δ tr(
k,R
) ≅≅≅≅
1 12
σ2 X (
1-ρ2 )
(k-1
)/k
α
G 2
-2R
,
αG =
2π
33
/2 =
32
.6 f
or F
RC
2π
e =
17
.08 fo
r VR
C
→
1 12
σ2 X (
1-ρ2 )
αG
2-2
R
as
k →
∞
Tr-
48
SNR
FO
R F
IXE
D-R
AT
E T
RA
NSF
OR
M C
OD
ING
Gau
ss A
R S
ourc
e --
cor
r. c
oeff.
ρ
= .9
, R
= 3
dim
ensi
on k
SNR, dB
101214161820222426
12345678
1216244896
192256
1024
Lob
SN
R*k
(3)
SN
R*(
3)
SNR
tr,k
(3)
Lcu
Lpt
(Ign
ore
the
loss
es.
In
this
plo
t, th
ey a
re d
efin
ed in
a d
iffer
ent
way
tha
n be
fore
.)
Tr-
49
Co
mp
aris
on
of
Tra
nsf
orm
Co
din
g a
nd
DP
CM
Con
side
r a
stat
iona
ry,
Gau
ssia
n so
urce
and
larg
e R
.F
or k
-dim
ensi
onal
tra
nsfo
rm c
odin
g:
δ tr,
k(R
) ≅
1 12
|K(k
)
|1/k
α
G 2
-2R
For
DP
CM
with
kth
-ord
er li
near
pre
dict
ion
δ dpc
m,k
(R)
≅
1 12
Mk
α G 2
-2R
whe
re
Mk
is M
SE
of
optim
al k
th-o
rder
line
ar p
redi
ctor
for
X
i, i.
e.
from
Xi-k
,…,X
i-1.
Fac
t 14:
M
k+1
≤ M
k
Pro
of:
Sin
ce k
th o
rder
pre
dict
ion
is a
spe
cial
cas
e of
(k+
1)th
ord
erpr
edic
tion,
the
bes
t (k
+1)
th o
rder
pre
dict
or m
ust
be a
t le
ast
as g
ood
asth
e be
st k
th o
rder
pre
dict
or;
i.e.
Mk+
1 ≤
Mk.
Tr-
50
Fac
t 15:
|K
(k)
|1/
k
≥ M
k
Pro
of:
Fro
m F
acts
10
and
14
|K(k
)
|1/k
=
(σ2 X
∏ i=1
k-1 M
i)1/k
=
(σ2 X
∏ i=1
k-1 M
k-1)
1/k
≥
Mk-
1 ≥
Mk
Fac
t 16:
lim k→∞ M
k =
lim k→
∞ |K
(k)
|1/
k
= e
xp{
1 2π
∫ -ππ ln S
(ω) d
ω }
Pro
of:
Firs
t w
e no
te t
hat
sinc
e th
e M
k's
are
non
nega
tive
and
noni
ncre
asin
g,th
ey c
onve
rge
to a
lim
it. S
econ
dly,
fro
m F
act
10
|K(k
)
|1/k
=
(σ2 X
∏ i=1
k-1 M
i)1/k
and
the
latte
r ex
pres
sion
con
verg
es t
o l
im k→
∞ M
k b
ecau
se w
hen
one
take
sth
e ge
omet
ric a
vera
ge o
f th
e fir
st
k t
erm
s of
a c
onve
rgen
t se
quen
ce (
the
Mk'
s),
that
geo
met
ric a
vera
ge c
onve
rges
to
the
sam
e va
lue
as t
hese
quen
ce.
Hen
ce,
lim k→∞ |K
(k)
|1/
k
=
lim k→∞ M
k
•B
y co
nsid
erin
g F
acts
11,
14,
15
and
16,
we
see
that
the
per
form
ance
s of
both
DP
CM
and
tra
nsfo
rm c
odin
g im
prov
e m
onot
onic
ally
with
k
to
the
sam
elim
it.
Mor
eove
r, t
he p
erfo
rman
ce o
f D
PC
M c
onve
rges
mor
e ra
pidl
y to
the
limit
than
doe
s th
e pe
rfor
man
ce o
f tr
ansf
orm
cod
ing.
Tr-
51
Exa
mp
le:
Firs
t-or
der
AR
, G
auss
ian
sour
ce:
Xi
= a
Xi-1
+ Z
i
whe
re
Zi's
ar
e IID
Gau
ssia
n, z
ero
mea
n w
ith v
aria
nces
σ2 Z
, an
d Z
i is
inde
pend
ent
of p
ast
X's
. I
n th
is c
ase,
M1
= M
2 =
M3
= …
= σ
2 Z =
σ2 X (
1-a2
) =
exp
{ 1 2π
∫ -ππ ln S
(ω) d
ω },
whi
ch m
eans
tha
t D
PC
M w
ith f
irst-
orde
r lin
ear
pred
ictio
n is
as
good
as
kth
-or
der
linea
r pr
edic
tion
for
any
k.
Der
ivat
ion:
F
or a
ny
k ≥
1, t
he b
est
linea
r pr
edic
tor
for
Xi
from
X
i-1,…
,Xi-k
is
~ Xi
= a
Xi-1
. T
his
can
be v
erifi
ed v
ia t
he o
rtho
gona
lity
prin
cipl
e, i.
e. b
y ch
ecki
ngth
at t
he e
rror
due
to
this
pre
dict
ion
is o
rtho
gona
l to
each
X
j, i-
k≤j≤
i-1:
E (
Xi-a
Xi-1
)Xj
= E
Zi X
j =
0
beca
use
Zi
is in
depe
nden
t of
X
j an
d ha
s ze
ro m
ean.
T
he r
esul
ting
MS
E is
:
Mk
= E
(Xi-~ X
i)2 =
E (
Xi-a
Xi-1
)2 = E
Z2 i
= σ
2 Z =
σ2 X (
1-a2
)
For
k-d
imen
sion
al t
rans
form
cod
ing
|K(k
)
|1/k
=
(σ2 X
(∏ i=
1
k-1 M
i))1/
k
= (
σ2 X (
∏ i=1
k-1 σ
2 Z))
1/k
= (σ
2 X (σ
2 X(1
-a2 )
)k-1
)1/
k
=
σ
2 X (
1-a2 )
(k-1
)/k
Thi
s is
larg
er t
han
M1
= σ
2 X (
1-a2
),
but
conv
erge
s to
it a
s k
→ ∞
.
Tr-
52
The
SN
R g
ain
in d
B o
f D
PC
M w
ith k
th-o
rder
line
ar p
redi
ctio
n ov
erk-
dim
ensi
onal
tra
nsfo
rm c
odin
g is
10 lo
g 10
δ tr,
k(R
)δ d
pcm
,k =
10
log 1
0 (1
-a2 )
(k-1
)/k
1-a2
= -
10 k lo
g 10
(1-a
2 )
For
a
= .
9,
this
gai
n is
plo
tted
belo
w.
05
1015
2025
3035
40012345678
k=tr
ansf
orm
cod
e di
men
., or
der
of D
PC
M li
near
pre
dict
or
dB
gain
of D
PC
M o
ver
tran
sfor
m c
odin
g
Tr-
53
Oth
er t
rans
form
s ar
e of
ten
used
:
DC
T,
FF
T a
nd w
avel
ets
The
y ar
e
fast
er,
sig
nal i
ndep
ende
nt a
nd h
ence
mor
e ro
bust
, p
erce
ptua
lly r
elev
ant,
appr
oxim
atio
ns t
o th
e K
LT
Rec
onsi
der
JPE
G
uses
DC
T in
stea
d of
KLT
, f
or a
ll of
the
abo
ve r
easo
ns
uses
US
Q w
ith V
L co
ding
, be
caus
e th
is is
the
bes
t ki
nd o
f sc
alar
quan
tizat
ion.
it
is a
lso
sim
ple
the
VLC
is n
ot s
impl
y H
uffm
an c
odin
g be
caus
e m
any
of t
he c
oeffi
cien
tsha
ve
H(U
i) <
1
and
in s
uch
case
s H
uffm
an c
odin
g yi
elds
giv
es p
oor
rate
.so
it u
ses
a m
ore
soph
istic
ated
sch
eme
in o
rder
to
get
Ri ≅
H(U
i)
JPE
G is
not
opt
imiz
ed f
or M
SE
, if
it w
ere
it w
ould
sho
ot f
or s
omet
hing
like
equa
l dis
tort
ion
of a
ll co
effic
ient
s.
Wha
t bl
ock
size
to
use?
If da
ta is
sta
tiona
ry G
auss
ian,
the
n pe
rfor
man
ce im
prov
es w
ith d
imen
sion
,to
a li
mit.
How
ever
, fo
r re
al d
ata
this
is n
ot a
lway
s th
e ca
se.
Ins
tead
the
dat
a is
mul
timod
al,
inho
mog
eneo
us,
blo
bby,
m
ixtu
res
of G
auss
ian,
non
stat
iona
ry
In t
his
case
, ch
oosi
ng t
oo la
rge
a bl
ockl
engt
h w
ill h
urt.
