§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2...
-
Upload
arlene-bishop -
Category
Documents
-
view
218 -
download
1
Transcript of §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2...
§3 Discrete memoryless sources and their rate-distortion function
§3.1 Source coding
§3.2 Distortionless source coding theorem
§3.3 The rate-distortion function
§3.4 Distortion source coding theorem
§3.1 Source coding
§3.2 Distortionless source coding theorem
§3.3 The rate-distortion function
§3.4 Distortion source coding theorem
1. Source coder
§3.1 Source coding
Source coder}1...,1,0{ rA },...,,{ 110 rC
|| iin
}1,...,1,0{ sS
)...,,( 110 rpppp
1 2( ), , ( 1, ..., )ii i i in ij is s s s S j n
Source alphabet
Channel input alphabet
Code
mnm
1 2( ), , ( 1, ..., )ii i i in ij is s s c S j n
Extendedsource coder
mA
}1,...,1,0{ sS
1 2( )mU U U U
iU A
Example 3.1
§3.1 Source coding
2. Examples
1) ASCII source coder
ASCII coder
{0,1}
{English symbol , command} {binary code, 7 bits}
2) Morse source coder
Source coder(1)
Source coder(2)
{0,1}{. , —}
{A,B,…,Z} Binary code
2. Examples
§3.1 Source coding
3) Chinese telegraph coder
“中”
“0022”
“01101 01101 11001 11001”
2. Examples
§3.1 Source coding
Constant-length codes
Variable-length codes
Distortionless codes
Distortion codes
2. Classification of the source coding
Uniquely decodable (UD) codes
Non-UD codes
§3.1 Source coding
in n
in
( ; ) ( )I A C H A
( ; ) ( )I A C H A
The code C is called uniquely decodable (UD) if each string in each Ck arises in only one way as a concatenation of codewords. This means that if say
and each of the τ’s and σ’s is a codeword, then
Thus every string in Ck can be uniquely decoded into a concatenation of codewords.
1 2 1 2* * * * * *k k
1 1 2 2, , , .k k
2. Classification of the source coding
Example 3.2
§3.1 Source coding
Source symbol
si
Symbol probability
P(si) Code
1
Code
2
Code
3
Code
4
Code
5
S1 1/2 0 0 1 1 00 S2 1/4 11 10 10 01 01 S3 1/8 00 00 100 001 10 S4 1/8 11 01 1000 0001 11
3. Parameters about source coding
1) Average length of coding
1
0
1
0
||)(r
iii
r
iii nppn
mr
i
mi
mim npn
1
For extended source coding:
(code/sig)
code/m-sigs
Length of codeword
§3.1 Source coding
mnn
m (code/sig)
2) Information rate of coding
( ) /R H p n
( )
/mm
H pR
n m
(bit/code)
(bit/code)
3. Parameters about source coding
§3.1 Source coding
3) Coding efficiency
smn
pH
m log
)(
Actual rate
Maximum rate
3. Parameters about source coding
§3.1 Source coding
log
R
s
For extended source coding:
s
npH
log
/)(
§3.2 Distortionless source coding theorem
§3 Discrete memoryless sources and their rate-distortion function
§3.1 Source coding
§3.2 Distortionless source coding theorem
§3.3 The rate-distortion function
§3.4 Distortion source coding theorem
Example 3.3
The binary DMS has the probability space:
4
1
4
3)(
21 aa
aP
A
i
§3.2 Distortionless source coding theorem
1) “0” a1, “1” a2
2) a1a1: 0 a1a2: 10 a2a1: 110 a2a2: 111
)/(811.03
4log
4
34log
4
1)( signbitAH
Average length of coding: )/(11 sigcoden
Code efficiency: 811.01
§3.2 Distortionless source coding theorem
“0” a1, “1” a2
Rate:1
1
( )0.811 ( / )
H pR bit code
n
Example 3.3
4
1
4
3)(
21 aa
aP
A
i
Extended source coding
i )( iP code Length of codeword
a1a1 16
90 1
a1a2 16
310 2
a2a1 16
3110 3
a2a2 16
1111 3
Average length of coding :3
16
13
16
32
16
31
16
92 n
Code efficiency:
)/(961.0844.0
811.02 codebitR
Rate:
961.02
)2/(688.1 sigscode
)/(844.02
2 sigcoden
§3.2 Distortionless source coding theorem
Example 3.3
m times extended source coding
m = 3: 985.03
R3 = 0.985 (bit/code)
m = 4: 991.04
R4 = 0.991 (bit/code)
m 1m
§3.2 Distortionless source coding theorem
Example 3.3
§3.2 Distortionless source coding theorem
Distortionless source coding theorem
Theorem 3.1 If the code C is UD, its average length mustexceed the s-ary entropy of the source , that is,
1
0
log)(r
iisis pppHn
(Theorem 11.3 in textbook)
§3.2 Distortionless source coding theorem
Distortionless source coding theorem
Theorem 3.2
1)()()( pHpnpH sss
(Theorem 11.4 in textbook)
§3.2 Distortionless source coding theorem
Theorem 3.3 )()(
1lim pHpn
m sm
sm
(Theorem 11.5 in textbook)
Distortionless source coding theorem
The source can indeed be represented faithfully using s-ary symbols per source symbol.
p
( )sH p
§3.2 Distortionless source coding theorem
Distortionless source coding theorem
corollary
The efficient UD codes are achievable if rate R ≤ C.
(C is the capacity of s-ary lossless channel )
Review
• KeyWords:
Source coder
Variable-length codes
distortionless codes
Uniquely decodable codes
Average length of coding
Information rate of coding
Coding efficiency
Shannon’s TH1
Homework
1. p344: 11.12
2. p345:11.20
§3 Discrete memoryless sources and their rate-distortion function
§3.1 Source coding
§3.2 Distortionless source coding theorem
§3.3 The rate-distortion function
§3.4 Distortion source coding theorem
§3.3 The rate-distortion function
1. IntroductionReview
Distortionless source coding theorem (corollary) The efficient UD codes are achievable if rate R ≤ C.
(C is the capacity of s-ary lossless channel )
Conversely, any sequence of (2nR, n) codes with must have R ≤ C.
0EP
The channel coding theorem (Statement 2 ):
All rates below capacity C are achievable. Specifically,
for every rate R ≤ C, there exists a sequence of (2nR,n) codes
with maximum probability of error .0EP
§3.3 The rate-distortion function
1. Introduction
Review
For distortionless coding: R≤C - (PE→0, R→C - )
But actually……
Given a source distribution and a distortion measure, what is the minimum expected distortion achievable at aparticular rate?what is the minimum rate description required to achieve a particular distortion?
§3.3 The rate-distortion function
2. Distortion measure
coding channelui
vj
AU={u1,u2,…,ur} AV={v1,v2,…,vs}
k
iii vudvud
1
),(),(
kV
kUkk AAvvvuuuvuif ),...,,;,...,,(),( 2121
( , )i jd u v
Source symbol Destination symbol
2. Distortion measure
Average distortion measure:
, ,
( ) [ ( , )] ( ) ( , ) ( ) ( | ) ( , )u v u v
D k E d U V p uv d u v p u p v u d u v
Let the input and output of the channel be U=(U1,U2,…,Uk)and V=(V1,V2,…,Vk) respectively
kV
kUkk AAvvvuuuvu ),...,,;,...,,(),( 2121
where,
§3.3 The rate-distortion function
,
,
( ) [ ( , )] ( ) ( , )
( ) ( | ) ( , )
i j i j i jU V
i j i i jU V
D E d E d u v p u v d u v
p u p v u d u v
Example 3.3.1
AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix
01
10D
2. Distortion measure
§3.3 The rate-distortion function
Example 3.3.2
AU = {-1,0,+1}, AV = {-1/2, +1/2};source statistics(1/3,1/3,1/3)and distortion matrix
12
11
21
D
2. Distortion measure
§3.3 The rate-distortion function
2. Distortion measure
Fidelity criterion:
§3.3 The rate-distortion function
( )D or D k k ,
Test channel:
Let the source statistics p(u) and distortion measure d(u,v) are fixed.
( | ) :j iB P v u D
( | ) : ( )or B P V U D k k
3. Rate-distortion function
1) Definition
The function is a function of the source statistics(p(u)) ,the distortion matrix D, and the real number .
)(kR
§3.3 The rate-distortion function
The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as
1 1, ( ,
( )
) (( , ..., ), ( ,
min{ ( ; )
..., ))
: ( ) }k
k kwhere U V
R I U V D k k
U U V V
The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as
3. Rate-distortion function
UAu
vvudup ),(min)(min
21 )()( 21 kk RR ③If , then
§3.3 The rate-distortion function
②The minimum possible value of is ,wheremink( )D k
R(δ) and C
①The function I(U;V) actually achieves its minimum value on the region of ;D
3. Rate-distortion function
2) Properties
Theorem 3.4 is a convex function of .)(kR min (Theorem 3.1 in textbook)
§3.3 The rate-distortion function
R(0)=H(U)
maxmin
( )kR
max
( )kR
maxmin},:);(min{)( DVUIR
max( ) 0, iff.R
3. Rate-distortion function
2) Properties
Theorem 3.4 is a convex function of .)(kR min
Theorem 3.5 For a DMS, for all k and .min )()( 1 kRRk
(Theorem 3.1 in textbook)
(Theorem 3.2 in textbook)
§3.3 The rate-distortion function
3. Rate-distortion function
Example 3.3.1 (continued)
AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix
01
10D
§3.3 The rate-distortion function
min
max
( ) (0) ( ) ( ),
( ) 0
R R H U H p
R
D
2) Properties
with different
(bit/sig)
0.0
1.0
0.8
0.6
0.4
0.2
0.50.40.30.20.1
0.5p
0.2p
0.1p
( )R
( )R p
0.3p
§3.3 The rate-distortion function
AU = AV = {0,1,…,r-1},
P{U=u}=1/r
Distortions are given by:
vuif
vuifvud
,1
,0),(
0.6 1.0
3.0
2.0
1.0
with different r
(bit/sig)
0.0 0.80.40.2
( )R
( )R
8r4r2r
2) Properties
Example 3.3.3
3. Rate-distortion function
§3 Discrete memoryless sources and their rate-distortion function
§3.1 Source coding
§3.2 Distortionless source coding theorem
§3.3 The rate-distortion function
§3.4 Distortion source coding theorem
1. Distortion source coding theorem
(modified on Theorem 3.4 in textbook)
§3.4 Distortion source coding theorem
Theorem 3.6 (Shannon’s source coding theorem with a fidelity criterion) If , there exists a source code C of length k with M codewords, where:
)(RR
Db
Ma kR
)(
2)(
If ,no such codes exist.)(RR A source symbol can be compressed into R(δ) bits
if a distortion δ is allowable.
2. Relation of shannon’s theorems
§3.4 Distortion source coding theorem
Source Distortion
source coder
Distortionless source coder
Sink Distortion
source decoder
Distortionless source decoder
channel
Channel coder
Channel decoder
A general communication system
Review
• KeyWords:
Distortion measure
Average distortion measure
Fidelity criterion
Test channel
Rate-distortion function
Shannon’s TH3
thinking
Source X has the alphabet set {a1,a2,…,a2n},P{X = ai}=1/2n,i = 1,2,…,2n. The distortion measure is Hamming distortionmeasure ,that is
ji
jid ij ,0
,1
Design a source coding method with δ=1/2.
§3.4 Distortion source coding theorem