Ai Wei Wei innovative / renowned / influential /leading figure.
Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof....
-
Upload
bridget-tucker -
Category
Documents
-
view
220 -
download
1
Transcript of Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof....
Deblocking Algorithms in Video and Image Compression Coding
Speaker: Wei-Yi Wei
Advisor: Prof. Jian-Jung Ding
Digital Image and Signal Processing Lab
GICE, National Taiwan University
Outline
Introduction Observations of Blocking Artifact Four Categories of Deblocking Algorithms Block-Edge Impairment Metric Comparison Simulation Conclusions and Future Work References
2009/04/11
Deblocking Algorithms in Video and Image Compression Coding 2
Introduction What is Blocking Artifacts ?
– Discontinuity over the block boundaries– Result from coarse quantization
2009/04/11 3
Original Image
Highly Compressed Image
Deblocking Algorithms in Video and Image Compression Coding
A Simple Coarse Quantization Example
02
46
8
0
2
4
6
8-100
0
100
200
300
400
500
2009/04/11 4
02
46
8
0
2
4
6
80
20
40
60
80
100
02
46
8
0
2
4
6
80
100
200
300
400
500
02
46
8
0
2
4
6
852.5
53
53.5
54
54.5
55
DCT
IDCT
Q/Q-1
Deblocking Algorithms in Video and Image Compression Coding
Observations of Blocking Artifacts
Two observations of blocking artifact in block-based transform coding– The blocking artifact are more noticeable in flat
areas than in complex areas– The motion compensation propagates the
blocking artifacts into the next frames
2009/04/11 5
Deblocking Algorithms in Video and Image Compression Coding
Four Categories of Deblocking Algorithms
In-loop Deblocking Filter Post-processing Deblocking Algorithms Pre-processing Algorithms Overlapping Block Methods
2009/04/11 6
Deblocking Algorithms in Video and Image Compression Coding
In-loop Deblocking Filter
Mainly used in video compression coding Add deblocking filtering to the coding loop Not compatible with the existing standards Case Study
– H.264/AVC– Optimal Post-Process/In-Loop Filtering
2009/04/11 7
Deblocking Algorithms in Video and Image Compression Coding
H.264/AVC In-loop Filter (1/2)
2009/04/11 8
IntraInter
Transform Quantization
MotionEstimation
FrameBuffer
CoefficientScanning
EntropyCoding
MotionCompensation
Intra FramePrediction
InverseQuantization
InverseTransform
In-LoopFilter
Video Source
Motion Vector
Bitstream
In-LoopFilter
Deblocking Algorithms in Video and Image Compression Coding
H.264/AVC In-loop Filter (2/2)
BS Pixels to be modified
4 p0, p1, p2, q0, q1, q2
3 p0, p1, q0, q1
2 p0, p1, q0, q1
1 p0, p1, q0, q1
0 No filtering is applied
2009/04/11 9
p3 p2 p1 p0
q0 q1 q2 q3
H.264/AVC encoder must determine the boundary strength (BS) before filtering.
Apply the filters with different strength based on the boundary strength
Deblocking Algorithms in Video and Image Compression Coding
Optimal Post-process/In-loop Filtering (1/3)
2009/04/11 10
Transform Quantization
Multiplex
Find OptimalFilter Parameters
for Kernel G
Video SourceDCT
CoefficientsEntropyCoding
CoderControl
InverseQuantization
InverseTransform
FrameBuffer
Filter withG
Buffer
MotionCompensation
MotionEstimation
Buffer Fullness
EnablePrediction
QP
PreviousDecodedFrame
Motion Vector
Residue
Data out
OPF
ILOF0
Deblocking Algorithms in Video and Image Compression Coding
Optimal Post-process/In-loop Filtering (2/3)
2009/04/11 11
arg min || || G
I I G
Input Frame with size W × H
Filter Kernel with size l × l
Reconstructed Frame with size W × H
I
G
I
I I GThe relationship between , and can be expressed asIGI
The optimal filter coefficient matrix must satisfy
( ) ( , ), 0 , 1m nl m n m n l g GRow-stacked form of G
Deblocking Algorithms in Video and Image Compression Coding
Optimal Post-process/In-loop Filtering (3/3)
2009/04/11 12
0,0
0,1
, ( ) (1)
T
T
TW H WH
a
a
a
A
Ag b
Ag b
Tl l A A A T
1l b A b
( ) ( , ), 0 1 and 0 1i jl i j i W j H b IRow-stacked form of I
Window Matrix
Now the equation can be expressed as I I G
Multiply both side of with , we can obtainAg b
Solve this equation, then we
can obtain the optimal filter coefficients
Deblocking Algorithms in Video and Image Compression Coding
G
l
l I(i,j)^
TA
Post-processing Deblocking Algorithms
The most popular deblocking methods Improve the quality of the compressed
image and video without the source data Case Study
– Reduction of Blocking Artifacts in DCT Domain– Deblocking Using Weighted Sums of Symmetrically
Aligned Pixels– Deblocking Using Offset and Shift Technique
2009/04/11 13
Deblocking Algorithms in Video and Image Compression Coding
Reduction of Blocking Artifacts in DCT Domain (1/2)
2009/04/11 14
,, , ,
1( , ) ( , )
h hk l
m n k l m nk h l h
B i j w B u vW
,
h h
k lk h l h
W w
, 1, , 2,..., 2k lw k l
,
3, for ( , ) (0,0)
1, k l
k lw
otherwise
1,m nb
,m nb
1, 1m nb
, 1m nb
1, 1,m nb
, ,, ,( , ) { ( , )}k l k l
m n m nB u v DCT b u v
Low Activity
High Activity
DCT-domain filtering
where
Deblocking Algorithms in Video and Image Compression Coding
Reduction of Blocking Artifacts in DCT Domain (2/2)
The neighboring DCT coefficients at the same frequency do not vary radically within a small range
Apply lowpass filter to each frequency to filter the high frequency components resulting from blocking effect
2009/04/11 15
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (1/4)
Deblocking Frame
2009/04/11 16
{ , }, 0 -1, 0 -1i jI p i R j C
0,0,r cB 0,2
,r cB
2,0,r cB 2,2
,r cB
,r cB ,r cB
0,0,r cB
1,0,r cB
0,1,r cB
1,1,r cB
,, 1 1
8 ( ) ,8 ( )2 2
{ }, 0 1, 0 1f f
m nr c f f
r m S i c n S jB p i S j S
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (2/4)
2009/04/11 17
' ' ' ' ' ' ' ',
', , , , , , , ,i j f f f f f f f f
i j i j S i j S i j i S j i S j S i S j S i S jp p p p p
, ( ) ( )i j i j
'
'
,( , ) ( )
ffS i j
S i j j
'
'
,( ) ( , )
ffi S j
i i S j
' '
' '
,( ) ( )
f ff fS i S j
S i S j
The filtered pixel
where the filter coefficients are
2,1p1,3p 1,4p
2,6p
6,3p 6,4p5,6p
5,1p
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (3/4)
2009/04/11 18
Linear Solution
Quadratic Solution
[0, 1]( ) , if x [0, 1]kL x x k
k
[0, 1] (2 )( ) , if x [ , 2 1]k
L
kx x k k
k k
2 ( )(1 2 )( )
( (Q
kx x x
k k k k
2fS
k [0,1] whereDeblocking Algorithms in Video and Image
Compression Coding
Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (4/4)
2009/04/11 19
Linear Solution Quadratic Solution
05
10
05
100
0.5
1
05
10
05
100
0.2
0.4
05
10
05
100
0.2
0.4
05
10
05
100
0.1
0.2
05
10
05
100
0.5
1
05
10
05
100
0.2
0.4
05
10
05
100
0.2
0.4
05
10
05
100
0.1
0.2
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Offset and Shift Technique (1/3)
Estimate the block of activity
2009/04/11 20
6
0H k
k
ACH C
6
0V k
k
ACH R
7
, , 10
| |k i k i kk
C q q
7
, , 10
| |k k j k jk
R q q
1,1g1,0g 1,2g
0,1g0,0g 0,2g
2,1g2,0g 2,2g
1,4g1,3g 1,5g
0,4g0,3g 0,5g
2,4g2,3g 2,5g
1,6g 1,7g
0,6g 0,7g
2,6g 2,7g
4,2g4,1g
3,1g3,0g 3,2g
5,1g5,0g 5,2g
4,5g4,4g
3,4g3,3g 3,5g
5,4g5,3g 5,5g
4,7g4,0g
3,6g 3,7g
5,6g 5,7g
6,1g6,0g 6,2g
7,1g7,0g 7,2g
6,4g6,3g 6,5g
7,4g7,3g 7,5g
6,6g 6,7g
7,6g 7,7g
4,6g4,3g
Block boundary
Horizontal Activity
Vertical Activity
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Offset and Shift Technique (2/3)
Filtering for Uniform Deblocking Blocks
2009/04/11 21
3 4offset p p
' | |( ) ( ), for i=1,2,3i i
i
offsetp p sign offset
' | |( ) ( ), for i=4,5,6i i
i
offsetp p sign offset
{8,4,2,2,4,8}i
p1 p2 p3 p4 p5 p6
offset
Block boundary
p1 p2 p3 p4 p5 p6
Block boundary
Offset/8Offset/4
Offset/2
Offset/2Offset/4
Offset/8
Deblocking Algorithms in Video and Image Compression Coding
Deblocking Using Offset and Shift Technique (3/3)
Filtering for Directional Deblocking Blocks
Filtering for Complex Deblocking Blocks
2009/04/11 22
'3 3
| |( ) ( )
4
offsetp p sign offset
'4 4
| |( ) ( )
4
offsetp p sign offset
' 1 9 3 7 2 8 4 65 0 5 1 2 3 42 2 2 2
g g g g g g g gg w g w w w w
5g4g 6g
2g1g 3g
8g7g 9g
Deblocking Algorithms in Video and Image Compression Coding
Pre-processing Algorithms (1/2) Achieve deblocking and enhance the coding
efficiency
2009/04/11 23
S
p3 p2p1
p0
q0q1
q2
q3
'S ''S
pS 'pS ''
pS
Encoding Decoding
Encoding Decoding
Deblocking Algorithms in Video and Image Compression Coding
Pre-processing Algorithms (2/2)
Our objective is to minimize the difference between I and V and obtain the coefficients
2009/04/11 24
3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q
V I
2( ) 0T
V I VV I
Q Q Q
' ' ' '0 1 2 3( , , , )Q q q q q
The difference between I and V
Original (I)
Pre-processed (R)' ' ' ' ' ' ' '3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q
Filtered (V)' ' '' '' '' '' ' '3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q
Taking derivative
Solve this equation, then we
can obtain the optimal filter coefficients
Deblocking Algorithms in Video and Image Compression Coding
Overlapped Block Methods
Source of Blocking Artifact – Block-based transform does not take the correlation
across the block boundary– Motion Estimation in video coding will propagate the
blocking artifact to the next frame How to solve these two problems
– Lapped Orthogonal Transform– Overlapped Block Motion Compensation
2009/04/11 25
Deblocking Algorithms in Video and Image Compression Coding
Lapped Orthogonal Transform (1/3) The blocks overlap slightly and the redundant
information is transmitted for the samples in the block boundaries
Reduce the blocking artifact
2009/04/11 26
0 0y Tx
1
0
0
2
0
0
P
P
T
P
P
N N
LCurrent Block Neighboring Block
Transform CoefficientsAssume the 1D discrete time signal is a sequence of MN samples
Segments the signal into M blocks and each block with size N
Deblocking Algorithms in Video and Image Compression Coding
Lapped Orthogonal Transform (2/3)
2009/04/11 27
0 0 0T
xxR P R P
2
11/
2
1
1 N
ii
TC NN
ii
NG
Energy Compaction Measurement
Where is the i-th diagonal entry of the matrix 2i 0R
2
1
1
2
1
1
1
1
L
L
xx
L
L
R
where
Deblocking Algorithms in Video and Image Compression Coding
Lapped Orthogonal Transform (3/3)
2009/04/11 28
cos i
cos i
sin i
sin i
i
0
1
2
3
4
5
6
7
0
2
4
6
1
3
5
7
0
1
2
3
4
5
6
7
0
2
4
6
1
3
5
7
Y(0)Y(2)Y(4)Y(6)
Y(1)Y(3)Y(5)Y(7)
1
2
3
DCT
DCT
-
-
-
-
-
-
-
-
DCTE
O
Z
DCTE
O
Z
DCTE
O
Z
eJH
eH
Z
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
2
2
-
-
-
-
-
-
-
… …
Block 1
Block 2
Block M
LOT ofBlock 1
LOT ofBlock 2
LOT ofBlock M
Deblocking Algorithms in Video and Image Compression Coding
Overlapped Block Motion Compensation (1/2)
2009/04/11 29
( , ) ( , ) ( , )i iv vE x y P x y S x y
( , ) ( , ) ( , )i iv W vE x y E x y W x y
21 1
1( , )
i
N N
v Wx y
MAE E x yN
EvaluationTarget Block
Overlapped Block
Reference blocks
Motion Vector OutputCurrent Frame
Reference Frame
Window Function
Deblocking Algorithms in Video and Image Compression Coding
Overlapped Block Motion Compensation (2/2)
2009/04/11 30
( , ) ( , ) ( , )vW vP x y P x y W x y
Overlapped Block
Motion Vectors
Prediction Frame
Reference Frame
Window Function
Deblocking Algorithms in Video and Image Compression Coding
Block-Edge Impairment Metric (1/2)
The disadvantages of PSNR– It does not always reveal the real quality
perceived by the HVS– It can not real the blockiness
Several block-edge impairment metrics have been proposed– GBIM– It can reveal the real blockiness and evaluate
the performance of deblocking algorithms
2009/04/11 31
Deblocking Algorithms in Video and Image Compression Coding
Block-Edge Impairment Metric (2/2)
2009/04/11 32
Reconstructed Image Reconstructed Image after deblocking
=3.11GBIMM =1.05
PSNR=21.75 dB
GBIMM
PSNR=21.86 dBDeblocking Algorithms in Video and Image Compression Coding
Comparison (1/2)
In-loop Filter– Achieve better improvement because it can
refer to the new input video frame– Not compatible with the existing standards
Post-processing Deblocking Algorithms– Good potential to be integrated into existing
standards– Blind to the target image and video
2009/04/11 33
Deblocking Algorithms in Video and Image Compression Coding
Comparison (2/2)
Pre-processing Deblocking Algorithms– Modify the source image and video in advance
and reduces the bit rate– Achieve the quality close to the direct
compressed image and video Overlapped Block Methods
– Prevents the blocking artifacts from happening in advanced
2009/04/11 34
Deblocking Algorithms in Video and Image Compression Coding
Simulation (1/3)
2009/04/11 35
Original Image Highly Compressed Image
Offset and Shift
WSSAP DCT DomainDeblocking Algorithms in Video and Image Compression Coding
Simulation (2/3) LOT and DCT
2009/04/11 36
0 2 4 6 8 10 12 14 16 18 2024
26
28
30
32
34
36
38
40
42
Quantization step
PS
NR
(dB
)
DCT
LOT
LOT with Q=10
DCT with Q=10
PSNR under different quantization strength
Deblocking Algorithms in Video and Image Compression Coding
Simulation (3/3) Overlapped Block Motion Compensation
2009/04/11 37
0 10 20 30 40 50 60 70 80 90 10029.5
30
30.5
31
31.5
32
32.5
Frame Number
PS
NR
(dB
)
weather-slow motion video :PSNR
Video Compression
Video Compression with OBMC
0 10 20 30 40 50 60 70 80 90 10027
28
29
30
31
32
33
Frame Number
PS
NR
(dB
)
stefan-fast motion video :PSNR
Video Compression
Video Compression with OBMC
0 10 20 30 40 50 60 70 80 90 1000.4
0.6
0.8
1
1.2
1.4
1.6x 10
7
Frame Number
Energ
y
weather-fast motion video :Energy
Video Compression
Video Compression with OBMC
0 10 20 30 40 50 60 70 80 90 1000
1
2
3
4
5
6
7
8
9
10x 10
7
Frame Number
Energ
y
stefan-fast motion video :Energy
Video Compression
Video Compression with OBMC
Deblocking Algorithms in Video and Image Compression Coding
Conclusions and Future Work
Many conventional methods do not exploit the characteristics of the blocking artifacts
The new algorithms apply different filter with different strength to different areas
Adopting the blockiness metric to measure the blockiness will be the future trend
2009/04/11 38
Deblocking Algorithms in Video and Image Compression Coding
References (1/2)In-loop Filter
[1] Lain E.G. Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia”, John Wiley & Sons, Jan. 2004.
[2] Lee, Jae-Beom, Kalva and Hari, ”The VC-1 and H.264 Video Compression Standards for Broadband Video Services”, Springer, 2008.
[3] “Text of ISO/IEC FDIS 14496-10/Draft ITU-T H.264: Information Technology – Coding of Audio-Visual Objects: Advanced Video Coding”, International Organization for Standardization, 2003.
[4] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007.
[5] S. Romero, and L.F. Romero, “An Optimized Preconditioned Conjugate Gradient Algorithm”, Technicah Report No: UMA-DAC-02/11, University of Malaga, Sept. 2002.
Post-Filtering [6] Tao Chen, Hong Ren Wu and Bin Qiu, “Adaptive Postfiltering of Transform Coefficients for the
Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 11, no. 5, Dec. 2001.
[7] Shizhong Liu and Alan C. Bovik, “Efficient DCT-Domain Blind Measurement and Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 12, no. 12, May 2002.
[8] Ci Wang, Wen-Jun Zhang and Xiang-Zhong Fang, “Adaptive Reduction of Blocking Artifacts in DCT Domain for Highly Compressed Images”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 50, no. 2, May 2004.
[9] A. Z. Averbuch, A. Schclar and D. L. Donoho, “Deblocking of block-transform compressed images using weighted sums of symmetrically aligned pixels,” IEEE Trans. on Circuits Syst. Video Technology, vol.14, pp.200-212, Feb. 2005.
2009/04/11 39
Deblocking Algorithms in Video and Image Compression Coding
References (2/2) [10] Jongho Kim, Minseok Choi, and Jechang Jeong, “Reduction of Blocking Artifact for HDTV using
Offset-and-Shift Techniques”, IEEE Trans. on Consumer Electronics, vol. 53, no.4, November 2007. [11] Zixiang Xiong, Michael T. Orchard, and Ya-Qin Zhang, “A Deblocking Algorithm for JPEG-
Compressed Images Using Overcomplete Wavelet Representations”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 7, no. 2, April 1997.
[12] Jongho Kim and Jechang Jeong, “Adaptive Deblocking Technique for Mobile Video”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007.
Pre-Filtering [13] Sheng-Ho Wang, Sung-Wen Wang, Yi-Shin Tung, Ja-Ling Wu, Jau-Hsiung Huang, "Pre-Process for
maximizing the effect of in-loop deblocking filtering in H.264/AVC encoding," 5th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services (EC-SIP-M 2005), Smolenice, Slovak, 29 June - 2 July, 2005.
Overlapped Block Methods [14] H. S. Malvar and D. H. Staelin, “The LOT: transform coding without blocking effects”, IEEE Trans. on
Acoustic., Speech, Signal Processing, vol. 37, no. 4, pp. 553-559, April 1989. [15] M. T. Orchard and G. J. Sullivan, “Overlapped block motion compensation: an estimation-theoretic
approach”, IEEE Trans. on Image Processing, vol. 3, no. 4, pp. 693-699, Sep. 1994.
Block-Edge Impairment Metric [16] H. R. Wu and M. Yuen, “A Generalized Block-Edge Impairment Metric for Video Coding”, IEEE Signal
Processing Letters, vol. 4, no. 11, Nov. 1997. [17] Athanasios Leontaris, Pamela C. Cosman and Amy R. Reibman, “Quality Evaluation of Motion-Compensated
Edge Artifacts in Compressed Video”, IEEE Trans. on Image Processing, vol. 16, no. 4, April 2007.
2009/04/11 40
Deblocking Algorithms in Video and Image Compression Coding