Master Thesis Presentation:

Real-time Rate-distortion Oriented Joint

Video Denoising and Compression

Junsheng Fu

Signal Processing Department

Tampere University of Technology

03/12/2011

1/30

Outline

1. A real-time filter

2. Joint video pre-filtering and compression

3. Joint video in-loop filtering and compression

03/12/2011

2/30

Part 1:

A real-time filter

03/12/2011

3/30

VBM3D (Video Block-Matching 3D Filtering)

03/12/2011

K. Dabov, A. Foi, and K. Egiazarian, “Video denoising by sparse 3D transform-domain collaborative filtering,” in Proc. 15th

European Signal Processing Conference, EUSIPCO, Poznan, Poland, September 2007

4/30

Filter performance

03/12/2011

Res: 640x480 Sigma:5

Standard VBM3D

Vassar

Noisy (dB) 34.15

Denoised (dB) 40.81

Speed (fps) 1.12

ballroom

Noisy (dB) 34.15

Denoised (dB) 41.55

Speed (fps) 1.14

Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.

Computer Platform

Processor: Intel Core 2 Duo 3GHz

Memory: 3.2Gb of RAM

5/30

Optimization Task

1. Accelerate filter speed: above 25 fps for video (640 x 480)

In a computer

• Processor: Intel Core 2 Duo 3GHz

• Memory: 3.2Gb of RAM

03/12/2011

Approaches

1. Simplify VBM3D filter by only using most influential parts for

noise attenuation.

2. Propose a real-time integer implementation of simplified

VBM3D.

6/30

03/12/2011

Filters

Thresholding filtering Wiener filtering

Temporal searching

frames

Max grouping

blocks Spatial search

Spatial transform

Temporal search

Thres- holding

Temporal transform

Spatial search

Spatial transform

Temporal search

Wiener filtering

Temporal transform

Standard VBM3D + + + + + + + + + + 9 8

*Simplified VBM3D - - + + + - - - - - 5 4

* Simplified VBM3D means turn off some features of VBM3D filter.

Approach 1:

What to remove to Simplify VBM3D?

7/30

Filter performance

03/12/2011

Res: 640x480 Sigma:5

Standard VBM3D

Simplified VBM3D

Vassar

Noisy (dB) 34.15

Denoised (dB) 40.81 38.61

Speed (fps) 1.12 7.62

ballroom

Noisy (dB) 34.15

Denoised (dB) 41.55 37.58

Speed (fps) 1.14 7.61

Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.

8/30

Approach 2:

Proposed real-time implementation

03/12/2011

Filter version Proposed implementation VBM3D standard version

Data type Integer float

Memory buffer only 4 frames buffer whole video

Block matching Modified diamond search Full search

Temporal searching frames

4 5

9/30

Practical Results

03/12/2011

Res: 640x480 Sigma: 5

Standard VBM3D

Simplified VBM3D

Proposed Implementation

Vassar

Noisy (dB) 34.15

Denoised (dB) 40.81 38.61 38.49

Speed (fps) 1.12 7.62 30.49

ballroom

Noisy (dB) 34.15 34.15 34.15

Denoised (dB) 41.55 37.58 37.43

Speed (fps) 1.14 7.61 29.59

Video sequences vassar and ballroom are from http://www.merl.com/pub/avetro/mvc-testseq/orig-yuv/.

10/30

1. 30 time faster than standard VBM3D

2. Ready to be used for real-time application.

3. Good video denoising performance in a low noise level.

03/12/2011

Part 1 summary: A real-time filter

11/30

Part 2:

Joint video pre-filtering and compression

03/12/2011

12/30

Typical scheme of Pre-filtering and

compression

Usually we have two separate processes for pre-filtering and compression, and we

choose parameters separately.

03/12/2011

Pre-filtering

Rate control

x(t) y(t) z(t) Video Encoder

Filter control

F* Q*

13/30

R max

03/12/2011

(1)

Optimization Task

Joint Rate

control F* Q*

Pre-filtering x(t) y(t) z(t) Video Encoder

14/30

R max

Practical Results

Pre-filtering: VBM3D

Video Encoder: H.264/AVC (JM codec)

Experiments modes:

• Constant quantization mode

• Constant bitrates mode

03/12/2011

15/30

Practical Results

Constant Quantization Mode

In compression part, quantization parameters {Q} include:

• QPI ={21,22,…45}, respectively QPP = QPI + 5

• Other fixed codec setting

In filtering part, filtering parameters {F} include:

• Sigma ={0,0.5,1,…5},

• Other fixed filter setting

Full search:

• For each Q, find the best output under rate-distortion

framework.

03/12/2011

16/30

03/12/2011

PSNR gains up to 0.5 dB.

Bitrates savings up to 13.4%.

Practical Results

0 50 100 150 200 250 300 350 400 450

26

28

30

32

34

36

38

Bitrate (kbit/s)

Y-P

SN

R

hall.yuv, 25 points, QPI=21:1:45, QPP=26:1:50

H.264/AVC compression

joint VBM3D prefiltering and H.264/AVC compression

17/30

Practical Results

Constant Bitrates Mode

In compression part:

• Enable the constant bitrates control.

In filtering part, filtering parameters {F} include:

• Sigma ={0,0.5,1,…5},

• Other fixed filter setting

Full search:

• For each Q, find the best output under rate-distortion

framework.

03/12/2011

18/30

03/12/2011

PSNR gains up to 1.2 dB .

0 50 100 150 200 250 30035

35.5

36

36.5

37

37.5

38

38.5hall.yuv, Constant bitrate=215 kbit/s

frames

Y-P

SN

R

H.264/AVC compression

joint VBM3D prefiltering and H.264/AVC compression

Practical Results 19/30

H.264/AVC, frame 23 VBM3D+H.264/AVC, frame 23

03/12/2011

20/30

Part 2 summary:

Joint video pre-filtering and compression

1. Output video frames have less compression artifacts.

2. Output videos have consistent PSNR gains.

• can be up to 1.2 dB under constant bitrates mode.

• can be up to 0.5 dB under constant quantization mode.

3. Can save the bitrates up to 13.4% in comparison with only

compression.

03/12/2011

21/30

Part 3:

Joint video in-loop filtering and

compression

03/12/2011

22/30

Typical scheme of in-loop filtering and

compression

03/12/2011

23/30

03/12/2011

Optimization Task

24/30

Practical Results

Enhanced in-loop filtering: real-time filter from Part 1.

Video Encoder: H.264/AVC (JM codec)

Experiments modes:

• Inter mode

• QPI ∈ {21, 22 . . . 45} for I frame, and respective QPP =

QPI + 5 for P frames.

• Intra mode

• QPI ∈ {21, 22 . . . 45} for I frame.

03/12/2011

25/30

03/12/2011

PSNR gains up to 0.87 dB.

Bitrates savings up to 10.5%.

Practical Results

500 1000 1500 2000 2500 3000 3500

28

30

32

34

36

38

40

42

Bitrate (kbit/s)

Y-P

SN

R

hall.yuv, 24 points, QPI=21:1:45, QPP=26:1:50

H.264/AVC(intra)

H.264/AVC(intra) + Enhancing filter

26/30

03/12/2011

PSNR gains up to 0.35 dB.

Bitrates savings up to 6.3%.

Practical Results

200 250 300 350 400 450 500 550

26

28

30

32

34

36

38

Bitrate (kbit/s)

Y-P

SN

R

hall.yuv, 24 points, QPI=21:1:45, QPP=26:1:50

H.264/AVC(inter)

H.264/AVC(inter) + Enhancing filter

27/30

Part 3 summary :

Joint video in-loop filtering and compression

1. This joint approach consistently improves the compression

performance of H.264/AVC under intra mode, but it gains little

under inter mode.

2. Under intra mode

• PSNR gains up to 0.87 dB.

• Bitrates saving up to 10.5% in comparison with only compression.

3. Under inter mode

• PSNR gains up to 0.35 dB.

• Bitrates saving up to 6.3% in comparison with only compression.

03/12/2011

28/30

Conclusion

1. We propose

• a real-time video filter which has good de-noising performance.

• a joint pre-filtering and compression approach

• a joint in-loop de-noising and compression approach .

2. Results show that these two joint approaches enhance the

performance of the H.264/AVC standard:

• less compression artifact

• increased PSNR

• saved the bitrates.

03/12/2011

29/30

Thank you!

03/12/2011

30/30