MPEG-4 Essentials 108

MPEG-4 Essentials

E.P.J.Tozer

broadcast communications

Cover photos show from top to bottom: Camera optical splitter & CCD pickups Conditional Access Smart Cards Transmission Area at USA Cable, photo courtesy Ken Alexander, USA Cable

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com

broadacst communications

1

Contents

1 INTRODUCTION 3

2 LEVELS AND PROFILES 4

2.1 Profiles .........................................................................................................................................4

2.2 Detail of Profile content..............................................................................................................4

2.3 Levels............................................................................................................................................6

3 VIDEO CODING LAYER 8

3.1 Introduction.................................................................................................................................8

3.2 Intraframe Prediction.................................................................................................................8

3.3 Interframe Prediction ...............................................................................................................10 3.3.1 Variable prediction block size.............................................................................................11 3.3.2 Sub pixel motion vectors.....................................................................................................12 3.3.3 Coding motion vectors ........................................................................................................14

3.4 B Slices .......................................................................................................................................14

3.5 SI and SP Slices (Switch Slices)................................................................................................15

3.6 Transform Coding.....................................................................................................................16

3.7 MB-AFF (Macroblock Adaptive Frame Field Mode)............................................................17

3.8 Exp-Golomb, CAVLC and CABAC Entropy Coding............................................................17 3.8.1 Exponential-Golomb Coding ..............................................................................................18 3.8.2 CAVLC (Context based Variable Length Coding) .............................................................18 3.8.3 CABAC (Context based Adaptive Binary Arithmetic Coding)...........................................19

3.9 De-blocking filter ......................................................................................................................20

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


2

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


3

1 Introduction MPEG-4 originally started development around 1998 with VCEG (Video Coding Experts Group) as a call for proposals on H.26L. The aim of the project was to double the video coding efficiency of the extant MPEG-2 system, in order to achieve a 50% saving in bandwidth requirements for a given transmission picture quality. This document looks specifically at MPEG-4 part 10 (a subset of the extremely broadly scoped MPEG-4), which is also known as AVC (Advanced Video Coding) and is the standard H.264.

Date ITU-T ISO/IEC 1984 – 1988 H120 1990 H261 1993 MPEG-1 1994-6 H262 / MPEG-2 1995-2000 H263 1998-2001 MPEG-4 2003-2006 H264 / MPEG-4 AVC

Fig 1 History of ISO/IEC and ITU-Tvideo compression standards

Fig 2 Performance comparison of H264/AVC to earlier standards

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


4

2 Levels and Profiles Like MPEG-2 AVC/H264 breaks the broadly scoped standard into subsets of the standards to allow interoperability between systems without all systems having to comply with the full scope of the standard. The profiles define which processing tools may be used to code the video and the levels define the quality of coding through limits on data rates.

2.1 Profiles

Baseline Profile (BP): Used for low end applications, typically using less processing power used in videoconference and mobile applications.

Main Profile (MP): Originally created as the main consumer profile for broadcast and storage applications, it has now been superseded by the use of High Profile.

Extended Profile (XP): is the video streaming profile, this profile provides higher compression ratios and additional tools to provide increased robustness to data loss and stream switches.

High Profile (HiP): The primary profile for broadcast and disc storage applications, used for SD and HD TV applications (this profile is also used by HD DVD and Blu-ray disc systems).

High 10 Profile (Hi10P): An enhancement on HiP providing higher video resolution through the use of 10 bit video samples.

High 4:2:2 Profile (Hi422P) is the professional broadcast profile extending Hi10P to support 4:2:2 sampling format.

High 4:4:4 Profile (Hi444P) The use of this high end production profile is now depracted and is to be removed from the standard, while a new 4:4:4 profile is developed. Features include 4:4:4 sampling, up to 12 bits per sample, efficient lossless region coding plus coding RGB video.

2.2 Detail of Profile content All profiles support:

• 4:2:0 Chroma Format • Multiple Reference Frames • In-Loop Deblocking Filter • CAVLC Entropy Coding • 8 Bit Sample Depth • I and P Slices

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


5

Baseline Extended Main High High 10

High 4:2:2

High 4:4:4

B Slices No Yes Yes Yes Yes Yes Yes SI and SP Slices No Yes No No No No No CABAC Entropy Coding No No Yes Yes Yes Yes Yes

Ordering (FMO) Yes Yes No No No No No Arbitrary Slice Ordering (ASO) Yes Yes No No No No No

Redundant Slices (RS) Yes Yes No No No No No

Data Partitioning No Yes No No No No No Interlaced Coding (PicAFF, MBAFF) No Yes Yes Yes Yes Yes Yes

Monochrome Video Format (4:0:0) No No No Yes Yes Yes Yes

4:2:2 Chroma Format No No No No No Yes Yes 4:4:4 Chroma Format No No No No No No Yes 9 and 10 Bit Samples No No No No Yes Yes Yes 11 and 12 Bit Sample No No No No No No Yes 8x8 vs. 4x4 Transform Adaptation

No No No Yes Yes Yes Yes

Quantisation Scaling Matrices No No No Yes Yes Yes Yes

Separate Cb and Cr QP control No No No Yes Yes Yes Yes

Residual Colour Transform No No No No No No Yes

Predictive Lossless Coding No No No No No No Yes

Fig 3 Table of H264/AVC Profiles

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


6

Fig 4 AVC/H264 Profiles set

2.3 Levels In addition to Profiles H264/AVC sub-divides the standard into a number of levels that define the maximum rates and resolutions that are required to be supported. Level 3 is typically used for Standard Definition and Level 4 for High Definition.

Level number

Max macroblocks per second

Max frame size (macroblocks)

Max video bit rate (VCL) for Baseline, Extended and Main Profile

Max video bit rate (VCL) for High Profile

Max video bit rate (VCL) for High 10 Profile

Max video bit rate (VCL) for High 4:2:2 and High 4:4:4 Profile

Examples for high resolution @ frame rate (max stored frames)in Level

1 1485 99 64 kbit/s 80 kbit/s

192 kbit/s

256 kbit/s

[email protected] (8) [email protected] (4)

1b 1485 99 128 kbit/s 160 kbit/s

384 kbit/s

512 kbit/s


1.1 3000 396 192 kbit/s 240 kbit/s

576 kbit/s

768 kbit/s

[email protected] (9) [email protected] (3) [email protected] (2)

1.2 6000 396 384 kbit/s 480 kbit/s

1152 kbit/s

1536 kbit/s


1.3 11880 396 768 kbit/s 960 kbit/s

2304 kbit/s

3072 kbit/s


2 11880 396 2 Mbit/s 2.5 Mbit/s

6 Mbit/s

8 Mbit/s


2.1 19800 792 4 Mbit/s 5 Mbit/s

12 Mbit/s

16 Mbit/s


2.2 20250 1620 4 Mbit/s 5 Mbit/s

12 Mbit/s

16 Mbit/s

[email protected](10) [email protected] (7) [email protected] (6) [email protected] (5)

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


7

3 40500 1620 10 Mbit/s 12.5 Mbit/s

30 Mbit/s

40 Mbit/s

[email protected] (12) [email protected] (10) [email protected] (6) [email protected] (5)

3.1 108000 3600 14 Mbit/s 17.5 Mbit/s

42 Mbit/s

56 Mbit/s

[email protected] (13) [email protected] (11) [email protected] (5)

3.2 216000 5120 20 Mbit/s 25 Mbit/s

60 Mbit/s

80 Mbit/s


4 245760 8192 20 Mbit/s 25 Mbit/s

60 Mbit/s

80 Mbit/s

[email protected] (9) [email protected] (4)[email protected] (4)

4.1 245760 8192 50 Mbit/s 62.5 Mbit/s

150 Mbit/s

200 Mbit/s

[email protected] (9) [email protected] (4)[email protected] (4)

4.2 522240 8704 50 Mbit/s 62.5 Mbit/s

150 Mbit/s

200 Mbit/s

[email protected] (4)[email protected] (4)

5 589824 22080 135 Mbit/s 168.75 Mbit/s

405 Mbit/s

540 Mbit/s

[email protected] (13)[email protected] (13)[email protected] (12)[email protected] (5)3680x1536/26.7 (5)

5.1 983040 36864 240 Mbit/s 300 Mbit/s

720 Mbit/s

960 Mbit/s

[email protected] (16)

[email protected] (5)[email protected] (5)

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


8

3 Video Coding Layer

3.1 Introduction The H264/AVC video coder is essentially similar in structure and concept to the MPEG-2 video coder but takes the compression and optimisation processes further to achieve lower stream bitrates for a given video SNR (Signal to Noise Ratio).

VideoFrame Difference Transform Quantise Zig-Zag

ScanEntropy

Code

IntraPrediction

InterPrediction

DecodedReference

Frame

Fig 5 General process of H264/AVC encoder

EntropyDecode Zig Zag Inverse

QuanatiseInverse

TransformInverse

PredictionDeblock

FilterVideoFrame

Fig 6 General process of H264/AVC dencoder The two key additions H264/AVC makes to the MPEG-2 processes are the addition of prediction for Intra Frames, and the addition of a picture conditioning ‘Deblocking’ filter. Typically the H264/AVC encoder will achieve 50% the bit-rate of an MPEG-2 video coder for the same picture quality.

3.2 Intraframe Prediction In MPEG-2 systems Intraframes are directly coded by the lossy and lossless BRR processes. In order to improve the efficiency of I frame coding further H264 encoders use a content prediction process on I frame macroblocks. There are two basic versions of Intraframe MB prediction, one for 4x4 macroblocks and one for 16x16 macroblocks. The basic concept is shown below for a 16x16 macroblock:

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


9

Fig 7 mode 1 I frame prediction A prediction for the 16x16 macroblock for an I frame is generated by extrapolating the edge pixel values of the adjacent block, in the example above mode 1 I frame prediction is used, in which the rightmost column of pixels in adjacent macroblock are used to form the prediction block. The prediction block values are subtracted from the actual macro block values to form difference block for coding.

Vertical AverageHorizontal Diagonal

Fig 8 four16x16 macro block I frame prediction modes

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


10

There are four 16x16 I frame prediction modes:

• 0 vertical predictions • 1 horizontal predictions • 2 flat average (DC average) predictions • 3 diagonal predictions

In addition to the 16x16 macroblock modes I frame prediction can also be performed on 4x4 pixel macro blocks There are nine 4x4 block prediction modes as shown below:

0: vert

1: horizontal

3: diag down L 4: diag down R

5: vertical R

6: horiz down

7: vertical L

2: DC8: horiz up

Fig 9 nine 4x4 block I frame prediction modes To reduce the data required to indicate the prediction mode in use there is a “use most probable” prediction mode flag, which is calculated from the best prediction mode for the previous two blocks. If the encoder wishes to use a mode other than the most probable mode it sends a “new mode” flag and indicates which of the other eight modes it is using.

3.3 Interframe Prediction As with MPEG-2 the prediction for Interframe data may be performed using data from other frames, (Motion Estimation) and is typically one of the most significant contributors to bit rate reduction in the compression system. MPEG-4 extends on the available tools of MPEG-2 with the addition of three key methods:

• Variable prediction block size • ¼ pixel prediction resolution • Extended, multiple frame prediction

Variable block size is from 16x16 down to 4x4 (in the luminance channel) and frames up to 32 frames old may be used for prediction.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


11

3.3.1 Variable prediction block size The pixel dimensions of the set of prediction block sizes available to H264 is shown below:

16

16

8 8

8

8

4 4

8

8

8 816

8 4 4

44

0 0 11

0 0 1

2 3

0 0 101

0 12 3

Fig 10 nine 4x4 block I frame prediction modes The upper set, from 16x16 to 8x8 pixels are referred to as macroblock partitions, the lower set from 8x8 to 4x4 are referred to as sub partitions. The block sizes shown are for the luminance channel, if the component video is 4:2:2 the chroma block sizes will be half horizontally, and if the component video is 4:2:0 the chroma block sizes will be half both horizontally and vertically. Providing a variable block size facility in H264 allows the encoder to choose the optimum prediction size for each area of the picture, typically a large block size in ‘flat’ picture areas and a small block size in detailed areas of the image, or in areas where movement has occurred between frames.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


12

Fig 11 Prediction block size selected by encoder on P frame image data The example above shows example prediction block size decisions made by an AVC encoder on a section of P difference frame.

3.3.2 Sub pixel motion vectors MPEG-2 allows for ½ pixel motion vector accuracy, AVC improves this resolution to ¼ pixel accuracy. The resolution increase allows for higher compression through a more accurate definition of interframe motion. Specifically the half-pixel values are generated using a six tap FIR filter, the six tap filter points are (1/32,-5/32,20/32,20/32,-5/32,1/32). The half resolution pixel points must be generated in two phases. And the quarter pixel resolution values, are generated from the ½ pixel values using linear interpolation.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


13

Original Pixel

1/2 Pixel res (1st generation)

1/2 Pixel res (2nd generation)

1/4 Pixel res

Source pixel for interolation

Result pixel from interolation

Interpolation path

Full to half pixel resolution, pass 1 Full to half pixel resolution, pass 2 1/2 to 1/4 pixel resolution

Fig 12 Creation of ½ and ¼ pixel images through interpolation

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


14

Fig 13 Prediction errors resulting from1, ½ and ¼ pixel offsets

3.3.3 Coding motion vectors The small prediction block size coded by H264 creates a large number of motion vectors, one per block, in order to reduce the quantity of data H264 uses motion vector prediction. Motion vector prediction works by using the average value of the motion vectors for blocks above, diagonally above and horizontally adjacent to the current lock, and transmitting just the offset from that predicted value. Generally this will be a highly efficient method of coding as motion vectors are normally highly correlated locally.

3.4 B Slices In addition to P slices all Profiles other than Baseline H264/AVC, allow the use of B slices for coding. B-slices are similar in concept to MPEG-2’s B-frames in that they can be predicted from both past and future frames. MPEG-2 limits the coding of B-frames to the closest past and future I or P frame. H264/AVC extends the concept by allowing prediction from reference frames an aritbrary distance in the past and future. The reference frames may also both be in the past or the future,

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


15

CurrentFrame

2x Past Frames

Past and Future Frames

2x Future Frames

Fig 14 B prediction from multiple past and future frames

In order to able to use reference frames arbitrarily in the past or future reference frames are tagged and stored by the decoder and referenced via a look up table to link the current frames to its references.

3.5 SI and SP Slices (Switch Slices) SI and SP slices are support by some of the H264/AVC Profiles, they are designed to allow the efficient switching between two coded streams of video. Normally if two streams were to be switched between it would only be possible at an I frame, in order to allow rapid switching between two streams an many I frames would need to be present in the streams. SP and SI slices are specially coded predicted slices that are a hybrid between the two streams. In the SP switching example illustrated below when decoding stream B, B frame 4 would normally be decoded directly from B frame 3. If a stream switch were taking place from stream A, B frame 3 is reconstructed from A frame 2 and SP frame 3, then B frame 4 may be reconstructed. The intended use for SP and SI slices is for streaming multiple rate version of the same video, a decoder is able to efficiently switch between the different bit rate version depending of the (varying) bit rate available from the server. SI slices are only supported by extended profile and perform a similar function to SP slices, but additionally allow switching between dissimilar streams of video.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


16

A Frame 6A Frame 5A Frame 4A Frame 3A Frame 2A Frame 1 A Frame 7I

B Frame 6B Frame 5B Frame 4B Frame 3IB Frame 2B Frame 1 B Frame 7

A Frame 6SPA Frame 5A Frame 4A Frame 3

SPA Frame 2A Frame 1 A Frame 7

B Frame 6SP

B Frame 3SP

B Frame 6SPB Frame 5B Frame 4B Frame 3

SPB Frame 2B Frame 1 B Frame 7

IB or P frame

I frame only

SP/SI frame set

Stream Switching at I Frames

Stream Switching using SP Frame sets

Fig 15 Stream Switching using SI and SP Frames

3.6 Transform Coding Like MPEG-2 H264/AVC uses a transform coder to create a bit stream suitable for entropy (lossless) coding. The transform utilised by H264/AVC is an evolution from MPEG-2’s Discrete Cosine Transform but in essence produces similar results in a similar manner. The key differences between MPEG-2’s DCT and H264/AVC’s transform are:

• Integer arithmetic only is used, meaning that mismatches and errors between the encoder and decoder are eliminate, this is particularly important in H264/AVC as, due to the highly efficient prediction it is more susceptible to error drift between the encoder and decoder.

• Multiplication free transform, only addition and binary shifts are required to perform the transform making the transform less processor intensive.

• Hierarchical transform, DC values are retransformed in a second layer of (Hadamard) transformation

• Smaller 4x4 transform block reduce ringing artefacts (mosquito noise) • Quantisation is performed as part of the transform process using

multiplication but not division, reducing processor performance needs. The quantisation process is more finely controllable than in MPEG-2.

The output of the transform process is zig-zag scanned in a similar manner to MPEG-2, and the output fed to the entropy encoder.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


17

Fig 16 H264/AVC zig zag scan sequence (left frame coding mode, right field coding mode)

3.7 MB-AFF (Macroblock Adaptive Frame Field Mode) H264AVC MB-AFF mode allows an encoder to separate the field 1 and field 2 pixels before coding as in interlaced source video the two fields will be generated at separate points in time.

PictureMacroblocks

Codingblocks

Field 1

Field 2

PictureMacroblocks

Codingblocks

Frame1

16x16 pixel

16x16 pixel

Fig 17 H264/AVC MBAFF adaptive frame field Macroblock coding

3.8 Exp-Golomb, CAVLC and CABAC Entropy Coding Exp-Golomb, CAVLAC (Context Adaptive Variable Length Coding) and CABAC (Context Adaptive Binary Arithmetic Coding) are the three entropy coding techniques used for the lossless BRR(Bit Rate Reduction) process in H264/AVC. Exp-Golomb is used to code the higher syntax elements and CAVLC is used to code the quantised video transform coefficients in addition in many profiles CABAC is used to for optimise the data coding. The coding is made Context Adaptive through the selection of a large set of code lookup tables, the choice of which depends on prior coding decisions of local ares of the picture.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


18

3.8.1 Exponential-Golomb Coding Exponential-Golomb coding is performed on data other than the transformed picture data. An Exponential-Golomb code is a structured variable length coding system, the initial elements of which are shown in the table below.

Code_num Exp-Golomb word 0 1 1 010 2 011 3 00100 4 00101 5 00110 6 00111 7 0001000 8 0001001 9 0001010 10 0001011

Fig 18 First eleven elements of the Exponential-Golomb code

The form of the code is [length][1][value] and the H264/AVC control values are mapped to the code such that frequently occurring values are given short codes, i.e. are low numbers. Different (context based) mappings are made depending on the data to be coded:

ue Unsigned, direct mapping te truncated table mapping se signed mapping, e.g. for motion vector value me Mapped to a specified lookup table

3.8.2 CAVLC (Context based Variable Length Coding) The entropy code used for transform data is a variable length code, which uses a variety of look up tables to code the data, the table choice being context based, i.e. the look up table selected is dependant on prior coded data, which is done as generally the most efficient coding method is common to local areas of the picture.

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


19

0

0

0

0

0

0

0

00

0

-2

+4 -1

+1+1

+1

Fig 19 example output from transform block The CAVLC process is perform in a number of stages:

• The number of non zero coefficients and the quantity of training 1s encoded, in the above example that would be 6.3. This data is encoded using a context selected table.

• The sign of the trailing ones sequence is encoded in reverse order, bit=0

for positive bit=1 for minus . In the above example that would be 100.

• Encode the remaining values in reverse order 0+100-2+4 using adaptive table lookup.

• Encode the zero sequences before the last coefficient In the above

example that would b 0 This is efficient in that leading and training zero sequences don’t need to be coded.

• Encode runs of zeros in reverse order 0,00 in the above example

3.8.3 CABAC (Context based Adaptive Binary Arithmetic Coding) In some Profiles CABAC may be used as an alternative to CAVLC, CABAC codes according using:

• Optimised coding according to a syntax element’s type • Optimised coding according to the element’s local context • Binary arithmetic coding

The process used to code CABAC is as follows:

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


20

The value to be coded is binarised according to the following table:

Value Binarised value 0 0 1 10 2 110 3 1110 4 11110 5 111110 5 1111110 Etc.

The binarised value is the coded using a lookup table selected according to the data type and the probability of the current bit value being a 1 or a 0 using a binary decision process to select the lookup table from a tree of tables. The H264/AVC standard defines around 400 context tables for the arithmetic coding values, as well as different initialisation values to be used at start of slice for I, P and B slices which is dependant of the quantisation value..

3.9 De-blocking filter In transform based compression systems if high degrees of quantising are used to increase the compression ratio then picture distortions will occur. This distortion is most objectionable once errors start to become severe enough that ‘blocking’ appears in the decoded image. Blocking is caused by the high frequency components of the image being quantised out (or all but quantised out), leaving just the DC component of the block to represent the image. H264/AVC reduces the visibility of the blocking effect firstly through the use of a small transform block size of 4x4 pixels, and secondly through the use of a deblocking filter. The de-blocking filter operates by filtering the boundaries between reconstructed transform blocks, smoothing or blending their edges. The illustration below shows the range of a single pass of the filter, which is up to +/- 3 pixels at a block boundary. The actual range/strength of the filter depends on whether the block is intra coded, a macroblock boundary, predicted from different reference frames or have differing motion vectors

Fig 20 de-blocking filter edges and range, left 16x16 luminance block, right 8x8 chrominance block

MPEG-4 Essentials

E.P.J.Tozer v108

www.zetrox.com


21

Fig 21 Image quantised with QP=36, left de-blocking filter off, right de-blocking on

MPEG-4 Essentials 108

Documents

Transcript of MPEG-4 Essentials 108