Outline Lecture 12 Image and Video Coding Standards

1

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Lecture 12Image and Video Coding

Standards

July 8, 2009


Outline

Multimedia Coding Standardization ActivitiesImage Coding Standards

Video Coding Standards

1


Multimedia Coding Standardization Activities


Image and video encoding: A big picture

Pre-Processing

LossyCoding

Lossless Coding

Post-Processing

(Post-filtering)

EncodedImage/Video

Predictive Coding

QuantizationTransform Coding

Model-Based Coding…

Entropy CodingDictionary-Based Coding

Run-Length Coding…

A/D ConversionColor Space Conversion

Pre-FilteringPartitioning

…

Differential CodingMotion Estimation and

CompensationContext-Based Coding

…

InputImage/Video

3

2


Why do we need standards?

Common formats for data exchange• ⇒ Good for multimedia data creators, distributors, multimedia

device manufacturers, multimedia software developers, consumers, transcoders, …

Reduce development costs of the whole society and research communityFacilitate further development of new algorithms, products, services and standardsReduce risk of deploying new technology…

4


Types of standards

International standardsRegional (cross-nation) standards• Example: European standards

National standards• Example: Chinese standards

Industrial/Commercial standardsOpen (Royalty-free) standardsDe facto standardsNon-standard specifications…

5


What are standardized?

A common bitstream format of encoded multimedia dataA way of how a format-compliant encoded bitstream should be decodedSome different configurations: profiles and levelsSome optional featuresInformative description of encoder…

6


Who are making coding standards?and (International Electrotechnical Commission)

• JTC1 (Joint Technical Committee on Information Technology)• JPEG (Joint Photographic Experts Group) and JBIG (Joint Bi-level

Image Experts Group): JTC1/SC29/WG1• (Moving Picture Experts Group): JTC1/SC29/WG11

(International Telecommunication Union)• ITU-T (ITU Telecommunication Standardization Sector) ⇐ CCITT• ITU-R (ITU Radio Communication Sector) ⇐ CCIR• VCEG (Video/Visual Coding Experts Group): ITU-T / Study Group 16 /

Working Party 3 / Question 6• VQEG (Video Quality Experts Group)

JVT (Joint Video Team) ⇐ VCEG @ ITU-T + MPEG @ ISO/IEC(Society of Motion Picture and Television Engineers)

7

3


Who are making coding standards?

(Internet Engineering Task Force)SAC (Standardization Administration of China, 国家标准化管理委员会) + MIIT (Ministry of Industry and Information Technology of China, 中国信息产业部)• Audio Video Coding Standard Working Group (数字视音频编解码技术标准化工作组)

…

8


How are standards developed?

C4P: Call for proposalsCE: Core Experiments ProcessCE: Core Experiments DescriptionVM: Verification ModelWD: Working DraftCD: Committee DraftFCD: Final Committee DraftDIS/DS: Draft (International) StandardIS/S: (International) Standard

9


Who are making standards?

Industry• They want to keep their products interoperable with new

standards.• They want to put their patents into standards ⇒ So they can earn

royalties and reduce their own expenses on making related products and/or providing related services

• They want to create new markets

Academia• They want to transfer new coding technologies• They want to influence the real world• They want to learn from industry• …

10


Image Coding Standards

4


Bi-level image coding standards

CCITT T.4/T.6 (for Group 3/4 Fax machines, 1988-)• 1-D run-length coding + 2-D run-length coding + Huffman coding

JBIG/JBIG1 (ISO/IEC 11544, ITU-T T.82, 1993-) ⊃JBIG for Fax (ITU-T T.85, 1995-)• Predictive coding + Arithmetic coding (QM-coder) + Multi-

resolution progressive coding

JBIG 2 (ISO/IEC 14492, ITU-T T.88, 2000-) ⊃ JBIG2 for Fax (ITU-T T.89, 2000-)• Support both lossless and lossy modes• Partitioning (text, halftone, others) + Arithmetic coding (MQ-coder)

+ Huffman coding + 2-D Run-length coding (MMR) + Dictionary coding (symbol and pattern dictionaries)

12


Lossless image coding standards

GIF (Graphics Interchange Format, CompuServe Incorporated Specification, 1987-)• Dictionary coding (LZW)• Up to 256 colors• Support animation

PNG (Portable Network Graphics = PNG’s Not GIF, IETF RFC 2083, 1996-; ISO/IEC 15948, 2003-)• Predictive coding (2-D DPCM via filtering) + Dictionary/Entropy

coding (DELATE = LZ77 + Huffman coding)• Support true-color images• Do not support animation

13


Lossy image coding standard: JPEG

ISO/IEC 10918, ITU-T T.81 (1992-)• Partitioning + Color space conversion (RGB ⇒ YCbCr) +

Chroma subsampling (4:4:4/4:2:2/4:2:0) + Transform coding (DCT) + Differential coding (DC coefficients) + Run-length coding + Scalar quantization + Entropy coding (Huffman coding)

14


JPEG: Quantization tables (matrices)

For DCT coefficients at different locations, use different quantization steps: Cq=round(C/Q)

15

5


JPEG: DC coefficients coding

DCT coefficients are classified into 12 categories according to their amplitudes ⇒ The category labels are Huffman coded + Additional bits (FLC) to refine the amplitudes and the signs• Category 0 (0), Category 1 (1), Category 2 (2/3), Category 3 (4-

7), …, Category 11 (1024-2047)

16

LuminanceChrominance


JPEG: AC coefficients coding

All the AC coefficients are zig-zag scanned and run-length coded as RRRRSSSS• RRRR denotes the number of preceding zero coefficients• SSSS (1-11) denotes the amplitude of the next non-zero

coefficients• RRRRSSSS is Huffman encoded and refined by additional bits

(same as DC coefficients)• Example: (0,0,0,0,-7) ⇒ RRRR=0100 (4 zero proceding

coefficients), SSSS=0011 (Category 3), additional bits =000• Exceptions: 11110000 denotes “16 zeros plus one more zero”,

00000000 denotes EOB (end of block) ⇒ No additional bits follow Huffman-encoded RRRRSSSS

17


JPEG: Other features

Arithmetic coding is also allowed, but seldom usedProgressive DCT-based mode (vs. Baseline JPEG = Sequential coding)• Spectral selection: DCT coefficients are partitioned into different

bands, which are coded progressively• Successive approximation: DCT coefficients are partitioned into

different bitplanes, which are coded progressively

Hierarchical mode (multi-resolution progressive coding via downsampling)Lossless mode (Lossless JPEG)• Predictive coding (1-D/2-D DPCM) + Entropy coding

(Huffman/Arithmetic coding)

18


JPEG: Lossy compression examples

19

Quality = 100 (2.6:1) Quality = 50 (15:1)

Quality = 25 (23:1) Quality = 10 (46:1) Quality = 1 (144:1)

6


JPEG-LS

ISO/IEC 14495, ITU-T T.87 (1998-)A lossless/near-lossless image coding standard• Near-lossless means the coding error of each pixel is

below a predefined (small) threshold.LOCO-I (LOw COmplexity LOsslessCOmpression for Images) algorithm• Predictive coding (2-D DPCM + Adaptive context-based

predictive coding)+ Run-length coding + Golomb Rice coding

20


JPEG2000

ISO/IEC 15444, ITU-T T.800 (2000-)Wavelet-based image coding standard• Superior low bit-rate performance• Support both continuous-tone and bi-level images• Large dynamic range of pixel values• Large images and large image components• Support both lossless and lossy image compression• Progressive coding (scalability)• ROI (Region-of-interest) coding • Robustness to bit-errors• Compress once, decompress many ways• …

21


JPEG vs. JPEG2000

22

0.25 bit/pixel

0.5 bit/pixel


JPEG2000: Image coding componentsPreprocessing (Optional)• Spatial partitioning (Tiling)• DC level shifting = Making the mean close to 0• Multi-component transformation (RGB ⇒ YCbCr)

• Reversible color transform (RCT)• Irreversible color transform (ICT) = The color space transform used in

baseline JPEG without chroma subsampling

Transform coding (2-D DWT)Uniform scalar quantization with deadzoneSpectral partitioning and fractional bit-place coding (BPC) of DWT coefficient coding (EBCOT)Binary arithmetic coding (MQ-coder)Packetization…

23

7


JPEG2000: Encoding process

24


JPEG2000: 2-D DWT

2-D DWT are performed on each tile.Both reversible and irreversible DWT are used.• Cohen-Daubechies-Feauveau (CDF) wavelets are used• Irreversible DWT (lossy compression) = CDF 9/7• Reversible DWT (biorthogonal integer DWT for lossless

compression ⇒ Quantization is not needed!) = CDF 5/3 = Le Gall 5/3

DWT can be implemented by the lifting scheme

25


JPEG2000: EBCOT

EBCOT (Embedded Block Coding with Optimized Truncation)• Fractional bit-plane coding + Context-based binary

arithmetic coding• Each code-block is coded independently• Each bit-plane is scanned in a striped (4 rows form a

stripe) manner• A multi-pass process

• Pass 1: significance propagation• Pass 2: magnitude refinement• Pass 3: clean-up

26


JPEG2000: EBCOT

Some features• MSB-plane is coded with only the clean-up pass.• All the other bit-planes are coded with all the three passes.• Each coefficient bit is coded only in one of the three passes.

Three passes• Significance propagation pass: new significant bits, and

insignificant but context-significant bits are coded with the neighbor context.

• Magnitude refinement pass: all bits becoming significant in previous bit-planes are encoded with neighbor context.

• Clean-up pass: all the other bits are coded with neighbor context and run-length context.

27

8


JPEG2000: EBCOT

28

Sign bit

1-bit 0-bit


JPEG2000: EBCOT

Context-significance state of each bit is determined by its 8 neighbors.The context-significance states are initialized to be 0.

29


JPEG2000: Quality layer

A quality layer is a collection of some consecutive bit-planes of all code-blocks in all subbands and all components.

30


JPEG2000: Bitstream format

31

A precinct

Code-blocks

9


JPEG2000: Precinct

32

A precinct (packet partition location) is a group of code-blocks within a resolution level.


JPEG2000: Bitstream format

How packets are organized?• Quality: layer, resolution, component, position

(precinct number)• Resolution 1: resolution, layer, component,

position• Resolution 2: resolution, position, component,

layer• Position: position, component, resolution, layer• Component: component, position, resolution,

layer33


JPEG2000: ROI coding

MAXSHIFT (JPEG2000 Part 1): Arbitrary shape possible, shape coding unnecessary

34


JPEG2000: Scalability

Scalable coding• More than one quality/resolution levels, which can

be decoded from part of the whole bitstream

Scalabilities• SNR scalability

• Base/Lower layer: basic quality• Enhancement layer(s): added higher quality

• Spatial/Resolution scalability• Lower spatial resolution ⇒ Higher spatial resolution(s)

35

10


JPEG2000: SNR and spatial scalabilities

36


JPEG2000: Error resilence

Independent coding of code-blocks• ⇒ Error will not propagated out of a code-block

Termination of arithmetic coding and reset of contexts are allowed at the end of each coding passLazy coding mode• In BPC, after the 4th bit-plane is coded, the first two passes output

raw bits (arithmetic coder is skipped) and only the third pass uses arithmetic coding

Segmentation markers

37


JPEG-XR (Microsoft HD Photo)

ISO/IEC 29199 (2009-) ⇒ The latest image coding standardLapped bi-orthogonal transform (LBT) = PCT (Photo Core Transform) + POT (Photo Overlap Transform)Smaller transform blocks: n×n = 4×4, 2×4 and 2×2A second level of transform on DC coefficients (2n×2n)Run-length coding + Adaptive Huffman codingLossless color space conversionFlexible quantizationInter-block predictive coding

Competitor of JPEG2000

38


JPEG-XR: Spatial hierarchy

Spatial hierarchy• Image ⇒ Tile ⇒ Macroblock ⇒ Block ⇒ Sample

39

11


JPEG-XR: Spectral hierarchy

Spectral hierarchy (⇐ Two-level PCT)• Highpass (HP) band ⇒ Lowpass (LP) band ⇒ DC

band

40


JPEG-XR: Transforms

Lapped Bi-orthogonal Transform (LBT)• Photo Core Transform (PCT) +• Photo Overlap Transform (POT) = Overlapped

prefiltering ⇒ Anti-blocking

41


Video Coding Standards


Chronology of video coding standards

43

SVC

2007

MVC

2009

H.265

SMPTE

VC-1

AVS-China

12


H.120

ITU-T H.120 (1984-)The first video coding standardTarget bit-rate = 1.5 – 2 MbpsUnfortunately not practical: temporal quality is bad• Frame replenishment• DPCM• Scalar quantization• VLC• Motion compensation and background prediction (since version 2,

1988-)• NO transform coding!

44


H.261

ITU-T H.261 (1990-)The first truly practical video coding standard ⇒The basis of modern video coding standards• Macroblock + blockwise DCT + zig-zag scan + run-

length coding + entropy coding + loop filter (anti-blocking) + …

Target bit-rate = 64 kbps – 2MbpsInput formats = CIF/QCIF 4:2:0 (CIF4 4:2:0 added in 1993)

45


The general video encoding structure

46


The general video coding structure

47

13


Multimedia coding: A big picture

Pre-Processing

LossyCoding

Lossless Coding

Post-Processing

(Post-filtering)

EncodedImage/Video

Predictive Coding

QuantizationTransform Coding

Model-Based Coding…

Entropy CodingDictionary-Based Coding

Run-Length Coding…

A/D ConversionColor Space Conversion

Pre-FilteringPartitioning

…

Differential CodingMotion Estimation and

CompensationContext-Based Coding

…

InputImage/Video

48


MPEG-1

ISO/IEC 11172 (1993-)The first widely used video coding standardsTarget bit-rate = 1.5 Mbps (VCD)Features added on H.261• B-pictures (Bi-directional predication)• Half-pel motion compensation• Slice-structure coding• D-pictures (DC coefficients only)• Quantization weight matrices

49


MPEG-1: Bitstream format

50

Video sequence ⇒ Picture ⇒ GOP (Group of pictures) ⇒ Picture ⇒ …



… ⇒ Slice ⇒ Macroblock (MB) ⇒ Block ⇒ Sample (Transform coefficient)

51

Y U/V

MB “header”

14



52

Encoded DCT coefficients


MPEG-2 (H.262)

ISO/IEC 13818, ITU-T H.262 (1994-)The most successful (so far) video coding standardsTarget bit-rate = 2 – 15 Mbps (Various applications, including DVD and HDTV)Backward compatible with MPEG-1Features added to MPEG-1• Support on interlaced video: one frame = two fields• Increased DC quantization precision• I-picture concealment motion vectors (error resilience)• Various forms of scalability

53


MPEG-2: New features on interlacing

54

Dual-prime prediction


MPEG-2: Scalability

Spatial scalability• Lower spatial resolution ⇒ Higher spatial resolution

Temporal scalability• Lower frame rate ⇒ Higher frame rate

SNR scalability• Lower perceptual quality ⇒ Higher perceptual quality

Data partitioning• More significant data ⇒ Less significant data

55

15


H.263

ITU-T H.263 (1995-)Features added to H.261• Half-pel motion estimation and compensation• 3-D VLC of DCT coefficients• Median motion vector prediction• Motion vectors over picture boundaries• PB-frames• Four-MV Motion Compensation (INTER4V)• Overlapped Block Motion Compensation (OBMC)• Arithmetic coding• Rounding control• … (see H.263+ and H.263++)

56


H.263+ (v2, 1997/98)Annex I - Advanced INTRA Coding modeAnnex J - Deblocking Filter modeAnnex K - Slice Structured modeAnnex L - Supplemental Enhancement Information SpecificationAnnex M - Improved PB-frames modeAnnex N - Reference Picture Selection modeAnnex O - Temporal, SNR, and Spatial Scalability modeAnnex P - Reference picture resamplingAnnex Q - Reduced-Resolution Update modeAnnex R - Independent Segment Decoding modeAnnex S - Alternative INTER VLC modeAnnex T - Modified Quantization modeAnnex X - Profiles and levels definition

57


H.263++ (v3, 2000/2001)

Annex U - Enhanced reference picture selection modeAnnex V - Data-partitioned slice modeAnnex W - Additional supplemental enhancement information specification

Annex X - Profiles and levels definitionAppendix III - Examples for H.263 encoder/decoderimplementations

58


MPEG-4 Part 2 (Visual)ISO/IEC 14496-2 (1999-)Target bit-rate = Low bit-rate (initially scheduled) ⇒ O(1) kbps –O(10) MbpsPopular codecs: DivX, Xvid, …New features added to MPEG-2• Object-oriented video coding standards: rectangular video frame, video

objects with arbitrary shapes, 2-D/3-D mesh objects, animated human faces/bodies, still textures

• Half- or Quarter-pel motion compensation• Zerotree wavelet coding of still textures• 10/12-bit sampling• Error resilience / packet loss enhancements• …

59

16


MPEG-4 Visual: VOPs

GOP in MPEG-1/2 ⇒ VOP (Video Object Plane) in MPEG-4

60


MPEG-4: Bitstream hierarchy

61


MPEG-4: Wavelet texture coding

62


MPEG-4: Encoding process

63

17


H.264 / MPEG-4 AVC (Part 10)ISO-IEC 14496-10, ITU-T H.264 (2003-)The most popular new video coding standards (HDTV, DVB, …)Difference between MPEG-4 Part 2 (Visual )• Rectangular video frames only• Switching slices (SP/SI slices)• Quarter-pel and tree-structured motion compensation• Smaller DCT transform block: 4×4• Deblocking filter• More reference frames for macroblocks in an encoded frame• More complicated predication modes• CAVLC (context-adaptive VLC) and CABAC (context-adaptive binary arithmetic

coding)• VCL (Video Coding Layer) = RBSP (Raw Byte Sequence Payload) @ NAL

(Network Abstraction Layer)• …

64


H.264: Encoding process

65


H.264: Three profiles

66


H.264: MC – Macroblock partition

Macroblock partition and sub-macroblock partition

67

18


H.264: MC – Macroblock partition

68


H.264: MC – Subpixel interpolation

Quarter-pel motion compensation for luminance blocks. (8th-pel for chroma blocks.)

69

Original samples


H.264: MC – MV prediction

The current MVs of the current block can be predicted from previously encoded MVs.

70

The current block


H.264: Intra prediction

4×4 luma prediction modes

16×16 luma prediction modes

71

19


H.264: Scan order of DCT coefficients

72


More video coding standardsH.264 SVC (Scalable Video Coding, Annex G of H.264 standard, 2007)H.264 MVC (Multiview Video Coding, to be includd in H.264)H.265 (successor of H.264, 2010?)VC-1 = Microsoft WMV9 (SMPTE 421M, 2005-)Dirac (BBC) ⇒ SMPTE VC-2 (Intra coding part)• Wavelet-based video coding specification• Competitor of H.264

Motion JPEG/JPEG2000/JPEG-XRChina’s AVS1-P2: National royal-free standardMPEG RVC (Reconfigurable Video Coding)…

73


References


References for further readingYun Q. Shi and Huifang Sun, Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms, and Standards, 2nd Edition, CRC Press, 2008Iain E. G. Richardson, Video Codec Design: Developing Image and Video Compression Systems, John Wiley & Sons Ltd, 2002David S. Taubman and Michael W. Marcellin, JPEG2000: Image Compression Fundamentals, Standards and Practice, KAP, 2002Tinku Acharya and Ping-Sing Tsai, JPEG2000 Standard for Image Compression: Concepts, Algorithms and VLSI Architectures, John Wiley & Sons, Inc., 2004Mihaela van der Schaar, Deepak S. Turaga and Thomas Stockhammer, MPEG-4 Beyond Conventional Video Coding: Object Coding, Resilience, and Scalability, Morgan & Claypool, 2005John Watkinson, The MPEG Handbook: MPEG-1, MPEG-2, MPEG-4, 2nd Edition, Focal Press, 2004

75

Outline Lecture 12 Image and Video Coding Standards

Documents

Transcript of Outline Lecture 12 Image and Video Coding Standards