Video Processing

Dinesh Ganotra

Terminologies• Frames = Images• frame size : WxH• Color depth : Pixel represented by fixed amount of bits• Duration: T• FPS: Frames per Second• Pixel per frame: WxHxCD• Bits per frame: • Bit rate:• Video Size:• Low Level profile: Minimum video specification that any video

decoder should be able to handle

Peak Signal-to-Noise Ratio (PSNR), a metric for the "quality" of compressed video images, while the horizontal axis shows the transmission bit rate.10 frames per second, 176x144 (QCIF) resolution, 10 seconds of video, (100 frames)

Microsoft

• AVI : Audio/Video Interleaved File Format• ( Video for Windows: 1992)• Do not have pixel aspect ratio information

Hdrl tagWidth , height , frame rate

movi tagAudio , video data

Idex1 tagIndexing info

Apple

• MOV: Quicktime File FormatContainer file that contains one or more

tracks, each of which stores a particular type of data: audio, video, effects, or text (e.g. for subtitles).– Can also be data reference to the media stream

located in another file– Suited for editing.

MPEG-1

• color-space is transformed to YCbCr• Chroma is also sub-sampled to 4:2:0, meaning it is

reduced by one half vertically and one half horizontally

• Because of sub-sampling, YCbCr video must always be stored using even dimensions (divisible by 2), otherwise chroma mismatch ("ghosts") will occur, and it will appear as if the color is ahead of, or behind the rest of the video, much like a shadow.

Frame Types• Intra-frames: can be decoded independently of any other frame.

Effectively identical to baseline JPEG. • GOP: group of pictures: typically 17-18 non I-frames are their

between two I –frames• Predicted-frames: improve compression by exploiting the temporal

(over time) redundancy in a video. P-frames store only the difference in image from the frame (either an I-frame or P-frame) immediately preceding it.

• Bidirectional Frames: Can make prediction using both previous and future frames. Often avoided, not fully supported by hardware.

• D-frame: only DC component is encoded. Poor quality images, only used to find frames in videos. Obsolete.

MacroBlocks

• 8x8 blocks for quantization. However, because chroma (color) is subsampled by a factor of 4, each pair of (red and blue) chroma blocks corresponds to 4 different luma blocks. This set of 6 blocks, with a resolution of 16x16, is called a macroblock.

Motion Vector

The encoder compares the current frame with adjacent parts of the video from the anchor frame (previous I- or P- frame) in a diamond pattern, up to a (encoder-specific) predefined radius limit from the area of the current macroblock. If a match is found, only the direction and distance (i.e. the vector of the motion) from the previous video area to the current macroblock need to be encoded into the inter-frame (P- or B- frame).

MPEG-2

• Interlacing and high-definition

MPEG-4

• Intellectual property protection• Support B-Frames

DIVX

• Digital Video Express• Interactive video menus • Multiple subtitles (XSUB) • Multiple audio tracks

WMV

• Window Media Video

3GP

• Third Generation Partnership Project

FLV

• Flash Live Video• Originally developed by Macromedia , now

maintained by Adobe Systems• Contain material encoded with codecs

following the Sorenson Spark or VP6 video compression formats, protected by patents

Year Wise Progress• Year Standard Application• 1984 H.120 • 1990 H.261 Videoconferencing, Videotelephony • 1993 MPEG-1 ISO, IEC Video-CD • 1995 H.262/MPEG-2 DVD Video , Blu-Ray , Digital

Video Broadcasting, SVCD • 1996 H.263 T Videoconferencing, Videotelephony,

Video on Mobile Phones (3GP) • 1999 MPEG-4 Video on Internet (DivX, Xvid ...) • 2003 H.264/MPEG-4 Blu-Ray , Digital Video Broadcasting,

iPod Video , HD DVD

Video Codecs are generally installed in the windows installation

mrle Microsoft Run Length Encoding (msrle32.dll) msvc Microsoft Video 1 (msvidc32.dll) i420 Intel Indeo 4 codec (iyuv_32.dll) cvid Supermac Cinepak (iccvid.dll)

8,8 ,16 ,16

• All the video compression algorithms work on the DCT 8 cross 8 pixel blocks and macro-blocks are 16 cross 16 pixels block. So, its advisable to work with frames whose sizes are multiples of 16 and avoid exception handling algorithms and unnecessarily slowing down the compression/decompression algorithms.

Support for B-frames

Variable bit rate audio Variable frame rate Captions (Subtitles)

Video formats supported

Audio formats supported

Metadata/Tags supported Menu Support

3GP Yes Yes Yes 3GPP Timed Text

H.263, MPEG-4 Part 2 and H.264/MPEG-4 AVC

AMR-NB, AMR-WB, AMR-WB+, AAC, HE-AAC and HE-AAC v2 ? No

FLV Yes ? Yes No

Sorenson, VP6, Screen Video, H.264/MPEG-4 AVC

MP3, Nellymoser, ADPCM, Linear PCM, AAC[9], Speex[10] Yes No

mov Yes Yes Yes Yes

Limited to what is available to the QuickTime codec manager

Limited to what is available to Sound Manager or CoreAudio Yes

Can be done by using sprite tracks

divx Yes Yes Yes Yes

MPEG-4 Part 2 video encoded to meet DivX profiles MP3, PCM, AC-3 ? Yes

wmv Yes Yes Yes Yes

Almost anything through VFW or DMO; H.264/AVC is problematic

Almost anything through ACM or DMO; Vorbis is problematic Yes ?

AVI Yes Yes Yes Yes

Almost anything through VFW; H.264/AVC is problematic due to the limited B-frame support

Almost anything through ACM; Vorbis is problematic [5] not officially No

mpeg Yes Yes Yes No MPEG-1, MPEG-2

MPEG-1 Layers I, II, III (mp3), other formats only in private streams: LPCM[15] No No

Algorithm for m2v format

• BitStream [];• BitStream = Alignbits(BitStream)• BitStream = PutSequenceHeader(BitStream)• BitStream = Alignbits(BitStream)• BitStream = PutSequenceExtension(BitStream)• BitStream = Alignbits(BitStream)• BitStream = PutSequenceDisplayExtension(BitStream)• BitStream = Alignbits(BitStream)• BitStream = PutGOPHeader(BitStream)

• frame_Y_trfmd(row:row+7,col:col+7) = floor(dct2(frame_Y(row:row+7, col:col+7)) + 0.5);

• BitStream = PutPictureHeader(BitStream);• BitStream = Alignbits(BitStream);• BitStream = PutPictureCodingExtn(BitStream);• frame_Y_trfmd_Quant(row:row+7,col:col+7) =

round((frame_Y_trfmd(row:row+7,col:col+7)./(intra_quant_mat.*QUANT)).*tempmat);

• BitStream = Alignbits(BitStream);• BitStream = PutSliceHeader(ind_slice, BitStream); % Each row is a

slice•