School of Computer Science & Information Technology

G6DPMM - Lecture 12G6DPMM - Lecture 12

Digital VideoDigital Video

Sourcing Digital Video Film as DigitalFilm as Digital

Becoming more common as costs fallBecoming more common as costs fall May be compressed in the camera or uncompressedMay be compressed in the camera or uncompressed Stored on magnetic tapeStored on magnetic tape

D1, D2, D3, D5, D9 (Digital-S)D1, D2, D3, D5, D9 (Digital-S) Digital Betacam, BetacamSX, BetacamIMXDigital Betacam, BetacamSX, BetacamIMX DVCAM DVCPRODVCAM DVCPRO DV / MiniDV (consumer)DV / MiniDV (consumer) Digital-8 (uses Hi-8 tapes)Digital-8 (uses Hi-8 tapes)

Compressed vs UncompressedCompressed vs Uncompressed Film as AnalogueFilm as Analogue

Digitise with capture board (frame grabber)Digitise with capture board (frame grabber) Wildly variable in price and qualityWildly variable in price and quality

Digital Video Playback Playback Software / Multimedia FrameworkPlayback Software / Multimedia Framework Windows Media Player (Microsoft)Windows Media Player (Microsoft)

Bundled with Windows (3.1 - XP)Bundled with Windows (3.1 - XP) Free versions for Mac OSX and SolarisFree versions for Mac OSX and Solaris Media framework – includes video infrastructureMedia framework – includes video infrastructure File formats (e.g. AVI / WMV)File formats (e.g. AVI / WMV)

QuicktimeQuicktime Bundled with Macintosh (OS 7 – X)Bundled with Macintosh (OS 7 – X) Free version for WindowsFree version for Windows Media framework – includes video infrastructureMedia framework – includes video infrastructure MOV file formatMOV file format

RealNetworksRealNetworks RealPlayer (Windows / Macintosh / Linux / SolarisRealPlayer (Windows / Macintosh / Linux / Solaris Originally a streaming technology, now a full blown media Originally a streaming technology, now a full blown media

frameworkframework

Digital Video Compression Video consists of a time-series of framesVideo consists of a time-series of frames Frames are still imagesFrames are still images

NB still image compression techniquesNB still image compression techniques Sequences of frames may be compressed by:Sequences of frames may be compressed by:

Motion estimationMotion estimation Only updating the parts of the frame that have moved Only updating the parts of the frame that have moved E.g. consider a talking head!E.g. consider a talking head!

Motion compensationMotion compensation Finding static parts of the frame and moving them aroundFinding static parts of the frame and moving them around E.g. consider a pan or zoomE.g. consider a pan or zoom

Compression Codecs Playback software (media framework) does not Playback software (media framework) does not

usually contain compression/decompression usually contain compression/decompression softwaresoftware

Compression software implementing a specific Compression software implementing a specific algorithm is called a algorithm is called a codeccodec

Codecs are required by the framework – many Codecs are required by the framework – many different codecs can be installed into a given different codecs can be installed into a given frameworkframework

NB compatibility and distribution (legality) NB compatibility and distribution (legality) issuesissues

Common Codecs IndeoIndeo CinepakCinepak P*64P*64 SorensonSorenson ClearVideoClearVideo VDOwaveVDOwave RealVideoRealVideo MPEGMPEG

NB this is NB this is notnot a single codec a single codec MPEG also has many spin-offs – such as Microsoft MPEG also has many spin-offs – such as Microsoft

WMV or DivXWMV or DivX

Video Compression Uncompressed video is expensiveUncompressed video is expensive

Hard to store – very large volumes of dataHard to store – very large volumes of data Expensive equipment required to manipulateExpensive equipment required to manipulate

Compression is Compression is alwaysalways lossy lossy Multi-way trade-offMulti-way trade-off

File size / transfer rate (bandwidth)File size / transfer rate (bandwidth) Frame rate (typically 15fps for multimedia)Frame rate (typically 15fps for multimedia) Image qualityImage quality Frame sizeFrame size

Different codecs have differing strengths and Different codecs have differing strengths and weaknesses (e.g. Indeo vs Cinepak)weaknesses (e.g. Indeo vs Cinepak)

Indeo Algorithms developed for DVI – Intel i750 Algorithms developed for DVI – Intel i750

chipsetchipset Indeo – implemented in softwareIndeo – implemented in software Proprietary intel algorithmsProprietary intel algorithms Two levels of compressionTwo levels of compression

PLV (Production Level Video) – broadcast qualityPLV (Production Level Video) – broadcast quality RTV (Real Time Video)RTV (Real Time Video)

Typical compression ratio of 160:1Typical compression ratio of 160:1 Beware versions (5.1 safe on modern machines, Beware versions (5.1 safe on modern machines,

3.2 fairly ubiquitous)3.2 fairly ubiquitous)

MPEG Moving Picture Expert GroupMoving Picture Expert Group Established by ISO in 1988Established by ISO in 1988 Remit – define standards for movie with soundtracksRemit – define standards for movie with soundtracks

Digital televisionDigital television Video phonesVideo phones Consumer electronicsConsumer electronics Digital cinemaDigital cinema Video gamesVideo games

Family of standardsFamily of standards Always lossyAlways lossy Discrete Cosine Transform (DCT) algorithmDiscrete Cosine Transform (DCT) algorithm

MPEG-1 (1993) The original MPEG standardThe original MPEG standard Objective: deliver video and audio from CD-Objective: deliver video and audio from CD-

ROM with comparable quality to VHS tape and ROM with comparable quality to VHS tape and CD-AudioCD-Audio

Encodes an audiovisual signal in about 1.5 MbpsEncodes an audiovisual signal in about 1.5 Mbps Images are progressively scannedImages are progressively scanned

This fundamentally limits qualityThis fundamentally limits quality Motion compensationMotion compensation

I-Frames (reference)I-Frames (reference) P-Frames (predictive)P-Frames (predictive) B-Frames (bidirectional predictive)B-Frames (bidirectional predictive)

MPEG Frame Types I-FramesI-Frames

Reference frameReference frame Complete image – no missing informationComplete image – no missing information Subsequent P-frames are based upon an I-frameSubsequent P-frames are based upon an I-frame

P-FramesP-Frames Only contain the difference between themselves and the Only contain the difference between themselves and the

previous P-frame or I-frameprevious P-frame or I-frame Anything that does not change is not storedAnything that does not change is not stored

B-FramesB-Frames Contain the difference between themselves and both frames on Contain the difference between themselves and both frames on

either sideeither side B-frames are not always implemented because of memory B-frames are not always implemented because of memory

requirementsrequirements

The MPEG GOP GOP StructureGOP Structure Group of PicturesGroup of Pictures Specifies I / P frame structure (e.g. 1 I frame per 11 P Specifies I / P frame structure (e.g. 1 I frame per 11 P

frames.frames.

IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPIIPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPI

The fewer I frames the better the compression…The fewer I frames the better the compression… but frame to frame calculations are less accurate – leading to but frame to frame calculations are less accurate – leading to

artefactsartefacts PEL (picture elements)PEL (picture elements)

Block of pixels, used as a basis for compressionBlock of pixels, used as a basis for compression Motion of PEL is predicted based on I, P or B framesMotion of PEL is predicted based on I, P or B frames

The MPEG-1 standard Five parts (separate ISO standards)Five parts (separate ISO standards)

1. Systems1. Systems

2. Video2. Video

3. Audio (NB MPEG 1, level 3 – MP3)3. Audio (NB MPEG 1, level 3 – MP3)

4. Conformance testing4. Conformance testing

5. C implementation (encoding and decoding)5. C implementation (encoding and decoding)

MPEG Audio MP3 is MPEG 1, audio layer 3MP3 is MPEG 1, audio layer 3

Layer 1: 384, 4:1 compressionLayer 1: 384, 4:1 compression Layer 2: 192 – 256kbit/s, 6:1..8:1 compressionLayer 2: 192 – 256kbit/s, 6:1..8:1 compression Layer 3: 112 – 128 kbit/s, 10:1..12:1 compressionLayer 3: 112 – 128 kbit/s, 10:1..12:1 compression

CompressionCompression Modified discrete cosine transfom (MDCT)Modified discrete cosine transfom (MDCT) 32 band polyphase quadrature filter (PQF)32 band polyphase quadrature filter (PQF) Alias reduction postprocessingAlias reduction postprocessing

Severe design limitations of MP3Severe design limitations of MP3 Bitrate, time resolution, lack of scaleability – Bitrate, time resolution, lack of scaleability –

especially for high frequenciesespecially for high frequencies

Applications of MPEG-1 MPEG-1 compression is cheap and rapidMPEG-1 compression is cheap and rapid Consumer DevicesConsumer Devices Multimedia / CD-ROMMultimedia / CD-ROM Internet VideoInternet Video Video CD (production is extremely cheap)Video CD (production is extremely cheap) Internet audio (MP2)Internet audio (MP2) Audio only devices (MP3 players)Audio only devices (MP3 players)

MPEG-2 (1994) A completely different standard from MPEG 1.A completely different standard from MPEG 1. Objective:Objective:

Deliver video and audio at comparable quality to Deliver video and audio at comparable quality to broadcast video at roughly 4 times the typical MPEG-broadcast video at roughly 4 times the typical MPEG-1 bit rate1 bit rate

Requires between 3 and 15 MbpsRequires between 3 and 15 Mbps Improvements over MPEG-1Improvements over MPEG-1

Higher resolutionHigher resolution Higher image qualityHigher image quality Supports interlaced video formatsSupports interlaced video formats Multiresolution scalabilityMultiresolution scalability Multiple sound track featuresMultiple sound track features ScaleableScaleable

The MPEG-2 Standard ISO 13818 – 10 partsISO 13818 – 10 parts 640 patents, held by 24 corporations and one university640 patents, held by 24 corporations and one university Video Encoding (grossly simplified)Video Encoding (grossly simplified)

I, P and B frames – similar to MPEG-1I, P and B frames – similar to MPEG-1 Video is separated into:Video is separated into:

1 luminance and 2 chrominance channels, 1 luminance and 2 chrominance channels, 4 luminance macroblocks 4 luminance macroblocks variable number of chrominance macroblocksvariable number of chrominance macroblocks

Macroblocks of P and B frames are compared, and a motion Macroblocks of P and B frames are compared, and a motion vector calculatedvector calculated

Each block is treated with DCT, the result is run-length Each block is treated with DCT, the result is run-length encoded and a huffman compression is appliedencoded and a huffman compression is applied

Scaleable MPEG MPEG-2 is scaleable (layered coding)MPEG-2 is scaleable (layered coding) Single coded bitstream is scaleable to various bitratesSingle coded bitstream is scaleable to various bitrates Layers can be incrementally decodedLayers can be incrementally decoded

Basic layer – low quality videoBasic layer – low quality video Subsequent layers – higher quality videoSubsequent layers – higher quality video

If the bandwidth is low, only the bitstream from the If the bandwidth is low, only the bitstream from the base layer is sentbase layer is sent

If the bandwidth changes, more or fewer layers may If the bandwidth changes, more or fewer layers may be usedbe used

The same principle can be used on noisy connectionsThe same principle can be used on noisy connections

Applications of MPEG-2 DVD VideoDVD Video HDTVHDTV Ditgital satelite / cable TVDitgital satelite / cable TV Digital video recordersDigital video recorders Video conferencingVideo conferencing Some gamesSome games Some multimedia (usually where very high Some multimedia (usually where very high

quality is essential)quality is essential) NB encoding is processor intensive – thus more NB encoding is processor intensive – thus more

expensive than MPEG-1expensive than MPEG-1

Editing MPEG 1 & 2 High performance compression formatsHigh performance compression formats

Not designed for editingNot designed for editing Only I-frames are stored in their entiretyOnly I-frames are stored in their entirety

This makes editing, rewinding or even bookmarking This makes editing, rewinding or even bookmarking extremely difficultextremely difficult

MPEG editors tend to be crudeMPEG editors tend to be crude Serious editing involves decmopressing and Serious editing involves decmopressing and

recompressingrecompressing Quality lossQuality loss

Moral – don’t edit MPEG if at all possibleMoral – don’t edit MPEG if at all possible Therefore don’t store digital video as MPEG!Therefore don’t store digital video as MPEG!

MPEG-4 (1998) A different approach to videoA different approach to video Designed primarily for streaming video (e.g. on WWW), Designed primarily for streaming video (e.g. on WWW),

3G phones, and other multimedia.3G phones, and other multimedia. MPEG-4 absorbs most of MPEG-1 and MPEG-2MPEG-4 absorbs most of MPEG-1 and MPEG-2 Adds: Adds:

VRML (3D) supportVRML (3D) support Object-based codingObject-based coding User interactivityUser interactivity Hypermedia capabilityHypermedia capability Streams from different sources can be combinedStreams from different sources can be combined Supports uploading and downloadingSupports uploading and downloading Rights management protocolRights management protocol

Implementation of MPEG-4 Designed to be modularDesigned to be modular

Standard in 20 parts (levels)Standard in 20 parts (levels) Each level contains profilesEach level contains profiles Developers implement components of MPEG-4Developers implement components of MPEG-4 Probably no full implementationProbably no full implementation

Various partial implementations, such as:Various partial implementations, such as: DivXDivX Microsoft ASFMicrosoft ASF Quicktime 6Quicktime 6

Object-based coding Media objectsMedia objects Audio and visual objectsAudio and visual objects

Natural objects (captured by camera)Natural objects (captured by camera) Synthetic objects (created by computer)Synthetic objects (created by computer)

Visual objects can be any shape (not just rectangles)Visual objects can be any shape (not just rectangles) Video manipulation (insertion, deletion, rotation etc.)Video manipulation (insertion, deletion, rotation etc.) Video indexing and retrievalVideo indexing and retrieval Texture mapsTexture maps SpritesSprites 3-d models and animation3-d models and animation

Probably the future of multimedia!Probably the future of multimedia!

MPEG-7 Multimedia content description interfaceMultimedia content description interface Aimed at addressing issues of content retrievalAimed at addressing issues of content retrieval

Intended primarily for digital librariesIntended primarily for digital libraries Object based (as MPEG-4)Object based (as MPEG-4) Supported data:Supported data:

Still imagesStill images 3D models3D models Sampled audioSampled audio Speech Speech VideoVideo Composition information (combinations of the above)Composition information (combinations of the above)

Data representationData representation Binary objects, XML or bothBinary objects, XML or both

MPEG-21 Designed to manage digital media resourcesDesigned to manage digital media resources

Digital item declarationDigital item declaration Digital item identification and descriptionDigital item identification and description Content management and usageContent management and usage Intellectual property management and protectionIntellectual property management and protection Terminals and networks Terminals and networks Content representationContent representation Event reportingEvent reporting