Advance video distribution

Advance Video Distribution

www.dsp-ip.comFast Forward Your Development

Day I - Agenda

• Current Delivery technologies (RTP/RTSP)

• Components in depth

▫ File Formats

▫ Codecs

• Video Streaming Evolution

▫ Requirements


▫ Requirements

▫ Solutions

• CDN

• Basic Adobe Flash

Day II - Agenda

• Microsoft “Smooth Streaming”

• Adobe “HTTP Streaming”

• Apple “HTTP Streaming”

• Internet Gorillas positioning

• Adobe OSMF


Current Delivery Methods


Progressive Download

• Simple• Utilizing existing protocols & servers (HTTP)

• Media file is prepared: metadata up front

• Playback - after metadata is received

• Cache-ability - supported


• Cache-ability - supported

• Seek-ability – very limited support

• Poor user experience - seek, multi-rate

• Waste of bandwidth when not watched fully

• Low cost

Pseudo Streaming• Media is sent as a regular file like Progressive DW

• Server must understand how the media is

structured

• Playback: after metadata is received

• Existing protocols

▫ Non standard server


▫ Non standard server

▫ Non standard client component

• Cache-ability – Limited !

• Seek-ability – supported

• User experience – better than PD, support seek.

• Waste of bandwidth when not watched full

Streaming

• Seek-ability – supported

• Server side – proprietary technology (FMS)

• Cache-ability – requires special servers for streaming

• User experience – very good

• Cost – high


• Cost – high

HTTP Streaming Intro

• HTTP Streaming offers the advantages of:▫ Progressive download in terms of � Cost

� Standard Server

� Scalability

� Standard client components (OSMF)



▫ Streaming in terms of � User experience

� Seek-ability of streaming

RTSP Streaming


RTSP Protocol

• Real Time Streaming Protocol

• Used for controlling streaming data over the web.

• Designed to efficiently broadcast audio/video-on-demand to large groups.


on-demand to large groups.

• Using Directives to control the stream

▫ Options, Describe, Setup, Play, Pause, Record, Teardown.

SDP Protocol• Describes the metadata of the stream.

• Mainly used in: SIP, RTSP and other Multicast sessions.

• Sample SDP description:▫ v=0

▫ o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5

▫ s=SDP Seminar

▫ i=A Seminar on the session description protocol

Session ID

Protocol Version

Session Name

Session Info.

Description URI


▫ i=A Seminar on the session description protocol u=http://www.example.com/seminars/[email protected] (Jane Doe)

▫ c=IN IP4 224.2.17.12/127

▫ t=2873397496 2873404696

▫ a=recvonly

▫ m=audio 49170 RTP/AVP 0

▫ m=video 51372 RTP/AVP 99

▫ a=rtpmap:99 h263-1998/90000

Connection Info.

Active session time

Session Attribute lines

Media Name and

Transport address

Media Attribute lines

Client-Server flow

Web

Browser

Web

Server

Client Server

HTTP GET

Stream URI

DESCRIBE

OPTIONS

SDP Information


Media

Player

Media

Server

SETUP

PLAY

RTP Media Stream

PAUSE

TEARDOWN

RTP Media Stream

SDP Information

RTSP Protocol Parameters

• version

▫ The version of rtsp. (RTSP/1.0)

• URL

[rtsp/rtspu]://host:port/path


*port – the actual stream will be delivered in other port

Reliable

protocol

(TCP)

unreliable

protocol

(UDP)

legal domain name or IP

address

port used to control the

stream

the server stream path

RTSP Protocol Parameters (Ctnd.)• Session ID

▫ Generated by the server

▫ Stays constant for the entire session

• SMPTE – Relative timestamp

▫ A relative time from the beginning of the stream.

▫ Nested types: smpte-range, smpte-type, smpte-time.

▫ smpte-25=(starttime)-(endtime)


• UTC – Absolute time

▫ Absolute time using GMT.

▫ Nested types: utc-range, utc-time. utc-date

▫ utc-time = (utcdate)T(utctime).(fraction)Z

• NPT - Normal Play Time

▫ Absolute position from the beginning of the presentation.

▫ npt=123.45-125

RTSP Session Details

Initiation


Handling

Termination

RTSP - OPTIONS request

Media URL


OPTIONS – Request for information about the communication options available by

the Request-URI.

• CSeq – the request id, a response with the same id will be sent from the server.

•Media URL – the URL of the video.

•Client Player – the user agent of the client.

Media URL

Client Player

Request ID

RTSP – OPTIONS response

Response Code


• All RTSP response codes are divided into 5 ranges (RFC 2326 7.1.1) :

1xx – Informational, 2xx – Success 3xx – Redirection, 4xx – Client Error, 5xx – Server Error.

CSeq has the same value as the request CSeq field.

• The server response will return the available methods that it supports.

It May contain any arbitrary data the server want to expose.

Available

Options

RTSP – DESCRIBE request


DESCRIBE is used to retrieve the description of the media URL and the session.

The description response MUST contain all media and streaming data needed in order to initialize the session.

Fields: Accept - Used to inform the server which description methods the client supports.

Session Description Protocol (SDP) is highly used.

Notice that CSeq field is increased by one.

Description readers

RTSP – DESCRIBE response

The media URL the response is referring to

The description method used

The length of the SDP message


Description readers

SDP

The response will always return the details of the media.

SDP details will be next>

RTSP – GET_PARAMETER request


GET_PARAMETER is used to retrieve information about the stream.

The request can be initiated from the Client or from the Server.

The request/response message body is left to server/client implementation.

The parameters can be: packets received, jitter, bps or any other relevant information about the

stream.

RTSP – SETUP request


SETUP is used to specify the transport details used to stream the media.

The request/response message body is left to server/client implementation.

The parameters can be: packets received, jitter, bps or any other relevant information about the

stream.

Transport protocol Unicast/Multicast RTP/RTSP client

media portTrack ID


SETUP response will contain the session ID.

For each track ( audio/video ) a different SETUP request will be made

After the response is received, a PLAY request can be made to start receiving the media stream.

Transport

protocol

Unicast/Multicast

server option

Unicast

destination ip

Last gateway

source ip

The client port

to receive

media data

The server

port to receive

media data

RTSP – PLAY request


PLAY request tells the server to start send data through the streaming details defined in the

SETUP process.

PLAY request maybe queued so that a PLAY request arriving while a previous PLAY request is

still active is delayed until the first has been completed.

Normal Play

TIme

RTSP – PAUSE request


Stream URL

PAUSE request tells the server to pause the streaming.

When the user will want to start the stream again he’ll send a PLAY request to the same URL.

The request may contain time information to handle when the pause will take effect.

RTSP – TEARDOWN


TEARDOWN stops the stream delivery for the URL specified.

Informs the server that the client is disconnecting from it.

The response will include only the response code.

Description readers

RTSP – More Request types• RECORD:

▫ Initiates recording operation given a time information and stream URL.

• REDIRECT:

▫ Server to Client request that informs the client he needs to switch the server he connected to.The request will contain the new server URL.


The request will contain the new server URL.

• SET_PARAMETER:

▫ sends a request to change a value of the presentation stream.The response code will contain the answer.

• ANNOUNCE:

▫ Can be initiated both by client/server. Informs the recipient that the SDP table of the object has changed.

Components

•Containers•CODECs


•CODECs

File Format

www.dsp-ip.comFast Forward Your Developmentframe

samplesample

frame

sample

mdat

trak

moovAudio track

trak

Video track

Movie (meta-data)

Media Data

sample

Agenda

• Intro to file formats

• Second Generation formats

▫ RIFF: AVI, WAV

• Third Generation Containers

▫ MPEG4 FF

▫ MKV


▫ MKV

File Format Segmentation

File Formats

3rd Generation

2nd Generation

1st Generation


Generation

XML BasedObject Based

Generation

Media Muxer

Generation

Raw / Proprietary

2nd Generation File Formats


2ND Generation Files features

• Multiple media track in the same file

• Identification of codec

▫ Usually by FourCC

• Interleaving


2nd Generation File Formats

2nd Generation FF

RIFF ASF MPEG2 FLV


RIFF

WAV AVI

ASF

WMA WMV

MPEG2

MP2PS VOB

MP2TS

FLV

AVI FILE FORMAT


AVI Overview

• AVI files use the AVI RIFF format (like WAV)

• Introduced by Microsoft on 1992

• File is divided into:

▫ Streams – Audio, Video, Subtitles

▫ Blocks “Chunks” -


Blocks / Chunks

• A RIFF File logical unit

• Chunks are identified by four letters (FOUR-CC)

• RIFF file has two mandatory sub-chunks and one optional sub-chunk

• Mandatory Chunks:

▫ hdrl – File headerRIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI '

LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘


▫ hdrl – File header

▫ movi - Media Data

• Optional Chunk

▫ idx1 - Index

RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘

'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . )

LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) ['idx['idx['idx['idx1111'<AVI Index>]'<AVI Index>]'<AVI Index>]'<AVI Index>]

)

*This order is fixed

AVI main headerRIFF 'AVI ' - Identifies the file as RIFF file.

LIST 'hdrl' - Identifies a chunk containing sub-chunks that define the format of the data.

'avih' - Identifies a chunk containing general information about the file. Includes:▫ dwMicrosecPerFrame - Time between frames


▫ dwMicrosecPerFrame - Time between frames

▫ dwMaxBytesPerSec – number of bytes per second the player should handle

▫ dwReserved1 - Reserved

▫ dwFlags - Contains any flags for the file.

Example - headers

chunk ID chunk IDformat

Time between

frames

Data rate

chunk size

flages

Total no. of

Initial frame

streams

Avi file header


Total no. of

frames

Frame

width 320

Frame

height

Stream header

Junk chunk

identifier

Size of padding

reserved

Example – data chunks

Audio data chunk

(stream 01)


(stream 01)

video data chunk

(stream 00)

AVI Summary

• Advantages

▫ Includes both audio and video

▫ Index-able

• Disadvantage

▫ Not suited for progressive DW

▫ Very rigid format


▫ Very rigid format

▫ Insufficient support for: seeking, metadata multi-reference frames

3RD GENERATION FILE FORMATS


Why “Fix it”?

• 2nd Generation Formats are missing:

• Metadata

▫ Separate from Media

▫ Info on angle, language, Synchronization

▫ Versioning

• Better Streaming Support


• Better Streaming Support

▫ Reduce CPU per stream

▫ Better seeking support

• Better parsing

▫ XML

▫ Atom Based

Main Attributes

• File format is not just a Video / Audio multiplexer

• Separation between

▫ Media – Audio, Video, Images, Subtitles

▫ Metadata – Indexing, frame length, Tags


3rd Generation File Formats

3rd Generation

XML Based Object Based


Matruska (MKV) MOV MPEG4 FF

3GPP FFFragmented MPEG4 FF

MPEG4 FILE FORMAT


MP4 File Format

• File Structuring Concepts

▫ Separate the media data from descriptive (meta) data.

▫ Support the use of multiple files.

▫ Support for hint tracks:

� support of real time streaming over any protocol


� support of real time streaming over any protocol

Separate Metadata and Media

• Key meta-information is compact▫ The type of media present▫ Time-scales▫ Timing▫ Synchronization points etc.

• Enables


• Enables▫ Random access▫ Inspection, composition, editing etc.▫ Simplified update

Multiple file support

• Use URLs to ‘point to’ media

▫ Distinct from URLs in MPEG-4 Systems

• URLs use file-access service

▫ e.g. file://, http://, ftp:// etc.

• Permits assembly of composition without requiring data-copy


requiring data-copy

• Referenced files contain only media

▫ Meta-data all in ‘main’ file

Logical File Structure

• Presentation (‘movie’) contains…

• Tracks which contain…

• Samples


Physical Structure—File

• Succession of objects (atoms, boxes)

• Exactly one Meta-data object

• Zero or more media data object(s)

• Free space etc.


Example Layout

moovAudio track

trak

Video track

Movie (meta-data)


frame

samplesample

frame

sample

mdat

trak

Media Data

sample

Meta-data tables

• Sample Timing• Sample Size and position• Synchronization (random access) points, priority

etc.• Temporal/physical order de-coupled

▫ May be aligned for optimization


▫ May be aligned for optimization▫ Permits composition, editing, re-use etc. without

re-write

• Tables are compacted

Multi-protocol Streaming support

• Two kinds of track

• Media (Elementary Stream) Tracks

▫ Sample is Access Unit

• Protocol ‘hint’ tracks

▫ Sample tells server how to build protocol transmission unit (packet, protocol data unit etc.)


transmission unit (packet, protocol data unit etc.)

Track types

• Visual—’description’ formats

▫ MPEG4

▫ JPEG2000

• Audio—’description’ formats

▫ MPEG4 compressed tracks

▫ ‘Raw’ (DV) audio


▫ ‘Raw’ (DV) audio

• Other MPEG-4 tracks

• Hint Tracks (streaming)

Track Structure

• Sample pointers (time, position)

• Sample description(s)

• Track references

▫ Dependencies, hint-media links

• Edit lists

▫ Re-use, time-shifting, ‘silent’ intervals etc.


▫ Re-use, time-shifting, ‘silent’ intervals etc.

Hint Tracks

• May include media (ES) data by ref.

• Only ‘extra’ protocol headers etc. added to hint tracks — compact

▫ Make SL, RTP headers as needed

• May multiplex data from several tracks

• Packetization/fragmentation/multiplex through


• Packetization/fragmentation/multiplex through hint structures

• Timing is derived from media timing

Hint track structure

moov

Hint track

trak

Video track

Movie (meta-data)


frame

sample

header

pointer

hint

sample

frame

sample

header

pointer

hint

sample

mdat

trak

Hint track

Sample Data

Extensibility

• Other media types.

▫ Non-sc29 sample descriptions (e.G. Other video).

▫ Non-sc29 track types (e.G. Laboratory instrument trace).

• Copyright notice (file or track level) etc.

• General object extensions (GUIDs).


• General object extensions (GUIDs).

Advantages

• Compatibility

▫ files can be played by other companies players.

� Real Player with envivo plug-in.

� Windows media player etc.

▫ Files can be streamed by other companies streaming server


streaming server

� Darwin Streaming Server.

� Quick Time Streaming Server.

Single File-Multiple data types

• No need to do an export process for files, one file type is used for storage of video, audio, events, continues telemetry data from sensors and JPEG images in one file.


AudioAudio

VideoVideo

JPEGJPEG11

Sensor Continues dataSensor Continues data

eventsevents

Métadonnées

Métadonnées

Single file playback

• All video track of a site could be stored in one file. In order to view many cameras in a synchronized manner the MPEG-4 file format can hold all the views of multiple cameras in one file.


AudioAudio

Video cam 1Video cam 1

Video cam 2Video cam 2

Video cam …….Video cam …….

Video cam NVideo cam N

Métadonnées

Métadonnées

Skimming

• Skimming – shortening a long movie to its interesting points, much like creating a “promo”. For example skimming a surveillance movie of two hours to 2 minutes where there is movement and people are entering the building.

• MPEG-4 FF enables the creation of skims within the file through the use of edit-list (part of the


the file through the use of edit-list (part of the standard) without overhead.

MKV FILE FORMAT


XML Based File-Format

MKV - File Format

• Container file format for videos, audio tracks, pictures and subtitles all in one file.

• Announced on Dec. 2002 by Steve Lhomme.

• Based on Binary XML format called EBML (Extensible Binary Meta Language)


(Extensible Binary Meta Language)

• Complete Open-Standard format. (Free for personal use).

• Source is licensed under GNU L-GPL.

MKV - Specifications

• Can contain chapter entries of video streams

• Allows fast in-file seeking.

• Metadata tags are fully supported.

• Multiple streams container in a single file.


• Multiple streams container in a single file.

• Modular – Can be expanded to company special needs.

• Can be streamed over HTTP, FTP, etc.

MKV Support software & hardware

• Players:▫ All Player, BS.Player, DivX Player, Gstreamer-Based

players, VLC media, xine, Zoom Player, Mplayer, Media Player Classic, ShowTime, Media Player Classic and many more…

• Media Centers:


• Media Centers:▫ Boxee, DivX connected, Media Portal, PS3 Media

Server, Moovida, XBMC etc.

• Blu-Ray Players:▫ Samsung, LG and Oppo.

• Mobile Players:

▫ Archos 5 android device, Cowon A3 and O2.

MKV - EBML in details

• A binary format for representing data in XML-like format.

• Using specific XML tags to define stream properties and data.

• MKV conforms to the rules of EBML by defining a set of tags.


a set of tags.

▫ Segment , Info, Seek, Block, Slices etc.

• Uses 3 Lacing mechanisms for shortening small data block (usually frames).

▫ Uses: Xiph, EBML or fixed-sized lacing.

MKV – Simple representationDescriptionType

Version info, EBML type ( matroska in our case ).Header

Optional, Allows fast seeking of other level 1 elements in file.Meta Seek Information

File information - title, unique file ID, part number, next file ID.

Segment Information

Basic information about the track – resolution, sample rate, codec info.

Track


codec info.

Predefines seek point in media.Chapters

Video and audio frames for each trackClusters

Stores cue points for each track. Allows fast in track seeking. Cueing Data

Any other file relates to this. ( subtitles, Album covers, etc… )Attachment

Tags that relates to the file and for each track (similar to MP3 ID3 tags).

Tagging

MKV – Streaming

• Matroska supports two types of streaming.

• File Access

▫ Used for reading file locally or from remote web server.

▫ Prone to reading and seeking errors.

▫ Causes buffering issues on slow servers.


▫ Causes buffering issues on slow servers.

• Live Streaming

▫ Usually over HTTP or other TCP based protocol.

▫ Special streaming structure – no Meta seek, Cues, Chapters or attachments are allowed.

File Format Summary - Trends

• Metadata is important

▫ Simple metadata or XML

▫ Separated from media

• Forward compatibility

▫ Not crash if don’t understand a data entry

• Progressive download oriented


• Progressive download oriented

• Multi-bitrate oriented

• Fragmentation -> Lower granularity

▫ Self contained File fragments

• CDN-ability

Video Codecs

www.dsp-ip.comFast Forward Your Developmentframe

samplesample

frame

sample

mdat

trak

moovAudio track

trak

Video track

Movie (meta-data)

Media Data

sample

Why Advance ? MPEG2 Works….• Coding efficiency

• Packetization

• Robustness

• Scalable profiles

• Internet requires Interaction

▫ Scalable & On demand


▫ Scalable & On demand

▫ Fast-Forward / Fast Rewind / Random Access

▫ Stream switching

• Multi

▫ Bitrate

▫ resolution /screen

Coding efficiency Motivation


Sorenson Spark video Codec

• H263 variant

• Low footprint (code size) ~100K

• Good performance for 2002

• Quality SPARK vs Optimal MPEG (H263+)

▫ 20-30% less efficient


• SPARK Quality RT vs Offline

▫ RT has Considerably lower quality due to processing power and RT (delay) constraints

Sorenson Spark - 2

• Does Not support:▫ Arithmetic coding▫ Advance prediction▫ B-frames

• Features▫ De-blocking filter mode


▫ De-blocking filter mode▫ UMV - Unrestricted Motion Vector mode ▫ Arbitrary frame dimensions▫ Supported by FFMPEG ▫ D – Frames

D-Frames

• D (Disposable) frames

▫ One way prediction

▫ Provides flexible bit-rate: I-D-P-D-P-D-P

▫ D-frames used only when feeding a flash communication server


On2 TrueMotion VP6

• Features

▫ Compressed I-frames (Intra-compression makes use of spatial predictors)

▫ unidirectional predicted frames (P-frames)

▫ Multiple reference P-frames

▫ 8x8 iDCT-class transform (4x4 in VP7)


▫ 8x8 iDCT-class transform (4x4 in VP7)

▫ improved quantization strategy (preserves image details)

▫ Advance Entropy Coding

VP6 Features

• Entropy Coding ▫ various techniques are used based on complexity

and frame size including:� VLC � Context modeled binary coding (like H264 CABAC)

• Bit Rate Control


▫ To reach the requested data rate, VP6 adjusts� Quantization levels� Encoded frame dimensions� Entropy Coding� Drop frames

VP6 motion prediction

• Motion Vectors▫ One vector per MacroBlock (16x16)

or ▫ 4 vectors for each block (8x8)

• Quarter pel motion compensation support• Unrestricted motion compensation support


• Unrestricted motion compensation support• Two reference frames:

▫ The previous frameor

▫ Previously bookmarked frame

VP6 vs H264

• VP6 is much simpler than H.264

▫ Requires less CPU resourced for decoding & encoding

▫ Code size is considerably smaller.

• Simpler means less efficient? NO! Techniques


• Simpler means less efficient? NO! Techniques used:

▫ Mix of adaptive sub-pixel motion estimation

▫ Better prediction of low-order frequency coefficients

▫ Improved quantization strategy

▫ de-blocking and de-ringing filters

▫ Enhanced context based entropy coding,

720p High Profile H.264 vs VP7

45

45.5

46

46.5

47

Alexander Trailer

Tips for reading this kind of a graph (a PSNR graph):

Draw a line straight

Draw a line straight across until you intersect the lower line ( in this case x264. i.e. keep the quality/ psnr constant )

Draw a line straight

What this means: On this clip VP7 at 2750 kbps has the same quality / PSNR as x264 high profile at 3620 kbps. i.e. you’d need 30% higher

PSNR Graphs are used for comparative analysis of compression quality. Each line represents the encode quality on a given clip at multiple datarates. The highest line represents the codec with the best quality. In this case VP7 clearly is better than x264.

Th

is a

xis

re

pre

sen

ts q

ua

lity

. H

igh

er

is b

ett

er

Pick any point on the top line, in this case it’s VP7.


42.5

43

43.5

44

44.5

45

1400 1900 2400 2900 3400 3900 4400

PSNR

Kbps

Vp7

x264

2750 kbps 3620 kbps

Draw a line straight down from that point to the datarate axis. The crossing point tells you the datarate at that point.

Draw a line straight down from that point to the datarate axis. The crossing point tells you the datarate at that point.

at 3620 kbps. i.e. you’d need 30% higher datarate to get the same quality out of x264 that you got from vP7.

Th

is a

xis

re

pre

sen

ts q

ua

lity

. H

igh

er

is b

ett

er

This axis represents datarate in kilobits per second.

VP6 vs. H264• There is a difference between the codec

technology and a codec implementation.


On2 VP7

• Not open source

• Non-standard royalties model

• Better video quality than H264

• Used by:

▫ Part of EVD – China standard for HD-DVD

▫ Skype Beta (V 2.0)


▫ Skype Beta (V 2.0)

▫ Flash Player

Windows Media

• Windows media is a format used by Microsoft for encoding and distributing Audio and Video.

• Windows Media has two types of media:▫ Windows Media Audio (WMA) ▫ Windows Media video (WMV)

• Windows Media Video


• Windows Media Video ▫ A modified version of MPEG 4▫ Codec version has initially started from version 7

for windows media player 7 and then evolved to version 8-10

Windows Media 9 - VC1 Format

• Microsoft has submitted Version 9 codec to the Society of Motion Picture and Television Engineers (SMPTE), for approval as an international standard. SMPTE is reviewing the submission under the draft-name "VC-1")

• This codec is also used to distribute high definition video on standard DVDs in a format Microsoft has branded as


on standard DVDs in a format Microsoft has branded as WMV HD. This WMV HD content can be played back on computers or compatible DVD players.

• The Trial version of standards were published by SMPTE in September 2005

• WMV9 was approved by SMPTE, April 2006

H.264


H.264 Terminology

• The following terms are used interchangeably:▫ H.26L

▫ “JVT CODEC”

▫ The “AVC” or Advanced Video CODE

• Proper Terminology going forward:


▫ MPEG-4 Part 10 (Official MPEG Term)

� ISO/IEC 14496-10 AVC

▫ H.264 (Official ITU Term)

H264 Standard ideas

• “Blocks” size fixed ->Variable

▫ Slice

▫ Block

• Block Size order/scanning –> different orders

▫ Zig-zag, Flexible Macroblock Order

• Additional spatial prediction - >Intra prediction


• Additional spatial prediction - >Intra prediction

• Inter prediction 1 frame only ->Multiple frames

▫ P and B picture

▫ Multiple reference frame

H264 Standard Ideas

▫ Pixel interpolation

▫ Motion vectors

• In-loop Deblocking filter

• Improved Entropy coding


• Improved Entropy coding

New Features of H.264 - summarized

• SP, SI - Additional picture types

• NAL (Network Abstraction Layer)

• CABAC - Additional entropy coding mode

• ¼ & 1/8-pixel motion vector precision

• In-loop de-blocking filter

• B-frame prediction weighting


• B-frame prediction weighting

• 4×4 integer transform

• Multi-mode intra-prediction

• NAL - Coding and transport layers separation

• FMO - Flexible MacroBlock ordering.

Block diagram


Profiles and Levels

• Profiles: Baseline, Main, and X▫ Baseline: Progressive, Videoconferencing & Wireless

▫ Main: esp. Broadcast

▫ Extended: Mobile network

• Wireless <> Mobile


Baseline Profile

• Baseline profile is the minimum

implementation▫ No CABAC, 1/8 MC, B-frame, SP-slices

• 15 levels▫ Resolution, capability, bit rate, buffer, reference #

▫ Built to match popular international production and


▫ Built to match popular international production and

emission formats

▫ From QCIF to D-Cinema

• Progressive (not interlaced)• I and P slices types

Baseline Profile

• 1/4-sample Inter prediction • Deblocking filter, Redundant slices • VLC-based entropy coding (no CABAC)• 4:2:0 chroma format • Flexible Macroblock Ordering (FMO) • Arbitrary Slice Order (ASO)


• Arbitrary Slice Order (ASO)▫ Decoder process slices in an arbitrary order as they

arrive to the decoder.▫ The decoder dose not have a wait for all slices to

be properly arranged before it starts processing them.▫ Reduces the processing delay at the decoder.

Baseline Profile

• FMO: Flexible Macroblock Ordering ▫ With FMO, macroblocks are coded according to a

macroblock allocation map that groups, within a given slice.

▫ Macroblocks from spatially different locationsin the frame.


in the frame.▫ Enhances error resilience

• Redundant slices:▫ allow the transmission of duplicate slices.

H.264 Profiles & Levels - Main

• All Baseline features Plus� Interlace � B slice types (bi directional reference )� CABAC� Weighted prediction

• All features included in the Baseline profile


• All features included in the Baseline profile except:� Arbitrary Slice Order (ASO)� Flexible Macroblock Order (FMO)� Redundant Slices

Main Profile

• CABAC

• Good performance (bit rate reduction) by

▫ Selecting models by context

▫ Adapting estimates by local statistics

▫ Arithmetic coding reduces computational complexity

• Improve computational complexity more than


• Improve computational complexity more than10%~20% of the total decoder execution time at medium bitrate

• Average bit-rate saving over CAVLC 10-15%

Extended Profile

• All Baseline features plus

▫ Interlace

▫ B slice types

▫ Weighted prediction


Variable block size

Slices�A picture split into 1 or several slices

�Slices are a sequence of macroblocks

Macroblock�Contains 16x16 luminance samples and


Slice 0

Slice 1

Slice 2

�Contains 16x16 luminance samples and

two 8x8 chrominance samples

�Macroblocks within a slices depend on

each others

�Macroblocks can be further partitioned

Basic Marcoblock Coding Structure

Entropy

Coding

Scaling & Inv.

Transform

Control

Data

Quant.

Transf. coeffs

Coder

Control

Decoder

Transform/

Scal./Quant.-

Input

Video

Signal

Split into

Macroblocks

16x16 pixels


Motion-

Compensation

Motion

Data

Intra/Inter

Motion

Estimation

Intra-frame

Prediction

De-blocking

Filter

Output

Video

Signal

Motion Compensation

Entropy

Scaling & Inv.

Transform

Control

Data

Quant.

Transf. coeffs

Coder

Control

Decoder

Transform/

Scal./Quant.-

Input

Video

Signal

Split into

Macroblocks

16x16 pixels


Entropy

Coding

Motion-

Compensation

Motion

Data

Intra/Inter

Motion

Estimation

16x16 pixels

Intra-frame

Prediction

De-blocking

Filter

Output

Video

Signal

Various block sizes and shapes

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8

Types

0

16x16

0 1

8x16

MBTypes

8x8

0 1

2 3

16x8

1

0

Variable block size

0

0

1

0 1

2 30 1

Mode 1 Mode 2 Mode 3 Mode 4

�Block sizes of

16x8, 8x16, 8x8,

8x4 , 4X8 and

4X4 are

available.

www.dsp-ip.comFast Forward Your DevelopmentYossi Cohen

DSP-IP

0 1 2 3

4 5 6 7

0 1 2 3

4 5 6 7

8 9 1

0

1

1

1

2

1

3

1

4

1

5

0 1

2 3

4 5

6 7

Mode 11 16x16 block

Mode 22 16x8 blocks

Mode 32 8x16 blocks

Mode 44 8x8 blocks

Mode 58 8x4 blocks

Mode 68 4x8 blocks

Mode 716 4x4 blocks

�Using seven different

block sizes can translate

into bit rate savings of

more than 15% as

compared to using only a

16x16 block size.

How to select the partition size?


The partition size that minimizes the coded

residual and motion vectors

Inter prediction modesmotion vectors

• MVs for neighboring partitions are often highly correlated.

• So we encode MVDs instead of MVs

• MVD = predicted MV – MVp

• ¼ pixel accurate motion compensation


Multiple Reference Frames

Entropy

Coding

Scaling & Inv.

Transform

Control

Data

Quant.

Transf. coeffs

Coder

Control

Decoder

Transform/

Scal./Quant.-

Input

Video

Signal

Split into

Macroblocks

16x16 pixels


Coding

Motion-

Compensation

Motion

Data

Intra/Inter

Motion

Estimation

Intra-frame

Prediction

De-blocking

Filter

Output

Video

Signal

Multiple Reference Frames for

Motion Compensation

Multiple Reference Frames


Intra prediction modes4x4 luminance prediction modes

0(vertical) 1(Horizontal) 2(DC) 3(Diagonal 4(Diagonal

Down/left) Down/right)

5(Vertical-right) 6(Horizontal-down) 7(Vertical-left) 8(Horizontal-top)


Mode 2 (DC)

Predict all pixels from

(A+B+C+D+I+J+K+L+4)/8 or

(A+B+C+D+2)/4 or (I+J+K+L+2)/4

Intra prediction modes4x4 luminance prediction modes


Intra prediction modesIntra 16x16 luminance and 8x8 chrominance prediction modes


Inter prediction modeschrominance Pixel interpolation

dx

dy

S-dx

Quarter chrominance Pixels are

interpolated by tacking weighted

averages of distance from the new

pixel to four surrounding original

A B


S-dx

S-dy

V=(s-dx)(s-dy)A+dx(s-dy)B+(s-dx)dyC+dxdyD+s2/2

S2

pixel to four surrounding original

pixels.C D

In-loop De-blocking Filter

• Highly compressed decoded inter picture

• Significantly reduces prediction residuals


Without filter with H.264/AVC De-blocking

Entropy coding


Entropy coding

• Entropy coding methods:• CABAC - Discussed• UVLC

▫ H.264 offers a single Universal VLC (UVLC) table for all symbol

• CAVLC


• CAVLC▫ CAVLC (Context-based variable Length Coding

)

▫ Probability distribution is static▫ Code words must have integer number of bits (Low

coding efficiency for highly peaked pdfs)

CABAC - Binarization


Context Probability Coding

update probability estimation

CABAC: Technical OverviewCABAC: Technical Overview


Context

modelingBinarization Probability

estimation

Coding

engine

Adaptive binary arithmetic coder

Chooses a model

conditioned on

past observations

Maps non-binary

symbols to a

binary sequence

Uses the provided model

for the actual encoding

and updates the model

H.264 NAL & RTP


H264 Layer Structure

Video Coding Layer

Data PartitioningContr

ol D

ata

Macroblock


Data Partitioning

Network Abstraction Layer

H.320 H.324 H.323/IP MPEG2

Slice/Partition

H264 & NAL

• Motivation

▫ Many delivery methods are based on packet based networks

▫ Its better to do the packetization inside the encoder where all coding information than in other separate modules

• Architecture: NAL units as the transport entity


• Architecture: NAL units as the transport entity

▫ NAL units may be mapped into a bit stream▫ NAL units are self-contained-independently

decodable▫ The decoding process assumes NAL units are in

decoding order

Network Abstraction Layer (NAL)

• H.264 encoder is composed of two layers:

• VCL - Video Coding Layer – unit which translates the video information into a stream of bits

• NAL - Network Abstraction Layer (NAL). Which maps and packetize the VCL biststream into unitsprior to transmission or storage

• Each NAL unit contains:


• Each NAL unit contains:

▫ Payload – RBSP (Raw Byte Sequence Payload), which contains set of data corresponding to coded video data or header information

▫ NAL Unit Header – which contains the NAL header

NAL

• The coded video sequence is represented by a sequence of NAL units that can be transmitted over a packet-based network or a bitstream transmission link or stored in a file

• There are two NAL Types

▫ VCL Units – NAL units which represents encoded


▫ VCL Units – NAL units which represents encoded video data

▫ Non-VCL Units – Parameters sets

NAL Unit Header

NAL unit header- 1 byte consisting of:• Forbidden_bit(1 bit) may be used to signal that a NAL unit is

corrupt

• nal_storage_idc(2 bit): signal relative importance, and if the picture is stored in the reference picture buffer.

NAL unit header NAL unit payload


picture is stored in the reference picture buffer.

• nal_unit_type(5 bit): signal 1 of 10 different NAL unit types:▫ Coded slice (regular VCL data)

▫ Coded data partition. (DPA,DPB,DPC)

▫ Instantaneous decoder refresh (IDR)

▫ Supplemental enhancement information (SEI)

▫ Sequence and picture parameter sets (SPS,PPS)

▫ Picture delimiter (PD) and filler data (FD)

RBSP (NAL Payload) Types• Parameter Set – global parameters for a sequence

includes: Resolution, video format, macroblock, allocation map

• Supplemental Enhancement Information

• Picture Delimiter – boundary between video pictures

• Coded slice – header and data for a slice, this unit contains actual coded video data


contains actual coded video data

• Data Partition A,B or C – Data Partitioned slice layer data (A – header data for all MBs in the slice, B – intra coded data, C – inter coded data)

• End of sequence

• End of stream

• Filler data

RTP payload format for H.264

• Based on IETF RFC 3984, February 2005

• Described how to use H.264 NAL inside RTP with proper packetization Employs the native NAL (Network Abstraction Layer) interface, based on NAL units (NALUs)

• NALU – byte string of variable length that contains syntax elements of a certain class

• NALU header – defines the information within the NAL


• NALU header – defines the information within the NAL Unit (Corrupted, Type etc)

• There are two basic methods for RTP packetization of NAL units:

▫ Non fragmented NAL units

▫ Fragmented NAL units

RTP Payload for H.264 NAL

• The most common method is to configure the encoder to output one NAL unit for each RTP packet. Each NAL unit is ~1.4KB


• Fragment a large NAL unit (Frame) into many RTP Packets. The difference is in the RTP Header information

RTP and H.264

• RTP Packetization of NAL allows both aggregation of many NAL units into one RTP Packet and fragmentation of one NAL units into many RTP packets


Comparison


Summary• New key features are:

▫ Enhanced motion compensation

▫ Small blocks for transform coding

▫ Improved de-blocking filter

▫ Enhanced entropy coding

• Substantial bit-rate savings (up to 50%) relative to

other standards for the same quality


other standards for the same quality

• The complexity of the encoder triples that of the

prior ones

• The complexity of the decoder doubles that of the

prior ones

Google VP8


Before we start

• VP8 goal is NOT to delivery the best video quality in any given bitrate

• VP8 was designed as a mobile video decoder and should be examined in this context:

▫ VP8 vs H.264 base profile


Google VP8

• Last month, in Google IO (its developer confrence), Google released VP8 as open source

• VP8 is a light weight video codec developed by On2.

• VP8 provide quality which is the same/higher than H.264 base profile


than H.264 base profile

• VP8 memory requirements are lower than H.264 base profile

• After optimization, VP8 might have better MIPS performance than H.264 base profile

Genealogy • VP8 is part of a well know codec family

• VP3 was released to open source to become XIPH Theora

• VP6 is used in Flash video

• VP7 is used in Skype

Sorenson

Spark


• Motivation:

▫ “No Royalties” CODEC

VP8

VP7 VP6

VP3Theora

ADAPTATION – WHO USE IT?

Software

Hardware


Hardware

Platform & Publishers

Software Adaptation

• Android, Anystream, Collabora

• Corecodec, Firefox, Adobe Flash

• Google Chrome, iLinc,

• Inlet, Opera, ooVoo

• Skype, Sorenson Media

• Theora.org, Telestream, Wildform.


• Theora.org, Telestream, Wildform.

Hardware adaptation

• AMD, ARM, Broadcom

• Digital Rapids, Freescale

• Harmonic ,Logitech, ViewCast

• Imagination Technologies, Marvell

• NVIDIA, Qualcomm, Texas Instruments

• VeriSilicon, MIPS


• VeriSilicon, MIPS

Platforms and Publishers

• Brightcove

• Encoding.com

• HD Cloud

• Kaltura

• Ooyala

• YouTube


• YouTube

• Zencoder

VP8 MAIN FEATURES


(According to On2/Google)

Adaptive Loop Filter

• Improved Loop filter provides better quality & performance in comparison to H.264

• ALF is a feature of H.265


Source: On2

Golden Frames

• Golden frames enables better decoding of background which is used for prediction in later frames

• Could be used as resync-point:

▫ Golden frame can reference an I frame

• Could be hidden (not for display)


• Could be hidden (not for display)

Source: On2

Decoding efficiency

• CABAC is an H.264 feature which improves coding efficiency but consumes many CPU cycles

• VP8 has better entropy coding than H.264, this leads to relatively lower CPU consumption under the same conditions

• Decoding efficiency is


• Decoding efficiency is important for smooth operation and long battery life in netbooks and mobile devices

Source: On2

Resolution up-scaling & downscaling

• Supported by the decoder

• Encoder could decide dynamically (RT applications) to lower resolution in case of low bit rate and let the decoder scale.

• Remove decision from the application

• No need for an I frame


• No need for an I frame

VP8 BASICS

Definitions

Bitstream structure


Bitstream structure

Frame structure

Definitions

• Frame – same as H.264

• Segment – Parallel to slice in H.264. MB in the same segment will use the settings such as:

▫ Entropy encoder/decoder context

▫ De-blocking filter settings

• Partition – block of byte aligned compressed


• Partition – block of byte aligned compressed video bits.

Definitions

• Block – 8x8 matrix of pixels

• Macro-block –processing unit, contains a 16x16 Y pixels, and 2 8x8 matrix of U and V:

▫ 4* 8x8Y block

▫ 1* 8x8U block

▫ 1* 8x8V block


▫ 1* 8x8V block

• Sub-block – 4x4 matrix of pixels. All DCT / WHT operations are done on sub-blocks.

Frame Types

• I Frame

• P Frame

• No B Frames due to patents / delays but we have “Future Alt-Ref” frame. What is the diff?

• Prediction

▫ Previous frame


▫ Previous frame

▫ Golden Frame

▫ Alt-ref frame

Frame Structure

• Include three sections:

• Frame Header

• Partition I

• Partition II


Frame

Header Partition I Partition II

partitions

Frame Header

• Byte aligned uncompressed information

• Frame type - 1-bit frame type

▫ 0 for key frames, 1 for inter-frame.

• Level - A 3-bit version number

▫ 0 - 3 are defined as four different profiles with different decoding complexity; other values for future use


• show_frame - A 1-bit show_frame flag

▫ 0 – current frame not for display

▫ 1 - current frame is for display

• Length - A 19-bit field containing the size of the first data partition in bytes.

Partition I

• Header information for the entire frame

• Per-macroblock information specifying how each macroblock is predicted.

• This information is presented in raster-scan order


Partition II

• Texture information - DCT/WHT quantized coefficients

• Optionally each macroblock row could be mapped to a separate partition.

• Partition II might be divided to several partitions for parallel processing


partitions for parallel processing

Frame

Header Partition I

Texture Data

Partition IIA Partition IIB Partition IIn

Decoder

• Holds 4 frames:

▫ Current remonstrated frame

▫ Previous frame

▫ Previous “Golden Frame”

▫ Previous Alt-ref frame

• Frame dimension can change in every frame


• Frame dimension can change in every frame

VP8 block diagram

Entropy

Coding

Scaling & Inv.

Transform

Control

Data

Quant.

Transf. coeffs

Coder

Control

Decoder

Transform/

Scal./Quant.-

Input

Video

Signal

Split into

Macroblocks


Motion-

Compensation

Motion

Data

Intra/Inter

Motion

Estimation

Intra-frame

Prediction

Dynamic

De-blocking

Output

Video

VP8 BLOCK CODING


VP8 Macroblock coding

Divide to

16x16

Macroblock

Divide to

8x8

blocks

Process as

4x4

sub blocks 4x4

WHT

4x4

DCT

Each Macroblock is divided into 25 sub-blocks

DC/AC Coeff


Each Macroblock is divided into 25 sub-blocks•16Y sub-blocks•4 U sub-blocks, •4 V sub-blocks•1 Y2 DC values sub-block (WHT)aka Hierarchical transform

VP8 Transforms

• Very inefficient raw implementation of the transform – uses 16bit multipliers

• Uses exact values of pixels

▫ +Memory

▫ +Accuracy and no drift


static const int cospi8sqrt2minus1 = 20091; //sqrt(2) * cos(pi/8)

static const int sinpi8sqrt2 = 35468; //sqrt(2) * sin (pi/8)

temp1 = (ip[4] * sinpi8sqrt2 + rounding) >> 16;

iDCT

• Like in H.264 or VP6 the 2D transform is done by two separable 1D transforms (LLM method) there are two methods of implementation:

1. Transpose, vertical T, Transpose, vertical T.

2. Vertical T., transpose, vertical T, transpose.

• Duo to SIMD command its better to do it as 1 and


• Duo to SIMD command its better to do it as 1 and eliminate the first transpose like H.264 does

• VP8 does it as 2 (Why?!) wasted 1-2% CPU

Quantization

• There are 6 quantizer types each has its own levels

• The quantizer depends on (multiplication of)

▫ Plane: Y,U, V

▫ Coefficient AC, DC

• Quantizer level is indicated by a 7 digit number which is an entry into one of the 6 quantization


which is an entry into one of the 6 quantization levels

VP8 PREDICTION

Inter-prediction


Inter-prediction

Intra prediction

Macroblock Intra Prediction

• Intra-prediction exploits the spatial coherence between Macro-blocks without referring to other frames.

• Modes

▫ Same as H.264 in i16x16 and i4x4

▫ Missing modes like i8x8 which exists in H.264


▫ Missing modes like i8x8 which exists in H.264

Intra prediction - blocks used

Not Relevant

MNot Available Not Available


M

Not Available Not Available Not Available Not Available

Luma i4x4 Intra Prediction

• 4x4 block are predicated by

▫ Orig four 16x16 prediction methods

▫ six “diagonal” prediction methodsDiagonal Down/left Diagonal Down/right


Diagonal Down/left

Vertical-right Horizontal-down Vertical-left Horizontal-top

Diagonal Down/right

8x8 Chroma prediction modes• U,V, Y prediction are done separately and one

channel prediction does not affect the other channels.


Inter-frame prediction

• Definition - Inter-prediction exploits the temporal coherence between frames to save bitrate.

• Luma sub-block prediction

▫ Method - each Y 4x4 sub-blocks is related to a 4x4 sub-block of the prediction frame.


sub-block of the prediction frame.

▫ Precision – motion vectors precision is q-pel.

▫ interpolation pixel is calculated by applying a kernel filter three pixels horizontally and vertically.

▫ Based on deltas from previous MVs “NEARST” and NEAR modes

Inter-frame prediction

• SPLITMV mode

▫ Supports up to 16 MV inside a single MB

▫ MV reuse (delat) is done not only on MB level but also on sub-block level

• Example: Partition of Block into 3 MV areas


16

3

Inter-frame prediction - Chroma• Chroma prediction - motion vector for each

8X8 chroma block is calculated separately by one of four prediction methods listed below:

1. Vertical - Copying the row from above throughout the prediction buffer.

2. Horizontal - Copying the column from left


2. Horizontal - Copying the column from left throughout the prediction buffer.

3. DC - Copying the average value of the row and column throughout the prediction buffer.

4. Extrapolation from the row and column using the (fixed) second difference (horizontal and vertical) from the upper left corner.

Inter-frame Prediction - Chroma

• Chroma precision - the calculated chroma motion vectors have 1/8 pixel resolution

• averaging the vectors of the four Y sub-blocks that occupy the same area of the frame.


Entropy Coding

• Entropy coding is based on binary tree like CABAC but unlike H.264 its not context adaptive like in H.264 and does not adapt on every operation

• Tables does not change all the time

• Unlike H.264 which makes all Symbol->Bit


• Unlike H.264 which makes all Symbol->Bit decision as text, VP8 has a tree which represent the transform of each symbol -> faster implementation (not faster processing time)

16

6

Entropy Coding

• According to X264Developer “probability values are constant over the course of the frame”

• According to Google document, there could be up to 4 “Entropy coding context” in a frame

• We can view this as “switched context BAC” and not as CABAC


not as CABAC

• Obviously “switched context BAC”<<CABAC on a CPU level

• However setting the context for each fragment is hard to implement in hardware

16

7

PARALLEL PROCESSING

Segment


Segment

Partition

Segment Processing

• Segmentation enables creation of MB groups within one logical unit.

• MB are associated with a segment by the MB Segment ID

• All MBs in a segment has the same adaptive adjustments which includes:


adjustments which includes:

▫ Same Quantization level

▫ Loop filter strength (0-2)

Frame Processing Architecture

• Frame Header and Partition I are processed first to initialize probabilistic decoder and prediction scheme for each MB. A Serial operation

• Each sub-partition might be processed in parallel to other partitions. probabilistic model of one sub-partition does not interact with another sub-


sub-partition does not interact with another sub-partition

Frame

HeaderPartition I

Sub-partition

Partition

IIALength

IIA-IIn-1

Partition

IIB

Partition

IIn

VP VS H.264 DIFFRENCES


H.264 Loop Filter

• H.264 Loop filter strength depends on boundary strength, MB type


17

2

VP8 Loop Filter

• Lop filter is adjusted by 6 bit variable on two levels:

▫ Global Frame Loop filter level

▫ Per MB level

• VP8 supports two filter types.

• Per MB level is set as “delta” from frame filter level

• L1 – enable/Disable Loop filter in code


• L1 – enable/Disable Loop filter in code mode_ref_lf_delta_update

17

3

VP8 Loop filter complexity

• Before final optimization 70% of CPU time is

deblocking filter!!! According to

http://x264dev.multimedia.cx/?p=486

• a 4×4 transform requires a total of 8 length-16 and 8 length-8 loopfilter calls per macroblock while theora 8x8 filter requires half the amount


while theora 8x8 filter requires half the amount of calls

17

4

AltRef Noise Reduction

• One of the usages of Alt-Ref frame is for noise reduction

• For noisy sources using a filtered image as reference improves compression (so is pre-processing)

• Improvement 0.25DB for noisy clip


• Improvement 0.25DB for noisy clip

• Example with noise reduction set to 5

17

5

AltRef Frame Future Frame

• Just don’t call it a B-Frame and it will give you 1db


17

6

H.264 vs VP8 Transforms differences

• No 8x8 transforms

• H.264 simplifies the DCT transform and implement it as a series of:

1. Add

2. Subtract

3. Right shift by 1


3. Right shift by 1

It sacrifice accuracy (~1%) for speed/CPU

VP8 uses large,16bit multipliers for accuracy (20091, 35468), this is redundant unlike VC which uses small multiplier

17

7

H.264 vs VP8 Transforms differences

• Unlike H.264 which de-corellates DC values (Hadamard transform) ONLY in Intra i116x16 MB, VP8 uses this method also on some p16x16 MBs


17

8

VP8 / H.264 summary

• “Golden Frames” – exist at some level in H.264 slice group map type 2

• Slice granularity is better in H.264

▫ support MB instead of MB line

• Interlacing – not supported


• AltRef Frame – a reference frame which is not displayed. Could aggregate all sorts of useful MBs

• Filter – VP has an adaptive complex and slow filter

17

9

Install

• Visual Studio 2010 version – I couldn’t make it work

• Download Visual Studio 2005/2008 code version

• Download and Install YASM

• Follow instructions on


• Follow instructions on http://www.tortall.net/projects/yasm/wiki/VisualStudio2005 to integrate with Visual Studio 2005/2008 (copy files and set paths)

18

0

Basic Decoder work

• Compile IVFDEC Project

• Set parameters

▫ --codec=vp8 -o a.i420 qcif1.ivf

• Compile IVFEnc Project

▫ --codec=vp8 --i420 -v a.i420 qcif1.ivf

• Check that the decoder unders


• Check that the decoder unders

18

1

COMPARISON (FINALLY)


Talking heads, Low motion

• Low motion videos like talking heads are easy to compress, so you'll see no real difference


Low motion

In another low motion video with a terrible background for encoding (finely detailed wallpaper), the VP8 video retains much more detail than H.264. Interesting result.


Medium motion

VP8 holds up fairly well


High motion• In high motion videos, H.264 seems superior. In this

sample, blocks are visible in the pita where the H.264 video is smooth. The pin-striped shirt in the right background is also sharper in the H.264 video, as is the striped shirt on the left.


Very High motion

In this very high motion skateboard video, H.264 also looks clearer, particularly in the highlighted areas in the fence, where the VP8 video has artifacts.


Final

In the final comparison, I'd give a slight edge to VP8, which was clearer and showed fewer artifacts.


Quality Comparison


Test yourself

1. Why VP8 is less effective in high motion?

2. Is it patent free?

3. Will you use it?


Adobe Flash Video

www.dsp-ip.comFast Forward Your Development www.dsp-ip.com

Flash File Format

Flash Movie / SWF

� File type: .swf

� Includes:

� Graphics, text, video

Web Page

� File type: .htm, .html, .asp, etc>

Flash video / FLV

� File type: .flv

� Includes:

�Video

� Plays in:

.swf


� Graphics, text, video

� Video controls

� Client logic

� Plays in:

� Flash Player

� Served from:

� Web server

� Synonyms:

� Application

� Video player

� .swf

� Served from:

�Web server

�Flash Communication Server

FLV (Flash Video) File Format

• Headers in the beginning of the file. Why?

• FLV and MP4 File Formats

• Video support:

▫ SPARK

▫ TrueMotion VP6


▫ H264 and VP8 (future support)

• Audio

▫ Nelly Moser codec by ASAO

▫ MP3 Codec

▫ ADPCM (not compressed)

• Alpha Channel

SWF File

• Vector Graphic Format

• Container file

• Includes FLV files

• Action Scripts

• Players


• Players

SWF and Personalized Ads

• Encoding a special video version for each user is:

▫ Expensive

▫ Degrades the video

• SWF Enables:

▫ Using one video version stored in FLV

▫ Changing the text or image stored in the outer SWF


▫ Changing the text or image stored in the outer SWF per user

Video & Audio

FLV File

Images, Scripts, special Effects

Personalized Video

HTTP Live Streamingby

Apple


Agenda

• System Overview

• Components

• Session


Apple’s Note

Note: Many existing streaming services require specialized servers to distribute content to end users. It requires specialized skills to set up and maintain these servers, and in a large-scale deployment these servers can be costly. Apple


deployment these servers can be costly. Apple has designed a system that avoids this by using standard HTTP to deliver the streams.

System Overview


Components Review

• Server

▫ Encoder

▫ Segmenter

• Distributer

▫ Basic HTTP Server

• Client


• Client

Server

• Receives Digital / Analog input stream

• Encodes / Transcode video/audio

▫ H.264 Video

▫ AAC audio (HE-AAC or AAC-LC)

• Encodes / Transcode audio only:

▫ MPEG-2 elementary streams, HE-AAC or AAC-LC


▫ MPEG-2 elementary streams, HE-AAC or AAC-LC files, or MP3 files

• Encapsulate in MPEG2

▫ Transport Stream

▫ Program Stream

Segmenter

• All segments should be with the same duration

• All segments are placed in a separate file

• Creates Index file with references to segment files

• For protection, the Segmenter might encrypt each media segment and create a key file


each media segment and create a key file

Distribution

• Distribution system is a regular HTTP Server

• Could be Apache or small embedded Server


Files

• Segments – stored as *.ts files

• Index files – stored as *.m3u8

• Index file format example:

#EXTM3U#EXT-X-TARGETDURATION:10#EXTINF:10,


#EXTINF:10,http://media.example.com/segment1.ts#EXTINF:10,http://media.example.com/segment2.ts#EXTINF:10,http://media.example.com/segment3.ts#EXT-X-ENDLIST

Session types

• Live Stream Broadcast

▫ Index file is continues updated

▫ Include a moving window of segments around “live” part of the session

▫ Client should continuously refresh the Index file

• VoD Session


• VoD Session

▫ Index file static

▫ Includes ALL the segments of the file

▫ Enables “Seek” operation

Multi-bitrate multi–device support

• Multi-bitrate is enabled via multiple index files

• Index files are pointed by a global index files

• Client can select a stream according to:

▫ Device properties

▫ Available bit rate

• This method is less efficient


• This method is less efficientthan Silverlight Global File

Test yourself

1. What are the two Live streaming file types?

2. What is the role of the Segmenter?

3. On which delivery protocol is the live streaming based?


Silverlight Smooth Streaming


SILVERLIGHT INTRODUCTION


Smooth Streaming

• Microsoft’s implementation of HTTP-based adaptive streaming

• A hybrid media delivery method that acts like streaming but is in fact a series of short progressive downloads


• Leverages existing HTTP caches• Client can seamlessly switch video quality and bit

rate based on perceived network bandwidth and CPU resources

Streaming or Progressive Download?

Traditional Streaming

• Responsive User Experience

• Bandwidth Use• User Tracking

Progressive Download

• Works from a Web Server

• World-wide scale w/HTTP


Challenges

• No cache-ability• Separate,

smaller streaming networks

Challenges

• Limited User Experience

• User tracking• Bandwidth Use

(20% watched)

Smooth Streaming Design

• Smooth Streaming File Format based on MP4 (ISO Base Media File Format)

• Video is encoded and stored on disk as one contiguous MP4 file▫ Separate file for each bit rate

• Each video Group of Pictures (GOP) is stored in a


• Each video Group of Pictures (GOP) is stored in a Movie Fragment box▫ This allows easy fragmentation at key frames

• Contiguous file is virtually split up into chunks when responding to a client request

Content Provider Benefits

• Cheaper to deploy

▫ Can utilize any generic HTTP caches/proxies

▫ Doesn’t require specialized servers at every node

• Better scalability and reach

▫ Reduces “last mile” issues because it can


▫ Reduces “last mile” issues because it can dynamically adapt to inferior network conditions

• Audience can adapt to the content, rather than requiring the content providers to guess which bit rates are most likely to be accessible to their audience

End User Benefits• Fast start-up and seek times

▫ Start-up/seeking can be initiated on the lowest bit rate before moving up to a higher bit rate

• No buffering, no disconnects, no playback stutter

▫ As long as the user meets the minimum


▫ As long as the user meets the minimum bit rate requirement

• Seamless bit rate switching based on network conditions and CPU capabilities.

• A generally consistent, smoothplayback experience

Evolution

• Previous versions of MS streaming divide the file into many chunkc 0001.vid 0002.vid etc

• Problematic in caching, CDNs, CMS etc

• Today all fragments of a file are contained in a single bitstream container. Typically 1 fragment = 1 video GOP.


= 1 video GOP.

SILVERLIGHT FILES


Containers & Configuration files

Format options

• ASF/WMV – native Microsoft Format

• MPEG4 File-Format

• AVI

• OGG


MP4 over ASF file format• MP4 is a lightweight container format with less

overhead than ASF

• MP4 is easier to parse in managed (.NET) code

• MP4 is based on a widely used standard, making 3rd party adoption and support easier

• MP4 has native H.264 video support


• MP4 has native H.264 video support

• MP4 was designed to natively support payload fragmentation within the file

MP4 File format

• MP4 has two format types

▫ Disk Format - for file storage

▫ Wire format - for transport

• Wire format enables easy CDN support and integration


Smooth Streaming File Format


Smooth Streaming Wire Format


File extensions

• Media Files

▫ *.ismv - Audio & Video

▫ *.isma – Audio only

• Manifest Files

▫ *.ism – Server manifest. Describes to the server Relation between tracks, bitrates & files on disk.


Relation between tracks, bitrates & files on disk. Based on SMIL 2.0 XML format specification

▫ *.ismc – Describes to the client the available streams, CODECS used, bitrates encoded, video resolutions, markers, captions. First file delivered to client. It’s the first file delivered to client (“SDP” like).

Directory Structure

Manifest Files

Media file in different bitrates


Manifest files• VC-1, WMA, H.264 and AAC codecs

• Text streams

• Multi-language audio tracks

• Alternate video & audio tracks (i.e. multiple camera angles, director’s commentary, etc.)

• Multiple hardware profiles (i.e. same bitrates


• Multiple hardware profiles (i.e. same bitrates targeted at different playback devices)

• Script commands, markers/chapters, captions

• Client manifest Gzip compression

• URL obfuscation

• Live encoding and streaming

ISM file sample<?xml version="1.0" encoding="utf-16" ?>

- 

- <smil xmlns="http://www.w3.org/2001/SMIL20/Language">

- <head>

<meta name="clientManifestRelativePath" content="NBA.ismc" />

</head>

- <body>

- <switch>

- <video src="NBA_3000000.ismv" systemBitrate="3000000">


<param name="trackID" value="2" valuetype="data" />

</video>



</video>



</video>

ISM file sample- <video src="NBA_1300000.ismv" systemBitrate="1300000">


</video>



</video>



</video>


- <audio src="NBA_3000000.ismv" systemBitrate="64000">


</audio>

</switch>

</body>

</smil>

*.ISMC sample<?xml version="1.0" encoding="utf-16" ?>

- 

- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="4084405506">

- <StreamIndex Type="video" Subtype="WVC1" Chunks="208" Url="QualityLevels({bitrate})/Fragments(video={start time})">

<QualityLevel Bitrate="3000000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3FE27F1678A27F859E80C9082DB8D44A9C00000010E5A67F840" />

<QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592"


<QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3FE20F1278A20F849E80C9082493DEDDCC00000010E5A67F840" />

<QualityLevel Bitrate="1800000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBF81A70EF8A1A783BE80C908236EE5265400000010E5A67F840" />

<QualityLevel Bitrate="1300000" FourCC="WVC1" Width="640" Height="352" CodecPrivateData="250000010FCBE813F0AF8A13F82BE80C9081A7ABF704400000010E5A67F840" />

ISMC File - 2- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">


<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" /> …..

<c n="0" d="20000000" />

<c n="1" d="20000000" />

.....

<c n="298" d="5000001" />

</StreamIndex>


</StreamIndex>

- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">

<QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />

<c n="0" d="20433560" /> ....

<c n="297" d="20433560" />

<c n="298" d="4393197" />

</StreamIndex>

</SmoothStreamingMedia>

SILVERLIGHT SESSION


Initiation and Flow

Smooth Streaming Protocol

• Smooth Streaming Protocol uses HTTP [RFC2616] as its underlying transport .

• The Server role in the protocol is stateless

▫ Enabling (potentially) different instance of the server to handle client requests


server to handle client requests

▫ Request can utilize any generic HTTP caches/proxies - > Lowering CDN costs

Messages

• Smooth Streaming Protocol uses 4 different messages:

▫ Manifest Request

▫ Manifest Response

▫ Fragment Request

▫ Fragment Response


▫ Fragment Response

• All messages follow the HTTP/1.1 specification

Messages Flow

Server Client

Manifest Request

Manifest Response

Fragment Request


Fragment Response

Fragment Request(s)

Messages

• Manifest Request and Fragment Request message MUST use the HTTP "GET" method, generated by the client.

• Manifest Request and Fragment Request message use the HTTP Response messages.


message use the HTTP Response messages.

Status-Code SHOULD be 200.

Smooth Streaming Transport Protocol Session Details

Manifest Request

Manifest ResponseVideo Fragment Request

Fragment ResponseAudio Fragment Request


Session Details - Manifest Request

• In order to initiate a presentation the Client MUST send the server a Manifest Request using the HTTP GET method.


the HTTP GET method.

Session Details - Manifest Response

• The Response is a ISMC Manifest file describing the session. - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">


<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />

…..

<c n="0" d="20000000" />

<c n="1" d="20000000" />


<c n="1" d="20000000" />

.....

<c n="297" d="20000000" />

<c n="298" d="5000001" />

</StreamIndex>



<c n="0" d="20433560" />

....

<c n="297" d="20433560" />

<c n="298" d="4393197" />

</StreamIndex>


Manifest Response reviewed• We can see in the ISMC file that the server can support 8 different levels

of quality (bitrate) for the client can chose from between 2.75Mbit to 0.35 Mbit.

- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">


<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />

<QualityLevel Bitrate="2040000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3BE20F1278A20F849E80450823E414DD1400000010E5AE7F840" />

<QualityLevel Bitrate="1520000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" />


CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" />

<QualityLevel Bitrate="1130000" FourCC="WVC1" Width="704" Height="400" CodecPrivateData="250000010FCBA215F0C78A15F831E8045081A27BD635C00000010E5AE7F840" />

<QualityLevel Bitrate="845000" FourCC="WVC1" Width="576" Height="320" CodecPrivateData="250000010FCB9A11F09F8A11F827E804508199C94077400000010E5AE7F840" />

<QualityLevel Bitrate="630000" FourCC="WVC1" Width="448" Height="256" CodecPrivateData="250000010FCB920DF07F8A0DF81FE804508113396020C00000010E5AE7F840" />

<QualityLevel Bitrate="470000" FourCC="WVC1" Width="368" Height="208" CodecPrivateData="250000010FC38E0B70678A0B7819E80450810E5747B6C00000010E5AE7F840" />

<QualityLevel Bitrate="350000" FourCC="WVC1" Width="320" Height="176" CodecPrivateData="250000010FC38A09F0578A09F815E80450808AADEACF400000010E5AE7F840" />

Manifest Response – reviewed• The client also receives the number of chunks for audio and video tracks

and the duration of each chunk so it can request the chunk which fits the desired position in the file

<c n="0" d="20000000" />

<c n="1" d="20000000" />

<c n="2" d="20000000" />

<c n="3" d="20000000" />

....

<c n="297" d="20000000" />

<c n="298" d="5000001" />

</StreamIndex>


</StreamIndex>



<c n="0" d="20433560" />

<c n="1" d="19969161" />

<c n="2" d="19969161" />

<c n="3" d="20433560" />

<c n="4" d="20433560" />

<c n="297" d="20433560" />

<c n="298" d="4393197" />

</StreamIndex>


Session Details – Fragment Request

• Client-Server requests are based on RESTFull URLs:GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBunny_720p.ism/QualityLevels(350000)/Fragments(video=0)

• The URL includes reference to:


• The URL includes reference to:

▫ Bitrate as QualityLevels which maps to a media file

▫ Fragment number

Session Details – Fragment Response

• The Server:

▫ checks “BigBuckBunny_720p.ism” server manifest file to find the media file associated with the quality level(350000)

▫ Opens and parses the associated media file to get the chunk with requested time offset (0).


requested time offset (0).

▫ Sends the requested media fragment to the client as HTTP response with status code set to 200

Refrences

• Most valuable refrence: http://alexzambelli.com/blog/2009/02/10/smooth-streaming-architecture/


SILVERLIGHT PROGRAMMING


MediaElement Code Walkthrough

Creating New Silverlight Project

• Choose new “Silverlight Application” in the new project dialog. C# or VB.Net is available.

• Choose the default settings at the dialog opened


New Project configuration

• A new project has been created.

• The starting point of the application is “App.xaml”.

▫ The starting page of the application is called by default: “MainPage.xaml” ( can be changed at “App.xaml.cs”)


• Media element Represents a control that contains audio and/or video.

• Media files cannot be used as project resource.


▫ Must be delivered separately.

• To preserve video original aspect ratio DON’T change both Width and Height.

Media Element – Main Events

• Media Events

▫ MediaOpened Occurs when media loading has finished.� ‘ActualWidth’ and ‘ActualHeight’ are set after this event is called.

▫ MediaEnded Occurs when the media has ended.

▫ MediaFailed Occurs when an error is encountered.� Use ‘ExceptionRoutedEventArgs’ to get failure details.


• Buffering Events:

▫ BufferingStarted Occurs when media buffering has begun.

▫ BufferingEnded Occurs when media buffering has ended.

Controlling Media Element

• Setting Audio/Video input.

• Playing file from current position.

▫ Call this only after MediaOpened event is called.


• Stop Playing current

▫ Stops the media playing and sets current to start

• Pause command

Full Source - MainPage.xaml


Full Source - MainPage.xaml.cs


Adobe HTTP Streaming


Agenda

• Overview

• Components

• Files

• Session


HTTP Streaming Intro

• HTTP Streaming offers the advantages of:▫ Progressive download in terms of � Cost

� Standard Server

� Scalability




▫ Streaming in terms of � User experience

� Seek-ability of streaming

Overview

Adobe HTTP Dynamic Streaming is a solution that allows you to stream live and on-demand content over HTTP to Adobe Flash Player. When content streams over HTTP, clients can seek quickly to any


HTTP, clients can seek quickly to any location.

HTTP Dynamic Streaming supports adaptive streaming, DVR functionality, and Adobe Flash Access protection (DRM).

COMPONENTS

• Content Ingest

• Server


• Server

• Client

Main components

• Preparation

▫ File Packager

▫ Live Packager for HTTP Dynamic Streaming

• Server

▫ Apache module (HTTP Origin Module)

▫ Flash Access


▫ Flash Access

• Client

▫ Player with OSMF classes

▫ Flash Player version 10.1+

▫ Air 2.0+

Adobe Solution main components


Ingest- File Packager

• A command-line tool

• Used for converting offline content to formats required for Adobe HTTP streaming

• Translates on-demand media files into fragments and writes the fragments to F4F files.


Ingest - Live Packager

• The Live Packager for HTTP Dynamic Streaming is part of Adobe Flash Media Server.

• The server ingests a live stream over RTMP and translates it into F4F files in real-time.

• The built-in Apache HTTP Server uses the HTTP Origin Module to deliver the live content over


Origin Module to deliver the live content over HTTP.

Server - Apache module (HTTP Origin)

• Extension to Apache HTTP Server version 2.2.

• Enables processes of Adobe Files:

▫ F4F, F4M,F4X

▫ .bootstrap and

▫ .drmmeta

• Flash Media Interactive Server


• Flash Media Interactive Server 3.8 includes Apache HTTP Server.

Server - Flash Access

• DRM Server

• Flash Access delivers protected media to Flash Player

• For content protection, both File Packager and Flash Media Server are required to package and encrypt the


required to package and encrypt the content

Client - OSMF classes

• The OSMF Player uses the ActionScript 3.0 NetStream.appendBytes() API to deliver data to Flash Player.

• OSMF is a robust framework designed to deliver high-quality video.

• Adobe strongly recommends using OSMF


• Adobe strongly recommends using OSMF to build HTTP Dynamic Streaming players.

ADOBE HTTP STREAMING FILES

F4F, F4M,F4X

.bootstrap and


.bootstrap and

.drmmeta

Files

• The files required for HTTP streaming are:▫ F4F - MPEG4 media format. Holds the media

▫ F4M – Media description file(codec, resolution)

▫ F4X - Fragments location file

▫ .bootstrap – bootstrap information for each segment


segment

▫ .drmmeta – DRM encryption information

*.F4F File

• Standard MP4 format with open file specification

• Each file contains a segment of the source file.

• Each segment contains one or more fragments of content.

• The file formats stores any flash supported codec

• A player can use a URL to address each fragment.


• A player can use a URL to address each fragment.

HTTP Streaming file types

• *.F4X File▫ Flash Index file.

▫ Contains the location of specific fragments within a stream.

• *.F4M File▫ Flash Media Manifest file.


▫ Flash Media Manifest file.

▫ Contains information about the media codecs, resolution, and the availability of multi-bitratefiles.

F4M File Sample<?xml version="1.0" encoding="utf-8" ?>

- <manifest xmlns="http://ns.adobe.com/f4m/1.0">

<id>Dynamic Streaming</id>

<duration>253</duration>

<mimeType>video/mp4</mimeType>

<baseURL>rtmp://cp67126.edgefcs.net/ondemand</baseURL>

<media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" />


ream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" />



</manifest>

HTTP Streaming file types

• .bootstrap

▫ Bootstrap file.

▫ Contains the bootstrap information for each segment of the file

• .drmmeta

▫ DRM Header file.


▫ DRM Header file.

▫ Contains additional header information about the encryption.

Folder example

• Example for File “foo” Folder will include :

▫ fooSeg#.f4f (many fragments)

▫ foo.f4x

▫ foo.meta

▫ foo.bootstrap

▫ foo.drmmeta (if the stream is configured for


▫ foo.drmmeta (if the stream is configured for encryption)

Messages Flow

Server Client

Manifest Request

Manifest Response

Fragment Request


Fragment Response

Fragment Request(s)

Session

• The player fetches foo.f4m manifest.

• The player request fragment#

• HTTP Origin Module locates the required f4f files by looking at F4X index file and send to client.

• If the client seeks further, the Origin module searches at the f4m and again locates the f4f file that fits, starts streaming again.


streaming again.

• If the player constantly monitor bandwidth and performance and ask the Origin Module for the proper bit rate associated f4f.

HTML5 Video


HTML5

• Drafts by WHAT WG

▫ Web Hypertext Application Technologies

• Merging into W3C specifications• “One of HTML5’s goals is to move the Web away from

proprietary technologies such as Flash, Silverlight, and JavaFX, says Ian Hickson, co-editor of the HTML5


JavaFX, says Ian Hickson, co-editor of the HTML5 specification.”—Paul Krill, reporting for InfoWorld, June 16, 2009

• Browser support

Fragmented Web - Description

• Multimedia coding on the web is fragmented

• Many video codecs:

▫ DIVX, XVID, H.264

▫ WMV, VC-1, VP6

• Many containers (File Format)

▫ AVI, MKV


▫ AVI, MKV

▫ MPEG4 FF, 3GPP

• Many delivery methods

▫ RTSP/RTP Streaming, Progressive download

▫ Live HTTP, Smooth Streaming

Fragmented Web - Challenges

• Proprietary Plug-ins - like Flash

• Vertical market control on media distribution –like Apple

• Media Distributers need to support many:

▫ Codecs

▫ Containers


▫ Containers

▫ Delivery Formats

• in order to support all device and audiences

XIPH

• XIPH.org is a non profit organization which aims to create free multimedia coding standards

• XIPH defined

▫ Vorbis – Audio codec

▫ Ogg – a free file format media container

▫ Speex – voice codec


▫ Speex – voice codec

▫ Theora – Video Codec

• HTML5 Video first based its video codec and container standard on XIPH Standards

HTML5 Video

• HTML5 video first defined XIPH formats as the base HTML5 video: “User agents should support Theora video and Vorbis audio, as well as the Ogg container format.” December 10, 2007, the HTML5 specification

• This was later replaced by a statement which


• This was later replaced by a statement which basically stated: we cant make up our mind, use whatever you like.

HTML5 Video - Fragmented• Support Theora (version of VP3)

▫ Old codec

▫ Poor performance (BR/Quality ratio)

▫ Free no royalties

▫ Hardware support?

• Also H.264


▫ Much better quality per bitrate

▫ But it requires royalties….

• Google opens VP8

▫ Good Quality

▫ No Royalties (?)

HTML5 Video Code

<videosrc="movie.ogg" controls="controls">If you can see this text, your browser does not support the HTML5 video tag.</video>


Source W3C School

Browser CODEC Support

Browser Ogg Theora H.264/MPEG-4 AVC

Internet Explorer NO 9.0

Mozilla Firefox 3.5 No


Google Chrome 3.0 3.0

Safari No 3.1

Opera 10.50

What is missing

• Standard Multi-bitrate support

• HTTP Streaming (not PD)

• Option for live streams

• Transmit your camera (ChatRoulette Style)

• P2P Interaction


Is that the Flash Killer?

WebM Project


WebM Overview

• Google Sponsored Project

• Aims to create: Open, Royalty free media coding formats for the open web

• Defines

▫ File Format / Container

▫ Audio CODEC


▫ Audio CODEC

▫ Video CODEC

H.264 vs Theora

• Theora is a royalty free code but has lower quality than H.264

• H.264 requires royalties for IP owners but has better quality

• Some browser used H.264 video some stayed with Theora.


with Theora.

• HTML5 video fragments the web instead of unit it

Browser Ogg Theora H.264

Internet Explorer

NO 9.0

Mozilla Firefox

3.5 No

Google Chrome

3.0 3.0

Safari No 3.1

Opera 10.50

Source: Wikipedia

WebM

• WebM fills the gap left by HTML5 standardization.

• Defines: video, audio and container formats

• Solves the royalty free Theora vs the superior quality H.264 by providing a royalty free video codec with the same (or better) video quality as H.264


H.264

Source: On2

CDNs

www.dsp-ip.comFast Forward Your DevelopmentSource: Limelight

Previous CDN classification

• Server based – Incumbents, Akamai

• Traditional P2P

▫ “Bittorrent”-s

▫ Pando Networks

Classify CDN based on delivery technology


▫ Pando Networks

• Grid Based

• Hybrid Architecture

▫ “Peer Assisted Streaming”

▫ Joost

Classic CDN Segmentation

CDN

PoPBased

P2PMulti-stream

Grid


Distributed Centralized Pure P2PPeer

Assisted

Modern CDN Classification

Classify CDN based on

▫ Who pays for the service?

▫ What is the service goal?

not on the delivery technology

• Content Provider CDN


• Content Provider CDN

• External Service provider CDN

• Internal Service Provider CDN

• Application CDN

Modern CDN Classification

CDN

Service Provider -

Service Provider -

Content Based


Provider -Internal

Premium Content

OTT

Provider -External

Based

Content Provider CDN

• Goal – Assure delivery, Reduce load and delay from content providers’ sites.

• Examples

▫ Akamai

▫ Limelight

• Located – in main service provider’s POP


• Located – in main service provider’s POP deployed globally

• Client - Content distributers and owners such as CNN, Hulu and “Youtube”-like sites.

External Service providers CDN

• Goal - Reduce EXTERNAL bandwidth cost for ISP by caching content within the ISP network.

• Examples

▫ Oversi

▫ PeerApp

• Located – in main ISP POPs


• Located – in main ISP POPs

• Equipment is PC server based or embedded

• Clients - service providers like 012 or 014 (Israeli examples)

Internal Service Provider CDN• Goal - reduce INTERNAL bandwidth load inside

the service provider network by caching.

• Used mainly in IPTV to distribute premium content (due to cost)

• Examples: Mostly VoD service providers

▫ Bitband (Motorola)


▫ Bitband (Motorola)

▫ Velolcix (Alcatel-Lucent)

• Located – Distributed within service provider’s network

• Equipment is PC server based or embedded

• Clients – IPTV service providers

Peer based CDNs

• Bittorrent

• Giraffic

• RayV


Video Technology PositioningMicrosoft

Apple


Apple

Adobe

Google

Microsoft

• Large Install base 60%+ for Silverlight

• Good connection to media and content companies (DRM – Play Ready)

• Innovative in video technologies

▫ HTTP Streaming

▫ CDN-oriented


▫ CDN-oriented

▫ Moving from proprietary to standard technology

Apple

• User base

▫ Relatively small and fanatic

• Video innovation – minor

▫ HTTP Streaming FF is based on MPEG2

▫ Many fragments - hard to manage and CDN

▫ First use of front video cam on iPhone 4.0


▫ First use of front video cam on iPhone 4.0

• Closed platform

• Vertical market control

• Relaying too much on HTML5

• Annoying clients and business partners

Adobe

• Install base

▫ Biggest 99% of connected computers

▫ Company image suffers from bad PR lately

• Innovation

▫ Very innovative in the past

▫ First massive use of progressive download


▫ First massive use of progressive download

▫ Good on Integration Solution – OSMF

▫ Losing its edge:

� Last to implement HTTP Streaming

Adobe

• Acquisition strategy - Poor

▫ Can’t monetize past acquisitions: Macromedia, Amicima

▫ Paid too much for last acquisition: Omniture

• Technology

▫ Got good technologies


▫ Got good technologies

▫ Focused on developing technologies and less on monetizing them

Google

• Install base

▫ Medium

▫ Good growth potential on mobile / browser

• Innovation

▫ Good Innovation Internally

▫ Aggressive acquisition strategy (Android, On2)


▫ Aggressive acquisition strategy (Android, On2)

▫ Open system as innovation

• Open platform strategy

▫ Not technology oriented

• Partnerships

▫ Good relations with partners

Google

• Acquisition strategy – Great!

▫ Does not pay too much on companies

▫ Monetize and build acquired companies IP: Android

▫ Great move with On2 IP

• Technology


• Technology

▫ Good mix of internal and external technologies

▫ Use standard API & Protocols

▫ Standardize proprietary Technologies (VP8)

▫ Does not depend on a single delivery technology: HTML5 or Flash

How do we define success?

• Who is first in the video race:

▫ Install base – Speed

▫ Innovation - acceleration Install

Base


Innovation

Positioning

• This is my understanding.

• What do you think?Install

Base


Innovation

Summary

• Scaling Video delivery:

▫ CDNs

▫ HTTP Streaming

▫ Standards

▫ Interoperability

• Lowering costs


• Lowering costs

▫ HTTP Streaming

• User experience

▫ PD -> Streaming

▫ Multi-bit-rate

DSP-IP Contact information

Download slides at: Download slides at: www.dspwww.dsp--ip.comip.com

Course materials & lecture requestCourse materials & lecture request

Adi YakovAdi YakovTraining ManagerTraining Manager

adi@[email protected]

++972972--99--86519338651933


Projects development services:Projects development services:

++972972--99--86519338651933

wwwwww..dspdsp--ipip..comcomMailMail :: info@[email protected]:: ++972972--99--88509568850956,,FaxFax :: ++972972--5050-- 89629108962910

Alona AshkenaziAlona Ashkenazi

Development Development ServicesServices

alona@[email protected]

++972972--99--88509568850956

APPENDIX OPTIONAL SLIDES


OSMF


Goal

• Accelerate Development

• Standardize API between modules

For Video Players


Support Media Types

• Streaming audio

▫ Mp3 , AAC , Speex and Nellymoser

• Streaming video

▫ FLV, F4V, MP4, MPEG:MP4, M4V, 3GPP


3GPP

• Progressive audio & video

▫ Mp3, FLV, F4V, MP4, MP4V-ES, M4V,3GPP, QuickTime

Basic Structure an OSMF player

Media player Control playback of a media

element

Media containerDisplay media element (s)

On the stage

Play, Stop, Pause etc Add to stage, show element(s)


Media element Loads any support media type

Video, images, sound, etc or

Composite element

load stuff

Imports

Import org. osmf. containers. Media container;

Import org. osmf. Elements. Video element;

Import org. osmf. Media .media player ;

Import org. osmf. Media. URL Resource;


The media container




Import org. osmf. Media. URL Resource;


// create the container class that displays the media .

var container: media container = new media container ( ) ;

The media container




Import org. osmf. Media. URL Resource; // create the container class that displays the media .var container: media container = new media container ( ) ;


//add the Media container instance to the stage

Add child (container) ;

The media container




Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage


Var video Element =

new video Elements ( new URL resource ) ;

//add the Media container instance to the stage Add child (container) ;

The media container




Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage


// cerate the media player instance

Var media player : media player = mew media player ( );

//add the Media container instance to the stage Add child (container) ;Var video Element =

new video Elements ( new URL resource ) ;

The media containerImport org. osmf. containers. Media container;



Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage Add child (container) ;


Add child (container) ;Var video Element = new video Elements ( new URL resource ) ;// cerate the media player instance Var media player : media player = mew media player ( );//Set the media Elements on the media player// because auto play defaults to true , play back , begin immediately media player . media = Video Elements ;

ADDING CAPABILITIES

Playback control


Media Factory

Layouts

Default Playback

// set auto play to false

Media player .auto play=false;


// listen for the click event on the stage

Stage . Add Event Listener ( mouse Event . CLICK,_ on click );

Function _on click (evt ): void

{

// if the media player is playing

If (media player .playing )

{

// pause it

Default Playback


// pause it

media player. pause ( );

}

else {

// resume it

media player . Play ( );

}

}

Media Factory

Create

Media Elements, with a URL

Media Factory


Media Elements:

video ElementImage Element sound Elements

The Media Factory

Import org .osmf .media .default media factory ;

// create the default media factory instance

Var media factory = new default media factory ();

// have the media factory create the video element


Var

Media Element = media factory create media element ( new URL resource )

LayoutsMedia container

Layout metadata:


Controls how a media element is

Displayed in the media container

Media element

Layout metadata

Width

Height

X

Y

>

Layouts


>

Can be absolute or relative

Layout metadata

// Create a Layout metadata object

Var layout = new Layout metadata( ) ;

// set the width and height of the Layout metadata object

Layout . width = 640

Layout height =480

// tie the Layout metadata instance to the


// tie the Layout metadata instance to the

Media element instance

Media element . add metadata (Layout metadata. LAYOUT_MANESPACE, layout ) ;

Media element

Netloader


Netloader

HTTP Streaming NetLoader

RTMPDynamic Streaming NetLoaderNet Loader

Netloader


DVRCastNetLoader

Change

var video element : video element

= new video element ( resource ) ;

var video element : video element = new video element (

To


var video element : video element = new video element ( resource , new RTMPDynamicstreamingNetLoader ( ) )

Media Element

Vector.<DynamicstreamingReousorcecletem>


Dynamic Streaming

// create a new Dynamic streaming Resource pointing to FMS application

Var resource : Dynamic streaming Resource = new

Dynamic streaming Resource ( RMTP_ URL ) ;

// create a Vector to store the

Dynamic Streaming Item object


Var vector : vector . < Dynamic Streaming Item > =

New vector . < Dynamic Streaming Item > ( 3 ) ;

// create 3 entries in the vector, each a DynamicStreamingItem , with a file name and a minimum bit rate

Vector ( 0 ) = new DynamicStreamingItem

( “ 2012 _ high “ , 1500 ) ;

Dynamic Streaming


( “ 2012 _ high “ , 1500 ) ;


( “ 2012 _ low “ , 400 ) ;


( “ 2012 _ medium “ , 600 ) ;

// set the StreamingItems property on the

DynamicStreamingResource to our vector

Resource. StreamingItems = vector ;

// create a new VideoElement , based on the resource

Var videoElement = new Video Element ( resource ) ;

Dynamic Streaming


Var videoElement = new Video Element ( resource ) ;

Dynamic Streaming in a few linesVar resource = new Dynamic Streaming resource ( RMTP_ URL ) ;

Var vector = new vector . < DynamicStreamingItem > ( 3 ) ;


( “ 2012 _ high “ , 1500 ) ;


( “ 2012 _ low “ , 400 ) ;



( “ 2012 _ medium “ , 600 ) ;

Resource . StreamItems = vector ;

Var VideoElement = new VideoElement (resource ) ;

Advance video distribution

Technology

Transcript of Advance video distribution