Brightcove Inc.

Video Coding and Delivery… at Scale

Yuriy A. Reznik, PhD

Brightcove, Inc.

Mile High Video Workshop, Denver, CO, July 31, 2018

Brightcove Inc.

About Brightcove

2

Brightcove Inc.

Global Presence & Reach

3

Brightcove Inc. 4

Brightcove Inc.

VideoCloud Architecture

Main components / chain of processing steps:

5

Mux, DRM

& SSAI

Publisher Transcoder (Zencoder)

Analytics

Video players

Encoders

packager

packager

packager

…p

BitrateTablet

TVTablet PC

MezzanineMezzanineMezzanine

Dashboard

Ingest

API

Back-

end

engine

Job

request APIbitrate=375,

res=480x270, …

bitrate=700,

res=768x432, …

bitrate=5800,

res=1920x1080,

…

Analytics

engine

Network & Usage statistics

Trans-

coder

Pipeline

Brightcove Inc.

VideoCloud Architecture

Same chain with Context-Aware Encoding (CAE) module:

6

Mux, DRM

& SSAI

Publisher Transcoder (Zencoder)

Analytics

Video players

Encoders

packager

packager

packager

…p

BitrateTablet

TVTablet PC

MezzanineMezzanineMezzanine

Dashboard

Ingest

API

Back-

end

engine

Job

request APIbitrate=375,

res=480x270, …

bitrate=700,

res=768x432, …

bitrate=5800,

res=1920x1080,

…

Analytics

engine

CAE

Profile

Generator

Trans-

coder

Pipeline

Network & Usage statistics

Profile statistics

Brightcove Inc.

Key Challenges

Scale

• need to process millions of concurrent streams

Reliability

• things have to work (even if input is totally broken, cloud region is down, etc.)

• everything needs to be redundant and fault-tolerant

Reach

• streams must be decodable on most devices in existence

• including some legacy TVs, game consoles, Blackberries, etc.

Fragmentation

• new codecs, flavors of HDR, DRMs, delivery formats, etc.

Operating costs

• cloud, bandwidth, CDN bills…

Quality

• delivered streams must be of high quality

7

Brightcove Inc.

Supporting multiple devices & formats

System complexity:

8

Core

system

Delivery

formats

Video

formats

Codecs

DRMs

H.264, HEVC, VP9,

AV1, …

SDR

HLG

HDR10

DV

HDR10+

SL-HDR1

….

FairPlay

PlayReady

Widevine

Marvin

CMLA-OMA

….

HLS v1..v7, DASH,

Smooth, progressive, …

Brightcove Inc.

A thought experiment…

Number of streams one needs to generate to reach existing devices:

7 streams per media asset

x 3 codecs (H.264/baseline, H.264, HEVC)

x 3 DRMs (Fairplay (iOS), PlayReady (TVs), Widevine Modular (Android))

x 3 formats (HLS (iOS), DASH (Android, TVs), progressive (legacy))

= 7 * 27 = 189 streams !!!

This is simply not practical!

9

Platform / DRM support matrix:

Source: https://castlabs.com/resources/drm-comparison/

Brightcove Inc.

Minimizing costs of supporting multiple formats / devices

Just-in-time packaging

• packaging and applying DRM dynamically, when clients request streams

• avoids creation of redundant (or unused) streams

Device detection

• detecting and recognizing device capabilities

• targeting manifests and streams accordingly

Rules engine

• allows operators to set specific limits on stream profiles delivered to devices of each kind

• e.g. one can specify max resolutions, bitrates, choices of codecs/profiles, etc.

Optimized multi-codec profile generation

• minimizes number of streams needed based on characteristics of content and intended set of delivery devices

10

Brightcove Inc.

The use of just-in-time packaging reduces the number of format/DRM combinations to one that is necessary to support active players.

This reduces CDN, storage and transcoding costs!

Transcoder

(Zencoder)

Transcoders

Dynamic Delivery system in VideoCloud

11

Players

Media

CDNsMedia

CDNs

Manifest

CDNsManifest

CDNs

Dynamic delivery system

Playback

API

Device

detection

Manifest

generators

Just-in-time

packagers

Media Files

+

Metadata

Content

URLs

Rules

Engine

Rules

API

API, orchestrator

Operator

Jobs

Files

Delivery

preferences

Brightcove Inc.

Video transcoding

Single-rate transcoding:

Transcoding for ABR:

Key operations:

• ABR ladder design

• video processing

• encoding

12

DecodeVideo

processingEncode

Mezzanine

video ES

Delivery

video ES

Targets

DecodeVideo

processingEncode

Mezzanine

video ES

Delivery

video ES 1

Video

processingEncode

Delivery

video ES 2

Video

processingEncode

Delivery

video ES 3

Video

processingEncode

Delivery

video ES n

Ladder of

ABR targets

Encode

request

… … …

Brightcove Inc.

Transcoding in the cloud

Single transcoding operation:

• executed on a single system

Plurality of transcoder processes executed in the cloud:

• executed on plurality of systems (cloud instances)

Key aspects:

• API

• Distribution of work

• Cloud orchestration

• Monitoring

• Ensuring reliability

• Minimizing costs

13

Transcoder process

Decode A/V processing EncodeMezzanine Delivery stream

Job

parameters

Demux Mux

Operator

Mezzanine

Cloud transcoder

CDN

MezzanineMezzanines

Job

parameters

Delivery streamDelivery streamDelivery

streams

API Orchestration

Dashboard

g processCocessCloud Instance

Transcoding processCloud Instance

Transcoding processTranscoding processes

Brightcove Inc.

Distribution of work

Types of parallelization

• stream-level

• each stream is encoded at dedicated instance

• segment-level

• stream is split in segments

• each segment is encoded at dedicated instance

• instance-level

• each instance has multiple CPU/GPU cores

• encoding job is split and allocated to those cores

• codec-level parallelization (slices, tiles, wavefronts, etc.)

When and what works?

• instance-level: always works

• stream level: works in most cases; packages of ABR streams can be more effectively processed jointly

• segment-level: only for VOD

14

Operator

Mezzanine

Job parameters

Cloud

CDN

MezzanineMezzanines

Job parameters

Job parametersDelivery stream

Delivery stream

Delivery streams

API

Orchestratio

n

Dashboard

Cloud Instance

Transcoding processCocessCloud Instance

Transcoding processCloud Instance

Transcoding processTranscoding processes

Brightcove Inc.

Transcoding API, notifications, monitoring

Example: encoding Job API in Zencoder:

An HTTP POST to: https://app.zencoder.com/api/v2/jobs

To receive notifications about job status/completion, operator can specify:

15

{

"input": "s3://zencodertesting/test.mov",

"output": [

{

"label": “stream-1-240k",

"audio_bitrate": 56,

"audio_sample_rate": 22050,

"base_url": "s3://my-bucket/",

"decoder_bitrate_cap": 300,

"decoder_buffer_size": 800,

"max_frame_rate": 15,

"public": 1,

"type": "segmented",

"video_bitrate": 200,

"width": 400,

"format": "ts"

},

…

]

}

{

"notifications": [

"http://user:password@example.com/zencoder",

"admin@example.com"

],

…

Brightcove Inc.

Optimizing cloud-based transcoder

Key topics:

• optimizing job allocations

• ensuring reliability

• redundancy

• reliable ingest & upload techniques

• support for legacy & broken formats

• managing costs

• selecting cloud, region, instances, etc.

In addition, the transcoder by itself must be good!

Attention must be given to:

• pre-processing

• setting right codec constraints

• optimizing encoding ladder design

16

Brightcove Inc.

Pre-processing

Key functions:

• understand what type of content it is (which may be different from the way it was previously encoded)

• progressive / interlace / telecine, cadence type, field order, etc.

• minimize artifacts introduced by prior generation encoder or sampling process

• blocking, ringing, broken lines, temporal noise, etc.

• perform conversion from source format to format needed for delivery. This includes conversions of:

• spatial resolution

• chroma sampling type (4:4:4, 4:2:2, 4:2:0, different kinds of 4:2:0)

• frame rate

• temporal sampling type (progressive, telecine, interlace, field order)

• color (gamma, matrix, primaries, EOTF, mastering display color volume, other display-related metadata, etc.).

Typical chain:

17

Decoder

Content analysis

Artifact

removal

filters

Temporal sampling

type conversion

Spatial

resolution

conversion

Color space

conversion

Frame-rate

conversion

Sequence and frame-level metadata, bitstream elements, SEIs, etc.

Detected temporal sampling type/pattern, cuts, broken lines, black bars, etc.

Raw video

Brightcove Inc.

Encoding: quality-rate tradeoffs

Each ABR stream (“representation”) is basically a tradeoff between:

• bitrate 𝑅 used to send video over the network, and

• quality 𝑄 achieved when video is rendered by receiver

A combination of (𝑅, 𝑄) pairs achievable for a particular sequence and codec is commonly

called quality-rate function 𝑄 𝑅 . It is normally monotonic and saturating with 𝑅 → ∞.

Notes on quality metrics:

• broadly available metrics (MSE, PSNR, SSIM, FSIM, etc.) measure only codec noise

• full-reference, specific to each resolution (spatial, framerate, sampling type, etc.)

• have no means for discrimination between resolutions, reproduction volume (HDR/SDR), etc.

• tools/models accounting for different resolutions, display & environmental parameters:

• VDP-derivatives, Tektronix PQA (PQR metric), SSIMWave SSIM+, etc.

• but… none of these metrics/models is perfect

• human-measured scores typically scatter around

• the most one can infer from an objective quality score is the probability that user assessment of

quality is above or below a given threshold.

18

Hypothetical shape of quality-rate function:

Some existing metrics (Zhang, et al 2011):

Brightcove Inc.

Range of resolutions

Note: 10x downscaled videos

start losing details that are

most prominently visible in

normal reproduction setting.

This suggests that the range of

resolutions that can be used

for encoding should not be

larger than 10.

Coincidentally, this is precisely

the range supported by the

DVB ladder.

Encoding: choices of resolutions

In theory, the more resolutions are available the better:

• allows better quality / rate tradeoffs:

In practice, the choices of resolutions are also constrained by

• content owners

• industry forum guidelines: DVB, HBBTV, DTV, etc.

• closed ecosystem guidelines: Apple TV, etc.

• capabilities of playback devices

19

– 960x540

-- 768x432

– 640x360

𝑄 𝑅𝑄

𝑅

DVB recommended resolutions

~10x

Brightcove Inc.

Setting right codec constraints

VBR control

• for ABR streaming streams must be capped VBR !!!

• the use of highly-variable VBR encoding may confuse DASH clients and cause buffering or

inefficient use of bandwidth

• typically, maximum bitrate cap is set to about 10-35% above average bitrate

• must be lower than next target bitrate in the ABR encoding ladder

• decoder buffer size must also be limited

GOP length and type

• GOP length must be shorter or equal than GCD of delivery segment lengths

• E.g. for 4,6, and 10-sec segments, GOP <= gcd(4,6,10) = 2 sec.

• GOP length may also be affected by the need to support ad-insertions/splicing, etc.

• closed GOP is most commonly used

Profiles & levels

• H.264 Baseline profile is needed for legacy devices (e.g. mobile devices prior to 2012)

• Main profile is adequate for streams of up to 720p

• High is better for 1080p and beyond

• Level must be sufficient to carry stream at given resolution, framerate, bitrate, CPB size

Reference frames, B frames:

• for legacy devices (e.g. mobiles of 2012 and earlier) – no B frames, 1 reference

• most STBs can support up to 4 reference frames, 3 B-frames

20

Example level/profile combinations:

Example of uncapped VBR behavior:

Brightcove Inc.

Optimizing ABR encoding profiles

Encoding profile parameters:

• bitrates: 𝑅 = (𝑅1… , 𝑅𝑛)

• resolutions: 𝑆 = (𝑆1, … , 𝑆𝑛), 𝑆𝑖 = 𝑤𝑖 , ℎ𝑖 , 𝑓𝑖 (width, height, framerate)

• codec-related constraints 𝐶 = (𝐶1, … , 𝐶𝑛)

Typical ladder design constraints:

• bitrate monotonicity: 𝑅min ≤ 𝑅1 < ⋯ < 𝑅𝑛 ≤ 𝑅max

• bound on first rate/start-up latency: 𝑅1 ≤ 𝑅1,max

• bitrate granularity constraints: 𝛾min ≤𝑅𝑖+1

𝑅𝑖− 1 ≤ 𝛾max, 𝑖 = 1,… , 𝑛 − 1

• resolution monotonicity: 𝑆1 ≤ ⋯ ≤ 𝑆𝑛 , 𝑆𝑖 = 𝑤𝑖 ⋅ ℎ𝑖 ⋅ 𝑓𝑖

• codec noise thresholds: 𝑞min ≤ 𝑞𝑆𝑖(𝑅𝑖), i = 1,… , n

• quality monotonicity: 𝑄min ≤ 𝑄(𝑅1) < ⋯ < 𝑄(𝑅𝑛)

Optimal ladder design:

• find: number of points 𝑛∗, bitrates 𝑅∗ = 𝑅1∗, …𝑅𝑛∗

∗ , resolutions 𝑆∗ = 𝑆1∗, … 𝑆𝑛∗

∗ , and constraints 𝐶∗ = 𝐶1∗, … 𝐶𝑛∗

∗ , such that:

Φ 𝑛∗, 𝑅∗, 𝑆∗, 𝐶∗ = max𝑛∈ℕ, 𝑅∈ℝ𝑛, 𝑆∈ℝ3×𝑛

+all constraints

Φ 𝑛, 𝑅, 𝑆, 𝐶

where: Φ 𝑛, 𝑅, 𝑆, 𝐶 is a figure of merit function (e.g. best overall quality, lowest cost, etc.)

21

Brightcove Inc.

Brightcove ABR profile generator

Available as “CAE” ingest profiles in VideoCloud and Zencoder: Example custom CAE profile:

In VideoCloud there are 3 standard CAE profiles, as shown above.

Advanced users can also define custom CAE profiles using JSON descriptor.

This way users may specify limits for:

• number of renditions

• range of bitrates to be used

• list of allowed video resolutions, framerates, codec profiles & levels

• allowed granularity of rate steps in the profile

• operator-specific network and usage statistics, etc.

22

{

"id": "1234567890",

"version": 1,

"name": "custom-cae-profile",

"dynamic_origin": {

"dynamic_profile_options": {

"min_renditions": 2,

"max_renditions": 6,

"max_resolution": {

"width": 1920,

"height": 1080

},

"max_bitrate": 4200,

"max_first_rendition_bitrate": 400,

"max_frame_rate": 30,

"keyframe_rate": 0.5,

"select_baseline_profile_configuration": true,

"max_granularity": 100,

"video_configurations": [

{"width": 1280, "height": 720},

{"width": 960, "height": 540},

{"width": 640, "height": 360}

]

}}}

Brightcove Inc.

Brightcove CAE: an example

Example of static vs CAE-generated encoding profiles:

Advantages:

• CEA saves bandwidth: 2366kbps vs 3800kbps at 1080p => 36.77% saving!

• CAE saves number of renditions / transcoding costs: 4 vs 8

• CAE delivers more pixels / better quality at low rates: 432p @ 777kbps vs 360p @ 900kpbs

23

Demo: https://www.brightcove.com/en/blog/2017/05/context-aware-encoding-improves-video-quality-while-cutting-costs

Brightcove Inc.

Summary

Modern-era OTT video delivery poses a number of challenges, and requires some highly specialized tools:

• Dynamic packaging & delivery

• addresses the need to support multiple devices, codecs, DRMs, and delivery formats

• minimizes storage, bandwidth and CDN costs

• High-quality mass-scale transcoder

• scalable at instance, segment, and stream-levels

• supporting all sorts of input formats

• achieving high encoding quality

• incorporating intelligent ABR profile generator

• Means for playback, monitoring and end-to-end optimizations

• optimized clients

• end-to-end analytics

• customizable profiles, delivery rules, etc.

One can have much fun building them, or just pick some of the existing solutions.

To try our products, please click:

https://register.brightcove.com/en/video-cloud – VideoCloud

https://zencoder.com/en/ – Zencoder

24

Brightcove Inc.

Thank you!

Contact:

Yuriy A. Reznik

yreznik@brightcove.com