SmartLabs

SmartMEDIA 5.0

Administration Guide

Document released on 13.06.2018 ©2018 SmartLabs LLC

Table of Contents

Services and Components 4 Conveyor 4 Playlist Generator 5 RTSPServer 5 Redirector 6 SmartPVR 6 SmartMPicker 6 SmartCETS 7 Utilities 7

vodconvert 7 siege 7 scanTs 8 cets 8 tsSnoop 8

Planning 8 Local Deployment 8 Cloud Deployment 8 System Requirements 9

General Recommendations for the Hardware Platform 9 Requirements for Transcoding Servers 10 Calculating Storage Requirements for nDVR Services 11

Storage Capacity 11 Storage Performance (Write) 11 Storage Performance (Read) and Caching 11

Using the CDN 12 Installing SmartMEDIA 13

Preparing for the Install 13 Configure the SmartLabs yum repository 13 Disable SELinux 13 Make Basic Configuration for IPTables Firewall 14

Installing Packages 14 Quick "All-in-one" Installation 14 Individual Component Installation 15

Stream Processing and Record Servers (SmartMEDIA Conveyor) 15 Playlist Generator (SmartMEDIA Playlist Generator) 16 Index Storage Servers (MongoDB) 17 HLS / DASH Streamers 17 Installing Third-Party Components for Transcoding Support 18

Installing the License 19 First Start 19

Backup and Fault Tolerance 20 Building Failover Clusters 20

Redundancy of VoD Services 20

– 2 –

Redundancy of LiveTV-based Services (Live, nDVR, TSTV, etc.) 21 Solution 1: Record and Storage Servers Duplication 21 Solution 2: Using S3 Object Storage 22 Solution 3: Using an S3 Object Storage and

Redundant SmartMEDIA Services (All-in-One) 23 Solution 4: Using a Ceph (S3) Object Storage,

Combined With the Record Service 23 Load Balancing 24

Using Content Protection Systems (CAS / DRM) 26 Encrypt Live and nDVR Content 26

HLS (AES-128) 26 HLS and DASH with the ISOBMFF Container (MP4 Fragmented) 27 HLS and RTSP for IPTV Systems (DVB Simulcrypt) 28 Encryption of Multicast MPEG2-TS Content 28

VoD Encryption 28 Configuring DASH Encryption 28

Performance Optimization 29 Disk Subsystem 29 Network Subsystem 32

Solving the main tasks 36 Live Streams Recording 36 Preparing VoD Content 43 Redundancy and Load Balancing with HTTP Redirector 43 Protecting Multicast Streams (CETS) 45

Appendix 1. Supported protocols 46 HLS 46 MPEG-DASH 48 Smooth Streaming 51 RTSP 51

Appendix 2. Linux Network Subsystem Configuring 51 Calculating the TCP Window for the Bandwidth Delay Product 51 TCP/IP Stack Configuration for Linux 52

Appendix 3. Example of Running FFmpeg for VoD Content Preparation 57 Appendix 4. Examples of the Vodconvert Utility Using 59

– 3 –

Services and Components

Depending on the services you wish to provide to your subscribers, the SmartMEDIA solution may include a different set of components. Below is the SmartMEDIA component interaction diagram, which illustrates the component location in the content processing workflow from the source TV streams and VoD files up to subscriber devices. This is followed by a short description of each component.

Conveyor

The SmartMEDIA Conveyor service (smcvr) is responsible for processing (transcoding, remultiplexing, segmenting), indexing and recording of LiveTV streams to the storage.

Technically, the nDVR and Unicast Live services do not differ. In both cases, the stream must be written to the storage, but in the case of Unicast Live, the record can be made to a temporary storage (for example, tmpfs) and deleted after a minute or two.

Content can be delivered to the SmartMEDIA Conveyor server within the MPEG2-TS container using one of the following protocols:

● UDP over IP Multicast (without RTP encapsulation), ● HTTP Stream, ● HLS (according to draft-pantos-http-live-streaming-07), without encryption.

– 4 –

SmartMEDIA Conveyor performs:

● transcoding of incoming streams (if necessary), ● remultiplexing, ● recording to the storage, ● index generation and writing indexes to MongoDB, ● media data encryption (if necessary).

Adaptive Streaming Support To record several streams of the single channel which have different bit rates (Adaptive Streaming), SmartMEDIA Conveyor synchronizes them at the recording stage and create one index record for all the chunks of different bitrates of the same time interval. Thus, all streams (audio, video, subtitles) must be synchronized in time (DTS discrepancy within the stream should be minimal) and have the same GOP structure of the video stream with synchronous keyframes. Otherwise, streams will not be synchronized and will not be written.

Management There are two ways to manage SmartMEDIA Conveyor's settings:

● via the channel configuration file, ● via the JSON-RPC API.

Playlist Generator

The SmartMEDIA Playlist Generator (smplgen) generates HLS playlists and DASH manifest files for recorded and indexed content, and also checks the availability of content for service requests from SmartMEDIA Redirector.

To acquire playlists, the subscriber device sends a request to the web server (nginx web server is included in the standard package), which in turn requests a playlist/manifest from the Playlist Generator service using the FastCGI protocol. The Playlist Generator forms a playlist based on the index stored in MongoDB, and returns it to the device.

Functionality: ● HLS playlist generation (m3u8) for the streams recorded in the MPEG2-TS container; ● HLS (m3u8), DASH (MPD) and SmoothStreaming playlists generation for the streams

recorded in the MP4 Fragmented/ISO BMFF container.

– 5 –

RTSPServer

The SmartMEDIA RTSPServer (rtspServer) service is designed to support devices that do not support the HLS protocol, but support RTSP. The main task of RTSPServer is to translate RTSP requests to HLS and to transfer media content to RTSP-compatible end-user devices. Supports playback and forward/backward rewind (trickplay).

RTSPServer can also be used to build distributed content delivery solutions. In this case, the content delivery between different geographical sites can be done over HTTP, and RTSP will be used only for the "last mile", to deliver content to the end-user devices from the EDGE nodes.

Redirector

The SmartMEDIA Redirector service (httpRedirector and rtspRedirector) allows you to define server groups (farms) and achieve uniform and flexible load balancing between servers or server groups, depending on client location and requested content. In particular, it:

● provides failover to backup sources (servers / farms) when one or more sources fail; ● monitors the health of servers and services; ● checks the availability of requested content on servers; ● provides caching of information about the availability of servers or content.

For a more detailed description of HTTPRedirector workflow, see "Load Balancing".

SmartPVR

The SmartPVR service generates, upon the request, a standalone record of the specified TV program from the records generated by the SmartMEDIA Conveyor service, based on the requested channel and the broadcast time (so-called Live-to-VoD). Creation and management of records are performed with JSON-RPC API.

SmartMPicker

SmartMEDIA SmartMPicker (smpicker) allows you to pick the working/best source of incoming stream and send it to a user or another service.

SmartMPicker can be used to ensure the fault tolerance of incoming traffic and automatically switch to the backup stream for the Conveyor service.

The source streams are defined in the SmartMPicker configuration file in the order of their priority. If there are no data in the stream or it is ignored at the current time of the day, according to the

– 6 –

configuration, SmartMPicker moves to the next stream from the list. If none of the streams match the conditions, a "stub" stream from the file can be generated.

The list of incoming streams maps to the address of the outgoing multicast stream, which will be delivered to the user or another service.

SmartCETS

The SmartCETS service encrypts MPEG2 TS streams in real time according to ISO/IEC 23001-9 (part: "Common encryption of MPEG-2 transport streams"). The encryption keys are requested over the Widevine Modular DRM/SmartLabs UDRM protocol (see the document "SmartMEDIA. Integration Guide").

Utilities

vodconvert

The vodconvert utility is designed for converting VOD content to the required format and generating DASH/Smooth Streaming/HLS playlists, as well as content encryption.

The utility can receive several files with several tracks. Track selection and processing setting are defined in command-line parameters.

MPEG-DASH / Microsoft Smooth Streaming

vodconvert can accept TS, MP4 and MKV files. After processing:

● each selected track will be saved to a separate MP4 file; ● if encryption is required, the content will be encrypted when saving; ● an MPD file (manifest) will be created.

HLS

vodconvert can accept TS files as a source content. After processing:

● a playlist will be created for each source TS file, files won't be fragmented;

or:

● the source TS files will be divided into chunks of the specified duration, after which playlists will be generated.

– 7 –

siege

The siege utility emulates the load on the video servers created by a specified number of devices when working with HLS or RTSP protocol. The utility allows you to flexibly configure the user's behavior emulation. You can specify:

● possible player states — play, rewind, pause, stop; ● possible transitions between states; ● time spent in this or that state.

scanTs

The scanTs utility analyzes MPEG2 TS files and displays a brief description of structure found: PAT, PMT, ECM, PCR, PES, PTS, IFrame, CC errors, packet format, PCR jumps.

cets

The cets utility is a simplified version of the SmartCETS service. It allows you to work with files more conveniently, but has some restrictions. Accepts all necessary parameters from the command line; when reading from a file, stops as the end of the file is reached. The cets utility requests encryption keys only once — before starting encryption, and cannot perform keys rotation (i.e. encrypts the entire source with one set of keys).

tsSnoop

The tsSnoop utility is intended for viewing the headers and structures of media containers. It outputs to the console the binary data of the source file in hex-format (optional) and a textual representation of the media data structures found (for example, TS packets, MP4 atoms, etc.). The following formats are supported: MP4, MKV, TS, H.262, H.264, H.265, MP3, DTS, AC3.

Planning

Local Deployment

For permanent loads (operators of IPTV / OTT services, TV broadcasters, etc.), it is usually expedient to purchase (or use previously purchased) equipment for content delivery tasks. Peak

– 8 –

loads can be leveled by using CDN services and / or building an infrastructure with the required performance margin.

Cloud Deployment

SmartMEDIA can also be installed on leased resources and VPSes, so-called "cloud services" (for example, Amazon EC2) and cloud storage services. This solution has a number of advantages:

● absence of capital expenses for hardware resources and technological infrastructure; ● use of systems and architectures of reservation provided by the cloud service provider; ● rapid, flexible modification of the hardware platform to meet the system performance

requirements when changing the service needs.

Cloud computing services are reasonable to use for irregular loads on the broadcasting system (for example, broadcasting of certain events — sports events, concerts, etc.) and, accordingly, low profitability when purchasing your own hardware.

Storage of nDVR / VoD content in this case is provided by using storage with access via the S3 protocol.

System Requirements

The SmartMEDIA product can be installed on servers with the x86_64 architecture and with the CentOS 7 or RedHat Enterprise Linux 7 operating system.

Regardless of the way resources are allocated — purchase of equipment or rental of capacity from hosting providers and cloud service providers, the equipment performance must meet the load requirements. Exact performance requirements for a certain load (the number of sessions, the number and characteristics of recorded streams, etc.) can only be found in experimental way. General recommendations, explanations and calculations will be given below. They can be used as a starting point in the organization of the service.

General Recommendations for the Hardware Platform

SmartMEDIA's performance in stream recording and streaming is mostly determined by the performance of the storage system used (IOPS and bandwidth) and in most cases is limited by the speed of writing media data in the storage and data reading speed. The best performance can be achieved, for example, by increasing the number of disks in the storage system (when using HDD) or using solid-state drives (SSDs).

The main consumers of computing resources (CPU) are:

● SmartMEDIA Conveyor service; ● the operating system in terms of networking (nonblocking I/O);

– 9 –

● the operating system in terms of working with storage (the processor is idle in the I/O Wait state).

The SmartMEDIA Conveyor service, in turn, uses CPU time (CPU) for:

● incoming streams processing; ● streams encryption if it is used. To increase performance, it is recommended to use a CPU

with hardware support for AES algorithms (AES-NI); ● if transcoding is used — for decoding and encoding of audio streams (video streams will be

transcoded by QSV, see below).

Requirements for Transcoding Servers

If you plan to use the transcoding functionality, you'll need to use CPUs and chipsets with the Intel Quick Sync Video (QSV) hardware accelerated encoding / decoding technology support. These are some Intel Core and Intel Xeon processors.

For this task you can use processors of the 5th (Broadwell) and the 6th (Skylake) generations with the following built-in GPUs:

● Intel® HD Graphics 5500, 5600, 5700; ● Intel® Iris™ Graphics 6100, 6200; ● Intel® HD Graphics 510, 520, 530; ● Intel® Iris™ Graphics 540, 550; ● Intel® Iris™ Pro Graphics 580; ● Intel® Iris™ Pro Graphics P580.

On the processors of the 7th (Kabylake) and 8th (Coffee Lake) generations Intel Media SDK / Intel QSV is currently not supported.

The performance of the transcoder (the number of simultaneous streams, resolution, frame rate) will depend primarily on the GPU used. The performance of the CPU itself (model, frequency) affects the decoding / encoding of audio streams, demultiplexing of incoming streams and multiplexing of transcoded elementary streams.

The estimated performance of the Intel® Xeon® processor E3-1585Lv5 with the Intel® Iris™ Pro Graphics P580 is shown below:

Resolution Input/Output Codecs:

Number of realtime streams (30 fps)

Number of realtime streams (60 fps)

1080p → 1080p

AVC → AVC 18 9

AVC → HEVC 15 7

HEVC → HEVC 8 4

4k → 4k AVC → HEVC 4 2

– 10 –

HEVC → HEVC 2 1

The exact performance depends not only on the above parameters, but also on the characteristics of the incoming stream (codec settings, progressive/interlaced scanning, etc.), applied filters, used audio codecs and the number of audio tracks, etc.

For more information, visit the Intel website: https://software.intel.com/en-us/media-sdk.

Calculating Storage Requirements for nDVR Services

To calculate data storage requirements, the following values are required:

● D — number of channels to be recorded; ● Cndvr — number of concurrent nDVR viewing sessions; ● Bavg — average bitrate of the stream towards the subscriber, kbps; ● Bsum — total bitrate of one channel (the sum of all bitrates of all streams — audio, video,

subtitles — of one channel), kbps; ● W — depth (window) of nDVR recording, hours.

Storage Capacity

Vstorage (Kbyte) = Bsum × W × D × 3600 ÷ 8

For example, for recording of 10 channels with a total bitrate of ~ 8000 kbps (video: 3000, 2000, 1500, 1000 kbps, audio: 4 tracks of 128 kbps) within 48 hours, the required storage capacity will be:

8000 (kbps) × 10 (channels) × 3600 (sec / hour) ÷ 8 (bit / byte) × 48 (hours) = ~ 1.65 Tbyte

When calculating the storage capacity, it is necessary to lay some stock (about 10–15%) for cases when input streams' bitrates won't be constant, and also due to performance degradation of some file systems which occurs when a certain level of their fullness is exceeded.

Storage Performance (Write)

To calculate the number of IOPS, you need to determine the amount of data per disk operation, which depends on a significant number of factors: the read and write block size, the 'readahead' settings, the data availability in PageCache, and so on. As a rated value, it is suggested to use the size of the operation 200 Kbyte (1600 Kbit).

In this case:

IOPS = Bsum × D ÷ 1600

For example, to record 10 channels of 8000 Kbit/s each will require storage performance of about 50 IOPS.

– 11 –

Storage Performance (Read) and Caching

Video streaming services (Unicast Live and nDVR) provide delivery of a relatively small amount of content to a large number of subscribers. At the same time, the popularity of different content items is far from the equality: some programs are watched more often than others; the Live content is usually more popular than the content shown a few days ago; there can be some "blockbusters", which will be watched much more often than others, etc.

This facilitates good data caching. The cache can be implemented on the operating system's level (PageCache), and as a simple HTTP cache. In this case, the OS level cache will always be on storage servers, and the HTTP cache can be placed on separate hardware.

For the best performance of the storage system when using a local POSIX storage, it is recommended to provide at least 256 Mbyte of RAM per one recording channel for the OS level cache (PageCache).

When using third-party storage systems or software solutions (for example, CEPH), follow the vendor's recommendations for these products.

Caching in memory using HTTP server, as a rule, is more effective and in the further calculations this method is considered.

For the nDVR services, it is desirable to provide an HTTP cache volume corresponding to 1–2 hours of content storage, i.e. Sndvr_cache = Bavg × 3600 ÷ 8.

For Unicast Live services, the cache should be approximately 5–10 minutes of content. At the same time, for channels with adaptive streaming, all bitrates must be included in this volume:

Slive_cache = Bsum × 600 ÷ 8.

Depending on the cache size and the load profile, you can achieve different caching performance (HitRate).

In practice, HitRate can meet 90% or more, but for new service calculations it is better to use lower values — about 65–70% for services implementing both nDVR and Live (i.e. 65–70% of traffic will be given out from the cache, load on the storage will be 30–35%).

Based on this, the storage read performance should be:

IOPS = Bavg × Cndvr ÷ 1600 × (100-HitRate)% ,

where "1600" is the size of one disk operation in kilobits (200 Kbyte).

For example, to support 1,000 sessions with a bitrate of 3 Mbps if HitRate = 65%, you will need the storage with a performance of approximately 670 IOPS.

– 12 –

Using the CDN

If you expect a broad geography of the service, and/or also want to reduce the impact of short-term peak loads on service quality, one of the possible solutions will be the using of CDN provider services or building your own CDN network.

SmartMEDIA supports virtually all the major CDN providers, including Akamai, Amazon CloudFront and others.

When using CDN, it is necessary to take into account that:

● all requests for the VoD content can be cached for an infinite period; ● requests for the Live / nDVR chunks can be cached for an infinite period (but it is

reasonable to limit the caching period to the recording window duration); ● requests for the HLS / DASH playlists can only be cached for a period equal to the playlist

update interval (usually half the duration of the chunk). Lifetime DASH manifests can also be cached for an indefinite period.

Installing SmartMEDIA

Preparing for the Install

Configure the SmartLabs yum repository

All installation steps require the SmartLabs YUM repository to be configured properly. To configure the repository, run the following commands on the command line (bash):

sudo rpm -iUhv http://smartlabs-public:6vTh5WVrQlgF@repo.smartlabs.tv/repo/public/centos/7/x86_64/smartlabs-smartmedia-v-repo-1.0-1.el7.centos.x86_64.rpm

Disable SELinux

SmartMEDIA installation packages do not include SELinux rules. If necessary, you can configure them by yourself later.

To disable SELinux, run the following command on the command line:

sudo setenforce 0

– 13 –

In the /etc/selinux/config file, set SELINUX to disabled:

SELINUX = disabled

After the reboot, check the status of SELinux:

sudo getenforce

The command should return Disabled.

Make Basic Configuration for IPTables Firewall

In this document, we assume that IPTables is used as the firewall with the default policy set in DROP for all the chains of the Filter table.

To install IPTables instead of FirewallD on CentOS7/RHEL7, run the following commands:

sudo systemctl stop firewalld sudo systemctl disable firewalld sudo yum -y install iptables iptables-services sudo systemctl enable iptables

After setting up IPTables, the default policy for all chains of the filter table will be set to ACCEPT.

Allow traffic for already established TCP connections and traffic on the loopback interface:

# IPTables basic rules sudo iptables -I INPUT -i lo -m comment --comment "Allow Loopback traffic" -j ACCEPT sudo iptables -I INPUT -p tcp -m state -state ESTABLISHED -m comment --comment "Allow incoming traffic for ESTABLISHED TCP connections" -j ACCEPT sudo iptables -I OUTPUT -o lo -m comment --comment "Allow Loopback traffic" -j ACCEPT sudo iptables -I OUTPUT -p tcp -m state -state ESTABLISHED -m comment --comment "Allow outgoing traffic for ESTABLISHED TCP connections" -j ACCEPT

Warning: after completing the installation and configuration of SmartMEDIA, remember to save the configured IPTables rules with:

sudo /usr/libexec/iptables/iptables.init save

– 14 –

Installing Packages

Quick "All-in-one" Installation

To install basic SmartMEDIA packages, run the following command:

sudo yum install smartmedia

As a result:

● MongoDB will be installed; ● the following SmartMEDIA packages will be installed:

○ smartmedia-conveyor, ○ smartmedia-plgen, ○ smartmedia-utils, ○ smartmedia-nginx;

● the /video directory will be created;

If you're going to mount another filesystem partition or external storage to /video mount point, do not forget to change its owner by running the command:

sudo chown -R smartmedia / video

● the time will be synchronized from the Internet resource clock.isc.org over the NTP protocol and the following services will be started:

○ ntpd, ○ mongod, ○ smartmedia-nginx, ○ smcvr, ○ smplgen.

To install other components, see "Individual Component Installation."

Configure the IPTables rules as follows:

● Allow incoming TCP connections to ports 80 and 443 for the Nginx service:

sudo iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m comment --comment "SM: Allow HTTP requests from Media Streamers" -j ACCEPT sudo iptables -I INPUT -p tcp --dport 443 -m state --state NEW -m comment --comment "SM: Allow HTTPS requests from Media Streamers" -j ACCEPT

● Allow incoming multicast streams for the Conveyor service:

sudo iptables -I INPUT -d 224.0.0.0/4 -m comment --comment "SM: Allow incoming multicast" -j ACCEPT

– 15 –

# Outgoing multicast traffic (JOIN requests) sudo iptables -I OUTPUT -d 224.0.0.0/4 -m comment --comment "SM: Allow outgoing multicast" -j ACCEPT

Individual Component Installation

SmartMEDIA allows individual installation of components on different servers or virtual machines. Also, you can partially use the existing infrastructure (HTTP server or an existing MongoDB installation), or configure a MongoDB cluster with replication.

Stream Processing and Record Servers (SmartMEDIA Conveyor)

Install SmartMEDIA Conveyor:

sudo yum install smartmedia-conveyor

Configure IPTables rules:

# Incoming multicast traffic sudo iptables -I INPUT -d 224.0.0.0/4 -m comment --comment "SM: Allow incoming multicast" -j ACCEPT # Outgoing multicast traffic (JOIN requests) sudo iptables -I OUTPUT -d 224.0.0.0/4 -m comment --comment "SM: Allow outgoing multicast" -j ACCEPT # Outgoing traffic to MongoDB and S3 storage sudo iptables -I OUTPUT -d <mongodb-ip-subnet> -p tcp --dport 27017 -m state --state NEW -m comment --comment "SM: Allow outgoing connections to MongoDB" -j ACCEPT sudo iptables -I OUTPUT -d <s3-ip-subnet> -p tcp --dport <s3-port> -m state --state NEW -m comment --comment "SM: Allow outgoing connections to S3 Data Storage" - j ACCEPT

If you're going to use POSIX storage (local disk, RAID array, iSCSI volume) to store media data, install the HTTP server too and allow incoming TCP connections to port 80:

# Install nginx http server sudo yum install smartmedia-nginx # IPTables rules for incoming connections:

– 16 –

sudo iptables -I INPUT -s <streamers-ip-subnet> -p tcp --dport 80 -m state --state NEW -m comment --comment "-M ACCEPT

If you are using HTTP or HLS stream sources, configure the appropriate IPTables rules for outbound connections to source servers.

Playlist Generator (SmartMEDIA Playlist Generator)

Install the SmartMEDIA Playlist Generator package:

sudo yum install smartmedia-plgen

Configure IPTables rules:

# Incoming traffic sudo iptables -I INPUT -s <streamers-ip-subnet> -p tcp --dport 9000 -m state --state NEW -m comment --comment "SM: Allow FastCGI requests to plgen from Media Streamers" -j ACCEPT # Outgoing traffic sudo iptables -I OUTPUT -d <mongodb-ip-subnet> -p tcp --dport 27017 -m state --state NEW -m comment --comment "SM: Allow outgoing connections to MongoDB" -j ACCEPT

Index Storage Servers (MongoDB)

Install the MongoDB database software:

sudo yum install mongodb-org

To ensure fault tolerance, use MongoDB replication (clustering).

It is recommended to host MongoDB nodes on the playlist generation servers. This will increase the speed of data access and reduce server-to-server traffic when generating playlists. It is undesirable to use sharding, because it excludes the ability to retrieve all data from the local replica for generating playlists.

According to MongoDB recommendations, the odd number of nodes is preferred for replicated setup.

For more details on MongoDB replication, see https://docs.mongodb.com/manual/replication/.

Allow incoming requests to port 27017 from the record servers (to record indexes) and Playlist Generators (for reading indexes when creating playlists):

– 17 –

# Incoming traffic sudo iptables -I INPUT -s <conveyor-ip-subnet> -p tcp --dport 27017 -m state --state NEW -m comment --comment "SM: Allow incoming connections to MongoDB from conveyor servers" -j ACCEPT sudo iptables -I INPUT -s <plgen-ip-subnet> -p tcp --dport 27017 -m state --state NEW -m comment --comment "SM: Allow incoming connections to MongoDB from plgen servers" -j ACCEPT

HLS / DASH Streamers

Install the smartmedia-nginx package on servers that will deliver content to the end users:

sudo yum install smartmedia-nginx

Configure IPTables Rules:

# IPTables rules for incoming connections from everywhere: sudo iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m comment --comment "SM: Allow HTTP requests from Media Streamers" -j ACCEPT sudo iptables -I INPUT -p tcp --dport 443 -m state --state NEW -m comment --comment "SM: Allow HTTP requests from Media Streamers" -j ACCEPT

For more information about SmartMEDIA components and their functionality, see "Services and Components".

The complete list of packages included in the SmartMEDIA is shown below:

● smartmedia-conveyor, ● smartmedia-plgen, ● smartmedia-utils, ● smartmedia-nginx, ● smartmedia-cets, ● smartmedia-mpicker, ● smartmedia-pvr, ● smartmedia-rtspserver, ● smartmedia-hlsredirector.

You can also find it in the document "SmartMEDIA. Configuration Reference."

– 18 –

Installing Third-Party Components for Transcoding Support

The transcoding functionality is performed by the SmartMEDIA Conveyor service using the Intel QuickSync Video (QSV)/Intel Media SDK libraries. To enable this functionality, install the following packages on the transcoding servers:

• Linux Kernel 3.10.0-514,

• intel-linux-media — Intel Media SDK / Driver,

• kmod-ukmd — QSV Kernel Module (kernel version 3.10.0-514.el7.x86_64 required),

• libdrm — Direct Rendering Manager runtime library,

• libva — Video Acceleration API,

• intel-gpu-tools — a set of utilities for Intel graphics processors.

You can do this by running the single command:

yum install smartmedia-trc

As a result, all necessary packages will be installed. After that, reboot the OS to switch to the installed kernel version.

Installing the License

When you start SmartMEDIA services for the first time after installing the product, they will start and immediately stop with an error indicating the absence of a license file. Information about the error can be found in the services log files (in the directory /var/log/smartmedia):

[error] [default] Can not read license file "/opt/smartlabs/smartmedia/license.bin", error = No such file or directory [info] [default] SmartMEDIA license verification failed, exit

To add the license:

1. Send the system UUID to the address support@smartlabs.tv. The UUID can be found in the log of any service after its launch. For example: [2017-06-01 13: 15: 32.687637] [0x00007f0c77d7a777] [info] [default] System UUID: 00000000-0000-0000-0000-0011101DD111;

2. Obtain a license file from the SmartLabs support service. 3. Copy the resulting file to /opt/smartlabs/smartmedia/license.bin on the server. 4. Repeat the procedure for each server on which SmartMEDIA is installed.

– 19 –

First Start

After installing the license, start the SmartMEDIA services. For example, Conveyor and Playlist Generator are run as follows (other components are the same):

sudo systemctl reset-failed sudo systemctl start smcvr smplgen

Note: after the server reboot, SmartMEDIA services will be started automatically.

To get started, edit the SmartMEDIA component configuration files (see the SmartMEDIA Configuration Guide document). In particular, specify the addresses of the channel sources to be recorded on the video server in the configuration file of the SmartMEDIA Conveyor component /opt/smartlabs/smartmedia/conveyor/config/channels.json.

Changes in the channels.json file are applied automatically, service restart is not required.

The process of updating channel settings can be controlled using the Conveyor's log.

Backup and Fault Tolerance

The SmartMEDIA architecture allows to create the IPTV / OTT broadcasting system with guaranteed high availability of content and services. Depending on business objectives, budget and other factors, the system administrator can design an optimal backup architecture.

SmartMEDIA allows you to build the backup system in two main ways:

1. Building fault-tolerant clusters allows you to achieve high availability of the service by reserving individual components (storage, Conveyor services, playlist generators, HTTP servers). Failure of any of the components does not lead the cluster to fail and is invisible to the end user. Usually, it is used for reservation on the one site scale (data center). Below are typical solutions description that will help you to understand the SmartMEDIA architecture better and choose the most suitable clustering scheme.

2. Load balancing and redundancy by redirecting the client to the nearest working server / cluster. Unfortunately, even the most reliable clusters can not provide 100% availability of services — the "human factor" remains (for example, the administrator removes data from the cluster storage) , the probability of one of the fail-safe components failure (for example, an index database cluster failure) or a whole data center failure (power outage, communication failure, etc.). Improving service availability can be achieved by backing up between individual servers, clusters or entire data centers.

– 20 –

Building Failover Clusters

Redundancy of VoD Services

The VoD broadcasting service redundancy is generalized to providing fault tolerance for files storage and delivery over HTTP. Due to triviality of this task, it will not be described in detail in this document.

The following things should be considered when building a system:

● Potential failures during the content preparation stage are possible. So, it is necessary to provide the content verification after each stage of its preparation (transcoding, segmentation by the 'vodconvert' utility) before writing to the repository / repositories. Otherwise, the client may receive incomplete content (for example, if a failure occurred during the transcoding or segmentation and the resulting file was not recorded to the end) or the content not suitable for playback;

● Verify the content after writing to each of the repositories; ● The use of several separate repositories is more reliable than using one "cluster" repository,

because the probability of a clustering software failure is remained, as is the probability of data corruption due to external factors. This requires more careful verification of data on each of the repositories (including, on a regular basis). In some cases, such a solution is more complex in terms of load balancing between storages.

Redundancy of LiveTV-based Services (Live, nDVR, TSTV, etc.)

The nDVR service provisioning system is distinguished by a more complex architecture and, as a result, more complex backup schemes. Examples of various solutions, their benefits and drawbacks will be discussed below.

– 21 –

Solution 1: Record and Storage Servers Duplication

The solution is to record each channel to several independent servers. Each of them runs all the services and components necessary for the Live / nDVR services functioning, the data is stored to the local POSIX storage.

Because data recorded on different servers is not synchronized, and the content can be segmented differently, you cannot use simple load balancing in this case. Redundancy is provided by redirecting requests to a working server using HTTP Redirector.

The disadvantage of such a solution is that in the event of a failure of any service on any of the servers, content delivery to the subscriber will be interrupted and resumed only after the content will be re-requested via SmartMEDIA Redirector. In this case, SmartMEDIA Redirector will redirect the client to a working server after the failure detection on one of the servers.

– 22 –

Solution 2: Using S3 Object Storage

Record services redundancy — 2N (Active/Standby); other services redundancy — N+M (Active/Active)

The solution architecture means record servers redundancy in Active / Standby mode; each server works with its own group of TV channels. Recording and storage of nDVR content is performed on an external service (local or cloud — for example, Amazon S3), compatible with the S3 protocol.

In this architecture, all the recording servers and delivery servers in the cluster store an equal copy of the index database. Its relevance on all nodes is maintained by the MongoDB internal mechanisms.

Because all delivery servers of the same cluster ("Video Server" in the diagram) have access to the same copy of the content and to the same index database, load balancing between them can be implemented both by the SmartMEDIA HTTP Redirector service and standard HTTP load balancing tools (virtual addresses + DNS RoundRobin, LVS, hardware HTTP balancer, or others).

Each content delivery server (Video Server) is both a playlist generation server and an HTTP access proxy to the S3 storage: the HTTP server requests a playlist from the Playlist Generator component and returns it to the device, and then starts sending the nDVR content from the S3 storage system to the subscriber.

– 23 –

Solution 3: Using an S3 Object Storage and Redundant SmartMEDIA Services (All-in-One)

Record services redundancy — 2N (Active/Standby); other services redundancy — N+M (Active/Active)

Solution 3 differs from the previous one by moving of record servers (primary and backup) from individual servers to delivery servers. This solution reduces the cost of equipment, but increases the load on media servers and increases the requirements for the stability of data transmission over the network (losses, jitter, etc.).

Solution 4: Using a Ceph (S3) Object Storage, Combined With the Record Service

– 24 –

Record services redundancy — 2N (Active/Standby); other services redundancy — N+M (Active/Active)

The Ceph is an example of a relatively inexpensive and reliable storage solution with S3 access. Other solutions, such as SwiftStack and others, can also be used as storage.

The Ceph cluster (not included in the SmartMEDIA product) can consist of any number of storage servers ("MSTORx" in the diagram). The local disks of each server are joined into a single cluster, which provides a sufficiently high level of nDVR content storage reliability. The load between storages is balanced by Ceph internal mechanisms.

In addition to the Ceph storage system, the SmartMEDIA Conveyor component (primary or backup), recording the streams, can be installed on the same servers. If the primary service fails, the live streams recording switches to the standby one. Some servers in the cluster can only perform the nDVR content storage function.

Similar to "Solution 2", the load on playlists generation and nDVR content broadcasting is distributed among several video servers running in parallel.

The relevance of the index database, copies of which are stored on the record servers and video servers, is maintained with MongoDB internal mechanisms.

Load Balancing

Uniform and flexible load balancing between multiple video servers or their groups can be provided by the SmartMEDIA Redirector component, which:

● allows you to define server groups and load balancing rules within the group and between groups, depending on the location of the client and the requested content;

● redirects subscriber requests to working servers if one or more source servers fail; ● monitors the servers' performance; ● checks the content availability.

IP addresses of subscriber devices are grouped into several subbands (subnets), each of which is serviced by a group of video servers. Subnets allocation and their service procedure maintenance is performed by the system administrator taking into account the specifics of network organization, number of subscribers, etc. To add new video servers under the SmartMEDIA Redirector control, you must specify them in the configuration file.

The component has two services:

● for protocols based on HTTP (HLS, DASH) — httpRedirector; ● for the RTSP protocol — rtspRedirector.

The simplified diagram of HTTP Redirector workflow is shown below:

– 25 –

SmartMEDIA Redirector accepts content viewing requests from subscriber devices, identifies video servers with the best content access conditions and redirects subscribers to these video servers. When the request is received, Redirector determines the group of video servers (so-called a "farm"), that processes requests from devices on this subnet. Video servers of each group process incoming requests in accordance with the specified load balancing method (see below). In general, Redirector selects the next video server from the group and checks the requested content availability in its repository:

● if the content exists, Redirector forwards the request to the selected video server; ● otherwise Redirector switches to the next video server in the group; ● if none of the servers in the group can process the request, the another most suitable

group is searched; ● if no group can process the request, an error is returned to the subscriber device.

Information about the error causes is recorded in the log.

Selection of the request processing server within the group are performed according to one of the following policies:

● On the first server responded. If some server is underloaded, it will respond faster than others and the probability of its selection increases.

● By content ID. In this case, the content (TV channel, VoD, etc.) is "anchored" to the serving server. This increases the chance of cache using on this server, but may result in uneven load distribution between the servers.

– 26 –

In addition to the balancing policy, a "weight" parameter can be assigned to each server in the group. All other conditions being equal, the request will be redirected to the server with a higher weight.

SmartMEDIA Redirector monitors the status of video servers, periodically sending requests to prevent the request sending to a failed video server. Information about the unavailability of the server or content is cached for a specified time. If the server or content is unavailable, the server is re-requested after a specified time. When the operability of monitored video servers is restored, HTTP Redirector begins to distribute requests taking into account the changed state of video servers.

Using Content Protection Systems (CAS / DRM)

SmartMEDIA can use third-party content protection systems (Verimatrix, Widevine, etc.) to encrypt content before recording to the storage.

The content protection functionality is implemented in the components Conveyor, SmartCETS, vodconvert and cets. CAS/DRM access settings and file encryption parameters are specified in the components' configuration files or in the utilities' command line parameters.

SmartMEDIA encrypts DASH and HLS content with keys retrieved from CAS/DRM systems, records encrypted content, indexes it, and delivers to the client devices. These keys are stored only in the external CAS/DRM system, which transmits them securely to the subscriber device using proprietary protocols.

Depending on the end-user devices used, delivery protocols, media containers, and content security requirements (in particular, the DRM system), content encryption methods may also vary.

Encrypt Live and nDVR Content

HLS (AES-128)

HLS-AES encryption standard is supported for chunks recorded in the MPEG2-TS container. As a DRM server, Verimatrix Adaptive CAS or another server that implements the VCAS Scrambler HTTP protocol can be used.

The content is encrypted using the AES-CBC algorithm with a key length of 128 bits and PKCS#7 padding.

Some SmartTV models and possibly some other subscriber devices decrypt the stream incorrectly: PKCS#7 padding is not discarded, random "garbage" appears at the end of each chunk and that's why the first TS packet of each followed chunk will be decoded incorrectly. As a workaround for this problem, you can enable the insertion of a PAT/PMT table to the beginning of each chunk.

– 27 –

With this workaround applied, the first TS packet in each chunk will always be PAT, it will be decoded incorrectly and rejected by the TS demuxer, but media data will not be affected. For more information about adding PAT/PMT to the beginning of the chunk, see the document "SmartMEDIA. Configuration Reference."

Encryption settings are specified in the drms section of the main SmartMEDIA Conveyor configuration file.

HLS and DASH with the ISOBMFF Container (MP4 Fragmented)

The ISO BMFF container supports ISO Common Encryption standard. SmartMEDIA services can interact with the key server through the SmartLabs UDRM Encryption API, which is an extension of the Widevine Modular DRM Encryption API.

Until the first key for the track is received, the SmartMEDIA Conveyor will skip all media data and won't record the content of this track.

To improve the content security, Conveyor supports the periodic updates of encryption keys (key rotation).

Depending on the requirements of content owners and internal operator's policies, the duration of the period of using one key can be adjusted from one second to the complete rotation disabling. It should be taken into account that the smaller rotation interval will increase security, but, as well, it will increase the load on the key generation servers (on the encryption side) and, especially, the license servers (from the client devices side).

Encryption settings are specified in the drms section of the main SmartMEDIA Conveyor configuration file.

For the key rotation, the entire timeline can be divided into periods of fixed duration, each of these periods has an unique sequence number. The number of periods for which keys are requested within the single request to the key server is specified in the configuration parameter crypto_period_count (the default value is 5); the duration of one period should be defined in crypto_period_length parameter (by default — 1 hour).

Depending on the functionality and settings of the key generation server, the response may contain both the one PSSH for all periods (which in turn contains as many keys as requested), and various PSSHs for each period (each contains only one key).

Google WideVine Modular DRM service supports only PSSH generation with one key per PSSH.

For details about encryption settings, see the document "SmartMEDIA. Configuration Reference."

– 28 –

HLS and RTSP for IPTV Systems (DVB Simulcrypt)

A stream already encrypted using DVB Simulcrypt standards can be injected to the SmartMEDIA Conveyor input. Recording is possible only in the MPEG2-TS container "as is", without remultiplexing and other transformations in the stream.

This method is not standard for the HLS protocol, but it's often used in IPTV systems that historically inherited the DVB CAS.

For more information on setting the "pass through" mode, see the document "SmartMEDIA. Configuration Reference."

Encryption of Multicast MPEG2-TS Content

The SmartCETS service and the cets utility encrypt MPEG-TS content in real time according to ISO/IEC 23001-9 (Common encryption of MPEG-2 transport streams). After receiving the list of input multicast streams, SmartCETS/cets requests encryption keys from the key server using the Common Encryption API for Widevine DRM (hereinafter Widevine API) or SmartLabs UDRM Encryption API. Encrypted content can be either written to a file, or broadcasted to a multicast group using the UDP protocol. For the SmartCETS service, the list of multicast groups of input streams for encryption along with other parameters must be specified in the configuration file. For the cets utility, all parameters are passed by command-line arguments.

The following formats are supported:

● Video: H.262, H.264, H.265; ● Audio: AAC / ADTS, AC3, DTS.

To improve the encryption security of streaming content, the SmartCETS service supports the periodic rotation of encryption keys. It requires using of SmartLabs UDRM Encryption API because of support of multiple keys in one PSSH.

The cets utility does not support the keys rotation.

The encryption and keys rotation settings are similar to the SmartMEDIA Conveyor service for encrypting the ISO BMFF container.

VoD Encryption

Configuring DASH Encryption

Encryption of VoD content is performed by the utility vodconvert. Description of the vodconvert parameters is presented in the document "SmartMEDIA. Configuration Reference." Examples of vodconvert running can be found in Appendix 4.

– 29 –

Performance Optimization

Disk Subsystem

When using storage on local disks or RAID arrays you should paid an attention to I/O subsystem settings. Below are some recommendations that are applicable in most cases.

1. Use the deadline scheduler for all block devices responsible for storing content

You can install the scheduler for the sdX device with the following command:

echo 'deadline' > /sys/block/sdX/queue/scheduler

Do not forget that these settings will be reset and need to be made again after the OS restart. To persist, you can add them to the file /etc/rc.local or create an init.d script.

To configure the scheduler globally, add the elevator=deadline parameter to the kernel configuration string GRUB_CMDLINE_LINUX in the file /etc/default/grub, then rebuild the grub configuration (grub.cfg). For example:

GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="vconsole.font=latarcyrheb-sun16 vconsole.keymap=us rd.lvm.lv=vgroot/root elevator=deadline crashkernel=auto rhgb quiet" GRUB_DISABLE_RECOVERY="true"

You can rebuild the bootloader configuration with these commands:

● on servers with BIOS: # grub2-mkconfig -o /boot/grub2/grub.cfg ● on servers with UEFI (RHEL7): # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg ● on servers with UEFI (CentOS7): # grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

In RHEL7/CentOS7, deadline is the default scheduler for all block devices except SATA disks.

– 30 –

2. Ensure that you have enough RAM

The amount of RAM should be sufficient to accommodate the "hot" data of OS's page cache. As you know, the amount of memory reserved for PageCache can be recognized using the command free (column "buff/cache" or "cache", depending on OS version):

# free -m total used free shared buff/cache available Mem: 257756 5999 2003 241580 249753 8852 Swap: 0 0 0

Zero or low values (in conjunction with the zero value of the "free" column) can be the sign of low memory and associated low performance.

Avoid using swap. If it cannot be disabled, reduce the probability of using it:

# sysctl vm.swappiness=0 echo "vm.swappiness=0" >> /etc/sysctl.conf

3. Select the appropriate readahead values

Read Ahead can increase the speed of data access by reading multiple blocks in a row instead of reading one requested data block. Reading is done into a PageCache (we do not consider the case with the O_DIRECT flag set, because it is inappropriate for the expected load), so the query of the next block will not lead to an I/O operation, but will be processed from the RAM.

Setting too small readahead values will increase the number of IOPS for the same number of read requests. Setting too large values can result in "knocking out" the necessary data from PageCache before it was used, by unnecessary blocks (those that will not be read or will be read after they will be "knocked out" from the cache by another request).

The readahead value can be applied to a block device. You can find out the current value for the sdX device as follows:

blockdev -getra /dev/sdX

Set the value for the sdX device:

blockdev -setra N /dev/sdX

where N is the number of sectors (512 bytes).

– 31 –

Do not forget that these settings will be reset and need to be reinstalled after the OS restart. You can persist them by adding to the /etc/rc.local file or creating an init.d script.

4. Use the optimal file system settings

Disable files access time modification (atime) and write boundaries:

mount -o remount,noatime,nobarrier /your/mountpoint

Do not forget to make the appropriate changes to /etc/fstab.

If you are using the XFS file system over a RAID array, optimize the file system settings for the RAID array you are using. In particular, set the values sunit, swidth, agcount, agsize according to the number of disks and stripe size of your array. More information can be found at http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance.

The XFS file system allows you to put the log on a separate partition. This can lead to increased performance (if you are using a solid-state drive (SSD) for logging tasks), or reduce it (if one device serves the logs of several file systems, but it's performance is not enough).

Note that most of the parameters that optimize XFS performance can only be set at the time the filesystem is created and can not be overridden at the time of mounting.

For more information about XFS, see man mkfs.xfs and man mount.

5. Use WriteBack for hardware RAID controllers with a backup battery (BBU)

WriteBack allows you to significantly increase recording performance by combining several small write operations into single one. The use of WriteBack is especially important for RAID5/RAID6 arrays, in which the checksum also must be recalculated and rewritten to write one block to one physical disk.

DO NOT USE WRITE-BACK MODE WITHOUT BACKUP BATTERY OR BATTERY FAILURE! THIS MAY LEAD TO DATA LOSS!

6. Align the HDD partitions

Sometimes, when using discs with Advanced Format, you need to align the partitions so that the partition beginning coincides with the physical sector beginning, otherwise performance degradation is possible.

To check whether partitions are created correctly, you can use the parted utility:

– 32 –

root@smartmedia:~# parted /dev/sdb "align-check" alignment type(min/opt) [optimal]/minimal? Partition number? 1 1 aligned

To automatically align the partition at the creation stage, pass the -a optimal option to the parted utility:

root@smartmedia:~# parted -a optimal /dev/sdb GNU Parted 3.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted)

For more details about parted, see man parted or https://www.gnu.org/software/parted/manual/html_node/index.html.

7. Monitor the hardware

The smartctl system utility and the smartd service allow to monitor the state of the block devices: the number of HDD corrupted blocks, the SSD health, etc.

smartctl can be used both for controlling single disks, and, in most cases, for controlling disks in a hardware RAID array.

Network Subsystem

1. Use the manufacturer's recommended drivers and firmware for network adapters

Often, the low performance of the network subsystem is caused by drivers and low-level firmware of network adapters (NIC). Use recommended versions of drivers and firmware (as a rule, these are the latest public versions available).

2. Reduce the number of interrupts (IRQs) and distribute interrupts across multiple cores

Most of modern network adapters (NICs) allow you to configure the frequency of interrupts and the number of send (Tx) and receive (Rx) queues depending on the number of CPU cores. In some cases, the number of queues is set automatically, but for some adapters this value must be set manually when the NIC driver is loaded.

– 33 –

Use the irqbalance service to distribute interrupts across various OS kernels. Do not allow a large number of interrupts to be concentrated on one core.

In some cases, irqbalance redistributes interrupts too often, which can result in more than necessary NUMA memory accesses and performance degradation. In this case, it may make sense to stop the irqbalance service or manually redistribute the interrupts without it.

To find out the current interrupt distribution by CPU cores (the total number of interrupts processed by each kernel), you can run the following command:

cat / proc / interrupts

Unfortunately, the Linux OS does not allow to reset the interrupt counters after the kernel interrupts are redistributed.

3. Use the hardware acceleration capabilities of network adapters

In most cases, TCP Offloading and UDP Offloading allow to reduce CPU load while calculating TCP/UDP checksums and their defragmentation, as well as the number of interrupts, which also reduces the CPU load.

4. Optimize the OS network stack

Linux provides a wide range of network subsystem settings via procfs or using the sysctl utility.

For more information about OS network stack settings, see Appendix 2.

5. Choose the best congestion control algorithm for TCP

The Linux network stack can use various congestion control algorithms for TCP. The list of algorithms loaded into the kernel and the currently used algorithm can be obtained using the sysctl utility:

# sysctl net.ipv4.tcp_available_congestion_control net.ipv4.tcp_available_congestion_control = cubic reno # sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_congestion_control = cubic

The list of available modules in the system can be obtained as follows:

# find /lib/modules/`uname -r`/kernel/net/ipv4 -name tcp_* /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_diag.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_hybla.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_westwood.ko

– 34 –

/lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_illinois.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_lp.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_veno.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_highspeed.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_dctcp.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_scalable.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_vegas.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_yeah.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_bic.ko /lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/tcp_htcp.ko

You can load the module to the system using the modprobe command:

# modprobe tcp_yeah You have mail in /var/spool/mail/root # lsmod | grep yeah tcp_yeah 12635 0 tcp_vegas 13839 1 tcp_yeah

Depending on the characteristics of the data transmission channel (RTT, packet loss, throughput), different algorithms will show different results. For example, mechanisms that work well at low losses (for example, cubic) can give unsatisfactory results at high losses (> 3–5%) and large RTT (> 100ms), and vice versa.

According to our observations, the yeah algorithm is suitable for most circumstances, but we recommend you to conduct your own testing and choose the best algorithm for your conditions.

HTTP Server (nginx)

1. Turn on the use of sendfile

Using the sendfile system call allows to significantly reduce the number of memory copying operations, reduce the CPU load and increase the overall system performance.

When using HTTPS or data compression (gzip), sendfile is not used, even if the appropriate setting is present (because the data must be encrypted and / or compressed before sending to the socket).

To enable sendfile globally, in the http section of the nginx configuration file, specify:

http { sendfile on; }

The sendfile system call allows you to initiate the data transfer from a file descriptor to a TCP socket (or other non-blocking socket). The call will be terminated either after sending all the necessary data, or after the socket buffer is full and it becomes not ready for transmitting.

– 35 –

For very fast connections, this can mean that the blocking sendfile call will be completed only after the whole file is transferred. To avoid this, specify the maximum amount of data nginx will try to send per one sendfile call:

http { sendfile_max_chunk 131072; }

Decreasing sendfile_max_chunk can lead to too many reads (can be partially compensated by read ahead, see above), increasing — to too long blocking of the primary execution thread or worker thread (see below) to send data.

2. Enable AIO threads on servers that perform blocking I/O

All read and write operations on block devices (disks, RAID arrays, SSDs, etc.) in Linux are blocking — the main execution thread goes into the D-state and waits for the operation to complete. In the case of nginx, when one thread serves a large number of client requests, this leads to the blocking of all requests by single I/O intensive query.

To prevent the thread from blocking by the I/O task, Linux supports asynchronous operations (AIO). But using AIO requires, at first, disabling the use of PageCache (operations must be performed with the O_DIRECT flag) and taking into account a number of other shortcomings.

To work around the problem of using the system AIO nginx, since version 1.7.11, supports the use of a thread pool to handle I/O operations. All blocking operations can be transferred to a separate thread pool, which eliminates the blocking of the main execution thread.

To enable the thread pool:

1. Configure the thread pool in the main context of the configuration file, for example:

thread_pool iopool threads=32 max_queue=65536;

2. Enable AIO for the desired location:

location /video/ { sendfile on; aio threads=iopool; }

AIO threads does not make sense to include on locations that are reading from tmpfs (if swap is not used).

The expediency of using AIO threads on proxying data locations (without caching) depends on the probability of recording this data to a "slow" device (disk / SSD) — respectively, on the data type and the total system load.

– 36 –

3. Select a sufficient number of workers

Typically, the optimal number of workers is equal to the number of cores or number of threads (in case of HyperThreading) CPU.

Increasing the number of workers makes sense only if the workers go into the I/O standby mode (the so-called "D-state"), but in this case, it is usually better to use the thread pools.

4. Optimize the nginx service performance

Nginx has a large number of configuration options, allowing you to configure the interaction with a network stack, OS, memory for specific loads.

The basic settings can be:

timer_resolution 1ms; events { worker_connections 10240; use epoll; } http { keepalive_timeout 60; tcp_nodelay on; tcp_nopush on; }

Solving the main tasks

Live Streams Recording

1. Prepare the source streams

The streams in the MPEG2-TS container delivered by one of the following protocols can be used as the source streams for the SmartMEDIA server:

● UDP over IP Multicast (without RTP encapsulation); ● HTTP Stream (SmartMEDIA acts as an HTTP client); ● HLS according to draft-pantos-http-live-streaming-07, without encryption.

– 37 –

If you do not have streams that meet these requirements, you can create test streams in any available way:

● Broadcast an existing TS file using the tsplay utility (included in the tstools package). For example:

tsplay -loop test.ts 239.65.9.1:5000

● Broadcast a static image using ffmpeg. For example:

ffmpeg -loop 1 -i teaser_1920x1080.jpg -f lavfi -i "sine=frequency=1000" -c:v libx264 -pix_fmt yuv420p -b:v 1000k -g 75 -r 25 -c:a aac -ac 2 -ar 48000 -ab 128k -f mpegts "udp://@239.65.9.13:5000?pkt_size=1316&buffer_size=131070&reuse=1&ttl=10"

● Broadcast existent TS file using the service smartMPicker. ● Any other way you prefer.

2. Determine the storage type and make the appropriate settings

Local POSIX Storage

POSIX storage should be available on the SmartMEDIA Conveyor server as a local mount point. The /video directory is a standard directory for the SmartMEDIA Conveyor server used in the basic configuration.

In the main Conveyor service configuration file, add the POSIX storage section (the configuration for /video is already present in the basic configuration) to the data_storages section:

"data_storages": [ { "id": "fs_hdd", "fs": { "path": "/video" } }, { "id": "fs_tmpfs", "fs": { "path": "/video-tmpfs" } } ]

– 38 –

Object Storage with Access Using S3 protocol

You will need to find out the following information for configuration:

● storage HTTP URL, ● bucket identifier, ● the access key, ● the access key identifier.

In the main Conveyor service configuration file, add the S3 storage section to the data_storages section:

"data_storages": [ { "id": "s3", "s3": { "host": "192.168.2.12:8080", "bucket": "test", "access_key_id": "60ZXGDH7Z8L2K86L6AJ3", "access_key": "Bnc19iicY61Lc8Y9x3zbqB7FmW89EokGCJ3p9FrO", "authorization_version": "v4", "max_connections": 50 } } ]

3. Make other settings for the service

If you plan to use the encryption, add the drms section with the DRM-specific configurations in the main Conveyor configuration file:

"drms": [ { "id": "verimatrix", "verimatrix": { "address": "192.168.32.32", "port": 12684, "key_update_period_sec": 1800 } }, { "id": "widevine_test", "cenc": { "url": "http://10.10.10.10/cenc/getkey", "signer_name": "conveyor", "signer_key":

– 39 –

"1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9", "signer_iv": "d58ce954203b7c9a9a9d467f59839249", "crypto_period_length": 0, "crypto_period_count": 0, "key_server_type": "udrm" } } ]

4. Define a channel and configure the basic properties of the channel

In the Conveyor channel configuration file, in the channels section, add the channel element:

{ "id": "CH_TEST1", "container_type": "mp4", "index_storage": "mongo_local", "data_storages": [ "fs_hdd" ], "recording_window": "1d" }

5. Set the source for the recording channel

In the Conveyor channel configuration file, add the sources and demuxers sections to the configuration of the created channel:

"sources": [ { "id": "ts_multicast_1", "url": "udp://239.255.0.55:1136" } ], "demuxers": [ { "id": "dmx1", "source": "ts_multicast_1", "tracks": [ "vid0", "aud0", "aud1" ] } ]

– 40 –

6. Set the transcoding of incoming streams (if necessary)

To support transcoding, the Conveyor service must be installed on equipment supporting the Intel QSV technology. For details, see "Requirements for transcoding servers".

In the Conveyor channel configuration file , add video_transcoders and audio_transcoders sections to the channel:

"video_transcoders": [ { "id": "vtrc", "source": "dmx1:vid0", "use_qsv": true, "out": [ { "bitrate": "2800K", "gop_size": "60", "width": "1280", "height": "720" }, { "bitrate": "2200k", "gop_size": "60", "width": "1280", "height": "720" }, { "bitrate": "1300k", "gop_size": "60", "width": "720", "height": "576" } ] } ], "audio_transcoders": [ { "id": "atrc_eng", "source": "dmx1:aud0", "out": [ { "bitrate": "384k" } ] }, { "id": "atrc_afr", "source": "dmx1:aud1",

– 41 –

"out": [ { "bitrate": "128k" } ] } ]

7. Configure the stream recording

In the Conveyor channel configuration file , add the muxers section to the channel:

"muxers": [ { "bw":"3000000", "dir":"video\/bw3000000", "sources":[ { "master":"true", "source":"vtrc:vid0" } ] }, { "bw":"2200000", "dir":"video\/bw2200000", "sources":[ { "source":"vtrc:vid1" } ] }, { "bw":"1300000", "dir":"video\/bw1300000", "sources":[ { "source":"vtrc:vid2" } ] }, { "bw":"384000", "dir":"audio\/eng\/aac", "lang":"en", "sources":[ {

– 42 –

"source":"atrc_eng:aud0" } ] }, { "bw":"128000", "dir":"audio\/afr\/aac", "lang":"af", "sources":[ { "source":"atrc_afr:aud0" } ] } ]

8. Configure the encryption (if necessary)

In the Conveyor channel configuration file in the channel muxers section, add the encryptor section:

"encryptor" : { "drm" : "widevine_test" }

9. Check out the result

If you use the MP4 container, the recorded content will be available as DASH and as HLS with the mp4 container. The URL for the channel playback in this case will look like:

● for DASH: http://streamer1.example.com/dash/CH_TEST1/playlist.mpd ● for HLS: http://streamer1.example.com/hls/CH_TEST1/master.m3u8

When using the MPEG2 TS container, the recorded content will be available only as HLS: http://streamer1.example.com/hls/CH_TEST_TS/variant.m3u8

You can check the playback on any player supporting the corresponding protocol. For example:

● for DASH — Shaka Player (https://shaka-player-demo.appspot.com/demo/). Please note that the security settings of modern browsers do not allow the use of HTTP and HTTPS on the same page at the same time, therefore, the channel playback via HTTP or with an untrusted / expired certificate may not be available without additional browser settings;

● for HLS — http://osmfhls.kutu.ru/ or any Apple device (iPad, iPhone, Mac). To play a channel on Apple devices, just open its URL in the Safari browser.

– 43 –

Preparing VoD Content

1. Prepare the original content

To prepare VoD content, the original content must be in one of the following containers: MKV, TS, MP4. The correspondence of containers to supported codecs is given in the table:

Container Video Codec Audio Codec

MKV H.264 AAC

TS H.262, H.264, H.265 AAC, AC3, MP3, DTS

MP4/ISO BMFF H.264 AAC, AC3

DTS - DTS

If the source files don't meet these requirements or you want to prepare adaptive content from the master copy, the content must be pre-transcoded (for example, using ffmpeg — see Appendix 3).

2. Convert the content to the required format

Once you have prepared the content with the necessary profiles, you need to prepare the playlists and segment content using the vodconvert utility (if necessary).

Examples of running the vodconvert utility for preparing VoD content can be found in Appendix 4.

Redundancy and Load Balancing with HTTP Redirector

The HTTP Redirector service is designed to provide redundancy of streaming servers, balance the load between them and optimize traffic flows from the platform to subscribers. It allows you to configure server groups (farms) and the weight of each server within the group. Also you can assign your own balancing policy for the group.

The redirections section within the farms array defines farms with their names (identifiers) and servers list.

The locations array defines subnets lists and a farms list in the order in which the content and servers availability will be polled.

An example of the configuration with requests distribution between sites is given below:

{ "core":{ "port":9201,

– 44 –

"interface":"0.0.0.0" }, "redirection":{ "vod_range_cache_time_sec":20, "request_timeout_sec":5.0, "channel_regex":"([^/]+)/.*", "server_status_cache_time_sec":30.0, "uri_path":"/", "live_cache_time_sec":10, "clear_cache_interval_sec":60, "channel_regex_index":1, "range_cache_time_fraction":0.05, "farms":[ { "name":"site6", "servers":[ { "url":"http://10.72.10.18:80/" }, { "url":"http://10.72.10.110:80/" } ] }, { "name":"site1", "servers":[ { "url":"http://10.72.10.50:80/" }, { "url":"http://10.72.10.106:80/" } ] }, { "name":"default", "servers":[ { "url":"http://10.72.5.4:80/" }, { "url":"http://10.72.5.5:80/" }, { "url":"http://10.72.5.8:80/" } ] } ], "locations":[ { "farms":[ "site6", "default" ], "uri_regex":".*", "ip_mask":[ "172.21.128.0/17", "172.22.0.0/17", "172.27.0.0/20" ] },

– 45 –

{ "farms":[ "site1", "default" ], "uri_regex":".*", "ip_mask":[ "172.24.112.0/20", "172.24.192.0/19", "172.25.0.0/18" ] }, { "farms":[ "default" ], "uri_regex":".*", "ip_mask":[ "10.65.1.97/32", "172.16.0.0/12", "0.0.0.0/0" ] } ] } }

Protecting Multicast Streams (CETS)

In the global section of the SmartCETS configuration file, you'll need to set the values for key_server_url, signer_name, signer_key, signer_iv. Then, in the streams section, list content IDs, source URLs and destination URLs in udp://GROUP:PORT format .

Example:

{ "key_server_url": "http://10.10.10.10/cenc/getkey", "signer_name": "cets", "signer_key": "1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9", "signer_iv": "d58ce954203b7c9a9a9d467f59839249", "crypto_period_length": "24h", "crypto_period_count": "3", "key_request_policy": "each_key_expires", "smooth_transmission_by_pcr": "true", "packets_between_pssh": 200, "msec_between_audio_ecm": 50, "streams": [ { "content_id_string": "R24.ism", "source_url": "udp://239.65.40.4:5001", "sink_url": "udp://239.65.70.4:5001" }, { "content_id_string": "CH_7_145_LOCAL", "source_url": "udp://239.253.7.145:1234", "sink_url": "udp://239.253.8.145:1234" },

– 46 –

{ "content_id_string": "CH_7_96_LOCAL", "source_url": "udp://239.253.7.96:1234", "sink_url": "udp://239.253.8.96:1234" } ] }

Appendix 1. Supported protocols

HLS

The HLS (HTTP Live Streaming) protocol, developed by Apple, allows Live and VoD content to be delivered over the HTTP protocol. Currently standardized as RFC8216.

The protocol is supported by a large number of devices: set-top boxes (STB), SmartTV, different browsers (some have built-in support, some with JavaScript players), etc. But the standard implementation of the protocol can be considered Apple products — QuickTime Player, Safari web-browser, AVPlayer (a part of AVFoundationFramework), etc.

The basic principle of the HLS protocol is to segment the stream into small fragments (chunks), subsequently downloaded via the HTTP protocol. Using the HTTP protocol allows the stream to pass firewalls or proxy servers (as opposed to UDP-based protocols such as RTSP). The standard HTTP caches and HTTP-based content delivery networks can be used with HLS without any specialized software for video delivery.

At the beginning of the session, the device requests a master playlist containing metadata about the available streams (the so-called "variants") and their characteristics: the type of tracks (audio, video, subtitles), bitrate, resolution, language, etc. After retrieving a list of available variants from the main playlist, the client device, based on the bandwidth and user preferences, selects the appropriate streams and requests media playlists for each stream containing media URLs.

When bandwidth changed or user choose another playback quality settings, the player can select another stream and, if possible, smoothly switch to it. As a rule, the player requires synchronization of key frames of video tracks and correspondence of chunks between all variants for smooth switching.

All playlists has the m3u8 format.

The main playlist can contain:

● list of content variants depending on the bitrate (tag #EXT-X-STREAM-INF); ● list of alternative tracks: audio, video, subtitles, Closed Captions (tag #EXT-X-MEDIA); ● list of playlists for fast playback: the so-called "trickplay" (tag #EXT-X-I-FRAME-STREAM-INF),

and other information.

Note, that the HLS standard does not support the changing of variants set on the fly. Many players request the master playlist once and never update it.

– 47 –

The simplest master playlist for content with one "variant" may look like this:

#EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1492102,CODECS="avc1.66.31, mp4a.40.2",RESOLUTION=1024x576 playlist.m3u8

The simplest media playlist (playlist.m3u8) for Live-broadcast may look like this:

#EXTM3U #EXT-X-VERSION:3 #EXT-X-ALLOW-CACHE:NO #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:26256 #EXTINF:10.0, chunk_1.ts #EXTINF:10.0, chunk_2.ts #EXTINF:10.0, chunk_3.ts

Support for multiple audio and subtitle languages in HLS can be implemented in two ways:

• In accordance with RFC8216: for each language, there is a individual set of chunks and a individual playlist specified in the tag #EXT-X-MEDIA. So, alternative tracks are downloaded only when they are needed. As a rule, in this case, "main" data chunks contains only video and no other tracks — audio or subtitles (for the M4F container this is the only possible option).

• In accordance with the MPEG2-TS standard: inside single TS-chunk. Each chunk contains a video track and a full set of audio tracks and subtitles. This method is not supported by all devices and is less preferred, because the subscriber device downloads all tracks, even when they are not needed.

The playlist with multiple audio tracks may look like this:

#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3000000,AUDIO="aud0",CODECS="avc1.640028",RESOLUTION=1280x720 video/bw3000000/mp4_playlist.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2200000,AUDIO="aud0",CODECS="avc1.640028",RESOLUTION=1280x720 video/bw2200000/mp4_playlist.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1300000,AUDIO="aud0",CODECS="avc1.64001F",RESOLUTION=720x576 video/bw1300000/mp4_playlist.m3u8 #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud0",NAME="English",LANGUAGE="en",DEFAULT=YES,AUTOSELECT=YES,URI="audio/eng/aac/mp4_playlist.m3u8"

– 48 –

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud0",NAME="African",LANGUAGE="af",DEFAULT=NO,AUTOSELECT=NO,URI="audio/afr/aac/mp4_playlist.m3u8"

Only MPEG2-TS container was supported on Apple devices until 2016; in 2016 (iOS10 and later, macOS 10.12 and later, tvOS10 and later), support for the M4F container was added. The protocol does not impose limitations on codecs. The H.265 codec is supported on iOS11 devices (only with M4F container) since 2017.

MPEG-DASH

The MPEG-DASH (Dynamic Adaptive Streaming over HTTP) protocol, just like HLS, is based on the HTTP protocol and segmenting of a continuous stream into chunks. Unlike HLS, it was developed by the MPEG group and standardized as an ISO standard in 2012. The current standard version is ISO/IEC 23009-1:2014. In addition to the main standard, there are several sets of recommendations. The most known is DASH-IF Interoperability Points (DASH-IF-IOP). At the time of writing, the current version of DASH-IF-OP is v4.0.

All information about media data (the list of chunks, their duration, events in the stream, etc.) is contained in an XML file called Media Presentation Descriptor (MPD).

MPD has a multi-level nested structure. The simplified representation of this structure can be demonstrated as follows:

The MPEG-DASH standard involves dividing a continuous stream into periods. In the simplest case, there will be only one period, but in some cases (they will be considered below) there will be several periods.

– 49 –

Each period includes one or more so-called AdaptationSets. AdaptationSet, in turn, contains one or more Representations.

In fact, AdaptationSet is a "logical" stream: one video track, one audio track in a specific language, one subtitle track, etc. If the content contains several audio tracks (in different languages, for example), there will be several AdaptationSets. If several video tracks (for example, from different view angles or with different localisations), there also will be several AdaptationSets.

The stream, that will be delivered to the client, is a Representation. For example, if the video stream is encoded in several bitrates and the player can automatically switch between them — these are several Representations within the same AdaptationSet.

The composition of AdaptationSets within the period and the composition of Representations within the AdaptationSet of one period can not be changed. If such changes are necessary, the media server must generate a new period with a new set of AdaptationSets and Representations. In this case, Representation's identifiers representing the same stream must be preserved.

Representation contains information about the media chunks list. Unlike HLS, where each chunk should be explicitly listed in a playlist, MPEG-DASH has three options for content encoding and enumerating chunks in MPD:

● SegmentBase. Suitable only for VoD content. The list of chunks should be contained in the sidx atom of the MP4 container. Only the sidx atom URL is described in MPD. If the content is encoded in the MP4 container, the sidx atom is usually placed in the same file, and sidx is specified by ByteRange in the indexRange attribute:

<Representation id="rep1" width="1280" height="720" mimeType="video/mp4" codecs="avc1.64001F" bandwidth="1743409"> <BaseURL>vid_bw2500000.mp4</BaseURL> <SegmentBase timescale="10000000" indexRange="685-2192"> <Initialization range="0-684"/> </SegmentBase> </Representation>

In this example, there is a so-called Initialization Data (atom moov of the MP4 container) in bytes 0–684, and sidx atom in bytes 685–2192 .

In case the content is encoded in the MPEG2-TS container, the sidx atom may be placed in a separate file:

<Representation id="rep1" bandwidth="5000000" width="1920" height="1080"> <BaseURL>content.ts</BaseURL> <SegmentBase> <RepresentationIndex sourceURL="content.sidx"/> </SegmentBase> </Representation>

– 50 –

● SegmentList. The URL of each chunk is specified explicitly. The duration of each chunk should be the same (in this case it is described in the duration attribute of the SegmentList tag) or described in the SegmentTimeline tag, as in the example below:

<Representation id="0" width="1920" height="1080" mimeType="video/mp4" codecs="avc1.640028" bandwidth="4500000"> <SegmentList timescale="10000000"> <Initialization sourceURL="vid_bw4500000_.mp4"/> <SegmentTimeline> <S d="59999904" r="99"/> </SegmentTimeline> <SegmentURL media="vid_bw4500000_1.mp4"/> <SegmentURL media="vid_bw4500000_2.mp4"/> <SegmentURL media="vid_bw4500000_3.mp4"/> .... <SegmentURL media="vid_bw4500000_100.mp4"/> </SegmentList> </Representation>

● SegmentTemplate. The URL of each chunk is generated from a template containing:

○ Representation's identifier — $ RepresentationID $ ○ Representation's bitrate — $ Bandwidth $ ○ chunk's sequence number (identifier) — $ Number $ ○ chunk's start time — $ Time $

The chunk duration must be either the same or described in the SegmentTimeline tag. If all chunks have the same duration, an "endless" playlist, that describes the Live stream and does not require to update, can be created.

Representation example of the playlist with SegmentTimeline:

<Representation width="1920" height="1080" mimeType="video/mp4" codecs="avc1.640028" id="0" bandwidth="3141752"> <SegmentTemplate timescale="10000000" startNumber="1" media="vid_bw4500000_$Number$.mp4" initialization="vid_bw4500000_.mp4"> <SegmentTimeline> <S d="59999904" r="99"/> </SegmentTimeline> </SegmentTemplate> </Representation>

Example of "endless" playlist's Representation:

<Representation width="1920" height="1080" mimeType="video/mp4" codecs="avc1.640028" id="0" bandwidth="3141752"> <SegmentTemplate duration="59999904" timescale="10000000" startNumber="1" media="vid_bw4500000_$Number$.mp4"

– 51 –

initialization="vid_bw4500000_.mp4" /> </Representation>

The MPEG-DASH standard implies the use of two container types: ISO BMFF and MPEG-2 TS. It does not depend on the audio/video codecs used and supports various content protection schemes. However, de facto, MPEG-DASH implies the use of ISO-BMFF as a container and Common Encryption (CENC) for content protection, allowing you to encrypt and broadcast a single copy of the content to clients supporting different DRM systems.

Smooth Streaming

Smooth Streaming protocol allows to broadcast Live and VoD content similar to HLS and DASH protocols. As the client devices supporting the Smooth Streaming protocol, there may be various Silverlight players and devices, mobile devices running Windows Phone/Windows 10 Mobile and other devices.

Similar to the HLS and MPEG-DASH protocols, the Smooth Streaming protocol uses a fragmented structure of the media stream with HTTP delivery. The content is segmented and packed by the SmartMEDIA server, eliminating the need to install the IIS web server.

Smooth Streaming is the predecessor of the MPEG-DASH protocol and has fewer possibilities (for example, periods are not supported and the chunk duration of all tracks must match).

RTSP

The SmartMEDIA server also supports VoD and Live content streaming to devices supporting Real Time Streaming Protocol (RTSP). It can be software players and hardware devices, including QuickTime player (version 10 and higher), VideoLAN VLC player, various set-top boxes and other devices.

The RTSP protocol is now considered obsolete, and the number of modern RTSP-enabled subscriber devices that do not support HTTP-based delivery (HLS, MPEG-DASH, Smooth Streaming), is steadily decreasing. However, SmartMEDIA supports such devices, allowing you to cover a wider range of devices.

Appendix 2. Linux Network Subsystem Configuring

– 52 –

Calculating the TCP Window for the Bandwidth Delay Product

TCP performance depends on several factors. The two most important are link bandwidth (the speed of packets transmitted over the network), and round-trip time (RTT, the delay between sending a segment and receiving a notification from the node). These two values define so-called Bandwidth Delay Product (BDP).

Knowing network bandwidth and RTT gives you information for BDP calculation. BDP provides an easy way to calculate the optimal (in theory) TCP socket buffer size. If the buffer is too small, the TCP window can't fully open, and it'll limit performance. If it is too large, valuable memory will be spent without reason. If you set the buffer precisely, you'll be able to use all the available bandwidth and minimize resources usage.

BDP calculation can be made this way:

BDP = link_bandwidth × RTT

For example, if your application interacts through a 100Mbps network with RTT 50ms, BDP will be:

100 Mbps × 0.050 sec ÷ 8 = 0.625 MB = 625 KB

Note: BDP is defined in (K/M/G) bytes, while the bandwidth is defined in (K/M/G) bits / sec . To cast types, divide bandwidth by 8, as in example above.

So, your TCP window should be set to BDP (625 KB). But by default the TCP window in Linux 2.6 is 110 KB, which limits the connection bandwidth to ~18 Mbps:

bandwidth = window size ÷ RTT 110 KB ÷ 0.050 × 8 = 17.6 Mbps

TCP/IP Stack Configuration for Linux

A standard Linux installation tries to be optimised for a wide range of applications. This means that the standard distribution may not be optimal for your environment.

GNU/Linux provides a wide range of configurable kernel parameters that you can use for the dynamic operation system configuring for your specific applications. Let's look at some of the most important options that affect the performance of sockets.

Configurable kernel parameters exist inside the /proc virtual file system. Each file in it represents one or more parameters, which can be read by the cat utility or changed with the echo command. You can also use the sysctl utility to configure it.

net.core.rmem_default

– 53 –

Default value: 110592

Unit: byte

Defines the default receive window size; affects only protocols other than TCP. For TCP redefined in tcp_rmem.

The size should be increased for larger BDP values.

Recommendation: set the value equal to the minimum BDP per client (i.e. based on the bandwidth that should be allocated to every client at any load).

net.core.rmem_max

Default value: 110592

Unit: byte

Defines the maximum size of the receive window; affects all protocols, including TCP. For TCP, the maximum value in tcp_rmem cannot exceed the value of this parameter.

The size should be increased for larger BDP values.

Recommendation: set the value equal to the minimum BDP per client (i.e. based on the bandwidth that should be allocated to every client at any load).

net.core.wmem_default

Default value: 110592

Unit: byte

Defines the default transmit window size; affects only protocols other than TCP. For TCP, it is redefined in tcp_wmem.

The size should be increased for larger BDP values.

Recommendation: according to http://www-01.ibm.com/support/knowledgecenter/SSQPD3_2.6.0/com.ibm.wllm.doc/UDPSocketBuffers.html the value is always set to 64 KB (65536). Otherwise, the send buffer ring (TX) overflows may occur.

net.core.wmem_max

Default value: 110592

Unit: byte

Defines the maximum size of the transmit window; affects all protocols, including TCP. For TCP, the maximum value in tcp_wmem cannot exceed the value of this parameter.

The size should be increased for larger BDP values.

Recommendation: set the value to the maximum host BDP (i.e. based on the planned host bandwidth).

net.ipv4.tcp_window_scaling

Default value: 1

– 54 –

Type: boolean (1/0)

Enables window scaling, as defined in RFC1323. It should be enabled to support windows larger than 64 KB.

net.ipv4.tcp_sack

Default value: 1

Type: boolean (1/0)

Activates a selective acknowledgment, which improves performance by selectively confirming packets received out of turn (as a result, the sender retransmits only the missed segments).

May increase the CPU load.

ATTENTION: increases the potentially susceptibility to DoS attacks.

net.ipv4.tcp_fack

Default value: 1

Type: boolean (1/0)

Enables Forward Acknowledgment, which operates with Selective Acknowledgment (SACK) to reduce congestion.

Recommendation: must be enabled when tcp_sack is enabled.

net.ipv4.tcp_timestamps

Default value: 1

Type: boolean (1/0)

Activates RTT calculation in a more accurate way (see RFC1323) than the retransmission interval.

Recommendation: should be enabled to improve performance.

ATTENTION: sometimes causes interaction problems between two hosts. Repeatedly there was a situation when it was impossible to establish an SSH connection to the server when the tcp_timestamps was enabled on the client.

net.ipv4.tcp_mem

Default value: "24576 32768 49152"

Unit: memory page

Defines the memory limit for the TCP stack and memory usage behavior of the TCP stack (total memory usage, regardless of the sockets number).

The value is defined in memory pages (usually 4 KB).

Increase the value for large BDP (but remember, it is defined in memory pages, not in bytes).

The first value is the lower memory usage limit (initially allocated memory capacity). Recommendation: set the value to BDP ÷ PageSize.

– 55 –

The second value is the threshold for the memory multiplex mode, upon which the multiplexing to use the buffer is activated. Recommendation: set the value to BDP × 2 ÷ PageSize.

The third value is the maximum limit. At this level, packets can be skipped to reduce memory usage. Recommendation: set the value to BDP × 4 ÷ PageSize.

net.ipv4.tcp_wmem

Default value: "4096 16384 131072"

Specifies the memory usage for each socket for automatic configuration.

This setting is important for content stream servers, where each client should receive data at a speed not lower than the speed of the audio / video stream.

The first value is the minimum number of bytes allocated for the socket send buffer. Recommendation: set the value to 4 KB.

The second value is the default value (overrides wmem_default), to which the buffer size can grow at low system load. Recommendation: set the value to BDP per client (based on the minimum required speed of delivery to the client).

The third value is the maximum send buffer length (overrides wmem_max, but cannot exceed it). With static soft memory allocation by the SO_RCVBUF socket option this parameter is ignored. Recommendation: set the value to BDP per client (based on the desired speed of delivery to the client).

net.ipv4.tcp_rmem

Default value: "4096 87380 174760"

Same as tcp_wmem, except that it refers to receive buffers for automatic tuning.

net.ipv4.tcp_low_latency

Default value: 0

Type: boolean (1/0)

Allows the TCP/IP stack to prefer a low latency than a higher bandwidth.

Recommendation: should be disabled.

net.ipv4.tcp_westwood

Default value: 0

Type: boolean (1/0)

Activates the sender's congestion control algorithm, which supports estimated throughput values and tries to optimize the full use of bandwidth.

This parameter is also useful for wireless interfaces, since packet losses can be caused not by overloading.

Recommendation: must be enabled for WAN connections.

net.ipv4.tcp_bic

Default value: 1

– 56 –

Type: boolean (1/0)

Activates Binary Increase Congestion for fast, long-distance networks.

Improves the use of links operating at gigabit speeds.

Recommendation: must be enabled for WAN connections.

net.ipv4.tcp_max_orphans

Default value: 65536

An integer value that specifies the maximum number of TCP sockets allowed in the system that are not associated with any user file handle. Each orphan-connection absorbs about 64 Kbytes of unswappable memory.

When the threshold is reached, the orphan-connections are immediately discarded with a warning.

Do not reduce the threshold value (rather increase it in accordance with the system requirements, for example, after additional memory installing).

This threshold helps prevent simple DoS attacks.

net.ipv4.tcp_fin_timeout

Default value: 60

Specifies the remaining time of a socket in the FIN-WAIT-2 state after it is closed by the local side. The partner may never close this connection, so you should close it on your own initiative after the timeout expires. The default timeout is 60 seconds.

The 2.2 series kernels typically used a value of 180 seconds, and you can keep that value, but be aware that you run the risk of wasting a lot of memory to save the half-ruptured dead connections on the high loaded servers. Sockets in the FIN-WAIT-2 state are less dangerous than FIN-WAIT-1, since they absorb no more than 1.5 KB of memory, but they can exist longer.

net.ipv4.tcp_max_syn_backlog

Integer value; determines the maximum number of stored connection requests for which no confirmation was received from the connecting client (i.e., the TCP SYN came from the client, the server answered SYN/ACK, and the next ACK was not received).

If overloads occur on the server, you can try to increase this value.

Too large backlog may be the result of a SYN-flood attack. In this case, the inclusion of SYN cookies will help. If you are sure that the attack is not performed, then you need to increase the value to the appropriate one.

net.ipv4.tcp_synack_retries

Specifies the number of retry attempts to send SYNACK packets for passive TCP connections (resends will be made if the response ACK is not received, which may be the result of a SYN-flood attack).

The number of attempts should not exceed 255.

A value of 5 corresponds to approximately 180 seconds for connection attempts.

– 57 –

net.ipv4.tcp_syncookies

The flag indicating whether to enable the SYN-Cookies mechanism when the SYN-Flood attack is detected.

net.ipv4.tcp_tw_reuse

Default value: 0

Type: boolean (1/0)

In case the tcp_tw_reuse flag is set, sockets in the TIME_WAIT state can be reused before the timeout expires. The kernel will try to eliminate collisions based on the TCP Sequence Number.

Activated tcp_timestamps excludes collisions, but it must be enabled on both sides (server and client).

Recommendation: must be activated on streaming servers.

net.ipv4.tcp_tw_recycle

Default value: 0

Type: boolean (1/0)

In case the tcp_tw_recycle flag is set, the kernel will make assumptions about the possibility to reuse the TIME_WAIT-stated socket basing on the remote host timestamps. The kernel will monitor the last sent timstamp values from remote hosts that "hold" connections in the TIME_WAIT state, and will allow to reuse the socket if the timestamp increases correctly.

At the same time, if the timestamp changed incorrectly (for example, "rolled back"), the SYN packet will be "silently" discarded and the connection will not be established.

Appendix 3. Example of Running FFmpeg for VoD Content

Preparation

The sequence of commands for the source HD content transcoding to 5 video bitrates and 3 audio bitrates is given below (the source file contains 2 audio tracks). To demonstrate the switching between bitrates, each output file will be overlaid with text indicating the resolution and bitrate. It is supposed that:

● the target directory path should be defined in OUTDIR variable; ● the source file path should be defined in the INFILE variable.

mkdir -p $OUTDIR ffmpeg -threads 2 -y -i "$INFILE" -filter_complex "[0:v]split=5[r1][r2][r3][r4][r5]; \ [r1]drawtext=fontfile=DroidSansMono.ttf:text='1920x1080 4500

– 58 –

Kbps':fontcolor=white:shadowcolor=black:shadowx=2:fontsize=24:x=15:y=1026,scale=1920:1080[v1]; \ [r2]drawtext=fontfile=DroidSansMono.ttf:text='1920x1080 3300 Kbps':fontcolor=white:shadowcolor=black:shadowx=2:fontsize=24:x=15:y=1026,scale=1920:1080[v2]; \ [r3]drawtext=fontfile=DroidSansMono.ttf:text='1280x720 2500 Kbps':fontcolor=white:shadowcolor=black:shadowx=2:fontsize=24:x=15:y=1026,scale=1280:720[v3]; \ [r4]drawtext=fontfile=DroidSansMono.ttf:text='720x576 2400 Kbps':fontcolor=white:shadowcolor=black:shadowx=2:fontsize=24:x=15:y=1026,scale=720:576[v4]; \ [r5]drawtext=fontfile=DroidSansMono.ttf:text='720x576 2000 Kbps':fontcolor=white:shadowcolor=black:shadowx=2:fontsize=24:x=15:y=1026,scale=720:576[v5]; \ [0:a:0]asplit=3[ar1][ar2][ar3]; \ [0:a:1]asplit=3[ae1][ae2][ae3]" \ -map [v1] -an -pix_fmt yuv420p -vcodec libx264 -preset medium -b:v 4500k -minrate 4500k -maxrate 4500k -bufsize 4500k -x264-params keyint=72:min-keyint=72:scenecut=0 $OUTDIR/vid_bw4500000.mp4 \ -map [v2] -an -pix_fmt yuv420p -vcodec libx264 -preset medium -b:v 453300k -minrate 3300k -maxrate 3300k -bufsize 3300k -x264-params keyint=72:min-keyint=72:scenecut=0 $OUTDIR/vid_bw3300000.mp4 \ -map [v3] -an -pix_fmt yuv420p -vcodec libx264 -preset medium -b:v 24500k -minrate 2500k -maxrate 2500k -bufsize 2500k -x264-params keyint=72:min-keyint=72:scenecut=0 $OUTDIR/vid_bw2500000.mp4 \ -map [v4] -an -pix_fmt yuv420p -vcodec libx264 -preset medium -b:v 24500k -minrate 2400k -maxrate 2400k -bufsize 2400k -x264-params keyint=72:min-keyint=72:scenecut=0 $OUTDIR/vid_bw2400000.mp4 \ -map [v5] -an -pix_fmt yuv420p -vcodec libx264 -preset medium -b:v 420500k -minrate 2000k -maxrate 2000k -bufsize 2000k -x264-params keyint=72:min-keyint=72:scenecut=0 $OUTDIR/vid_bw2000000.mp4 \ -map [ar1] -vn -strict -2 -acodec aac -ab 192k -ac 2 -ar 48000 $OUTDIR/aud_rus_192.mp4 \ -map [ar2] -vn -strict -2 -acodec aac -ab 128k -ac 2 -ar 48000 $OUTDIR/aud_rus_128.mp4 \ -map [ar3] -vn -strict -2 -acodec aac -ab 96k -ac 2 -ar 48000 $OUTDIR/aud_rus_096.mp4 \ -map [ae1] -vn -strict -2 -acodec aac -ab 192k -ac 2 -ar 48000 $OUTDIR/aud_eng_192.mp4 \ -map [ae2] -vn -strict -2 -acodec aac -ab 128k -ac 2 -ar 48000 $OUTDIR/aud_eng_128.mp4 \ -map [ae3] -vn -strict -2 -acodec aac -ab 96k -ac 2 -ar 48000 $OUTDIR/aud_eng_096.mp4

Note: the text will be rendered with the DroidSansMono.ttf font. This font file shall be present in the current directory. You can also use any other available TTF font.

– 59 –

Subtitlings for VoD must be provided in the WebVTT format. Subtitling files should be copied to the output directory manually (also they need to be specified at the content package stage, see below). To convert SRT subtitlings to WebVTT you can also use ffmpeg:

ffmpeg -i Tears\ of\ Steel\ 2012\ 2160p\ DM\ FLAC\ 5.1\ Hi444PP\ 10-bit\ x264-DON.mkv -map 0:s:0 -vn -an -scodec webvtt subtitles.vtt

The subtitles.vtt file containing the first (in sequence) subtitling track will be created.

Appendix 4. Examples of the Vodconvert Utility Using

Multi-bitrate content in DASH format, Base profile:

/opt/smartlabs/smartmedia/utils/bin/vodconvert --type dash -a input=TOS/vid_bw2000000.mp4,adaptation=0,track=vid0,output=vid_bw2000000.mp4 -a input=TOS/vid_bw2400000.mp4,adaptation=0,track=vid0,output=vid_bw2400000.mp4 -a input=TOS/vid_bw2500000.mp4,adaptation=0,track=vid0,output=vid_bw2500000.mp4 -a input=TOS/vid_bw3300000.mp4,adaptation=0,track=vid0,output=vid_bw3300000.mp4 -a input=TOS/vid_bw4500000.mp4,adaptation=0,track=vid0,output=vid_bw4500000.mp4 -a input=TOS/aud_eng_096.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_096.mp4 -a input=TOS/aud_eng_128.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_128.mp4 -a input=TOS/aud_eng_192.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_192.mp4 -a input=TOS/aud_rus_096.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_096.mp4 -a input=TOS/aud_rus_128.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_128.mp4 -a input=TOS/aud_rus_192.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_192.mp4 -a input=sub_eng.vtt,adaptation=3,track=sub0,output=sub_eng.vtt,lang=en,itype=vtt,mime=text/vtt -a input=sub_esp.vtt,adaptation=4,track=sub0,output=sub_esp.vtt,lang=es,itype=vtt,mime=text/vtt -a input=sub_rus.vtt,adaptation=5,track=sub0,output=sub_rus.vtt,lang=ru,itype=vtt,mime=text/vtt -m dash.mpd -d /video/DTE_DASH_AUTO/TOS_clear_a_base -f 5 -t 10000000 --segmentation Base

Encrypted multi-bitrate content in DASH format, Template profile:

– 60 –

/opt/smartlabs/smartmedia/utils/bin/vodconvert --type dash -a input=TOS/vid_bw2000000.mp4,adaptation=0,track=vid0,output=vid_bw2000000_$Number$.mp4 -a input=TOS/vid_bw2400000.mp4,adaptation=0,track=vid0,output=vid_bw2400000_$Number$.mp4 -a input=TOS/vid_bw2500000.mp4,adaptation=0,track=vid0,output=vid_bw2500000_$Number$.mp4 -a input=TOS/vid_bw3300000.mp4,adaptation=0,track=vid0,output=vid_bw3300000_$Number$.mp4 -a input=TOS/vid_bw4500000.mp4,adaptation=0,track=vid0,output=vid_bw4500000_$Number$.mp4 -a input=TOS/aud_eng_096.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_096_$Number$.mp4 -a input=TOS/aud_eng_128.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_128_$Number$.mp4 -a input=TOS/aud_eng_192.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_192_$Number$.mp4 -a input=TOS/aud_rus_096.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_096_$Number$.mp4 -a input=TOS/aud_rus_128.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_128_$Number$.mp4 -a input=TOS/aud_rus_192.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_192_$Number$.mp4 -a input=sub_eng.vtt,adaptation=3,track=sub0,output=sub_eng.vtt,lang=en,itype=vtt,mime=text/vtt -a input=sub_esp.vtt,adaptation=4,track=sub0,output=sub_esp.vtt,lang=es,itype=vtt,mime=text/vtt -a input=sub_rus.vtt,adaptation=5,track=sub0,output=sub_rus.vtt,lang=ru,itype=vtt,mime=text/vtt -m dash.mpd -d /video/DTE_DASH_AUTO/TOS_skey_a_tmpl -f 5 -t 10000000 --segmentation Template --key_server_url http://10.65.50.167/cenc/getkey --signer_name widevine_test --signer_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9 --signer_iv d58ce954203b7c9a9a9d467f59839249 --content_id 544f535f736b65795f615f746d706c0a --sd_key_only

Encrypted multi-bitrate content in DASH format, List profile:

/opt/smartlabs/smartmedia/utils/bin/vodconvert --type dash -a input=TOS/vid_bw2000000.mp4,adaptation=0,track=vid0,output=vid_bw2000000_$Number$.mp4 -a input=TOS/vid_bw2400000.mp4,adaptation=0,track=vid0,output=vid_bw2400000_$Number$.mp4 -a input=TOS/vid_bw2500000.mp4,adaptation=0,track=vid0,output=vid_bw2500000_$Number$.mp4 -a input=TOS/vid_bw3300000.mp4,adaptation=0,track=vid0,output=vid_bw3300000_$Number$.mp4 -a input=TOS/vid_bw4500000.mp4,adaptation=0,track=vid0,output=vid_bw4500000_$Number

– 61 –

$.mp4 -a input=TOS/aud_eng_096.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_096_$Number$.mp4 -a input=TOS/aud_eng_128.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_128_$Number$.mp4 -a input=TOS/aud_eng_192.mp4,adaptation=1,track=aud0,lang=en,output=aud_eng_192_$Number$.mp4 -a input=TOS/aud_rus_096.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_096_$Number$.mp4 -a input=TOS/aud_rus_128.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_128_$Number$.mp4 -a input=TOS/aud_rus_192.mp4,adaptation=2,track=aud0,lang=ru,output=aud_rus_192_$Number$.mp4 -a input=sub_eng.vtt,adaptation=3,track=sub0,output=sub_eng.vtt,lang=en,itype=vtt,mime=text/vtt -a input=sub_esp.vtt,adaptation=4,track=sub0,output=sub_esp.vtt,lang=es,itype=vtt,mime=text/vtt -a input=sub_rus.vtt,adaptation=5,track=sub0,output=sub_rus.vtt,lang=ru,itype=vtt,mime=text/vtt -m dash.mpd -d /video/DTE_DASH_AUTO/TOS_skey_a_list -f 5 -t 10000000 --segmentation List --key_server_url http://10.65.50.167/cenc/getkey --signer_name widevine_test --signer_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9 --signer_iv d58ce954203b7c9a9a9d467f59839249 --content_id 544f535f736b65795f615f6c6973740a --sd_key_only

Clear single-bitrate content in HLS format:

/opt/smartlabs/smartmedia/utils/bin/vodconvert --type hls -a input=test.ts,bandwidth=6000000,iframes_bandwidth=300000,output=playlist.m3u8,output_iframes=iframes.m3u8 --variant_file_name variant.m3u8

– 62 –