Master ThesisElectrical EngineeringDecember 2012

A Comparision of RTMP and HTTP Protocolswith respect to Packet Loss and Delay Variation

based on QoE

Ramesh Goud Guniganti and Srikanth Ankam

School of ComputingBlekinge Institute of Technology37179 KarlskronaSweden

This thesis is submitted to the School of Computing at Blekinge Instituteof Technology in partial fulfillment of the requirements for the degree ofMaster of Science in Electrical Engineering. The thesis is equivalent to 20weeks of full time studies.

This Master Thesis is typeset using LATEX

Contact Information

Author 1:Ramesh Goud GunigantiAddress: Karlskrona, SwedenE-mail: ramesh493@gmail.comAuthor 2:Srikanth AnkamAddress: Karlskrona, SwedenE-mail: srikanthankam@hotmail.com

University advisor:

Dr. Adrian Popescu, Prof.COM/BTH

School of ComputingBlekinge Institute of Technology371 79 KARLSKRONA, SWEDEN

Internet: www.bth.se/comPhone: +46 455 385000SWEDEN

Abstract

In recent year’s multimedia services like Video-on-Demand (VoD) and mo-bile video streaming, videos for e-learning, video conferencing are growingpredominantly, and the user’s expectations towards the quality video are in-creasing as the technology is developing. There are different video streamingprotocols are used for streaming videos from servers to the client. Recently,Adobe Systems developed Real Time Messaging Protocol (RTMP) (propri-etary) for streaming audio, video and data over the Internet between a Flashplayer and a Media Server. On the other hand, Hypertext Transfer Protocol(HTTP) is a well-known and efficient protocol; it has achieved the popu-larity in multimedia services like VoD. Hence, a qualitative research is tobe performed on comparing the two Transmission Control Protocol (TCP)based protocols, under sustainable network conditions for tracing the QoEresults from acquired User Ratings (UR).

This thesis investigates the quality assets on network parameters overVoD streaming. The study addresses the subjective assessment of RTMP andHTTP streaming protocols, by varying network parameters (like packet lossand delay variation) in a controlled and repeatable environment. The packetloss and delay variation are altered by the network emulator NetEm [1,2] inbetween the server and client. The video collected at the client end are eval-uated by using subjective assessment, MOS (Mean Opinion Score), followingthe International Telecommunication Union (ITU) Recommendations [3].

Based on our results it was found that HTTP is having better ratings,when there are more packet losses compared to RTMP. RTMP accomplishedbetter at minimum loss of packets. However, in the case of delay perfor-mance of HTTP is better than RTMP.

Keywords: HTTP, Mean Opinion Score, Quality of Experience, RTMP,Subjective Assessment, Network Emulator.

i

Acknowledgements

First and foremost, we offer our sincerest gratitude to my supervisor, Prof.Adrian Popescu, who has supported throughout our Thesis with his pa-tience and knowledge. We attribute the level of our Masters degree to hisencouragement and effort and without him this Thesis would not have beencompleted or written.

We also owe our deepest gratitude to Dr. Patrik Arlos for providing usthe experimental test bed and support throughout the Thesis work.

We would like to thank all the video quality assessment survey partici-pants, who have contributed towards survey part of this work. Finally, wewould also like to thank our loved parents for supporting us both morallyand financially. Without their encouragement and motivation we could notable to complete this project. We would also like to thank our friends whohelped us with their valuable suggestions and support.

Ramesh Goud GunigantiSrikanth Ankam

ii

Contents

Abstract i

Acknowledgements ii

Contents iii

List of Figures vi

List of Tables vii

Acronyms viii

Introduction 1

1 Introduction 21.1 Aims and Objectives . . . . . . . . . . . . . . . . . . . . . . . 31.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Expected outcomes . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Research Methodology . . . . . . . . . . . . . . . . . . . . . . 51.5 Outline of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 5

Background 6

2 Background and Related Work 72.1 Flash Player . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 H.264 Video Codec . . . . . . . . . . . . . . . . . . . . . . . . 92.3 FFmpeg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Quality of Experience . . . . . . . . . . . . . . . . . . . . . . 102.5 Subjective Quality Assessment . . . . . . . . . . . . . . . . . 102.6 SSCQE (Single Stimulus Continuity Quality Evaluation) . . 11

iii

Implementation 12

3 Experimental Setup 133.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 133.3 Setup Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3.1 Measurement Point . . . . . . . . . . . . . . . . . . . . 143.3.2 MArC . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.3.3 Consumer . . . . . . . . . . . . . . . . . . . . . . . . . 153.3.4 Wowza Media Server . . . . . . . . . . . . . . . . . . . 153.3.5 Wowza Client . . . . . . . . . . . . . . . . . . . . . . . 153.3.6 NetEm . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.3.7 Packet Loss . . . . . . . . . . . . . . . . . . . . . . . . 163.3.8 Packet Delay Variation . . . . . . . . . . . . . . . . . . 163.3.9 RTMP at Client . . . . . . . . . . . . . . . . . . . . . 173.3.10 HTTP at Client . . . . . . . . . . . . . . . . . . . . . 173.3.11 FFmpeg Encoding . . . . . . . . . . . . . . . . . . . . 173.3.12 FFmpeg Splitting . . . . . . . . . . . . . . . . . . . . . 173.3.13 FFmpeg Grabbing . . . . . . . . . . . . . . . . . . . . 173.3.14 Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.4 Experimental setup and procedure . . . . . . . . . . . . . . . 193.5 Assessment of Videos . . . . . . . . . . . . . . . . . . . . . . . 19

Results 22

4 Results and Discussions 234.1 Mean scores calculations . . . . . . . . . . . . . . . . . . . . . 234.2 Confidence Interval Calculations . . . . . . . . . . . . . . . . 244.3 RTMP and HTTP Packet loss . . . . . . . . . . . . . . . . . . 244.4 RTMP and HTTP Delay Variation . . . . . . . . . . . . . . . 294.5 Validity Threats . . . . . . . . . . . . . . . . . . . . . . . . . 32

Conclusions and Future Work 33

5 Conclusions and Future Work 345.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

iv

Bibliography 36

Appendix 42

A Apendix A 43A.1 Network Configuration . . . . . . . . . . . . . . . . . . . . . . 43A.2 Speed and Duplex settings . . . . . . . . . . . . . . . . . . . . 43A.3 Emulator Commands . . . . . . . . . . . . . . . . . . . . . . . 44

B Apendix B 45B.0.1 BTH HTTP Packet loss ratings . . . . . . . . . . . . . 46B.0.2 BTH RTMP Packet loss ratings . . . . . . . . . . . . . 47B.0.3 Cable car HTTP Packet loss ratings . . . . . . . . . . 48B.0.4 Cable car RTMP Packet loss ratings . . . . . . . . . . 49B.0.5 Big Buck Bunny HTTP Packet loss ratings . . . . . . 50B.0.6 Big Buck Bunny RTMP Packet loss ratings . . . . . . 51B.0.7 BTH HTTP Delay variation ratings . . . . . . . . . . 52B.0.8 BTH RTMP Delay variation ratings . . . . . . . . . . 53B.0.9 Cable car HTTP Delay Variation ratings . . . . . . . . 54B.0.10 Cable car RTMP Delay Variation ratings . . . . . . . 55B.0.11 Big Buck Bunny Delay Variation HTTP ratings . . . . 56B.0.12 Big Buck Bunny Delay Variation RTMP ratings . . . . 57

v

List of Figures

3.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 143.5.1 Login Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.5.2 Assessment tool for video survey . . . . . . . . . . . . . . . . 20

4.3.1 MOS with 95% CI vs. Packet loss for HTTP BTH and RTMPBTH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.3.2 MOS with 95% CI vs. Packet loss for HTTP cablecar andRTMP cablecar . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3.3 MOS with 95% CI vs. Packet loss for HTTP Big Buck Bunnyand RTMP Big Buck Bunny . . . . . . . . . . . . . . . . . . . 26

4.3.4 Standard deviation for HTTP and RTMP packet loss . . . . . 274.4.1 MOS vs. Delay Variation for HTTP BTH and RTMP BTH . 294.4.2 MOS vs. Delay Variation for HTTP Cable car and RTMP

Cable car . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.4.3 MOS vs. Delay Variation for HTTP Big Buck Bunny and

RTMP Big Buck Bunny . . . . . . . . . . . . . . . . . . . . . 314.4.4 Standard deviation for HTTP and RTMP Delay variation . . 31

vi

List of Tables

3.3.1 Description of Videos . . . . . . . . . . . . . . . . . . . . . . . 183.3.2 Characteristics of Videos . . . . . . . . . . . . . . . . . . . . . 183.3.3 ITU-T SCALE OF MEDIA QUALITY IMPAIRMENT . . . 18

4.3.1 MOS Ratings for Packet Loss . . . . . . . . . . . . . . . . . . 284.4.1 MOS Ratings for Packet Delay Variation . . . . . . . . . . . . 32

B.0.1Packet loss user ratings on BTH HTTP Video . . . . . . . . . 46B.0.2Packet loss user ratings on BTH RTMP Video . . . . . . . . . 47B.0.3Packet loss user ratings on Cable car HTTP Video . . . . . . 48B.0.4Packet loss user ratings on Cable car RTMP Video . . . . . . 49B.0.5Packet loss user ratings on Big Buck Bunny HTTP Video . . 50B.0.6Packet loss user ratings on Big Buck Bunny RTMP Video . . 51B.0.7Delay variation user ratings on BTH HTTP Video . . . . . . 52B.0.8Delay Variation user ratings on BTH RTMP Video . . . . . . 53B.0.9Delay Variation user ratings on Cable car HTTP Video . . . . 54B.0.10Delay Variation user ratings on Cable car RTMP Video . . . 55B.0.11Delay Variation user ratings on Big Buck Bunny HTTP Video 56B.0.12Delay Variation user ratings on Big Buck Bunny RTMP Video 57

vii

Acronyms

3GPP 3rd Generation Partnership Project

AVC Advance Video Codec

BTH Blekinge Tekniska Högskola

CGA Colour Graphics Adapter

CI Confidence Interval

DAG Data Acquisition and Generation

DSCQS Double Stimulus Continuous Quality Scale

DPMI Distributed Passive Measurement Infrastructure

EGA Enhanced Graphic Adapter

FPS Frames Per Second

GPS Global Positioning System

HAS HTTP Adaptive Streaming

HTTP Hypertext Transfer Protocol

IP Internet Protocol

IPTV Internet Protocol TeleVision

ISO International Organization for Standard

IEC International Electro technical Commission

ITU-R International Telecommunication Union, Radio Communication Sec-tor

ITU-T International Telecommunication Union, Telecommunication Stan-dardization Sector

JVT Joint Video Team

viii

LTE Long Term Evolution

MArC Measurement Area Controller

MOS Mean Opinion Score

MP Measurement Point

MPEG Moving Pictures Expert Group

PL Packet Loss

PDV Packet Delay Variation

PEVQ Perceptual Evaluation Video of Quality

PQoS Perceived Quality of Service

PSNR Peak Signal-to-Noise Ratio

QoE Quality of Experience

QoS Quality of Service

QoP Quality of Presentation

QoD Quality of Delivery

RTMP Real Time Messaging Protocol

RTP Real-Time Transport Protocol

RTSP Real Time Streaming Protocol

SP Service Providers

SS Single Stimulus

SSCQE Single Stimulus Continuous Quality Evaluation

TC Traffic Control

TCP Transport Control Protocol

TS Traffic Shaper

UR User Rating

VGA Video Graphics Array

VoD Video-on-Demand

VCEG Video Coding Expers Group

VQEG Video Quality Expert Group

WMS Wowza Media Server

ix

Introduction

1

Chapter 1

Introduction

In recent years, multimedia technology has been widely developed. In thevideo streaming there is enormous development and research in the field ofstreaming protocols like RTSP (Real Time Streaming Protocol), RTP (Real-Time Transport Protocol), HTTP and RTMP . However, these protocolsperformance may vary in streaming due to traffic influencing parameterssuch as packet loss and delays in the network. Because of this, choosing anappropriate video streaming protocol becomes challenging task. Using anefficient protocol in video streaming would improve the Quality of Service(QoS) when there is large data transmission over the network. Recently,RTMP protocol was designed by Adobe for video streaming. HTTP is wellknown protocol for video streaming now-a-days [4].

To stream a video over a network, protocols are required. In a case,when a client wants to view the video client sends a request by using aprotocol, when the servers receives request from the client it starts sendingthe packets. To stream videos over any network, many protocols are availablebut an appropriate protocol is to be chosen.

RTMP refers to the proprietary protocol developed by Adobe Systemsfor streaming audio, video, and data over the Internet between a Flash playerand a Flash Media Server. Like RTSP, RTMP is an example of a traditionalstreaming protocol, though it is only one of many versions of streamingprotocols for the web. RTMP is defined as a stateful protocol, meaningthat from the first time a client connects until the time it disconnects, thestreaming server keeps track of the clients actions. The client communicatesits actions, or “states”, to the server by issuing commands such as PLAY orPAUSE. When a command between the client and the server is established,the server begins sending the media as a steady stream of small informa-tion packets. This behavior continues and repeats until the server or playerclient closes the session [4, 5]. RTMP was designed for high-performancetransmission such as audio, video, and data between Adobe Flash Platformtechnologies, including Adobe Flash Player [6].

2

CHAPTER 1. INTRODUCTION 3

HTTP refers to the protocol used to deliver webpages images and videoscross the Internet worldwide. HTTP is an adopted, open standard the mostubiquitous mode of online delivery. It is a “stateless” protocol think of it asan airline ticket to anywhere. It can be delivered by a variety of web servers,both commercial and open source [4].

When the network factors such as packet delay variation (PDV)and packet loss (PL) are in the network, the quality of the video is affectedby the video artifacts such as blockiness, blur jerkiness, freezes affects thestreaming video. When these artifacts deploy on the video quality degradesdue to the network behaviour. In user point of view, the user will not showinterest in knowing about the video codec or video bit rate while experiencingthe streaming video in VoD or Live. The end-user considers the quality ofvideo instead of Quality of Service.

This work focuses on Quality of video presenting to the end-user whilestreaming over protocols like HTTP and RTMP. To continue our experi-mentation, VoD streaming has been done on RTMP and HTTP by varyingpacket loss and delay variation with network emulator NetEm [1, 2] in be-tween the server and client. Wowza server was used on the server side, anda flash player is embedded in a web page at the client side. The videos arestreamed over RTMP and HTTP protocols from server to client. FFmpegis used at client side to record the video sequences, that are being streamedfrom the server.

In our thesis, we are investigating the impact of network parameters suchas packet loss and packet delay variation for both RTMP and HTTP proto-cols by streaming three different video sequences over a controlled network.In the quality of video assessment by human subjects, the subjective ratingswere obtained for the resulting test sequences using SSCQE (Single Stim-ulus Continuous Quality Evaluation) defined by ITU-R RecommendationBT.500-11 [3]. Similarly, Mean Opinion Score (MOS) quality assessmentover videos is conducted with No-reference metric design [7].

1.1 Aims and Objectives

The aim of this research is to investigate the performance of the RTMP andHTTP protocols for video streaming, and also to provide recommendationson suitable protocol.

The objectives are as follows:

• Literature review is performed to identify the drawbacks andperformance of RTMP and HTTP protocols for video streamingservers.

CHAPTER 1. INTRODUCTION 4

• Streaming videos over RTMP and HTTP video streaming servervia a controlled traffic network.

• Relating the network parameters quality of streamed videos fromserver to client with QoE by MOS.

• Analyze the threshold level of delay in both RTMP and HTTPstreaming servers based on MOS.

• Giving recommendations on the performance of protocols basedon the analysis of results.

1.2 Research Questions

1 What is the impact of packet loss on RTMP and HTTP videostreaming with regard to video quality?

2 Which one of the protocols RTMP and HTTP achieves bettervideo quality when delays are introduced in the network?

3 What recommendations can be given for RTMP and HTTP pro-tocols based on the network performance in video streaming?

1.3 Expected outcomes

The following results are reported based on the work done:

1 The impact of packet loss and delay variation on RTMP andHTTP protocols on video quality is investigated by measuringMOS for the two protocols under varying network disturbances.

2 By performing a subjective video quality assessment of videostreamed over RTMP and HTTP protocols, conclusions are drawnregarding their performance.

3 Final results also contain QoS parameters that are taken intoconsideration while relating QoE.

4 By analyzing the obtained results for RTMP and HTTP protocolsfor various network disturbances like delay variation, packet loss,recommendations will be given based on the results.

CHAPTER 1. INTRODUCTION 5

1.4 Research Methodology

1 In the early stage of our research, we do literature review relatedto the video streaming protocols RTMP, HTTP and QoE, stan-dards. Finally, QoS parameters like delay and packet loss.

2 Study of different video streaming servers such as Adobe flashmedia, Red5, Wowza, VLC. Selecting the appropriate one thatsuits our research in video streaming.

3 In the next stage we choose the videos recommended by standardgroups that are encoded into H.264 using FFmpeg encoder forconducting experiments using RTMP and HTTP protocols.

4 Study about the constraints that affect the QoE.

5 Once the literature survey is done, experimentations are donewith appropriate video streaming servers for RTMP and HTTP,traffic shaper and standard videos

6 During the experimentations packet loss and delay variation areintroduced by using appropriate traffic shaper in the network.

7 Streaming selected videos over RTMP and HTTP protocols. Westream from the server to client, in a controlled environment.

8 Videos streamed at the client are observed and MOS are collectedusing subjective analysis method.

9 The results obtained from the user related survey of differentvideos of both these protocols are quantitatively analysed.

10 Based on the final results and analysis, we provide conclusionsand recommendations between these two protocols.

1.5 Outline of Thesis

This report is organized as follows. The Chapter 2 describes back-ground work and research related to this thesis work. The Chapter 3describes the experimental setup. The Chapter 4 contains results anddiscussion and final chapter 5 describes conclusion and future work.

Background and Related Work

6

Chapter 2

Background and Related Work

This section provides the background related to the thesis work. Sev-eral factors like packet loss and delay variation in network may varythe video quality, and because of some unreliable protocols are influ-encing the end user video applications. However, there may be smoothplay out interruptions due to bandwidth fluctuations or long retrans-missions, which affect the QoE for the video applications [8].

In [9] the author has explained about the overview of HTTP Adap-tive Streaming (HAS) concepts, and also an end-to-end QoE evaluationstudy on HAS conducted over 3GPP LTE networks. In paper [8], theauthors proposed no reference QoE monitoring module for adaptiveHTTP streaming based on Random Neural Networks, models the im-pact of both factors. In [10] authors has proposed a design for theacademic institutions to distribution, uploading videos and scheduledbroadcast by using Wowza media server, Stream Class Scheduler Mod-ule, Flash media encoder, FFmpeg, VLC and JWplayer tools are usedin this work. In paper [11], to investigate the relation between QoSand QoE, the author has conducted a set of experiments on differentvideos, by using different shapers like NetEm, NISTnet and KauNetby varying packet loss, delay variation and bit rate, and also to chosenthe best shaper for the future. Finally, authors concluded on NetEmshowing the accurate results based on one-way delay evaluation. Inpaper [12] the author focuses on revealing challenges and offering con-cepts associated with the incorporation of the Quality of Experience(QoE) into the design of mobile video systems. In [13] the author ex-plained concept of QoE in engineering is which also known as PerceivedQuality of Service (PQoS) and the term QoS which is finally perceivedby the end-user. The quality perception of H.264/AVC coded videocontaining packet loss is analyzed based on the results of a combinedsubjective video quality and eye tracking experiment. In the field of

7

CHAPTER 2. BACKGROUND AND RELATED WORK 8

networking, multidimensional QoS analysis has not yet received muchattention with regards to quality services [14].

The QoE of users degrades due to defective encoding, bottlenecks,channel change time too long, excessive compression, order failure forVoD, Transmission Unavailable, audio-video out of sync [15]. In pa-per [16] the authors explained the technical and non-technical param-eters aspects of QoE, in technical, application- and network-level QoSand in non-technical user perception, experience and expectations. Inpaper [17] the author discussed about video streaming effects by usingQoS parameters like delay, loss and bandwidth. To characterize theeffects of the parameters such as delay and loss are accessed by QoEwhere the MOS assigns each video sample by human subjects. In pa-per [18] the authors had done series of experiments on 3G networks,to observe the effect of PDV on the end-user QoE. These results areadequate to predict UR (User Ratings) values over PDV, related toQoE of streaming video users.

In paper [19], the author describes about applications and accesstechnologies that best suit the user needs, thereby combining thesefeatures of access technologies to provide an enhanced user perceivedexperience. In [20], work was done for small set of experiments, con-sidering the web based server for RTMP video streaming. The authorsmentioned that they had Mean Opinion Score for only 10 users. Theserver communicates with the flash media player by using the RealTime Messaging Protocol. RTMP was designed for high speed audio,video and data with maintaining persistent connection [21].

In paper [5], RTMP is used as real-time internet class room for web-based collaborative work between teachers and students to interactover exchanging messages such as flash data, audio, video. RTMPis basically a TCP/IP based protocol which delivers flash content. In[22,23], with the RTMP specification, developers and companies will beable to provide users with optimized audio, video and data streaming,no matter what kind of device the user is on, or where the content iscoming from.

The end user satisfaction is very much important, in evaluating aproduct or a service. There is a correlation between QoS and QoE toevaluate the end user satisfaction, of a product on the internet [18].In paper [9] HAS (HTTP Adaptive Streaming) provides the abilityat client to fully control over the streaming session, i.e., HTTP canintelligently manage the on-time request and smooth play out of videoframes, potentially adjusting bitrates.

The evolution of new multimedia solutions requires new ways tooptimize future wireless networks for video services towards delivery to

CHAPTER 2. BACKGROUND AND RELATED WORK 9

the end user. In the case of HTTP Adaptive Streaming, the contentdelivered at client side is in multiple bit rates that are higher or lower bitrates. HTTP Dynamic Streaming supports both live and on-demandmedia content that adjusts to viewer connection speed and processingpower using standard HTTP protocol Infrastructures [24].

In this [25] authors considered an Eclipse IDE application to streamHTTP and RTMP, for the streaming they used a web server (ApacheTomcat and Red5) and flash client. They fetched that flash clientbrowser is used to stream in both protocols. In scalable 3G live stream-ing over HTTP and RTMP protocols will stream different encodedvideos over various computers and mobile devices. Here, videos areencoded with On2 VP6 and H.264 baseline, it showed better encodingand decoding performance against High profile [26].

2.1 Flash Player

Here, a web based flash player is used at client side to receive en-coded video sequences. Flash Player 10.1 has more features and it willincrease the options available to deliver high quality media includingH.264 over many different protocols such as RTMP [27].

2.2 H.264 Video Codec

H.264 is a video compression technology, or codec, that was jointly de-veloped by the ITU as H.264 and International Organization for Stan-dardization/International Electro technical Commission (ISO/IEC) Mov-ing Picture Experts Group (as MPEG-4 Part 10, Advanced Video Cod-ing, or AVC). Thus, the terms H.264 and AVC mean the same thingand are interchangeable. A video codec in H.264 integrated into mul-timedia container format, and is frequently produced in the MPEG-4container format, which uses the .MP4 extension [28]. In [29] ITU-TH.264/MPEG-4 (Part 10) Advanced Video Coding (commonly referredas H.264/AVC) is the newest entry in the series of international videocoding standards. H.264/AVC is currently the most powerful andstate-of-the-art standard. It was developed by a Joint Video Team(JVT) consisting of experts from ITU-T Video Coding Experts Group(VCEG) and ISO/IEC Moving Picture Experts Group (MPEG).

2.3 FFmpeg

FFmpeg is an open source tool that can be used as both encoder anddecoder. It has a powerful multimedia processing capability [30]. FFm-

CHAPTER 2. BACKGROUND AND RELATED WORK 10

peg provides multimedia visual adaptation functionalities such as res-olution reduction, frame rate reduction, cropping etc. [31]. It is a veryfast video converter that can grab live video source [32].

2.4 Quality of Experience

It is the perceived quality of service by the end-user. In [33] the authorsportrayed the purview of QoE definition, QoE is the overall perfor-mance of a system from the user point of view, i.e., what a user reallyperceives in terms of usability, accessibility, retain-ability and integrity.The author discusses about QoE parameters, has more important thanthe QoS parameter [11]. To know about the better service of the net-work the QoE determines in the form of results from user perspective.In [11, p. 26- 28] the QoE depends on services like Quality of Presenta-tion (QoP), Quality of Delivery (QoD) and QoS to get user experienceover network services. Research has been done on user satisfaction hasbeen mainly forward motion in multimedia technology such as in videostreaming [8–12, 14]. In paper [34] for the service providers (SP) is toprovide PQoS services, assurance based on multimedia content to theend-user. Subjective analysis test should be measured via users. Thequality of perceived video is rated in the subjective analysis by theuser.

The ITU-T definitions for QoE are:Quality of Experience includes the complete end-to-end system ef-

fects (client, Terminal, network, services infrastructure, etc)

• The overall acceptability may be influenced by user expectationsand con- text. [35].

• The overall acceptability of an application or service, as perceivedsubjectively by the end-user by ITU-T P.10/G.100.

2.5 Subjective Quality Assessment

Subjective assessment methods are thus often considered as a groundtruth for quality prediction by user [13, p. 7]. The feeling of the in-dividual taking part in the analysis process determines the outcome.In QoE the subjective results vary from user-to-user [36]. There aretwo types of standards to measure subjective analysis defined by In-ternational Telecommunication Union, single and double stimulus. InSSCQE (Single Stimulus Continuous Quality Evaluation) method thequality of the distorted video is rated without any reference to theoriginal stimulus.

CHAPTER 2. BACKGROUND AND RELATED WORK 11

In DSCQS (Double Stimulus Continuous Quality Scale) method thereference video is given to measure the distorted video quality. In thisproject SSCQE method is used in subjective analysis. In this scenariosingle SSCQE allows end-users to dynamically rate the quality of anarbitrarily long video sequence using quality scale, be more useful forevaluating real-time quality monitoring systems [37]. This subjectivetest was performed in a room conforming to ITU-R BT.500 [3].

2.6 SSCQE (Single Stimulus Continuity Quality Eval-uation)

In the SSCQE continuous video sequences are once presented to user,to rate the video quality. The presented video sequences may containimpairments. Human subjects will evaluate the instantaneous qualityin real time using a slider with a continuous scale. This SSCQE isrecommended by ITU-R BT. 500-7. This method is preferred in QoEsubjective analysis.similarly, another method for objective analysis is No-reference method.It is also known as reference free method [11]. In this case, the QoEis not measured by comparing an original video. This method triesto detect artifacts such as blockiness, blur, jerkiness directly in thevideo [13] for quality prediction. This approach is based on the ideathat customers don’t know the original content. In this method theusers will rate the video with experienced quality and distortions.

Implementation

12

Chapter 3

Experimental Setup

3.1 Method

This section describes the methodology and methods to answer ourresearch questions. There are two steps in our proposed method. Theyare experimentation and user related survey i.e. subjective assessment.

The first step of our methodology is discussed in the next section3.2, and the second phase involves video quality assessment. The videoquality assessment can be done in two ways namely objective assess-ment and subjective assessment.

In objective assessments involves mathematical algorithms and mod-els that can estimate the Human perceptual behavior. The models likepeak signal-to-noise ratio (PSNR), Mean Square error are some ex-amples of objective assessment. These models are may vary for thedifferent codecs and others parameters. They may or may correlatewith the subjective assessment values.

The other method for video assessments is subjective assessmentwhere a user will participate in the survey to provide ratings for thevideos. In our research we have chosen subjective assessment to ratethe videos, because the objective assessment ratings may or may notcorrelate the perceptual ratings of the human behavior.

3.2 Experimental Setup

This chapter elucidates the experimental setup, to observe the charac-teristic of packet loss and packet delay variation affects in video qualityat end user and a set of experiments conducted in different scenarios.

13

CHAPTER 3. EXPERIMENTAL SETUP 14

3.3 Setup Design

In this experiment, streaming is done form server to client over HTTP,RTMP. Here, server and client machines are on Linux operating sys-tem. In between server and client the required equipments are placed totrace out the results for packet loss and packet delay variations. In thissetup two hosts are equipped with the traffic shaper in between. Thetraffic shaper is used to control the traffic in network going from hostA (server) to host B (client). The experimental setup is configured tostream video from host A (video originator) to host B (video receiver)at the other end. The connections are made with Ethernet cables withrequired static IP address. In this scenario host A sends packets tohost B via emulator. In the emulator losses and delays are introducedto find out the video quality at host B client side while streaming videoover RTMP, HTTP. The setup is mount to Measurement Point (MP)to capture the traffic flowing form sender to receiver. The experimentsetup is shown in Figure 3.3.1.

Figure 3.3.1: Experimental Setup

3.3.1 Measurement Point

MP is used for measuring overall timestamps of packets. MP obtainsthe sender time as well as receiver time of packet. With the helpof wiretaps MP collects the data at sending and receiving side. Therole of wiretaps is to recognize the traffic flowing in network. It willduplicate the traffic and passes to MP. The MP consists of two Endace

CHAPTER 3. EXPERIMENTAL SETUP 15

DAG 3.5E cards [38]. These DAG cards are used near sender andnear receiver. These cards are synchronized with respect to time andfrequency respectively; by utilizing Global positioning System (GPS)antenna for obtaining timestamps accuracy of 60ns in network DAGcards [39].

3.3.2 MArC

MArC (Measurement Area Controller) is a central subsystem in Mea-surement Area manages the MP’s. According to applied rules of MArCthe MP will tap the traffic and filters. It recognizes the users requestinformation and send it to MP’s [39].

3.3.3 Consumer

Consumer is system controlled by the user. It is a Linux environmentbased system. The replicated packets captured by Data Acquisitionand Generation (DAG) cards are stored in the consumer. It containslibcap-utils-7.0.8 is used to convert the binary traces to human readabletext format. Here, the network traffic are saved in text files. Thesestored files are used for further analysis of protocols [38].

3.3.4 Wowza Media Server

Wowza Media Systems has server software that delivers streaming videoand audio content as streaming content on a variety of platforms [40].Wowza Media Server (WMS) can deliver multi-bitrate live and on-demand media [41]. The paper [10] discusses about both VoD andlive streaming by using Wowza meadia server. It is fully interactiveserver for streaming multimedia content with full support for H.264,they have used FFmpeg tool for video conversion and video length.In [42, 43] WMS software produces the broadest any-screen coverageover Flash and Silverlight-capable computers, tablets, phones, set-topboxes, media players, and game consoles. In this project Wowza mediaserver 3 is used on Linux platform, which is and the DELL Laptop withIntel i5 M370 2.40GHz is used.

3.3.5 Wowza Client

Wowza client is a flash player, which is on a web browser. It is usedto receive the flash content delivered by WMS. Here, the flash contentare encoded video sequences. The Wowza client side HP Pavilion g6laptop with an Intel i3 M370 2.40GHz processor architecture runningLinux Ubuntu 10.04 LTS (Lucid Lynx) is used.

CHAPTER 3. EXPERIMENTAL SETUP 16

3.3.6 NetEm

Traffic shaping and emulation of network properties is useful in caseslike delay, loss, duplication and re-ordering [1]. NetEm or TC shaperbelongs to the Traffic Control (TC) bandwidth provisioning package ofLinux [2]. In this project NetEm is placed in between server and clientto disturb the ongoing traffic. To capture the video traffic before andbehind the shaper, the Distributed Passive Measurement Infrastruc-ture (DPMI) is used [39]. It acts as an interface. All the packets aretravelling through the interface will be affected with properties like lossand packet delay variation. The paper [38] explains about the NetEmshaper having more accuracy emulation property, compared to NIST-net and KauNet. It is installed on Intel Celeron(tm) 710 MHz. RAM:392192KB with Ubuntu 10.04 LTS (Lucid Lynx), and Kernel version2.6.32-38-generic-pae.

3.3.7 Packet Loss

The packet loss determines the number of packets that have been lostat the destination side compared to the source. It illustrates as ratioof packets lost (PL) at destination to packets sent (PS) by source onan Ethernet packet switched network.

Packetloss =PL

PS

In our experiments we have used 0%, 2.5%, 5%, 10%, 15% and 20%packet loss.

3.3.8 Packet Delay Variation

Packet Delay Variation (PDV) requires end-to-end measurements be-tween source and destination. In this era Packets can arrive at itsdestination node with a random time distortion. That clearly states,the time between packets at the destination is different from that atthe source [44]. ITU-T recommendations have been applied for delayvariations in audio and video transmissions. The delay and delay vari-ation values are expressed as D ±4D, where D is the fixed delay and4D is variable delay. [45] ITU-T Rec. G.114 suggests that 0 to 150ms limits can be used for one-way transmission time. 150 to 400 ms:acceptable when provided that administrations are aware of the trans-mission time impact on the transmission quality of user applications.The constant delay (D) value is considered as 150 ms. The delay (D)and variable delay (4D) settings used for these experiments are D±4D=150 ms±{0 ms, 10 ms, 15 ms, 25 ms, 50 ms, 100 ms, 150 ms}.

CHAPTER 3. EXPERIMENTAL SETUP 17

3.3.9 RTMP at Client

RTMP is a sophisticated TCP-based real time networking protocol. Itsupports an efficient exchange of multimedia messages like video [5].

3.3.10 HTTP at Client

In [8] HTTP streaming using TCP is the popular choice of many webbased applications. HTTP Dynamic Streaming supports both live andon-demand media content that adjusts to viewer connection speed, itenables high-quality H.264, network efficient HTTP streaming for mul-timedia delivery that is tightly integrated with flash platform for videocontent [4].

3.3.11 FFmpeg Encoding

It is a cross-platform tool. In this project the role of FFmpeg is tointeract with videos in digitizing different dimensions such as encoding,splitting and capturing. It acted as an interface between Wowza clientvideo and user, here QoP (Quality of Presentation) of FFmpeg mattersto present the videos to users by an Application. The raw videos Big-Buck-Bunny, Cable car and BTH are encoded to one of the most widelyused codecs are H.264/AVC (Advanced Video Coding) [46]. We hadused FFmpeg with libx264 for encoding purpose. The high profile ofH.264 was used. The encoder used was x264 [29]. The each videosequence has a frame rate of 25 fps and a bit rate of 800 kbps.

3.3.12 FFmpeg Splitting

The videos are taken from different sources, are split to 50 secs due tothe long duration for the convenience of subjective assessment.

3.3.13 FFmpeg Grabbing

In this two offsets are used i.e., X offset and Y offset, mainly theseterms are defined as XOFF and YOFF of application. The Geometryspecifications are considered to grab the videoed using X, Y OFF. Off-sets must be given as pairs. The layout of window can be placed infour edges of display i.e., +0+0 (upper left edge of window), -0+0 (up-per right edge of window), -0-0 (lower right edge of window) and +0-0(lower left edge of window) [47]. In this case +0+0 to Grab the res-olution of 640x480 (EGA) video with required parameters at client side.

CHAPTER 3. EXPERIMENTAL SETUP 18

Video name DescriptionBig-Buck-Bunny A computer animation video. Big-Buck-Bunny from

the Netherlands computer graphics teacher SachaGoedegebure, a comedy about a fat rabbit and three ir-ritating rodents [48].

Cable car This is video took from Wowza. These video visualsare scene at mountain area, in a same motion.

BTH This is real time advertisement video of Blekinge In-stitute of Technology.

Table 3.3.1: Description of Videos

Video Sequences Big-Buck-Bunny, Cable car, BTH.Codec H.264/AVCResolution EGA (640X350)

Frame−rate 25fpsExtension name .MP4Encoder Libx264

Table 3.3.2: Characteristics of Videos

The video resolution 640x350 is used in this project. This resolu-tion is preferred in computer screens. In [49] IBM display standardis 640x350 (EGA) introduced in 1984. It is a subset of VGA. EGA(Enhanced Graphic Adapter) is in between CGA and VGA.

3.3.14 Scale

MOS scale which is recommended by ITU-T. The QoE subjectiveratings for videos is generally given on a scale from 1 to 5. To measurethe video quality by the user a five point scale Mean Opinion Score(MOS) is used. The User Rating (UR) for videos sequences using thescale Excellent (5), Good (4), Fair (3), Poor (2) and Bad (1). Thisscale is recommended by ITU-T P.910 [50] shown in table 3.3.3.

Scale Quality Impairment5 Excellent Imperceptible4 Good Perceptible, but not annoying3 Fair Slightly, annoying2 poor Annoying1 Bad Very Annoying

Table 3.3.3: ITU-T SCALE OF MEDIA QUALITY IMPAIRMENT

CHAPTER 3. EXPERIMENTAL SETUP 19

3.4 Experimental setup and procedure

The experimental setup consists of a video streaming server, videoplayer at client, shaper and MP. The Streaming server namely WowzaMedia Server is used to send the encoded video sequences to the Wowzaclient. The video originator will send packets to client using HTTP andRTMP protocols. The NetEM traffic shaper is used to control the lossand variable delay shaping of traffic in the network going from serverto client. To capture the video traffic at sender and receiver side of theshaper. Distributed Passive Measurement Infrastructure (DPMI) [39]is used, with Endace DAG cards equipped in MP [51] to verify thepacket loss and delay variation shaping. Streamer and player were in-stalled on Linux Ubuntu 10.04 platform the shaper also run on sameLinux platform. These connections are installed with Ethernet cables.Here, a full duplex link bandwidth of 10Mbps is concerned. The IPaddresses are assigned in a network. Here, static IP address for theserver is: 192.168.0.2 and for the client: 192.168.0.3 with net mask:255.255.255.0. Here, two port numbers are concerned. The port num-ber for RTMP is: 1935 and port number for HTTP is: 8080. Theseare default port numbers for RTMP as well HTTP. Intially wire sharkis used to check the traffic that is flowing from one device to anotherdevice [52] and finding out the extra traffic.

The different videos are collected by varying packet loss and delayvariation at client by using FFmpeg grabbing, as the client side videois played by flash player. Here, three video sequences preferred namelyBTH, Big Buck Bunny and Cable Car.

3.5 Assessment of Videos

In this work, subjective analyses were conducted among 35 viewers.Among 35 viewers, 30 of them were males and 5 females are partici-pated. We had not considered 5 user ratings as they were not givengenuine ratings. According to the recommendation by the ITU-R [3]the number of human viewers participating in a subjective quality ex-periments should not be lower than 15. The standard group VQEG(Video Quality Expert Group) suggest at least 24 human observersare needed for quality assessment [53]. The observers belong to BTHuniversity students. Nine students had been participated before in thevideo quality assessment. Participants with an average age of 23. Forthe survey of videos we had design a tool shown in the Figures 3.5.1and 3.5.2 With local database in the backend. When user click onthe opinion score and submit, the corresponding value will store in the

CHAPTER 3. EXPERIMENTAL SETUP 20

database. The values collected from the database are mathematicallyanalyzed.

Figure 3.5.1: Login Screen

Figure 3.5.2: Assessment tool for video survey

Subjective testing for visual quality assessment has been formalizedin ITU-R Rec. BT.500 and ITU-T Rec. P.910. Our survey was ac-cording to the ITU-R recommendations [3], using the Single Stimulus

CHAPTER 3. EXPERIMENTAL SETUP 21

(SS) method. Total 78 videos are rated by the end users, according to5 point scale as shown in the Table 3.3.3. When each user rates thevideo, the score will be stored in the local database. All the valuesfrom the database are taken and analyzed using statistical methodsdiscussed in the next chapter.

Results

22

Chapter 4

Results and Discussions

This chapter explains the detailed description of the obtained results.These results are based on the experiments performed in the previ-ous chapter. These results gives the Subjective assessment and qualityof the two video streaming protocols RTMP and HTTP based on thepacket loss and delay variation. However, MOS rating gives best opin-ion on video quality.

To discuss and achieve these results we had used statistical methodssuch as Mean and standard deviation acquired from the set of data ofthe subjective assessment of video quality survey. Later these resultsare analyzed.

Based on the Recommendation BT. 500 subjective assessment in thequality of television pictures, of the International TelecommunicationsUnion Radio communications Sector (ITU-R) [3], we had calculatedmean scores of the MOS and Confidence Interval (CI) for the set ofvideos like BTH, Big Bunky video and Cable car for both the protocolsHTTP and RTMP, based on the packet loss and delay variation.

4.1 Mean scores calculations

We have to calculate, the mean score for each and every single presen-tation, and the mean is defined as,

X̄jk =1

N

N∑i=1

X̄ijk

The standard deviation defined as,

23

CHAPTER 4. RESULTS AND DISCUSSIONS 24

δjk =

√√√√ N∑i=1

(X̄jk − X̄ijk)2

N − 1,

Where N = Number of observers Xijk = Score of the ith observerfor test condition j and video sequence k .

4.2 Confidence Interval Calculations

Once all the results of mean scores are calculated, and as the meanscores are always associated with CI, 95 % Confidence Intervals for allthe mean scores are calculated.

With the 95% Confidence Interval, the exact value of differencebetween experimental mean score and the true mean score, will be ob-tained.

The 95% of CI is given by:[X̄jk − γjk, X̄jk + γjk]

Where marginal error γjk,

γjk = 1.96 ∗ δjk√N

γjk = Standard deviationN = Number of observers

4.3 RTMP and HTTP Packet loss

The below Figures 4.3.1, 4.3.2, 4.3.3 gives graphical representation ofBTH advertisement, Cable Car and Big Buck Bunny streamed videoswith MOS ratings on Y-axis and packet loss % on X-axis respectivelyfor both RTMP and HTTP.

CHAPTER 4. RESULTS AND DISCUSSIONS 25

Figure 4.3.1: MOS with 95% CI vs. Packet loss for HTTP BTH and RTMPBTH

In Figure 4.3.1 the BTH RTMP and BTH HTTP videos have “GOOD”ratings at 0% packet loss, above “FAIR” ratings from 2.5%- 10% packetloss. Once the packet loss is increasing from 10% - 15%, 20% the ratingsof the users are degrading to “POOR” and “BAD” for BTH advertise-ment video for the HTTP and RTMP protocols. The both protocolshave similar performance as the user ratings are matching for BTHadvertisement video.

The Figure 4.3.2 it illustrates that the Cable car RTMP and Cablecar HTTP videos are having slightly “GOOD” ratings at 0% packetloss, “FAIR” ratings from 2.5% - 10% packet loss. Once the packet lossis increasing from 10% to 20% the ratings of the users are degrading to“POOR” and “BAD” for RTMP protocol. For HTTP video the ratingsare slightly less when compared to RTMP up to 10% packet loss. Aspacket loss is increasing 10%-20% HTTP is having “POOR” and “BAD”ratings, and slightly better than RTMP.

CHAPTER 4. RESULTS AND DISCUSSIONS 26

Figure 4.3.2: MOS with 95% CI vs. Packet loss for HTTP cablecar andRTMP cablecar

Figure 4.3.3: MOS with 95% CI vs. Packet loss for HTTP Big Buck Bunnyand RTMP Big Buck Bunny

Figure 4.3.3 it illustrates that Big Buck Bunny HTTP video is hav-ing “GOOD” ratings at 0% packet loss, “FAIR” ratings from 2.5% -15% packet loss, and when the packet loss is increasing from 15% to20% the ratings of the users are degrading to “POOR”. For RTMP pro-

CHAPTER 4. RESULTS AND DISCUSSIONS 27

tocol it has similar ratings from 0%-5%, compared to HTTP, “FAIR”ratings from 5% - 10%, and “POOR” ratings from 10% - 20%. As thepacket loss is increasing from 5% RTMP protocol is having less ratingscompared to HTTP protocol.

Figure 4.3.4 illustrates the standard deviation of all videos for bothRTMP and HTTP on Y-axis and Packet loss % on X-axis. The valuesare ranging from 0.49 to 1.19. Comparing all the standard deviations,there is minimum divergence for all the videos up to 5% packet lossand more divergence from 10% - 20% packet loss.

Figure 4.3.4: Standard deviation for HTTP and RTMP packet loss

95% Confidence intervals for all the three video sequences for bothRTMP and HTTP are shown with MOS respectively in the Figures4.3.1, 4.3.2, 4.3.3 BTH, Cable Car, Big Buck Bunny videos. CI ismainly depends on total number of people participated during the as-sessment. X-axis shows the packet loss % and Y-axis shows the MOSof UR. From the graphs we can observe that the UR are decreasingas packet loss is increasing. But when we compare the ratings of bothRTMP and HTTP, it is clear that user ratings are better for RTMPwhen there is minimum loss of packets and as the loss is increasingHTTP is having better ratings.

CHAPTER 4. RESULTS AND DISCUSSIONS 28

PacketLoss % BTHHTTP

BTHRTMP

CableCar

HTTP

CableCar

RTMP

BigBuckBunnyHTTP

BigBuckBunnyRTMP

0% 4.433 4.6 4 4.0666 4.56 4.62.5% 3.7666 3.9 3.1 3.3 3.7 3.95% 3.7666 3.7 3 3.166 3.7 3.66610% 3.633 3.533 2.8 3.133 3.633 3.2315% 2.7666 2.633 2.466 2.033 3.066 2.2620% 1.533 1.666 1.566 1.633 1.866 1.933

Table 4.3.1: MOS Ratings for Packet Loss

From the both cases we can observe that users are giving “GOOD”ratings at 0% packet loss, “FAIR” ratings from 2.5% to 10%, and aspacket loss is increasing MOS ratings are “POOR” by the users. RTMPis having better performance at low packet loss compared to HTTP,but when observed from the graphs the behavior of RTMP streamedvideos quality is degrading compared to HTTP streamed videos.

CHAPTER 4. RESULTS AND DISCUSSIONS 29

4.4 RTMP and HTTP Delay Variation

The below Figures 4.4.1, 4.4.2, 4.4.3, gives the graphical representa-tion of BTH advertisement, Cable Car and Big Buck Bunny streamedvideos with MOS ratings on Y-axis and Delay variation in ms on X-axis respectively for both RTMP and HTTP. In Figure 4.4.1 the BTHHTTP streamed video has “FAIR” ratings until 0ms, as the delay vari-ation is increasing from 10ms - 50ms it has “POOR” ratings by users.From 50ms - 150ms user ratings are tending towards “BAD”. For theBTH RTMP video at 0ms - 15ms user ratings are above “FAIR”. As thedelay variation is increasing from 15ms - 150ms user ratings are tend-ing towards “BAD” from “POOR”. As the delay variation is increasingBTH HTTP is having better performance compared to BTH RTMP.

Figure 4.4.1: MOS vs. Delay Variation for HTTP BTH and RTMP BTH

From the Figure 4.4.2 it illustrates that Cable Car HTTP video ishaving “FAIR” ratings at 0ms. As delay variation is increasing from10ms - 50ms it has above “FAIR” ratings, and from 100ms to 150msthe ratings are tending towards “POOR” to “BAD”. In case of CableCar RTMP video, it has “BAD” ratings at 0ms and as delay variation isincreasing from 10ms to 150ms, the ratings are changing from “POOR”to “BAD”.

From the Figure 4.4.3 it illustrates that Big Buck Bunny HTTPvideo is having “GOOD” ratings by the user at 0ms and while delayvariation is varying from 0ms - 15ms it has “FAIR” ratings. From15ms to 150ms the HTTP video has “POOR” ratings. For Big Buck

CHAPTER 4. RESULTS AND DISCUSSIONS 30

Figure 4.4.2: MOS vs. Delay Variation for HTTP Cable car and RTMPCable car

Bunny RTMP video, it has “FAIR” ratings from at 0ms - 25ms andvery “POOR” ratings from 25ms - 150ms.

The Figure 4.4.4 illustrates the standard deviation of all videos forboth RTMP and HTTP on Y-axis and delay variation in ms on X-axis. The values are ranging from 0.46 to 1.014. As comparing all thestandard deviations, there is minimum divergence for RTMP comparedto HTTP videos until 25ms and more divergence of ratings for RTMPvideos compared to HTTP videos. This intends that users are giving“GOOD” ratings for HTTP rather than RTMP when there are moredelays.

CHAPTER 4. RESULTS AND DISCUSSIONS 31

Figure 4.4.3: MOS vs. Delay Variation for HTTP Big Buck Bunny andRTMP Big Buck Bunny

Figure 4.4.4: Standard deviation for HTTP and RTMP Delay variation

The 95% Confidence intervals for all the three video sequences forboth RTMP and HTTP are shown in the Figures 4.4.1, 4.4.2, 4.4.3refers to CI of Big Buck Bunny, Cable Car and BTH videos respectively.CI mainly depends on total number of people participated during theassessment. X-axis shows the packet loss % and Y-axis shows the MOS

CHAPTER 4. RESULTS AND DISCUSSIONS 32

of user ratings. From the graphs we can observe that user ratings aredecreasing as delay variation is increasing. But when we compare theratings of both RTMP and HTTP, it is clear that user ratings are“GOOD” for HTTP when compared to RTMP.

DelayVaria-tion(ms)

BTHHTTP

BTHRTMP

CableCar

HTTP

CableCar

RTMP

BigBuck

BunnyHTTP

BigBuck

BunnyRTMP

150±0 3.166 3.7 3.4 3.3233 4.166 3.866150±10 2.366 3.4 2.866 2.833 3.266 3.633150±15 2.466 3.3 2.5666 2.166 3.166 3.266150±25 2.266 1.933 2.4666 1.766 2.8 2.5150±50 2.2 1.6 2.4666 1.7333 2.433 1.866150±100 1.7 1.433 1.9 1.6 2.233 1.566150±150 1.5 1.366 1.7 1.5 2.233 1.666

Table 4.4.1: MOS Ratings for Packet Delay Variation

From both cases we can observe that users are giving “FAIR” ratingsat 0ms delay variation, as we observe from 10 ms to 50 ms and as delayvariation is increasing MOS ratings are given “POOR” by the users. Byobserving the graphs the behaviour of RTMP streamed videos qualityis degrading Compared to HTTP streamed videos.

4.5 Validity Threats

In our thesis we are lacking real time physical network, in order toovercome we had proposed a small network in a lab environment.

We had recorded the videos by using ffmpeg tool, for which therecorded video quality depends on tool. The videos are recorded forboth RTMP and HTTP to compare at different packet loss and delayvariation. We have chosen ffmpeg to record the videos at client as thereare no other tools to record the RTMP videos.

Conclusions and Future Work

33

Chapter 5

Conclusions and Future Work

5.1 Conclusion

Experimental results of our thesis shows the video quality of RTMPand HTTP video streaming protocols based on subjective video qualityassessment. As a part of our experiment, we had streamed the videosfrom the server to client over traffic shaper, by varying the factors likepacket loss and delay variation, for different video sequences such asBTH, Cable car and Big Buck Bunny. All the videos at the client sideare recorded by changing the network factors. By using QoE subjectiveassessment methodology, we had collected feedback from different userperceptions. The collected feedback is later analyzed by statisticalmethods like mean and Confidence intervals.

Our first research question deals with impact of packet loss onRTMP and HTTP protocols on video quality. From the subjectiveassessment analysis, we can observe the quality between both RTMPand HTTP protocols for different video sequences. As the packet lossis increasing the MOS ratings are changing from “GOOD“ to “POOR”by the users, as there is a degrade in the video quality. RTMP pro-tocol is having better ratings than HTTP from 0%-5% packet loss, aspacket loss metric is increasing, HTTP is slightly better as comparedto RTMP.

Our second research question deals with impact of delay variationon RTMP and HTTP protocols on video quality. From the subjectiveassessment analysis, we can observe the quality between both RTMPand HTTP protocols for different video sequences. As delay variationis increasing MOS ratings are pointed less for the video sequences.When the delay variation is very low, users are giving better ratingsfor RTMP streamed videos when compared to HTTP video sequences.When high delay variations are applied, HTTP performs better than

34

CHAPTER 5. CONCLUSIONS AND FUTURE WORK 35

RTMP. Form the analysis of results HTTP is having good performancethan RTMP.

Our third research question deals with what recommendations canbe given for both protocols when network disturbances like packet lossand delays places. Form the analysis of the results, we can say thatRTMP protocol is having less performance with respect to video qual-ity, and more performance examined by HTTP when there is morepacket loss and delay variation. We recommend, using HTTP protocolis preferred for streaming the videos in wide networks like web, publicnetwork domains. As RTMP is a very secure protocol, and it havingbetter quality when there are fewer disturbances in the network, wesuggest using RTMP in small set of networks for video conferences ande-learning video lectures in the universities, offices.

5.2 Future Work

Future research has to be done on 3G and 4G networks with largeset of experiments should be conducted for both RTMP and HTTP.Comparing the two protocols videos by using PEVQ software withsubjective and objective assessments.

Bibliography

[1] (2012) NetEm. Linuxfoundation.org. [Online]. Avail-able: http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

[2] S. Hemminger, “Network emulation with netem,” in Linux ConfAu. Citeseer, 2005, pp. 18–23.

[3] ITU-R Recommendation BT.500-11, “Methodology for the subjec-tive assessment of the quality of television pictures,” InternationalTelecommunication Union, Geneva, Switzerland, Research Report,2002.

[4] C. group, “HTTP versus RTMP: Which Way to Go and Why?,.”

[5] W. Premchaiswadi, A. Tungkasthan, and N. Jongsawat, “En-hancing learning systems by using virtual interactive classroomsand web-based collaborative work,” in Education Engineering(EDUCON), 2010 IEEE, pp. 1531–1537.

[6] (2012) Dynamic streaming in Flash Media Server.Adobe.com. [Online]. Available: http://www.adobe.com/devnet/flashmediaserver/articles/dynstream_advancedpt1.html

[7] ITU-T Telecommunication G.1070, “Opinion model for video-telephony applications,” International Telecommunication Union,Research Report, 2007.

[8] K. Singh, Y. Hadjadj-Aoul, and G. Rubino, “Quality of experi-ence estimation for adaptive HTTP/TCP video streaming usingh. 264/AVC,” in Consumer Communications and Networking Con-ference (CCNC), 2012 IEEE, pp. 127–131.

[9] O. Oyman and S. Singh, “Quality of experience for HTTPadaptive streaming services,” IEEE Communications Magazine,vol. 50, no. 4, pp. 20 – 7, 2012/04/. [Online]. Available:http://dx.doi.org/10.1109/MCOM.2012.6178830

36

BIBLIOGRAPHY 37

[10] D. Cymbalák, F. Jakab, and M. Michalko, “Next generation iptvsolution for educational purposes,” in Emerging eLearning Tech-nologies and Applications (ICETA), 2011 9th International Con-ference on. IEEE, 2011, pp. 41–46.

[11] T. Minhas, “Network Impact on Quality of Experience of MobileVideo,” Ph.D. dissertation, Blekinge Institute of Technology, 2012.

[12] M. Fiedler, H. Zepernick, L. Lundberg, P. Arlos, and M. Petters-son, “Qoe-based cross-layer design of mobile video systems: Chal-lenges and concepts,” in Computing and Communication Technolo-gies, 2009. RIVF’09. International Conference on. IEEE, 2009,pp. 1–4.

[13] U. Engelke, “Modelling perceptual quality and visual saliency forimage and video communications,” Ph.D. dissertation, Licentiate,Blekinge Institute of Technology, 2012.

[14] K. D. Vogeleer, “On the Quality of Delivery for Variable Bit RateVideo,” Ph.D. dissertation, Licentiate, Blekinge Institute of Tech-nology, 2012.

[15] (2012) Gain control over your audiences quality of experience.ip-label.co.uk. [Online]. Available: http://www.ip-label.co.uk/end-user-experience-monitoring-solutions/image-tv-video/

[16] T. Hosfeld, M. Fiedler, and T. Zinner, “The QoE provisioning-delivery-hysteresis and its importance for service provisioning inthe Future Internet,” in Next Generation Internet (NGI), 2011 7thEURO-NGI Conference on, june 2011, pp. 1 –8.

[17] M. Venkataraman, M. Chatterjee, and S. Chattopadhyay, “Eval-uating Quality of Experience for Streaming Video in Real Time,”in Global Telecommunications Conference, GLOBECOM 2009.IEEE, pp. 1 –6.

[18] S. Ickin, K. De Vogeleer, M. Fiedler, and D. Erman, “The effectsof Packet Delay Variation on the perceptual quality of video,” inLocal Computer Networks (LCN), 2010 IEEE 35th Conference on,oct. 2010, pp. 663 –668.

[19] J. Kellokoski and T. Hamalainen, “User-centric approach toalways-best-connected networks,” in Ultra Modern Telecommuni-cations and Control Systems and Workshops (ICUMT), 3rd Inter-national Congress on, oct. 2011, pp. 1 –6.

BIBLIOGRAPHY 38

[20] H. French, J. Lin, T. Phan, and A. Dalal, “Real time videoQoE analysis of RTMP streams,” in Performance Computing andCommunications Conference (IPCCC), IEEE 30th International.IEEE, 2011, pp. 1–2.

[21] W. Wei, “Flash Media Server and Its Application in the Simula-tor of VHF Marine Radio Station,” in Information Engineeringand Computer Science. ICIECS 2009. International Conferenceon, pp. 1 –4.

[22] (2012) Adobe developers connection. Adobe.com. [Online].Available: http://www.adobe.com/devnet/rtmp.html

[23] (2012) Adobe to Open Flash Platform Mes-saging Protocol. Adobe.com. [Online]. Avail-able: http://www.adobe.com/aboutadobe/pressroom/pressreleases/pdfs/200901/012009RTMP.pdf

[24] (2012) HTTP Dynamic Streaming. Adobe.com. [Online]. Avail-able: http://www.adobe.com/products/hds-dynamic-streaming.html

[25] X. Gong and D. Zhang, “Research on web server based on red5,tomcat and apache,” Advanced Materials Research, vol. 282, pp.721–725, 2011.

[26] Q. Chang, Z. Yang, and Y. Song, “A scalable mobile live videostreaming system based on RTMP and HTTP transmissions,” Ad-vanced Research on Computer Science and Information Engineer-ing, pp. 113–118, 2011.

[27] (2012) RTMP, RTMFP and HTTP - The multi-protocolexperience with Flash Media Delivery on Flash Player 10.1.Adobe.com. [Online]. Available: http://blogs.adobe.com/ktowes/2009/10/rtmp_rtmfp_and_http_the_multi.html

[28] O. Jan. (2011) what is H.264? . Streamingmedia.com. [Online].Available: http://www.streamingmedia.com/Articles/Editorial/What-Is-.../What-is-H.264-74735.aspx

[29] Joint Video Team, “Draft ITU-T Recommendation and FinalDraft International Standard of Joint Video Specification (ITU-T Rec. H.264 | ISO/IEC 14496-10 AVC),” 8th Meeting: Geneva,Switzerland, 23-27 May, Tech. Rep., 2003.

BIBLIOGRAPHY 39

[30] B. Zhang, S. Zeng, R. Xu, D. Guo, J. Yan, and W. Wang, “Designand implementation of a scalable system architecture for embed-ded multimedia terminal,” in Electrical and Control Engineering(ICECE), 2011 International Conference on. IEEE, 2011, pp.618–621.

[31] F. Caron, S. Coulombe, and T. Wu, “A Transcoding Serverfor the Home Domain,” in Portable Information Devices, 2007.PORTABLE07. IEEE International Conference on, may 2007, pp.1 –5.

[32] (2012) FFmpeg Document. ffmpeg.org. [Online]. Available:http://ffmpeg.org/ffmpeg.html

[33] J. Schumacher, M. Dobler, E. Dillon, G. Power, M. Fiedler, D. Er-man, K. De Vogeleer, M. Ramos, and J. Argente, “Providing anuser centric always best connection,” in Evolving Internet (IN-TERNET), 2010 Second International Conference on. IEEE,2010, pp. 80–85.

[34] M. Sidibe, H. Koumaras, I. Kofler, A. Mehaoua, A. Kourtis, andC. Timmerer, “A novel monitoring architecture for media servicesadaptation based on network QoS to perceived QoS mapping,”Signal, Image and Video Processing, pp. 307 – 20, 2008. [Online].Available: http://dx.doi.org/10.1007/s11760-008-0083-2

[35] ITU-T FG IPTV, “Focus group on IPTV standardization:Definition of quality of experience (QoE),” International Telecom-munication Union Telecommunication Sector Publications,Research Report, 2007. http://ties.itu.int/ftp/public/itu-t/fgiptv/readonly/PreviousMeetings/20070122MountainView/il/T05-FG.IPTV-IL-0050-E.html.

[36] S. Baraković, J. Baraković, and H. Bajrić, “QoE dimensions andQoE measurement of NGN services,” in Proceedings of the 18thTelecommunications Forum, TELFOR, 2010.

[37] M. Pinson and S. Wolf, “Comparing subjective video quality test-ing methodologies,” in SPIE Video Communications and ImageProcessing Conference, vol. 5150, 2003, pp. 573–582.

[38] P. Arlos, “On the Quality of Computer Network Measurements,”Ph.D. dissertation, Blekinge Institute of Technology, 2005.

[39] P. Arlos, M. Fiedler, and A. A. Nilsson, “A Distributed PassiveMeasurement Infrastructure,” in PAM2005, 2005, pp. 215–227.

BIBLIOGRAPHY 40

[40] M. McClure, Managing Costs in a Multipaltform World, Vol. 35Issue 1 ed., Jan/Feb2012, pp. 20-24.

[41] (2012) Wowza Media Systems, LLC. Wowza.com. [Online]. Avail-able: www.wowza.com/resources/WowzaMediaServerUsersGuide.pdf

[42] (2012, July) Wowza Media Systems at IBC. wowza.com. [Online].Available: http://www.wowza.com/news/ibc2012-preview

[43] (2012) Wowza Media Server 3.1. wowza.com. [On-line]. Available: http://broadcastengineering.com/automation/wowza-server-streaming-media-east

[44] J. Welch and J. Clark, “A Proposed Media Delivery Index (MDI),”2006.

[45] ITU-T G.114, “One-way transmission time,” Internationaltelecommunications union telecommunication sector, Research Re-port, 2006.

[46] ITU-T REC-H.264, “H.264: Advanced video coding for genericaudiovisual services,,” International telecommunications uniontelecommunication sector publications, Tech. Rep., 2010.

[47] (2012) How to Specify the location of windows. mit.edu. [Online].Available: http://stuff.mit.edu/afs/athena/project/consultdev/answers/xwindows/xwindows_geometry.html

[48] (2012) Bigbuckbunny Homepage. bigbuckbunny.org. [Online].Available: http://www.bigbuckbunny.org/

[49] IBM, IBM Enhanced Graphics Adapter. minuszerodegrees.net.[Online]. Available: http://www.minuszerodegrees.net/oa/OA%20-%20IBM%20Enhanced%20Graphics%20Adapter.pdf

[50] ITU-T Rec. P.910, “Subjective video quality assessment meth-ods for multimedia applications,” International telecommunica-tions union telecommunication sector, Research Report, 1999.

[51] (2012) Endace time distribution server. Endace.com. [Online].Available: http://www.endace.com/time-distribution-servers.html

[52] (2012) Wireshark. Wireshark.com. [Online]. Available: http://www.wireshark.org/docs/

BIBLIOGRAPHY 41

[53] Video Quality Experts Group, “Final report from the video qualityexperts group on the validation of objective models of multimediaquality assessment, phase I,” VQEG, Sept, 2008.

Appendix

42

Appendix A

Apendix A

A.1 Network Configuration

To get network in a proper condition the permanent settings are changedat network interface configuration used /etc/network/interfaces at ter-minal. Where ifup and ifdown settings are generated. These conditionswill change the interface identity.

# ifconfig ethX ipaddress netmask 255.255.255.0 up

The above command is used to change the settings at Client andServer. This feature will change physical interface to logical interface.

To take place the above settings restart is required to networkingservice under Linux OS.

# sudo /etc/init.d/networking restart

A.2 Speed and Duplex settings

A full duplex bandwidth of 10Mpbs is prefered, to change the speedand duplex settings the following command is used

# ethtool -s ethX speed 10 duplex full advertise 0x002 autoneg on

To get speed and other information of ethX

# ethtool ethX

43

APPENDIX A. APENDIX A 44

A.3 Emulator Commands

Packet loss

# tc qdisc add dev eth2 root netem delay X %# tc qdisc change dev eth2 root netem delay X %

Delay Variation

# tc qdisc add dev eth2 root netem delay X.ms# tc qdisc change dev eth2 root netem delay X ms Y ms

In the above stated command the term X stands for Delay, the termY stand for delay variation.

The FFmpeg commands used in this project are:

FFmpeg Grabbing

ffmpeg −f x11grab −sameq −r25 −s 640x350 −i : 0.0+X−offset, Y−offset −vcodec libx264 −vpre losslessultrafast −threads 0 video.MP4

FFmpeg Splitting

ffmpeg −i file.mp4 −sameq −ss hh : mm : ss −t hh : mm : ss outfile.mp4

Appendix B

Apendix B

This section gives the detail MOS ratings over videos took for subjectiveanalysis. In upcoming figures clearly explains the ratings on threevideos in packets loss and delay variation.

45

APPENDIX B. APENDIX B 46

B.0.1 BTH HTTP Packet loss ratings

Table B.0.1: Packet loss user ratings on BTH HTTP Video

APPENDIX B. APENDIX B 47

B.0.2 BTH RTMP Packet loss ratings

Table B.0.2: Packet loss user ratings on BTH RTMP Video

APPENDIX B. APENDIX B 48

B.0.3 Cable car HTTP Packet loss ratings

Table B.0.3: Packet loss user ratings on Cable car HTTP Video

APPENDIX B. APENDIX B 49

B.0.4 Cable car RTMP Packet loss ratings

Table B.0.4: Packet loss user ratings on Cable car RTMP Video

APPENDIX B. APENDIX B 50

B.0.5 Big Buck Bunny HTTP Packet loss ratings

Table B.0.5: Packet loss user ratings on Big Buck Bunny HTTP Video

APPENDIX B. APENDIX B 51

B.0.6 Big Buck Bunny RTMP Packet loss ratings

Table B.0.6: Packet loss user ratings on Big Buck Bunny RTMP Video

APPENDIX B. APENDIX B 52

B.0.7 BTH HTTP Delay variation ratings

Table B.0.7: Delay variation user ratings on BTH HTTP Video

APPENDIX B. APENDIX B 53

B.0.8 BTH RTMP Delay variation ratings

Table B.0.8: Delay Variation user ratings on BTH RTMP Video

APPENDIX B. APENDIX B 54

B.0.9 Cable car HTTP Delay Variation ratings

Table B.0.9: Delay Variation user ratings on Cable car HTTP Video

APPENDIX B. APENDIX B 55

B.0.10 Cable car RTMP Delay Variation ratings

Table B.0.10: Delay Variation user ratings on Cable car RTMP Video

APPENDIX B. APENDIX B 56

B.0.11 Big Buck Bunny Delay Variation HTTP ratings

Table B.0.11: Delay Variation user ratings on Big Buck Bunny HTTP Video

APPENDIX B. APENDIX B 57

B.0.12 Big Buck Bunny Delay Variation RTMP ratings

Table B.0.12: Delay Variation user ratings on Big Buck Bunny RTMP Video