Understanding the Effectiveness of Video Ads: A...

14
Understanding the Effectiveness of Video Ads: A Measurement Study S. Shunmuga Krishnan Akamai Technologies [email protected] Ramesh K. Sitaraman University of Massachusetts, Amherst & Akamai Technologies [email protected] ABSTRACT Online video is the killer application of the Internet. Videos are expected to constitute more than 85% of the trac on the consumer Internet within the next few years. However, a vexing problem for video providers is how to monetize their online videos. A popular monetization model pursued by many major video providers is inserting ads that play in-stream with the video that is being watched. Our work represents the first rigorous scientific study of the key fac- tors that determine the eectiveness of video ads as mea- sured by their completion and abandonment rates. We col- lect and analyze a large set of anonymized traces from Aka- mai’s video delivery network consisting of about 65 million unique viewers watching 362 million videos and 257 million ads from 33 video providers around the world. Using novel quasi-experimental techniques, we show that an ad is 18.1% more likely to complete when placed as a mid-roll than as a pre-roll, and 14.3% more likely to complete when placed as pre-roll than as a post-roll. Next, we show that completion rate of an ad decreases with increasing ad length. A 15- second ad is 2.9% more likely to complete than a 20-second ad, which in turn is 3.9% more likely to complete than a 30-second ad. Further, we show the ad completion rate is influenced by the video in which the ad is placed. An ad placed in long-form videos such as movies and TV episodes is 4.2% more likely to complete than the same ad placed in short-form video such as news clips. Finally, we show that about one-third of the viewers who abandon leave in the first quarter of the ad, while about two-thirds leave at the half- way point in the ad.Our work represents a first step towards scientifically understanding video ads and viewer behavior. Such understanding is crucial for the long-term viability of online videos and the future evolution of the Internet. Categories and Subject Descriptors C.4 [Performance of Systems]: Performance attributes, Measurement techniques; C.2.4 [Computer-Communication Networks]: Distributed Systems—Client/server Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. IMC’13, October 23–25, 2013, Barcelona, Spain. Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-1953-9/13/10 ...$15.00. http://dx.doi.org/10.1145/2504730.2504748. General Terms Economics, Human Factors, Measurement Keywords Online videos; Advertisements; Monetization; User behav- ior; Internet content delivery; Multimedia. 1. INTRODUCTION Online video is the killer application of the Internet. Ac- cording to a recent Cisco study more than half of the con- sumer trac on the Internet today is related to videos and that fraction is expected to exceed 85% in 2016 [9]. As all forms of traditional media such as news, entertainment and sports migrate to the Internet, video on-demand trac is expected to triple by 2016 from the levels seen in 2011. Video providers who oer online videos include news chan- nels (such as CNN and Fox News), sports channels (such as ESPN and MLB), movie outlets (such as Hulu and Net- Flix), and entertainment providers (such as NBC, ABC, and CBS). Video providers bear the costs of acquiring and de- livering the videos to their audience of viewers. Acquisition costs may include production costs for original content or li- censing costs and/or revenue sharing for third-party content. The delivery costs often involve contracting with a content delivery service (such as Akamai [18]), who in turn incur the costs for the servers, software, bandwidth, colocation, and power. The runaway success of online videos leaves video providers and the media industry with perhaps their single most vexing problem. How can online videos be monetized? How can they be made viable and profitable? While successful models for video monetization are still evolving, there are broadly three monetization models that are gaining popularity in the industry. The subscription- based model requires users to pay a fee on a periodic basis (usually monthly) to watch videos. The pay-per-view model requires users to pay a fee usually on a per-event basis. Fi- nally, a popular model more relevant to our work is the ad- based model where viewers do not pay a fee but are shown ads that are placed in-stream in the video content. 1 Driven by both the rapid increase in online video con- sumption and the intense need to monetize that consump- tion, it is perhaps not surprising that video ads were the 1 We use the term “video” to describe the video being watched, such as a news clip, sports event, or movie. We use the term “ad” to indicate the ad that is played in-stream with the video that is being watched.

Transcript of Understanding the Effectiveness of Video Ads: A...

Understanding the Effectiveness of Video Ads:

A Measurement Study

S. Shunmuga Krishnan

Akamai Technologies

[email protected]

Ramesh K. Sitaraman

University of Massachusetts, Amherst

& Akamai Technologies

[email protected]

ABSTRACTOnline video is the killer application of the Internet. Videosare expected to constitute more than 85% of the tra�c onthe consumer Internet within the next few years. However,a vexing problem for video providers is how to monetizetheir online videos. A popular monetization model pursuedby many major video providers is inserting ads that playin-stream with the video that is being watched. Our workrepresents the first rigorous scientific study of the key fac-tors that determine the e↵ectiveness of video ads as mea-sured by their completion and abandonment rates. We col-lect and analyze a large set of anonymized traces from Aka-mai’s video delivery network consisting of about 65 millionunique viewers watching 362 million videos and 257 millionads from 33 video providers around the world. Using novelquasi-experimental techniques, we show that an ad is 18.1%more likely to complete when placed as a mid-roll than as apre-roll, and 14.3% more likely to complete when placed aspre-roll than as a post-roll. Next, we show that completionrate of an ad decreases with increasing ad length. A 15-second ad is 2.9% more likely to complete than a 20-secondad, which in turn is 3.9% more likely to complete than a30-second ad. Further, we show the ad completion rate isinfluenced by the video in which the ad is placed. An adplaced in long-form videos such as movies and TV episodesis 4.2% more likely to complete than the same ad placed inshort-form video such as news clips. Finally, we show thatabout one-third of the viewers who abandon leave in the firstquarter of the ad, while about two-thirds leave at the half-way point in the ad.Our work represents a first step towardsscientifically understanding video ads and viewer behavior.Such understanding is crucial for the long-term viability ofonline videos and the future evolution of the Internet.

Categories and Subject DescriptorsC.4 [Performance of Systems]: Performance attributes,Measurement techniques; C.2.4 [Computer-Communication

Networks]: Distributed Systems—Client/server

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’13, October 23–25, 2013, Barcelona, Spain.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-1953-9/13/10 ...$15.00.http://dx.doi.org/10.1145/2504730.2504748.

General TermsEconomics, Human Factors, Measurement

KeywordsOnline videos; Advertisements; Monetization; User behav-ior; Internet content delivery; Multimedia.

1. INTRODUCTIONOnline video is the killer application of the Internet. Ac-

cording to a recent Cisco study more than half of the con-sumer tra�c on the Internet today is related to videos andthat fraction is expected to exceed 85% in 2016 [9]. Asall forms of traditional media such as news, entertainmentand sports migrate to the Internet, video on-demand tra�cis expected to triple by 2016 from the levels seen in 2011.Video providers who o↵er online videos include news chan-nels (such as CNN and Fox News), sports channels (suchas ESPN and MLB), movie outlets (such as Hulu and Net-Flix), and entertainment providers (such as NBC, ABC, andCBS). Video providers bear the costs of acquiring and de-livering the videos to their audience of viewers. Acquisitioncosts may include production costs for original content or li-censing costs and/or revenue sharing for third-party content.The delivery costs often involve contracting with a contentdelivery service (such as Akamai [18]), who in turn incur thecosts for the servers, software, bandwidth, colocation, andpower. The runaway success of online videos leaves videoproviders and the media industry with perhaps their singlemost vexing problem. How can online videos be monetized?How can they be made viable and profitable?

While successful models for video monetization are stillevolving, there are broadly three monetization models thatare gaining popularity in the industry. The subscription-based model requires users to pay a fee on a periodic basis(usually monthly) to watch videos. The pay-per-view modelrequires users to pay a fee usually on a per-event basis. Fi-nally, a popular model more relevant to our work is the ad-based model where viewers do not pay a fee but are shownads that are placed in-stream in the video content.1

Driven by both the rapid increase in online video con-sumption and the intense need to monetize that consump-tion, it is perhaps not surprising that video ads were the

1We use the term “video” to describe the video beingwatched, such as a news clip, sports event, or movie. Weuse the term“ad” to indicate the ad that is played in-streamwith the video that is being watched.

fastest growing category of online ads with spending increas-ing by about 50% in 2012 [8]. But, how e↵ective are videoads? Are there general causal rules of viewer behavior thatgovern their e↵ectiveness? What key factors of the ad, ofthe video, and of the viewer influence an ad’s e↵ectiveness?These questions are of great importance to the long-termviability of online videos that are a key part of the Internetecosystem. However, to our knowledge, they have not beenstudied in a rigorous scientific fashion, and hence our focus.

1.1 Understanding Ad EffectivenessThe question of how to measure the e↵ectiveness of a video

ad is complex. Ads convey a message to the viewer and thekey metric for ad e↵ectiveness that is widely used in themedia industry is ad completion rates. Ad completion rateis the percentage of ads that the viewer watched completelywithout abandoning in the middle. Completion rates areperhaps the most tracked metric in an ad campaign sincea viewer watching an ad to completion is more likely tobe influenced by it. A related metric is ad abandonmentrate that measures what fraction of viewers watched whatfraction of the ad. The goal in any advertising campaign isto maximize completion rates and minimize abandonment.

In addition to ad completion, there are a few other met-rics that are tracked that attempt to measure the responseof the viewer after watching the ad. Primary among those isthe click-through rate (CTR) that measures the percentageof users who click on a link associated with the ad dur-ing or after watching the ad. CTR has the advantage overad completion rates of capturing an active user response.Though many have argued that CTRs capture only a imme-diate response but not the long-term impact of the ad thatadvertiser is hoping to achieve [12].

Another class of metrics for ad e↵ectiveness take the moredirect approach of surveying a sample of users who haveviewed the ad to determine how much the ad may have in-creased brand awareness, brand loyalty, and the viewer’sintent to buy. While the directness of the approach is anadvantage, such surveys are di�cult to do at scale and suf-fer from biases that relate to how the questions are framedand who opts to participate in the survey.

While the video ad industry is yet to evolve a consensuson how to integrate the di↵erent ways of measuring ad ef-fectiveness, there is consensus that a basic and importantmeasure is ad completion rate. Thus, we focus on ad com-pletion rate and the associated metric of ad abandonmentrate as indicators of ad e↵ectiveness in our study. Our cur-rent data set does not currently allow us to measure CTRsor survey responses. But, comparing the di↵erent metrics ofad e↵ectiveness is an interesting avenue for future work.

1.2 Our contributionsTo our knowledge, our work is the first in-depth scientific

study of video ads and their e↵ectiveness. We explore howad e↵ectiveness as measured by completion rate is impactedby key properties of the ad, of the video, and of the viewer.A key contribution of our work is that we go beyond simplecharacterization to derive causal rules of viewer behaviorthat are predictive and more generally applicable. To derivesuch rules we develop and use a novel technique based onquasi-experimental designs (QEDs).

Our data set is one of the most extensive cross-sectionsof enterprise videos used in a scientific study of this kind.

The data used in our analysis was collected from 33 videoproviders over a period of 15 days consisting of 362 millionvideos and 257 million ad impressions that were watched by65 million unique viewers located across the world.

The metrics that we study such as completion and aban-donment rates are critical in the media industry and arewidely tracked and reported by ad networks and analyticsproviders. We expect that the deeper scientific understand-ing that our work provides for these metrics will have a sig-nificant impact on the evolution of monetization models forvideo. We now list our specific key contributions below.

(1) “Mid-roll” ads placed in the middle of a video had thehighest completion rate of 97% while “pre-roll” ads placedin the beginning and “post-roll” ads placed in end yieldeddrastically smaller completion rates of 74% and 45% respec-tively. The intuitive reason is that viewers are more engagedwith the video during a mid-roll ad causing them to be morepatient, while viewers are less engaged in the beginning andat the end of the video. By designing a quasi-experiment, weverify the above intuition by showing that the position of anad can causally impact its completion rate. We show that anad is 18.1% more likely to complete when placed as mid-rollthan as a pre-roll, and 14.3% more likely to complete whenplaced as pre-roll than as a post-roll.

(2) 20-second ads had the least completion rate of 60%in our data set, with 15-second and 30-second ads complet-ing at higher rates of 84% and 90% respectively. However,using a quasi-experiment, we show that longer ads are infact less likely to complete. Our causal analysis bolsters ourintuition that viewers have less patience to wait for longerads and would complete fewer of them, provided the otherconfounding factors are kept similar.

(3)Ads played within long-form video such as TV episodesand movies completed at a higher rate of 87%. While adsplayed within short-form video such as news clips completedat a lower rate of 67%. A plausible reason is that viewers aremore willing to complete an ad that they view as a “cost” ifthey perceive a greater “benefit” in return from watching theassociated video. And, on average, viewers tend to perceivegreater benefit from a long-form video than a short-formone. Using a quasi-experiment, we confirm this intuitionby showing that an ad is 4.2% more likely to complete ifplaced in a long-form video than if it is placed in a short-form video, provided all other factors are similar. Note thatthe magnitude of impact of video length on ad completionrates is smaller when confounding factors are accounted forthan what a simple correlation suggests (4.2% versus 20%).

(4) Using information gain ratios, we show that the con-tents of the video and the ad have high relevance for com-pletion rates, while the connectivity of the viewer had thelowest relevance.

(5) Industry folklore suggests that viewers are less likelyto abandon ads when watching them in the evenings or week-ends when they tend to be more relaxed and have more sparetime. However, we did not find any supporting evidence inour data as we did not observe a significant influence of ei-ther time-of-day or day-of-week on ad completion rates.

(6) In our study of ad abandonment rates, we show thata significant set of viewers abandon soon after the ad starts.The abandonment rate is initially higher but tapers o↵ overtime as the ad plays. About one-third of the viewers whoeventually abandon leave in the first quarter of the ad, whileabout two-thirds leave at the half-way point in the ad.

2. BACKGROUND

2.1 The Video Ads EcosystemThe video ad ecosystem consists of four types of entities.

Video providers own and manage video content, e.g., NBC,CBS, CNN, Hulu, Fox News, etc. Advertisers o↵er ads thatcan be played in-stream with the video. Ad insertion ismanaged by an ad delivery network such as Freewheel [5],Adobe Auditude [1], or Video Plaza [7]. The ad networkbrings together the video providers (i.e., publishers) whoo↵er videos and the advertisers who o↵er ads. An ad net-work has an ad decision component that decides what adsto play with which videos and where to position those ads.Both the ads from the advertisers and the videos from thevideo providers need to be streamed to the viewer with highperformance. For that reason, both the ads and videos aretypically delivered to the viewers using content delivery net-works (CDNs) such as Akamai [10, 18]. Thus, CDNs are cog-nizant of both the video content and ads embedded withinthem. The mechanism for inserting the ad is commonlyperformed by the user’s media player when it is playing thevideo. When it is time to play an ad, the media playerredirects to the ad network that choses the ad and plays itwithin an ad player.When the ad completes the control re-turns to the user’s media player that continues to play thevideo content.

2.2 Views, viewers, and visitsWe describe a user watching a video with ads, defining

terms along the way that we will use in this paper.Viewer. A viewer is a user who watches one or more videos

using a specific media player installed on the user’s device.A viewer is uniquely identified and distinguished from otherviewers by using a GUID (Globally Unique Identifier) valuethat is set as a cookie when the media player is accessed. Toidentify the viewer uniquely, the GUID2 value is generatedto be distinct from other prior values in use.

Views. A view represents an attempt by a viewer to watcha specific video. A typical view would start with the viewerinitiating the video playback, for instance, by clicking theplay button of the media player3 (see Figure 1). During aview, the media player might first play an ad called a pre-roll before the actual video content begins. (We use theterm video to denote the actual video that the viewer wantsto watch to distinguish it from the ad that the viewer isalso shown.) Further, the video may be interrupted in themiddle with one or more ads called mid-rolls. Finally, an admight be shown when the video ends called a post-roll. Eachshowing of an ad, whether or not it is watched completely,is called an ad impression. Ad completion rate is the percentof ad impressions that were played to completion4.

Typically, viewers do not have the ability to “skip” adsand must either watch the ad in order to watch the videocontent that follows, or just abandon the ad and the view

2In most implementations, the GUID is tied to the deviceor the desktop of the viewer. Thus, we cannot always detectcases where one user watches video on another user’s device.3In other cases, a view may be initiated automatically usinga play list or by other means.4If an ad is played completely it is likely that it was watchedcompletely by the viewer. However, we are not able to mea-sure whether or not a viewer is actually watching the ad orif he/she has shifted focus elsewhere when the ad is playing.

Type Factor Description

AdContent defined by unique namePosition Pre-, mid-, post-rollLength 15-, 20-, and 30-second

VideoContent defined by unique urlLength Short-form, Long-formProvider News, Movie, Sports, En-

tertainment

ViewerIdentity defined by unique GUIDGeography Country and ContinentConnection Type Mobile, DSL, Cable, FiberTemporal Time of day, Day of week

Table 1: Potential factors that influence viewer behavior andad completion.

all together. Our data sets have such non-skippable adsthat is the standard for enterprise video. Recently YouTubethat has a large fraction of user-generated videos has beenexperimenting with pre-roll ads that have a mandatory non-skippable part that must be viewed but can be skipped be-yond that point. But, it is not yet common within enterprisevideos and is not represented in our data set.

Ad Video Ad VideoPlayerStates

View begins

Pre-roll Mid-roll

Post-roll

View

Visit Visit

More than T minutes of

inactivity

Views

Ad

Viewends

Figure 1: Views and Visits

Visits. A visit is intended to capture a single session ofa viewer visiting a content provider’s site to view videos. Avisit is a maximal set of contiguous views from a viewer at aspecific video provider site such that each visit is separatedfrom the next visit by at least T minutes of inactivity, wherewe choose T = 30 minutes5(see Figure 1).

2.3 Potential factors that impact ad comple-tion

We study three sets of key factors that potentially influ-ence ad completion that we list in Table 1 and discuss below.

Ad-related factors. The actual ad and its contents as iden-tified by its unique name is a first factor. Ad position relateswhere it was inserted in a video view and can either be pre-roll, mid-roll, or post-roll. The most common ad lengths inour study are 15-second, 20-second, and 30-second ads.

5Our definition is similar to the standard notion of a visit(also called a session) in web analytics where each visit isa set of page views separated by a period of idleness of atleast 30 minutes (say) from the next visit.

Video-related factors. The first factor is the video contentitself as identified uniquely by its url6. Besides the actualcontent of the video itself, we isolate two important factors.The video length can be used to di↵erentiate short-form fromlong-form videos. The IAB (Interactive Advertising Bureau)which is a major industry body for online video advertisingdefines long-form video as videos lasting over 10 minutesand short-form as those under 10 minutes [6]. We adoptthis standard definition in our work. Typically, short-formand long-form videos are qualitatively di↵erent. Short-formvideo are usually smaller clips for news, weather, etc. Long-form videos are typically TV episodes, movies, sports events,etc. Most long-form videos possess a “content arc” with abeginning, middle and end.

Viewer-related factors. A viewer can be uniquely andanonymously identified by their GUID. Besides their iden-tity, we consider three important attributes of the viewer.The geographical location of the viewer at the country levelencapsulates several social, economic, religious, and culturalaspects that could influence his/her viewing behavior. Inaddition, the manner in which a viewer connects to the In-ternet, both the device used and typical connectivity char-acteristics can influence viewer behavior. The primary con-nection types are mobile, DSL, cable, and fiber (such asAT&T’s Uverse and Verizon’s FiOS). Finally, it is plausiblethat the time-of-day and the day-of-week7 when the ad waswatched could potentially influence its completion rate. Forinstance, folklore holds that people have more time in theweekend and evenings, leading them to be more relaxed andmore patient with video ads. However, as we show in Sec-tion 5.3.3, we did not observe a significant influence of eithertime-of-day or the day-of-week on ad completion rates.

It is worth noting that many of the video and viewer re-lated factors considered here are known to significantly im-pact viewer behavior in the context of viewer tolerance toperformance degradation from our prior work [14]. So, it isnatural to consider these in the di↵erent behavioral contextof viewer tolerance to ads. The ad-related factors consideredhere, besides being natural to consider, are widely trackedin industry benchmarks.

3. DATA SETSThe data sets that we use for our analysis are collected

from a large cross section of actual users around the worldwho play videos using media players that incorporate thewidely-deployed Akamai’s client-side media analytics plug-in [2]. When video providers build their media player, theycan choose to incorporate the plugin that provides an accu-rate means for measuring a variety of video and ad metrics.When the viewer uses the media player to play a video, theplugin is loaded at the client-side and it “listens”and recordsa variety of events that can then be used to stitch togetheran accurate picture of exactly what the viewer experiencedand what the viewer did.

When a view is initiated, say with the user clicking theplay button, metrics such as the time when the view was

6Note that if two di↵erent providers are showing the samemovie with di↵erent urls, we consider them di↵erent videos.Detecting them to be the same content is intrinsically verydi�cult as there is no universally accepted naming systemacross video providers.7Time-of-day and day-of-week is computed using the localtime for the viewer based on his/her geographical location.

initiated, the video url that uniquely identifies the content,video length, whether it is live or on-demand, the videoprovider, the amount of video watched, the bitrate(s) atwhich it was streamed, and several other detailed character-istics pertaining to the video are recorded. Likewise, whenan ad is inserted, ad-related metrics such at what point inthe video the ad was inserted, the ad name that uniquelyidentifies the content of the ad, the ad length, the amountof the ad that was actually played, and whether the adcompleted or not are recorded. Further, detailed informa-tion about the viewer is recorded including the GUID thatuniquely identifies the viewer, current ip address, network,geography, and connection type. Once the metrics are cap-tured by the plugin, the information is “beaconed” to ananalytics backend that we use to process the huge volumesof data. From every media player at the beginning and endof every view, the relevant measurements are sent to the an-alytics backend. Further, incremental updates are sent ata configurable periodicity as the video is playing, typicallyonce every 300 seconds. All relevant fields in the data setused in our study are measured and stored in an anonymizedfashion so as to not include any PII or sensitive information.

3.1 Data CharacteristicsThe Akamai CDN serves a significant fraction of world’s

online videos and ads. We selected a large, characteris-tic cross-section of 33 video providers including news sites,sports sites, movie providers, and entertainment channelswho have an ad-based monetization model. We tracked allvideos and ads for these providers over a period of 15 daysin April 2013. About 94% of the video views were for on-demand content and the rest were live events. We only con-sider on-demand videos that currently form the bulk of thevideos for our study.

Our data is among the most extensive ever studied forvideo ads and consisted of 257 million ad impressions thatwere watched by over 65 million unique viewers located inall major continents of the world. On average, viewers spent8.8% of their time watching ads as opposed videos. Table 2summarizes some basic statistics of our data. The geography

Total Per Per PerView Visit Viewer

Views 362 mil N/A 1.3 5.6Ad 257 mill 0.71 0.92 3.95

ImpressionsVideo Play 777 mil 2.15 2.79 11.96(in minutes)Ad Play 75 mil 0.21 0.27 1.15

(in minutes)

Table 2: Key statistics of our data set.

of the viewer was mostly concentrated in North America,and Europe that together originate the bulk of video tra�c.One continent where we could not obtain proportional rep-resentation is Asia that accounts for significant video tra�cbut where many video providers do not yet have the soft-ware changes required in their media player to track ads.The connection types were dominated by cable, though theother categories have a solid representation as well (cf. Ta-ble 3).

Viewer Percent Connection PercentGeography Views Type Views

North America 65.56% Fiber 17.14%Europe 29.72% Cable 56.95%Asia 1.95% DSL 19.78%Other 2.77% Mobile 6.05%

Table 3: Geography and connection type.

Figure 2: CDF of ad length showing the three major clustersat 15-, 20- and 30-seconds.

The ad length distribution is shown in Figure 2. The adlengths clustered around 15-, 20- and 30-second marks andwere clustered into those categories respectively.

The distribution of the video lengths for short-form andlong-form videos are shown in Figure 3. The mean lengthof a short-form video is 2.9 minutes and that of long-formvideo is 30.7 minutes. The most popular duration for long-form video was 30 minutes that is typical for a TV episode.

4. ANALYSIS TECHNIQUESWe seek to understand how a set of key factors such as

those in Table 1 impact viewer behavior metrics such as adcompletion and abandonment rates. We use correlationaltools such as Kendall correlation and information gain de-scribed in Section 4.1 to characterize the observed data.While correlational analysis is important as a descriptionof what is, they don’t necessarily have the ability to predictwhat will be. The ability to predict often requires a deeperinference of a causal rule between the key factor and theviewer behavior metric. To provide an example, a correla-tional analysis of the observed data will be able to say thatmid-roll ads have a higher completion rate than pre-roll adsin the observed data. However, this fact does necessarily im-ply a causal rule that states that placing an ad as mid-rollwill likely cause higher completions than placing the same

Figure 3: CDF of video length for long-form and short-formvideos.

ad as a pre-roll. The value of causal inference over a purelycorrelational one is that it extracts general rules of viewerbehavior from the data that can be applied to a more generalor even di↵erent context. In our prior work [14], we intro-duced an innovative tool called Quasi Experimental Design(QED) that we adapted from the social and medical sciencesfor use in network measurement research. In this work, wetake the next step and further evolve this technique to ex-tract causal rules pertaining to video ads. We describe ourtechnique in Section 4.2.

4.1 Correlational AnalysisTo study the impact of a key factor X (say, ad length) with

a viewer behavioral metric Y (say, completion rate), we startby visually plotting factor X versus metric Y in the observeddata. Then, when relevant, we compute Kendall’s correla-tion coe�cient ⌧ that takes values in the interval [�1, 1]where ⌧ near 1 means that larger values of X are associatedwith larger values for Y , ⌧ near �1 means that larger valuesof X are associated with smaller values of Y , and ⌧ near 0means that X and Y are independent.

A key technique that we use to quantify the influence offactor X on metric Y is the information gain ratio [13]. In-formation gain ratio measures the extent to which the vari-ability of Y is reduced by knowing X. That is, informationgain is the entropy of Y (denoted by H(Y )) minus the en-tropy of Y given X (denoted by H(Y |X)). Normalizing theinformation gain, we obtain the information gain ratio de-noted by

IGR(Y,X) =H(Y )�H(Y |X)

H(Y )⇥ 100.

It is instructive to view the two extreme cases. Supposeknowing X perfectly predicts Y , then H(Y |X) is zero sincethere is no variability left in Y and IGR(X,Y ) is 100%. Inthe other extreme, suppose that X and Y are statisticallyindependent. In that case, H(Y |X) simply equals H(Y )

since knowing X says nothing about Y and IGR(Y,X) is0%. In all our results, IGR is somewhere in between and isa quantitative indicator of the extent of a factor’s influenceon a viewer behavioral outcome.

4.2 Causal Analysis using QEDsA correlational analysis of factor X (say, ad length) and a

viewer behavior metric Y (say, completion rate) could showthat X and Y are associated with each other. A primarythreat to a causal conclusion that an independent variable Xcauses the dependent variable Y is the existence of confound-ing variables that can impact both X and Y . To take anexample that we describe in greater detail in Section 5.1.3,suppose we want to infer a causal rule that a longer ad causesviewers to complete the ad less often. Simply correlatingcompletion rate and ad length is not su�cient. In fact, 20-second ads complete less often than 30-second ads in theobserved data, apparently violating the rule. To derive acausal conclusion, one would need to account for the con-founding factor of ad position, since 30-second ads are oftenplaced as mid-rolls and as we show mid-rolls have a highercompletion rate independent of length.

A primary technique for showing that an independentvariable X (also called the treatment variable) has a causalimpact on a dependent variable Y (called the outcome vari-able) is to design a quasi-experiment . Quasi-experimentswere developed by social and medical scientists and has morethan 150 years of history in those domains [20]. In partic-ular, we use a specific type of QED called the matched de-sign [19] where a treated individual (in our case, a view orviewer) is randomly matched with an untreated individual,where both individuals have similar values for the confound-ing variables. Consequently, any di↵erence in the outcomefor this pair can be attributed to the treatment. By creatinga large collection of matched pairs and assessing the di↵er-ential outcome of the paired individuals, one can isolate thecausal impact of X on Y .

Adapting QEDs to our situation, our population typicallyconsists of views. The independent variable is one of thefactors in Table 1 (say, ad position). The treated and un-treated sets have two di↵erent values of the independentvariable that we want to compare (say, mid-roll versus pre-roll). Our outcome is a function of the behavioral metricunder study, such as ad completion. The confounding fac-tors that need to be matched so that they have similar valuesare typically other key factors in Table 1 except the inde-pendent variable that is varied, since the other factors couldconfound the outcome. We form comparison sets by ran-domly matching each treated view with an untreated viewsuch that they have similar values for the confounding vari-ables and di↵er only in the independent variable. For in-stance, to study the impact of ad position, we match viewsthat belong to similar viewers watching the same ad withinthe same video, neutralizing the impact of the confoundingvariables. By forming a large number of such pairs and bystudying the behavioral outcomes of matched pairs one candeduce whether or not the treatment variable X has a causale↵ect on variable Y , with the influence of the confoundingvariables neutralized.

Statistical Significance of the QED Analysis.

As with any statistical analysis, it is important to evalu-ate whether the results are statistically significant or if they

Type Factor IGR

AdContent 32.29%Position l5.1%Length 12.79%

VideoContent 23.92%Length 18.24%Provider 15.24%

ViewerIdentity 59.2%Geography 9.57%Connection Type 1.82%

Table 4: Information gain ratio (IGR) for ad completionrate.

could have occurred by random chance. As is customaryin hypothesis testing [16], we compute the p-value whichevaluates the probability that the observed outcome from aQED happened by chance, assuming that the null hypothe-sis holds. The null hypothesis states that there is no impactof the treatment on the outcome. To evaluate the p-valuewe use the sign test that is a non-parametric test that makesno distributional assumptions and is particularly well-suitedfor evaluating matched pairs in a QED setting [21]. A lowp-value means that our results are statistically significant.The choice of the threshold is somewhat arbitrary and inmedical sciences a treatment is considered e↵ective if the p-value is at most 0.05. We can achieve much higher levels ofsignificance owing to the large numbers of treated-untreatedpairs in our QEDs (order of 100,000) in relation to what istypical in the medical sciences (in the 100’s). However, ourresults are unambiguously significant and not very sensitiveto the choice of significance level. We refer to our prior work[14] for a more detailed treatment of QEDs.

Some Caveats.

It is important to understand the limitations of our QEDtools, or for that matter any experimental technique of infer-ence. Care should be taken in designing the quasi-experimentto ensure that the major confounding variables are explicitlyor implicitly captured in the analysis. If there exists con-founding variables that are not easily measurable (example,the gender of the viewer) and/or are not identified and con-trolled, these unaccounted dimensions could pose a risk to acausal conclusion, but only if they turn out to be significant.Our work on deriving a causal relationship by systematicallyaccounting for the confounding variables must not be viewedas a definitive proof of causality, as indeed there can be nodefinitive proof of causality. But, rather, our work increasesthe confidence in a causal conclusion by accounting for po-tential major sources of confounding. This is of course ageneral caveat that holds for all domains across the sciencesthat attempt to infer causality from observational data.

5. AD COMPLETION RATEWe study the ad completion rate metric that is a key mea-

sure of ad e↵ectiveness. The completion rate can be influ-enced by the characteristics of three entities that we examinein turn: the ads themselves, the videos that the ads are em-bedded in, and the viewer who is watching the videos andads. We evaluate the relevance of each of these factors to thecompletion rate by computing their information gain ratioshown in Table 4. Enormous e↵ort goes into creating the

Figure 4: The percent of ad impressions y attributed toads with ad completion rate smaller than x. 50% of the adimpressions are from ads with completion rate at most 91%.

ad and video content to make it as captivating as possible.It is interesting that both show high information gain, per-haps indicating that content does matter.The informationgain ratio of the viewer is very high. This is at least in partdue to the fact that 51% of the viewers watched only one adresulting in either a 0% or 100% completion rate. In thosecases, knowing the viewer perfectly predicts the completionrate. Information gain is known to be counter-intuitive forfactors like viewer that can take millions of values each withsmall individual weights. The information gain from con-nection type was the least, as viewers showed lesser varia-tions in their patience for completing ads across the di↵erentconnection types. This is in contrast with our earlier workon viewer patience in the context of video performance [14]where viewers with worse connectivity had more patiencefor a video to start up.

5.1 Impact of Ad-related FactorsWe examine three factors that relate to the ad itself: the

ad’s content as identified by its unique name, the positionin which the ad was played, and the length of the ad.

5.1.1 Ad Content

For each unique ad, we can define its completion rateas simply the fraction of ad impressions where the ad waswatched to completion by the viewer. We plot the percentof ad impressions (y-axis) attributed to ads with completionrate smaller than some x-value (cf. Figure 4). We can seefrom the figure that ads complete at varying rates with somehaving low completion rates with others completing 90+%of the time. Further, 25% the ad impressions come from adswith completion rate under 66%, and 50% come from adswith completion rate under 91%.

5.1.2 Ad Position

We analyze the impact of ad position on the likelihoodthat a viewer watches the ad to completion. We first take

Figure 5: Mid-roll ads complete most often as the viewer isalready engaged by the video and wants to watch more.

a simple correlational approach of categorizing the positionin which the ad was played and computing the completionrates for each category. Our analysis shows that mid-rollads completed most often, followed by pre-roll and post-rollads (cf. Figure 5).

Assessing Causal Impact.

Our observational results support the intuition that adsplaced in the middle of the content has the most likelihoodof being watched, since the viewer is engaged with the con-tent when the ad is shown, wants to watch the rest of thevideo, and is thus more willing to tolerate the ad. Whereasads placed as pre-roll run a greater risk of viewers aban-doning and going elsewhere, since they have not yet startedwatching their content and hence are not yet engaged withit. Further, an ad placed at the end of the content as apost-roll runs an even greater risk of viewers leaving sincethe content that they wanted to watch has completed, andso they are less motivated to sit through an ad. Based onour observational results, we assert the following causal rule.

Rule 5.1. On average, a viewer is more likely to completewatching an ad that is placed as a mid-roll than when thesame ad is placed as a pre-roll. In turn, a viewer is morelikely to complete watching an ad that is placed as a pre-rollthan when the same ad is placed as a post-roll.

Note that the correlational analysis in Figure 5 is not su�-cient to show that the rule holds, as there are potential con-founding factors such as the ad length, video length, contentprovider, viewer geography, and viewer connectivity that cannegate such an assertion. For instance, the following plausi-ble scenario could be threat to our asserted rule. It could bepossible that mid-roll ads appear largely in longer contentsuch as TV episodes and movies, and perhaps ads placed inlonger content have higher completion rates than ads placedin shorter content irrespective of their position. Thus, mid-roll ads could have a higher completion, not by virtue of

Matching Algorithm

Matched: similar viewer, same ad, same video.Independent: ad position.

1. Match step. Let the treated set T be the set of allviews that had a mid-roll ad and let the untreatedset C be the set of all views that had a pre-roll ad.For each u 2 T that had some ad ↵ as mid-roll,we pick uniformly and randomly from the set ofcandidate views v 2 C such that u and v belongto similar viewers with the same geography andconnection type who are watching the same videoand the same ad ↵, except that the ad ↵ wasplayed as mid-roll in u but played as pre-roll inva. The matched set of pairs M ✓ T ⇥ C havethe same or similar attributes for the confoundingvariables that are matched and di↵er only on theindependent variable.

2. Score step. For each pair (u, v) 2 M , we computean outcome(u, v) to be +1 if the matched ad wascompleted in u but not completed in v, - 1 if thematched ad was completed in v but not in u, and0 otherwise. Now,

Net Outcome =

P(u,v)2M outcome(u, v)

|M | ⇥ 100.

aIf no match v exists for a u, then no pair is formed.

Figure 6: The matching algorithm that compares ads placedas mid-roll (treated) versus pre-roll (untreated) while ac-counting for the other confounding variables such as the ad,video, and viewer characteristics.

them being placed in the middle of the content, but simplyby being more likely to be placed in longer content.

QED. To carefully assess the impact of ad position in iso-lation by accounting for other potential confounding factors,we design a quasi-experiment as described by the matchingalgorithm in Figure 6. To compare the e↵ect of placingan ad as mid-roll versus placing as pre-roll, the algorithmfinds matched views (u, v) from two similar viewers who havethe same connection type and geography. Further, the twomatched views are for exactly the same video and the samead. The primary di↵erence between the matched views isthat the ad was played in di↵erent positions, i.e., one view uhas the ad as a mid-roll while the other view v has the samead as a pre-roll. Note that a positive value for net outcomeprovides positive (supporting) evidence for the rule that anad in mid-roll completes more often than the same ad aspre-roll, while a negative value provides negative evidencefor the asserted rule. The algorithm in Figure 6 can be usedto compare any pair of ad positions with minor modifica-tions. For instance, to compare pre-roll with post-roll, wecan apply the same algorithm with pre-roll as the treatedset T and post-roll as the untreated set C.

QED Results. The results for the quasi-experiment areshown in Table 5. These results show that on average adsrun in the mid-roll position are 18.1% percent more likely tocomplete than the same ad run in the pre-roll position forthe same video content for a similar viewer. Further, ads

Treated/Untreated Net Outcomemid-roll/pre-roll 18.1%pre-roll/post-roll 14.3%

Table 5: Net QED outcomes support the rule that placingan ad as a mid-roll can cause greater completions than as apre-roll or as a post-roll.

run in the pre-roll position are 14.3% percent more likelyto complete than the same ad run in the post-roll posi-tion for the same video content for a similar viewer. Theresults confirm the causal impact of ad position on comple-tion rates and establish Rule 5.1 in a causal and quantitativemanner. Further, using the sign test, the p-value for eachquasi-experiment was at most 1.98⇥ 10�323, confirming thestatistical significance of the results.

Discussion.

(1) Note that the impact of ad position on ad completionrates turns out to be smaller (but still significant) when theconfounding factors are accounted for than in the simplercorrelational analysis of Figure 5.(2) If mid-rolls are so e↵ective, why not place only mid-rollads? While our results show that positioning an ad as mid-roll increases its likelihood of completion, it is not a recom-mendation for advertisers to place only mid-roll ads. If anad network wants to achieve a certain number of completedad impressions one needs to worry about both the audiencesize and the ad completion rate. Audience size for pre-rollads are larger than mid-roll ads simply because viewers dropo↵ before the video progresses to a point where a mid-rollad can be played. Likewise, the audience size of a mid-rollad is typically larger than that of a post-roll ad. Thus, anad positioning algorithm would have to carefully considerthis tradeo↵ when deciding where to place ads. Our workprovides an important input to such an algorithm, thoughdesigning optimal ad placement algorithms is beyond thescope of our work. However, our results do show that post-roll ads are generally inferior to mid-roll and pre-roll ads,since post roll ads have both smaller audience sizes and lesserad completion rates.

5.1.3 Ad Length

We classify each ad into the three common categories,15-second, 20-second, and 30-second ads, and compute thecompletion rate for each category (cf. Figure 7). Ads oflength 30 seconds had the highest completion rate and 20-second ads have the least. A fundamental question is howthe ad length causally influences its completion rate. With apurely correlational analysis such as that shown in Figure 7,one is liable to incorrectly conclude that 20-second ads aredetrimental to ad completion and the sweet spots are 15-second and 30-second ads. Further, the results appear tocontradict the intuition that longer ads are more likely to beabandoned, since viewers are more likely to loose patience.

To dig deeper, we analyzed the ad positions of the di↵erentad lengths (cf. Figure 8). We noticed that 30-second ads areplaced most often as mid-rolls since advertisers intuitivelyrealize what we quantified in Section 5.1.2 that the viewer ismore engaged in the middle of video and tend to place theirlongest ads there. Thus, the observed high completion ratefor 30-second ads could be an influence of its ad position

Figure 7: The measured ad completion rates in our data setdid not decrease with ad length as expected. The 30-secondad, while longer, had the highest completion rate in part dueto being placed more frequently in the mid-roll position.

that counteracts its larger length. Further, 15-second adsare placed most often as pre-roll and 20-second ads have agreater chance of being a post-roll than other ad lengths. Wehave to compensate for the confounding e↵ect of variablessuch as ad position to isolate the true impact of ad lengthon ad completion rates. To that end, we design the quasi-experiment described below.

Assessing Causal Impact.

QED. We design a quasi-experiment where the indepen-dent variable is the ad length (15-second, 20-second, or 30-second) and the other potential confounding variables arematched. For a given pair of ad-lengths x 6= y and x, y 2{15, 20, 30} seconds, we design a quasi-experiment with thetreated set consisting of videos that contained an ad of lengthx and the untreated set consisting of videos that containedan ad of length y. The matching algorithm that we use issimilar to that in Figure 6 with the following di↵erences.When forming the matched pair of views (u, v) 2 M , weensure that view u played an ad of length x and view vplayed an ad of length y. To account for the influence ofad position, we ensure that the ads were played in the sameposition. Further, we ensure that the viewers of u and vare similar with the same geography and connection typeand are watching exactly same video. The scoring step isidentical to the matching algorithm of Figure 6.QED Results. The results of the quasi-experiments are

shown in Table 6. Our results show that 15-second adscompleted 2.86% more often than the 20-second ones in ahead-to-head comparison that accounts for the confound-ing factors. Likewise, 20-second ads completed 3.89% moreoften than the 30-second ones in the head-to-head compari-son. Further, using the sign test, the p-value for the quasi-experiment is at most 8.52⇥10�30, confirming the statisticalsignificance of the results. Thus we state the following rule.

Figure 8: 30-second ads are most commonly mid-rolls, and15-second ads most commonly pre-rolls. 20-second ads aremore often post-rolls than other lengths.

Treated/Untreated Net Outcome15 sec/20 sec 2.86%20 sec/30 sec 3.89%

Table 6: Net QED outcomes support the assertion thatlonger ads result in fewer completions.

Rule 5.2. On average, a shorter ad is more likely to com-plete than a longer ad, when the other confounding factorsare neutralized.

5.2 Impact of Video-related FactorsWe examine two factors that relate to the video: its con-

tent as identified by its unique url, and its length.

5.2.1 Video Content

People typically watch ads so that they are allowed towatch the video. Therefore, it is reasonable to ask whatinfluence the video itself has on the completion rate of theads embedded within it. Videos in our traces are uniquelyidentified by their urls. A video could have been viewedmultiple times, and multiple ads could have been shown aspart of each view. The ad completion rate of a video issimply the percentage of all ad impressions shown with thatvideo that completed. Ad completion rate of a video is notto be confused with the unrelated metric of video completionrate that relates to whether the video itself completed or not.One could imagine that the ad completion rates vary fromvideo to video, with videos with compelling content havinghigh ad completion rates and videos with boring contenthaving lower ad completion rates. In Figure 9, we do indeedsee a large variation in ad completion rate across di↵erentvideos with half the ad impressions coming from videos thathave an ad completion rates of 90% or smaller.

5.2.2 Video Length

We narrow our focus to the length of video to assess howit relates to the ad completion rate. We bucket the video

Figure 9: The percentage of ad impressions y% from videoswith ad completion rate at most x%, plotted in 5% bucketsof ad completion rate. Half the ad impressions belonged tovideos with completion rate 90% or smaller.

length into one minute buckets and compute the average adcompletion rate of the videos in each bucket. (Each video isweighted by the number of ad impression shown with thatvideo for computing the average.) We plot ad completionrate as a function of the video length in Figure 10. Thead completion rate shows a positive correlation with videolength with Kendall correlation of 0.23. One can furtherbucket the videos according to whether they are short-formor long-form and one can see that short-form video has asmaller ad completion rate than the long-form (cf. Fig-ure 11).

Our initial correlational results support the intuition thata viewer exhibits more patience for an ad to complete if theyare watching long-form content such as a TV episode or amovie that are often perceived to be of greater value thanshort-form content. Such a phenomena is known to holdin the physical world where researchers who study the psy-chology of queuing [15] have shown that people have morepatience for waiting in longer queues if the perceived valueof the service that they are waiting for is greater. Dura-tion of the service often influences its perceived value withlonger durations often perceived as having greater value. In[14], we showed that viewers are more likely to wait withoutabandoning for a longer video to startup than a shorter one.Our current work implies that a similar phenomenon holdsfor viewer patience for ads to complete.

Assessing Causal Impact.

Based on our correlational evidence above, we would liketo establish the following causal rule by a carefully designedquasi-experiment.

Rule 5.3. On average, placing an ad in long-form videocan cause a greater completion rate in comparison to placingthe same ad in a short-form video.

Figure 10: Ad completion rate and video length have a pos-itive correlation with a Kendall coe�cient of 0.23.

QED. We conduct a quasi-experiment where the indepen-dent variable is the video length (long-form versus short-form) and other potential confounding variables are matched.The matching algorithm that we use is similar to that in Fig-ure 6 with the following di↵erences. Since the independentvariable is video length, the treated set consists of long-formvideos with ads, while the untreated set consists of short-form videos with ads. When forming the matched pair ofviews (u, v) 2 M , we ensure that the paired views playedthe same ad in the same position, i.e., the ad was pre-roll,mid-roll, or post-roll in both views. Further, the viewers ofu and v are similar in that they are from the same geographyand have the same connection type. Finally, even though uand v are watching di↵erent videos, one long-form and theother short-form, we ensure that they are watching videosfrom the same video provider. The scoring step is identicalto the matching algorithm of Figure 6.

QED Results. The results of the quasi-experiment pro-duced a net outcome of 4.2%. The positive net outcomesupports Rule 5.3 by showing that on average an ad that isplaced in long-form video is 4.2% more likely to completethan the same ad placed in short-form video. Further, usingthe sign test, the p-value for the quasi-experiment was atmost 9.9 ⇥ 10�324, confirming the statistical significance ofthe results.

Discussion.

The impact of video length on ad completion rate is con-founded by factors such as ad position. For instance, mid-roll ads that tend to have higher completion rates are morecommonly embedded in long-form video than in short-formvideo. Thus, the influence of ad position must be neutral-ized to get a clearer picture of the impact of video lengthin isolation. Accounting for such confounding factors in theQED analysis shows a smaller (but still significant) impactof video length, though that impact is smaller than what isimplied by the simpler analysis of Figure 11.

Figure 11: Ads embedded in long-form video such as a TVepisode or a movie complete more often than ads embeddedin short-form video such as a news clip.

5.3 Impact of Viewer-related FactorsWe examine three factors that relate to the viewer in more

depth: the viewer as identified by his/her unique GUID,the viewer’s geographical location, and the temporal factorswhen the ad was played.

5.3.1 Viewer’s identity

We compute the ad completion rate of each viewer as sim-ply the percentage of ad impressions that the viewer watchedto completion. In Figure 12, we plot the percent of ad im-pressions y% that were watched by viewers with completionrate less than or equal to x%. One can notice the concen-trations of viewers around completion rates of 0%, 50%, and100%. These concentrations are an artifact of the fact thata large fraction of viewers see a small number of ads. Forinstance, 51.2% see one ad contributing to concentrationsaround 0% and 100%. And, 20.9% see only two ads, con-tributing to concentrations around integer multiples of 1/2.More generally, one can observe concentrations around inte-ger multiples of 1/i, where i is a small integer.

5.3.2 Geography

In Figure 13, we show the ad completion rates across dif-ferent continents in the world. Perhaps the most strikingcontrast are between the two most tra�cked continents withEurope having the lowest completion rate and North Amer-ica having the highest.

5.3.3 Temporal factors

A plausible hypothesis that exists as a folklore is thatviewers are more likely to watch ads (and complete them)in the weekend or in the evenings where they tend to be morerelaxed, more patient, and have more spare time. Indeed,both video and ad viewership peaks in the late evening asshown in Figures 14 and 15 respectively. However, as shownin Figure 16, ad completion rates did not show much time-

Figure 12: The percentage of ad impressions y% from view-ers with completion rate at most x%.

Figure 13: Europe has the smallest completion rate whileNorth America has the greatest.

of-day variation and were nearly identical between weekdayand weekend.

6. AD ABANDONMENT RATEWhile ad completion rates measure whether viewers com-

plete watching an ad or not, ad abandonment rates mea-sure what portion of the ad was played before the viewerabandoned. Thus, abandonment rates provide more granu-lar information than completion rates. We define metricswe use to study abandonment. Suppose we have an ad oflength L time units. Ad play time x, 0 x L, refers tothe amount of time that the ad was played by the viewer

Figure 14: Video viewership is high during the day, dipsslightly in the evening, and peaks in the late evening.

during an ad impression. The abandonment rate at time x,0 x L, is the percentage of ad impressions that have adplay time less than x, i.e., the percentage of ad impressionswhere the ad was watched for fewer than x time units. Bydefinition, the abandonment rate of the ad at time x = L is100 minus that ad’s completion rate, since viewers who didnot abandon and watched all L time units completed thead. When aggregating abandonment rates across ads withdi↵erent lengths, we plot ad abandonment rate as a func-tion of a normalized value called ad play percentage which is(ad play time/ad length)⇥ 100. Further, we define normal-ized abandonment rate to be

(ad abandonment rate/(100� ad completion rate))⇥ 100.

Aggregated over all ad impressions in our study, the aban-donment rate when ad play percentage equals 100% is 17.9%,which equals 100 minus the system-wide completion rate of82.1%. In Figure 17, we plot the normalized abandonmentrate as a function of the ad play percentage. Normalized adabandonment rate is a concave function with viewers aban-doning at a greater rate initially that subsequently taperso↵. One can observe from the figure that when 25% of thead is played, the normalized abandonment rate is already33.3%, i.e., one-third of the viewers who eventually aban-don have abandoned on or before the quarter-way mark inthe ad. Likewise, at 50%, the normalized abandonment rateis 67%, i.e., two-thirds of the viewers who eventually aban-don have abandoned on or before the half-way mark in thead.

Next, in Figure 18, we plot the normalized abandonmentrate as a function of ad play time to examine how viewersabandon for each of the three ad lengths. By definition,the three abandonment curves reach the normalized aban-donment rate of 100% at 15, 20, and 30 seconds respectively.However, the normalized abandonment rates are nearly iden-tical for the first few seconds and diverge beyond that point.This suggests that perhaps a significant fraction of viewersabandon as soon as the ad starts independent of its length.

Figure 15: Ad viewership roughly follows the same trend asvideo viewership.

Figure 16: However, ad completion rates do not show majorweekday/weekend or time-of-day variations.

Finally, in Figure 19, we show the normalized abandon-ment rate for the di↵erent connection types. Our results donot show major di↵erences between the four major connec-tion types for when viewers who eventually abandon stopwatching the ad. One plausible explanation could be thatviewers have a similar expectations on how long they wouldhave to wait for an ad to complete, independent of theirconnectivity. This could be contrasted with the situationwhere viewers are waiting for a video to start up after aplay is initiated. In this situation, viewers with high-speedconnectivity (say, fiber) rightfully expect the video to startup sooner than viewers on a mobile connection. Indeed, inthis situation we showed in our prior work [14] that viewerswith faster connectivity abandoned the video sooner than

Figure 17: Normalized abandonment rate as a function ofad play percentage has a concave form. Of the viewers whoeventually abandon the ad, about a third of the them haveabandoned before the quarter-way mark and two-thirds ofthem have abandoned before the half-way mark.

those with slower connectivity, presumably because the for-mer had greater expectations for a quicker startup and henceshowed less patience for the video to start up.

7. RELATED WORKWe are not aware of large-scale scientific studies of video

ads and their impact akin to our work. However, given itsimportance, the metrics that we study such as ad comple-tion rate, abandonment rate are widely reported on a quar-terly or yearly basis by ad networks such as FreeWheel [5],Adobe[1], and Bright Roll [3] and analytics providers suchas comScore [4]. Since the business of online video relies onad completion rates, audience size and other such metrics,the major industry standards body IAB [6] provides guide-lines on how such video monetization metrics ought to bemeasured. Our work on systematically understanding theimpact of various factors on ad viewing behavior and ex-tracting general rules via quasi-experiments is unique andsignificantly contributes to our scientific understanding ofad e�cacy and video monetization.

There has recently been research on understanding theimpact of video performance on viewer behavior [11, 14],and in the use of client-side measurements for better videodelivery [17]. These work share a commonality with ourcurrent work in the sense of using large amounts of datacollected from media players, but are targeted towards verydi↵erent research goals.

In terms of the techniques, our prior work [14] used quasi-experiments in a network measurement setting. In this pa-per, we develop the QED technique further and use it in adi↵erent context for the study of video ads. While seldomused in measurement studies of networked systems prior toour work in [14], quasi-experiments have a long and distin-

Figure 18: Normalized abandonment rate for di↵erent adlengths.

Figure 19: Normalized abandonment rates are roughly sim-ilar for the di↵erent connection types.

guished history of use in the social and medical sciences thatis well documented in [20].

8. CONCLUSIONSTo our knowledge, our work is the first in-depth scientific

study of video ads and their e↵ectiveness. We explored howad e↵ectiveness as measured by ad completion rate is im-pacted by key properties of the ad, of the video, and of theviewer. A key contribution of our work is that we go beyondsimple characterization to derive causal rules of viewer be-havior using quasi-experimental designs (QEDs). We showthat an ad is 18.1% more likely to complete when placed asa mid-roll than as a pre-roll, and 14.3% more likely to com-

plete when placed as pre-roll than as a post-roll. Next, weshow that completion rate of an ad decreases with increasingad length. A 15-second ad is 2.9% more likely to completethan a 20-second ad, which in turn is 3.9% more likely tocomplete than a 30-second ad. Further, we show that the adcompletion rate is influenced by the video in which the adis placed. An ad placed in long-form videos such as moviesand TV episodes is 4.2% more likely to complete than thesame ad placed in short-form video such as news clips. Wealso studied the abandonment rate metric and showed thatviewers abandon more quickly in the beginning of the adand abandon at slower rates as the ad progresses. Our workrepresents a first step towards scientifically understandingvideo ads and viewer behavior. Such understanding is cru-cial for the long-term viability of online videos and the futureevolution of the Internet ecosystem.

9. ACKNOWLEDGEMENTSWe thank Girish Bettadpur for his deep insights into the

video ecosystem, Rick Weber for providing computing re-sources, and Harish Kammanahalli for his support. Anyopinions expressed in this work are solely those of the au-thors and not necessarily those of Akamai Technologies.

10. REFERENCES[1] Adobe Auditude. http://goo.gl/iI7Mc.[2] Akamai Media Analytics. http://goo.gl/lnFwF.[3] Bright roll. http://www.brightroll.com.[4] comscore. http://www.comscore.com.[5] Freewheel. http://www.freewheel.tv.[6] Interactive Advertising Bureau (IAB).

http://www.iab.net.[7] Video Plaza. http://www.videoplaza.com.[8] Cynthia Boris. What is the fastest growing online ad

format? Video! http://goo.gl/ylnX8.[9] Cisco. Visual Networking Index.

http://bit.ly/KXDUaX.[10] John Dilley, Bruce M. Maggs, Jay Parikh, Harald

Prokop, Ramesh K. Sitaraman, and William E. Weihl.Globally distributed content delivery. IEEE InternetComputing, 6(5):50–58, 2002.

[11] Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica,Dilip Joseph, Aditya Ganjam, Jibin Zhan, and HuiZhang. Understanding the impact of video quality onuser engagement. In Proceedings of the ACMSIGCOMM Conference on Applications, Technologies,Architectures, and Protocols for ComputerCommunication, pages 362–373, 2011.

[12] eMarketer. Click rates complicate online video admetrics. http://goo.gl/vPGzs.

[13] John T Kent. Information gain and a general measureof correlation. Biometrika, 70(1):163–173, 1983.

[14] S Shunmuga Krishnan and Ramesh K Sitaraman.Video stream quality impacts viewer behavior:inferring causality using quasi-experimental designs.In Proceedings of the ACM conference on Internetmeasurement conference, pages 211–224. ACM, 2012.

[15] R.C. Larson. Perspectives on queues: Social justiceand the psychology of queueing. Operations Research,pages 895–905, 1987.

[16] E.L. Lehmann and J.P. Romano. Testing statisticalhypotheses. Springer Verlag, 2005.

[17] X. Liu, F. Dobrian, H. Milner, J. Jiang, V. Sekar,I. Stoica, and H. Zhang. A case for a coordinatedinternet video control plane. In Proceedings of theACM SIGCOMM Conference on Applications,Technologies, Architectures, and Protocols forComputer Communication, pages 359–370, 2012.

[18] E. Nygren, R.K. Sitaraman, and J. Sun. The AkamaiNetwork: A platform for high-performance Internetapplications. ACM SIGOPS Operating SystemsReview, 44(3):2–19, 2010.

[19] H. Oktay, B.J. Taylor, and D.D. Jensen. Causaldiscovery in social media using quasi-experimentaldesigns. In Proceedings of the First Workshop onSocial Media Analytics, pages 1–9. ACM, 2010.

[20] W.R. Shadish, T.D. Cook, and D.T. Campbell.Experimental and quasi-experimental designs forgeneralized causal inference. Houghton, Mi✏in andCompany, 2002.

[21] D.A. Wolfe and M. Hollander. Nonparametricstatistical methods. Nonparametric statistical methods,1973.