MLCONF SEATTLE — MAY 1, 2015
A large scale online natural experiment Measuring causal impact of display ads
Robert Moakler — [email protected] | [email protected]
@ MLconf Seattle 2015
MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?
$104.57 $120.05
$140.15
$160.18 $178.45
$196.05
$213.89 Digital ad spending!% change!
2012 2013 2014 2015 2016 2017 2018!Source: www.emarketer.com, “Global Ad Spending Growth to Double This Year”
20.4% 14.8% 16.7% 14.3%
11.4% 9.9% 9.1%
MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?
MLCONF SEATTLE — MAY 1, 2015
The $100+ billion question!
Does online advertising really work?
Do online ads cause you to take some action?
MLCONF SEATTLE — MAY 1, 2015
Measuring causal impact!Option 1: Randomized A/B test • Pros
– If setup correctly, gives unbiased causal estimates
• Cons – Control ads cost as much as real
ones – Planned before campaign starts – Coordination of multiple media
partners – Too many levers to test them all
Campaign Ad PSA
MLCONF SEATTLE — MAY 1, 2015
Measuring causal impact!Option 2: Observational study • Pros
– Cheap – Flexible
• Cons – Enormous amount of selection bias
MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!• Why is there selection bias in observational techniques?
– Online ads are targeted to specific segments of the population based on particular demographics, user interests and behaviors, etc.
– Targeting ads to specific populations makes comparing users that have received an ad to those that did not very problematic; estimates of causal impact will be overestimated.
MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!• Why is there selection bias in observational techniques?
W User
features
A Served
ads
Y Convert
MLCONF SEATTLE — MAY 1, 2015
Confounding in digital advertising campaigns!• Why is there selection bias in observational techniques?
W User
features
A Served
ads
Y Convert
Unless we know what information targeters are using, we will never be able to fully adjust for selection bias.
MLCONF SEATTLE — MAY 1, 2015
Viewability!• Web page layout, ad placement details, and user browsing behavior
and setup can all impact the way in which ads are seen online. – Some ads are served far down on the page (below the fold) – Ads can be loaded in hidden tabs or windows – Users may not stay on a page long enough for it to finish loading
MLCONF SEATTLE — MAY 1, 2015
Viewability!
MLCONF SEATTLE — MAY 1, 2015
Viewability!
MLCONF SEATTLE — MAY 1, 2015
Viewability!
MLCONF SEATTLE — MAY 1, 2015
Viewability as a natural experiment!Introduce a mediating variable — viewability
W User
features
A Served
ads
Y Convert
V Viewable
ad
MLCONF SEATTLE — MAY 1, 2015
Methodology!
Conversion (Y=1)
Web page visit
Effect window
T 0 Untreated user
Treated user
Viewable ad (V=1)
Unviewable ad (V=0)
MLCONF SEATTLE — MAY 1, 2015
Methodology!
Conversion (Y=1)
Web page visit
Effect window
T 0 Untreated user
Treated user
Viewable ad (V=1)
Unviewable ad (V=0)
Parameter of interest
MLCONF SEATTLE — MAY 1, 2015
Data!• Seven display advertising campaigns run during the 4th quarter of
2014 – Diverse industries such as auto insurance, beauty products, finance, and
online marketing – 3 million - 29 million impressions – 2,000 - 2 million conversions
MLCONF SEATTLE — MAY 1, 2015
Using viewability as a natural experiment!Compared to the naïve analysis of comparing users that were served and not served ads, we find a drastic decrease in estimated lift when utilizing viewability.
MLCONF SEATTLE — MAY 1, 2015
Validation!• How do we know a reduction in lift means our new estimates are
correct? • Use negative control tests
– Use the impressions of one campaign to predict an unrelated conversion
MLCONF SEATTLE — MAY 1, 2015
Validation!• How do we know a reduction in lift means our new estimates are
correct? • Use negative control tests
– Use the impressions of one campaign to predict an unrelated conversion
W User
features
A Served
ads
Y Convert
V Viewable
ad
Y-
Unrelated outcome
MLCONF SEATTLE — MAY 1, 2015
Validation!Focusing on Campaign B from the previous example, we measure the ads’ impact on unrelated outcomes
MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t account for
MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t account for
W User
features
A Served
ads
Y Convert
V Viewable
ad
W’ User
features
MLCONF SEATTLE — MAY 1, 2015
Bias in the natural experiment!• We don’t see zero effect on many of our negative controls
– There can be other factors that affect viewability and conversion that we don’t account for
W User
features
A Served
ads
Y Convert
V Viewable
ad
W’ User
features
Parameter of interest
MLCONF SEATTLE — MAY 1, 2015
Validation!Returning to Campaign B from the previous example, we measure the ads’ impact on irrelevant outcomes
MLCONF SEATTLE — MAY 1, 2015
Summary!• Viewability enables a natural experiment
– Combines the benefits of A/B tests and observational analysis
– Adjustment for viewability features is easier than adjusting for targeting features
– Results in a large reduction in bias
• Negative controls allow for validation of models when the true value being estimated is unknown – As the true effect of a natural experiment is usually unknown, negative controls provide a
method for validation
MLCONF SEATTLE — MAY 1, 2015
Versatility!• Features that can be used in a natural experiment can be found in data sets from a
wide array of industries – Viewability of stories in a user’s news feed
– Listening to songs on shuffle
– Winning bids in online advertising real-time bidding systems
• Valid negative controls naturally exist in many industries – Purchasing unrelated products
– Clicking unrelated links
MLCONF SEATTLE — MAY 1, 2015
Acknowledgments!Integral Ad Science Daniel Hill Ekaterina Eliseeva Gijs Joost Brouwer Kiril Tsemekhman
NYU Stern Foster Provost UC Berkley Alan Hubbard
MLCONF SEATTLE — MAY 1, 2015
Thanks! Robert Moakler — [email protected] | [email protected]
Top Related