Decisions, Causality and All That…
description
Transcript of Decisions, Causality and All That…
![Page 1: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/1.jpg)
Decisions, Causality and All That…
BIG DATAFrom knowing ‘what’ to understanding ‘why’?.
![Page 2: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/2.jpg)
an important decision…
I think she is hot!
Hmm – so what should I write to her to get her number?
![Page 3: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/3.jpg)
The word pretty is a perfect case study for our point. As an adjective, it’s a physical compliment, but as an adverb (as in, “I’m pretty good at sports.”) it is just another word.
On the other hand,
more general
compliments work
quite well.
Source: OK Trends
?
![Page 4: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/4.jpg)
hardships of causality.
Beauty is Confounding
determines both the probability of getting the number and of the probability that James will say it
need to control for the actual beauty or it can appear that making compliments is a bad idea
“You are beautiful.”
![Page 5: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/5.jpg)
causal analysis inonline display advertising.
![Page 6: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/6.jpg)
The life of a browser process.
2. Use observed data to build list of prospects
3. Subsequently observe same browser surfing the web the next day
4. Browser visits a site where a display ad spot exists and bid requests are made
5. Auction is held for display spot
6. If auction is won display the ad
7. Observe browsers actionsafter displaying the ad
1. Observe people taking actions and visiting content
![Page 7: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/7.jpg)
what do advertisers want?Conversions?
1.05X
2.62X
1.11X
1.31X
0.92X2.26X
TELECOM COMPANY A
TELECOM COMPANY B
TELECOM COMPANY C
TELECOM COMPANY A
TELECOM COMPANY B
TELECOM COMPANY C
![Page 8: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/8.jpg)
questionof interest.
what is the causal effect of m6d’s display advertisingon customer conversion?
display advertisingShowing/Not showing a browser a display ad.
customer conversionVisiting the advertisers website in the next 5 days.
![Page 9: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/9.jpg)
general approach.
1. Ask the right question
3. Translate question into a formal quantity
4. Try to estimate it
2. Understand/express the causal process
![Page 10: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/10.jpg)
What is the effect ofdisplay advertising on customerconversion?
1. state question.
display advertisingShowing/Not showing a browser a display ad.
customer conversionVisiting the advertisers website in the next 5 days.
![Page 11: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/11.jpg)
2. express causal process.
O = (W,A,Y) ~ P0
W – Baseline VariablesA – Binary Treatment
(Ad)Y – Binary Outcome
(Purchase)
“You are beautiful.”
![Page 12: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/12.jpg)
data structure: our viewers.
CHARACTERISTICS(W)
TREATMENT(A)
CONVERSION(Y)
Color Sex HeadShape
Ad No Ad
No Yes
![Page 13: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/13.jpg)
3. define quantity.
E[YA=ad] – E[YA=no ad]
E[YA=ad]/E[YA=no ad]
additive impact
relative impact
![Page 14: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/14.jpg)
4. estimate quantity.
1.A/B testing
2.Modeling Observational Data
![Page 15: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/15.jpg)
common approach: A/B testing.
Since we can not both treat and not treat the SAME individuals. Randomization is used to create “EQUIVALENT” groups to treat and not treat.
3.4 per 1,000
1.6 per 1,000
![Page 16: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/16.jpg)
practical concerns.associated with doing A/B testing
1. Cost of displaying PSAs to the control (untreated group).
2. Overhead cost of implementing A/B test and ensuring that it is done CORRECTLY.
3. Wait time necessary to evaluate the results.
4. No way to analyze past or completed campaigns.
![Page 17: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/17.jpg)
non invasive causal estimation (NICE).
Estimate The Effects In The Natural Environment (Observed Data)
![Page 18: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/18.jpg)
“what if”causal analysis adjusting for confounding
Need to adjust for the fact that the group that saw the advertisement and the group that didn’t may be very different.
![Page 19: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/19.jpg)
estimation – a primer.1. When can we estimate it? Necessary conditions:
– no unmeasured confounding– experimental variability/positivity
2. Be VERY careful with data collection– Define cohorts and follow them over time
3. Estimation techniques – Unadjusted
– Adjust through gA
– MLE estimate of QY
– Double robust combining gA and QY
– TMLE
4. Many tools exist for estimating binary conditional distributions– Logistic regression, SVM, GAM, Regression Trees, etc.
P(W) P(A|W) P(Y|A,W)
QWQY
gA
![Page 20: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/20.jpg)
summary results.median relative lift of 90%
![Page 21: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/21.jpg)
method validation:A/B Test vs. analytic estimate
![Page 22: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/22.jpg)
method validation: negative test
Impact of Telecommunication company’s advertisement on fast food conversion
![Page 23: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/23.jpg)
gross conversion rates.
Additive Impact
-0.2%
TELECOM COMPANY
A
TELECOM COMPANY
B
TELECOM COMPANY
C
![Page 24: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/24.jpg)
effectiveness varies by marketer.
B2B COMPANY
A
B2B COMPANY
B
1.08X
1.08X4.23X
3.77X B2B COMPANY A
B2B COMPANY B
![Page 25: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/25.jpg)
NO LIFT
NO LIFT
creative matters.
This campaign drove no significant lift from either retargeting or new customer
prospects, likely due to ineffective creative.
Brand is buried; sweepstakes, not the brand, is the primary message
Call to action is inconsistent with primary message
![Page 26: Decisions, Causality and All That…](https://reader036.fdocuments.in/reader036/viewer/2022062723/56813e57550346895da84a18/html5/thumbnails/26.jpg)
references.1. O. Stitelman, B. Dalessandro, C. Perlich, and F.
Provost. Estimating The Effect Of Online Display Advertising On Browser Conversion. In Proceedings of KDD, Annual International Workshop on Data Mining and Audience Intelligence for Online Advertising, ADKDD ’11.
2. M. van der Laan and S. Rose. Targeted Learning: Causal Inference for Observational and Experimental Data. New York, NY: Springer Publishing Company, 2011. http://www.targetedlearningbook.com/
3. ‘tmle’ R Package http://cran.r-project.org/web/packages/tmle/index.html
4. R. Kohavi and R. Longbotham. Unexpected results in online controlled experiments. ACM SIGKDD Explorations Newsletter, 12(2):31–35, 2010.
5. R. Lewis and D. Reiley. Does retail advertising work: Measuring the effects of advertising on sales via a controlled experiment on yahoo. Technical report, Working paper, 2010.
6. D. Chan, R. Ge, O. Gershony, T. Hesterberg, and D. Lambert. Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of KDD, KDD ’10, pages 7–16, New York, NY, USA, 2010. ACM.
Claudia’s Office Hours:Thursday 2:20 PMExhibition Hall
Data Science Team:Ori StitelmanBrian DalessandroTroy RaederCharlie Guthrie
Foster Provost