Building a Dynamic Bidding system for a location based Display advertising Platform

Experimentation to Productizationof a Location based Dynamic Bidding system

Ekta GroverData Scientist, AdNear

26th July, 2014

@ektagrover(Twitter)/[email protected] The Fifth Elephant, 2014 1/22

Structure of this Talk

Introduction to a Real Time Bidder(RTB)

System design to deliver performance at scale

Three specific Data products that we built

Building a low latency self learning system

The Data we get our hands on

Real time bidding

What happens when a mobile user logs in

System design

Simulation

A/B testingframework

Reporting

Data products &Experimentation

Bidder

Spark-In-memoryprocessing

of logs in δt

Update snap-shots in Redis

to consume

(Multiple) Kafkaconsumers

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

Online experimentation at Microsoft - Kohavi, Crook, Longbotham(2009)

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

System design

Simulation

Reporting

Bidder

of logs in δt

to consume

AccessBusi-

nessris

target&control

groups

Parse json

logs & dump

to Spark

Feedback Loop

Jsonlogs

Experiments

run live

Bidder getsall

attributes it

ExperimentsProbablityToConvert()Bid()Price()

InputsFeatureEngineering()MetaDataExtraction()

BidderExperiments()OverRideTargetGroup()RollBack()Log()Reporting()

InputsPreTargetingChecks()AB Testing()ReadStateFromRedis()

DataFeedAggregatorExperiments()LogAndMaintainState()UpdateRedis()NotifyUser()

A/B TestingAssignExperiment()AssignTargets()

InputsSplits()

SimulationDollarSpend()MinSampleSize()

ReportingPacingRate()BidRate()WinRate()CTR()eCPM()eCPC()PercentageLift()

Sandbox

Tested

Set Traffic across Experiments

Signal

Update

Snapshots

Go Live

Low Latency feedback Loop

Acceptable Risk & TimeFrame

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

InputsSplits()

Sandbox

Tested

Signal

Update

Snapshots

Go Live

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

StaggeredInventoryBidding

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Productizing your Experimentation:Structuring your Data Products

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

Performanceat Scale

RightRealEstate

AppRankingsystem

RightTime

In-Appengage-

RightUser

Mobility&

affinityanalysis

BehavioralProfiling

Powerusers

RightPrice

DynamicPrice

Bidding

RightCreative

Verticalaffinity

Windowing&

Memory

RightGeo-

location

Georelevance

What signals can we extract from a weblog(andmore..)

Problem # 1: Dynamic bidding system

Guiding principlePrice that we bid at should reflect the quality of inventory

probclick = function(Engagementapp,appcategory ,

SessionContextdepth,length,

Engagementcreativeattributes ,

Engagementvertical ,

Engagementuserprofile,collaborativeprofile ,

EngagementHandsetattributes ,

Timeday ,week,seasonality )

Problem # 1 : And, hence the price should reflect thisQuality

price|probclick = Constant1 ∗ (0 ≤ probclick ≤ threshold1)+

Constant2 ∗ (threshold1 ≤ probclick ≤ threshold2)+

Constant3 ∗ (threshold2 ≤ probclick ≤ threshold3)

Modelled as a logistic regression with L1 regularization1 withbaggingConverges & scales faster for large datasets: Use the start˙paramsfrom the last optimization call - Better fit & AUC

1Bid optimizing and inventory scoring in targeted online advertising - Perlich, Dalessandro, Hook, Stitelman,

Raeder, Provost(2012)

Representation : Variable importance

Problem #2 :Setting the context

Chih Lee,Jalali,Dasdan: Real time bid optimization with smooth budget delivery in onlineadvertising(ADKDD, 2013)

Problem #2: Comprehensive Mobile App-Ranking system

Primary goal:

Capture the cream in the apps, at the right time, for a right price

Conventional approach : Scrape the # downloads, appcat,in-app-purchases, trends - Lot of noise !

Key observations : Peculiarity in apps wrt time of day, winrate & demand signals

Combat Winner’s curse - Uncover a right spread for theprice to bid, so we the bid reflects the of click, and at a rightprice

The Sweet Spot - tames the market in long turn by shavingoff the bid price & helps in price discovery

Problem #2: How we approached this problem

Guiding principle:

An app that ”delivers” should be in high-demand, and hence”should” show up with low win rate in the live feeds

Stage 1:H0 : CTR depends on Winrate, BidFloor, PriceSpread, Density, Category

CTRappidi ,timet = function(Winrateappidi ,timet ,Bidfloorappidi ,timet ,

PriceSpreadappidi ,timet , densityappidi ,timet ,Categoryappidi ,timet )

δappidi ,time(t+1) = function(Winrateappidi ,timet ,Bidfloorappidi ,timet

CTRappidi ,timet , densityappidi ,timet ,Categoryappidi ,timet )

BidPriceappidi ,time(t+1) = δappidi ,timet + Winpriceappidi ,timet

Problem #2 : But Mobile Apps have their own nuances

The ”outcome” class is yet to come

Low win rate could also be because of expectation of highCTR

Stage 1 is tightly coupled with campaign budgets

Over penalizes a rock star app with too few moments oftruth in the last snapshot

Other idiosyncracies, broken input pipes, incoherent data

Problem #2 cont..

Stage 2: Decay factor of 80% every hour

This means that the signal is only 26.2% informative after 6 time periods

CTRappidi ,timet = function((0.80)1 ∗ CTRappidi ,timet−1 ,

(0.80)2 ∗ CTRappidi ,timet−2 , (0.80)3 ∗ CTRappidi ,timet−3 ,

(0.80)4 ∗ CTRappidi ,timet−4 , (0.80)5 ∗ CTRappidi ,timet−5)

Stage 3: Different time periods decay differently, for each appidi

CTRappidi ,timet = function(CTRappidi ,timet−1 ,CTRappidi ,timet−2 ,

CTRappidi ,timet−3 ,CTRappidi ,timet−4 ,

Winrateappidi ,timet ,Winrateappidi ,timet−1 ,

Winrateappidi ,timet−2 ,Winrateappidi ,timet−3 ,

Winrateappidi ,timet−4 ,Winrateappidi ,timet−5 ,

categoryappidi , densityappidi ,timet ..)

Trade-off: dampens the CTR signal, while cushioning for system

failures, broken pipes & outliers

Problem #2 : But still has another limitation

Chicken & Egg Problem !

We need to have sufficient mass of each mobile application. EnterPooled learning algorithms - Hybrid of Fuzzy, Levenshtein distance

Distance metric helps map the performing & non-performingapps from mutiple exchanges

which means, we have larger ”support”

And, we create data points that are better than blind/naivebidding strategy

Can we reduce the candidate set? Lot of Bookkeeping tomaintain the appids across exchanges

What we actually implemented : Hybrid approach of all thesemodels and Iterate multiple times over !

Levenshtein with C bindings & pandas itertuples

Problem #3: User-mobility patterns to generate userprofiles

Guiding principleGenerate a probabilistic of picture activity patterns & affinity towardsactivities

Data nodes : Users, Categories of places checked in, Category ofAppsRepresent this as a bipartite graph, then just need to get the top-k,or activate k segments over a certain critical mass

Can we do better ?

Need to get this for each lat-long, once - Memoization & Book-keeping

Problem #3: Representation

User 1

User 2

User 3

User 4

Category6

Category7

Category8

Category9

Movies

Entertainment

Workplace

UsersApp category

Checked in

0.0568

0.0043

0.0029

0.0091

0.0033

0.09030.

0.0953

0.06670.0867

Problem #3 :Why Graphs?

Tractability of the problem

Interesting properties :∑

v∈V deg+, deg−, sink, joiningcommunities

Abstraction & reusability - Multiple ways of Similarity ofApps, Users, Places

Behavior Dilution & Ghost clicks

Better Hypothesis - maturing your data Products

What we learnt: Build a self learning, assisted healingsystem

Own the Statistics

Cover the base-line : Universal sink

Log everything

Forward lookup - Abstract the error & Try-catch

Proximity to data producer

Reuse data Prep cycles, Better still productize it

Loose couple each system - fail fast & forward

Feature engineering & Meta-data Engine - It’s more than justYOUR data

Building a Dynamic Bidding system for a location based Display advertising Platform

Engineering

Transcript of Building a Dynamic Bidding system for a location based Display advertising Platform