RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack

Virtual Meetup – January 2015

Agenda

• Developer Spotlight – Nicolas Emiliani – RTB Dev Lead

• RTBkit Behind the Scenes

• More than a Bidder – The RTBkit Stack

• Open Q&A

Copyright © 2015 Datacratic Inc. All rights reserved

Getting an RTBKit installation running.

More info here : http://datacratic.com/site/blog/tales-rtbkit-adventure

Nicolas Emiliani, RTB dev. team Tech Leader at Motrixi.

A little about our traffic :

We are a mobile centered DSP

We are connected to Nexage, Rubicon, Mopub and more recently to ADX

We take up to 40k qps in peak hours from US and Canada

Life on the Amazon (not the rain forest)

Create a VPC (or you will regret it, I promise) Create all your production nodes inside that VPC Be smart about security : do not expose your nodes unnecessarily to the internet. Use a VPN software (ie: OpenVPN) to access the nodes safely inside your VPC.

Isolating the RTBKit stack (BR loop) Chose an HTTP reverse proxy (we use NGINX)

Set up the proxy as a gateway to your RTBKit installation, we use elastic IP s on the NGINX nodes and allow incoming traffic only for the used HTTP ports. Elastic IPS are cool and they will let you do all sort of handy things.

You can use the proxy as a load balancer if you have more than one node

You can use the proxy to automatically reply with 204s to the exchanges in case you need to work on your installation

You can use the proxy to clone the traffic and send it to a staging or development environment (we recently cracked how to do this with NGINX)

Isolating the RTBKit stack (BR loop)

Isolating the RTBKit PAL As with the BR loop use a reverse proxy with an Elastic IP

Type of nodes Different responsibilities → Different nodes → Easier scaling

Kernel parameters

fs.file-max=300000 : RTBKit likes to chew up your file descriptors.

net.ipv4.tcp_tw_reuse=1 : allow reusing sockets in TIME_WAIT state for new connections.

net.ipv4.tcp_fin_timeout=15 Close connections in the TCP FIN timeout state in 15 seconds ( default 60 )

Tweaking this parameters will prevent crashes due to exceptions related to too many open file descriptors.

Use the APIs, Luke

Banker API : comes in handy when implementing pacing strategies thus keeping logic simple in agents.

Agent Configuration Service API : is useful to implement agent configuration related APIs as the Agent Gateway (https://github.com/nemiliani/agent_gateway).

https://github.com/nemiliani/agent_gateway

Things that will help you out

All our processes run under a watchdog, currently we are using monit

Zabbix is your friend, particularly when things go wrong.

Don't ever forget about Graphite. RTBKit is a huge black box otherwise. Always keep an eye on your metrics. Yous should be doing it right now!, no, really!

… QUESTIONS ?

Transition To 14.04

What’s taking so long?

● Ancient Node.js version in platform-depso Upgrading requires fixing the bindings

o Lots of work, very little pay-off


Alternative?

● Transition to HTTP interfaceso Support for multiple languages

o Cleaner well defined OpenRTB protocol

o Start of RTBkit 2.0 implementation


How will it work?

● Abstracted agent communicationo HTTP Bidder Interface class

o Agent config specifies bidder URL


Details

● Added two new HTTP bidder examples:o Python bidder based on tornado

o Node.js bidder

● Detailed walkthrough on the wiki


Backwards compatibility?

● Node.js support will be dropped for 14.04

● ZMQ C++ interface will remain for now.

➔ Will require an upgrade.

➔ Should be fairly painless.


Platform-deps

● Fork of platform-deps for RTBkito Upgrade dependencies to 14.04

o Cleanup of pointless dependencies


How long?

● Platform-deps fork is almost completedo Should be released in a week or two

o Work done by Michael

● Currently re-writing our agents in golango Will iron-out kinks in the bidder interface

o Should be done in a month or two

More Than a Bidder

A Brief Introduction to an RTB Stack

But there is more to it than that ...

Campaign

Management

UI

Campaign

Reporting UI

ETL / Log

Processing

Offline

Optimization /

Bid Strategy

Logs / Log

Management

Ad Server

Online

Audience

Data Store

Offline Data

Prep

Monitoring Alerting 24/7 Ops

Conversion

Collection

Data

Collection /

DMP


You need an RTB Solution

Requirement Provided by RTBkit?

Reliable, high-performance core bidder system Yes

Online Impression / Click / Conversion matching Yes

Pacing No

Logging Customer has to set up and manage logs

Monitoring and Alerting Some support for Carbon and Graphite

24-7 operations and support No

Capacity planning and system evolution No

Campaign management UI integrated with the bidder No

Seats on exchanges / integration with exchanges No

ETL for campaign reporting and bidding agents No

Other Issues …

Keep it Simple

● There is an abundance of inventory. Your goal is to process enough bids to

achieve your goals. Over-engineering is expensive.

● Unreliability is probably OKo Campaigns run for a long time and buy millions of impressions. Short outages

probably don't matter

o Most partners fine with noticeable error rates (~2%)


Other Issues …

Pacing and Strategy / Optimization

● Hard to do in real time, at high volume of transactions, in a distributed

system

● Hard to understand what data and how much data to use

● Riskyo blown daily caps

o uneven or poor performance


Other Issues … Ad Ops

Existing Ad Ops

● Used to feature set of DSP UI

● Comfortable delivering campaigns with current tool set

● May be resistant to using a new system

● May be resistant to trusting a new system

● You may spend a lot of time on UI features and reports, not on your real-

time platform


Host an RTBkit Meetup!


Q & A


RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack

Technology

Transcript of RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack