CleverDATA_HighLoadStrategy_presentation

Company Profile

Make your DMP process 100% of incoming bid requests

Ad at the speed of light

Sergey Zhemzhitsky, CTO, CleverDATA January 30, 2015

cleverdata.ru | [email protected]

International market business development since 2012

One of three leading IT companies in Russia

43 branches in Russia and abroad

+5500 employees

100K projects for 10K customers

Data management innovative

platform (Data Exchange Service)

Cloud Service

In-house development

Internet advertising solutions

Data Management Platforms

Customers Base Management

Web Analytics

Marketing automation

Big Data Data Mining Digital Intelligence Operational Intelligence Low Latency and NoSQL Cloud Computing


Agenda

• RTB intro; • Challenges; • Choice difficulties; • Results; • Do’s and Don’ts.


RTB intro


Real Time Bidding (RTB)

publishers

AD NETWORK AD NETWORK

AD NETWORK


AD NETWORK

advertisers


How RTB works

publishers


AD NETWORK


AD NETWORK

advertisers


Demand Side Platform (DSP)

advertisers publishers


AD NETWORK


AD NETWORK

DS P


Supply Side Platform (SSP)

publishers


AD NETWORK


AD NETWORK

advertisers

SS P

TRACKING DATA


Data Management Platform (DMP)

publishers

COOKIE SYNCs

ACCESS LOGS

PARTNERS’ DATA

3rd PARTY DATA

CLICK STREAMS

advertisers

SS P

DS P

WHATEVER ELSE DATA


RTB challenges visitors publishers

advertisers

10

0 m

s

20 ms

10.000+ rps

TRACKING DATA


publishers

COOKIE SYNCs

ACCESS LOGS

PARTNER’S DATA

3rd PARTY DATA

CLICK STREAMS

advertisers

SS P

DS P

DMP in the RTB ecosystem

DMP


Wishes…


Choice criteria

• Linear scalability; • Sharding “out of the box”; • Distributed by design; • Data redundancy & replication; • Low latency.


Which NoSQL … again


Aerospike Cassandra Mongo Redis

Scalability

Sharding

Redundancy

Latency

SPoF

Maintainability

Monitoring tools

Features


Public results


nginx 1.2.x ngx-http-redis lua-nginx-module lua-resty-mongol ngx-aerospike

aerospike 2.x redis 2.6.x mongodb 2.4.x wrk iperf nmon

• Intel® Core™ i7-920 Quad-Core • 48 GB RAM • 1 Gbit/s NICs (*) • 20 * 106 msgs • 512 bytes msg

Tools & Materials


• No logging; • No plugins; • CPU Affinity.

NGINX

WRK

Measuring Nginx


Nginx results. Base line

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

1.35 1.46 1.64 1.90 68851


Measuring Redis

NGINX

WRK

Node 1 Shard 1

Shard 2

Slave 3

Slave 4

Node 2 Shard 3

Shard 4

Slave 5

Slave 6

Node 3 Shard 5

Shard 6

Slave 2

Slave 1


Redis results


2.68 3.10 3.25 3.90 34769


Redis results


Measuring Mongo

NGINX :: MONGOS

WRK

Node 1 mongod :: cfg

mongod :: repl

mongod :: repl

mongod :: repl


mongod :: repl

mongod :: repl

mongod :: repl


mongod :: repl

mongod :: repl

mongod :: repl

Shard 1 Replica Set 1




Mongo results


6.70 8.22 10.22 15.46 14220


Mongo results


Node 1

Chunk 1

Replica 2

Node 2

Chunk 2

Replica 3

Node 3

Chunk 3

Replica 1

NGINX

WRK

Measuring Aerospike



8.93 14.99 26.83 106.48 3402

Aerospike results


Aerospike results


Wks L, 50% L, 75% L, 90% L, 99% T, r/s Ngx, CPU Ngx, p/s As, CPU As, p/s

4 8.22 10.22 15.46 97.3 3402 7 8 K 1 1 K

8 7.89 15.16 92.4 89.8 6028 10 13 K 1 2 K

12 4.76 8.74 103.1 121.6 10233 15 20 K 2 3 K

16 3.91 6.21 99.6 111.3 13178 22 26 K 3 6 K

24 2.13 2.87 4.68 76.81 25744 21 60 K 5 10 K

32 2.01 2.60 4.42 81.29 28925 25 70 K 6 11 K

64 2.54 3.66 112.4 118.1 26468 27 70 K 6 11 K

… more Aerospike results


nginx + blocking I/O

How to shoot yourself in the foot…


Libevent to the rescue (2nd attempt)

Node 1

Chunk 1

Replica 2

Node 2

Chunk 2

Replica 3

Node 3

Chunk 3

Replica 1

LIBEVENT :: LIBEVHTP

WRK


Finally…


2.64 3.09 3.27 3.95 35746


Finally…


Nginx Redis Mongo Aerospike, b Aerospike, nb

Latency, 50% 1.35 2.68 6.70 8.93 2.64

Latency, 75% 1.46 3.10 8.22 14.99 3.09

Latency, 90% 1.64 3.25 10.22 26.83 3.27

Latency, 99% 1.90 3.90 15.46 106.48 3.95

Throughput, m/s 68851 34769 14220 3402 35746

CPU, Http, % 29 20 70 7 33

Network, Http, p/s, K 71 71 37 8 71

CPU, Db, % - 6 25 1 6

Network, Http, p/s, K - 12 21 1 12

Unofficial results


Aerospike


Let’s start building…


• Do use non-blocking I/O;

• Do computations locally;

• Be lazy. Don’t do more than needed;

• Don’t afraid of using other’s good ideas;

• Don’t trust anyone.

Do’s & Don’ts


Thanks a lot for your questions!

+7 (495) 967-66-50 [email protected]

cleverdata.ru facebook.com/CleverData

CleverDATA_HighLoadStrategy_presentation

Technology

Transcript of CleverDATA_HighLoadStrategy_presentation