How not to reinvent the wheel - Literature Searching for ENCH400 2012
Why reinvent the wheel at Criteo?
-
Upload
criteolabs -
Category
Engineering
-
view
171 -
download
1
Transcript of Why reinvent the wheel at Criteo?
![Page 1: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/1.jpg)
Cedrick MONTOUT
2016 June 8th
A story about C# @ Criteo
Why do we reinvent the wheel ?
![Page 2: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/2.jpg)
2 | Copyright © 2016 Criteo
• Criteo• Global leader on retargeting
• Scalability• One of many groups in the Criteo R&D department
• WebScale• We write code to help scale up the real-time Criteo Platform
• https://www.linkedin.com/in/kerdrek• https://github.com/kerdrek
• @kerdrek
Who am I ?
![Page 3: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/3.jpg)
Client Side Load Balancing
PLACEHOLDER IMAGE
Click icon to add picture
![Page 4: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/4.jpg)
4 | Copyright © 2016 Criteo
The situation then … 1/5
Footer:
App A Pool App B Pool
HA Proxy
Service Pool
![Page 5: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/5.jpg)
5 | Copyright © 2016 Criteo
The situation then … 2/5
Footer:
App A Pool App B Pool
HA Proxy
Service Pool
Random DC Service PoolTraffic 185k QPS
Input size ~5K bytes
Output size ~3,5K bytes
Network traffic ~12Gbits/s
![Page 6: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/6.jpg)
6 | Copyright © 2016 Criteo
The situation then … 3/5
Footer:
App A Pool App B Pool
HA Proxy
Service Pool
Random DC Service PoolTraffic 185k QPS
Input size ~5K bytes
Output size ~3,5K bytes
Network traffic ~12Gbits/s
Compression 30% gain
![Page 7: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/7.jpg)
7 | Copyright © 2016 Criteo
The situation then … 4/5
Footer:
App A Pool App B Pool
HA Proxy
Service Pool
Random DC Service PoolTraffic 185k QPS
Input size ~5K bytes
Output size ~3,5K bytes
Network traffic ~12Gbits/s
Compression 30% gain
Bonding 2 ports
Combine two physical data links into one logical link, by connecting 2 ports of the switch to 2 network interfaces of the HAProxy
![Page 8: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/8.jpg)
8 | Copyright © 2016 Criteo
The situation then … 5/5
Footer:
App A Pool App B Pool
HA Proxy
Service Pool
Random DC Service PoolTraffic 185k QPS
Input size ~5K bytes
Output size ~3,5K bytes
Network traffic ~12Gbits/s
Compression 30% gain
Bonding 2 ports
Combine two physical data links into one logical link, by connecting 2 ports of the switch to 2 network interfaces of the HAProxy
4 pairs
![Page 9: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/9.jpg)
9 | Copyright © 2016 Criteo
The new wheel : Client Side Load Balancing
• Bypass HA Proxy• Implemented inside Twitter/Finagle• Reuse existing health Check • Re-implement monitoring
PLACEOLDER IMAGE
App Pool A App Pool B
Service Pool
![Page 10: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/10.jpg)
DevHost
PLACEHOLDER IMAGE
![Page 11: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/11.jpg)
11 | Copyright © 2016 Criteo
The situation then …
Footer:
• Harder and harder to release• No available memory for new feature• Fragmented feature in production• Tightly coupled with the HTTP stack
![Page 12: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/12.jpg)
12 | Copyright © 2016 Criteo
The new wheel: Component
• One Input • One Output• On Process
PLACEHOLDER IMAGE
![Page 13: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/13.jpg)
13 | Copyright © 2016 Criteo
The new wheel: A Host with Services
• A collection of services
PLACEHOLDER IMAGE
![Page 14: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/14.jpg)
14 | Copyright © 2016 Criteo
The new wheel: DevHost
• The DevHost• Several Components• Several Services• Asynchronous Process• Transport agnostic• Not front facing
PLACEHOLDER IMAGE
![Page 15: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/15.jpg)
15 | Copyright © 2016 Criteo
The new wheel: DevHost
• Still use in production• ~45% of the windows production machine• Next iteration will use .net core (WiP)
PLACEHOLDER IMAGE
Web (IIS)300020%
De-vHost250017%
Other5003%
CentOS
900060%
![Page 16: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/16.jpg)
Monitoring @ Task level
PLACEHOLDER IMAGE
![Page 17: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/17.jpg)
17 | Copyright © 2016 Criteo
• Synchronous processing everywhere• Timeout on several pipeline (loosing money)• No clear diagnostic on execution path
The situation then …
Footer:
![Page 18: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/18.jpg)
18 | Copyright © 2016 Criteo
The new Wheel : Asynchronous Token Framework
• TPL was not a solution at that time• Asynchronous Completion Token• Delegate based• Execution is time boxed• Task underneath• Timing for every thing• Metrics available on the machine• Metrics available aggregated on Graphite
PLACEHOLDER IMAGE
![Page 19: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/19.jpg)
PLACEHOLDER IMAGE
Apache Kafka Driver in C#
![Page 20: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/20.jpg)
20 | Copyright © 2016 Criteo
The situation then …
Footer:
• Syslog• Text based• Fire and Forget• Single messages• No built-in resiliency• No API for consuming
![Page 21: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/21.jpg)
21 | Copyright © 2016 Criteo
The situation then …
Footer:
• Apache Kafka• Binary• Acknowledged message• Batched messages• Partitioning and replication• Consuming support
• Syslog• Text based• Fire and Forget• Single messages• No built-in resiliency• No API for consuming
![Page 22: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/22.jpg)
22 | Copyright © 2016 Criteo
Apache Kafka where is your C# driver ?
We looked at several drivers
![Page 23: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/23.jpg)
23 | Copyright © 2016 Criteo
Goldilocks Conundrum all over again
We looked at several drivers
First driver never used in production
![Page 24: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/24.jpg)
24 | Copyright © 2016 Criteo
Goldilocks Conundrum all over again
We looked at several drivers
First driver never used in production
Second driver was impossible to unit test
![Page 25: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/25.jpg)
25 | Copyright © 2016 Criteo
Goldilocks Conundrum all over again
We looked at several drivers
First driver never used in production
Second driver was impossible to unit test
Third driver was not recently maintained
![Page 26: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/26.jpg)
26 | Copyright © 2016 Criteo
The new Wheel: kafka-sharp
• Yet another C# driver • Highly tuneable• Written with perf and scale in mind• Battle tested in production• Available here: https://github.com/criteo/kafka-sharp
![Page 27: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/27.jpg)
27 | Copyright © 2016 Criteo
The wheel lists
• Distributed load balancer between clients.• Lightweight hosting server.• Low level asynchronous execution framework. • Yet another C# driver for Apache Kafka.
![Page 28: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/28.jpg)
28 | Copyright © 2016 Criteo
PLACEHOLDER IMAGE
![Page 29: Why reinvent the wheel at Criteo?](https://reader035.fdocuments.in/reader035/viewer/2022062522/589c48fe1a28ab227d8b5621/html5/thumbnails/29.jpg)
Click to add text