dataops BCN 2019 - Size does matter · 2019-06-27 · PARQUET Athena PRESTO dt=2018-08-21-01-51...
Transcript of dataops BCN 2019 - Size does matter · 2019-06-27 · PARQUET Athena PRESTO dt=2018-08-21-01-51...
Lucas Ceballos Size does matter @ Dataops BCN 2019
�1
Lucas Ceballos Size does matter @ Dataops BCN 2019
�2
Lucas Ceballoslinkedin.com/in/lucas-ceballos
CTO @ Headway Born @ ArgentinaCTO @ Smadex
Lucas Ceballos Size does matter @ Dataops BCN 2019
�3
SIZEDOES MATTER
Lucas Ceballos Size does matter @ Dataops BCN 2019
�3
SIZEDOES MATTERHOW TO BUILD A SERVERLESS PETA-BYTE SCALE DATABASE
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �4
CHAPTER 1CHAPTER 2
CHAPTER 3 - 5CHAPTER 6
CONTEXTPROBLEMATTEMPTSQ&A
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �5
CONTEXTCHAPTER 1
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �6
D I G I T A L A N D A P P G R O W T H M A R K E T I N G C O M P A N Y
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �7
“We help our clients to grow through digital advertising”
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �8
DeliverYou
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �9
NEW USERSUSER ACQUISITION
NEW ORDERSUSER RE-ENGAGEMENT
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER
Click IDImpression ID
�10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
△t△t
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
△t△t<1 min 1 - ∞ min
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
△t△t<1 min 1 - ∞ min
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �10
IMPRESSION+50K / SEG
CLICK+5K / SEG
CONVERSION+15K / HR
AD L IFE CYCLE
△t△t<1 min Attribution Window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �11
In the world of digital attribution, an attribution window is a time period during which eligible conversion events can be claimed. The conversion events can be an app install, added to cart, or purchase.
ATTRIBUTION WINDOW:
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �13
1808210149a523932a4ceaed
CLICK ID
Date & Time (up to minute) Random Hexadecimal String
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �14
{“_id”:"1808210149a523932a4ceaed","version":5,"earn":0,"spent":0,"converted":false,"campaignId":280392,"categoryGroup":"mainstream","publisherId":"10466_581_34_863796162656735232","adSlot":null,"siteId":null,"sellerId":null,"formatId":null,"sourceUid":"59af4b01e9cbfd600085ca9df0e709ed","appId":"","bundleId":null,"deviceId":null,"ip":"170.79.119.28","referer":null,"userAgent":"Mozilla/5.0(Linux;Android8.0;N820Build/KOT49H)AppleWebKit/537.36(KHTML,likeGecko)Chrome/60.0.3112MobileSafari/537.36","countryCode":"BR","countryName":"Brazil","region":"RioGrandedoNorte","city":"Natal","isp":"JDSMedeirosProvedorDeInternet-Me","connectionType":"wifi","latitude":-5.8002,"longitude":-35.2107,"device":"N820","deviceType":"mobile","browser":"Chrome","browserVersion":"60.0.3112","os":"Android","osVersion":"8.0","payout":3.23,"cost":1.87,"share":0,"impressions":0,"clicks":1,"conversions":0,"createdAt":"20180821014947","updatedAt":"20180821014947","date":"2018-08-21T04:49:47.205Z","datetime":"2018-08-2101:49:47","dateUpdated":"2018-08-21T04:49:47.205Z","callbackUrl":"http://click.foo-bar.com/fass/?atoken=f8ad6505fa1b53fc948c8543dc0afc7a05296200&version=1&c=tracker.postback&click_id={sid}"}
CLICK TAXONOMY
JSONFORMAT
1KBNO COMP.
650BGZIP
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �15
BIGGERATTRIBUTION WINDOW
LONGERCLICK RETENTION PERIOD
=
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �16
THE PROBLEMCHAPTER 2
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �17
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Conversions
W R
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �18
500.000.000 x 650
x 30
Daily clicks Avg. click size Days/month
~ 9 TB / month
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �19
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Events
W R
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �19
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �19
COST
2
1 3LATENCYSIZE
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �20
FIRST ATTEMPTCHAPTER 3
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Events
W R
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Events
W R
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Events
W R
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �22
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �22
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �23
2ND ATTEMPTCHAPTER 4
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �24
2ND ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming Clicks
Incoming Events
W R
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �25
FASTSCALABLE FIXED SIZE COSTLY
PROS CONS
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �26
10TB x 3 x 1
x 102,4
500M clicks/month Mongo replicas Months Retention GP2 cost/tb/month
~ 6500 USD / month
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �27
A NEW HOPECHAPTER 4
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �28
REQUIREMENTS
KEY VALUE CHEAP TB SCALE FAST
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory Costly
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �30
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �30
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �31
THINK OUT OF THE BOXCHAPTER 5
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �32
t1 t2
attribution window
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �32
t1 t2
attribution window
24H
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �32
t1 t2
attribution window
24H
90%<24H
10%>24H
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �32
t1 t2
attribution window
1H
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �32
t1 t2
attribution window
1H
70%<1H
30%>1H
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �33
“DIVIDE AND CONQUER”FINAL CHAPTER
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �34
H TLast hour clicks
C LD> 1 hour clicks
FREQUENTLY ACCESSED
SMALL NUMBER
INFREQUENTLY ACCESSED
HUGE NUMBER
+ +
= =
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �34
H TLast hour clicks
C LD> 1 hour clicks
+ +
= =
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �35
UNDER THE HOOD
EC2
EC2
EC2
EC2
Incoming Clicks EC2
EC2
EC2
EC2
Incoming Conversions
H T H T
C LDC LDFridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �36H TEC2
EC2
EC2
EC2
Incoming Clicks
CSV
EC2
EC2
EC2
EC2
Incoming Conversions
(450b)
1 hour ~ 9GB
ElasticacheHADES
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �37
EC2
EC2
EC2
EC2
Incoming Clicks
C LD
JSON
1m BUFFERING
PARQUET
Fridge
ALFREDDO
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �38
EC2
EC2
EC2
EC2
Incoming Conversions
C LDSQS Athena Clicks
H TSQS
Retry
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �38
EC2
EC2
EC2
EC2
Incoming Conversions
C LDSQS Athena Clicks
H TSQS
Retry
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �38
EC2
EC2
EC2
EC2
Incoming Conversions
C LDSQS Athena Clicks
H TSQS
Retry
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �39
JSON
PARQUET
AthenaPRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �39
JSON
PARQUET
AthenaPRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �39
JSON
PARQUET
AthenaPRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �40
EC2
EC2
EC2
EC2
Incoming Conversions
C LDSQS Athena Clicks
H TSQS
Retry
Fridge
Lucas Ceballos Size does matter @ Dataops 2019
SIZEDOES MATTER �41
443 + 300 + 290 + 140
S3 Storage Elasticache Athena Lambdas
~ 1173 USD / month
Lucas Ceballos Size does matter @ Dataops BCN 2019
�42
Moltes Gracies!Q&A
lucas-ceballos
We’re hiring!