AERPAW: Aerial Experimentation and Research Platform for ...
Building an Experimentation Platform in Clojure
-
Upload
srihari-sriraman -
Category
Software
-
view
230 -
download
1
Transcript of Building an Experimentation Platform in Clojure
![Page 1: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/1.jpg)
![Page 2: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/2.jpg)
![Page 3: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/3.jpg)
• built at Staples-SparX • one box serving all Staples’s
experimentations • 8 GB of data per day • 5 million sessions a day • 500 requests per second • SLA of 99.9th percentile at 10ms
what we built
![Page 4: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/4.jpg)
• values of different experiments setup • how to efficiently use traffic • some nice things about clojure • building assembly lines using core.async • putting a complex system under simulation testing
what you will learn
![Page 5: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/5.jpg)
1. explaining experimentation
2. implementation
3. simulation testing
structure of the talk
![Page 6: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/6.jpg)
explaining
experimentation
![Page 7: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/7.jpg)
experimentation is the step in the scientific method that helps people decide between two
or more competing explanations – or hypotheses.
the experimental method
![Page 8: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/8.jpg)
experimentation in business
• a process for where business ideas can be evaluated at scale, analyzed scientifically and in a consistent manner
• data driven decisions
![Page 9: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/9.jpg)
hypotheses
• “a red button will be more compelling than a blue button”
• algorithms, navigation flows • measurement of overall performance of an
entire product
![Page 10: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/10.jpg)
treatment
• values for the variables in the system under investigation
• control (no treatment) vs test (some treatment) • red/blue/green
![Page 11: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/11.jpg)
coverage
• effect of external factors (business rules, integration bug, etc.)
• fundamental in ensuring a precise measurement
• design: not covered by default
![Page 12: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/12.jpg)
sequence of interactions
![Page 13: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/13.jpg)
experiment
infrastructure
![Page 14: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/14.jpg)
A/B
traffic is split
![Page 15: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/15.jpg)
A/B/C
no limitation in the number of treatments you can associate to an experiment
![Page 16: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/16.jpg)
messy
testing orthogonal hypotheses
![Page 17: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/17.jpg)
precise
testing non-orthogonal hypotheses
![Page 18: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/18.jpg)
messy/precise
first version of experiment infrastructure
![Page 19: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/19.jpg)
traffic is precious
![Page 20: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/20.jpg)
nested
![Page 21: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/21.jpg)
![Page 22: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/22.jpg)
![Page 23: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/23.jpg)
shared bucket
![Page 24: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/24.jpg)
A/A
null hypothesis test
![Page 25: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/25.jpg)
why build ep?
• capacity to run a lot of experiments in parallel • eCommerce opinionated • low latency (synchronous) • real time reports • controlled ramp-ups • layered experiments • statistically sound (needs to be auditable by data
scientists, CxOs, etc.) • deeper integration
![Page 26: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/26.jpg)
परन्तु
• the domain is quite complex • significant investment of time, effort and
maintenance (takes years to build correctly) • you might not need to build this if your
requirements can be met with existing 3rd party services.
![Page 27: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/27.jpg)
implementation
![Page 28: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/28.jpg)
![Page 29: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/29.jpg)
![Page 30: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/30.jpg)
postgres cluster
• data centered domain • data integrity • quick failover mechanism
• no out of the box postgres cluster management solution • built it ourselves using repmgr • multiple lines of defense
• repmgr pushes • applications poll • zfs - mirror and incremental snapshots
![Page 31: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/31.jpg)
reporting on postgres
• sweet spot of a medium sized warehouse • optimized for large reads • streams data from master (real time reports) • crazy postgres optimizations • maintenance (size, bloat) is non trivial • freenode#postgresql rocks!
![Page 32: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/32.jpg)
real OLAP solution
• reporting on historical data (older than 6 months) • reporting across multiple systems’ data • tried greenplum
• loading, reporting was pretty fast • has a ‘merge’/upsert strategy for loading data • not hosted, high ops cost
• leveraged existing ETL service built for Redshift • assembly line built using core.async
![Page 33: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/33.jpg)
![Page 34: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/34.jpg)
why clojure?
• lets us focus on the actual problem
• expressiveness (examples ahead)
• jvm: low latency, debugging, profiling
• established language of choice among the teams
• java, scala, go, haskell, rust, c++
![Page 35: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/35.jpg)
![Page 36: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/36.jpg)
![Page 37: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/37.jpg)
![Page 38: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/38.jpg)
![Page 39: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/39.jpg)
परन्तु
![Page 40: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/40.jpg)
realize your lazy seqs!
![Page 41: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/41.jpg)
simulation testing
![Page 42: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/42.jpg)
why• top of the test pyramid • generating confidence that your system will
behave as expected during runtime • humans can't possibly think of all the test cases • simulation testing is the extension of property
based testing to whole systems • testing a system or a collection of systems as a
whole
![Page 43: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/43.jpg)
tools
• simulant - library and schema for developing simulation-based tests
• causatum - library designed to generate streams of timed events based on stochastic state machines
• datomic - data store
![Page 44: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/44.jpg)
![Page 45: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/45.jpg)
state machine to create streams of actions
![Page 46: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/46.jpg)
run the simulation, record the data
![Page 47: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/47.jpg)
setting up and teardown of target system
![Page 48: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/48.jpg)
validate the recorded data
![Page 49: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/49.jpg)
examples of validations
• are all our requests are returning non-500 responses under the given SLA.
• invalidity checks for sessions, like no conflicting treatments were assigned
• traffic distribution • the reports match
![Page 50: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/50.jpg)
running diagnostics
• all the data is recorded • you can create a timeline for a specific session
from the data recorded for diagnostics purposes
![Page 51: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/51.jpg)
![Page 52: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/52.jpg)
परन्तु
• requires dedicated time and effort • was difficult to for us to put into CI • many moving parts
![Page 53: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/53.jpg)
conclusions• traffic is precious, take it account when
you are designing your experiments • ETL as assembly line work amazingly well • test your system from the outside • use simulation testing • use clojure ;)
![Page 54: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/54.jpg)
• Overlapping Experiment Infrastructure • More, Better, Faster Experimentation (Google)
• A/B testing @ Internet Scale • LinkedIn, Bing, Google
• Controlled experiments on the web • survey and practical guide
• D. Cox and N. Reid • The theory of the design of experiments, 2000
• Netflix Experimentation Platform • Online Experimentation at Microsoft • Practical Guide to Controlled Experiments on the Web:
Listen to Your Customers not to the HiPPO (Microsoft)
Great Material on Experiment Infrastructure
![Page 55: Building an Experimentation Platform in Clojure](https://reader031.fdocuments.in/reader031/viewer/2022030316/5871bd7e1a28ab55058b5f03/html5/thumbnails/55.jpg)