DataEngConf: Building the Next New York Times Recommendation Engine
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
-
Upload
hakka-labs -
Category
Technology
-
view
226 -
download
0
Transcript of DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
![Page 1: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/1.jpg)
BYOMQ: Why We [re]Built IronMQ
Reed Allman
@rdallman10
![Page 2: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/2.jpg)
Making [another!] Case for NIHReed Allman
@rdallman10
![Page 3: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/3.jpg)
Beating A Dead Horse [again!]Reed Allman
@rdallman10
![Page 4: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/4.jpg)
There
![Page 5: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/5.jpg)
Are
![Page 6: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/6.jpg)
Too
![Page 7: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/7.jpg)
Many
![Page 8: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/8.jpg)
Options
![Page 9: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/9.jpg)
Me Showing .gifs for 30 MinutesReed Allman
@rdallman10
![Page 10: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/10.jpg)
![Page 11: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/11.jpg)
NIH: Not Invented Here Syndrome
A quick refresher:
...the tendency towards reinventing the wheel (reimplementing something that is
already available) based on the belief that in-house developments are inherently better
suited, more secure, more controlled, quicker to develop, and incur lower overall cost
(including maintenance cost) than using existing implementations.
![Page 12: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/12.jpg)
Proof by Contradiction
![Page 13: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/13.jpg)
Throw this at ${USE_CASE}
![Page 14: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/14.jpg)
Out of the Box
● Massive scale on commodity hardware in hours of easy set up
● Battle tested, many bugs worked out, lots of docs, tools, guides, etc.
● Operate JVM, hard not to code on JVM
● Any and all problems associated with any of the boxes or how they fit together
● And these are open source! Many boxes are not and suffer vendor lock-in
● At mercy of those who understand it / time for fixing bugs, making improvements
![Page 15: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/15.jpg)
Could you do better?
![Page 16: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/16.jpg)
Different people come from different backgrounds and, based on that, find different
tools useful. Ultimately you need to use the tools that let you get your job done most
effectively, however that is defined in your particular case.
- Russ Cox
Go
![Page 17: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/17.jpg)
If it's a core business function -- do it yourself, no matter what.
- Joel Spolsky
Excel
![Page 18: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/18.jpg)
I felt that my team, which was supposed to be made up of distributed systems
engineers, was really acting more as distributed system plumbers.
- Jay Kreps
Kafka
![Page 19: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/19.jpg)
![Page 20: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/20.jpg)
Pain meow or later?
![Page 21: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/21.jpg)
To the Queues!
![Page 22: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/22.jpg)
Our Requirements:
● Easy to run (single binary ideal) in any environment & not hosted
● Persisted, consistent and performant
● Highly available -- fast recovery, no data loss, minimal downtime
● Horizontally scalable
● Timing based
● FIFO
● >= 1 time delivery
● Multi-Tenant capable+
● Preferably built in Go, HTTP interface
![Page 23: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/23.jpg)
...Scalable, Low-Latency, Persistent MQ for Job Processing
![Page 24: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/24.jpg)
![Page 25: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/25.jpg)
![Page 26: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/26.jpg)
What, briefly● Single Go binary per server
● RocksDB embedded to store all metadata and messages
● Built the distributed database - viewstamped replication, multi-master, gossip
membership - scales across queues across nodes, auto-balanced
● Linear scaling factor of .98
● Recovery time < 300ms
● Simple HTTP interface, stupidly simple clients
● Enqueue / Dequeue avg latency < 10ms for < 4KB messages
![Page 27: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/27.jpg)
![Page 28: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/28.jpg)
boom
![Page 29: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/29.jpg)
Results
![Page 30: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/30.jpg)
Full understanding and control makes for quick fixes, easy to add features
![Page 31: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/31.jpg)
Don’t have to build a daemon to add features, can bake everything in
![Page 32: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/32.jpg)
Takes a long time to get to stable
![Page 33: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/33.jpg)
Make all your own tools, plumb any data you want out
![Page 34: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/34.jpg)
Build exactly what you need, optimized for what you are doing
![Page 35: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/35.jpg)
Use the language(s) you know and love
![Page 36: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/36.jpg)
My attempt to say profound shit:
Be a builder, not a plumber.
![Page 37: DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://reader031.fdocuments.in/reader031/viewer/2022030315/587e47ef1a28abeb1a8b46e5/html5/thumbnails/37.jpg)
Questions?