Claudius MbembaCascadiaJS / Seattle, WA
Scalable Distributed Messaging
w/ Kafka on Containers
Who am I?CTO, Neu Inc.
@mbembashipsclaudiusmbemba.com
github.com/user1mgithub.com/neucleans
What we’ll cover...● The Problem● The Solution● The Demo● The End
The Problemwe had to solve
at Neu
We needed to write our
own solution
Image from Kafka.org
“Kafka® is used for building real-time data pipelines and streaming apps.
It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies”
Image from USPS.com
A Simple Analogy...● Kafka = Post Office● Producer = Sender● Consumer = Receiver● Topic = P.O. Box Wall
○ Partitions = P.O. Boxes○ Messages = Mail
Image from Google
Image from Google Images
Image from cloudkarafka.com
Consumer Group
Node+Kafka
Image from medium.com
Kafka+Docker
Image from medium.com
Bring in the Real Time!
The Solution
Kafka-Pub-Sub
Github.com/NeuCleans/kafka-pub-sub
Code Walkthrough
Github.com/NeuCleans/kafka-pub-sub
Commented out code hidden
Github.com/NeuCleans/kafka-pub-sub
The End
What we covered...● The Problem● The Solution● The Demo● The End
Github.com/NeuCleans/kafka-pub-sub
Takeaway Resources
● http://podplayer.net/?id=82304318 - Podcast w/ Kafka co-creator, Neha Narkhede
● https://kafka-tutorials.confluent.io/ - Kafka tutorials
● https://www.youtube.com/watch?v=udnX21__SuU - Kafka Tutorial - Core Concepts
● https://www.youtube.com/watch?v=UEg40Te8pnE - Introduction to Apache Kafka
● https://kafka.apache.org/ - Kafka Homepage
Thank You!- claudiusmbemba.com/public/slides/cjs2019.pdf
- github.com/NeuCleans/kafka-pub-sub
- github.com/User1m/socketio-kafka-pubsub-sample
- github.com/User1m/kafka-pub-sub-investigation
Top Related