Event your life with Kafka
-
Upload
mari-carmen-gutierrez -
Category
Software
-
view
232 -
download
0
Transcript of Event your life with Kafka
Event your life with Kafka
Mari Gutiérrez :: Madrid.rb :: Dec 2014
@valakirka
kafka /ˈkɑːfkɑː/
“Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a
messaging system, but with a unique design.”
zookeeper /ˈzuːˌkiːpə/
“ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed
synchronization, and providing group services.”
topic
partition
broker
producer
consumer
consumer group
kafka in Ruby: Poseidon /bpot/poseidon
require 'poseidon' !class MyProducer def initialize partitioner = lambda { |key, partition_count| key.to_i % partition_count } @producer = Poseidon::Producer.new( ['localhost:9092'], self.class.name, partitioner: partitioner) end ! def send_message(body) message = Poseidon::MessageToSend.new( 'mytopic', body, Time.now.to_i.to_s) ! @producer.send_messages([message]) end end
require 'poseidon' !class MyConsumer def initialize(offset) @consumer = Poseidon::PartitionConsumer. consumer_for_partition(self.class.name, ['localhost:9092'], 'mytopic', 0, offset) end ! def run loop do @consumer.fetch.each do |event| puts event.value end end end end
consumer = MyConsumer.new(0) consumer.run !producer = MyProducer.new producer.send_message('hola Madrid.rb')
require 'poseidon' require 'poseidon_cluster' !class MyConsumerGroup def initialize @group = Poseidon::ConsumerGroup.new( self.class.name, ['localhost:9092'], ['localhost:2181'], 'mytopic') end ! def run @group.fetch_loop do |partition, messages| messages.each { |m| puts m.value } end end end
why kafka? usage pattern
offset & retention
batch/offline consumers
ordered delivery
event sourcing /ɪˈvɛnt ˈsɔːsɪŋ/
“Capture all changes to an application state as a sequence of events.”
¡gracias!
sourceshttp://kafka.apache.org/documentation.html !http://zookeeper.apache.org/ !https://kafka.apache.org/08/ops.html !https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol !http://www.infoq.com/articles/apache-kafka !https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines !https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem !https://github.com/bpot/poseidon !https://github.com/bsm/poseidon_cluster !http://martinfowler.com/eaaDev/EventSourcing.html !http://harpers.org/wp-content/uploads/whykafka-final.jpg !https://github.com/valakirka/kafka_example