101 ways to configure kafka - badly
-
Upload
henning-spjelkavik -
Category
Technology
-
view
347 -
download
4
Transcript of 101 ways to configure kafka - badly
![Page 1: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/1.jpg)
101* ways to configure Kafka - badly
Hva FINN.no har lært av å kjøre Kafka
Audun Fauchald StrandLead Developer Infrastructure
@audunstrandbio: gof, mq, ejb, wli, bpel eda, soa esb, ddd, k8s
Henning Spjelkavik Architect
@spjelkavikbio: Skiinfo (Vail Resorts),
FINN.noenjoys reading jstacks
![Page 2: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/2.jpg)
agenda
Introduction to kafka
kafka @ finn.no
101* mistakes
questions
“From a certain point onward
there is no longer any turning
back. That is the point that must
be reached.”
― Franz Kafka, The Trial
![Page 3: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/3.jpg)
introduction to kafka
![Page 4: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/4.jpg)
why use kafka
#notAnESB
what is a log
terminology
components
giant leap“A First Sign of the Beginning of Understanding is the Wish to Die.”― Franz Kafka
https://commons.wikimedia.org/wiki/File:Kafka.jpg
![Page 5: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/5.jpg)
Why use Kafka?
“Apache Kafka is publish-subscribe messaging
rethought as a distributed commit log.”
Fast
Scalable
Durable
Distributed by design
Sweet spot: High volume, low latency
Quora:“Use Kafka if you have a fire hose of events (100k+/sec)
you need delivered in partitioned order 'at least once' with a mix of online and batch consumers, you want to be able to re-read messages”
“Use Rabbit if you have messages (20k+/sec) that need to
be routed in complex ways to consumers, you want per-message delivery guarantees, you don't care about ordered delivery”
![Page 6: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/6.jpg)
What is a (data) log / journal?
A log is perhaps the simplest possible storage abstraction.
It is an append-only, totally-ordered sequence of records ordered by time.
Appended to the end of the log, reads proceed left-to-right.
Each entry is assigned a unique sequential log entry number.
The ordering of records defines a notion of "time" since entries to the left are
defined to be older then entries to the right.
This is a data log, not an application log (i.e not log4j)
![Page 7: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/7.jpg)
Changelog 101: Tables and Events are Dual
Duality: a log of changes and a table.
Accounting
log: credit and debit (events pr key)
table: all current balances (i.e state pr key)
In a sense the log is the more fundamental data structure: in addition to creating the
original table you can also transform it to create all kinds of derived tables.
![Page 8: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/8.jpg)
producers writes to brokers
consumers reads from brokers
everything is distributed
data is stored in topics
topics are split into partitions
which are replicated
kafka cluster
consumer
producerproducerproducer producer
consumerconsumerconsumer
consumerconsumer
producerproducer
terminology
![Page 9: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/9.jpg)
producer“ad.new”
consumergroup.1
“ad.new”
Broker1
Broker2
Broker3
P0R1
P0R2
P1R3
P2R1
P1R2
P2R3
zookeeper
components consumergroup.id
“ad.new”
consumergroup.1
“ad.new”
![Page 10: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/10.jpg)
1:data 2:... 3:... 4:... 5:... 6:... 7:...
old messages newer messages
...
consumer.group.1
consumer.group.2
producer a1
![Page 11: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/11.jpg)
Giant leap?
In fact, persistent replicated messaging is such a giant leap in messaging architecture that it may be worthwhile to point out a few side
effects:
a. Per-message acknowledgments have disappeared
b. ordered delivery
c. The problem of mismatched consumer speed has disappeared. A slow consumer can peacefully co-exist with a fast
consumer now
d. Need for difficult messaging semantics like delayed delivery, re-delivery etc. has disappeared. Now it is all up to the
consumer to read whatever message whenever - onus has shifted from broker to consumer
e. The holy grail of message delivery guarantee: at-least-once is the new reality - both Kafka and Azure Event Hub
provides this guarantee. You still have to make your consumers and downstream systems idempotent so that recovering
from a failure and processing the same message twice does not upset it too much, but hey - that has always been the
case
http://blogs.msdn.com/b/opensourcemsft/archive/2015/08/08/choose-between-azure-event-hub-and-kafka-_2d00_-what-
you-need-to-know.aspx
![Page 12: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/12.jpg)
Confluent platform
![Page 13: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/13.jpg)
Top 5
1. no consideration of data on the
inside vs outside
2. schema not externally defined
3. same config for every
client/topic
4. 128 partitions as default config
5. running on 8 overloaded nodes
![Page 14: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/14.jpg)
reklame
![Page 15: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/15.jpg)
FINN.nohttps://finn.no/apply-here
https://twitter.com/@FINN_techhttp://tech.finn.no
https://github.com/finn.no
60 millions pageviews a day
250-300 microservices
130 developers
1479 deployed changes into
production last week
6 minutes from commit to deploy
(median)
![Page 16: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/16.jpg)
#javazone @spjelkavik @audunstrand
Schibsted Media Group6800 people in 30 countries
FINN.no is a part of
6900
employees30 countries
200 million
users
jobs.schibsted.com
![Page 17: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/17.jpg)
kafka @ finn.no
![Page 18: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/18.jpg)
kafka @finn.no architecture
use cases
tools
![Page 19: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/19.jpg)
#javazone @spjelkavik @audunstrand
in the beginning ...
Architecture governance board decided to use RabbitMQ as message queue.
Kafka was installed for a proof of concept, after developers spotted it januar 2013.
![Page 20: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/20.jpg)
#javazone @spjelkavik @audunstrand
2013 - POC
“High” volume
Stream of classified ads
Ad matching
Ad indexed
mod05
zk
kafka
mod07
zk
kafka
mod01
zk
kafka
mod03
zk
kafka
mod06
zk
kafka
mod08
zk
kafka
mod02
zk
kafka
mod04
zk
kafka
dc 1
dc 2
Version 0.8.1
4 partitions
common client
java library
thrift
![Page 21: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/21.jpg)
#javazone @spjelkavik @audunstrand
2014 - Adoption and complaininglow volume/ high
reliability
Ad Insert
Product Orchestration
Payment
Build Pipeline
click streams
mod05
zk
kafka
mod07
zk
kafka
mod01
zk
kafka
mod03
zk
kafka
mod06
zk
kafka
mod08
zk
kafka
mod02
zk
kafka
mod04
zk
kafka
dc 1
dc 2
Version 0.8.1
4 partitions
experimenting
with
configuration
common java
library
![Page 22: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/22.jpg)
#javazone @spjelkavik @audunstrand
toolingalerting
![Page 23: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/23.jpg)
#javazone @spjelkavik @audunstrand
2015 - Migration and consolidation“reliable messaging”
asynchronous
communication
between services
store and forward
zipkin
slack notifications
dc 1
dc 2
Version 0.8.2
5-20 partitions
multiple
configurations
broker05
zk
kafka
broker01
zk
kafka
broker03
zk
kafka
broker04
zk
kafka
broker02
zk
kafka
![Page 24: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/24.jpg)
#javazone @spjelkavik @audunstrand
toolingGrafana dashboard visualizing jmx stats
via Prometheus
Kafka-manager from Yahoo
Kafka-cat cli
![Page 25: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/25.jpg)
#javazone @spjelkavik @audunstrand
2016 - Confluent
broker01
broker05
Schema
Registry
kafka
kafka
broker03
kafka
broker04
kafka
broker02
kafka
zk02 zk zk03 zk
Rest Proxy
zk01 zk
platform
schema registry
data replication
kafka connect
kafka streams
![Page 26: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/26.jpg)
101* mistakes
“God gives the nuts, but he does not crack them.”― Franz Kafka
![Page 27: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/27.jpg)
Pattern Language
why is it a mistake
what is the consequence
what is the correct solution
what has finn.no done
![Page 28: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/28.jpg)
Top 5
1. no consideration of data on the
inside vs outside
2. schema not externally defined
3. same config for every
client/topic
4. 128 partitions as default config
5. running on 8 overloaded nodes
![Page 29: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/29.jpg)
#javazone @spjelkavik @audunstrand
mistake: no consideration of data on the inside vs outside
https://flic.kr/p/6MjhUR
![Page 30: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/30.jpg)
#javazone @spjelkavik @audunstrand
why is it a mistakeeverything published on Kafka (0.8.2) is visible to any client that can access
![Page 31: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/31.jpg)
#javazone @spjelkavik @audunstrand
what is the consequencedirect reads across services/domains is quite normal in legacy and/or enterprise
systems
coupling makes it hard to make changes
unknown and unwanted coupling has a cost
Kafka had no security per topic - you must add that yourself
![Page 32: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/32.jpg)
#javazone @spjelkavik @audunstrand
what is the correct solutionConsider what is data on the inside, versus data on the outside
Convention for what is private data and what is public data
If you want to change your internal representation often, map it before publishing it
publicly (Anti corruption layer)
![Page 33: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/33.jpg)
#javazone @spjelkavik @audunstrand
what has finn.no doneDecided on a naming convention (i.e Public.xyzzy) for public topics
Communicates the intention (contract)
![Page 34: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/34.jpg)
#javazone @spjelkavik @audunstrand
mistake: schema not externally defined
![Page 35: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/35.jpg)
#javazone @spjelkavik @audunstrand
why is it a mistakedata and code needs separate versioning strategies
version should be part of the data
defining schema in a java library makes it more difficult to access data from non-
jvm languages
very little discoverability of data, people chose other means to get their data
difficult to create tools
![Page 36: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/36.jpg)
#javazone @spjelkavik @audunstrand
what is the consequencedevelopment speed outside jvm has been slow
change of data needs coordinated deployment
no process for data versioning, like backwards compatibility checks
difficult to create tooling that needs to know data format, like data lake
and database sinks
![Page 37: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/37.jpg)
#javazone @spjelkavik @audunstrand
what is the correct solutionconfluent.io platform has a separate schema registry
apache avro
multiple compatibility settings and evolutions strategies
connect
more automatic tooling
![Page 38: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/38.jpg)
#javazone @spjelkavik @audunstrand
what has finn.no donestill using java library, with schemas in builders
confluent platform 2.0 is planned for the next step, not (just) kafka 0.9
![Page 39: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/39.jpg)
#javazone @spjelkavik @audunstrand
mistake: running mixed load with a single, default configuration
https://flic.kr/p/qbarDR
![Page 40: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/40.jpg)
#javazone @spjelkavik @audunstrand
why is it a mistakeHistorically - One Big Database with Expensive License
Database world - OLTP and OLAP
Changed with Open Source software and Cloud
Tried to simplify the developer's day with a single config
Kafka supports very high throughput and highly reliable
![Page 41: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/41.jpg)
#javazone @spjelkavik @audunstrand
what is the consequenceTrade off between throughput and degree of reliability
With a single configuration - the last commit wins
Either high throughput, and risk of loss - or potentially too slow
![Page 42: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/42.jpg)
#javazone @spjelkavik @audunstrand
what is the correct solutionUnderstand your use cases and their needs!
Use proper pr topic configuration
Consider splitting / isolation
![Page 43: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/43.jpg)
#javazone @spjelkavik @audunstrand
Defaults that are quite reliable
Exposing configuration variables in the client
Ask the questions;
at least once delivery
ordering - if you partition, what must have strict ordering
99% delivery - is that good enough?
what level of throughput is needed
what has finn.no done
![Page 44: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/44.jpg)
#javazone @spjelkavik @audunstrand
ConfigurationConfiguration for production
Partitions
Replicas (default.replication.factor)
Minimum ISR (min.insync.replicas)
Wait for acknowledge when producing messages (request.required.acks, block.on.buffer.full)
Retries
Leader election
Configuration for consumer
Number of threads
![Page 45: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/45.jpg)
#javazone @spjelkavik @audunstrand
Gwen Shapira recommends...akcs = all
block.on.buffer.full = true
retries = MAX_INT
max.inflight.requests.per.connect = 1
Producer.close()
replication-factor >= 3
min.insync.replicas = 2
unclean.leader.election = false
auto.offset.commit = false
![Page 46: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/46.jpg)
#javazone @spjelkavik @audunstrand
mistake: default configuration of 128 partitions for each topic
https://flic.kr/p/6KxPgZ
![Page 47: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/47.jpg)
#javazone @spjelkavik @audunstrand
why is it a mistakepartitions are kafkas way of scaling consumers, 128 partitions can handle 128
consumer processes
in 0.8; clusters could not reduce the number of partitions without deleting data
highest number of consumers today is 20
![Page 48: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/48.jpg)
#javazone @spjelkavik @audunstrand
what is the consequenceour 0.8 cluster was configured with 128 partitions as default, for all topics.
many partitions and many topics creates many datapoints that must be coordinated
zookeeper must coordinate all this
rebalance must balance all clients on all partitions
zookeeper and kafka went down (may 2015)
(500 topics * 128 partitions)
![Page 49: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/49.jpg)
#javazone @spjelkavik @audunstrand
what is the correct solutionsmall number of partitions as default
increase number of partitions for selected topics
understand your use case
reduce length of transactions on consumer side
partitions per topic: max(t/p, t/c)
Max partitions on a broker 100 x brokers x replication factor => 1500 in our case
http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
![Page 50: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/50.jpg)
#javazone @spjelkavik @audunstrand
what has finn.no done5 partitions as default
4 topics have more than 5 partitions, topics with lots of traffic
![Page 51: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/51.jpg)
#javazone @spjelkavik @audunstrand
mistake: deploy a proof of concept hack - in production ; i.e why we had 8 zk nodes
https://flic.kr/p/6eoSgT
![Page 52: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/52.jpg)
#javazone @spjelkavik @audunstrand
why is it a mistakeKafka was set up by Ops for a proof of concept - not for hardened production use
By coincidence we had 8 nodes for kafka, the same 8 nodes for zookeeper
Zookeeper is dependent on a majority quorum, low latency between nodes
The 8 nodes were NOT dedicated - in fact - they were overloaded already
![Page 53: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/53.jpg)
#javazone @spjelkavik @audunstrand
what is the consequenceZookeeper recommends 3 nodes for normal usage, 5 for high, and any more is
questionable
More nodes leads to longer time for finding consensus, more communication
If we get a split between data centers, there will be 4 in each
You should not run Zk between data centers, due to latency and outage
possibilities
![Page 54: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/54.jpg)
#javazone @spjelkavik @audunstrand
what is the correct solutionHave an odd number of Zookeeper nodes - preferrably 3, at most 5
Don’t cross data centers
Check the documentation before deploying serious production load
Don’t run a sensitive service (Zookeeper) on a server with 50 jvm-based services,
300% over committed on RAM
Watch GC times
![Page 55: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/55.jpg)
#javazone @spjelkavik @audunstrand
what has finn.no done
dc 1
dc 2
broker05
zk
kafka
broker01
zk
kafka
broker03
zk
kafka
broker04
zk
kafka
broker02
zk
kafka
Version 0.8.2
5-20 partitions
multiple
configurations
![Page 56: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/56.jpg)
#javazone @spjelkavik @audunstrand
“They say ignorance is
bliss.... they're wrong ”
― Franz Kafka
![Page 57: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/57.jpg)
![Page 58: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/58.jpg)
“It's only because of their stupidity that they're able to be so sure of themselves.” ― Franz Kafka, The
Trial
Audun Fauchald Strand
@audunstrand
Henning Spjelkavik
@spjelkavik
Q?
https://www.finn.no/apply-here
https://tech.finn.no
https://twitter.com/@FINN_tech
https://github.com/finn.no
http://www.schibsted.com/en/Career/
![Page 59: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/59.jpg)
#javazone @spjelkavik @audunstrand
Runner upUsing pre-1.0 software
Have control of topic creation
Kafka is storage - treat it like one also ops-wise
Client side rebalancing
Commiting on all consumer threads, believing that you only commited on one
![Page 60: 101 ways to configure kafka - badly](https://reader031.fdocuments.in/reader031/viewer/2022021923/58ef214b1a28ab135e8b4609/html5/thumbnails/60.jpg)
#javazone @spjelkavik @audunstrand
References / Further readingDesigning data intensive systems, Martin Kleppmann
Data on the inside - data on the outside, Pat Helland
The Confluent Blog, http://confluent.io/
Kafka - The definitive guide
This presentation, in English: http://www.confluent.io/blog/the-top-sessions-from-
kafka-summit-2016
www.finn.no/apply-here
jobs.schibsted.com/
tech.finn.no
twitter.com/@FINN_tech
github.com/finn.no