Making Apache Kafka Elastic with Apache Mesos
Transcript of Making Apache Kafka Elastic with Apache Mesos
![Page 2: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/2.jpg)
Joe Stein
CEO of Elodina, Inc. Elodina http://www.elodina.net/ is a startup focusing on the support & maintenance of third party open source software (like Mesos frameworks) as well as its own open source products & SaaS solutions.
Joe Stein is also the Founder and Principal Consultant of Big Data Open Source Security http://stealth.ly and has been working for the last couple of years on implementing and assisting organizations with their Kafka, Mesos, Hadoop, Cassandra, Accumulo, Storm, Spark, etc, Big Data systems.
Prior to this, Joe Stein was responsible for building out a platform that ingested and processed the analytics for 6 billion unique mobile devices.
![Page 3: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/3.jpg)
◉ Quick intro to Mesos◉ Quick intro to Kafka◉ Making Kafka elastic
with Mesos
Overview
![Page 4: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/4.jpg)
Quick intro to Mesos
![Page 5: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/5.jpg)
Static partitioning
![Page 6: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/6.jpg)
Static partitioning
![Page 7: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/7.jpg)
Static partitioning
![Page 8: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/8.jpg)
Static partitioning
![Page 9: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/9.jpg)
Better option
![Page 10: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/10.jpg)
Data Center Operating System
![Page 11: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/11.jpg)
![Page 12: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/12.jpg)
Mesos
![Page 13: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/13.jpg)
Quick intro to Kafka
![Page 14: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/14.jpg)
Not Streaming Data
![Page 15: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/15.jpg)
Kafka decouples data pipelines
![Page 16: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/16.jpg)
Making Kafka elastic with Mesos
![Page 17: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/17.jpg)
Goals we set out with
● smart broker.id assignment.● preservation of broker placement (through
constraints and/or new features).● ability to-do configuration changes.● rolling restarts (for things like configuration
changes).● scaling the cluster up and down with
automatic, programmatic and manual options.● smart partition assignment via constraints visa
vi roles, resources and attributes.
![Page 19: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/19.jpg)
Scheduler● Provides the operational automation for a Kafka
Cluster.● Manages the changes to the broker's
configuration. ● Exposes a REST API for the CLI to use or any other
client.● Runs on Marathon for high availability.
Executor● The executor interacts with the kafka broker as an
intermediary to the scheduler
Scheduler & Executor
![Page 20: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/20.jpg)
CLI & REST API
● scheduler - starts the scheduler.● add - adds one more more brokers to the cluster.● update - changes resources, constraints or broker properties one or
more brokers.● remove - take a broker out of the cluster.● start - starts a broker up.● stop - this can either a graceful shutdown or will force kill it
(./kafka-mesos.sh help stop)● rebalance - allows you to rebalance a cluster either by selecting the
brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic.
● help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}
![Page 21: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/21.jpg)
Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1./kafka-mesos.sh start 1000..1019
![Page 22: Making Apache Kafka Elastic with Apache Mesos](https://reader036.fdocuments.in/reader036/viewer/2022081504/55bec283bb61eb0d7b8b480b/html5/thumbnails/22.jpg)
Kafka is available on DCOShttps://mesosphere.com/product/
dcos install kafka dcos kafka help
Mesosphere DCOS