Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to...
Transcript of Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to...
![Page 1: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/1.jpg)
1
Fundamentals for Apache Kafka®
Apache Kafka Architecture & Fundamentals Explained
Joe Desmond, Sr. Technical Trainer, Confluent
![Page 2: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/2.jpg)
2
Session Schedule
● Session 1: Benefits of Stream Processing and Apache Kafka Use Cases
● Session 2: Apache Kafka Architecture & Fundamentals Explained
● Session 3: How Apache Kafka Works
● Session 4: Integrating Apache Kafka into your Environment
![Page 3: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/3.jpg)
3
Learning ObjectivesAfter this module you will be able to:
● Identify the key elements in a Kafka cluster
● Name the essential responsibilities of each key element
● Explain what a Topic is and describe its relation to Partitions and Segments
![Page 4: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/4.jpg)
4
The World Produces Data
![Page 5: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/5.jpg)
5
Producers
![Page 6: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/6.jpg)
6
Kafka Brokers
![Page 7: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/7.jpg)
7
Consumers
![Page 8: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/8.jpg)
8
Architecture
![Page 9: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/9.jpg)
9
Decoupling Producers and Consumers● Producers and Consumers are decoupled
● Slow Consumers do not affect Producers
● Add Consumers without affecting Producers
● Failure of Consumer does not affect System
![Page 10: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/10.jpg)
10
How Kafka Uses ZooKeeper
![Page 11: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/11.jpg)
11
ZooKeeper Basics● Open Source Apache Project
● Distributed Key Value Store
● Maintains configuration information
● Stores ACLs and Secrets
● Enables highly reliable distributed coordination
● Provides distributed synchronization
● Three or five servers form an ensemble
![Page 12: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/12.jpg)
12
Topics● Topics: Streams of “related” Messages in Kafka
○ Is a Logical Representation
○ Categorizes Messages into Groups
● Developers define Topics
● Producer Topic: N to N Relation
● Unlimited Number of Topics
![Page 13: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/13.jpg)
13
Topics, Partitions, and Segments
![Page 14: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/14.jpg)
14
Topics, Partitions, and Segments
![Page 15: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/15.jpg)
15
The Log
![Page 16: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/16.jpg)
16
Log Structured Data Flow
![Page 17: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/17.jpg)
17
The Stream
![Page 18: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/18.jpg)
18
Data Elements
![Page 19: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/19.jpg)
19
Brokers Manage Partitions● Messages of Topic spread across Partitions
● Partitions spread across Brokers
● Each Broker handles many Partitions
● Each Partition stored on Broker’s disk
● Partition: 1..n log files
● Each message in Log identified by Offset
● Configurable Retention Policy
![Page 20: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/20.jpg)
20
Broker Basics● Producer sends Messages to
Brokers
● Brokers receive and store Messages
● A Kafka Cluster can have many Brokers
● Each Broker manages multiple Partitions
![Page 21: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/21.jpg)
21
Broker Replication
![Page 22: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/22.jpg)
22
Producer Basics● Producers write Data as Messages
● Can be written in any language
○ Native: Java, C/C++, Python, Go,, .NET, JMS
○ More Languages by Community
○ REST Server for any unsupported Language
● Command Line Producer Tool
![Page 23: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/23.jpg)
23
Load Balancing and Semantic Partitioning● Producers use a Partitioning Strategy to assign each message to a Partition
● Two Purposes:
○ Load Balancing
○ Semantic Partitioning
● Partitioning Strategy specified by Producer
○ Default Strategy: hash(key) % number_of_partitions
○ No Key Round-Robin
● Custom Partitioner possible
![Page 24: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/24.jpg)
24
Consumer Basics● Consumers pull messages from 1..n topics
● New inflowing messages are automatically retrieved
● Consumer offset
○ Keeps track of the last message read
○ Is stored in special topic
● CLI tools exist to read from cluster
![Page 25: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/25.jpg)
25
Consumer Offset
![Page 26: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/26.jpg)
26
Distributed Consumption
![Page 27: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/27.jpg)
27
Scalable Data Pipeline
![Page 28: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/28.jpg)
28
Q&AQuestions:
● Why do we need an odd number of ZooKeeper nodes?
● How many Kafka brokers can a cluster maximally have?
● How many Kafka brokers do you minimally need for high availability?
● What is the criteria that two or more consumers form a consumer group?
![Page 29: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/29.jpg)
29
Continue your Apache Kafka Education!
● Confluent Operations for Apache Kafka
● Confluent Developer Skills for Building Apache Kafka
● Confluent Stream Processing using Apache Kafka Streams and KSQL
● Confluent Advanced Skills for Optimizing Apache Kafka
For more details, seehttp://confluent.io/training
![Page 30: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/30.jpg)
3030
CertificationsConfluent Certified Developer for Apache Kafka(aligns to Confluent Developer Skills for Building Apache Kafka course)
Confluent Certified Administrator for Apache Kafka(aligns to Confluent Operations Skills for Apache Kafka)
What you Need to Know○ Qualifications: 6-to-9 months hands-on
experience
○ Duration: 90 mins
○ Availability: Live, online 24/7
○ Cost: $150
○ Register online: www.confluent.io/certification
![Page 31: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/31.jpg)
3131
cnfl.io/slack
Stay in touch!
cnfl.io/kafka-trainingcnfl.io/download
![Page 33: Slides - Apache Kafka® Architecture & Fundamentals Explained€¦ · for Apache Kafka (aligns to Confluent Developer Skills for Building Apache Kafka course) Confluent Certified](https://reader034.fdocuments.in/reader034/viewer/2022052500/5f10c5827e708231d44abd63/html5/thumbnails/33.jpg)
33
Copyright ©Confluent, Inc. 2014-2019. Privacy Policy | Terms & Conditions.Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of
the Apache Software Foundation