Building a Replicated Logging System with Apache Kafka
Click here to load reader
-
Upload
guozhang-wang -
Category
Software
-
view
1.491 -
download
1
Transcript of Building a Replicated Logging System with Apache Kafka
Building a Replicated Logging System with Apache Kafka
Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam Mammad Zadeh, Neha Narkhede, Jun Rao, Jay
Kreps, Joe Stein
We All Love Logs!
Apache Kafka
• A distributed messaging system
..that store messages as a log!
Example: LinkedIn back in 2010Point-to-Point Pipelines
What We Want: A Centralized Data Pipeline
Log-centric Data Flow• Logical Ordering
• Persistent Buffering
• “Source-of-Truth”
Store Messages as a Log
4 5 5 7 8 9 10 11 12...
Producer Write
Consumer1 Reads (offset 7)
Consumer2 Reads (offset 10)
Messages3
Partition the Log across Machines
Topic 1
Topic 2
Partitions
Producers
Producers
Consumers
Consumers
Brokers
Apache Kafka
Example: Kafka at LinkedIn
“Source-of-Truth” should not be lost even when..
Replicas and Layout
Logs
Broker-1
topic1-part1
topic1-part3
topic1-part2
Logs
topic1-part2
topic1-part1
topic1-part3
Logs
topic1-part3
topic1-part2
topic1-part1
Broker-2 Broker-3
Consensus for Log Replication
Logs
Broker-1
Logs Logs
Broker-2 Broker-3
Write
Consensus Protocol
Consensus Protocol
Key Idea
Separate membership configuration from data replication
Primary-backup Replication
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write
Conventional Quorum Commits
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write
Conventional Quorum Commits
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write
Conventional Quorum Commits
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Conventional Quorum Commits
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
• Leader maintains in-sync-replicas (ISR)• Failed / slow follower => drop from ISR• Caught-up follower => re-join ISR
• Producer specifies required ACK based on ISR
Configurable ISR Commits
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with all ISRs
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: ACK with Leader-only
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“leader”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2, 3}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2}
Example: Slow Follower
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
Write (ack=“all”)
ISR {1, 2}
Configurable ISR CommitsACK mode Latency On Failures
“no" no network delay some data loss
“leader" 1 network roundtrip a few data loss
“all" ~2 network roundtrips no data loss
• Use an embedded controller• Detect broker failure via ZooKeeper• Leader failure => elect new leader from ISR
• Leader and ISR persisted in Zookeeper• For Controller fail-over
Membership Management
Example: Broker Failure
Logs
Broker-1 *
Logs Logs
Broker-2 Broker-3
ISR {1, 2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 Broker-3ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 * Broker-3
ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 * Broker-3
ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 * Broker-3
ISR {2}
Example: Broker Failure
Logs
Broker-1
Logs Logs
Broker-2 * Broker-3
ISR {2, 3}
• Overview: Logs and Kafka
• Log Replication in Kafka
• Kafka Usage at LinkedIn
• Conclusion
Agenda
Change Log Replication
Apache Kafka
Example: Kafka at LinkedIn
Example: Espresso
• A distributed document store
• Primary online data serving platform at LI
• Member profile, homepage, InMail, etc
[SIGMOD 2013]
Old Espresso ReplicationData Center-1
Storage Node
Storage NodeMySQL
Replication
MySQL MySQL
Search Index
Hadoop …
…DatabusCross-DC Replicator
Data Center-1
Storage Node
Storage NodeMySQL
ReplicationMySQL MySQL
Search Index
Hadoop …
DatabusCross-DC Replicator
Problems with MySQL Replication
Master Storage Node
P1
Slave Storage Node
P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Binary Log Shipping
Replicate Logs with KafkaStorage Node
Kafka Logs
P1
Storage Node
P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Kafka Producer Kafka Consumer Kafka Consumer Kafka Producer
Key-based Log Compaction
...
Partition Messages
Segment-3 Segment-4 Segment-6 *
Key-based Log Compaction
d: 3 f: 8 b: 0 c: null...
Partition Messagesc: 3 a: 5 a: 6 a: 5 f: 9 ...
Segment-3 Segment-4
b: 2 d: 4a: 1
Key-based Log Compaction
... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...
Segment-3 Segment-4
c: 3 a: 5 a: 6b: 2 d: 4a: 1 c: 3 a: 5 a: 6b: 2 d: 4a: 1 d: 3 f: 8 b: 0 a: 5 f: 9
New Segment
Partition Messages
Key-based Log Compaction
... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...
Segment-3 Segment-4
c: 3 a: 5 a: 6b: 2 d: 4a: 1
c: 3 a: 6 d: 3 f: 8 b: 0
c: null a: 5 f: 9
New Segment
Partition Messages
Key-based Log Compaction
... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...
Segment-3 Segment-4
c: 3 a: 5 a: 6b: 2 d: 4a: 1
d: 3 b: 0 a: 5 f: 9
New Segment
Partition Messages
Key-based Log Compaction
... d: 3 f: 8 b: 0 c: null a: 5 f: 9 ...
Segment-3 Segment-4
c: 3 a: 5 a: 6b: 2 d: 4a: 1
d: 3 b: 0 a: 5 f: 9
New Segment
Partition Messages
New Espresso ReplicationData Center-1
Storage Node
Storage Node
Storage Node
Kafka Logs
MySQL MySQL MySQL
Data Center-n
Storage Node
Storage Node
Storage Node
Kafka Logs
MySQL MySQL MySQL
Kafka MirrorMaker
Search Index
Hadoop …
…
Search Index
Hadoop …
* In Progress
Stream Processing
Apache Kafka
Example: Kafka at LinkedIn
• Data flow streaming on Kafka and YARN• Stateful processing
• Re-processing
• Failure Recovery
Example: Samza [CIDR 2015]
Kafka
Kafka
Samza
StateProcess
Protocol
StateProcess
Protocol
StateProcess
Protocol
Samza Processing
Kafka
Kafka
Samza
StateProcess
Protocol
StateProcess
Protocol
StateProcess
Protocol
Samza Processing Kafka Changelog
Kafka
Kafka
Samza
StateProcess
Protocol
StateProcess
Protocol
StateProcess
Protocol
Samza Processing Kafka Changlog
Kafka
Kafka Samza
StateProcess
Protocol
StateProcess
Protocol
StateProcess
Protocol
Samza Processing Kafka Changlog
StateProcess
Protocol
Take-aways• Log-centric data flow helps scaling your
systems
• Kafka: replicated log streams for real-time platforms
We are Hiring
Take-aways• Log-centric data flow helps scaling your
systems
• Kafka: replicated log streams for real-time platforms
THANKS!