Microservices with Kafka Ecosystem

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Microservices with KafkaEcosystemGuido SchmutzFrankfurt, 12.12.2017

@gschmutz guidoschmutz.wordpress.com

Guido Schmutz

Working at Trivadis for more than 21 yearsOracle ACE Director for Fusion Middleware and SOAConsultant, Trainer Software Architect for Java, Oracle, SOA andBig Data / Fast DataHead of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 30 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Microservices with Kafka Ecosystem2

COPENHAGEN

MUNICH

LAUSANNEBERN

ZURICHBRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

14 Trivadis branches and more than600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:CHF 5.0 million

Financially self-supporting and sustainably profitable

Experience from more than 1,900 projects per year at over 800customers


Agenda

1. Where do we come from?2. What are Microservices?3. Why not Event Driven?4. Why Kafka for Event-Driven Microservices?5. Asynchronous Microservices in Enterprise Architecture6. Summary7. References



Where do we come from?

Shop Rich UI

Shop Backend Application

Traditional Approach

Search Facade

Customer DAO

Order DAO

Order Facade

Shop UI

Product DAO

UI Logic

DataBusiness

GUI

Customer Fat Client App

Customer BOCustomer UIDataGUI

Data Storage

Shared Database


sync request/response

Shop UI App

Business Activity Service

SOA Approach

Search BAS

Customer DAO

Order DAO

Order BAS

Shop UI

Product DAO

UI Logic

Data

GUI

Service

Business Entity ServiceShop Web App

Shop UI UI LogicGUI

Data Storage

CustomerDatabase

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order andProduct DBSOAP

SOAP

SOAP SOAP

SOAP

SOAP

SOAP


Shop UI App


Virtualized SOA Approach

Search BASCustomer DAO

Order DAO

Order BAS

Shop UI UI LogicGUI

Business Entity Service

Shop Web App

Shop UI UI LogicGUI

Data Storage

CustomerDatabase

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order andProduct DB

Service Virtualization Layer

Service Bus

SOAP SOAP

SOAP

SOAP

SOAP

SOAP

SOAP

9


What are Microservices?

Key Principles of Microservices

Tightly Scoped

Tightly Encapsulated behind concrete interfaces

Responsible for managing their own data

Highly cohesive

Highly decoupled

Independently deployable, self-contained and autonomous


Microservice Approach Customer Microservice

{}

Customer APICustomer

Customer Logic

Order Microservice

{}

Order API OrderOrder Logic

Product Microservice

{}

Product API ProductProduct Logic

Stock Microservice

{}

Stock API StockStock Logic

Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST


Microservice Approachwith API Gateway

Customer Microservice

{}


Customer Logic

Order Microservice

{}



{}


Stock Microservice

{}


Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

API Gateway


Synchronous World of Request-Response leads to tight, point-to-point couplings


problem in lower end of chain have a ripple effect on the other service• crash of service• overloaded service / slow response time• change of interface

Service 2Service 1

{}

APILogic

{}

API Logic

StateState

Service 3

{}

API Logic

State

Service 4

{}

API Logic

State

Service 5

{}

API Logic

State

Service 7

{}

API Logic

State

Service 6

{}

API Logic

State


Why not Event-Driven?

3 mechanisms through which services interact

Request-Driven Event Driven

Service

Logic

State

Event Consume“IPad OrderedEvent”Command

”Order IPad”order(iPad) : boolean

EventPublish“OrderValidatedEvent”

Query”Retrieve my Orders)getAllOrders(myself)

Event Broker


Request-Driven Communication – Highest Coupling

Shop UI Application

UI Logic

Order Microservice

Logic State

{}

API

Stock Microservice

State

{}

API Logic

Call PlaceOrderAPI

Call UpdateStockAPI

Maybe Call Reorder API


Decoupling through Events – Lowest Coupling

Event Broker


Order Microservice

Logic State

{}

API

Stock Microservice

State

{}

API Logic3) Raise OrderConfirmedEvent

4) Listen toOrderConfirmedEvent

5) Maybe Raise Reorder Event

2) Listen toOrderRequested

Event

1) RaiseOrderRequestedEvent


async request/response

async, event pub/sub

Event-Driven (Async)Microservice Approach

Customer Microservice

{}


Customer Logic

Order Microservice

{}



{}


Stock Microservice

{}


Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

API Gateway


Event Broker


async request/response

async, event pub/sub

Shop UI App


Process & Virtualized SOA Approach – Sync & Async

Search BASCustomer DAO

Order DAO

Order BAS

Shop UI UI LogicGUI

Business Entity Service

Shop Web App

Shop UI UI LogicGUI

Data Storage

CustomerDatabase

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order andProduct DB

Service Virtualization Layer

Service Bus

Orchestration


Command Query Responsibility Segregation (CQRS)

Optimize different nonfunctional requirements for read and write behavior

interface to the service is split between • commands that trigger changes in state• queries that provide read access to the state

of resources

support services with higher performance and capacity requirements for readingdata than for writing data

scare reading capacity independently of writing

Data Storage

Write Data Store

Read Data Store(Materialized Views)

Service

Command Service

Query Service

App

UI

ProjectionService

UI Logic


Event Sourcing

persists the state of a business entity as a sequence of state-changing events

Whenever state of business entity changes, a new event is appended to the list of events

Saving an event is a single operation and is inherently atomic

The application reconstructs an entity’s current state by replaying the events

Data Storage

Event Store

Service

Event Service

Publisher

App

UI

UI Logic Replayer

Other App


Event Sourcing & CQRS

Event sourcing is commonly combined with the CQRS pattern

materializing views from the stored events

Optionally Commands can be stored in event store and transformed into events by the command handler

Data Storage

Event Store

Service

Command Service

App

UI

UI Logic

Query Service Read Data Store(Materialized Views)

ProjectionService

Command Handler


Using Event Sourcing with Microservices


“Event sourcing enables building a forward-compatible application architecture — the ability to add more applications in the future that need to process the same event but create a different materialized view.”

Neha Narkhede, Confluent Blog

Microservice

State

Command Handler

{}

API

REST

Event Store

Projection Handler(s)

Query Logic

Event Handler(s)

Event Subscribe

How many Event Stores do we need ?

Microservice

{}

API StateLogic

REST

Event Store

Event

Microservice

{}

API StateLogic

REST

Event Store

Event

Microservice

{}

API StateLogic

REST

Event Store

Event

Microservice

{}

API StateLogic

REST

Microservice

{}

API StateLogic

REST

Event Store

Event

Microservice

{}

API StateLogic

REST

OR


Have only one „source of truth“

Avoid double write!• Would need distributed transactions

Write Event first then consume it from same micro service • “eat your own dog food”

Microservice

{}

API StateLogic

REST

Event Store

Event

Microservice

{}

API StateConsumer

REST

Event Store

Event

Publisher


Data Store

Publish Events through Database

Customer

Event Store

Event

Integration

Microservice

StateLogic

CDCCDC Connector

Customer Fat Client App

Customer BOCustomer UI


Why Kafka for Event-Driven Microservices?


What is apache Apache Kafka?

The who is who• Producers write data to brokers.• Consumers read data from

brokers.• All this is distributed.

The data• Data is stored in topics.• Topics are split into partitions,

which are replicated.

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

Broker 1 Broker 2 Broker 3

ZookeeperEnsemble


Apache Kafka – scalable message processing and more!


Source Connector

trucking_driver

Kafka BrokerSink

Connector

Stream Processing

Leverage the Log

at the heart of Apache Kafka sits a distributed log

collection of messages, appended sequentially to a file

service ‘seeks’ to the position of the last message it read, then scans sequentially, reading messages in order

log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing

Event Hub

01020304050607080910111213141516171819202122

Reads are a singleseek & scan

Writes are append only


Scale-Out Architecture


topic consists of many partitions

producer load load-balanced over all partitions

consumer can consume with as many threads as there are partitions

Producer 1

Consumer 1Broker 1

Producer 2

Producer 3

Broker 2

Broker 3

Consumer 2

Consumer 3

Consumer 4

ConsumerGroup1

ConsumerGroup2

KafkaCluster

Strong Ordering Guarantees

most business systems need strong ordering guarantees

messages that require relative ordering need to be sent to the same partition

supply same key for all messages that require a relative order

To maintain global ordering use a single partition topic

Producer 1

Consumer 1

Broker 1

Broker 2

Broker 3

Consumer 2

Consumer 3

Key-1

Key-2

Key-3Key-4

Key-5

Key-6

Key-3

Key-1


Replay-ability – Logs never forget

by keeping events in a log, we have a version control system for our data

if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable

sequence of events provides an audit point, so that you can examine exactly what happened

rewind and reply events, once service is back and bug is fixed

Event Hub

01020304050607080910111213141516171819202122

Replay

RewindService

Logic State


Hold Data for Long-Term – Data Retention

Producer 1

Broker 1

Broker 2

Broker 3

1. Never

2. Time based (TTL) log.retention.{ms | minutes | hours}

3. Size based log.retention.bytes

4. Log compaction based (entries with same key are removed):

kafka-topics.sh --zookeeper zk:2181 \--create --topic customers \--replication-factor 1 \--partitions 1 \--config cleanup.policy=compact


Keep Topics in Compacted Form

0 1 2 3 4 5 6 7 8 9 10 11

K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2

V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11

OffsetKey

Value

3 4 6 8 9 10

K1 K3 K4 K5 K2 K6

V4 V5 V7 V9 V10 V11

OffsetKey

Value

Compaction


V1 V2

V3

V4 V5

V6

V7

V8

V9V10 V11

K1 K3 K4 K5K2 K6

Topic Viewed as Event Stream or State Stream (Change Log)

Event Stream State Stream (Change Log Stream)

2017-10-02T20:18:46 11,Normal,41.87,-87.67

2017-10-02T20:18:55 11,Normal,40.38,-89.17

2017-10-02T20:18:59 21,Normal,42.23,-91.78

2017-10-02T20:19:01 21,Normal,41.71,-91.32

2017-10-02T20:19:02 11,Normal,38.65,-90.2

2017-10-02T20:19:23 21,Normal41.71,-91.32

11 2017-10-02T20:18:46,11,Normal,41.87,-87.67

11 2017-10-02T20:18:55,11,Normal,40.38,-89.17

21 2017-10-02T20:18:59,21,Normal,42.23,-91.78

21 2017-10-02T20:19:01,21,Normal,41.71,-91.32

11 2017-10-02T20:19:02,11,Normal,38.65,-90.2

21 2017-10-02T20:19:23,21,Normal41.71,-91.32


Keep Topics both in Original and Compacted Form


Kafka BrokerABCBAEKAEACK

BEACK

Producer

Kafka BrokerABCBAEKAEACK

BEACK

Producer

ConsumeandProduce

OR

TX

Legacy Microservice

Change Data Capture (CDC)


RDBMS cdc-source trucking_driver

Driver Topicelasticsearch

-sink NoSQL

Enrich Stream with Static Data with Kafka Streams


Movement Topic ConsumeandProduce

DriverTable

Driver Topic DriverHandler

Join

Driver Local State

Movement & Driver Topic

Building Embedded, Materialized View with Kafka Streams


Movement & Driver TopicConsumeandProduce

JoinStore

Engine Metric Topic

Join

Join State

WindowAggregation

ResultStore

Result State

Building Embedded, Queryable Materialized View with Kafka Streams


Movement & Driver Topic ConsumeandProduce

JoinStore

Engine Metric Topic

Join

Join State

WindowAggregation

ResultStore

Result State

API Application

Asynchronous Microservices in Enterprise Architecture


Hadoop ClusterdHadoop ClusterBig Data Cluster

Asynchronous Microservices in Enterprise Architecture

Location

Social

Clickstream

Sensor Data

Billing &Ordering

CRM / Profile

MarketingCampaigns

CallCenter

MobileApps

SQL

Search

BITools

Enterprise Data Warehouse

Search/Explore

Online&MobileApps

File Import / SQL Import

WeatherData

Event Hub

Parallel Processing

Storage

Storage

Raw

Ref

ined

Results

Microservice Cluster

Microservice State

{}

API

Stream Analytics Cluster

StreamProcessor

State

{}

API

EventStream

EventStream

Service

48

Summary


Summary


• service autonomy is key in a Microservices Architecture!

• not all communication need to be synchronous => separate into• commands

• events

• queries

• Kafka is well suited as an event broker / event store• brings many more interesting features beyond just “message passing”

References


References


Microservices Blog Series, Ben Stopford, Confluent:• https://www.confluent.io/blog/tag/microservicesApache Kafka for Microservices: A Confluent Online Talk Series:• https://www.confluent.io/landing-page/microservices-online-talk-series/Turning the database inside-out with Apache Samza, Martin Kleppmann, Con• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/Immutability Changes Everything, Pat Helland, Salesforce:• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdfCommander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:• https://www.youtube.com/watch?v=B1-gS0oEtYc

Technology on its own won't help you.You need to know how to use it properly.


Microservices with Kafka Ecosystem

Data & Analytics

Transcript of Microservices with Kafka Ecosystem