Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh
Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment...
Transcript of Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment...
![Page 1: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/1.jpg)
Real Time Investment Alerts using Apache Kafka at ING Bank
Marcos Maia & Tim van Baarsen
GOTO AMSTERDAM – JUNE 2019
![Page 2: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/2.jpg)
Introduction
Tim van BaarsenSoftware Engineer @ ING
@TimvanBaarsen
Marcos MaiaSoftware Engineer @ ING
@thegroo
![Page 4: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/4.jpg)
Agenda
5
• Use case @ ING• Mifid II• Mobile Investments App
• Overall Architecture• Demo• Metrics / Monitoring• Kafka streams• Local development / Code walk through• Lessons learned• Questions
![Page 5: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/5.jpg)
Disclaimer
6
Both the code and high level architecture we show here today is
not the actual product we developed within ING!This example is inspired by our work
![Page 6: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/6.jpg)
Domain Lingo
7
Tick Price update on stock=
![Page 7: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/7.jpg)
Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!”
8
Mifid IIRegulations
At the end of the day?!
Markets in Financial Instruments Directive II = Regulatory reform financial markets European Union came into effect on January 3, 2018
![Page 8: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/8.jpg)
Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!”
9
Mifid IIRegulations
"No, in real time!Markets in Financial Instruments Directive II =
Regulatory reform financial markets European Union came into effect on January 3, 2018
![Page 9: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/9.jpg)
Screens of our mobile Investments app
10
price going up!
specific price
-10% drop in price
Recurring alerts
![Page 10: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/10.jpg)
2.000 – 12.000 stock updates a second!
11
# $
![Page 11: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/11.jpg)
The perfect “match”
12
❤
![Page 12: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/12.jpg)
Overall architecture
13
Alert serviceAPI
Matching Service
Stocks update receiver
Internet
“SMS Platform”
API
Receive stock price updates ~ 12.000/sec
Publish stock price updates to Kafka
Create alertsDelete alerts
List alerts Customer receives
investment alert
Consume updates & match against alerts created by
customers
In case of a matchsend a sms to
customer
Customer alert(s) stored in database
Notification Service
Schema Registry Zookeeper
“Push Platform”
API
"
![Page 13: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/13.jpg)
Quick Demo
14
![Page 14: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/14.jpg)
Monitoring - Metrics
15
• Micrometer
• Think SLF4J, but for metrics (Jon Schneider)
• Application metrics
• Out of the box metrics
• Business metrics
• Vendor Neutral
![Page 15: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/15.jpg)
Monitoring – Real-time?
16
![Page 16: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/16.jpg)
Kafka Streams Join
17
Matching Service
raw-stock-quote stock-quote-tick stock-quote-metalog-compaction
Raw-tick
stock-quote-ticker RocksDB
Key value
ING Raw-tick
GOOGL Raw-tick
AAPL Raw-tick
Key value
ING Meta-data
GOOGL Meta-data
AAPL Meta-data
GlobalKTableKStream
![Page 17: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/17.jpg)
Local development
18
• Docker• Kafka• Zookeeper• Kafka manager• All our Spring Boot applications
• Producers for development• Custom Kafka command line producer
• Lab project
![Page 18: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/18.jpg)
Spring Kafka
19
• Producing: • ProducerFactory• KafkaTemplate
• “The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.”
• Consuming• Kafka consumers are not thread safe• @KafkaListener• MessageListener (many different interfaces)• Access to ConsumerRecord• Access to manual control to commit the offset• Scale up multiple consumer (threads) within the application
![Page 19: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/19.jpg)
Spring Kafka
20
• Integration tests
• Junit rule: EmbeddedKafkaRule• @EmbeddedKafka
• Metrics
• Kafka consumer metrics out of the box (Micrometer)
![Page 20: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/20.jpg)
Lessons learned
21
• Use Spring Kafka after you understand the Kafka APIs
• Error handling • (De)serialization -> Apache Avro
• Think about # of partitions per topic upfront if possible (routing)• Can your consumers keep up? Can you scale up?
• Monitor your topics / streams in production• Based on metrics -> Micrometer
![Page 21: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/21.jpg)
Lessons learned
22
• Kafka clients are backward compatible with Kafka brokers
• Kafka Streams• KTables• GlobalKTables• Interactive Queries• RocksDB build for ssd• Describe• processing.guarantee=exactly_once
• Kafka topology• Plan ahead• small & multiple topologies
![Page 22: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/22.jpg)
Questions
23
&
![Page 23: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/23.jpg)
Contact
Tim van BaarsenSoftware Engineer @ ING
@TimvanBaarsen
Marcos MaiaSoftware Engineer @ ING
@thegroo
![Page 24: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/24.jpg)
![Page 25: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/25.jpg)
Backup slides
![Page 26: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/26.jpg)
Matching in memory
28
Matching Service
Customer alert(s) stored in database
Matching Service
Matching Service
Matching Service
Database state of alerts in
memory cache
In memory cache is
updated based on event
published to Kafka
Alert created event
Alert matched event
Alert deleted event
ING AAPL NFLX ESTC
![Page 27: Real Time Investment Alerts using Apache Kafka at ING Bank · 2019-06-19 · Real Time Investment Alerts using Apache Kafka at ING Bank Marcos Maia & Tim van Baarsen GOTO AMSTERDAM–JUNE](https://reader034.fdocuments.in/reader034/viewer/2022042112/5e8d423ae20af31a9c7dd565/html5/thumbnails/27.jpg)
Kafka Topology
29