codecentric AG: CQRS and Event Sourcing Applications with Cassandra
-
Upload
datastax-academy -
Category
Technology
-
view
4.563 -
download
0
Transcript of codecentric AG: CQRS and Event Sourcing Applications with Cassandra
![Page 1: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/1.jpg)
CQRS and Event Sourcing Applications with Cassandra_Matthias Niehoff #CassandraSummit 2015
1
![Page 2: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/2.jpg)
! The Use Case ! Event Sourcing ! CQRS ! Cassandra for Storage ! Spark for Processing ! Benefits & Pitfalls ! Q&A
Agenda_
2
![Page 3: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/3.jpg)
The Use Case
3
![Page 4: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/4.jpg)
24x7 Proxy_
4
Lega
cy S
yste
ms
(Not
24x
7)
“Inte
rnet
Rea
dy“
App
licat
ions
(2
4x7
avai
labl
e)
24x7 Proxy
•Caches data •Provides data •Stores changes •Provides changes •No business logic/validation
![Page 5: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/5.jpg)
•Solution needs to be highly scalable (up to 100.000 reads/s, 10.000 writes/s)
•Read and write access needs to be low latency
•Read/write ratio is 10:1 or higher
•Solution needs to deal with up to 500.000.000 customers
Assumptions_
5
![Page 6: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/6.jpg)
Event Sourcing
6
![Page 7: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/7.jpg)
Traditional Pattern: Saving Application State_
7
Store
IDAddress
Article
NameStockSize updateInventory()
getInventory()
sells
![Page 8: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/8.jpg)
A series of sales and replenishments for
• a tablet • Starting with 60, sell 20, replenish 10
• a stove • Starting with 25, sell 5, no replenishments
What is different with Event Sourcing?_
8
![Page 9: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/9.jpg)
Saving only application state
What is the Difference?_
9
:ArticleInventory
Fancy Tablet50
:ArticleInventory
Gas Stove20
![Page 10: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/10.jpg)
Saving events instead of state
What is the Difference?_
10
:ArticleInventory
Fancy Tablet3915-08-14T19:..
:ArticleInventory
Gas Stove2015-08-14T19:..
:ArticleInventory
Fancy Tablet4515-08-14T19:..
:ArticleInventory
Gas Stove2015-08-14T19:..
:ArticleInventory
Fancy Tablet5015-08-14T19:..
:ArticleInventory
Gas Stove2015-08-14T19:..
![Page 11: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/11.jpg)
• Log of all stock changes
• Complete rebuild of the state
• Temporal query
• Event replay and rollback
Benefits of Storing Events_
11
![Page 12: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/12.jpg)
CQRS
12
![Page 13: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/13.jpg)
Default Application Architecture_
13
Use
r Int
erfa
ce
Dom
ain
Mod
el
App
licat
ion
Serv
ices
DB
![Page 14: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/14.jpg)
CQRS Application Architecture_
14
Use
r Int
erfa
ce
Que
ry
Serv
ices
Com
man
d Se
rvic
es
Dom
ain
Mod
el
DB
![Page 15: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/15.jpg)
• The pattern is simple
• Going further • Split up the domain model • Independent scaling of models • Not using a query model at all • Different databases for models
A Pattern Changing Your Mindset_
15
![Page 16: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/16.jpg)
Event Sourcing & CQRS_
16
Com
man
d Se
rvic
es
Com
man
d M
odel
Read
Lay
er
Que
ry
Serv
ices
Que
ry
Serv
ices
Que
ry
Serv
ices
Asynchronous
DB
Event Store
Query Stores
ProcessorEvent Processor
DBDBDB
![Page 17: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/17.jpg)
Storage with Cassandra
17
![Page 18: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/18.jpg)
• Not only an event sink • Compaction • Selective replay
• No single point of failure
• Horizontal scale & Geo Replication
•Write ahead of unmodified data
• Plays well with further processing
• Open source & a huge community
• Easy operations
Why Cassandra…
18
![Page 19: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/19.jpg)
For accessing all entities of a given type
Event Store_
19
CREATE TABLE event_source_by_type (entity_type TEXT,bucket INT,entity_key TEXT,insert_time TIMESTAMP,update_time TIMESTAMP,payload TEXT,PRIMARY KEY((entity_type,bucket),insert_time,entity_key)
) WITH CLUSTERING ORDER BY (created_at DESC,entity_key ASC);
e.g. as JSON, XML, protobuf, Avro
prevent huge partitions
![Page 20: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/20.jpg)
CREATE TABLE event_source_by_key (entity_type TEXT,entity_key TEXT,insert_time TIMESTAMP,update_time TIMESTAMP,payload TEXT,PRIMARY KEY((entity_type,entity_key),created_at)
) WITH CLUSTERING ORDER BY (created_at DESC);
For accessing an entity directly
Optional: Second Table_
20
e.g. as JSON, XML or protobuf
![Page 21: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/21.jpg)
• Create tables that fit your queries!
• E.g. „Get articles in category ‚computer‘“
Query Stores_
21
CREATE TABLE articles_by_category ( category TEXT PRIMARY KEY, article_id UUID, article_info TEXT);
may need bucketing
could also be a JSON document
![Page 22: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/22.jpg)
Query Stores_
22
„I need ad-hoc queries“
„I need specific queries with a lot of different filters“
![Page 23: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/23.jpg)
Query Stores_
23
![Page 24: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/24.jpg)
Processing with Spark
24
![Page 25: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/25.jpg)
• Command model triggers event processor
• Event processor updates query views
From Event Store to Query Store_
25
Command Model
Event Processor DBDBDB
Event Processor
Event Processor
![Page 26: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/26.jpg)
Event Processing in Detail_
26
Command Model DBDBDB
![Page 27: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/27.jpg)
• Easy scale out
• Easy deployment
• Intuitive Scala & Java API
• Fault tolerant
• Out-of-the-box Kafka adapter
• Integrates well with Cassandra
Why Spark?
27
![Page 28: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/28.jpg)
• Spark Streaming application
• Consumes only topics of interest
• Joins the stream of events with the current view • Use primary key of entity for correlation • Use joinWithCassandraTable
Spark Job in Detail_
28
![Page 29: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/29.jpg)
1. Create a table for the query view 2. Create a Spark job filling your table 3. Deploy the Spark job 4. Init reprocess of the event DB • same transformation logic as in normal processing • source can be different
5. Mark view as initialized
If you need a new query view_
29
Query DB
Event DB
![Page 30: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/30.jpg)
Benefits & Pitfalls
30
![Page 31: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/31.jpg)
• Scalability • On storage & processing: just add nodes • Efficient queries due to separation
• Collaboration • Every client gets its own data access • Easy to support new queries
Benefits_
31
![Page 32: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/32.jpg)
•More complexity than simple CRUD
• Side effects on event replay
• Eventual consistency in query views
• Concurrent writes
• Performance of replay
Pitfalls_
32
![Page 33: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/33.jpg)
Lost Updates • Due to parallel processing • Two events A and B as sequential input • A is processed after B
• Solution • Partition Spark RDD by entity key • Use a lambda architecture
Pitfalls_
33
spee
d
Data Stream
Serving Layer
batc
h
![Page 34: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/34.jpg)
• Event Store Compaction • Compact store to improve processing time • Only store latest entry of a entity key • e.g. a Spark batch job / Cassandra TTL
• Snapshot / Master State • Constantly build a complete state of all data • Can be used • To speed up initialization • As a store for a search engine
Pitfalls_
34
![Page 35: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/35.jpg)
The Use Case Solved with ES & CQRS
35
![Page 36: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/36.jpg)
24x7 Proxy
24x7 Proxy_
36
Lega
cy C
ore
Syst
ems
(Not
24x
7)
“Inte
rnet
Rea
dy“ A
pplic
atio
ns
(24x
7 av
aila
ble)
![Page 37: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/37.jpg)
37
Questions?
![Page 38: codecentric AG: CQRS and Event Sourcing Applications with Cassandra](https://reader031.fdocuments.in/reader031/viewer/2022021500/5876b4d51a28abad1a8b5735/html5/thumbnails/38.jpg)
Thank You!
Matthias Niehoff,IT-Consultantcodecentric AG Zeppelinstraße 2 76185 Karlsruhe, Germany
www.codecentric.de blog.codecentric.de
matthiasniehoff
38