Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... •...
Transcript of Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... •...
![Page 1: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/1.jpg)
Using transactional BPM for serviceorchestration on NoSQL
Joram Barrez – Principal Engineer
Paul Holmes-Higgin - Chief Product Officer
![Page 2: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/2.jpg)
What is it? How is this useful to me? Architecture and some code samples.
Flowable
Our journey: implementation details – challenges & pitfalls - benefits?
MongoDB
Scope & Goal (we’ve got lots to cover)
2
MicroservicesWhat does this mean for Flowable?
![Page 3: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/3.jpg)
Manage complex business activities that require
a mix of human and electronic actions, such as an incoming application, a submitted claim, a
complaint, or incident management
Case management
Coordinate or orchestrate the behaviour of
people, systems, information, and things to produce business outcomes. Processes are
typically structured and repeatable
Business process management
Case and Business Process ManagementThe heart of Digitalization
3
![Page 4: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/4.jpg)
BPMNCMMN
DMN
4
Open Standards
![Page 5: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/5.jpg)
Flowable: the Project
• Open Source• Fork of Activiti by the Activiti developers• Active forums and contributions
• BPMN 2.0, CMMN 1.1 and DMN 1.1 engines• High performance, high scale, years of real world use
• Last release 6.4.0• https://github.com/flowable
• https://flowable.org/
5
![Page 6: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/6.jpg)
Flowable Engines in a nutshell
Models
Deployed- Standalone (e.g REST)- Embedded in any Java env(with mimimal resources needed)
Engines APIs
Process (bpmn)Case (cmmn)Rules (dmn)
- Web apps- Mobile- Machine to Machine
- Open standards- Pluggability & extensibility
![Page 7: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/7.jpg)
What does it give me as a developer?
- Yuck, visual programming!
- The model is “metadata”. It’s the glue between business and dev. It takes away (a big part of) the dreaded requirements document(s).
- Flowable makes working with these models easy and developer-friendly
- Flowable allows you to write unit tests, deploy and work with these models and its instances in a dev-natural way
![Page 8: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/8.jpg)
DIY quick demo
8
Code needed to create a REST endpoint that starts a process instance
![Page 9: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/9.jpg)
Role in a microservices world
• As an orchestrator/conductor: a central component managing a business goal that stretches across service boundaries, keeping services simple and focused.
9
Flowable Engine
Service A
Service B
Service Z
…
![Page 10: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/10.jpg)
Role in a microservices world
• As an orchestrator/conductor: a central component managing a business goal that stretches across service boundaries, keeping services simple and focused.
10
Flowable Engine
Service A
Service B
Service Z
…
events
![Page 11: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/11.jpg)
Role in a microservices world
• As an aggregator
11
New service
Service A
Service B
Service Z…
![Page 12: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/12.jpg)
Role in a microservices world
• As exposing one service, where the function or domain just happens to be implemented by a process engine
12
![Page 13: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/13.jpg)
Evolution of our own architecture
DMN
Form
JobTaskVariable Content
Content
BPMN
FormIDM
CMMN
DMN
BPMN
DMN Form
FormIDMBPMN
REST API
Engine
Service
Spring Boot
OSGI
LDAP
JMX
CDI
Identity Link Camel
Integration
Persistence
Task Job IDM VariableIDM
…
CMMN
CMMN
![Page 14: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/14.jpg)
MongoDB• Flowable relies on the transactional semantics of a
relational db
• “Atomically” moving from one stable state to another
• This doesn’t free you from forgetting about service failures,
but understanding the transactional model of Flowable sure
makes it easier to write resilient processes
• MongoDB 4.0 added support for transactions (June)
14
![Page 15: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/15.jpg)
MongoDB• Open-source NoSql document store
• Short history
• Started in 2007 by 10gen as component of their PaaS
• Was known in the early days (2.2 versions and before) as the dev/null db
• Acquired WiredTiger end of 2014
• WiredTiger default storage engine in 3.2
• WiredTiger enables transactional semantics (ACID) on multi-document operations in 4.0 (*)
15* “Path to transactions” series on https://www.youtube.com/user/MongoDB/videos
![Page 16: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/16.jpg)
Flowable – MongoDB• All code: https://github.com/flowable/flowable-mongodb
• 6.0.4.alpha1 release last week
16
Service call
Command Interceptor/ Commands
Agenda / operations
EntityManagers
DataManagers
Engine core logic
Low-level data access
High-level data functions
![Page 17: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/17.jpg)
Implementation
• MongoDB’s transactions follow a familiar programming model
• Concept of clientSession
• Matches Flowable’s low-level session concept nicely
17
![Page 18: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/18.jpg)
Demo
18
Relational vs MongoDB implementation
![Page 19: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/19.jpg)
Implementation• Replace all Datamanager interface implementations with
a MongoDB counterpart
• -> alpha releases
• Using the existing test suite to validate the implementation
• Completed -> beta / stable release
• (Almost) 1-1 translation of the relational data structure
• Optimizations along the way
• MongoDb-specific structure optimization surely will follow19
![Page 20: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/20.jpg)
Challenges
• com.mongodb.MongoCommandException: Command failed with error 112 (WriteConflict): 'WriteConflict' on server exethanter.local:27017. The full response is { "errorLabels" : ["TransientTransactionError"], "operationTime" : { "$timestamp" : { "t" : 1537701066, "i" : 3 } }, "ok" : 0.0, "errmsg" : "WriteConflict", "code" : 112, "codeName" : "WriteConflict", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1537701066, "i" : 3 } }, "signature" : { "hash" : { "$binary" : "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type" : "00" }, "keyId" : { "$numberLong" : "0" } } } }
20
![Page 21: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/21.jpg)
Challenges
• Taking joins for granted
• Denormalization needed
• Way more work as a developer to guarantee data consistency
• E.g simple example: see ‘latest’ of Process definition
• Exchange writes/updates for faster reads
21
![Page 22: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/22.jpg)
Luckily• Over the past years
• We’ve made Flowable a lot faster by keeping in mind that one exchange over a network is extremely expensive
• Denormalization, prefetching, entity counts
22
![Page 23: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/23.jpg)
Performance
• Is the performance acceptable?
• Benchmark on AWS
• Setup (see GitHub repo)
23
Process Service
Postgres
MongoDB
m5d.2xlarge (8 cores/32Gb RAM), 100GB SSD
m5d.2xlarge (8 cores/32Gb RAM), 100GB SSD
t3.2xlarge (8 cores/32Gb RAM), 100GB SSD
max_connections = 100shared_buffers = 8GBeffective_cache_size = 24GBmaintenance_work_mem = 2GBcheckpoint_completion_target = 0.7wal_buffers = 16MBdefault_statistics_target = 100random_page_cost = 1.1effective_io_concurrency = 200work_mem = 20971kBmin_wal_size = 1GBmax_wal_size = 2GBmax_worker_processes = 8max_parallel_workers_per_gather = 4max_parallel_workers = 8listen_addresses = '*'
![Page 24: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/24.jpg)
Process Service• Bring process into a stable state
• One transaction
• Fixed threadpool of 8 threads
24
- 6 executions- 2 user tasks- 1 hist. proc inst- 11 hist. activities- 2 hist. user tasks- 31 variables- 31 hist. variables- 1 timer job
![Page 25: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/25.jpg)
Results
• Reverse of what we expected J
25
![Page 26: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/26.jpg)
Results
• Reverse of what we expected J
• Although the graphs seem to indicate a relative large difference, we’re talking about sub-ms differences!
• Relational db’s have not been idling
• See our recent performance benchmarks
• https://blog.flowable.org/2018/03/05/flowable-6-3-0-performance-benchmark/
• https://blog.flowable.org/2018/03/13/async-history-performance-benchmark/
26
![Page 27: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/27.jpg)
Conclusion
• The transactional support in MongoDB is impressive
• Data consistency perspective
• Performance perspective
• Using Flowable on MongoDB is a valid alternative
27
![Page 28: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/28.jpg)
Current limitations
• Read/Write to primary only
• Adding replica nodes seemed to have a negative effect
• Even though read/write to primary (current MongoDB transactions limitation)
• MongoDB transactions are still under development
28https://www.youtube.com/watch?v=dQh03YLkmyg
![Page 29: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/29.jpg)
Future work
• MongoDB is designed for horizontal scale
• (Yes, (for example) postgres has partitioning, but …)
• Sharded clusters + Flowable à interesting use cases
• Shard by tenant
• Shard on process definition key
• BigData use cases … like ML!
29
![Page 30: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/30.jpg)
Machine Learning
• MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens up use cases for ML• Demo• Run processes a lot from start to end• Feed historical data into ML• See if human work is repetitive and suggest optimizations
30
![Page 31: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/31.jpg)
Machine Learning
1. Look for Human Decision patterns
31
![Page 32: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/32.jpg)
Machine Learning
1. Look for Human Decision patterns2. Gather possible data inputs and backtrack
![Page 33: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/33.jpg)
Architecture
• vs last year
33
![Page 34: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/34.jpg)
Architecture
34
Process Service
UI
Stream as RDD
Decision Analysis Service
suggestions
Spark (cluster) + MLlib
Process ServiceProcess
Service
Decision Analysis Service
Decision Analysis Service
![Page 35: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/35.jpg)
DemoProcesses with Machine Learning
35
![Page 36: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/36.jpg)
Source Codehttps://github.com/flowable/flowable-
examples/tree/master/demo/demo-jax-2018
36
![Page 37: Using transactional BPM for service orchestration on NoSQL · Role in a microservices world ... • MongoDB being “BigData” (e.g better suited for streaming, reactive, etc.) opens](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d40fe581cf2aa161c6d4/html5/thumbnails/37.jpg)
Thank you!