Picking a Message Queue.
Transcript of Picking a Message Queue.
![Page 1: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/1.jpg)
Picking a Message Queue.
Vladislav Kirshtein
Softwareengineer/consultant
1
![Page 2: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/2.jpg)
2
![Page 3: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/3.jpg)
About me
Senior consultant at CodeValue since 2015
Experienced in large scale software development
10 years of software engineering experience
Particularly interested in highload projects, distributed systems, and cloud computing
3
![Page 4: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/4.jpg)
Agenda
Message oriented protocols
AMQP - RabbitMQ
ZeroMQ
Cloud
NATS
4
![Page 5: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/5.jpg)
Protocols
▪ Stomp – Simple (or Streaming) Text Oriented Message Protocol
▪ MQTT – Message Queue Telemetry Transport
▪ AMQT – Advanced Message Queuing Protocol
▪ Vendor specific (Kafka,NATS,ZeroMQ etc)
![Page 6: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/6.jpg)
STOMP – Simple Text Oriented Message Protocol
• Simple interoperable protocol for asynchronous messaging• coming from HTTP school of design
• Text Based
• Communication between client and server is through a 'frame'
• Client frames : SEND,SUBSCRIBE/ UNSUBSCRIBE, ACK/NACK…
• Server frames: MESSAGE,RECEIPT,ERROR
STOMPSERVER
Client Client
Send Subscribe
MessageReceipt
ACK/NACK
![Page 7: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/7.jpg)
MQTT – Message Queue Telemetry Transport• Created by IBM for
embedded devices telemetry
• Ideal for constrained networks (low bandwidth, high latency, data limits, and fragile connections)
• Binary format (2-byte header)
• Pub/Sub
• Reliable – QoS for reliability on unreliable networks
• Security• Authentication• TLS/SSL
• Simple – 43 page spec.
• Connect/Disconnect + Publish + Subscribe/Unsubscribe
7
![Page 8: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/8.jpg)
AMQP – Advanced Message Queue Protocol
Conceived by JP Morgan in 2006
Answer to current Middleware Hell
100’s of applications
10,000’s of links
every connection different
massive waste of effort
Goal to make interoperable Message Oriented Middleware
8
![Page 9: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/9.jpg)
AMQP 0-10 vs 1.0
![Page 10: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/10.jpg)
Direct Exchange
10
Producer Exchange
Consumer
Consumer
Routing key Binding
![Page 11: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/11.jpg)
Fan-out exchange
11
Producer Exchange
C
Routing key
C
C
C
C
![Page 12: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/12.jpg)
Topic Exchange
12
Producer Exchange
C
Routing key
C
C
C
C
order.bissli
![Page 13: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/13.jpg)
RabbitMQ
Open source
Support multiple protocols AMQP,MQTT,STOMP,HTTP
Routing capabilities
Reliability
Scalability through clustering
High availability
Management and monitoring
Security
Traceability and debugging
Pluggable architecture
13
![Page 14: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/14.jpg)
RabbitMQ
14
• Clustering• Act as single logical unit• Consumer seens cluster as a single
node• Automatic metadata replication
• Mirrored queues
• Federation/Shovel• Consumes and republishers• Exchange to exchange• Works well accros WAN• Two main difference
• Shovel is low level• Shovel is more flexible
![Page 15: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/15.jpg)
RabbitMQ
15
• Know your rabbit (DevOps!)• Erlang• Configuration are Erlang tuples• You will need to deal with Erlang's message
passing framework• Reading source and extend the system will
be difficult
• Optimized for discrete message handling
• Content filtering• It's not too fast
• Optimized for discrete message handling• Fan-out ~35000 mps• Direct ~35000 mps• Topic ~10000 mps• Consistent-Hash ~40000 mps
![Page 16: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/16.jpg)
RabbitMQ
• Community support• https://groups.google.com/forum/#!forum/rabbitmq-users
• Slack - https://rabbitmq-slack.herokuapp.com/
• Official site - http://www.rabbitmq.com/
• Books• RabbitMQ in Action
• RabbitMQ in Depth
• RabbitMQ Essentials
• RabbitMQ best practices – CloudAMQP• https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html
• Video
16
![Page 17: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/17.jpg)
RabbitMQ - Demo
17
Client Server
Request queue
Response queue
https://github.com/vgrokk/RabbitMQDemo
![Page 18: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/18.jpg)
RabbitMQ - Demo
18
Producer Exchange
C
C
C
C
C
order.bissli
https://github.com/vgrokk/RabbitMQDemo
![Page 19: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/19.jpg)
No more brokers
19
• Brokers are single point of failure
• Not horizontally scalable
• Reduce reliability with addition of nodes
• HA provides minimal benefit, and adds complexity
![Page 20: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/20.jpg)
20
![Page 21: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/21.jpg)
ZeroMQ
What Ø come for?
The original Ø meant:• Zero broker
• Zero latency (as close as possible)
With the time• Zero administration
• Zero cost
• Zero waste
21
![Page 22: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/22.jpg)
ZeroMQ
22
![Page 23: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/23.jpg)
ZeroMQCreators - iMatix CEO Pieter Hintjens and Martin Sustrik
Super socket libraryLanguage agnostic (40+ languages)Multiple network protocols
Multiple connection patterns
High throughput/low latency (~5m mps, 30usec latency)
MessagesBatching Atomic
Small (~25k lines of code)
Open source. Large supported community. Multiple forks (nanomessage)
23
![Page 24: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/24.jpg)
ZeroMQ - Cons
24
Durability
Static routing
No out of the box solutionService discovery
Management
Delivery deadlines
QoS
Low level
Internal queue management
![Page 25: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/25.jpg)
ZeroMQ - Sockets
• Unicast – inproc, ipc (except on Windows), tcp
• Multicast – pgm, epgm
• Asynchronous
• Designed for particular messaging patterns
• Connect to multiple endpoints
• Automatically reconnect
• Queue messages when under heavy load
25
![Page 26: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/26.jpg)
ZeroMQ - Sockets
26
• PUB - Outgoing broadcast
• SUB - Broadcast listener
• REQ - Synchronous (1 msg at a time) request
• REP - Synchronous reply
• DEALER - Async request
• ROUTER - Async reply
• PUSH - One way outgoing to PULL(s)
• PULL - One way incoming from PUSH(es)
• PAIR - One-to-one 2-way with another PAIR
PUB and SUB
REQ and REP
REQ and ROUTER
DEALER and REP
DEALER and ROUTER
DEALER and DEALER
ROUTER and ROUTER
PUSH and PULL
PAIR and PAIR
![Page 27: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/27.jpg)
ZeroMQ - Demo
27
https://github.com/vgrokk/ZeroMQDemo
![Page 28: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/28.jpg)
ZeroMQ - Demo
28
https://github.com/vgrokk/ZeroMQDemo
![Page 29: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/29.jpg)
ZeroMQ - Demo
29
https://github.com/vgrokk/ZeroMQDemo
![Page 30: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/30.jpg)
30
![Page 31: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/31.jpg)
Cloud
You host your services on cloud
You don’t have a dedicated budget or skills to maintain your own queueing system
You aren’t necessarily interested in lowest latency
You prefer easy solutions over powerful solutions when queuing is considered
You don’t plan on processing large amounts of messages
31
![Page 32: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/32.jpg)
Self - hosted
32
Your system is distributed or hosted outside of public cloud
You want an open solution that you can reuse
You’re worried about latency
You want to configure exchanges or use some other non-trivial methods of message distribution
You want your queuing system to be extendable and configurable, and
You want maximum performance and have dedicated resources to invest in your messaging solution.
![Page 33: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/33.jpg)
Azure
33
![Page 34: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/34.jpg)
Storage queuePROS
• Can store over 80Gb in queue
• Low cost (100x cheaper than Azure Service Bus)
• Batch receive (up to 32 messages)
• Only pay for storage and operations
• Redundancy(LRS, ZRS, GRS, RA-GRS)
• Rest api
CONS
• Maximum size of a message is 64KB
• Maximum mps is 2000
• Maximum TTL for each message is 7 days
• No order guaranty
• Only polling (for receiver)
• No batch support for sender
34
![Page 35: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/35.jpg)
Service bus
35
![Page 36: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/36.jpg)
Service Bus
36
Scheduled delivery
QoS
Poison message handling
Defer
Auto Forward
Sessions
Batching
Ordering
Auto-delete on idle
OnMessage (.NET only)
Duplicate detection
Content filters
Transactions
Dead lettering
![Page 37: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/37.jpg)
Amazon
37
AWS SQS Amazon SNS
AmazonMQ
Amazon SQS
Amazon Kinesis
AWS IOT
Managed queue
Managed pub/sub
Managed Apache
ActiveMQ
Managed event
streaming
![Page 38: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/38.jpg)
Amazon SQS
• Unlimited queues and messages
• Payload Size: up to 256KB
• Batches
• Long polling
• Retain messages in queues for up to 14 days
38
• Send and read messages simultaneously
• Message locking
• Queue sharing
• Server-side encryption (SSE)
• Dead Letter Queues (DLQ)
• Amazon CloudWatch metrics + alerts
![Page 39: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/39.jpg)
Amazon SNS
39
• Fan-out pattern• Reliability• Flexible• Nearly unlimited throughput• Secure• Payloads up to 256 KB• Amazon CloudWatch
![Page 40: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/40.jpg)
Amazon MQ
40
Amazon MQ is a managed message broker service
for Apache ActiveMQ that makes it easy to set up and operate
message brokers in the cloud.
![Page 41: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/41.jpg)
Amazon MQ
41
ActiveMQ Features• Transient & Persistent• Industry standard
APIs and protocols• Local & Distributed
transactions• Queues & topics (FIFO)• Message filtering• Request/reply• Scheduled messages• Unlimited message size• Unlimited retention
AmazonMQ Features• Provisioning• Updates• Security• Maintenance• Monitoring• Troubleshooting• High availability
![Page 42: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/42.jpg)
42
Demo time
![Page 43: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/43.jpg)
43
• Was created by Derek Colission in 2010• Open source, lightweight, high-performance cloud
native messaging system• Capable of sending 11-12 million messages per
second• Written in Go• Available in two interoperable modules
• NATS Server• NATS Streaming Server
![Page 44: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/44.jpg)
44
![Page 45: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/45.jpg)
NATS
45
Chart source: bravenewgeek.com/dissecting-message-
queues
![Page 46: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/46.jpg)
NATS Server
Highly performant, extremely lightweight
Clustered servers/clustered aware clients
Built in resilience and high availabilityAuto discovery for topology
Text based protocol (payload as array of bytes)
Monitorable on a dedicated port
TLS support
Auto Pruning of Clients
Pure Pub/Sub
Request/Reply ,Queueing pattern
Smart routing with wildcards
46
Durability
Transactions
Enhanced Delivery Models
Flow control (rate matching/limitng)
![Page 47: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/47.jpg)
NATS use cases
High throughput
Addressing, discovery
Command and control (control plane)
Load balancing
N-way scalability .
Location transparency
Fault tolerance
47
![Page 48: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/48.jpg)
NATS Streaming
Layer on top of NATS
Enhanced message protocol - NATS Streaming implements its own enhanced message format using Google Protocol Buffers.
Message/event persistence
QoS
Flow control (rate matching/limiting)
Message replay by subjectAll available messagesLast published valueAll messages since specific sequence numberAll messages since a specific date/time
Durable subscribers
48
![Page 49: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/49.jpg)
NATS Streaming
49
![Page 50: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/50.jpg)
NATS
50
Chart source: https://dzone.com/articles/benchmarking-nats-streaming-and-apache-kafka
![Page 51: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/51.jpg)
NATS cluster
51
• For high availability, a full mash of NATS servers can be setup
• Clients can connect to any of nodes to communicate with other clients, the NATS cluster would then route the messages
• Routing will be done if clients showed interests in subject
• Whenever a new NATS servers joins then members already in cluster connected as well to form the full mash
![Page 52: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/52.jpg)
52
![Page 53: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/53.jpg)
Summary
Define priorities
Ask questions
Dive deep
Call to consultants
53
![Page 54: Picking a Message Queue.](https://reader031.fdocuments.in/reader031/viewer/2022021406/62097be6671ceb418c3c90ef/html5/thumbnails/54.jpg)
54