Consuming RabbitMQ at TTL
-
Upload
jeanml -
Category
Engineering
-
view
1.474 -
download
2
Transcript of Consuming RabbitMQ at TTL
Consuming Rabbit MQ
JM LaplanteDeveloper @ Trainline.com
What can you use it for?
Pub-Sub
Publisher 1
Subscriber 1
Publisher X
Publisher 2
...
Subscriber 2
Subscriber Y
...
Event A
Event B
Event C
Event A
Event B
Event B
Remote ProcedureCall
Provider
Requester
ResponseCorId: 123
RequestCorId: 123
RequestCorId: 123
ResponseCorId: 123
AsyncJobProcessing
Worker 1
Requester
Worker N...
Job Request 1
...
Job Request N
Job Request 1 Job Request N
Warnings
•RabbitMQ is happier when queues are empty
•RabbitMQ ≠ Data Store
•Connections are precious
•Shared infrastructure*• Poor design decisions have an impact on other components
What are routing topologies?
Publisher
Consumer
PublishingExchange
ConsumingQueue
Routing Topology (Simplest)
Publisher
Consumer
ConsumingQueue
PublishingExchange
Fanout
Routing Topology (Complex)
Publisher
Consumer
ConsumingQueue
PublishingExchange
Fanout
Exchange A Exchange B Exchange C
Subscription Exchange ?
Remote Location
Local Broker B
Remote Consumer
OtherConsumer
Federated FederatedRouting Key
Routing Topology
Proposed approachfor Pub-Sub
Publisher
Consumer
ConsumingQueue
PublishingExchange
Fanout
Routing Exchange
Subscription Exchange
Routing Key Topic
Routing Key
Fanout
What if things fail?
Designing applications for failure•Connections can drop at all times•Nodes can die at all times
•Protect yourself:• Client library should manage connections• Talk to load balancer, not specific nodes• Publish with a circuit breaker library
Designing applications for failure•RabbitMQ delivers messages at least once
RabbitMQ delivers messages at least
once
RabbitMQ
delivers messag
es
at least ONCE
Designing applications for failure•Message processing should be idempotent
•What will happen if a message is processed twice?•No single answer• Immutability goes a long way• Be mindful of side effects
Designing applications for failure•What happens if message processing fails?• Error queues•Dead-letter exchanges
•How will you know?
•How will you replay messages?• Custom-built application• Shovel plugin
How fast is it?
Performance
•RabbitMQ can be very fast• Tens of thousands of messages per second
•Key considerations•Manage connections well•Do you really need publish confirmations?•Do you need message persistence?•Do you need high availability?•How much security should be put in place?
AMQP Transactions vs ‘Publisher Confirms’
Publisher
SynchronousAck
Publisher
AsynchronousAckSLOW FASTER
(100x)
Message Persistence
•Messages that ‘should’ survive broker restart
• Inconsistent publishing times•Disk writes not always consistent (e.g. cheap disk/AWS)
•Do you really need it?
Message Persistence
Website(Matrix)
Add Item To Shopping Basket
Message Persistence
Mobile API
Take Money from Credit Card
Message Persistence
Payment System
Payment Accepted
What about security?
Authentication
• Username/Password for
client applications
• Virtual Host segregation
Client A
ValidCredential
Client A
InvalidCredential
Default vHost
Super Secure vHost
Authorisation
Publisher
Publishing Exchange A
Publishing Exchange B
Consumer
Queue A Queue B
Encryption
•AMQP• Communication channel is not encrypted
•AMQPS• Communication channel is encrypted using SSL• Optional: Exchange signed certificates• Optional: Verify certificates to establish chain of trust
•How much is enough for your data?
How can I monitor my app?
New Relic
•Plugin developed internally•Queue monitoring (Ready/Total messages)• Consumers (Active/Total)•Memory consumption
Circuit Breaker Dashboard
•SCOM alerts
Any .NET client libraries?
Libraries
•RabbitMQ Client API•EasyNetQ (http://easynetq.com/)
• Wrapper around RabbitMQ Client API• Open Source, Well supported, MIT license• Using IAdvancedBus, not IBus*
Questions?