The Art ofMessage QueuesConFoo.ca 2010

Mike WillbanksSr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.comTwitter: mwillbanks IRC: lubs on freenode Talk: http://joind.in/1375

Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging the passing of control or of content.

http://en.wikipedia.org/wiki/Message_queue

Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.

http://www.rabbitmq.com/faq.html#what-is-messaging

A Basic Message Queue

What are Message Queues

An application framework for sending and receiving messages.Message Oriented Middleware aka Enterprise Service Bus

A way to communicate between applications / systems.

A way to decouple components.

A way to offload work.

Why are Message Queues Useful?

Asynchronous Processing

Communication between Applications / Systems

Chat

Image Resizing

Video Processing

Sending out Emails

Auto-Scaling Virtual Instances

Log Analysis

You Might Use a Message Queue If...

The request can be fulfilled in the background.

You need to communicate between multiple applications / systems.

You have a limited number of processing slots.

You need to support legacy applications or use multiple programming languages.

Message Queue Servers

Picking a Message Queue Server

Durability (Memory, Disk, DBMS)

Scalability (Horizontal / Vertical)

Synchronous / Asynchronous

Batching Policies

Delivery Policies

Purging Policies

Routing Policies

Security Policies

Standards

http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

Standard Protocols

AMQP Advanced Message Queue Protocol

STOMP Streaming Text Oriented Messaging Protocol

XMPP - Extensible Messaging and Presence Protocol

What I'll Be Using

RabbitMQ AMQP based

ActiveMQ Stomp Transport

Beanstalkd Non-standard based

Installing Servers

RabbitMQ AMQPDownload Server: http://www.rabbitmq.com/server.html

Experimental Client: php-rabbitRequires rabbitmq-c and rabbitmq-codegen

http://code.google.com/p/php-rabbit/

ActiveMQDownload binary: http://activemq.apache.org/activemq-530-release.html

Client: pecl install stomp

BeanstalkdDownload from: http://kr.github.com/beanstalkd/

Client from: http://github.com/pda/pheanstalk/

Some Code

A Simple Publish with AMQP

A Simple Receive with AMQP

A Simple Publish with Stomp

A Simple Receive with Stomp

A Simple Publish with Beanstalk

A Simple Receive with Beanstalk

How to UseYour Queues

Message Formatting

Anything you want it to be!XML

JSON

Remember the purpose is to decouple and application agnostic.

Workers / Consumers to 1 Task

Workers / Consumers should be dumb, they should not need to do more than a single task.If they do, write another message to the queue for a separate worker to do the work.Exception: Legacy Code

A Simple Architecture

Client / PublisherMessage QueueServerWorker / Consumer

Queue

Receive

A Complex Architecture

ClientMessage QueueServerWorker

WorkerWorkerWorkerWorkerMessage QueueServerMessage QueueServerClientClient

Things to Keep in Mind

Consumers and Servers should be monitoredSupervisord (http://supervisord.org/) is excellent at this.

Memory Usage of Consumers (in the event of leaks, kill and let supervisord re-create it)

Persistence of DataMany have multiple backends or built-in backends.

High AvailabilityNot all clients have built in support for multiple servers, you may need to abstract it.

Message queues do not handle scheduled processing (aka wait 4 days to process the message).

Use Cases

Email Notifications

A user takes an action on a page that needs to notify all subscribing members.

Send a message to the queue regarding the action, then queue emails to send in another queue.Why 2 queuesAllows us to handle getting everyone that needs the notification.

Then allows us to focus on sending email through a single queue with multiple workers.

Auto-Scaling

A scheduled task polls the servers looking for potential load issues. If load exceeds 80% or 80% of web server processes are taken send a message to the queue to boot up a new instance.

The worker or consumer then utilizes a service call or logic to boot up a new instance.

Video / Photos

Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.

When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.

Remember to monitor how many jobs are getting backed up :)

Questions?

Mike Willbanks Blog: http://blog.digitalstruct.comTwitter: mwillbanks IRC: lubs on freenode Talk: http://joind.in/1375