Do More With Message Queue

Leong Hean Hong (mrleong.net)2nd Aug, 2015

Objective

Optimizing web application using message queue

Agenda

● Problem● Solution● Message Queue● Queue Systems● Protocols● Benchmark● Choosing Queue System● Example

Problem

● An author publishes an article on your website

● The website needs to ○ send push notifications to 2000 followers○ send email notification to 2000 followers○ optimise images in the article○ announce it on various (social) media

When user clicked ‘Submit’ button, how long does she have to wait?

Solution

● Show a publish success message to user immediately

● Schedule the tasks to be performed later● Perform multiple tasks concurrently

Queue your tasks

Message Queue

Producer

● Request for tasks to be performed● Tasks may not be performed immediately

upon request

Message

● Contains information about a task● Information must be understood by task

processor (consumer)

Message Broker

● Manage multiple queues● Receive messages and store in queue(s)● Allow consumer to retrieve messages

Consumer

● Retrieve message from message broker● Process the task● Producer and consumer might be the same

application

● RabbitMQ● ActiveMQ (most popular)● SQS (Amazon Simple Queue Service)● ZeroMQ (decentralized, blazing fast)● HornetQ● Apollo● QPID

Reference: http://queues.io/

Queue Systems

Protocols

● AMQP○ Advanced Message Queuing Protocol○ Most popular

● STOMP○ Streaming Text Oriented Messaging Protocol

Benchmark

Reference: http://bit.ly/1MsOu70

Choosing Queue System

● Persistent / non-persistent● Setup/integration effort● (Community) support● Licensing cost● Maintenance effort/cost● Performance

Example

Custom made message forwarder: https://github.com/hongster/message-forwarder

Example

1. Web app sends a message to RabbitMQ server.a. Message contains worker’s URL in custom AMQP

headerb. Message content might be ID of posted article

2. We developed an app (Message Forwarder) that constantly a. get message from RabbitMQ serverb. forwards message to worker using HTTP POSTc. multiple messages are sent concurrently

Example

● Worker is another web app that do work upon receiving message.○ Sometime it is the the original sender web app itself

● Worker might be a webservice, and URL is the API endpoint