RabbitMQ Model and Some Example Applications

houcheng@gmail.comCCMA ITRI

Outline

• RabbitMQ model and standard• Simplified message queue model• 3 example code• 5 real case studies

RabbitMQ Model

directexchange

topicexchange

filterexchange

key: q1reply-to: tmp1

cor-id: 1

to: topickey: STK.US.C

to: fan-outkey:key1

STK.US.*

STK.US.* ANDtype == pdf

STK.IND.*

q1: ! private

C2

C3

C1

fan-out q3:

q4:

q5:!durable

s1

key1

(Distribution)

(Fan-out)

tmp1: private

s2

s3

s4

q6:

key1

Notations in RabbitMQ Model

TCP connection

channel

message

application thread

s4

exchange

binding

consuming

virtual host

queue

RabbitMQ Model

• Virtual Host• Exchange

– direct– fan-out– topic

• Binding– topic– cascading– message select

• queue– flags: private, durable

• Connection– channel: every thread work

with one channel

• Message– content header

• Properties: Reply-To, Cor-Id, Message-Id, Key

• queue server may add properties , wont remove/ modify

– content body (wont modify)• binary/ file/ stream

• Application– client/ server

Other Items in AMQP 0.8, not included in Model

– AMQP transaction – deliver ACK to sender after message received

– AMQP distributed transaction – deliver ACK to a 3rd party software after message received

– Access – a realm based access control model

we think these are irrelevant to ADDM transaction detection

Simplified Message Queue Model

reply-to: tmp1cor-id: (2)

C1s1

(Distribution)s3

s4cor-id: 2

send rpc:q1.1

send update:q34

queue server

rpc q1.*

q2.*

updateq34

rpc-result

tmp1

s2

C2

s5

rpc q5

q5s6

(Fan out)

Model Notation

C1

application thread

$properties

($destination: key)

message

queue server

queue

binding

consuming

exchange$name

$key

3 Example Code

1. RabbitMQ tutorial code (synchronous)2. RabbitMQ RPC (synchronous)3. Celery framework (distributed task queue /

asynchronous)

– one request queue per service; one reply queue per client – properties used on message

• correlate-id • message-id• reply-to

– steps1. client thread send REQUEST and wait on reply queue2. server thread receive REQUEST, process it and send REPLY to reply queue3. client thread get reply from reply queue4. if asynchronous mode, client may polls processing status by another status

update REQUEST message, and server reply another status update REPLY message

Classical Scenarioin Example Code

Example Code 1: RabbitMQ tutorial

reply-to: (tmp1)

C1

tmp1 q1

s1

C1

tmp1 q1

s1

1

C1

tmp1 q1reply-to: (tmp1)

s1

2

3 4

C1

tmp1 q1

s1

(q1.1) (q1)

(tmp1) (tmp1)

• coordinate REQUEST-REPLY into transaction by reply queue id

Example Code 2:RabbitMQ RPC

C1

tmp1 rpcq1

s1

C1

tmp1 rpcq1

s1

1

C1

tmp1 q1reply-to: (tmp1)cor-id: (1)

s1

2

3 4

C1

tmp1 rpcq1cor-id: (1)

s1

(rpcq1) (rpcq1)

(tmp1) (tmp1)

reply-to: (tmp1)cor-id: (1)

cor-id: (1)

• Coordinate REQUEST-REPLY into transaction by correlate-id

Example Code 3:Celery (Distributed Task Queue)

add (1,1

)cor-id: (1)

C1

celeryresult:1

celery:celery

s1 C1 s1

step 1 step 2

(celery:celery)

C1

step a

( )*step 3: polling status of result queue many times:

celeryresult:1

celery:celery

add (1,1

)cor-id: (1)

(celery:celery)

poll

(celery-result:1)C1

empty

(celery-result:1)

step b

C1 worker

step 4 step 5

C1

cor-id: (1)

worker

(celery-result:1) (celery-result:1)

cor-id: (1)

celery-result:1

celery:celery

celery-result:1

celery:celery

• Coordinate REQUEST-REPLY into transaction by (correlate-id)

Example Code 3:Celery (Distributed Task Queue)

1. client make two calls via the same connection

C1service

1+1

cor-id: (1)

2+2

cor-id: (2)

2. worker send REPLY via its own connection3. client receive REPLYs via the same connection

C1

w1

w2

cor-id: (1)

2

cor-id: (2)

4

w1

w1

worker thread pool

Celery: one client application with two worker

5 Real Cases Study

• Applications use RabbitMQ1. Zenoss4‘s message bus between daemons (non

transaction app)2. Nova controller’s RPC call between hosts (transaction

app)

• Enterprise use RabbitMQ3. Instagram : Celery + RabbitMQ4. MercadoLibre: RabbitMQ5. Indeed.comMailboxApp

Case 1: Zenoss 4• A message bus to pass event and heartbeat between daemons

– RabbitMQ as message queue server– Protobufs to serialization

• Non-transaction application

RabbitMQ

zenossevent server

zeventd

heartbeat

event

rawevent

zenhub

zenhubworker

Heartbeat

zenhubworker.py---(1)-->[RabbitMQ] publish exchange: zenoss.heartbeats key: zenoss.heartbeat.localhostzenhubworker.py---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.DaemonHeartbeatzenhubworker.py---(1)-->[RabbitMQ] content with len: 33[RabbitMQ]---(3)-->java deliver[RabbitMQ]---(3)-->java content header: org.zenoss.protobufs.zep.DaemonHeartbeat[RabbitMQ]---(3)-->java content with len: 33

message bus

zenhub

zeneventd

zenhubworker

zenossevent server

HB

heartbeat

HB

HB

HB

Ping Failure

RabbitMQ

zenossevent server

event

rawevent

zenping

rawping

event1ping

event

zeventd2

Ping Failure Messages

zenping---(1)-->[RabbitMQ] publish exchange: zenoss.zenevents.raw key: zenoss.zenevent.status.pingzenping---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.Eventzenping---(1)-->[RabbitMQ] content with len: 235[RabbitMQ]---(1)-->zeneventd.py deliver[RabbitMQ]---(1)-->zeneventd.py content header: org.zenoss.protobufs.zep.Event[RabbitMQ]---(1)-->zeneventd.py content with len: 235zeneventd.py---(1)-->[RabbitMQ] publish exchange: zenoss.zenevents.zep key: zenoss.zenevent.status.pingzeneventd.py---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.ZepRawEventzeneventd.py---(1)-->[RabbitMQ] content with len: 432[RabbitMQ]---(1)-->java deliver[RabbitMQ]---(1)-->java content header: org.zenoss.protobufs.zep.ZepRawEvent[RabbitMQ]---(1)-->java content with len: 432java---(6)-->[RabbitMQ] publish exchange: zenoss.zenevents key: zenoss.zenevent.status.pingjava---(6)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.EventSummaryjava---(6)-->[RabbitMQ] content with len: 433

Case 2: Nova Controller• A message queue server for RPC call between hosts• Each RPC call has its own unique message-id generated by UUID library

Case 3: Instagram• Web server uses nginx• Messaging system use

RabbitMQ– Scalability: self-made message

broker, dispatch message in RR– Reliability: RabbitMQ cluster

• Worker system use Celery– Celery is distributed task queue– worker concurrency model

• slow task: pre-fork pool• fast task: gevent

• Overall throughput– 25k application threads

publishing task– 4K tasks per second– Also monitor queue length

http://lanyrd.com/2013/pycon/scdyrp/

Nginx Server with Celery

Nginx• single thread• event driven• asynchronous (non-

blocking)

nginx

queue server

cor-id: (c1’s request)

1

w1

cor-id: (c2’s request)

2

w2

client1

client 10k

client 10k

client 10k

• Task complete and return REPLY messages

Case 4: MercadoLibre

• MercadoLibre is 8th largest ecommerce company in Latin American

• original architecture of their ecommerce system:

nodeJS

Grails

Groovy

7000+ vm insance (by openstack)

nginxproxyto routerequests

requests

20 million request per minutes, 4GB per seconds

Redis

MongoDb

requests

requests

• Changes from LB + web app to enterprise service bus architecture

• Benefit– error handling– routing – scalability– event driven for higher

performance

Case 4: MercadoLibre

enterprise service bus

app1

app2

app3

req

persist

webserver

itemfeed

req

MercadoLibre’sNews Update System

• The items-app is not transaction based

MercadoLibre’sSelectively Receive Message

• The items-app is not transaction based

Case 5: indeed.COM

crawler engine

RabbitMQ

crawler engine

RabbitMQ

crawler engine

RabbitMQ

RabbitMQ

Job writeservice

Job writeservice

Job writeservice

Asian DC US DCpacificocean

RabbitMQHA

DB

DB

DB

• crawler engine app is not transaction based

Reference

• RabbitMQ specification 0.8• RabbitMQ Tutorial, example 1/6 and its packet dump• Celery: Distributed Task Queue and its packet dump• Messaging at Scale at Instagram• How is nginx different from apache• Zenoss source code• nova controller source code