Post on 10-May-2015
Outline
• RabbitMQ model and standard• Simplified message queue model• 3 example code• 5 real case studies
RabbitMQ Model
directexchange
topicexchange
filterexchange
key: q1reply-to: tmp1
cor-id: 1
to: topickey: STK.US.C
to: fan-outkey:key1
STK.US.*
STK.US.* ANDtype == pdf
STK.IND.*
q1: ! private
C2
C3
C1
fan-out q3:
q4:
q5:!durable
s1
key1
(Distribution)
(Fan-out)
tmp1: private
s2
s3
s4
q6:
key1
Notations in RabbitMQ Model
TCP connection
channel
message
application thread
s4
exchange
binding
consuming
virtual host
queue
RabbitMQ Model
• Virtual Host• Exchange
– direct– fan-out– topic
• Binding– topic– cascading– message select
• queue– flags: private, durable
• Connection– channel: every thread work
with one channel
• Message– content header
• Properties: Reply-To, Cor-Id, Message-Id, Key
• queue server may add properties , wont remove/ modify
– content body (wont modify)• binary/ file/ stream
• Application– client/ server
Other Items in AMQP 0.8, not included in Model
– AMQP transaction – deliver ACK to sender after message received
– AMQP distributed transaction – deliver ACK to a 3rd party software after message received
– Access – a realm based access control model
we think these are irrelevant to ADDM transaction detection
Simplified Message Queue Model
reply-to: tmp1cor-id: (2)
C1s1
(Distribution)s3
s4cor-id: 2
send rpc:q1.1
send update:q34
queue server
rpc q1.*
q2.*
updateq34
rpc-result
tmp1
s2
C2
s5
rpc q5
q5s6
(Fan out)
Model Notation
C1
application thread
$properties
($destination: key)
message
queue server
queue
binding
consuming
exchange$name
$key
3 Example Code
1. RabbitMQ tutorial code (synchronous)2. RabbitMQ RPC (synchronous)3. Celery framework (distributed task queue /
asynchronous)
– one request queue per service; one reply queue per client – properties used on message
• correlate-id • message-id• reply-to
– steps1. client thread send REQUEST and wait on reply queue2. server thread receive REQUEST, process it and send REPLY to reply queue3. client thread get reply from reply queue4. if asynchronous mode, client may polls processing status by another status
update REQUEST message, and server reply another status update REPLY message
Classical Scenarioin Example Code
Example Code 1: RabbitMQ tutorial
reply-to: (tmp1)
C1
tmp1 q1
s1
C1
tmp1 q1
s1
1
C1
tmp1 q1reply-to: (tmp1)
s1
2
3 4
C1
tmp1 q1
s1
(q1.1) (q1)
(tmp1) (tmp1)
• coordinate REQUEST-REPLY into transaction by reply queue id
Example Code 2:RabbitMQ RPC
C1
tmp1 rpcq1
s1
C1
tmp1 rpcq1
s1
1
C1
tmp1 q1reply-to: (tmp1)cor-id: (1)
s1
2
3 4
C1
tmp1 rpcq1cor-id: (1)
s1
(rpcq1) (rpcq1)
(tmp1) (tmp1)
reply-to: (tmp1)cor-id: (1)
cor-id: (1)
• Coordinate REQUEST-REPLY into transaction by correlate-id
Example Code 3:Celery (Distributed Task Queue)
add (1,1
)cor-id: (1)
C1
celeryresult:1
celery:celery
s1 C1 s1
step 1 step 2
(celery:celery)
C1
step a
( )*step 3: polling status of result queue many times:
celeryresult:1
celery:celery
add (1,1
)cor-id: (1)
(celery:celery)
poll
(celery-result:1)C1
empty
(celery-result:1)
step b
C1 worker
step 4 step 5
C1
cor-id: (1)
worker
(celery-result:1) (celery-result:1)
cor-id: (1)
celery-result:1
celery:celery
celery-result:1
celery:celery
• Coordinate REQUEST-REPLY into transaction by (correlate-id)
Example Code 3:Celery (Distributed Task Queue)
1. client make two calls via the same connection
C1service
1+1
cor-id: (1)
2+2
cor-id: (2)
2. worker send REPLY via its own connection3. client receive REPLYs via the same connection
C1
w1
w2
cor-id: (1)
2
cor-id: (2)
4
w1
w1
worker thread pool
Celery: one client application with two worker
5 Real Cases Study
• Applications use RabbitMQ1. Zenoss4‘s message bus between daemons (non
transaction app)2. Nova controller’s RPC call between hosts (transaction
app)
• Enterprise use RabbitMQ3. Instagram : Celery + RabbitMQ4. MercadoLibre: RabbitMQ5. Indeed.comMailboxApp
Case 1: Zenoss 4• A message bus to pass event and heartbeat between daemons
– RabbitMQ as message queue server– Protobufs to serialization
• Non-transaction application
RabbitMQ
zenossevent server
zeventd
heartbeat
event
rawevent
zenhub
zenhubworker
Heartbeat
zenhubworker.py---(1)-->[RabbitMQ] publish exchange: zenoss.heartbeats key: zenoss.heartbeat.localhostzenhubworker.py---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.DaemonHeartbeatzenhubworker.py---(1)-->[RabbitMQ] content with len: 33[RabbitMQ]---(3)-->java deliver[RabbitMQ]---(3)-->java content header: org.zenoss.protobufs.zep.DaemonHeartbeat[RabbitMQ]---(3)-->java content with len: 33
message bus
zenhub
zeneventd
zenhubworker
zenossevent server
HB
heartbeat
HB
HB
HB
Ping Failure
RabbitMQ
zenossevent server
event
rawevent
zenping
rawping
event1ping
event
zeventd2
Ping Failure Messages
zenping---(1)-->[RabbitMQ] publish exchange: zenoss.zenevents.raw key: zenoss.zenevent.status.pingzenping---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.Eventzenping---(1)-->[RabbitMQ] content with len: 235[RabbitMQ]---(1)-->zeneventd.py deliver[RabbitMQ]---(1)-->zeneventd.py content header: org.zenoss.protobufs.zep.Event[RabbitMQ]---(1)-->zeneventd.py content with len: 235zeneventd.py---(1)-->[RabbitMQ] publish exchange: zenoss.zenevents.zep key: zenoss.zenevent.status.pingzeneventd.py---(1)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.ZepRawEventzeneventd.py---(1)-->[RabbitMQ] content with len: 432[RabbitMQ]---(1)-->java deliver[RabbitMQ]---(1)-->java content header: org.zenoss.protobufs.zep.ZepRawEvent[RabbitMQ]---(1)-->java content with len: 432java---(6)-->[RabbitMQ] publish exchange: zenoss.zenevents key: zenoss.zenevent.status.pingjava---(6)-->[RabbitMQ] content header: org.zenoss.protobufs.zep.EventSummaryjava---(6)-->[RabbitMQ] content with len: 433
Case 2: Nova Controller• A message queue server for RPC call between hosts• Each RPC call has its own unique message-id generated by UUID library
Case 3: Instagram• Web server uses nginx• Messaging system use
RabbitMQ– Scalability: self-made message
broker, dispatch message in RR– Reliability: RabbitMQ cluster
• Worker system use Celery– Celery is distributed task queue– worker concurrency model
• slow task: pre-fork pool• fast task: gevent
• Overall throughput– 25k application threads
publishing task– 4K tasks per second– Also monitor queue length
http://lanyrd.com/2013/pycon/scdyrp/
Nginx Server with Celery
Nginx• single thread• event driven• asynchronous (non-
blocking)
nginx
queue server
cor-id: (c1’s request)
1
w1
cor-id: (c2’s request)
2
w2
client1
client 10k
client 10k
client 10k
• Task complete and return REPLY messages
Case 4: MercadoLibre
• MercadoLibre is 8th largest ecommerce company in Latin American
• original architecture of their ecommerce system:
nodeJS
Grails
Groovy
7000+ vm insance (by openstack)
nginxproxyto routerequests
requests
20 million request per minutes, 4GB per seconds
Redis
MongoDb
requests
requests
• Changes from LB + web app to enterprise service bus architecture
• Benefit– error handling– routing – scalability– event driven for higher
performance
Case 4: MercadoLibre
enterprise service bus
app1
app2
app3
req
persist
webserver
itemfeed
req
MercadoLibre’sNews Update System
• The items-app is not transaction based
MercadoLibre’sSelectively Receive Message
• The items-app is not transaction based
Case 5: indeed.COM
crawler engine
RabbitMQ
crawler engine
RabbitMQ
crawler engine
RabbitMQ
RabbitMQ
Job writeservice
Job writeservice
Job writeservice
Asian DC US DCpacificocean
RabbitMQHA
DB
DB
DB
• crawler engine app is not transaction based
Reference
• RabbitMQ specification 0.8• RabbitMQ Tutorial, example 1/6 and its packet dump• Celery: Distributed Task Queue and its packet dump• Messaging at Scale at Instagram• How is nginx different from apache• Zenoss source code• nova controller source code