Plone, RabbitMQ and messaging that just worksAsko Soukka & Jukka Ojaniemi

Speakers' backgroundUniversity of Jyväskyläthe largest Plone user in Finland● 83 Plone sites● 2.5 TB data served through Plone● 700 content editors

Demo (serializing writes)

RabbitMQ

QueueQueueBrowser B

Browser A

PloneM1M1

M1

M1

Why message queues?

Why message queues?We needed a reliable, effective and scalable solution for:

● running asynchronous tasks● being a common integration point

between services (decoupling).

Why message queues?

”None of the existing ZODB-based mechanism really scales. When you are really in need for a real and working queuing mechanism one would use some external queuing solution (e.g. something AMQP compatible like RabbitMQ).”

– Andreas Jung

Advanced Message Queue Protocol (AMQP)

Think of it as an highly advancedpost office which can delivermessage to multiple recipientsbased on address.

AMQP jargon 1/2

● Message consists of label and payload.

● Producer is a program which sends messages to exchange.

● Consumer is a program which mostly waits to receive messages.

AMQP jargon 2/2

● Exchange receives the messages from producers and pushes them to queues.

● Bindings are how the messages get routed from the exchange to a queue.

● Queue is a buffer that stores messages.

Message broker

Basic messaging

Producer

Consumer

Exchange(direct) Queue

Work queues

Consumer

M1M2

M1 M1M2

M1

M2

M2

Basic messaging

Producer

Message broker

Exchange(fanout)

Queue

Publish/Subscribe queues

Queue

Consumer

Consumer

M1M1

M1

M1

M1

M1

Basic messaging

Client/producer

Message broker

RPC queue

RPC implementation

Exchange

Server

M1 M1Exchange M1

Reply_to queue

R1R1

M1

R1

R1

RabbitMQ● Implements AMQP (Advanced Message

Queue Protocol) open standard

● Written in Erlang○ made for messaging○ easy to cluster○ fast (enough)

● Industry tested and reliable

Sounds complicated?

Example withcollective.zamqp

Site A

Message broker

Exchange(fanout)

Queue

Publish and subscribe announcements

Queue

Site B

Site C

M1M1

M1

M1

M1

M1

Produce messages withcollective.zamqpfrom zope.component import getUtility

from collective.zamqp.interfaces import IProducer

producer = getUtility( IProducer, name="announcer")

producer.register() # for transaction

producer.publish("Hello World!")

Handle messages withcollective.zamqpfrom five import grok

from collective.zamqp.interfaces import IMessageArrivedEvent

@grok.subscribe(IAnnouncement,

IMessageArrivedEvent)

def handleMessage(message, event):

logger.info(message.body)

message.ack()

Overview ofcollective.zamqp

Producers

Message broker

AMQP Broker Connections

Message handlers(as event subscribers)

Consuming Servers

IMessageArrivedEvent

produce messages

consume messages

Consumers

stamp messages

Connections incollective.zamqpbuildout.cfg:

[instance]

zope-conf-additional =

%import collective.zamqp

<amqp-broker-connection>

connection_id example

...

</amqp-broker-connection>

Producers incollective.zamqpclass Announcer(Producer):

grok.name("announcements")

connection_id = "example"

exchange = "announcements"

exchange_type = "fanout"

serializer = "text/plain"

durable = False

Consuming servers incollective.zamqpbuildout.cfg:

zope-conf-additional =

%import collective.zamqp

....

<amqp-consuming-server>

connection_id example

site_id Plone

</amqp-consuming-server>

Consumers incollective.zamqpclass Announcements(Consumer):

grok.name("announcements")

connection_id = "example"

exchange = "announcements"

exchange_type = "fanout"

queue = "" # anonymous temp. queue

durable = False

marker = IAnnouncement

Message handlers incollective.zamqp

@grok.subscribe(IAnnouncement, IMessageArrivedEvent)

def handleMessage(message, event):

logger.info(message.body)

message.ack()

Overview ofcollective.zamqp

Producers

Message broker

AMQP Broker Connections

Message handlers(as event subscribers)

Consuming Servers

IMessageArrivedEvent

produce messages

consume messages

Consumers

stamp messages

It really is!

It's so simple!

Real world exampleMoniviestin video publishing service● Plone 4.1 site● Distributed encoder servers (48 CPUs

combined)● First version used XML-RPC based

communication - lots of error handling code.● New version uses AMQP-messaging. ● Very fast and scalable. Recovers from

network outages.

Mediainfo

Plone

CherryPyuploadservice

memcached

FS

RabbitMQ

Encoder

Thank youc.zamqp and examples are available at:● https://github.com/datakurre/collective.zamqp● https://github.com/datakurre/collective.zamqpdemos● https://github.com/datakurre/chatbehavior● https://github.com/datakurre/pubsubannouncementsother relevant links:● https://www.rabbitmq.com/getstarted.html● https://www.amqp.org/about/examples● http://www.manning.com/videla/ (”RabbitMQ in Action”)