Evented Applications

with RabbitMQ , Python, and CakePHP

What the wibblefish?

All about offloading work, and doing things in ‘realtime’

Make applications that complete tasks in a distributed/scalable fashion.

Doing work outside the request/response cycle of a UI/API.

Work to offload

Noti"cations (SMS, Email, Postal mail).

Logging.

Import & export.

Process image & video.

Reporting.

Webhooks.

Isn’t cron good enough?

Cron tabs are great with one server.

Cron doesn’t scale to multiple machines.

Cron doesn’t solve the whole problem space.

Problems with Cron

Isn’t suitable for ‘real-time’ results.

Requires a set schedule. You might spin up processes that do nothing. Or you might not have enough capacity.

Parallelizing is a pain.

Queues to the rescue!

Queues accept and dispatch events.

Events can contain almost anything*.

Events can be roughly equal to ‘jobs’.

A single event can trigger lots of downstream work.

Architectural benefits

Separation of concerns.

Collection of mini/small applications.

No monolithic applications.

Decoupled, and easier to test.

Easy to scale. Busy service = more of them.

Queue technologies

RabbitMQ

ZeroMQ

SQS

Celery

Gearman

RabbitMQ

Written in Erlang.

Open source and Free.

Really Fast.

Socket protocol that is has cheap connections.

Data persists after a crash.

Queue basics

Producers create events.

Consumers eat messages, and do work.

messages == events.

Queue Basics

Producer

Queue Basics

Producer

Queue Basics

Producer Consumer

Queue Basics

Producer

Consumer

Consumer

Queue Basics

Producer

Consumer

Consumer

Consumer

Exchanges

Exchanges abstract queues from the producer.

Route messages to queues, that are bound by routing keys.

Allows consumers to listen to speci"c routing keys.

Multiple consumers can listen to the same events.

Exchanges

Producer

Exchanges

Producer

Exchanges

Producer

Consumer

page_view

Exchanges

ProducerConsumerimport

Consumer

page_view

Exchanges

Producer

Consumererror,

page_view

Consumerimport

Consumer

page_view

Topic subscription

Consumers can also listen to topics. Which are keys separated by dots.

e.g page_view.search, page_view.update

Consumer could listen to page_view.*

Producing

Can create messages with many tools.

PHP, python, node, ruby, etc.

Messages can contain any stringy data (JSON)

Data for messages

Only string-y things can be stored in messages.

JSON encoded data in message.

Key to value stored in Redis/Memcache.

Keep messages small. RabbitMQ performs better with small messages.

Consuming

Use a daemon process.

Daemons in PHP can be done, but they kinda suck.

Usually use sleep(2) or other workaround.

sleep(2) moves you away from ‘real time’.

Aw bugger..

Use a better tool

Other tools are much better for making daemons.

Python (pika, kombu, sparkplug)

NodeJS (node-amqp)

Ruby (carrot, ruby-amqp)

Managing mixed applications

Many ways to get it done.

Avoid duplicating code.

Thin daemons + worker processes.

Thin daemons

Python/node daemon accepts messages.

Delegate to PHP worker processes.

Keeps PHP’s fork + die work#ow.

Re-use application code by calling shells.

Thin daemons

PHP

Thin daemons

PHP

Thin daemons

PHP

Py

Thin daemons

PHP Py

Py

Thin daemons

PHP Py

Py

Py

Thin daemons

PHP Py

Py

Py PHP

Thin daemons

PHP Py PHP

Py

Py PHP

Thin daemons

PHP Py PHP

Py PHP

Py PHP

Demo Time

Overview

Use CakeLog to "re events.

Use CakePHP shell’s for actual work.

Thin python daemon for processing messages.

Python + sparkplug

Python module for creating queue listeners.

Created by a developer at FreshBooks.

Ini con"guration "le de"nes behaviour.

What we’ll try:

User events logged during application.

Log events to the database.

Insert code

Limitations of using shells

Daemon and shell processes have to be on the same box.

This can be limiting as you scale, depends on how you structure applications.

Solutions

Use HTTP.

Have thin daemons dispatch back to the front-end application using HTTP.

Use Basic/Digest authentication.

Use checksums.

HTTP daemons

PHP

HTTP daemons

PHP

HTTP daemons

PHP Py

HTTP daemons

PHP Py

Web

HTTP daemons

PHP Py Web

Web

HTTP daemons

PHP Py Web

Web

Web

Thanks; Questions?