Evented applications with RabbitMQ and CakePHP
-
Upload
markstory -
Category
Technology
-
view
7.633 -
download
1
description
Transcript of Evented applications with RabbitMQ and CakePHP
Evented Applications
with RabbitMQ , Python, and CakePHP
What the wibblefish?
All about offloading work, and doing things in ‘realtime’
Make applications that complete tasks in a distributed/scalable fashion.
Doing work outside the request/response cycle of a UI/API.
Work to offload
Noti"cations (SMS, Email, Postal mail).
Logging.
Import & export.
Process image & video.
Reporting.
Webhooks.
Isn’t cron good enough?
Cron tabs are great with one server.
Cron doesn’t scale to multiple machines.
Cron doesn’t solve the whole problem space.
Problems with Cron
Isn’t suitable for ‘real-time’ results.
Requires a set schedule. You might spin up processes that do nothing. Or you might not have enough capacity.
Parallelizing is a pain.
Queues to the rescue!
Queues accept and dispatch events.
Events can contain almost anything*.
Events can be roughly equal to ‘jobs’.
A single event can trigger lots of downstream work.
Architectural benefits
Separation of concerns.
Collection of mini/small applications.
No monolithic applications.
Decoupled, and easier to test.
Easy to scale. Busy service = more of them.
Queue technologies
RabbitMQ
ZeroMQ
SQS
Celery
Gearman
RabbitMQ
Written in Erlang.
Open source and Free.
Really Fast.
Socket protocol that is has cheap connections.
Data persists after a crash.
Queue basics
Producers create events.
Consumers eat messages, and do work.
messages == events.
Queue Basics
Producer
Queue Basics
Producer
Queue Basics
Producer Consumer
Queue Basics
Producer
Consumer
Consumer
Queue Basics
Producer
Consumer
Consumer
Consumer
Exchanges
Exchanges abstract queues from the producer.
Route messages to queues, that are bound by routing keys.
Allows consumers to listen to speci"c routing keys.
Multiple consumers can listen to the same events.
Exchanges
Producer
Exchanges
Producer
Exchanges
Producer
Consumer
page_view
Exchanges
ProducerConsumerimport
Consumer
page_view
Exchanges
Producer
Consumererror,
page_view
Consumerimport
Consumer
page_view
Topic subscription
Consumers can also listen to topics. Which are keys separated by dots.
e.g page_view.search, page_view.update
Consumer could listen to page_view.*
Producing
Can create messages with many tools.
PHP, python, node, ruby, etc.
Messages can contain any stringy data (JSON)
Data for messages
Only string-y things can be stored in messages.
JSON encoded data in message.
Key to value stored in Redis/Memcache.
Keep messages small. RabbitMQ performs better with small messages.
Consuming
Use a daemon process.
Daemons in PHP can be done, but they kinda suck.
Usually use sleep(2) or other workaround.
sleep(2) moves you away from ‘real time’.
Aw bugger..
Use a better tool
Other tools are much better for making daemons.
Python (pika, kombu, sparkplug)
NodeJS (node-amqp)
Ruby (carrot, ruby-amqp)
Managing mixed applications
Many ways to get it done.
Avoid duplicating code.
Thin daemons + worker processes.
Thin daemons
Python/node daemon accepts messages.
Delegate to PHP worker processes.
Keeps PHP’s fork + die work#ow.
Re-use application code by calling shells.
Thin daemons
PHP
Thin daemons
PHP
Thin daemons
PHP
Py
Thin daemons
PHP Py
Py
Thin daemons
PHP Py
Py
Py
Thin daemons
PHP Py
Py
Py PHP
Thin daemons
PHP Py PHP
Py
Py PHP
Thin daemons
PHP Py PHP
Py PHP
Py PHP
Demo Time
Overview
Use CakeLog to "re events.
Use CakePHP shell’s for actual work.
Thin python daemon for processing messages.
Python + sparkplug
Python module for creating queue listeners.
Created by a developer at FreshBooks.
Ini con"guration "le de"nes behaviour.
What we’ll try:
User events logged during application.
Log events to the database.
Insert code
Limitations of using shells
Daemon and shell processes have to be on the same box.
This can be limiting as you scale, depends on how you structure applications.
Solutions
Use HTTP.
Have thin daemons dispatch back to the front-end application using HTTP.
Use Basic/Digest authentication.
Use checksums.
HTTP daemons
PHP
HTTP daemons
PHP
HTTP daemons
PHP Py
HTTP daemons
PHP Py
Web
HTTP daemons
PHP Py Web
Web
HTTP daemons
PHP Py Web
Web
Web
Thanks; Questions?