one system to rule them all

the challenge

• write a backend that scales well

• handle various frontends (web, iPhone, ...)

• runs “somewhere in the cloud”

• easy to maintain and extend

the solution

RabbitMQ+CouchDB=Awesome

Part I

• AMQP based Message Queue

• lots of additional transports (XMPP, HTTP, STOMP, ...)

• erlang based

RabbitMQ

RabbitMQ

daemon

daemon

daemon daemon

frontend frontend frontend

• every functionality gets a backed daemon

• every daemon has a queue associated

• communication via a JSON message that gets passed around

the JSON message

{ "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" }}

• RPC style communication

• if the flow through the system is static

• fast transactions

works great for ...

gets fiddly when ...

• there are asynchronous callbacks from outside

• message flow can change based on various factors

• tasks can take days to complete

Part II

workflows

... again ...

• various frontends

• backend daemons

• JSON messages

what we have ...

what we need

• a way to describe message flow

• a easy flexible way to write and update it

CouchDB

... to the rescue

workflow

• really only a path definition through backends

• some sort of condition management

• a backend definition

{ "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ]}

... and the backend definition

{ "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc"}

now we need a workflow daemon

RabbitMQ

daemon

daemon

daemon daemon

frontend frontend frontend

workflow

where are we now?

• we can call backends as before

• we can call workflows via the workflow queue

• we can manage workflows in CouchDB

Part III

asynchronous callbacks

problem

• we start a workflow

• we call some external API

• the external API somehow notifies us (Mail, Callback URL, ...) about progress

• we need to continue in the workflow

we need a persistent message store

{ "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill"}

one more problemhow do we correlate messages

... with some CouchDB awesome ...

• the backend daemon for the callback polls a view in CouchDB

• we can search for any value in the request we sent

• all we need is a view that searches for the very value this external API uses as correlation ID

RabbitMQ

daemon

daemon

daemon daemon

frontend frontend frontend

workflow

externalcallback

Lenz Gschwendtner

CTO iWantMyName

@norbu09

me

credits

• RabbitMQ

• CouchDB

• Erlang and Perl

more credits

• http://www.flickr.com/photos/djpd

• http://www.flickr.com/photos/lilyapp

• http://www.flickr.com/photos/b-tal

• http://www.flickr.com/photos/mikebaird

• http://www.flickr.com/photos/eyetwist