One System to Rule Them All

download One System to Rule Them All

of 44

Transcript of One System to Rule Them All

  • 8/9/2019 One System to Rule Them All

    1/44

    one system to rulethem all

  • 8/9/2019 One System to Rule Them All

    2/44

  • 8/9/2019 One System to Rule Them All

    3/44

    the challenge

  • 8/9/2019 One System to Rule Them All

    4/44

    write a backend that scales well

    handle various frontends (web, iPhone, ...)

    runs somewhere in the cloud

    easy to maintain and extend

  • 8/9/2019 One System to Rule Them All

    5/44

    the solution

  • 8/9/2019 One System to Rule Them All

    6/44

    RabbitMQ

    +CouchDB=Awesome

  • 8/9/2019 One System to Rule Them All

    7/44

    Part I

  • 8/9/2019 One System to Rule Them All

    8/44

    AMQP based Message Queue

    lots of additional transports (XMPP, HTTP,STOMP, ...)

    erlang based

    RabbitMQ

  • 8/9/2019 One System to Rule Them All

    9/44

    RabbitMQ

    daemon

    daemon

    daemon daemon

    frontend frontend frontend

  • 8/9/2019 One System to Rule Them All

    10/44

    every functionality gets a backed daemon

    every daemon has a queue associated

    communication via a JSON message thatgets passed around

  • 8/9/2019 One System to Rule Them All

    11/44

    the JSON message

  • 8/9/2019 One System to Rule Them All

    12/44

  • 8/9/2019 One System to Rule Them All

    13/44

    {"response": {

    "expiry": 710,"ts": "1234425918","name": "Lenz Gschwendtner","last4": 0,"cc_type": "master"

    },"data": {

    "options": {"billing_id": "E16003D4-B312-BEA23D9AB789"

    },"command": "billing_info"

    },"meta": {

    "lang": "en","reply_to": "B312-E16003D4-BEA23D9AB789","platform": "iwmn"

    }}

  • 8/9/2019 One System to Rule Them All

    14/44

    RPC style communication

    if the flow through the system is static

    fast transactions

    works great for ...

  • 8/9/2019 One System to Rule Them All

    15/44

    gets fiddly when ...

    there are asynchronous callbacks from

    outside

    message flow can change based on variousfactors

    tasks can take days to complete

  • 8/9/2019 One System to Rule Them All

    16/44

    Part II

  • 8/9/2019 One System to Rule Them All

    17/44

    workflows

  • 8/9/2019 One System to Rule Them All

    18/44

  • 8/9/2019 One System to Rule Them All

    19/44

    ... again ...

  • 8/9/2019 One System to Rule Them All

    20/44

    various frontends

    backend daemons

    JSON messages

    what we have ...

  • 8/9/2019 One System to Rule Them All

    21/44

    what we need

    a way to describe message flow

    a easy flexible way to write and update it

  • 8/9/2019 One System to Rule Them All

    22/44

    CouchDB

  • 8/9/2019 One System to Rule Them All

    23/44

    ... to the rescue

  • 8/9/2019 One System to Rule Them All

    24/44

  • 8/9/2019 One System to Rule Them All

    25/44

    workflow

    really only a path definition throughbackends

    some sort of condition management

    a backend definition

  • 8/9/2019 One System to Rule Them All

    26/44

    {"_id": "f7a4408898e5...a05b4181045","_rev": "3-633060011","name": "billing_info","type": "workflow",

    "user": "iwmn","path": [

    "logger", "billing",

    "billing_iwmn_cc"]}

  • 8/9/2019 One System to Rule Them All

    27/44

    ... and the backenddefinition

  • 8/9/2019 One System to Rule Them All

    28/44

    {"_id": "33adf2e3efc3...f423929f057ac","_rev": "3-3507820969","conn": "rpc",

    "type": "worker","name": "billing_iwmn_cc","location": "billing.cc","description": "the CC interface",

    "command": "billing_verify_cc"}

  • 8/9/2019 One System to Rule Them All

    29/44

    now we need aworkflow daemon

  • 8/9/2019 One System to Rule Them All

    30/44

    RabbitMQ

    daemon

    daemon

    daemon daemon

    frontend frontend frontend

    workflow

  • 8/9/2019 One System to Rule Them All

    31/44

    where are we now?

    we can call backends as before

    we can call workflows via the workflowqueue

    we can manage workflows in CouchDB

  • 8/9/2019 One System to Rule Them All

    32/44

    Part III

  • 8/9/2019 One System to Rule Them All

    33/44

    asynchronous callbacks

  • 8/9/2019 One System to Rule Them All

    34/44

  • 8/9/2019 One System to Rule Them All

    35/44

    problem

    we start a workflow

    we call some external API

    the external API somehow notifies us(Mail, Callback URL, ...) about progress

    we need to continue in the workflow

  • 8/9/2019 One System to Rule Them All

    36/44

    we need a persistentmessage store

    {" id" "22 d37 fd06 82129 db665d3150f570"

  • 8/9/2019 One System to Rule Them All

    37/44

    "_id": "22cd37afd06c82129adb665d3150f570","_rev": "3-2669449653","status": "done","last_update": 1252296819,"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789","message": {

    "response": {

    "expiry": 710,"name": "Lenz Gschwendtner","last4": 0,"cc_type": "master"

    },"data": {

    "options": {"billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789"

    },

    "command": "billing_info"},"meta": {

    "lang": "de","name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789","orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB","reply_to": "workflow","user": "iwmn",

    "log": [ "logger", "billing", "billing.iwmn.info"

    ],"platform": "iwmn"

    }},"created": 1252296815,

    "type": "bill"}

  • 8/9/2019 One System to Rule Them All

    38/44

    one more problemhow do we correlate messages

  • 8/9/2019 One System to Rule Them All

    39/44

    ... with some CouchDBawesome ...

  • 8/9/2019 One System to Rule Them All

    40/44

    the backend daemon for the callback polls aview in CouchDB

    we can search for any value in the requestwe sent

    all we need is a view that searches for the

    very value this external API uses ascorrelation ID

  • 8/9/2019 One System to Rule Them All

    41/44

    RabbitMQ

    daemon

    daemon

    daemon daemon

    frontend frontend frontend

    workflow

    external

    callback

  • 8/9/2019 One System to Rule Them All

    42/44

    Lenz Gschwendtner

    CTO iWantMyName

    @norbu09

    me

  • 8/9/2019 One System to Rule Them All

    43/44

    credits

    RabbitMQ

    CouchDB

    Erlang and Perl

  • 8/9/2019 One System to Rule Them All

    44/44

    more credits

    http://www.flickr.com/photos/djpd

    http://www.flickr.com/photos/lilyapp

    http://www.flickr.com/photos/b-tal

    http://www.flickr.com/photos/mikebaird

    http://www.flickr.com/photos/eyetwist

    http://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/mikebairdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpdhttp://www.flickr.com/photos/djpd

One ring to rule them all telephone fundraising

One ring to rule them all telephone fundraising

One language to rule them all: TypeScript - SDD Conferencesddconf.com/brands/sdd/library/One_language_to_rule_them_all_TypeScript.pdf · One language to rule them all: TypeScript

One language to rule them all: TypeScript - SDD Conferencesddconf.com/brands/sdd/library/One_language_to_rule_them_all_TypeScript.pdf · One language to rule them all: TypeScript

One Framework to Rule Them All

One Framework to Rule Them All

Pillar: One Format to Rule them All

Pillar: One Format to Rule them All

Aegir One Drupal to Rule Them All

Aegir One Drupal to Rule Them All

CMIS: One ECM API to Rule Them All

CMIS: One ECM API to Rule Them All

One Phone to Rule Them All

One Phone to Rule Them All

LibCT: One Lib to Rule Them All

LibCT: One Lib to Rule Them All

One site to rule them all

One site to rule them all

Delafloxacin: The One Fluoroquinolone to Rule Them Allsites.utexas.edu/pharmacotherapy-rounds/files/2017/...Stauffer 1 . Delafloxacin: The One Fluoroquinolone to Rule Them All . The

Delafloxacin: The One Fluoroquinolone to Rule Them Allsites.utexas.edu/pharmacotherapy-rounds/files/2017/...Stauffer 1 . Delafloxacin: The One Fluoroquinolone to Rule Them All . The

XAML: One Language to Rule Them All

XAML: One Language to Rule Them All

.Imaging Radar - .The One Sensor To Rule Them All.autonomoustechtlv.com/Portals/90/Amit.pdf1 .Imaging Radar - .The One Sensor To Rule Them All. .Imaging Radar - .The One Sensor To

.Imaging Radar - .The One Sensor To Rule Them All.autonomoustechtlv.com/Portals/90/Amit.pdf1 .Imaging Radar - .The One Sensor To Rule Them All. .Imaging Radar - .The One Sensor To

One ring to rule them all CIDOC CRM

One ring to rule them all CIDOC CRM

Marketing Automation : One ring to rule them all

Marketing Automation : One ring to rule them all

One Temple to Rule them All - RTS

One Temple to Rule them All - RTS

One Newsfeed to Rule Them All

One Newsfeed to Rule Them All

One to rule them all

One to rule them all

Parrot -- "one bytecode to rule them all"

Parrot -- "one bytecode to rule them all"

One scrum to rule them all

One scrum to rule them all

One Ring to rule them all, One Ring to find them, One Ring ...

One Ring to rule them all, One Ring to find them, One Ring ...