Disbatch BarCamp Buffalo

DisbatchDistributed batch processing: open-source release

Thursday, June 9, 2011

Distributed Batch Processing

Wanted the best way to manage distributed, long-running tasks

Needed excellent and centralised traceability, logging & visibility


About us

Matt Busigin - Sr Email Engineer

( a n d o c c a s i o n a l syndicated author on economics)

Matt Berg - Lead Email Engineer, promised to be h e r e , b u t i n s t e a d p r e s e n t l y e n j o y i n g copious vendor-supplied alcohol at MAAWG in SF.


The ProblemsMass customer migrations

Spamming Sending millions of legitimate pieces of mail

DDOS Load testing


Requirements

Dependent/chained tasks

Visibility

Distributed & elastic

Platform agnostic

Able to handle large queues


Features

Durable persistence (using MongoDB as the core)

Multiple active queues

Multiple queue types via plugin architecture


ArchitectureIt’s all just JSON or BSON!

MongoDB

.NET Disbatch Execution Node

Ruby Disbatch Execution Node

Perl Disbatch Execution Node

Disbatch Command Node


Per-node ArchitectureConcurrency implementation up to the Disbatch Execution Node implementor

Disbatch Execution Node MongoDB

Queue

Queue

Queue

Thread

Thread

Thread

Thread

Thread


Disbatch ScreenshotsThe main screen (queues)


Disbatch ScreenshotsTasks inside of a queue


Disbatch ScreenshotsTask visibility


Disbatch ScreenshotsCreating a task from a Mongo Query


Other cool stuff

We can queue up millions (effectively unlimited) tasks from a Mongo query cursor

Full web interface based on Qooxdoo

Dynamically reload plugin classes (thanks Module::Reload::Selective)


Other possible uses

Parallel processing of classical maths problems

Fractal generation

Factoring large prime numbers

Any batch/bulk business operation with many different individual task parameters based on a pattern


Disbatch BarCamp Buffalo

Documents

Transcript of Disbatch BarCamp Buffalo