DisbatchDistributed batch processing: open-source release

Thursday, June 9, 2011

Distributed Batch Processing

Wanted the best way to manage distributed, long-running tasks

Needed excellent and centralised traceability, logging & visibility

Thursday, June 9, 2011

About us

Matt Busigin - Sr Email Engineer

( a n d o c c a s i o n a l syndicated author on economics)

Matt Berg - Lead Email Engineer, promised to be h e r e , b u t i n s t e a d p r e s e n t l y e n j o y i n g copious vendor-supplied alcohol at MAAWG in SF.

Thursday, June 9, 2011

The ProblemsMass customer migrations

Spamming Sending millions of legitimate pieces of mail

DDOS Load testing

Thursday, June 9, 2011

Requirements

Dependent/chained tasks

Visibility

Distributed & elastic

Platform agnostic

Able to handle large queues

Thursday, June 9, 2011

Features

Durable persistence (using MongoDB as the core)

Multiple active queues

Multiple queue types via plugin architecture

Thursday, June 9, 2011

ArchitectureIt’s all just JSON or BSON!

MongoDB

.NET Disbatch Execution Node

Ruby Disbatch Execution Node

Perl Disbatch Execution Node

Disbatch Command Node

Thursday, June 9, 2011

Per-node ArchitectureConcurrency implementation up to the Disbatch Execution Node implementor

Disbatch Execution Node MongoDB

Queue

Queue

Queue

Thread

Thread

Thread

Thread

Thread

Thursday, June 9, 2011

Disbatch ScreenshotsThe main screen (queues)

Thursday, June 9, 2011

Disbatch ScreenshotsTasks inside of a queue

Thursday, June 9, 2011

Disbatch ScreenshotsTask visibility

Thursday, June 9, 2011

Disbatch ScreenshotsCreating a task from a Mongo Query

Thursday, June 9, 2011

Other cool stuff

We can queue up millions (effectively unlimited) tasks from a Mongo query cursor

Full web interface based on Qooxdoo

Dynamically reload plugin classes (thanks Module::Reload::Selective)

Thursday, June 9, 2011

Other possible uses

Parallel processing of classical maths problems

Fractal generation

Factoring large prime numbers

Any batch/bulk business operation with many different individual task parameters based on a pattern

Thursday, June 9, 2011