Background Processing

Kerry BuckleyIPRUG 3 November 2010

Why?

• Sending e-mail, SMS, tweets etc

• Resizing images

• Generating large PDFs

• Updating search indexes

• etc

Approaches• Process/thread based

• Spawn

• Database-backed• BackgroundDRb, BackgroundJob,

DelayedJob, JobFu, Background-Fu

• Queue-backed• Sparrow, Starling, Kestrel, RabbitMQ,

ActiveMQ, SQS, Beanstalkd, Resque

Delayed::Job

Delayed::Job

• Extracted from Shopify

• Daemon runs with Rails environment

• Use collectiveidea’s fork

• 2.0 for Rails 2.x

• 2.1RC (from GitHub) for Rails 3

• ActiveRecord/MongoMapper/DataMapper

SetupGemfile:gem "delayed_job", :git => "git://github.com/collectiveidea/delayed_job.git"

Then run:$ rails g delayed_job

$ rake db:migrate

$ script/delayed_job start

Delaying methods

Mailer.send_alerts(event)

Delaying methods

Mailer.delay.send_alerts(event)

Options

Mailer.delay(:priority => 3, run_at => 1.hour.from_now). send_alerts(event)

Running workers

Foreground$ rake jobs:work

$ script/delayed_job run

Daemonised$ script/delayed_job start

$ RAILS_ENV=production script/delayed_job -n 4 start

Custom Jobs

class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end endend

emails = Customers.all.collect(&:email)Delayed::Job.enqueue NewsletterJob.new(text, emails)

Hooksdef enqueue(job)end

def before(job)end

def after(job)end

def success(job)end

def error(job, exception)end

def failureend

Resque

Resque

• Created by GitHub

• Jobs persisted to Redis as JSON

• Multiple queues

• Parent/child forking architecture

• Web monitoring interface

SetupRedis:

$ brew install redis

Gemfile:

gem "resque"

Rakefile:

require "resque/tasks"task "resque:setup" => :environment

Delaying methods

class Archive @queue = :file_serve

def self.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) endend

Resque.enqueue(Archive, repository.id, branch)

Running queues

$ QUEUE=queue_name rake resque:work

$ QUEUES=file_serve,warm_cache rake resque:work

$ QUEUES=critical,high,low rake resque:work

$ QUEUE=* rake resque:work

$ COUNT=5 QUEUE=* rake resque:workers

Monitoring$ resque-web

Which to use?

Delayed::Job Resque

How much of your site is background work?

Less than 50% More than 50%

Demo