Background job processing in Rails with delayed_job

Running jobs in background is part of every non-trivial web application – for either speeding up user response, or running periodically scheduled tasks. Here, I explain how to install, configure and use the elegant delayed_job plugin to process background jobs.

Why background processing is needed

Since Ruby 1.8 only supports green threads, if you want to process something longish as part of a web request in Rails, it will delay the response. Of course, if the processing is not critical to returning a response to the client, it’s wiser to push the processing to a background queue and return a response immediately.

There are other incidental benefits to using a background job processing queue like, letting a dedicated server, other than the customer facing app server, handle it, assigning priority to jobs, retry on failure etc.

Most common situations where such a background processing is needed are: sending emails to users, rebuilding search indices, image and video processing, etc.

Available options

There are several ways to process background jobs in Rails. In fact, So many that Geoffrey Grosenbach joked on the RailsEnvy podcast about the number of continuous integration servers finally being equal to number of queuing servers available in Ruby.

My favorite one so far is: delayed_job. Because its simple to set up, simple to use and has lot of powerful features. However, don’t forget to look at others to see what fits you needs and style better. For an overview see here. In the rest of the article, I will explain how to install and use delayed_job.

Installation

Firstly, install it in your rails project just as you would any other plugin:

$ ruby script/plugin install git://github.com/tobi/delayed_job.git

Delayed_job stores the jobs in a table in the database. Create the table using a migration:

$ ruby script/generate migration create_table_for_delayed_job

class CreateTableForDelayedJob < ActiveRecord::Migration
  def self.up
    create_table :delayed_jobs, :force => true do |t|
      t.integer  :priority, :default => 0
      t.integer  :attempts, :default => 0
      t.text     :handler
      t.text     :last_error
      t.datetime :run_at
      t.datetime :locked_at
      t.datetime :failed_at
      t.string   :locked_by
      t.timestamps
    end
  end

  def self.down
    drop_table :delayed_jobs
  end
end

$ rake db:migrate

Note: Compared to the documentation, I have changed last_error column to text from string to handle larger error messages

Delay jobs

Restart your server, and you can start delaying jobs. There are two ways to push a job to background. First, the easy way:

You have seen Ruby’s send method, right.

"Bangalore".length # returns 9
"Bangalore".send(:length) # same as above, returns 9

send just invokes the given method on the object.

Analogous to that, delayed_job adds send_later method. Say, if you were sending a mail to user on registration:

NotificationMailer.deliver_welcome_user(@user)

Just change that to:

NotificationMailer.send_later(:deliver_welcome_user, @user)

And you are done. delayed_job will push this into the queue (the database table you created above), to be executed later.

There is another way to background a job. First, create a job class.

# put this in lib/delayed_notification.rb
class DelayedNotification
  attr_accessor :user_id
  def initialize(user_id)
    self.user_id = user_id
  end

  def perform
   NotificationMailer.deliver_welcome_user(User.find(@user_id))
  end
end

# Add this where you were sending the mail earlier
Delayed::Job.enqueue DelayedNotification.new(@user.id)

Executing the jobs

Now, there are several ways to run the job which are there in the queue. In development mode, you can just issue the following command in your terminal:

$ rake jobs:work

This will cause the worker to run in a loop. Stop it by pressing Control-C.

On production, you would want the worker to be running all the time. Also, it will be nice to have the ability to stop the worker just before the deployment and start it again once the deployment finishes. It sounds complicated, but its pretty easy to set up.

First, install the daemons gem:

# Add in environment.rb
 config.gem 'daemons'

$ rake gems:install

Then, copy this script in your rails project in file script/delayed_job and give it execute permission. This creates a worker daemon, which when started keeps running as a background process.

Alternately, instead of daemons gem, you can use daemon-spawn gem.

Install the gem on the host where you want the delayed_job daemon to run:

$ sudo gem sources -a http://gems.github.com
$ sudo gem install alexvollmer-daemon-spawn

And then copy this script instead in script/delayed_job. Rest all of the following steps are identical, whichever gem and script/delayed_job you choose.

# start the worker daemon
$ ruby script/delayed_job start

# stop it
$ ruby script/delayed_job stop

You would of course want to start and stop it on your production server using Capistrano. Here’s a recipe to do that (courtesy collectiveidea)

# add this to config/deploy.rb
namespace :delayed_job do
  desc "Start delayed_job process" 
  task :start, :roles => :app do
    run "cd #{current_path}; script/delayed_job start #{rails_env}" 
  end

  desc "Stop delayed_job process" 
  task :stop, :roles => :app do
    run "cd #{current_path}; script/delayed_job stop #{rails_env}" 
  end

  desc "Restart delayed_job process" 
  task :restart, :roles => :app do
    run "cd #{current_path}; script/delayed_job restart #{rails_env}" 
  end
end

after "deploy:start", "delayed_job:start" 
after "deploy:stop", "delayed_job:stop" 
after "deploy:restart", "delayed_job:restart" 

And you are done. Just deploy as usual.

$ cap deploy:migrations

Further tips and tricks

• Get a separate server to process your delayed job so as not to take CPU away from customer facing app server.

• You can customize some parameters like how many times to retry on failure and prune runaway tasks. See here.

• Delayed_job allows you to assign priority to jobs.

• In development mode, if you just want to discard pending jobs instead of running them:

$ rake jobs:clear

• If you want to control the time at which the job gets executed, you can specify it as an argument to enqueue while delaying the job:

Delayed::Job.enqueue DelayedNotification.new(@user.id), 0, 15.minutes.from_now

Here, 0 denotes the (default) priority of the task. This task will get executed at least 15 minutes later than it would have been executed normally.

References

For more about this wonderful plugin, refer to these resources:

• A nice tutorial covering the basics.
• Home of delayed_job (also look at the wiki).