Background jobs with Redis and Resque

December 12, 2010

Today we will talk about various technologies that can be used to schedule background jobs using a Redis datastore and Resque to manage them, we will assume that you have already installed redis in your machine.

Installing resque

Install the resque gem, this will also pull the redis client library if you didnt had it oun your system

$   gem install resque

Running redis

$   redis-server

Start the resque web interfase

$   resque-web

Integrating Redis and Resque with Rails

Lets create a redis.yml configuration file

defaults: &defaults
host: localhost
port: 6379

development:
<<: *defaults

test:
<<: *defaults

staging:
<<: *defaults

production:
<<: *defaults

Now lets add the Resque initializer

Dir[File.join(Rails.root, 'app', 'jobs', '*.rb')].each { |file| require file }

config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
Resque.redis = Redis.new(:host => config['host'], :port => config['port'])

and finally we need to create a resque rake task

require 'resque/tasks'
task "resque:setup" => :environment

Using Resque

Create a WorkerClass

WorkerClass will be the class that will be called by Resque to dequeue and execute the job with the given parameters it needs to be told which queue within Resque belongs to, and it must implement a perform method, which will be executed by Redis when a new job is retrieved

class WorkerClass
  @queue = "demo"

  def self.perform(args)
    sleep 3 
    puts "Doing something complex with #{args}"
  end
end

Launch Resque worker

Finally lets tell our worker to be ready to accept new jobs

rake resque:work QUEUE=*

Put a new job in the queue

require "resque"

Resque.enqueue(WorkerClass, args)

Jobs scheduled at future dates

Resque can create both delayed jobs and recurring ‘cron style’ jobs

Creating recurring jobs

Resque.set_schedule("create_fake_leaderboards", {
  :cron => "30 6 * * 1",
  :class => "CreateFakeLeaderboards",
  :queue => scoring
})

Create a job to be executed at exactly one set date

Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)

this will store the job for 5 days in the resque delayed queue at which time the scheduler process will pull it from the delayed queue and put it in the appropriate work queue for the given job and it will be processed as soon as a worker is available.

Canceling a scheduled delayed job

# after you've enqueued a job like:
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
# remove the job with exactly the same parameters:
Resque.remove_delayed(SendFollowUpEmail, :user_id => current_user.id)

Resque-scheduler is another option to queue items to be executed in the future

Resque-scheduler is an extension to Resque that adds support for queueing items in the future.

####Installing resque-scheduler

gem install resque-scheduler

Recurring (or scheduled) jobs are logically no different than a standard cron job

####Scheduling a job Jobs are stored in the resque_schedule.yml file

queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  args: 
  description: "This job queues all content for indexing in solr"

####Load the scheduler and scheduled jobs from your application

require 'resque_scheduler'
Resque.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))

####Run the scheduler process

rake resque:scheduler

Comments

comments powered by Disqus