Llamarada

Tomar de la élite lo que pertenece a todos

Background Jobs With Redis and Resque

| Comments

Today we will talk about various technologies that can be used to schedule background jobs using a Redis datastore and Resque to manage them, we will assume that you have already installed redis in your machine.

Installing resque

Install the resque gem, this will also pull the redis client library if you didnt had it oun your system

install resque gem
1
$   gem install resque

Running redis

starting redis server
1
$   redis-server

Start the resque web interfase

starting redis http interfase
1
$   resque-web

Integrating Redis and Resque with Rails

Lets create a redis.yml configuration file

config/redis.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
defaults: &defaults
host: localhost
port: 6379

development:
<<: *defaults

test:
<<: *defaults

staging:
<<: *defaults

production:
<<: *defaults

Now lets add the Resque initializer

config/initializers/resque.rb syntax
1
2
3
4
Dir[File.join(Rails.root, 'app', 'jobs', '*.rb')].each { |file| require file }

config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
Resque.redis = Redis.new(:host => config['host'], :port => config['port'])

and finally we need to create a resque rake task

lib/tasks/resque.rake
1
2
require 'resque/tasks'
task "resque:setup" => :environment

Using Resque

Create a WorkerClass

WorkerClass will be the class that will be called by Resque to dequeue and execute the job with the given parameters it needs to be told which queue within Resque belongs to, and it must implement a perform method, which will be executed by Redis when a new job is retrieved

WorkerClass
1
2
3
4
5
6
7
8
class WorkerClass
  @queue = "demo"

  def self.perform(args)
    sleep 3
    puts "Doing something complex with #{args}"
  end
end

Launch Resque worker

Finally lets tell our worker to be ready to accept new jobs

starting our worker
1
rake resque:work QUEUE=*

Put a new job in the queue

enqueuing a job
1
2
3
require "resque"

Resque.enqueue(WorkerClass, args)

Jobs scheduled at future dates

Resque can create both delayed jobs and recurring ‘cron style’ jobs

Creating recurring jobs

dynamically creating recurring jobs
1
2
3
4
5
Resque.set_schedule("create_fake_leaderboards", {
  :cron => "30 6 * * 1",
  :class => "CreateFakeLeaderboards",
  :queue => scoring
})

Create a job to be executed at exactly one set date

Queueing jobs for a future date
1
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)

this will store the job for 5 days in the resque delayed queue at which time the scheduler process will pull it from the delayed queue and put it in the appropriate work queue for the given job and it will be processed as soon as a worker is available.

Canceling a scheduled delayed job

1
2
3
4
# after you've enqueued a job like:
Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
# remove the job with exactly the same parameters:
Resque.remove_delayed(SendFollowUpEmail, :user_id => current_user.id)

Resque-scheduler is another option to queue items to be executed in the future

Resque-scheduler is an extension to Resque that adds support for queueing items in the future.

Installing resque-scheduler

1
gem install resque-scheduler

Recurring (or scheduled) jobs are logically no different than a standard cron job

Scheduling a job

Jobs are stored in the resque_schedule.yml file

resque_schedule.yml
1
2
3
4
5
queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  args:
  description: "This job queues all content for indexing in solr"

Load the scheduler and scheduled jobs from your application

loading resque scheduled jobs
1
2
require 'resque_scheduler'
Resque.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))

Run the scheduler process

1
rake resque:scheduler

Comments