advertise
« 10 NoSQL Systems Reviewed | Main | A Yes for a NoSQL Taxonomy »
Friday
Nov062009

Product: Resque - GitHub's Distrubuted Job Queue

Queuing work for processing in the background is a time tested scalability strategy. Queuing also happens to be one of those much needed tools where it easy enough to forge for your own that we see a lot of different versions made. Resque is GitHub's take on a job queue and they've used it to process million and millions of jobs so far.

What is Resque?

Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

GitHub tried and considered many other systems: SQS, Starling, ActiveMessaging, BackgroundJob, DelayedJob, beanstalkd, AMQP,  and Kestrel, but found them all wanting in one way are another. The latency for SQS was too high. Others didn't make full use of Ruby. Others still had a lot of overhead. Some didn't have enough features. And still others weren't reliable enough.

You may think GitHub is being too picky, but for GitHub background work is a core competency. Fifty percent of their work is background processing so they need a serious, robust, monitorable, performant queue system that meets their needs. These are exactly the circumstances when building your own makes perfect sense.

Resque is very full featured, especially for high volume sites, supporting: Persistence, See what's pending, Modify pending jobs in-place, Tags, Priorities, Fast pushing and popping, See what workers are doing, See what workers have done, See failed jobs, Kill fat workers, Kill stale workers, Kill workers that are running too long, Keep Rails loaded / persistent workers, Distributed workers (run them on multiple machines), Workers can watch multiple (or all) tags, Don't retry failed jobs, Don't "release" failed jobs.

One innovation is to use Redis, a newgen key-value store instead of MySQL as the persistent store. A key element is the Sinatra-based based front-end for managing and monitoring the system. Often manageability is what separates simple libraries and servers from a complete, reliable system and Resque has taken that next step of professionalism.

Of course the downside of Resque is that it is Ruby centric and it's yet another queue system, but it definitely looks to be worth a look if you are in the market for a solid production quality queue system.

References (2)

References allow you to track sources for this article, as well as articles that were written in response to this article.

Reader Comments (2)

The article says " Fifty percent of their work is background processing so they need a serious, robust, monitorable, performant queue system that meets their needs." and then it keeps describing a dozen of features typically found in job-schedulers.

So yes, some queues like MQSeries are often used as one-size-fits-all solution, but most of the others have evolved into messaging products and not work managing. If one needs to manage jobs across a pool of workers, then take a look at schedulers like Quartz and Flux or grid solutions like Platform and DataSynapse. Some of them are even free (as in no-money).

November 9, 2009 | Unregistered CommenterDimitar Dimitrov

I wonder why no one mentioned gearman.

September 15, 2010 | Unregistered CommenterTill

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>