Product: Resque - GitHub's Distrubuted Job Queue

Queuing work for processing in the background is a time tested scalability strategy. Queuing also happens to be one of those much needed tools where it easy enough to forge for your own that we see a lot of different versions made. Resque is GitHub's take on a job queue and they've used it to process million and millions of jobs so far.

What is Resque?

Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

GitHub tried and considered many other systems: SQS, Starling, ActiveMessaging, BackgroundJob, DelayedJob, beanstalkd, AMQP,  and Kestrel, but found them all wanting in one way are another. The latency for SQS was too high. Others didn't make full use of Ruby. Others still had a lot of overhead. Some didn't have enough features. And still others weren't reliable enough.

You may think GitHub is being too picky, but for GitHub background work is a core competency. Fifty percent of their work is background processing so they need a serious, robust, monitorable, performant queue system that meets their needs. These are exactly the circumstances when building your own makes perfect sense.

Resque is very full featured, especially for high volume sites, supporting: Persistence, See what's pending, Modify pending jobs in-place, Tags, Priorities, Fast pushing and popping, See what workers are doing, See what workers have done, See failed jobs, Kill fat workers, Kill stale workers, Kill workers that are running too long, Keep Rails loaded / persistent workers, Distributed workers (run them on multiple machines), Workers can watch multiple (or all) tags, Don't retry failed jobs, Don't "release" failed jobs.

One innovation is to use Redis, a newgen key-value store instead of MySQL as the persistent store. A key element is the Sinatra-based based front-end for managing and monitoring the system. Often manageability is what separates simple libraries and servers from a complete, reliable system and Resque has taken that next step of professionalism.

Of course the downside of Resque is that it is Ruby centric and it's yet another queue system, but it definitely looks to be worth a look if you are in the market for a solid production quality queue system.