Product: Gearman - Open Source Message Queuing System
Update: New Gearman Server & Library in C, MySQL UDFs.
Gearman is an open source message queuing system that makes it easy to do distributed job processing using multiple languages. With Gearman you: farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, to call functions between languages, spread CPU usage around your network.
Gearman is used by companies like LiveJournal, Yahoo!, and Digg. Digg, for example, runs 300,000 jobs a day through Gearman without any issues. Most large sites use something similar. Why would anyone ever even need a message queuing system?
Message queuing is a handy way to move work off your web servers (like image manipulation), to generate thousands of documents in the background, to run the multiple requests in parallel needed to build a web page, or to perform tasks that can comfortably be run in the background and not part of the main request loop for servicing a web request.
There's a gearmand server and clients written in Perl, Ruby, Python or C. Use at least two gearmand server daemons for higher availability. The tasks each client can perform are registered with gearman distributes requests for those functions to the client that can implement them.
Gearman uses a very robust, if somewhat higher latency, signal-and-pull architecture.
* worker connects to all gearmand servers.
* worker registers what functions it supports.
* worker asks for jobs.
* if no jobs, sends command 'pre_sleep' to all gearmand's and sleeps.
* Connect to gearmand.
* submit's a job for a particular func.
* Acks the job, finds all *sleeping workers* related to the function.
* Sends them all a 'noop' command to wake them up.
* Urk, I'm awake now.
* Worker asks for jobs.
* If jobs, do work.
* If no jobs, sends command 'pre_sleep' to all gearmand's, etc.
Gearman uses an efficient binary protocol and no XML. There's an a line-based text protocol for admin so you can use telnet and hook into Nagios plugins.
The system makes no guarantees. If there's a failure the client is told about the failure and the client is responsible for retries. And the queue isn’t persistent. If gearman is restarted the queue is gone.
