job queue

Job queue and search engine

Thu, 05/29/2008 - 09:15 — mat

Hi,

I want to implement a search engine with lucene.
To be scalable, I would like to execute search jobs asynchronously (with a job queuing system).

But i don't know if it is a good design... Why ?

Search results can be large ! (eg: 100+ pages with 25 documents per page)
With asynchronous sytem, I need to store results for each search job.
I can set a short expiration time (~5 min) for each search result, but it's still large.

What do you think about it ?
Which design would you use for that ?

Thanks
Mat

3 comments
1242 reads

Strategy: Asynchronous Queued Virus Scanning

Thu, 01/17/2008 - 15:01 — Todd Hoff

Atif Ghaffar has a nice strategy to deal with virus checking uploads:

Upload item into a safe area. If necessary, the uploader blocks waiting for a result.

Queue a work order into a job system so all the work can be distributed throughout your cluster.

A service in your cluster performs the virus scan and informs the uploader of the result.

Move the vetted item into your system.

This removes the CPU bottleneck from your web servers and distributes it through your cluster. Keep your web servers providing prompt service to users. Let your cluster do the heavy lifting. This minimizes response time and maximizes throughput. A similar system can be used for creating thumbnails, transcoding, copyright checks, updating indexes, event notification or any other kind of intensive work.

Bookmark/Search this post with: