« Moving old to new. Do not be afraid of the re-write -- but take some help | Main | Does Sun Buying MySQL Change Your Scaling Strategy? »

Strategy: Asynchronous Queued Virus Scanning

Atif Ghaffar has a nice strategy to deal with virus checking uploads:

  • Upload item into a safe area. If necessary, the uploader blocks waiting for a result.
  • Queue a work order into a job system so all the work can be distributed throughout your cluster.
  • A service in your cluster performs the virus scan and informs the uploader of the result.
  • Move the vetted item into your system.

    This removes the CPU bottleneck from your web servers and distributes it through your cluster. Keep your web servers providing prompt service to users. Let your cluster do the heavy lifting. This minimizes response time and maximizes throughput. A similar system can be used for creating thumbnails, transcoding, copyright checks, updating indexes, event notification or any other kind of intensive work.
  • Reader Comments (1)

    Thanks for the heads-up Todd,

    One more comment,

    The same can be applied for your file uploads.
    Dont set your frontend webservers to wait for a max_execution_time of 60 seconds etc.
    Anything that does not finish in 1 second is a bug.

    Set up a different server that handles only uploads and knows nothing else.
    The upload server can respond back with a temp URL that can be used to move/copy the uploaded file to the permanent area.

    This can be used for image/whatever.

    best regards

    November 29, 1990 | Unregistered Commenteratif.ghaffar

    PostPost a New Comment

    Enter your information below to add a new comment.
    Author Email (optional):
    Author URL (optional):
    Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>