Home

Strategy: Asynchronous Queued Virus Scanning

Todd Hoff's picture
Thu, 01/17/2008 - 15:01 — Todd Hoff

Atif Ghaffar has a nice strategy to deal with virus checking uploads:

  • Upload item into a safe area. If necessary, the uploader blocks waiting for a result.
  • Queue a work order into a job system so all the work can be distributed throughout your cluster.
  • A service in your cluster performs the virus scan and informs the uploader of the result.
  • Move the vetted item into your system.

    This removes the CPU bottleneck from your web servers and distributes it through your cluster. Keep your web servers providing prompt service to users. Let your cluster do the heavy lifting. This minimizes response time and maximizes throughput. A similar system can be used for creating thumbnails, transcoding, copyright checks, updating indexes, event notification or any other kind of intensive work.

    • Comments

    Thu, 01/17/2008 - 20:39 — atif.ghaffar
    atif.ghaffar's picture

    Re: Strategy: Asynchronous Queued Virus Scanning

    Thanks for the heads-up Todd,

    One more comment,

    The same can be applied for your file uploads.
    Dont set your frontend webservers to wait for a max_execution_time of 60 seconds etc.
    Anything that does not finish in 1 second is a bug.

    Set up a different server that handles only uploads and knows nothing else.
    The upload server can respond back with a temp URL that can be used to move/copy the uploaded file to the permanent area.

    This can be used for image/whatever.

    best regards

    Comment viewing options

    Select your preferred way to display the comments and click "Save settings" to activate your changes.

    Post new comment

    The content of this field is kept private and will not be shown publicly.
    • Web page addresses and e-mail addresses turn into links automatically.
    • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd><div ?=?><p ?=?> <img ?=?><h1 ?=?><h2 ?=?><h3 ?=?>
    • Lines and paragraphs break automatically.
    • Glossary terms will be automatically marked with links to their descriptions
    • You may link to webpages through the weblinks registry

    More information about formatting options

    To combat spam, please enter the code in the image.