advertise
« GIS Application Hosting | Main | Numbers Everyone Should Know »
Thursday
Feb192009

Heavy upload server scalability

Hi,

We are running a backup solution that uploads every night the files our clients worked on during the day (Cabonite-like).
We have currently about 10GB of data per night, via http PUT requests (1 per file), and the files are written as-is on a NAS.
Our architecture is basically compound of a load balancer (hardware, sticky sessions), 5 servers (Tomcat under RHEL4/5, ) and a NAS (nfs 3).

Since our number of clients is rising, (as is our system load) how would you recommend we could scale our infrastructure? hardware and software? Should we go towards NAS sharding, more servers, NIO on tomcat...?


Thanks for your inputs!

Reader Comments (2)

Usually people with a lot of files to store use a distributed file system like MogileFS on the backend to handle the write load. Then you need as much bandwidth and as many servers as you want to handle simultaneous users. Messing with Tomcat probably won't buy you much, but that's when benchmarking is for.

Another option would be to load into S3 or some other cloud service. Then you can drop the computers and NAS and let them worry about scaling, backup, etc.

November 29, 1990 | Unregistered CommenterTodd Hoff

Bob,
How are the files transfered to you system? From what I understood, its via http-PUT.
Why not use ftp to transfer the files and then store then in your NAS via batch?

November 29, 1990 | Unregistered Commenteratif.ghaffar

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>