High Load on production Webservers after Sourcecode sync

Hi everybody :)

We have a bunch of webservers (about 14 at this time) running Apache. As application framework we're using PHP with the APC Cache installed to improve performance. For load balancing we're using a Big F5 system with dynamic ratio (SNMP driven)

To sync new/updated sourcecode we're using subversion to "automaticly" update these servers with our latest software relases. After updating the new source to these production servers the load of the mashines is raising to hell.

While updating the servers, they are still in "production", serving webpages to the users. Otherwise the process of updating would take ages. Most of the time we're only updating in the morning hours while less users are online, because of the above issue.

My guess is, that the load is raising that high, because APC needs to recompile a bunch of new files each time. Before and while compiling the performance simply is "bad".

My goal would be to find a better solution. We want to "sync" code no matter how many users are online (in case of emergency) without taking the whole site down.

How you're handling this issues ? What do you think about the process above ? Do you may find the "problem" ? Do you have similiar issues ?

Feedback is highly welcome :)

Greetings,

Stephan Tijink
Head of Web Development

|  fotocommunity GmbH & Co. KG
|  Rheinwerkallee 2
|  53227 Bonn