This is what Mike Peters says he can do: make your site run 10 times faster. His test bed is "half a dozen servers parsing 200,000 pages per hour over 40 IP addresses, 24 hours a day." Before optimization CPU spiked to 90% with 50 concurrent connections. After optimization each machine "was effectively handling 500 concurrent connections per second with CPU at 8% and no degradation in performance."
Mike identifies six major bottlenecks:
Database write access (read is cheaper)
Database read access
PHP, ASP, JSP and any other server side scripting
Client side JavaScript
Multiple/Fat Images, scripts or css files from different domains on your page
Slow keep-alive client connections, clogging your available sockets
Mike's solutions:
Switch all database writes to offline processing
Minimize number of database read access to the bare minimum. No more than two queries per page.
Denormalize your database and Optimize MySQL tables
Implement MemCached and change your database-access layer to fetch information from the in-memory database first.
Store all sessions in memory.
If your system has high reads, keep MySQL tables as MyISAM. If your system has high writes, switch MySQL tables to InnoDB.
Limit server side processing to the minimum.
Precompile all php scripts using eAccelerator
If you're using WordPress, implement WP-Cache
Reduce size of all images by using an image optimizer
Merge multiple css/js files into one, Minify your .js scripts
Avoid hardlinking to images or scripts residing on other domains.
Put .css references at the top of your page, .js scripts at the bottom.
Install FireFox FireBug and YSlow. YSlow analyze your web pages on the fly, giving you a performance grade and recommending the changes you need to make.
Optimize httpd.conf to kill connections after 5 seconds of inactivity, turn gzip compression on.
Configure Apache to add Expire and ETag headers, allowing client web browsers to cache images, .css and .js files
Consider dumping Apache and replacing it with Lighttpd or Nginx.
Find more details in Mike's article.