High Scalability Building bigger, faster, more reliable websites.

Update 5: In Load Balancer Update Barry describes how WordPress.com moved from Pound to Nginx and are now "regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!".
Update 4: Nginx better than Pound for load balancing. Pound spikes at 80% CPU, Nginx uses 3% and is easier to understand and better documented.
Update 3: igvita.com combines two cool tools together for better performance in Nginx and Memcached, a 400% boost!.
Update 2: Software Project on Installing Nginx Web Server w/ PHP and SSL. Breaking away from mother Apache can be a scary proposition and this kind of getting started article really helps easy the separation.
Update: Slicehost has some nice tutorials on setting up Nginx.

From their website:
Nginx ("engine x") is a high-performance HTTP server and reverse proxy, as well as an IMAP/POP3/SMTP proxy server. Nginx was written by Igor Sysoev for Rambler.ru, Russia's second-most visited website, where it has been running in production for over two and a half years. Igor has released the source code under a BSD-like license. Although still in beta, Nginx is known for its stability, rich feature set, simple configuration, and low resource consumption.

lighttpd (pronounced "lighty") is a web server which is designed to be secure, fast, standards-compliant, and flexible while being optimized for speed-critical environments. Its low memory footprint (compared to other web servers), light CPU load and its speed goals make lighttpd suitable for servers that are suffering load problems, or for serving static media separately from dynamic content. lighttpd is free software / open source, and is distributed under the BSD license. lighttpd runs on GNU/Linux and other Unix-like operating systems and Microsoft Windows.

* Load-balancing FastCGI, SCGI and HTTP-proxy support
* chroot support
* select()-/poll()-based web server
* Support for more efficient event notification schemes like kqueue and epoll
* Conditional rewrites (mod_rewrite)
* SSL and TLS support, via openSSL.
* Authentication against an LDAP server
* rrdtool statistics
* Rule-based downloading with possibility of a script handling only authentication
* Server-side includes support
* Flexible virtual hosting
* Modules support
* Cache Meta Language (currently being replaced by mod_magnet)
* Minimal WebDAV support
* Servlet (AJP) support (in versions 1.5.x and up)
* HTTP compression using mod_compress and the newer mod_deflate ( 1.5.x )

Information Sources

* http://en.wikipedia.org/wiki/Lighttpd
* http://highscalability.com/paper-lightweight-web-servers

This paper is a great overview of different lightweight web servers. A lot of websites use lightweight web servers to serve images and static content. YouTube is one example: http://highscalability.com/youtube-architecture.

So if you need to improve performance consider changing over a different web server for some types of content.